Information Technology: DOD's Acquisition Policies and Guidance  
Need to Incorporate Additional Best Practices and Controls	 
(30-JUL-04, GAO-04-722).					 
                                                                 
The way in which the Department of Defense (DOD) has historically
acquired its business systems has been cited as a root cause for 
its limited success in delivering promised system capabilities	 
and benefits on time and within budget. In response, DOD recently
revised its systems acquisition policies and guidance to	 
incorporate best practices, including those pertaining to	 
business systems. GAO was asked to determine whether DOD's	 
revised systems acquisition policies and guidance (1) are	 
consistent with industry best practices, including those	 
pertaining to commercial component-based systems, and (2) provide
the necessary controls to ensure that DOD component organizations
adhere to the practices.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-04-722 					        
    ACCNO:   A11167						        
  TITLE:     Information Technology: DOD's Acquisition Policies and   
Guidance Need to Incorporate Additional Best Practices and	 
Controls							 
     DATE:   07/30/2004 
  SUBJECT:   ADP procurement					 
	     Best practices					 
	     Defense procurement				 
	     Information technology				 
	     Procurement policy 				 
	     Procurement practices				 
	     Systems analysis					 
	     Best practices methodology 			 
	     Policy evaluation					 
	     Internal controls					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-04-722

                 United States Government Accountability Office

                     GAO Report to Congressional Requesters

July 2004

                                  INFORMATION
                                   TECHNOLOGY

  DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best
                             Practices and Controls

                                       a

GAO-04-722

Highlights of GAO-04-722, a report to congressional requesters

The way in which the Department of Defense (DOD) has historically acquired
its business systems has been cited as a root cause for its limited
success in delivering promised system capabilities and benefits on time
and within budget. In response, DOD recently revised its systems
acquisition policies and guidance to incorporate best practices, including
those pertaining to business systems.

GAO was asked to determine whether DOD's revised systems acquisition
policies and guidance (1) are consistent with industry best practices,
including those pertaining to commercial component-based systems, and (2)
provide the necessary controls to ensure that DOD component organizations
adhere to the practices.

To improve DOD's ability to acquire IT business systems, GAO recommends
that the Secretary of Defense incorporate additional best practices in
DOD's acquisition policies and guidance, and that the department
strengthen controls for ensuring that best practices are appropriately
followed.

In commenting on a draft of this report, DOD agreed or partially agreed
with our recommendations for incorporating additional best practices, but
did not agree that it needed (1) a plan to govern its incorporation of
these practices or (2) stronger controls for ensuring that best practices
are followed.

www.gao.gov/cgi-bin/getrpt?GAO-04-722.

To view the full product, including the scope and methodology, click on
the link above. For more information, contact Randolph C. Hite at (202)
512-3439 or [email protected].

July 2004

INFORMATION TECHNOLOGY

DOD's Acquisition Policies and Guidance Need to Incorporate Additional Best
Practices and Controls

DOD's revised policies and guidance largely incorporate 10 best practices
for acquiring any type of information technology (IT) business system. For
example, the revisions include the requirement that acquisitions be
economically justified on the basis of costs, benefits, and risks.
However, the revisions generally do not incorporate 8 best practices
relating to the acquisition of commercial component-based systems. For
example, they do not address basing any decision to modify commercial
components on a thorough analysis of the impact of doing so or on
preparing system users for the business process and job roles and
responsibilities changes that are embedded in the functionality of
commercial IT products. In total, GAO found that DOD's acquisition
policies and guidance fully incorporate 8 of the 18 best practices that
they were evaluated against, partially incorporate 5 practices, and do not
incorporate the remaining 5 practices (see figure). DOD intends to expand
its acquisition guidance to incorporate additional best practices by
September 30, 2004, but department officials cite other priorities as a
reason why they have not been able to complete this effort and could not
provide a plan specifying how this will be accomplished. Until DOD's
revised policies and guidance incorporate key systems acquisition best
practices, the risk that system investments will not consistently deliver
promised capabilities and benefits on time and within budget is increased.

DOD's revised policies also do not contain sufficient controls to ensure
that DOD components appropriately follow the best practices that are
incorporated in its policies and guidance. According to acquisition best
practices experts, as well as GAO's internal control guidance, controls
are effective if they are backed by measurements that are verified.
Although the revised policies and guidance require acquisition managers to
examine and, as appropriate, adopt best practices, they do not cite what
that examination entails. DOD believes existing controls are sufficient,
even though these controls do not provide for measuring and validating the
practices' use. Without specific requirements to measure and validate the
use of best practices, the risk that they will not be followed increases,
which, in turn, increases the risk that system investments will not meet
expectations.

DOD Incorporation of Best Practices

Relevant to any IT business systems Relevant to commercial component-based

acquisition IT business systems acquisitions

Number of best practices Number of best practices

10 10

99

88

77

66

55

44

33

22

11

00Fully Partially Not Fully Partially Not

 incorporated incorporated incorporated incorporated incorporated incorporated

Source: GAO, based on analysis of DOD data.

Contents

  Letter

Results in Brief
Background
DOD's Acquisition Policy and Guidance Are Consistent with Some,

but Not All, Key Acquisition Best Practices

DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure
That the Requirement Is Met for Appropriately Applying Best Practices

Conclusions
Recommendations for Executive Action
Agency Comments and Our Evaluation

1 1 3

10

22 23 23 25

Appendixes

Appendix I: Appendix II:

Appendix III: Appendix IV:

Objectives, Scope, and Methodology Best Practices

Best Practices Relevant to Any IT Business Systems Acquisition
Complementary Best Practices Relevant to Commercial Component-Based IT
Business Systems Acquisitions

Comments from the Department of Defense

GAO Comments

GAO Contact and Staff Acknowledgments

GAO contact
Staff acknowledgments

                                       28

                                     31 31

                                       37

                                     42 47

50 50 50

Tables	Table 1: Table 2: Table 3:

Table 4:

Organizational Responsibilities for the DOD 5000 Series
Documents
Summary of Business Systems Acquisition Best
Practices
Activity-by-activity Comparison of the 5000 Series to Best
Practices Relevant to Any IT Business Systems
Acquisition
Activity-by-Activity Comparison of the 5000 Series to Best
Practices Relevant to Commercial Component-based
Business Systems Acquisitions

6 11

17

20

Figures Figure 1: Revisions to the DOD 5000 Series Documents 7 Figure 2:
Simplified Diagram of DOD's Acquisition Management Framework 9

Contents

Figure 3:	DOD Acquisition Policies and Guidance Incorporation of Best
Practices 15

Abbreviations

AT&L Acquisition, Technology, and Logistics
DOD Department of Defense
IT information technology
NII Networks and Information Integration
OT&E Operational Test and Evaluation

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

A

United States Government Accountability Office Washington, D.C. 20548

July 30, 2004

The Honorable John Ensign
Chairman
The Honorable Daniel K. Akaka
Ranking Minority Member
Subcommittee on Readiness and Management Support
Committee on Armed Services
United States Senate

This report responds to your request that we assess the Department of
Defense's (DOD) recently revised policies and guidance for acquiring
business systems for incorporation of acquisition best practices. As you
know, the way in which DOD has historically acquired information
technology (IT) systems has been cited as a root cause of these systems
failing to deliver promised capabilities and benefits on time and within
budget. The use of these best practices-which includes those practices
pertaining to any business system, whether custom developed or based on
commercially available product components, as well as those unique to
commercial component-based systems-is intended to improve on this
performance.

As agreed with your offices, our objectives were to determine whether
DOD's recently revised policies and guidance for acquiring business
systems (1) are consistent with industry best practices, including those
pertaining to commercial component-based systems, and (2) provide the
necessary controls to ensure that DOD component organizations adhere to
best practices. We conducted our work between December 2003 and May
2004 in accordance with generally accepted government auditing
standards. Details of our objectives, scope, and methodology are included
in appendix I.

Results in Brief	DOD's revisions to its systems acquisition policies and
guidance incorporate many best practices for acquiring business systems.
For example, the revisions recognize such practices as (1) economically
justifying system investments on the basis of costs, benefits, and risks
and (2) continually measuring an acquisition's performance, cost, and
schedule against approved baselines. However, the revised policies and
guidance do not incorporate a number of other best practices, particularly
those associated with acquiring commercial component-based business
systems. For example, they do not address basing a decision to modify a
commercial

component on a thorough analysis of the impact of doing so; evaluating
contractors on their ability to implement commercial components and using
the results in source selection decisions; and preparing system users for
the business process and job roles and responsibilities changes that are
embedded in the functionality of commercial software products. According
to officials responsible for revising DOD acquisition policies and
guidance, additional best practices and lessons learned will be
incorporated into the acquisition guidance by September 30, 2004. However,
documented plans for this task do not exist, and the associated resources
needed to complete this task have not been assigned due to higher priority
needs. Until these missing best practices are included in DOD's
acquisition policies and guidance, the risk is increased that systems
acquisitions will not deliver planned capabilities and benefits on time
and within budget.

DOD's revised acquisition policies also do not contain sufficient controls
to ensure that military services and defense agencies appropriately follow
the practices. Controls are considered effective if they are measured and
verified. Although DOD's revised policies require both the project manager
and the investment decision authority to examine the adoption of best
practices, neither the policies nor the associated guidance provide for
measuring and verifying the use of the practices. As a result, DOD is
increasing the risk that best practice adoption and use will not occur,
which, in turn, increases the risk that systems acquisitions will not
deliver what is planned-on time and within budget.

This report makes 14 recommendations to the Secretary of Defense that are
aimed at strengthening DOD's acquisition policy and guidance by including
additional business systems acquisition best practices and controls for
ensuring that best practices are followed.

In its written comments (reprinted in app. III) on a draft of this report
signed by the Principal Director, Deputy Assistant Secretary of Defense
(Command, Control, Communications, Space and Information Technology
Programs), DOD agreed with the importance and relevance of the best
practices that we cite in the report. DOD also agreed or partially agreed
with most of our recommendations, stating that it would either incorporate
those practices that we reported as missing from the department's
acquisition policies and guidance, or consider augmenting its coverage of
those practices that deserve greater emphasis in its policies and
guidance. Further, while DOD acknowledged that incorporation of additional
best practices in its acquisition policies and guidance should be
undertaken or considered, it did not agree that it needed an explicit plan
to govern its

ongoing and future policy and guidance revision activities, stating that
our recommendation to this effect was inappropriate. We do not agree with
DOD that a plan governing incorporation of the practices is not needed.
Given the importance of DOD's acquisition policies and guidance as well as
best practices, we believe that having an explicit plan that defines how
and when incorporation of best practices will be added is essential. Among
other things, a plan would highlight the resource constraints that this
revision effort has been subject to, would allow measurement against
defined milestones, and would allow disclosure of progress and
impediments. In its comments, DOD also did not agree that stronger
controls are needed for ensuring adherence to the best practices contained
in its acquisition policies and guidance. We do not agree with the
department's position on this matter because its existing controls do not
provide for either the measurement or verification of whether the
practices are employed-both recognized elements of effective process
controls.

Background	Best practices are tried and proven methods, processes,
techniques, and activities that organizations define and use to minimize
risks and maximize chances for success. As we have previously reported,
using best practices can result in better outcomes-including cost savings;
improved service and product quality; and, ultimately, a better return on
investment. For example, two software engineering analyses of nearly 200
systems acquisitions projects indicated that teams using systems
acquisition best practices produced cost savings of at least 11 percent
over similar projects conducted by teams that did not employ the kind of
rigor and discipline embedded in these practices.1 In addition, our
research2 shows that best practices are a significant factor in successful
acquisition outcomes, including increasing the likelihood that programs
and projects will be executed within cost and schedule estimates.

DOD, GAO, and the Congress have all advocated the use of best practices.
For example, in September 2000, DOD established a steering group to

1Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, "Effects
of Process Maturity on Quality, Cycle Time, and Effort in Software Product
Development," Management Science, vol. 46, no. 4, 2000; and Bradford K.
Clark, "Quantifying the Effects of Process Improvement on Effort," IEEE
Software (November/December 2000).

2U.S. General Accounting Office, Defense Acquisitions: Stronger Management
Practices Are Needed to Improve DOD's Software-Intensive Weapon
Acquisitions, GAO-04-393 (Washington, D.C.: Mar. 1, 2004).

promote the use of systems acquisition best practices and lessons learned.
Further, our 2001 report3 cited the benefits of DOD adoption of best
practices and provided recommendations for establishing a mechanism for
sharing best practices throughout DOD. In the fiscal year 2003 Defense
Authorization Act,4 the Congress used our recommendations in directing DOD
to expand its use of best practices. Specifically, it required the Under
Secretary of Defense (Acquisition, Technology, and Logistics (AT&L)) and
the Assistant Secretary of Defense (Command, Control, Communications, and
Intelligence-now called Networks and Information Integration (NII))-to
identify and serve as a clearinghouse for information regarding software
acquisition best practices in the public and private sectors. In response,
DOD assigned AT&L responsibility for serving as that clearinghouse.
Further, the Defense Information Systems Agency created a Web site to
provide information about acquisition best practices.

DOD Relies Extensively on IT Systems to Perform a Variety of Business
Functions

DOD is one of the largest and most complex organizations in the world. In
fiscal year 2003, DOD reported that its operations involved over $1
trillion in assets, nearly $1.6 trillion in liabilities, disbursements of
more than $416 billion, and approximately 3.3 million military and
civilian personnel. Execution of these operations spans a wide range of
defense organizations, including the military services, defense agencies
and field activities, and various combatant and joint operation commands.
To execute these military operations, the department performs an
assortment of business functions, including logistics management,
procurement, healthcare management, and financial management.

To support its business functions, DOD reports that it currently relies on
about 2,300 IT systems, including accounting, acquisition, logistics, and
personnel systems. Moreover, its future investment in business systems is
expected to be sizable. For fiscal year 2004, DOD requested approximately
$28 billion in IT funding to support a wide range of military and business
operations. Approximately $9 billion of this amount is to support
primarily command and control systems, and the remaining $19 billion is to
support the operation, maintenance, and modernization of business systems.

3U.S. General Accounting Office, DOD Information Technology: Software and
Systems Process Improvement Programs Vary in Use of Best Practices,
GAO-01-116 (Washington, D.C.: Mar. 30, 2001).

4Bob Stump National Defense Authorization Act for 2003 (Pub.L. No.
107-314).

Overview of DOD's Acquisition Management Framework

Since the 1980s, DOD's oversight of its systems acquisitions had been
defined by a series of three documents-commonly called the 5000
series-that provided the policies and guidance for departmental efforts to
acquire service capabilities and systems:

o 	DOD Directive 5000.1, The Defense Acquisition System-describes the
management principles for DOD's acquisition programs.

o 	DOD Instruction 5000.2, Operation of the Defense Acquisition
System-outlines the framework for managing acquisition programs.

o 	DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition
Programs (MDAPS) and Major Automated Information System (MAIS) Acquisition
Programs-provides the mandatory procedures for acquiring major defense
programs.

These documents have been revised several times. Most recently, in October
2002, the Deputy Secretary of Defense determined that the existing
versions of these three documents required further revisions to improve
acquisition efficiency, flexibility, creativity, and innovation. As a
result, the Deputy Secretary canceled the existing versions of each
document and instructed the Under Secretary for AT&L the Assistant
Secretary for NII; and the Director, Operational Test and Evaluation
(OT&E) to jointly revise the documents. (Table 1 describes selected
responsibilities of these three entities.) The revised directive and
instruction were issued in May 2003. Both were shortened and modified to
focus on required outcomes and legal requirements and to eliminate the
"how-to" details in the previous versions. In doing so, DOD intended the
revisions to provide program managers with more flexibility in executing
their respective programs.

Table 1: Organizational Responsibilities for the DOD 5000 Series Documents
                          Organization Responsibility

Office of the Under Secretary of Advises the Secretary of Defense on all
matters Defense (AT&L)/Defense Acquisition pertaining to DOD's acquisition
framework as well Executive as research and development; advanced

technology; developmental test and evaluation; production; logistics;
installation management; military construction; procurement; environmental
security; and nuclear, chemical, and biological matters.

Office of the Assistant Secretary of Defense (NII)/DOD Chief Information
Officer Advises the Secretary of Defense on achieving and maintaining
information superiority through the collection, processing, and
dissemination of an uninterrupted flow of information in support of DOD
missions while exploiting or denying an adversary's ability to do the
same.

Serves as the principal assistant to the secretary for electronic
business, information management, information operations and assurance,
and IT.

Office of the Director (OT&E)	Advises the Secretary of Defense on OT&E.
Issues DOD OT&E policy and procedures, and reviews and analyzes OT&E
conducted for each major acquisition program and provides reports on
adequacy and results to the Secretary, the Under Secretary for AT&L, and
the Congress.

Source: GAO, based on DOD documentation.

DOD 5000.2-R was renamed the Interim Defense Acquisition Guidebook and was
made optional guidance on best practices and lessons learned. (See fig.
1.) According to DOD officials, improvements to the guidebook are
currently under way. Until they are completed, DOD 5000.2-R serves as the
guidebook.

Figure 1: Revisions to the DOD 5000 Series Documents

DOD acquisition policy as of

DOD acquisition policy/

October 2002

guidance as of May 2003

Source: GAO, based on DOD documentation.

According to the revised policy, an acquisition is a directed, funded
effort that provides a new, improved, or continuing materiel, weapon or
information system, or service capability. The revised 5000 series applies
to acquisitions conducted by all of the department's organizational
components.5 These components include the military services and the
defense agencies, such as the Defense Information Systems Agency.

5Collectively, these oversight policy and guidance documents cover
most-but not all-major acquisitions. The Secretary of Defense has
delegated authority to the Missile Defense Agency and to the National
Security Space Team to develop separate guidance for missile defense and
space systems, respectively.

The 5000 series describes a management framework that is intended to
translate mission needs and requirements into systems acquisition
programs. To accomplish this, the framework specifies five phases:

o 	Concept refinement: Intended to refine the initial system concept and
produce a strategy for acquiring a system capability. A decision is made
at the end of this phase (milestone A decision) whether to move to
technology development.

o 	Technology development: Intended to determine the appropriate set of
technologies to be integrated into the system by iteratively assessing the
viability of various technologies while simultaneously refining user
requirements. Once the technology has been demonstrated in a relevant
environment, a decision is made at the end of this phase (milestone B
decision) whether to move to system development and demonstration.

o 	System development and demonstration: Intended to develop a system or a
system increment and demonstrate through developer testing that the
system/system increment can function in its target environment. A decision
is made at the end of this phase (milestone C decision) whether to move to
production and deployment.

o 	Production and deployment: Intended to achieve an operational
capability that satisfies the mission needs, as verified through
independent operational test and evaluation, and ensures that the system
is implemented at all applicable locations.

o 	Operations and support: Intended to provide a support program to meet
operational support requirements and sustain the system in the most
cost-effective manner over its total life cycle.

According to the framework, an acquisition program may begin at milestone
A, B, or C, and its progress depends on obtaining sufficient knowledge to
make an informed decision about whether to continue to the next
acquisition phase. Although the framework permits programs to be managed
as a single project, DOD Instruction 5000.2 states that the department
prefers an evolutionary acquisition strategy that delivers a mature
product in increments. Under such a strategy, the instruction states that
each increment is to begin with a milestone B decision, and the production
and deployment phase of each increment is to begin with a milestone C
decision. Figure 2 provides a simplified diagram of the department's
acquisition management framework.

Figure 2: Simplified Diagram of DOD's Acquisition Management Framework

                                            System     Production  Operations 
                 Concept    Technology   development       and        and     
               refinement  development       and       deployment   support   
                                        demonstration              

Milestone decision points

Source: GAO, based on DOD Instruction 5000.2.

Typically, the Under Secretary for AT&L or a designee serves as the
investment decision authority6 for DOD acquisitions, but the Assistant
Secretary for NII/Chief Information Officer or a designee serves as the
decision authority for IT systems acquisitions.

6DOD policy establishes a decision authority, called a milestone decision
authority, as the designated individual who has overall responsibility for
an IT investment. This person has the authority to approve an investment's
move from one phase to the next phase of the acquisition process and is
responsible for reporting cost, schedule, and performance results to
higher authorities, including the Congress.

Past Evaluations of DOD Business Systems Have Revealed Acquisition
Management Weaknesses

Due in part to its long-standing and pervasive IT acquisition management
weaknesses, DOD has had limited success in acquiring IT resources to
replace its outdated business systems. Both inspector general and
departmental studies have cited these weaknesses on a variety of
acquisition projects. We have also reported on business systems
acquisition weaknesses.7 For example, in 2002 we reported that the Defense
Logistics Agency did not have effective corporate processes for
consistently acquiring software (the most costly and complex component of
systems), and that the agency did not have a software process improvement
program in place to effectively strengthen its software acquisition
processes. In 2002, we also reported acquisition management problems with
the Military Health System's acquisition of DOD's primary medical
information system, including weaknesses in incremental economic
justification, risk management, and contract management. In 2003, we
reported that DOD had not economically justified four finance and
accounting systems that have an estimated cost of more than $1 billion. In
each of our reports, we have made recommendations for strengthening
acquisition management through the adoption of best practices. DOD has
largely agreed with our recommendations, but its progress to date in
implementing them across the department has been uneven.

DOD's Acquisition Policy and Guidance Are Consistent with Some, but Not
All, Key Acquisition Best Practices

We and others, such as Carnegie Mellon University's Software Engineering
Institute, have identified and promoted the use of a number of best
practices associated with acquiring IT systems. For the purposes of this
report, we have identified 18 relevant best practices and grouped them
into two categories: (1) 10 best practices for acquiring any type of
business system and (2) 8 complementary best practices that relate
specifically to acquiring commercial component-based business systems.
Examples of best practices relevant to any business systems acquisition
include ensuring that (1) reasonable planning for all parts of the
acquisition occur, (2) a clear understanding of system requirements
exists, and (3) risks are proactively identified and systematically
mitigated. Examples of best

7U.S. General Accounting Office, Information Technology: Inconsistent
Software Acquisition Processes at the Defense Logistics Agency Increase
Project Risks, GAO-02-9 (Washington, D.C.: Jan. 10, 2002); Information
Technology: Greater Use of Best Practices Can Reduce Risks in Acquiring
Defense Health Care System, GAO-02-345 (Washington, D.C.: Sept. 26, 2002);
and DOD Business Systems Modernization: Continued Investment in Key
Accounting Systems Needs to be Justified, GAO-03-465 (Washington, D.C.:
Mar. 28, 2003).

practices relevant to commercial component-based systems acquisitions
include ensuring that (1) commercial product modification is effectively
controlled, (2) relationships among commercial products are understood
before acquisition decisions are made, and (3) the organizational impact
of using new system functionality is proactively managed. Each of these
practices is composed of from one to eight activities and is described in
table 2. DOD officials responsible for revising the 5000 series told us
that each of these 18 practices are relevant to DOD business systems
acquisitions. Appendix II provides additional details on each of these
practices.

        Table 2: Summary of Business Systems Acquisition Best Practices

Best practices Activity

Best practices relevant to any business systems acquisition

Acquisition planning  o  Plans are prepared during acquisition planning
and maintained throughout the
To ensure that reasonable planning for all acquisition.
parts of the acquisition is conducted.  o  Planning addresses the entire
acquisition process, as well as life cycle support of the

products being acquired.

o  The acquisition organization has a written policy for planning the
acquisition.

o  Responsibility for acquisition planning activities is designated.

Architectural alignment  o  The system being acquired is assessed for
alignment with the enterprise architecture To ensure that the acquisition
is consistent at key life cycle decision points, and any deviations from
the architecture are explicitly with the organization's enterprise
architecture. understood and justified by an explicit waiver to the
architecture.

o  Product line requirements--rather than just the requirements for the
system being acquired--are an explicit consideration in each acquisition.

Contract tracking and oversight

To ensure that contract activities are performed in accordance with
contractual requirements.

o  The acquiring organization has sufficient insight into the contractor's
activities to manage and control the contractor and ensure that contract
requirements are met.

o  The acquiring organization and contractor maintain ongoing
communication; commitments are agreed to and implemented by both parties.

o  All contract changes are managed throughout the life of the contract.

o  The acquisition organization has a written policy for contract tracking
and oversight.

o  Responsibility for contract tracking and oversight activities is
designated.

o  The acquiring organization involves contracting specialists in the
execution of the contract.

o  A quantitative set of software and system metrics are used to define
and measure product quality and contractor performance.

o  In addition to incentives for meeting cost and schedule estimates,
measurable, metrics-based product quality incentives are explicitly cited
in the contract.

Economic justification  o  System investment decisions are made on the
basis of reliable analyses of estimated
To ensure that system investments have an costs, expected benefits, and
anticipated risks.
adequate economic justification.  o  Large systems acquisitions are (to
the maximum extent practical) divided into a series

of smaller, incremental acquisition efforts, and investment decisions on
these smaller efforts are made on the basis of reliable analyses of
estimated costs, expected benefits, and anticipated risks.

(Continued From Previous Page)

Best practices Activity

Evaluation  o  Evaluation requirements are developed in conjunction with
the contractual To ensure that evidence showing that the requirements and
are maintained over the life of the acquisition. contract products satisfy
the defined  o  Evaluations are planned and conducted throughout the total
acquisition period to requirements are provided prior to accepting provide
an integrated approach that satisfies evaluation requirements and takes
contractor products. advantage of all evaluation results.

o  Evaluations provide an objective basis to support the product
acceptance decision.

o  The acquiring organization has a written policy for managing the
evaluation of the acquired products.

o  Responsibility for evaluation activities is designated.

Project management  o  Project management activities are planned,
organized, controlled, and communicated.
To ensure that the project office and its  o  The performance, cost, and
schedule of the acquisition are continually measured,
supporting organizations function efficiently compared with planned
objectives, and controlled.
and effectively.  o  Problems discovered during the acquisition are
managed and controlled.

o  The acquisition organization has a written policy for project
management.

o  Responsibility for project management is designated.

Requirements development and management

To ensure that contractual requirements are clearly defined and understood
by the acquisition stakeholders.

o  Contractual requirements are developed, managed, and maintained.

o  The end user and other affected groups have input to the contractual
requirements over the life of the acquisition.

o  Contractual requirements are traceable and verifiable.

o  The contractual requirements baseline is established prior to release
of the solicitation package.

o  The acquisition organization has a written policy for establishing and
managing the contractual requirements.

o  Responsibility for requirements development and management is
designated.

o  Requirements that are mandatory versus optional are clearly delineated
and used in deciding what requirements can be eliminated or postponed to
meet other project goals, such as cost and schedule constraints.

Risk management  o  Projectwide participation in the identification and
mitigation of risks is encouraged. To ensure that risks are proactively
identified  o  The defined acquisition process provides for the
identification, analysis, and mitigation and systematically mitigated. of
risks.

o  Milestone reviews include the status of identified risks.

o  The acquisition organization has a written policy for managing
acquisition risk.

o  Responsibility for acquisition risk management activities is
designated.

Solicitation

To ensure that a quality solicitation is produced and a best-qualified
contractor is selected.

o  The solicitation package includes the contractual requirements and the
proposal evaluation criteria.

o  The technical and management elements of proposals are evaluated to
ensure that the requirements of the contract will be satisfied.

o  The selection official selects a supplier who is qualified to satisfy
the contract's requirements.

o  The acquiring organization has a written policy for conducting the
solicitation.

o  Responsibility for the solicitation is designated.

o  A selection official has been designated to be responsible for the
selection process and decision.

o  The acquiring team includes contracting specialists to support contract
administration.

(Continued From Previous Page)

Best practices Activity

Transition to support

To ensure proper transfer of the system from the acquiring organization to
the support organization.

o  The acquiring organization ensures that the support organization has
the capacity and capability to provide the required support.

o  There is no loss in continuity of support to the products during
transition from the supplier to the support organization.

o  Configuration management of the products is maintained throughout the
transition.

o  The acquiring organization has a written policy for transitioning the
products to the support organization.

o  The acquiring organization ensures that the support organization is
involved in planning for transition to support.

o  Responsibility for transition to support activities is designated.

Complementary best practices relevant to commercial component-based
business systems acquisitions

Component modification  o  Modification of commercial components is
discouraged and allowed only if justified by
To ensure that commercial product a thorough analysis of life-cycle costs
and benefits.
modification is effectively controlled.

Configuration management  o  Project plans explicitly provide for
evaluation, acquisition, and implementation of new,
To ensure the integrity and consistency of often frequent, product
releases.
system commercial components.  o  Modification or upgrades to deployed
versions of system components are centrally

controlled and unilateral user release changes are precluded.

Dependency analysis  o  Decisions about acquisition of commercial
components are based on deliberate and
To ensure that relationships between thorough research, analysis, and
evaluation of the components' interdependencies.
commercial products are understood before
acquisition decisions are made.

Legacy systems integration planning  o  Project plans explicitly provide
for the necessary time and resources for integrating
To ensure reasonable planning for integration commercial components with
legacy systems.
of commercial products with existing systems.

Organization change management  o  Project plans explicitly provide for
preparing users on the impact that the business
To ensure that the organizational impact of processes embedded in the
commercial components will have on the users' respective
using new system functionality is proactively roles and responsibilities.
managed.  o  The introduction and adoption of changes to how users will be
expected to execute

                        their jobs are actively managed.

Solicitation  o  Systems integration contractors are explicitly evaluated
on their ability to implement
To ensure that a quality solicitation is produced commercial components.
and a best-qualified contractor is selected.

Tradeoff analysis  o  Investment decisions throughout a system's life
cycle are based on tradeoffs among To ensure that system requirements
alone do the availability of commercial products (current and future), the
architectural not drive the system's solution. environment in which the
system is to operate (current and future), defined system

            requirements, and acquisition cost/schedule constraints.

Vendor and product research and  o  Commercial component and vendor
options are researched, evaluated/tested, and
evaluation understood, both early and continuously.
To ensure that vendor and product  o  A set of evaluation criteria for
selecting among commercial component options is
characteristics are understood before established that includes both
defined system requirements and vendor/commercial
acquisition decisions are made. product characteristics (e.g., customer
satisfaction with company and product line).

           Sources: See sources listed in appendix I of this report.

DOD's acquisition policies and guidance largely incorporate the 10 best
practices that are relevant to any business systems acquisition.8 More
specifically, they fully incorporate 7 of the 10 practices and partially
incorporate the other 3 practices. However, they generally do not
incorporate the 8 best practices that relate to acquiring commercial
component-based business systems. In particular, they fully incorporate 1
best practice, partially incorporate 2, and do not incorporate the
remaining

5. (See fig. 3 for a summary of our analysis.) At our request, DOD
officials responsible for the 5000 series also assessed it against those
18 practices, and we incorporated information from their assessment into
ours. These officials also told us that the acquisition guidebook is
currently being expanded to incorporate additional best practices, but
they did not provide us with a plan for accomplishing this. Until this is
accomplished, DOD is increasing the risk that important and beneficial
best practices will not be followed and that DOD business systems
investments will not deliver promised capabilities and benefits on time
and within budget.

8We defined "fully incorporate" to mean the 5000 series addressed all of
the practice's activities; "partially incorporate" to mean it addressed
some, but not all, of the activities; and "not incorporate" to mean it did
not address any of the activities.

Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best Practices

Not incorporated

Partially incorporated Fully incorporated Source: GAO, based on analysis
of DOD data.

The 5000 Series Largely Incorporates Best Practices Relevant to Any
Business Systems Acquisition

Of the 10 best practices that we categorized as relevant to the
acquisition of any business system, whether custom-developed or developed
using commercial packages and products, essentially all have been
incorporated into DOD's acquisition policies and guidance. (See table 3
for our detailed comparative analysis of the 5000 series against the 10
best practices.) For example, those practices aimed at ensuring that the
acquisition is well planned, that the system is adequately tested and
evaluated against contractual requirements, and that the requirements are
clearly defined and understood by all stakeholders are all addressed in
the 5000 series. Similarly, for the 3 practices that are not fully
addressed, this is the case because one activity associated with the
practice is not addressed. According to DOD officials responsible for
revising the 5000 series, the policies contain those best practices
mandated by either law or DOD regulation, and other, optional best
practices are contained in the interim guidance represented by the former
DOD 5000.2-R.

Nevertheless, the activities that are missing from the 3 practices are
important, and their absence increases the risk that the activities, and
thus the practice, will not be adequately performed. In turn, this
increases the risk that acquisition projects will fall short of
expectations. The best practice aimed at ensuring that risks are
proactively identified and systematically mitigated is a case in point.
This practice has five activities associated with it, one of which the
5000 series does not address-project reviews include the status of
identified risks. As with all the activities under this practice, this
activity plays an important role in ensuring that the appropriate level of
attention and visibility is regularly given to risk identification and
mitigation to ensure that it is effectively executed. Conversely, if the
activities are not provided for in policy and guidance, it is unlikely
that they will be performed, and it is likely that acquisition risks will
become cost, schedule, and performance problems.

 Table 3: Activity-by-activity Comparison of the 5000 Series to Best Practices
                Relevant to Any IT Business Systems Acquisition

                                              5000 series      5000 series    
                                             addresses this incorporates this 
            Best practice/Activity             activity?     best practice?   
             Acquisition planning                                 Fully       
     Plans are prepared during acquisition        Yes       
    planning and maintained throughout the                  
                 acquisition.                               
Planning addresses the entire acquisition                
process, as well as life cycle support of      Yes       
                      the                                   
           products being acquired.                         
      The acquisition organization has a                    
        written policy for planning the           Yes       
                 acquisition.                               
    Responsibility for acquisition planning       Yes       
           activities is designated.                        
            Architectural alignment                             Partially     

The system being acquired is assessed for alignment with the enterprise
architecture at Yes key life cycle decision points, and any deviations
from the architecture are understood and justified by an explicit waiver
to the architecture.

Product line requirements-rather than just the requirements for the system
being No acquired-are an explicit consideration in each acquisition.

Contract tracking and oversight Fully

The acquiring organization has sufficient insight into the contractor's
activities to manage Yes and control the contractor and ensure that
contract requirements are met.

The acquiring organization and contractor maintain ongoing communication;
Yes commitments are agreed to and implemented by both parties.

All contract changes are managed throughout the life of the contract. Yes

The acquisition organization has a written policy for contract tracking
and oversight. Yes

Responsibility for contract tracking and oversight activities is
designated. Yes

The acquiring organization involves contracting specialists in the
execution of the contract. Yes

A quantitative set of software and system metrics are used to define and
measure product Yes quality and contractor performance.

In addition to incentives for meeting cost and schedule estimates,
measurable, Yes metrics-based product quality incentives are explicitly
cited in the contract.

Economic justification Fully

System investment decisions are made on the basis of reliable analyses of
estimated Yes costs, expected benefits, and anticipated risks.

Large system projects are (to the maximum extent practical) divided into a
series of Yes smaller, incremental acquisition efforts, and investment
decisions on these smaller efforts are made on the basis of reliable
analyses of estimated costs, expected benefits, and anticipated risks.

(Continued From Previous Page)

                                              5000 series      5000 series    
                                             addresses this incorporates this 
            Best practice/Activity             activity?     best practice?   
                  Evaluation                                      Fully       
Evaluation requirements are developed in                 
       conjunction with the contractual           Yes       
                 requirements                               
    and are maintained over the life of the                 
                 acquisition.                               

Evaluations are planned and conducted throughout the total acquisition
period to provide Yes an integrated approach that satisfies evaluation
requirements and takes advantage of all evaluation results.

Evaluations provide an objective basis to support the product acceptance
decision. Yes

The acquiring organization has a written policy for managing the
evaluation of the Yes acquired products.

Responsibility for evaluation activities is designated. Yes

                          Project management Partially

Project management activities are planned, organized, controlled, and
communicated.	Partly- communication not cited.

The performance, cost, and schedule of the acquisition are continually
measured, Yes compared with planned objectives, and controlled.

Problems discovered during the acquisition are managed and controlled. Yes

The acquisition organization has a written policy for project management.
Yes

Responsibility for project management is designated. Yes

Requirements development and management Fully

Contractual requirements are developed, managed, and maintained. Yes

The end user and other affected groups have input into the contractual
requirements over Yes the life of the acquisition.

Contractual requirements are traceable and verifiable. Yes

The contractual requirements baseline is established prior to release of
the solicitation Yes package.

The acquisition organization has a written policy for establishing and
managing the Yes contractual requirements.

Responsibility for requirements development and management is designated.
Yes

Requirements that are mandatory versus optional are clearly  Yes 
                      delineated and used in                        
deciding what requirements can be eliminated or postponed to     
                    meet other project goals,                       
              such as cost and schedule constraints.                
                         Risk management                            Partially 

Projectwide participation in the identification and mitigation of risks is
encouraged. Yes

The defined acquisition process provides for the identification, analysis,
and mitigation of Yes risks.

Milestone reviews include the status of identified risks. No

(Continued From Previous Page)

                                              5000 series      5000 series    
                                             addresses this incorporates this 
            Best practice/Activity             activity?     best practice?   
      The acquisition organization has a                    
    written policy for managing acquisition       Yes       
                     risk.                                  
      Responsibility for acquisition risk         Yes       
     management activities is designated.                   
                 Solicitation                                     Fully       
     The solicitation package includes the        Yes       
contractual requirements and the proposal                
             evaluation criteria.                           
The technical and management elements of                 
    proposals are evaluated to ensure that        Yes       
                      the                                   
     requirements of the contract will be                   
                  satisfied.                                
The selection official selects a supplier                
        who is qualified to satisfy the           Yes       
                  contract's                                
                 requirements.                              
The acquiring organization has a written       Yes       
    policy for conducting the solicitation                  
    Responsibility for the solicitation is        Yes       
                  designated.                               
A selection official has been designated                 
      to be responsible for the selection         Yes       
                  process and                               
                   decision.                                
    The acquiring team includes contracting                 
        specialists to support contract           Yes       
                administration.                             
             Transition to support                                Fully       
    The acquiring organization ensures that                 
the support organization has the capacity      Yes       
                      and                                   
      capability to provide the required                    
                   support.                                 

There is no loss in continuity of support to the products during
transition from the supplier Yes to the support organization.

Configuration management of the products is maintained throughout the
transition. Yes

The acquiring organization has a written policy for transitioning the
products to the Yes support organization.

The acquiring organization ensures that the support organization is
involved in planning Yes for transition to support.

Responsibility for transition to support activities is designated. Yes

                  Source: GAO, based on analysis of DOD data.

The 5000 Series Generally Of the 8 best practices relevant to acquiring
commercial component-based Does Not Incorporate Best business systems, few
have been incorporated into DOD systems Practices Relevant to acquisition
policies and guidance. (See table 4 for our detailed comparative

analysis of the 5000 series against the 8 best practices.) For example,
whileCommercial the practice aimed at ensuring that adequate planning
takes place for Component-Based Business integrating commercial products
with legacy systems is incorporated into Systems Acquisitions the 5000
series, practices associated with closely controlling any

modification to the software of these packages and products, thoroughly

analyzing and understanding the dependencies among commercial

products before acquiring them, and proactively managing the institutional
change that results from implementing the functionality in commercial
packages and products are not incorporated. According to DOD officials
responsible for revising the 5000 series, these practices were not
included in the recently revised version of DOD's acquisition policies
because they included only those in existing law or regulation.

Nevertheless, the absence of these practices from the 5000 series
increases the risk that the practices will not be performed, which, in
turn, increases the risk that acquisition projects will fall short of
expectations. The practice intended to ensure development of a quality
solicitation and selection of a best-qualified contractor illustrates
this. Specifically, this practice calls for contract bidders to be
evaluated on their ability to implement commercial components. This
evaluation is important because integrating and implementing these
component products is sufficiently different from developing customized
system solutions; it requires different core competencies and experiences
to be successful. By explicitly taking this into consideration in
evaluating and selecting a contractor, the risk of contract award to a
less-than-best-qualified contractor is reduced.

Table 4: Activity-by-Activity Comparison of the 5000 Series to Best
Practices Relevant to Commercial Component-based Business Systems
Acquisitions

Does the 5000 Does the 5000 series series address this incorporate this
best

Best practice/Activity activity? practice?

Component modification No

Modification of commercial components is discouraged and allowed only if
justified by a No thorough analysis of life-cycle costs and benefits.

Configuration management Partially

Project plans provide for evaluation, acquisition, and implementation of
new, often Yes frequent, product releases.

Modification or upgrades to deployed versions of system components are
centrally No controlled and unilateral user release changes are precluded.

Legacy systems integration planning Fully

Project plans explicitly provide for the necessary time and resources for
integrating Yes commercial components with legacy systems.

Dependency analysis No

Decisions about acquisition of commercial components are based on
deliberate and No thorough research, analysis, and evaluation of the
components' interdependencies.

(Continued From Previous Page)

Does the 5000 Does the 5000 series series address this incorporate this
best

Best practice/Activity activity? practice?

Organization change management No

Project plans provide for preparing users for the impact that the business
processes No embedded in the commercial components will have on the users'
respective roles and responsibilities.

The introduction and adoption of changes to how users will be expected to
use the No system to execute their jobs are actively managed.

Solicitation No

Systems integration contractors are explicitly evaluated on their ability
to implement No commercial components.

Tradeoff analysis No

Investment decisions throughout a system's life cycle are based on
tradeoffs among the No availability of commercial products (current and
future), the architectural environment in which the system is to operate
(current and future), defined system requirements, and acquisition
cost/schedule constraints.

Vendor and product research and evaluation Partially

Commercial component and vendor options are researched, evaluated/tested,
and Yes understood, both early and continuously.

A set of evaluation criteria for selecting among commercial component
options is Partly-vendor/ established that includes both defined system
requirements and vendor/commercial commercial product product
characteristics (e.g., customer satisfaction with company and product
line). characteristics not

cited.

Source: GAO, based on analysis of DOD data.

DOD Officials Report That They Are in the Process of Revising the Interim
Defense Acquisition Guidebook

The DOD officials responsible for revising the 5000 series told us they
recognize the need for the 5000 series to incorporate additional best
practices. To this end, they reported that efforts are under way to expand
the Interim Defense Acquisition Guidebook to include additional best
practices and lessons learned across the department. However, the
officials could not provide us with a documented plan and associated
documentation showing how this task will be accomplished, what resources
are needed and assigned to accomplish it, when it will be accomplished,
and where the department stands in accomplishing it. Instead, the
officials told us that progress on it has been slowed by other priorities,
such as the need to first revise DOD Directive 5000.1 and DOD Instruction
5000.2. They also said that there is only a small number of staff
available to work on what they described as being an extensive revision of
the guidebook. According to these officials, 80 to 90 percent of the
revision has been completed and reviewed and their goal is to publish the
initial version of the revised guidebook by September 30, 2004. In our

view, until the missing best practices that we cite in this report are
included in DOD's acquisition policies and guidance, the chance that
business systems acquisitions will follow the policies and guidance and
consistently produce a successful outcome is diminished.

DOD's Acquisition Policies Do Not Contain Sufficient Controls to Ensure
That the Requirement Is Met for Appropriately Applying Best Practices

Federal laws and regulations define the need for effective controls over
agency programs, and controls are a key factor in achieving program
results, minimizing operational problems, and managing evolving demands
and priorities.9 Controls over defined processes, procedures, and support
activities are considered effective if they entail measuring and verifying
whether a given practice is followed. Without sufficient controls, it is
unlikely that practices will be consistently employed, which, in turn,
increases the probability that the positive program and project outcomes
these practices are designed to produce will not occur.

DOD's acquisition policy requires program managers and investment decision
authorities to examine and, as appropriate, adopt best practices. However,
neither the policies nor the accompanying guidance explain what "examine"
means, including whether practice use is to be measured and verified.
Instead, the policies state that any issues regarding the intent of the
5000 series, which would include whether practice adoption is to be
measured and validated, shall be resolved by the investment decision
authority, meaning that it is entirely up to this individual what
information relative to the use of best practices is relevant and
necessary to ensure that best practices are appropriately followed.
According to the Chairman of the Defense Acquisition Policy Working
Group,10 this control is sufficient, and explicit requirements for
measuring and validating the use of best practices are not necessary.

In our view, not requiring that decision authorities examine the
measurement and validation of best practices' use increases the chance
that important best practices will not be appropriately followed, as

9The Federal Managers' Financial Integrity Act of 1982 (Pub.L. No.
97-255); Government Performance and Results Act of 1993 (Pub.L. No.
103-62); Office of Management and Budget Circular A-123, June 21, 1995.

10The Defense Acquisition Policy Working Group is the standing DOD
acquisition policy working group that revised the 5000 series.

required by DOD policy. As we have previously reported,11 a lack of
explicit controls that require review of relevant information at key
decision points raises the risk of making uninformed project decisions, as
well as the risk that investments will not meet cost, schedule,
capability, and benefit commitments.

Conclusions	DOD recognizes the importance of business systems acquisition
best practices by including best practices in their revised policy and
guidance. However, other practices that, if followed, could increase the
odds of acquisitions delivering promised system capabilities and benefits
on time and within budget have yet to be similarly included. In
particular, those practices associated with the successful acquisition of
commercial component-based business systems have not been sufficiently
incorporated into either the policies or the guidance. Moreover, effective
controls for ensuring that best practices are appropriately followed are
not adequately provided for in the policies. Although DOD officials intend
to expand the coverage of best practices in future versions of DOD's
acquisition guidance, it is unclear what the scope, nature, and status of
these intentions are because explicit plans for revising the guidance and
associated progress reports were not available. Until DOD incorporates the
best practices we found missing in the 5000 series, and until it
strengthens the means by which the appropriate use of these practices will
be ensured, its business systems acquisitions will be exposed to
unnecessary risk. Therefore, it is important for DOD to treat further
revisions of its acquisition policy and guidance relative to business
systems as a priority and move quickly to incorporate missing best
practices and associated controls for ensuring that the practices are
followed.

Recommendations for Executive Action

To improve DOD's ability to acquire business systems, we recommend that
the Secretary of Defense direct the Under Secretary for AT&L, in
collaboration with the Assistant Secretary for NII and the Director, OT&E,
to take the following actions:

1.	Develop and implement an explicit plan for incorporating into the 5000
series the best practices and associated activities currently missing

11U.S. General Accounting Office, Defense Acquisitions: DOD's Revised
Policy Emphasizes Best Practices, but More Controls Are Needed, GAO-04-53
(Washington, D.C.: Nov. 10, 2003).

from the series. We recommend that the plan specify tasks to be performed,
resources needed and assigned, and milestones for completing tasks.

2.	We further recommend that progress against this plan be tracked and
reported as appropriate, and that the plan, at a minimum, incorporate each
of the following best practice activities:

o 	Product line requirements-rather than just the requirements for the
system being acquired-are an explicit consideration in each acquisition.

o 	Acquisition project management activities are communicated to all
stakeholders.

o  Acquisition reviews include the status of identified risks.

o 	Modification of commercial components is discouraged and allowed only
if justified by a thorough analysis of life-cycle costs and benefits.

o 	Modification or upgrades to deployed versions of system components are
centrally controlled, and unilateral user release changes are precluded.

o 	Acquisition decisions about commercial components are based on
deliberate and thorough research, analysis, and evaluation of the
components' interdependencies.

o 	Acquisition plans provide for preparing users for the impact that the
business processes embedded in the commercial components will have on
their respective roles and responsibilities.

o 	Changes affecting how users will be expected to use the system to
execute their jobs are actively managed.

o 	Systems integration contractors are explicitly evaluated on their
ability to implement commercial components.

o 	Investment decisions throughout a system's life cycle are based on a
continuous set of tradeoffs among capabilities available in commercial
components (current and future), the architectural environment in

which the system is to operate, defined system requirements, and existing
cost/schedule constraints.

o 	Evaluation criteria are established for selecting among commercial
component options that include both defined system requirements and
vendor/commercial product characteristics.

3.	To ensure that the best practices provided for in DOD acquisition
policies and guidance are appropriately followed, we also recommend that
the above recommended plan incorporate steps to include in DOD's
acquisition policies a provision for measurement and verification of best
practices.

Agency Comments and Our Evaluation

In written comments on a draft of this report, signed by the Principal
Director for Command, Control, Communications, Space, and Information
Technology Programs in the office of the DOD Assistant Secretary for
Networks and Information Integration, DOD agreed with the importance and
relevance of the best practices that we cite in the report. Additionally,
DOD agreed with 2 of our 13 recommendations for incorporating additional
best practices, stating that the department would incorporate the 2
practices in its policies and guidance.

DOD also partially agreed with 9 of our recommendations for incorporating
additional practices, stating that it would consider augmenting its
coverage of 5 of the practices and that it believed that 4 practices
already existed in its policies and guidance. With regard to the 5
practices, DOD stated that it needed to review each practice further and
determine the need for its emphasis or endorsement in the 5000 series. We
understand DOD's desire to carefully consider changes to its acquisition
policies and guidance, and believe that such careful deliberation is
consistent with the spirit of our recommendations.

With regard to the remaining 4 practices that DOD partially agreed with,
we do not agree with the department's comment that these best practices
adequately exist in the 5000 series. For example, DOD commented that
because its existing policies and guidance provide for the use of
integrated product teams, which, according to DOD, are a means for
promoting collaboration and facilitating communication among stakeholders,
its policies and guidance therefore already provide for communicating
information about management of a given acquisition project to all
relevant project stakeholders. While we do not question the use of
integrated

product teams as a way to communicate information, the point of our
recommendation is that there needs to be an explicit recognition in policy
or guidance of the type of information to be communicated and with whom it
is to be communicated. Restated, our recommendation for incorporating the
best practice of communicating acquisition management activities to all
stakeholders is intended to permit communication vehicles, such as
integrated product teams, to be more effective by explicitly providing for
this best practice in relevant policies and guidance. As another example,
we do not agree with DOD's comment that 2 of the best practices that we
recommended for incorporation in its policies and guidance-preparing users
for the impact that business process changes embedded in commercial
components will have on their roles and responsibilities, and actively
managing changes in how users will use new systems-are already
sufficiently contained in the 5000 series. In particular, while we agree
that the series references an acquisition management toolkit that
addresses these 2 best practices, this reference is provided only once in
the 5000 series, and this reference is only in relation to one phase of
the acquisition cycle (the technology development phase). Given the
importance and relevance of these practices to successful implementation
of commercial component-based systems, our position, and thus the basis
for our recommendations, is that the practices' implementation would be
more likely to occur if the practices were visible and better recognized
in all relevant stages of DOD's acquisition cycle.

Also in its comments, DOD did not agree with our recommendations to
develop and implement an explicit plan to govern its ongoing and future
policy and guidance revision activities, specifically stating that the
recommendation was inappropriate and offering updated information on the
status of and associated milestone for completing its activities. While we
have updated our report to include the revised status and milestone
information, we do not agree with DOD that a plan governing these efforts
is not needed. Given the importance of DOD's acquisition policies and
guidance, and the need for their continuous review and update to reflect
new acquisition best practices, we believe that having an explicit plan
that defines how and when these policies and guidance will be incorporated
is essential. Among other things, a plan would highlight the resource
constraints that this revision effort has been subject to, would allow
measurement against defined milestones, and would allow disclosure of
progress and impediments.

DOD also did not agree with our recommendation to add stronger controls
for ensuring adherence to the best practices that are contained in its

acquisition policies and guidance, stating that its existing oversight
process includes the necessary compliance activities. We disagree. As we
state in the report, DOD's existing policy leaves these compliance
activities to the discretion of the program manager and the investment
decision authority, and it does not provide for measurement and
verification of the use of best practices, both of which are recognized
components of effective control processes.

A copy of DOD's comments is reprinted in appendix III, along with our
response.

We are sending copies of this report to the Chairmen and Ranking Minority
Members of the Senate and House Committees on Armed Services;
Subcommittees on Defense, Senate and House Committees on Appropriations;
and the Subcommittee on Military Readiness, House Committee on Armed
Services. We are also sending copies to the Director, Office of Management
and Budget; the Secretary of Defense; the Under Secretary of Defense
(AT&L); the Assistant Secretary of Defense (NII)/Chief Information
Officer; and the Director, OT&E. We will make copies available to others
on request. This report will also be available at no charge on our Web
site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please
contact me at (202) 512-3439. I can also be reached by e-mail at
[email protected]. Other contacts and key contributors to this report are
listed in appendix IV.

Randolph C. Hite Director, Information Technology Architecture

and Systems Issues

Appendix I

                       Objectives, Scope, and Methodology

Our objectives were to determine whether the Department of Defense's (DOD)
revised systems acquisition policies for acquiring information technology
(IT) business systems (1) are consistent with industry best practices,
including those pertaining to commercial component-based systems, and (2)
provide the necessary controls to ensure that the department's component
organizations adhere to the practices.

To accomplish the first objective, we identified the DOD policies and
guidance relevant to business systems. These policies and guidance are
contained in three documents-DOD Directive 5000.1, DOD Instruction 5000.2,
and the Interim Defense Acquisition Guidebook-and are generally referred
to as the 5000 series. We then reviewed each of these documents and
discussed with DOD officials responsible for developing and revising the
documents what steps were taken to incorporate best practices into each
document. The DOD officials that we interviewed were from the offices of
the Under Secretary of Defense (Acquisition, Technology, and Logistics
(AT&L)); the Assistant Secretary of Defense (Networks and Information
Integration (NII)); and the Director, Operational Test and Evaluation
(OT&E). Next, we researched prior GAO reports; the work of federally
funded research and development organizations, such as the Software
Engineering Institute and The Aerospace Corporation;1 and other
authoritative sources to identify business systems acquisition best
practices. Our research produced 18 best practices, including associated
activities, that we placed into two categories-one category for the
practices that are relevant to any business systems acquisition and one
category for the practices that are relevant to commercial component-based
business systems acquisitions. In particular, we drew extensively from the
Software Engineering Institute's Software Acquisition Capability Maturity
Model.2 In doing so, we selected practices from the model's repeatable
level of process maturity, which is level two on the model's five-level
scale. We used the repeatable level of process maturity because it is
intended to provide the necessary process discipline

1For example, the Software Engineering Institute is a federally funded
research and development center operated by Carnegie Mellon University and
sponsored by DOD. The Software Engineering Institute's objective is to
provide leadership in software engineering and in the transition of new
software engineering technology into practice. The Aerospace Corporation
is a private, nonprofit organization that operates a federally funded
research and development center for DOD that focuses on the government's
need to develop spacerelated hardware and software.

2Software Engineering Institute, Software Acquisition Capability Maturity
Model(R) version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: March 2002).

Appendix I
Objectives, Scope, and Methodology

to allow an organization to repeat earlier successes on similar projects.
In addition, we included one Software Acquisition Capability Maturity
Model level-three process area-risk management-because many experts
consider it to be one of the most important process areas. We did not
attempt to develop an exhaustive list of best practices and, in fact,
fully recognize that additional best practices exist, such as ensuring
that the appropriate level of human capital knowledge, skills, and
abilities are employed, as well as that additional activities for the
practices that we have identified exist, such as those configuration
management activities associated with identifying, controlling, reporting
on, and auditing configuration items and components. For the purposes of
this report, we identified those practices that are embodied, recognized,
and accepted acquisition models or frameworks, as well as those practices
that are now being recognized as being unique to commercial
component-based systems and for which there appears to be general
agreement, including agreement with DOD officials responsible for revising
the 5000 series, that the practices are relevant and important. Last, we
analyzed each of the DOD 5000 series documents to determine whether the
documents addressed, either directly or indirectly by reference to another
authoritative document, the 18 best practices that we identified. Based on
this analysis, we judged whether the 5000 series documents fully,
partially, or did not incorporate each best practice. In making these
judgments, we used the following criteria:

o To fully incorporate the practice, the 5000 series addressed all of the
practice's activities.

o To partially incorporate the practice, the 5000 series addressed some,
but not all, of the practice's activities.

o To not incorporate the practice, the 5000 series did not address any of
the practice's activities.

Additionally, we provided the DOD officials responsible for revising the
5000 series with the 18 practices that we identified to obtain their views
on whether the practices were relevant to DOD business systems
acquisitions. The officials agreed that they were. We also requested that
these officials perform their own assessment of the 5000 series against
these practices, and we used these officials' assessment in making our
judgments as to whether the practices were fully, partially, or not
incorporated into DOD's acquisition policies and guidance. For a number of
the activities, DOD identified the Federal Acquisition Regulation and the
Defense Federal

Appendix I
Objectives, Scope, and Methodology

Acquisition Regulation Supplement as evidence that the activity was being
performed within a particular practice. We accepted that as proof that the
activity was being covered within DOD's business systems acquisition
policy.

To address our second objective, we researched federal internal control
standards and controls inherent in the business systems acquisition best
practices that we identified. In particular, we reviewed the Software
Engineering Institute's Software Acquisition Capability Maturity Model
framework and GAO's internal control standards.3 We then analyzed DOD's
revised acquisition policies and guidance to identify whether these
controls were cited and to provide assurance that relevant best practices
were being followed. We also interviewed DOD officials responsible for
revising the 5000 series to determine reasons why controls were addressed
or not addressed in the policies and guidance.

We conducted our work at DOD offices in Arlington, Virginia, between
December 2003 and May 2004 in accordance with generally accepted
government auditing standards

3U.S. General Accounting Office, Standards for Internal Control in the
Federal Government, AIMD-00-21.3.1 (Washington, D.C.: November 1999).

Appendix II

Best Practices

Additional information on each of the 18 best practices that we identified
is provided in this appendix.

Best Practices Relevant to Any IT Business Systems Acquisition

1. Acquisition Planning

Purpose: To ensure that reasonable planning for all parts of the
acquisition is conducted.

Description: Acquisition planning is the process for conducting and
documenting acquisition planning activities beginning early and covering
all parts of the project. It extends to all acquisition areas, such as
budgeting, scheduling, resource estimating, risk identification, and
requirements definition, as well as the overall acquisition strategy.
Acquisition planning begins with the earliest identification of a
requirement that is to be satisfied through an acquisition.

Activities: (1) Plans are prepared during acquisition planning and
maintained throughout the acquisition. (2) Planning addresses the entire
acquisition process, including life cycle support of the products being
acquired. (3) The acquisition organization has a written policy for
planning the acquisition. (4) Responsibility for acquisition planning
activities is designated.

2. Architectural Alignment

Purpose: To ensure that the acquisition is consistent with the
organization's enterprise architecture.

Description: Architectural alignment is the process for analyzing and
verifying that the proposed architecture of the system being acquired is
consistent with the enterprise architecture for the organization acquiring
the system. Such alignment is needed to ensure that acquired systems can
interoperate and are not unnecessarily duplicative of one another.
Exceptions to this alignment requirement are permitted, but only when
justified and only when granted an explicit waiver from the architecture.
A particular architectural consideration is whether requirements that
extend beyond the specific system being acquired should be considered when
selecting system components. Such product line (i.e., systems that are
developed from a common set of assets and share a common and managed set
of features) considerations can provide substantial production economies
over acquiring systems from scratch.

Appendix II Best Practices

Activities: (1) The system being acquired is assessed for alignment with
the enterprise architecture at key life cycle decision points, and any
deviations from the architecture are explicitly understood and justified
by an explicit waiver to the architecture. (2) Product line requirements-
rather than just the requirements for the system being acquired-are an
explicit consideration in each acquisition.

3. Contract Tracking and Oversight

Purpose: To ensure that contract activities are performed in accordance
with contractual requirements.

Description: Contract tracking and oversight is the process by which
contractual agreements are established and contractor efforts to satisfy
those agreements are supervised. It involves information sharing between
the acquirer and contractor to ensure that contractual requirements are
understood, that there are regular measurements to disclose overall
project status and whether problems exist, and that there are appropriate
incentives for ensuring that cost and schedule commitments are met and
that quality products are delivered. Contract tracking and oversight
begins with the award of the contract and ends at the conclusion of the
contract's period of performance.

Activities: (1) The acquiring organization has sufficient insight into the
contractor's activities to manage and control the contractor and ensure
that contract requirements are met. (2) The acquiring organization and
contractor maintain ongoing communication; commitments are agreed to and
implemented by both parties. (3) All contract changes are managed
throughout the life of the contract. (4) The acquiring organization has a
written policy for contract tracking and oversight. (5) Responsibility for
contract tracking and oversight activities is designated. (6) The
acquiring organization involves contracting specialists in the execution
of the contract. (7) A quantitative set of software and system metrics is
used to define and measure product quality and contractor performance.1
(8) In

1Richard J. Adams, Suellen Eslinger, Karen L. Owens, and Mary A. Rich,
Reducing Risk in the Acquisition of Software-Intensive Systems: Best
Practices from the Space System Domain (Los Angeles, CA: 2003).

Appendix II Best Practices

addition to incentives for meeting cost and schedule estimates,
measurable, metrics-based product quality incentives are explicitly cited
in the contract.2

4. Economic Justification

Purpose: To ensure that system investments have an adequate economic
justification.

Description: Economic justification is the process for ensuring that
acquisition decisions are based on reliable analyses of the proposed
investment's likely costs versus benefits over its useful life, as well as
an analysis of the risks associated with actually realizing the
acquisition's forecasted benefits for its estimated costs. Moreover, it
entails minimizing the risk and uncertainty of large acquisitions that
require spending large sums of money over many years by breaking the
acquisition into smaller, incremental acquisitions. Economic justification
is not a one-time event, but rather is performed throughout an
acquisition's life cycle in order to permit informed investment decision
making.

Activities: (1) System investment decisions are made on the basis of
reliable analyses of estimated system life cycle costs, expected benefits,
and anticipated risks. (2) Large systems acquisitions are (to the maximum
extent practical) divided into a series of smaller, incremental
acquisition efforts, and investment decisions on these smaller efforts are
made on the basis of reliable analyses of estimated costs, expected
benefits, and anticipated risks.

5. Evaluation

Purpose: To ensure that evidence showing that the contract products
satisfy the defined requirements are provided prior to accepting
contractor products.

Description: Evaluation is the process by which contract deliverables are
analyzed to determine whether they meet contract requirements. It includes
developing criteria such as product acceptance criteria to be

2Adams and Eslinger, "COTS-Based Systems: Lessons Learned from Experiences
with COTS Software Use on Space Systems." (Paper presented to the Southern
California SPIN: Oct. 6, 2003.)

Appendix II Best Practices

included into both the solicitation package and the contract. It should be
conducted continuously throughout the contract period as products are
delivered. It begins with development of the products' requirements and
ends when the acquisition is completed.

Activities: (1) Evaluation requirements are developed in conjunction with
the contractual requirements and are maintained over the life of the
acquisition. (2) Evaluations are planned and conducted throughout the
total acquisition period to provide an integrated approach that satisfies
evaluation requirements and takes advantage of all evaluation results. (3)
Evaluations provide an objective basis to support the product acceptance
decision. (4) The acquisition organization has a written policy for
managing the evaluation of the acquired products. (5) Responsibility for
evaluation activities is designated.

6. Project Management

Purpose: To ensure that the project office and its supporting
organizations function efficiently and effectively.

Description: Project management is the process for planning, organizing,
staffing, directing, and managing all project-office-related activities,
such as defining project tasks, estimating and securing resources,
scheduling activities and tasks, training, and accepting products. Project
management begins when the project office is formed and ends when the
acquisition is completed.

Activities: (1) Project management activities are planned, organized,
controlled, and communicated. (2) The performance, cost, and schedule of
the acquisition are continually measured, compared with planned
objectives, and controlled. (3) Problems discovered during the acquisition
are managed and controlled. (4) The acquisition organization has a written
policy for project management. (5) Responsibility for project management
is designated.

7. Requirements Development and Management

Purpose: To ensure that contractual requirements are clearly defined and
understood by the acquisition stakeholders.

Description: Requirements development is the process for developing and
documenting contractual requirements, including evaluating opportunities

Appendix II Best Practices

for reusing existing assets. It involves participation from end users to
ensure that product requirements are well understood, and that optional
versus mandatory requirements are clearly delineated. Requirements
management is the process for establishing and maintaining agreement on
the contractual requirements among the various stakeholders and for
ensuring that the requirements are traceable, verifiable, and controlled.
This involves baselining the requirements and controlling subsequent
requirements changes. Requirements development and management begins when
the solicitation's requirements are documented and ends when system
responsibility is transferred to the support organization.

Activities: (1) Contractual requirements are developed, managed, and
maintained. (2) The end user and other affected groups have input into the
contractual requirements over the life of the acquisition. (3) Contractual
requirements are traceable and verifiable. (4) The contractual
requirements baseline is established prior to release of the solicitation
package. (5) The acquisition organization has a written policy for
establishing and managing the contractual requirements. (6) Responsibility
for requirements development and management is designated. (7)
Requirements that are mandatory versus optional are clearly delineated and
used in deciding what requirements can be eliminated or postponed to meet
other project goals, such as cost and schedule constraints.3

8. Risk Management

Purpose: To ensure that risks are identified and systematically mitigated.

Description: Risk management is the process for identifying potential
acquisition problems and taking appropriate steps to avoid their becoming
actual problems. It includes risk identification and categorization based
on estimated impact, development of risk mitigation strategies, and
execution of and reporting on the strategies. Risk management occurs early
and continuously in the acquisition life cycle.

Activities: (1) Projectwide participation in the identification and
mitigation of risks is encouraged. (2) The defined acquisition process
provides for the

3Software Engineering Institute, Real-Time Systems Engineering: Lessons
Learned from Independent Technical Assessments, CMU/SEI-2001-TN-004
(Pittsburgh, PA: June 2001); and Adams, Eslinger, Owens, and Rich,
Reducing Risk in the Acquisition of Software Intensive-Systems: Best
Practices from the Space System Domain.

Appendix II Best Practices

identification, analysis, and mitigation of risks. (3) Milestone reviews
include the status of identified risks. (4) The acquisition organization
has a written policy for managing acquisition risk. (5) Responsibility for
acquisition risk management activities is designated.

9. Solicitation

Purpose: To ensure that a quality solicitation is produced and a
bestqualified contractor selected.

Description: Solicitation is the process for developing, documenting, and
issuing the solicitation package; developing and implementing a plan to
evaluate responses; conducting contract negotiations; and awarding the
contract. Solicitation ends with contract award.

Activities: (1) The solicitation package includes the contractual
requirements and the proposal evaluation criteria. (2) The technical and
management elements of proposals are evaluated to ensure that the
requirements of the contract will be satisfied. (3) The selection official
selects a supplier who is qualified to satisfy the contract's
requirements. (4) The acquiring organization has a written policy for
conducting the solicitation. (5) Responsibility for the solicitation is
designated. (6) A selection official has been designated to be responsible
for the selection process and decision. (7) The acquiring team includes
contracting specialists to support contract administration.

10. Transition to Support

Purpose: To ensure proper transfer of the system from the acquisition
organization to the eventual support organization.

Description: Transition to support is the process for developing and
implementing the plans for transitioning products to the support
organization. This includes engaging relevant stakeholders in the
acquisition and sharing information about the system's supporting
infrastructure. Transition to support begins with requirements development
and ends when the responsibility for the products is turned over to the
support organization.

Activities: (1) The acquiring organization ensures that the support
organization has the capacity and capability to provide the required
support. (2) There is no loss in continuity of support to the products
during

                           Appendix II Best Practices

transition from the supplier to the support organization. (3)
Configuration management of the products is maintained throughout the
transition. (4) The acquiring organization has a written policy for
transitioning products to the support organization. (5) The acquiring
organization ensures that the support organization is involved in planning
for transition to support. (6) Responsibility for transition to support
activities is designated.

Complementary Best Practices Relevant to Commercial Component-Based IT
Business Systems Acquisitions

1. Component Modification

Purpose: To ensure that commercial product modification is effectively
controlled.

Description: Component modification is the process for limiting the
chances of a commercial product being modified to the point that it
becomes a one-of-a-kind solution because doing so can result in extensive
life cycle costs. Such modifications, if not incorporated into the
commercially available version of the product by the supplier, mean that
every product release has to be modified in accordance with the custom
changes, thus precluding realization of some of the benefit of using a
commercial product.

Activity: (1) Modification of commercial components is discouraged and
allowed only if justified by a thorough analysis of life cycle costs and
benefits.4

2. Configuration Management

Purpose: To ensure the integrity and consistency of system commercial
components.

Description: Configuration management relative to commercial
component-based systems is the process for ensuring that changes to the
commercial components of a system are strictly controlled. It recognizes
that when using commercial components, it is the vendor, not the
acquisition or support organization, that controls the release of new
component versions and that new versions are released frequently. Thus,

4Adams and Eslinger, "COTS-Based Systems: Lessons Learned from Experiences
with COTS Software Use on Space Systems." (Paper presented to the Southern
California SPIN: Oct. 6, 2003.)

Appendix II Best Practices

acquisition management needs to provide for both receiving new product
releases and controlling the implementation of these releases.

Activities: (1) Project plans explicitly provide for evaluation,
acquisition, and implementation of new, often frequent, product releases.5
(2) Modification or upgrades to deployed versions of system components are
centrally controlled, and unilateral user release changes are precluded.

3. Dependency Analysis

Purpose: To ensure that relationships between commercial products are
understood before acquisition decisions are made.

Description: Dependency analysis relative to commercial componentbased
systems is the process for determining and understanding the
characteristics of these products so that inherent dependencies among them
can be considered before they are acquired. It involves recognizing that
the logical and physical relationships among products impact one another.
This is necessary because commercial products are built around each
vendor's functional and architectural assumptions and paradigms, such as
approaches to error handling and data access, and these assumptions and
paradigms are likely to be different among products from different
sources. Such differences complicate product integration. Further, some
commercial products have built-in dependencies with other products that,
if not known, can further complicate integration.

Activity: (1) Decisions about the acquisition of commercial components are
based on deliberate and thorough research, analysis, and evaluation of the
components' interdependencies.6

4. Legacy Systems Integration Planning

Purpose: To ensure reasonable planning for integration of commercial
products with existing systems.

5Donald J. Reifer, Victor R. Basili, Barry W. Boehm, Betsy Clark,
"COTS-Based Systems- Twelve Lessons Learned about Maintenance."
(Presentation, 3rd International Conference on COTS-Based Software
Systems, Redondo Beach, CA, Feb. 4, 2004.)

6Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An
Activity Framework for COTS-Based Systems, Technical Report
CMU/SEI-2000-TR-010 (Pittsburgh, Pa.: Software Engineering Institute,
Carnegie Mellon University, October 2000).

Appendix II Best Practices

Description: Legacy systems integration planning is the process for
ensuring that the time and resources needed to integrate existing systems
with the system being acquired are identified and provided for. It
involves identifying which legacy systems will interact with the system
being acquired and what kinds and levels of testing are required.
Integration planning recognizes that, although some commercial products
may provide mechanisms and information that is helpful in integration with
legacy systems, the unavailability of the source code for commercial
products and the different organizations that are responsible for the two
will likely require additional time and effort.

Activity: (1) Project plans explicitly provide for the necessary time and
resources for integrating commercial components with legacy systems.

5. Organization Change Management

Purpose: To ensure that the organizational impact of using new system
functionality is proactively managed.

Description: Organization change management relative to commercial
component-based systems is the process for preparing system users for the
business process changes that will accompany implementation of the system.
It involves engaging users and communicating the nature of anticipated
changes to system users through training on how jobs will change. This is
necessary because commercial products are created with the developers'
expectations of how they will be used, and the products' functionality may
require the organization implementing the system to change existing
business processes.

Activities: (1) Project plans explicitly provide for preparing users on
the impact that the business processes embedded in the commercial
components will have on the user's respective roles and responsibilities.
(2) The introduction and adoption of changes to how users will be expected
to execute their jobs are actively managed.7

7Suzanne Garcia, John Roberts, and Len Estrin, "Managed Technology
Adoption Risk." (Presentation, 3rd International Conference on COTS-Based
Software Systems, Redondo Beach, CA, Feb. 2, 2004).

Appendix II Best Practices

6. Solicitation

Purpose: To ensure that a quality solicitation is produced and a
bestqualified contractor is selected.

Description: Solicitation relative to commercial component-based systems
is the process for ensuring that a capable contractor is selected. It
involves ensuring that the selected contractor has experience with
integrating commercial component products. This is important because
expertise in developing custom system solutions is different from
expertise in implementing commercial components; it requires different
core competencies and experiences to be successful.

Activity: (1) Systems integration contractors are explicitly evaluated on
their ability to implement commercial components.8

7. Tradeoff Analysis

Purpose: To ensure that system requirements alone do not drive the system
solution.

Description: Tradeoff analysis relative to commercial product-based
systems is the process for analyzing and understanding the tradeoffs among
competing acquisition variables so as to produce informed acquisition
decision making. It involves planning and executing acquisitions in a
manner that recognizes four competing interests: defined system
requirements, the architectural environment (current and future) in which
the system needs to operate, acquisition cost and schedule constraints,
and the availability of products in the commercial marketplace (current
and future). This analysis should be performed early and continuously
throughout an acquisition's life cycle.

Activity: (1) Investment decisions throughout a system's life cycle are
based on tradeoffs among the availability of commercial products (current
and future), the architectural environment in which the system is to

8Adams and Eslinger, "COTS-Based Systems: Lessons Learned from Experiences
with COTS Software Use on Space Systems." (Paper presented to the Southern
California SPIN: Oct. 6, 2003.)

Appendix II Best Practices

operate (current and future), defined system requirements, and acquisition
cost/schedule constraints.9

8. Vendor and Product Research and Evaluation

Purpose: To ensure that vendor and product characteristics are understood
before acquisition decisions are made.

Description: Vendor and product research and evaluation relative to
commercial component-based systems is the process for obtaining reliable
information about both the product being considered and the vendor
offering the product. It involves taking additional steps beyond vendor
representations, such as obtaining information about the vendor's history,
obtaining information on the vendor's business strategy relative to
evolution and support of the product, and evaluating copies of the product
in a test environment.

Activities: (1) Commercial component and vendor options are researched,
evaluated/tested, and understood, both early and continuously. (2) A set
of evaluation criteria for selecting among commercial component options is
established that includes both defined system requirements and
vendor/commercial product characteristics (e.g., customer satisfaction
with company and product line).

9Software Engineering Institute, Evolutionary Process for Integrating
COTS-Based Systems (EPIC): An Overview, CMU/SEI-2002-TR-009 (Pittsburgh,
PA: July 2002).

                                  Appendix III

                    Comments from the Department of Defense

Note: GAO comments supplementing those in the report text appear at the
end of this appendix.

Appendix III
Comments from the Department of Defense

                                 See comment 1.

                                 See comment 2.

                                 See comment 3.

Appendix III
Comments from the Department of Defense

                                 See comment 4.

                                 See comment 5.

Appendix III
Comments from the Department of Defense

                                 See comment 6.

                                 See comment 7.

                                 See comment 8.

Appendix III
Comments from the Department of Defense

                                 See comment 9.

                                See comment 10.

                                See comment 11.

                                  Appendix III
                    Comments from the Department of Defense

The following are GAO's comments on the Department of Defense's letter
dated July 1, 2004.

GAO Comments 1.

2.

3.

4.

We disagree. While we have updated our report to reflect the additional
information provided in DOD's comments on the status of its efforts and
the associated milestone, the importance of revising and maintaining DOD's
acquisition policies and guidance, and their incorporation of acquisition
best practices, makes it essential to have an explicit plan. Among other
things, a plan would highlight the resource constraints that this revision
effort has been subject to, would allow measurement against defined
milestones, and would allow disclosure of progress and impediments.

See comment 1.

We do not question DOD's statement that it has an initial repository of
custom software components that can help adapt commercial components to
defense use and reuse. However, this repository does not satisfy the
product line requirements in this best practice and our recommendation.
According to the Software Engineering Institute, the product line
requirements best practice involves more than just reuse. Under the
approach described in DOD's comments, reuse generally involves items, such
as software modules or components, that developers are encouraged to use.
However, the product line requirements best practice is not simply
encouraging reuse of items in a repository. Rather, it is planned,
enabled, and enforced reuse of such assets as requirements, models, and
architectures that have been designed and optimized for use in multiple
systems. In short, it is proactive rather than reactive reuse.

We have modified our recommendation to use the terminology "acquisition
project management activities" instead of "acquisition management
activities" to eliminate any confusion. Further, we do not question DOD's
use of integrated product teams as a way to communicate information.
Further, the point of our recommendation is that there needs to be an
explicit recognition in policy and guidance of the type of information to
be communicated and to whom. Incorporating this best practice is based on
the need to ensure that communication vehicles, such as integrated product
teams, are effective.

Appendix III
Comments from the Department of Defense

5.	We agree that the 5000 series contains information on acquisition risk
management, as we state in our report, and that one of the purposes of
DOD's acquisition framework is to reduce risks. However, there is no
provision in DOD's acquisition policy or guidelines to ensure that the
status of identified risks are discussed at key decision points. For
example, DOD policy states that a criterion for passing milestone A is
"management and mitigation of technology risk." However, it does not
provide for what is to be done to manage and mitigate risks, and it does
not provide for reviewing risk status at milestones B or C.

6.	While we agree that the toolkit provides relevant change management
information, the toolkit is referenced only once in the 5000 series; and
the reference is only in relation to one phase of the acquisition cycle
(the technology development phase). Given the importance and relevance of
this practice to the successful implementation of commercial
component-based systems, our position, and thus the basis for our
recommendation, is that the best practice's implementation would more
likely occur if it was visible and recognized in all relevant stages of
DOD's acquisition cycle.

7. See comment 6.

8.	While the regulations and guidance that DOD cited and referenced in its
acquisition policies discuss the use of information on contractors' past
performance, they do not discuss evaluating systems integration
contractors on their ability to implement commercial components, which is
the point of the best practice. Further, DOD's objection to incorporating
this best practice is not consistent with its own comments. Specifically,
DOD commented that it has already taken steps, through its enterprise
systems initiative, to establish blanket agreements with five contractors
who were evaluated on, among other things, "explicitly proven abilities to
implement commercial systems solutions." Additionally, while we appreciate
DOD's concern that incorporation of this best practice can have unintended
consequences, it is important to also recognize that our recommendation is
not intended to restrict a contracting officer's options. Rather, it is
intended to be one of the many factors considered in the source selection
process when it is relevant.

9.	We support the Defense Acquisition Executive's decision to add more
specifics on systems engineering to the 5000 series, including provisions
that address this best practice and recommendation.

Appendix III
Comments from the Department of Defense

10. See comment 8.

11. We disagree. While we do not question that statutory and regulatory
compliance are referenced in DOD's integrated process team and milestone
decision point processes, we do not believe that these reviews are
adequately defined with respect to implementation of best practices
because DOD's policy does not require that the practices' use be measured
and verified. Rather, it leaves these reviews to the discretion of the
program manager and investment decision authority. As we state in our
report, not requiring that the use of best practices be measured and
verified increases the chance that the practices will not be followed.
Therefore, our position remains that DOD's policies do not provide
effective controls for ensuring that best practices are appropriately
followed.

Appendix IV

                     GAO Contact and Staff Acknowledgments

GAO contact Carl L. Higginbotham, (404) 679-1824

Staff acknowledgments	In addition to the individual named above, key
contributors to this report included Nabajyoti Barkakati, Nancy Glover,
Madhav Panwar, Morgan Walts, and Thomas Wright.

GAO's Mission	The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost

is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to

www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone:	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Congressional	Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125

Relations Washington, D.C. 20548

Public Affairs	Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

                               Presorted Standard
                              Postage & Fees Paid
                                      GAO
                                Permit No. GI00

United States
Government Accountability Office
Washington, D.C. 20548-0001

Official Business
Penalty for Private Use $300

Address Service Requested
*** End of document. ***