Defense Acquisitions: Risks Posed by DOD's New Space Systems	 
Acquisition Policy (29-JAN-04, GAO-04-379R).			 
                                                                 
On November 18, 2003, we testified before Congress on the	 
Department of Defense's (DOD) new acquisition policy for space	 
systems. The new acquisition policy, issued in October 2003, sets
the stage for decision making for DOD's investment in space	 
systems, which currently stands at more than $18 billion annually
and is expected to grow considerably over the next decade.	 
Congress requested that we provide additional comments on several
issues relating to the new policy and other space acquisition	 
issues. 							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-04-379R					        
    ACCNO:   A09186						        
  TITLE:     Defense Acquisitions: Risks Posed by DOD's New Space     
Systems Acquisition Policy					 
     DATE:   01/29/2004 
  SUBJECT:   Competition					 
	     Competitive procurement				 
	     Defense procurement				 
	     Procurement planning				 
	     Strategic planning 				 
	     Computer software					 
	     Weapons systems					 
	     Cost estimates					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-04-379R

United States General Accounting Office Washington, DC 20548

January 29, 2004

The Honorable Wayne Allard
Chairman
The Honorable Bill Nelson
Ranking Minority Member
Subcommittee on Strategic Forces
Committee on Armed Services
United States Senate

Subject: Defense Acquisitions: Risks Posed by DOD's New Space Systems
Acquisition Policy

On November 18, 2003, we testified before the Subcommittee on the
Department of Defense's (DOD) new acquisition policy for space systems.
The new acquisition policy, issued in October 2003, sets the stage for
decision making for DOD's investment in space systems, which currently
stands at more than $18 billion annually and is expected to grow
considerably over the next decade. You requested that we provide
additional comments on several issues relating to the new policy and other
space acquisition issues. Your specific questions and our answers are
discussed below.

1. 	The Air Force maintains that its Defense Space Acquisition Board
(DSAB) process allows earlier identification of problems and senior level
attention, which will improve management and lower risk. Does GAO have any
concerns with the DSAB process?

Our concern is not with earlier identification of problems or the added
senior level attention the new process calls for, but with earlier
investment decisions, which are also called for. Under the new process,
the DSAB may approve product development to begin before DOD knows whether
technologies can work as intended. As a result, it will make major
investment commitments without really knowing what resources will be
required to deliver promised capability. Moreover, the policy encourages
development of leading-edge technology within product development, that
is, at the same time the program manager is designing the system and
undertaking other product development activities. DOD believes this
approach will allow space systems to better incorporate leading-edge
technologies. But as our work has

repeatedly shown, such concurrency within space and other weapon system
programs increases the risk that significant problems will be discovered
as the system is integrated and built, when it is more costly and
time-consuming to fix them.

Moreover, as we testified, the knowledge-building approach for space
stands in sharp contrast to that followed by successful programs and the
approach recommended by DOD's revised acquisition policy for weapon
systems. Successful programs will not commit to undertaking product
development unless they have high confidence that they have achieved a
match between what the customer wants and what the program can deliver.
Technologies that are not mature continue to be developed in an
environment that is focused solely on technology development. This system
puts programs in a better position to succeed because they can focus on
design, system integration, and manufacturing. Further, our work has shown
that taking an evolutionary approach to improving capability increases the
likelihood of delivering that capability to the war fighter sooner than
the revolutionary approach the Air Force continues to support in the new
space policy.

2. 	Does GAO believe that the process put into place in the new space
acquisition policy by which cost estimates are derived will provide better
cost estimates?

No. Although some process changes will be made, the underlying causes of
underestimating costs remain.

DOD is adopting new methodologies and tools to enhance cost estimates, and
it is enlisting assistance from DOD's Cost Analysis Improvement Group
(CAIG) to conduct independent cost estimates using cost estimating teams
drawn from a broad spectrum of the cost-estimating community. Moreover,
programs are now required to resolve differences between their cost
estimates and estimates produced by the independent teams. In the past,
cost-estimating groups have developed estimates that were different,
leaving decision makers to select one estimate or combine a few.

However, under the new space acquisition policy, cost estimates do not
have to be based on the knowledge that technologies can work as intended.
History has shown that cost estimates not based on such knowledge are
significantly understated. Moreover, incentives that work against
providing good estimates have not changed. Unlike the commercial world
where the focus is on delivering a product to market, DOD's system focuses
on competing for resources from oversubscribed budgets. In the competition
for funding, managers are encouraged to launch product developments before
technologies are mature. Because funding is competitive and DOD's
forecasts of costs, schedules, and performance are largely based on
immature technologies and other unknowns, estimates tend to be squeezed
into insufficient profiles of available funding. In fact, pressures to
underestimate costs may increase over the next decade as DOD plans to
undertake a number of new, challenging space programs-which are expected
to require an additional $4 billion in the next 4 years alone. Costs
beyond that period are as yet unknown but are likely to be considerably
higher.

3. What is GAO's view on the Air Force policy related to full funding?

DOD's acquisition policy for other weapon systems requires a commitment
for full funding at milestone B-the start of product development and the
point at which DOD should have knowledge that technologies can work as
intended. However, the new space acquisition policy does not require DOD
to commit to fully fund a space program either when this knowledge has
been obtained or at any point in the development process. Hence, there is
no guarantee that the resources needed to meet requirements on any
individual program will be there when needed- particularly as DOD moves
forward with its new programs.

This represents another important departure from the development approach
followed by successful programs. Our prior work1 has found that if a
product's business case measures up, that is a company is assured that
there is a market or need for the product and that it has the right
knowledge in hand to develop the product with firm cost and schedule
estimates, the company then commits to the entire development of the
product, including the financial investment. In other words, corporate
resources are made available to the development team so that product
success is not compromised. As noted earlier, because DOD begins too many
programs, its resources are always oversubscribed. By requiring program
managers to continually justify funding, DOD runs a risk of foreclosing
the ability for sound planning and execution.

4. 	The Young Panel was not convinced of the merits of competition in some
circumstances, particularly when the incumbent has performed well and
"owns" the expertise and the government would incur significant cost in
choosing another contractor for follow-on systems. Does GAO have a view on
the merits or demerits of competition in space programs?

Competition can provide natural incentives for an organization to be more
efficient and more innovative. These incentives work in DOD's favor.
However, it is also important to recognize that competition can take
various forms. For example, DOD can increase competition by using shadow
contractors, pursuing alternative sensor designs, and breaking
acquisitions into smaller blocks. DOD can also optimize its investment in
weapon systems by competing air, land, sea, and space-based capabilities.
By pursuing these various options, DOD would have greater assurance that
it is obtaining the best value when it must select a prime contractor for
follow-on systems.

5. 	How effective can competition be when we have so few major contractors
capable of executing large and complex space programs?

While there are only a few contractors currently capable of implementing
large and complex space programs, there are many more capable of building
specific satellite components and technologies. Thus, by increasing
competition at the mission payload or sensor level and breaking
acquisitions into smaller pieces, DOD can

1 U.S. General Accounting Office, Best Practices: A More Constructive Test
Approach Is Key to Better Weapon System Outcomes, GAO/NSIAD-00-199
(Washington, D.C., July 31, 2000).

expand the universe of contractors competing for work. Over the long run,
this could enable more contractors to build the expertise and knowledge
needed to manage large space programs. It would also require DOD to have
significant insight into the lower tiers of the industry.

6. 	Is there a path to making competition a useful element in healthy
programs?

Managing the industrial base is one of the most critical determinants of
acquisition success. According to DOD studies, this not only means
injecting competition early on to ensure that the highest performing and
most cost-effective technologies and designs are being pursued, but
adequately defining work; establishing shorter, more manageable contract
periods; and providing the right incentives for contractors. Following an
evolutionary development path would better enable programs to take these
kinds of actions. It would also foster a healthier industrial base because
it would get programs into production sooner. Also important is ensuring
that programs have the right capability to evaluate contractor proposals
and to manage the contracts once they are in place. As DOD's studies of
space programs show, the government will invariably encounter problems
when too much responsibility is handed over to contractors and too little
oversight is provided.

We have also found that the path to healthier programs is characterized by
having an open systems design. Such a design is characterized by (1) well
defined, widely used, preferably nonproprietary interfaces and protocols
between systems, subsystems, and components and (2) an explicit provision
for system expansion or upgrade through incorporation of additional higher
performance subsystems and components with minimal negative impact on the
existing system. Open systems design allows competing developers to offer
additional features and capabilities. With this approach, the government
might be able to minimize dependence on a specific contractor. Also,
upgrades can be added without replacing the entire system. Costs across
the board--development, production, operations, and support--can thereby
be reduced.

7. 	Does GAO believe that space programs will be less schedule driven
under the new acquisition policy?

No. In the past, DOD has taken a schedule-driven versus a knowledge-driven
approach to the acquisition process for space and other weapons systems
with the justification that capabilities were urgently needed. In other
words, commitments were made to achieving certain capabilities without
knowing whether technologies being pursued could really work as intended.
As a result, time and costs estimates were consistently exceeded, and
steps essential to containing costs, maximizing competition among
contractors, and testing technologies were shortchanged. Perversely,
programs actually took longer when rushed at the start. Moreover, DOD
often lacked assurance that it was even pursuing the best technical
solution because alternatives were not analyzed or they were eliminated in
order to meet schedule pressures. When technology did not perform as
planned, assigning additional resources in terms of time and money became
the primary option for solving problems, since customer expectations about
the products' performance already became hardened.

The new space acquisition process does not change this approach or the
incentives that drive it. Rather, it encourages programs to enter into
product development without knowledge that technologies can work as
intended. Moreover, for new programs like the Transformational Satellite
(TSAT) and Space Based Radar (SBR), DOD is still setting initial satellite
launch dates before this knowledge has been obtained. By contrast, DOD's
acquisition policy for non-space systems establishes mature
technologies-that is, technologies demonstrated in a relevant environment-
as critical before entering product development. By encouraging programs
to do so, the policy for non-space systems puts programs in a better
position to deliver capability to the war fighter in a timely fashion and
within funding estimates because program managers can focus on the design,
system integration, and manufacturing tasks needed to produce a product.

8. 	The requirements for the Space-Based Infrared System High system
(SBIRS High) still continue to change. In GAO's report, you highlight
several examples, including batteries and solar cell panels. From the
report, I gather that GAO finds that the Air Force's efforts to limit
requirements changes to only those that are "urgent and compelling" are
better, but that they are not successfully eliminating the growth of
requirements. You mention at least $203 million in new requirements. Is
this a correct interpretation of the new "urgent and compelling approach?"

Prior to the restructuring, the SBIRS High program office exerted no
control over requirements changes, leaving many decisions on requirements
to its contractors or within lower management levels of the program
office. As part of the SBIRS High program restructuring, the Air Force
established an advisory program management board to oversee requirements
changes. The board's role is to ensure that new requirements are urgent
and compelling, that they reflect an appropriate use of funds, and that
decisions about requirements are more transparent. Air Force leadership,
not the SBIRS High program office, made the decision that the new
requirements were urgent and compelling enough to address.

We believe that establishing the board is a positive step and should help
manage requirements changes more effectively. Nevertheless, the board will
still be challenged to ensure some discipline in requirements setting,
since there is a diverse group of Air Force and other DOD users that have
an interest in SBIRS High and there are increasing demands for
surveillance capabilities. Currently, there are several proposed
requirements changes on the table that could have a significant impact on
the program.

9. 	The GAO report also indicates that software development problems
continue to be a problem. This problem is not limited to SBIRS High,
however. What recommendations can you make to address this continuing
problem?

Problems with software development in DOD weapons systems are well known.
For example, the Defense Science Board reviewed selected DOD software
intensive

systems and found that programs lacked well thought-out, disciplined
program management and/or software development processes. The programs
lacked meaningful cost, schedule, and requirements baselines, making it
difficult to track progress against them. These findings are echoed by the
work of DOD's Tri-Service initiative. Because weapon systems are becoming
increasing dependent on software, lax management and oversight over
software development can be detrimental to a program, as it was for SBIRS
High.

There are steps we have identified in an ongoing review for the Senate
Committee on Armed Services that DOD could take to address this problem.
Chief among them is to require programs to apply best practices for
software development and acquisition, many of which have been identified
by the Software Engineering Institute at Carnegie Mellon University and
packaged into continuous improvement models and guidance. In adopting
these models, organizations would take a more disciplined and rigorous
approach toward managing or overseeing software development. At the same
time, organizations need to provide the right environment to reduce
software development risk. This means establishing an environment
comprised of an evolutionary software development approach that relies on
well-understood, manageable requirements and a desire to continuously
improve development processes. It also means adopting and using a host of
metrics to track cost and scheduling deviations; requirements changes and
their impact on software development efforts; testing efforts; as well as
efforts to detect and fix defects. Also important is to integrate these
practices into existing acquisition policies and improvement plans as well
as to enforce the use of these practices within individual programs.

10. The GAO conclusion is that SBIRS High is still a program in trouble.
To remedy this problem, GAO recommends that the Secretary reconvene the
independent review team, or a similar body, to provide an assessment of
the restructured program and concrete guidance for addressing the
program's underlying problems. To play devil's advocate for a moment, how
will another review of this program improve its chances of technical,
budget, and schedule success?

The fundamental problem with the SBIRS High program has been the failure
to develop key knowledge at critical junctures early in the development of
the system, that is, before major investments were made. The program is
now paying the price for this lack of knowledge development. Although the
restructuring of the program in 2002 improved management and oversight
capabilities, it did not go far enough in addressing the underlying
problems with system design, integration, and software development.
Another independent and in-depth technical review of the program is
important to ensure that these problems are more clearly understood and
that there are no other hidden problems lurking. At the same time, such a
review will keep attention focused and heighten oversight of the program.
Moreover, until it becomes standard to make knowledge-based decisions on
DOD programs, ad hoc reviews such as the one we call for may be the only
way to bring transparency to the decision making process.

11. SBIRS High is clearly a highly visible troubled program. How
representative is it of space programs in general? Is it unique or are the
problems identified present throughout the space acquisition effort?

We recently reported2 that the majority of satellite programs over the
past couple decades, like SBIRS High, cost more than expected and took
longer to develop than planned. SBIRS High is one of the few weapon
systems programs to exceed the 25 percent cost threshold established in 10
U.S.C. 2433, but the problems affecting other programs have been equally
dramatic. For example, cost estimates for the Advanced Extremely High
Frequency (AEHF) communications satellite program grew by $1.2 billion
from 1999 through 2001, while the program experienced a 2-year delay in
the launch of the first satellite. And while DOD has spent several billion
dollars over the past 2 decades to develop low-orbiting satellites that
can track ballistic missiles throughout their flight, it has not launched
a single satellite to perform this capability.

A key underlying problem with many programs has been the desire to achieve
revolutionary advancements in capability instead of evolutionary
advancements. Such an approach meant that requirements exceeded resources
(time, money, and technology) at the time of product development, setting
the stage for costly and timeconsuming rework later in the program. More
specifically, in reviewing our past reports, we found that: (1)
requirements for what the satellite needed to do and how well it must
perform were not adequately defined at the beginning of a program or were
changed significantly once the program had already begun; (2) investment
practices were weak, e.g., cost estimates were optimistic or potentially
more costeffective approaches were not examined; (3) acquisition
strategies were poorly executed, e.g., competition was reduced for the
sake of schedule or DOD did not adequately oversee contractors; and (4)
technologies were not mature enough to be included in product development.
All of these problems affected SBIRS High and AEHF. One or more affected
the Space Tracking and Surveillance System (STSS) and the predecessor
SBIRS programs as well as Milstar, the Global Positioning System (GPS),
and the National Polar-orbiting Operational Environmental Satellite System
(NPOESS).

Because DOD took a schedule-driven approach instead of a knowledge-driven
approach to the acquisition process, activities essential to containing
costs, maximizing competition among contractors, and testing technologies
were compressed or not done. Like SBIRS High, many programs also
encountered problems in setting requirements due to the diverse array of
organizations with competing interests involved in overall satellite
development-from the individual military services, to testing
organizations, contractors, civilian agencies, and in some cases
international partners. Requirements setting for SBIRS High was
particularly problematic because the government put too much
responsibility on its contractors to balance these competing interests-a
problem recognized in DOD's own study of SBIRS High and other studies of
space acquisition problems.

2 U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions,
GAO-03-825R (Washington, D.C.: June 2, 2003).

In our view, new programs like the Transformational Satellite (TSAT) will
likewise be unable to make a match between needs and resources at the
onset of product development because DOD's new space acquisition policy
encourages product development to begin without knowing that technologies
can work as intended to meet capability needs.

In preparing answers to your questions, we relied on our prior work on
DOD's new
space acquisition policy,3 best practices in weapon system acquisitions,
and our
reviews of specific space acquisitions. Because we relied on previously
issued work,
we did not obtain comments from DOD on a draft of this letter. We
conducted our
work from December 2003 through January 2004 in accordance with generally
accepted government auditing standards.

We are sending copies of this letter to the Secretaries of Defense and the
Air Force
and interested congressional committees. We will also make copies
available to
others upon request. In addition, the report will be available at no
charge on the GAO
Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please
contact either
me or Katherine Schinasi at (202) 512-4841. Key contributors to this
letter were
Cristina Chaplain, Sigrid McGinty, Art Gallegos, Maricela Cherveny, John
Oppenheim,
and Mike Hazard.

Robert E. Levin
Director, Acquisition and Sourcing Management

(120318)

3U.S. General Accounting Office, Defense Acquisitions: Improvements Needed
in Space Systems Acquisition Management Policy, GAO-03-1073 (Washington,
D.C.: September 15, 2003) and Defense Acquisitions: Improvements Needed in
Space Systems Acquisition Policy to Optimize Growing Investment in Space,
GAO-04-253T (Washington, D.C.: November 18, 2003).

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

The General Accounting Office, the audit, evaluation and investigative arm
of

GAO's Mission 	Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and full-

Obtaining Copies of GAO Reports and Testimony

text files of current reports and testimony and an expanding archive of
older products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO email this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud,	Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm

                  Waste, and Abuse in E-mail: [email protected]

Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470

Public Affairs 	Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***