Defense Acquisitions: Space-Based Radar Effort Needs Additional
Knowledge before Starting Development (19-JUL-04, GAO-04-759).
Missing among the Department of Defense's (DOD) portfolio of
systems is a capability to track stationary and moving enemy
vehicles on land or at sea in any type of weather, day or night,
from space. To meet this need, DOD and the intelligence community
are collaborating on the ambitious Space-Based Radar (SBR)
program. By leveraging the newest generation of radar
technologies, the SBR concept promises to deliver high-quality
data to a wide array of users. DOD intends to start product
development in 2006 and to field SBR satellites as quickly as
possible so that warfighters, the intelligence community, and
national decision makers can gain a better understanding of what
adversaries are doing in specific locations around the world. GAO
reviewed the SBR program to assess DOD's progress in attaining
the knowledge it needs by 2006 in terms of customer needs (or
requirements) and resources.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-759
ACCNO: A10956
TITLE: Defense Acquisitions: Space-Based Radar Effort Needs
Additional Knowledge before Starting Development
DATE: 07/19/2004
SUBJECT: Aerospace research
Defense capabilities
Defense procurement
Federal intelligence agencies
Information technology
Military intelligence
Radar equipment
Satellites
Weapons research and development
Information resources management
Program management
Performance measures
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-759
United States Government Accountability Office
GAO Report to Congressional Committees
July 2004
DEFENSE ACQUISITIONS
Space-Based Radar Effort Needs Additional Knowledge before Starting Development
The Web version of this report was reposted on July 26, 2004, to reflect
congressional developments since the original version was posted July 23,
2004. The following items were revised to 1) add new addressees to the
report on (page 1), which shifted text throughout;
2) add a sentence to reflect the appropriations committee's action on the
SBR program on July 21, 2004 (page 7); and 3) add a sentence to the
introduction explaining that this work was done under the Comptroller
General's authority and clarifying why we were addressing the report to
the committees (page 2).
The printed version reflects these changes.
a
GAO-04-759
Highlights of GAO-04-759, a report to congressional committees
Missing among the Department of Defense's (DOD) portfolio of systems is a
capability to track stationary and moving enemy vehicles on land or at sea
in any type of weather, day or night, from space. To meet this need, DOD
and the intelligence community are collaborating on the ambitious
Space-Based Radar (SBR) program. By leveraging the newest generation of
radar technologies, the SBR concept promises to deliver high-quality data
to a wide array of users. DOD intends to start product development in 2006
and to field SBR satellites as quickly as possible so that warfighters,
the intelligence community, and national decision makers can gain a better
understanding of what adversaries are doing in specific locations around
the world. GAO reviewed the SBR program to assess DOD's progress in
attaining the knowledge it needs by 2006 in terms of customer needs (or
requirements) and resources.
Before committing to SBR's acquisition program in 2006, GAO recommends
that DOD and intelligence partners close gaps in the requirements approval
process in terms of documenting decisions and be prepared to add time and
money or make trade-offs with other DOD space programs to address SBR's
requirements and resources. DOD generally agreed with our findings and
partially agreed with our recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-759.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Katherine V. Schinasi at
(202) 512-4841 or [email protected].
July 2004
DEFENSE ACQUISITIONS
Space-Based Radar Effort Needs Additional Knowledge before Starting Development
Although SBR is 2 years away from product development, the program already
faces major challenges. DOD officials say SBR will likely be the most
expensive and technically challenging space system ever built by DOD. The
acquisition time frame is much shorter than what has been achieved in the
past for other complex satellite systems. Finally, DOD is setting
precedence by taking the lead on developing SBR with the intelligence
community as a partner. Most DOD space programs that GAO has reviewed in
the past several decades were hampered by schedule and cost growth and
performance shortfalls. Problems were largely rooted in a failure to match
requirements with resources when starting product development. Commitments
were made without knowing whether technologies being pursued would work as
intended. To avoid these problems, leading commercial firms have adopted a
knowledge-based model that enables decision makers to be reasonably
certain about their products at critical junctures and helps them make
informed investment decisions.
Although DOD has taken positive steps to strengthen the involvement of
senior leaders within DOD and the intelligence community in setting
requirements, SBR's concept of operations has not been approved and signed
by requirements boards for either of the two partners. Without
documentation and formal approval, it is unclear who will be held
accountable for setting requirements or how disagreements among SBR's
partners will be resolved when DOD moves SBR into ensuing phases of
acquisition.
DOD has adopted noteworthy practices to gain knowledge about SBR's
resources. These include maximizing the use of systems engineering to
close gaps between requirements and resources; estimating all of SBR's
costs; exploring alternatives for SBR if the Transformational
Communications Architecture (TCA)-the communications infrastructure that
is expected to relay SBR data across a network of users-incurs schedule
and performance shortfalls; and asking contractors to propose multiple
operations concepts for SBR with or without TCA. Despite these
accomplishments, DOD is at risk of knowledge gaps. SBR's critical
technologies will not be mature when product development starts, as called
for by best practices. One of TCA's primary components may not be ready in
time to support SBR data. These knowledge gaps make it harder for DOD to
reliably estimate how much time and money are needed to complete SBR's
development. If TCA is delayed, DOD's alternatives may involve reducing
SBR's capabilities or significantly increasing program cost. Without
sufficient knowledge, DOD may not be able to determine by the time SBR's
product development starts in 2006 whether space-based radar is best
suited to tracking moving targets on land or at sea or whether air-based
radar would provide enough capabilities at far less cost. More specific
analyses would help DOD weigh the merits of various alternatives and
assess how much to invest in the SBR acquisition program versus air
platforms with similar capabilities.
Contents
Letter
Scope and Methodology
Results in Brief
Background
Gaining Knowledge about Requirements and Resources before
Product Development Is Important for Space Acquisition Success DOD Moving
Forward on Acquiring Critical Knowledge but Gaps Remain in Approval for
SBR Requirements DOD Taking Proactive Steps to Gain Knowledge about
Resources,
but Critical Gaps May Remain at Product Development Conclusions
Recommendations for Executive Action Agency Comments and Our Evaluation 1
3 4 6
7
9
15 22 23 24
Appendix I TRL Scale for Assessing Critical Technologies
Appendix II Comments from the Department of Defense
Tables
Table 1: Knowledge Provided by SBR's Initial Capabilities
Document and Concept of Operations 14
Table 2: Technology Readiness Levels of SBR Critical 18
Technologies
Figures
Figure 1: SBR's Acquisition Schedule (in fiscal years) 7
Figure 2: SBR's Oversight Structure 11
Figure 3: Requirements-Setting Can Impact Acquisition Process 12
Abbreviations
AOA analysis of alternatives
DOD Department of Defense
R&D research and development
SBR Space-Based Radar
TCA Transformational Communications Architecture
TRL Technology Readiness Level
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office Washington, DC 20548
July 23, 2004
The Honorable John Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Ted Stevens
Chairman
The Honorable Daniel Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Duncan Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives
The Honorable Jerry Lewis
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives
The Department of Defense (DOD) currently has ground-based and
air-based radars but no capability to track moving targets from space.
To meet a need for persistent global observation, DOD and the intelligence
community are working together to develop the Space-Based Radar (SBR)
system to find, identify, track, and monitor ground and sea targets-
mobile or immobile-under all-weather conditions and on a near-continual
basis across large swaths of the earth's surface. SBR is to enhance
information gathering by providing intelligence, surveillance, and
reconnaissance data in a meaningful and timely manner.
This is an ambitious program for DOD. DOD's initial total cost estimate
for SBR is about $28.6 billion from fiscal year 2003 to 2024. And although
the technologies for tracking moving targets from space are still in
development, DOD is scheduling almost 7 years between the start of product
development and launch of the first SBR satellite-a time frame that is
considerably shorter than what has been achieved in the past for other
complex satellite systems. DOD would like to field SBR satellites as
quickly as possible because it believes that SBR represents a major leap
forward in providing the warfighters, intelligence community, and national
decision makers with significant tactical, operational, and strategic
advantages over potential adversaries. For example, DOD envisions that SBR
will be able to see deep inside enemy territory without risk to personnel
or resources and that it will operate over areas where conventional
airborne surveillance systems are at risk from the enemy's surface-to-air
missiles, which can travel increasingly longer ranges. DOD also envisions
that SBR will generate high-quality radar imagery of targets and terrain
and interface with ground, air, and other space systems so that users can
gain a better understanding of what is occurring in specific locations.
DOD is in the early exploratory phase of the SBR program-a period when it
gathers knowledge about its needs, or requirements, and explores the
feasibility of meeting those requirements. DOD expects to begin product
development in fiscal year 2006; at which point it will make a commitment
to invest in a formal acquisition program for SBR. Our past work has shown
that successful weapon system programs are able to match their needs to
their resources-that is, money, technology, and time-before product
development. With achievable requirements and commitment of sufficient
investment to complete development, programs are better able to deliver
products at cost and on schedule. Most space programs over the past
several decades have not been able to achieve a match between needs and
resources before product development and have incurred significant cost
and schedule increases due in part to the need to rework technologies in
the later stages of their acquisition.
We conducted our review of the SBR program on the initiative of the
Comptroller General. Given SBR's overall importance to DOD and the
intelligence community, we reviewed the SBR program to assess DOD's
progress in attaining the knowledge it needs by 2006 in terms of (1)
requirements and (2) resources-technology, communications infrastructure,
and funding.
Scope and Methodology
We are addressing this report to you because of your jurisdiction over
weapon systems acquisition.
To assess DOD's progress in attaining the knowledge it needs before the
start of product development, we examined the resources (technology,
communications infrastructure, and funding) committed and planned for the
program as well as the users' needs for an SBR system. We considered DOD's
plans for maturing the critical technologies when we obtained
technology-readiness information for each critical technology (as well as
its mature backup technology) against best practice standards to determine
if they will be sufficiently mature when DOD plans to start product
development. We also reviewed the SBR risk management plans and concept
development contract information. We discussed these documents and issues
with representatives from Air Force Space Command, Peterson Air Force
Base, Colorado; and the SBR Joint Program Office, Space and Missile
Systems Center, Los Angeles Air Force Base, California.
To determine SBR's role in a larger DOD architecture, we met with
officials from the Joint Chiefs of Staff, Washington, D.C.; and the Air
Force Directorate of Space Acquisitions, Arlington, Virginia. We also
consulted past GAO reports to determine the relationship between SBR and
the Transformational Communications Architecture.
To determine the scope and completeness of the analysis of alternatives
and its follow-on study to identify the optimal ways to gather information
on ground moving targets from radars based in space versus air, we met
with officials from Air Force Space Command; DOD Office of the Director,
Program Analysis and Evaluation, Washington, D.C.; Air Force Directorate
of Requirements for Space, Crystal City, Virginia; and the Air Force
Studies and Analyses Agency, Arlington, Virginia. We also talked with an
official from the Air Force Office of Aerospace Studies, Kirtland Air
Force Base, New Mexico.
We discussed overarching programmatic issues-including the level of
coordination between DOD and the intelligence community-with
representatives from the Air Force Directorate of Space Acquisitions. We
were not able to obtain meetings with members of the Mission Requirements
Board (a board within the intelligence community responsible for approving
program requirements) or the intelligence agencies to discuss their stake
in the SBR program.
Results in Brief
We performed our work from November 2003 through June 2004 in accordance
with generally accepted government auditing standards.
DOD has inserted a great deal of management and stakeholder involvement
into developing SBR so that it can gain greater knowledge about
requirements before product development than it has in past programs.
Senior-level officials from DOD, the military services, and the
intelligence community are heading up three new SBR oversight groups to
discuss what can and cannot be accomplished in terms of the desired time
frame, available funding, and achievable technologies. The groups have so
far attained informal agreement on requirements. However, this commitment
has not been formalized, and it is unclear as to whether or how it will be
formalized. Moreover, it is also unclear how disagreements that may occur
later on among SBR's partners will be resolved. Given the varied interests
of SBR's partners and past problems with securing agreement on
requirements for space programs,1 it is important that DOD build on the
positive steps it has already taken and find ways to formalize commitment
to requirements as well as SBR's concept of operations.
DOD is also taking positive steps in its effort to gain knowledge about
SBR's resources. These steps include strengthening systems engineering
applications; estimating not just direct costs but all of SBR's life-cycle
costs; exploring alternatives for SBR if the new Transformational
Communications Architecture (TCA), the infrastructure that is to help
relay SBR data, falls short of its schedule and performance goals; and
asking concept development contractors to propose multiple design concepts
for SBR with or without TCA. Despite these accomplishments, however,
decision makers will have significant knowledge gaps about SBR's resources
if SBR's product development phase starts when currently planned. This is
because DOD intends to start product development in fiscal year 2006 even
though the two critical technologies that would enable the tracking of
surface-moving targets and the timely delivery of imagery data will not
have been tested in space or even in a relevant environment. Without the
knowledge derived from such tests, DOD cannot adequately assess whether
the technologies will work as intended, making it harder to reliably
estimate how much time and money
1 U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions,
GAO-03-825R (Washington, D.C.: June 2, 2003).
is needed to complete development. Adding to SBR's overall risk are
uncertainties as to whether TCA will be available to transmit SBR's vast
volumes of radar data and imagery. Although DOD has plans underway to
develop alternatives for SBR if TCA is delayed, the primary alternatives
would involve either reducing SBR's capabilities or significantly
increasing program cost. Lastly, DOD and the Air Force may not have
sufficient knowledge to make a corporate decision as to how much it should
invest in space-based radar capabilities versus air-based capabilities at
the time it makes a commitment to the formal SBR acquisition in 2006.
While the Air Force has undertaken two analyses of the SBR investment,
neither presents a comprehensive assessment as to whether space- or
air-based radar platforms (or a combination of both) are better suited for
tracking moving targets on land or at sea, nor does either analysis weigh
the capabilities and costs of each suitable radar option in space and in
the air.
Before committing to SBR's acquisition program in 2006, we recommend that
senior-level officials in DOD and the intelligence community close gaps in
the requirements-setting process in terms of documenting decisions and
establishing a formal mechanism for addressing unresolved issues or how
changes to approved requirements will be assessed. We recommend that DOD's
space acquisition policy be modified to identify the requirements-setting
process when DOD partners with the intelligence community on space
programs. We recommend that DOD delay approval to commit funding to
product development for SBR until technologies are matured to higher
levels, but if DOD determines that the SBR program should proceed to
product development with less than mature technologies, we recommend that
DOD be prepared to add time and money or make trade-offs with other DOD
space programs to address SBR's requirements and resources. DOD generally
agreed with our findings and our recommendation to strengthen its study of
SBR alternatives. DOD partially agreed with our recommendations to
strengthen its requirement setting process for SBR and to demonstrate SBR
technologies in a relevant or operational environment before committing to
product development. DOD did not agree with our recommendation to modify
its acquisition policy to strengthen requirements setting. In commenting
on our recommendations, DOD cited concerns about supplanting current
requirements setting processes. Our recommendation does not advocate
replacing this process, but rather strengthening it to provide more
transparency, discipline, and accountability.
Background
SBR represents the first time that DOD has taken the lead on developing a
major national security space capability with the intelligence community
as a partner. Because of this partnership, SBR's acquisition process is
more complex than that used for typical DOD programs. While DOD and the
intelligence community will likely use all the data that SBR produces,
their priorities differ. DOD's warfighting community is particularly
interested in tracking targets moving over land or sea as well as other
objects of interest. The intelligence community is more focused on
obtaining detailed global imagery and combining it with other data for
advanced processing. SBR is expected to meet both needs and be fully
integrated with other space and non-space systems, including TCA, which is
to transmit SBR's data to receivers in the air, at sea, or on the ground.
A key advantage of radar in space is having the ability to "see" through
clouds and sand storms and any type of weather, day or night.
Radar-equipped aircrafts, on the other hand, require U.S. air dominance to
collect radar information and must steer clear of hostile areas-the result
being limited radar coverage. The SBR concept offers other added features,
including electronic steering of the radar signal toward a particular area
and capturing high volumes of very fine resolution radar images of targets
and terrain. With the ability to perform these functions almost
simultaneously, SBR is expected to help analysts gain a better
understanding of what is occurring in specific locations.
To help meet some of its goals, DOD plans to leverage key technologies
that were developed in the late 1990s to demonstrate a space-based radar
capability. According to DOD officials, contractors developed some
satellite hardware and prototype components under the Discoverer II
program, which began in 1998 and was to identify and validate by 2008 the
capability of tracking mobile ground targets from space. Discoverer II,
comprising two radar demonstration satellites, was a joint initiative by
the Air Force, DOD's Defense Advanced Research Projects Agency, and the
intelligence community's National Reconnaissance Office. DOD officials
told us that the Discoverer II program had reached the preliminary design
review phase when it was cancelled in 2000 because of cost and schedule
uncertainties, poorly explained requirements, and the lack of a coherent
vision to transition the system to operational use.
The Secretary of Defense concluded that space-based radar could provide a
military advantage and in 2001 approved SBR as a new major defense
acquisition program, delegating it to the Air Force. In July 2003, an
independent cost assessment team consisting of representatives from DOD
and the intelligence community estimated that $28.6 billion would be
needed to pay for SBR's life-cycle costs-development, production, launch,
and operation. The program entered the study phase in August 2003. The Air
Force has requested $328 million for SBR in fiscal year 2005 and has
programmed about $4 billion for the program from fiscal years 2005 to
2009. Given concerns about affordability and readiness, the Fiscal Year
2005 Defense Appropriations Conference Report reduced funding for SBR to
$75 million, with the direction to return this effort back to the
technology development phase. In 2003, Congress reduced the Air Force's
$274 million budget request for SBR by $100 million due to concerns about
technology maturity and schedule. DOD has scheduled the start of product
development for mid-fiscal year 2006, with production starting at the end
of fiscal year 2008 and the first satellite to be launched at the end of
fiscal year 2012. Figure 1 shows SBR's acquisition schedule in fiscal
years.
Figure 1: SBR's Acquisition Schedule (in fiscal years)
Sources: Air Force (data); GAO (analysis).
Gaining Knowledge about Requirements and Resources before Product Development
Is Important for Space Acquisition Success
In the past several decades, DOD's space acquisitions have experienced
problems that have driven up costs by hundreds of millions, even billions,
of dollars; have stretched schedules by years; and have increased
performance risks. In some cases, capabilities have not been delivered to
the warfighter after decades of development. Our reports have shown that
these problems, common among many weapon acquisitions, are largely rooted
in a failure to match the customer's requirements (desired capabilities)
with the developer's resources (technical knowledge, timing, and funding)
when starting an acquisition program.
In particular, our past work has shown that for space systems, product
development was often started based on a rigid set of requirements that
proved to be unachievable within a reasonable development time frame.
Other cases involved unstable requirements. In some cases where
requirements had been identified and approved, even more requirements were
added after the program began. When technology did not perform as planned,
adding resources in terms of time and money became the primary option for
solving problems because the customer's expectations about the product's
performance capabilities already had been set.
The path traditionally taken by space programs-and other DOD weapon system
programs-stands in sharp contrast to that taken by leading commercial
firms. Our extensive body of work shows that leading companies use a
product development model that helps reduce risks and increase knowledge
when developing new products. This best practices model enables decision
makers to be reasonably certain about their products at critical junctures
during development and helps them make informed investment decisions. This
knowledge-based process can be broken down into three cumulative knowledge
points.
o Knowledge point 1: A match must be made between the customer's
requirements and the developer's available resources before product
development starts. As noted earlier, DOD plans to start SBR product
development in 2006.
o Knowledge point 2: The product's design must be stable and must meet
performance requirements before initial manufacturing begins.
o Knowledge point 3: The product must be producible within cost,
schedule, and quality targets and demonstrated to be reliable before
production begins.
Systems engineering is a technical management tool that provides the
knowledge necessary at knowledge point 1 to translate requirements into
specific, achievable capabilities. With systems engineering knowledge in
hand, acquisition decision makers and developers can work together to
close gaps between requirements and available resources-well before
product development starts. Some gaps can be resolved by the developer's
investments, while others can be closed by finding technical or design
alternatives. Remaining gaps-capabilities the developer does not have or
cannot get without increasing the price and timing of the product beyond
what decision makers will accept-must be resolved through trade-offs and
negotiations. Effective use of this tool enables decision makers to move
on to knowledge point 2 and to produce a stable product design.
DOD has recently issued a new acquisition policy for space systems, partly
intended to address past acquisition problems and provide capability to
users quicker. However, we recently reported that the policy is not likely
to achieve these goals because it allows programs to continue to develop
technologies after product development starts. Our past work has shown
that this approach makes it more difficult to estimate cost and schedule
at the onset of product development and increases the likelihood that
programs will encounter technical problems that could disrupt design and
production and require more time and money to address than anticipated.
Over the long run, the extra investment required to address these problems
could reduce funding for developing other technological advances, slow the
overall modernization effort, delay capabilities for the warfighter, and
force unplanned-and possibly unnecessary-trade-offs between space and
other weapon system programs. By contrast, DOD's revised policy for other
weapon acquisitions encourages programs to mature technologies to the
point of being tested in an operational environment before beginning
product development. We recommended that DOD modify its policy to separate
technology development from product development so that needs can be
matched with available technology, time, and money at the start of a new
program.
We also reported that DOD's space acquisition policy does not require DOD
to commit to setting aside funding for space acquisitions. Hence, there is
no guarantee that the resources needed to meet requirements will be there
on any individual program when needed. This makes it difficult for DOD as
a whole to make corporate-level and trade-off decisions- which will likely
be needed when DOD begins the SBR acquisition because (1) costs are
significantly increasing for other critical space systems such as the
Space-Based Infrared System High, the Transformational Satellite, and the
Evolved Expendable Launch Vehicle and (2) DOD is planning to undertake
additional new programs, such as the Space-Based Space Surveillance system
and a new version of the Global Positioning System.
DOD is revising its new space acquisition policy partly to address these
issues; however, the revision was not available for review at the time of
this review.
DOD Moving Forward on Acquiring Critical Knowledge but Gaps Remain in Approval
for SBR Requirements
DOD has bolstered the SBR acquisition program by increasing senior leader
and stakeholder involvement in setting requirements. However, DOD is not
fully documenting commitments made during the requirements approval
process before progressing to the next acquisition phase, nor has it
established a process to resolve potential disagreements that may occur
after approval. Clouding the approval of requirements is that DOD's
current space acquisition policy does not provide specific guidance for
acquisitions that involve partnerships between DOD and the intelligence
community.
SBR Managed by New Executive Oversight Structure
Providing senior-level oversight are three new groups created expressly
for the SBR program: the Executive Steering Group, which advises the
Requirements/Capabilities Group and the Joint Senior Acquisition Group.
Members of these groups come from DOD, National Reconnaissance Office, and
National Geospatial-Intelligence Agency. All key stakeholders are expected
to have open and honest discussions about what can and cannot be done
within desired time frames, budgetary constraints, and achievable
technologies. Figure 2 shows how these groups work with SBR's joint
program office and requirements review boards for DOD and the intelligence
community.
Figure 2: SBR's Oversight Structure
Sources: DOD (data); GAO (analysis).
A primary benefit of having an oversight structure for the SBR program,
which involves many decision makers from across multiple organizations, is
that the right people are involved in the decision-making process and can
work together to lock in their requirements. The intent is to avoid
problems of the past in which a program incurs cost, schedule, and
performance risks because decision makers continue to negotiate and make
trade-offs even after designers and engineers have started technology
development and design work. Figure 3 shows the likely outcomes if
requirements are poorly defined and are not approved or, in the case of
SBR, if requirements are adequately defined and approved early in the
study phase.
Figure 3: Requirements-Setting Can Impact Acquisition Process
Source: GAO (analysis).
SBR's Requirements-Setting Process Lacks Formal Approval and Documentation
DOD officials reported to us that the oversight groups have achieved
informal consensus on requirements for SBR. However, this approval has not
been formalized and it is unclear as to whether and how it might be
formalized. Moreover, it is unclear how disagreements that may occur after
initial approval will be resolved.
Regardless of how many stakeholders have been invited to join in decision
making or how much expertise is included in SBR's oversight function,
overall success of the SBR program hinges in part on whether the
requirements are clear, stable, and achievable and whether DOD and the
intelligence community demonstrate commitment and accountability by
formally approving the requirements. In an acquisition decision
memorandum, the Under Secretary of the Air Force requested that DOD and
the intelligence community approve the initial capabilities document and
concept of operations before the request for proposals was released in
January 2004 for concept development contracts. DOD officials told us that
the Joint Requirements Oversight Council and the intelligence community's
Mission Requirements Board approved the initial capabilities document, and
there are memoranda documenting these decisions. The Joint Requirements
Oversight Council reviewed the concept of operations, provided comments,
but did not approve it. According to DOD officials, during a meeting of
the SBR Executive Steering Group, high-level officials from the
intelligence community verbally approved the concept of operations, but
there is no documentation recording this approval.
Agreement is critical because DOD and the intelligence community are
placing different emphasis on desired capabilities for SBR. An independent
assessment of the SBR program determined that requirements were adequate
to enter the study phase, which started in August 2003, but cautioned that
the requirements needed to be converged among all stakeholders and users.
Table 1 shows the type of knowledge that decision makers expect to gain
from the initial capabilities document and the concept of operations.
Table 1: Knowledge Provided by SBR's Initial Capabilities Document and
Concept of Operations
Document Key knowledge provided
Initial capabilities document Identifies current gaps in capability
Identifies the overall desired capabilities of the SBR system Lists users'
performance requirements for SBR Identifies the functional relationships
between users Specifies the desired capability for DOD's capability
development document, which is prepared at the end of the study phase
Identifies validated requirements
Concept of operations Describes the components of the SBR system
Identifies how SBR information is to be processed and disseminated to the
warfighters and others Describes how SBR fits into architectures involving
other space and non-space systems in meeting requirements Identifies
external threats to SBR and the perceived operational environment
Sources: DOD (data); GAO (analysis).
A defined requirements approval process helps decision makers resolve
disagreements that may occur and ensure they will remain committed to
their decisions after formal approval. Based on our past reports on
uncovering problems and our best practice work, we believe that the steps
in a formal approval process include:
o explaining how decision makers' requirements and comments are obtained
and addressed;
o identifying the officials and/or the organizations responsible for
taking specific approval action;
o establishing a mechanism and time frame for providing approval or
disapproval;
o establishing a system for addressing unresolved issues as they relate
to key program documentation; and
o assessing changes to approved requirements based on their effect on
the program's cost and schedule.
While DOD has taken steps to increase senior leader and stakeholder
involvement in setting requirements and addressing acquisition issues, DOD
is not fully documenting commitments made during the requirements approval
process, nor has it established a process to resolve potential
disagreements that may occur after approval.
DOD Taking Proactive DOD is also taking positive steps to attain the
knowledge needed to understand what resources will be needed to develop
SBR's capabilities
Steps to Gain Knowledge about Resources, but Critical Gaps May Remain at
Product Development
and to mitigate risks. These include:
o relying on systems engineering to translate requirements into
specific, achievable capabilities and to close gaps between requirements
and resources;
o adopting a more comprehensive cost estimating technique to identify
SBR's life-cycle costs;
o exploring alternatives for SBR if TCA-the infrastructure that DOD is
depending on to transmit SBR's data-incurs schedule slips; and
o asking two concept development contractors to each propose at least
two different operations concepts for SBR with and without TCA.
However, the path that SBR is on has potential for knowledge gaps when
making investment decisions, the types of gaps that have hampered other
space programs in the past. Specifically, it is expected that some
critical SBR technologies will not be mature when product development
starts, that is, not tested in a relevant or operational environment.
Typical outcomes of this lack of knowledge are significant cost and
schedule increases because of the need to fix problems later in
development. Furthermore, TCA, a new, more robust communications
infrastructure that could transmit SBR's imagery data much more quickly
than the current infrastructure, is facing uncertainties. Specifically,
one of TCA's primary components, the Transformational Satellite, may not
be ready in time to support SBR.2 Without mature technologies and faced
with a possible slip in the Transformational Satellite's schedule, DOD
will be less able to accurately estimate total system costs before the
start of product development. In addition, DOD and the Air Force may not
have knowledge needed to make corporate level trade-offs between SBR and
other air-based radar systems at the time it plans to make a commitment to
invest in the SBR acquisition program. DOD has undertaken an analysis to
weigh the merits of space-based radar. At this time, it is not known
whether this analysis will be a detailed examination of the capabilities
and costs of each individual radar option and combined with other radar
platforms or whether the analysis will be a less rigorous examination of
the mix of radar options.
2 U.S. General Accounting Office, Space Acquisitions: Committing
Prematurely to the Transformational Satellite Program Elevates Risks for
Poor Cost, Schedule, and Performance Outcomes, GAO-04-71R (Washington,
D.C.: Dec. 4, 2003).
DOD Taking Positive Steps to Build Foundation of Knowledge about SBR
Resources
DOD is planning to aggressively address technology, affordability, and
integration issues by, in part, instituting robust systems engineering
processes and procedures. Systems engineering is a technical management
tool for gaining information on a broad array of activities related to the
development of a system. For SBR, DOD plans to perform systems engineering
work on requirements and their allocation, interface definitions, trade
studies, risk management, performance analysis and modeling, environmental
and safety planning, test planning, program protection planning,
information assurance, and configuration control. Applying systems
engineering to these activities would give DOD the insight and knowledge
it needs to better manage the program, including ways to reduce risk and
ensure the viability of concepts and requirements.
DOD has also decided to take a more comprehensive approach to estimating
SBR's life-cycle costs. According to the SBR program director, this marks
the first time DOD has willingly presented all related costs to develop,
acquire, produce, maintain, operate, and sustain the system. DOD officials
stated that they wanted to identify not just direct costs, but also costs
for associated infrastructure such as the costs related to modifying the
ground system that will be used to support SBR as well as other systems.
According to DOD, about $8 billion of the $28.6 billion life-cycle cost
estimate represents costs that in the past, would not have been included
in space program total cost estimates. Taking steps to more
comprehensively identify SBR and SBR related costs is a positive step and
will help DOD manage its portfolio of space programs.
Although DOD hopes to rely on TCA to support SBR data transmissions, it is
taking a proactive approach to identify and assess the viability of TCA
alternatives. First, in April 2004, DOD awarded two 2-year contracts for
concept development efforts that call for the identification of
alternatives to TCA. For each alternative identified, the contractor is to
conduct an assessment of the cost, risk, and effect on SBR's performance.
DOD officials told us that when SBR initiates product development in 2006,
it would know whether TCA will be available to support SBR or whether to
pursue a TCA alternative. In addition, DOD also awarded two contracts
totaling $510,000 for a yearlong study to propose several alternatives to
TCA capable of supporting SBR's communications requirements and to analyze
the viability of such alternatives. These actions have put DOD in a better
position to ensure the program is successful.
The two 2-year contracts that DOD awarded in April 2004 also require that
at least two different viable SBR operations concepts be proposed. DOD is
expecting each contractor to fully develop the alternative operations
concepts. These alternative concepts could involve using unique radar
processing techniques. According to DOD, it will work with each of the
contractors to pare down the alternatives to a single best concept for
each contractor. For the remainder of the contract performance period, the
contractors would focus their attention on fleshing out the details
associated with these concepts. This approach will put DOD in a better
position when the time comes to select a single contractor to design the
SBR system.
Technologies Will Not Be Mature at Product Development Start
DOD officials have said that SBR will likely be the most technically
challenging, software-intensive, and complex space system ever built by
DOD. The two key pieces of hardware needed to give SBR a radar capability
from space-the electronically scanned array (which steers the radar signal
to an area of interest) and the on-board processor (the radar-processing
unit aboard SBR)-face the highest amount of risk. The electronically
scanned array can scan multiple areas of interest virtually
simultaneously, allowing for simplified satellite design over conventional
technology offering mechanical slew radar. The on-board processor is
expected to allow the processing radar data to assure the timely and
thorough delivery of imagery data that will be downlinked for transmission
to the warfighter.
To minimize the potential for technology development problems after the
start of product development, DOD uses an analytical tool to assess
technology maturity for many weapon system acquisition programs. Called
Technology Readiness Levels (TRL), this tool associates a TRL with
different levels of demonstrated performance, ranging from paper studies
to actual application of the technology in its final form. The value of
using a tool based on demonstrated performance is that it can presage the
likely consequences of incorporating a technology at a given level of
maturity into a product's development, enabling decision makers to make
informed choices. Our previous reviews have found the use of TRLs, which
range from 1 to 9, to be a best practice. (See app. I for a description of
the TRL levels.)
The critical technologies that will support the SBR program currently
range from TRL 3 to 5. A TRL 3 means that most of the work performed so
far has been based on analytical and laboratory studies. At a TRL 5, the
basic technology components are integrated and tested in a simulated or
laboratory environment. Table 2 shows the current TRL for each of SBR's
critical technologies and the expected TRL at product development start in
2006. In general, the program office's key risk reduction efforts are
scheduled to mature these technologies to TRL 5 by the middle of fiscal
year 2006. These efforts include the awarding of research and development
contracts to three payload contractors for efforts to continue to develop
and mature these components (the electronically scanned array and on-board
processor). The period of performance of each contract is about 2.5 years.
Table 2: Technology Readiness Levels of SBR Critical Technologies
Current TRL Expected TRL Fiscal
Critical technology levela level year b
Electronically scanned array 4 5 2006
On-board processor 3 5 2006
Signal processing algorithms (for
moving
target indication) 4-5 5 2006
Information management system 3 5 2006
Moving target indication
exploitation
hardware and software 3-4 5 2006
Sources: DOD (data); GAO (analysis).
aA TRL range is shown because of varying maturities between the viable
suppliers.
bEven if the SBR program office chooses to pursue a different SBR concept
alternative, these technologies still would be considered critical;
however, the specific technology readiness date could be different for
each proposed alternative.
To mature the electronically scanned array and on-board processor
technologies from a TRL 3/4 to 5, the contractors plan to conduct various
developmental and integrative tasks in about 3 years. For example, one
contractor plans to conduct 18 tasks to develop the electronically scanned
array and 8 tasks to integrate the on-board processor with other system
components. In addition, the development of the integrated circuits and
programmable microcircuits that support the on-board processor requires
extensive tests and evaluations and the radiation-hardening requirement
further complicates the development. Given the challenges of the
state-of-the-art technologies being developed and the algorithms involved,
the testing programs must be rigorous and transparent and the results
fully documented. We have determined that the time allotted to mature the
SBR technologies to TRL 5 is ambitious given the tasks that need to be
accomplished. Furthermore, the development of the signal processing
algorithms and communications downlink involves significant software
development. Based on our past experience of software assessments in other
programs, the establishment of a structured testing regime for software
development has always been underestimated.
By planning to start product development in fiscal year 2006 with
technologies at TRL 5, DOD is very likely to continue designing the system
and to conduct other program activities at the same time it builds
representative models of key technologies and tests them in an environment
that simulates space conditions (such as a vacuum chamber). This approach
is common with DOD space acquisitions but has a problematic history. Our
past work3 has shown that it can lead to significant cost and schedule
increases because of the need to fix problems later in development. A
continuing problem is that software needs are poorly understood at the
beginning of a program. We have previously recommended that DOD not allow
technologies to enter into a weapon system's product development until
they are assessed at a TRL 7, meaning a prototype has been demonstrated in
an operational environment.4 DOD has accepted lower TRL thresholds for
space programs because testing in an operational environment-in space, for
example, or even in a relevant environment-is difficult and costly.
However, DOD's new space acquisition policy does not identify what the
minimum TRL level should be before starting product development for space
programs, how risks should be mitigated if technologies are included in
programs without full testing, or how lower TRL levels affect the
confidence of cost and schedule estimates. Moreover, the policy does not
address the option of maturing technologies outside a program and pulling
them in once they prove to be viable.
One way to mitigate technology risk is to rely on backup technologies,
should newer technologies prove to be problematic during product
development. According to DOD officials, there are backup technologies
that are more mature for each of SBR's critical technologies. The backups
are the same technologies but rely on a previous and more mature version.
Using previous versions of these technologies would result in a lower
level of desired performance-such as a reduced area collection rate, a
reduction in the total number of targets collected per satellite per day,
increased product delivery time frames to the user, an increased weight of
the spacecraft, and higher cost. For example, more mature versions of the
3 U.S. General Accounting Office, Military Space Operations: Common
Problems and Their Effects on Satellite and Related Acquisitions,
GAO-03-825R (Washington, D.C.: June 2, 2003).
4 U.S. General Accounting Office, Best Practices: Better Management of
Technology Development Can Improve Weapon System Outcomes,
GAO/NSIAD-99-162 (Washington, D.C.: July 30, 1999).
electronically scanned array exist and if used, would result in a
reduction in its performance level. In addition, some previous versions of
SBR technologies have not been demonstrated or tested in space. But
according to DOD officials, even with backup technologies, the total
performance of the SBR system can be maintained through systems
engineering trades. DOD says it has been able to leverage some of the key
technologies (such as the electronically scanned array) that were under
development during the previous effort, Discoverer II, to demonstrate a
space-based radar capability.
Communications Infrastructure May Not Be Ready in Time to Support SBR
Current plans call for TCA to transmit SBR's large volume of data to
ground-, air-, ship-, and space-based systems. However, one of TCA's
primary components, the Transformational Satellite-which will use
technologies that DOD has never before tried in space-is facing
uncertainties in its scheduled 2011 launch. DOD started product
development for the Transformational Satellite in December 2003 even
though technologies were immature. If the Transformational Satellite
falters but SBR launches as expected in 2012, then DOD will have a fully
operational, new-generation satellite that is missing its primary means of
data transmission. Recognizing the challenges, DOD is to decide by
November 2004 whether to move forward or delay the Transformational
Satellite's acquisition program and instead procure another Advanced
Extremely High Frequency satellite, which already are under development
and are based on mature technologies.
Our analysis shows that alternatives to TCA may involve a greater reliance
on processing aboard the SBR satellites, thereby increasing software
development efforts. This approach would reduce the volume of data
requiring transmission, allowing conventional satellite systems, such as
the Advanced Extremely High Frequency satellites, to handle the
transmission. Another likely alternative is to have SBR satellites
transmit only selected portions of data, again, so that the Advanced
Extremely High Frequency satellite could handle the lower volume of
information. Finally, a dedicated system of satellites could be fielded
for the sole purpose of transmitting SBR data, significantly increasing
program cost and raising affordability issues. Currently, DOD is working
closely with officials from the Transformational Satellite program office
to evaluate the relative merits of various alternatives and to document
the interfaces needed between SBR and the Transformational Satellite for
each alternative. During the course of our audit work, SBR program
officials met weekly with the Transformational Satellite program's
integrated product teams and were coordinating efforts on a memorandum of
agreement on
requirements development, joint engineering practices, and studies of
air-and space-based options.
SBR's Cost Estimate Unlikely to Be Realistic Because of Multiple
Uncertainties
Based on a notional constellation of nine (plus one spare) satellites
operating in low-earth orbit, an independent cost assessment in 2003 put
SBR's cost at the $28.6 billion mark, making SBR the most expensive DOD
space system ever built. When this initial cost estimate is revised in
2006, before SBR's product development starts, DOD is to have decided a
number of issues, such as how many satellites are to be acquired, what
their capabilities will be, and at what altitude(s) the satellites are to
operate. This system refinement allows DOD to develop a more realistic
total system cost estimate-a critical knowledge point if a successful
match between requirements and resources is to be made. However, if DOD
begins product development with less than mature technologies and without
knowing the availability of TCA, accurate cost estimates for SBR will be
much more difficult to prepare. We have previously reported that improving
the reliability of cost estimates is critical5 and affords DOD decision
makers with the appropriate information to decide whether a weapon system
is worth the overall investment and whether the time is right to proceed
with such an investment. Once a total cost is known, DOD needs to secure
the funding so it can design, produce, operate, and sustain the system.
DOD may also lack knowledge needed to make a corporate-level decision as
to how much it should invest in SBR versus air platforms with similar
capabilities at the time it begins the SBR acquisition program. In
November 2003, the Air Force completed an analysis of alternatives (AOA)
for SBR, which was supposed to evaluate whether space-or air-based radar
platforms (such as manned and unmanned aircraft with radar capabilities)
or a combination of both are better suited for tracking moving targets on
land or at sea and analyze the capabilities and costs of each suitable
option. However, DOD officials raised a concern that the AOA only weighed
the merits of various space-based solutions. The Air Force decided to
undertake a follow-on study to explore the optimal ways to gather
information on ground moving targets from radars based in space versus
air. The plan is to also use this follow-on study as part of DOD's
preparations for submitting a fiscal year 2006 budget to Congress to
5 U.S. General Accounting Office, Defense Acquisitions: Improvements
Needed in Space Systems Acquisition Management Policy, GAO-03-1073
(Washington, D.C.: Sept. 15, 2003).
Conclusions
secure funding for SBR and other radar systems on air platforms. A more
thorough AOA, completed before the start of the study phase, might
conceivably have determined that air-based radar could provide many or
most of the capabilities promised by space-based radar but at a fraction
of the cost. Moreover, this type of analysis could help DOD officials
better decide whether SBR should be initiated at a later date, when
critical technologies will have been matured, or when the communications
infrastructure to support SBR will be available.
DOD officials have mentioned other ongoing studies that are examining the
optimal mix between SBR and other platforms for specific capabilities,
such as ground-moving target indication. However, it is unclear as to the
extent these studies will be factored into the SBR product development
start decision.
DOD has recently embarked on a discovery and exploration phase for its SBR
program. During this period, it is critical for programs to work toward
closing knowledge gaps about requirements, technologies, funding, and
other resources so they can be positioned to succeed when DOD decides to
commit to making significant investments. For SBR, this would mean testing
technologies to the point of knowing they can work as intended before
starting program development, securing agreement on requirements with the
intelligence community, and fully assessing the cost and benefits and
risks of relying on TCA and alternatives, including different mixes of air
and space-based platforms. DOD is taking positive steps toward this end,
but without maturing critical technologies or securing formal commitment
on requirements, it will not be able to assure decision makers that the
program can be completed within cost and schedule estimates. Should DOD
decide to proceed on a path that leaves open important questions,
including those about technologies, then it should do so with (1)
assessments of technical risks and what additional resources (in terms of
time and money) would be needed to address problems that may occur during
development as well as what trade-offs would need to be made with other
space programs should DOD need to invest additional resources in SBR, and
(2) a formal commitment for providing additional resources if problems do
occur.
Recommendations for To better ensure that DOD and its intelligence
community partners obtain the additional knowledge they need to determine
whether and when to
Executive Action begin the SBR acquisition program, we recommend that the
Secretary of Defense direct the Under Secretary of the Air Force to:
o Direct the SBR Executive Steering Group to ensure that outcomes from
the requirements management process are formally approved and documented
as the program proceeds through product development before an investment
is made beyond technology and concept development for the SBR program.
This group should identify how key document review comments are to be
obtained and addressed and identify all the officials and/or organizations
responsible for taking specific approval action. In addition, the group
should establish a mechanism and time frame for providing
approval/disapproval. Finally, the group should establish a formal
mechanism for addressing unresolved issues as they relate to key program
documentation, as well as how changes to approved requirements will be
assessed.
o Modify DOD's space acquisition policy to reflect protocols for setting
requirements when DOD undertakes programs in partnership with the
intelligence community.
o Delay approval to commit funding to product development (key decision
point B) for SBR until technologies have been demonstrated in a relevant
or operational environment so DOD can more reliably estimate the resources
needed to complete the program. If the Under Secretary determines that the
program should go forward with less mature technologies, then we recommend
that the Under Secretary (1) undertake an assessment of the backup
technologies that may lessen capability and add cost to the program and
the additional time and money that may be required to meet SBR's
performance objectives to address those risks, (2) undertake an assessment
of trade-offs that may need to be made with other space programs to assure
SBR's successful outcome, and (3) secure formal commitments from DOD to
provide funding for total estimated costs as well as costs estimated to
address potential technical risks.
o Strengthen the ongoing study of options for tracking ground-moving
targets by ensuring this work includes: (1) a full range of air and space
options; (2) measures of effectiveness that would help justify choosing
SBR over air options; and (3) the possibility of having to rely on TCA
alternatives for space options. This work should also consider the results
of analyses being conducted by other DOD entities on tracking groundmoving
targets.
Agency Comments
and Our Evaluation
We received written comments on a draft of this report from the Deputy
Under Secretary of Defense (Programs, Requirements, and Resources) within
the Office of the Under Secretary of Defense for Intelligence. DOD
generally agreed with our findings and our recommendation to strengthen
its study of SBR alternatives. DOD partially agreed with our
recommendations to strengthen its requirement setting process for SBR and
to demonstrate SBR technologies in a relevant or operational environment
before committing to product development. DOD did not agree with our
recommendation to modify its acquisition policy to strengthen requirements
setting.
In commenting on our recommendations, DOD agreed in principle with the
need to extensively define, analyze, and validate requirements for SBR,
but it did not believe this necessitated a different requirements setting
process than the one that is in place for SBR or changes to its space
acquisition policy or that additional controls were needed within the
program's study phase. To clarify, our recommendation was not intended to
construct a new requirements setting process or supplant activities
undertaken by the Joint Requirements Oversight Council or the Mission
Requirements Board, as DOD asserts. Rather, we recommend that DOD build on
the positive requirements setting procedures it has already put in place
by instituting controls and mechanisms that ensure transparency,
discipline, and accountability with requirements setting. As noted in our
report, while DOD has taken steps to increase senior leader and
stakeholder involvement in requirements setting, it is not fully
documenting commitments made during the requirements approval process, nor
has it established a process to resolve potential disagreements that may
occur after approval. It is important that this discipline be instilled in
the study phase and throughout the SBR effort. As noted in previous
reports, many space programs have not been executed within cost and
schedule estimates because of an inability to establish firm requirements
and to make and enforce trade-off decisions. For SBR, the potential for
difficulty in requirements setting is higher because of the distinct needs
of the intelligence community and DOD's desire to integrate SBR with other
radar platforms. Moreover, revising the acquisition policy to clearly
communicate protocols that should be followed when DOD undertakes space
programs in the future involving diverse users-such as the intelligence
community, military services, industry, and/or other agencies-would
further help DOD to rationalize requirements setting and to solidify
relationships with users, which DOD reported was a top SBR management
issue.
In regard to our recommendation to delay product development until SBR
technologies are sufficiently matured, DOD stated that it has planned for
critical and most other enabling technologies to be demonstrated at least
at the component level in a relevant environment on the ground. DOD also
stated that where technically feasible and fiscally feasible, it planned
to pursue on-orbit demonstrations. It also stated it has taken some
actions relating to our recommendation such as accounting for technical
risks in the costing and budgeting process. DOD asserted, however, that
our recommendation encourages pursuit of older, more proven, technologies.
We recommended that DOD pursue relevant or operational environment
demonstrations of all critical technologies and even an integrated system
before committing to a formal acquisition program because this practice
enables a program to align customer expectations with resources, and
therefore minimize problems that could hurt a program in its design and
production phase and drive up costs and schedule. Further, we agree that
continuing to develop leading edge technology is important for space
system capabilities. However, history has shown and we have repeatedly
reported that conducting technology development within a product
environment consistently delays the delivery of capability to the user,
robs other programs of necessary funds through unanticipated cost
overruns, and, consequently, can result in money wasted and fewer units
produced than originally stated as necessary. A technology development
environment is more forgiving and less costly than a delivery-oriented
acquisition program environment. Events such as test "failures," new
discoveries, and time spent in attaining knowledge are considered normal
in this environment. Further, judgments of technology maturity have proven
to be insufficient as the basis for accurate estimates of program risks as
it relates to cost, schedule, and capability. Lastly, our report noted
that DOD was taking positive actions to gain knowledge about technology
readiness, including strengthening systems engineering, undertaking risk
assessments, and assessing various technical concepts. Given the potential
cost of the program, our recommendation focuses on taking these steps
further by assessing what trade-offs may need to be made with other space
programs should the program encounter technical problems that require more
time and money than anticipated and securing commitments to provide
resources needed to address such problems.
DOD's detailed comments are provided in appendix II.
We plan to provide copies of this report to the Secretary of Defense, the
Secretary of the Air Force, and interested congressional committees.
We will make copies available to others upon request. In addition, the
report will be available on the GAO Web site at http://www.gao.gov.
If you or your staff has any questions concerning this report, please
contact me at (202) 512-4841 or Arthur Gallegos at (303) 572-7368. Other
key contributors to the report include Tony Beckham, Cristina Chaplain,
Lily Chin, Maria Durant, Nancy Rothlisberger, and Hai V. Tran.
Katherine V. Schinasi
Managing Director
Acquisition and Sourcing Management
Appendix I: TRL Scale for Assessing Critical Technologies
TRL TRL description
Basic principles observed and reported Lowest level of technology
readiness. Scientific research begins to be translated into applied
research and development (R&D). Examples might include paper studies.
Technology concept and/or application Invention begins. Once basic
principles are observed, practical applications can be formulated
invented. Examples are still limited to paper studies.
Analytical and experimental critical function Active R&D is initiated.
This includes analytical studies and laboratory studies to
and/or characteristic proof of concept physically validate analytical
predictions of separate elements of the technology. Examples include
components that are not yet integrated or representative.
Component and/or breadboard validation in Basic technological components
are integrated to establish that they will work
laboratory environment together. This is relatively "low fidelity"
compared to the eventual system. Examples include integration of "ad hoc"
hardware in a laboratory.
Component and/or breadboard validation in Fidelity of breadboard
technology increases significantly. Basic technological
relevant environment components are integrated with reasonably realistic
supporting elements so that the technology can be tested in simulated
environment. Examples include "high fidelity" laboratory integration of
components.
System/subsystem model or prototype demonstration in a relevant
environment Representative model or prototype system, which is well beyond
the breadboard tested for level 5, is tested in a relevant environment.
Represents a major step up in a technology's demonstrated readiness.
Examples include testing a prototype in a high-fidelity laboratory
environment or in a simulated operational environment.
System prototype demonstration in an operational environment
Prototype near, or at, planned operational system. Represents a major step
up from TRL 6, requiring demonstration of an actual system prototype in an
operational environment such as an aircraft, vehicle, or space.
Actual system completed and qualified through test and demonstration
Technology has been proven to work in its final form and under expected
conditions. In almost all cases, this TRL represents the end of true
system development. Examples include Developmental Test and Evaluation of
the system in its intended weapon system to determine if it meets design
specifications.
Actual system proven through successful mission operations
Actual application of the technology in its final form and under mission
conditions, such as those encountered in Operational Test and Evaluation.
Examples include using the system under operational mission conditions.
Source: DOD Interim Defense Acquisition Guidebook, app.6 (Oct. 30, 2002).
Appendix II: Comments from the Department of Defense
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
Note: Page numbers in the draft report may differ from those in this
report.
Appendix II: Comments from the Department of Defense
See comment 1.
Now on p. 23. See comment 2.
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
Now on p. 23.
See comment 3. Now on p. 18.
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
Now on p. 23.
Appendix II: Comments from the Department of Defense
Appendix II: Comments from the Department of Defense
GAO's Comments
The following are GAO's comments on the Department of Defense's letter
dated June 29, 2004.
1. DOD stated that DOD does not require formal approval for the concepts
of operations from the Joint Requirements Oversight Council or the Mission
Requirements Board, but noted that the Joint Requirements Oversight
Council communicated agreement in a memo. As we reported, the Under
Secretary of the Air Force requested that both DOD and the intelligence
community approve the initial capabilities document and concept of
operations in light of the complexity of SBR's acquisition process, the
partnership with the intelligence community, and the proposed integration
with other radar platforms.
2. DOD stated that it is not engaged in a partnership with the
intelligence community on SBR, as our report states. Specifically, DOD
stated that SBR is wholly funded in the defense budget and that a
programmatic commitment with the intelligence community does not exist.
DOD's SBR System Acquisition Strategy was signed by senior-level officials
from DOD, National Reconnaissance Office, and the National
Geospatial-Intelligence Agency and approved on January 14, 2004. This
strategy states that the Air Force, in close partnership with the National
Reconnaissance Office and National Geospatial-Intelligence Agency, is
responsible for leading development of an SBR capability. This strategy
further identifies the responsibilities related to SBR that each mission
partner (National Reconnaissance Office and National
Geospatial-Intelligence Agency) is supposed to carry out. We disagree with
DOD's assertion that these organizations must provide funding to SBR in
order to consummate a partnership. Because SBR is being justified on the
basis of the system's ability to provide intelligence, surveillance, and
reconnaissance products to both DOD and the intelligence community, the
part of the budget used is not relevant to our finding.
3. To clarify, we did not recommend that DOD pursue lower risk
technologies that would result in lower levels of desired performance.
Instead, we reported that DOD might have to resort to using backup
technologies if the current ones prove to be problematic during product
development. We recommended that DOD should assess the cost to the program
of having to use the backup technologies DOD has already identified in
terms of time and money.
GAO's Mission The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125
Relations Washington, D.C. 20548
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard Postage & Fees Paid GAO Permit No. GI00
United StatesEURGovernment Accountability OfficeEURWashington, D.C.
20548-0001EUR
Official BusinessEURPenalty for Private Use $300EUR
Address Service RequestedEUR
*** End of document. ***