Quadrennial Defense Review: Future Reviews Could Benefit from
Improved Department of Defense Analyses and Changes to
Legislative Requirements (14-SEP-07, GAO-07-709).
The Department of Defense (DOD) is required by law to conduct a
comprehensive examination of the national defense strategy, force
structure, modernization plans, infrastructure, and budget every
4 years including an assessment of the force structure best
suited to implement the defense strategy at low-to-moderate level
of risk. The 2006 Quadrennial Defense Review (QDR), completed in
February 2006, represents the first comprehensive review that DOD
had undertaken since the military forces have been engaged in
operations in Iraq and Afghanistan. GAO was asked to assess (1)
the strengths and weaknesses of DOD's approach and methodology
for the 2006 QDR and (2) what changes, if any, in the QDR
legislation could improve the usefulness of the report, including
any changes that would better reflect 21st century security
conditions. To conduct its review, GAO analyzed DOD's
methodology, QDR study guidance, and results from key analyses
and also obtained views of defense analysts within and outside of
DOD.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-709
ACCNO: A76267
TITLE: Quadrennial Defense Review: Future Reviews Could Benefit
from Improved Department of Defense Analyses and Changes to
Legislative Requirements
DATE: 09/14/2007
SUBJECT: Defense budgets
Defense capabilities
Defense contingency planning
Defense economic analysis
Force structure
Future budget projections
Internal controls
Military personnel
Program management
Reporting requirements
Requirements definition
Risk assessment
Risk management
Strategic planning
Program implementation
DOD Quadrennial Defense Review
GAO High Risk Series
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-709
* [1]Results in Brief
* [2]Background
* [3]The QDR's Approach Had Several Strengths but Analytical Weak
* [4]Senior Leadership Involvement, Interagency and Allied Partic
* [5]Weaknesses in Assessment of Force Structure, Personnel Requi
* [6]DOD Did Not Conduct a Comprehensive, Integrated Analysis
of
* [7]DOD Did Not Conduct a Thorough Review of Personnel
Requireme
* [8]DOD Did Not Conduct an Analytically Sound Risk Assessment
of
* [9]Options for Modifying Some Legislative Requirements Could Im
* [10]Strategic Focus of Some QDR Legislative Requirements Is Cons
* [11]Legislative Options Are Available to Improve Usefulness of F
* [12]Conclusions
* [13]Recommendations for Executive Action
* [14]Matters for Congressional Consideration
* [15]Agency Comments and Our Evaluation
* [16]GAO Contact
* [17]Acknowledgments
* [18]GAO's Mission
* [19]Obtaining Copies of GAO Reports and Testimony
* [20]Order by Mail or Phone
* [21]To Report Fraud, Waste, and Abuse in Federal Programs
* [22]Congressional Relations
* [23]Public Affairs
* [24]PDF6-Ordering Information.pdf
* [25]Order by Mail or Phone
Report to the Committee on Armed Services, U.S. Senate
United States Government Accountability Office
GAO
September 2007
QUADRENNIAL DEFENSE REVIEW
Future Reviews Could Benefit from Improved Department of Defense Analyses
and Changes to Legislative Requirements
GAO-07-709
Contents
Letter 1
Results in Brief 5
Background 8
The QDR's Approach Had Several Strengths but Analytical Weaknesses Limited
Its Usefulness in Assessing Force Structure, Personnel Requirements, and
Risk 13
Options for Modifying Some Legislative Requirements Could Improve
Usefulness of Future QDRs 26
Conclusions 32
Recommendations for Executive Action 34
Matters for Congressional Consideration 34
Agency Comments and Our Evaluation 34
Appendix I Scope and Methodology 38
Appendix II Quadrennial Defense Review Legislation in Effect for the 2006
Quadrennial Defense Review 40
Appendix III Summary of New Changes in 10 U.S.C. S118 for Future
Quadrennial Defense Reviews 43
Appendix IV DOD Comments 44
Appendix V GAO Contact and Staff Acknowledgments 48
Related GAO Products 49
Table
Table 1: Key Elements of Capabilities-Based Planning 17
Figure
Figure 1: Organizational Structure for the Development of the 2006 QDR 12
Abbreviations
DOD Department of Defense
OSD Office of the Secretary of Defense
QDR Quadrennial Defense Review
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
September 14, 2007
The Honorable Carl Levin
Chairman
The Honorable John Warner
Committee on Armed Services
United States Senate
Among the 21st century challenges facing the Department of Defense (DOD)
and the nation are difficult decisions concerning how to strike an
affordable balance between national security and domestic needs. Aided by
annual and supplemental funding of over $400 billion per year since fiscal
year 2003, DOD has been maintaining a high pace of operations while
simultaneously transforming its military forces to meet emerging threats
of the new security environment. However, as we have emphasized in
previous reports, the federal government now faces increasing fiscal
challenges, and DOD may face increasing competition for federal dollars.^1
The 2006 Quadrennial Defense Review (QDR) represents the first
comprehensive review of the national defense strategy that DOD has
undertaken since military forces have been engaged in operations in Iraq
and Afghanistan. Further, the 2006 QDR provided an opportunity for DOD to
move beyond its long-standing approaches and methods and identify the
capabilities required to meet current, emerging, and future threats.
The QDR is a key component of national security planning. To ensure that
the country's defense needs are reviewed periodically, Congress directed
DOD to conduct comprehensive QDRs every 4 years to examine elements of the
defense program and policies of the United States including the national
defense strategy, force structure,^2 modernization, infrastructure, and
budget plan.^3 Key assessments required during the review that relate to
national security planning include: (1) the force structure best suited to
implement the defense strategy at low-to-moderate level of risk; (2) the
budget plan that would be required to provide sufficient resources to
execute successfully the full range of missions called for in the national
defense strategy at a low-to-moderate level of risk;^4 (3) the Secretary
of Defense's assessment of the nature and magnitude of the political,
strategic, and military risks associated with executing the missions
called for under the national defense strategy; and (4) the Chairman of
the Joint Chiefs of Staff's assessment of risk.
^1See examples, GAO, 21st Century Challenges: Reexamining the Base of the
Federal Government, [26]GAO-05-325SP (Washington, D.C.: February 2005) and
Fiscal Stewardship and Defense Transformation, [27]GAO-07-600CG
(Washington, D.C.: Mar. 8, 2007).
^2Force structure represents the numbers, size, and composition of the
units that compromise U.S. forces, for example, ships or air wings.
^3The National Defense Authorization Act for Fiscal Year 2000, Pub. L. No.
106-65 S901 (1999), codified at 10 U.S.C. S118.
DOD submitted its report on the third quadrennial review to Congress on
February 6, 2006.^5 In the report, DOD concluded that "For the foreseeable
future, steady-state operations including operations as part of a long war
against terrorist networks and associated rotation base and sustainment
requirements will be the main determinant for sizing U.S. forces." DOD
also confirmed that for the long term it must size and shape U.S. forces
for three main types of missions: homeland defense, the war on
terrorism/irregular warfare,^6 and conventional campaigns. In addition,
DOD acknowledges that it must implement departmentwide change to ensure
that organizational structures, processes, and procedures effectively
support its strategic priorities. For example, DOD created the Defense
Business Transformation Agency to integrate and oversee corporate-level
business systems and initiatives in areas such as acquisition and
logistics. Further, DOD reached several key decisions in the 2006 QDR that
emphasized the need to continue changing the mix of joint capabilities and
forces, such as stabilizing Army and Marine Corps active duty personnel at
fiscal year 2006 congressionally authorized levels while increasing
special operations forces in areas such as civil affairs units and special
forces battalions, and military personnel for sea, air, and land teams.
DOD viewed the 2006 QDR as a refinement of the concepts it introduced in
its 2001 QDR report, such as shifting the basis of force planning from
focusing on specific adversaries and geographic locations to
capabilities-based planning that identifies the capabilities the military
will need to prepare for a range of potential military operations against
unknown enemies. In the 2001 QDR report, DOD introduced a risk management
framework designed to help address the tension between preparing for
future threats and meeting the demands of the present with finite
resources and to size, shape, and manage the department to accomplish its
strategic priorities. DOD planned to use the framework in conducting the
2006 QDR.
^4DOD is also required to identify any additional resources (beyond those
programmed in the current future-years defense program) required to
achieve such a level of risk.
^5The first Quadrennial Defense Review was submitted to Congress in May
1997 before the current legislation was enacted. The second Quadrennial
Review was submitted on September 30, 2001, pursuant to 10 U.S.C. S118.
^6DOD refers to irregular warfare as conflicts in which enemy combatants
are not regular military forces of nation states.
Our work on past QDRs^7 has shown long-standing weaknesses in DOD's
assessment of force structure requirements. In past QDRs, DOD has not
focused on longer-term threats and requirements for support capabilities,
and its QDR reports have provided little information on some required
issues, such as assumptions used in its analyses. Moreover, we have
reported that force structure decisions were not clearly supported by
analysis and linked to strategic plans. Further, in November 2005,^8 we
reported that DOD has not fully implemented a risk management approach and
it planned to refine its risk management framework during the 2006 QDR.
In 2006, Congress passed legislation^9 which added new reporting elements
that will apply to the next QDR in 2010 as well as future QDRs. For
example, the Secretary of Defense must establish an independent panel to
conduct a postreview assessment of the QDR including the recommendations,
assumptions used, and vulnerabilities of the strategy and force structure
underlying the review. The new legislation also required that the
Secretary of Defense submit to the Senate and House Committees on Armed
Services quarterly reports on the status of the department's
implementation of the 2006 QDR decisions, beginning in January 2007.^10
DOD submitted its first and second quarterly reports to Congress on
January 31, 2007, and June 14, 2007, respectively.^11
^7GAO, Quadrennial Defense Review: Future Reviews Can Benefit from Better
Analysis and Changes in Timing and Scope, [28]GAO-03-13 (Washington, D.C.:
Nov. 4, 2002) and GAO, Quadrennial Defense Review: Opportunities to
Improve the Next Review, [29]GAO/NSIAD-98-155 (Washington, D.C.: June 25,
1998).
^8GAO, Defense Management: Additional Actions Needed to Enhance DOD's
Risk-Based Approach for Making Resource Decisions, [30]GAO-06-13
(Washington, D.C.: Nov. 15, 2005).
^9John Warner National Defense Authorization Act for Fiscal Year 2007,
Pub. L. No. 109-364, S1031 (2006).
You asked us to evaluate DOD's overall approach and supporting analysis in
preparing the 2006 QDR and assess whether the QDR legislative requirements
could be reevaluated to improve the usefulness of the report, including
any changes needed to better reflect the security conditions of the 21st
century. Accordingly, this report assesses (1) the strengths and
weaknesses of DOD's approach and methodology for the 2006 QDR and (2) what
changes, if any, in the QDR legislation could improve the usefulness of
future QDRs.
To assess the 2006 QDR's strengths and weaknesses, we reviewed DOD's study
guidance for the QDR and discussed the advantages and disadvantages of the
review's process with DOD's officials. We also examined the methodology
and results of the QDR key analyses and assessed how capabilities-based
planning principles were applied during the assessments. To understand how
the Secretary of Defense and the Chairman of the Joint Chiefs of Staff
conducted their risk assessments, we reviewed the QDR's study guidance on
assessing risk and held discussions with officials responsible for
conducting risk assessments during the QDR. We reviewed DOD's quarterly
report to Congress on the status of implementation for the 2006 QDR and
post-QDR study team reports and implementation plans to review the
processes that DOD has to implement QDR initiatives. Further, we held
discussions with Office of the Secretary of Defense (OSD) officials
responsible for monitoring the status of initiatives related to the QDR
and ongoing work in the post-QDR study teams. To determine whether changes
to the QDR legislation could improve the usefulness of future QDRs, we
identified potential options from our analyses of prior QDRs and obtained
the views of DOD civilian and military leaders who participated in the
2006 QDR as well as nongovernmental defense analysts, many of whom had
played key roles in previous QDRs or in prior defense strategy reviews. We
performed our review from May 2006 through May 2007 in accordance with
generally accepted government auditing standards. Further information on
our scope and methodology appears in appendix I.
^10Pub. L. No. 109-364, S1032 (2006).
^11Department of Defense, Quarterly Report to Congress on Implementation
of the 2006 Quadrennial Defense Review (Washington, D.C.: Jan. 31, 2007)
and Second Quarterly Report to Congress on Implementation of the 2006
Quadrennial Defense Review (Washington, D.C.: June 14, 2007).
Results in Brief
While DOD's approach and methodology for the 2006 QDR had several
strengths, several weaknesses significantly limited the review's
usefulness in addressing force structure, personnel requirements, and risk
associated with executing the national defense strategy. On the positive
side, the 2006 QDR benefited from the sustained involvement of key senior
DOD officials who provided top-down leadership and oversight of the review
process. Second, for the first time, DOD collaborated extensively with
several major interagency partners, such as the Department of Homeland
Security, as well as representatives of some allied countries to identify
capabilities that would address current and future security threats.
Third, leaders of the QDR's six study teams collaborated with each other
to avoid duplication of work as they developed options to address DOD's
challenges. Fourth, DOD has developed a database for monitoring the
implementation of about 170 QDR initiatives, which range from changing
organizational structures to enhancing military capabilities. However,
weaknesses in three key areas--force structure analysis, assessment of
personnel requirements, and assessment of the level of risk--hampered
DOD's ability to determine the military force best suited to implement the
defense strategy, which is a fundamental QDR goal, and thoroughly
demonstrate how the risks associated with desired capabilities were
evaluated. As a result, DOD is not well positioned to balance capability
needs and risks within future budgets given the nation's serious budget
pressures.
o First, although the 2001 QDR and the 2006 QDR study guidance
emphasized that DOD planned to use capabilities-based planning to
perform its analysis, DOD did not conduct a comprehensive,
integrated assessment of alternative force structures. A key
reason why DOD did not conduct such an assessment of its force
structure was that it has not developed an integrated
capabilities-based planning approach for comparing alternative
force structures. Although DOD relied on several analyses of
different parts of the force structure to make decisions about
capabilities, it did not integrate these analyses into a
comprehensive assessment. For example, while DOD conducted
separate studies about tactical aircraft and ground forces, these
were not integrated into an overall assessment of the numbers and
size of units needed. Further, instead of assessing different
levels of forces and their capabilities and evaluating the
trade-offs among capabilities, DOD's primary assessment approach
was to compare currently planned forces to potential scenarios to
determine whether and to what extent the planned force structure
would experience shortages.
o Second, DOD did not provide a clear analytical basis for its
conclusion that the number of personnel in the active and reserve
components across the military departments was appropriate to meet
current and projected operational demands. A key reason why DOD
did not provide a clear basis for its personnel requirements is
that existing personnel levels were taken as a given, and DOD
focused on analyzing options on how to change the skill mix of
active and reserve military personnel and civilians. Further,
within 1 year after the QDR was published, the Secretary of
Defense announced plans to seek congressional approval to increase
Army and Marine Corps personnel by 92,000. These plans call into
question the analytical basis of the QDR conclusion that the
number of personnel and the size of the services' force structure
were appropriate to meet current and future requirements. Without
performing a comprehensive analysis of the number of personnel it
needs, DOD cannot provide an analytical basis for its conclusion
that its military and civilian personnel levels reflect the number
of personnel needed to fill DOD's combat force structure and
provide institutional support.
o Third, the risk assessments conducted by the Secretary of
Defense and the Chairman of the Joint Chiefs of Staff, which are
required by the QDR legislation, did not fully apply DOD's risk
management framework to demonstrate how risks associated with the
proposed force structure were evaluated. Although tasked to use
the risk management framework to demonstrate how risks were
evaluated, several of the QDR study teams relied primarily on
professional judgment to assess risks and examine the consequences
of not investing in various capabilities. The Chairman was not
tasked to use the risk management framework in assessing risks and
did not choose to use it in his assessment. Our prior work has
shown that performing a data-driven risk assessment can provide a
guide to help organizations shape, focus, and prioritize
investment decisions to develop capabilities. DOD did not conduct
a comprehensive data-driven risk assessment because, according to
DOD officials, it had difficulties in developing the
department-level measures that would be necessary to assess risk
and, as a result, the assessment tools were not available for use
during the QDR.
Without thorough alternative force structure assessments, analyses
of personnel requirements, and comprehensive risk assessments, DOD
cannot provide comprehensive analytical support for significant
decisions so that Congress can effectively evaluate the benefits,
costs, and risks of alternative force structures and associated
expenditures of federal resources. We are recommending that the
Secretary of Defense develop appropriate methods for conducting
comprehensive, data-driven assessments in future QDRs of (1) the
capabilities related to alternative force structures and related
personnel requirements and (2) the risks associated with
capabilities. DOD partially agreed with these recommendations. In
its comments, DOD agreed that the 2006 QDR did not comprehensively
assess alternatives to the planned force structure and instead
assessed force requirements within capability areas. DOD stated
that it is taking steps to provide more robust analysis of
capabilities for future QDRs. However, until DOD comprehensively
assesses alternative force structures that include examining
alternatives across capability areas, it will not have the
detailed information it needs to determine the force structure
best suited to implement the defense strategy and to demonstrate
risks associated with the planned force structure. DOD also agreed
that the study teams inconsistently applied DOD's risk management
framework, although it noted the senior leaders discussed a
consolidated analysis of risk related to the proposed force
structure as the QDR decisions were finalized in November 2005.
DOD noted that further development of the department's risk
management methodology is necessary to appropriately assess risks
and it discussed some steps the department intends to take to
identify performance goals and develop metrics. However, DOD did
not provide detailed information or a time frame for the
improvements it discussed.
Several options exist for refining legislative language that
Congress could consider to focus QDR statutory requirements on
strategic issues and eliminate some reporting elements that are
already required under other laws and that may no longer be as
useful in the new security environment. Specifically, some defense
analysts we interviewed suggested requiring DOD to focus its
efforts on broad strategic issues and provide more information on
the analytic basis for its key assumptions and strategic planning
decisions. In addition, some defense analysts suggested that to
facilitate congressional oversight and decision making, the QDR
legislation should clarify Congress' expectations for information
related to budget plans and planned trade-offs among capabilities.
In addition, most analysts agreed that many of the detailed
requirements requiring reporting on more operational issues, such
as reporting on the unified command plan, may divert the QDR's
focus from strategic issues and should be eliminated from the QDR
and assessed separately. Finally, most defense analysts we
interviewed believe that recent legislation, which requires DOD to
appoint an independent panel to complete a post-QDR assessment of
the results of future QDRs,^12 could be expanded to include
providing advice to the Secretary of Defense before or during the
QDR process. We are suggesting that Congress consider options to
(1) clarify expectations for how the QDR should address the budget
plan that supports the national defense strategy and (2) eliminate
some reporting requirements. DOD agreed with these suggestions. In
our draft report we also suggested that Congress consider
broadening the scope of the post-QDR assessment panel for future
QDRs to include providing advice before or during the QDR process.
In its written comments, DOD stated that having an independent
panel that could provide advice and alternatives to the Secretary
of Defense before and during the QDR process would be useful.
However, DOD raised the concern that having the same panel advise
the department before and during the QDR as well as critiquing the
results could create mistrust between the department's leadership
and the panel. To reflect DOD's concern, we have revised our
matter to state that Congress should consider requiring an
independent panel to provide advice and alternatives to the
Secretary of Defense before and during the QDR. This change is
intended to provide Congress with the flexibility to establish
separate independent panels prior to and following the QDR.
Background
In May 1995, the Commission on Roles and Missions of the Armed
Forces proposed the idea of a comprehensive quadrennial review by
DOD of the country's defense strategy and force structure. In
August 1995, the Secretary of Defense endorsed the idea, and the
following year legislation directed DOD to conduct the 1997
QDR.^13
Congress created a permanent requirement for DOD to conduct a QDR
every 4 years in the National Defense Authorization Act for Fiscal
Year 2000, passed in 1999.^14 According to this legislation, DOD
was to conduct a comprehensive examination of the national defense
strategy, force structure, force modernization plans,
infrastructure, budget plan, and other elements of the country's
defense program and policies with a view toward determining and
expressing the nation's defense strategy and establishing a
defense program for the next 20 years. Originally the legislation
identified 14 specific issues for DOD to address, such as a
comprehensive discussion of the national defense strategy of the
United States and the force structure best suited to implement
that strategy at a low-to-moderate level of risk. In addition, it
allowed the Secretary of Defense to review any other issues he
considers appropriate. The legislation in effect during the 2006
QDR reflected several amendments to the original legislation, for
example, requiring DOD to assess the national defense mission of
the Coast Guard. (See app. II for the legislation in effect during
the 2006 QDR.)
^12Pub. L. No. 109-364 S1031 (2006).
^13National Defense Authorization Act for Fiscal Year 1997, Pub. L. No.
104-201 S923 (1996).
^14Pub. L. No. 106-65 S901 (1999), codified at 10 U.S.C. S118.
Among other requirements, the 1999 QDR legislation required that
the Secretary of Defense assess the nature and magnitude of the
political, strategic, and military risks associated with executing
the missions called for under the national defense strategy. In
the 2001 QDR report, DOD introduced a new risk management
framework that identified four areas of risk--operational, force
management, future challenges, and institutional. According to the
2001 QDR report, the framework would enable DOD to address the
tension between preparing for future threats and meeting the
demands of the present with finite resources. Further, the
framework was intended to ensure that DOD was sized, shaped,
postured, committed, and managed with a view toward accomplishing
the strategic priorities of the 2001 QDR.
Future QDRs will be affected by the new reporting elements added
to the QDR legislation by the John Warner National Defense
Authorization Act for Fiscal Year 2007.^15 Specifically, the
legislation requires DOD to establish an independent review panel
to conduct an assessment of the QDR no later than 6 months before
the date that DOD's report on the QDR is submitted to Congress.
The panel is required to submit, within 3 months after the date on
which the QDR is submitted, an assessment of the review, including
its recommendations, the stated and implied assumptions
incorporated in the review, and the vulnerabilities of the
strategy and force structure underlying the review. The
legislation also specifies that the QDR review should not be
constrained to comply with the budget submitted to Congress by the
President. In addition, the legislation added several specific
issues that DOD is required to address such as providing the
specific capabilities, including the general number and type of
specific military platforms, needed to achieve the strategic and
warfighting objectives. Lastly, the authorization act directs DOD
to submit to the Senate and House Armed Services Committees a
report on the implementation of recommendations identified in the
2006 QDR report no later than 30 days after the end of each fiscal
year quarter.^16 (See app. III for a summary of additions to the
QDR legislation, 10 U.S.C. S118 as a result of the John Warner
National Defense Authorization Act for Fiscal Year 2007.)
^15Pub. L. No. 109-364, SS1031, 1032 (2006).
DOD considers the 2006 QDR a refinement of its predecessor 2001
QDR, which detailed the department's intent to shift the basis of
defense planning from the long-standing "threat-based" model,
which focused on specific adversaries and geographic locations, to
a "capabilities-based" construct that seeks to prepare for a range
of potential military operations against unknown enemies.
According to the 2001 QDR report, the capabilities-based model
focuses on how an adversary might fight rather than specifically
who the adversary might be or where the war might occur.
The Under Secretary of Defense (Policy) had the lead role in
conducting the 2006 QDR. The Joint Staff played a supporting role
in the process and had primary responsibility for leading the
analytical work to support the Chairman of the Joint Chief of
Staff's risk assessment. In March 2005, the Secretary of Defense
approved guidance, called the Terms of Reference, for the review.
The Terms of Reference identified four focus areas and provided
guidance to senior officials to develop capabilities and make
investment decisions to shape the future force and reduce risks in
these areas. The four focus areas were 1) defeating terrorist
networks, 2) defending the homeland in depth, 3) shaping the
choices of countries at strategic crossroads, and 4) preventing
hostile states and nonstate actors from acquiring or using weapons
of mass destruction. During the spring of 2005, DOD senior leaders
held meetings on the focus areas with interagency partners from
across the federal government and international allies to identify
the potential threats and the types of capabilities needed to
address the challenges associated with the focus areas. Officials
from the intelligence community, such as the Defense Intelligence
Agency, provided threat assessments for each of the focus areas.
^16DOD submitted its first and second quarterly reports to Congress on
January 31, 2007, and June 14, 2007, respectively. This reporting
requirement will terminate upon the publication of the next QDR or when
the Secretary of Defense notifies the Senate and House Armed Services
Committees in writing that implementation is complete for the 2006 QDR
recommendations.
The Terms of Reference also established six study teams to assess
capabilities associated with the QDR focus areas and directed the
teams to develop options to reduce risk in these areas. Top-level
civilian and military leaders from OSD and Joint Staff led the
study teams, which included officials from the services and
Combatant Commands. The Deputy Secretary of Defense and the Vice
Chairman of the Joint Chiefs of Staff co-chaired a senior level
group, which was eventually referred to as the Deputy's Advisory
Working Group, and this group reviewed the work of the study teams
during the summer and fall of 2005. Other members of the review
group included the Under Secretaries of Defense, the services'
Under Secretaries, the services' Vice Chief of Staffs, and the
Deputy Commander, U.S. Special Operations Command. The Deputy
Secretary and his working group determined what information each
study team would provide to the senior-level review group, which
was led by the Secretary of Defense. Figure 1 shows the structure
that OSD established to conduct the QDR.
Figure 1: Organizational Structure for the Development of the 2006 QDR
According to the 2006 QDR report, the foundation of this QDR is the
National Defense Strategy, published in March 2005. The Secretary of
Defense's National Defense Strategy is implemented through the National
Military Strategy, which is developed by the Chairman of the Joint Chiefs
of Staff. The National Military Strategy provides focus for military
activities by defining a set of interrelated military objectives from
which the service chiefs and combatant commanders identify desired
capabilities and against which the Chairman of the Joint Chiefs of Staff
assesses risk.
The QDR's Approach Had Several Strengths but Analytical Weaknesses Limited Its
Usefulness in Assessing Force Structure, Personnel Requirements, and Risk
While DOD's approach and methodology for the 2006 QDR had several
strengths, several weaknesses significantly limited the review's
usefulness in addressing force structure, personnel requirements, and risk
associated with executing the national defense strategy. On the positive
side, the 2006 QDR benefited from the sustained involvement of key senior
DOD officials, interagency and allied participation, and internal
collaboration among the QDR's participants. However, weaknesses in the
assessment of three key areas--force structure, personnel requirements,
and risk--hampered DOD's ability to undertake a fundamental reassessment
of the national defense strategy and U.S. military forces. As a result of
these weaknesses, Congress lacks assurance that DOD has conducted the
analysis needed to determine the force best suited to implement the
defense strategy. Further, DOD is not well positioned to demonstrate to
Congress how it considered risks and made difficult trade-offs among its
capabilities to balance investments within future budgets, given the
nation's fiscal challenges.
Senior Leadership Involvement, Interagency and Allied Participation, Internal
Collaboration, and a System to Monitor Implementation Provided Benefits
DOD's approach for the 2006 QDR benefited from several strengths. First,
key senior DOD leaders maintained sustained involvement throughout the
review. As we have noted in previous reports,^17 best practices clearly
indicate that top-level leadership is crucial for engineering major
changes in an organization. Top leaders establish the framework for change
and provide guidance and direction to others to achieve change. During the
2006 QDR process, the Deputy Secretary of Defense and the Vice Chairman of
the Joint Chiefs of Staff co-chaired a senior level review group, now
referred to as the Deputy's Advisory Working Group, to review and approve
initiatives of varying complexity presented by the six study team leaders
and leaders of specialized issue areas, such as special operations forces.
According to an official in the Office of the Secretary of Defense, during
most of the QDR process, this senior level group met several times a week
to review the study teams' options and provide guidance to the teams to
ensure that the QDR's strategic priorities were addressed. Since the QDR
report was issued in February 2006, the Deputy's Advisory Working Group
continues to meet regularly to oversee implementation of the QDR's
strategic priorities, such as improving DOD's management structures and
business processes to support effective decision making.
^17For examples, GAO, DOD's High Risk Areas Successful Business
Transformation Requires Sound Strategic Planning and Sustained Leadership,
[31]GAO-05-520T (Washington, D.C.: Apr. 13, 2005), and Force Structure
Army Needs to Provide DOD and Congress More Visibility Regarding Modular
Force Capabilities and Implementation Plans, [32]GAO-06-745 (Washington,
D.C.: Sept. 6, 2006).
Second, DOD collaborated with interagency partners, such as the Department
of Homeland Security, and U.S. international allies, such as the United
Kingdom, to discuss potential strategic challenges and determine
capabilities that are required to meet current and future challenges.
According to DOD officials, senior officials from the Department of
Homeland Security including the U.S. Coast Guard and the Departments of
Energy, State, and other federal agencies participated in DOD's
discussions establishing the strategic direction of the QDR during the
spring of 2005. U.S. agency officials discussed with DOD officials the
types of capabilities and investments needed to reduce risk in the QDR's
four focus areas--defeating terrorist networks, defending the homeland in
depth, shaping the choices of countries at strategic crossroads, and
preventing hostile states and nonstate actors from acquiring or using
weapons of mass destruction. For example, DOD officials who coordinated
the QDR stated that U.S. Coast Guard officials identified current and
planned maritime defense capabilities as part of DOD's discussion on
combating weapons of mass destruction. Further, officials from U.S.
allies, such as the United Kingdom, participated in the discussions to
share their perspectives about how DOD, its allies, and global partners
could address the nontraditional, asymmetric warfighting challenges of the
21st century, such as preventing the acquisition or use of weapons of mass
destruction by nonstate actors. As a result of contributions from the
interagency partners and allies, DOD was in a better position to identify
and develop the four focus areas that eventually shaped the scope of the
QDR.
Third, leaders of the six study teams collaborated with each other to
avoid duplication of work as they developed options to address challenges
associated with the focus areas. The study team leaders held weekly
meetings to discuss whether their issues could be better addressed by
another study team, the progress of their work plans, and whether they
could provide each other with mutually supporting analysis. Further, a
group of senior officials, led by an official in the Office of the
Secretary of Defense for Policy, attended the study teams' weekly meetings
to ensure that the options addressed the capabilities associated with the
four focus areas and helped identify overlaps or gaps in the development
of options. For example, three study teams, which developed and identified
options related to force structure, personnel requirements, and roles and
missions respectively, coordinated their work to minimize any overlap and
identify any gaps in the development of options to increase the number of
military and civilian personnel proficient in key languages such as
Arabic, Farsi, and Chinese.
Fourth, following the release of the 2006 QDR, the Deputy Secretary of
Defense requested that officials in OSD establish procedures to track the
implementation of the 2006 QDR initiatives which encompassed a range of
military capabilities, from implementing its new personnel management
system to developing a new land-based, penetrating long-range strike
capability by 2018. Senior officials from the Office of the Director,
Administration and Management created a departmentwide database and
established criteria to categorize the implementation status of each
initiative. Specifically, implementation of an initiative was categorized
as "completed" if the initiative was fully implemented or if DOD had taken
actions that officials determined as having met the intent of the
initiative, even though the initiative may take years to fully implement.
OSD officials have provided periodic briefings on the status of QDR
initiatives to the Deputy Secretary of Defense and his advisory group
since the publication of the 2006 QDR report. DOD reported to Congress in
January 2007 that it had completed implementation of about 90, or 70
percent, of the 130 initiatives.^18 Further, in January 2006 at the end of
the QDR process, the Deputy Secretary of Defense identified eight study
areas and established a process to continue developing DOD's approaches
for the issues associated with these study areas.^19 According to senior
DOD officials, these areas identified for post-QDR study were generally
complex and involved multiple organizations, such as developing
interoperable strategic communications.^20 The Deputy Secretary provided
guidance for the teams that included requirements to (1) define
objectives, timelines, and performance metrics and (2) establish an
oversight process as part of an implementation plan to ensure the
decisions made during the QDR were achieved. According to DOD officials,
DOD plans to provide Congress with information about the status of the
post-QDR study teams' implementation in its quarterly reports. For
example, in DOD's January 2007 report to Congress, DOD reported that one
of the Institutional Reform and Governance study team's objectives is to
continue developing concepts and overseeing initiatives related to
reforming governance and management functions such as capabilities-based
planning.
^18Department of Defense, Quarterly Report to Congress on Implementation
of the 2006 Quadrennial Defense Review (Washington, D.C.: Jan. 31, 2007).
We did not find documentation in DOD officials' briefings to the Deputy's
Advisory Working Group or DOD's report to Congress that indicates how
effectively DOD is implementing the initiatives, such as providing
information on whether the activity is on schedule or assessing the
effectiveness of the initiatives. Further, in June 2007, DOD reported that
it had closed an additional 19 initiatives, which brings the number of QDR
initiatives that DOD considers completed to about 110, or 85 percent, of
the 130 initiatives.
^19The eight post-QDR study teams are Authorities; Irregular Warfare;
Building Partnership Capacity; Strategic Communications; Institutional
Reform and Governance; Joint Command and Control; Locate, Tag, Track; and
Sensor-based Management of Intelligence, Surveillance, and Reconnaissance
Enterprise.
^20For example, DOD officials, including the Assistant Secretary for
Public Affairs and the Chairman of the Joint Chief of Staff Director for
Strategic Communication, plan to develop formal processes that will enable
DOD to better coordinate and synchronize the assessment and delivery of
information to key allies and coalition partners.
Weaknesses in Assessment of Force Structure, Personnel Requirements, and Risk
Limited the QDR's Usefulness in Linking Force Structure to the Defense Strategy
and Addressing Affordability Challenges
Weaknesses in the assessment of three key areas--force structure,
personnel requirements, and risk--significantly limited the review's
usefulness in reassessing the force structure best suited to implement the
defense strategy at low-to-moderate level of risk, which is a key
requirement of the review. Our previous reporting on DOD's prior QDRs and
other work has shown that weaknesses in establishing a substantive basis
for force structure, personnel requirements, and risk have been
long-standing issues for the department. Further, until DOD can
demonstrate an analytical basis for its force structure and personnel
requirements, it will not be well-positioned to balance capability needs
within budgets that are likely to be constrained in the future, given the
nation's fiscal challenges.
DOD Did Not Conduct a Comprehensive, Integrated Analysis of Alternative Force
Structures Using a Capabilities-Based Approach
Although the 2006 QDR study guidance emphasized that DOD would use
capabilities-based planning to focus on how a range of potential enemies
might fight, DOD did not conduct a comprehensive, integrated assessment of
alternative force structures during the QDR using a capabilities-based
approach. Based on our discussions with DOD officials and our review of
DOD documents and non-DOD published studies, a capabilities-based approach
requires a common understanding of how a capability will be used, who will
use it, when it is needed, and why it is needed. Further, each capability
should be assessed based on the effects it seeks to generate and the
associated operational risk^21 of not having the capability. A
capabilities-based approach also seeks to identify capability gaps^22 or
redundancies and make trade-offs among the capabilities in order to
efficiently use fiscal resources. In table 1 we identify several key
elements of a capabilities-based planning approach and provide
descriptions of these elements.
^21DOD defines operational risk as the ability to achieve military
objectives in a near-term conflict or other contingency.
^22According to DOD, a capability gap is the military inability to achieve
a desired effect by performing a set of tasks under specified standards
and conditions. The gap may be the result of not having an existing
capability or the lack of proficiency or sufficiency in an existing
capability.
Table 1: Key Elements of Capabilities-Based Planning
Key element Description
Establish an organizational o Identify roles, responsibilities, and
structure organizational changes
o Establish methods for recording and
communicating decisions and tracking their
execution
Establish an analytic o Use traceable and analytically-based
framework data, information sources, and standards
consistently when conducting assessments
o Refine and further develop an approach
to assess risk
Develop a national defense o Identify strategic goals so capabilities
strategy can be developed to support these goals
Develop a wide range of o Identify potential threats by using
specific and generic threats intelligence sources, strategic studies,
and professional military experience
Develop a wide range of o Develop scenarios that address various
scenarios time frames and do not solely focus on one
or two major conventional campaigns
o Ensure that the scenarios challenge the
force and do not simplify existing
weaknesses and problems, e.g., do not
assume that overseas locations have a
developed infrastructure
o Identify the capabilities that are
needed to perform missions outlined in the
scenarios
Conduct a capability survey o Identify the capabilities within the
existing and planned force
o Identify any gaps or excesses
Develop capability options o Perform comprehensive assessments to
determine trade-offs among the
capabilities, such as identifying which
capabilities can be substituted with other
capabilities
o Link capability options to strategic
goals to determine whether the goals are
being addressed
o Assess the risk of each trade-off using
a data-driven approach
o Prioritize the best balance of
investment across major capability areas
Link capability solutions to o Identify the near- and long-term
well-defined budget, budgetary implications for the capability
acquisition, performance options
plans o Develop detailed acquisition plans
o Establish mechanisms to establish clear
authority and accountability, milestones,
and performance measures
Source: GAO analysis of DOD and non-DOD capabilities-based planning
studies.
DOD's primary basis for assessing the overall force structure best suited
to implement the national defense strategy, according to several DOD
officials, was a Joint Staff-led study known as Operational Availability
06. The study compared the number and types of units in DOD's planned
force structure to the operational requirements for potential scenarios to
determine whether and to what extent the planned force structure would
experience shortages. However, the Joint Staff's Operational Availability
06 Study did not assess alternatives to planned force structures and
evaluate trade-offs among capabilities.
In conducting the Operational Availability 06 Study, the Joint Staff
completed two different analyses. The first analysis, referred to as the
base case, relied on a set of operational scenarios that created
requirements for air, ground, maritime, and special operations forces.
During this study, the Joint Staff examined requirements for a broad range
of military operations over a 7-year time frame. Two overlapping
conventional campaigns served as the primary demand for forces with
additional operational demands created by 23 lesser contingency
operations, some of which represented the types of operations that
military forces would encounter while defending the homeland and executing
the war on terrorism. The Joint Staff then compared the number of military
units in DOD's planned air, ground, maritime, and special operations
forces to the operational demands of the scenarios. The Joint Staff made
two key assumptions during the analysis. First, the Joint Staff assumed
that reserve component units could not deploy more than once in 6 years.
Second, the Joint Staff assumed that while forces within each service
could be reassigned or retrained to meet shortfalls within the force
structure, forces could not be substituted across the services.^23 Results
of the Joint Staff's first analysis showed that maritime forces were
capable of meeting operational demands and air, ground, and special
operations forces experienced some shortages.
In response to a tasking from top-level officials the Joint Staff
performed a second analysis that developed a different set of operational
demands reflecting the high pace of operations in Iraq. In this analysis,
the Joint Staff used the same 2012 planned force structure that was
examined in the first analysis. When it compared the operational demands
that were similar to those experienced in Iraq with DOD's planned force
structure, the Joint Staff found that the air, ground, maritime, and
special operations forces experienced shortages and they could only meet
operational demands for a security environment similar to Iraq, one
conventional campaign, and 11 of the 23 lesser contingency scenarios.
^23Our reporting shows that DOD's experience has been different than this
assumption. To meet operational demands in Iraq and Afghanistan, DOD has
relied increasingly on reassigning and retraining personnel to meet
requirements. See [33]GAO-06-962 .
While the Operational Availability 06 Study had some benefits, several
weaknesses significantly limited the study's usefulness for integrating a
capabilities-based approach that assessed force structure options. On the
positive side, top leaders maintained sustained involvement in the
Operational Availability Study; for example, based on their guidance, the
Joint Staff conducted a second analysis that depicted operational demands,
which more accurately represented the current security environment. That
study demonstrated that significant shortages in military forces exist
when forces are not retrained or reassigned to meet operational demands.
However, weaknesses in the study's methodology to assess different levels
of force structure and use a capabilities-based planning approach limited
the study's usefulness in reassessing the fundamental relationship between
the national defense strategy and the force structure best suited to
implement the strategy. First, the Joint Staff did not vary the number and
types of units to demonstrate that it assessed different levels or mixes
of air, ground, maritime, and special operations force structure in its
second analysis. Second, the Joint Staff did not identify capabilities of
the force structure and make recommendations about trade-offs among
capabilities.
Further, concurrent with the Operational Availability 06 Study, DOD
conducted separate assessments of some segments of its force structure to
inform decisions about investments for capabilities. For example, DOD
conducted a departmentwide study that assessed options about different
levels and types of tactical air assets, such as the Joint Strike Fighter.
However, in this study DOD did not fully address whether and to what
extent future investment plans are affordable within projected funding
levels, and in April 2007, we reported that DOD does not have a single,
comprehensive, and integrated investment plan for recapitalizing and
modernizing fighter and attack aircraft.^24 In another example, DOD also
conducted a study to determine whether ground forces in the Army, Marine
Corps, and Special Operations Command could meet operational demands for a
broad range of scenarios without relying extensively on reserve
personnel.^25 However, options to increase ground forces were not part of
the study's scope, and the implications of the ongoing operations in Iraq,
such as the number of active brigade combat teams that would be needed and
their length of time in theater, were not fully considered.^26
^24GAO, Tactical Aircraft: DOD Needs a Joint and Integrated Investment
Strategy, [34]GAO-07-415 (Washington, D.C.: Apr. 2, 2007).
^25In December 2004, the Deputy Secretary of Defense directed the
Quadrennial Defense Review to review ground forces capability.
A key reason why DOD did not use an integrated capabilities-based approach
to assessing force structure options is that DOD did not have a unified
management approach to implement capabilities-based planning principles
into the QDR assessment. At the time of the QDR, no one individual or
office had been assigned overall responsibility and authority necessary
for implementing an integrated capabilities-based planning approach.
Further, DOD had not provided comprehensive written guidance to implement
departmentwide methods for capabilities-based planning that specifies the
need to identify capabilities at the appropriate level of detail, identify
redundant or excess capabilities that could be eliminated, facilitate
trades among capabilities, assess and manage risk, and balance decisions
about trade-offs with near- and long-term costs. Currently, DOD is
undertaking some initiatives related to capabilities-based planning.
However, these select initiatives do not represent the type of
comprehensive, unified management approach needed to assess the force
structure requirements to address a range of potential military operations
against unknown enemies. For example:
o The Joint Staff initiated the Joint Capabilities Integration and
Development System in 2003 to assess gaps in joint capabilities
and recommend solutions to resolve those gaps. Under this system,
boards comprised of high-level DOD civilians and military
officials are convened to identify future capabilities needed in
key functional areas, such as battle space awareness, and to make
recommendations about trade-offs among air, space, land, and sea
platforms. While this process may be important to assess gaps in
joint warfighting capabilities, we have reported that its focus is
to review and validate the initial need for proposed capabilities.
However, we have also reported that the process is not yet
functioning as envisioned to define gaps and redundancies in
existing and future military capabilities across the department
and to identify solutions to improve joint capabilities.^27
Further, we reported that programs assessed by the Joint Staff's
process build momentum and move toward starting product
development with little if any early department-level assessment
of the costs and feasibility.^28 According to senior DOD
officials, the Joint Staff's process does not thoroughly link
capabilities to the strategic priorities identified in the QDR.
o The Deputy Secretary of Defense tasked the Institutional Reform
and Governance post-QDR study team to develop departmentwide
approaches that would allow DOD to integrate and facilitate its
capabilities-based planning initiatives. Based on the study team's
work, in March 2007 the Deputy Secretary of Defense tasked several
DOD organizations to develop plans to facilitate a
capabilities-based planning approach. For example, the Joint
Requirements Oversight Council is tasked with developing a process
for identifying capability priorities and gaps at the appropriate
level of detail and ranking all capabilities from high to low
priority by October 2007. Further, the Deputy Secretary of Defense
has reaffirmed the department's commitment to portfolio management
and expanded the scope of responsibility for the four capability
portfolio test case managers.^29 Among their new responsibilities,
each portfolio manager is required to provide the Deputy's
Advisory Working Group with an independent portfolio assessment to
inform investment decisions during DOD's fiscal year 2009 program
review. DOD may establish more portfolios as the roles and
responsibilities of the existing managers evolve and operate in
DOD's existing decision processes, such as the Deputy's Advisory
Working Group.
^26The operational demands related to Iraq's security environment were not
part of DOD's planning scenarios used in the ground forces study.
^27GAO, Force Structure: Joint Seabasing Would Benefit from a
Comprehensive Management Approach and Rigorous Experimentation before
Services Spend Billions on New Capabilities, [60]GAO-07-211 (Washington,
D.C.: Jan. 26, 2007).
^28GAO, Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition Outcomes,
[61]GAO-07-388 (Washington, D.C.: Mar. 30, 2007).
^29In September 2006 the Deputy Secretary of Defense announced a new DOD
management effort to explore whether managing groups of military
capabilities across the entire department will enable DOD to improve the
interoperability of future capabilities, minimize capability redundancies
and gaps, and maximize the effectiveness of capabilities. DOD identified
the following test cases: battlespace awareness, joint command and
control, joint net centric operations, and joint logistics.
DOD made some changes to the current force structure to address
perceived gaps in capabilities based on the QDR review, although
these did not represent major changes to the composition of the
existing force structure. For example, among the key
force-structure-related decisions highlighted in the QDR were to
(1) increase Special Operations forces by 15 percent and the
number of Special Forces Battalions by one-third; (2) expand
Psychological Operations and Civil Affairs units by 3,700
personnel, a 33 percent increase; (3) develop a new land-based
penetrating long-strike capability to be fielded by 2018 and fully
modernize the current bomber force (B-52s, B-1s, and B-2s); and
(4) decrease the number of active component brigade combat teams
from 43 to 42 and the number of planned Army National Guard
brigade combat teams from 34 to 28. In January 2007--about a year
after the QDR was completed--DOD approved the Army's plan to
increase the number of active component brigade combat teams to
48. Since DOD did not conduct a comprehensive, data-driven
assessment of force structure alternatives during the QDR, it is
not in the best position to assure itself or Congress that it has
identified the force best suited to execute the national defense
strategy.
DOD Did Not Conduct a Thorough Review of Personnel Requirements
Although DOD concluded in the 2006 QDR report that the size of
today's forces--both the active and reserve components across all
four military services--was appropriate to meet current and
projected operational demands, it did not provide a clear
analytical basis for its conclusion. In January 2007, the
Secretary of Defense announced plans to permanently increase the
size of the active component Army and the Marine Corps by a total
of 92,000 troops over the next 5 years. But again, DOD did not
identify the analysis that it used to determine the size of the
increase. In February 2005,^30 we recommended that DOD review
active personnel requirements as part of the QDR, and in doing so,
discuss its conclusions about the appropriate personnel levels for
each of the services and describe the key assumptions guiding the
department's analysis, the methodology used to evaluate
requirements, and how the risks associated with various
alternative personnel force levels were evaluated. While DOD
agreed with our recommendation, it did not perform a
comprehensive, data-driven analysis of the number of personnel
needed to implement the defense strategy as part of its 2006 QDR.
Until DOD performs a comprehensive review of personnel
requirements, it cannot effectively demonstrate to Congress a
sound basis for the level of military and civilian personnel it
requests.
Our prior work has shown that valid and reliable data about the
number of personnel required to meet an agency's needs are
critical because human capital shortfalls can threaten an
organization's ability to perform missions efficiently and
effectively.^31 Data-driven decision making is one of the critical
factors in successful strategic workforce management.
High-performing organizations routinely use current, valid, and
reliable data to inform decisions about current and future
workforce needs, stay alert to emerging mission demands, and
remain open to reevaluating their human capital practices.
Further, federal agencies have a responsibility to provide
thorough analytical support over significant decisions affecting
requirements for federal dollars so that Congress can effectively
evaluate the benefits, costs, and risks.
^30 [62]GAO-05-200 .
^31GAO, A Model of Strategic Human Capital Management, [63]GAO-02-373SP
(Washington, D.C.: Mar.15, 2002).
Rather than conducting a comprehensive assessment of its personnel
requirements, DOD's approach to active and reserve military
personnel and civilian personnel levels was to limit growth and
initiate efforts to use current personnel levels more efficiently.
Consequently, the study team that was assigned to review issues
related to manning and balancing the force took the existing force
size as a given. From that basis, the study team identified
alternative courses of action for changing the mix of specific
skills, such as civil affairs, in the active and reserve
components to meet future operational requirements. The team also
considered whether changes in the mix of skills would require more
military and civilian personnel at headquarters staffs. While
these reviews are important for understanding how to use the force
more efficiently, they cannot be used to determine whether U.S.
forces have enough personnel to accomplish missions successfully
because these reviews did not systematically assess the extent to
which different levels of end strength could fill DOD's combat
force structure and provide institutional support at an acceptable
level of risk.
Although DOD's 2006 QDR concluded that the Army and Marine Corps
should plan to stabilize their personnel levels at 482,400 and
175,000 active personnel respectively, by 2012, in February 2007
the President's fiscal year 2008 budget submission documented a
plan to permanently increase the size of the active components of
the Army by 65,000 to 547,400 and the Marine Corps by 27,000 to
202,000 over the next 5 years; and the Army National Guard by
8,200 to 358,200 and the U.S. Army Reserve by 6,000 to 206,000 by
2013. Shortly after the increase was announced, we testified
before Congress^32 that DOD's record in providing an analytically
driven basis for requested military personnel levels needs to be
improved and suggested that Congress should carefully weigh the
long-term costs and benefits in evaluating DOD's proposal for the
increases. Both the Army and Marine Corps are coping with
additional demands that were not fully reflected in the QDR. For
example, the Marine Corps decided to initiate a new study to
assess active military personnel requirements shortly after the
2006 QDR was completed due to its high pace of operations and the
QDR-directed changes in force structure, such as establishing a
Special Operations Command requiring about 2,600 military
personnel. Without performing a comprehensive analysis of the
number of personnel it needs, DOD cannot ensure that its military
and civilian personnel levels reflect the number of personnel
needed to execute the defense strategy. Further it cannot ensure
that it has a sufficient basis for understanding the risks
associated with different levels of military and civilian
personnel. For example, while too many active military personnel
could be inefficient and costly, having too few could result in
other negative consequences, such as the inability to provide the
capabilities that the military forces need to deter and defeat
adversaries.
^32 [64]GAO-07-397T .
DOD Did Not Conduct an Analytically Sound Risk Assessment of Its
Proposed Force Structure
During the 2006 QDR, the risk assessments conducted by the
Secretary of Defense and the Chairman of the Joint Chiefs of Staff
did not fully apply DOD's risk management framework to demonstrate
how risks associated with its proposed force structure were
evaluated.^33 DOD introduced its risk management approach in 2001;
however, we have reported that it has faced difficulty
implementing this approach.^34 For example, we found that DOD
faced challenges in integrating its risk management framework and
reform initiatives into a unified management approach. We have
reported that an emerging challenge for the federal government
involves the need for completion of comprehensive national threat
and risk assessments in a variety of areas.^35 For example,
evolving requirements from the changing security environment,
coupled with increasingly limited fiscal resources across the
federal government, emphasize the need for agencies to adopt a
sound approach to establishing resource decisions. We have
advocated that the federal government, including DOD, adopt a
comprehensive risk management approach as a framework for decision
making that fully links strategic goals to plans and budgets,
assesses values and risks of various courses of actions as a tool
for setting priorities and allocating resources, and provides for
the use of performance measures to assess outcomes.^36 A risk
management approach represents a series of analytical and
managerial steps that can be used to assess risk, evaluate
alternatives for reducing risks, choose among those alternatives,
implement the alternatives, monitor their implementation, and that
incorporate new information to adjust and revise the assessments
and actions, as needed.^37 Further, such a data-driven risk
assessment can provide a guide to help shape, focus, and
prioritize investment decisions to develop capabilities.
^33The QDR legislation requires that the Secretary of Defense assess the
nature and magnitude of the political, strategic, and military risks
associated with executing the missions called for under the national
defense strategy. The legislation also requires that upon the completion
of each QDR, the Chairman of the Joint Chiefs of Staff shall prepare and
submit to the Secretary of Defense the Chairman's assessment of the
review, including the assessment of risk. (10 U.S.C. S118.).
^34 [65]GAO-06-13 .
^35 [66]GAO-05-325SP .
A key reason why DOD did not apply its risk framework during the
QDR is that it had difficulty in developing department-level
measures that would be necessary to assess risk and as a result,
the assessment tools were not available for use during the QDR.
The QDR's study guidance tasked the QDR coordination group, led by
officials in the Office of the Under Secretary of Defense
(Policy), to review the QDR risk management guidelines and provide
these guidelines to the QDR's study teams for review. The
guidelines were to provide some examples of how to measure
performance related to DOD's key areas identified in its
framework--operational, force management, institutional, and
future challenges. The QDR coordination group was to incorporate
the study teams' feedback about recommended changes. Lastly, the
QDR coordination group was to issue the guidelines and monitor the
application of performance measures during the QDR. According to
an official in the Office of the Under Secretary of Defense
(Policy), the QDR coordination group had difficulty developing the
measures and thus did not issue guidelines. As a result, the study
teams did not have the assessment tools to assess risk during the
QDR.
Since department-level measures for assessing risk were not
available during the 2006 QDR, several of the study teams relied
primarily on professional judgment to assess the risks of not
investing in various capabilities. For example, the study team
responsible for developing capabilities told us that they examined
information about potential future threats and determined that DOD
needed medical countermeasures to address the threat of
genetically engineered biological agents. Members of the study
team discussed the consequences of not developing the medical
procedures and treatments that would be needed to increase
survival rates if U.S. military personnel were to encounter the
highly advanced genetic material. Further, the Chairman of the
Joint Chiefs of Staff was not tasked to use the risk management
framework in assessing risks and did not choose to use it in his
assessment. Rather, the Chairman's assessment examined the extent
to which the 2006 QDR initiatives would address combatant
commanders' operational needs for potential future requirements.
^36GAO, High-Risk Series: An Update, [67]GAO-05-207 (Washington, D.C.:
January 2005).
^37 [68]GAO-06-13 .
Without a sound analytical approach to assess risk during future
QDRs, DOD will not have a sufficient basis to demonstrate how the
risks associated with the capabilities of its proposed force
structure were evaluated. Further, DOD may be unable to
demonstrate how it will manage risk within current and expected
resource levels. Without an analytically based risk assessment,
DOD may not be able to prioritize and focus the nation's
investments to combat 21st century security threats efficiently
and wisely.
Options for Modifying Some Legislative Requirements Could Improve
Usefulness of Future QDRs
The security environment of the 21st century has been
characterized by conflicts that are very different from
traditional wars among states. This environment has created the
need for DOD to reexamine the fundamental operations of the
department and the capabilities needed to continue to execute its
missions. In addition, DOD has created new organizations, such as
the U.S. Northern Command and the Assistant Secretary of Defense
for Homeland Defense, to counter new threats to the homeland and
support the federal response to any potential catastrophic event,
natural, or man-made. Through our discussions with defense
analysts, we have identified options for modifying several QDR
legislative requirements that could be considered in light of the
changed security environment, to make the QDR process and report
more useful to Congress and DOD. The QDR legislation contains
numerous issues for DOD to address, some that require reporting on
broad issues, such as the national defense strategy and the force
structure needed to execute that strategy, and some that are more
detailed, such as the requirement that DOD examine the appropriate
ratio of combat forces to support forces under the national
defense strategy. Many defense analysts we spoke with thought some
of the strategic issues are of great importance and should remain
for future QDRs. Further, they believe DOD should focus its
efforts on providing more information on the analytic basis for
its key assumptions and strategic planning decisions. However,
they also asserted that several of the QDR's detailed reporting
elements detract attention from strategic issues, are already
required and reported under other laws, or are no longer relevant
in the new security environment. Options to improve the usefulness
of future QDRs include (1) clarifying expectations for how the QDR
should address the budget plan, (2) eliminating some reporting
elements for the QDR legislation that could be addressed in
different reports, (3) eliminating some reporting elements in the
QDR legislation for issues that may no longer be as relevant due
to changes in the security environment, and (4) establishing an
independent advisory group to work with DOD prior to and during
the QDR to provide alternative perspectives and analyses.
Strategic Focus of Some QDR Legislative Requirements Is Considered
Useful for Establishing Defense Plans and Programs
Several defense analysts we spoke with asserted that the permanent
requirement for DOD to conduct a comprehensive strategic review of
the defense program every 4 years is important and that Congress
should continue to require that DOD conduct future QDRs. Moreover,
several defense analysts acknowledge that certain key requirements
remain critical to the QDR's purpose of fundamentally reassessing
the defense strategy and program. Specifically, the requirements
that task the Secretary of Defense to (1) delineate a defense
strategy and (2) define sufficient force structure, force
modernization, budget plan, and other elements of a defense
program that could successfully execute the full range of missions
called for by the defense strategy at low to medium risk over 20
years were seen as critical elements needed to ensure that
Congress understands DOD's strategies and plans. Several defense
analysts told us that it is in the national interest to ensure
that DOD conducts the kind of long-range strategic planning that
can provide meaningful recommendations for meeting future national
security challenges and that enables debate on the costs and
benefits of requirements for future military and capabilities as
well as risks in capability gaps in light of national fiscal
challenges.
The QDR legislation also directs DOD to define the nature and
magnitude of the political, strategic, and military risks
associated with executing the missions called for under the
national defense strategy in the QDR and include a comprehensive
discussion of the force structure best suited to implement that
strategy at low-to-moderate level of risk. Analysts saw these
areas as important for DOD to provide Congress the assurance that
there is a sound analytical basis for its risk assessment that
includes how DOD identified risks and evaluated alternatives for
reducing risks. Additionally, analysts viewed this discussion as
important in assuring that the department has incorporated a
variety of perspectives in its risk assessments. Some analysts
stated that the requirements to discuss the assumed or defined
national security interests, the threats to the assumed or defined
national security interests, and the scenarios developed in the
examination of those threats are several key elements that should
remain to enable the department to demonstrate that principles of
risk assessment have been addressed. Similarly, analysts suggest
that the Chairman of the Joint Chiefs of Staff's requirement to
assess the results of the QDR review, including an independent
assessment of risk, is helpful to provide another assessment that
DOD and Congress can use to understand the risks associated with
the force structure and consider the courses of actions the
department might want to take to reduce risks.
Legislative Options Are Available to Improve Usefulness of Future QDRs
Some DOD defense analysts told us that the QDR legislation
includes numerous detailed requirements that may impede DOD's
focus on high-priority areas. Based on our discussions with
analysts, we identified several options that Congress should
consider to enhance the focus of future QDRs on high-priority
issues and improve the thoroughness of DOD's analysis:
o Clarify expectations for how the QDR should address the budget
plan that supports the national defense strategy. The QDR
legislation has several reporting elements that relate to budget
planning to support the defense strategy. First, the QDR
legislation requires DOD "to delineate a national defense
strategy..." and "to identify the budget plan that would be
required to provide sufficient resources to execute successfully
the full range of missions called for in that national defense
strategy at a low-to-moderate level of risk."^38 Second, the
legislation requires DOD "to conduct a comprehensive
examination...of the national defense strategy...with a view
toward establishing a defense program for the next 20 years."
Third, based on recent changes to the legislation that will apply
to the next QDR in 2010 as well as future QDRs, DOD is required to
"make recommendations that are not constrained to comply with the
budget submitted to Congress by the President."
Some defense analysts raised concerns about whether these
reporting requirements provide sufficient and clear guidance for
DOD to use in conducting QDRs. For example, they questioned
whether the planning time frame of 20 years established by the QDR
legislation is most useful in providing Congress with information
to perform its oversight of the defense program. Although DOD
officials and defense analysts acknowledged the benefits of
forecasting threats and capabilities for a 20-year period, they
stated it would be difficult to develop a detailed budget plan for
a 20-year period given the uncertain nature of threats in the new
security environment. Further, analysts asserted that rather than
enabling DOD to set strategic priorities without regard to current
budgets, the requirement to "make recommendations that are not
constrained to comply with the budget..," could lead the services
and the capability portfolio managers to push for inclusion of
every program in their plans. This could make it more difficult
for DOD to prioritize investments to meet key capability needs and
assess the affordability of new capabilities across the
department.
^38DOD is also required to identify any additional resources (beyond those
programmed in the current future years defense program) required to
achieve such a level of risk.
Moreover, DOD's three QDR reports since 1997 have not fully
described DOD's methodology or approach for assessing its budget
needs or budget plans that explained how DOD intended to fund the
full range of missions called for in the national defense
strategy. For the 2006 QDR, DOD included several QDR initiatives
in the President's fiscal year 2007 budget that was submitted to
Congress at the same time as the QDR report but stated that it
would continue to define a budget plan for the QDR by identifying
the funding details in DOD's future years defense program for
fiscal years 2008 through 2013. In addition, the report did not
provide information about the extent to which DOD considered the
long-term affordability of the overall defense program. We have
emphasized in previous reports that the federal government now
faces increasing fiscal challenges, and DOD may face increasing
competition for federal dollars.^39 Further, in November 2005, we
reported that DOD has not demonstrated discipline in its
requirements and budgeting processes, and its costly plans for
transforming military operations and expensive acquisitions may
not be affordable in light of the serious budget pressures facing
the nation.^40 For example, we reported that DOD's planned annual
investment in acquisition programs it has already begun is
expected to rise from $149 billion in fiscal year 2005 to $178
billion in fiscal year 2011. Given these pressures, Congress may
want a clearer view of how DOD should budget for the capabilities
associated with the proposed force structure, and how it evaluated
the trade-offs in capabilities to maximize the effectiveness of
future investments. If Congress decides that it needs additional
budget-related information to carry out its oversight of future
QDRs, then it might consider clarifying the reporting element
relating to the required budget plan to specify what information
DOD should include in the QDR. Further, Congress may want to
consider clarifying its expectations for the information DOD
provides in the QDR as to how it has addressed the long-term
affordability challenges of transforming military operations.
^39For example, see [69]GAO-05-325SP .
^40GAO, DOD Acquisition Outcomes: A Case for Change, [70]GAO-06-257T
(Washington, D.C.: Nov. 15, 2005).
o Eliminate some reporting elements in the QDR legislation for
issues that could be addressed in different reports. According to
some defense analysts, some requirements contained in the QDR
legislation are not essential to the strategic purpose of the QDR
and may divert DOD's focus from that strategic purpose. While
important, some reporting elements are already examined in other
DOD reviews, and Congress has access to the results of these
periodic reviews. These reporting elements include the following:
o An evaluation of "the strategic and tactical
airlift, sealift, and ground transportation
capabilities required to support the national defense
strategy." In November 2002 we reported that the QDR
may not be the appropriate venue for addressing
mobility issues because examination of this issue
requires detailed analysis that can best be conducted
after DOD decides on a defense strategy, identifies a
range of planning scenarios consistent with the new
strategy, and completes its detailed analysis of
requirements for combat forces.^41 Furthermore, DOD
routinely conducts analyses of its mobility
requirements outside of the QDR process, according to
DOD officials. Since 1992, DOD has issued four major
analyses of the U.S. military strategic lift
requirements: the 1992 Mobility Requirements Study,
Bottom Up Review; the 1995 Bottom Up Review Update;
the 2001 Mobility Requirements Study--2005, issued in
2001; and the Mobility Capability Study, issued in
2005.
o An assessment of the "advisability of revisions to
the Unified Command Plan as a result of the national
defense strategy." DOD has a process for assessing
the Unified Command Plan and is required to report
changes to the plan to Congress under other
legislation. Specifically, the Chairman of the Joint
Chiefs of Staff is required to review periodically
and not less than every 2 years the missions,
responsibilities, and forces of each combatant
command and recommend any changes to the President,
through the Secretary of Defense.^42 This legislation
also requires that, except during times of
hostilities or the imminent threat of hostilities,
the President notify Congress not more than 60 days
after either establishing a new combatant command or
significantly revising the missions,
responsibilities, or force structure of an existing
command. As such, a major event or change in the
political or security landscape could trigger the
need for a change in the plan. For example, in the
spring of 2007, the President announced that DOD
intends to establish a U.S. Africa Command to oversee
military operations on the African continent.
According to an OSD official, DOD will revise the
2002 Unified Command Plan and report on the changes
in the military command structure after plans for
U.S. Africa Command are more fully developed.
^41 [71]GAO-03-13 .
^4210 U.S.C. S161(b).
o Eliminate some reporting elements in the QDR legislation for
issues that may no longer be as relevant due to changes in the
security environment. As we reported in our assessment of DOD's
2001 QDR,^43 a DOD official and some defense analysts said that
two reporting elements should be eliminated because they are
related to the allocation of forces under the old
two-major-theater-war construct, which is more limited than DOD's
current force planning construct that includes a broader range of
threats. These reporting elements include the following:
o A discussion of the "appropriate ratio of combat
forces to support forces (commonly referred to as the
`tooth-to-tail ratio') under the national defense
strategy." DOD's goal has been to reduce the number
of personnel and costs associated with the support
forces, or "tail." However, during the 2006 QDR
process and report DOD did not identify which units
should be considered support and which should be
considered combat. Given rapidly changing
technologies, differentiating between support and
combat troops has become increasingly irrelevant and
difficult to measure. For example, as the United
States moves toward acquiring greater numbers of
unmanned aircraft piloted from remote computer
terminals and relies increasingly on space-based
assets operated by personnel in the United States, it
will be more difficult to distinguish between combat
and support personnel.
o Assessments of "the extent to which resources must
be shifted among two or more theaters under the
national defense strategy in the event of conflict in
such theaters," and the assumptions used regarding
"warning time." Both these reporting elements relate
to the allocation of forces under the old
two-major-theater-war planning construct. Under this
construct, the amount of time that was assumed
available for warning and the separation time between
major theater wars were critical factors in planning
the size and composition of U.S. forces and assessing
operational risk, particularly for assets that might
need to be shifted between theaters. However, under
the new defense strategy, along with DOD's new force
planning construct, DOD assumes that it will continue
to be involved in a wide range of military operations
around the world. Given the full spectrum of threats
that DOD is planning to address, it may be more
useful for DOD's force structure assessments to be
tied to requirements for a broad range of potential
threats.
^43 [72]GAO-03-13 .
o Establish an independent advisory group to work with DOD prior
to or during the QDR to provide alternative perspectives and
analyses. As part of our assessment of the 1997 QDR, we suggested
that a congressionally mandated panel, such as the 1997 National
Defense Panel, could be used to encourage DOD to consider a wider
range of strategy, force structure, and modernization options.
Specifically, we noted that such a review panel, if it preceded
the QDR, could be important because it is extremely challenging
for DOD to conduct a fundamental reexamination of defense needs,
given that its culture rewards consensus building and often makes
it difficult to gain support for alternatives that challenge the
status quo. One of the recent additions to the QDR legislation
requires the establishment of an independent panel to conduct an
assessment of future QDRs after the process is completed; however,
most defense analysts we spoke with agreed that an independent
analysis of key issues for the Secretary of Defense either prior
to or during the next review would complement a post-QDR
assessment and strengthen DOD's ability to develop its strategic
priorities and conduct a comprehensive force structure and
capabilities analysis. The analysts agreed that an advisory group
established before or during the QDR process could function as an
independent analytical team to challenge DOD's thinking, recommend
issues for DOD to review and review assumptions, and provide
alternative perspectives in activities such as identifying
alternative force structures and capabilities, and performing risk
assessments. An independent group's assessments could be useful to
DOD in future QDRs to identify the capabilities of the nation's
current and future adversaries because potential enemies will
likely be more difficult to target than the adversaries of the
Cold War era.
Conclusions
The 2006 QDR represented an opportunity for DOD to perform a
comprehensive review of the national defense strategy for the
first time since military forces have been engaged in the Global
War on Terrorism. Sustained DOD leadership facilitated decision
making, and extensive collaboration with interagency partners and
allies provided a range of perspectives on threats and
capabilities. However, weaknesses in DOD's analysis of force
structure, personnel requirements, and risk limited its
reassessment of the national defense strategy and U.S. military
forces. For example, by not fully incorporating capabilities-based
planning into a comprehensive assessment of alternative force
structures, DOD could not comprehensively identify capabilities
gaps, associated operational risks, and trade-offs that must be
made to efficiently use limited fiscal resources. Therefore, DOD
was not in a good position to assure Congress that it identified
the force best suited to execute the national defense strategy.
Moreover, the Secretary of Defense's announcement of plans to
increase the sizes of the Army and Marine Corps in January 2007
calls into question the analytical basis of the QDR conclusion
that the number of personnel and the size of the force structure
for the services were appropriate to meet current and future
requirements. Further, without a comprehensive approach to
assessing risk, DOD's 2006 QDR did not provide a sufficient basis
to demonstrate how risks associated with its proposed force
structure were evaluated. Unless DOD takes steps to provide
comprehensive analytical support for significant decisions in
future QDRs, the department will not be in the best position to
distinguish between the capabilities it needs to execute the
defense strategy versus those capabilities it wants but may not be
able to afford at a time when the nation's fiscal challenges are
growing. Moreover, Congress will be unable to effectively evaluate
the benefits, costs, and risks associated with decisions flowing
from future QDRs.
Opportunities exist for Congress to consider further changes to
the QDR legislation that may encourage DOD to concentrate its
efforts on high-priority matters such as developing a defense
strategy and identifying the force structure best suited to
execute the strategy. Unless Congress clearly identifies its
expectations for DOD to develop a budget plan that supports the
strategy, DOD may not thoroughly address the challenges it will
face as it competes with other federal agencies and programs for
taxpayers' dollars and may spend considerable effort assessing
options for capabilities that could be unaffordable given our
nation's fiscal challenges. Moreover, the large number of
reporting elements in the QDR legislation presents DOD with a
challenge in conducting data-driven comprehensive analyses of many
significant complex issues. A reassessment of the QDR's scope
could provide greater assurances that DOD will thoroughly assess
and report on the most critical security issues that the nation
faces and could help it decide what actions it needs to take to
establish the most effective military force to counter 21st
century threats. Lastly, although Congress has established a new
legislative requirement for an independent panel to conduct a
post-QDR review, there is currently no mechanism for Congress and
the Secretary of Defense to obtain an independent perspective
prior to and during the QDR. Without an independent group of
advisors that could provide comprehensive data-driven analyses to
DOD prior to and during future QDR reviews, DOD may not consider a
wider range of perspectives, such as force structure options, thus
limiting the analytic basis of its QDR decisions.
Recommendations for Executive Action
To enhance the usefulness of future QDRs and assist congressional
oversight, we recommend that the Secretary of Defense take the
following two actions:
o Develop appropriate methods for the department to use in a
comprehensive, data-driven capabilities-based assessments of
alternative force structures and personnel requirements during
future QDRs.
o Develop appropriate methods for the department to use in
conducting a comprehensive, data-driven approach to assess the
risks associated with capabilities of its proposed force structure
during future QDRs.
Matters for Congressional Consideration
To improve the usefulness of future QDRs, Congress should consider
revisions to the QDR legislation, including (1) clarifying
expectations on how the QDR should address the budget plan that
supports the national defense strategy, (2) eliminating some
detailed reporting elements that could be addressed in different
reports and may no longer be relevant due to changes in the
security environment, and (3) requiring an independent panel to
provide advice and alternatives to the Secretary of Defense before
and during the QDR process.
Agency Comments and Our Evaluation
The Principal Deputy Under Secretary of Defense for Policy
provided written comments on a draft of this report. The
department partially agreed with our recommendations and agreed
with the matters we raised for congressional consideration
regarding possible changes to the QDR legislative language. In
addition, the comments provided information about steps the
department is taking to update its methodologies for analyzing
force structure requirements and assessing risks. DOD's comments
are reprinted in their entirety in appendix IV. DOD also provided
technical comments which we incorporated as appropriate.
In its comments, the department partially agreed with our
recommendation that the Secretary of Defense develop appropriate
methods for conducting comprehensive, data-driven
capabilities-based assessments of alternative force structures and
personnel requirements. DOD agreed with our conclusion that the
2006 QDR did not comprehensively assess alternatives to planned
structure; rather, its analysis was limited to identifying
shortfalls in current structure when compared to various
illustrative operational scenarios. However, in its comments, the
department noted that it has developed or is developing new
illustrative security environments to use to demonstrate the
demands associated with force structures and personnel
requirements for each strategic environment. The department also
pointed out the difficulty of undertaking an evaluation of the
defense strategy and producing a defense program within the QDR
process, as required under current QDR legislation. It said that
as the department further develops the underlying assumptions for
the force planning construct and refreshes the illustrative
scenarios available for analysis, it will be in a better position
to analyze overall needed capabilities, including personnel
requirements. Finally, the department noted that the 2006 QDR was
based on information available in 2005, which included a different
demand than what military forces face today. At that time, the
department's collective decision, approved by the then Secretary
of Defense, was that the size of the force was about right,
although the force mix should be adjusted. As a result of this
change in demand since the 2006 QDR, according to DOD's comments,
DOD has responded by increasing Army and Marine Corps end
strength. We believe that the steps DOD outlined in its comments,
such as revising the illustrative scenarios and developing force
demands for new security environments, will help DOD to improve
its force structure analyses. However, we believe that a
comprehensive assessment that identifies and documents the basis
for trade-off decisions across capability areas is critical to
developing the force structure best suited to execute the defense
strategy. Until DOD undertakes a comprehensive assessment of
alternative force structure options that clearly documents how the
department reached its force structure decisions, it will not be
in the best position to determine the force structure best suited
to execute the missions called for in the defense strategy at
low-to-moderate risk.
DOD also partially concurred with our recommendation to develop
appropriate methods for conducting comprehensive, data-driven
assessments of the risks associated with the capabilities of its
proposed force structure during future QDRs. In its comments, the
department agreed that improving the department's risk methodology
is necessary to appropriately assess risk. It noted that in
addition to risks associated with capabilities, strategic,
operational, force management, and institutional risks need to be
addressed in a risk assessment methodology. The department cited
several post-QDR initiatives the department is undertaking to
improve how the department assesses risk, including new measures
to help link strategic goals to plans and budgets and develop
performance metrics. Also in its comments, the department
described efforts to strengthen and integrate existing assessments
to allow decision makers to better set priorities, allocate
resources, and assess outcomes and risks and stated its intent to
improve risk assessment methods to inform risk measurement in
future QDRs. We agree that assessing risk associated with
capabilities is only one type of risk facing the department and
that the initiatives the department is undertaking to link
strategic goals with plans and budgets and improve its risk
assessment methodology can, when implemented, help it improve its
ability to identify and manage risks. Until the department's risk
management framework is sufficiently developed that it can support
comprehensive assessments of risk across domains, assess progress
toward accomplishing strategic goals, and provide senior leaders
reliable analysis to inform decisions among alternative actions,
DOD will not be in the best position to identify or assess risks
to establish investment priorities.
DOD also provided its views on matters we raised for congressional
consideration in a draft of this review regarding possible
revisions to the QDR legislation. Specifically, DOD agreed with
clarifying expectations for addressing the budget plan and
eliminating some reporting requirements. In a draft of this
report, we originally raised as a matter for congressional
consideration broadening the QDR legislation by requiring the
legislatively required independent advisory panel, which would
provide a post-QDR critique of the results of the process, to
provide DOD with alternative perspectives and analysis prior to or
during the QDR. The department stated that having an independent
panel that could provide advice and alternatives to the Secretary
of Defense before and during the QDR process would be useful.
However, it raised the concern that tasking the same independent
panel that is required to provide a post-QDR critique to also
perform an advisory function before and during the review could
create mistrust between the department leadership and the
independent advisory panel. To address DOD's concerns we have
modified the matter for consideration to suggest that an
independent panel be required to provide advice and alternatives
to the Secretary of Defense before and during the QDR. This change
is intended to provide Congress with the flexibility to establish
separate independent panels to provide advice prior to and
following the next QDR.
We are sending copies of this report to other appropriate
congressional committees and the Secretary of Defense. We will
also make copies available to other interested parties upon
request. In addition, the report will be available at no charge on
the GAO Web site at [35]http://www.gao.gov . If you have any
questions about this report, please contact me at (202) 512-4402.
Contact points for our Offices of Congressional Relations and
Public Affairs may be found on the last page of this report. Major
contributors to this report are listed in appendix V.
Janet A. St. Laurent
Director, Defense Capabilities and Management
Appendix I: Scope and Methodology
To assess the strengths and weaknesses of the Department of
Defense's (DOD) approach and methodology for the 2006 Quadrennial
Defense Review (QDR), we examined the relevant documentation
including the John Warner National Defense Authorization Act for
Fiscal Year 2007; the National Defense Strategy of the United
States of America (March 2005); the 1997, 2001, and 2006 QDRs; the
QDR Terms of Reference (March 2005); the Under Secretary of
Defense (Policy) issue papers for the QDR's focus areas; and the
2006 QDR's study teams' briefings and other documentation for the
DOD's senior-level review group, as well as our reports on aspects
of previous QDRs. We also examined documents identifying the
methodology and results of the QDR's key force structure analyses
and risk assessments. We reviewed studies on capabilities-based
planning and compared the key elements of capabilities-based
planning identified in the studies to the QDR's Terms of Reference
and DOD's documented methodology for the Operational Availability
06 Study to assess the extent to which capabilities-based planning
concepts were used during the QDR. We also discussed these issues
with officials from the Office of the Under Secretary of Defense
(Policy); the Office of Program Analysis and Evaluation; the Joint
Chiefs of Staff Directorate for Force Assessment; U.S. Special
Operations Command; and officials from the Army, Air Force, and
Marine Corps who participated in the QDR process. To understand
how DOD established processes to ensure that QDR initiatives are
implemented, we examined internal DOD documents, DOD's January
2007 quarterly report to Congress on the status of implementation
of the 2006 QDR, and post-QDR study teams' reports to understand
the methodology that was developed to oversee implementation. We
discussed the implementation status of the QDR initiatives with
officials from the Office of the Director, Administration and
Management and the Under Secretary of Defense (Policy). We did not
undertake an assessment of the effectiveness of implementation of
the QDR initiatives because it was outside of scope of our review.
We obtained and examined documents from the Deputy Secretary of
Defense and the post-QDR study teams and discussed the status of
the teams' work with officials from the Under Secretary of Defense
(Policy), the Institutional Reform and Governance team, and the
Joint Command and Control and Battlespace Awareness capability
portfolios. Moreover, we reviewed the internal controls on DOD's
tracking system for QDR initiatives and evaluated the reliability
of that data for DOD's use. We applied evidence standards from the
generally accepted government auditing standards in our evaluation
of DOD's database. As a result, we determined the information we
used meets these evidence standards and is sufficiently reliable
for our purposes.
To determine whether changes to the QDR legislation could improve
the usefulness of future reviews including any changes needed to
better reflect the security conditions of the 21st century, we
examined a wide variety of studies that discussed the strengths
and weaknesses of DOD's 2006 QDR and prior reviews. Our review
included studies from the RAND Corporation, the National Defense
University, and the Center for Strategic and Budgetary
Assessments. To obtain opinions and develop options to improve the
usefulness of future QDRs, we interviewed several DOD officials
who participated in the 2006 QDR from the services and the Joint
Staff. Further, we met with 11 defense analysts who had detailed
knowledge of DOD's QDR process and/or participated in DOD's 1997,
2001, or 2006 QDRs. We used a standard set of questions to
interview each of these analysts to ensure we consistently
discussed the reporting elements of the QDR legislation and DOD's
approach and methods for its three QDRs. To develop the questions,
we reviewed the QDR legislation, DOD's QDR reports, and our prior
work on DOD's strategic reviews. One of the defense analysts
served in various positions within and outside of DOD such as the
former Chairman of the Defense Science Board and the Chairman of
the 1997 National Defense Panel. Other defense analysts were
senior officials from the following organizations: the American
Enterprise Institute, the Center for American Progress, the Center
for Naval Analysis, the Center for a New American Security, the
Center for Strategic and Budgetary Assessments, the Center for
Strategic and International Studies, the Lexington Institute, the
National Defense University's Institute for National Strategic
Studies, the RAND Corporation, and the Heritage Foundation. Based
on our review of QDR literature and our discussions with DOD
analysts, we developed a matrix summarizing these individuals'
concerns regarding the QDR legislative requirements and their
views on the options to address them.
Our work was conducted in the Washington, D.C., metropolitan area
and Tampa, Florida. We performed our review from May 2006 through
May 2007 in accordance with generally accepted government auditing
standards.
Appendix II: Quadrennial Defense Review Legislation in Effect for
the 2006 Quadrennial Defense Review
TITLE 10 U.S.C. S118. Quadrennial Defense Review
(a) Review required.--The Secretary of Defense shall every four
years, during a year following a year evenly divisible by four,
conduct a comprehensive examination (to be known as a "quadrennial
defense review") of the national defense strategy, force
structure, force modernization plans, infrastructure, budget plan,
and other elements of the defense program and policies of the
United States with a view toward determining and expressing the
defense strategy of the United States and establishing a defense
program for the next 20 years. Each such quadrennial defense
review shall be conducted in consultation with the Chairman of the
Joint Chiefs of Staff.
(b) Conduct of review.--Each quadrennial defense review shall be
conducted so as--
(1) to delineate a national defense strategy consistent with the
most recent National Security Strategy prescribed by the President
pursuant to section 108 of the National Security Act of 1947 (50
U.S.C.404a);
(2) to define sufficient force structure, force modernization
plans, infrastructure, budget plan, and other elements of the
defense program of the United States associated with that national
defense strategy that would be required to execute successfully
the full range of missions called for in that national defense
strategy;
(3) to identify (A) the budget plan that would be required to
provide sufficient resources to execute successfully the full
range of missions called for in that national defense strategy at
a low-to-moderate level of risk, and (B) any additional resources
(beyond those programmed in the current future-years defense
program) required to achieve such a level of risk; and
(c) Assessment of risk.--The assessment of risk for the purposes
of subsection (b) shall be undertaken by the Secretary of Defense
in consultation with the Chairman of the Joint Chiefs of Staff.
That assessment shall define the nature and magnitude of the
political, strategic, and military risks associated with executing
the missions called for under the national defense strategy.
(d) Submission of QDR to Congressional committees.--The Secretary
shall submit a report on each quadrennial defense review to the
Committees on Armed Services of the Senate and the House of
Representatives. The report shall be submitted in the year
following the year in which the review is conducted, but not later
than the date on which the President submits the budget for the
next fiscal year to Congress under section 1105(a) of title 31.
The report shall include the following:
(1) The results of the review, including a comprehensive
discussion of the national defense strategy of the United States,
the strategic planning guidance, and the force structure best
suited to implement that strategy at a low-to-moderate level of
risk.
(2) The assumed or defined national security interests of the
United States that inform the national defense strategy defined in
the review.
(3) The threats to the assumed or defined national security
interests of the United States that were examined for the purposes
of the review and the scenarios developed in the examination of
those threats.
(4) The assumptions used in the review, including assumptions
relating to-- (A) the status of readiness of United States forces;
(B) the cooperation of allies, mission-sharing and additional
benefits to and burdens on United States forces resulting from
coalition operations; (C) warning times; (D) levels of engagement
in operations other than war and smaller-scale contingencies and
withdrawal from such operations and contingencies; and (E) the
intensity, duration, and military and political end-states of
conflicts and smaller-scale contingencies.
(5) The effect on the force structure and on readiness for
high-intensity combat of preparations for and participation in
operations other than war and smaller-scale contingencies.
(6) The manpower and sustainment policies required under the
national defense strategy to support engagement in conflicts
lasting longer than 120 days.
(7) The anticipated roles and missions of the reserve components
in the national defense strategy and the strength, capabilities,
and equipment necessary to assure that the reserve components can
capably discharge those roles and missions.
(8) The appropriate ratio of combat forces to support forces
(commonly referred to as the `tooth-to-tail' ratio) under the
national defense strategy, including, in particular, the
appropriate number and size of headquarters units and Defense
Agencies for that purpose.
(9) The strategic and tactical air-lift, sea-lift, and ground
transportation capabilities required to support the national
defense strategy.
(10) The forward presence, pre-positioning, and other anticipatory
deployments necessary under the national defense strategy for
conflict deterrence and adequate military response to anticipated
conflicts.
(11) The extent to which resources must be shifted among two or
more theaters under the national defense strategy in the event of
conflict in such theaters.
(12) The advisability of revisions to the Unified Command Plan as
a result of the national defense strategy.
(13) The effect on force structure of the use by the armed forces
of technologies anticipated to be available for the ensuing 20
years.
(14) The national defense mission of the Coast Guard.
(15) Any other matter the Secretary considers appropriate.
(e) CJCS review.--(1) Upon the completion of each review under
subsection (a), the Chairman of the Joint Chiefs of Staff shall
prepare and submit to the Secretary of Defense the Chairman's
assessment of the review, including the Chairman's assessment of
risk.
(2) The Chairman shall include as part of that assessment the
Chairman's assessment of the assignment of functions (or roles and
missions) to the armed forces, together with any recommendations
for changes in assignment that the Chairman considers necessary to
achieve maximum efficiency of the armed forces. In preparing the
assessment under this paragraph, the Chairman shall consider
(among other matters) the following:
(A) unnecessary duplication of efforts among the armed forces.
(B) changes in technology that can be applied effectively to
warfare.
(3) The Chairman's assessment shall be submitted to the Secretary
in time for the inclusion of the assessment in the report. The
Secretary shall include the Chairman's assessment, together with
the Secretary's comments, in the report in its entirety.
Appendix III: Summary of New Changes in 10 U.S.C. S118 for Future
Quadrennial Defense Reviews
This appendix provides a summary of changes to the Quadrennial
Defense Review (QDR) legislation (10 U.S.C. S118) as a result of
the John Warner National Defense Authorization Act for Fiscal Year
2007.^1 The new requirements will be in effect when the Department
of Defense issues its next quadrennial review in 2010.
o The QDR should make recommendations that are not constrained to
comply with the budget submitted to Congress by the President.
o The review shall include the following new reporting elements:
o the specific capabilities, including the general
number and type of specific military platforms,
needed to achieve the strategic and warfighting
objectives identified in the review; and
o the homeland defense and support to civil authority
missions of the active and reserve components,
including the organization and capabilities required
for the active and reserve components to discharge
each such mission.
o The Chairman shall describe the capabilities needed to address
the risk that he identified in his risk assessment.
o The Secretary of Defense shall establish an independent panel to
conduct an assessment of the QDR not later than 6 months before
the date on which the QDR will be submitted.
o Not later than 3 months after the date on which the
QDR is submitted, the panel shall submit an
assessment of the review, including the review's
recommendations, the stated and implied assumptions
incorporated in the review, and the vulnerabilities
of the strategy and force structure underlying the
review.
o The panel's assessment shall include analyses of
the trends, asymmetries, and concepts of operations
that characterize the military balance with potential
adversaries, focusing on the strategic approaches of
possible opposing forces.
^1Pub. L. No. 109-364 S1031 (2006).
Appendix IV: DOD Comments
Appendix V: GAO Contact and Staff Acknowledgments
GAO Contact
Janet St. Laurent (202) 512-4402, or [36][email protected]
Acknowledgments
In addition to the contact name above, Margaret Morgan, Assistant
Director; Deborah Colantonio; Alissa Czyz; Nicole Harms; Elizabeth
Morris; Brian Pegram; Rebecca Shea; and John Townes made major
contributions to this report.
Related GAO Products
Tactical Aircraft: DOD Needs a Joint and Integrated Investment
Strategy. [37]GAO-07-415 . Washington, D.C.: April 2, 2007.
Best Practices: An Integrated Portfolio Management Approach to
Weapon System Investments Could Improve DOD's Acquisition
Outcomes. [38]GAO-07-388 . Washington, D.C.: March 30, 2007.
United States Government Accountability Office: Supporting the
Congress through Oversight, Insight, and Foresight.
[39]GAO-07-644T . Washington, D.C.: March 21, 2007.
Fiscal Stewardship and Defense Transformation. [40]GAO-07-600CG .
Washington, D.C.: March 8, 2007.
Homeland Security: Applying Risk Management Principles to Guide
Federal Investments. [41]GAO-07-386T . Washington, D.C.: February
7, 2007.
Military Personnel: DOD Needs to Provide a Better Link between Its
Defense Strategy and Military Personnel Requirements.
[42]GAO-07-397T . Washington, D.C.: January 30, 2007.
Force Structure: Joint Seabasing Would Benefit from a
Comprehensive Management Approach and Rigorous Experimentation
before Service Spend Billions on New Capabilities. [43]GAO-07-211
. Washington, D.C.: January 26, 2007.
Force Structure: Army Needs to Provide DOD and Congress More
Visibility Regarding Modular Force Capabilities and Implementation
Plans. [44]GAO-06-745 .Washington, D.C.: September 6, 2006.
Force Structure: DOD Needs to Integrate Data into Its Force
Identification Process and Examine Options to Meet Requirements
for High-Demand Support Forces. [45]GAO-06-962 . Washington, D.C.:
September 5, 2006.
DOD Acquisition Outcomes: A Case for Change. [46]GAO-06-257T .
Washington, D.C.: November 15, 2005.
Defense Management: Additional Actions Needed to Enhance DOD's
Risk-Based Approach for Making Resource Decisions. [47]GAO-06-13 .
Washington, D.C.: November 15, 2005.
DOD's High-Risk Areas: Successful Business Transformation Requires
Sound Strategic Planning and Sustained Leadership. [48]GAO-05-520T
. Washington, D.C.: April 13, 2005.
Military Personnel: DOD Needs to Conduct a Data-Driven Analysis of
Active Military Personnel Levels Required to Implement the Defense
Strategy. [49]GAO-05-200 . Washington, D.C.: February 1, 2005.
21st Century Challenges: Reexamining the Base of the Federal
Government. [50]GAO-05-325SP . Washington, D.C.: February 1, 2005.
High-Risk Series: An Update. [51]GAO-05-207 . Washington, D.C.:
January 1, 2005.
Results-Oriented Cultures: Implementation Steps to Assist Mergers
and Organizational Transformations. [52]GAO-03-669 . Washington,
D.C.: July 2, 2003.
Quadrennial Defense Review: Future Reviews Can Benefit from Better
Analysis and Changes in Timing and Scope. [53]GAO-03-13 .
Washington, D.C.: November 4, 2002.
A Model of Strategic Human Capital Management. [54]GAO-02-373SP .
Washington, D.C.: March 15, 2002.
Quadrennial Defense Review: Opportunities to Improve the Next
Review. [55]GAO/NSIAD-98-155 . Washington, D.C.: June 25, 1998.
Quadrennial Defense Review: Some Personnel Cuts and Associated
Savings May Not Be Achieved. [56]GAO/NSIAD-98-100 . Washington,
D.C.: April 30, 1998.
Combating Terrorism: Threat and Risk Assessments Can Help
Prioritize and Target Program Investments. [57]GAO/NSIAD-98-74 .
Washington, D.C.: April 9, 1998.
Bottom-Up Review: Analysis of DOD War Game to Test Key
Assumptions. [58]GAO/NSIAD-96-170 . Washington, D.C.: June 21,
1996.
Bottom-Up Review: Analysis of Key DOD Assumptions.
[59]GAO/NSIAD-95-56 . Washington, D.C.: Jan. 31, 1995.
(350845)
[73]www.gao.gov/cgi-bin/getrpt?GAO-07-709
To view the full report, including the scope and methodology, click on the
link above. For more information, contact Janet A. St. Laurent at (202)
512-4402 or [email protected].
Highlights of [74]GAO-07-709 , a report to the Committee on Armed
Services, U.S. Senate
September 2007
QUADRENNIAL DEFENSE REVIEW
Future Reviews Could Benefit from Improved Department of Defense Analyses
and Changes to Legislative Requirements
The Department of Defense (DOD) is required by law to conduct a
comprehensive examination of the national defense strategy, force
structure, modernization plans, infrastructure, and budget every
4 years including an assessment of the force structure best suited to
implement the defense strategy at
low-to-moderate level of risk. The 2006 Quadrennial Defense Review (QDR),
completed in February 2006, represents the first comprehensive review that
DOD had undertaken since the military forces have been engaged in
operations in Iraq and Afghanistan.
GAO was asked to assess (1) the strengths and weaknesses of DOD's approach
and methodology for the 2006 QDR and (2) what changes, if any, in the QDR
legislation could improve the usefulness of the report, including any
changes that would better reflect 21st century security conditions. To
conduct its review, GAO analyzed DOD's methodology, QDR study guidance,
and results from key analyses and also obtained views of defense analysts
within and outside of DOD.
[75]What GAO Recommends
GAO recommends that for future QDRs, DOD develop methods to conduct a more
thorough analysis of force structure and risk. GAO is also providing
options for Congress to consider to revise QDR legislation. DOD generally
agreed with our recommendations.
DOD's approach and methodology for the 2006 QDR had several strengths, but
several weaknesses significantly limited the review's usefulness in
addressing force structure, personnel requirements, and risk associated
with executing the national defense strategy. Key strengths of the QDR
included sustained involvement of senior DOD officials, extensive
collaboration with interagency partners and allied countries, and a
database to track implementation of initiatives. However, GAO found
weaknesses in three key areas. First, DOD did not conduct a comprehensive,
integrated assessment of different options for organizing and sizing its
forces to provide needed capabilities. Without such an assessment, DOD is
not well positioned to balance capability needs and risks within future
budgets, given the nation's fiscal challenges. Second, DOD did not provide
a clear analytical basis for its conclusion that it had the appropriate
number of personnel to meet current and projected demands. During its
review, DOD did not consider changing personnel levels and instead focused
on altering the skill mix. However, a year after the QDR report was
issued, DOD announced plans to increase Army and Marine Corps personnel by
92,000. Without performing a comprehensive analysis of the number of
personnel it needs, DOD cannot provide an analytical basis that its
military and civilian personnel levels reflect the number of personnel
needed to execute the defense strategy. Third, the risk assessments
conducted by the Secretary of Defense and the Chairman of the Joint Chiefs
of Staff, which are required by the QDR legislation, did not fully apply
DOD's risk management framework because DOD had not developed assessment
tools to measure risk. Without a sound analytical approach to assessing
risk, DOD may not be able to demonstrate how it will manage risk within
current and expected resource levels. As a result, DOD is not in the best
position to demonstrate that it has identified the force structure best
suited to implement the defense strategy at low-to-moderate risk.
Through discussions with DOD officials and defense analysts, GAO has
identified several options for refining the QDR legislative language that
Congress could consider to improve the usefulness of future QDRs,
including changes to encourage DOD to focus on high priority strategic
issues and better reflect security conditions of the 21st century.
Congress could consider options to clarify its expectations regarding what
budget information DOD should include in the QDR and eliminate reporting
elements for issues that could be addressed in different reports. For
example, the requirement to assess revisions to the unified command plan
is also required and reported under other legislation. Further, some
reporting elements such as how resources would be shifted between two
conflicts could be eliminated in light of DOD's new planning approach that
focuses on capabilities to meet a range of threats rather than on the
allocation of forces for specific adversaries. GAO also presents an option
to have an advisory group work with DOD prior to and during the QDR to
provide DOD with alternative perspectives and analyses.
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( [76]www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
[77]www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [78]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [79][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [80][email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Susan Becker, Acting Manager, [81][email protected] (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149 Washington,
D.C. 20548
References
Visible links
26. http://www.gao.gov/cgi-bin/getrpt?GAO-05-325SP
27. http://www.gao.gov/cgi-bin/getrpt?GAO-07-600CG
28. http://www.gao.gov/cgi-bin/getrpt?GAO-03-13
29. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-155
30. http://www.gao.gov/cgi-bin/getrpt?GAO-06-13
31. http://www.gao.gov/cgi-bin/getrpt?GAO-05-520T
32. http://www.gao.gov/cgi-bin/getrpt?GAO-06-745
33. http://www.gao.gov/cgi-bin/getrpt?GAO-06-962
34. http://www.gao.gov/cgi-bin/getrpt?GAO-07-415
35. http://www.gao.gov/
36. mailto:[email protected]
37. http://www.gao.gov/cgi-bin/getrpt?GAO-07-415
38. http://www.gao.gov/cgi-bin/getrpt?GAO-07-388
39. http://www.gao.gov/cgi-bin/getrpt?GAO-07-644T
40. http://www.gao.gov/cgi-bin/getrpt?GAO-07-600CG
41. http://www.gao.gov/cgi-bin/getrpt?GAO-07-386T
42. http://www.gao.gov/cgi-bin/getrpt?GAO-07-397T
43. http://www.gao.gov/cgi-bin/getrpt?GAO-07-211
44. http://www.gao.gov/cgi-bin/getrpt?GAO-06-745
45. http://www.gao.gov/cgi-bin/getrpt?GAO-06-962
46. http://www.gao.gov/cgi-bin/getrpt?GAO-06-257T
47. http://www.gao.gov/cgi-bin/getrpt?GAO-06-13
48. http://www.gao.gov/cgi-bin/getrpt?GAO-05-520T
49. http://www.gao.gov/cgi-bin/getrpt?GAO-05-00
50. http://www.gao.gov/cgi-bin/getrpt?GAO-05-325SP
51. http://www.gao.gov/cgi-bin/getrpt?GAO-05-207
52. http://www.gao.gov/cgi-bin/getrpt?GAO-03-669
53. http://www.gao.gov/cgi-bin/getrpt?GAO-03-13
54. http://www.gao.gov/cgi-bin/getrpt?GAO-02-373SP
55. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-155
56. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-100
57. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-98-74
58. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-96-170
59. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-95-56
60. http://www.gao.gov/cgi-bin/getrpt?GAO-07-211
61. http://www.gao.gov/cgi-bin/getrpt?GAO-07-388
62. http://www.gao.gov/cgi-bin/getrpt?GAO-05-200
63. http://www.gao.gov/cgi-bin/getrpt?GAO-02-373SP
64. http://www.gao.gov/cgi-bin/getrpt?GAO-07-397T
65. http://www.gao.gov/cgi-bin/getrpt?GAO-06-13
66. http://www.gao.gov/cgi-bin/getrpt?GAO-05-325SP
67. http://www.gao.gov/cgi-bin/getrpt?GAO-05-207
68. http://www.gao.gov/cgi-bin/getrpt?GAO-06-13
69. http://www.gao.gov/cgi-bin/getrpt?GAO-05-325SP
70. http://www.gao.gov/cgi-bin/getrpt?GAO-06-257T
71. http://www.gao.gov/cgi-bin/getrpt?GAO-03-13
72. http://www.gao.gov/cgi-bin/getrpt?GAO-03-13
73. http://www.gao.gov/cgi-bin/getrpt?GAO-07-709
74. http://www.gao.gov/cgi-bin/getrpt?GAO-07-709
76. http://www.gao.gov/
77. http://www.gao.gov/
78. http://www.gao.gov/fraudnet/fraudnet.htm
79. mailto:[email protected]
80. mailto:[email protected]
81. mailto:[email protected]
*** End of document. ***