Defense Transportation: Study Limitations Raise Questions about  
the Adequacy and Completeness of the Mobility Capabilities Study 
and Report (20-SEP-06, GAO-06-938).				 
                                                                 
The Department of Defense (DOD) issued the Mobility Capabilities 
Study (MCS), which was intended to identify and quantify the	 
mobility capabilities needed to support U.S. strategic objectives
into the next decade. The MCS found that projected capabilities  
are adequate to achieve U.S. objectives with an acceptable level 
of risk--that is, current U.S. inventory of aircraft, ships,	 
prepositioned assets, and other capabilities are sufficient, in  
conjunction with host nation support, and assuming planned	 
investments take place. The Senate report accompanying the bill  
for the fiscal year 2005 Defense Authorization Act required GAO  
to report on the adequacy and completeness of the MCS. GAO	 
assessed the extent to which the MCS met generally accepted	 
research standards that this type of study would be expected to  
meet to be considered sound and complete.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-938 					        
    ACCNO:   A61198						        
  TITLE:     Defense Transportation: Study Limitations Raise Questions
about the Adequacy and Completeness of the Mobility Capabilities 
Study and Report						 
     DATE:   09/20/2006 
  SUBJECT:   Defense capabilities				 
	     Documentation					 
	     Evaluation methods 				 
	     Military forces					 
	     Reports management 				 
	     Standards						 
	     Strategic planning 				 
	     Transportation					 
	     C-17 Aircraft					 
	     C-5 Aircraft					 
	     DOD Mobility Capabilities Study			 
	     C-17A Aircraft					 
	     C-X Aircraft					 
	     Globemaster Aircraft				 
	     C-5A Aircraft					 
	     Galaxy Aircraft					 
	     C-5B Aircraft					 
	     C-5D Aircraft					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-938

     

     * Results in Brief
     * Background
     * Limitations in the MCS Study and Report Raise Questions abou
          * Aspects of Modeling and Data Were Inadequate
          * Exclusive Use of Warfighting Metrics in MCS Analyses Limited
          * Results Are Not Always Complete or Presented Clearly and Are
          * VV&A of Models and Data Was Not Complete
     * Conclusions
     * Recommendation for Executive Action
     * Agency Comments and Our Evaluation
     * Appendix I: Scope and Methodology
     * Appendix II: Comments from the Department of Defense
          * GAO's Responses to DOD's Technical Comments
     * Appendix III: GAO Contact and Staff Acknowledgments
          * GAO Contact
          * Acknowledgments
               * Order by Mail or Phone

Report to Congressional Committees

United States Government Accountability Office

GAO

September 2006

DEFENSE TRANSPORTATION

Study Limitations Raise Questions about the Adequacy and Completeness of
the Mobility Capabilities Study and Report

GAO-06-938

Contents

Letter 1

Results in Brief 2
Background 5
Limitations in the MCS Study and Report Raise Questions about Adequacy and
Completeness 7
Conclusions 13
Recommendation for Executive Action 13
Agency Comments and Our Evaluation 14
Appendix I Scope and Methodology 18
Appendix II Comments from the Department of Defense 22
Appendix III GAO Contact and Staff Acknowledgments 29

Table

Table 1: Generally Accepted Research Standards Relevant to MCS
Requirements 19

Figure

Figure 1: MCS Hypothetical Ongoing Contingencies during 7-year Baseline
Security Posture Time Frame 9

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

September 20, 2006

Congressional Committees

The National Security Strategy of the United States requires global
mobility through rapid, effective, and efficient projections of power at
home and abroad to deploy and sustain America's armed forces. To improve
its mobility capabilities, the Department of Defense (DOD) plans to spend
more than $50 billion from fiscal years 2006 through 2011 for aircraft,
ships, ground transportation, prepositioned assets, and other mobility
assets. DOD has conducted several studies to determine mobility
requirements and recently completed a study of its mobility capabilities
and issued a report in December 2005. The intent of the Mobility
Capabilities Study (MCS) was to identify and quantify the mobility
capabilities needed to support U.S. strategic objectives into the next
decade. The MCS determined that the projected mobility capabilities are
adequate to achieve U.S. objectives with an acceptable level of risk
during the period from fiscal years 2007 through 2013; that is, the
current U.S. inventory of aircraft, ships, prepositioned assets, and other
capabilities are sufficient, in conjunction with host nation support. The
MCS emphasized that continued investment in the mobility system, in line
with current departmental priorities and planned spending, is required to
maintain these capabilities in the future. This includes, for example,
fully funding Army prepositioned assets as planned and completing a
planned reengineering of the C-5 aircraft. The MCS report also made
recommendations to conduct further studies, develop plans and strategies,
and improve data collection and mobility models. In fact, DOD officials
told us that a Mobility Capabilities Study--2006 is underway.

In the Senate report accompanying the bill for the fiscal year 2005
Defense Authorization Act, you asked us to monitor the process used to
conduct the MCS and report on the adequacy and completeness of the study.1
Specifically, our objective was to determine whether the MCS was adequate
and complete. On March 1, 2006, we briefed your staff on our preliminary
observations. This report expands on that briefing and makes
recommendations to the Secretary of Defense.

1 S. Rep. 108-260, at 126 (2004).

To conduct our review of the MCS, we analyzed the final MCS report, the
MCS Terms of Reference and MCS Study Plan, as well as other DOD policies
and guidance concerning how DOD would conduct the MCS and the databases
and models used in the study. We identified generally accepted research
standards that define a sound and complete quality study that were
relevant to the MCS, and assessed the extent to which the MCS report met
these standards. We interviewed study officials, study participants, and
subject matter experts from several DOD entities, including the combatant
commands and the military services. As we monitored the development of the
MCS, we requested that DOD provide documentation supporting and verifying
key analytical and decision-making processes.2 DOD officials could not
produce this documentation during the development of the MCS or following
issuance of the report. Consequently, we were unable to fully determine
whether the analytical and decision-making processes that we believe are
significant to the credibility of the study supported the MCS effort and
its conclusions. Our scope and methodology are discussed in more detail in
appendix I. We conducted our work from July 2004 through July 2006 in
accordance with generally accepted government auditing standards.

                                Results in Brief

DOD used an innovative approach in conducting the study and acknowledged
some methodological limitations in its report, as any sound study should.
However, it did not fully disclose how these limitations could affect the
MCS conclusions and recommendations. Therefore, it is not transparent how
the analyses done for the study support DOD's conclusions. As measured
against relevant generally accepted research standards, we identified
limitations in the MCS study and report that raise questions about their
adequacy and completeness. Among our findings:

2 Examples of the documentation we requested to support and verify key
analytical and decision-making processes used by DOD to conduct the MCS
included (1) the accreditation report and supporting documentation or
evidence of the verification, validation, and accreditation process for
the models and data used in the MCS; (2) copies of MCS working group
meeting minutes that verify and validate the analytical processes the
various MCS study teams and study participants used to vet and agree upon
data, scenarios, assumptions, models, and associated risk; and (3) copies
of MCS General Officer Steering Committee and Executive Committee meeting
minutes that verify and validate the analytical and decision-making
processes the DOD senior leadership used to vet and agree upon the key
data, scenarios, assumptions, models, and associated risk used to conduct
the MCS, as well as agreement with the study results.

           o  Aspects of modeling and data were inadequate in some areas
           because data were lacking and some of the models used could not
           simulate all relevant aspects of the missions. The report did not
           explain how these limitations could affect the study results or
           what the effect on the projected mobility capabilities might be.
           Relevant research standards require that models used are adequate
           for the intended purpose, represent a complete range of
           conditions, and that data used are properly generated and
           complete. For example, the MCS modeled hypothetical homeland
           defense missions rather than homeland defense demands derived from
           a well defined and approved concept of operations for homeland
           defense, because the specific details of the missions were still
           being determined, and DOD acknowledged that the data used may be
           incomplete. The MCS also was unable to model the flexible
           deterrent options/deployment order process to move units and
           equipment into theater due to lack of data, but the study assumed
           a robust use of this process, which in one scenario accounted for
           approximately 60 percent of the airlift prior to beginning combat
           operations.3 In addition, the MCS report contains more than 80
           references to the need for improved modeling, and 12 of these
           references call for additional data or other refinements.
           Additionally, the MCS modeled the year 2012 to determine the
           transportation capabilities needed for the years 2007 through
           2013. The year 2012 did not place as much demand for mobility
           assets in support of smaller military operations, such as
           peacekeeping, as other years. However, DOD officials considered
           2012-the year modeled-as "most likely" to occur and stated that
           statistically it was not different from other years in the 2007 to
           2013 period even though the number of smaller military operations
           is the least of any of the years reviewed.
           o  While the MCS concluded that combined U.S. and host nation
           transportation assets were adequate to meet U.S. objectives with
           acceptable risk, the report, in describing the use of warfighting
           metrics in its analyses, does not provide a clear understanding of
           the direct relationship of warfighting objectives to
           transportation capabilities. Acknowledging this point, the report
           stated that further analysis is required to understand the
           operational impact of increased or decreased strategic lift on
           achieving warfighting objectives. Relevant generally accepted
           research standards require that conclusions be supported by
           analyses. The use of warfighting metrics is a measure to determine
           whether combat tasks, such as achieving air superiority, are
           achieved. However, they do not measure whether appropriate
           personnel, supplies, and equipment arrived in accordance with
           timelines. As a result, we could not determine how the study
           concluded that planned transportation assets were adequate because
           the study did not contain a transparent analysis to support its
           conclusion or a clear roadmap in the report to help decision
           makers understand what that conclusion meant in terms of type and
           number of mobility assets needed. Previous DOD mobility studies
           primarily used mobility metrics, which measured success in terms
           of tons of equipment and personnel moved per day to accomplish
           military objectives. The use of both warfighting and mobility
           metrics to measure success would allow decision makers to know
           whether combat tasks were achieved and how much strategic
           transportation is needed to accomplish those tasks.
           o  In some cases, the MCS results were incomplete, unclear, or
           contingent on further study, making it difficult to identify
           findings and evaluate evidence. Relevant research standards
           require results to be presented in a complete, accurate, and
           relevant manner. For example, the report contains several
           recommendations for further studies and assessments, five of which
           are under way. However, DOD has no plans to report the effect of
           these studies on the MCS results after the studies are complete.
           In addition, the report contains qualified information that is not
           presented clearly, such as varying assessments of intratheater
           assets in three different places in the report. The lack of
           clarity and conciseness of the reported results can limit the
           study's usefulness to decision makers and stakeholders.
           o  Verification, validation, and accreditation (VV&A) of models
           and data used to conduct the study was not complete because it was
           not done in accordance with DOD policy or relevant research
           standards. Moreover, relevant research standards state that a
           study report should include a VV&A accreditation report that is
           signed by the study director and addresses the models and data
           certification. DOD officials acknowledged that they did not comply
           with DOD VV&A policy when using legacy models in the MCS because
           they contended that long-term use of models and data constitutes
           an equivalent VV&A process. Other than a description of the
           process contained in the MCS report, DOD officials could provide
           no additional documentation to verify and validate this equivalent
           process to provide the assurance that models and data used in the
           MCS reduced the risk inherent in modeling and simulation and added
           to the credibility of the results. Moreover, officials could not
           provide documentation to support key analytical and
           decision-making processes used by senior DOD leadership, thus
           undermining the credibility of the reported study results.

           These limitations to the study's methodology raise questions
           concerning the accuracy of the study's finding that projected
           capabilities are adequate to achieve U.S. objectives with an
           acceptable level of risk. Until DOD conducts an adequate and
           complete future MCS and clearly discloses all limitations and
           their effects on the study results, decision makers may be unable
           to clearly understand the operational implications of the study
           results and make fully informed programmatic investment decisions
           concerning mobility capabilities. We are recommending that the
           Secretary of Defense, when conducting future mobility capabilities
           studies, beginning with any study currently under way, develop and
           use models and data for critical missions and processes that are
           verified, validated, and accredited as required; include in study
           reports an explanation of how stated limitations might impact the
           study results and, at a minimum, describe how recommended future
           studies might be conducted to enhance the results of the original
           study; and incorporate both mobility and warfighting metrics in
           determining capabilities.

           In commenting on a draft of this report, DOD concurred with the
           first and third recommendations and claimed that they did not
           understand the second. We have clarified that recommendation to
           the Secretary of Defense to include in study reports an
           explanation of how stated limitations might impact the study
           results and, at a minimum, describe how recommended future studies
           might be conducted to enhance the results of the original study.
           In its comments, DOD also stated that the report contained
           misleading information and factual errors. We disagree with DOD's
           assertion. We did modify our report to respond to a DOD technical
           comment related to homeland defense missions. DOD's comments and
           our evaluation of them are discussed in the agency comments
           section of this report.

           Background
			  
			  The MCS was the first assessment of DOD's mobility system since
           2000. The study was designed to identify changes in DOD's
           transportation force structure due to changes in threats and
           national security and military strategies. The MCS is the fourth
           in a series of major mobility studies that DOD has conducted since
           the end of the Cold War. The first study, the Mobility
           Requirements Study, conducted in 1992, was undertaken because of
           concern about the DOD's strategic mobility capabilities in the
           wake of Operation Desert Shield and Operation Desert Storm. That
           study established mobility requirements for the post-Cold War era;
           defined baseline requirements for intertheater, or strategic,
           mobility; and proposed a long-range investment plan to meet these
           requirements. The Mobility Requirements Study Bottom-Up Review
           Update, conducted in 1994, reaffirmed the need for increases in
           key mobility components and validated the prior study's
           recommendation for the procurement of additional ships for afloat
           prepositioning and for surge deployments of forces based in the
           continental United States. The Mobility Requirements Study-2005,
           issued in 2001, projected future mobility requirements based on
           two nearly simultaneous major regional contingencies. It included
           a broader range of factors, including host nation support and
           enemy use of weapons of mass destruction, than the previous
           studies.

           The current MCS, which began in May 2004, reassessed DOD's
           mobility capabilities against the backdrop of a revised National
           Military Strategy that included the ongoing war against violent
           extremism, an evolving global defense posture, a new force-sizing
           construct, revised campaign scenarios, and ongoing departmentwide
           transformation efforts. The study results were intended to support
           decisions on future strategic airlift, aerial refueling aircraft,
           and sealift procurements needed to meet varying military
           requirements. The study used an innovative "capabilities-based"
           approach, measuring existing and currently projected mobility
           capabilities against warfighting demands that could be expected in
           fiscal year 2012 while also considering mobility demands during
           the 7-year period from fiscal year 2007 through fiscal year 2013.
           According to DOD officials, the Secretary of Defense believed this
           approach would give him greater flexibility in deciding which
           capabilities to fund in a constrained budget environment. In
           considering each aspect of the National Military Strategy,4 the
           MCS modeled warfighting scenarios in the year 2012 using different
           premises with varying assumptions to develop and evaluate mobility
           capability mix alternatives. The models were used to evaluate
           transportation alternatives, including variations in alternative
           transportation modes (air, land, sea) and sources (military,
           civilian, foreign), as well as factors that affect transportation
           mode and source decisions. The scope of the MCS described the
           study as an assessment of the full range of transportation needs
           required to support (1) combat operations; (2) smaller military
           efforts, such as peacekeeping or overseas evacuation of American
           citizens; (3) homeland defense/civil support, such as disaster
           relief and antiterrorism response; and (4) other strategic
           missions, such as strategic nuclear and global strike missions.
           The study was coauthored by the Office of the Secretary of
           Defense, Office of the Director, Program Analysis and Evaluation
           (PA&E), and the Chairman, Joint Chiefs of Staff, Office of the
           Director of Logistics. Other DOD components involved in the study
           included the U.S. Transportation Command and its subordinate
           commands, the Office of the Under Secretary of Defense
           (Acquisition, Technology and Logistics), the combatant commanders,
           the military services, and others. The final report was signed on
           December 19, 2005, by the Deputy Secretary of Defense.

           Limitations in the MCS Study and Report Raise Questions about
			  Adequacy and Completeness
			  
			  As measured against relevant generally accepted research
           standards, limitations in the MCS study and report raise questions
           about their adequacy and completeness. For example, aspects of
           modeling and data were inadequate in some areas because data were
           lacking and some of the models used could not simulate all
           relevant aspects of the missions. Furthermore, the exclusive use
           of warfighting metrics in the MCS analyses limited the usefulness
           of the report. Moreover, in some cases the MCS results were
           incomplete, unclear, or contingent on further study, making it
           difficult to identify findings and evaluate evidence. Finally,
           verification, validation, and accreditation of models and data
           used to conduct the study were incomplete because they were not
           done in accordance with DOD policy or relevant research standards,
           and supporting documentation for key processes could not be
           provided.

           Aspects of Modeling and Data Were Inadequate
			  
			  Aspects of modeling and data were inadequate in some areas because
           data were lacking and some of the models used could not simulate
           all relevant aspects of the missions. Relevant research standards
           require that models used are adequate for the intended purpose,
           represent a complete range of conditions, and that data used are
           properly generated and complete. As DOD acknowledged in the MCS
           report as a study limitation, some modeling tools were not
           available to analyze key missions. The MCS cited deficiencies in
           several existing mobility models and the need for follow-on MCS
           analyses. The MCS report also acknowledged that the identified
           deficiencies in data hindered analysis of future requirements and
           transportation system performance. However, the report did not
           explain how these limitations could affect the study results or
           what the effect on the projected mobility capabilities might be.

           For example, the MCS modeled hypothetical homeland defense
           missions rather than homeland defense demands derived from a well
           defined and approved concept of operations for homeland defense,
           because the specific details of the missions were still being
           determined, and DOD acknowledged that the data used are
           incomplete. The MCS report recommended further analysis of
           mobility capabilities after homeland defense needs are refined.
           However, the report did not identify the potential effect that
           using these hypothetical scenarios might have on the MCS results.
           The MCS also was unable to model the flexible deterrent
           options/deployment order process to move units and equipment into
           theater because of lack of data on how deployment orders have been
           issued in the past for major combat operations. However, the MCS
           assumed a robust use of the flexible deterrent option/deployment
           order process, which in one scenario accounted for approximately
           60 percent of the early airlift movement prior to the beginning of
           combat operations. Instead, the MCS modeled the flow of forces and
           equipment contained in the time-phased force deployment data
           process.5 Based on the scenarios provided for the MCS analyses, we
           could not determine how the deployment order process would affect
           the mobility assets required for major combat operations. The MCS
           report noted that additional analysis is required to determine the
           implications of the deployment order process and to provide
           sufficient information for decision makers concerning the amount
           of future mobility assets actually needed.

           In addition to these modeling and data issues, the MCS report
           contains more than 80 references to the need for improved
           modeling, and 12 of these references call for additional data or
           other refinements, such as follow-on analyses, further
           assessments, future evaluations, additional study, and
           investigation of alternatives in a wide range of areas, such as
           antiterrorism response, infrastructure availability, intratheater
           airlift, strategic sealift, air refueling, and availability of
           civil reserve aircraft. Some of these studies are currently
           underway, as discussed later in this report.

           Moreover, our analysis of the MCS report showed that the year
           modeled (2012) to determine the DOD transportation capabilities
           needed for the years 2007 through 2013 did not place as much
           demand for mobility assets in support of smaller military
           operations, such as peacekeeping, as other years. To establish
           transportation demands for mission requirements, DOD developed and
           used a baseline security posture6 that covered a 7-year period.
           This baseline was developed, in part, using a historical analysis
           of DOD's movement of personnel, equipment, supplies, and other
           items. According to DOD officials, Office of the Under Secretary
           of Defense for Policy, which developed the baseline security
           posture, selected the year modeled in the MCS because it was
           deemed the "most likely" to occur in terms of transportation
           demands and because it was not statistically different from other
           years in the 7-year period. However, our analysis showed that 2012
           involved the least demand for transportation assets in support of
           smaller military efforts than any year in the 7-year period and
           did not fully stress the defense transportation system. Figure 1
           depicts the number of hypothetical ongoing contingencies for each
           year in the baseline as shown in the MCS.

           Figure 1: MCS Hypothetical Ongoing Contingencies during 7-year
           Baseline Security Posture Time Frame

           Note: A particular contingency may be ongoing in more than 1 year.
           Each contingency has unique cargo and passenger requirements. For
           example, a contingency that may be ongoing over a 2- or 3-year
           time frame may require more or less mobility capability than a
           6-month contingency.

           Although not transparent in the study, DOD officials said
           scenarios in the year modeled were not intended to fully stress
           the defense transportation system. DOD officials provided no
           further explanation for the year selected to develop the DOD
           transportation capabilities other than it was directed by Office
           of the Under Secretary of Defense for Policy and agreed to by the
           study leadership. We believe that selection of a different year
           that placed an increased demand on transportation assets for
           smaller military efforts may have revealed gaps in mobility
           requirements. Therefore, we found that the selection of 2012 as
           the modeling year was a limitation in the MCS with respect to
           smaller military efforts.

           Because of these modeling and data limitations, the MCS may have
           incorrectly estimated the future mobility requirements needed to
           support homeland defense missions, major combat operations, and
           smaller contingencies. Until DOD improves aspects of the modeling
           and data used to conduct the MCS-to include defining its homeland
           defense mission needs, developing models for the deployment order
           process, and explaining how identified modeling and data
           limitations could affect the study results-decision makers may not
           have adequate and complete information about DOD's mobility
           capabilities.

           Exclusive Use of Warfighting Metrics in MCS Analyses Limited
			  Usefulness of Report
			  
			  While the MCS concluded that combined U.S. and host nation
           transportation assets were adequate to meet U.S. objectives with
           acceptable risk, the report, in describing the use of warfighting
           metrics in its analyses, does not provide a clear understanding of
           the direct relationship of warfighting objectives to
           transportation capabilities. The report acknowledged that further
           analysis is required to understand the operational impact of
           increased or decreased strategic lift on achieving warfighting
           objectives. Relevant generally accepted research standards require
           that conclusions be supported by analyses. The use of warfighting
           metrics is a measure to determine whether combat tasks, such as
           establishing air superiority, are achieved. However, they do not
           measure success in terms of whether appropriate personnel,
           supplies, and equipment arrived in accordance with timelines. As a
           result, we could not determine how the study concluded that
           planned transportation assets were adequate because the study did
           not contain a transparent analysis to support its conclusion. In
           our opinion, it is important for decision makers to have an
           understanding of both combat tasks that must be achieved and the
           amount of transportation assets needed to achieve those tasks with
           some level of success. This understanding would allow creation of
           a clear roadmap for investment decisions. However, we could not
           determine how the study calculated the specific numbers of
           transportation assets needed or whether there are specific gaps,
           overlaps, or excesses in transportation assets, a key objective of
           the study. Previous DOD mobility studies, including the Mobility
           Requirements Study-2005, primarily used mobility metrics, which
           measured success in terms of tons of equipment and personnel moved
           per day to accomplish military objectives. Million-ton-miles per
           day is a commonly accepted measure of airlift performance and
           reflects how much cargo can be delivered over a given distance in
           a given period of time based on the capability of each type of
           aircraft. A similar mobility metric-short tons-is used to measure
           ship capability. However, these studies did not fully integrate
           combat tasks as a metric. The use of both warfighting and mobility
           metrics to measure success would allow decision makers to know
           whether there is sufficient capability to achieve warfighting
           objectives, as well as to understand the number, type, and mix of
           mobility assets that are actually needed.

           Results Are Not Always Complete or Presented Clearly and Are
			  Qualified or Contingent on Further Study or Analysis
			  
			  In some cases, the MCS results were incomplete, unclear, or
           contingent on further study, making it difficult to identify
           findings and evaluate evidence. Relevant research standards
           require results to be presented in a complete, accurate, and
           relevant manner; conclusions to be sound and complete; and
           recommendations to be supported by analyses. Our analysis of the
           MCS report found that it contains several recommendations for
           further studies and assessments, five of which are under way. The
           five studies address intratheater lift capabilities; sealift
           petroleum, oil, and lubricants; logistics contingency operations
           capabilities; aerial refueling; and integrated global presence and
           basing. However, the report does not explain the potential effect
           of these ongoing studies on the MCS results after the studies are
           complete, nor does DOD have plans to report the effect of these
           studies on the MCS results.

           In addition, the report contains qualified information that is not
           presented clearly in the report, such as varying assessments of
           intratheater assets in three different places. For example, the
           report states in the assessment section of the executive summary
           that projected transportation assets are sufficient to address
           intratheater demands in the fiscal years 2007 through 2013 time
           frame. However, in the recommendations section of the executive
           summary, the report states that DOD should take action to
           determine the proper mix of intratheater assets needed to meet
           requirements. Then, in the part of the report that discusses
           intratheater airlift, the report states that a detailed analysis
           of intratheater airlift needs would require improved modeling
           tools to accurately capture interactions among land, sealift, and
           airlift capabilities and that data sets must be developed that
           accurately describe the requirement in light of emerging concepts.

           VV&A of Models and Data Was Not Complete
			  
			  VV&A of models and data used to conduct the study was not complete
           because it was not done in accordance with DOD policy or relevant
           research standards. DOD policy issued by the Under Secretary of
           Defense for Acquisition, Technology and Logistics requires that
           DOD models and data go through a VV&A process. Moreover, relevant
           research standards state that a study report should include a VV&A
           accreditation report that is signed by the study director and
           addresses the models and data certification. DOD officials
           acknowledged that they did not comply with the VV&A policy when
           using legacy models in the MCS because they believed such an
           approach was not warranted for legacy models that have been used
           for many years and have proved reliable. Moreover, these officials
           believe that such long-term use constitutes a VV&A process
           equivalent to that required in the policy. However, the DOD policy
           does not specify that the actual use of a model constitutes an
           equivalent VV&A process. VV&A of models and data reduces the risk
           inherent in the use of models and simulations by improving the
           credibility of modeling and simulation results. We previously
           reported our concerns that DOD did not follow its policy in
           executing the MCS and had little documentation to support the VV&A
           process used.7 We found that the final MCS report contained a
           description of the equivalent VV&A process, but DOD officials
           could provide no further documentation to verify and validate this
           equivalent process other than the description included in the
           report. We also found no documentation in the study report to
           support DOD claims that the models have proven reliable.

           Furthermore, DOD officials were unable to provide documentation to
           support and verify key analytical and decision-making processes
           used by senior DOD leadership throughout the study. Relevant
           research standards support documenting the study's analytical and
           decision-making processes to ensure the accuracy, completeness,
           and credibility of study results. DOD officials told us that the
           study's key analytical and decision-making processes were
           validated and approved by study participants during working group
           meetings and by senior leadership during General Officer Steering
           Committee meetings and Executive Committee meetings. PA&E
           officials could not produce documentation of these meetings
           because they said documentation did not exist. Nor could they
           produce other documents we requested during the development of the
           MCS or following issuance of the report. Consequently, we were
           unable to determine the adequacy and completeness of the
           analytical and decision-making processes that supported the MCS
           effort and that we believe are significant to the credibility of
           the study and its conclusions.

           Conclusions
			  
			  The methodological limitations in the MCS that we identified-some
           of which were acknowledged by DOD in the MCS report-raise
           questions about the adequacy and completeness of the study and its
           report. Until DOD improves aspects of the modeling and data used
           to conduct the MCS-such as defining its homeland defense mission
           needs and developing models for the deployment order
           process-decision makers may not have adequate and complete
           information about DOD's mobility capabilities to enable them to
           make fully informed investment decisions. Furthermore, in the
           absence of an explanation of how identified modeling and data
           limitations could affect the study results or how such limitations
           could affect projected mobility capability requirements, the
           accuracy of the study's finding that projected capabilities are
           adequate to achieve U.S. objectives with an acceptable level of
           risk during the period from fiscal years 2007 through 2013 is
           unclear. Moreover, without a transparent comparison between
           existing mobility assets and projected needed assets, decision
           makers will be unable to use study results to identify and
           quantify the specific types and mix of mobility assets needed to
           address mobility gaps, overlaps, and excesses. Until DOD conducts
           an adequate and complete future MCS and clearly discloses all
           limitations and their effects on the study results, decision
           makers likely will not have full information concerning DOD's
           mobility capabilities. As a result, we suggest that Congress and
           other decision makers exercise caution in using the MCS to make
           programmatic investment decisions.

           Recommendation for Executive Action
			  
			  To provide decision makers with adequate and complete information
           concerning mobility capabilities so they are able to clearly
           understand the operational implications of the study and make
           fully informed programmatic investment decisions, and to improve
           the usefulness of future mobility capabilities studies, we
           recommend that the Secretary of Defense take the following three
           actions, when conducting future mobility capabilities studies
           beginning with any study currently underway:

           o  develop models and data for all critical missions, such as
           homeland defense, and processes, such as the flexible deterrent
           options/deployment order process;
           o  include in study reports an explanation of how stated
           limitations might impact the study results and, at a minimum,
           describe how recommended future studies might be conducted to
           enhance the results of the original study; and
           o  incorporate both mobility and warfighting metrics in
           determining capabilities.

           Agency Comments and Our Evaluation
			  
			  In commenting on a draft of this report, DOD concurred with our
           first and third recommendations. DOD stated it did not understand
           our second recommendation that the Secretary of Defense, when
           conducting future mobility studies, beginning with any study
           currently underway, include in study reports an explanation of how
           ongoing and follow-on studies and modeling and data limitations
           that are referenced in the report could affect the reported
           results. DOD also noted that it plans to continue its ongoing
           efforts to enhance the models and data collection processes used
           to assess mobility capabilities across the full range of strategic
           missions and supports the notion that continual improvements are
           needed. As we noted throughout our report, the MCS report contains
           numerous and repeated references to the need for improved modeling
           and additional data or other refinements, such as follow-on
           analyses, further assessments, future evaluations, additional
           study, and investigation of alternatives in a wide range of areas.
           DOD further commented that while a completed study can recommend
           that follow-on studies be conducted, it cannot explain how future
           studies might affect the results of the current study. We
           acknowledge that DOD cannot quantitatively predict the outcome of
           an ongoing or future study. However, we believe DOD should be able
           to explain what ongoing follow-on studies or evaluations seek to
           determine, what changes are being made to the data inputs and
           modeling tools that are being used to conduct the studies, and how
           DOD expects the results may differ from current study results.
           While the explanation may be hypothetical, as are many operations
           research study hypotheses, it can provide decision makers with a
           better understanding of the current study's limitations and
           results and how an ongoing or future study's results may differ.
           Therefore, we refined our recommendation to recommend that the
           Secretary of Defense, when conducting future mobility studies,
           beginning with any study currently under way, include in study
           reports an explanation of how stated limitations might impact the
           study results and, at a minimum, describe how recommended future
           studies might be conducted to enhance the results of the original
           study. For example, if modeling and data are limitations to a
           study, the report should discuss the ways in which the results
           might change with better models and data.

           DOD also commented that our report contained misleading
           information and factual errors and that it stands by the adequacy
           and completeness of the MCS. The department provided examples in
           its technical comments where it believed our report contained
           misleading information and factual errors. We disagree with the
           department's comments regarding the facts in our report and have
           addressed each of the department's comments in appendix II.

           Lastly, DOD stated that the MCS and its conclusions are well
           accepted by the civilian and military leadership of the
           department, and pointed out that in March 2006 testimony before
           the House Armed Services Committee, the Commander, U.S.
           Transportation Command, stated that the planned strategic airlift
           fleet determined by the MCS is "about the right capacity". However
           we note that in the same hearing, the Commander also stated that
           he thought DOD needed "somewhere in the neighborhood of" 20 C-17
           cargo aircraft beyond what is planned. We also note that in the
           Commander's April 2006 testimony before the Senate Armed Services
           Committee, Subcommittee on Sealift, he stated that, in an internal
           Focused Mobility Analysis to study strategic mobility from a
           Transportation Command perspective, the MCS will be the baseline,
           "but we will explore how changes in key assumptions may impact the
           analytical outcome."

           We are sending copies of this report to the Secretary of Defense;
           the Director of PA&E and the Office of the Chairman, Joint Chiefs
           of Staff. We will also make copies available to others upon
           request. In addition, the report is available at no charge on the
           GAO Web site at http://www.gao.gov . If you or your staff have any
           questions regarding the briefing or this report, please contact me
           at (202) 512-8365 or [email protected] . Contact points for our
           Offices of Congressional Relations

           and Public Affairs may be found on the last page of this report.
           GAO staff who made key contributions to this report are listed in
           appendix III.

           William M. Solis Director, Defense Capabilities and Management

           List of Congressional Committees

           The Honorable John Warner Chairman The Honorable Carl Levin
           Ranking Minority Member Committee on Armed Services United States
           Senate

           The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye
           Ranking Minority Member Subcommittee on Defense Committee on
           Appropriations United States Senate

           The Honorable Duncan L. Hunter Chairman The Honorable Ike Skelton
           Ranking Minority Member Committee on Armed Services House of
           Representatives

           The Honorable C.W. Bill Young Chairman The Honorable John P.
           Murtha Ranking Minority Member Subcommittee on Defense Committee
           on Appropriations House of Representatives

           Appendix I: Scope and Methodology
			  
			  To conduct our review of the Mobility Capabilities Study (MCS), we
           reviewed and analyzed the final MCS report; the MCS Terms of
           Reference; the MCS Study Plan; applicable Department of Defense
           (DOD) strategic planning guidance; as well as other DOD guidance,
           directives, instructions, and memos that describe how DOD would
           conduct its MCS. We also reviewed the National Security Strategy
           of the United States of America and the National Military Strategy
           of the United States of America; DOD guidance concerning data
           collection, development, and management in support of strategic
           analysis; DOD modeling and simulation instruction; Defense
           Modeling and Simulation Office guidance; descriptions of models
           used to conduct the study; and the databases used in the models.
           We interviewed study officials from the Office of the Secretary of
           Defense, Program Analysis and Evaluation (PA&E), and the office of
           the Chairman, Joint Chiefs of Staff, Logistics, as well as study
           participants and subject matter experts from the U.S.
           Transportation Command, Air Mobility Command, Surface Deployment
           and Distribution Command, the combatant commands, and the military
           services concerning the extent of their input to the study. We
           also interviewed officials from the Office of the Secretary of
           Defense, Acquisition, Technology and Logistics, and the Modeling
           and Simulation Technical Director at the Defense Modeling and
           Simulation Office.

           Additionally, we reviewed research literature and DOD guidance and
           identified frequently occurring, generally accepted research
           standards that are relevant for defense studies such as the MCS
           that define a quality or sound and complete study. The following
           were our sources for these standards:

           o  GAO, Government Auditing Standards: 2003 Revision, GAO-03-673G
           (Washington, D.C.: June 2003);
           o  GAO, Designing Evaluations, GAO/PEMD-10.1.4 (Washington, D.C.:
           March 1991);
           o  GAO, Dimensions of Quality, GAO/QTM-94-1 (Washington, D.C.:
           February 2004);
           o  RAND Corporation, RAND Standards for High-Quality Research and
           Analysis (Santa Monica, Calif.: June 2004);
           o  Air Force Office of Aerospace Studies, Analysts Handbook: On
           Understanding the Nature of Analysis (January 2000);
           o  Air Force, Office of Aerospace Studies, Air Force Analysis
           Handbook, A Guide for Performing Analysis Studies: For Analysis of
           Alternatives or Functional Solution Analysis (July 2004);
           o  Department of Defense, DOD Modeling and Simulation (M&S)
           Verification, Validation, Accreditation (VV&A), Instruction
           5000.61 (Washington, D.C.: May 2003);
           o  Department of Defense, Data Collection, Development, and
           Management in Support of Strategic Analysis, Directive 8260.1
           (Washington, D.C.: December 2, 2003); and
           o  Department of Defense, Implementation of Data Collection,
           Development, and Management for Strategic Analyses, Instruction
           8260.2 (Washington, D.C.: January 21, 2003).

           During the process of identifying generally accepted research
           standards we noted that not all studies are conducted the same
           way. For example, while all studies use data, not all use baseline
           data. Likewise, all studies require analyses, but not all use
           models or simulation to conduct analyses. We tailored the research
           standards we identified as relevant to the MCS, as shown in table
           1.

3 Deployment orders are issued to deploy specific capabilities as
commitment decisions are made, rather than a deploying unit's full set of
equipment or capabilities. Flexible Deterrent Options (FDOs) provide
escalation options during the initial stages of a conflict. FDOs are
employed under certain conditions to deter adversarial actions contrary to
U.S. interests.

                                   Background

4 The 2004 National Military Strategy of the United States calls for a
force sized to defend the homeland, proactively dissuade adversaries in
and from four global regions, and conduct two overlapping "swift defeat"
campaigns. Even when committed to a limited number of lesser
contingencies, the force must be able to "win decisively" in one of the
two campaigns. This "1-4-2-1" force-sizing construct places a premium on
increasingly innovative and efficient methods to achieve objectives.

Limitations in the MCS Study and Report Raise Questions about Adequacy and
                                  Completeness

Aspects of Modeling and Data Were Inadequate

5 In joint military planning, time-phased force deployment data are
defined as a computer database that contains detailed personnel and cargo
planning data; it usually includes priority and sequencing of deploying
forces.

6 The baseline security posture projects the position from which combatant
commanders will perform future missions, including how they will address
the global war on terrorism, ongoing operations, and other day-to-day
activities to which U.S. forces remain committed and from which they are
not likely to disengage entirely.

Exclusive Use of Warfighting Metrics in MCS Analyses Limited Usefulness of
Report

Results Are Not Always Complete or Presented Clearly and Are Qualified or
Contingent on Further Study or Analysis

VV&A of Models and Data Was Not Complete

7 See GAO, Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies,
GAO-05-659R (Washington, D.C.: Sept. 14, 2005), for a more detailed
discussion.

                                  Conclusions

                      Recommendation for Executive Action

                       Agency Comments and Our Evaluation

Appendix I: Scope and Methodology Appendix I: Scope and Methodology

Table 1: Generally Accepted Research Standards Relevant to MCS
Requirements

Design: The Study is well designed
I.                    Study plan, scope, and objectives follow Defense
                         Planning Guidance                  
I.a                   (Do the study scope and objectives fully address the
                         charter presented in the 2004 Defense Planning
                         Guidance?)                         
I.a.1                 Does the study plan address specified guidance?
I.b                   Was the study plan followed?       
I.c                   Were deviations from the study plan explained and
                         documented?                        
I.d                   Was the study plan updated over the course of the
                         study and the updates explicitly identified in the
                         study and updated study plan?      
II                    Assumptions and constraints are reasonable and
                         consistent                         
II.a                  Are assumptions and constraints explicitly
                         identified?                        
II.a.1                (Are the study assumptions necessary and
                         reasonable?)                       
II.b                  Do the study assumptions support a sound analysis?
II.c                  Are the assumptions used in analyses common
                         throughout the study and models?   
II.d                  Do the assumptions contribute to an objective and
                         balanced research effort?          
III                   Scenarios and threats are reasonable
III.a.                Are scenarios traceable back to formal guidance?
III.b                 Were the threat scenarios validated and Defense
                         Intelligence Agency approved and documented?
III.c                 Do scenarios represent a reasonably complete range
                         of conditions?                     
III.d                 (Were the threats varied to allow for the conduct of
                         sensitivity analysis?)             
Execution: The study is well executed
IV                    Methodology is successfully executed
IV.a                  Was the study methodology executed consistent with
                         the (MCS) study plan and schedule? 
IV.b                  (Does the methodology support accomplishing the
                         objectives presented in the study plan?)
IV.c                  Were the models used to support the analyses
                         adequate for their intended purpose?
IV.d                  Were the model input data properly generated to
                         support the methodology?           
V                     (Analytical ) Baseline data and other data used to
                         support study and analyses validated, verified, and
                         approved                           
V.a                   Is the (analytical) baseline fully and completely
                         identified and used consistently throughout the
                         study for the various analyses?    
V.b                   Were data limitations identified (and the impact of
                         the limitations fully explained?)  
V.c                   Were the (baseline security posture) data verified
                         and validated?                     
V.d                   Was the data verification and validation process
                         documented?                        
VI                    Models, simulations, and verification, validation,
                         and accreditation are reasonable   
VI.a                  Was a VV&A accreditation report that addresses the
                         models and data certification signed by the study
                         director and included in the report?
VI.b                  Were modeling and simulation limitations identified
                         and explained?                     
VI.c                  Has each model in the study been described?
VI.d                  Are the model processes clearly explained,
                         documented and understood?         
VII                   Measures of effectiveness (MOEs) and essential
                         elements of analysis (EEAs) are addressed
VII.a                 (Do MOEs adhere to the guidance in the study terms
                         of reference?)                     
VII.b                 (Are the MOEs fully addressed in the study?)
VII.c                 (Are the EEAs addressed in the study?)
Presentation of results: Timely, complete, accurate,     
concise, and relevant to the client and stakeholders     
VIII                  Presentation of results support findings
VIII.a                Does the report address the objectives?
VIII.b                Does the report present an assessment that is well
                         documented and conclusions that are supported by the
                         analyses?                          
VIII.c                Are conclusions sound and complete?
VIII.d                Are recommendations supported by analyses?
VIII.e                Is a realistic range of options provided?
VIII.f                Are the study results presented in the report in a
                         clear manner?                      
VIII.g                Are study participants/stakeholders (i.e., services
                         and Combatant Commands) informed of the study
                         results and recommendations?       

Source: GAO analysis of industry and DOD study and research standards.

We used these relevant standards as our criteria to assess the reported
MCS results. All eight key areas of the study process were considered to
have equal importance relative to the soundness and completeness of the
study; that is, a sufficiently serious concern in any category could raise
questions concerning the adequacy and completeness of the report. The
analysts independently reviewed evidence relevant to each subquestion,
including the study itself, the study Terms of Reference, and its
strategic planning guidance. For each of the subquestions in the key study
process areas, the analysts determined whether (1) the evidence had no
limitations or raised no concerns, (2) the evidence had some limitations
or raised some concerns, (3) the evidence had significant limitations or
raised significant concerns, or (4) we could not determine the extent of
limitations or concerns because there was not sufficient information. The
analysts then met, compared, and discussed their individual assessments,
and reached an overall assessment for each subquestion. Areas of the study
where we identified either "some" or "significant" limitations or concerns
were considered to affect the adequacy or completeness of the study.
Additionally, areas of the study that could not be assessed because of the
lack of supporting documentation were considered to affect the credibility
of the study.

Throughout our review PA&E officials told us that the documentation needed
to support and verify the key analytical and decision-making processes
used to conduct the MCS, documentation that was vetted and approved by DOD
leadership and all of the study participants, would not be completed and
available for our review until the study report was issued. However, after
the report was issued, we were told that the report provides all of the
supporting documentation needed and that the other documentation we
requested could not be provided. As a result, we were unable to determine
the adequacy and completeness of the analytical and decision-making
processes that supported the MCS effort to evaluate the credibility of the
study. We believe these processes are significant to the credibility of
the study and its results.

We conducted our review from July 2004 through July 2006 in accordance
with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense

See comment 3.

See comment 2.

See comment 1.

See comment 5.

See comment 4.

                  GAO's Responses to DOD's Technical Comments

           1. DOD disagreed with our assessment that the modeled
           year-2012-did not place as much demand for mobility assets in
           support of smaller military operations, such as peacekeeping, as
           other years. DOD also stated that we incorrectly focused on the
           number of operations, not the level of effort. We disagree. The
           MCS report (Annex A to Appendix F) made no distinction between the
           number of lesser contingencies and the level of effort.
           Specifically, the Vignettes for Baseline Security Posture Analysis
           did not report the level of effort by year and instead aggregated
           the data, in many instances across several modeled years.
           Consequently, we compared the number of operations conducted in
           the model year. Throughout our review, PA&E officials consistently
           told us that the completed MCS report would contain all the
           documentation needed to support its analyses. Furthermore,
           although demand in the modeled year may exceed previous efforts,
           the MCS was chartered to assess the ability of the mobility system
           to support the National Military Strategy into the next decade.
           The size of the selected model year in relation to efforts
           conducted between 1941 and 2006 is not at issue. As our report
           makes clear, our concern is that modeling what appears to be the
           least demanding year does not address whether the United States
           has sufficient capability to support national objectives during a
           peak demand period and may underestimate and underreport demands
           to senior decision makers.
           2. DOD disagreed with our observation that the MCS report does not
           provide a clear understanding of the direct relationship of
           warfighting objectives to transportation capabilities. We
           disagree. We understand that achieving a combat task requires
           delivering the right commodity to the right place at the right
           time. However, the specific combat tasks (e.g., attaining air
           superiority) necessary to satisfy the commander's campaign
           objectives are not a direct measure of mobility capability. For
           example, the problems in using a single metric are reflected in
           the MCS Appendix H, where the MCS report states that "the study
           itself still had difficulty in evaluating the operational impact
           of the delivery of theater support elements," adding that "we
           [DOD] were unable to develop a satisfactory mechanism to capture
           the linkage of the closely related, but delayed, follow-on support
           needed..." Finally, the MCS concludes that "there was no way to
           model a decrease in [Air Force] squadron effectiveness if this
           support was late. Additional effort is required to develop a
           methodology for evaluation the operational impact of support
           equipment availability." We continue to believe, and DOD agreed
           with our recommendation, that warfighting metrics, in conjunction
           with mobility metrics, can give decision makers a full picture of
           the capabilities needed to meet a specific warfighting goal.
           3. DOD disagreed with our statement that the MCS was unable to
           model the flexible deterrent options/deployment order process and
           that the study in fact analyzed flexible deterrent option (FDO)
           movements to theater. We do not dispute that DOD analyzed FDO
           movements as part of the MCS analysis. However, the degree to
           which the MCS analyses successfully modeled FDOs is in question.
           The MCS report, Appendix H, stated that an individual FDO is
           essentially the same as a deployment order. It also states in that
           section that "Deployment orders [DEPORDS] are issued to deploy
           specific capabilities as commitment decisions are made. This was
           not modeled due to lack of data on how DEPORDS would have been
           issued for an MCO [major combat operation] deployment." In the
           same paragraph, the MCS concludes that "the impact on the mobility
           system of the DEPORD process should be assessed in follow-on MCS
           analyses," adding that "there is a data deficit on how to model
           and execute a DEPORD process." Furthermore, the MCS report states
           that "additional analysis is required to investigate the
           implications of the DEPORD process decisions and provide data for
           future decision-makers to develop a DEPORD execution process."
           4. DOD believes our statement concerning homeland defense missions
           is misleading and is not sure what the report means by "actual
           homeland defense demands." We removed the word "actual" and
           clarified our report to discuss "demands derived from a well
           defined and approved concept of operations for homeland defense",
           which were not available for the study according to the MCS
           report. Furthermore, in chapter 4, the MCS report states that
           "maintaining a dedicated capability to support multiple, nearly
           simultaneous homeland defense/civil support events concurrent with
           the peak demand period of two overlapping warfights, greatly
           exceeds programmed lift capabilities". This raises questions about
           the conclusions of the MCS that there are adequate mobility
           capabilities to meet national security objectives. Also, in
           Chapter 3 of the MCS report, it states that the DOD homeland
           defense concept of operations required refinement and was one of
           nine issue areas cited within the homeland defense portion of the
           study that "need to be addressed and investigated in more detail."
           All of these nine areas potentially impact mobility support for
           homeland defense operations. Moreover, the MCS Executive Summary
           notes that reassessment of these missions is required as DOD's
           role in homeland defense evolves. The MCS report, chapter 4,
           concludes by calling for further refinement of mission
           requirements, continuing risk assessments, and an effort to
           determine corresponding mobility solutions. We continue to believe
           that the MCS conclusion that adequate mobility capability exists
           is conditional given the results of the homeland defense portion
           of the study and that the accuracy and completeness of the data,
           modeling, and results for this portion of the MCS remain in
           question.
           5. We disagree with DOD's characterization that our information
           was "misleading" regarding the adequacy of some aspects of the
           MCS' modeling and data. Furthermore, we continue to disagree with
           DOD's statement that the models and data used by the study were
           sound and adequate to assess relevant aspects of missions required
           to support the National Military Strategy, and that the results of
           the study are valid. In this report, as in our September 14, 2005
           report,1 we reaffirm our concern that the data and models used by
           the study may not be sound and the results may not be valid since
           the verification, validation, and accreditation (VV&A) of the
           models and data used to conduct the study was not done in
           accordance with DOD policy or relevant research standards. VV&A of
           models and data reduces the risk inherent in the use of models and
           simulations by improving the credibility of modeling and
           simulation results. We do not dispute DOD's assertion that it has
           relied upon the same models to produce mobility studies done
           "since the end of the Cold War". However, as we discuss in our
           report, the MCS report fails to explain or qualify the impact that
           identified data or modeling limitations might have on its results.
           For example, in the MCS chapter 4, entitled Operational Data, the
           MCS states that "data deficiencies negatively affected MCS's
           ability to use current execution data to project future
           requirements and assess system performance." Unclear is the extent
           to which these deficiencies impacted the MCS ability to meet the
           objective of identifying mobility capability gaps, overlaps, or
           excesses and provide associated risk assessments, an MCS
           objective. Similarly, in the section of chapter 4 entitled
           Analysis Tools, the report states that "MCS analysis revealed
           several deficiencies in existing mobility models." The section
           concludes with five recommended tool enhancements but it does not
           explain the impact that the absence of these enhanced tools may
           have. We continue to believe that because of these modeling and
           data limitations, the MCS may have incorrectly estimated the
           future mobility requirements needed to support homeland defense
           missions, smaller contingencies, and major combat operations.

1 See GAO, Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies, 
GAO-05-659R (Washington, D.C., September 14, 2005), for a more detailed
discussion.

Appendix III: GAO Contact and Staff Acknowledgments

                                  GAO Contact

William M. Solis, (202) 512-8365 or [email protected]

                                Acknowledgments

Ann Borseth, Assistant Director; Brian Lepore, Assistant Director;
Nabajyoti Barkakati; Renee Brown; Claudia Dickey; Ron La Due Lake; Oscar
Mardis; Deborah Owolabi; Kenneth Patton; and Stephen Woods made
significant contributions to this report.

(350558)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149 Washington,
D.C. 20548

transparent illustrator graphic

www.gao.gov/cgi-bin/getrpt? GAO-06-938 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact William M. Solis at (202) 512-8365 or
[email protected].

Highlights of GAO-06-938 , a report to congressional committees

September 2006

DEFENSE TRANSPORTATION

Study Limitations Raise Questions about the Adequacy and Completeness of
the Mobility Capabilities Study and Report

The Department of Defense (DOD) issued the Mobility Capabilities Study
(MCS), which was intended to identify and quantify the mobility
capabilities needed to support U.S. strategic objectives into the next
decade. The MCS found that projected capabilities are adequate to achieve
U.S. objectives with an acceptable level of risk-that is, current U.S.
inventory of aircraft, ships, prepositioned assets, and other capabilities
are sufficient, in conjunction with host nation support, and assuming
planned investments take place.

The Senate report accompanying the bill for the fiscal year 2005 Defense
Authorization Act required GAO to report on the adequacy and completeness
of the MCS. GAO assessed the extent to which the MCS met generally
accepted research standards that this type of study would be expected to
meet to be considered sound and complete.

What GAO Recommends

GAO recommends that the Secretary of Defense, in future mobility s
capabilities studies beginning with any study currently under way, develop
models and data for critical missions and processes; include in study
reports an explanation of how stated limitations might impact results;
and, incorporate both mobility and warfighting metrics to determine
capabilities. In comments, DOD concurred with two of the recommendations
and claimed they did not understand the third, which GAO clarified.

DOD used an innovative approach in conducting the study and acknowledged
methodological limitations in its report; however, it did not fully
disclose how these limitations could affect the MCS conclusions and
recommendations. Therefore, it is not transparent how the analyses done
for the study support DOD's conclusions. Measured against relevant
generally accepted research standards, GAO has identified limitations in
the MCS and its report that raise questions about their adequacy and
completeness. GAO suggests that Congress and other decision makers
exercise caution in using the MCS to make investment decisions. Among
GAO's findings:

           o  Aspects of modeling and data were inadequate in some areas
           because data were lacking and the models used could not simulate
           all relevant aspects of the missions. The report did not explain
           how these limitations could affect the study results or what the
           impact on projected mobility capabilities might be. Generally
           accepted research standards require that models used are adequate
           for the intended purpose, represent a complete range of
           conditions, and that data used are properly generated and
           complete. For example, the MCS modeled hypothetical homeland
           defense missions rather than homeland defense demands derived from
           a well defined and approved concept of operations for homeland
           defense, because the specific details of the missions were still
           being determined and the data used may be incomplete. The MCS also
           was unable to model the flexible deterrent options/deployment
           order process to move units and equipment into theater because of
           lack of data, but the study assumed a robust use of this process.
           In addition, the MCS report contains over 80 references to the
           need for improved modeling or data.
           o  While the MCS concluded that combined U.S. and host nation
           transportation assets were adequate, in describing the use of
           warfighting metrics in its analyses, the report does not provide a
           clear understanding of the direct relationship of warfighting
           objectives to transportation capabilities. Additionally, the
           report stated that further analysis is required to understand the
           operational impact of increased or decreased strategic lift on
           achieving warfighting objectives. Relevant generally accepted
           research standards require that conclusions be supported by
           analyses. The use of both warfighting and mobility metrics would
           allow decision makers to know whether combat tasks were achieved
           and how much strategic transportation is needed to accomplish
           those tasks.
           o  In some cases, the MCS results were incomplete, unclear, or
           contingent on further study, making it difficult to identify
           findings and evaluate evidence. Relevant research standards
           require results to be presented in a complete, accurate, and
           relevant manner. For example, the report contains recommendations
           for further studies and assessments, five of which are under way.
           However, DOD has no plans to report the impact of these studies on
           the MCS results after the studies are complete. In addition, the
           report contains qualified information that is not presented
           clearly, such as varying assessments of intra-theater assets in
           three different places.
*** End of document. ***