Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities
Studies (14-SEP-05, GAO-05-659R).
We are reviewing the processes the Department of Defense (DOD) is
using to conduct its Mobility Capabilities Study (MCS). The MCS
is to address changes in DOD's transportation force structure and
mobility requirements due to changes in threats and certain
national security and military strategies. The study results may
underpin decisions on future strategic airlift, aerial refueling
aircraft, and sealift procurements. The study relies on the use
of various models and data inputs to develop and evaluate
transportation alternatives, including variations in alternative
transportation modes (air, land, sea) and sources (military,
civilian, foreign), as well as factors that affect transportation
mode and source decisions. The Senate Armed Services Committee
directed us to monitor the conduct of the MCS and report on the
adequacy and completeness of the report no later than 30 days
after DOD completes the study. DOD plans to issue the MCS report
during 2005. This letter is intended to bring to the Secretary of
Defense's attention preliminary observations on certain aspects
of the MCS methodology to permit you to ensure the credibility of
this and future studies. In our letter, we address the adequacy
of the department's verification, validation, and accreditation
(VV&A) of the models and simulations being used to conduct the
MCS--that is, the process the MCS team is using to identify the
models' capabilities, limitations, and performance relative to
the real world events they simulate. We will continue to monitor
the MCS and will report on the adequacy and completeness of the
methodology after DOD issues its report.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-659R
ACCNO: A36758
TITLE: Defense Transportation: Opportunities Exist to Enhance
the Credibility of the Current and Future Mobility Capabilities
Studies
DATE: 09/14/2005
SUBJECT: Defense capabilities
Defense contingency planning
Documentation
Program evaluation
Strategic mobility forces
Strategic planning
Transportation
Computer modeling
Statistical data
DOD Mobility Capabilities Study
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-659R
* Results in Brief
* Background
* Documentation of VV&A Process Is Lacking
* Conclusions
* Recommendations for Executive Action
* Agency Comments and Our Evaluation
United States Government Accountability Office Washington, DC 20548
September 14, 2005
The Honorable Donald H. Rumsfeld Secretary of Defense
Subject: Defense Transportation: Opportunities Exist to Enhance the
Credibility of the Current and Future Mobility Capabilities Studies
Dear Mr. Secretary:
We are reviewing the processes the Department of Defense (DOD) is using to
conduct its Mobility Capabilities Study (MCS). The MCS is to address
changes in DOD's transportation force structure and mobility requirements
due to changes in threats and certain national security and military
strategies. The study results may underpin decisions on future strategic
airlift, aerial refueling aircraft, and sealift procurements. The study
relies on the use of various models and data inputs to develop and
evaluate transportation alternatives, including variations in alternative
transportation modes (air, land, sea) and sources (military, civilian,
foreign), as well as factors that affect transportation mode and source
decisions.
The Senate Armed Services Committee directed us to monitor the conduct of
the MCS and report on the adequacy and completeness of the report no later
than 30 days after DOD completes the study. 1 As you may be aware, DOD
plans to issue the MCS report during 2005. This letter is intended to
bring to your attention preliminary observations on certain aspects of the
MCS methodology to permit you to ensure the credibility of this and future
studies. In our letter, we address the adequacy of the department's
verification, validation, and accreditation (VV&A) of the models and
simulations being used to conduct the MCS-that is, the process the MCS
team 2 is using to identify the models' capabilities, limitations, and
performance relative to the real world events they simulate. We will
continue to monitor the MCS and will report on the adequacy and
completeness of the methodology after DOD issues its report.
To do our work, we reviewed applicable DOD guidance, directives,
instructions, and memos that describe how DOD would conduct its mobility
capabilities
1
S. Rep. No. 108-260, at 126 (2004).
2
The MCS study team includes officials from the Office of the Secretary of
Defense, Program Analysis and Evaluation and the office of the Director of
Logistics, Joint Chiefs of Staff, identified as co-leads and study
management, as well as study participants to include representatives from
the military services, combatant commands, and contractors employed by any
of the aforementioned DOD organizations to provide input to or services in
support of the MCS.
GAO-05-659R Defense Transportation
assessments to include the National Security and Military Strategies; DOD
Strategic Planning Guidance; DOD data collection, development, and
management in support of strategic analysis directives; DOD modeling and
simulation instruction; Defense Modeling and Simulation Office guidance;
MCS Study Plan and Terms of Reference; descriptions of models used to
conduct the study; and the databases used in the models. Additionally, we
reviewed and analyzed previously published DOD mobility studies and past
GAO reports related to the studies. We interviewed study officials from
the Office of the Secretary of Defense, Program Analysis and Evaluation
and the office of the Joint Chiefs of Staff, as well as study participants
and subject matter experts from the U.S. Transportation Command, Air
Mobility Command, Surface Deployment and Distribution Command, the
combatant commands, and military services concerning the extent of their
input to the study. We interviewed a modeling and simulation subject
matter expert at the Defense Modeling and Simulation Office. We
interviewed DOD officials to try and identify the process used to ensure
the validation and verification of the models and the reliability of the
data used in the study models, and conducted a comparative analysis of
this process with applicable DOD guidance. We did not evaluate the
relevancy of the DOD guidance because it was outside the scope of our
work. We conducted this portion of our review from July 2004 through July
2005 in accordance with generally accepted government auditing standards.
Results in Brief
We are unable to assess the adequacy of the process DOD used to verify,
validate, and accredit the models used to conduct the MCS. Although
officials in the Office of Program Analysis and Evaluation stated that
they have performed an equivalent VV&A 3 process for the models used in
the MCS, there is little documentation available to describe the
equivalent process that was used. An adequate evaluation of this
self-described equivalent process cannot be conducted due to this absence
of documentation, which is compounded because DOD currently does not plan
to disclose how it conducted its equivalent VV&A process in its MCS
report. This could negatively impact the credibility of the MCS report.
DOD guidance requires that models and data go through a VV&A process, but
officials in the Office of Program Analysis and Evaluation believe that
this guidance is not relevant for models that have been used for many
years, called legacy models, because, in their view, the models and data
have already undergone an equivalent VV&A process consisting of actual
use, although the guidance does not identify actual use as an appropriate
equivalent process. Moreover, DOD was conducting VV&A on one legacy model
being used in the MCS, raising questions about the need for such actions
given the department's statements that it is unnecessary.
3 DOD, DOD Modeling and Simulation (M&S) Verification, Validation,
Accreditation (VVA), Instruction 5000.61 (Washington, D.C.: May 2003).
We are making recommendations to improve DOD's documentation of any
equivalent VV&A process the department may have and to establish the
relevancy of VV&A guidance for use with legacy models. In commenting on a
draft of this report, DOD concurred or partially concurred with all of our
recommendations. DOD's comments are reprinted in their entirety in
enclosure I.
Background
The Office of the Secretary of Defense directed its Office of Program
Analysis and Evaluation to conduct the MCS. DOD was using an array of
models and baseline data to develop transportation alternatives and
evaluate their impact on the department's capability to support military
strategy. As with past mobility requirements studies, 4 the MCS uses a
variety of models and data analyses to achieve the overall study
objectives 5 and determine the effect of the study variables on the
defense transportation system and its resultant effect on the capabilities
required to meet the mobility needs for all aspects of the National
Military Strategy. The baseline data used in the mobility models are the
foundations for the strategic analyses and contain such data as a specific
warfight scenario, concept of operations for the scenario, needed forces
and equipment, battlefield terrain and weather, and time frames. According
to DOD officials, the models have become increasingly complex over the
past 15 years and are used to analyze large volumes of data to define
mobility requirements, assess risk based on the forces' ability to achieve
war-fighting objectives, identify mobility gaps, and determine alternative
methods to achieve desired capabilities. For example, the analysis would
identify tons of equipment or number of passengers to be moved, the number
of aircraft and ships needed to move equipment and forces, and the number
of aircraft to be aerially refueled.
Modeling and simulation are assuming a larger role in military
assessments, driven in part by an appreciation for the cost, logistics,
and acquisition implications associated with DOD programs. DOD models and
simulations are to be developed in accordance with DOD policies, plans,
and guidance. 6 Generally,
4
Past mobility requirements studies conducted since the early 1990s
include: Mobility Requirements Study (1992); Mobility Requirements
Study-Bottom Up Review Update (1995); and Mobility Requirements Study for
Fiscal Year 2005 (2001).
5
The overall study objectives are to identify and/or quantify (1) how
variations in mobility capabilities support the defense strategy from
point of origin to point of use and return in the 2012 time frame; (2)
mobility capability gaps, overlaps, or excesses and associated risk
assessments with regard to conducting operations; (3) mobility capability
alternatives that mitigate operational logistic impacts caused by
challenges; (4) combinations of mobility, engineering, and infrastructure
capabilities required to support deployments and distributions required by
the defense strategy; (5) new metrics for assessing mobility capabilities;
(6) potential impact of evolving service force transformation and research
and development efforts that integrate mobility concepts for the 2024 time
frame; and (7) transformational accelerants to enable the defense
transportation system to operate in a net-centric environment.
6
DOD, DOD modeling and Simulation Verification, Validation and
Accreditation, DOD Instruction 5000.61 (Washington, D.C.: May 2003); and
DOD Modeling and Simulation Office, DOD
overall VV&A policy is established in DOD modeling and simulation master
plans. DOD and service instructions clarify the policy and guidance
indicates how to implement the policy. VV&A constitutes processes that
gather and evaluate evidence to determine, based on the simulation's
intended use, the simulation's capabilities, limitations, and performance
relative to the real objects or events it simulates.
The VV&A process entails the review, analysis, evaluation, and testing of
models and simulations, incrementally over time as the models are being
developed, by an independent agent or authority to improve the credibility
of the process. Furthermore, VV&A provides enhanced user confidence,
improved performance and reliability for the subject model and simulation
results, more predictable and accurate modeling/simulation behavior, and
reduced risk of inaccurate model outputs. Verification is the process of
determining that a model implementation and its associated data accurately
represent the developer's conceptual design. Validation is generally
understood as an independently administered process where multiple parties
that have no vested interest in the outcomes participate in developing (1)
an appropriate set of standard protocols for a simulation and (2) protocol
reviews across several occasions and settings. Generally, before formal
validation is applied, the goals of the simulation's performance are
thoroughly developed and specified. The validation process establishes the
credibility of a simulation by evaluating its capability and accuracy
relative to its intended use. Successfully completing validation enhances
the credibility of the simulation by offering assurances that it can be
relied on for reproducible results appropriate for its objectives.
Additionally, data validation is to be put in the context of its
suitability for use in models. For this reason, the data cannot be
validated independently of the models for which they are intended. The
results of the verification and validation phase are used to support the
accreditation decision, which is the user's official certification that a
model, simulation, or federation of models and simulations and the
associated data are acceptable for use for a specific purpose.
Documentation of VV&A Process Is Lacking
Officials in the Office of Program Analysis and Evaluation stated that
they have performed an equivalent VV&A process for the legacy models used
in the MCS, but there is little documentation available to describe the
equivalent process that was used. As a result, we are not able to assess
the adequacy of DOD's self-described equivalent process. Office of Program
Analysis and Evaluation and Joint Data Support officials told us that most
of the documentation does not yet exist and will not be completed until
after the MCS is completed. Moreover, at the time of our review, DOD had
not planned to describe the equivalent VV&A process in its MCS report. The
absence of VV&A documentation for the models
Verification, Validation and Accreditation Recommended Practices Guide
(Washington, D.C.: August 2004).
Page 4 GAO-05-659R Defense Transportation
and data used to conduct the MCS and the lack of disclosure in the
published MCS report could limit the study's credibility.
DOD guidance, issued by the Under Secretary of Defense for Acquisition,
Technology and Logistics, requires that DOD models and data go through a
VV&A process. The Office of Program Analysis and Evaluation acknowledged
that it did not comply with the guidance because it believes such an
approach is not warranted for legacy models that have been used for many
years. Moreover, these officials believe that such long-term use
constitutes a VV&A process equivalent to that required in the DOD
guidance. However, the DOD guidance does not identify the actual use of a
model as constituting an equivalent VV&A process.
DOD is using the following nine mobility models to conduct the MCS:
o Aerial Port of Debarkation
o Air Mobility Operations Simulation
o Combined Mating and Ranging Planning System
o CONUS (Continental U.S.) Enhanced Logistics Intra-theater Support Tool
o Model for Inter-theater Deployment by Air and Sea
o Integrated Computerized Development System
o Joint Throughput Modeling Tool
* TRANSPORT
officials, eight of the nine models did not go through
the VV&A process specified in the DOD guidance. 7 Office of Program
Analysis and Evaluation, U.S. Transportation Command, Air Mobility
Command, and Surface Deployment and Distribution Command officials told us
that (1) most of the models have been used within DOD for many years and
have proved reliable, and
(2) many subject matter experts work with the models and the output daily
to assure ongoing error detection and swift corrections when needed. These
officials maintain that actual use of the models for a long period of time
constitutes an equivalent VV&A process.
Nonetheless, DOD officials were conducting VV&A on one of the legacy
models being used in the MCS while MCS officials were simultaneously
questioning the relevancy of DOD's guidance for the legacy models. For
example, Surface Deployment and Distribution Command officials told us
that the CONUS Enhanced Logistics Intra-theater Support Tool model was
undergoing VV&A at the time of our review. It is unclear why DOD is
conducting VV&A on this model
DOD conducted VV&A on the Combined Mating and Ranging Planning System
model about 15 years ago in compliance with departmental guidance but
could not locate documentation to demonstrate how VV&A was done. As a
result, we could not evaluate the adequacy of the VV&A process.
given Office of Program Analysis and Evaluation and other officials'
belief that it is unnecessary. Furthermore, the extent to which DOD's
guidance may be irrelevant as asserted by these officials is unknown,
because the department has not evaluated the current VV&A guidance to
determine its relevancy for use with legacy models.
Conclusions
Models and simulations approximate the real world. The approximations must
be justified to assure modeling and simulation users that their
predictions are credible within the bounds of specific situations,
environments, and circumstances. When modeling and simulation are
credible, decision makers have greater assurance that they are well
informed and thus can make well-founded decisions. VV&A reduces the risk
inherent in the use of models and simulations by improving the credibility
of modeling and simulation results. VV&A also enhances credibility by
applying a process of incremental review, analysis, evaluation, and
testing by an independent agent. In light of the fact that DOD did not
follow its guidance, the absence of model and baseline data VV&A
documentation for the models and data used to conduct the MCS, and the
planned lack of disclosure in the soon to be published MCS report, could
limit the study's credibility. Moreover, MCS officials maintain that DOD
guidance regarding VV&A is not relevant to legacy models and data.
However, until the department evaluates the guidance to determine its
relevancy for use with legacy models or incorporates guidance showing how
actual model usage is to be applied as an equivalent VV&A process, the
validity of DOD's assertion is uncertain. When conducted as intended, VV&A
provides greater assurance that the MCS outputs are accurate. Ultimately,
if the MCS inaccurately identifies mobility requirements, DOD officials
may be less well informed and may therefore inadvertently obtain
insufficient mobility assets or more than needed and thus waste resources.
Recommendations for Executive Action
We recommend that you direct the Director, Office of Program Analysis and
Evaluation, to take the following three actions:
o develop documentation that describes the equivalent VV&A process used
to verify and validate the mobility models and baseline data used to
conduct the MCS prior to publishing any portion of the study results,
o disclose in the published MCS report the equivalent VV&A process used
on the models and baseline data, and
o work with the Office of the Under Secretary of Defense for
Acquisition, Technology and Logistics to evaluate the current DOD VV&A
guidance to determine its relevance for use with legacy models and to
change the guidance if appropriate.
Agency Comments and Our Evaluation
In commenting on a draft of this report, the DOD concurred with two of our
recommendations and partially concurred with the third.
In its comments, DOD concurred with our first and second recommendations
that the Office of Program Analysis and Evaluation develop documentation
that describes the equivalent VV&A process used to verify and validate the
mobility models and baseline data used to conduct the MCS prior to
publishing any portion of the study results, and disclose in the published
MCS report the equivalent VV&A process used on the models and baseline
data. In its comments, DOD stated that the mobility modeling community has
amassed substantial expertise during the past 25 years and that a
"vigorous, collaborative VV&A process that is fully consistent with and in
many respects exceeds the intent of DOD VV&A guidance has been put into
place." DOD noted the MCS report will provide information on the VV&A
process, and will exceed the level of documentation provided in past
reports on DOD mobility studies.
DOD partially concurred with our recommendation to evaluate the current
DOD VV&A guidance to determine its relevance for use with legacy models
and change the guidance if deemed appropriate. In its response, DOD
essentially agreed with our recommendation, but pointed out that the
Office of the Under Secretary of Defense for Acquisition, Technology and
Logistics is responsible for VV&A guidance. Therefore, we refined our
recommendation to recommend that the Director of the Office of Program
Analysis and Evaluation work with the Office of the Under Secretary of
Defense for Acquisition, Technology and Logistics to evaluate the current
DOD VV&A guidance to determine its relevance for use with legacy models
and to change the guidance if appropriate.
Additionally, DOD expressed concern that it was premature to reach a
conclusion as to the adequacy of the department's VV&A process because
some of the documentation requested cannot be provided until the final
report is written and that our draft report should be amended. We agree
that we cannot assess the adequacy until the documentation is available
and have so stated in our report. We also stated in our report that at the
time of our review, DOD had not planned to describe the VV&A process in
its MCS report nor had it planned to perform VV&A because the legacy
models being used were reliable. As we noted in our report, at least one
of the models was already undergoing a VV&A.
Moreover, DOD questioned our conclusion that if an adequate VV&A process
could not be documented, then the credibility of the MCS could be limited.
Specifically, DOD stated that convincing empirical evidence indicates that
the MCS report's credibility will not be limited by the VV&A documentation
associated with the legacy models, because DOD has used the models for two
decades with no credibility limitations noted. As we noted in our report,
DOD guidance indicates that a well-documented VV&A process for the models
used to conduct the study will add to the MCS report's credibility. Also
as we noted in our report, DOD guidance states that the VV&A process
provides enhanced user confidence, improved performance and reliability
for the subject model and simulation results, reduced risk of inaccurate
model outputs, and offers assurances that a particular model or simulation
can be relied on for reproducible results appropriate for its objectives.
While we support the use of empirical evidence, such evidence is normally
verifiable.
Finally, DOD expressed concern that we linked the VV&A of mobility models
used to conduct the MCS with the prospect of inaccurate identification of
mobility requirements. We disagree. DOD guidance states that VV&A reduces
the risk inherent in the use of models and simulations by improving the
credibility of their results and provides greater assurance that the study
outputs are accurate. If the models supporting the MCS do not effectively
simulate the real world and DOD uses the results to complete the MCS and
subsequently base acquisition decisions, then we maintain our caution--DOD
could inadvertently obtain insufficient mobility assets or could acquire
more than needed and thus waste resources.
Enclosure I contains the full text of DOD's comments.
- - - -
We are sending copies of this report to the Chairman and Ranking Members
of the Senate Armed Services Committee as well as to the Chairmen and
Ranking Members of the Senate Appropriations Committee, Subcommittee on
Defense, the House Armed Services Committee, and the House Appropriations
Committee, Subcommittee on Defense. This letter is also available at no
charge on the GAO's Web site at http:www.gao.gov. If you or your staff
have any questions on the matters discussed in this letter, please contact
me at (202) 512-8365 or solisw@gao.gov. Key contributors to this report
are listed in enclosure II.
Sincerely yours,
William M. Solis, Director Defense Capabilities and Management
Enclosures
Enclosure I
Comments from the Department of Defense
Enclosure I
Enclosure I
Enclosure II
GAO Contact and Staff Acknowledgments
GAO Contact William M. Solis, (202) 512-8365
Acknowledgments Key contributors to this report include Brian J.
Lepore, Claudia Dickey, Ron La Due Lake,
Oscar Mardis, Deborah Owolabi, Kenneth
Patton, and R.K. Wild.
(350708)
*** End of document. ***