Contract Management: Service Contract Approach to Aircraft
Simulator Training Has Room for Improvement (22-SEP-06,
GAO-06-830).
The Air Force has turned to service contracts for the F-15C,
F-16, Airborne Warning and Control System, and F-15E, and the
Army has done the same for helicopter simulator training at its
Flight School XXI. The contractors own, operate, and maintain the
simulator hardware and software. The military services rely on
industry to capitalize the required up-front investment, with the
understanding that the contractors will amortize this investment
by selling training services by the hour. GAO was asked to
address (1) the factors that led the Air Force and Army to
acquire simulator training as a service and whether the decision
to use this approach was adequately supported; (2) whether
implementation of the approach has resulted in the planned number
of simulator training sites being activated; and (3) whether the
Air Force and Army are effectively tracking the return on their
expenditure of taxpayer dollars. GAO makes recommendations to the
Secretary of Defense intended to improve management and oversight
of these service contracts to help ensure that the best approach
is used to provide the war-fighter with needed training. In
written comments on a draft of this report, DOD concurred with
all but one of the recommendations, only partially concurring
with one pertaining to the Army's simulator utilization rates.
GAO continues to believe that the Army needs to track the extent
to which it is using simulator availability.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-830
ACCNO: A61306
TITLE: Contract Management: Service Contract Approach to
Aircraft Simulator Training Has Room for Improvement
DATE: 09/22/2006
SUBJECT: Cost analysis
Defense procurement
Department of Defense contractors
Federal funds
Financial analysis
Flight training
Funds management
Military training
Policy evaluation
Schedule slippages
Service contracts
Training utilization
Airborne Warning and Control System
AWACS
Eagle Aircraft
F-15C Aircraft
F-15E Aircraft
F-16 Aircraft
Fighting Falcon Aircraft
Strike Eagle Aircraft
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-830
*
* Results in Brief
* Background
* Contract Information
* Acquisition Reforms Shaped Contract Structures
* Key Players
* Service Contract Approach Had Perceived Benefits but Was Not
* Simulators Had Lost Out in Competition for Procurement Funds
* Service Contracts Intended to Yield More Concurrent Simulato
* Decision to Use Service Approach Not Supported with Thorough
* While Training Has Improved, the Expected Number of Simulato
* Some Air Force and Army Sites Are Operating and Officials Ci
* Air Force Funding Uncertainties Have Resulted in Schedule Sl
* Army's Simulator Activation Schedule Was Also Delayed
* F-16 Simulator Training Gap Anticipated
* Return on the Expenditure of Taxpayer Dollars Is Not Effecti
* Information on Utilization Rates Is Lacking
* Payments for Tasks during Development Period Are Not Transpa
* Award-term Evaluations Do Not Measure Key Acquisition Outcom
* Conclusion
* Recommendations for Executive Action
* Agency Comments and Our Evaluation
* Appendix I: Scope and Methodology
* Appendix II: Comments from the Department of Defense
* Appendix III: Comparison of Contractor Requirements under Co
* Appendix IV: Potential Duration of Air Force and Army Simula
* Appendix V: GAO Contact and Staff Acknowledgments
* Order by Mail or Phone
Report to the Subcommittee on Readiness, Committee on Armed Services, U.S.
House of Representatives
United States Government Accountability Office
GAO
September 2006
CONTRACT MANAGEMENT
Service Contract Approach to Aircraft Simulator Training Has Room for
Improvement
GAO-06-830
Contents
Letter 1
Results in Brief 2
Background 4
Service Contract Approach Had Perceived Benefits but Was Not Adequately
Supported 9
While Training Has Improved, the Expected Number of Simulator Sites Has
Not Been Activated 12
Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked
19
Conclusion 28
Recommendations for Executive Action 29
Agency Comments and Our Evaluation 30
Appendix I Scope and Methodology 32
Appendix II Comments from the Department of Defense 34
Appendix III Comparison of Contractor Requirements under Commercial and
Non-commercial Acquisition Procedures 38
Appendix IV Potential Duration of Air Force and Army Simulation Contracts
43
Appendix V GAO Contact and Staff Acknowledgments 44
Tables
Table 1: Air Force and Army Simulator Training Contracts 6
Table 2: Operational Simulator Training Locations and Start Dates as of
July 2006 13
Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal
Year 2002 Program Objective Memorandum Funding Levels 14
Table 4: Key Events That Have Affected Funding for the DMO Program 15
Table 5: Comparison of Planned and Actual Site Activations for AWACS 15
Table 6: Comparison of Planned and Actual Site Activations for F-15C 16
Table 7: Comparison of Planned and Actual Site Activations for F-16 16
Table 8: Comparison of Monthly Utilization Rate Calculation Methods for
DMO System Simulatorsa 21
Table 9: Comparison of Site Survey Costs at Selected DMO System Sites 24
Table 10: Comparison of Training Capability Requirements Assessments and
Training Capability Assessments for DMO Systems 24
Table 11: Comparison of Award-term Areas of Evaluation and the Weight
Given to Each Area by System 26
Figures
Figure 1: Flight School XXI Schedule Rebaselines 17
Figure 2: Comparison of Monthly Utilization Rates for AWACS 20
Figure 3: Original Rate Structure for F-15C Simulator Service Contracts 23
Abbreviations
AAVS Advanced Aircraft Virtual Simulators AB air base AFB Air Force base
AWACS Airborne Warning and Control System DMO Distributed Mission
Operations DOD Department of Defense FAR Federal Acquisition Regulation
O&M operation and maintenance OMB Office of Management and Budget TH
training helicopter
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
September 22, 2006 September 22, 2006
The Honorable Joel Hefley Chairman The Honorable Solomon P. Ortiz Ranking
Minority Member Subcommittee on Readiness Committee on Armed Services
House of Representatives The Honorable Joel Hefley Chairman The Honorable
Solomon P. Ortiz Ranking Minority Member Subcommittee on Readiness
Committee on Armed Services House of Representatives
In the late 1990s, the Air Force embarked upon an innovative strategy for
acquiring flight simulator training for several of its aircraft platforms.
Faced with increasingly obsolete simulators and growing competition for
procurement dollars, the Air Force decided to buy simulator services from
contractors-using operation and maintenance (O&M) funds-rather than
continue the practice of having the government own and operate the
simulators. This strategy was the beginning of an approach to
war-fighting, known as Distributed Mission Operations (DMO), in which
high-fidelity simulators would be used for training across dispersed
locations. The initial platforms included in the DMO program were the
F-15C, F-16, Airborne Warning and Control System (AWACS), and F-15E. In
2003, the Army followed the Air Force lead and awarded a service contract
to obtain simulator training for the helicopters in its Flight School XXI
program. The estimated value of the Air Force and Army contracts is nearly
$2 billion. A key aspect of the service contract approach is the increased
reliance on contractors to keep the simulators concurrent with aircraft
upgrades and ensure they are available for use. In the late 1990s, the Air
Force embarked upon an innovative strategy for acquiring flight simulator
training for several of its aircraft platforms. Faced with increasingly
obsolete simulators and growing competition for procurement dollars, the
Air Force decided to buy simulator services from contractors-using
operation and maintenance (O&M) funds-rather than continue the practice of
having the government own and operate the simulators. This strategy was
the beginning of an approach to war-fighting, known as Distributed Mission
Operations (DMO), in which high-fidelity simulators would be used for
training across dispersed locations. The initial platforms included in the
DMO program were the F-15C, F-16, Airborne Warning and Control System
(AWACS), and F-15E. In 2003, the Army followed the Air Force lead and
awarded a service contract to obtain simulator training for the
helicopters in its Flight School XXI program. The estimated value of the
Air Force and Army contracts is nearly $2 billion. A key aspect of the
service contract approach is the increased reliance on contractors to keep
the simulators concurrent with aircraft upgrades and ensure they are
available for use.
At your request, we addressed (1) the factors that led the Air Force and
Army to acquire simulator training as a service and whether the decision
to use this approach was adequately supported, (2) whether implementation
of this approach has resulted in the planned number of simulator training
sites being activated, and (3) whether the Air Force and Army are
effectively tracking the return on their expenditure of taxpayer dollars.
At your request, we addressed (1) the factors that led the Air Force and
Army to acquire simulator training as a service and whether the decision
to use this approach was adequately supported, (2) whether implementation
of this approach has resulted in the planned number of simulator training
sites being activated, and (3) whether the Air Force and Army are
effectively tracking the return on their expenditure of taxpayer dollars.
To address these objectives, we interviewed Air Force and Army officials,
as well as contractor representatives; performed a detailed analysis of
the Air Force and Army service contracts; reviewed contractor performance
measurements, annual evaluations, and payment invoices; and analyzed
historical documents such as acquisition plans, briefings, and decision
memorandums. We visited Langley Air Force Base (AFB), Virginia, to view To
address these objectives, we interviewed Air Force and Army officials, as
well as contractor representatives; performed a detailed analysis of the
Air Force and Army service contracts; reviewed contractor performance
measurements, annual evaluations, and payment invoices; and analyzed
historical documents such as acquisition plans, briefings, and decision
memorandums. We visited Langley Air Force Base (AFB), Virginia, to view
F-15C simulator training; Shaw AFB, South Carolina, to view F-16 training;
and the U. S. Army Aviation Center, Fort Rucker, Alabama to view the use
of Flight School XXI training helicopters (TH), the TH-67 Virtual
Simulators and the Blackhawk and Chinook simulators. At each of these
installations, we met with military officials and the contractor officials
responsible for the training simulators. See appendix I for additional
details on our scope and methodology. We performed our review from
September 2005 to July 2006 in accordance with generally accepted
government auditing standards.
Results in Brief
The Air Force and Army turned to service contracts for simulator training
primarily because efforts to modernize existing simulator hardware and
software had lost out in the competition for available procurement funds.
Buying simulator training as a service meant that O&M funds could be used
to fund the requirement. Shifting the responsibility for simulator
ownership, operation, and maintenance from the government to the
contractor was thought to provide quicker state-of-the-art pilot and
aircrew training capabilities and allow for automatic simulator upgrades
to match the changing configurations of aircraft. However, the decision to
use service contracts was not supported by a thorough analysis of the
costs and benefits of this approach versus alternative approaches, despite
a Department of Defense (DOD) directive on training that provides for an
evaluation of the benefits and trade-offs of potential alternative
training solutions. While the Air Force and Army provided us with
briefings that showed the decision-making process, the information does
not provide adequate rationale for why certain alternatives were
eliminated from consideration. A more in-depth analysis could have
proactively addressed and possibly mitigated the risks associated with
this approach, where if a contractor fails to perform, the government is
left without simulator hardware, software, or the anticipated level of
training services.
While Air Force and Army officials told us the new simulators are
significant improvements over what they had previously, the expected
number of Air Force simulator training sites has not been activated. For
the Air Force, O&M funds have not been allocated to the contracts at the
anticipated levels, leading to schedule slippages and fewer simulator
sites in place than planned. Efforts to add funds, for example by shifting
flying hour funds into the DMO program in 2003, have not been sufficient
to regain the schedule and activate the number of sites anticipated.
Further, the F-16 simulator training contractor, citing the funding
problems and subsequent schedule slips, notified the Air Force in March
2004 that its situation was no longer financially viable under the
contract and that it could not continue to provide simulator training as
originally agreed. The Air Force will allow the contract to expire in June
2007. It is re-competing the F-16 training contract, which will likely
result in a training gap for pilots-possibly over 2 years-and additional
costs to the Air Force. Two aspects of the original contract, awarding it
as a commercial acquisition and including an award-term provision,1 will
not be included in the new contract.
The return on expenditure of taxpayer dollars is not being effectively
tracked in three key ways:
o Air Force utilization of simulator training frequently falls
well below the hours for which the government is paying. In
addition, utilization rates at different sites are reported based
on very different criteria, leading to decisions to activate new
training sites based on noncomparable information. The Army is not
collecting data on utilization rates at all.
o The government has little insight into what it is paying for
during the development period before training is activated, which
can take more than a year. Our analysis of invoices and
discussions with contractor officials suggest that the government
is, in effect, funding the private sector for a portion of its
up-front costs to acquire and develop the simulators, although the
invoices merely reflect discrete tasks such as site surveys.
o Award-term evaluations are being used to measure performance,
but the evaluation factors do not always measure key acquisition
outcomes such as simulator availability and concurrency. In
addition, the ability to "roll over" positive points on Air Force
contracts can result in the contractor being awarded an additional
year for only satisfactory performance-a scenario that is highly
unlikely under the Army's award-term plan. Air Force, Army, and
some contractor officials indicated that award-term provisions may
not be an effective incentive for improved performance.
We make recommendations in this report to DOD on actions that can
be taken to help ensure that the best approach is used to provide
the war-fighter with needed training, that the incentives motivate
contractor performance toward achieving desired training outcomes,
and that available simulator training is used in the most
effective and efficient manner. In written comments on a draft of
this report, DOD concurred with all but one of our
recommendations. DOD partially concurred with our recommendation
that the Army track and record monthly utilization rates on
simulators at Flight School XXI. Nevertheless, DOD stated that the
contractor is required to submit utilization data and that the
data are available. Our recommendation was intended to encourage
DOD to fully understand its student training requirements and to
collect the information to decide whether it needs to adjust those
requirements or contract provisions regarding simulator
availability. Whether the utilization rates pertain to individual
simulators or the student training load as a whole, we believe
that the Army needs to know the extent to which it is using the
simulator availability it is buying. DOD also offered two
corrections to information in our draft report; we made changes as
appropriate. DOD's comments are included in their entirety in
appendix II.
Background
Air Force and Army cite the increasingly complex training
requirements needed to prepare for the ever more lethal
battlefield environment as a factor that has led to greater
reliance on flight simulators. A flight simulator is a system that
tries to realistically replicate, or simulate, the experience of
flying an aircraft. Flight simulators range from video games to
full-sized cockpit replicas mounted on hydraulic (or
electromechanical) actuators and controlled by state-of-the-art
computer technology. According to Air Force and Army officials,
aircraft simulators are a cost-effective way of helping to develop
and refine operational flight skills. Simulators can facilitate
training that might be impractical or unsafe if done with actual
systems and allow for concentrated pilot practice in selected
normal and emergency actions. Simulators also can train operators
and maintainers to diagnose and address possible equipment faults,
and enhance proficiency despite shortages of equipment, space,
ranges, or time.
In the late 1990s, the Air Force and Army were faced with
increasingly obsolete simulators and the need to quickly acquire
up-to-date pilot and aircrew training. In 1997, the then-Commander
of the Air Force's Air Combat Command proposed an innovative
approach of buying training as a service, under which the
contractors would own, operate, and maintain the simulator
hardware and software. The simulator service contracts are one
component of a much broader effort, now known as the DMO program.
The DMO goal is to provide state-of-the-art simulator training
on-demand at the location of the trainee, with the ultimate vision
of networking different sites together to create more realistic
flying scenarios. Plans call for each fighter unit eventually to
be equipped with high-fidelity simulators. As of the fiscal year
2002 budget, the DMO program was formalized in the Air Force
budget with the assignment of a program element line item that
combined previous program elements for the various simulator
systems. For fiscal year 2006, over $200 million was budgeted for
the program.
In the early 2000s, Army use of rotary-wing aviation simulation
training was limited because the simulators being used were
grossly obsolete and based on late 1970s' technology. To revamp
its helicopter training, the Army in late 2001 began the Flight
School XXI program. Following the Air Force's lead, the Army
decided to acquire up-to-date simulator training using a service
contract.
Congress and the Office of Management and Budget (OMB) have
recently addressed the growing level of procurement of services.
For example, Congress included provisions in Section 801 of the
National Defense Authorization Act for Fiscal Year 20022 designed
to improve management and oversight of procurement of services. To
ensure that DOD acquires services by means that are in its best
interest and managed in compliance with applicable statutes,
regulations, directives, and other requirements, the Act required
DOD to establish a service acquisition management structure,
comparable to the management structure that applies to the
procurement of products.3 In September 2003, we reported that DOD
and the military services had a management structure in place for
reviewing individual services acquisitions valued at $500 million
or more, but that approach did not provide a departmentwide
assessment of how spending for services could be more effective.4
Also, OMB Circular A-11's Appendix B, "Budgetary Treatment of
Lease-Purchases and Leases of Capital Assets," was amended in 2005
to require agencies to submit to OMB for review any service
contracts that require the contractor to acquire or construct
assets valued over $50 million. While these provisions do not
apply to the previously-awarded simulator training contracts,
future replacement contracts will be covered.
Contract Information
All of the Air Force and Army simulator service contracts are
funded with O&M funds. O&M funds are typically used for such
things as military force operations, training and education, and
depot maintenance. The contracts are requirements contracts,
meaning that the government, within available funds, shall order
from the contractor all the training services specified for each
of the aircraft platforms that are required during the effective
performance periods. Additionally, each contract contains language
limiting the government's liability in the event the contract is
terminated. For example, the Air Force F-15C contract states that
the government reserves the right to terminate the contract for
its sole convenience and that such termination prior to the
issuance of a funded task order shall result in no payment to the
contractor of any amount for any work performed or costs incurred.
Table 1 provides additional descriptive information for each
contract.
Table 1: Air Force and Army Simulator Training Contracts
Source: Air Force and Army contracts.
aEstimated costs for contract base years and all potential
award-term or option years.
bThe F-15C and F-15E contracts were originally awarded to
McDonnell Douglas Corporation, which became a wholly-owned
subsidiary of the Boeing Company.
cTraining services for the F-15E are ordered under a much larger
umbrella contract, Training Systems Acquisition II.
The military services are relying on industry to capitalize the
required up-front investment needed to acquire simulator hardware
and software, with the understanding that the contractors will
amortize this investment by selling training services by the hour.
Each contract establishes operating hours and the hourly payment
rates for the life of the contracts, with rates structured to
provide the contractor with higher income in the initial years of
service. In calendar year 2004, for example, if the F-16
contractor provided Shaw AFB with simulator availability that met
95 percent of the required system elements,5 the hourly rate would
be $5,225, whereas in calendar year 2006, it would drop to $709
per hour.
We have previously identified the need to examine the appropriate
role for contractors to be among the challenges in meeting the
nation's defense needs in the 21st century.6 We recently reported
that the government's increasing reliance on contractors for
missions previously performed by government employees highlights
the need for sound planning and contract execution.7
Acquisition Reforms Shaped Contract Structures
The structure of the simulator service contracts was heavily
influenced by mid-1990s' acquisition reform initiatives such as
the Federal Acquisition Streamlining Act of 19948 and the
Clinger-Cohen Act of 1996.9 These Acts encouraged agencies to use
commercial acquisition procedures as a way to streamline the
acquisition process. Differences under commercial versus
non-commercial procedures pertain, for example, to the contracting
officer's determination of price reasonableness, the government's
right to inspect and test, and government rights to acquire
technical data. Appendix III outlines these and other key
differences. The Air Force contracts for simulator training are
structured as commercial acquisitions, but the Army's is not. Army
officials told us they could not justify calling the requirement
"commercial" because the simulators would be configured to reflect
combat helicopters, which do not exist in the commercial market.
In August 2005, a DOD Inspector General review of the procurement
procedures for the F-16 contract concluded that the simulator
service did not meet the definition or intent of a commercial
service and recommended that the Air Force not use commercial
procedures for the re-competed F-16 contract. The Air Force is
using non-commercial procedures for the new contract.
To allow for contractor recoupment of up-front investment, the
strategy to acquire simulator services envisioned longer duration
contracts. This coincided with practices in commercial industry,
where long-term relationships between buyer and seller were
becoming common. The Air Force and Army adopted this approach by
including award-term incentives in the contracts. This incentive
can best be described as a variant of an award-fee incentive,
where the contractor is rewarded for excellent performance with an
extension of the contract period instead of additional fee. Under
the award-term concept, an assessment of the contractor's
performance is presented to the term determining official, who
unilaterally determines whether to award an extension or a
reduction to the contract ordering period. The potential total
years of contract performance under the simulator contracts range
from 13 to 19.5 years. Appendix IV contains the details of each
contract. Award-term incentives are relatively new in government
contracting and are not addressed in the Federal Acquisition
Regulation (FAR).
Key Players
Several key players are involved with acquisition and use of
simulator training.
For the Air Force:
o Air Combat Command: The requiring entity-the user of simulator
training services-is located at Langley AFB, Virginia. The command
trains, equips and maintains combat-ready forces for rapid
deployment and employment.
o Aeronautical Systems Center: This organization is the
acquisition agency for the simulator contracts. Located at
Wright-Patterson AFB, Ohio, it manages development, acquisition,
modification, and in some cases, sustainment for a wide variety of
aircraft and related equipment programs. The center develops
attack, bomber, cargo, fighter, trainer, and reconnaissance
aircraft for the Air Force.
o Air Force fighter units: These are the users of the simulator
training, which currently is taking place in 10 fighter units.
For the Army:
o Fort Rucker, Alabama: Fort Rucker is the requiring entity for
the Army's helicopter flight simulator services. It is the home of
all Army aviation flight training and the location of the initial
training for new aviators, known as Flight School XXI. The types
of helicopters used in Flight School XXI training are the TH-67
basic training helicopter, Chinook, Blackhawk, Apache, and the
Attack Reconnaissance Helicopter. Unlike the Air Force's multiple
sites, the helicopter simulators provided under the service
contract are located at only this one training site, not at each
operational unit.
o Army Program Executive Office for Simulation, Training and
Instrumentation: This office's mission is to provide training,
testing, and simulation solutions for soldier readiness. The
office is co-located in Orlando, Florida, with the Naval Air
Systems Command, which awarded the contract on behalf of the Army.
Service Contract Approach Had Perceived Benefits but Was Not
Adequately Supported
Both the Air Force and Army were faced with obsolete simulators
due to decisions to not devote sufficient procurement funds to
upgrade existing simulator hardware and software. The decision to
buy simulator training as a service allowed use of O&M funds,
which would alleviate the need to compete for procurement funds.
Further, it was envisioned that service contracts would allow for
automatic simulator upgrades to match the changing aircraft
configurations, because industry would be responsible for
acquiring, operating, and maintaining the simulators and keeping
them concurrent. However, the decision to embark on a services
approach was not supported by a thorough analysis of the costs and
benefits, despite a DOD directive providing that the acquisition
of simulators is to be based on an evaluation of the benefits and
trade-offs of potential alternative training solutions.
Simulators Had Lost Out in Competition for Procurement Funds
The difficulty associated with competition for limited procurement
dollars was a key factor in the decision to turn to service
contracts for war-fighting training. Frequently, simulators have
lost out in this competition and ended up under-funded. In 1997,
the Air Force identified simulators for four aircraft-the F-15C,
F-16, F-15E, and AWACS-as "obsolete or grossly non-concurrent" due
to age, technological obsolescence, and lack of concurrency with
operational aircraft. By early 2002, the Army was also faced with
non-concurrent helicopter simulators, and field unit commanders
were reporting decreased unit readiness. For example, while the
goal of the training at Fort Rucker is to produce aviators trained
at a proficiency level of two (with level one being the highest),
Army officials reported that most of the aviators were leaving
school with only a proficiency level of three.
These degraded situations existed despite a DOD directive that
provides for the military services to ensure that all development,
procurement, operation, and support costs for the acquisition of
training simulators were programmed and funded. Recognizing the
need to keep simulators current with aircraft configurations,
particularly as the use of simulators to substitute for live
flying hours was rising, the Air Force issued specific guidance on
training devices. For example, Air Force Instruction 36-2248,
Operation and Maintenance of Aircrew Training Devices, provides
that funding be established for simulator modifications
concurrently with modifications to the weapon system. Also, Air
Force Instruction 36-2251, Management of Air Force Training
Systems, provides that the training system receive the same
precedence rating as the prime mission system it supports and the
same visibility, funding, and documentation. Nevertheless, Air
Force funding decisions had not kept flight training simulators
for the four aircraft systems concurrent with aircraft
configurations.
Also in the late 1990s, the Air Combat Command had unexpended O&M
flying hour funds available due to flight crew deployments and
obstacles in scheduling training. Use of these funds for service
contracts would alleviate the need to compete for procurement
funds in an increasingly tight arena. The competition for
procurement dollars was also a factor for the Army, which noted
that the funds necessary to maintain and upgrade its helicopter
training simulators had "not competed effectively against other
Army operational and logistics requirements."
Service Contracts Intended to Yield More Concurrent Simulators
Air Force acquisition officials conducted market research to
determine how civilian airlines acquired flight training. They
found "turnkey" training services contracts10 in place in the
commercial airline industry. These officials envisioned that
services contracts would provide quicker state-of-the-art pilot
and aircrew training and keep up with the rapid pace of technology
development by shifting the responsibilities for simulator
ownership, operation, and maintenance from the government to the
contractor. Further, with the contractor responsible for any
development, production, and testing necessary to ready the
simulators for use, the Air Force saw that it would be relieved of
these multiple acquisition efforts, an important factor given the
recently downsized acquisition offices. In addition, a stated
benefit of the service contracting approach for simulator training
as initially implemented was the streamlining or reduction in
government oversight. Since commercial acquisition procedures were
used to buy these services, fewer government system reviews were
required.
When it decided to take a new approach to solve its helicopter
simulator concurrency problems, the Army conducted its own market
research, solicited business solutions from industry, and
conferred with Air Force DMO officials.
Decision to Use Service Approach Not Supported with Thorough
Analysis of Costs and Benefits of Alternatives
Neither the Air Force nor the Army thoroughly analyzed the costs
and benefits of alternative approaches before pursuing this new
approach. DOD's August 1986, Directive 1430.13, Training
Simulators and Devices, provides that the acquisition of
simulators be based on an analysis of the training need, the
potential use of existing devices to satisfy that need, and an
evaluation of the benefits and trade-offs of potential alternative
training solutions.11 A 1999 report to the Air Force on the DMO
program also noted the importance of identifying key business
factors before embarking on a major acquisition. As a result of
the failure to conduct a thorough review of the various
alternatives to solving the problem of non-concurrent simulators,
decision makers lacked information on the potential cost and
benefit estimates that would be encountered should facts,
circumstances, and assumptions change.
The historical documents we reviewed demonstrate that within the
Air Force there was uncertainty about the cost-effectiveness of
the service contract approach to simulator training. Although the
potential for reduced costs through outsourcing certain
responsibilities and eliminating government logistics support were
cited in some decision documents, other documents indicated that
the service contract approach would not cost significantly more or
less than the traditional ownership strategy. Air Force officials
told us that a comprehensive study of various options for
providing simulator training had been commissioned. However, they
have been unable to locate it.
In preparing to re-compete the F-16 simulator contract, the DMO
program office completed a formal business case analysis in
November 2005, in response to a July 2005 congressional request.
Air Force officials acknowledged that, if not for the request, the
formal business case analysis would not have been completed.
The Army completed two business case analyses prior to contracting
for simulator services under the Flight School XXI program, but
the analyses lacked sufficient detail to provide a thorough
examination of the pros and cons of the new approach. The scope of
the analyses was limited to determining (1) what length of service
contract would be appropriate to justify the large up-front
investment required of the contractor and (2) whether projected
funding was sufficient to meet program costs in the event the Army
was required to follow the traditional acquisition approach. The
Army provided us with decision briefings that set forth various
options for simulator training, but the documents ruled out all
but the service contract approach without providing supporting
analyses of the costs and benefits associated with each
alternative. Further, the traditional method, where the government
bought the simulators, was eliminated as an option due its
perceived inability to meet the Flight School XXI 15-month
start-up time frame. This schedule eventually slipped more than 10
months with, according to Army officials, no detrimental effect on
student training schedules. The briefings do not address the
possibility that the 15-month time frame was flexible.
While Training Has Improved, the Expected Number of Simulator Sites
Has Not Been Activated
Air Force and Army officials told us the new simulators are big
improvements over what they had previously. However, the Air Force
has faced funding uncertainties using O&M funds for the contracts,
and subsequent schedule slippages have resulted in fewer simulator
sites activated than planned. In particular, the F-16 simulator
training contractor, citing the reduced activations, notified the
Air Force as early as May 2001, that it was unable to provide
simulator services as originally agreed and wished to restructure
the contract. Later, the company cited Air Force funding problems
and schedule slips as the basis for claims against the Air Force
and notified the Air Force that its financial situation under the
contract was no longer viable. The Air Force will let the current
F-16 simulator training contract expire in June 2007 and is in the
process of re-competing the contract, which will likely result in
a training gap for pilots and additional costs to the Air Force.
Some Air Force and Army Sites Are Operating and Officials Cite
Improved Training Results
At the locations we visited, officials told us they were pleased
with the quality of the simulator training, particularly when
compared with the level of training they had in the past. Pilots
are routinely surveyed about the training they receive, and
officials told us that, generally, the results have been very
positive. For example, the Director of Operations for the F-16
mission training center at Shaw AFB told us that the simulation
hardware and software are outstanding and that the training
received by young pilots is great. Initial training began under
the Army's Flight School XXI contract in November 2005. While all
planned simulators have not yet been activated, according to
Flight School XXI officials the school is now meeting its training
goal and producing aviators with a proficiency level of two, an
improvement over the old regime.
As of July 2006, the Air Force had 16 training simulator sites
operational, as shown in table 2.
Table 2: Operational Simulator Training Locations and Start Dates
as of July 2006
Source: GAO analysis of Air Force data.
aSome locations have multiple mission training centers.
Air Force Funding Uncertainties Have Resulted in Schedule Slippages
The use of O&M funds under the service contract approach was
intended to overcome the situation the military services had faced
in the past, when internal decisions on funding priorities had
resulted in inadequate procurement funds being made available for
simulators. However, almost from the start of the DMO program,
funding has been less than projected. As a result, schedule
slippages have occurred for many sites compared with original Air
Force requirements set forth in acquisition plans. Army officials
told us that, to date, O&M funding for the Flight School XXI
program has not been reduced. Army officials committed at the
outset to fully fund the contract in accordance with the
originally projected funding profile and, to date, the funding
level has remained stable.
As early as the 2002 budget planning process, Air Force budget
requests did not fully fund planned activations, with a total
difference between estimated requirements and funding of $524
million over the future year defense plan, as shown in table 3.
1 Award-term incentives are similar to award-fee incentives, but the
contractor is rewarded for excellent performance with an extension of the
contract period instead of additional fee.
2 Pub. L. No. 107-107 (2001) codified at 10 U.S.C. 2330.
3 Implemented by Defense Federal Acquisition Regulation Supplement
237.170.
4 GAO, Contract Management: High-Level Attention Needed to Transform DOD
Services Acquisition, GAO-03-935 (Washington, D.C.: Sept. 10, 2003).
Dollars in
millions
Contract award
System date Prime contractor Contract valuea
AWACS Feb. 1999 Plexsys Interface Products,
Inc. $101.0
F-15C Nov. 1997 Boeing Companyb 277.0
F-15Ec Aug. 2003 Boeing Company 287.0
F-16 June 1999 Lockheed Martin Integrated
Systems, Inc. 178.0
Flight School Sept. 2003 Computer Sciences Corporation
XXI - Defense 1108.0
5 Required system elements include instructor stations, pilot cockpits,
manned combat stations, local and long haul networks, and observation
rooms.
6 GAO, 21st Century Challenges: Reexamining the Base of the Federal
Government, GAO-05-325SP (Washington, D.C.: February 2005).
7 GAO, Contract Security Guards: Army's Guard Program Requires Greater
Oversight and Reassessment of Acquisition Approach, GAO-06-284
(Washington, D.C.: Apr. 3, 2006); and Hurricane Katrina: Improving Federal
Contracting Practices in Disaster Recovery Operations, GAO-06-714T
(Washington, D.C.: May 4, 2006).
8 Pub. L. No. 103-355, Title VIII (1994).
9 Pub. L. No. 104-106, Div. D (1996), formerly Federal Acquisition Reform
Act of 1996 and renamed in Treasury, Postal Service and General Government
Appropriations Act, 1997, Pub. L. No. 104-208, Sec. 808 (1996) (contained
in Omnibus Consolidation Appropriations Act, 1997, Pub. L. No. 104-208,
(1996)).
10 Turnkey contracts are those under which the contractor provides all
needed supplies, services, equipment, facilities, etc., to produce the
desired product or service, so that the customer has only to "turn the
key" to avail himself of the result.
Decision to Use Service Approach Not Supported with Thorough Analysis of Costs
and Benefits of Alternatives
11 This directive was cancelled in 2005.
System Location Start date
F-15C Eglin AFB, Florida May 1999
Langley AFB, Virginia May 2000
Elmendorf AFB, Alaska July 2003
Kadena air base (AB), Okinawa Apr. 2005
Lakenheath AB, United Kingdom Jan. 2006
AWACS Tinker AFB, Oklahomaa Dec. 2001
Elmendorf AFB, Alaska Sept. 2003
Tinker AFB, Oklahoma Dec. 2003
Kadena AB, Okinawa May 2005
Tinker AFB, Oklahoma Sept. 2005
F-16 Shaw AFB, South Carolina May 2002
Shaw AFB, South Carolina Jan. 2003
Mountain Home AFB, Idaho Sept. 2002
Mountain Home AFB, Idaho Feb. 2004
Spangdahlem AB, Germany Apr. 2004
Misawa AB, Japan June 2005
Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal
Year 2002 Program Objective Memorandum Funding Levels
Dollars in millions
FY 2002 FY 2003 FY 2004 FY 2005 FY 2006 FY 2007 Total
DMO estimated
funding requirement $97 $169 148 157 133 86 790
Fiscal year 2002
funding approved in
the program
objective memorandum 46 49 43 42 43 43 266
Funding difference -51 -120 -105 -115 -90 -43 -524
Source: GAO analysis of Air Force documents.
An October 2000 Air Force "roadmap" report stated that this funding
scenario would "severely impact the executability of the current
contracted efforts, as well as the entire [DMO] vision." Further, other
Air Force decisions, in reaction to fiscal constraints and programs viewed
as higher priority, have led to additional funding differences. The Air
Combat Command sought to mitigate the impact of these funding differences
by shifting flying hour funds into the DMO program in 2003. Table 4
depicts some key events pertaining to the program's funding impacts and
the command's attempts to secure additional O&M funds.
Table 4: Key Events That Have Affected Funding for the DMO Program
Event Impact
1999: Unexpended flying Air Combat Command no longer had flexibility
hour funds centralized at to use its unexpended flying hour funds for
Air Force headquarters DMO; it now had to request, justify, and
compete for funding.
2001: Budget decision Decisions made by the Air Force to fund other,
results for fiscal years higher-priority programs resulted in a
2002 - 2007 difference of $524 million over the 6-year
period, compared to the DMO plan.
2004: Decision to Office of the Secretary of Defense mandated
accelerate F/A-22 DMO that the schedule for linking the F/A-22 into
compliance the DMO network be accelerated. The $57
million required to do this was unexpectedly
taken from the DMO budget for fiscal years
2006-08, instead of from the F/A-22 program's
budget.
2004: Budget decision Decision was made by Air Force not to fund
results for fiscal year $250 million to facilitate access to the DMO
2006 network by other aircraft (B-1, B-2, B-52,
A-10), although flying hour funds had already
been shifted out of these programs to support
the DMO program.
2004: Budget decision Differences ranging from over $225 million in
results for fiscal years fiscal year 2006 to over $50 million in fiscal
2006-2011 year 2011 as compared to the DMO plan could
not be alleviated, even with a 5-percent
transfer of flying hour funds from other
programs.
2005: Decision to end F-16 The Air Force decision to end the F-16
simulator training contract simulator contract in 2007 will result in
additional costs for re-competition efforts
and upgrades of old training devices.
Source: GAO analysis of Air Force documents.
Largely as a result of these funding uncertainties, many Air Force mission
training centers have been activated significantly behind the planned
schedule contained in acquisition management plans. These schedule
slippages for AWACs, F-15C, and F-16 are shown in tables 5, 6, and 7,
respectively.
Table 5: Comparison of Planned and Actual Site Activations for AWACS
Quarters behind
Scheduled Actual activation scheduled activation
Location activation date date date
Tinker AFB (#1) 1QFY01 1QFY02 4
Tinker AFB (#2) 1QFY03 2QFY03 1
Tinker AFB (#3) No date set 4QFY05 Cannot be determined
Elmendorf AFB 4QFY02 4QFY03 4
Kadena AB 4QFY02 3QFY05 11
Source: GAO analysis of Air Force data.
Table 6: Comparison of Planned and Actual Site Activations for F-15C
Scheduled Actual activation Quarters behind scheduled
Location activation date date activation date
Eglin AFB 2QFY99 2QFY99 0
Langley AFB 3QFY99 3QFY99 0
Kadena AB 1QFY02 3QFY05 14
Lakenheath AB 1QFY02 2QFY06 17
Elmendorf AFB 1QFY03 4QFY03 3
Source: GAO analysis of Air Force data.
Table 7: Comparison of Planned and Actual Site Activations for F-16
Quarters behind
Scheduled Actual scheduled activation
Location activation date activation date date
Shaw AFB (#1) 1QFY02 3QFY02 2
Shaw AFB (#2) 3QFY02 2QFY03 3
Mt. Home AFB (#1) 3QFY02 4QFY02 1
Mt. Home AFB (#2) 4QFY03 2QFY04 2
Misawa AB 2QFY03 3QFY05 9
Spangdahlem AB 3QFY03 3QFY04 4
Source: GAO analysis of Air Force data.
Air Force officials told us that since most of the original dates were
"notional," meaning that they were not firm requirements, but rather were
intended to provide contractors with information about potential mission
training center sites, the timely achievement of the schedules was not
required. However, contractor representatives told us that their proposals
relied upon the planned site activation schedules contained in the
contracts, and delays could directly affect their profitability.
Army's Simulator Activation Schedule Was Also Delayed
The Army has twice rebaselined the activation schedules for the Flight
School XXI simulators-the TH-67 and the advanced aircraft virtual
simulators (AAVS)-as shown in figure 1.
Figure 1: Flight School XXI Schedule Rebaselines
In the original contract, the TH-67 basic training helicopter simulators
were scheduled to begin operation in December 2004, 15 months after
contract award. The Flight School XXI project manager could not provide
documentation to support this time frame and, in fact, told us that the
flight school could not have been ready for students at that time. The
Army subsequently rebaselined the schedule to allow for an 8-month delay.
Similarly, the Army revised the AAVS activation schedule-originally set at
18 months after contract award-to allow a 7-month delay. According to the
project manager, these delays resulted from a protest of the contract
award by a competitor and the contractor's renegotiation with its
subcontractors. The schedule was rebaselined a second time, as shown
above, because the contractor was not able to meet the adjusted schedule.
The Army agreed to the further slippages in exchange for the contractor's
providing two extra terrain databases as consideration. Despite these
schedule changes, the necessary simulators and facilities were ready for
the first flight school class in November 2005, in accordance with the
final revisions to the contract schedule.
F-16 Simulator Training Gap Anticipated
The risk the government faces if a contractor fails to perform as expected
under the service contracts is heightened because the government does not
own anything-the hardware, software, and data rights are owned by the
contractor. In the traditional approach, the government would own the
hardware and any software or data it had acquired rights to. While there
would be no guarantees as to the condition of these items if the
contractor had failed to perform, the government would at least be able to
provide them to the replacement contractor, who could potentially make use
of them under a new contract. The situation the Air Force has faced with
the F-16 simulator contract is illustrative of the potential for not only
a degradation in training, but also increased costs to the government when
contract performance does not occur as planned.
From the outset, the Air Force believed that the F-16 simulator
contractor's cost estimate was low, as it was about $70 million less than
the government's estimate. According to Air Force and contractor
officials, the reason for the low cost estimate was that the contractor
amortized its development costs over all the sites that were planned to be
activated rather than the minimum number that were contractually required.
When schedule delays occurred and the expected sites were not activated,
the contractor reported that it lacked the financial viability to continue
work under the contract. In April 2003, the contractor stopped work toward
making the simulators concurrent with the aircraft, stating that it
considered the tasks beyond the contract scope. Subsequently, it told the
Air Force it was not in its best interest to activate additional training
sites.12
The Air Force will allow the F-16 simulator training contract to expire in
June 2007 because, according to DOD, the contractor failed to earn enough
award-term points to extend the period of performance. The Air Force plans
to re-compete the contract. Two aspects of the original contract, awarding
it as a commercial acquisition and including an award- term provision,
will not be included in the new contract.
12 The contractor claimed that it had incurred $73.3 million in
non-recurring costs and that it had only recovered $31.7 million of these
costs through payment for preparatory services. According to a March 2006
DOD Inspector General report, an improper contract restructure directed by
the Air Force Principal Deputy Assistant Secretary for Acquisition and
Management committed the Air Force to pay the remaining $41.6 million in
claimed non-recurring costs even though the government received minimal
value.
Because of the time needed to re-compete the contract and for the winning
contractor to provide initial training capabilities, the Air Force faces a
potential training gap of over 2 years, during which even the current
degraded level of F-16 simulator training services will not be available
to pilots. In an effort to ensure some level of continued training during
that period, the Air Force plans to award a contract for interim service
capability at three air bases. This interim capability will be available
for block 50 aircraft only.13 For the block 40 aircraft, the Air Combat
Command plans to spend approximately $20 million to refurbish old F-16
unit training devices. These devices are limited in training potential
compared to the current level of simulation.
Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked
The Air Force and the Army are not effectively tracking the return on
their expenditure of taxpayer dollars to acquire simulator training
services. The extent to which the simulators are being used is either not
measured or is measured inconsistently. The government is paying for
activities conducted during the simulator development period but lacks
insight into what it is actually paying for. Finally, award-term
evaluations that were established to encourage excellent contractor
performance do not always measure key acquisition outcomes such as
simulator availability and concurrency, and can result in additional
contract years being awarded for only "satisfactory" performance.
Information on Utilization Rates Is Lacking
The utilization rate is the percentage of available hours the simulators
are actually used. The Army is not tracking the extent to which aviators
are using the contracted service for Flight School XXI simulators, even
though for simulators the Army owns, utilization rates are tracked.
Program officials told us that, because the Army is contracting for
simulator training to be available, there was no need to track the extent
to which the government is using this availability. Without data on
utilization rates, the Army has no basis for determining the extent to
which it is using the services it is buying.
We found that Air Force installations are collecting information on
monthly utilization rates, as provided for in a May 1998 Air Force
instruction.14 However, rates at the locations we examined were often far
less than the hours the government purchased. For the three AWACS mission
training centers at Tinker AFB, for example, we found that, during the
2-year period ending December 2005, monthly utilization rates were
frequently reported at less than 50 percent, as shown in figure 2.
13 The primary mission of F-16 block 50 aircraft is the destruction and
suppression of enemy air defenses. Block 40 aircraft have the primary
mission of filling the air-to surface attack role.
Figure 2: Comparison of Monthly Utilization Rates for AWACS
The Air Force Audit Agency has reported that installations had acquired
excess simulator capacity and unnecessarily consumed O&M funds that could
have been applied to other mission requirements. At Shaw AFB, for example,
the agency found that the Air Force had paid to use the simulator 10 hours
a day, but only used it about 6 hours per day over a 4-month period.15 The
underutilization was attributed to missions being either not scheduled or
cancelled. Deployment requirements and range training were identified as
contributing factors. At Spangdahlem AB, the audit agency reported that
the Air Force had contracted for excess hours of simulator availability to
provide the maximum flexibility for pilot schedules.16 As a result, the
Air Force paid for enough simulator availability to hold 3,952 training
events in fiscal year 2005, even though it needed only 1,982 training
events to meet training requirements.
14 Air Force Instruction 36-2248, Operation and Management of Aircrew
Training Devices, provides that training units calculate the utilization
rate.
15 Air Force Audit Agency, Flight Simulator Utilization, 20th Fighter
Wing, Shaw AFB, SC; F2004-0051-FDM0000 (Mid-Atlantic Area Audit Office,
May 5, 2004).
Our analysis also found that monthly utilization rate calculations are
inconsistent among DMO system sites, even though an Air Force instruction
provides guidance on how to calculate and report utilization data. We
asked six installation quality assurance representatives how they
calculated utilization rates. Four of the six representatives were unaware
of the instruction, telling us that they had not received any guidance for
calculating simulator use. Several different calculation methods are being
used, as described in table 8.
Table 8: Comparison of Monthly Utilization Rate Calculation Methods for
DMO System Simulatorsa
Simulator Calculation method
AWACS Number of hours where some training was given
divided by the number of available training
hours.
F-15C Number of training periods in which one or more
simulators was in use divided by the number of
training periods available.
F-16 (Shaw AFB) Number of system elements used that were
effective divided by the number of possible
system elements.
F-16 (Mountain Home AFB) Number of simulators used divided by the number
of available simulators.
Source: Air Force quality assurance representatives.
aF-15E service not yet available.
Air Combat Command officials told us the reported utilization rates are
used to determine whether or when to activate another training center at a
site. They also said they are using utilization rate information to
determine how many additional "live" flying hours can be moved to the
simulators, in particular to alleviate the burden of high fuel costs for
aircraft. Because of the very different methods being used to calculate
the rates, however, decisions are being made based on non-comparable
information.
16 Air Force Audit Agency, Distributed Mission Operations, 52d Fighter
Wing, Spangdahlem AB, Germany; F2006-0018-FDE000 (European Area Audit
Office, Jan. 25, 2006).
In addition, we found that the Air Force's instruction for calculating
monthly simulator utilization rates could result in overstating the rates,
thus overstating the return on the expenditures made. The instruction
directs that utilization be reported when any or all devices at a given
location are used. Thus, the Air Force can pay to have four simulators
available at a site, use only one of the four during a training period,
and still report that simulator utilization was 100 percent as opposed to
25 percent of the paid availability.
Payments for Tasks during Development Period Are Not Transparent
Under the services approach, contractors commit to major investment at the
front end, with the return on their investment to come from hourly fees
received for providing simulator service. As an additional way to help the
contractor recoup its costs earlier, the government added "preparatory"
tasks during the development period prior to the start of service. These
tasks are defined in the contracts as discrete events, such as site
surveys and training capability assessments, that are ordered and paid for
prior to the start of service. Payments for these tasks provide the
contractor cash flow between contract award and the planned service start
dates and give the government a contractual avenue for contract oversight
prior to receiving services. We found that the Air Force and Army have
little insight into what they are paying for under the preparatory tasks.
Although the invoices reflect only the discrete tasks, such as training
capabilities assessments, the wide range of invoice amounts-from $91,000
to more than $6.5 million for similar tasks-and our discussions with
contractor officials suggest that the government is actually making
milestone payments to the contractors for a portion of their up-front
costs to acquire and develop the simulators.
The original service contract concept for the F-15C, the first simulator
contract awarded, had no provision for the contractor to recoup any costs
during the development period, which usually lasts more than a year.
Figure 3 shows the development period before the start of simulator
services and the original hourly rate structure under the F-15C contract.
Figure 3: Original Rate Structure for F-15C Simulator Service Contracts
This original approach, according to Air Force and contractor officials,
contributed to schedule and certification delays with the F-15C. Air Force
officials told us that they had no contractual avenue to obtain insight
into the contractor's performance during the development period and thus
were not aware that the contractor had encountered delays in obtaining
information from other programs and in determining the complexity of some
simulation elements. As a result, full service was not implemented on
schedule and certification of simulation service was delayed until after
the start of initial service. Further, according to the contractor, it
suffered an unrecoverable loss of income during the high-rate, initial
service period. Subsequently, based on feedback received from industry,
the Air Force changed its approach and incorporated preparatory services
into the F-15C contract and all subsequent DMO system contracts to obtain
more visibility into contractor activities during the development period.
The Army also paid for preparatory services during the development period
of the Flight School XXI contract.
Our analysis of the Air Force's payments for preparatory services found
significantly disparate costs for site surveys and training assessments,
as reflected in tables 9 and 10, respectively.
Table 9: Comparison of Site Survey Costs at Selected DMO System Sites
Dollars in thousands
Systems
Sites AWACS F-15C F-15E F-16
Tinker AFB (#1) $130
Tinker AFB ( #2) 216
Elmendorf AFB 322 $55 $90
Kadena AB 330 67
Lakenheath AB 56 100
Shaw AFB (#1) $625
Shaw AFB (#2) 1,000
Shaw AFB (#3) 1,100
Source: GAO analysis of contractor invoices.
Table 10: Comparison of Training Capability Requirements Assessments and
Training Capability Assessments for DMO Systems
Dollars in thousands
Systems
Task AWACS F-15Cc F-15E F-16
Training capability requirements assessmentsa
Number 1 $122 $2,000 $2,850
Number 2 2,600 4,850
Number 3 200 1,800
Number 4 100
Training capability assessmentsb
Number 1 356 $91 1,644
Number 2 454 722 6,575
Number 3 261 565 3,161
Number 4 289 1,000
Number 5 115
Number 6 150
Source: GAO analysis of contractor invoices.
aTraining capability requirements assessments trace the capabilities of
the simulator to the training task list, which provides descriptions of
training for various mission profiles.
bTraining capability assessments include reviews of system performance
evaluations and verification and validation tests to authorize shipment of
simulator service to designated destinations.
cThe F-15C contractor was not paid for training capability requirements
assessments as preparatory services.
We asked Air Force and Army officials what was specifically included in
these preparatory services and how they determined what they received in
return for payments made. They told us that the contractors determine what
is included and needed for each service at each site.
Three of the four contractors we spoke with agreed that funding for
preparatory tasks helped defray their development costs. They said that,
in effect, they bill for these tasks as milestone payments rather than for
the discrete tasks themselves. Thus, they are able to begin defraying
hardware and software development costs before the start of services.
Officials from the fourth contractor stated that site survey tasks are
standard but that there is some leeway in what is to be done for training
capability assessments and training capability requirements assessments.
With the upcoming re-competition of the F-16 simulator training contract,
the Air Force may pay again for the preparatory service tasks in the new
contract's development period, having already spent nearly $42 million on
these tasks in the initial contract. Air Force officials told us they
cannot assume that potential offerors would make use of the preparatory
work the original contractor has performed.
Award-term Evaluations Do Not Measure Key Acquisition Outcomes
In an effort to measure performance and encourage the contractors to
perform in an efficient and effective manner, both the Air Force and Army
employ award-term incentives. However, while the award-term evaluation
areas include pilot and crew satisfaction, they do not always measure the
key acquisition outcomes of system availability and concurrency with
aircraft upgrades.
While the Air Force does include system availability as an evaluation
area, it is assigned only 25 to 30 percent of the total score. Concurrency
is not included as a separate evaluation area. The Army's evaluation
areas, on the other hand, include concurrency but not system availability.
While the Army requires the tabulation and submission of such data as
operational availability and training service completion rate, these data
are not included in the award-term evaluations.17 In addition, several of
the evaluation areas include assessments of such things as responsiveness
to government requests for cost and pricing data for proposed work not in
the initial contract. We recently recommended that DOD move toward more
outcome-based award-fee criteria that would promote accountability for
acquisition outcomes, rather than include criteria such as responsiveness
to government customers or the quality of proposals submitted.18
17 Operational availability is the total time of effective training
service divided by the total time of scheduled simulation services. The
training service completion rate is the number of completed training
periods divided by the number of required training periods.
Table 11 compares the award-term evaluation areas and the weight given to
each area.
Table 11: Comparison of Award-term Areas of Evaluation and the Weight
Given to Each Area by System
Weight of evaluation area (percent of
total)
Flight School
Evaluation area AWACS F-15C F-16 XXI
Pilot/crew satisfaction 50 50 50 40
System availability 30 25 20
Technical insertion 10 15 10 10
Proposal quality (also called 10 5 15 5
affordability)
Small business participation 5 5
Concurrency 20
Management (includes small business 25
participation)
Total 100 100 100 100
Source: Air Force and Army award-term plans.
Note: F-15E does not include an award-term incentive.
The Air Force and Army both assign the largest weight to "pilot/crew
satisfaction." However, this measure has limitations, particularly when it
is heavily relied on to inform award-term decisions. Air Force officials
told us that it is in the pilots' best interests to assign a high rating
to this factor; otherwise, they could be viewed as not having received
adequate training and could be asked to retake it. Additionally, pilots
are frequently hurried in completing their surveys and dash off check
marks without much consideration. Also, the distinction between the levels
of satisfaction can be blurry. For the Army, for example, if training and
support are adversely impacted for an "extended period," user satisfaction
is to be rated as unsatisfactory. However, if the adverse impact occurs
"infrequently or temporarily," it is considered marginal. Because the
terms are not defined, the Army cannot be certain that pilots are
providing consistent ratings.
18 GAO, Defense Acquisitions: DOD Has Paid Billions in Award and Incentive
Fees Regardless of Acquisition Outcomes, GAO-06-66 (Washington, D.C.: Dec.
19, 2005).
We also found that, under the Air Force's award-term plan, contractors can
earn an additional award-term year for only satisfactory performance
because awarded points are rolled over to the next evaluation period. A
contractor with only satisfactory performance in each of five rating areas
can receive up to 51 points each year; thus, within 2 years, it can
accumulate the 100 points needed for a 1-year contract extension. The F-16
simulator training contractor, for example, which recently notified the
government that it could not continue to perform under the contract,
received overall award-term evaluations of "very good" for the first two
rating periods (May 2002 through July 2003) and "satisfactory" in the
third and fourth periods (July 2003 through January 2005) and earned one
contract year extension. The Army has taken a different approach; under
its award-term plan it is very unlikely that the contractor can be awarded
contract extensions for "satisfactory" performance because rollover is
allowed only when more than 100 points are earned. Thus, the contractor
with only satisfactory performance cannot accumulate enough points for an
additional contract year.
While service is not yet available, the F-15E simulator training contract,
awarded in August 2003, does not include an award-term incentive because,
according to the contracting officer, "it doesn't work." Contractor
officials told us that the subjective nature of the criteria and the
manner in which they are applied negate the award term as a performance
incentive. Both the Air Force and Army indicated that they are moving away
from using award-term incentives on future contracts. The Air Force will
not include such an incentive in its re-competition for the F-16 simulator
training contract because, according to the DMO director, it has not been
found to be a significant motivator to the contractor; experience has
shown that withholding payment for poor service is a much more effective
tool to induce improved performance. In addition, since a recent statutory
provision limits future total contract periods of performance to 10 years,
an award-term provision can no longer be used to implement long-term
arrangements such as those in place for the existing simulator training
contracts.19
We recently reported that DOD has little evidence to support its belief
that award fees improve contractor performance and acquisition outcomes
and, in fact, frequently pays out most of the available award fee to
contractors regardless of their performance outcomes.20 We also found that
DOD contracts frequently included rollover provisions, where unearned
award fee from one evaluation period was shifted to a subsequent
evaluation period or periods, thus providing the contractor an additional
opportunity to earn previously unearned fee. We recommended that DOD issue
guidance on when rollover of award fee is appropriate. A March 2006
guidance on award fee contracts states, among other things, that use of
rollover provisions should be the exception rather than the rule and that
the decision to use rollover provisions should be addressed in the
acquisition strategy, including a rationale as to why a rollover provision
is appropriate.
Conclusion
Because simulator training had lost out in the internal competition for
procurement funds, the Air Force and Army turned to service contracts,
expecting that O&M funds would be made available to meet requirements. In
the case of the Air Force, this expectation has not materialized and
planned site activations have been slowed. In addition, although the Air
Force and Army plan to continue with the service contract approach for
simulator training, neither supported the decision with a thorough
analysis of the costs and benefits of alternative approaches to delivering
the training. Finally, the heightened risks associated with increased
reliance on contractors to deliver simulator training calls for careful
attention to contract management and oversight. Effective and well-managed
incentives for motivating performance are especially important. Better
government visibility into the contractors' activities, such as
preparatory tasks, during the development period is critical so that the
government can understand the basis for what are essentially milestone
payments during that phase. In addition, unless utilization rates are
tracked in a consistent manner, the government will not know whether it is
making the best use of what it is buying.
19 Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005, Pub. L. No.108-375, Sec. 813 (2004) limits the total contract period
for multiyear task and delivery order contracts, as extended pursuant to
options, to 10 years unless the head of the agency determines in writing
that exceptional circumstances necessitate a longer contract period.
20 GAO-06-66 .
Recommendations for Executive Action
To help ensure that the best approach is used to provide the war-fighter
with needed training, we recommend that the Secretary of Defense direct
the Secretaries of the Air Force and Army to conduct a thorough analysis
of the costs and benefits of using service contracts for simulator
training to determine if it is indeed the best approach. The analysis
should proactively address potential risks associated with the service
contract approach and identify the level of simulator training needed to
meet requirements.
To help ensure that the required training is provided to pilots, we
recommend that the Secretary of the Air Force reconcile the funding level
needed for simulator training with the requirements identified in the
evaluation of costs and benefits of the service contract approach and take
steps to allocate funds accordingly.
To help ensure that the incentives motivate contractor performance toward
achieving desired training outcomes, we recommend that the Secretary of
Defense direct the Secretaries of the Air Force and Army to take the
following two actions:
o Determine whether it is in the government's best interest to
retain the award-term incentive under these service contracts.
o If the award-term incentive is retained, take appropriate steps
to improve the approach by reassessing the areas to be rated and
the definitions of performance levels for the various grade
categories. For the Air Force, improvements to the approach should
include a determination as to whether to continue allowing
rollover of award-term points.
To help ensure greater transparency into what the government is
paying for preparatory tasks during the development phase, we
recommend that the Secretary of Defense direct the Secretaries of
the Air Force and Army to take the following two actions:
o Reassess the pricing of any up-front payments made to the
contractors during the development period on future replacement or
restructured contracts.
o If retained, take appropriate measures to (1) create an
appropriate and transparent contract payment mechanism, separate
from the preparatory tasks, if development costs are to be
reimbursed; and (2) increase visibility into the percentage of
upfront development costs contractors are recouping from these
preparatory tasks and development payments.
To help ensure that available simulator training for the
warfighter is used in the most effective and efficient manner, we
recommend that the Secretary of Defense take the following four
actions:
o Direct the Secretaries of the Air Force and Army to determine
whether and how simulator utilization can be increased in order to
maximize use of taxpayer dollars.
o Direct the Secretary of the Army to track and record monthly
utilization rates on Flight School XXI contracted simulator
training in order to have the data necessary to adjust training
requirements and contract provisions, as necessary.
o Direct the Secretary of the Air Force to revise Air Force
Instruction 36-2248, Operation and Management of Aircrew Training
Devices, to ensure that, for the purposes of reporting utilization
rates, the usage of individual training simulators is calculated.
o Direct the Secretary of the Air Force to ensure that all sites
consistently track and report simulator utilization.
Agency Comments and Our Evaluation
In written comments on a draft of this report, DOD concurred with
all but one of our recommendations. DOD partially concurred with
our recommendation that the Army track and record monthly
utilization rates on simulators at Flight School XXI. DOD stated
that the service contract approach requires only that the vendor
meet the programmed student training load. Nevertheless, DOD
stated that the contractor is required to submit utilization data
and that the data are available for use in future adjustments to
the contracting strategy, requirements, or provisions. Our
recommendation was intended to encourage DOD to fully understand
its student training requirements and to collect the information
to decide whether it needs to adjust requirements or contract
provisions regarding simulator availability. Whether the
utilization rates pertain to individual simulators or the student
training load as a whole, we believe that the Army needs to know
the extent to which it is actually using the simulator
availability it is buying. DOD also offered two corrections to
information in the draft, and we made changes as appropriate.
DOD's comments are included in their entirety in appendix II.
We will send copies of this report to the Secretaries of Defense,
the Air Force, and the Army; appropriate congressional committees;
and other interested parties. We make copies available to others
upon request. In addition, the report will be available at no
charge on the GAO Web site at http://www.gao.gov .
If you or your staff has questions concerning this report, please
contact me at (202) 512-4841 or by e-mail at [email protected] .
Contact points for our Offices of Congressional Relations and
Public Affairs may be found on the last page of this report.
Lisa Shames, Acting Director Acquisition and Sourcing Management
Appendix I: Scope and Methodology
To determine which factors led the Air Force and Army to acquire
simulator training as a service contract using operation and
maintenance funds, we analyzed historical documents such as
acquisition plans, briefings, and decision memorandums. For the
Air Force, we interviewed Air Force management, including
officials at the Office of the Assistant Secretary of the Air
Force, Acquisition; Aeronautical Systems Center (responsible for
contracting the simulator training services) and the Air Combat
Command (funds and uses the simulator training). For the Army, we
interviewed officials at the Office of the Assistant Secretary of
the Army for Acquisition, Logistics, and Technology; and Army
officials responsible for managing the Army's Flight School XXI
initiative, including officials of the Program Executive Office
for Simulation, Training and Instrumentation. We visited Langley
Air Force Base, Virginia, to observe F-15C simulator training;
Shaw Air Force Base, South Carolina, to observe F-16 simulator
training; and Fort Rucker, Alabama, to observe the Flight School
XXI helicopter simulator training. Additionally, to evaluate
whether the military services adequately justified the new service
contract approach, we reviewed the Office of Management and
Budget's Circular A-11, Appendix B, "Budgetary Treatment of
Lease-Purchases and Leases of Capital Assets," and Air Force and
Army regulations and guidance regarding business case analyses. We
also drew from our prior reviews of Department of Defense systems,
in particular our recent review of the Army's Future Combat
System.1
To assess whether the new approach has resulted in the planned
number of simulator training sites being activated, we evaluated
contract documents and information provided by the Air Combat
Command and Aeronautical Systems Center to compare planned to
actual schedule activations. We gathered and analyzed budget data
related to program schedules and interviewed program officials. We
analyzed contract documents and other program documents from
Flight School XXI and discussed the schedule rebaselining with
Army officials. We analyzed the Air Force's request for proposals
for the F-16 simulator training contract re-competition to
determine whether key differences in the acquisition approach were
incorporated.
To determine if the Air Force and Army are effectively tracking
the return on their expenditure of taxpayer dollars, we analyzed
simulator utilization data and military service guidance on
utilization rates; analyzed contractor performance measurements,
annual evaluations, and award-term plans for the simulator
training contracts; and compared preparatory service costs charged
to the government under each of the four Air Force contracts and
the Army contract. We also interviewed contractor representatives
and government officials.
1 GAO, Defense Acquisitions: Improved Business Case Is Needed for Future
Combat System's Successful Outcome, GAO-06-367 (Washington, D.C.: Mar.
14, 2006).
Appendix II: Comments from the Department of Defense
Appendix III: Comparison of Contractor Requirements under Commercial and
Non-commercial Acquisition Procedures
The following table shows differences, as set forth in the Federal
Acquisition Regulation (FAR), for contractor requirements under commercial
versus non-commercial acquisition procedures.
Under noncommercial Under commercial
Requirement procedures procedures
Type of contract A wide selection of Limited contract types
contract types is are authorized.
available in order to Agencies shall use
provide flexibility. (FAR firm-fixed-price
16.101(a)) contracts or
fixed-price contracts
with economic price
adjustment. These
contract types may be
used in conjunction
with an award fee and
performance or
delivery incentives
when the award fee or
incentive is based
solely on factors
other than cost. (FAR
12.207)
To implement the
Services Acquisition
Reform Act of 2003
(contained in Section
1432 of the National
Defense Authorization
Act for Fiscal Year
2004, Pub. L. No.
108-136 (2003)), a
proposed amendment to
FAR would expressly
authorize the use of
time-and-materials and
labor-hour contracts
for certain categories
of commercial services
under specified
conditions. (FAR Case
2003-027, 70 Federal
Register 56318, Sept.
26, 2005.)
Inspection and testing Government has right to Contracts for
inspect and test. (FAR commercial items shall
46.102 & 46.202-3) rely on contractors'
existing quality
assurance systems as a
substitute for
Government inspection
and testing before
tender for acceptance
unless customary
market practices for
the commercial item
being acquired include
in-process inspection.
Any in-process
inspection by the
Government shall be
conducted in a manner
consistent with
commercial practice.
(FAR 12.208)
Determination of price Price must be determined While price
reasonableness fair and reasonable reasonableness must be
through various proposal established, the
analysis techniques. (FAR contracting officer
15.404-1) should be aware of
customary commercial
terms and conditions
when pricing
commercial items.
Commercial item prices
are affected by
factors that include,
but are not limited
to, speed of delivery,
length and extent of
warranty, limitations
of seller's liability,
quantities ordered,
length of the
performance period,
and specific
performance
requirements. (FAR
12.209)
Cost or pricing data Required for contract Commercial items are
award and modifications exempt (FAR
(Requirements based on unless applicable 15.403-1(b)(3) and
Truth in Negotiations exception, such as (c)(3))
Act, 10 U.S.C. 2306a; 41 adequate competition or
USC 254b) prices agreed upon are
based on prices set by law
or regulation. Threshold
for application is
$550,000. (FAR 15.403-1
and -4)
Contract financing The contracting officer For purchases of
must consider the commercial items,
following order of financing of the
preference when a contract is normally
contractor requests the contractor's
contract financing., (a) responsibility.
Private financing without (32.202-1) However,
Government guarantee.(b) customary market
Customary contract practice for some
financing. (c) Loan commercial items may
guarantees. (d) Unusual include buyer contract
contract financing. (e) financing. In these
advance payments. (FAR circumstances, the
32.106) contracting officer
may offer Government
financing in
accordance with the
policies and
procedures in Part 32.
(FAR 12.210)
However, government Government financing
financing provided only to of commercial
extent actually needed for purchases is expected
prompt and efficient to be different from
performance, considering that used for
availability of private non-commercial
financing and probable purchases. While the
impact on working capital contracting officer
of predelivery may adapt
expenditures and non-commercial
productions lead-times. techniques and
(FAR 32.104) procedures for use in
implementing
commercial contract
financing
arrangements, the
contracting officer
must have a full
understanding of
effects of the
differing contract
environments and of
what is needed to
protect the interests
of the Government in
commercial contract
financing. (FAR
32.202-1(c))
Types of payments for
commercial item
purchases. (FAR
32.202-2)
1. Commercial
advance payment:
payment made before
any performance of
work (not to exceed
15 percent of
contract price)
2. Commercial
interim payment:
payment made after
some, but not all,
work has been
performed
3. Delivery
payment: payment
made for accepted
supplies or
services, including
partial deliveries
(FAR 32.001)
Technical data The Government may acquire Generally, the
technical data and rights Government shall
in technical data for acquire only the
multiple purposes. technical data and the
Agencies shall strike a rights in that data
balance between the customarily provided
government's need and the to the public with a
contractor's legitimate commercial item or
proprietary interest. (FAR process. The
27.4) contracting officer
shall presume that
data delivered under a
contract for
commercial items was
developed exclusively
at private expense.
When a contract for
commercial items
requires delivery of
technical data, the
contracting officer
shall include
appropriate provisions
and clauses
delineating the rights
in the technical data
in the contract. (FAR
12.211)
Computer The Government may acquire Commercial computer
software/documentation computer software or commercial
software/documentation for computer software
multiple purposes. documentation shall be
Agencies shall strike a acquired under
balance between the licenses customarily
government's need and the provided to the public
contractor's legitimate to the extent such
proprietary interest. (FAR licenses are
27.402) consistent with
federal law and
otherwise satisfy the
government's needs.
Generally, offerors
and contractors shall
not be required to-
1. Furnish
technical
information related
to commercial
computer software
or commercial
computer software
documentation that
is not customarily
provided to the
public; or
2. Relinquish to,
or otherwise
provide, the
Government rights
to use, modify,
reproduce, release,
perform, display,
or disclose
commercial computer
software or
commercial computer
software
documentation
except as mutually
agreed to by the
parties.
(FAR 12.212(a))
Cost accounting Compliance generally Cost Accounting
standards required for contractors Standards do not apply
in connection with to contracts for
41 U.S.C. 422 negotiated contracts in acquisition of
excess of $500,000. commercial items when
they are
Contractors must disclose firm-fixed-price or
and consistently follow fixed-price with
their cost accounting economic price
practices. (FAR 30.101) adjustment. (FAR
12.214)
Preaward Survey In determining whether a If the contemplated
potential awardee is a contract will involve
responsible contractor, the acquisition of
per criteria in FAR commercial items, the
9.104-1, contracting contracting officer
officers may require a should not request a
preaward survey when the preaward survey unless
information on hand or circumstances justify
readily available is not its cost. (FAR
sufficient to make such a 9.106-1(a))
determination. (FAR
9.106-1)
Audit Rights When contracting by Commercial item
negotiation, the contracts exempted.
contracting officer shall (FAR
insert the clause at FAR 15.209(b)(1)(iii))
52.215-2, Audit and
Records-Negotiation in
solicitations and
contracts which allows
contracting officer
examination of costs when
cost or pricing data is
required or for
cost-reimbursement,
incentive,
time-and-materials,
labor-hour, or price
redeterminable contracts.
(FAR 15.209(b) and
52.215-2)
Applicability of certain
laws:
Walsh-Healy Public On contracts for supplies Not applicable. (FAR
Contracts Act over $10,000, contractors 12.503(a))
must adhere to provisions
pertaining to minimum
wages, maximum hours,
child labor, convict
labor, safe/sanitary
working conditions. (FAR
22.602)
Contingent fees Contractor must warrant Not applicable. (FAR
that it has not employed 12.503(a))
or retained anyone, on a
contingent fee basis, to
obtain this contract.
(FAR 3.404, 52.203-5)
Drug-Free Workplace Act Contractor must agree that Not applicable. (FAR
of 1988 it will provide a 12.503(a))
drug-free workplace (FAR
23.504(a))
Vietnam Era Veterans' Contractor must report on Law's limitation on
Readjustment Assistance its affirmative actions to use of appropriated
Act of 1972 employ and advance covered funds for contracts
veterans (FAR 22.1302(a)) with entities not
meeting veterans
employment reporting
requirements is not
applicable. (FAR
12.503(a))
Trafficking Victims Contracts for services Not applicable. (FAR
Protection must prohibit contractor 12.503(a))
Reauthorization Act of activities regarding, and
2003 require contractor
policies to combat, severe
forms of trafficking in
persons, the procurement
of commercial sex acts,
and use of forced labor.
(FAR 22.1705)
Contract Work Hours and Contract clause required Requirements for a
Safety Standards Act providing that contractors certificate and
employing laborers or contract clause
mechanics are required to related to the Act are
compensate them for not applicable. (FAR
overtime. (FAR 52.222-4) 12.503(b))
Anti-Kickback Act of Contract clause requires Requirements for a
1986 prime contractors to (1) clause and certain
have in place and follow other requirements
reasonable procedures related to the Act are
designed to prevent and not applicable. (FAR
detect violations of the 12.503(b))
Act; and (2) cooperate
fully with any Federal
agency investigating a
possible violation of the
Act. (FAR 3.502-2(i))
International Air Contracts must include Requirement for a
Transportation Fair clause requiring use of clause related to the
Competitive Practices U.S-Flag Air Carriers by Act is not applicable.
Act of 1974 (Fly America government contractors (FAR 12.503(b))
Act) when available (FAR
47.405)
Prohibition on Limiting Contracts must include Contractors may
Subcontractor Direct clause precluding restrict
Sales to the United contractors from subcontractors' sales
States restricting direct to the Government, as
subcontractor sales to the long as the Government
Government. (FAR 3.503-2 is treated no
and 52.203-6(a)) differently than other
prospective purchaser.
(FAR 52.203-6,
Alternate I)
Selected contract
administration
provisions:
Changes Generally, contracting Changes may be made
officer permitted to make only by written
unilateral changes within agreement of the
the scope of the contract parties (bilateral).
and to require continued (FAR 12.301(b)(3);
contractor performance of 52.212-4(c))
the contract as changed.
(FAR 43.201)
Termination for the Generally, termination Termination costs
convenience of the costs for fixed-price limited to percentage
government contracts limited to total of contract price
contract price less reflecting percentage
payments made or to be of work performed
made under contract plus prior to termination
reasonable costs incurred plus reasonable
in performance of work charges resulting from
terminated, to include termination. For
fair and reasonable payments thereunder,
profit, and reasonable contractor not
settlement costs. Cost required to comply
principles and procedures with cost accounting
of FAR Part 31 apply to standards or contract
costs. (FAR 49.502(b); cost principles in FAR
52.249-2) Part 31. (FAR
12.301(b)(3);
52.212-4(l))
Source: GAO analysis of Federal Acquisition Regulation requirements.
Appendix IV: Potential Duration of Air Force and Army Simulation Contracts
Additional years
Base performance possible via award Potential total years of
System period term contract performance
Air Force:
F-15C Nov. 1997 to Nov.
2005 (8 years) 7 years 15 years
AWACS Feb.1999 to Feb.
2006 (7 years) 8 years 15 years
F-16 June 1999 to June
2006 (7 years) 8 years 15 years
F-15E Aug. 2003 to July
2016 (13 years) N/A 13 years
Army:
Flight Sept. 2003 to Mar.
School XXI 2015 (11.5 years) 8 years 19.5 years
Source: GAO analysis of contract data.
Appendix V: GAO Contact and Staff Acknowledgments
GAO contact
Lisa Shames (202) 512-4841 or [email protected]
Acknowledgments
In addition to the individual named above, Michele Mackin, Assistant
Director; Marie Ahearn; Christine Bonham; Gary Delaney; Carlos Diz;
Benjamin Federlein; Victoria Klepacz; and Sanford Reigle made key
contributions to this report.
(120492)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [email protected] (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149 Washington,
D.C. 20548
www.gao.gov/cgi-bin/getrpt? GAO-06-830 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Lisa Shames at (202) 512-4841 or
[email protected].
Highlights of GAO-06-830 , a report to the Subcommittee on Readiness,
Committee on Armed Services, House of Representatives
September 2006
CONTRACT MANAGEMENT
Service Contract Approach to Aircraft Simulator Training Has Room for
Improvement
The Air Force has turned to service contracts for the F-15C, F-16,
Airborne Warning and Control System, and F-15E, and the Army has done the
same for helicopter simulator training at its Flight School XXI. The
contractors own, operate, and maintain the simulator hardware and
software. The military services rely on industry to capitalize the
required up-front investment, with the understanding that the contractors
will amortize this investment by selling training services by the hour.
GAO was asked to address (1) the factors that led the Air Force and Army
to acquire simulator training as a service and whether the decision to use
this approach was adequately supported; (2) whether implementation of the
approach has resulted in the planned number of simulator training sites
being activated; and (3) whether the Air Force and Army are effectively
tracking the return on their expenditure of taxpayer dollars.
GAO makes recommendations to the Secretary of Defense intended to improve
management and oversight of these service contracts to help ensure that
the best approach is used to provide the war-fighter with needed training.
In written comments on a draft of this report, DOD concurred with all but
one of the recommendations, only partially concurring with one pertaining
to the Army's simulator utilization rates. GAO continues to believe that
the Army needs to track the extent to which it is using simulator
availability.
The Air Force and Army turned to service contracts for simulator training
primarily because efforts to modernize existing simulator hardware and
software had lost out in the competition for procurement funds. As a
result, the simulators were becoming increasingly obsolete. Buying
training as a service meant that operation and maintenance (O&M) funds
could be used instead of procurement funds. Shifting the responsibility
for simulator ownership, operation, and maintenance from the government to
the contractor was thought to more quickly enable simulator upgrades to
match the changing configurations of aircraft. However, the decision to
take a service contract approach was not supported by a thorough analysis
of the costs and benefits as compared to other alternatives, despite a
Department of Defense directive that provided for such an analysis.
While Air Force and Army officials told GAO the new simulators are
significant improvements over the previous ones, the expected number of
Air Force training sites have not been activated. For the Air Force, O&M
funds have not been allocated at the anticipated levels, leading to
schedule slippages. The F-16 simulator contractor cited the funding
problems and subsequent schedule slippages as the basis for notifying the
Air Force that its situation under the contract was no longer financially
viable. The Air Force is in the process of re-competing the F-16 training
contract, which will likely result in a training gap for pilots-possibly
over 2 years-and additional costs to the Air Force. The start date of the
Army's flight simulator training was rebaselined twice, but Army officials
told us that adequate training was in place for the flight school
participants.
The return on expenditure of taxpayer dollars is not being effectively
tracked in three key ways:
o Air Force utilization of simulator training frequently falls
well below the hours for which the government is paying. The Army
is not collecting data on utilization rates at all.
o The government has little insight into what it is paying for
during the development period before training is activated, which
can take more than a year. While invoices for preparatory efforts
reflect only discrete tasks such as training capabilities
assessments, the wide range of invoice amounts and GAO's
discussions with contractor representatives suggest that the
government is actually making milestone payments to the
contractors for a portion of their up-front costs to acquire and
develop the simulators.
o Most of the contracts contain award-term provisions, where the
contractors can earn an extension of the contract period for good
performance. GAO found that the award-term evaluation factors do
not always measure key acquisition outcomes such as simulator
availability and concurrency with aircraft upgrades.
*** End of document. ***