Contract Management: Service Contract Approach to Aircraft	 
Simulator Training Has Room for Improvement (22-SEP-06, 	 
GAO-06-830).							 
                                                                 
The Air Force has turned to service contracts for the F-15C,	 
F-16, Airborne Warning and Control System, and F-15E, and the	 
Army has done the same for helicopter simulator training at its  
Flight School XXI. The contractors own, operate, and maintain the
simulator hardware and software. The military services rely on	 
industry to capitalize the required up-front investment, with the
understanding that the contractors will amortize this investment 
by selling training services by the hour. GAO was asked to	 
address (1) the factors that led the Air Force and Army to	 
acquire simulator training as a service and whether the decision 
to use this approach was adequately supported; (2) whether	 
implementation of the approach has resulted in the planned number
of simulator training sites being activated; and (3) whether the 
Air Force and Army are effectively tracking the return on their  
expenditure of taxpayer dollars. GAO makes recommendations to the
Secretary of Defense intended to improve management and oversight
of these service contracts to help ensure that the best approach 
is used to provide the war-fighter with needed training. In	 
written comments on a draft of this report, DOD concurred with	 
all but one of the recommendations, only partially concurring	 
with one pertaining to the Army's simulator utilization rates.	 
GAO continues to believe that the Army needs to track the extent 
to which it is using simulator availability.			 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-06-830 					        
    ACCNO:   A61306						        
  TITLE:     Contract Management: Service Contract Approach to	      
Aircraft Simulator Training Has Room for Improvement		 
     DATE:   09/22/2006 
  SUBJECT:   Cost analysis					 
	     Defense procurement				 
	     Department of Defense contractors			 
	     Federal funds					 
	     Financial analysis 				 
	     Flight training					 
	     Funds management					 
	     Military training					 
	     Policy evaluation					 
	     Schedule slippages 				 
	     Service contracts					 
	     Training utilization				 
	     Airborne Warning and Control System		 
	     AWACS						 
	     Eagle Aircraft					 
	     F-15C Aircraft					 
	     F-15E Aircraft					 
	     F-16 Aircraft					 
	     Fighting Falcon Aircraft				 
	     Strike Eagle Aircraft				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-06-830

     

     *  
     * Results in Brief
     * Background
          * Contract Information
          * Acquisition Reforms Shaped Contract Structures
          * Key Players
     * Service Contract Approach Had Perceived Benefits but Was Not
          * Simulators Had Lost Out in Competition for Procurement Funds
          * Service Contracts Intended to Yield More Concurrent Simulato
          * Decision to Use Service Approach Not Supported with Thorough
     * While Training Has Improved, the Expected Number of Simulato
          * Some Air Force and Army Sites Are Operating and Officials Ci
          * Air Force Funding Uncertainties Have Resulted in Schedule Sl
          * Army's Simulator Activation Schedule Was Also Delayed
          * F-16 Simulator Training Gap Anticipated
     * Return on the Expenditure of Taxpayer Dollars Is Not Effecti
          * Information on Utilization Rates Is Lacking
          * Payments for Tasks during Development Period Are Not Transpa
          * Award-term Evaluations Do Not Measure Key Acquisition Outcom
     * Conclusion
     * Recommendations for Executive Action
     * Agency Comments and Our Evaluation
     * Appendix I: Scope and Methodology
     * Appendix II: Comments from the Department of Defense
     * Appendix III: Comparison of Contractor Requirements under Co
     * Appendix IV: Potential Duration of Air Force and Army Simula
     * Appendix V: GAO Contact and Staff Acknowledgments
          * Order by Mail or Phone

Report to the Subcommittee on Readiness, Committee on Armed Services, U.S.
House of Representatives

United States Government Accountability Office

GAO

September 2006

CONTRACT MANAGEMENT

Service Contract Approach to Aircraft Simulator Training Has Room for
Improvement

GAO-06-830

Contents

Letter 1

Results in Brief 2
Background 4
Service Contract Approach Had Perceived Benefits but Was Not Adequately
Supported 9
While Training Has Improved, the Expected Number of Simulator Sites Has
Not Been Activated 12
Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked
19
Conclusion 28
Recommendations for Executive Action 29
Agency Comments and Our Evaluation 30
Appendix I Scope and Methodology 32
Appendix II Comments from the Department of Defense 34
Appendix III Comparison of Contractor Requirements under Commercial and
Non-commercial Acquisition Procedures 38
Appendix IV Potential Duration of Air Force and Army Simulation Contracts
43
Appendix V GAO Contact and Staff Acknowledgments 44

Tables

Table 1: Air Force and Army Simulator Training Contracts 6
Table 2: Operational Simulator Training Locations and Start Dates as of
July 2006 13
Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal
Year 2002 Program Objective Memorandum Funding Levels 14
Table 4: Key Events That Have Affected Funding for the DMO Program 15
Table 5: Comparison of Planned and Actual Site Activations for AWACS 15
Table 6: Comparison of Planned and Actual Site Activations for F-15C 16
Table 7: Comparison of Planned and Actual Site Activations for F-16 16
Table 8: Comparison of Monthly Utilization Rate Calculation Methods for
DMO System Simulatorsa 21
Table 9: Comparison of Site Survey Costs at Selected DMO System Sites 24
Table 10: Comparison of Training Capability Requirements Assessments and
Training Capability Assessments for DMO Systems 24
Table 11: Comparison of Award-term Areas of Evaluation and the Weight
Given to Each Area by System 26

Figures

Figure 1: Flight School XXI Schedule Rebaselines 17
Figure 2: Comparison of Monthly Utilization Rates for AWACS 20
Figure 3: Original Rate Structure for F-15C Simulator Service Contracts 23

Abbreviations

AAVS Advanced Aircraft Virtual Simulators AB air base AFB Air Force base
AWACS Airborne Warning and Control System DMO Distributed Mission
Operations DOD Department of Defense FAR Federal Acquisition Regulation
O&M operation and maintenance OMB Office of Management and Budget TH
training helicopter

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

September 22, 2006 September 22, 2006

The Honorable Joel Hefley Chairman The Honorable Solomon P. Ortiz Ranking
Minority Member Subcommittee on Readiness Committee on Armed Services
House of Representatives The Honorable Joel Hefley Chairman The Honorable
Solomon P. Ortiz Ranking Minority Member Subcommittee on Readiness
Committee on Armed Services House of Representatives

In the late 1990s, the Air Force embarked upon an innovative strategy for
acquiring flight simulator training for several of its aircraft platforms.
Faced with increasingly obsolete simulators and growing competition for
procurement dollars, the Air Force decided to buy simulator services from
contractors-using operation and maintenance (O&M) funds-rather than
continue the practice of having the government own and operate the
simulators. This strategy was the beginning of an approach to
war-fighting, known as Distributed Mission Operations (DMO), in which
high-fidelity simulators would be used for training across dispersed
locations. The initial platforms included in the DMO program were the
F-15C, F-16, Airborne Warning and Control System (AWACS), and F-15E. In
2003, the Army followed the Air Force lead and awarded a service contract
to obtain simulator training for the helicopters in its Flight School XXI
program. The estimated value of the Air Force and Army contracts is nearly
$2 billion. A key aspect of the service contract approach is the increased
reliance on contractors to keep the simulators concurrent with aircraft
upgrades and ensure they are available for use. In the late 1990s, the Air
Force embarked upon an innovative strategy for acquiring flight simulator
training for several of its aircraft platforms. Faced with increasingly
obsolete simulators and growing competition for procurement dollars, the
Air Force decided to buy simulator services from contractors-using
operation and maintenance (O&M) funds-rather than continue the practice of
having the government own and operate the simulators. This strategy was
the beginning of an approach to war-fighting, known as Distributed Mission
Operations (DMO), in which high-fidelity simulators would be used for
training across dispersed locations. The initial platforms included in the
DMO program were the F-15C, F-16, Airborne Warning and Control System
(AWACS), and F-15E. In 2003, the Army followed the Air Force lead and
awarded a service contract to obtain simulator training for the
helicopters in its Flight School XXI program. The estimated value of the
Air Force and Army contracts is nearly $2 billion. A key aspect of the
service contract approach is the increased reliance on contractors to keep
the simulators concurrent with aircraft upgrades and ensure they are
available for use.

At your request, we addressed (1) the factors that led the Air Force and
Army to acquire simulator training as a service and whether the decision
to use this approach was adequately supported, (2) whether implementation
of this approach has resulted in the planned number of simulator training
sites being activated, and (3) whether the Air Force and Army are
effectively tracking the return on their expenditure of taxpayer dollars.
At your request, we addressed (1) the factors that led the Air Force and
Army to acquire simulator training as a service and whether the decision
to use this approach was adequately supported, (2) whether implementation
of this approach has resulted in the planned number of simulator training
sites being activated, and (3) whether the Air Force and Army are
effectively tracking the return on their expenditure of taxpayer dollars.

To address these objectives, we interviewed Air Force and Army officials,
as well as contractor representatives; performed a detailed analysis of
the Air Force and Army service contracts; reviewed contractor performance
measurements, annual evaluations, and payment invoices; and analyzed
historical documents such as acquisition plans, briefings, and decision
memorandums. We visited Langley Air Force Base (AFB), Virginia, to view To
address these objectives, we interviewed Air Force and Army officials, as
well as contractor representatives; performed a detailed analysis of the
Air Force and Army service contracts; reviewed contractor performance
measurements, annual evaluations, and payment invoices; and analyzed
historical documents such as acquisition plans, briefings, and decision
memorandums. We visited Langley Air Force Base (AFB), Virginia, to view
F-15C simulator training; Shaw AFB, South Carolina, to view F-16 training;
and the U. S. Army Aviation Center, Fort Rucker, Alabama to view the use
of Flight School XXI training helicopters (TH), the TH-67 Virtual
Simulators and the Blackhawk and Chinook simulators. At each of these
installations, we met with military officials and the contractor officials
responsible for the training simulators. See appendix I for additional
details on our scope and methodology. We performed our review from
September 2005 to July 2006 in accordance with generally accepted
government auditing standards.

                                Results in Brief

The Air Force and Army turned to service contracts for simulator training
primarily because efforts to modernize existing simulator hardware and
software had lost out in the competition for available procurement funds.
Buying simulator training as a service meant that O&M funds could be used
to fund the requirement. Shifting the responsibility for simulator
ownership, operation, and maintenance from the government to the
contractor was thought to provide quicker state-of-the-art pilot and
aircrew training capabilities and allow for automatic simulator upgrades
to match the changing configurations of aircraft. However, the decision to
use service contracts was not supported by a thorough analysis of the
costs and benefits of this approach versus alternative approaches, despite
a Department of Defense (DOD) directive on training that provides for an
evaluation of the benefits and trade-offs of potential alternative
training solutions. While the Air Force and Army provided us with
briefings that showed the decision-making process, the information does
not provide adequate rationale for why certain alternatives were
eliminated from consideration. A more in-depth analysis could have
proactively addressed and possibly mitigated the risks associated with
this approach, where if a contractor fails to perform, the government is
left without simulator hardware, software, or the anticipated level of
training services.

While Air Force and Army officials told us the new simulators are
significant improvements over what they had previously, the expected
number of Air Force simulator training sites has not been activated. For
the Air Force, O&M funds have not been allocated to the contracts at the
anticipated levels, leading to schedule slippages and fewer simulator
sites in place than planned. Efforts to add funds, for example by shifting
flying hour funds into the DMO program in 2003, have not been sufficient
to regain the schedule and activate the number of sites anticipated.
Further, the F-16 simulator training contractor, citing the funding
problems and subsequent schedule slips, notified the Air Force in March
2004 that its situation was no longer financially viable under the
contract and that it could not continue to provide simulator training as
originally agreed. The Air Force will allow the contract to expire in June
2007. It is re-competing the F-16 training contract, which will likely
result in a training gap for pilots-possibly over 2 years-and additional
costs to the Air Force. Two aspects of the original contract, awarding it
as a commercial acquisition and including an award-term provision,1 will
not be included in the new contract.

The return on expenditure of taxpayer dollars is not being effectively
tracked in three key ways:

           o  Air Force utilization of simulator training frequently falls
           well below the hours for which the government is paying. In
           addition, utilization rates at different sites are reported based
           on very different criteria, leading to decisions to activate new
           training sites based on noncomparable information. The Army is not
           collecting data on utilization rates at all.

           o  The government has little insight into what it is paying for
           during the development period before training is activated, which
           can take more than a year. Our analysis of invoices and
           discussions with contractor officials suggest that the government
           is, in effect, funding the private sector for a portion of its
           up-front costs to acquire and develop the simulators, although the
           invoices merely reflect discrete tasks such as site surveys.

           o  Award-term evaluations are being used to measure performance,
           but the evaluation factors do not always measure key acquisition
           outcomes such as simulator availability and concurrency. In
           addition, the ability to "roll over" positive points on Air Force
           contracts can result in the contractor being awarded an additional
           year for only satisfactory performance-a scenario that is highly
           unlikely under the Army's award-term plan. Air Force, Army, and
           some contractor officials indicated that award-term provisions may
           not be an effective incentive for improved performance.

           We make recommendations in this report to DOD on actions that can
           be taken to help ensure that the best approach is used to provide
           the war-fighter with needed training, that the incentives motivate
           contractor performance toward achieving desired training outcomes,
           and that available simulator training is used in the most
           effective and efficient manner. In written comments on a draft of
           this report, DOD concurred with all but one of our
           recommendations. DOD partially concurred with our recommendation
           that the Army track and record monthly utilization rates on
           simulators at Flight School XXI. Nevertheless, DOD stated that the
           contractor is required to submit utilization data and that the
           data are available. Our recommendation was intended to encourage
           DOD to fully understand its student training requirements and to
           collect the information to decide whether it needs to adjust those
           requirements or contract provisions regarding simulator
           availability. Whether the utilization rates pertain to individual
           simulators or the student training load as a whole, we believe
           that the Army needs to know the extent to which it is using the
           simulator availability it is buying. DOD also offered two
           corrections to information in our draft report; we made changes as
           appropriate. DOD's comments are included in their entirety in
           appendix II.

           Background
			  
			  Air Force and Army cite the increasingly complex training
           requirements needed to prepare for the ever more lethal
           battlefield environment as a factor that has led to greater
           reliance on flight simulators. A flight simulator is a system that
           tries to realistically replicate, or simulate, the experience of
           flying an aircraft. Flight simulators range from video games to
           full-sized cockpit replicas mounted on hydraulic (or
           electromechanical) actuators and controlled by state-of-the-art
           computer technology. According to Air Force and Army officials,
           aircraft simulators are a cost-effective way of helping to develop
           and refine operational flight skills. Simulators can facilitate
           training that might be impractical or unsafe if done with actual
           systems and allow for concentrated pilot practice in selected
           normal and emergency actions. Simulators also can train operators
           and maintainers to diagnose and address possible equipment faults,
           and enhance proficiency despite shortages of equipment, space,
           ranges, or time.

           In the late 1990s, the Air Force and Army were faced with
           increasingly obsolete simulators and the need to quickly acquire
           up-to-date pilot and aircrew training. In 1997, the then-Commander
           of the Air Force's Air Combat Command proposed an innovative
           approach of buying training as a service, under which the
           contractors would own, operate, and maintain the simulator
           hardware and software. The simulator service contracts are one
           component of a much broader effort, now known as the DMO program.
           The DMO goal is to provide state-of-the-art simulator training
           on-demand at the location of the trainee, with the ultimate vision
           of networking different sites together to create more realistic
           flying scenarios. Plans call for each fighter unit eventually to
           be equipped with high-fidelity simulators. As of the fiscal year
           2002 budget, the DMO program was formalized in the Air Force
           budget with the assignment of a program element line item that
           combined previous program elements for the various simulator
           systems. For fiscal year 2006, over $200 million was budgeted for
           the program.

           In the early 2000s, Army use of rotary-wing aviation simulation
           training was limited because the simulators being used were
           grossly obsolete and based on late 1970s' technology. To revamp
           its helicopter training, the Army in late 2001 began the Flight
           School XXI program. Following the Air Force's lead, the Army
           decided to acquire up-to-date simulator training using a service
           contract.

           Congress and the Office of Management and Budget (OMB) have
           recently addressed the growing level of procurement of services.
           For example, Congress included provisions in Section 801 of the
           National Defense Authorization Act for Fiscal Year 20022 designed
           to improve management and oversight of procurement of services. To
           ensure that DOD acquires services by means that are in its best
           interest and managed in compliance with applicable statutes,
           regulations, directives, and other requirements, the Act required
           DOD to establish a service acquisition management structure,
           comparable to the management structure that applies to the
           procurement of products.3 In September 2003, we reported that DOD
           and the military services had a management structure in place for
           reviewing individual services acquisitions valued at $500 million
           or more, but that approach did not provide a departmentwide
           assessment of how spending for services could be more effective.4
           Also, OMB Circular A-11's Appendix B, "Budgetary Treatment of
           Lease-Purchases and Leases of Capital Assets," was amended in 2005
           to require agencies to submit to OMB for review any service
           contracts that require the contractor to acquire or construct
           assets valued over $50 million. While these provisions do not
           apply to the previously-awarded simulator training contracts,
           future replacement contracts will be covered.

           Contract Information
			  
			  All of the Air Force and Army simulator service contracts are
           funded with O&M funds. O&M funds are typically used for such
           things as military force operations, training and education, and
           depot maintenance. The contracts are requirements contracts,
           meaning that the government, within available funds, shall order
           from the contractor all the training services specified for each
           of the aircraft platforms that are required during the effective
           performance periods. Additionally, each contract contains language
           limiting the government's liability in the event the contract is
           terminated. For example, the Air Force F-15C contract states that
           the government reserves the right to terminate the contract for
           its sole convenience and that such termination prior to the
           issuance of a funded task order shall result in no payment to the
           contractor of any amount for any work performed or costs incurred.
           Table 1 provides additional descriptive information for each
           contract.

           Table 1: Air Force and Army Simulator Training Contracts

           Source: Air Force and Army contracts.

           aEstimated costs for contract base years and all potential
           award-term or option years.

           bThe F-15C and F-15E contracts were originally awarded to
           McDonnell Douglas Corporation, which became a wholly-owned
           subsidiary of the Boeing Company.

           cTraining services for the F-15E are ordered under a much larger
           umbrella contract, Training Systems Acquisition II.

           The military services are relying on industry to capitalize the
           required up-front investment needed to acquire simulator hardware
           and software, with the understanding that the contractors will
           amortize this investment by selling training services by the hour.
           Each contract establishes operating hours and the hourly payment
           rates for the life of the contracts, with rates structured to
           provide the contractor with higher income in the initial years of
           service. In calendar year 2004, for example, if the F-16
           contractor provided Shaw AFB with simulator availability that met
           95 percent of the required system elements,5 the hourly rate would
           be $5,225, whereas in calendar year 2006, it would drop to $709
           per hour.

           We have previously identified the need to examine the appropriate
           role for contractors to be among the challenges in meeting the
           nation's defense needs in the 21st century.6 We recently reported
           that the government's increasing reliance on contractors for
           missions previously performed by government employees highlights
           the need for sound planning and contract execution.7

           Acquisition Reforms Shaped Contract Structures
			  
			  The structure of the simulator service contracts was heavily
           influenced by mid-1990s' acquisition reform initiatives such as
           the Federal Acquisition Streamlining Act of 19948 and the
           Clinger-Cohen Act of 1996.9 These Acts encouraged agencies to use
           commercial acquisition procedures as a way to streamline the
           acquisition process. Differences under commercial versus
           non-commercial procedures pertain, for example, to the contracting
           officer's determination of price reasonableness, the government's
           right to inspect and test, and government rights to acquire
           technical data. Appendix III outlines these and other key
           differences. The Air Force contracts for simulator training are
           structured as commercial acquisitions, but the Army's is not. Army
           officials told us they could not justify calling the requirement
           "commercial" because the simulators would be configured to reflect
           combat helicopters, which do not exist in the commercial market.
           In August 2005, a DOD Inspector General review of the procurement
           procedures for the F-16 contract concluded that the simulator
           service did not meet the definition or intent of a commercial
           service and recommended that the Air Force not use commercial
           procedures for the re-competed F-16 contract. The Air Force is
           using non-commercial procedures for the new contract.

           To allow for contractor recoupment of up-front investment, the
           strategy to acquire simulator services envisioned longer duration
           contracts. This coincided with practices in commercial industry,
           where long-term relationships between buyer and seller were
           becoming common. The Air Force and Army adopted this approach by
           including award-term incentives in the contracts. This incentive
           can best be described as a variant of an award-fee incentive,
           where the contractor is rewarded for excellent performance with an
           extension of the contract period instead of additional fee. Under
           the award-term concept, an assessment of the contractor's
           performance is presented to the term determining official, who
           unilaterally determines whether to award an extension or a
           reduction to the contract ordering period. The potential total
           years of contract performance under the simulator contracts range
           from 13 to 19.5 years. Appendix IV contains the details of each
           contract. Award-term incentives are relatively new in government
           contracting and are not addressed in the Federal Acquisition
           Regulation (FAR).

           Key Players
			  
			  Several key players are involved with acquisition and use of
           simulator training.

           For the Air Force:

           o  Air Combat Command: The requiring entity-the user of simulator
           training services-is located at Langley AFB, Virginia. The command
           trains, equips and maintains combat-ready forces for rapid
           deployment and employment.

           o  Aeronautical Systems Center: This organization is the
           acquisition agency for the simulator contracts. Located at
           Wright-Patterson AFB, Ohio, it manages development, acquisition,
           modification, and in some cases, sustainment for a wide variety of
           aircraft and related equipment programs. The center develops
           attack, bomber, cargo, fighter, trainer, and reconnaissance
           aircraft for the Air Force.

           o  Air Force fighter units: These are the users of the simulator
           training, which currently is taking place in 10 fighter units.

           For the Army:

           o  Fort Rucker, Alabama: Fort Rucker is the requiring entity for
           the Army's helicopter flight simulator services. It is the home of
           all Army aviation flight training and the location of the initial
           training for new aviators, known as Flight School XXI. The types
           of helicopters used in Flight School XXI training are the TH-67
           basic training helicopter, Chinook, Blackhawk, Apache, and the
           Attack Reconnaissance Helicopter. Unlike the Air Force's multiple
           sites, the helicopter simulators provided under the service
           contract are located at only this one training site, not at each
           operational unit.

           o  Army Program Executive Office for Simulation, Training and
           Instrumentation: This office's mission is to provide training,
           testing, and simulation solutions for soldier readiness. The
           office is co-located in Orlando, Florida, with the Naval Air
           Systems Command, which awarded the contract on behalf of the Army.

           Service Contract Approach Had Perceived Benefits but Was Not
			  Adequately Supported
			  
			  Both the Air Force and Army were faced with obsolete simulators
           due to decisions to not devote sufficient procurement funds to
           upgrade existing simulator hardware and software. The decision to
           buy simulator training as a service allowed use of O&M funds,
           which would alleviate the need to compete for procurement funds.
           Further, it was envisioned that service contracts would allow for
           automatic simulator upgrades to match the changing aircraft
           configurations, because industry would be responsible for
           acquiring, operating, and maintaining the simulators and keeping
           them concurrent. However, the decision to embark on a services
           approach was not supported by a thorough analysis of the costs and
           benefits, despite a DOD directive providing that the acquisition
           of simulators is to be based on an evaluation of the benefits and
           trade-offs of potential alternative training solutions.

           Simulators Had Lost Out in Competition for Procurement Funds
			  
			  The difficulty associated with competition for limited procurement
           dollars was a key factor in the decision to turn to service
           contracts for war-fighting training. Frequently, simulators have
           lost out in this competition and ended up under-funded. In 1997,
           the Air Force identified simulators for four aircraft-the F-15C,
           F-16, F-15E, and AWACS-as "obsolete or grossly non-concurrent" due
           to age, technological obsolescence, and lack of concurrency with
           operational aircraft. By early 2002, the Army was also faced with
           non-concurrent helicopter simulators, and field unit commanders
           were reporting decreased unit readiness. For example, while the
           goal of the training at Fort Rucker is to produce aviators trained
           at a proficiency level of two (with level one being the highest),
           Army officials reported that most of the aviators were leaving
           school with only a proficiency level of three.

           These degraded situations existed despite a DOD directive that
           provides for the military services to ensure that all development,
           procurement, operation, and support costs for the acquisition of
           training simulators were programmed and funded. Recognizing the
           need to keep simulators current with aircraft configurations,
           particularly as the use of simulators to substitute for live
           flying hours was rising, the Air Force issued specific guidance on
           training devices. For example, Air Force Instruction 36-2248,
           Operation and Maintenance of Aircrew Training Devices, provides
           that funding be established for simulator modifications
           concurrently with modifications to the weapon system. Also, Air
           Force Instruction 36-2251, Management of Air Force Training
           Systems, provides that the training system receive the same
           precedence rating as the prime mission system it supports and the
           same visibility, funding, and documentation. Nevertheless, Air
           Force funding decisions had not kept flight training simulators
           for the four aircraft systems concurrent with aircraft
           configurations.

           Also in the late 1990s, the Air Combat Command had unexpended O&M
           flying hour funds available due to flight crew deployments and
           obstacles in scheduling training. Use of these funds for service
           contracts would alleviate the need to compete for procurement
           funds in an increasingly tight arena. The competition for
           procurement dollars was also a factor for the Army, which noted
           that the funds necessary to maintain and upgrade its helicopter
           training simulators had "not competed effectively against other
           Army operational and logistics requirements."

           Service Contracts Intended to Yield More Concurrent Simulators
			  
			  Air Force acquisition officials conducted market research to
           determine how civilian airlines acquired flight training. They
           found "turnkey" training services contracts10 in place in the
           commercial airline industry. These officials envisioned that
           services contracts would provide quicker state-of-the-art pilot
           and aircrew training and keep up with the rapid pace of technology
           development by shifting the responsibilities for simulator
           ownership, operation, and maintenance from the government to the
           contractor. Further, with the contractor responsible for any
           development, production, and testing necessary to ready the
           simulators for use, the Air Force saw that it would be relieved of
           these multiple acquisition efforts, an important factor given the
           recently downsized acquisition offices. In addition, a stated
           benefit of the service contracting approach for simulator training
           as initially implemented was the streamlining or reduction in
           government oversight. Since commercial acquisition procedures were
           used to buy these services, fewer government system reviews were
           required.

           When it decided to take a new approach to solve its helicopter
           simulator concurrency problems, the Army conducted its own market
           research, solicited business solutions from industry, and
           conferred with Air Force DMO officials.

           Decision to Use Service Approach Not Supported with Thorough
			  Analysis of Costs and Benefits of Alternatives
			  
			  Neither the Air Force nor the Army thoroughly analyzed the costs
           and benefits of alternative approaches before pursuing this new
           approach. DOD's August 1986, Directive 1430.13, Training
           Simulators and Devices, provides that the acquisition of
           simulators be based on an analysis of the training need, the
           potential use of existing devices to satisfy that need, and an
           evaluation of the benefits and trade-offs of potential alternative
           training solutions.11 A 1999 report to the Air Force on the DMO
           program also noted the importance of identifying key business
           factors before embarking on a major acquisition. As a result of
           the failure to conduct a thorough review of the various
           alternatives to solving the problem of non-concurrent simulators,
           decision makers lacked information on the potential cost and
           benefit estimates that would be encountered should facts,
           circumstances, and assumptions change.

           The historical documents we reviewed demonstrate that within the
           Air Force there was uncertainty about the cost-effectiveness of
           the service contract approach to simulator training. Although the
           potential for reduced costs through outsourcing certain
           responsibilities and eliminating government logistics support were
           cited in some decision documents, other documents indicated that
           the service contract approach would not cost significantly more or
           less than the traditional ownership strategy. Air Force officials
           told us that a comprehensive study of various options for
           providing simulator training had been commissioned. However, they
           have been unable to locate it.

           In preparing to re-compete the F-16 simulator contract, the DMO
           program office completed a formal business case analysis in
           November 2005, in response to a July 2005 congressional request.
           Air Force officials acknowledged that, if not for the request, the
           formal business case analysis would not have been completed.

           The Army completed two business case analyses prior to contracting
           for simulator services under the Flight School XXI program, but
           the analyses lacked sufficient detail to provide a thorough
           examination of the pros and cons of the new approach. The scope of
           the analyses was limited to determining (1) what length of service
           contract would be appropriate to justify the large up-front
           investment required of the contractor and (2) whether projected
           funding was sufficient to meet program costs in the event the Army
           was required to follow the traditional acquisition approach. The
           Army provided us with decision briefings that set forth various
           options for simulator training, but the documents ruled out all
           but the service contract approach without providing supporting
           analyses of the costs and benefits associated with each
           alternative. Further, the traditional method, where the government
           bought the simulators, was eliminated as an option due its
           perceived inability to meet the Flight School XXI 15-month
           start-up time frame. This schedule eventually slipped more than 10
           months with, according to Army officials, no detrimental effect on
           student training schedules. The briefings do not address the
           possibility that the 15-month time frame was flexible.

           While Training Has Improved, the Expected Number of Simulator Sites
			  Has Not Been Activated
			  
			  Air Force and Army officials told us the new simulators are big
           improvements over what they had previously. However, the Air Force
           has faced funding uncertainties using O&M funds for the contracts,
           and subsequent schedule slippages have resulted in fewer simulator
           sites activated than planned. In particular, the F-16 simulator
           training contractor, citing the reduced activations, notified the
           Air Force as early as May 2001, that it was unable to provide
           simulator services as originally agreed and wished to restructure
           the contract. Later, the company cited Air Force funding problems
           and schedule slips as the basis for claims against the Air Force
           and notified the Air Force that its financial situation under the
           contract was no longer viable. The Air Force will let the current
           F-16 simulator training contract expire in June 2007 and is in the
           process of re-competing the contract, which will likely result in
           a training gap for pilots and additional costs to the Air Force.

           Some Air Force and Army Sites Are Operating and Officials Cite
			  Improved Training Results
			  
			  At the locations we visited, officials told us they were pleased
           with the quality of the simulator training, particularly when
           compared with the level of training they had in the past. Pilots
           are routinely surveyed about the training they receive, and
           officials told us that, generally, the results have been very
           positive. For example, the Director of Operations for the F-16
           mission training center at Shaw AFB told us that the simulation
           hardware and software are outstanding and that the training
           received by young pilots is great. Initial training began under
           the Army's Flight School XXI contract in November 2005. While all
           planned simulators have not yet been activated, according to
           Flight School XXI officials the school is now meeting its training
           goal and producing aviators with a proficiency level of two, an
           improvement over the old regime.

           As of July 2006, the Air Force had 16 training simulator sites
           operational, as shown in table 2.

           Table 2: Operational Simulator Training Locations and Start Dates
           as of July 2006

           Source: GAO analysis of Air Force data.

           aSome locations have multiple mission training centers.

           Air Force Funding Uncertainties Have Resulted in Schedule Slippages
			  
			  The use of O&M funds under the service contract approach was
           intended to overcome the situation the military services had faced
           in the past, when internal decisions on funding priorities had
           resulted in inadequate procurement funds being made available for
           simulators. However, almost from the start of the DMO program,
           funding has been less than projected. As a result, schedule
           slippages have occurred for many sites compared with original Air
           Force requirements set forth in acquisition plans. Army officials
           told us that, to date, O&M funding for the Flight School XXI
           program has not been reduced. Army officials committed at the
           outset to fully fund the contract in accordance with the
           originally projected funding profile and, to date, the funding
           level has remained stable.

           As early as the 2002 budget planning process, Air Force budget
           requests did not fully fund planned activations, with a total
           difference between estimated requirements and funding of $524
           million over the future year defense plan, as shown in table 3.

1 Award-term incentives are similar to award-fee incentives, but the
contractor is rewarded for excellent performance with an extension of the
contract period instead of additional fee.

2 Pub. L. No. 107-107 (2001) codified at 10 U.S.C. 2330.

3 Implemented by Defense Federal Acquisition Regulation Supplement
237.170.

4 GAO, Contract Management: High-Level Attention Needed to Transform DOD
Services Acquisition, GAO-03-935 (Washington, D.C.: Sept. 10, 2003).


Dollars in                                                 
millions                                                   
                 Contract award                                               
System        date           Prime contractor              Contract valuea
AWACS         Feb. 1999      Plexsys Interface Products,                   
                                Inc.                                   $101.0
F-15C         Nov. 1997      Boeing Companyb                         277.0 
F-15Ec        Aug. 2003      Boeing Company                          287.0 
F-16          June 1999      Lockheed Martin Integrated                    
                                Systems, Inc.                           178.0
Flight School Sept. 2003     Computer Sciences Corporation                 
XXI                          - Defense                              1108.0

5 Required system elements include instructor stations, pilot cockpits,
manned combat stations, local and long haul networks, and observation
rooms.

6 GAO, 21st Century Challenges: Reexamining the Base of the Federal
Government, GAO-05-325SP (Washington, D.C.: February 2005).

7 GAO, Contract Security Guards: Army's Guard Program Requires Greater
Oversight and Reassessment of Acquisition Approach, GAO-06-284
(Washington, D.C.: Apr. 3, 2006); and Hurricane Katrina: Improving Federal
Contracting Practices in Disaster Recovery Operations, GAO-06-714T
(Washington, D.C.: May 4, 2006).

8 Pub. L. No. 103-355, Title VIII (1994).

9 Pub. L. No. 104-106, Div. D (1996), formerly Federal Acquisition Reform
Act of 1996 and renamed in Treasury, Postal Service and General Government
Appropriations Act, 1997, Pub. L. No. 104-208, Sec. 808 (1996) (contained
in Omnibus Consolidation Appropriations Act, 1997, Pub. L. No. 104-208,
(1996)).

10 Turnkey contracts are those under which the contractor provides all
needed supplies, services, equipment, facilities, etc., to produce the
desired product or service, so that the customer has only to "turn the
key" to avail himself of the result.

Decision to Use Service Approach Not Supported with Thorough Analysis of Costs
and Benefits of Alternatives

11 This directive was cancelled in 2005.

System Location                      Start date 
F-15C  Eglin AFB, Florida            May 1999   
          Langley AFB, Virginia         May 2000   
          Elmendorf AFB, Alaska         July 2003  
          Kadena air base (AB), Okinawa Apr. 2005  
          Lakenheath AB, United Kingdom Jan. 2006  
AWACS  Tinker AFB, Oklahomaa         Dec. 2001  
          Elmendorf AFB, Alaska         Sept. 2003 
          Tinker AFB, Oklahoma          Dec. 2003  
          Kadena AB, Okinawa            May 2005   
          Tinker AFB, Oklahoma          Sept. 2005 
F-16   Shaw AFB, South Carolina      May 2002   
          Shaw AFB, South Carolina      Jan. 2003  
          Mountain Home AFB, Idaho      Sept. 2002 
          Mountain Home AFB, Idaho      Feb. 2004  
          Spangdahlem AB, Germany       Apr. 2004  
          Misawa AB, Japan              June 2005  

Table 3: Comparison of DMO Estimated Funding Requirement with the Fiscal
Year 2002 Program Objective Memorandum Funding Levels

Dollars in millions                                                  
                        FY 2002 FY 2003 FY 2004 FY 2005 FY 2006 FY 2007 Total 
DMO estimated                                                              
funding requirement      $97    $169     148     157     133      86   790
Fiscal year 2002                                                           
funding approved in                                                  
the program                                                          
objective memorandum      46      49      43      42      43      43   266
Funding difference       -51    -120    -105    -115     -90     -43  -524 

Source: GAO analysis of Air Force documents.

An October 2000 Air Force "roadmap" report stated that this funding
scenario would "severely impact the executability of the current
contracted efforts, as well as the entire [DMO] vision." Further, other
Air Force decisions, in reaction to fiscal constraints and programs viewed
as higher priority, have led to additional funding differences. The Air
Combat Command sought to mitigate the impact of these funding differences
by shifting flying hour funds into the DMO program in 2003. Table 4
depicts some key events pertaining to the program's funding impacts and
the command's attempts to secure additional O&M funds.

Table 4: Key Events That Have Affected Funding for the DMO Program

Event                       Impact                                         
1999: Unexpended flying     Air Combat Command no longer had flexibility   
hour funds centralized at   to use its unexpended flying hour funds for    
Air Force headquarters      DMO; it now had to request, justify, and       
                               compete for funding.                           
2001: Budget decision       Decisions made by the Air Force to fund other, 
results for fiscal years    higher-priority programs resulted in a         
2002 - 2007                 difference of $524 million over the 6-year     
                               period, compared to the DMO plan.              
2004: Decision to           Office of the Secretary of Defense mandated    
accelerate F/A-22 DMO       that the schedule for linking the F/A-22 into  
compliance                  the DMO network be accelerated. The $57        
                               million required to do this was unexpectedly   
                               taken from the DMO budget for fiscal years     
                               2006-08, instead of from the F/A-22 program's  
                               budget.                                        
2004: Budget decision       Decision was made by Air Force not to fund     
results for fiscal year     $250 million to facilitate access to the DMO   
2006                        network by other aircraft (B-1, B-2, B-52,     
                               A-10), although flying hour funds had already  
                               been shifted out of these programs to support  
                               the DMO program.                               
2004: Budget decision       Differences ranging from over $225 million in  
results for fiscal years    fiscal year 2006 to over $50 million in fiscal 
2006-2011                   year 2011 as compared to the DMO plan could    
                               not be alleviated, even with a 5-percent       
                               transfer of flying hour funds from other       
                               programs.                                      
2005: Decision to end F-16  The Air Force decision to end the F-16         
simulator training contract simulator contract in 2007 will result in      
                               additional costs for re-competition efforts    
                               and upgrades of old training devices.          

Source: GAO analysis of Air Force documents.

Largely as a result of these funding uncertainties, many Air Force mission
training centers have been activated significantly behind the planned
schedule contained in acquisition management plans. These schedule
slippages for AWACs, F-15C, and F-16 are shown in tables 5, 6, and 7,
respectively.

Table 5: Comparison of Planned and Actual Site Activations for AWACS

                                                              Quarters behind 
                   Scheduled       Actual activation     scheduled activation 
Location        activation date date                                  date 
Tinker AFB (#1) 1QFY01          1QFY02                                   4 
Tinker AFB (#2) 1QFY03          2QFY03                                   1 
Tinker AFB (#3) No date set     4QFY05                Cannot be determined 
Elmendorf AFB   4QFY02          4QFY03                                   4 
Kadena AB       4QFY02          3QFY05                                  11 

Source: GAO analysis of Air Force data.

Table 6: Comparison of Planned and Actual Site Activations for F-15C

                 Scheduled       Actual activation  Quarters behind scheduled 
Location      activation date date                         activation date 
Eglin AFB     2QFY99          2QFY99                                     0 
Langley AFB   3QFY99          3QFY99                                     0 
Kadena AB     1QFY02          3QFY05                                    14 
Lakenheath AB 1QFY02          2QFY06                                    17 
Elmendorf AFB 1QFY03          4QFY03                                     3 

Source: GAO analysis of Air Force data.

Table 7: Comparison of Planned and Actual Site Activations for F-16

                                                              Quarters behind 
                     Scheduled       Actual              scheduled activation 
Location          activation date activation date                     date 
Shaw AFB (#1)     1QFY02          3QFY02                                 2 
Shaw AFB (#2)     3QFY02          2QFY03                                 3 
Mt. Home AFB (#1) 3QFY02          4QFY02                                 1 
Mt. Home AFB (#2) 4QFY03          2QFY04                                 2 
Misawa AB         2QFY03          3QFY05                                 9 
Spangdahlem AB    3QFY03          3QFY04                                 4 

Source: GAO analysis of Air Force data.

Air Force officials told us that since most of the original dates were
"notional," meaning that they were not firm requirements, but rather were
intended to provide contractors with information about potential mission
training center sites, the timely achievement of the schedules was not
required. However, contractor representatives told us that their proposals
relied upon the planned site activation schedules contained in the
contracts, and delays could directly affect their profitability.

Army's Simulator Activation Schedule Was Also Delayed

The Army has twice rebaselined the activation schedules for the Flight
School XXI simulators-the TH-67 and the advanced aircraft virtual
simulators (AAVS)-as shown in figure 1.

Figure 1: Flight School XXI Schedule Rebaselines

In the original contract, the TH-67 basic training helicopter simulators
were scheduled to begin operation in December 2004, 15 months after
contract award. The Flight School XXI project manager could not provide
documentation to support this time frame and, in fact, told us that the
flight school could not have been ready for students at that time. The
Army subsequently rebaselined the schedule to allow for an 8-month delay.
Similarly, the Army revised the AAVS activation schedule-originally set at
18 months after contract award-to allow a 7-month delay. According to the
project manager, these delays resulted from a protest of the contract
award by a competitor and the contractor's renegotiation with its
subcontractors. The schedule was rebaselined a second time, as shown
above, because the contractor was not able to meet the adjusted schedule.
The Army agreed to the further slippages in exchange for the contractor's
providing two extra terrain databases as consideration. Despite these
schedule changes, the necessary simulators and facilities were ready for
the first flight school class in November 2005, in accordance with the
final revisions to the contract schedule.

F-16 Simulator Training Gap Anticipated

The risk the government faces if a contractor fails to perform as expected
under the service contracts is heightened because the government does not
own anything-the hardware, software, and data rights are owned by the
contractor. In the traditional approach, the government would own the
hardware and any software or data it had acquired rights to. While there
would be no guarantees as to the condition of these items if the
contractor had failed to perform, the government would at least be able to
provide them to the replacement contractor, who could potentially make use
of them under a new contract. The situation the Air Force has faced with
the F-16 simulator contract is illustrative of the potential for not only
a degradation in training, but also increased costs to the government when
contract performance does not occur as planned.

From the outset, the Air Force believed that the F-16 simulator
contractor's cost estimate was low, as it was about $70 million less than
the government's estimate. According to Air Force and contractor
officials, the reason for the low cost estimate was that the contractor
amortized its development costs over all the sites that were planned to be
activated rather than the minimum number that were contractually required.
When schedule delays occurred and the expected sites were not activated,
the contractor reported that it lacked the financial viability to continue
work under the contract. In April 2003, the contractor stopped work toward
making the simulators concurrent with the aircraft, stating that it
considered the tasks beyond the contract scope. Subsequently, it told the
Air Force it was not in its best interest to activate additional training
sites.12

The Air Force will allow the F-16 simulator training contract to expire in
June 2007 because, according to DOD, the contractor failed to earn enough
award-term points to extend the period of performance. The Air Force plans
to re-compete the contract. Two aspects of the original contract, awarding
it as a commercial acquisition and including an award- term provision,
will not be included in the new contract.

12 The contractor claimed that it had incurred $73.3 million in
non-recurring costs and that it had only recovered $31.7 million of these
costs through payment for preparatory services. According to a March 2006
DOD Inspector General report, an improper contract restructure directed by
the Air Force Principal Deputy Assistant Secretary for Acquisition and
Management committed the Air Force to pay the remaining $41.6 million in
claimed non-recurring costs even though the government received minimal
value.

Because of the time needed to re-compete the contract and for the winning
contractor to provide initial training capabilities, the Air Force faces a
potential training gap of over 2 years, during which even the current
degraded level of F-16 simulator training services will not be available
to pilots. In an effort to ensure some level of continued training during
that period, the Air Force plans to award a contract for interim service
capability at three air bases. This interim capability will be available
for block 50 aircraft only.13 For the block 40 aircraft, the Air Combat
Command plans to spend approximately $20 million to refurbish old F-16
unit training devices. These devices are limited in training potential
compared to the current level of simulation.

    Return on the Expenditure of Taxpayer Dollars Is Not Effectively Tracked

The Air Force and the Army are not effectively tracking the return on
their expenditure of taxpayer dollars to acquire simulator training
services. The extent to which the simulators are being used is either not
measured or is measured inconsistently. The government is paying for
activities conducted during the simulator development period but lacks
insight into what it is actually paying for. Finally, award-term
evaluations that were established to encourage excellent contractor
performance do not always measure key acquisition outcomes such as
simulator availability and concurrency, and can result in additional
contract years being awarded for only "satisfactory" performance.

Information on Utilization Rates Is Lacking

The utilization rate is the percentage of available hours the simulators
are actually used. The Army is not tracking the extent to which aviators
are using the contracted service for Flight School XXI simulators, even
though for simulators the Army owns, utilization rates are tracked.
Program officials told us that, because the Army is contracting for
simulator training to be available, there was no need to track the extent
to which the government is using this availability. Without data on
utilization rates, the Army has no basis for determining the extent to
which it is using the services it is buying.

We found that Air Force installations are collecting information on
monthly utilization rates, as provided for in a May 1998 Air Force
instruction.14 However, rates at the locations we examined were often far
less than the hours the government purchased. For the three AWACS mission
training centers at Tinker AFB, for example, we found that, during the
2-year period ending December 2005, monthly utilization rates were
frequently reported at less than 50 percent, as shown in figure 2.

13 The primary mission of F-16 block 50 aircraft is the destruction and
suppression of enemy air defenses. Block 40 aircraft have the primary
mission of filling the air-to surface attack role.

Figure 2: Comparison of Monthly Utilization Rates for AWACS

The Air Force Audit Agency has reported that installations had acquired
excess simulator capacity and unnecessarily consumed O&M funds that could
have been applied to other mission requirements. At Shaw AFB, for example,
the agency found that the Air Force had paid to use the simulator 10 hours
a day, but only used it about 6 hours per day over a 4-month period.15 The
underutilization was attributed to missions being either not scheduled or
cancelled. Deployment requirements and range training were identified as
contributing factors. At Spangdahlem AB, the audit agency reported that
the Air Force had contracted for excess hours of simulator availability to
provide the maximum flexibility for pilot schedules.16 As a result, the
Air Force paid for enough simulator availability to hold 3,952 training
events in fiscal year 2005, even though it needed only 1,982 training
events to meet training requirements.

14 Air Force Instruction 36-2248, Operation and Management of Aircrew
Training Devices, provides that training units calculate the utilization
rate.

15 Air Force Audit Agency, Flight Simulator Utilization, 20th Fighter
Wing, Shaw AFB, SC; F2004-0051-FDM0000 (Mid-Atlantic Area Audit Office,
May 5, 2004).

Our analysis also found that monthly utilization rate calculations are
inconsistent among DMO system sites, even though an Air Force instruction
provides guidance on how to calculate and report utilization data. We
asked six installation quality assurance representatives how they
calculated utilization rates. Four of the six representatives were unaware
of the instruction, telling us that they had not received any guidance for
calculating simulator use. Several different calculation methods are being
used, as described in table 8.

Table 8: Comparison of Monthly Utilization Rate Calculation Methods for
DMO System Simulatorsa

Simulator                Calculation method                                
AWACS                    Number of hours where some training was given     
                            divided by the number of available training       
                            hours.                                            
F-15C                    Number of training periods in which one or more   
                            simulators was in use divided by the number of    
                            training periods available.                       
F-16 (Shaw AFB)          Number of system elements used that were          
                            effective divided by the number of possible       
                            system elements.                                  
F-16 (Mountain Home AFB) Number of simulators used divided by the number   
                            of available simulators.                          

Source: Air Force quality assurance representatives.

aF-15E service not yet available.

Air Combat Command officials told us the reported utilization rates are
used to determine whether or when to activate another training center at a
site. They also said they are using utilization rate information to
determine how many additional "live" flying hours can be moved to the
simulators, in particular to alleviate the burden of high fuel costs for
aircraft. Because of the very different methods being used to calculate
the rates, however, decisions are being made based on non-comparable
information.

16 Air Force Audit Agency, Distributed Mission Operations, 52d Fighter
Wing, Spangdahlem AB, Germany; F2006-0018-FDE000 (European Area Audit
Office, Jan. 25, 2006).

In addition, we found that the Air Force's instruction for calculating
monthly simulator utilization rates could result in overstating the rates,
thus overstating the return on the expenditures made. The instruction
directs that utilization be reported when any or all devices at a given
location are used. Thus, the Air Force can pay to have four simulators
available at a site, use only one of the four during a training period,
and still report that simulator utilization was 100 percent as opposed to
25 percent of the paid availability.

Payments for Tasks during Development Period Are Not Transparent

Under the services approach, contractors commit to major investment at the
front end, with the return on their investment to come from hourly fees
received for providing simulator service. As an additional way to help the
contractor recoup its costs earlier, the government added "preparatory"
tasks during the development period prior to the start of service. These
tasks are defined in the contracts as discrete events, such as site
surveys and training capability assessments, that are ordered and paid for
prior to the start of service. Payments for these tasks provide the
contractor cash flow between contract award and the planned service start
dates and give the government a contractual avenue for contract oversight
prior to receiving services. We found that the Air Force and Army have
little insight into what they are paying for under the preparatory tasks.
Although the invoices reflect only the discrete tasks, such as training
capabilities assessments, the wide range of invoice amounts-from $91,000
to more than $6.5 million for similar tasks-and our discussions with
contractor officials suggest that the government is actually making
milestone payments to the contractors for a portion of their up-front
costs to acquire and develop the simulators.

The original service contract concept for the F-15C, the first simulator
contract awarded, had no provision for the contractor to recoup any costs
during the development period, which usually lasts more than a year.
Figure 3 shows the development period before the start of simulator
services and the original hourly rate structure under the F-15C contract.

Figure 3: Original Rate Structure for F-15C Simulator Service Contracts

This original approach, according to Air Force and contractor officials,
contributed to schedule and certification delays with the F-15C. Air Force
officials told us that they had no contractual avenue to obtain insight
into the contractor's performance during the development period and thus
were not aware that the contractor had encountered delays in obtaining
information from other programs and in determining the complexity of some
simulation elements. As a result, full service was not implemented on
schedule and certification of simulation service was delayed until after
the start of initial service. Further, according to the contractor, it
suffered an unrecoverable loss of income during the high-rate, initial
service period. Subsequently, based on feedback received from industry,
the Air Force changed its approach and incorporated preparatory services
into the F-15C contract and all subsequent DMO system contracts to obtain
more visibility into contractor activities during the development period.
The Army also paid for preparatory services during the development period
of the Flight School XXI contract.

Our analysis of the Air Force's payments for preparatory services found
significantly disparate costs for site surveys and training assessments,
as reflected in tables 9 and 10, respectively.

Table 9: Comparison of Site Survey Costs at Selected DMO System Sites

Dollars in thousands                    
                                 Systems
Sites                 AWACS F-15C F-15E  F-16 
Tinker AFB (#1)        $130             
Tinker AFB ( #2)        216             
Elmendorf AFB           322   $55   $90 
Kadena AB               330    67       
Lakenheath AB                  56   100 
Shaw AFB (#1)                            $625 
Shaw AFB (#2)                           1,000 
Shaw AFB (#3)                           1,100 

Source: GAO analysis of contractor invoices.

Table 10: Comparison of Training Capability Requirements Assessments and
Training Capability Assessments for DMO Systems

Dollars in thousands                                  
                                              Systems
Task                              AWACS F-15Cc  F-15E   F-16 
Training capability requirements assessmentsa
Number 1                          $122         $2,000 $2,850 
Number 2                                       2,600  4,850  
Number 3                                       200    1,800  
Number 4                                       100    
Training capability assessmentsb 
Number 1                          356   $91           1,644  
Number 2                          454   722           6,575  
Number 3                          261   565           3,161  
Number 4                          289                 1,000  
Number 5                                              115    
Number 6                                              150    

Source: GAO analysis of contractor invoices.

aTraining capability requirements assessments trace the capabilities of
the simulator to the training task list, which provides descriptions of
training for various mission profiles.

bTraining capability assessments include reviews of system performance
evaluations and verification and validation tests to authorize shipment of
simulator service to designated destinations.

cThe F-15C contractor was not paid for training capability requirements
assessments as preparatory services.

We asked Air Force and Army officials what was specifically included in
these preparatory services and how they determined what they received in
return for payments made. They told us that the contractors determine what
is included and needed for each service at each site.

Three of the four contractors we spoke with agreed that funding for
preparatory tasks helped defray their development costs. They said that,
in effect, they bill for these tasks as milestone payments rather than for
the discrete tasks themselves. Thus, they are able to begin defraying
hardware and software development costs before the start of services.
Officials from the fourth contractor stated that site survey tasks are
standard but that there is some leeway in what is to be done for training
capability assessments and training capability requirements assessments.

With the upcoming re-competition of the F-16 simulator training contract,
the Air Force may pay again for the preparatory service tasks in the new
contract's development period, having already spent nearly $42 million on
these tasks in the initial contract. Air Force officials told us they
cannot assume that potential offerors would make use of the preparatory
work the original contractor has performed.

Award-term Evaluations Do Not Measure Key Acquisition Outcomes

In an effort to measure performance and encourage the contractors to
perform in an efficient and effective manner, both the Air Force and Army
employ award-term incentives. However, while the award-term evaluation
areas include pilot and crew satisfaction, they do not always measure the
key acquisition outcomes of system availability and concurrency with
aircraft upgrades.

While the Air Force does include system availability as an evaluation
area, it is assigned only 25 to 30 percent of the total score. Concurrency
is not included as a separate evaluation area. The Army's evaluation
areas, on the other hand, include concurrency but not system availability.
While the Army requires the tabulation and submission of such data as
operational availability and training service completion rate, these data
are not included in the award-term evaluations.17 In addition, several of
the evaluation areas include assessments of such things as responsiveness
to government requests for cost and pricing data for proposed work not in
the initial contract. We recently recommended that DOD move toward more
outcome-based award-fee criteria that would promote accountability for
acquisition outcomes, rather than include criteria such as responsiveness
to government customers or the quality of proposals submitted.18

17 Operational availability is the total time of effective training
service divided by the total time of scheduled simulation services. The
training service completion rate is the number of completed training
periods divided by the number of required training periods.

Table 11 compares the award-term evaluation areas and the weight given to
each area.

Table 11: Comparison of Award-term Areas of Evaluation and the Weight
Given to Each Area by System

                                        Weight of evaluation area (percent of
                                                       total)
                                                               Flight School  
Evaluation area                       AWACS   F-15C   F-16       XXI       
Pilot/crew satisfaction                50      50      50         40       
System availability                    30      25      20   
Technical insertion                    10      15      10         10       
Proposal quality (also called          10       5      15         5        
affordability)                                              
Small business participation                    5      5    
Concurrency                                                       20       
Management (includes small business                               25       
participation)                                              
Total                                  100     100    100        100       

Source: Air Force and Army award-term plans.

Note: F-15E does not include an award-term incentive.

The Air Force and Army both assign the largest weight to "pilot/crew
satisfaction." However, this measure has limitations, particularly when it
is heavily relied on to inform award-term decisions. Air Force officials
told us that it is in the pilots' best interests to assign a high rating
to this factor; otherwise, they could be viewed as not having received
adequate training and could be asked to retake it. Additionally, pilots
are frequently hurried in completing their surveys and dash off check
marks without much consideration. Also, the distinction between the levels
of satisfaction can be blurry. For the Army, for example, if training and
support are adversely impacted for an "extended period," user satisfaction
is to be rated as unsatisfactory. However, if the adverse impact occurs
"infrequently or temporarily," it is considered marginal. Because the
terms are not defined, the Army cannot be certain that pilots are
providing consistent ratings.

18 GAO, Defense Acquisitions: DOD Has Paid Billions in Award and Incentive
Fees Regardless of Acquisition Outcomes, GAO-06-66 (Washington, D.C.: Dec.
19, 2005).

We also found that, under the Air Force's award-term plan, contractors can
earn an additional award-term year for only satisfactory performance
because awarded points are rolled over to the next evaluation period. A
contractor with only satisfactory performance in each of five rating areas
can receive up to 51 points each year; thus, within 2 years, it can
accumulate the 100 points needed for a 1-year contract extension. The F-16
simulator training contractor, for example, which recently notified the
government that it could not continue to perform under the contract,
received overall award-term evaluations of "very good" for the first two
rating periods (May 2002 through July 2003) and "satisfactory" in the
third and fourth periods (July 2003 through January 2005) and earned one
contract year extension. The Army has taken a different approach; under
its award-term plan it is very unlikely that the contractor can be awarded
contract extensions for "satisfactory" performance because rollover is
allowed only when more than 100 points are earned. Thus, the contractor
with only satisfactory performance cannot accumulate enough points for an
additional contract year.

While service is not yet available, the F-15E simulator training contract,
awarded in August 2003, does not include an award-term incentive because,
according to the contracting officer, "it doesn't work." Contractor
officials told us that the subjective nature of the criteria and the
manner in which they are applied negate the award term as a performance
incentive. Both the Air Force and Army indicated that they are moving away
from using award-term incentives on future contracts. The Air Force will
not include such an incentive in its re-competition for the F-16 simulator
training contract because, according to the DMO director, it has not been
found to be a significant motivator to the contractor; experience has
shown that withholding payment for poor service is a much more effective
tool to induce improved performance. In addition, since a recent statutory
provision limits future total contract periods of performance to 10 years,
an award-term provision can no longer be used to implement long-term
arrangements such as those in place for the existing simulator training
contracts.19

We recently reported that DOD has little evidence to support its belief
that award fees improve contractor performance and acquisition outcomes
and, in fact, frequently pays out most of the available award fee to
contractors regardless of their performance outcomes.20 We also found that
DOD contracts frequently included rollover provisions, where unearned
award fee from one evaluation period was shifted to a subsequent
evaluation period or periods, thus providing the contractor an additional
opportunity to earn previously unearned fee. We recommended that DOD issue
guidance on when rollover of award fee is appropriate. A March 2006
guidance on award fee contracts states, among other things, that use of
rollover provisions should be the exception rather than the rule and that
the decision to use rollover provisions should be addressed in the
acquisition strategy, including a rationale as to why a rollover provision
is appropriate.

                                   Conclusion

Because simulator training had lost out in the internal competition for
procurement funds, the Air Force and Army turned to service contracts,
expecting that O&M funds would be made available to meet requirements. In
the case of the Air Force, this expectation has not materialized and
planned site activations have been slowed. In addition, although the Air
Force and Army plan to continue with the service contract approach for
simulator training, neither supported the decision with a thorough
analysis of the costs and benefits of alternative approaches to delivering
the training. Finally, the heightened risks associated with increased
reliance on contractors to deliver simulator training calls for careful
attention to contract management and oversight. Effective and well-managed
incentives for motivating performance are especially important. Better
government visibility into the contractors' activities, such as
preparatory tasks, during the development period is critical so that the
government can understand the basis for what are essentially milestone
payments during that phase. In addition, unless utilization rates are
tracked in a consistent manner, the government will not know whether it is
making the best use of what it is buying.

19 Ronald W. Reagan National Defense Authorization Act for Fiscal Year
2005, Pub. L. No.108-375, Sec. 813 (2004) limits the total contract period
for multiyear task and delivery order contracts, as extended pursuant to
options, to 10 years unless the head of the agency determines in writing
that exceptional circumstances necessitate a longer contract period.

20 GAO-06-66 .

                      Recommendations for Executive Action

To help ensure that the best approach is used to provide the war-fighter
with needed training, we recommend that the Secretary of Defense direct
the Secretaries of the Air Force and Army to conduct a thorough analysis
of the costs and benefits of using service contracts for simulator
training to determine if it is indeed the best approach. The analysis
should proactively address potential risks associated with the service
contract approach and identify the level of simulator training needed to
meet requirements.

To help ensure that the required training is provided to pilots, we
recommend that the Secretary of the Air Force reconcile the funding level
needed for simulator training with the requirements identified in the
evaluation of costs and benefits of the service contract approach and take
steps to allocate funds accordingly.

To help ensure that the incentives motivate contractor performance toward
achieving desired training outcomes, we recommend that the Secretary of
Defense direct the Secretaries of the Air Force and Army to take the
following two actions:

           o  Determine whether it is in the government's best interest to
           retain the award-term incentive under these service contracts.

           o  If the award-term incentive is retained, take appropriate steps
           to improve the approach by reassessing the areas to be rated and
           the definitions of performance levels for the various grade
           categories. For the Air Force, improvements to the approach should
           include a determination as to whether to continue allowing
           rollover of award-term points.

           To help ensure greater transparency into what the government is
           paying for preparatory tasks during the development phase, we
           recommend that the Secretary of Defense direct the Secretaries of
           the Air Force and Army to take the following two actions:

           o  Reassess the pricing of any up-front payments made to the
           contractors during the development period on future replacement or
           restructured contracts.

           o  If retained, take appropriate measures to (1) create an
           appropriate and transparent contract payment mechanism, separate
           from the preparatory tasks, if development costs are to be
           reimbursed; and (2) increase visibility into the percentage of
           upfront development costs contractors are recouping from these
           preparatory tasks and development payments.

           To help ensure that available simulator training for the
           warfighter is used in the most effective and efficient manner, we
           recommend that the Secretary of Defense take the following four
           actions:

           o  Direct the Secretaries of the Air Force and Army to determine
           whether and how simulator utilization can be increased in order to
           maximize use of taxpayer dollars.

           o  Direct the Secretary of the Army to track and record monthly
           utilization rates on Flight School XXI contracted simulator
           training in order to have the data necessary to adjust training
           requirements and contract provisions, as necessary.

           o  Direct the Secretary of the Air Force to revise Air Force
           Instruction 36-2248, Operation and Management of Aircrew Training
           Devices, to ensure that, for the purposes of reporting utilization
           rates, the usage of individual training simulators is calculated.

           o  Direct the Secretary of the Air Force to ensure that all sites
           consistently track and report simulator utilization.

           Agency Comments and Our Evaluation
			  
			  In written comments on a draft of this report, DOD concurred with
           all but one of our recommendations. DOD partially concurred with
           our recommendation that the Army track and record monthly
           utilization rates on simulators at Flight School XXI. DOD stated
           that the service contract approach requires only that the vendor
           meet the programmed student training load. Nevertheless, DOD
           stated that the contractor is required to submit utilization data
           and that the data are available for use in future adjustments to
           the contracting strategy, requirements, or provisions. Our
           recommendation was intended to encourage DOD to fully understand
           its student training requirements and to collect the information
           to decide whether it needs to adjust requirements or contract
           provisions regarding simulator availability. Whether the
           utilization rates pertain to individual simulators or the student
           training load as a whole, we believe that the Army needs to know
           the extent to which it is actually using the simulator
           availability it is buying. DOD also offered two corrections to
           information in the draft, and we made changes as appropriate.
           DOD's comments are included in their entirety in appendix II.

           We will send copies of this report to the Secretaries of Defense,
           the Air Force, and the Army; appropriate congressional committees;
           and other interested parties. We make copies available to others
           upon request. In addition, the report will be available at no
           charge on the GAO Web site at http://www.gao.gov .

           If you or your staff has questions concerning this report, please
           contact me at (202) 512-4841 or by e-mail at [email protected] .
           Contact points for our Offices of Congressional Relations and
           Public Affairs may be found on the last page of this report.

           Lisa Shames, Acting Director Acquisition and Sourcing Management

           Appendix I: Scope and Methodology
			  
			  To determine which factors led the Air Force and Army to acquire
           simulator training as a service contract using operation and
           maintenance funds, we analyzed historical documents such as
           acquisition plans, briefings, and decision memorandums. For the
           Air Force, we interviewed Air Force management, including
           officials at the Office of the Assistant Secretary of the Air
           Force, Acquisition; Aeronautical Systems Center (responsible for
           contracting the simulator training services) and the Air Combat
           Command (funds and uses the simulator training). For the Army, we
           interviewed officials at the Office of the Assistant Secretary of
           the Army for Acquisition, Logistics, and Technology; and Army
           officials responsible for managing the Army's Flight School XXI
           initiative, including officials of the Program Executive Office
           for Simulation, Training and Instrumentation. We visited Langley
           Air Force Base, Virginia, to observe F-15C simulator training;
           Shaw Air Force Base, South Carolina, to observe F-16 simulator
           training; and Fort Rucker, Alabama, to observe the Flight School
           XXI helicopter simulator training. Additionally, to evaluate
           whether the military services adequately justified the new service
           contract approach, we reviewed the Office of Management and
           Budget's Circular A-11, Appendix B, "Budgetary Treatment of
           Lease-Purchases and Leases of Capital Assets," and Air Force and
           Army regulations and guidance regarding business case analyses. We
           also drew from our prior reviews of Department of Defense systems,
           in particular our recent review of the Army's Future Combat
           System.1

           To assess whether the new approach has resulted in the planned
           number of simulator training sites being activated, we evaluated
           contract documents and information provided by the Air Combat
           Command and Aeronautical Systems Center to compare planned to
           actual schedule activations. We gathered and analyzed budget data
           related to program schedules and interviewed program officials. We
           analyzed contract documents and other program documents from
           Flight School XXI and discussed the schedule rebaselining with
           Army officials. We analyzed the Air Force's request for proposals
           for the F-16 simulator training contract re-competition to
           determine whether key differences in the acquisition approach were
           incorporated.

           To determine if the Air Force and Army are effectively tracking
           the return on their expenditure of taxpayer dollars, we analyzed
           simulator utilization data and military service guidance on
           utilization rates; analyzed contractor performance measurements,
           annual evaluations, and award-term plans for the simulator
           training contracts; and compared preparatory service costs charged
           to the government under each of the four Air Force contracts and
           the Army contract. We also interviewed contractor representatives
           and government officials.

1 GAO, Defense Acquisitions: Improved Business Case Is Needed for Future
Combat System's Successful Outcome, GAO-06-367  (Washington, D.C.: Mar.
14, 2006).

Appendix II: Comments from the Department of Defense

Appendix III: Comparison of Contractor Requirements under Commercial and
Non-commercial Acquisition Procedures

The following table shows differences, as set forth in the Federal
Acquisition Regulation (FAR), for contractor requirements under commercial
versus non-commercial acquisition procedures.

                            Under noncommercial        Under commercial       
Requirement              procedures                 procedures             
Type of contract         A wide selection of        Limited contract types 
                            contract types is          are authorized.        
                            available in order to      Agencies shall use     
                            provide flexibility. (FAR  firm-fixed-price       
                            16.101(a))                 contracts or           
                                                       fixed-price contracts  
                                                       with economic price    
                                                       adjustment. These      
                                                       contract types may be  
                                                       used in conjunction    
                                                       with an award fee and  
                                                       performance or         
                                                       delivery incentives    
                                                       when the award fee or  
                                                       incentive is based     
                                                       solely on factors      
                                                       other than cost. (FAR  
                                                       12.207)                
                                                                              
                                                       To implement the       
                                                       Services Acquisition   
                                                       Reform Act of 2003     
                                                       (contained in Section  
                                                       1432 of the National   
                                                       Defense Authorization  
                                                       Act for Fiscal Year    
                                                       2004, Pub. L. No.      
                                                       108-136 (2003)), a     
                                                       proposed amendment to  
                                                       FAR would expressly    
                                                       authorize the use of   
                                                       time-and-materials and 
                                                       labor-hour contracts   
                                                       for certain categories 
                                                       of commercial services 
                                                       under specified        
                                                       conditions. (FAR Case  
                                                       2003-027, 70 Federal   
                                                       Register 56318, Sept.  
                                                       26, 2005.)             
Inspection and testing   Government has right to    Contracts for          
                            inspect and test. (FAR     commercial items shall 
                            46.102 & 46.202-3)         rely on contractors'   
                                                       existing quality       
                                                       assurance systems as a 
                                                       substitute for         
                                                       Government inspection  
                                                       and testing before     
                                                       tender for acceptance  
                                                       unless customary       
                                                       market practices for   
                                                       the commercial item    
                                                       being acquired include 
                                                       in-process inspection. 
                                                       Any in-process         
                                                       inspection by the      
                                                       Government shall be    
                                                       conducted in a manner  
                                                       consistent with        
                                                       commercial practice.   
                                                       (FAR 12.208)           
Determination of price   Price must be determined   While price            
reasonableness           fair and reasonable        reasonableness must be 
                            through various proposal   established, the       
                            analysis techniques. (FAR  contracting officer    
                            15.404-1)                  should be aware of     
                                                       customary commercial   
                                                       terms and conditions   
                                                       when pricing           
                                                       commercial items.      
                                                       Commercial item prices 
                                                       are affected by        
                                                       factors that include,  
                                                       but are not limited    
                                                       to, speed of delivery, 
                                                       length and extent of   
                                                       warranty, limitations  
                                                       of seller's liability, 
                                                       quantities ordered,    
                                                       length of the          
                                                       performance period,    
                                                       and specific           
                                                       performance            
                                                       requirements. (FAR     
                                                       12.209)                
Cost or pricing data     Required for contract      Commercial items are   
                            award and modifications    exempt (FAR            
(Requirements based on   unless applicable          15.403-1(b)(3) and     
Truth in Negotiations    exception, such as         (c)(3))                
Act, 10 U.S.C. 2306a; 41 adequate competition or    
USC 254b)                prices agreed upon are     
                            based on prices set by law 
                            or regulation. Threshold   
                            for application is         
                            $550,000. (FAR 15.403-1    
                            and -4)                    
Contract financing       The contracting officer    For purchases of       
                            must consider the          commercial items,      
                            following order of         financing of the       
                            preference when a          contract is normally   
                            contractor requests        the contractor's       
                            contract financing., (a)   responsibility.        
                            Private financing without  (32.202-1) However,    
                            Government guarantee.(b)   customary market       
                            Customary contract         practice for some      
                            financing. (c) Loan        commercial items may   
                            guarantees. (d) Unusual    include buyer contract 
                            contract financing. (e)    financing. In these    
                            advance payments. (FAR     circumstances, the     
                            32.106)                    contracting officer    
                                                       may offer Government   
                                                       financing in           
                                                       accordance with the    
                                                       policies and           
                                                       procedures in Part 32. 
                                                       (FAR 12.210)           
                            However, government        Government financing   
                            financing provided only to of commercial          
                            extent actually needed for purchases is expected  
                            prompt and efficient       to be different from   
                            performance, considering   that used for          
                            availability of private    non-commercial         
                            financing and probable     purchases. While the   
                            impact on working capital  contracting officer    
                            of predelivery             may adapt              
                            expenditures and           non-commercial         
                            productions lead-times.    techniques and         
                            (FAR 32.104)               procedures for use in  
                                                       implementing           
                                                       commercial contract    
                                                       financing              
                                                       arrangements, the      
                                                       contracting officer    
                                                       must have a full       
                                                       understanding of       
                                                       effects of the         
                                                       differing contract     
                                                       environments and of    
                                                       what is needed to      
                                                       protect the interests  
                                                       of the Government in   
                                                       commercial contract    
                                                       financing. (FAR        
                                                       32.202-1(c))           
                                                       Types of payments for  
                                                       commercial item        
                                                       purchases. (FAR        
                                                       32.202-2)              
                                                                              
                                                          1. Commercial       
                                                          advance payment:    
                                                          payment made before 
                                                          any performance of  
                                                          work (not to exceed 
                                                          15 percent of       
                                                          contract price)     
                                                          2. Commercial       
                                                          interim payment:    
                                                          payment made after  
                                                          some, but not all,  
                                                          work has been       
                                                          performed           
                                                          3. Delivery         
                                                          payment: payment    
                                                          made for accepted   
                                                          supplies or         
                                                          services, including 
                                                          partial deliveries  
                                                          (FAR 32.001)        
Technical data           The Government may acquire Generally, the         
                            technical data and rights  Government shall       
                            in technical data for      acquire only the       
                            multiple purposes.         technical data and the 
                            Agencies shall strike a    rights in that data    
                            balance between the        customarily provided   
                            government's need and the  to the public with a   
                            contractor's legitimate    commercial item or     
                            proprietary interest. (FAR process. The           
                            27.4)                      contracting officer    
                                                       shall presume that     
                                                       data delivered under a 
                                                       contract for           
                                                       commercial items was   
                                                       developed exclusively  
                                                       at private expense.    
                                                       When a contract for    
                                                       commercial items       
                                                       requires delivery of   
                                                       technical data, the    
                                                       contracting officer    
                                                       shall include          
                                                       appropriate provisions 
                                                       and clauses            
                                                       delineating the rights 
                                                       in the technical data  
                                                       in the contract. (FAR  
                                                       12.211)                
Computer                 The Government may acquire Commercial computer    
software/documentation   computer                   software or commercial 
                            software/documentation for computer software      
                            multiple purposes.         documentation shall be 
                            Agencies shall strike a    acquired under         
                            balance between the        licenses customarily   
                            government's need and the  provided to the public 
                            contractor's legitimate    to the extent such     
                            proprietary interest. (FAR licenses are           
                            27.402)                    consistent with        
                                                       federal law and        
                                                       otherwise satisfy the  
                                                       government's needs.    
                                                       Generally, offerors    
                                                       and contractors shall  
                                                       not be required to-    
                                                                              
                                                          1. Furnish          
                                                          technical           
                                                          information related 
                                                          to commercial       
                                                          computer software   
                                                          or commercial       
                                                          computer software   
                                                          documentation that  
                                                          is not customarily  
                                                          provided to the     
                                                          public; or          
                                                          2. Relinquish to,   
                                                          or otherwise        
                                                          provide, the        
                                                          Government rights   
                                                          to use, modify,     
                                                          reproduce, release, 
                                                          perform, display,   
                                                          or disclose         
                                                          commercial computer 
                                                          software or         
                                                          commercial computer 
                                                          software            
                                                          documentation       
                                                          except as mutually  
                                                          agreed to by the    
                                                          parties.            
                                                                              
                                                       (FAR 12.212(a))        
Cost accounting          Compliance generally       Cost Accounting        
standards                required for contractors   Standards do not apply 
                            in connection with         to contracts for       
41 U.S.C. 422            negotiated contracts in    acquisition of         
                            excess of $500,000.        commercial items when  
                                                       they are               
                            Contractors must disclose  firm-fixed-price or    
                            and consistently follow    fixed-price with       
                            their cost accounting      economic price         
                            practices. (FAR 30.101)    adjustment. (FAR       
                                                       12.214)                
Preaward Survey          In determining whether a   If the contemplated    
                            potential awardee is a     contract will involve  
                            responsible contractor,    the acquisition of     
                            per criteria in FAR        commercial items, the  
                            9.104-1, contracting       contracting officer    
                            officers may require a     should not request a   
                            preaward survey when the   preaward survey unless 
                            information on hand or     circumstances justify  
                            readily available is not   its cost. (FAR         
                            sufficient to make such a  9.106-1(a))            
                            determination. (FAR        
                            9.106-1)                   
Audit Rights             When contracting by        Commercial item        
                            negotiation, the           contracts exempted.    
                            contracting officer shall  (FAR                   
                            insert the clause at FAR   15.209(b)(1)(iii))     
                            52.215-2, Audit and        
                            Records-Negotiation in     
                            solicitations and          
                            contracts which allows     
                            contracting officer        
                            examination of costs when  
                            cost or pricing data is    
                            required or for            
                            cost-reimbursement,        
                            incentive,                 
                            time-and-materials,        
                            labor-hour, or price       
                            redeterminable contracts.  
                            (FAR 15.209(b) and         
                            52.215-2)                  
Applicability of certain                            
laws:                                               
Walsh-Healy Public       On contracts for supplies  Not applicable. (FAR   
Contracts Act            over $10,000, contractors  12.503(a))             
                            must adhere to provisions  
                            pertaining to minimum      
                            wages, maximum hours,      
                            child labor, convict       
                            labor, safe/sanitary       
                            working conditions. (FAR   
                            22.602)                    
Contingent fees          Contractor must warrant    Not applicable. (FAR   
                            that it has not employed   12.503(a))             
                            or retained anyone, on a   
                            contingent fee basis, to   
                            obtain this contract.      
                                                       
                            (FAR 3.404, 52.203-5)      
Drug-Free Workplace Act  Contractor must agree that Not applicable. (FAR   
of 1988                  it will provide a          12.503(a))             
                            drug-free workplace (FAR   
                            23.504(a))                 
Vietnam Era Veterans'    Contractor must report on  Law's limitation on    
Readjustment Assistance  its affirmative actions to use of appropriated    
Act of 1972              employ and advance covered funds for contracts    
                            veterans (FAR 22.1302(a))  with entities not      
                                                       meeting veterans       
                                                       employment reporting   
                                                       requirements is not    
                                                       applicable. (FAR       
                                                       12.503(a))             
Trafficking Victims      Contracts for services     Not applicable. (FAR   
Protection               must prohibit contractor   12.503(a))             
Reauthorization Act of   activities regarding, and  
2003                     require contractor         
                            policies to combat, severe 
                            forms of trafficking in    
                            persons, the procurement   
                            of commercial sex acts,    
                            and use of forced labor.   
                            (FAR 22.1705)              
Contract Work Hours and  Contract clause required   Requirements for a     
Safety Standards Act     providing that contractors certificate and        
                            employing laborers or      contract clause        
                            mechanics are required to  related to the Act are 
                            compensate them for        not applicable. (FAR   
                            overtime. (FAR 52.222-4)   12.503(b))             
Anti-Kickback Act of     Contract clause requires   Requirements for a     
1986                     prime contractors to (1)   clause and certain     
                            have in place and follow   other requirements     
                            reasonable procedures      related to the Act are 
                            designed to prevent and    not applicable. (FAR   
                            detect violations of the   12.503(b))             
                            Act; and (2) cooperate     
                            fully with any Federal     
                            agency investigating a     
                            possible violation of the  
                            Act. (FAR 3.502-2(i))      
International Air        Contracts must include     Requirement for a      
Transportation Fair      clause requiring use of    clause related to the  
Competitive Practices    U.S-Flag Air Carriers by   Act is not applicable. 
Act of 1974 (Fly America government contractors     (FAR 12.503(b))        
Act)                     when available (FAR        
                            47.405)                    
Prohibition on Limiting  Contracts must include     Contractors may        
Subcontractor Direct     clause precluding          restrict               
Sales to the United      contractors from           subcontractors' sales  
States                   restricting direct         to the Government, as  
                            subcontractor sales to the long as the Government 
                            Government. (FAR 3.503-2   is treated no          
                            and 52.203-6(a))           differently than other 
                                                       prospective purchaser. 
                                                       (FAR 52.203-6,         
                                                       Alternate I)           
Selected contract                                   
administration                                      
provisions:                                         
Changes                  Generally, contracting     Changes may be made    
                            officer permitted to make  only by written        
                            unilateral changes within  agreement of the       
                            the scope of the contract  parties (bilateral).   
                            and to require continued   (FAR 12.301(b)(3);     
                            contractor performance of  52.212-4(c))           
                            the contract as changed.   
                            (FAR 43.201)               
Termination for the      Generally, termination     Termination costs      
convenience of the       costs for fixed-price      limited to percentage  
government               contracts limited to total of contract price      
                            contract price less        reflecting percentage  
                            payments made or to be     of work performed      
                            made under contract plus   prior to termination   
                            reasonable costs incurred  plus reasonable        
                            in performance of work     charges resulting from 
                            terminated, to include     termination. For       
                            fair and reasonable        payments thereunder,   
                            profit, and reasonable     contractor not         
                            settlement costs. Cost     required to comply     
                            principles and procedures  with cost accounting   
                            of FAR Part 31 apply to    standards or contract  
                            costs. (FAR 49.502(b);     cost principles in FAR 
                            52.249-2)                  Part 31. (FAR          
                                                       12.301(b)(3);          
                                                       52.212-4(l))           

Source: GAO analysis of Federal Acquisition Regulation requirements.

Appendix IV: Potential Duration of Air Force and Army Simulation Contracts

                                   Additional years                           
              Base performance   possible via award  Potential total years of 
System     period                           term      contract performance
Air Force:                                       
F-15C      Nov. 1997 to Nov.                                               
              2005 (8 years)                7 years                  15 years
AWACS      Feb.1999 to Feb.                                                
              2006 (7 years)                8 years                  15 years
F-16       June 1999 to June                                               
              2006 (7 years)                8 years                  15 years
F-15E      Aug. 2003 to July                                               
              2016 (13 years)                   N/A                  13 years
Army:                                            
Flight     Sept. 2003 to Mar.                                              
School XXI 2015 (11.5 years)             8 years                19.5 years

Source: GAO analysis of contract data.

Appendix V: GAO Contact and Staff Acknowledgments

GAO contact

Lisa Shames (202) 512-4841 or [email protected]

Acknowledgments

In addition to the individual named above, Michele Mackin, Assistant
Director; Marie Ahearn; Christine Bonham; Gary Delaney; Carlos Diz;
Benjamin Federlein; Victoria Klepacz; and Sanford Reigle made key
contributions to this report.

(120492)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: [email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800 U.S.
Government Accountability Office, 441 G Street NW, Room 7149 Washington,
D.C. 20548

www.gao.gov/cgi-bin/getrpt? GAO-06-830 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Lisa Shames at (202) 512-4841 or
[email protected].

Highlights of GAO-06-830 , a report to the Subcommittee on Readiness,
Committee on Armed Services, House of Representatives

September 2006

CONTRACT MANAGEMENT

Service Contract Approach to Aircraft Simulator Training Has Room for
Improvement

The Air Force has turned to service contracts for the F-15C, F-16,
Airborne Warning and Control System, and F-15E, and the Army has done the
same for helicopter simulator training at its Flight School XXI. The
contractors own, operate, and maintain the simulator hardware and
software. The military services rely on industry to capitalize the
required up-front investment, with the understanding that the contractors
will amortize this investment by selling training services by the hour.

GAO was asked to address (1) the factors that led the Air Force and Army
to acquire simulator training as a service and whether the decision to use
this approach was adequately supported; (2) whether implementation of the
approach has resulted in the planned number of simulator training sites
being activated; and (3) whether the Air Force and Army are effectively
tracking the return on their expenditure of taxpayer dollars.

GAO makes recommendations to the Secretary of Defense intended to improve
management and oversight of these service contracts to help ensure that
the best approach is used to provide the war-fighter with needed training.
In written comments on a draft of this report, DOD concurred with all but
one of the recommendations, only partially concurring with one pertaining
to the Army's simulator utilization rates. GAO continues to believe that
the Army needs to track the extent to which it is using simulator
availability.

The Air Force and Army turned to service contracts for simulator training
primarily because efforts to modernize existing simulator hardware and
software had lost out in the competition for procurement funds. As a
result, the simulators were becoming increasingly obsolete. Buying
training as a service meant that operation and maintenance (O&M) funds
could be used instead of procurement funds. Shifting the responsibility
for simulator ownership, operation, and maintenance from the government to
the contractor was thought to more quickly enable simulator upgrades to
match the changing configurations of aircraft. However, the decision to
take a service contract approach was not supported by a thorough analysis
of the costs and benefits as compared to other alternatives, despite a
Department of Defense directive that provided for such an analysis.

While Air Force and Army officials told GAO the new simulators are
significant improvements over the previous ones, the expected number of
Air Force training sites have not been activated. For the Air Force, O&M
funds have not been allocated at the anticipated levels, leading to
schedule slippages. The F-16 simulator contractor cited the funding
problems and subsequent schedule slippages as the basis for notifying the
Air Force that its situation under the contract was no longer financially
viable. The Air Force is in the process of re-competing the F-16 training
contract, which will likely result in a training gap for pilots-possibly
over 2 years-and additional costs to the Air Force. The start date of the
Army's flight simulator training was rebaselined twice, but Army officials
told us that adequate training was in place for the flight school
participants.

The return on expenditure of taxpayer dollars is not being effectively
tracked in three key ways:

           o  Air Force utilization of simulator training frequently falls
           well below the hours for which the government is paying. The Army
           is not collecting data on utilization rates at all.
           o  The government has little insight into what it is paying for
           during the development period before training is activated, which
           can take more than a year. While invoices for preparatory efforts
           reflect only discrete tasks such as training capabilities
           assessments, the wide range of invoice amounts and GAO's
           discussions with contractor representatives suggest that the
           government is actually making milestone payments to the
           contractors for a portion of their up-front costs to acquire and
           develop the simulators.
           o  Most of the contracts contain award-term provisions, where the
           contractors can earn an extension of the contract period for good
           performance. GAO found that the award-term evaluation factors do
           not always measure key acquisition outcomes such as simulator
           availability and concurrency with aircraft upgrades.
*** End of document. ***