NASA Procurement: Use of Award Fees for Achieving Program	 
Outcomes Should Be Improved (17-JAN-07, GAO-07-58).		 
                                                                 
Cost-plus-award-fee contracts accounted for almost half of the	 
National Aeronautic and Space Administration's (NASA) obligated  
contract dollars for fiscal years 2002-2004. Since 1990, we have 
identified NASA's contract management as a high-risk area--in	 
part because of a lack of emphasis on end results. Congress asked
us to examine (1) the extent NASA's guidance on award fees	 
addresses problems previously identified with the use of	 
award-fee contracts and (2) whether NASA follows its guidance in 
using award fees to achieve desired outcomes. We reviewed the top
10 dollar value award-fee contracts active from fiscal years 2002
through 2004.							 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-58						        
    ACCNO:   A64921						        
  TITLE:     NASA Procurement: Use of Award Fees for Achieving Program
Outcomes Should Be Improved					 
     DATE:   01/17/2007 
  SUBJECT:   Evaluation criteria				 
	     Contract administration				 
	     Contract performance				 
	     Contractor payments				 
	     Program evaluation 				 
	     Cost control					 
	     Standards evaluation				 
	     Cost plus award fee contracts			 
	     Performance management				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-58

   

     * [1]Results in Brief
     * [2]Background

          * [3]Cost-Plus-Award-Fee Contracts
          * [4]NASA's Use of Cost-Plus-Award-Fee Contracts
          * [5]Award-Fee Criteria
          * [6]Evaluation of Award-Fee Contracts
          * [7]Acquisition Environment Can Affect Acquisition Outcomes

     * [8]NASA's Award-Fee Policy Addresses Many Cost-Plus-Award-Fee C
     * [9]NASA Has Not Always Followed the Preferred Approach Laid Out

          * [10]Some Evaluation Factors Were Not Outcome Factors as Preferre
          * [11]Some Contracts Did Not Limit the Number of Subfactors for Ev
          * [12]NASA Did Not Perform Cost-Benefit Analyses
          * [13]Award-Fee Payments at Times Did Not Reflect Program Outcomes
          * [14]NASA Has Not Assessed the Effectiveness of Award Fees in Ach

     * [15]Conclusions
     * [16]Recommendations for Executive Action
     * [17]Agency Comments and Our Evaluation
     * [18]Scope and Methodology

          * [19]NAS5-60000-Earth Observing System Data and Information Syste
          * [20]NAS15-10000-International Space Station-Johnson Space Center
          * [21]NAS5-32633-Landsat-7 Spacecraft-Goddard Space Flight Center
          * [22]NAS8-60000-Program Information Systems Mission Services-Mars
          * [23]NAS2-14263-Engineering and Technical Support for Life Scienc
          * [24]NAS9-19100-Science, Engineering, Analysis, and Test-Johnson
          * [25]NAS9-98100-Consolidated Space Operations Contract- Johnson S
          * [26]NAS10-99001-Joint Base Operation and Support-Kennedy Space C
          * [27]NAS5-01090-Mechanical System Engineering Services-Goddard Sp
          * [28]NAS7-03001-Jet Propulsion Laboratory

     * [29]GAO Contact
     * [30]Staff Acknowledgments
     * [31]GAO's Mission
     * [32]Obtaining Copies of GAO Reports and Testimony

          * [33]Order by Mail or Phone

     * [34]To Report Fraud, Waste, and Abuse in Federal Programs
     * [35]Congressional Relations
     * [36]Public Affairs

Report to the Chairman, Committee on Science, House of Representatives

United States Government Accountability Office

GAO

January 2007

NASA PROCUREMENT

Use of Award Fees for Achieving Program Outcomes Should Be Improved

GAO-07-58

Contents

Letter 1

Results in Brief 2
Background 3
NASA's Award-Fee Policy Addresses Many Cost-Plus-Award-Fee Contracting
Issues Identified as Problematic 8
NASA Has Not Always Followed the Preferred Approach Laid Out in Its Award-
Fee Guidance 11
Conclusions 17
Recommendations for Executive Action 18
Agency Comments and Our Evaluation 18
Appendix I Scope and Methodology 20
Appendix II Summary Description of the 10 NASA Contracts GAO Reviewed 22
Appendix III Comments from the National Aeronautics and Space
Administration 27
Appendix IV GAO Contact and Staff Acknowledgments 29

Tables

Table 1: Award Fees as a Negotiated Percentage of Contract Value 5
Table 2: Percentage of Available Award Fee Pool Paid on Contracts Examined
15

Abbreviations

CPAF cost-plus-award-fee
CSOC Consolidated Space Operations Contract
DOD Department of Defense
ECS EOSDIS Core System
EOSDIS Earth Observing System Data and Information System
ETM+ Enhanced Thematic Mapper Plus
FAR Federal Acquisition Regulation
FDO fee determination official
IEMP Integrated Enterprise Management Program
ISS International Space Station
JPL Jet Propulsion Laboratory
MSES Mechanical System Engineering Services
NASA National Aeronautics and Space Administration
PEB performance evaluation board
SEAT Science, Engineering, Analysis, and Test

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

United States Government Accountability Office

Washington, DC 20548

January 17, 2007

The Honorable Bart Gordon
Chairman
Committee on Science
House of Representatives

The Federal Acquisition Regulation (FAR) states that cost-plus-award-fee
(CPAF) contracts are intended to motivate excellent contractor performance
in areas such as quality, timeliness, technical ingenuity, and
cost-effective management.^1 During the early 1990s, the National
Aeronautics and Space Administration (NASA) Inspector General and NASA
internal studies raised concerns about NASA's use of CPAF contracts.^2 As
a result, NASA developed specific guidance to improve the effectiveness of
award fees. The CPAF contract type continues to be used extensively by
NASA for obtaining both goods and services, accounting for almost half of
NASA contract dollars for fiscal years 2002 through 2004.^3 Given this,
you requested that we examine NASA's use of award-fee contracts and
determine (1) the extent NASA's guidance addresses the problems previously
identified with the use of award-fee contracts and (2) whether NASA
follows its guidance in using award fees to achieve desired outcomes.

To address these objectives, we reviewed a sample of NASA CPAF contracts
that were active from fiscal years 2002 through 2004.^4 We reviewed
contract files, obtained information from program and contracting
officials through the use of a structured questionnaire, and discussed the
application of award-fee criteria with NASA officials involved in the
award-fee process. Our work was conducted between August 2005 and October
2006 in accordance with generally accepted government auditing standards.
For a complete description of our scope and methodology, see appendix I.

^1FAR Part 16.405-2(a)(2).

^2In December 2005, we issued a report on the use of award and incentive
fees at the Department of Defense that identified many of the same issues
as had been identified by NASA. GAO, Defense Acquisitions: DOD Has Paid
Billions in Award and Incentive Fees Regardless of Acquisition Outcomes,
[37]GAO-06-66 (Washington, D.C., Dec. 19, 2005).

^3Fiscal year 2004 was the last year for which award-fee contract data
were available in the Federal Procurement Data System when we began our
work.

^4The sample was 10 contracts that were the top 10 dollar value award-fee
contracts in terms of obligations active in that time period. The
estimated value of the 10 contracts totaled over $31 billion as of June
2006, and the contracts accounted for 44 percent of obligated CPAF dollars
for the 3-year period. Three of the contracts were for end items, while 7
were service contracts.

Results in Brief

NASA regulations and guidance on the use of cost-plus-award-fee contracts
address many of the issues and problems identified by NASA and the NASA
Inspector General and provide criteria for appropriately using award-fee
contracts. For example, NASA encourages tying fees to outcomes, prohibits
the rollover of unearned award fee, authorizes the use of interim fees on
end item contracts until final product delivery, and encourages the use of
performance-based contracts for the procurement of services and supplies.
Further, because of the cost and administrative burden associated with
managing this type of contract, the FAR and NASA's Award Fee Contracting
Guide recommend evaluating the costs and benefits of using a
cost-plus-award-fee contract before committing to this contract type.

NASA did not consistently implement key aspects of the agency's guidance
on major award-fee contracts that we reviewed. NASA's award-fee guide
states that while it is sometimes valuable to consider input factors when
evaluating contractor performance, it is NASA's preference to use
outcome-based criteria, and each of the contracts we reviewed had some
outcome based criteria. However, some criteria used to evaluate
performance were process or input factors, such as awarding portions of
the fee for the quality and effectiveness of the contractor's scheduling
system, and program planning and organizational management. Other
contracts used numerous subfactors for evaluating contractor performance,
which, according to NASA's award-fee guide, can dilute emphasis on any
specific criteria. For example, one contract specified 3 primary
performance evaluation factors, but included 96 subfactors for fiscal year
2004 and 108 subfactors for fiscal year 2005. Also, although the FAR and
NASA's award-fee guide specify consideration of the costs and benefits
before using a CPAF contract because of the cost and administrative burden
associated with these contracts, no analysis of costs and benefits was
conducted for the contracts we reviewed. Finally, NASA officials expressed
satisfaction with the results of the contracts we reviewed. NASA's
satisfaction was based on its evaluations of contractor performance
against criteria established in the award-fee plan. While NASA's
evaluations would indicate generally good contractor performance, such
performance did not always translate into desired program outcomes. That
disconnect raises questions as to the extent NASA is achieving the
effectiveness it sought through the establishment of guidance on award
fees. Specifically, our analysis showed that NASA paid significant amounts
of available fee on all of the contracts we reviewed, including those end
item contracts that did not deliver a capability within initial cost,
schedule, and performance parameters. For example, NASA paid the
contractor for the Earth Observing System Data and Information System Core
System 97 percent of the available award fee despite a delay in the
completion of the contract of more than 2 years and an increase in the
cost of the contract of more than 50 percent. Since revising its guidance,
NASA has not evaluated the effectiveness of award fees in achieving
program results and does not have metrics in place for measuring the
effectiveness of award fees.

We are recommending that NASA improve its current implementation of
cost-plus-award-fee contracts by reemphasizing its current guidance on
tying award-fee payments, particularly on major end item contracts, to
outcome factors and limiting the number of subfactors used in evaluations
and to use this contract type only when justified by a consideration of
costs and benefits. Further, we are recommending that NASA develop metrics
for measuring and a mechanism for evaluating the effectiveness of award
fees in achieving desired outcomes. In commenting on a draft of this
report, NASA concurred with all three recommendations.

Background

In January 2004, the President announced a new "Vision for Space
Exploration" calling for human and robotic missions to the Moon, Mars, and
beyond. Over the next two decades, NASA plans to spend over $100 billion
to develop a number of new capabilities, supporting technologies, and
facilities that are critical to enabling space exploration missions.
Development of the critical capabilities and technologies will be largely
dependent on NASA contractors, who constitute more than two-thirds of
NASA's workforce.^5 According to NASA officials, 87 percent of NASA's
$16.6 billion budget for fiscal year 2006 was spent on work performed by
its contractors.

Since 1990, we have designated NASA's contract management as a high-risk
area. This is based primarily on NASA's lack of a modern integrated
financial management system that can provide reliable information on
contract spending and performance as well as NASA's lack of emphasis on
end results, product performance, and cost control. For example, our most
recent high-risk report stated that while NASA has taken actions to
improve its contract management function, it continues to face
considerable challenges in implementing its contracts effectively.^6

5This workforce composition also includes NASA grantees.

NASA is organized under four mission directorates--Aeronautics Research,
Exploration, Science, and Space Operations--each of which covers a major
area of the agency's research and development efforts. The agency is
composed of NASA headquarters, 10 field centers, and the
contractor-operated Jet Propulsion Laboratory.^7

Cost-Plus-Award-Fee Contracts

NASA and other federal agencies can choose among numerous contract types
for acquiring goods and services that can differ in part according to the
nature of the fee that agencies offer to the contractor for achieving or
exceeding specified objectives or goals. According to the FAR, a CPAF
contract is appropriate to use when it is difficult to measure key
elements of performance. It is widely used to procure nonroutine services
such as the development of new systems. Typically, award-fee contracts
emphasize several aspects of contractor performance, such as schedule
performance, technical performance, and cost control. Because development
and administration of award-fee contracts involve substantially more
effort over the life of a contract than other types of contracts, the FAR
and NASA's Award Fee Contracting Guide specify that the expected benefits
of using an award-fee contract must exceed the additional administrative
effort and cost involved.^8

The theory behind CPAF contracts is that although the government assumes
most of the cost risk, it retains control over most or all of the
contractor's potential profit as leverage. On CPAF contracts, the award
fee is often the only source of potential fee for the contractor.
According to the NASA FAR Supplement and NASA's Award Fee Contracting
Guide^9, these contracts can include a base fee of anywhere from 0 to 3
percent of the estimated value of a nonservice contract. However, NASA's
regulations and guide do not allow the use of a base fee on service
contracts. Table 1 shows the percentage of award fee available on the
contracts we examined. (See app. II for a description of these contracts.)

^6GAO, High-Risk Series: An Update, [38]GAO-05-207 (Washington, D.C.:
January 2005).

^7NASA's field centers include Ames Research Center, Dryden Flight
Research Center, Glenn Research Center, Goddard Space Flight Center,
Johnson Space Center, Kennedy Space Center, Langley Research Center,
Marshall Space Flight Center, NASA Shared Services Center, Stennis Space
Center, and the Jet Propulsion Laboratory, which is a federally funded
research and development center and operated by the California Institute
of Technology.

^8FAR Part 16.404(b)(1) and 16.405-2(b)(1)(iii).

Table 1: Award Fees as a Negotiated Percentage of Contract Value

Contract                                              Award-fee percentage 
Jet Propulsion Laboratory^a                                            1.5 
International Space Station^b                                         11.0 
Consolidated Space Operations                                         10.0 
Joint Base Operation and Support                                       8.0 
Science, Engineering, Analysis, and Test                               6.0 
Engineering and Technical Support for Life Sciences                    5.0 
Program Information Systems Mission Services                           6.0 
Earth Observing System Data and Information System                     9.9 
Core System                                                                
Mechanical Systems Engineering Services                                9.2 
Landsat-7                                                              7.0 

Sources: NASA submissions to GAO and contract and award-fee documentation;
GAO (analysis).

Note: Half of the 10 contracts we reviewed also included other types of
fee or incentives including performance incentives and fixed fees.

^aAlthough this contract for the operation and management of the Jet
Propulsion Laboratory is a CPAF contract, it differs from the other
contracts we reviewed in that the contractor, the California Institute of
Technology (Caltech), is a nonprofit educational institution. According to
the NASA FAR Supplement 1815.404-471-6(a), it is NASA's policy not to pay
profit or fee on contracts with educational institutions. This contract is
an exception. According to NASA officials, the fee paid becomes part of
Caltech's investment in the institution's educational programs and the
infrastructure supporting the research efforts made by Caltech.

^bEleven percent is the maximum award fee available as negotiated in the
December 2003 International Space Station contract extension.

^9NASA's regulatory award-fee policy is found in the NASA FAR Supplement.
We refer to the NASA FAR Supplement as NASA's "regulations" throughout
this report. NASA's Award Fee Contracting Guide, dated June 2001, states
that the purpose of the guide is to explain and elaborate on NASA's
regulatory policy, providing examples and dealing with practical concerns
that could not be addressed in the NASA FAR Supplement.

NASA's Use of Cost-Plus-Award-Fee Contracts

NASA relies heavily on CPAF contracts. This contract type accounted for 48
percent of obligated contract dollars and 7.7 percent of contract actions
from fiscal years 2002 through 2004. By comparison, between fiscal years
1999 and 2003, award-fee contracts accounted for 13 percent of the
contract dollars and 3.4 percent of contract actions at the Department of
Defense (DOD). A CPAF contract includes an estimate of the total cost of
what is being contracted for, may include a fee with a possible base
amount fixed at the inception of the contract, and includes an award
amount that is intended to motivate excellence in contract performance.
The award fee is paid based upon the government's periodic judgmental
evaluations of contractor performance.

Award-Fee Criteria

When developing evaluation plans, NASA's award-fee guide indicates that
evaluation plans may include outcomes, outputs, inputs, or a combination
of these elements. NASA's guide expresses a preference for outcome
factors. It notes that while it is sometimes valuable to consider input
and output factors when evaluating contractor performance, outcome factors
are better indicators of success relative to the desired result.

           o An outcome factor is an assessment of the results of an activity
           compared to its intended purpose. Outcome-based factors are the
           least administratively burdensome type of performance evaluation
           factor, and should provide the best indicator of overall success.
           Outcome-based factors should therefore be the first type of
           evaluation factor considered for use, and are often ideal for
           nonroutine efforts.

           o An output factor is the tabulation, calculation, or recording of
           activity or effort and can be expressed in a quantitative or
           qualitative manner. Output factors may be more desirable for
           routine efforts, but are administratively more burdensome than
           outcome factors due to the tabulation, calculation, or recording
           requirements. When output factors are used, care should be taken
           to ensure that there is a logical connection between the reported
           measures and the program's mission, goals, and objectives.

           o Input factors refer to intermediate processes, procedures,
           actions, or techniques that are key elements influencing
           successful contract performance. These may include testing and
           other engineering processes and techniques; quality assurance and
           maintenance procedures; subcontracting plans; purchasing
           department management; and inventory, work assignment, and
           budgetary controls.

Evaluation of Award-Fee Contracts

For CPAF contracts, NASA personnel conduct periodic, typically semiannual
evaluations of contractor's performance against the criteria specified in
a performance evaluation plan. During the course of the evaluation period,
performance monitors track contractor performance, and once the period is
over they assess the performance and report to the performance evaluation
board (PEB). The PEB considers the reports as well as any other pertinent
information and prepares a report for the fee determination official (FDO)
with findings and recommendations. The contractor is given an opportunity
to provide a self-assessment of its performance during the evaluation
period, which is often a written report. The FDO may meet with the PEB to
discuss the report, after which a final determination is made in writing
as to the amount of fee to be paid. The FDO provides the determination to
the contracting officer and a copy of the related document to the
contractor.

Acquisition Environment Can Affect Acquisition Outcomes

When discussing award-fee contracts, it is important to acknowledge the
acquisition environment in which they are used. Award fees are intended to
motivate excellent contractor performance, which should result in
excellent program outcomes. However, award fees should not be used to make
up for factors internal or external to the acquisition environment that
hinder the success of acquisition outcomes. These factors may include
inadequate resources and financial management systems, lack of knowledge
prior to starting the acquisition, or unsound acquisition practices. We
have reported that in some cases, NASA's failure to define requirements
adequately and develop realistic cost estimates resulted in projects
costing more, taking longer, and achieving less than originally
planned.^10 The persistence of these problems in NASA contract management
is not only indicative of undisciplined processes or practices such as
these, but may also reflect the fact that the design, development, and
production of major space systems are extremely complex technical
processes that must operate within complex budget and political processes.
Even properly run programs can experience problems that may arise from
unknowns, such as technical obstacles and changes in circumstances. Only a
few things need to go wrong to cause major problems, and many things must
go right for a program to be successful.

^10GAO, NASA: Long-Term Commitment to and Investment in Space Exploration
Program Requires More Knowledge, [39]GAO-06-817R (Washington, D.C.: July
17, 2006).

NASA's Award-Fee Policy Addresses Many Cost-Plus-Award-Fee Contracting Issues
Identified as Problematic

The NASA FAR Supplement and NASA's Award Fee Contracting Guide address
many of the issues and problems identified by NASA on the use of award-fee
contracts and provide criteria for appropriately using such contracts.
Much of the guidance on award-fee contracting was issued in response to
weaknesses in CPAF contracting practices identified by NASA internal
reviews and NASA's Office of Inspector General in the early 1990s.^11
Those weaknesses included the awarding of excessive fees with limited
emphasis on acquisition outcomes (end results, product performance, and
cost control); rollover of unearned fee; use of base fee; and the failure
to use both positive and negative incentives. NASA updated its award-fee
guide in 1994, 1997, and 2001 to explain and elaborate on its award-fee
policy. The 2001 revision also reflects the FAR's additional emphasis on
using performance-based contracts.

NASA's award-fee guide emphasizes tying fees to outcome factors. The guide
states that outcome-based factors are the least administratively
burdensome type of evaluation factor and should provide the best indicator
of overall success. The award-fee guide warns against micromanaging
performance and diluting the emphasis of criteria by spreading the
potential award fee over a large number of performance evaluation factors.
Instead, the guide recommends selecting broad performance evaluation
factors, such as technical factors, project management, and cost control
supplemented by a limited number of subfactors under these factors.

Cost control is required to be a key performance evaluation factor in
award-fee performance evaluation plans, largely because of past
performance issues in which contractors were paid millions of dollars in
fees on contracts that were experiencing hundreds of millions of dollars
in cost overruns.^12 The NASA FAR Supplement states that cost control
shall be no less than 25 percent of the total weighted evaluation factors
when explicit evaluation factor weightings are used. The NASA FAR
Supplement states that emphasis on cost control should be balanced against
other performance requirement objectives, and the contractor should not be
incentivized to pursue cost control to the point that overall performance
is significantly degraded.

^11These weaknesses in NASA CPAF contracting practices were cited in our
1994 report, GAO, NASA Procurement: Challenges Remain in Implementing
Improvement Reforms, [40]GAO/NSIAD-94-179 (Washington, D.C., Aug. 18,
1994).

^12 [41]GAO/NSIAD-94-179 .

NASA's regulations prohibit rolling over unearned fee to subsequent
evaluation periods for service contracts. For such contracts, each interim
evaluation and the last evaluation are final. Another key element of the
current award-fee regulations is an increased emphasis on overall
contractor performance and the end product, rather than on incremental
progress. NASA requires conducting interim evaluations on end item
contracts until final product delivery to monitor performance prior to
contract completion and establish the basis for making interim payments.
At the end of the contract, a final evaluation is conducted and the
contractor's total performance is evaluated against the award-fee plan to
determine total earned award fee. For example, the contractor may be
evaluated and paid an interim fee once every 6 months until the product is
delivered. During the final evaluation, the contractor's performance is
evaluated to determine total earned award fee. The final evaluation may
result in the contractor retaining the fee previously awarded or receiving
additional or less fee than previously awarded and thus refunding a
portion of the fee to the government. The final evaluation provides NASA
the opportunity to make an award-fee decision based on actual quality,
total cost, and ability to meet the contract schedule at the point the
final product is delivered.

Further, under the award-fee policy in effect prior to the 1994 and
subsequent revisions to the guidance, base fee was allowed on all CPAF
contracts. NASA's current regulations prohibit the use of base fee on
service contracts and restrict the use of base fee on end item contracts,
such as for hardware. When base fee is used, it is not to exceed 3 percent
of estimated contract costs and it should only be paid if the final
award-fee evaluation is satisfactory or better. We note that base fee,
which was paid on two of the three end item contracts we reviewed, did not
exceed 3 percent, and none of the seven service contracts included base
fee.

Another issue addressed by NASA's regulations is the use of both positive
and negative performance incentives in its CPAF contracts. The NASA FAR
Supplement provides that award-fee contracts with primary deliverables of
hardware and with a total estimated cost and fee of greater than $25
million require both kinds of incentives based on measurements of hardware
performance against objective criteria.^13 Performance incentives are
separate and distinct from award fee and measure contractor performance up
to delivery and acceptance. Performance incentives are designed to reward
contractors when performance of delivered hardware is above minimum
contract requirements. For example, if the government establishes a
specified level of objective performance for a product that the contractor
exceeds, the contractor can be paid a performance incentive in addition to
the award fee already paid. If the contractor just meets this measure, it
cannot receive an additional performance incentive and keeps the award fee
already paid. If the contractor fails to meet the measure, however, it
must pay a negative performance incentive fee that reduces or eliminates
the entire award fee.

^13An exception is made for those awarded under the commercial item
procedures of FAR Part 12.

To address inconsistencies among NASA centers in how they evaluate
contractor performance, the current award-fee regulations also provide a
uniform rating system to be used for all NASA award-fee contracts. It
includes adjectival ratings as well as a numerical scoring system of
0-100. Scores of 61-70 percent are considered satisfactory, and the
regulations specify that contractors receiving a rating of less than 61
percent will not receive any fee. A contractor is not to be paid any base
fee or award fee for less than satisfactory overall performance.

NASA's award-fee guide encourages the use of performance-based contracts
for the procurement of services and supplies. The guide states that
constructing performance-based contracts that clearly define performance
requirements, include easily understood performance standards, and have an
objective incentive mechanism will result in contractors having a clearer
understanding of the government's expectations and will ultimately
facilitate enhanced contractor performance.

Finally, because of the cost and administrative burden associated with
administering award-fee contracts, the FAR and NASA's award-fee guide
specify consideration of the costs and benefits of using a CPAF contract
before committing to this contract type. Through an evaluation of the
administrative costs versus the expected benefits, the contracting officer
should be able to assess whether the benefits the government gains through
a CPAF contract will outweigh the additional costs of overseeing and
administering the contract. The award-fee guide provides an example of how
to calculate the administrative cost and states that benefits could be
measured in dollars saved through cost control or enhanced technical
capability.

NASA Has Not Always Followed the Preferred Approach Laid Out in Its Award- Fee
Guidance

Although the revisions in NASA's regulations and guidance on award-fee
contracts address many weaknesses previously identified, the contracts
that we reviewed did not always demonstrate use of award fees by the
centers in the way that NASA prefers as outlined in its guidance. Some
performance evaluation plans or reports included input evaluation factors,
which are not the best indicators of success relative to the desired
result, although they are allowed by the guidance. Other contracts
included numerous subcategories for evaluating the contractor that can
lessen the importance of any particular subcategory and reduce the
leverage of the award fee on any particular criterion. Also, although the
FAR and NASA's award-fee guide calls for a consideration of the costs and
benefits of using cost-plus-award-fee contracts because of the cost and
administrative burden involved, we found no examples of a documented
analysis of costs and benefits. Finally, NASA officials expressed
satisfaction with the results of the contracts based on their evaluations
of contractor performance against criteria established in the award-fee
plan. Those evaluations would indicate generally good performance.
However, that performance did not always translate into desired program
outcomes. NASA paid a majority of the available award fee on all of the
contracts we reviewed, including those end item contracts that did not
deliver a capability within initial cost, schedule, and performance
parameters.^14 That disconnect raises questions as to the extent NASA is
achieving the effectiveness it sought through the establishment of
guidance on the use of award fees. Further, NASA has not evaluated the
overall effectiveness of award fees in promoting program outcomes and does
not have metrics in place for measuring their effectiveness in achieving
program outcomes.

Some Evaluation Factors Were Not Outcome Factors as Preferred by NASA Guidance

Some performance evaluation subfactors included in performance evaluation
plans or reports were not outcome oriented. NASA's award-fee guide states
that while it is sometimes valuable to consider input and output factors
when evaluating contractor performance, it is NASA's preference when
feasible to tie fees to evaluation factors that are based on outcomes
because outcome-based factors provide the best indicator of overall
success. The award-fee guide recommends selecting broad performance
evaluation factors, such as technical factors, project management, and
cost control, and cautions that factors related to intermediate processes,
procedures, and actions may cause the contractor to divert its attention
from the overall desired outcome. The guide states that those types of
factors, while allowed, are not always true indicators of the contractor's
performance and should be relied on with caution. Further, with service
contracts, input factors may be of little or no value as a basis for
evaluation. While the contracts we reviewed generally used outcome factors
as part of the evaluation of performance, some supporting subfactors that
formed the basis of the ratings for performance measured compliance with
process or input factors that may not provide the best indicators of
success relative to the desired results.

^14 By initial cost, schedule, and performance parameters, we mean those
parameters agreed to at initial contract award.

For example, a part of the award fee on the Mechanical System Engineering
Services (MSES) contract was to be awarded for program and business
management performance. There were five subfactors under this primary
performance factor. Two of these subfactors, program planning and
organizational management and business management were input subfactors.
These two input subfactors measure contractor processes or inputs, but do
not focus on final results. Subfactors in the Landsat-7 contract included
input subfactors such as responsiveness of the contractor's corporate
management, quality and effectiveness of the contractor's scheduling
system, and prudent utilization of manpower and timely removal of manpower
upon completion of tasks.

Some Contracts Did Not Limit the Number of Subfactors for Evaluating Contractor
Performance as NASA Guidance Recommends

The NASA award-fee guide cautions that spreading the potential award fee
over a large number of performance evaluation factors dilutes emphasis on
any particular performance evaluation criterion, increases the prospect of
any one item being too small and thus overlooked, and increases the
administrative burden. It encourages broad performance evaluation factors
such as technical factors, project management, and cost control, which
should be supplemented by only a limited number of subfactors describing
significant evaluation elements over which the contractor has effective
management control. Our analysis showed that a large number of subfactors
were used to evaluate contractor performance for some contracts.

For example, the Jet Propulsion Laboratory (JPL) contract, which includes
both service and product deliverables defined in task orders under the
contract, uses three primary performance evaluation factors for measuring
contractor performance--programmatic, scientific, and engineering;
institutional management; and support to outreach initiative programs.
Although the JPL performance evaluation plan characterizes award-fee
subfactors as representing major areas of emphasis during the performance
period, the award-fee subfactors used to support the broad performance
evaluation factors were numerous--96 subfactors were used to evaluate the
contractor's performance in fiscal year 2004, and 108 subfactors were used
in fiscal year 2005.^15 The Engineering and Technical Support for Life
Sciences contract used three broad performance evaluation factors
also--technical performance, schedule performance and contract management,
and cost control--but evaluated the contractor on numerous supporting
subfactors identified as tasks or subtasks in the contractor performance
evaluation reports. For example, on one task order under this contract,
performance evaluation reports for various evaluation periods showed as
many as 50 different subtasks used to evaluate the contractor's
performance for the primary evaluation criteria: (1) technical performance
and (2) schedule performance and contract management.

The Landsat-7 contract also included a number of subfactors. Contractor
performance under this contract was evaluated in several different areas
each time the performance evaluation board met. Technical performance and
program management were grouped together in one primary performance
evaluation factor, and business management and cost performance were
grouped together in the other primary performance evaluation factor. There
were 9 subfactors under technical performance and 12 subfactors under
program management, including quality and effectiveness of the
contractor's scheduling system. Under business management and cost
performance, 17 evaluation subfactors and elements were to be considered,
including compliance with general contract provisions and clauses and
weekly scheduling of teleconferences to determine schedule status. In
addition to the number of subfactors that fell under the two primary
performance evaluation factors, there were nine additional evaluation
criteria, including resourcefulness, communication, and responsiveness.

NASA Did Not Perform Cost-Benefit Analyses

Although the FAR and NASA's award-fee guide require consideration of the
costs and benefits of using a CPAF contract before committing to this
contract type to determine whether the benefits outweigh the additional
cost and administrative burden of managing the contract, we found no
instances where a documented cost-benefit analysis had been done for any
of the contracts under our review. According to the guidance, since
award-fee contracts require additional administrative effort, they should
be used only when the contract values, performance period, and expected
benefits are sufficient to warrant that additional management effort.
Careful selection of the most appropriate contract type and careful
tailoring should prevent a situation in which the burden of administering
the award fee is out of proportion to the improvements expected in the
quality of the contractor's performance and in overall management. In
addition, CPAF contracts can be particularly costly and burdensome for
NASA to administer because of contract reporting and review requirements.
Major cost drivers include the number of award-fee periods, performance
monitors, and performance evaluation board members necessary for
implementing the award-fee process. For example, according to NASA's Award
Fee Contracting Guide's conservative estimate, it would cost about
$387,000 to administer the award-fee process over the life of a 5-year
contract. The guide notes that the estimate does not represent all
associated administrative cost that might arise. Although NASA procurement
officials acknowledged that formal cost-benefit analyses were not
prepared, some officials referred to determination and findings statements
or acquisition strategy meeting documents associated with specific
contracts as providing some evidence of consideration given to whether or
not CPAF contracts should be used.

^15 In comments on our draft report, NASA officials told us that the
number of subfactors used to evaluate the contractor's performance in
fiscal year 2006 was 57, and the number of subfactors planned for fiscal
year 2007 will be further reduced to 45.

Award-Fee Payments at Times Did Not Reflect Program Outcomes

While NASA officials expressed satisfaction with the results of the
contracts, in some cases there appeared to be a disconnect between the fee
paid and program results. NASA paid most of the available fee on all of
the contracts we reviewed--including on projects that showed cost
increases, schedule delays, and technical problems. The total estimated
value of the 10 contracts we reviewed was more than $31 billion. NASA paid
between 80 and 99 percent of the maximum award fee possible on these
contracts. The average was 90 percent, which equated to almost a billion
dollars in total award fees paid under the 10 contracts. Table 2 shows the
percentage of award fee paid for each of the 10 contracts we reviewed.

Table 2: Percentage of Available Award Fee Pool Paid on Contracts Examined

Contract                                              Award-fee percentage 
Jet Propulsion Laboratory^a                                             90 
International Space Station^a                                           92 
Consolidated Space Operations                                           80 
Joint Base Operation and Support^a                                      90 
Science, Engineering, Analysis, and Test                                80 
Engineering and Technical Support for Life Sciences                     91 
Program Information Systems Mission Services                            89 
Earth Observing System Data and Information System                      97 
Core System                                                                
Mechanical Systems Engineering Services                                 90 
Landsat-7                                                               99 

Sources: NASA contract and award-fee documentation; GAO (analysis).

Note: Numbers are rounded to the nearest whole percentage.

^aThese contracts were still active at the conclusion of our audit work.
The remaining 7 contracts were either closed or were in the process of
being closed.

NASA officials expressed satisfaction with contract results, which was
further evidenced by its evaluations of contractor performance against
criteria established in the award-fee plan. While NASA's evaluations would
indicate generally good performance, such performance did not always
translate into desired program outcomes. That disconnect raises questions
as to the extent NASA is achieving the effectiveness it sought through the
establishment of guidance on the use of award-fees. On the end item
contracts we reviewed, although there were some periods in which NASA paid
a lesser percentage of the available fee, NASA ultimately paid more than
90 percent of the available fee based on its evaluation of contractor
performance against criteria in the award-fee plan even when those
contracts did not deliver capability within initial cost, schedule, and
performance parameters. For example:

           o The prime contractor for the International Space Station (ISS)
           has received 92 percent of the total award fee available--$425.3
           million--although the cost increased by 131 percent, from $5.6
           billion to $13 billion, in part due to increased contract scope
           and delays caused by the Columbia accident, but also contractor
           cost overruns. In addition, the contractor estimates that it will
           incur an additional $76 million in overruns by the time the
           contract is completed. Further, the completion date for space
           station assembly under the prime contract was delayed by 8 years.
           In some cases these delays were caused by actions not within the
           control of the contractor, such as problems with the shuttle
           program and actions by the international partners.

           o The contractor for the Earth Observing System Data and
           Information System (EOSDIS) Core System (ECS) was paid 97 percent
           of the available award fee--$103.2 million--despite a delay in the
           completion of the contract by more than 2 years and an increase in
           the cost of the contract from $766 million to $1.2 billion.
           Technical problems, schedule delays, and cost control problems led
           to a major restructuring of the contract.

           o The Landsat-7 contractor was paid 99 percent of the available
           award fee or more than $17 million.^16 The original contract was
           managed by the Air Force but was subsequently transferred to NASA
           and rebaselined. The cost of the contract when transferred to NASA
           and rebaselined was $342.7 million. The Landsat-7 launch was
           delayed by 9 months and although the original scope of the work
           under the contract was significantly reduced^17, the cost of the
           contract increased. By the time the contract was complete, costs
           had risen 20 percent to $409.6 million.

While some NASA officials pointed out that problems encountered on these
contracts were at times outside the control of the contractor,
difficulties such as these with achieving program results have resulted in
NASA contract management being considered a high-risk area by GAO. We did
not review these contracts to determine responsibility for undesirable
results and therefore make no conclusion as to whether the fee paid was
appropriate on each particular contract. However, the high fees paid on
contracts where programs experienced disappointing results raise questions
as to the effectiveness of award fees as a tool for obtaining desired
program outcomes.

For the service contracts we reviewed, NASA officials reported that they
were satisfied with the results and quality of services provided. While we
could not assess these contracts against cost, schedule, and performance
outcomes as we could with the end item contracts, we did assess the
award-fee criteria used in these contracts against NASA guidance. Here we
found instances of process and input-oriented subfactors and the inclusion
of numerous subfactors in evaluating performance. Further, we found no
evidence that a cost-benefit analysis had been performed prior to choosing
the contract type. Taken together, this is not the preferred approach
according to NASA guidance, which raises questions as to the degree to
which performance outcomes--getting the quality of service desired--was
actually the basis for judging contractor performance and awarding fee.

^16The 99 percent of award fee paid on the Landsat-7 contract includes 6
years of on-orbit incentive fee, which is based on the satellite meeting
the government's requirements.

^17Under the original contract, managed by the Air Force, the contractor
was to assemble the satellite and its payload---the Enhanced Thematic
Mapper Plus (ETM+) instrument. The original estimated cost of the
contract, before it was transitioned to NASA and rebaselined, was $372.4
million. When the contract was transferred from the Air Force to NASA, the
subcontract for the ETM+ instrument was split into a separate prime
contract.

NASA Has Not Assessed the Effectiveness of Award Fees in Achieving Program
Outcomes or Developed Metrics for Conducting Such Evaluations

NASA views CPAF contracts as a viable and often preferred mechanism for
acquiring the types of goods and services that the agency procures. NASA's
satisfaction with the results of these contracts is evidenced by the level
of fee paid on all of the contracts we reviewed and is based on NASA's
evaluation of compliance with criteria contained in its award-fee plans.
However, the agency has not evaluated the overall effectiveness of award
fees in promoting desired outcomes. As noted, NASA developed its new
policies on award-fee contracts because the agency and its Office of
Inspector General found that it was paying excessive fees with limited
emphasis on acquisition outcomes. However, according to NASA officials,
the agency has not completed any assessments of the effectiveness of award
fees since the award-fee policy was restructured in the 1990s, nor has it
developed metrics or performance measures to conduct such evaluations.
Further, NASA lacks an agencywide system with the capability of compiling
and aggregating award-fee information and for identifying trends and
outcomes. According to NASA officials, even NASA's modern Integrated
Enterprise Management Program (IEMP) will not provide this capability.
Thus, NASA cannot meaningfully judge how well award fees are improving or
can improve contractor performance and program outcomes.

Conclusions

NASA could better link its award fees to desired results by making greater
use of outcome factors, its preferred criteria for evaluating award fee
contracts. While NASA has established policies and guidance that provide
an appropriate framework for their use, the agency has not always used
award fees as preferred by its guidance. To the extent that NASA uses
input evaluation factors and numerous subfactors for evaluating
performance, NASA may be diluting the leverage of award fees in achieving
desired results. Our review raises questions as to the extent NASA is
achieving the effectiveness it sought through the establishment of
guidance on the use of award fees. However, NASA has not evaluated the
overall effectiveness of its implementation of award fees.

Recommendations for Executive Action

We are making the following three recommendations to increase the
likelihood that the award fees NASA pays incentivize high performance from
its suppliers.

           o We recommend that the NASA Administrator reemphasize to the NASA
           centers the importance of tying award-fee criteria to desired
           outcomes and limiting the number of subfactors used in
           evaluations.

           o To ensure that cost-plus-award-fee contracts are used only when
           their benefits outweigh the costs, we recommend that the NASA
           Administrator direct the centers to consider costs and benefits in
           choosing this contract type by requiring documentation explaining
           how the perceived benefits will offset the additional cost
           associated with its administration as required by the FAR.

           o Finally, we recommend that the NASA Administrator require the
           development of metrics for measuring the effectiveness of award
           fees, establish a system for collecting data on the use of
           award-fee contracts, and regularly examine the effectiveness of
           award fees in achieving desired acquisition outcomes.

Agency Comments and Our Evaluation

In commenting on a draft of this report, NASA concurred with our
recommendations and indicated that it would reemphasize its current
guidance as recommended, address the issues raised by the report in
training, and cover those issues in its internal reviews of procurement
operations at the individual Space Centers. In terms of our recommendation
to develop metrics for measuring the effectiveness of award fees and
establish a system for collecting data on the use of award- fee contracts,
NASA concurred and indicated it would explore the best way to develop and
use metrics for evaluating the effectiveness of award fees and set up a
system for collecting data on award-fee contracts. NASA said it planned to
contact the Department of Defense to obtain information on its process,
since DOD is also developing such a data collection system and metrics for
measuring the effectiveness of award fees. NASA also provided technical
comments on the draft, which have been incorporated as appropriate.

As agreed with your office, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At
that time, we will send copies to interested congressional committees and
the NASA Administrator. We will also make copies available to others upon
request. In addition, the report will be available at no charge on the GAO
Web site at http://www.gao.gov .

If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or [email protected] . Contact
points for our Offices of Congressional Relations and Public Affairs may
be found on the last page of this report. Key contributors to this report
are acknowledged in appendix IV.

Ann M. Calvaresi-Barr Director Acquisition and Sourcing Management

Appendix I: Scope and Methodology

Scope and Methodology

Our objectives were to determine (1) the extent the National Aeronautics
and Space Administration's (NASA) guidance addresses the problems
previously identified with the use of award-fee contracts and (2) whether
NASA follows its guidance in using award fees to achieve desired outcomes.

We selected 10 NASA cost-plus-award-fee (CPAF) contracts to review. Our
selection was based on contract data from the Federal Procurement Data
System. We extracted information on all NASA contracts active between
fiscal years 2002 and 2004 that were coded as CPAF. To ensure the validity
of the database from which we drew our contracts, we confirmed the
contract type of each of the 10 contracts we selected through NASA
contracting officers and contract documentation. The contracts we selected
were the top 10 dollar value contracts active from fiscal years 2002
through 2004. These contracts account for about $7.6 billion, or 44
percent, of obligated cost-plus-award-fee-dollars for the 3-year period.

To determine the extent NASA's guidance addresses the problems previously
identified with the use of award-fee contracts and whether NASA follows
its guidance in using award fees to achieve desired outcomes, we
interviewed responsible program and procurement officials at NASA
headquarters and six NASA centers. We also reviewed the Federal
Acquisition Regulation (FAR), the NASA FAR Supplement, and NASA's Award
Fee Contracting Guide. We conducted a literature review and examined
previous reports, studies, and analyses done by GAO, NASA, the NASA
Inspector General, or others that included information related to NASA's
use of award fees and other relevant issues.

Additionally, we reviewed contract files, obtained information from
program and contracting officials through the use of a structured
questionnaire, and discussed the application of award-fee criteria with
NASA officials involved in the award-fee process. The contract documents
we reviewed contained information related to the development and
implementation of the award fee. This information included the basic
contract and statement of work, acquisition planning documents, award-fee
modifications, performance evaluation plan documentation describing fee
criteria for specific evaluation periods, contractor self-assessments,
performance evaluation board reports, and fee determination documents. We
used this information to corroborate and supplement the information
provided by NASA officials in response to structured questionnaires we
prepared and interviews we conducted. We e-mailed the questionnaires and
received written responses for all 10 of the contracts. We conducted
structured interviews with contracting and program officials concerning
the development, implementation, and effectiveness of the award-fee
structure for some of the contracts.

To accomplish our work, we visited NASA headquarters in Washington, D.C.
We also visited and held teleconferences with Goddard Space Flight Center
in Greenbelt, Maryland, responsible for managing 3 of the contracts we
reviewed; Johnson Space Center in Houston, Texas, responsible for managing
3 of the contracts; and Marshall Space Flight Center in Huntsville,
Alabama, responsible for managing 1 of the contracts. We held
teleconferences with officials at the Jet Propulsion Laboratory in
Pasadena, California; Kennedy Space Center in Cape Canaveral, Florida; and
Ames Research Center in Moffett Field, California, responsible for
managing 1 contract each under our review.

Our work was conducted from August 2005 through October 2006 in accordance
with generally accepted government auditing standards.

A NA Appendix II: Summary Description of the 10 NASA Contracts GAO
Reviewed

NAS5-60000-Earth Observing System Data and Information System Core
System-Goddard Space Flight Center

NAS5-60000 was an end item hardware cost-plus-award-fee contract between
NASA and Hughes Applied Information Systems Incorporation. Raytheon
Information Systems Company acquired Hughes in December 1999 and became
the prime contractor. The contract, currently closed, was managed by
Goddard Space Flight Center. The 10-year research and development contract
was awarded in March 1993 for the development and operation of the Earth
Observing System Data and Information System Core System. The period of
performance on the contract actually ended in April 2005, and the contract
has since been closed. According to Goddard Space Flight Center
procurement officials, the desired program outcome or objective of the
contract was to develop a technically capable system to process data from
NASA's satellites at a reasonable cost. Procurement officials stated that
the Earth Observing System Data and Information System Core System, a
state-of-the-art data-processing system, is currently dedicated to the
processing and dissemination of NASA Earth Science satellite data.

NAS15-10000-International Space Station-Johnson Space Center

NAS15-10000 is an end item hardware cost-plus-award-fee contract between
NASA and the Boeing Company. The contract, currently active, is managed by
the Johnson Space Center. A letter contract was awarded in November 1993
and was definitized in January 1995 as a cost-plus-incentive-fee award-fee
contract. In October 1999, during a restructuring of the contract, the
cost-plus-incentive-fee award-fee contract was converted to a
cost-plus-award-fee contract. The contract was extended in December 2003,
partially because of the Columbia accident. This planned 10-year contract
is for the design, development, manufacture, and on-orbit assembly of the
U.S. on-orbit segment of the International Space Station. The contract
also included provisions for a level of effort that included (1)
sustaining engineering, (2) multi-element integrated testing, (3)
logistics and maintenance-post production support, (4) technical
definition of contract changes, and (5) other engineering support.
According to Johnson Space Center procurement officials, the desired
program outcomes or objectives of this contract are (1) completion of the
U. S. on-orbit segment, delivery, and on-orbit acceptance of the space
station; (2) sustaining engineering of the U.S. on-orbit segment hardware
and software and common hardware and software provided to international
partners/participants and payloads; (3) post-production support of the
U.S. on-orbit segment hardware and common hardware provided to the
international partners/participants; and (4) space station end-to-end
subsystems management for the majority of the subsystems and specialty
engineering disciplines.

NAS5-32633-Landsat-7 Spacecraft-Goddard Space Flight Center

NAS5-32633 was an end item hardware cost-plus-award-fee contract between
NASA and Lockheed Martin Missiles and Space. The contract, currently
closed, was managed by Goddard Space Flight Center. The research and
development contract was initially awarded by the Air Force in October
1992 and transferred to NASA in May 1994. The contract was for the design,
development, fabrication, integration, test, and pre- and post-launch
support of the Landsat-7 spacecraft. Landsat-7 was launched in April 1999;
the contract was completed in 2005. The purpose of the Landsat-7 satellite
is to obtain continuous remotely sensed, high-resolution imagery of the
earth's surface for environmental monitoring, disaster assessment, land
use and regional planning, cartography, range management, and oil and
mineral exploration. According to Goddard Space Flight Center procurement
officials, the desired program outcome or objective of the contract was to
develop an operational satellite that met the science requirements of
users and the laws requiring the data be obtained at a reasonable cost.

NAS8-60000-Program Information Systems Mission Services-Marshall Space Flight
Center

NAS8-60000 was a cost-plus-award-fee service contract between NASA and
Computer Sciences Corporation. The contract, managed by the Marshall Space
Flight Center, was in the process of being closed as of June 2006. It was
awarded in May 1994, and covered a 2-year period of performance, but
included options to extend the period of performance for an additional 6
years--through April 30, 2002. The contract was extended three times, with
the period of performance ending on March 30, 2004. The primary purpose of
the contract was to provide services in the area of program information
system mission services. The contractor's responsibilities were to manage,
be responsible for, and provide information services to meet requirements
of the Information Systems Services Office and its customers. According to
Marshall Space Flight Center procurement officials, the desired program
outcome or objective of the contract was to provide services including
operating and maintaining existing equipment and software; gathering,
analyzing, defining, and documenting systems requirements; and planning,
designing, developing, acquiring, integrating, testing, and implementing
new systems or enhancements to existing systems.

NAS2-14263-Engineering and Technical Support for Life Sciences-Ames Research
Center

NAS2-14263 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin Engineering and Science Company, defined under task
orders. The contract, managed by Ames Research Center, was in the process
of being closed as of June 2006. Its period of performance ended in
September 2003. The 5-year research and development contract was awarded
in May 1995 for the provision of engineering and technical support
services for Ames Research Center life sciences. The work to be performed
included engineering and technical support for life sciences projects,
including space shuttle life sciences payloads, other life science
payloads, the Space Station Biological Research Project, ground-based life
sciences research, and advanced life support technology development.
According to Ames Research Center procurement officials, the desired
program outcome or objective of the contract was to achieve support for
space life science projects, life sciences research, and related
technology.

NAS9-19100-Science, Engineering, Analysis, and Test-Johnson Space Center

NAS9-19100 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin with indefinite delivery, indefinite quantity task orders;
performance-based; and level-of-effort provisions. Following the merger of
Lockheed and Martin in 1995, NASA consolidated two existing contracts to
form NAS9-19100 with an effective date of October 1, 1996. The contract,
managed by Johnson Space Center, was in the process of being closed as of
June 2006. The period of performance ended in January 2005. The contract
included requirements related to hardware, government-furnished crew
equipment, facilities, laboratory maintenance, life sciences, flight
hardware, and support for the science and engineering requirements of the
Space Shuttle Program and the International Space Station Program.
According to Johnson Space Center procurement officials, the desired
program outcomes or objectives of the contract were to provide engineering
and science support to all engineering directorates at Johnson Space
Center as well as support both the science and engineering requirements of
the shuttle and space station programs.

NAS9-98100-Consolidated Space Operations Contract- Johnson Space Center

NAS9-98100 was a cost-plus-award-fee service contract between NASA and
Lockheed Martin Space Operations Company, with task orders and
level-of-effort provisions. The contract, which was in the process of
being closed as of June 2006, was managed by the Johnson Space Center. It
was awarded on September 25, 1998, with a basic 5-year period of
performance and an option for an additional 5-year period. NASA chose not
to exercise the option for the second 5-year period of performance. The
contract required (1) developing an integrated operations approach to
spacecraft design, operations, and data processing that minimized cost and
the support infrastructure required to conduct space operations; (2)
obtaining a highly capable and accountable contractor that would be
responsible for providing space operations mission and data services; and
(3) providing a contract and management structure that would enable
outsourcing, commercialization, or privatization of some or all service
under the contract. According to Johnson Space Center procurement
officials, the desired program outcomes or objectives of the contract were
to (1) provide excellent quality and reliable mission and data services at
a significantly reduced cost; (2) move end-to-end mission and service
responsibility and accountability to industry; (3) implement an integrated
architecture that reduces overlap, eliminates unnecessary duplication, and
reduces life cycle costs; (4) define streamlined processes that minimize
intermediaries required to define requirements and deliver services; and
(5) adopt private sector commercial practices and services.

NAS10-99001-Joint Base Operation and Support-Kennedy Space Center

NAS10-99001 is a cost-plus-award-fee service contract between NASA and
Space Gateway Support. The contract, currently active, is managed by
Kennedy Space Center. The contract was awarded on October 1, 1998, for a
basic 5-year period of performance and included an option for an
additional 5 years. NASA exercised that option on October 1, 2003. The
purpose was to provide for base operations support at NASA's Kennedy Space
Center and the Air Force's Cape Canaveral Air Force Station, as well as
specific requirements at Patrick Air Force Base and Florida Annexes into
one consolidated contract. In addition to NASA and the Air Force, other
primary customers include the Navy, Department of Interior, Spaceport
Florida, and commercial customers such as Boeing, Lockheed Martin, Orbital
Science, and Astrotech. According to Kennedy Space Center procurement
officials, the desired program outcomes or objectives of the contract are
to (1) enhance safety for the public and on-site workforce; (2) provide
protection of human, national, and environmental resources; (3) provide
high-quality and responsive service to customers; (4) reduce the cost of
doing business for NASA and the Air Force; (5) provide flexibility to
respond to new requirements and unplanned events; (6) improve
supportability and reliability through innovative methodologies and
concepts; (7) provide common support practices and systems; and (8)
increase small business subcontracting goals.

NAS5-01090-Mechanical System Engineering Services-Goddard Space Flight Center

NAS5-01090 is a cost-plus-award-fee service contract between NASA and
Swales and Associates, with a line item for indefinite delivery,
indefinite quantity task orders. The contract, currently active, is
managed by Goddard Space Flight Center. NAS5-01090 was awarded in January
2001 with a period of performance of 5 years and 30 days. According to
Goddard Space Flight Center procurement officials, the period of
performance was extended and was currently scheduled to end on August 15,
2006. The purpose of the contract is to provide engineering services for
the study, design, development, fabrication, integration, testing,
verification, and operations of space flight and ground system hardware
and software, including development and validation of new technologies to
enable future science missions. According to Goddard Space Flight Center
procurement officials, the desired program outcomes or objectives of the
contract were to obtain high-quality performance, desired results, and
output.

NAS7-03001-Jet Propulsion Laboratory

NAS7-03001 is a cost-plus-award-fee contract between NASA and the
California Institute of Technology, a private nonprofit educational
institution, which establishes the relationship for the operation of the
Jet Propulsion Laboratory (JPL) federally funded research and development
center. The contract, currently active, is a 5-year research and
development contract awarded in November 2002 for the operation and
management of JPL. The contract allows for extension or decrease to the
initial period of performance in 3- or 9-month increments. JPL is a
NASA-owned facility as well as an operating division of Caltech. Caltech
has operated JPL as a federally funded research and development center
since 1959 to meet certain government research and development needs,
which, according to the contract, could not be met as effectively by
existing government resources or normal contractor relationships. The
contract includes both service and product deliverables, which are defined
in task orders issued under the contract. The contract encompasses a large
number of discrete programs and projects--approximately 500 active task
orders. According to NASA procurement officials, the desired program
outcomes or objectives of the contract are specific performance
requirements defined in task orders issued under the contract. The
contract encompasses support of exploration of the solar system, including
earth-based investigations, investigations and studies to support NASA
missions in the fields of earth science and astrophysics and astrobiology,
as well as development of supporting fundamental technologies.

Appendix III: Comments from the National Aeronautics and Space
Administration

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Ann M. Calvaresi-Barr, (202) 512-4841

Staff Acknowledgments

In addition to the individual named above, Thomas Denomme, Assistant
Director; James Beard; Shirley Johnson; Julia Kennon; Heather Barker
Miller; Kenneth Patton; Sylvia Schatz; and Robert Swierczek made key
contributions to this report.

(120481)

GAO's Mission

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone

The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Waste, and Abuse in Federal Programs

Contact:

Web site: [www.gao.gov/fraudnet/fraudnet.htm E-mail:
[email protected] Automated answering system: (800) 424-5454 or (202)
512-7470

Congressional Relations

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548

Public Affairs

Paul Anderson, Managing Director, [email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

www.gao.gov/cgi-bin/getrpt?GAO-07-58 .

To view the full product, including the scope
and methodology, click on the link above.

For more information, contact Ann M. Calvaresi-Barr, (202) 512-4841,
[email protected].

Highlights of [51]GAO-07-58 , a report to the Chairman, Committee on
Science, House of Representatives

January 2007

NASA PROCUREMENT

Use of Award Fees for Achieving Program Outcomes Should Be Improved

Cost-plus-award-fee contracts accounted for almost half of the National
Aeronautic and Space Administration's (NASA) obligated contract dollars
for fiscal years 2002-2004. Since 1990, we have identified NASA's contract
management as a high-risk area--in part because of a lack of emphasis on
end results. You asked us to examine (1) the extent NASA's guidance on
award fees addresses problems previously identified with the use of
award-fee contracts and (2) whether NASA follows its guidance in using
award fees to achieve desired outcomes. We reviewed the top 10 dollar
value award-fee contracts active from fiscal years 2002 through 2004.

[52]What GAO Recommends

We recommend NASA improve its current use of award fees by reemphasizing
tying award-fee payments to desired outcomes, limiting the number of
factors used in contractor evaluations as its guidance recommends, and by
using this contract type only when justified by a consideration of costs
and benefits. We also recommend that NASA develop metrics for measuring
the effectiveness of award fees, establish a system for collecting data on
the use of award-fee contracts, and regularly examine the effectiveness of
award fees in achieving desired outcomes. In commenting on a draft of this
report, NASA concurred with all three recommendations.

NASA guidance on the use of cost-plus-award-fee (CPAF) contracts provides
criteria to improve the effectiveness of award fees. For example, the
guidance emphasizes outcome factors that are good indicators of success in
achieving desired results, cautions against using numerous evaluation
factors, prohibits rollover of unearned fee, and encourages evaluating the
costs and benefits of such contracts before using this contract type.

However, NASA does notalways follow the preferred approach laid out in its
guidance. For example, some evaluation criteria contained input or process
factors, such as program planning and organizational management. Moreover,
some contracts included numerous supporting subfactors that may dilute
emphasis on any specific criteria. Although the Federal Acquisition
Regulation and NASA guidance require considering the costs and benefits of
choosing a CPAF contract, NASA did not perform such analyses.

In some cases there appears to be a significant disconnect between program
results and fees paid. For example, NASA paid the contractor for the Earth
Observing System Data and Information System Core System 97 percent of the
available award fee despite a delay in the completion of the contract by
over 2 years and an increase in the cost of the contract of more than 50
percent.

NASA officials expressed satisfaction with the results of the contracts we
reviewed, and this was further evidenced by the extent of fee paid. NASA's
satisfaction was based on its evaluations of contractor performance
against criteria established in the award-fee plan. While NASA's
evaluations would indicate generally good contractor performance, that
performance did not always translate into desired program outcomes. That
disconnect raises questions as to the extent NASA is achieving the
effectiveness it sought through the establishment of guidance on the use
of award fees. NASA has not evaluated the overall effectiveness of award
fees and does not have metrics in place for conducting such evaluations.

References

Visible links
  37. http://www.gao.gov/cgi-bin/getrpt?GAO-06-66
  38. http://www.gao.gov/cgi-bin/getrpt?GAO-05-207
  39. http://www.gao.gov/cgi-bin/getrpt?GAO-06-817R
  40. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-94-179
  41. http://www.gao.gov/cgi-bin/getrpt?GAO/NSIAD-94-179
  51. http://www.gao.gov/cgi-bin/getrpt?GAO-07-58
*** End of document. ***