Business Modernization: Disciplined Processes Needed to Better
Manage NASA's Integrated Financial Management Program (21-NOV-03,
GAO-04-118).
The National Aeronautics and Space Administration (NASA) has
struggled to implement a fully integrated financial management
system. The lack of such a system has affected the agency's
ability to control program costs, raising concerns about the
management of its most costly programs, including the space
shuttle program and the International Space Station. In April
2000 NASA initiated the Integrated Financial Management Program
(IFMP)--its third effort to improve the agencywide management of
its resources. Implementation is expected by fiscal year 2006
with an estimated life-cycle cost of nearly $1 billion. This
report (1) assesses NASA's methodology for preparing the current
life-cycle cost estimate for implementing IFMP, (2) determines
whether NASA's current schedule is reasonable, and (3) evaluates
NASA's processes for ensuring adequate cost contingencies.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-118
ACCNO: A08904
TITLE: Business Modernization: Disciplined Processes Needed to
Better Manage NASA's Integrated Financial Management Program
DATE: 11/21/2003
SUBJECT: Budgetary reserves
Cost control
Financial management systems
Future budget projections
Life cycle costs
Internal controls
Program management
NASA Integrated Financial Management
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-118
United States General Accounting Office
GAO Report to the Committee on Commerce, Science, and Transportation, U.S.
Senate, and the Committee on Science, House of Representatives
November 2003
BUSINESS MODERNIZATION
Disciplined Processes Needed to Better Manage NASA's Integrated Financial
Management Program
GAO-04-118
Highlights of GAO-04-118, a report to the Senate Committee on Commerce,
Science, and Transportation, and the House Committee on Science
The National Aeronautics and Space Administration (NASA) has struggled to
implement a fully integrated financial management system. The lack of such
a system has affected the agency's ability to control program costs,
raising concerns about the management of its most costly programs,
including the space shuttle program and the International Space Station.
In April 2000 NASA initiated the Integrated Financial Management Program
(IFMP)-its third effort to improve the agencywide management of its
resources. Implementation is expected by fiscal year 2006 with an
estimated life-cycle cost of nearly $1 billion.
This report (1) assesses NASA's methodology for preparing the current
life-cycle cost estimate for implementing IFMP, (2) determines whether
NASA's current schedule is reasonable, and (3) evaluates NASA's processes
for ensuring adequate cost contingencies.
GAO is recommending that IFMP follow best practices and NASA guidance in
preparing and updating the life-cycle cost estimate and establish
additional processes that would enable the agency to more accurately
estimate program cost and predict the impact of possible undesired events.
NASA concurred with GAO's recommendations for corrective action.
www.gao.gov/cgi-bin/getrpt?GAO-04-118.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact Allen Li at (202) 512-4841
or [email protected].
November 2003
BUSINESS MODERNIZATION
Disciplined Processes Needed to Better Manage NASA's Integrated Financial
Management Program
The uncertain reliability of cost estimates, optimistic schedules, and
insufficient processes for ensuring adequate funding reserves have put
NASA's latest financial management modernization effort at risk. Over the
past several years, IFMP's life-cycle cost estimates have fluctuated, and
NASA's current estimate is 14 percent greater than the previous estimate.
The reliability of these estimates is uncertain because disciplined
costestimating processes required by NASA and recognized as best practices
were not used in preparing them. For example, IFMP's current life-cycle
cost estimate did not include the full cost likely to be incurred during
the life of the program, including certain operations costs and costs to
retire the system. In addition, NASA did not consistently use breakdowns
of work in preparing the cost estimate, as recommended by NASA guidance.
In cases where work breakdowns were used, the agency did not always show
the connection between the work breakdown estimates and the official
program cost estimate. This has been a weakness since the inception.
Although more than half of the IFMP modules have been implemented-
including the Core Financial module, which is considered the backbone of
IFMP-the system may not be fully implemented by the end of fiscal year
2006 as planned. Efforts to complete the integrated system as quickly as
possible might have resulted in schedule margins that are insufficient to
manage program challenges-such as personnel shortages, uncertainties about
software availability, and Office of Management and Budget (OMB)
initiatives to implement electronic systems for agency business processes
governmentwide. These OMB initiatives have put IFMP in a reactive mode and
are already affecting planning for the payroll, procurement, and travel
components of the integrated system, which could result in additional
schedule delays and cost growth.
Finally, reserve funding for IFMP contingencies may be insufficient, which
is particularly problematic, given the program's unreliable cost estimates
and optimistic schedule. One module-Budget Formulation-is already
experiencing potential shortfalls in its reserves, and project officials
expressed concerns that the module's functionality may have to be reduced.
Yet the program continues to establish funding reserves based on reserve
levels set by other high-risk NASA programs, such as NASA's space flight
program-not on analyses of the potential cost impact of risks and unknowns
specific to IFMP, as required by NASA guidance. Moreover, the program did
not quantify the cost impact of high-criticality risks-also required by
NASA-or link its risks to funding reserves to help IFMP develop realistic
budget estimates.
Contents
Letter
Results in Brief Background Reliability of IFMP's Current Life-Cycle Cost
Estimate Is Uncertain
Owing to a Lack of Disciplined Processes Program Schedule May Be
Optimistic Processes Insufficient to Ensure Adequate Funding Set Aside
for Contingencies Conclusion Recommendations for Executive Action Agency
Comments and Our Evaluation Scope and Methodology 1
2 5
7 11
13 16 16 17 18
Appendix I Comments from the National Aeronautics and Space
Administration
Appendix II GAO Contact and Staff Acknowledgments
Tables
Table 1: IFMP's System Modules and Their Status 6 Table 2: IFMP's Cost
Estimates for Life Cycle 9 Table 3: IFMP's and NSSC's Share of Life-Cycle
Costs in the May
2003 Budget for Fiscal Years 1999-2010 9
Figure
Figure 1: IFMP's Life-Cycle Cost Estimate Trend
Abbreviations
GAO General Accounting Office IFMP Integrated Financial Management
Program NASA National Aeronautics and Space Administration NSSC NASA
Shared Services Center OMB Office of Management and Budget SAP System
Applications and Products SEI Software Engineering Institute WBS Work
Breakdown Structure
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States General Accounting Office Washington, DC 20548
November 21, 2003
The Honorable John McCain Chairman The Honorable Ernest F. Hollings
Ranking Minority Member Committee on Commerce, Science,
and Transportation United States Senate
The Honorable Sherwood L. Boehlert Chairman The Honorable Ralph M. Hall
Ranking Minority Member Committee on Science House of Representatives
About 90 percent of the National Aeronautics and Space Administration's
(NASA) annual budget is spent on contractors. Yet since 1990 we have
identified NASA's contract management as a high-risk area-in part,
because the agency has failed to implement a modern, fully integrated
financial management system. As we reported in January 2003, the lack of
such a system has hampered NASA's ability to oversee contracts; control
program costs; and ensure an effective human capital management
strategy, raising serious concerns about NASA's management of its largest
and most costly programs, including the space shuttle program and the
International Space Station.1
In April 2000 NASA initiated its third and most recent effort to implement
a modernized financial management system: the Integrated Financial
Management Program (IFMP).2 Through IFMP, NASA plans to employ multiple
software applications to improve the agencywide management of its
financial, physical, and human resources. NASA expects to complete IFMP's
implementation in fiscal year 2006 with an estimated life-cycle cost
1See U.S. General Accounting Office, Major Management Challenges and
Program Risks: National Aeronautics and Space Administration, GAO-03-114
(Washington, D.C.: January 2003).
2NASA abandoned two earlier efforts after spending about $180 million over
12 years.
totaling nearly $1 billion, including operations and maintenance costs.
However, several of the system's modules remain to be developed, and the
program faces significant challenges in completing them as planned. Given
its previous failures, NASA is under tremendous pressure to ensure that
IFMP is implemented quickly and can achieve its goals.
In April 2003 we issued an interim report on IFMP,3 which found that NASA
was not following key best practices for acquiring and implementing IFMP.
As agreed, we continued our review in three areas: (1) whether NASA is
acquiring and implementing IFMP in the context of an enterprise
architecture, (2) the extent to which the Core Financial module will
address NASA's external reporting requirements, and (3) NASA's life-cycle
cost estimate and schedule for IFMP. We are responding to the first two
issues in separate reports.4 This report addresses the third issue-IFMP's
life-cycle cost estimate and schedule. Specifically, you asked that we (1)
assess the reliability of NASA's methodology for preparing the current
cost estimate for implementing IFMP, (2) determine whether NASA's current
schedule is reasonable in terms of progress to date and available
resources, and (3) evaluate NASA's processes for ensuring the adequacy of
reserve funding for contingencies to mitigate the potential impact of
identified program risks and unknowns. In addition, we have summarized our
findings on the three areas previously cited in a separate report.5
The uncertain reliability of cost estimates, optimistic schedules, and
insufficient processes for ensuring the adequacy of funding reserves have
put NASA's latest financial management modernization effort at risk of
schedule delays and cost growth. The reliability of the current estimate-
which is 14 percent greater than the previous estimate established in
Results in Brief
3See U.S. General Accounting Office, Business Modernization: Improvements
Needed in Management of NASA's Integrated Financial Management Program,
GAO-03-507 (Washington, D.C.: Apr. 30, 2003).
4See U.S. General Accounting Office, Business Modernization: NASA's
Integrated Financial Management Program Does Not Fully Address Agency's
External Reporting Issues, GAO-04-151 (Washington, D.C.: Nov. 21, 2003).
Also, see U.S. General Accounting Office, Information Technology:
Architecture Needed to Guide NASA's Financial Management Modernization,
GAO-04-43 (Washington, D.C.: Nov. 21, 2003).
5See U.S. General Accounting Office, Business Modernization: NASA
Challenges in Managing Its Integrated Financial Management Program,
GAO-04-255 (Washington, D.C.: Nov. 21, 2003).
February 2002-is uncertain because disciplined cost-estimating processes
required by NASA and recognized as best practices were not used in
preparing the estimate. Specifically, IFMP's life-cycle cost6 estimate did
not include the full cost of all resources likely to be incurred during
the life of the program. In addition, NASA did not consistently use
breakdowns of the work to be performed in preparing the cost estimate, as
recommended by NASA guidance. Without using the work breakdowns to prepare
the cost estimate, NASA cannot ensure that all costs are accounted for. In
cases where work breakdowns were used, the agency did not always provide a
clear audit trail between the work breakdown estimates and the program's
life-cycle cost estimate.
IFMP is scheduled for completion at the end of fiscal year 2006. However,
efforts to complete the integrated system as quickly as possible might
have resulted in schedule margins that are insufficient to manage program
challenges-such as personnel shortages and uncertainties about software's
availability. In addition, Office of Management and Budget (OMB)
initiatives to implement electronic systems for agency business
processes---which are expected to improve agency operations
governmentwide-are affecting planning for the payroll, procurement, and
travel modules of the integrated system, which could result in additional
schedule delays and cost growth.
Finally, reserve funding for IFMP contingencies may be insufficient- which
is particularly problematic, given the uncertain reliability of cost
estimates and optimistic schedule for the program-because the program did
not consistently perform in-depth analyses of the potential cost impact of
risks and unknowns specific to IFMP, as required by NASA's guidance.
Instead, the program established funding reserves on the basis of reserve
levels set by other high-risk NASA programs. Moreover, the program did not
quantify the cost impact of identified risks or link its risks to funding
reserves.
To help decision makers better assess all costs associated with operating
and implementing IFMP, we are recommending that the program use current
processes dictated by best practices and NASA guidance for preparing and
updating the life-cycle cost estimate as well as establish
6Life-cycle cost is the total of the direct, indirect, recurring,
nonrecurring, and other related expenses incurred or estimated to be
incurred in the design, development, verification, production, operation,
maintenance, support, and retirement of a system over its planned life.
additional disciplined processes to better ensure that the agency more
accurately estimate program cost, predict the impact of possible undesired
events, and plan accordingly.
In written comments on a draft of this report, NASA's Deputy Administrator
concurred with our recommendations for corrective action but noted that
all IFMP projects completed to date stayed within budget, were completed
ahead of schedule, and delivered the committed scope. We do not agree that
this is the case for the Core Financial module, which NASA describes as
the "backbone" of the program. When NASA announced the implementation of
the Core Financial module at all of its centers, only about two-thirds of
the financial events needed for day-today financial operations and
external reporting had been implemented. In addition, we found that NASA
deferred the implementation of other key Core Financial module
capabilities and created new problems in recording certain financial
transactions.7 Thus, full functionality of the system has been deferred,
increasing the risk of additional costs and potentially affecting the
implementation of future modules. NASA's detailed comments also noted that
the program used a business case methodology and professional cost
estimators to perform cost and risk assessments. As discussed in this
report, the audit trail from these assessments to the program's cost
estimates required by NASA guidance and best practices is not clear.
NASA's detailed comments are included as appendix I.
7See GAO-04-151.
Background
For more than a decade, we have reported that the lack of a modern
integrated financial management system to produce accurate and reliable
information has hampered NASA's ability to oversee contracts and develop
good cost estimates for NASA's programs. In 1990 NASA's lack of effective
systems and processes for overseeing contractor's activities prompted us
to identify NASA's contract management as a high-risk area. In July 2002
we reported that the accuracy of NASA's $5 billion cost growth estimate
for the International Space Station was questionable and that the agency
might have difficulty preparing a reliable life-cycle cost estimate
because a modern integrated financial management system was not available
to track and maintain the data needed for estimating and controlling
costs. 8 NASA's lack of a fully integrated financial management system has
also hurt the agency's ability to collect, maintain, and report the full
cost of its projects and programs. For example, in March 2002 we testified
that NASA was unable to provide us with detailed support for the amounts
that it reported to the Congress as obligated against space station and
related shuttle program cost limits as required by the National
Aeronautics and Space Administration Authorization Act of 2000.9
IFMP is designed as an integrated system to replace the separate and
incompatible financial management systems used by NASA's 10 centers.10
According to the IFMP Program Director, the new system will provide better
decision data, consistent information across centers, and improved
functionality.
Unlike NASA's previous efforts to modernize its financial management
system, IFMP does not rely on a single contractor. NASA selected System
Applications and Products (SAP) to provide its "best of suite" software11
8See U.S. General Accounting Office, Space Station: Actions Under Way to
Manage Cost, but Significant Challenges Remain, GAO-02-735 (Washington,
D.C.: July 17, 2002).
9See U.S. General Accounting Office, National Aeronautics and Space
Administration: Leadership and Systems Needed to Effect Financial
Management Improvements, GAO-02-551T (Washington, D.C.: Mar. 20, 2002).
10NASA is composed of headquarters offices; nine centers located around
the country; and the Jet Propulsion Laboratory, which is operated by the
California Institute of Technology. For the purpose of this report, we
treat the Jet Propulsion Laboratory as a center.
11When acquiring and implementing commercial hardware and software
solutions, organizations can generally pursue one of two basic approaches:
An organization can opt for a single package of already integrated
software components, which is referred to as the "best of suite" approach,
or it can opt for different software components from different vendors,
which is referred to as the "best of breed" approach. "Best of suite"
components are easier and less costly to integrate.
and contracted for implementation services under a separate contract. NASA
has also broken the project into modules that will be implemented
individually-instead of all at once-on the basis of the availability of
proven commercial-off-the-shelf software products. IFMP initially
segmented implementation into 14 modules but has since reorganized the
program into 9. Some of these modules may be further broken out and others
added, depending on the scope of OMB's e-Government initiatives12 and
other considerations. Table 1 describes the modules that currently
comprise the system and their status.
Table 1: IFMP's System Modules and Their Status Module Description NASA-reported
status
Position Description Management
Web-based tool that enables the generation and classification of job
descriptions and automatic generation of associated documents.
Implemented September 2002.
Resume Management Web-based application that allows applicants to apply
for jobs online. Implemented November 2001. Erasmus Web-based system
providing financial performance information on NASA's
Implemented November 2002. programs and projects in a standardized format.
Travel Management Comprehensive system to streamline and unify the NASA
employee travel Implemented April 2003. system and to improve traveler and
vendor reimbursement.
Core Financial Accounting and financial reporting system that serves as
the "backbone" to Implemented June 2003. the integrated system.
Budget Formulation Web-based tool to formulate project, program,
institutional, enterprise, and Planned completion of agency-level budget
requirements. February 2004.
Integrated Asset System to manage NASA's physical To begin in late
assets through functions such as 2003.
Management physical inventory and financial
reporting.
Comprehensive tool to support
Contract Administration procurement, receiving, To begin in late
invoicing, and 2004.
payment of materials for NASA.
System allowing managers to fill
Human Resources positions with staff that possess To begin in late
the 2004.
appropriate skill sets and career
Management goals.
Source: NASA.
When NASA announced in June 2003 that the Core Financial module had been
implemented at all of its centers, only about two-thirds of the financial
events needed for day-to-day financial operations and external reporting
had been implemented. In addition, we found that NASA deferred
implementation of other key core financial module capabilities
12OMB's Electronic Government-or "e-Government"-initiatives advocate the
use of Internet-based technologies governmentwide for agency business
processes, such as payroll, travel management, and recruiting. The goal of
these initiatives is to eliminate redundant systems and improve the
government's quality of customer service.
and created new problems in recording certain financial transactions.13
Thus, full functionality of the system has been deferred, increasing the
risk of additional costs and potentially affecting the implementation of
future modules.
As we reported in April 2003, NASA is not following key best practices for
acquiring and implementing IFMP. For example, NASA has not analyzed the
interdependencies between selected and proposed IFMP components, and it
does not have a methodology for doing so. By acquiring IFMP components
without first understanding system component relationships, NASA has
increased its risk of implementing a system that will not optimize mission
performance and will cost more and take longer to implement than
necessary. In addition, in implementing the Core Financial module, NASA
faces risks in the areas of user needs and requirements management because
the agency did not consider the information needs of key system users and
is relying on a requirements management process that does not require the
documentation of detailed system requirements prior to system
implementation and testing.
The reliability of the current life-cycle cost estimate-which has
fluctuated since the initial estimate and is 14 percent greater than the
previous estimate established in February 2002-is uncertain because
disciplined cost-estimating processes required by NASA and recognized as
best practices were not used in preparing the estimate. Specifically,
IFMP's life-cycle cost estimate did not include the full cost likely to be
incurred during the life of the program. In addition, breakdowns of work
to be performed-or Work Breakdown Structure (WBS)14-were not consistently
used in preparing the cost estimate. In cases where work breakdowns were
used to prepare the estimate, the agency did not always provide a clear
audit trail. NASA has made some improvements in the program's financial
management, such as hiring personnel to provide oversight and consistency
for the cost-estimating process. However, until NASA uses more disciplined
processes such as breakdowns of work in preparing the program's cost
estimate, the reliability of the life-cycle cost estimate will be
uncertain and the program will have difficulty with controlling costs.
Reliability of IFMP's Current Life-Cycle Cost Estimate Is Uncertain Owing to
a Lack of Disciplined Processes
13See GAO-04-151.
14A WBS is a method of organizing a program into logical subdivisions at
lower and lower levels of detail.
IFMP's Life-Cycle Since the program began, cost estimates for IFMP's
10-year life cycle- fiscal years 2001 through 2010-have fluctuated and
increased overall, as shown in figure 1.
Cost Estimates Show
Overall Increase
Figure 1: IFMP's Life-Cycle Cost Estimate Trend
NASA's current IFMP life-cycle cost estimate totals $982.7 million-an
increase of $121.8 million, or 14 percent, over the previous IFMP
life-cycle cost estimate. The estimate comprises IFMP direct program
costs, NASA's enterprise support,15 and civil service salaries/benefits.
(See table 2.)
15NASA is organized into six strategic enterprises that function as
primary business areas for implementing NASA's mission. Each enterprise
draws on the capabilities of several NASA centers, while each center
contributes to multiple enterprises. For example, the Space Flight
Enterprise has oversight over NASA's human space flight program and
exercises management authority over the Johnson Space Center, Kennedy
Space Center, Marshall Space Flight Center, and Stennis Space Center.
Table 2: IFMP's Cost Estimates for Life Cycle
Dollars in millions of then-year dollars
IFMP's estimate component
Feb. 2002 estimate May 2003 estimate Change in dollars Percent change
Direct program $644.8 $635.3 $(9.5) (1.5)
Enterprise support 164.8 189.4 24.6
Civil Service salaries/benefits 51.3 158.0 106.7 208.0
Total life-cycle cost $860.9 $982.7 $121.8
Sources: NASA (data); GAO (analysis).
Although direct program costs decreased by $9.5 million, these costs were
shifted to the enterprise support component of the estimate with the
program's decision to fund only 1 year's worth of operations and
maintenance, rather than 2 years' worth from the direct program budget. In
addition, NASA anticipates that operations costs for fiscal years 2007
through 2010-estimated at $137.8 million-will be funded by the NASA Shared
Services Center (NSSC), a planned initiative to consolidate various agency
services such as purchasing and human resources. (See table 3.) As a
result, the fiscal year 2004 budget for the IFMP direct program portion of
implementing the system is $497.5 million.
Table 3: IFMP's and NSSC's Share of Life-Cycle Costs in the May 2003
Budget for Fiscal Years 1999-2010
Development and Operations and
implementation maintenance
IFMP estimate component (IFMP's share, FY (NSSC's share, FY Total
1999-2006) 2007-2010)
Direct program $497.5 $137.8 $635.3
Enterprise support 133.2 56.2 189.4
Civil Service salaries/benefits 125.3 32.7a 158.0
Total life-cycle cost $756.0 $226.7 $982.7
Source: NASA.
aThis number is not final and is still being reviewed by NASA.
In March 2003 an independent cost estimate team concluded that there is an
85 percent confidence level that the direct program portion can be
successfully completed with the available funding of $497.5 million.
However, the direct program portion represents only about half of the
total life-cycle cost estimate. In addition, the team's conclusion was
contingent on two optimistic assumptions: that there would be no schedule
disruptions and no increase in requirements.
Disciplined Processes Reflecting OMB guidance16 and the best practices of
government and Required by NASA Were industry leaders, NASA requires that
life-cycle cost estimates be prepared Not Used in Preparing on a full-cost
basis, that estimates be summarized according to the current IFMP's Cost
Estimates breakdown of work to be performed, and that major changes be
tracked to
the life-cycle cost. OMB guidance calls for a disciplined budget process
to
ensure that performance goals are met with the least risk and the lowest
life-cycle cost, which includes direct and indirect costs, operations and
maintenance, and disposal. The Software Engineering Institute (SEI)17
echoes the need for reliable cost-estimating processes in managing
software implementations-identifying tasks to be estimated, mapping the
estimates to the breakdown of work to be performed, and having a clear
audit trail are among SEI's requisites for producing reliable cost
estimates.
Despite NASA requirements and OMB and SEI guidance, IFMP did not prepare a
full life-cycle cost estimate-that is, all direct and indirect costs for
planning, procurement, operations and maintenance, and disposal were not
included. For example, the life-cycle cost estimate does not include the
following:
o the cost to operate and maintain the system beyond 2010;18
o the cost of retiring the system;
o enterprise travel costs, which are provided monthly by the NASA
centers; and
o the cost of nonleased NASA facilities for housing IFMP.
In addition, IFMP did not prepare WBS estimates for active modules-that
is, those currently being implemented. According to NASA guidance,
breaking down work into smaller units helps facilitate cost estimating and
project and contract management, and helps ensure that relevant costs are
not omitted. The guidance also states that the WBS should encompass both
in-house and contractor efforts. According to the IFMP Deputy Program
Director, WBS estimates are not prepared for active modules because
information such as contract task orders can be used to
16Planning, Budgeting, Acquisition, and Management of Capital Assets, OMB
Circular A-11, Part 7 (June 2002).
17SEI is a government-funded research organization that is widely
considered an authority on software implementations.
18NASA assumed a 10-year life cycle beginning in fiscal year 2001, but the
actual retirement date for the system is unknown, according to the Deputy
Program Director.
prepare the cost estimates. However, there is not one overriding contract
where each module is considered a deliverable at a fixed price. Rather,
numerous contracts at both the project and center level for a module's
implementation-many of which can be awarded for a level of effort at
agreed-upon fixed rates at various phases in the implementation. Without a
WBS estimate for the project as a whole, NASA cannot ensure that all
relevant contractor costs are included in the cost estimate. In addition,
using contract task orders to prepare the cost estimate would not ensure
that government in-house costs are included in the life-cycle cost
estimate.
Finally, for modules in the planning phase, the program utilized NASA's
subject matter experts and professional cost estimators to prepare
business case analyses. However, although these analyses contained WBS
cost estimates, the audit trail from the WBS estimate to the program's
lifecycle cost estimate was not always clear. Without a clear audit trail,
it is difficult to determine whether the differences between the detailed
WBS estimates and the official program cost estimate are appropriate. The
lack of a clear audit trail has been a weakness since the inception of the
program. For example, IFMP was unable to provide us with traceable support
for its baseline cost estimate for direct program costs.
NASA has made some improvements that should help the program prepare
better cost estimates. In May 2002 the NASA Administrator appointed an
executive to provide leadership and accountability in the direction and
operation of the system. The NASA headquarters program office also hired a
business manager to oversee and provide consistency for the
cost-estimating process and provide an analyst to review enterprise
support costs.
Although NASA guidance requires sufficient program schedule margins to
manage risks, efforts to complete the integrated system as quickly as
possible might have resulted in a schedule that is too compressed to
accommodate program challenges, such as personnel shortages and
uncertainties about software's availability. If the program schedule
margin is too compressed, the program could incur additional risks,
including added cost growth as well as failure to meet IFMP's schedule
objectives. OMB's e-Government initiatives-which aim to streamline agency
business processes and eliminate redundant systems governmentwide- could
also provide challenges for NASA's IFMP planning. As a result, the program
schedule may be optimistic.
Program Schedule May Be Optimistic
While implementing the Core Financial module (see table 1), IFMP has faced
human resource challenges, and the program continues to face these
challenges with other modules. For example, personnel shortages at
Marshall Space Flight Center for several months affected the Core
Financial project and other projects. In this case, a schedule slip was
avoided, but during fiscal year 2002, the shortages resulted in nearly
$400,000 for extra hours worked by center employees. Human resource
challenges are also affecting the Budget Formulation module. The
simultaneous implementation of this module with the Core Financial
module-an action advised against by a contractor conducting a
lessons-learned study-placed heavy demand on already scarce resources and
added complexity to the program. As a result the schedule for implementing
the Budget Formulation module has already slipped. Sometimes, relying more
on contractor personnel can alleviate shortfalls in civil service
personnel, but a recent Budget Formulation project status report indicated
that the implementation contractor might also have difficulties acquiring
and/or retaining qualified personnel. The implementation schedules for the
remaining modules overlap, putting the program at further risk of schedule
slippages.
Uncertainty regarding software availability also puts the program at risk
for completing the integrated system on schedule. For example, complete
software solutions and requirements for IFMP's Contract Administration
module have not yet been determined. Although contract-documentgeneration
software is available and tailored to meet the unique interface and
reporting requirements of the federal government,19 the "best of suite"
software solution-SAP-does not currently meet these requirements. NASA
faces the same challenge with IFMP's Human Resources Management module.
NASA's monthly status reports show that the program is working with SAP to
develop a software solution for the Human Resources Management module that
will meet federal government requirements, but the outcome is uncertain.
In addition, the program could adopt an e-Government solution for its
Human Resources Management module rather than the SAP solution.
19For example, according to the current business case analysis, NASA's
documentgeneration system would have to meet several federal requirements,
including providing the General Services Administration's Federal
Procurement Data System and the National Science Foundation's Federal
Assistance Awards Data System with data, along with reports to the
Department of Labor and the Small Business Administration.
Inserting e-Government solutions into IFMP planning-which calls for using
"best of suite" software-could create more difficult interface development
and a less-integrated system, thus interrupting the program's cost and
schedule. E-Government initiatives are already affecting NASA's planning
for the payroll, procurement, and travel modules in the integrated system.
For example, the payroll function, which was once part of the Human
Resources Management module, will likely become a separate module under
e-Government. Similarly, the Contract Administration module has been split
into two components: one for procurement document generation, for which
software is available although requirements are not finalized, and one for
the remainder of NASA's Contract Administration requirements, for which
requirements and software are currently unknown. Furthermore, e-Travel
could replace the Travel Management module, which has already been
implemented.
According to the program's fiscal year 2002 Independent Annual Review,
e-Government initiatives are forcing the program into a reactionary mode,
thus increasing risk to the program's success. The review specifically
noted that (1) the benefits of a fully integrated system could be lost
under e-Government, (2) the scope of IFMP and timing of future projects'
implementation have become uncertain, and (3) cost increases and schedule
slippage to accommodate directives may occur.
In addition to the uncertain reliability of IFMP's life-cycle cost
estimates and optimistic schedules, NASA cannot ensure that the funding
set aside for program contingencies is sufficient because the program did
not consistently perform in-depth analyses of the potential cost impact of
risks and unknowns specific to IFMP, as required by NASA guidance.
Moreover, the program did not quantify the cost impact of identified
risks, link its risks to funding reserves, or consistently set aside cost
contingencies for these risks.
Processes Insufficient to Ensure Adequate Funding Set Aside for Contingencies
In-Depth Analysis Not NASA guidance stipulates that programs incorporate
financial reserves, Performed in Establishing schedule margins, and
technical performance margins to provide the Cost Contingencies
flexibility needed to manage risks. According to the guidance, financial
reserves are to be established and maintained commensurate with
programmatic, technical, cost, and schedule risks. In other words, cost
contingencies should be tailored to the specific risks associated with a
particular program or project. In addition, NASA guidance suggests that
tools such as Probabilistic Risk Assessment20 can help in analyzing risk.
Although NASA's business case analyses include a risk assessment and
recommended reserve levels, we found no evidence that these recommended
levels were used in establishing the actual reserve levels for the IFMP
module projects. Regardless, the actual levels established did not match
the recommended levels in the business case analyses in most cases. We
found that reserves for some IFMP modules-both in the planning and active
phase-were based not on IFMP-specific risks but on reserve levels for
other high-risk NASA programs. For example, for a number of IFMP modules,
reserves were set at levels used for spacecraft implementations-typically
about 30 percent-because industry experience showed that large cost
overruns in system implementations such as IFMP are common. Yet it is
unclear whether this reserve margin is adequate for IFMP because the
effect of IFMP-specific risks and assumptions-such as uncertainties
relating to software, schedule, and OMB's e-Government initiatives-were
not analyzed. In addition, some of the enterprises supporting the module
projects described their method of establishing funding reserves as a
combination of rules of thumb and guesswork.
The Budget Formulation module has already experienced shortfalls in its
reserves, and project officials expressed concerns that the module's
functionality may have to be reduced. As of April 2003, the module had
expended its baseline reserves, which were established at about 20 percent
on the basis of the level of risk for space flight missions-not on the
risks specific to the module. Although the project was able to bring its
budget back into balance by obtaining an agreement with SAP to limit
overtime pay to time in excess of 50 hours per week, its remaining
reserves total only $83,000 to cover all contingencies-including those
that could require changes to the Budget Formulation module.
20Probabilistic risk assessment is a method of systematically examining
complex technical systems to measure both the likelihood that an undesired
event will occur and the consequences that will result.
Cost Impact of Identified Risks Not Quantified or Linked to Cost
Contingencies
NASA requires programs to quantify the cost impact of high-criticality
risks21 and to determine to what extent reserves may be exhausted, should
the risks become reality. According to SEI, estimating the potential cost
and schedule impact for all identified risks is an element of good
estimating practice. Quantifying the cost impact of identified risks and
clearly and consistently linking the risk database to funding reserves
helps programs develop realistic budget estimates.
While IFMP identifies program risks, analyzes their severity, and plans
mitigation actions, the program typically does not prepare a cost impact
analysis for identified risks nor does it consistently link identified
risks to funding reserves to ensure that funds are available, should the
risk occur. For example, in February 2003, the Travel Management Project
found that some components of the Travel Management module might not
satisfy individual centers, be funded, or be technically feasible.
However, the cost impact of this risk, as well as others, was not
quantified. Similarly, in June 2003, the Budget Formulation module did not
quantify the cost impact of a number of identified risks.22 Without
estimating the potential cost impact of these risks, NASA cannot determine
whether it has sufficient reserves to cover the risks-which is
particularly problematic for Budget Formulation, since virtually no
reserves remain for this module.
Furthermore, in its July 2003 monthly status report, the IFMP headquarters
office identified three high-criticality risks that could have a cost
impact on the overall program; however, no liens23 were set aside against
reserves for these risks:
o Reductions to out-year budgets could affect the implementation of
future integrated modules or the ongoing evolution of existing modules.
21Risk criticality is a function of the likelihood that an event will
occur and the severity of the consequences if the event does occur. The
criticality of each risk will be identified as low, medium, or high. Risks
with high criticality are also known as primary risks and typically have a
high likelihood of occurrence and a high magnitude of impact.
22A mitigation plan for the Budget Formulation module indicated that the
project was in the process of assessing the potential cost impact of four
of its six high-criticality risks, which the project manager confirmed.
23A lien is a potential cost to a project, direct or indirect, which may
or may not come to fruition, for which a portion of funding reserves is
set aside.
o An e-Government solution may be adopted for human resources management
rather than the IFMP solution, resulting in more difficult interface
development and a less-than-integrated solution.
o E-Government initiatives and policy decisions could disrupt IFMP
modules, resulting in delays or additional resource impacts.
An independent cost estimate team identified and quantified the impact of
two IFMP program risks, indicating that the cost and schedule impact of a
risk on a program or project can be sizeable. First, the team identified a
high-probability risk that NASA's "full cost requirement"-in which all
direct and indirect agency costs, including civil service personnel costs,
are tied to individual programs and projects-could affect the Budget
Formulation module.24 The team estimated this risk at $2 million to $3
million, with a potential schedule slip of 3 to 6 months. The Budget
Formulation Project is currently trying to determine what impact it may
have. The second risk identified by the independent cost review team- that
the Core Financial module may be transitioned to operations before all
integration points are addressed-could be more costly. The team estimated
this risk at $10.5 million to $20 million, also with a potential 3-to
6-month schedule slip. However, the team considered this risk as having a
low probability of occurrence.
NASA is at a critical juncture and faces major challenges in improving
contract management and controlling costs. These challenges seriously
Conclusion
affect the agency's ability to effectively manage its largest and most
costly programs. A modern integrated financial management system, as
envisioned in IFMP, is critical to ensuring that NASA has accurate and
reliable information to successfully meet these challenges. NASA has made
some improvements during the past year, such as hiring personnel to
provide the cost-estimating process with oversight and consistency.
However, if IFMP continues to ignore disciplined processes in estimating
program costs and impacts, it is unlikely that the program will meet its
goals.
To ensure that IFMP's life-cycle cost estimate conforms to NASA guidance
and best practices, we recommend that the NASA Administrator direct IFMP
to do the following:
Recommendations for
Executive Action
24NASA was to implement its full cost initiative October 1, 2003.
o Prepare cost estimates by the current Work Breakdown Structure for the
remaining modules.
o Provide a clear audit trail between detailed WBS estimates and the
program's cost estimate for the remaining modules.
o Prepare a full life-cycle cost estimate for the entire IFMP that meets
NASA's life-cycle cost and full cost guidance.
To ensure that contingencies are funded in accordance with NASA guidance
and best practices, we recommend that the NASA Administrator direct IFMP
to do the following:
o Utilize a systematic, logical, and comprehensive tool, such as
Probabilistic Risk Assessment, in establishing the level of financial
reserves for the remaining module projects and tailor the analysis to
risks specific to IFMP.
o Quantify the cost impact of at least all risks with a high likelihood
of occurrence and a high magnitude of impact to facilitate the continuing
analysis necessary to maintain adequate reserve levels.
o Establish a clear link between the program's risk database and
financial reserves.
Agency Comments and Our Evaluation
Although NASA concurred with our recommendations for corrective action,
NASA indicated that its current processes are adequate for (1) preparing
WBS cost estimates, (2) estimating life-cycle costs, and (3) establishing
reserves on the basis of IFMP-specific risks. The agency cited its
business case analyses as the methodology through which it is
accomplishing these tasks.
We disagree that NASA's current processes are adequate, and our
recommendations are aimed at improving these processes. As discussed in
this report, while NASA prepares WBS cost estimates for IFMP modules in
the planning phases by using business case analyses, it does not prepare
WBS cost estimates for active modules. And although IFMP indicates that
preparing cost estimates by using contract task orders is an appropriate
methodology, this approach will not ensure that all relevant costs,
including both contractor and government in-house costs, are included in
the life-cycle cost estimate. Regarding contract costs, there is not one
overriding contract where each module is considered a deliverable at a
fixed price. Rather, there are numerous contracts at both the project and
center level for implementing modules-many of which can be awarded for a
level of effort at agreed-upon fixed rates at various phases in the
implementation. Without a WBS estimate for the project as a whole, NASA
cannot ensure that all relevant contractor costs are included in the cost
estimate. In addition, using contract task orders to prepare the cost
estimate would not ensure that government in-house costs are included in
the life-cycle cost estimate.
According to NASA, IFMP will improve its business case analyses by
providing better estimates of operational costs through the expected life
of the module, retirement costs, and other full life-cycle costs. However,
as discussed in this report, an audit trail is needed between the detailed
estimates contained in the business case analyses and the program's
lifecycle cost estimate to ensure that these improvements are reflected in
the program's official cost estimate.
Finally, as discussed in this report, although NASA's business case
analyses include recommended reserve levels, we found no evidence that
these recommended levels were used in establishing the actual reserve
levels for the IFMP module projects. Regardless, the actual levels
established did not match the recommended levels in most cases. We found
that the program established funding reserves on the basis of reserve
levels set by other high-risk NASA programs, rather than on IFMPspecific
risks as required by NASA guidance.
To assess the reliability of NASA's methodology for preparing the current
cost estimate for IFMP, we reviewed program and project-level
documentation to obtain an understanding of NASA's current cost estimate
and its major components and the methodology used to develop the estimate.
We also interviewed program and project officials to clarify our
understanding of the cost estimate and how NASA derived it. In addition,
we compared the program's cost-estimating methodology with SEI best
practices, OMB requirements, and NASA's own procedures and guidance.
Finally, we reviewed internal and independent analyses of the cost
estimate. We did not attempt to validate NASA's estimate; rather, we
reviewed NASA's processes for preparing its estimate.
To determine whether NASA's current schedule is reasonable in terms of
progress to date and available resources, we reviewed the program's
schedule objectives and NASA's policies for managing program and project
schedules. We monitored the schedule and risks to the schedule through our
review of the program's monthly status reports and internal NASA
briefings. We interviewed program and project officials to ascertain
NASA's progress against the schedule.
Scope and Methodology
To evaluate NASA's processes for ensuring the adequacy of cost
contingencies to mitigate the potential impact of identified program risks
and unknowns, we reviewed governmentwide and NASA policies and SEI best
practices for managing risk and establishing cost contingencies. We also
interviewed program officials at NASA headquarters and project managers to
obtain an understanding of how reserve levels were established and
maintained for the program. We then compared IFMP's processes for ensuring
adequate cost contingencies with processes dictated by OMB and NASA
guidance and by best practices.
To accomplish our work, we visited NASA headquarters, Washington, D.C.;
Marshall Space Flight Center, Alabama; and Goddard Space Flight Center,
Maryland. We also contacted officials at Glenn Research Center, Ohio.
We performed our review from April through September 2003 in accordance
with generally accepted government auditing standards.
As agreed with your offices, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At
that time, we will send copies to interested congressional committees;
the NASA Administrator; and the Director, Office of Management and
Budget. We will make copies available to others upon request. In
addition, the report will be available at no charge on the GAO Web site
at http://www.gao.gov.
If you or your staff have any questions concerning this report, please
contact me at (202) 512-4841 or [email protected]. Key contributors to this
report are acknowledged in appendix I.
Allen Li Director Acquisition and Sourcing Management
Page 20 GAO-04-118 Business Modernization
Page 21 GAO-04-118 Business Modernization
Page 22 GAO-04-118 Business Modernization
Appendix II: GAO Contact and Staff Acknowledgments
GAO Contact Allen Li (202) 512-3600
Acknowledgments Staff making key contributions to this report were Jerry
Herley, Erin Schoening, LaTonya Miller, and Karen Sloan.
(120244)
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The General Accounting Office, the audit, evaluation and investigative arm
of Congress, exists to support Congress in meeting its constitutional
responsibilities and to help improve the performance and accountability of
the federal government for the American people. GAO examines the use of
public funds; evaluates federal programs and policies; and provides
analyses, recommendations, and other assistance to help Congress make
informed oversight, policy, and funding decisions. GAO's commitment to
good government is reflected in its core values of accountability,
integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud,Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Jeff Nelligan, Managing Director, [email protected] (202) 512-4800
Public Affairs U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***