Business Modernization: Improvements Needed in Management of
NASA's Integrated Financial Management Program (30-APR-03,
GAO-03-507).
The National Aeronautics and Space Administration's (NASA)
nonintegrated financial management systems have weakened its
ability to oversee its contractors, and its contract management
has been on GAO's high-risk list since 1990. In April 2000, NASA
began its Integrated Financial Management Program (IFMP), its
third attempt in recent years at modernizing financial management
processes and systems. GAO was asked to review whether NASA was
following key best practices in acquiring IFMP components and
implementing one of the first components--the core financial
module.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-03-507
ACCNO: A06781
TITLE: Business Modernization: Improvements Needed in Management
of NASA's Integrated Financial Management Program
DATE: 04/30/2003
SUBJECT: Computer software
Financial management
Financial management systems
Best practices
Information resources management
NASA Integrated Financial Management
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-03-507
A
Report to the Committee on Commerce, Science, and Transportation, U. S.
Senate, and the Committee on Science, House of Representatives
April 2003 BUSINESS MODERNIZATION Improvements Needed in Management of
NASA*s Integrated Financial Management Program
GAO- 03- 507
Letter 1 Results in Brief 3 Background 6 NASA*s Acquisition Management
Strategy Does Not Include
Analyzing Component Interdependencies 9 Core Financial Module Does Not
Fully Address Key User
Information Requirements 11 NASA*s Requirements Management Process for the
Core Financial
Module Is Ineffective 21 Conclusions 36 Recommendations for Executive
Action 37 Agency Comments and Our Evaluation 38
Appendixes
Appendix I: Objectives, Scope, and Methodology 47
Appendix II: Comments from the National Aeronautics and Space
Administration 51 GAO Comments 70
Appendix III: GAO Contacts and Staff Acknowledgments 73 GAO Contacts 73
Acknowledgments 73
Tables Table 1: Five Contractors and Their Responsibilities 8 Figures
Figure 1: Space Shuttle Flight Operations Contract 17
Figure 2: Example of Level of Detail Reported versus That Required by Cost
Estimators 19 Figure 3: System Requirements for the *Manage Accounts
Payable*
Process 27 Figure 4: Requirements for *Validate Payment* Subprocess 28
Figure 5: Relationship between Requirements Development and
Test i ng 30
Abbreviations
CSC Computer Services Corporation ERP Enterprise Resource Planning FFMIA
Federal Financial Management Improvement Act IBM International Business
Machines IEEE Institute of Electrical and Electronics Engineers IFMP
Integrated Financial Management Program IMCE International Space Station
Management and Cost Evaluation ISS International Space Station NASA
National Aeronautics and Space Administration OMB Office of Management and
Budget SEI Software Engineering Institute
This is a work of the U. S. Government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. It may contain
copyrighted graphics, images or other materials. Permission from the
copyright holder may be necessary should you wish to reproduce copyrighted
materials separately from GAO*s product.
Letter
April 30, 2003 The Honorable John McCain Chairman The Honorable Ernest
Hollings Ranking Minority Member Committee on Commerce, Science,
and Transportation United States Senate
The Honorable Sherwood L. Boehlert Chairman The Honorable Ralph Hall
Ranking Minority Member Committee on Science House of Representatives
Much of the National Aeronautics and Space Administration*s (NASA) success
depends on the work of its contractors* on which it spends $12.7 billion,
or 90 percent of its annual budget. For many years, NASA has not
effectively overseen its contracts, principally because it has lacked
accurate and reliable information on contract spending and performance and
it has placed insufficient emphasis on end results, product performance,
and cost control. Since 1990 we have identified NASA*s contract management
function as an area at high risk. 1 NASA*s ability to
collect, maintain, and analyze cost and performance data has been weakened
by nonintegrated, incompatible financial management systems and processes,
and uneven and nonstandard cost- reporting capabilities. NASA made two
efforts in the past to improve its financial management processes and
develop a supporting system intended to produce the kind of accurate and
reliable information needed to manage its contracts
effectively, but both of these efforts were eventually abandoned after a
total of 12 years and a reported $180 million in spending.
1 At that time, we began a special effort to review and report on the
federal program areas that our work had identified as high risk because of
vulnerabilities to waste, fraud, abuse, and mismanagement. We first issued
our High- Risk Series in December 1992 and have continued to include
NASA*s contract management as an area of high risk since. See U. S.
General Accounting Office, High- Risk Series: NASA Contract Management,
GAO/ HR- 93- 11 (Washington, D. C.: December 1992) and High- Risk Series:
NASA Contract Management,
GAO- 03- 119 (Washington, D. C.: January 2003).
In April 2000, NASA began its third attempt at modernizing its financial
management processes and systems. NASA has estimated the life cycle cost
of this effort through 2008 to be $861 million. 2 This effort, known as
the Integrated Financial Management Program (IFMP), is expected to produce
an integrated, NASA- wide financial management system through the
acquisition and incremental implementation of commercial software packages
and related hardware and software components. 3 Through the proven
business processes and centralized data management capabilities embedded
in these commercial components, NASA intends to reengineer its management
operations to *do business the way business does business.* The core
financial management module, which NASA considers to be the backbone of
IFMP, is currently operating at NASA headquarters and 6 of NASA*s 10
centers 4 and is expected to be fully operational in June 2003. According
to NASA*s business case analysis for the system, the core financial module
will provide NASA*s financial and program managers with timely,
consistent, and reliable cost and performance information for management
decisions.
Given the importance of IFMP to NASA*s mission performance, you asked us
to review the program. The purpose of this report is to alert you now to
concerns we have based on our work to date and to provide NASA management
with constructive recommendations for improvement that it can initiate as
soon as possible. We are continuing our work and plan to fully respond to
your request later this year.
Our work to date has focused on whether NASA has management processes in
place for effective system acquisition and implementation. This report
addresses three issues concerning the acquisition of IFMP components and
the implementation of one of the first components* the core financial
module. Specifically, we determined whether NASA (1) was
2 For this estimate, NASA has defined life cycle costs to include
implementation efforts through fiscal year 2008 and major upgrades, plus
operation and support costs for each system module for the first 2 years
after the module goes live.
3 The system is to consist of nine modules: core financial management,
resume management, travel management, position description management,
human resource management, payroll, budget formulation, contract
administration, and asset management. 4 NASA is comprised of its
headquarters offices, nine Centers located throughout the
country, and the Jet Propulsion Laboratory. The Jet Propulsion Laboratory
is operated by the California Institute of Technology, but for purposes of
this report, we treat the Jet Propulsion Laboratory as a center.
effectively evaluating the relationship among commercial systems component
options before acquiring them, (2) had adequately considered the
information needs of key users in implementing the core financial
module, and (3) had established and implemented an effective requirements
management process to support implementation of the core financial module.
We performed our work from April 2002 through February 2003 in accordance
with generally accepted government auditing standards. We had intended to
include our assessment of a key element of NASA*s acquisition strategy*
whether NASA was acquiring IFMP components in
the context of an agencywide blueprint, commonly called an enterprise
architecture* in this report. However, because NASA did not provide the
data needed to complete our assessment until after the conclusion of our
fieldwork, we plan to address NASA*s enterprise architecture in a future
product. Details on our objectives, scope, and methodology are in appendix
I.
Results in Brief If implemented as planned, IFMP may provide some
improvement to NASA*s current accounting system environment because it
should eliminate
the separate, incompatible systems that have previously been used at each
of NASA*s 10 centers and should result in standardized accounting data.
However, NASA is not following key best practices for acquiring and
implementing IFMP. Specifically, NASA has not established an analytical
capability to guide and constrain its acquisition of IFMP commercial
components. Further, in implementing the core financial module
component, NASA has deferred addressing the needs of key system users and
has not properly developed detailed system requirements. Consequently, the
agency is at risk of making a substantial investment in a system that will
fall far short of its stated goal of providing meaningful and reliable
information to support effective program management and congressional
oversight. NASA has not analyzed the interdependencies among selected and
proposed IFMP components, and it does not have a methodology for doing so.
For programs like IFMP, which involve building a system from commercial
components, it is essential to understand the characteristics and
credentials of each component in order to select ones that are compatible
and can be integrated without having to build and maintain expensive
interfaces. The alternative to such a structured and disciplined approach
to building a commercial component- based system is trial and
error, which is fraught with risk. Although NASA has already acquired the
core financial module and three other IFMP commercial components, the
agency has not performed the analysis necessary to understand the logical
and physical relationships among the component parts it has acquired. By
acquiring these IFMP components without first understanding system
component relationships, NASA has increased its risks of implementing a
system that will not optimize mission performance, and will cost more and
take longer to implement than necessary.
For the core financial module, NASA did not consider the information needs
of key system users and deferred addressing the requirements of program
managers, cost estimators, and the Congress. Since 1990, we have
identified NASA*s contract management function as an area at high risk, in
part because of the lack of effective systems and processes for managing
and overseeing its procurement dollars, producing credible cost estimates,
and providing the Congress with appropriate visibility over its large,
complex programs. However, despite these previous problems, program
managers, cost estimators, and congressional staffs were not included in
defining system requirements for NASA*s core financial module. Instead,
NASA*s financial managers and accountants have primary responsibility for
this process. In addition, little has been done to reengineer acquisition
management processes, particularly with respect to the consistency and
detail of budget and actual cost data provided by contractors. Although
capable of accepting the data needed to satisfy the information needs of
these key users, NASA*s new core financial module is not being
implemented to accommodate this information. According to IFMP program
officials, they chose to defer certain system capabilities and related
user requirements in order to expedite implementation of the core
financial module. As a result, program managers and cost estimators told
us that they will not rely on the core financial module and instead will
continue to rely on other systems or use other labor- intensive means to
capture the data they need to manage programs such as the International
Space Station (ISS). Further, NASA did not have an effective requirements
management process to support the implementation of the core financial
module. Specifically, NASA was relying on a systems requirements
management process that did not require documentation of detailed system
requirements prior to system implementation and testing. Although industry
best practices and NASA*s own system planning documents indicate that
detailed requirements are
needed to serve as the basis for effective system testing, NASA*s approach
instead relied on certain subject matter experts* knowledge of the
detailed
requirements necessary to evaluate the functionality actually provided. As
a result of this approach, our review of the core financial module
requirements found that, for many of them, (1) the functionality to be
delivered was not adequately described or stated in a manner that allowed
for quantitative evaluation and (2) the traceability between the various
requirement documents was not maintained. Accordingly, the potential for
these requirements defects to result in costly rework is significant and
increases the risk that the project will not meet its cost, schedule, and
performance objectives. Because of the direct relationship between
requirements and testing, the lack of complete and unambiguous
requirements also significantly impairs the testing phase of the system
implementation effort. For example, the core financial module could not
process vendor invoices that contained over 200 line items* a common
occurrence on NASA*s large contracts* because NASA did not design an
appropriate test case. If NASA had documented its requirements, it would
have recognized that a properly designed test case had not been developed
to cover this necessary functionality. Furthermore, NASA has not
effectively implemented the types of metrics that can help the
organization understand the effectiveness of its requirements management
process, such as identifying and quantifying any weaknesses and then
developing the corrective actions needed.
We are making recommendations that address the need for NASA to (1)
develop and implement a short- term plan to identify and mitigate the
risks currently associated with relying on already deployed IFMP
commercial components and to expeditiously stabilize these components*
operation capability and performance, (2) as part of the short- term plan,
develop and properly document requirements, reengineer acquisition
management processes, and fully engage stakeholders* including program
managers, cost estimators, and the Congress* in the development of user
requirements, and (3) develop a longer term strategy for acquiring
additional IFMP components that includes implementing a methodology for
commercial system component dependency analysis.
In written comments, which are reprinted in appendix II, NASA concurred
with the need for a short- term plan but disagreed with most of our
findings related to user needs and requirements and testing. We remain
convinced that, as we have stated, NASA needs to (1) reengineer its
acquisition
management processes to ensure that program and financial managers as well
as the Congress have needed budget and actual cost and schedule data and
(2) document detailed requirements to reduce, to acceptable levels, the
risks in implementing the selected processes. NASA also agreed with the
importance of having an approach for acquiring additional IFMP components,
but stated that it already has an effective strategy in place. We did not
find convincing evidence to support NASA*s contention that it is
using methodologically based dependency analysis* a best practice for
implementing commercial component- based systems* in acquiring IFMP.
Background NASA has a long and well- documented history of problems
overseeing its procurement dollars, producing credible cost estimates, and
providing the
Congress with appropriate visibility over its large, complex programs. We
first identified NASA*s contract management as an area at high risk in
1990 because NASA lacked effective systems and processes for overseeing
contractor activities. Over the past decade, other GAO, Inspector General,
and task force reports have shown that NASA*s cost estimates lack
credibility, in part because NASA does not collect the historical cost
data needed to accurately project future costs or assess the validity of
past estimates. Finally, because NASA had not provided the Congress with
adequate visibility over the ISS program, the Congress had little advance
warning when NASA reported that the estimated cost to complete the ISS had
grown by about $5 billion in 1 year.
Since we first identified NASA*s contract management as an area of high
risk, we have reported that one of NASA*s most formidable barriers to
sound contract management is the lack of a modern, integrated financial
management system. NASA*s ability to collect, maintain, and analyze cost
and performance data has been weakened by nonintegrated, incompatible
accounting systems and processes, and uneven and nonstandard costreporting
capabilities. The weaknesses in NASA*s financial management systems also
caused its independent auditor, PricewaterhouseCoopers, to conclude for
fiscal years 2001 and 2002 that NASA*s financial management systems do not
substantially comply with the requirements of the Federal Financial
Management Improvement Act (FFMIA). FFMIA builds on previous financial
management reform legislation by emphasizing the need for agencies to have
systems that can generate timely, accurate, and useful
information with which to make informed decisions and to ensure
accountability on an ongoing basis. While NASA*s efforts to design and
implement a new financial management system certainly move NASA forward in
this area, technology alone will not solve NASA*s problems. Our reviews,
as well as NASA*s, show that finance is not viewed as an integral part of
NASA*s program management decision process. Moreover, an independent task
force created by NASA to review
and assess ISS costs, budget, and management reached a similar conclusion.
In its November 1, 2001, report, the International Space Station
Management and Cost Evaluation (IMCE) Task Force found that the ISS
program office does not collect the historical cost data needed to project
future costs accurately and thus perform major program- level financial
forecasting and strategic planning. The task force also reported that
NASA*s ability to forecast and plan is weakened by diverse and often
incompatible
center- level accounting systems and uneven and non- standard cost
reporting capabilities. The IMCE Task Force also concluded that the
current weaknesses in financial reporting are a symptom, not a cause, of
the problem and that enhanced reporting capabilities, by way of a new
integrated financial management system, will not thoroughly solve the
problem. The root of the problem, according to the task force, is that
finance is not viewed as intrinsic to NASA*s program management decision
process.
NASA*s IFMP includes nine module projects supporting a range of financial,
administrative, and functional areas. According to NASA officials, of the
nine module projects, two are in operation, three are currently in
implementation, and four are future modules. The two projects in operation
are resume management and position description management; the three
projects in implementation are travel management, core financial, and
budget formulation; and the four future module projects are human
resources, payroll, asset management, and contract administration.
The core financial module project, which utilizes the SAP R/ 3 system, is
the backbone of IFMP and will become NASA*s standard, integrated
accounting system used agencywide. The other IFMP module projects will be
integrated/ interfaced with the core financial module, where applicable.
The scope of the core financial module, when fully implemented, includes:
standard general ledger, budget execution, purchasing, accounts
receivable, accounts payable, and cost management. NASA plans to implement
the core financial module at all 10 NASA centers by June 2003. The pilot
for the core financial module* conducted at Marshall Space Flight Center*
was implemented in October 2002. NASA is rolling out or deploying the core
financial module at the other nine NASA centers and headquarters in three
waves. The first wave, which consisted of Glenn Research Center, rolled
out in October 2002. The second wave, which consisted of NASA
headquarters, Johnson Space Center, Kennedy Space Center, and the Jet
Propulsion Laboratory rolled out in February 2003. Ames Research Center,
considered a second wave center, rolled out in April 2003. Finally, NASA
plans to roll out the third wave, which consists of
Dryden Flight Research Center, Goddard Space Flight Center, Langley
Research Center, and Stennis Space Center, in June 2003.
IFMP Acquisition NASA is contracting with multiple companies to assist in
the acquisition
Management Structure management, integration, and implementation of its
IFMP *system of
system components.* As shown in table 1, five contractors are assisting in
the integration and implementation of the core financial module. However,
none of these five contractors is responsible and accountable for
successfully implementing the entire IFMP system. Instead, NASA has
structured its IFMP acquisition so that NASA is the system integrator,
meaning that NASA is responsible for integrating multiple commercial
components and ensuring that they collectively perform in a manner that
meets the defined requirements.
Table 1: Five Contractors and Their Responsibilities Entity
Responsibility/ function
Accenture Implement the core financial module in accordance with agency
requirements, including interfacing it with NASA*s existing systems
environment. a
CSC (Computer Support the operations, maintenance, and administration of
the Services
new module, including integration efforts. Corporation)
IBM (International Develop training and user procedures and perform
security and Business Machines) internal control reviews to ensure that
the core financial module complies with accounting and financial reporting
standards. SAP Provide technical implementation support and training on
NASA*s
implementation of the core financial module. b Titan Systems
Perform independent verification and validation of requirements
Corporation- Civil
and testing processes and results, such as tracing requirements to
Government
test cases. Services Group
Source: NASA. a NASA plans to solicit additional contracts for
implementation of other selected and acquired modules.
b NASA has acquired SAP*s enterprise resource planning package and has
thus far planned to implement the core financial and budget formulation
modules. NASA has also acquired three other commercial software products*
Travel Manager, Resumix, and Position Description Management.
NASA*s Acquisition A key to effectively acquiring commercial component-
based systems that
Management Strategy are intended to support agencywide business needs,
like IFMP, is employing recognized acquisition management controls. One
such control
Does Not Include is to acquire system components only after deliberate and
comprehensive
Analyzing Component analysis and understanding of the components*
interdependencies.
Interdependencies Although NASA has already acquired the core financial
module and three
other IFMP commercial components, the agency has not performed the
analysis necessary to understand the logical and physical relationships
among the component parts it has acquired. By acquiring these IFMP
components without first understanding system component relationships,
NASA has increased its risks of implementing a system that will not
optimize mission performance and will cost more and take longer to
implement than necessary.
When acquiring a commercial component- based system or system of systems,
such as IFMP, industry best practices 5 recognize the critical importance
of understanding the logical and physical relationships among
the component parts. To provide for this understanding, these practices
advocate that the system integrator, which in the case of IFMP is NASA,
employ an explicit methodology, including a risk- based process for
deciding among product alternatives, that collects and verifies
information about each component*s characteristics and credentials,
evaluates the dependencies and constraints among these components, and
permits informed decisions about which products to acquire and how to
implement them. This is necessary because commercial products are built
around each vendor*s functional and architectural assumptions and
paradigms,
such as approaches to error handling and data access, and these
assumptions and paradigms are likely to be different among products from
different sources. Such differences complicate product integration.
Further, some commercial products have built- in dependencies with other
products that if not known can further complicate integration. For these
reasons, a structured and disciplined approach to systematically
evaluating product to product relationships is critical. The alternative
to such a
structured and disciplined approach to building a commercial
componentbased system is trial and error, which is fraught with risk.
5 See for example, Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge,
Ph. D., An Activity Framework for COTS- Based Systems, Technical Report
CMU/ SEI- 2000- TR- 010 (Pittsburgh, Pa.: Software Engineering Institute,
Carnegie Mellon University, October 2000).
In acquiring its IFMP components to date, NASA has not performed the
above- cited dependency analysis, and it does not have a methodology for
doing so. Despite this, NASA has acquired and is in the process of
implementing a commercial product (SAP*s R/ 3 core financial module) to
meet its needs in one business area (financial management), and it has
acquired three additional commercial products from three separate vendors
that are intended to meet its needs in other business areas (travel
management, resume management, and human capital position description
needs). 6 Beyond the four products that it has already acquired, NASA
plans to acquire an unspecified number of additional commercial components
that are intended to meet its needs in other business areas. To integrate
those separate commercial products into a *system of system components,*
NASA has executed several contracts and plans to execute more to build
interfaces (hardware and software) to permit the
components to interoperate. For example, a contractor is currently
building an interface between the core financial module of the SAP product
and the travel manager product. When acquiring and implementing commercial
hardware and software solutions, organizations can generally pursue one of
two basic courses of action. That is, an organization can opt for a single
package of already
integrated software components, which is referred to as the *best of
suite* approach, or it can opt for different software components from
different vendors, which is referred to as the *best of breed* approach.
NASA is currently following the *best of breed* approach. According to the
Integration Office Deputy Program Manager, NASA has not performed
dependency analyses among the various components acquired to date, and
those being considered for later acquisition, because NASA*s initial
acquisition strategy was to acquire a single commercial solution (i. e.,
*best of suite*) and thus it did not consider product interoperability to
be a concern. While NASA has since adopted a *best of breed* approach, the
Integration Office Deputy Program Manager stated that it still does not
plan to perform these analyses in the future because NASA will rely upon
commercial tools that support the development of interfaces between
commercial products, which the Integration Office Deputy Program
Manager claimed will make integration easy and relatively inexpensive and
negate the need for proactive dependency analysis. However, best
6 NASA has acquired the following commercial products: (1) SAP AG*s R/ 3,
version 4.62, (2) Gelco*s Travel Manager, version 8.0, (3) Resumix,
version 6, and (4) Avue Digital Services* Position Description Management,
which is a subscription service.
practices advocate that proactive dependency analysis and evaluation are
necessary for informed decision making regardless of whether integration
tools will be used, and particularly when a *best of breed* approach is
employed. What this means is that NASA is implementing its *best of breed*
approach using trial and error. This reactive method does not allow for
adequate understanding of commercial product dependencies until the only
alternative to integrating them is building and maintaining complex
interfaces, which unnecessarily increase system acquisition and
maintenance costs, delay promised capabilities and benefits, and do not
optimize agency performance. The results of a recent study 7 commissioned
by NASA recognize the added risk associated with the *best of breed*
approach, and thus the importance of proactive dependency analysis and
evaluation to minimize this risk. Specifically, the study states that
NASA*s *best of breed* approach will result in a higher total cost of
ownership
because the agency will need to (1) acquire and maintain multiple software
licenses, (2) hire and maintain technical staff knowledgeable about each
commercial product, (3) build and maintain interfaces to integrate the
various products, and (4) provide training to system users on each
commercial product.
Core Financial Module If implemented as planned, the core financial module
may improve NASA*s Does Not Fully
current system environment by eliminating the separate, incompatible
accounting systems that have been used at each of NASA*s 10 centers
Address Key User
previously. However, the core financial module currently being Information
implemented does not fully address the information requirements of key
users, such as program managers, cost estimators, or the Congress. Our
Requirements
previous work at leading public and private sector organizations 8 has
shown that user involvement and effectively reengineering business
processes are major factors in successfully implementing financial
management systems. In contrast, at NASA, key users such as program
managers and cost estimators were not involved in defining or implementing
NASA*s system requirements and have played a limited role 7 Gartner, Inc.,
A Report for NASA: IFMP Lessons Learned and Key Considerations for
Future Module Projects, January 20, 2003. 8 U. S. General Accounting
Office, Executive Guide: Creating Value Through World- class Financial
Management, GAO/ AIMD- 00- 134 (Washington, D. C.: April 2000).
in all aspects of the implementation of the core financial module.
Instead, NASA*s financial managers and accountants have primary
responsibility for this process. Consequently, NASA has not effectively
used this opportunity to reengineer the way it does business and implement
a financial management system that addresses many of its most significant
management challenges, including improving contract management, producing
credible cost estimates, and providing the Congress with
appropriate visibility over its large, complex programs. According to IFMP
officials, they chose to forgo certain system capabilities to expedite
implementation of the core financial module and have stated that these
capabilities can be added at a later date. In the meantime, program
managers and cost estimators will continue to rely on other nonintegrated
systems outside of IFMP and use other labor- intensive means to capture
the data they need to manage programs such as the ISS.
If Implemented as Planned, The core financial module, if implemented as
planned, may provide some
Core Financial Module May improvement to NASA*s current accounting system
environment.
Provide Some According to IFMP planning documents, the core financial
module should
Improvements (1) eliminate much of the inconsistent data and lack of
standardization,
(2) collect agency costs and allocate those costs to cost centers,
including civil service personnel costs, and (3) maintain a standard
general ledger to provide control over financial transactions, resource
balances, and assets and liabilities. If NASA is successful, the core
financial module could reduce the extensive amount of time and resources
currently required to consolidate NASA*s 10 different reporting entities
and close the books each
accounting period. However, as discussed later, our findings relating to
NASA*s requirements management and testing processes may affect NASA*s
ability to achieve these improvements.
Key Users Were Not The IFMP core financial module, although
technologically capable of
Involved in the meeting the needs of program managers, cost estimators,
and the
Implementation of the Core Congress, is not being configured to do so
because these key users have
Financial Module not been actively involved in the implementation of the
module. Our
previous work at leading public and private sector organizations has shown
that user involvement in reengineering business processes and establishing
and implementing system requirements are major factors in successfully
implementing financial management systems. 9 In fact, at these leading
organizations, not only did program and business managers participate in
the design and implementation of financial management systems, they
typically were responsible for driving the effort and played a key role in
reengineering core business processes. In contrast, at NASA, financial
managers and accountants have had primary responsibility for the
implementation of the core financial module, while the other key users
mentioned above have been largely excluded.
According to IFMP planning documents, financial managers and accountants
are considered direct customers responsible for the administrative
processes that will be reengineered and automated. Therefore, these
individuals, to date, have been engaged in defining system requirements
and priorities. On the other hand, stakeholders* including
program mangers, cost estimators, and the Congress* are described in NASA
documents as the ultimate beneficiaries of system improvements but are not
expected to be actively involved in the system*s implementation. While
NASA has formed teams to reengineer portions of the agency*s
administrative process, these teams primarily consisted of financial
managers. As a result, NASA has neither reengineered its core business
processes nor established adequate requirements of the system to address
many of its most significant management challenges, including improving
contract management, producing credible cost estimates, and providing the
Congress with appropriate visibility over its large, complex programs.
The Core Financial Module The core financial module is not being
implemented to provide program
Will Not Provide the managers with the information they need to fully
monitor the work being
Information Needed to performed by contractors. Based on our review of
NASA*s three largest
Manage Contracts 9 GAO/ AIMD- 00- 134.
space flight programs* Space Launch Initiative, 10 ISS, and Space Shuttle*
we found that the core financial module, as currently planned, will not
accommodate much of the information provided by NASA*s contractors and
needed by program managers to monitor contractor performance.
Specifically, the core financial module is not being implemented to
(1) accommodate the contract schedule information received from
contractors and needed by program managers to monitor contractor
performance and (2) maintain cost data at a sufficient level of detail for
certain contracts.
Core Financial Module Does Not To adequately oversee NASA*s largest
contracts, program managers need Integrate Cost and Schedule
reliable contract cost data* both budgeted and actual* and the ability to
Data Needed by Program integrate these data with contract schedule
information to monitor Managers
progress on the contract. However, because program managers were not
involved in defining system requirements or reengineering business
processes, the core financial module is not being designed to integrate
cost and schedule data needed by program managers. As a result, program
managers are resorting to using other systems that will result in
additional cost over and above IFMP costs.
The primary source of contract schedule information used by program
managers comes directly from NASA*s contractors in the form of monthly
hard copy or electronic cost and schedule performance reports. NASA tracks
contract schedule status by comparing the budgeted and actual cost of work
planned with budgeted and actual cost of work completed for specific time
periods. The term *schedule* incorporates both the concept of status of
work and whether a project or task is being completed within planned time
frames. Depending on the nature of the work being performed, the method of
measuring work progress varies. Work is measured in terms of tasks when a
specific end product or end result is
10 During the time of our review, NASA was pursuing a program* known as
the Space Launch Initiative* to build a new generation of space vehicles
to replace its aging space shuttle. This was part of NASA*s broader plan
for the future of space travel* known as NASA*s Integrated Space
Transportation Plan. On October 21, 2002, NASA postponed further
implementation of the program to focus on defining the Department of
Defense*s role, determining future requirements of the ISS, and
establishing the agency*s future space transportation needs. In November
2002, the administration submitted to the Congress an amendment to NASA*s
fiscal year 2003 budget request to implement a new Integrated Space
Transportation Plan. The new plan makes investments to extend the space
shuttle*s operational life and refocuses the Space Launch Initiative
program on developing an orbital space plane* which provides crew transfer
capability to and from the space station* and next generation launch
technology.
produced. But when work does not produce a specific end product or result,
level- of- effort or a more time- oriented method of measurement is used.
The type of information, level of detail, and reporting format provided by
contractors are determined during the contract negotiation process and
vary from contract to contract depending on the size, complexity, and
duration of the contract. In general, however, these reports show
contractor progress against cost and schedule targets set by the program
manager and against which contractor performance can be measured.
Contractors also report any significant variances from the targets and
explain how they will be mitigated.
NASA*s program managers need this contractor information to plan and
manage their programs effectively. However, the information from cost and
schedule performance reports is not recorded in the core financial module.
Instead, NASA uses only data from monthly contractor financial management
reports, commonly referred to as NASA form 533 reports, to update the core
financial module. NASA form 533 reports contain estimated and actual
contractor cost data but, according to NASA program managers, do not
contain the data needed to adequately assess schedule performance.
According to IFMP officials, the information needed to perform cost and
schedule analysis by program managers is outside the scope of the core
financial module and IFMP. IFMP program officials told us that they chose
to forgo certain system capabilities to expedite implementation of the
core financial module and have stated that these capabilities can be added
at a later date. However, NASA does not currently have a plan for
maintaining the data contained in cost and schedule performance reports in
the core financial module or IFMP.
Because contract schedule information is not currently maintained through
the core financial module, program managers will continue to rely on hard
copy reports, electronic spreadsheets, or other means to monitor
contractor performance. Several of NASA*s programs, including the Space
Launch Initiative and the ISS, are currently using other systems to
monitor contract cost and schedule performance, but these are stand- alone
efforts and have not been part of a coordinated NASA plan. Officials at
Marshall Space Flight Center have recognized the importance of maintaining
common cost and schedule performance data in a single integrated system
that is available to all NASA managers at all locations. As such, these
officials have proposed that NASA establish the system they currently use
as a NASA- wide standard.
NASA has stated that the core financial module is expected to result in a
single, integrated financial management system that is intended to serve
the needs of its program managers. By not including the cost and schedule
information needed by program managers in the core financial module, NASA
risks operating with two sets of books* one that is used to report to
management and the Congress and another that is used to manage NASA*s
programs.
Core Financial Module Will Rely Because NASA has not fundamentally changed
the way it operates by
on Legacy Coding Structure involving key users in business process
reengineering efforts, the core financial module is not being implemented
to capture cost information at
the same level of detail that it is received from NASA*s contractors.
Instead of implementing an accounting code structure that would meet the
information needs of program managers, NASA has embedded the same
accounting code structure that it uses in its legacy reporting system in
the core financial module. As a result, the availability of detailed cost
data is dependent on the adequacy of NASA*s legacy coding structure. In
some cases, the cost information received by NASA on monthly contractor
reports must be aggregated to a higher, less detailed level before it is
posted against the old accounting code structure. For example, as shown in
figure 1, program managers for the Space Shuttle receive monthly
contractor reports on the space flight operations contract that track
costs related to friction stir weld and propulsion safety upgrades
separately.
Figure 1: Space Shuttle Flight Operations Contract
Flight Space Shuttle
Hardware Flight Operations
$100 $100
Level captured
Flight hardware Solid rocket
in the core
upgrades booster
financial module
$40 $60 Data to
IFMP
Level received
Flight hardware Friction stir weld
Propulsion
from contractors
safety upgrades upgrades
upgrades $40
$10 $10
Source: GAO analysis of NASA cost data. Note: Amounts shown are for
illustrative purposes only.
However, because the NASA legacy accounting code structure embedded in the
core financial module only tracks the cost of space shuttle flight
hardware upgrades, the more detailed costs that program managers need,
such as friction stir weld and propulsion safety upgrades, are not
available through the core financial module. According to NASA officials,
the core financial module is capable of capturing this more detailed cost
data; however, due to the complexity associated with converting detailed
data from the centers* legacy systems, NASA has deferred this capability.
While this information is available to program managers from the
contractor, it is not available through the core financial module. In
fact, on this particular contract, program managers have access to the
contractor*s system and, therefore, have access to an even greater level
of detail than that reported by the contractor on hard copy reports.
On the other hand, in cases where the legacy coding structure adequately
reflects the programs* information needs, the cost data received from
contractors do not have to be aggregated prior to posting. For example,
program officials with the ISS program recently redesigned the program*s
cost coding structure in order to more precisely identify the cost of
specific work. This was not done as part of an IFMP reengineering effort,
but in response to external criticisms of the program*s failure to manage
its costs. Regardless of the reason, the program*s reengineering effort
has to some
extent improved the usefulness of the cost data being entered into the
core financial module.
Core Financial Module Will The core financial module, as currently
planned, will not provide Not Provide the Information
sufficiently detailed data for cost estimators. Although the core
financial Needed to Prepare Credible
module is technologically capable of maintaining the detailed data
required Cost Estimates
by cost estimators, cost estimators were not involved in defining the
system requirements or reengineering business processes. As a result, NASA
has not determined the most cost- effective way to satisfy the information
needs of its cost estimators nor reengineered its business process to
ensure that their needs are met.
According to members of NASA*s cost estimating community, they typically
need cost data at an even greater level of detail than that currently
being provided by NASA*s contractors. The cost estimators we spoke with
told us that their requests for more detailed cost data are often not
satisfied through the contract negotiation process. For example, as shown
in figure 2, while program managers may want* and contractors may provide*
the cost of an engine fan, cost estimators need to know more detailed
information, including the cost of the various tasks needed to make a
rotor assembly, which ultimately becomes part of the fan.
Figure 2: Example of Level of Detail Reported versus That Required by Cost
Estimators
Contract Engine
Auxiliary power
Fan Compressor Turbine
Level of detail Dual spool often provided
Fan assembly Full scale fan rig Minor fan rig compressor rig
by contractors Case assembly Rotor assembly
Level of detail required by cost
Work packages Tas k 1 estimators
Tas k 2 Tas k 3
Source: NASA Work Breakdown Structure Reference Guide.
The lack of sufficiently detailed information for cost estimators is due
to NASA*s lack of reengineering efforts for the acquisition management
process, which should have been done prior to implementing the core
financial module. Because the core financial module will not contain
sufficiently detailed historical cost data necessary for projecting future
costs, cost estimators will continue to rely on labor- intensive data
collection efforts after a program is completed. These efforts involve
searching through old hard copy and electronic contractor reports to
extract all relevant data. NASA pays its contractors extra to provide data
required but not contained in these reports, usually at a later point in
time.
Data collection after the fact is expensive but, according to some NASA
officials, is more cost effective than requiring contractors to provide
detailed cost data throughout the course of the contract. However, NASA
has not done the analysis needed to determine the appropriate mix of
routinely requiring contractors to provide detailed cost data and
capturing
that data in the core financial module versus purchasing the data after a
contract is complete.
Core Financial Module May NASA has identified the Congress as a key
stakeholder and ultimate
Not Provide Better beneficiary of system improvements. However, based on
our discussions
Information for with congressional staffs from NASA*s authorizing
committees, the agency
Congressional Oversight did not consult with them regarding their
information needs. Consequently, NASA cannot be sure that it is
implementing a system that will provide the
Congress with the information it needs for oversight. As discussed
previously, according to IFMP planning documents, financial managers and
accountants are considered direct customers and are responsible for
defining system requirements and priorities. On the other hand, NASA
considered the Congress a stakeholder and, therefore, did not seek input
from congressional staffs in defining system requirements.
Similar to the problems faced by program managers and cost estimators, the
core financial module may not address many of the information needs of the
Congress. To properly assess the agency*s annual budget submission and
make funding decisions, the Congress needs timely, reliable cost and
schedule information on the status of large, high- risk programs, such as
the ISS. As previously described, the module will not provide the type of
cost and schedule information that program managers need to adequately
monitor the status of NASA*s major programs and may not maintain
sufficient information to readily address any special congressional needs
that arise. Nevertheless, the Congress should be able to receive somewhat
better
information about NASA*s finances than it has in the past because, as
previously described, the core financial module may improve some aspects
of NASA*s ability to produce reliable financial information. For example,
the use of a standard general ledger will provide more standardized
accounting data and general ledger controls. As a result, the core
financial module should enable NASA to provide timelier, more reliable
high- level cost information to the Congress on some issues, such as
annual spending limits.
NASA*s Requirements NASA has not effectively implemented a requirements
management
process 11 to support the implementation of the core financial module and
Management Process
therefore has increased the risk that the agency will not be able to for
the Core Financial
effectively identify and manage the detailed system requirements that
Module Is Ineffective system developers and program managers use to
acquire, implement, and test a system. Specifically, based on discussions
with IFMP officials and a review of the process documents related to the
core financial module, we found that NASA was relying on a requirements
management process that did not require detailed documentation of system
requirements prior to system testing. Industry best practices, as well as
NASA*s own system planning documents, indicate that detailed system
requirements should be documented to serve as the basis for effective
system testing. Instead, NASA*s approach relied on the expertise of
certain subject matter experts to remember the detailed requirements
necessary to evaluate the functionality actually provided.
As a result of this approach, we found that (1) for over 80 percent of the
132 core financial module requirements we reviewed, the functionality to
be delivered was not adequately described or stated in a manner that
allowed
for quantitative evaluation and (2) the traceability among the various
requirement documents was not maintained. Accordingly, the potential for
these requirements defects to result in costly rework is significant and
increases the risk that the core financial module will not meet its cost,
schedule, and performance objectives. Because of the direct relationship
between requirements and testing, the lack of complete and unambiguous
requirements also significantly impairs the testing phase. Furthermore,
NASA has not effectively implemented the types of metrics that can help it
understand the effectiveness of its requirements management process,
such as identifying and quantifying any weaknesses in its process and then
developing the corrective actions needed.
11 According to the Software Engineering Institute, requirements
management is a process that establishes a common understanding between
the customer and the software project manager regarding the customer*s
business needs that will be addressed by a project. A critical part of
this process is to ensure that the requirements development portion of the
effort documents, at a sufficient level of detail, the problems that need
to be solved and the objectives that need to be achieved.
NASA Requirements Requirements are the specifications that system
developers and program Management Process Was
managers use to acquire, implement, and test a system. Requirements Not
Designed to Provide
should be consistent with one another, verifiable, and directly traceable
to Detailed System
higher- level business or functional requirements. It is critical that
requirements be carefully defined and that they flow directly from the
Requirements
organization*s concept of operations (how the organization*s day- to- day
operations are or will be carried out to meet mission needs). 12
Improperly defined or incomplete requirements have been commonly
identified as a root cause of system failure and systems that do not meet
their cost, schedule, or performance goals. Without adequately defined
requirements that have been properly reviewed and validated, a significant
risk exists that the system will need extensive and costly changes before
it will meet NASA*s needs.
As discussed previously, NASA is designing and fielding the core financial
module without having determined the specific information needs of its key
stakeholders, including program managers, cost estimators, and the
Congress. The omission of this critical step increased the risk that the
project would not effectively include all the detailed system requirements
that were needed to achieve management*s vision of a core financial
management module that provides timely, consistent, and reliable cost and
performance information for management decisions.
IFMP officials stated that their basic approach to developing the core
financial module system requirements was (1) defining high- level
requirements that could be used for making a software selection, (2)
defining the business processes that the core financial module needed to
address, (3) linking the requirements that were originally defined for the
software selection to those business processes, and (4) using subject
matter experts to determine whether the application met the business
processes envisioned by the users during their discussions of the needed
functionality. A key feature of the NASA approach is that the detailed
requirements covered in the discussion of the business processes are not
12 According to Institue of Electrical and Electronics Engineers Standard
1362- 1998, a concept of operations document is normally one of the first
documents that is produced during a disciplined development effort since
it describes system characteristics for a
proposed system from the user's viewpoint. This is important since a good
concept of operations document can be used to communicate overall
quantitative and qualitative system characteristics to the user,
developer, and other organizational elements. This allows the reader to
understand the user organizations, missions, and organizational objectives
from an integrated systems point of view.
required to be documented prior to testing. Rather, NASA depends on
subject matter experts, who are assigned to ensure that the core financial
module has the needed functionality, to know the detailed requirements
necessary to evaluate the functionality actually provided. Such an
approach relies on the subject matter expert being available throughout
the process and on the expert remembering the undocumented requirements
completely and consistently. Specifically, an individual assigned to
develop a test case 13 is relied on to understand the detailed
requirements associated with all facets of that test case and then to
ensure that the test will provide the information needed to understand
whether the functionality was
actually provided. IFMP officials also stated that the current approach
was based on discussions with their contractors and eliminated the need
for detailed documented requirements normally associated with efforts such
as IFMP. They also recognized that this approach was somewhat inconsistent
with their own Requirements Management Framework, issued in October 2000,
which stated that *[ i] n order to test the software, a more detailed
statement of a requirement or process may be required to insure [sic] the
successful completion of a test.* This document also recognized that these
detailed requirements would be needed for *a more refined testable set of
requirements . . . and needed to serve as a basis for the testing that
will occur . . .* In a January 2003 report 14 by a contractor on the
lessons learned
on the IFMP effort, the contractor noted that NASA would need to develop a
set of requirements and design specifications that had been validated by
the individuals responsible for managing each process. The contractor also
noted that although such an approach delays the first phase of the project
design, it reduces the overall implementation time.
As a result of NASA*s stated approach to requirements management, our
review of NASA*s system requirements related to the process documents for
the core financial module found that key attributes of effective
requirements were missing for many requirements. According to the
Institute of Electrical and Electronics Engineers (IEEE)* a leading source
13 A test case is a series of actions, performed serially, in parallel, or
in some combination, that creates the desired test conditions. Rex Black,
Managing the Testing Process: Practical Tools and Techniques For Managing
Hardware and Software Testing (Redmond, Wash.: Microsoft Press, 1999).
14 Gartner, Inc.
for defining the best practices for efforts such as this* good
requirements have several characteristics, including the following: 15
The requirements document contains all the requirements identified by
the customer, as well as those needed for the definition of the system.
The requirements fully describe the software functionality to be
delivered. Functionality is a defined objective or characteristic action
of a system or component. For example, a system may have inventory control
as its primary functionality.
The requirements are stated in clear terms that allow for quantitative
evaluation. Specifically, all readers of a requirement should arrive at a
single, consistent interpretation of it.
Traceability among various requirement documents is maintained.
Requirements for projects such as IFMP can be expressed at various levels
depending on user needs. They range from agencywide business requirements
to increasingly detailed functional requirements that eventually permit
the software project managers and other technicians to design and build
the required functionality in the new system. Adequate traceability
ensures that a requirement in one document is
consistent with and linked to applicable requirements in another document.
NASA established about 590 requirements for the core financial module. 16
We reviewed in detail one business process area of this module* the
*Manage Accounts Payable* process* that included 132 of these
requirements. We found that (1) for over 80 percent of the 132
requirements the functionality to be delivered was not adequately
described or stated in a manner that allowed for quantitative evaluation
and (2) the traceability between the various requirement documents was not
maintained.
Requirements Were Not Specific For over 80 percent of the 132 *Manage
Accounts Payable* requirements, the process documents lacked the specific
information necessary to understand the required functionality that should
be provided and how to
15 IEEE 830- 1998. 16 NASA originally identified about 590 requirements.
However, 51 of these were deleted and 86 were deferred.
determine quantitatively, through testing or other analysis, whether the
system will meet NASA's needs. The following are examples of core
financial module requirements that lacked the necessary specificity. One
requirement stated that the system must *[ a] llow the information
contained in the system to be queried to present detailed data as
requested (such as payee information). The capability to perform a Print
Screen must be available to all user screens.* This requirement did not
clearly state such items as (1) the data elements that must be supported,
(2) how the user would obtain the data definitions for the data elements
that could be used, (3) the tool or process that would be used to perform
these queries, and (4) the relationship between the ability to perform
such queries and the requirement to be able to print the screen.
The core financial module was required to support *multiple payment
addresses and/ or bank information for a single payee.* This requirement
did not clearly state the maximum number of payment address and bank
information entries that should be allowed.
Several requirements called for the core financial module to make
accounting entries; however, these requirements did not define the
specific accounting entries that should be made.
The lack of documented requirements that are complete and unambiguous not
only increases the risk that the project's functionality goals will not be
met, but also significantly impairs the testing phase of the system
implementation efforts, as discussed later in this report.
Traceability Was Not Maintained NASA has adopted a four- level approach to
defining its requirements* processes, subprocesses, activities, and tasks,
with processes stating highlevel requirements and tasks providing the most
detailed level. In reviewing the various requirement documents, we found
that (1) traceability was not always maintained through the various
documents and (2) the level of detail did not provide additional
specificity for a given requirement as it progressed through the
hierarchy. Traceability allows the user to follow the life of the
requirement both
forward and backward through these documents and from origin through
implementation. Traceability is also critical to understanding the
parentage, interconnections, and dependencies among the individual
requirements. This information in turn is critical to understanding the
impact when a requirement is changed or deleted. Without an effective
traceability approach, it is very difficult to perform such actions as (1)
accurately determining the impact of changes and making value- based
decisions when considering requirement changes, (2) maintaining the system
once it goes into production, (3) tracking the project's progress, and (4)
understanding the impact of a defect discovered during testing. To
illustrate these issues, we attempted to follow the hierarchy of
requirements for one of the core financial module's seven processes
through the four levels of requirements utilized by NASA for this project.
As shown in figure 3, the *Manage Accounts Payable* process area has 132
requirements associated with nine subprocesses. However, one of the 132
requirements, "Multiple User Access," contained in the *Manage Accounts
Payable* process, was not shown in any of the subprocesses, and it was
unclear where this requirement would be further defined.
Figure 3: System Requirements for the *Manage Accounts Payable* Process
Report on payment activities (14) Recertify payment activities (8) Execute
and manage payments (49)
Process IPAC payments (20) Manage accounts payable
Maintain AP master data (11) Process HHS (9) Enter invoice (41) Manage
travel (33) Validate payment (17) Source: NASA.
Note: The total number of requirements shown for the subprocesses (202)
exceeds the number of requirements shown for the *Manage Accounts Payable*
process (132) because some requirements apply to more than one activity.
Our review of the nine subprocesses found that 5 of the remaining 131
requirements contained in the subprocesses were not linked to any
activity. For example, a requirement that applies to all federal agencies
and is designed to ensure compliance with the Internal Revenue Service's
1099 reporting requirements was not included in any of the activities.
Therefore, the individuals responsible for implementing the requirements
contained in
the activities would not have the full universe of requirements they must
address. Conversely, as shown in figure 4, several of the activities for
the *Validate Payment* subprocess did not contain any related
requirements. Therefore, it was unclear whether these activities should
have been associated with this subprocess. For example, the *clear
advance* and
*adjust invoice* activities did not include any requirements related to
validating payments. Further, the lack of requirements for these
activities may cause confusion for the individuals assigned to test the
functionality
associated with this subprocess. We were also unable to trace the
requirements from activities to tasks, which should be the most detailed
level of requirements because, for the activities associated with the
*Validate Payment* subprocess, NASA used the same information for the
activities and tasks. In other words, the requirements for the tasks were
identical to those listed for the activities and therefore did not provide
any additional details.
Figure 4: Requirements for *Validate Payment* Subprocess
Adjust payment amount as neccessary (2) Compute holdback amount (1)
Determine holdback status (1) Check for other deductions (1)
Determine other adjustments (2) Validate Payment
Check for proper approvals (1)
(17)
Post invoice (11) Clear advance (0) Set flag (2) Adjust invoice (1)
Validate payment (0) Source: NASA.
Note: The total number of requirements shown for the activities (22)
exceeds the number of requirements shown for the *Validate Payment*
subprocess (17) because some requirements apply to more than one activity.
As can been seen in this example, NASA was unable to maintain adequate
traceability of the requirements for this subprocess as it progressed
through the hierarchy. More important, the level of specificity associated
with these requirements did not change. Based on our review, we generally
found that the wording of a given requirement was identical regardless of
the requirement document reviewed. For example, if a requirement was
listed in a subprocess area and flowed to an activity, the same wording
was used and the needed level of specificity to help ensure proper
implementation was not available. Therefore, although NASA appeared to
have adopted a requirements hierarchy that would facilitate the needed
specificity as the requirements flowed from subprocesses to tasks and
activities, the implementation of this approach did not address the
specificity problems discussed earlier. Accordingly, this is another
factor that increases the risk that this project will not meet its
schedule, cost, and functionality goals. A NASA contractor hired to help
evaluate the implementation of the core financial module had similar
findings. For example, the contractor found that NASA had not developed
documentation that explicitly details the relationship between lower-
level requirements and requirements of the next level.
Requirements Defects Because requirements provide the foundation for
system testing, the
Adversely Affect Testing of specificity and traceability defects in the
system requirements preclude
the Core Financial Module NASA from implementing a disciplined testing
process. That is, requirements must be complete, clear, and well
documented to design and
implement an effective testing program. Consequently, NASA is taking a
significant risk that its testing efforts will not detect significant
defects until after the system is placed into production. Industry best
practices indicate that the sooner a defect is recognized and corrected,
the cheaper it is to fix. This is especially true since NASA is depending
on the subject matter experts* knowledge, rather than documented
requirements, to ensure that the application does not have any significant
defects before the
system is placed into production. As shown in figure 5, there is a direct
relationship between requirements and testing.
Figure 5: Relationship between Requirements Development and Testing Stages
of system development Stages of testing Concept of operations
User acceptance testing
Specifies how system is used in Verifies that system operates operation
correctly with operational hardware and meets users' needs
Functional requirements System acceptance testing
Specifies the high- level functions Verifies that the complete system of
the system
satisfies functional requirements
Design requirements Integration testing
Specifies the tasks each software Verifies that units of software, when
component must perform
combined, work together as intended
For projects such as IFMP, these items are normally handled by the
commercial software vendor
Detailed design and coding Unit testing
Specifies the detailed steps for Verifies that each component of the each
software component and
software faithfully implements the implements those steps
detailed design Source: GAO.
Although the actual testing activities occur late in the development
cycle, test planning can help disciplined activities reduce requirements-
related defects. For example, developing conceptual test cases based on
the requirements derived from the concept of operations and functional
requirements stages can identify errors, omissions, and ambiguities long
before any code is written or a system is configured. Disciplined
organizations also recognize that planning testing activities in
coordination with the requirements development process has major benefits.
We have identified several indications that NASA*s testing program has
been adversely affected by the lack of complete and specific requirements.
Although we plan to continue our review of NASA*s testing plan for IFMP
implementation, we noted (1) significant defects that appeared to be
related to requirements occurred in the application after it was placed
into production and (2) several cases where NASA did not ensure that
modifications made to the application did not cause unintended effects and
that the system or component still complied with its specified
requirements after the change.
Significant Defects Appeared in Our review of the system test defect
reports for the core financial module
Production System disclosed that several defects considered by NASA to
have an initial
severity rating of critical 17 or high 18 had been identified after the
system was placed into production at Marshall Space Flight Center and
Glenn Research Center. Detecting such problems after the system goes into
production may lead to costly rework due to factors such as having to
reenter transactions and adjust reports manually. Furthermore, the manual
processes required to make these adjustments may introduce data integrity
errors. Our preliminary review indicated that the root cause of many of
these defects could be linked to the lack of complete requirements. For
example, see the following: Shortly after the system was placed into
production, NASA found that
the core financial module was not properly executing certain business
rules. An emergency fix was developed, and the defect report noted that a
long- term solution and requirements would need to be developed. It
was unclear why the subject matter experts did not include the business
rules in the test cases used to evaluate the functionality of the
17 NASA defines critical defects as those that *impact the ability to move
forward or complete an entire business function or task, and impacts
multiple business functions, multiple users and/ or locations. [It]
presents a failure that has no workaround or alternative.*
18 NASA defines high defects as those that have *a significant impact on
the completion of a business function or task, however, activities can
continue as far as the next function. A limited number of business
functions, business users, or locations are impacted, and may be an impact
to only one.*
application. However, we believe one cause is the lack of documented
requirements that the testers could use to develop effective test cases.
NASA was unable to process vendor invoices that contained over 200
line items, which, according to NASA officials, is a common occurrence on
NASA*s large contracts. It was unclear why the subject matter experts
responsible for testing this functionality had not developed the test
cases to ensure that large invoices were properly processed before the
system was placed into production. IFMP officials recognized that this was
an oversight in their testing process. If NASA had documented its
requirements, it would have recognized that a properly developed test case
had not been designed to cover this necessary functionality. About 3
months after the core financial module was placed into
production at one center, it was found that when the system produced
multiple bills for the same customer, only the first bill was sure to have
the proper account classification code printed. The remaining bills often
contained incorrect values since the program improperly assumed that the
account classification code would not change until the customer changed.
Since the account classification code is critical for these types of
bills, the center was required to manually make the necessary corrections
on the bills. Adequate Regression Testing Is
Efforts such as the core financial module undergo change constantly at
this Not Being Performed
stage of their development as functionality is being added and defects are
being corrected. However, before the revised application is released,
testing needs to be performed to ensure that any modifications have not
caused unintended effects and that the system or component still complies
with its specified requirements. This practice is commonly referred to as
regression testing. An effective regression testing program is critical
for ensuring that the functionality associated with requirements that has
been validated during previous testing efforts has not been impaired by
subsequent changes in the application. Although NASA officials stated that
they require regression testing before deploying any changes, we found
that they do not have an effective method to ensure that adequate
regression testing is being performed or that a
consistent approach is being taken in performing such testing. According
to NASA officials, the individual identifying a defect is responsible for
ensuring that the defect is corrected and for determining the amount of
testing necessary to ensure that the defect has been corrected. For
example, if a defect is identified when executing a test case, the tester
may
only test the section of the case where the error was originally
identified rather than performing all the steps in the test case. This
approach increases the risks that defects introduced into the application
during enhancements or defect corrections will not be detected until after
the application is deployed, which results in costly rework. We noted
several examples where NASA appeared to perform inadequate regression
testing. These include the following:
After adding an interface, it was found that an application screen for
recording advances did not operate as it had before the change was made.
A process for recording transactions provided by the Department of the
Treasury failed after an update to the application program.
One center was testing a certain type of invoice and received an error
message. This error was attributed to a system patch that had been applied
to the application.
IFMP officials agreed that they did not have a comprehensive regression
testing program with a consistent approach. However, they told us that
they believed that any defects would be detected by the centers as the
application progresses through its releases because they encourage each
center to completely retest the application before placing the application
into production. This approach is particularly risky in light of the
requirements defects discussed previously, which substantially increase
the risk that the testing conducted by the centers not yet operational may
not detect any negative impacts associated with a system change. However,
IFMP officials recognized that, after all centers are in production, a
regression testing program will be needed. A NASA contractor monitoring
this project also identified potential problems relating to regression
testing. According to the contractor performing this work, NASA has not
implemented the testing tools necessary to adequately perform the
regression testing to provide NASA reasonable assurance that changes made
in a given software release do not
have any adverse consequences for future releases.
Performance Metrics Could Without a well- documented set of requirements,
it is impossible to place an
Be Used to Assess Potential error in context and understand the cause of
the defect* for example,
Risks of Identified determining whether the error was caused by the
underlying requirements
Weaknesses or by some other process failure, such as inadequate testing or
inadequate controls over system configuration. NASA has not effectively
captured the
types of metrics that can help the organization understand the
effectiveness of its management processes, such as identifying and
quantifying any weaknesses in its requirements management process.
Accordingly, NASA is unable to implement a metrics measurement process
that allows it to understand (1) its capabilities to manage the IFMP
effort, (2) how its process problems will affect its cost, schedule, and
performance objectives, and (3) the corrective actions needed to reduce
the risks associated with the problems identified.
The Software Engineering Institute (SEI) has found that metrics
identifying important events and trends are invaluable in guiding software
organizations to informed decisions. Key SEI findings relating to metrics
include the following:
The success of any software organization depends on its ability to make
predictions and commitments relative to the products it produces.
Effective measurement processes help software groups succeed by enabling
them to understand their capabilities, so that they can develop achievable
plans for producing and delivering products and services.
Measurements enable people to detect trends and to anticipate problems,
thus providing better control of costs, reducing risks, improving quality,
and ensuring that business objectives are achieved. 19
A critical element in helping to ensure that a project meets its cost,
schedule, and performance goals is to ensure that defects are minimized
and corrected as early in the process as possible. Although NASA has a
system that captures the defects that have been identified during testing,
we found that the agency did not analyze its identified defects to
determine their root causes. Understanding the root cause of a defect is
critical to evaluating the effectiveness of a process. For example, if a
significant
19 William A. Florac, Robert E. Park, and Anita D. Carleton, Practical
Software Measurement: Measuring for Process Management and Improvement
(Pittsburgh, Pa.: Software Engineering Institute, Carnegie Mellon
University, 1997).
number of defects are caused by inadequate requirements definition, then
the organization knows that the requirements management process it has
adopted is not effectively reducing risks to acceptable levels. IFMP
officials stated that they do not capture the root causes of their
defects.
Our initial observations identified that the root cause of many defects
appeared to relate directly to the requirements management process. For
example, see the following:
About a week after the system was placed into production at a center,
NASA found that it was making payments to its vendors 1 day earlier than
required by Treasury regulations. This occurred because NASA thought that
Treasury would warehouse its payments. If NASA had researched and
documented the requirements associated with payment
warehousing for cash management purposes, it would have known that
Treasury does not warehouse payments such as these.
About 3 weeks after the system went into production, NASA found that one
of the payment processing tools was not working as required. It was
unclear whether this was caused by a requirements defect or failure to
properly test the functionality. A review of the requirements documents
relating to this functionality provided a different description of the
requirement than that included in the defect report.
In early October 2002, NASA found that the accounting entries for
certain advance transactions were incorrect. Properly documenting the
requirements, developing a test case that ensured the requirements were
met by the application, and executing that test case after the change was
made should have detected this problem.
After a patch was applied to the system, it was found that some code was
duplicated, which caused an error. The apparent reason for the duplication
of code was that manual adjustments were made to the code after the patch
had been applied. By analyzing the root causes of its identified defects,
NASA could determine whether the requirements management approach it has
adopted sufficiently reduces its risks of the system not meeting cost,
schedule, and
functionality goals to acceptable levels. Root cause analysis would also
help to quantify the risks inherent in the testing process that NASA has
selected for the core financial module. Because, as discussed previously,
its approach in both these areas includes elements that are not considered
industry best practices, such metrics would be particularly important to
NASA*s being reasonably assured that its processes will result in a system
that meets its business needs.
Conclusions NASA has established the right goal for IFMP, and its ongoing
implementation of several already- acquired system components,
particularly the core financial module, may provide some improvements to
NASA*s accounting data. However, implementation of these components will
only partially address NASA*s information needs related to its complex
space programs and contracts because NASA has deferred implementation of
the system capabilities needed to provide this information and has not
reengineered key business processes such as acquisition management. NASA*s
long- standing weaknesses in this area have been central to our
designation of NASA contract management as high risk. Moreover, NASA*s
approach to acquiring and implementing IFMP components has and will
continue to introduce risk and increase the chances that the agency will
fall short of meeting its IFMP goal.
NASA faces serious near- term risks in implementing the commercial
components that it has already acquired, including the core financial
module. However, it is too far along in deploying these components to its
centers, and relying on them to support operations, to stop and first
acquire and then implement them properly. Instead, NASA will be forced to
make the best of what it has acquired and implemented, meaning that NASA
will have to stabilize the components while they are operational by
identifying and correcting requirements defects and adequately testing the
components to ensure that completed requirements are met. Such rework of
already- deployed system components is a much more costly approach to
implementing systems than adequately defining requirements and effectively
testing system capabilities before they are deployed. However, NASA has
left itself no other viable option.
In the longer term, NASA has an opportunity to avoid the mistakes it has
made to date in acquiring system components, such as the core financial
module, by first determining whether proposed components are the best
solutions to meeting the agency*s corporate needs before it acquires them.
It is critically important that NASA*s acquisition management strategy for
future components includes a well- defined, risk- based methodology for
understanding the dependencies among commercial component options before
it acquires any additional components. Once these components are acquired,
it is also critically important that NASA employ effective
requirements management, testing, and performance metrics practices in
implementing the components. To do less will increase the risk of IFMP
becoming NASA*s third unsuccessful attempt to transform its financial
management and business operations.
Recommendations for Given that NASA has already largely deployed and
placed into production
Executive Action the IFMP commercial components acquired to date, we
recommend that
the NASA Administrator direct the Program Executive Officer for IFMP to
focus near- term attention on stabilizing the operational effectiveness of
these deployed commercial components.
Specifically, to mitigate the risks associated with relying on
alreadydeployed IFMP commercial components and to expeditiously stabilize
these components* operational capability and performance, we recommend
that the Administrator direct the Program Executive Officer for IFMP to
develop and implement a corrective action plan. At a minimum,
this plan should provide for identifying known and potential risks,
assessing the severity of the risks on the basis of probability and
impact, developing risk mitigation strategies, assigning
accountability and responsibility for implementing these
strategies, tracking progress in implementing these strategies, and
reporting progress regularly and frequently to relevant congressional
committees. Additionally, this plan should provide for developing and
properly documenting requirements that are consistent,
verifiable, and traceable, and that contain the necessary specificity to
minimize requirement- related defects;
conducting thorough regression testing before placing modified
components into production;
implementing a metrics program that will identify and address the root
causes of system defects;
reengineering acquisition management processes, particularly with
respect to the consistency and detail of budget and actual cost and
schedule data provided by contractors; and
engaging stakeholders* including program managers, cost estimators, and
the Congress* in developing a complete and correct set of user
requirements.
To mitigate future risks, we further recommend that the Administrator
require the Program Executive Officer for IFMP to complete the following
actions before the acquisition of any additional IFMP components:
establish and implement a methodology for commercial system component
dependency analysis and decision making, and
evaluate the suitability of already acquired, but not yet implemented,
IFMP component products within the context of a component dependency
analysis methodology.
Agency Comments and In its written comments on a draft of this report,
NASA stated that it
Our Evaluation recognized and was addressing several of the concerns we
raised and had
already implemented some of the recommendations. NASA also stated that it
disagreed with some of the issues in the report. NASA*s comments on our
recommendations included the following:
With regard to our recommendation to establish and implement a
methodology for IFMP commercial system component dependency analysis and
decision- making, NASA stated that it agreed with the importance of having
an approach for acquiring additional IFMP
components and believes that it has an effective strategy already in
place. We disagree that NASA has an effective strategy because it did not
provide convincing evidence to support its position that it is using
methodologically based dependency analysis* a best practice for
implementing commercial component- based systems* in acquiring IFMP.
Although NASA concurred with our recommendation regarding the need for a
short- term plan to mitigate the risks currently associated with
relying on already- deployed IFMP commercial components, it disagreed with
many of our findings in the areas of (1) its efforts to involve users and
reengineer its business process to ensure that the core financial module
would meet the needs of program managers and cost estimators and (2) the
need for detailed system requirements. We continue to believe that any
effort that falls short of end- to- end business process
reengineering will not result in a system that substantially improves the
data available for contract oversight and decision- making and that
documented, detailed requirements are necessary to reduce the risks of
implementing the selected processes to acceptable levels.
Overall, NASA disagreed with our findings related to three issues*
dependency analysis, user needs, and requirements and testing* which are
addressed in the following sections. NASA also included several technical
comments, which we have addressed as appropriate throughout the report.
Dependency Analysis In its written comments on our recommendation to
establish and implement a methodology for IFMP commercial component
dependency
analysis and decision making, NASA stated that it agreed with the
importance of having an approach for acquiring additional IFMP components
but disagreed with our finding that it has not performed such dependency
analysis to date in acquiring four IFMP commercial components. It also
disagreed that it lacked a methodology to guide its analysis, and
subsequent decision making, for future IFMP component acquisitions.
According to NASA*s comments, the agency already has an
effective strategy in place and it has followed this strategy to date in
acquiring four IFMP commercial components. NASA described this strategy as
consisting of two factors: following an enterprise resource planning (ERP)
suite integration strategy (i. e, *best of suite* approach) and using an
enterprise application integration framework and associated tool set for
integrating current and future IFMP components.
NASA said that prior to receiving our draft report, it had provided us
detailed documentation describing how it began performing its component
dependency analysis before selecting the SAP ERP product for IFMP*s core
financial module. The agency also noted that in a meeting following its
receipt of our draft report, it had provided us with clear evidence that
the program began performing dependency analysis before selecting the SAP
product. Further, NASA*s comments stated that it was using an enterprise
application integration tool to facilitate product integration and ease
the associated complexities of integrating multiple products. NASA added
that
there were *perhaps miscommunications* and *some misunderstanding* of its
approach, and opined that much of our concern about component dependency
stems from our belief that NASA is not following a *best of
suite* approach but rather a *best of breed* approach. To support its view
that it is following a *best of suite* approach, NASA offered several
statements, including that it is (1) on record in presentations and
letters, including responses to congressional inquiries, that it is
following *best of suite,* (2) developing business cases before
implementing an IFMP module, (3) working with SAP to extend its ERP
product, and (4) following a prioritization process when considering how
to introduce functionality that the SAP ERP product does not provide.
We agree with several of NASA*s comments and disagree with others.
Collectively, NASA*s comments do not change our finding and
recommendation. Specifically, we do not question that NASA is using an
enterprise application integration product and that this product
facilitates integration of system components, both commercial and legacy
components. Further, we do not challenge NASA*s statements regarding its
representation in presentations and briefings that it is following a *best
of suite* approach, its development of business cases, its interactions
with SAP, and its use of a prioritization process.
However, we do challenge NASA*s assertion that much of our concern is
based on our belief that it is following a *best of breed* approach. On
the contrary, our finding and recommendation do not hinge on the
distinctions between *best of breed* versus *best of suite,* despite
evidence supporting our statements in the report that NASA is indeed
following a best of breed approach. Such evidence includes (1) a report
from a NASA contractor hired to provide an independent evaluation of IFMP
stating that NASA is
following a *best of breed* approach, (2) NASA*s acquisition of four
separate commercial products from four vendors to satisfy the first five
of nine planned IFMP system modules, and (3) NASA*s statement in its
comments that additional products may be selected in the future. We fully
appreciate that when implementing an ERP solution, other vendor products
will likely be needed to fill gaps between agency requirements and the ERP
product*s capabilities. Accordingly, we state in our report that
proactive, methodologically based dependency analysis and evaluation is
needed regardless of whether an agency is following a *best of breed* or a
*best of suite* approach, although we appropriately recognize that this
analysis and evaluation is more vital in a *best of breed* effort.
Instead, our finding and recommendation is based on whether NASA is
following, and plans to follow, methodologically based dependency
analysis* a best practice for implementing commercial component- based
systems* in acquiring IFMP. In this regard, documentation that NASA
provided us during the course of our review, and that it provided
following its receipt of our draft report, both of which NASA cited in its
comments, does not offer convincing evidence that NASA is following this
best practice. For example, the documentation lacked product descriptions
and comparisons as well as any analysis of integration requirements.
Moreover, the Deputy Program Manager responsible for IFMP integration told
us during the course of our review that proactive analysis of prospective
IFMP
components* dependencies had not been performed and was not planned, and
that NASA did not have a methodology for doing such analysis. The Deputy
Program Manager for IFMP integration added, similar to NASA*s comments on
a draft of this report, that the agency*s use of an enterprise
application integration product and its associated tools will make
integration easy and will negate the need for proactive dependency
analysis. As noted above, we recognize that this product and tool set
facilitates integration of multiple system components. However, it does
not negate the need for dependency analysis and understanding to support
informed decision- making before integration begins. As we state in our
report, without such a proactive approach to acquiring system components,
the risk of component product incompatibilities increases, as do the
challenges and complexities that integration products and tools must
overcome in integrating the products.
User Needs NASA agreed that deployed and in- deployment modules do not yet
meet all the needs of program managers. NASA indicated that this was the
result of its *step- wise* approach in implementing the core financial
module first and then integrating follow- on modules at a later date. As
noted in our report, however, the deferral of many basic management
functions has resulted in critical NASA programs, such as the ISS, using
other systems to monitor contract cost and schedule performance. By not
including the cost and schedule information needed by program managers in
the core financial module, NASA risks operating with two sets of books*
one that is used to report to management and the Congress and another that
is used to manage NASA*s programs. NASA disagreed with our specific
findings related to user needs in three key areas:
NASA believes that we have understated its accomplishments and the
significance of the current capabilities delivered by the core financial
module. NASA took issue with our assessment of the level of detail
maintained in
the core financial module, but did not comment specifically on our
recommendation that the agency reengineer its acquisition management
processes, particularly with respect to the consistency and detail of
budgeted and actual cost and schedule data provided by contractors.
NASA disagrees with our characterization that key users were not
actively involved in the implementation of the core financial module or
defining system requirements, although NASA indicates that better
coordination was needed between program managers and the financial
management community. First, we acknowledge again the significant effort
that NASA has put into
this project. Moving from 10 separate, incompatible systems to a single
integrated financial management system is a major, complex undertaking.
However, as we discussed previously in this report, the core financial
module falls short of NASA*s own representation of the module*s
capabilities, which was to provide program managers with the information
required for day- to- day decision making. Specifically, it does not
provide integrated cost and schedule performance information needed by
program managers to oversee many of NASA*s largest and most complex
contracts. In commenting on a draft of this report, NASA officials stated
that it was never NASA*s intent to integrate schedule data with the
initial core financial module implementation. However, IFMP planning
documents (including its program plan and business case analysis),
congressional testimony by NASA*s Administrator, and NASA*s own press
releases clearly established an expectation that the core financial module
would remedy many of NASA*s long- standing management challenges by
providing program managers and other users with integrated financial and
performance information. For example, according to his testimony before
the House of Representatives Committee on Science on February 27, 2002,
the NASA Administrator stated that while all components of IFMP are
important, the successful completion of the core financial project will
satisfy the Office of Management and Budget requirement that the financial
and performance management systems supporting day- to- day operations are
fully
integrated. 20 NASA responded that it is currently in the process of
engaging program managers and defining specific requirements related to
needed cost and schedule performance data.
Second, we recognize that the commercial components NASA has selected for
its core financial module are technologically capable of capturing and
maintaining the detailed cost data required by program managers and cost
estimators. However, the level of detailed cost data currently maintained
in the core financial module depends on the level of detail provided by
NASA*s contractors and the coding structure embedded in the core financial
module. With respect to the level of detail provided by contractors, we
reported that NASA has not reengineered its acquisition management
processes to ensure that contractors are consistently providing the
detailed cost data needed by program managers and cost estimators and
recommended that NASA do so.
NASA did not specifically address our recommendation but stated that it is
incumbent upon program managers and cost estimators to learn and
understand the capabilities of the new module and take advantage of them
for their specific purposes. NASA*s comments also indicate that the data
structure in the core financial module would be extended beyond the
current legacy capabilities (i. e., the module will be able to record a
greater level of detail) in fiscal year 2004. However, increasing the
module*s capacity to store greater detail will not ensure that the
information needed
by program managers and cost estimators is requested and received from
contractors and subsequently updated in the module. Although NASA
commented that it would review its current project management process to
ensure that its contractors provide the appropriate levels of cost data,
we continue to believe that any effort that falls short of end- to- end
business
process reengineering will not result in a system that substantially
improves the data available for contract oversight and decision making.
Third, we acknowledge that the IFMP implementation team made an effort to
include resource management staff from program management offices in
various process teams. However, as we discussed previously in this report,
no effort was made to include the cost estimating community in these
20 The Chief Financial Officers Act of 1990 and subsequent related
financial management reform legislation, among other things, set
expectations for agencies to develop and deploy more modern financial
management systems, produce sound cost and operating performance
information, and design results oriented reports on the government*s
financial condition by integrating budget, accounting, and program
information.
efforts. While program management office staff did participate, these
efforts did not address the program cost and schedule needs of program
managers or cost estimators. For example, the program management staff
with whom we spoke, who worked on three of NASA*s largest programs (ISS,
Space Shuttle, and Space Launch Initiative), viewed the core financial
module as an *accounting system* that would be used by the accountants but
was not necessarily going to change the way that program managers manage
their programs. With this understanding, it is not surprising that the
core financial module does not meet the needs of program managers or cost
estimators. Implementing an integrated financial management system that is
intended to change the way an organization does business is extremely
complex and involves cultural, organizational, and process improvements.
It also means making financial management an agency- wide priority. Our
work at leading public and private sector organizations has shown that
implementing a financial management system that meets the organization*s
business needs takes more than just placing a handful of business or line
management representatives on the implementation team. NASA*s approach has
resulted in a core financial module that will be of limited value to
program managers and cost estimators, who will continue to rely on other
systems or ad hoc processes to get the data they need. As such,
implementation of the core financial module to date continues to foster
the concern that, at NASA, finance is not viewed as an integral part
of NASA*s program management decision process. Requirements and Testing
NASA generally agreed that improvements were needed in its requirements
management and testing processes and has stated that it has already begun
to make improvements. For example, NASA recognized the need to implement a
more rigorous regression testing methodology and stated that by October
2003 it would have an improved regression testing program. NASA also
recognized that its process for tracing requirements and testing needed
improvement and stated that it planned to implement improved capability
and functionality for traceability over the next few months.
However, according to NASA officials, they are following best practices
for implementing an ERP solution and have *defined and implemented
rigorous, closed- loop requirements and testing processes.* Further,
regarding the applicability of requirements management standards, NASA
did not agree that IEEE 830- 1998 was applicable to the IFMP since it was
an ERP implementation effort. NASA stated that specifying detailed
requirements for already- developed software is high risk and that other
leading industry experts have told them that NASA needed to change its
processes to conform to the capabilities of the commercial software
selected rather than attempt to change the software to conform to the
existing NASA processes. We agree with NASA*s position that it needs to
change its business processes to conform to the software; however, we do
not agree with the agency*s position that detailed requirements are not
needed. We continue to believe that NASA needs to properly configure the
software based on detailed requirements in a manner that supports the
business processes that have been adopted from the selected ERP solution.
Because it has not done so, we continue to believe that NASA has not
effectively implemented the types of disciplined processes necessary to
reduce this project*s risks to acceptable levels. Acceptable levels refer
to
the fact that any systems acquisition effort, such as that being
undertaken by NASA, will have some requirements- related defects. However,
the goal is to reduce the risks and prevent significant requirements
defects in order
to limit the negative impact of these defects on cost, timeliness, and
performance of the project.
During our review, we discussed with IFMP officials our concerns about the
lack of documented, detailed system requirements for implementing the core
financial module. In those meetings, we recognized that NASA*s approach
for developing requirements was based on a business process
model and did not disagree that this approach could be used to define how
NASA would implement the necessary functionality. However, we continue to
believe that once the business processes are defined and selected,
documented, detailed requirements are necessary to reduce the risks of
implementing the selected processes to acceptable levels. As NASA noted in
its comments, its consultants also recommended that NASA needed to
*determine the requirements while putting together the design process.*
Therefore, guidance provided by the IEEE standard is applicable to the
successful configuration and implementation of commercial software
packages and is useful to help gauge the effectiveness of those efforts.
As agreed with your offices, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At
that time, we will send copies to interested congressional committees, the
NASA Administrator, and the Director of the Office of Management and
Budget. We will make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site at http://
www. gao. gov.
If you or your staffs have any questions concerning this report, please
contact Gregory D. Kutz at (202) 512- 9505 or kutzg@ gao. gov, Randolph C.
Hite at (202) 512- 6256 or hiter@ gao. gov, Allen Li at (202) 512- 4841 or
lia@ gao. gov, or Keith A. Rhodes at (202) 512- 6412 or rhodesk@ gao. gov.
Key contributors to this report are acknowledged in appendix III.
Gregory D. Kutz Director Financial Management and Assurance
Allen Li Director Acquisition and Sourcing Management
Randolph C. Hite Director Information Technology Architecture
and Systems Issues Keith A. Rhodes Chief Technologist Applied Research and
Methods
Appendi Appendi xes x I
Objectives, Scope, and Methodology To determine whether the National
Aeronautics and Space Administration (NASA) is effectively managing the
Integrated Financial Management Program (IFMP) acquisition, we reviewed
relevant program- level acquisition management documentation to obtain an
understanding of NASA*s plans and strategy, including the program
overview, program- and project- level management plans, the acquisition
strategy, implementation and integration plans, briefing materials on the
agency*s plans to develop
an information architecture, and a report on IFMP lessons learned by
NASA*s consultant, Gartner, Inc. We also interviewed various program
officials, including the Program Executive Officer for IFMP, the IFMP
Program Director, the IFMP Deputy Program Director, the Core Financial
Project Manager, the Integration Office Deputy Program Manager, and the
Chief Information Officer to clarify our understanding of the agency*s
strategy and obtain current information on the status of the agency*s
efforts. Specifically, we inquired as to NASA*s basis for selecting
alreadyacquired commercial products and its plans for selecting future
modules. We then compared NASA*s plans and activities to relevant best
practices, Office of Management and Budget (OMB) requirements, federal
guidance,
and NASA procedures and guidance. We had also intended to include our
assessment of a key element of NASA*s acquisition strategy* whether NASA
was acquiring IFMP components in the context of an enterprise
architecture* in this report. However, because NASA did not provide the
data needed to complete our assessment until after the conclusion of our
fieldwork, we plan to address NASA*s enterprise architecture in a future
product.
To determine whether NASA had adequately considered the information needs
of key users in implementing the core financial module of IFMP, we
reviewed IFMP documents discussing the business case and properties of the
core financial module and spoke with IFMP implementers at Marshall Space
Flight Center* the lead center on this project* and NASA
headquarters. To determine whether the data requirements, as established
by the IFMP implementers, would address NASA*s known problems with cost
control and cost tracking, we spoke with program managers involved
in three of NASA*s largest programs and other NASA program and business
management staff at three centers* Marshall Space Flight Center, Johnson
Space Center, and Glenn Research Center. We also reviewed prior work on
NASA*s cost problems, including the report by the International Space
Station Management and Cost Evaluation Task Force, which reviewed the
recent cost growth in that program and identified causes and necessary
actions.
In addition to speaking with program managers and their staffs, we spoke
with cost estimators at the three centers mentioned above as well as
Langley Research Center and NASA headquarters. We also spoke with center
staff who oversee and support earned value management for programs that
use that tool, and with the congressional staffs of NASA*s authorization
committees. We asked them about the extent to which they
had been asked by IFMP implementers for input on their data needs, the
extent to which they had been involved in IFMP*s design and
implementation, and whether they had been briefed by IFMP implementers
on the capabilities of the core financial module. To determine what kind
of cost information program managers use to oversee their programs, we
reviewed selected large, cost- type contracts for NASA*s three largest
space flight programs* the International Space Station, the Space Shuttle,
and the Space Launch Initiative project (intended to develop technologies
for the next generation replacement for the Space Shuttle.) For all three
of these programs, cost control and cost tracking have been issues of
concern. The three programs together involve most of NASA*s work in the
human space flight area, which accounts for most of the agency*s spending.
These programs range from relatively new (Space Launch Initiative) to
quite mature (Space Shuttle) programs and require the procurement of a
wide range of goods and services. Each of these programs is being run at
multiple centers, involves the work of multiple contractors, and uses
cost- type contracts 1 that run for multiple years.
For the contracts we selected, we spoke to responsible personnel about how
costs are tracked and monitored, including the level of detail provided by
contractors, the format in which cost data are available, and how
contract cost data reporting requirements are developed. We also obtained
and reviewed copies of contractor financial management reports and cost
and schedule performance reports that we compared with contract or program
work breakdown structures, as well as contract cost data reporting
requirements and statements of work. We analyzed and discussed with agency
officials how all these documents and reports related to each other and to
the work breakdown structure. We also discussed how the
1 Cost- reimbursement contracts are often the most appropriate type for
developmental items or those for which the exact price of the goods or
services being purchased cannot be definitely known prior to contract
award. This type of procurement instrument places greater risk on the
government than contracts based on firm fixed prices.
reported information was used by the programs and the extent to which that
information would be included in the core financial module. We did not,
however, evaluate whether the information currently received from
contractors, or represented as needed by program managers and cost
estimators, was adequate for management purposes.
IFMP*s core financial module is intended to address known problems with
NASA*s program cost accounting and with its financial reporting. We did
not, however, review how the core financial module will address the
agency*s financial reporting issues, including property accounting and
budgetary information, and whether the module will reduce the time and
resources needed to close the books each accounting period and reduce the
number of postclosing adjusting entries. We plan to review and report on
these issues at a later date.
To assess whether NASA had established and implemented an effective
requirements management process to support implementation of the core
financial module, we wanted to determine whether NASA had effectively
implemented (1) the disciplined processes that can reduce project risks to
acceptable levels for its requirements management process and (2) the
types of metrics to identify and quantify any weaknesses in its
requirements management process. To accomplish these objectives, we
reviewed various requirements documents produced for the core
financial module project, including the over 500 contract requirements
used to acquire the SAP software;
performed an in- depth review and analysis of the 132 requirements,
which represent about 22 percent of the contract requirements, developed
for the *Managing Accounts Payable* process to determine whether they had
the attributes normally associated with good requirements and whether
these requirements traced between the various requirements documents;
reviewed NASA*s procedures for defining its requirements management
framework and compared these procedures to its current practices;
reviewed business processes, problem defect reports, test conditions,
test cases, and test execution logs contained in Accenture*s Method
Delivery Management system* NASA*s project management tool; and
reviewed guidance published by the Institute of Electrical and
Electronics Engineers (IEEE), Software Engineering Institute (SEI), and
publications by experts to determine the attributes that should be used
for developing good requirements and for identifying and quantifying
performance metrics.
To augment these document reviews and analyses, we interviewed officials
from NASA headquarters, Marshall Space Flight Center, and NASA*s
independent verification and validation contractor* Titan Systems
Corporation. In addition, we discussed with NASA officials the processes
they used to measure the effectiveness of their requirements management
process and compared NASA*s process to those used by disciplined
organizations. In order to determine the processes that can be used to
help an organization understand the effectiveness of its processes, we
used information from IEEE, SEI, and subject matter experts.
We conducted our work at NASA headquarters in Washington, D. C.; Marshall
Space Flight Center in Huntsville, Alabama; Glenn Research Center in
Cleveland, Ohio; Johnson Space Center in Houston, Texas; Langley Research
Center in Hampton Roads, Virginia; and Goddard Space Flight Center in
Greenbelt, Maryland. We received written comments on a draft of this
report from the NASA Deputy Administrator. These comments are addressed in
the *Agency Comments and Our Evaluation* section of this report and are
reprinted in appendix II. We performed our work from
April 2002 through February 2003 in accordance with generally accepted
government auditing standards.
Comments from the National Aeronautics and
Appendi x II
Space Administration Note: GAO comments supplementing those in the report
text appear at the end of this appendix.
See comment 1.
See comment 1. See comment 1.
See comment 1. See comment 1.
See comment 2. See comment 3.
See comment 4.
See comment 1.
See comment 1. See comment 1.
See comment 5. See comment 1.
See comment 1. See comment 1.
See comment 1.
See comment 1.
See comment 1.
See comment 6.
See comment 7.
See comment 8.
See comment 1.
See comment 9.
The following are GAO*s comments on the National Aeronautics and Space
Administration*s (NASA) letter dated March 25, 2003.
GAO Comments 1. See the *Agency Comments and Our Evaluation* section of
this report. 2. Although NASA indicates that it has extended the SAP suite
to include
Business Warehouse (for reporting) and Asset Management, it did not
provide us any documentation to support these selections during the course
of our fieldwork.
3. We did not assess the deployment and operation of the three modules to
which NASA referred. We understand that the NASA Inspector General has
recently begun a review of the Travel Management module.
4. The scope of our work did not include a review of the Integrated
Financial Management Program (IFMP) budget or schedule. We plan to address
these issues in a future product.
5. As stated in this report, although the core financial module will
provide some improvement to NASA*s current accounting system environment,
certain system capabilities have been deferred and will not be available
when the system becomes an agencywide operation in June. Without an effort
to reengineer NASA*s acquisition management processes, it is unlikely that
detailed cost information will be available to meet the needs of program
managers, cost estimators, and the Congress. Thus, NASA*s assertion that
it will be able to transition to *full cost management practices* in June
of 2003 is questionable.
6. NASA*s comments refer to the *Accounts Payable* process illustrated in
figure 3 of the report. While we did not verify or evaluate the extent to
which the additional requirements to which NASA refers in its response
were established or validated, the accounts payable requirements, as
described, do not provide for quantitative evaluation to determine whether
the system meets NASA*s needs. Furthermore, the additional requirements
did not provide the needed clarification for the
requirement cited in our report related to the ability to query
information. Moreover, given that NASA added new requirements for
reporting, it is unclear whether the existing accounts payable requirement
was to provide some other query functionality not included in the other
general reporting requirements.
7. As noted in our report, requirements provide the foundation for system
testing. In meetings with NASA, we acknowledged that the *processcentric*
approach that the agency adopted was an acceptable methodology for
understanding how the processes supported by the selected enterprise
resource planning (ERP) solution could be implemented at NASA. However, we
believe that this approach still requires the development and
documentation of the necessary requirements to fully understand the
functionality to be provided by a given process. Without such
requirements, a disciplined testing process is very difficult to implement
since requirements are a fundamental attribute of an effective testing
process. As discussed in our report, we continue to believe that the lack
of an effective requirements management process hampered NASA*s testing
efforts since significant defects in the production system should have
been detected before system implementation. Although NASA stated that it
will repeat its testing efforts at each
center implementing the system, without adequately documenting its
requirements and ensuring that the testing process adequately tests those
requirements, it does not have reasonable assurance that the testing
process will identify significant defects before a center is converted to
the production system. For example, NASA stated that it had developed over
1,700 test conditions. However, it was not until the system was placed
into production that NASA identified several significant weaknesses, as
discussed in our report. We continue to
believe that NASA will not have reasonable assurance that it has
adequately tested the system until it (1) documents its requirements and
(2) develops test conditions that fully test those requirements.
8. As noted in our report, discussions with IFMP officials recognized that
a test case was not properly developed to test large contracts that
contained over 200 line items* a common occurrence according to IFMP
officials* and that this was an oversight in their testing process. Had
NASA developed and documented a detailed requirement for this
functionality and then mapped its test conditions against those
requirements, it would have recognized that it had not developed a test
condition to properly demonstrate and test the functionality prior to the
system going into production. Properly processing these types of payments
may have enabled NASA to reduce the impact of the payment backlog.
9. As noted in our report, NASA does not have metrics that properly
analyze the cause of the defects so that it can improve its processes. For
example, although NASA was able to show the number of defects that were
related to subsequent implementation, referred to as the second wave, it
did not have information that could be used to analyze whether these
defects were caused by, for example, requirements or testing problems or
by not adequately correcting prior defects. Therefore, although NASA
states that it has a structured testing and problem analysis process in
place, we continue to believe that the examples provided in NASA*s
comments do not provide the data necessary to identify the causes of
defects or assess the effectiveness of processes such as the requirements
management and testing processes. As noted in our report, these types of
data can be used to prevent or anticipate problems before they occur,
resulting in less rework.
Appendi x III
GAO Contacts and Staff Acknowledgments GAO Contacts Diane Handley, (404)
679- 1986 Cynthia Jackson, (202) 512- 5086 Chris Martin, (202) 512- 9481
Acknowledgments Staff members who made key contributions to this report
were Erin Baker, Molly Boyle, Felicia Brooks, Francine DelVecchio, Michael
Giannone, Jamie Haynes, Richard J. Herley, Kristi Karls, LaTonya Miller,
Maria Storts, Eric Trout, Teresa Tucker, Carrie Wilson, and Jenniffer
Wilson.
(120148)
GAO*s Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities
and to help improve the performance and accountability of the federal
government for the American people. GAO examines the use of public funds;
evaluates federal programs and policies; and provides analyses,
recommendations, and other assistance to help Congress make informed
oversight, policy, and funding decisions. GAO*s commitment to good
government is reflected in its core values of accountability, integrity,
and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost is
through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext GAO Reports and
files of current reports and testimony and an expanding archive of older
Testimony
products. The Web site features a search engine to help you locate
documents using key words and phrases. You can print these documents in
their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this
list to you every afternoon, go to www. gao. gov and select *Subscribe to
GAO Mailing Lists* under *Order GAO Products* heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO
also accepts VISA and Mastercard. Orders for 100 or more copies mailed to
a single address are discounted 25 percent. Orders should be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www. gao. gov/ fraudnet/ fraudnet. htm
E- mail: fraudnet@ gao. gov Federal Programs
Automated answering system: (800) 424- 5454 or (202) 512- 7470 Public
Affairs Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512-
4800
U. S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D.
C. 20548
a
GAO United States General Accounting Office
The core financial module, if implemented as planned, may provide some
improvement to NASA*s accounting system environment. However, NASA is not
following key best practices for acquiring and implementing IFMP. In
acquiring IFMP components, NASA is facing risks in understanding
dependencies among commercial components. NASA has not analyzed the
interdependencies among selected and proposed IFMP components, and it does
not have a methodology for doing so. For programs like IFMP, which involve
building a system from commercial components, it is essential to
understand the characteristics and credentials of each component to select
ones that are compatible and can be integrated without having to build and
maintain expensive interfaces. By acquiring IFMP components without first
understanding system component relationships, NASA has increased its risks
of implementing a system that will not optimize mission performance and
will cost more and take longer to implement than necessary.
In implementing the core financial module, NASA is facing risks in two
additional areas:
User needs. NASA did not consider the information needs of key system
users and deferred addressing the requirements of program managers, cost
estimators, and the Congress. Although this module should eliminate NASA*s
separate, incompatible accounting systems, little has been done to
reengineer acquisition management processes. Program managers and cost
estimators indicated that they will continue
to rely on other means to capture the data needed to manage programs such
as the International Space Station. Requirements management. NASA is
relying on a requirements management process that does not require
documentation of detailed
system requirements prior to system implementation and testing. Over 80
percent of the requirements GAO reviewed lacked specificity, and several
could not be traced among various documents. These defects also
significantly impaired the testing phase of the system implementation
effort. Further, NASA has not implemented metrics to
help gauge the effectiveness of its requirements management process.
NASA*s approach will likely result in increasing amounts of time spent on
costly rework and reduced progress.
Unless these issues are successfully addressed, NASA is at increased risk
of having IFMP become its third unsuccessful attempt to transform its
financial management and business operations.
BUSINESS MODERNIZATION
Improvements Needed in Management of NASA's Integrated Financial
Management Program
www. gao. gov/ cgi- bin/ getrpt? GAO- 03- 507. To view the full report,
including the scope and methodology, click on the link above. For more
information, contact Gregory D. Kutz at (202) 512- 9095 or kutzg@ gao.
gov. Highlights of GAO- 03- 507, a report to the
Committee on Commerce, Science, and Transportation, U. S. Senate, and the
Committee on Science, House of Representatives
April 2003
The National Aeronautics and Space Administration*s (NASA) nonintegrated
financial management systems have weakened its ability to oversee its
contractors, and its contract management has been on GAO*s high- risk list
since 1990. In April
2000, NASA began its Integrated Financial Management Program (IFMP), its
third attempt in recent years at modernizing financial management
processes and
systems. GAO was asked to review whether NASA was following key best
practices in acquiring IFMP
components and implementing one of the first components* the core
financial module.
GAO is recommending that NASA develop and implement (1) a shortterm plan
to identify and mitigate the risks currently associated with relying on
already deployed IFMP
commercial components and (2) a longer term strategy for acquiring
additional IFMP components that includes implementing a
methodology for commercial system component dependency analysis. NASA
agreed with GAO*s
recommendation related to a shortterm plan but disagreed with many of the
findings related to user
needs and requirements management. NASA also agreed with the importance of
having an approach for acquiring additional IFMP components, but stated
that it already has an effective strategy in place. GAO reaffirms its
recommendations.
Page i GAO- 03- 507 NASA's IFMP
Contents
Contents
Page ii GAO- 03- 507 NASA's IFMP
Page 1 GAO- 03- 507 NASA's IFMP United States General Accounting Office
Washington, D. C. 20548 Page 1 GAO- 03- 507 NASA's IFMP
A
Page 2 GAO- 03- 507 NASA's IFMP
Page 3 GAO- 03- 507 NASA's IFMP
Page 4 GAO- 03- 507 NASA's IFMP
Page 5 GAO- 03- 507 NASA's IFMP
Page 6 GAO- 03- 507 NASA's IFMP
Page 7 GAO- 03- 507 NASA's IFMP
Page 8 GAO- 03- 507 NASA's IFMP
Page 9 GAO- 03- 507 NASA's IFMP
Page 10 GAO- 03- 507 NASA's IFMP
Page 11 GAO- 03- 507 NASA's IFMP
Page 12 GAO- 03- 507 NASA's IFMP
Page 13 GAO- 03- 507 NASA's IFMP
Page 14 GAO- 03- 507 NASA's IFMP
Page 15 GAO- 03- 507 NASA's IFMP
Page 16 GAO- 03- 507 NASA's IFMP
Page 17 GAO- 03- 507 NASA's IFMP
Page 18 GAO- 03- 507 NASA's IFMP
Page 19 GAO- 03- 507 NASA's IFMP
Page 20 GAO- 03- 507 NASA's IFMP
Page 21 GAO- 03- 507 NASA's IFMP
Page 22 GAO- 03- 507 NASA's IFMP
Page 23 GAO- 03- 507 NASA's IFMP
Page 24 GAO- 03- 507 NASA's IFMP
Page 25 GAO- 03- 507 NASA's IFMP
Page 26 GAO- 03- 507 NASA's IFMP
Page 27 GAO- 03- 507 NASA's IFMP
Page 28 GAO- 03- 507 NASA's IFMP
Page 29 GAO- 03- 507 NASA's IFMP
Page 30 GAO- 03- 507 NASA's IFMP
Page 31 GAO- 03- 507 NASA's IFMP
Page 32 GAO- 03- 507 NASA's IFMP
Page 33 GAO- 03- 507 NASA's IFMP
Page 34 GAO- 03- 507 NASA's IFMP
Page 35 GAO- 03- 507 NASA's IFMP
Page 36 GAO- 03- 507 NASA's IFMP
Page 37 GAO- 03- 507 NASA's IFMP
Page 38 GAO- 03- 507 NASA's IFMP
Page 39 GAO- 03- 507 NASA's IFMP
Page 40 GAO- 03- 507 NASA's IFMP
Page 41 GAO- 03- 507 NASA's IFMP
Page 42 GAO- 03- 507 NASA's IFMP
Page 43 GAO- 03- 507 NASA's IFMP
Page 44 GAO- 03- 507 NASA's IFMP
Page 45 GAO- 03- 507 NASA's IFMP
Page 46 GAO- 03- 507 NASA's IFMP
Page 47 GAO- 03- 507 NASA's IFMP
Appendix I
Appendix I Objectives, Scope, and Methodology
Page 48 GAO- 03- 507 NASA's IFMP
Appendix I Objectives, Scope, and Methodology
Page 49 GAO- 03- 507 NASA's IFMP
Appendix I Objectives, Scope, and Methodology
Page 50 GAO- 03- 507 NASA's IFMP
Page 51 GAO- 03- 507 NASA's IFMP
Appendix II
Appendix II Comments from the National Aeronautics and Space
Administration
Page 52 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 53 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 54 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 55 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 56 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 57 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 58 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 59 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 60 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 61 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 62 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 63 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 64 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 65 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 66 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 67 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 68 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 69 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 70 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 71 GAO- 03- 507 NASA's IFMP
Appendix II Comments from the National Aeronautics and Space
Administration
Page 72 GAO- 03- 507 NASA's IFMP
Page 73 GAO- 03- 507 NASA's IFMP
Appendix III
United States General Accounting Office Washington, D. C. 20548- 0001
Official Business Penalty for Private Use $300 Address Service Requested
Presorted Standard Postage & Fees Paid
GAO Permit No. GI00
*** End of document. ***