Business Modernization: NASA Must Consider Agencywide Needs to
Reap the Full Benefits of Its Enterprise Management System
Modernization Effort (20-JUL-07, GAO-07-691).
Since 1990, GAO has designated the National Aeronautics and Space
Administration's (NASA) contract management as an area of high
risk in part because it lacked modern systems to provide accurate
and reliable information on contract spending. In April 2000,
NASA began a system modernization effort, known as the Integrated
Enterprise Management Program (IEMP). When GAO last reported on
the status of IEMP in September 2005, NASA had begun to implement
disciplined processes needed to manage IEMP, but had yet to
implement other best practices such as adopting business
processes that improve information on contract spending. This GAO
report addresses (1) actions taken by NASA to effectively
implement the disciplined processes needed to manage IEMP and (2)
the extent to which NASA has considered the strategic issues
associated with developing a concept of operations and defining
standard business processes. GAO interviewed NASA officials and
obtained and analyzed documentation relevant to the issues.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-691
ACCNO: A73008
TITLE: Business Modernization: NASA Must Consider Agencywide
Needs to Reap the Full Benefits of Its Enterprise Management
System Modernization Effort
DATE: 07/20/2007
SUBJECT: Concept of operations
Performance measures
Risk management
Standards
Strategic planning
Systems management
Financial management systems
Program management
Contract administration
Systems conversions
Contract performance
Program evaluation
Business operations
Business planning
NASA Integrated Enterprise Management
Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-691
* [1]Results in Brief
* [2]Background
* [3]IEMP Implementation Status
* [4]Prior Reporting on IEMP
* [5]Progress Made in Developing Disciplined Project Management P
* [6]Improvements Made to NASA Requirements Management and Testin
* [7]NASA Has Implemented an Effective Metrics Program and Risk M
* [8]NASA Uses a Recognized Industry Best Practice to Move from O
* [9]Improved Requirements Development and Project Scheduling Nee
* [10]NASA Has Not Yet Fully Considered an Enterprise View of Its
* [11]Concept of Operations Would Provide an Important Foundation
* [12]Adopting Enterprise Business Processes Would Help NASA Trans
* [13]Conclusions
* [14]Recommendations for Executive Action
* [15]Agency Comments and Our Evaluation
* [16]GAO Contacts
* [17]Acknowledgments
* [18]GAO's Mission
* [19]Obtaining Copies of GAO Reports and Testimony
* [20]Order by Mail or Phone
* [21]To Report Fraud, Waste, and Abuse in Federal Programs
* [22]Congressional Relations
* [23]Public Affairs
Report to Congressional Requesters
United States Government Accountability Office
GAO
July 2007
BUSINESS MODERNIZATION
NASA Must Consider Agencywide Needs to Reap the Full Benefits of Its
Enterprise Management System Modernization Effort
GAO-07-691
Contents
Letter 1
Results in Brief 4
Background 6
Progress Made in Developing Disciplined Project Management Processes, but
Some Problems Remain 10
NASA Has Not Yet Fully Considered an Enterprise View of Its Operations and
Processes 16
Conclusions 24
Recommendations for Executive Action 25
Agency Comments and Our Evaluation 26
Appendix I Scope and Methodology 28
Appendix II Comments from the National Aeronautics and Space
Administration 30
Appendix III GAO Contacts and Staff Acknowledgments 34
Figures
Figure 1: Current NASA Processes for Oversight of Large, Complex Contracts
and for Asset Accounting 21
Figure 2: Example of How NASA Could Follow an Enterprise Process Using
IEMP for Program Management and External Reporting 23
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
United States Government Accountability Office
Washington, DC 20548
July 20, 2007
The Honorable Bart Gordon
Chairman
Committee on Science and Technology
House of Representatives
The Honorable Todd R. Platts
House of Representatives
As we and others have reported in the past, the National Aeronautics and
Space Administration (NASA) has fundamental problems with its financial
management operations that undermine its ability to effectively manage its
major programs and to report externally on its financial operations. Since
1990, we have designated NASA's contract management as an area of high
risk, in large part because NASA has lacked a modern financial management
system to provide accurate and reliable information on contract spending.1
In April 2000, NASA began a program expected to address many of its
financial and management challenges. This program, now known as the
Integrated Enterprise Management Program (IEMP),2 has been focused on
implementing a new integrated financial management system. Specifically,
NASA has invested in an enterprise resource planning (ERP) solution3--a
business system that is intended to meet the information needs of both
internal and external customers and to promote standardization and
integration of business processes and systems across the agency. NASA
plans to complete IEMP by 2009 for a total cost of over $800 million.
In April and November 2003--3 years into the IEMP implementation effort
and with significant investment already made in the program--we issued a
series of four reports4 that detailed weaknesses in NASA's acquisition and
implementation strategy for IEMP. Specifically, we reported that NASA had
not followed key best practices for acquiring and implementing IEMP and,
therefore, was at risk of making a substantial investment in a financial
management system that would fall far short of its stated goal of
providing meaningful, reliable, and timely information to support
effective day-to-day program management and external financial reporting.
1GAO, High-Risk Series: An Update, [24]GAO-07-310 (Washington, D.C.:
January 2007).
2The effort was formerly known as the Integrated Financial Management
Program (IFMP). According to NASA, IFMP was renamed to reflect the
addition of program management and labor distribution.
3An ERP solution is an automated system using commercial off-the-shelf
software consisting of multiple, integrated functional modules that
perform a variety of business-related tasks such as payroll, general
ledger accounting, contract management, and supply chain management.
NASA is not alone in its struggle to successfully implement an integrated
financial management system. Billions of dollars have been spent
governmentwide to modernize financial management systems that have often
exceeded budgeted cost, resulted in delays in delivery dates, and did not
provide the anticipated functionality when implemented. In our previous
report5 on government financial management systems failures, we provided
our views on actions that can be taken to help improve the management and
control of agency financial management system modernization efforts. Based
on industry best practices, we identified three key practices, or building
blocks, that are needed to ensure a solid foundation for agencies'
successful system implementation efforts: (1) developing a concept of
operations that would define how an agency will carry out its day-to-day
operations in order to meet mission needs; (2) defining standard business
processes that result in streamlined operations, rather than simply
automating old ways of doing business; and (3) effectively implementing
the disciplined processes necessary to manage the project.
When we last reported on NASA's IEMP effort, in September 2005,6 NASA had
begun to implement a number of recommendations from our earlier reports,
including taking steps toward implementing the disciplined processes
necessary to manage IEMP--one of the key building blocks discussed above
that is needed to ensure a solid foundation for agencies' system
implementation efforts. For example, we reported that NASA had implemented
new requirements management and testing processes and had developed
metrics to evaluate the effectiveness of its system implementation
processes. However, at that time, the agency had not implemented our
recommendation to properly define and document system requirements for
already-deployed IEMP modules, including the core financial module. This
was important not only because it would affect the way the core financial
module functions but also because it would affect NASA's ability to
implement future upgrades and other modules expected to interface with the
core financial module. In addition, we reported that additional
enhancements could be made in the area of regression testing and
performance metrics. Finally, NASA had yet to reengineer its business
processes so that the commercial off-the-shelf (COTS) software products it
had selected for IEMP could support these processes.
4GAO, Business Modernization: Improvements Needed in Management of NASA's
Integrated Financial Management Program, [25]GAO-03-507 (Washington, D.C.:
Apr. 30, 2003); Business Modernization: NASA's Integrated Financial
Management Program Does Not Fully Address Agency's External Reporting
Issues, [26]GAO-04-151 (Washington, D.C.: Nov. 21, 2003); Information
Technology: Architecture Needed to Guide NASA's Financial Management
Modernization, [27]GAO-04-43 (Washington, D.C.: Nov. 21, 2003); and
Business Modernization: Disciplined Processes Needed to Better Manage
NASA's Integrated Financial Management Program, [28]GAO-04-118
(Washington, D.C.: Nov. 21, 2003).
5GAO, Financial Management Systems: Additional Efforts Needed to Address
Key Causes of Modernization Failures, [29]GAO-06-184 (Washington, D.C.:
Mar. 15, 2006).
6GAO, Business Modernization: Some Progress Made toward Implementing GAO
Recommendations Related to NASA's Integrated Financial Management Program,
[30]GAO-05-799R (Washington, D.C.: Sept. 9, 2005).
Since we last reported, in September 2005, on NASA's efforts to implement
IEMP, NASA implemented its IEMP contract management module and upgraded
the version of the COTS software that is used for the core financial
module of IEMP--which was expected to enhance the functionality of the
core financial module. Because of your continued interest in NASA
financial management, you asked us to provide periodic updates on the
status of NASA's financial management improvement efforts--including its
effort to implement IEMP. Specifically, this report addresses (1) actions
taken by NASA to effectively implement the disciplined processes necessary
to manage IEMP and (2) the extent to which NASA has considered the
higher-level strategic issues associated with developing an agencywide
concept of operations and defining standard business processes--two key
building blocks critical to NASA's ability to successfully implementing
its planned financial management system.
To achieve these objectives, we interviewed the appropriate NASA officials
and obtained and analyzed documentation supporting the process
improvements that were cited by NASA. We performed our work from January
2006 through June 2007 in accordance with U.S. generally accepted
government auditing standards. Details on our scope and methodology are
included in appendix I.
Results in Brief
Since September 2005, when we last reported on NASA IEMP implementation
efforts, NASA has implemented some of the disciplined processes needed to
manage IEMP. Specifically, since 2005, NASA has improved its requirements
management and testing processes, enhanced its performance metrics program
related to tracking system defects, and developed an IEMP risk management
strategy--as we previously recommended. In addition, NASA has developed
quantitative entry and exit criteria for moving from one phase of an IEMP
project to another--a recognized industry best practice. However,
weaknesses in the areas of requirements development and project scheduling
offset some of the benefits associated with NASA's improved requirements
management and testing processes, causing NASA to compress the testing
phase of its core financial upgrade implementation and assume a greater
risk that it would not identify significant system defects prior to
implementation.
According to NASA officials, NASA's ability to complete testing for the
core financial upgrade within the planned implementation time frames was
not so much due to the use of disciplined processes as it was the result
of the extraordinary effort put forth by NASA's project implementation
team. Despite the implementation difficulties, NASA financial managers
have indicated that the core financial upgrade is now functioning as
expected for most transactions. As of the end of March 2007, the upgrade
was in a "stabilization" phase as NASA continued to work on correcting a
number of system errors, including posting errors for certain types of
transactions. Because the upgrade was still quite new and NASA was
continuing to stabilize the system, we were unable to determine whether
these weaknesses were significant.
Although NASA has taken action to improve its processes for managing the
implementation of individual IEMP projects, NASA has not yet fully
considered the higher-level strategic issues associated with developing an
agencywide concept of operations and defining standard business processes
that are supported by its software--two key building blocks critical to
the successful implementation of an integrated system such as IEMP. With a
planned investment of over $800 million, completion of these strategic
building blocks will be critical if IEMP is expected to address
long-standing management challenges, including overseeing contractor
performance and properly accounting for its property, plant, and equipment
(PP&E).
NASA officials indicated that they have undertaken a critical first
step--they have begun developing a concept of operations to describe how
all of its business processes should be carried out. According to NASA
officials, they expect to complete the agency's concept of operations by
the summer of 2008. Ideally, a concept of operations should be completed
before system development begins so that it can serve as a foundation for
system planning and requirements development. Although NASA's IEMP
development effort began in April 2000, the completion of such a document,
even at this late stage in NASA's IEMP effort, would be beneficial for the
development of the remaining IEMP modules as well as any future upgrades
to the core financial module. For NASA, an effective concept of operations
would describe, at a high level, (1) how all of the various elements of
NASA's business systems relate to each other and (2) how information flows
among these systems. A concept of operations would also provide a useful
tool to explain how business systems at the agency can operate cohesively.
It would be geared to a NASA-wide solution rather than individual
stovepiped efforts. Further, it would provide a road map that can be used
to (1) measure progress and (2) focus future efforts.
As part of an agencywide concept of operations, to best leverage its
investment in IEMP, NASA should also analyze its current business
processes and determine how these processes can be made more efficient and
effective. Specifically, it will be important to define standard business
processes supported by its IEMP software that result in streamlined
operations rather than simply automating the old ways of doing business.
To best leverage its investment in IEMP, NASA needs to ensure that the
business processes supported by this system are developed and implemented
to support the enterprise's needs rather than primarily focusing on the
parochial needs of a specific organizational entity. For example, system
efforts targeted at addressing accounting or external financial reporting
needs do not provide reasonable assurance that the needs of the program or
mission managers are addressed. With an ERP solution, one source of data
is used for multiple purposes and processes should be designed to ensure
that the data obtained and recorded meet the needs of the enterprise. NASA
can take advantage of the efficiencies inherent in its ERP solution by
allowing the data needed for external financial reporting to be produced
as a by-product of the processes it uses to manage its mission.
We are making five recommendations aimed at improving NASA's processes for
managing IEMP as well as addressing the higher-level strategic issues
associated with developing a concept of operations and defining standard
business processes supported by NASA's IEMP software. In written comments
on a draft of this report, NASA agreed with all five of our
recommendations and described the steps that it is taking to improve its
enterprise management system modernization efforts. NASA's comments are
discussed further in the Agency Comments and Our Evaluation section and
are reprinted in appendix II.
Background
For more than a decade, we have identified weak contract management and
the lack of reliable financial and performance information as posing
significant challenges to NASA's ability to effectively run its largest
and most costly programs. While NASA has made some progress in addressing
its contract management weaknesses through improved management controls
and evaluation of its procurement activities, NASA has struggled to
implement a modern, integrated financial management system. NASA made two
efforts in the past to improve its financial management processes and
develop a supporting system intended to produce the kind of accurate and
reliable information needed to manage its projects and programs and
produce timely, reliable financial information for external reporting
purposes. However, both of these efforts were eventually abandoned after a
total of 12 years and a reported $180 million in spending.
IEMP Implementation Status
In April 2000, NASA began its third attempt at modernizing its financial
management processes and systems. With its current financial management
system effort, known as IEMP, NASA has invested in an ERP solution that is
intended to meet the information needs of both internal and external
customers and to promote standardization and integration of business
processes and systems across the agency. NASA plans to complete IEMP by
2009 for a total cost of over $800 million.
As of March 2007, NASA had deployed the following nine IEMP functional
components: core financial, Travel Manager, ERASMUS,7 resume management,
position description management, budget formulation, Agency Labor
Distribution System, Project Management Information Improvement, and
contract management.8 Early in fiscal year 2007, NASA also implemented an
updated version of the core financial software, which includes several
critical enhancements to the previous core financial software.9 According
to NASA, the core financial upgrade provided the opportunity for it to
leverage the best practices inherent in the new version and allowed it to
redesign or enhance business processes. NASA updated its core financial
system in order to improve compliance with Federal Financial Management
System Requirements, Federal Accounting Standards, and the Federal
Financial Management Improvement Act, and to respond to GAO
recommendations. According to NASA, the software upgrade has enabled it to
implement critical process changes related to financial tracking and
reporting, support the goal of achieving financial management integrity,
and provide better project management information. NASA claims that the
updated software has also provided other enhancements, which should
contribute to NASA's goals of achieving a clean audit opinion and
achieving a "Green" rating on the President's Management Agenda scorecard
for "improved financial performance." Other IEMP modules that NASA plans
to implement in the future include aircraft management and asset
management.
7ERASMUS is an executive management information system that provides
information on costs, schedule, and risks for all significant NASA
programs and projects. According to the IEMP Program Director, ERASMUS was
discontinued in April 2007.
8The contract management module of IEMP is intended to support contract
and grant writing and administration, and procurement workload management.
Prior Reporting on IEMP
As discussed previously, we issued a series of four reports in April and
November 2003 that detailed weaknesses in NASA's acquisition and
implementation strategy for IEMP in general and the core financial module
in particular. The core financial module, which utilizes SAP software and
is considered the backbone of IEMP, was implemented in June 2003. Because
NASA did not follow key best practices or disciplined processes for
acquiring and implementing IEMP, we reported that NASA had made a
substantial investment in a financial management system that fell far
short of its stated goal of providing meaningful, reliable, and timely
information to support effective day-to-day program management and
external financial reporting. We noted problems in the areas of
requirements development, requirements management, testing, performance
metrics, risk management, and business process reengineering.
o Neither program managers nor cost estimators were involved in
the process of defining requirements for the core financial
module. As a result, the module was not designed to maintain the
level of detailed cost information needed by program managers to
perform contract oversight and by cost estimators to develop
reliable cost estimates.
o The requirements management methodology and tools used to
implement the core financial module did not result in requirements
that were consistent, verifiable, and traceable or that contained
enough specificity to minimize requirement-related defects.
Because NASA had not effectively implemented disciplined
requirements management processes,10 we reported that it had
increased the risk that it would not be able to effectively
identify and manage the detailed system requirements necessary to
properly acquire, implement, and test the core financial module.
o NASA's ability to effectively test the core financial module was
limited because of the lack of complete and specific requirements.
Industry best practices, as well as NASA's own system planning
documents, indicated that detailed system requirements should be
documented to serve as the basis for effective system testing.11
Because the link between these two key processes was not
maintained, NASA had little assurance that all requirements were
properly tested.
o NASA also did not effectively capture the type of metrics that
could have helped the agency understand the effectiveness of its
IEMP management processes. For example, NASA did not employ
metrics to help it identify and quantify weaknesses in its
requirements management processes. Because of its lack of
performance metrics, NASA was unable to understand (1) its
capabilities to manage IEMP projects; (2) how its process problems
affected cost, schedule, and performance objectives; and (3) the
corrective actions needed to reduce the risks associated with the
problems identified.
o NASA did not consistently identify known and potential risks for
the core financial module. Risk management processes are needed to
ensure that a project's risk is kept at an acceptable level by
taking actions to mitigate risk before it endangers the project's
success.
o NASA did not use the implementation of IEMP to fundamentally
change the way it did business. Instead of reengineering its
business processes, NASA automated many of its existing
ineffective business processes. First, NASA did not design the
system to accommodate the information needed to adequately oversee
its contracts and programs and to prepare credible cost estimates.
Second, NASA did not reengineer its contractor cost reporting
processes and therefore, did not always obtain sufficient contract
cost information needed by program managers to oversee contracts
and needed by financial managers for external financial reporting.
9Both the previous and updated versions of the core financial software are
from SAP, a company whose integrated software is used by many of the
world's largest corporations.
10According to the Software Engineering Institute, requirements management
is a process that establishes a common understanding between the customer
and the software project manager regarding the customer's business needs
that will be addressed by a project. A critical part of this process is to
ensure that the requirements development portion of the effort documents,
at a sufficient level of detail, the problems that need to be solved and
the objectives that need to be achieved.
11Testing is the process of executing a program with the intent of finding
errors.
When we last reported on NASA's IEMP effort, in September 2005,12 NASA had
begun to implement a number of recommendations from our earlier
reports--including steps toward implementing the disciplined processes
necessary to manage IEMP. For example, we reported that NASA had engaged
program managers to identify program management needs, implemented new
requirements management and testing processes, and developed metrics to
evaluate the effectiveness of its system implementation processes.
However, at that time, the agency had not implemented several of our other
recommendations, including the following:
o Properly define and document system requirements for
already-deployed IFMP modules, including the core financial
module. This is important not only because it would affect the way
the core financial module functions but also because it would
affect NASA's ability to implement future upgrades and other
modules expected to interface with the core financial module.
o Enhance regression testing processes and performance metrics.
o Develop a risk mitigation plan.
o Reengineer its business processes so that the commercial
off-the-shelf software products selected for IEMP could support
these processes.
At the time of our last report, NASA was making plans to reengineer some
of its business processes. However, because the agency was in the very
early planning stage of implementing this recommendation, the details for
how NASA would accomplish its objectives were still vague. Overall, our
September 2005 report concluded that it was not possible to assess whether
NASA's plans would accomplish its stated goal of enhancing the core
financial module to provide better project management information for
decision-making purposes.
12 [31]GAO-05-799R .
Progress Made in Developing Disciplined Project Management Processes, but Some
Problems Remain
Since September 2005, when we last reported on NASA IEMP implementation
efforts, NASA has implemented some of the disciplined processes needed to
manage IEMP. Specifically, NASA has, as we previously recommended,
implemented more effective requirements management and testing processes,
improved its performance metrics program related to tracking system
defects, and developed an IEMP risk management strategy. In addition, NASA
has developed quantitative entry and exit criteria for moving from one
phase of an IEMP project to another--a recognized industry best practice.
However, weaknesses in the areas of requirements development and project
scheduling have undermined some of the progress made in other key areas.
As a result, NASA struggled to complete required systems testing and
deliver the agency's core financial upgrade. Ultimately, through the
heroic efforts of the core financial upgrade team, NASA delivered the
upgrade within about 2 weeks of the October 30, 2006, planned completion
date. According to NASA officials, the system is functioning as expected
for most transactions. However, until the end of March 2007, the upgrade
was in a "stabilization" phase as NASA worked on correcting a number of
system errors, including posting errors for certain types of transactions.
Because the upgrade was still quite new and NASA was continuing to
stabilize the system, we were unable to determine the significance of
these weaknesses.
Improvements Made to NASA Requirements Management and Testing Processes
Since our September 2005 report, NASA has used its new requirements
management process--which documents sufficiently detailed requirements
that are traceable from the highest (most general) level to the lowest
(most detailed) level in NASA's requirements management system--for both
the core financial upgrade and the contract management module. For
example, we selected several requirements for both the core financial
module and the contract management module and validated that the
requirements management process (1) clearly linked related requirements
consistent with industry standards and (2) contained the information
necessary to understand how each requirement should be implemented and
tested in a quantitative manner.
Because NASA developed and is now using a disciplined requirements
management process, it has the quantitative information necessary to
support disciplined testing processes. NASA's disciplined testing
processes include (1) documentation of the scenarios that need to be
tested to obtain adequate test coverage, (2) requirements that are traced
to the test cases to ensure that all requirements are tested, (3)
instructions and other guidance for the testers, and (4) an effective
regression testing program.13 Although NASA had disciplined requirements
management and testing processes in place for the implementation of both
the contract management module and the core financial upgrade,
difficulties related to requirements development and project scheduling,
discussed later, forced NASA to compress the testing phase of its core
financial upgrade implementation. As a result, according to NASA
officials, completion of testing for the core financial upgrade required
an extraordinary effort on the part of NASA's implementation team.
NASA Has Implemented an Effective Metrics Program and Risk Management Strategy
Since we last reported, in September 2005, NASA has also enhanced its
metrics measurement program, which is used to evaluate the effectiveness
of its project management processes by identifying the causes of process
defects. Understanding the cause of a defect is critical to evaluating the
effectiveness of an organization's project management processes, such as
requirements management and testing. For example, if a significant number
of defects are caused by inadequate requirements definition, then the
organization knows that corrective actions are needed to improve the
requirements definition process. When we last reported, NASA had made
progress in this important area by collecting information on the causes of
system defects it identified in its regression testing efforts but was not
collecting similar information on defects identified by users and lacked a
formal process for fully analyzing the data related to system defects by
identifying the trends associated with them. Since that time, NASA has
developed additional metrics to track and analyze such things as the
number of changes made to requirements while a system is under
development. In addition, NASA has developed processes for tracking and
analyzing defects identified by IEMP users. For example, since
implementation of the core financial upgrade, NASA has maintained
spreadsheets showing specific information on each service request
submitted by users, including the type of defect involved and the status
of the request.
13Regression testing is the practice of testing changes to a software
application before it is released to ensure that modifications have not
caused unintended effects and that the system still complies with its
specified requirements.
Finally, NASA has also developed a comprehensive risk management strategy.
Specifically, NASA now has an IEMP Risk Management Plan that outlines the
standard processes and techniques for identifying, analyzing, planning,
tracking, and controlling risks as well as defining the roles and
responsibilities for each level of project risk management. In applying
these techniques to the core financial upgrade, NASA officials documented
the risks that they identified for the project, as well as their
mitigation strategies, likelihood, consequence, and criticality. According
to NASA officials, their risk management process worked well and was one
of the key reasons for the success of the core financial upgrade. For
example, using the metrics information discussed previously, NASA
officials said they were able to assess the risks of changing requirements
late in the project and then mitigate those risks by performing additional
testing.
NASA Uses a Recognized Industry Best Practice to Move from One Project Phase to
the Next
In addition to the disciplined processes discussed above, NASA has also
taken action to establish the use of quantitative entry and exit criteria
to move from one phase of an IEMP project to another. The use of such
criteria is considered an industry best practice. Entry criteria are the
minimum essential items considered necessary to enter into a given project
phase, while exit criteria are the minimum essential items necessary to
consider a given project phase successfully completed. For example, the
NASA entry criterion for moving into the regression testing phase requires
that all remaining significant defects from the integration testing phase
be resolved and successfully retested before regression testing can begin.
NASA demonstrated application of this criterion when it implemented the
contract management module. About 3 weeks before the scheduled start date
of regression testing, the project had not yet successfully completed all
test scenarios, and several significant defects had not been fully
resolved. In addition, a series of critical corrections from the software
vendor had not yet been delivered, and the project team agreed that there
would not be adequate time to test the corrections prior to beginning the
scheduled regression testing. Consequently, the team decided to push back
the scheduled date for the contract management module to begin operating.
For the core financial upgrade, NASA officials said that they used entry
and exit criteria as one of the management tools to determine whether the
project should move forward. However, rather than adopt a "hard stop"
approach when criteria were not met, they used the criteria to make sure
that all appropriate factors were considered before moving forward,
including the risks of not meeting certain criteria. Any instances in
which the project team thought exceptions to the criteria were warranted
were ultimately reviewed and decided on by higher levels of NASA
management, which helped ensure that such decisions were adequately
considered.
Improved Requirements Development and Project Scheduling Needed
Weaknesses in the areas of requirements development and project scheduling
offset some of the benefits associated with NASA's improved requirements
management and testing processes--causing NASA to assume a greater risk
that it would not identify significant system defects prior to
implementation. Weaknesses in requirements development and project
scheduling processes resulted in NASA having to compress the testing phase
of its core financial upgrade implementation. As a result, according to
NASA officials, NASA's ability to complete testing for the core financial
upgrade within the planned implementation time frames ultimately depended
on the extraordinary effort put forth by NASA's project implementation
team.
Because of weaknesses in NASA's requirements development process, it did
not have reasonable assurance that it identified all appropriate
requirements for the core financial upgrade when the project began.
Consequently, NASA continued making changes to the requirements very late
in the project's development, resulting in increased risks, delays, and a
compressed testing schedule. Improperly defined or incomplete requirements
have commonly been identified as a root cause of system failure. Although
NASA made a concerted effort, as part of its core financial system
upgrade, to involve program managers and other key stakeholders in the
requirements development process, it did not follow standard industry
practices for identifying and documenting user requirements.
According to the Software Engineering Institute (SEI),14 to help ensure
that critical requirements are identified, an organization should have a
well-documented, disciplined requirements development process that, among
other things, (1) defines how customer needs will be elicited, developed,
and validated; (2) specifies how to identify and ensure involvement of
relevant stakeholders; and (3) ensures that people involved in the
requirements development process are adequately trained in such topics as
requirements definition and analysis. In addition, it is critical that
requirements flow from an organization's business requirements or its
concept of operations. However, as discussed later, NASA has not yet
completed a concept of operations.
14Mary Beth Chrissis, Mike Konrad, and Sandy Shrum, CMMI: Guidelines for
Process Integration and Product Improvement, SEI Series in Software
Engineering (Boston, Mass.: Pearson Education, May 2004).
In developing its core financial upgrade requirements, NASA established a
task force, consisting of both financial and program managers, whose
primary objective was to "review, assess, and document Program/Project
Management requirements as they relate to financial management." In
addition, other groups of program managers were asked to review the
requirements and provide input to the task force. However, according to
NASA officials, they have not yet documented and institutionalized
requirements development procedures as recommended by SEI. Lacking
documentation, NASA cannot ensure that appropriate procedures are followed
and that all appropriate stakeholders are included in the process so that
all requirements are identified. Moreover, the requirements that were
addressed by the task force and user groups were at a very high or general
level and therefore, lacked a level of specificity that is needed to
ensure that users' needs are met.
Because it did not have a well-documented, disciplined requirements
development process in place to provide reasonable assurance that all
requirements had been identified, NASA delayed finalizing the system's
expected functionality until April 2006--about 6 months before the upgrade
was expected to be implemented--and continued to change some requirements
for several months after that. Delays in finalizing the requirements
contributed to delayed testing and a compressed testing schedule. To meet
the planned October 30, 2006, implementation date, the three rounds of
system testing for the core financial upgrade were scheduled to occur from
mid-June through September 22, with less than a week between each round. A
less compressed schedule could have allowed more time between the testing
cycles to perform necessary actions, such as additional development work
and testing to adequately address the defects that had been identified.
This, in turn, could have reduced the risk that significant system defects
would not be detected prior to implementation.
One key to developing a realistic project schedule is determining the
sequence of activities, which requires identifying and documenting the
dependencies among the various project activities. For example, testing
activities cannot be completed before the software being tested is
developed, and software should not be developed until requirements have
been defined. However, NASA did not document the dependencies among the
detailed project tasks for the core financial upgrade and therefore, did
not have reasonable assurance that the project schedule established at the
start of the project was realistic. According to NASA officials, they
recognized this risk and adopted several processes to identify and
mitigate the weakness, such as having knowledgeable project officials
review the schedule and holding weekly status meetings to determine
whether the tasks were on schedule.
While the techniques used by NASA to constantly evaluate and adjust the
schedule are considered best practices and allowed NASA to gain confidence
in the schedule as the core financial upgrade project progressed, they
were not sufficient to ensure that the original schedule was reasonable
because they relied on ad hoc processes rather than a formal task
dependency analysis. If NASA had also identified the task dependencies for
the core financial upgrade, it would likely not have had to rely on
extraordinary efforts to complete the project. Rather, project management
would have been in a better position to assess the difficulty in meeting
the planned schedule and to take further steps to reduce this risk, such
as scaling back some aspects of the project or adding more resources to
the project.
According to NASA officials, through the heroic efforts of IEMP
staff--their knowledge and experience with past projects and a
considerable amount of overtime invested--the core financial project team
was able to complete testing and other work within about 2 weeks of the
planned implementation date. Although NASA has made significant
improvements in its project management processes, NASA management
recognizes that weaknesses in its requirements development and project
scheduling processes have undermined some of the progress made. Despite
the implementation difficulties, NASA financial managers have indicated
that the core financial upgrade is now functioning as expected for most
transactions. Through the end of March 2007, the upgrade was in a
"stabilization" phase as NASA continued to work on correcting a number of
system errors, including posting errors for certain types of transactions.
Because NASA was continuing to stabilize the system during most of our
audit period, we were unable to determine the significance of these
weaknesses.
NASA Has Not Yet Fully Considered an Enterprise View of Its Operations and
Processes
Although NASA has significantly improved its processes for implementing
IEMP projects, these improvements are directed at implementing the desired
functionality for an individual project. NASA has not yet fully considered
the higher-level strategic issues that affect how useful IEMP will be in
addressing long-standing management challenges--including problems
associated with stovepiped systems and parochial interests of individual
NASA components as well as problems in overseeing contractor performance
and properly accounting for its property, plant, and equipment. NASA
envisions IEMP to be a leading-edge business system15 that will provide
management information needed for mission success, meet the needs of
internal and external customers, and promote standardization and
integration of business processes and systems across NASA. To achieve this
vision, it is critical that NASA develop an agencywide concept of
operations and adopt standard business processes that are supported by its
software.
Concept of Operations Would Provide an Important Foundation for IEMP
NASA officials stated that they have undertaken a critical first step to
achieving their vision for IEMP--they have begun developing a concept of
operations to describe how all of its business processes should be carried
out. NASA created a framework for developing a concept of operations in
fiscal year 2006 and plans to complete it by the summer of 2008, according
to NASA officials. Ideally, a concept of operations should be completed
before system development begins so that it can serve as a foundation for
system planning and requirements development. Nonetheless, the completion
of such a document even at this late stage in NASA's IEMP effort would be
beneficial for the development of the remaining IEMP modules as well as
any future upgrades to the core financial module. In addition, once a
concept of operations is complete, NASA could reassess the modules that
are already implemented and determine whether and how they might need to
be modified to best meet its agencywide needs.
A concept of operations defines how an organization's day-to-day
operations are (or will be) carried out to meet mission needs. The concept
of operations includes high-level descriptions of information systems,
their interrelationships, and information flows. It also describes the
operations that must be performed, who must perform them, and where and
how the operations will be carried out. Further, it provides the
foundation on which requirements definitions and the rest of the systems
planning process are built. Normally, a concept of operations document is
one of the first documents to be produced during a disciplined development
effort and flows from both the vision statement and the enterprise
architecture.16 According to Institute of Electrical and Electronics
Engineers (IEEE) standards,17 a concept of operations is a user-oriented
document that describes the characteristics of a proposed system from the
users' viewpoint. The key elements that should be included in a concept of
operations are major system components, interfaces to external systems,
and performance characteristics such as speed and volume.
15A business system is an information system that is used to support
business activities such as acquisition, financial management, logistics,
strategic planning and budgeting, installations and environment, and human
resources management.
For NASA, an effective concept of operations would describe, at a high
level, (1) how all of the various elements of NASA's business systems
relate to each other and (2) how information flows among these systems.
Further, a concept of operations would provide a useful tool to explain
how business systems at the agency can operate cohesively. It would be
geared to a NASA-wide solution rather than individual stovepiped
efforts.18 Further, it would provide a road map that can be used to (1)
measure progress and (2) focus future efforts. While NASA's enterprise
architecture efforts, when fully completed, can be used to help understand
the relationships between the various systems, a concept of operations
document presents these items from the users' viewpoint in nontechnical
terms. Such a document would be invaluable in getting various
stakeholders, including those in the programs and administrative
activities, to understand how the business systems are expected to operate
cohesively and how they fit into "the big picture."
16An enterprise architecture is a blueprint that defines, both in logical
terms (including integrated functions, applications, systems, users, work
locations, and information needs and flows) and technical terms (including
hardware, software, data, communications, and security), how an
organization's information technology systems operate today and how they
are to operate in the future and provides a road map for the transition.
17IEEE Std. 1362-1998.
18For example, as we discuss later, NASA receives two different types of
cost reports from its major contractors. Even though both types of reports
pertain to the same costs for a given contract, one report is used for
financial management while the other is used for program management. A
concept of operations might describe how NASA could use the information
from only one report for both purposes.
Adopting Enterprise Business Processes Would Help NASA Transform the Way It Does
Business
As part of an agencywide concept of operations, to best leverage its
investment in IEMP, NASA should also analyze the agency's current business
processes and determine how these processes can be made more efficient and
effective. Specifically, NASA needs to ensure that the business processes
supported by this system are developed and implemented to support the
enterprise's needs rather than primarily focusing on the needs of a
specific organizational entity. For example, system efforts targeted only
at addressing accounting or external financial reporting needs--as was
done during the initial implementation of the core financial module--do
not provide reasonable assurance that the needs of the mission managers or
other support organizations are addressed as well. Our review identified
an important opportunity for NASA to leverage its investment in IEMP by
using the system's inherent business processes to meet the enterprise's
needs.
Agencies such as NASA that invest in ERP solutions to meet their
enterprise needs often face difficulty in shifting from the stovepiped
processes of the past to the enterprise processes that underlie the ERP
concept. According to technical experts,19 a key benefit of an effective
ERP system is that the system provides the entire entity consistent data
regardless of which entity component generates a request or for what
purpose; the system maintains data based on the concept of "one truth." In
other words, in non-ERP environments, one system may have one amount for
an agency's obligations while another system has another amount for the
same obligations. While either of these systems may be the "official
system," actions and plans may be based on information in the other
system. In order for all of an organization's actions and plans to be
consistent, the same information needs to be available and used by all
segments of that organization. Under the ERP concept, it does not matter
whether an individual is in budget, accounting, procurement, or any other
organizational component; the answer to the question of "how much money
has been obligated and how much is still available" is consistent.
One example of an opportunity for NASA to use enterprise processes to
accomplish multiple needs is in the area of program oversight and
accounting for PP&E. NASA typically spends about 85 percent of its budget
procuring goods and services from its contractors each year. Therefore,
much of the cost information NASA needs to oversee its programs and
compile its external financial reports resides with its contractors. For
its larger contracts,20 NASA generally obtains cost data from monthly
contractor financial management reports, commonly referred to as NASA Form
533s. NASA Form 533 captures planned and actual contract costs and,
according to NASA officials, is used for budgeting, monitoring contract
costs, and controlling program resources. The Office of the Chief
Financial Officer (OCFO) also uses NASA Form 533 to capture the costs
reported on the agency's financial statements. However, NASA Form 533 does
not contain information related to the status of work performed on a
contract. Therefore, for all major acquisitions21 and for development or
production contracts and subcontracts valued at $20 million or more, in
addition to NASA Form 533s, NASA's contractors are also required to
provide monthly contract cost performance reports. Each of these reports
is treated as a stovepiped activity; that is, they provide cost
information for a given contract in two different formats and are used by
different organizations and for different purposes within NASA.
19Thomas F. Wallace and Michael H. Kremzar, ERP: Making It Happen; The
Implementers' Guide to Success with Enterprise Resource Planning (New
York: John Wiley & Sons, Inc., 2001).
For those contracts for which NASA receives contract cost performance
reports in addition to Form 533s, program managers use the cost
performance reports to monitor contract performance, while the OCFO uses
NASA Form 533 to accrue costs that, among other things, are reported on
the agency's financial statements. Although NASA Form 533 and the cost
performance report reflect cost data pertaining to the same contract, the
level of detail provided in each report may vary considerably depending on
the contractor cost reporting requirements negotiated as part of the
contract. For example, the cost data required by program managers to
manage major acquisitions are often more detailed than those required by
the OCFO. In addition, because neither the cost performance report nor
NASA Form 533 contains the information needed by the OCFO to properly
account for equipment and other property acquired from contractors, NASA
also relies on periodic, summary-level information provided by its
contractors to report property amounts on its financial statements.
20NASA requires its contractors to report monthly accrued costs on NASA
Form 533 for cost type, price redetermination, and fixed-price incentive
contracts with a performance period of 1 year or more and a contract value
of $500,000 to $999,000 or a performance period of less than a year but
with a contract value of $1 million or more.
21A major acquisition, as defined by OMB Circular A-11, means a system or
project requiring special management attention because of its importance
to the mission or function of the agency, a component of the agency, or
another organization; is for financial management and obligates more than
$500,000 annually; has significant program or policy implications; has
high executive visibility; has high development, operating, or maintenance
costs; or is defined as major by the agency's capital planning and
investment control process.
When NASA initially implemented its IEMP core financial module in June
2003, it did not adequately consider program managers' needs and did not
design the system to accommodate the more detailed cost data contained in
contractor cost performance reports. Since that time, NASA has redesigned
the coding structure embedded in the core financial module to be more
consistent with the work breakdown structure (WBS) coding used by program
managers. However, NASA continues to use cost data from NASA Form
533--generally reported by contract line items22--to populate the core
financial module. As a result, as shown in figure 1, NASA uses a complex,
NASA-specific process to allocate the costs reported on NASA Form 533 to
the WBS codes in IEMP based on available funding.
22Contract line items are usually consistent with higher levels of the
WBS, but do not contain the details that are found in the lower levels of
the WBS.
Figure 1: Current NASA Processes for Oversight of Large, Complex Contracts
and for Asset Accounting
In a very simplified example,23 if NASA received a Form 533 showing $1,000
of cost incurred for a particular contract line item and two WBS codes
pertained to that line item, NASA would allocate the costs to those two
WBS codes. Assuming WBS 1 had more funding available than WBS 2, NASA
might allocate $600 to WBS 1 and $400 to WBS 2. However, the contract cost
performance report might show that the actual costs were $500 for WBS 1
and $500 for WBS 2. Although this allocation process reorganizes cost data
reported on NASA Form 533 into the same reporting structure that is used
by program managers, it still results in different costs, maintained in
different systems, used for different purposes. Accordingly, these
separate processes do not result in the "one truth" that is provided when
an ERP view is taken.
23This example is for illustrative purposes only; the dollar amounts in
the example are not based on actual NASA data.
Further, this dual reporting approach has not addressed one of NASA's
long-standing financial reporting weaknesses: reporting on its PP&E. For
example, NASA's processes do not allow the agency to identify capital
costs--that is, those that should be recorded as assets--as they are
incurred. Instead, as we recently reported,24 the agency performs a
retrospective review of transactions entered into its property system to
determine which costs should be capitalized. This subsequent review is
labor-intensive and error-prone, and therefore increases the risk that not
all related costs will be properly captured and capitalized.
Figure 2 provides an example of how NASA could use IEMP to implement an
enterprise process that (1) provides the necessary data for the enterprise
operations and (2) reduces the burden on NASA and contractor officials.
24GAO, Property Management: Lack of Accountability and Weak Internal
Controls Leave NASA Equipment Vulnerable to Loss, Theft, and Misuse,
GAO-07-432 (Washington, D.C.: June 25, 2007).
Figure 2: Example of How NASA Could Follow an Enterprise Process Using
IEMP for Program Management and External Reporting
As shown in figure 2, if NASA received only one monthly report containing
contract cost data reported in sufficient detail for both program
management and financial reporting purposes, then it could record these
costs directly in IEMP without first going through an allocation process
as it does now. All individuals and components throughout NASA could then
use the same cost data that reside within IEMP for a given contract; IEMP
could provide different arrays of cost information based on each user's
needs, but all cost information for a given contract would come from one
source. For example, as shown in figure 2, the program manager could use
the cost data from IEMP along with other supplemental contractor
performance information, such as labor hours used, to see if the project
is meeting expectations. In addition, if discrete WBS codes were
established to identify the costs associated with the acquisition of
property, then IEMP could automatically capitalize those costs and
financial managers could readily determine how much cost has been recorded
for property.25 The key is that under the enterprise process concept,
single data entry is used for multiple purposes. Since the enterprise view
provides "one truth," an adequate audit trail over the data used to report
property can be maintained simply by reviewing the cost reports that were
provided by the contractors. Thus, NASA can take advantage of the
efficiencies inherent in an ERP solution by allowing the data needed for
external financial reporting to be produced as a by-product of the
processes it uses to manage its mission.
Conclusions
NASA has made significant strides in developing and implementing more
disciplined processes for supporting its IEMP efforts since our last
report in 2005. NASA has recognized the need for the disciplined processes
necessary to reduce risks to acceptable levels, as evidenced by its
implementation of several of our recommendations. More importantly, NASA
officials recognize that improving system implementation processes is a
continuous effort and that certain processes--particularly requirements
development and project scheduling--may need more attention. However, the
real key to realizing NASA's IEMP vision is for NASA's management to
develop an overarching strategy for managing its agencywide management
system development effort. We are encouraged that NASA has begun to
develop a concept of operations. As part of the development of this
document, it will be critical for NASA to define (1) the agency's business
processes and information needs and (2) the types of systems that will be
used to carry out these processes and produce the necessary information.
Another critical factor in developing a concept of operations will be
analyzing the agency's current business processes and determining how
these processes can be made more efficient and effective. For example,
NASA can take advantage of the efficiencies inherent in the solution it
has selected by utilizing an enterprise view to produce the data needed
for external financial reporting as a by-product of the processes it uses
to manage its mission. Unless NASA devotes immediate, focused attention to
taking these critical strategic planning steps, it will continue to face
the risk that its planned $800 million investment in IEMP will not achieve
the transformational changes necessary to provide NASA with the
information needed to make well-informed business decisions and to
effectively manage its operations.
25In its technical comments on a draft of this report, NASA stated that it
plans to establish unique WBS codes for contractors to use to report asset
costs on the Form 533. It is too early to determine the extent to which
these plans might address agencywide information needs.
Recommendations for Executive Action
To help ensure that disciplined processes are effectively implemented for
future IEMP modules, upgrades, or other business systems, we recommend
that the NASA Administrator direct the IEMP Program Director to take the
following two actions.
o Establish requirements development policies and procedures
regarding (1) how customer needs will be elicited, developed, and
validated; (2) how to identify and ensure the involvement of
relevant stakeholders; and (3) required training in such topics as
requirements definition and analysis to be provided to people
involved in the requirements process.
o Develop policies and procedures that require project schedules
to include the identification and documentation of dependencies
among various project tasks.
To help ensure that future IEMP projects are designed to carry out NASA's
mission in an efficient manner that meets the needs of all users, we
recommend that the NASA Administrator establish as a high priority the
completion of a concept of operations that addresses NASA's business
operations for both its mission offices and administrative offices (such
as financial management and human capital) before any new implementation
efforts begin.
Once the concept of operations is complete, we recommend that the NASA
Administrator review the functionality of previously implemented IEMP
modules for the purpose of determining whether enhancements or
modifications are needed to bring them into compliance with the concept of
operations.
To help ensure that NASA receives the maximum benefit from its reported
$800 million investment in IEMP, we recommend that the NASA Administrator
establish policies and procedures requiring approval to establish or
maintain business processes that are inconsistent with the processes
inherent in the COTS solutions selected for IEMP. The reasons for any
decisions made to not implement the inherent COTS processes should be
well-documented and approved by the Administrator or his designee. At a
minimum, approved documentation should address any decisions to maintain
current contractor cost reporting processes rather than revise these
processes to facilitate the use of one consistent source of cost data.
Agency Comments and Our Evaluation
We received written comments on a draft of this report from NASA, which
are reprinted in appendix II. NASA agreed with our recommendations and
described the approach and steps it is taking or plans to take to improve
its enterprise management system modernization efforts. We are encouraged
that a number of these steps are already under way, including the
establishment of an IEMP advisory body representing NASA's missions and
centers. As NASA progresses in addressing our recommendations, it is
important that it focuses on the concepts and underlying key issues we
discussed, such as considering the need to reengineer key business
processes to support agencywide needs and to take full advantage of its
ERP solution. We continue to believe that careful consideration of all of
the building blocks and key issues we identified will be integral to the
success of NASA's efforts. NASA also provided technical comments, which we
incorporated as appropriate.
As agreed with your offices, unless you announce its contents earlier, we
will not distribute this report further until 30 days from its date. At
that time, we will send copies to interested congressional committees, the
NASA Administrator, and the Director of the Office of Management and
Budget. We will make copies available to others upon request. In addition,
the report will be available at no charge on the GAO Web site at
[32]http://www.gao.gov .
If you or your staff have any questions concerning this report, please
contact McCoy Williams at (202) 512-9095 or [33][email protected] or
Keith Rhodes at (202) 512-6412 or [34][email protected] . Key contributors
to this
report are acknowledged in appendix III. Contact points for our Offices of
Congressional Relations and Public Affairs may be found on the last page
of this report.
McCoy Williams
Director
Financial Management and Assurance
Keith A. Rhodes
Chief Technologist
Applied Research and Methods
Center for Technology and Engineering
Appendix I: Scope and Methodology
To determine whether the National Aeronautics and Space Administration
(NASA) has improved its management processes for implementing the
Integrated Enterprise Management Program (IEMP), we reviewed project
management documentation for several IEMP projects, including the core
financial upgrade and the contract management module. The documentation we
reviewed for these projects included requirements management documents,
detailed testing plans, project schedules, risk management plans, and
metrics documentation. We also interviewed numerous IEMP officials,
including the IEMP Director, the Director and Assistant Director at the
IEMP Competency Center, and the Manager of IEMP Application Development
and Software Assurance. In addition, we interviewed the leader of a NASA
team that provided an independent assessment of the core financial upgrade
project to obtain his views of IEMP management processes.
To assess NASA's implementation of disciplined processes, we reviewed
industry standards and best practices from the Institute of Electrical and
Electronics Engineers, the Software Engineering Institute, and the Project
Management Institute. To assess the effectiveness of NASA's requirements
management processes, we performed a traceability analysis of several
requirements for both the contract management module and the core
financial upgrade, which demonstrated that there was traceability among
the different levels of requirements and with testing documentation. To
determine whether NASA had adequately and systematically determined the
information needs of key users of IEMP data when developing system
requirements, we reviewed documentation of NASA's requirements
identification effort for the core financial upgrade and interviewed a
number of program managers and staff who worked on various space and
science programs at three NASA centers--Marshall Space Flight Center,
Johnson Space Center, and Goddard Space Flight Center. We also met with
officials from the Office of the Chief Financial Officer (OCFO), including
the Deputy Chief Financial Officer, and with officials from the Office of
the Chief Engineer to obtain their opinions regarding the requirements of
the core financial upgrade. In addition, we discussed the requirements
development methodology with IEMP management.
To determine the results of the implementation of the core financial
upgrade, we met with both IEMP and OCFO officials. We reviewed data on the
amount and types of system defects that were identified by users during
the project's stabilization phase. We also obtained written responses to
specific questions about the results of the implementation from three NASA
centers.
To determine the extent to which NASA has considered the higher-level
strategic issues associated with developing an enterprisewide concept of
operations and defining standard business processes, we met with senior
management from IEMP and the OCFO. In addition, we also discussed these
issues with senior officials in the Office of the NASA Administrator. We
also interviewed IEMP officials about NASA's current processes for
recording contract costs. We also discussed this issue with officials from
the OCFO, the Office of the Chief Engineer, and the Office of Program and
Institutional Integration. In addition, we obtained documentation of
NASA's plans for reengineering processes related to the costs of capital
assets. We briefed NASA officials on the results of our audit, including
our findings and their implications. On May 25, 2007, we requested
comments from NASA and we received them on June 21, 2007. NASA also
separately provided technical comments. Our work was performed from
January 2006 through June 2007 in accordance with U.S. generally accepted
government auditing standards.
Appendix II: Comments from the National Aeronautics and Space
Administration
Appendix III: GAO Contacts and Staff Acknowledgments
GAO Contacts
McCoy Williams, (202) 512-9095 or [35][email protected]
Keith A. Rhodes, (202) 512-6412 or [36][email protected]
Acknowledgments
In addition to the contacts named above, staff members who made key
contributions to this report were Diane Handley, Assistant Director; J.
Christopher Martin, Senior Level Technologist; Francine DelVecchio; Kristi
Karls; and Lauren Catchpole.
(195100)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( [37]www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
[38]www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [39]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [40][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [41][email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [42][email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
[43]www.gao.gov/cgi-bin/getrpt?GAO-07-691 .
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact McCoy Williams at (202) 512-9095 or Keith
Rhodes at (202) 512-6412.
Highlights of [44]GAO-07-691 , a report to congressional requesters
July 2007
BUSINESS MODERNIZATION
NASA Must Consider Agencywide Needs to Reap the Full Benefits of its
Enterprise Management System Modernization Effort
Since 1990, GAO has designated the National Aeronautics and Space
Administration's (NASA) contract management as an area of high risk in
part because it lacked modern systems to provide accurate and reliable
information on contract spending. In April 2000, NASA began a system
modernization effort, known as the Integrated Enterprise Management
Program (IEMP). When GAO last reported on the status of IEMP in September
2005, NASA had begun to implement disciplined processes needed to manage
IEMP, but had yet to implement other best practices such as adopting
business processes that improve information on contract spending. This GAO
report addresses (1) actions taken by NASA to effectively implement the
disciplined processes needed to manage IEMP and (2) the extent to which
NASA has considered the strategic issues associated with developing a
concept of operations and defining standard business processes. GAO
interviewed NASA officials and obtained and analyzed documentation
relevant to the issues.
[45]What GAO Recommends
GAO recommends five new actions directed at improving the processes used
to manage IEMP, developing a concept of operations, and defining standard
business processes. NASA concurred with all five recommendations and
described steps it is taking to improve its enterprise management system
modernization efforts.
Since GAO last reported on NASA's IEMP efforts, NASA implemented its IEMP
contract management module and upgraded the software used for its core
financial module. NASA has also taken steps to improve its processes for
managing IEMP--including implementing improved requirements management and
testing processes, enhancing its performance metrics related to tracking
system defects, and developing an IEMP risk mitigation strategy. Further,
NASA has developed quantitative entry and exit criteria for moving from
one phase of an IEMP project to another--a recognized industry best
practice. However, NASA has not yet addressed weaknesses in the areas of
requirements development and project scheduling, which ultimately caused
the agency to assume a greater risk that it would not identify significant
system defects prior to implementation of the core financial upgrade.
Despite these difficulties, NASA financial managers have stated that the
core financial upgrade is now functioning as expected for most
transactions. As of the end of GAO's audit work in May 2007, NASA was
working to correct a number of system errors, including posting errors for
certain types of transactions. Because NASA was still working to stabilize
the system, GAO was unable to determine the significance of these
weaknesses.
Further, NASA has not yet fully considered higher-level strategic issues
associated with developing an agencywide concept of operations and
defining standard business processes. With a planned investment of over
$800 million for IEMP, NASA must immediately and effectively address these
strategic building blocks if IEMP is to successfully address long-standing
management challenges--including overseeing contractor performance and
properly accounting for NASA's property, plant, and equipment.
o NASA officials stated that they have begun developing a concept
of operations to describe how all of its business processes should
be carried out. According to NASA officials, they expect to
complete the concept of operations by the summer of 2008. Ideally,
a concept of operations should be completed before system
development begins so that it can serve as a foundation for system
planning and requirements development. Nonetheless, while NASA's
IEMP efforts are already well under way, the completion of such a
document remains essential for guiding the development of the
remaining IEMP modules as well as any future upgrades.
o As part of developing a concept of operations, NASA should also
define standard business processes that are supported by its IEMP
software. NASA needs to ensure that its business processes and the
information that flows from those processes support the
enterprise's needs. Efforts that primarily focus on the parochial
needs of a specific organizational unit, such as accounting, do
not provide reasonable assurance that NASA's agencywide management
information needs are addressed.
References
Visible links
24. http://www.gao.gov/cgi-bin/getrpt?GAO-07-310
25. http://www.gao.gov/cgi-bin/getrpt?GAO-03-507
26. http://www.gao.gov/cgi-bin/getrpt?GAO-04-151
27. http://www.gao.gov/cgi-bin/getrpt?GAO-04-43
28. http://www.gao.gov/cgi-bin/getrpt?GAO-04-118
29. http://www.gao.gov/cgi-bin/getrpt?GAO-06-184
30. http://www.gao.gov/cgi-bin/getrpt?GAO-05-799R
31. http://www.gao.gov/cgi-bin/getrpt?GAO-05-799R
32. http://www.gao.gov/
33. mailto:[email protected]
34. mailto:[email protected]
35. mailto:[email protected]
36. mailto:[email protected]
37. http://www.gao.gov/
38. http://www.gao.gov/
39. http://www.gao.gov/fraudnet/fraudnet.htm
40. mailto:[email protected]
41. mailto:[email protected]
42. mailto:[email protected]
43. http://www.gao.gov/cgi-bin/getrpt?GAO-07-691
44. http://www.gao.gov/cgi-bin/getrpt?GAO-07-691
*** End of document. ***