Information Technology: Improvements Needed to More Accurately
Identify and Better Oversee Risky Projects Totaling Billions of
Dollars (07-SEP-06, GAO-06-1099T).
The Office of Management and Budget (OMB) plays a key role in
overseeing federal IT investments. The Clinger-Cohen Act, among
other things, requires OMB to establish processes to analyze,
track, and evaluate the risks and results of major capital
investments in information systems made by agencies and to report
to Congress on the net program performance benefits achieved as a
result of these investments. OMB has developed several processes
to help carry out its role. For example, OMB began using a
Management Watch List several years ago as a means of identifying
poorly planned projects based on its evaluation of agencies'
funding justifications for major projects, known as exhibit 300s.
In addition, in August 2005, OMB established a process for
agencies to identify high risk projects, i.e., projects requiring
special attention because of one or more reasons specified by
OMB, and to report on those that are poorly performing or not
meeting performance criteria. GAO recently issued reports on the
Management Watch List, high risk projects, and agencies' exhibit
300s. GAO was asked to summarize (1) the number of projects and
the fiscal year 2007 dollar value of Management Watch List and
high risk projects, (2) previously reported results on how these
projects are identified and provided oversight, and (3)
recommendations it made to improve these processes.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-1099T
ACCNO: A60447
TITLE: Information Technology: Improvements Needed to More
Accurately Identify and Better Oversee Risky Projects Totaling
Billions of Dollars
DATE: 09/07/2006
SUBJECT: Agency evaluation
Budgeting
Cost analysis
Financial management
Information resources management
Information technology
Internal controls
Performance management
Program evaluation
Reporting requirements
Risk management
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-1099T
United States Government Accountability Office
Testimony
GAO
Before the Subcommittee on Federal Financial Management, Government Information
and International Security, Committee on Homeland Security and Governmental
Affairs, U.S. Senate
For Release on Delivery Expected at 9:30 a.m. EDT Thursday, September 7,
2006
INFORMATION TECHNOLOGY
Improvements Needed to More Accurately Identify and Better Oversee Risky
Projects Totaling Billions of Dollars
Statement of David A. Powner Director, Information Technology Management
Issues
GAO-06-1099T
INFORMATION TECHNOLOGY
Improvements Needed to More Accurately Identify and Better Oversee Risky
Projects Totaling Billons of Dollars
What GAO Found
As a result of the Management Watch List and high risk projects processes,
about 300 projects totaling about $12 billion in estimated IT expenditures
for fiscal year 2007 have been identified as being either poorly planned
or poorly performing. Specifically, of the 857 major IT projects in the
President's budget for fiscal year 2007, OMB placed 263 projects,
representing about $10 billion on its Management Watch List. In addition,
in response to OMB's memorandum, agencies reported that 79 of 226 high
risk projects, collectively totaling about $2.2 billion, had a performance
shortfall.
While this information helps to focus both agency and OMB management
attention on these poorly planned and poorly performing projects, GAO
identified opportunities to strengthen how these projects are identified
and provided oversight.
o The Management Watch List may be undermined by inaccurate and
unreliable data. OMB uses scoring criteria to evaluate agencies'
exhibit 300s to derive the projects on its Management Watch List.
GAO's detailed evaluation of exhibit 300s showed that the information
reported in them is not always accurate or supported by documentation.
o The criteria for identifying high risk projects were not always
consistently applied and projects that appeared to meet the criteria
were not identified as high risk. Without consistent application of
the high risk criteria, OMB and agency executives cannot have the
assurance that all projects that require special attention have been
identified.
o For both sets of projects, OMB did not develop a central list of
projects and deficiencies that could facilitate tracking progress and
reporting to Congress. Without such lists, OMB is not fully exploiting
the opportunity to analyze and track these projects on a
governmentwide basis and not involving Congress in the oversight of
these projects with risks.
To improve the way the Management Watch List and high risk projects are
identified and provided oversight, GAO has made a number of
recommendations to the Director of OMB. These recommendations include
directing agencies to improve the accuracy and reliability of exhibit 300
information and to consistently apply the high risk criteria defined by
OMB. In addition, GAO recommended that the Director develop a single,
aggregate list for both the Management Watch List and high risk projects
to facilitate tracking progress, performing governmentwide analysis, and
reporting the results to Congress. OMB generally disagreed with these
recommendations. However, GAO believes that they are needed to provide
greater assurance that poorly planned and poorly performing projects are
more accurately identified and provided oversight, and ultimately ensure
that potentially billions of taxpayer dollars are not wasted.
United States Government Accountability Office
Mr. Chairman and Members of the Subcommittee:
I am pleased to be here today to discuss the federal government's
processes for improving the management of IT investments that total $64
billion for fiscal year 2007. Effective management of these investments is
essential to the health, economy, and security of the nation. The Office
of Management and Budget (OMB) plays a key role in overseeing federal IT
investments. The Clinger-Cohen Act, among other things, requires OMB to
establish processes to analyze, track, and evaluate the risks and results
of major capital investments in information systems made by executive
agencies and to report to Congress on the net program performance benefits
achieved as a result of these investments.
To help carry out its role, OMB has developed several processes to improve
the management of federal IT projects, including the e-Gov scorecard,1
Management Watch List, and high risk projects. The Management Watch List
identifies projects with weaknesses in their funding justifications (or
exhibit 300s) based on an evaluation of these documents. High risk
projects are projects requiring special attention from oversight
authorities and the highest level of agency management because of one or
more of the following four reasons:2
(1) the agency has not consistently demonstrated the ability to manage
complex projects; (2) the projects has exceptionally high development,
operating, or maintenance costs; (3) the projects are addressing
deficiencies in the agencies' ability to perform an essential mission
program or function of the agency; or (4) the projects' delay or failure
would impact the agencies' essential
1
The quarterly e-Gov Scorecards are reports that use a red/yellow/green
scoring system to illustrate the results of OMB's evaluation of agencies'
implementation of e-government criteria in the President's Management
Agenda. The scores are determined in quarterly reviews, where OMB
evaluates agency progress toward agreed-upon goals along several
dimensions, and provides input to the quarterly reporting on the
President's Management Agenda. Key criteria used to score agencies
e-government process include acceptable business cases, cost and schedule
performance; and security accreditation. As of June 30, 2006, 21 of the 26
departments/major agencies were identified as having a yellow (mixed
results) or red (unsatisfactory) score.
2
These reasons are specified in OMB, Memorandum for Chief Information
Officers: Improving Information Technology (IT) Project Planning and
Execution , M-05-23 (Washington, D.C., Aug. 4, 2005)
mission functions. Agencies are also required to provide quarterly reports
to OMB on identified high risk projects that have performance shortfalls,
meaning that they do not meet one or more of four performance evaluation
criteria. The performance criteria are (1) establishing baselines with
clear cost, schedule, and performance goals;(2) maintaining the project's
cost and schedule variances within 10 percent; (3) assigning a qualified
project manager; or (4) avoiding duplication by leveraging interagency and
governmentwide investments.
These processes, among other things, are instrumental in helping to
identify and improve oversight of poorly planned and poorly performing
projects. Given the importance of these processes, you asked us to
summarize (1) the number of projects and fiscal year 2007 dollar value of
Management Watch List and high risk projects,
(2) previously reported results on how these projects are identified and
provided oversight, and (3) recommendations made to improve these
processes. In preparing this testimony, we summarized our previous reports
on initiatives for improving the management of federal IT investments.3
The work in these reports was performed in accordance with generally
accepted government auditing standards.
Results in Brief
As a result of the Management Watch List and high risk projects processes,
about 300 projects totaling about $12 billion in estimated IT expenditures
for fiscal year 2007 have been identified as being either poorly planned
or poorly performing. Of the 857 major IT projects in the President's
budget for fiscal year 2007, OMB placed 263 projects, representing about
$10 billion on its Management Watch List. In addition, in response to
OMB's memorandum, agencies reported that 79 of 226 high-risk projects,
collectively
3
GAO, Information Technology: OMB Can Make More Effective Use of Its
Investment Reviews, GAO-05-276 (Washington, D.C.: April 15, 2005 );
Information Technology: Agencies Need to Improve the Accuracy and
Reliability of Investment Information, GAO- 06-250 (Washington, D.C.:
Jan.12, 2006); GAO, Information Technology: Agencies and OMB Should
Strengthen Processes for Identifying and Overseeing High Risk Projects,
GAO-06-647 (Washington, DC, June 15, 2006)
Page 2 GAO-06-1099T
totaling about $2.2 billion, had a performance shortfall primarily
associated with cost and schedule variances that exceeded 10 percent.
While this information helps to focus both agency and OMB management
attention on these poorly planned and poorly performing projects, our
reviews identified opportunities to strengthen how these projects are
identified and provided oversight.
o The Management Watch List may be undermined by inaccurate and
unreliable data. OMB uses scoring criteria to evaluate each major
projects' justification for funding (known as exhibit 300s) to derive
the projects on its Management Watch List. Our detailed evaluation of
exhibit 300s showed that the information reported in them is not
always accurate or supported by documentation.
o For the high risk projects, the criteria for identifying projects were
not always consistently applied and we found examples of projects that
appeared to meet the criteria but were not identified as high risk.
Without consistent application of the high risk criteria, OMB and
agency executives cannot have the assurance that all projects that
require special attention have been identified.
o For both sets of projects, OMB did not develop a central list of
projects and deficiencies that could facilitate the tracking of
progress and reporting to Congress. By not having such lists, OMB is
not fully exploiting the opportunity to analyze and track these
projects on a governmentwide basis and to involve Congress in the
oversight of these projects with risks.
To improve the way the Management Watch List and high risk projects are
identified and provided oversight, we have made a number of
recommendations to the Director of OMB. These recommendations include
directing agencies to improve the accuracy and reliability of exhibit 300
information and to consistently applying the high risk criteria defined by
OMB. In addition, we recommended that the Director provide for training of
agency personnel responsible for completing exhibit 300s and to develop a
single, aggregate list for both the Management Watch List projects and for
high risk projects to facilitate tracking progress, performing
governmentwide analysis, and reporting the results to
Congress. OMB generally disagreed with our recommendations. However, we
continue to believe that they are needed to help accurately identify
poorly planned and performing projects and more effectively oversee these
projects.
Background
Each year, OMB and federal agencies work together to determine how much
the government plans to spend for IT and how these funds are to be
allocated. Federal IT spending has risen to an estimated $64 billion in
fiscal year 2007.
OMB plays a key role in overseeing federal IT investments and how they are
managed. To drive improvement in the implementation and management of IT
projects, Congress enacted the Clinger-Cohen Act in 1996 to further expand
the responsibilities of OMB and the agencies under the Paperwork Reduction
Act.4 In particular, the act requires agency heads, acting through agency
chief information officers (CIOs), to, among other things, better link
their IT planning and investment decisions to program missions and goals
and to implement and enforce IT management policies, procedures,
standards, and guidelines. The Clinger-Cohen Act requires that agencies
engage in capital planning and performance and resultsbased management.5
The act also requires OMB to establish processes to analyze, track, and
evaluate the risks and results of major capital investments in information
systems made by executive agencies. OMB is also required to report to
Congress on the net program performance benefits achieved as a result of
major capital investments in information systems that are made by
executive agencies.6
4
44 U.S.C. S: 3504(a)(1)(B)(vi)(OMB); 44 U.S.C. S: 3506(h)(5) (agencies).
5
40 U.S.C. S: 11312; 40 U.S.C. S: 11313.
6
These requirements are specifically described in the Clinger-Cohen Act, 40
U.S.C. S: 11302 (c).
Page 4 GAO-06-1099T
In response to the Clinger-Cohen Act and other statutes, OMB developed
policy for planning, budgeting, acquisition, and management of federal
capital assets. This policy is set forth in OMB Circular A-11 (section
300) and in OMB's Capital Programming Guide (supplement to Part 7 of
Circular A-11), which directs agencies to develop, implement, and use a
capital programming process to build their capital asset portfolios. Among
other things, OMB's Capital Programming Guide directs agencies to
o evaluate and select capital asset investments that will support core
mission functions that must be performed by the federal government and
demonstrate projected returns on investment that are clearly equal to
or better than alternative uses of available public resources;
o institute performance measures and management processes that monitor
actual performance and compare to planned results; and
o establish oversight mechanisms that require periodic review of
operational capital assets to determine how mission requirements might
have changed and whether the asset continues to fulfill mission
requirements and deliver intended benefits to the agency and
customers.
To further support the implementation of IT capital planning practices, we
have developed an IT investment management framework7 that agencies can
use in developing a stable and effective capital planning process, as
required by statute and directed in OMB's Capital Programming Guide.
Consistent with the statutory focus on selecting,8 controlling,9 and
evaluating10
7 GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.:
March 2004).
8
During the selection phase, the organization (1) identifies and analyzes
each project's risks and returns before committing significant funds to
any project and (2) selects those IT projects that will best support its
mission needs.
9
During the control phase, the organization ensures that, as projects
develop and investment expenditures continue, the project is continuing to
meet mission needs at the expected levels of cost and risk. If the project
is not meeting expectations or if problems have arisen, steps are quickly
taken to address the deficiencies.
investments, this framework focuses on these processes in relation to IT
investments specifically. It is a tool that can be used to determine both
the status of an agency's current IT investment management capabilities
and the additional steps that are needed to establish more effective
processes. Mature and effective management of IT investments can vastly
improve government performance and accountability. Without good
management, such investments can result in wasteful spending and lost
opportunities for improving delivery of services to the public.
Prior Reviews on Federal IT Investment Management Have Identified
Weaknesses
Only by effectively and efficiently managing their IT resources through a
robust investment management process can agencies gain opportunities to
make better allocation decisions among many investment alternatives and
further leverage their investments. However, the federal government faces
enduring IT challenges in this area. For example, in January 2004 we
reported on mixed results of federal agencies' use of IT investment
management practices.11 Specifically, we reported that although most of
the agencies had IT investment boards responsible for defining and
implementing the agencies' IT investment management processes, agencies
did not always have important mechanisms in place for these boards to
effectively control investments, including decisionmaking rules for
project oversight, early warning mechanisms, and/or requirements that
corrective actions for underperforming projects be agreed upon and
tracked. Executive-level oversight of project-level management activities
provides organizations with increased assurance that each investment will
achieve the desired cost, benefit, and schedule results. Accordingly, we
made several recommendations to agencies to improve their practices.
10
During the evaluation phase, actual versus expected results are compared
once projects have been fully implemented. This is done to (1) assess the
project's impact on mission performance, (2) identify any changes or
modifications to the project that may be needed, and (3) revise the
investment management process based on lessons learned.
11
GAO, Information Technology Management: Governmentwide Strategic Planning,
Performance Measurement, and Investment Management Can Be Further
Improved, GAO-04-49 (Washington, D.C.: Jan. 12, 2004).
Page 6 GAO-06-1099T
In previous work using our investment management framework, we reported
that the use of IT investment management practices by agencies was mixed.
For example, a few agencies that have followed the framework in
implementing capital planning processes have made significant
improvements.12 In contrast, however, we and others have continued to
identify weaknesses at agencies in many areas, including immature
management processes to support both the selection and oversight of major
IT investments and the measurement of actual versus expected performance
in meeting established performance measures.13
OMB's Management Watch List Intended to Correct Project Weaknesses and
Business Case Deficiencies
In helping to ensure that investments of public resources are justified
and that public resources are wisely invested, OMB began using the
Management Watch List, in the President's fiscal year 2004 budget request,
as a means to oversee the justification for and planning of agencies' IT
investments. This list was derived based on a detailed review of the
investments' Capital Asset Plan and Business Case, also known as the
exhibit 300.
The exhibit 300 is a reporting mechanism intended to enable an agency to
demonstrate to its own management, as well as OMB, that a major project is
well planned in that it has employed the disciplines of good project
management; developed a strong business case for the investment; and met
other Administration priorities in defining the cost, schedule, and
performance goals proposed for the investment.
12
These agencies include the Departments of Agriculture, Commerce, and the
Interior.
13
For example, GAO, Information Technology: Centers for Medicare & Medicaid
Services Needs to Establish Critical Investment Management Capabilities,
GAO-06-12 (Washington, D.C.: Oct. 28, 2005); Information Technology:
Departmental Leadership Crucial to Success of Investment Reforms at
Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); and United
States Postal Service: Opportunities to Strengthen IT Investment
Management Capabilities , GAO-03-3 (Washington, D.C.: Oct. 15, 2002).
We reported in 2005 that OMB analysts evaluate agency exhibit 300s by
assigning scores to each exhibit 300 based on guidance presented in OMB
Circular A-11.14 As described in this circular, the scoring of a business
case consists of individual scoring for 10 categories, as well as a total
composite score of all the categories. The 10 categories are
o acquisition strategy,
o project (investment) management,
o enterprise architecture,
o alternatives analysis,
o risk management,
o performance goals,
o security and privacy,
o performance-based management system (including the earned value
management system15),
o life-cycle costs formulation, and
o support of the President's Management Agenda.
Using these scores, projects were placed on the Management Watch List if
their exhibit 300 business case received a total composite score of 3 or
less, or if it received a score of 3 or less in the areas of performance
goals, performance-based management systems, or security and privacy, even
if its overall score was a 4 or 5. To derive the total number of projects
on the list that were reported for fiscal year 2005, OMB polled the
individual analysts and compiled the numbers.
According to OMB, agencies with weaknesses in these three areas were to
submit remediation plans addressing the weaknesses. OMB officials also
stated that decisions on follow-up and monitoring the
14 GAO-05-276.
Earned value management is a project management tool that integrates the
investment scope of work with schedule and cost elements for investment
planning and control. This method compares the value of work accomplished
during a given period with that of the work expected in the period.
Differences in expectations are measured in both cost and schedule
variances.
Page 8 GAO-06-1099T
progress were typically made by staff with responsibility for reviewing
individual agency budget submissions, depending on the staff's insights
into agency operations and objectives. According to OMB officials, those
Management Watch List projects that did receive specific follow-up
attention received feedback through the passback process, through targeted
evaluation of remediation plans designed to address weaknesses, and
through the apportioning of funds so that the use of budgeted dollars was
conditional on appropriate remediation plans being in place, and through
the quarterly e-Gov Scorecards.
OMB Issued August 2005 Memorandum on Improving Performance of High Risk IT
Projects
To improve IT project execution, OMB issued a memorandum in August 2005 to
all federal CIOs, directing them to begin taking steps to identify IT
projects that are high risk and to report quarterly on their
performance.16 As originally defined in OMB Circular A-11 and subsequently
reiterated in the August 2005 memorandum, high risk projects are those
that require special attention from oversight authorities and the highest
levels of agency management because of one or more of the following four
reasons:
o The agency has not consistently demonstrated the ability to manage
complex projects.
o The project has exceptionally high development, operating, or
maintenance costs, either in absolute terms or as a percentage of the
agency's total IT portfolio.
o The project is being undertaken to correct recognized deficiencies in
the adequate performance of an essential mission program or function
of the agency, a component of the agency, or another organization.
o Delay or failure of the project would introduce for the first time
unacceptable or inadequate performance or failure of an essential
mission function of the agency, a component of the agency, or another
organization.
16
OMB Memorandum, M-05-23 (Aug. 4, 2005).
As directed in the memorandum, agencies are to work with OMB to identify
their high risk IT projects using these criteria. Most agencies reported
that, to identify high risk projects, CIO office staff compared the
criteria against their current portfolio to determine which projects met
OMB's definition. They then submitted the list to OMB for review.
According to OMB and agency officials, after the submission of the initial
list, examiners at OMB worked with individual agencies to identify or
remove projects as appropriate. According to most agencies, the final list
was then approved by their CIO.
For the identified high risk projects, beginning September 15, 2005, and
quarterly thereafter, CIOs were to assess, confirm, and document projects'
performance. Specifically, agencies were required to determine, for each
of their high risk projects, whether the project was meeting one or more
of four performance evaluation criteria: (1) establishing baselines with
clear cost, schedule, and performance goals; (2) maintaining the project's
cost and schedule variances within 10 percent; (3) assigning a qualified
project manager; and (4) avoiding duplication by leveraging inter-agency
and governmentwide investments. If a high risk project met any of these
four performance evaluation criteria, agencies were instructed to document
this using a standard template provided by OMB and provide this template
to oversight authorities (e.g., OMB, agency inspectors general, agency
management, and GAO) on request. Upon submission, according to OMB staff,
individual analysts review the quarterly performance reports of projects
with shortfalls to determine how well the projects are progressing and
whether the actions described in the planned improvement efforts are
adequate using other performance data already received on IT projects such
as the e-Gov Scorecards, earned value management data, and the exhibit
300.
Poorly Planned and Performing Projects Identified, Totaling About $12 Billion
in Estimated Expenditures for Fiscal Year 2007
About 300 projects totaling about $12 billion in estimated IT expenditures
for fiscal year 2007 have been placed on OMB's Management Watch List or as
a high risk project with performance shortfalls. Specifically, the
President's budget for fiscal year 2007 included 857 major IT projects,
totaling approximately $64 billion. Of this, OMB reported that there were
263 proposed major projects that were poorly planned, totaling $10
billion. In addition, agencies reported that 79 of the 226 high-risk
projects identified as of March 2006, collectively totaling about $2.2
billion had a performance shortfall primarily associated with cost and
schedule variances that exceeded 10 percent.
OMB first reported on the Management Watch List in the President's budget
request for 2004. While the number of projects and their associated budget
have decreased since then, they still represent a significant percentage
of the total IT budget. Table 1 shows the budget information for projects
on the Management Watch List for fiscal years 2004, 2005, 2006, and 2007.
Table 1: Management Watch List Budget for Fiscal Years 2004, 2005, 2006, and
2007
IT budget for Percentage of Total IT Management budget for budget (in
Watch List
billions) Fiscal Years
Management projects (in Watch List billions) projects
2004 $59.0 $20.9 35%
2005 $60.0 $22.0 37%
2006 $65.0 $15.0 23%
2007 $64.0 $9.9 15%
Source: GAO analysis of OMB data.
Table 2 provides the number of projects on the Management Watch List for
fiscal years 2004, 2005, 2006, and 2007.
Table 2: Number of Projects on Management Watch List for Fiscal Years
2004, 2005, 2006, and 2007
Total IT Fiscal Years projects
Percentage of Management projects on Watch List Management projects Watch
List
1400 771 55%
1200 621 52%
1087 342 31%
857 263 31%
Source: GAO analysis of OMB data.
In addition, in response to OMB's August 2005 memorandum, the 24 major
agencies identified 226 IT projects as high risk, totaling about $6.4
billion in funding requested for fiscal year 200717. Agencies identified
most projects as high risk because their delay or failure would impact the
essential business functions of the agency. In addition, agencies reported
that about 35 percent of the high risk projects-or 79 investments,
collectively totaling about $2.2 billion in fiscal year 2007-had a
performance shortfall, primarily associated with cost and schedule
variances that exceeded 10 percent. 18
Figure 1 illustrates the number of agency high risk projects as of March
2006 with and without shortfalls. The majority of the agencies reported
that their high risk projects did not have performance shortfalls in any
of the four areas identified by OMB. In addition, six agencies-the
departments of Commerce, Energy, Housing and Urban Development, and Labor,
and the National Aeronautics and Space Administration and the National
Science Foundation- reported that none of their high risk projects
experienced any performance shortfalls.
17 GAO-06-647 .
Of the 79 projects with performance shortfalls, nineteen projects totaling
about $500 million in estimated IT expenditures for 2007 were also placed
on OMB's Management Watch List.
Page 12 GAO-06-1099T
Figure 1: Number of Agencies High Risk Projects with and without Performance
Shortfalls (as of March 2006)
Note: Department of Agriculture (USDA); Department of Health and Human
Services (HHS); Department of Homeland Security (DHS); Department of
Housing and Urban Development (HUD); Department of Veterans Affairs (VA);
Environmental Protection Agency (EPA); General Services Administration
(GSA); National Aeronautics and Space Administration (NASA); National
Science Foundation (NSF); Nuclear Regulatory Commission (NRC); Office of
Personnel Management (OPM); Small Business Administration (SBA); Social
Security Administration (SSA); Agency for International Development
(USAID)
Improvements Needed to Identify and Oversee Management Watch List and High
Risk Projects
While the Management Watch List and high risk processes serve to highlight
poorly planned and performing projects and focus attention on them, our
reviews identified opportunities to strengthen the identification and
oversight of projects for each.
Management Watch List May Be Based on Unreliable Data and High Risk
Project Criteria Are Not Always Consistently Applied
OMB's Management Watch List may be undermined by inaccurate and unreliable
data. While OMB uses the exhibit 300s as the basis for designating
projects as poorly planned, we have recently reported19 that the
underlying support was often inadequate for information provided in the
exhibit 300s GAO reviewed. Three general types of weaknesses were evident:
o All exhibit 300s had documentation weaknesses. Documentation either
did not exist or did not fully agree with specific areas of the
exhibit 300.
o Agencies did not always demonstrate that they complied with federal or
departmental requirements or policies with regard to management and
reporting processes. Also, none had cost analyses that fully complied
with OMB requirements for cost-benefit and cost-effectiveness
analyses. In contrast, most investments did demonstrate compliance
with information security planning and training requirements.
o In sections that required actual cost data, these data were unreliable
because they were not derived from cost-accounting systems with
19 GAO-06-250 .
adequate controls. In the absence of such systems, agencies generally
derived cost information from ad hoc processes.
Moreover, although agencies, with OMB's assistance, generally identified
their high risk projects using criteria specified by OMB, these criteria
were not always consistently applied.
o In several cases, agencies did not use OMB's criteria to identify high
risk projects. Some agencies reported using other reasons to identify
a total of 31 high risk projects. For example, the Department of
Homeland Security reported investments that were high risk because
they had weaknesses associated with their business cases based on the
evaluation by OMB. The Department of Transportation reported projects
as high risk because two did not have approved baselines, and four had
incomplete or poor earned value management assessments.
o Regarding the criterion for high risk designation that the agency has
not consistently demonstrated the ability to manage complex projects,
only three agencies reported having projects meeting this criterion.
This appears to be somewhat low, considering that we and others have
previously reported on weaknesses in numerous agencies' ability to
manage complex projects. For example, we have reported in our high
risk series on major programs and operations that need urgent
attention and transformation in order to ensure that our national
government functions in the most economical, efficient, and effective
manner possible.20 Specifically, the Department of Defense's efforts
to modernize its business systems have been hampered because of
weaknesses in practices for (1) developing and using an enterprise
architecture, (2) instituting effective investment management
processes, and (3) establishing and implementing effective systems
acquisition processes. We concluded that the Department of Defense, as
a whole, remains far from where it needs to be to effectively and
efficiently manage an undertaking with the size, complexity, and
significance of its
GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C., January
2005).
departmentwide business systems modernization. We also reported that,
after almost 25 years and $41 billion, efforts to modernize the air
traffic control program of the Federal Aviation Administration, the
Department of Transportation's largest component, are far from complete
and that projects continue to face challenges in meeting cost, schedule,
and performance expectations.21 However, neither the Department of Defense
nor the Department of Transportation cited the "inability to consistently
manage complex projects" criteria for any projects as being high risk.
0M Finally, while agencies have reported a significant number of IT
projects as high risk, we identified other projects on which we have
reported and testified that appear to meet one or more of OMB's criteria
for high risk designation including high development or operating costs
and recognized deficiencies in the adequate performance but were not
identified as high risk. Examples we have recently reported include the
following projects:
0M The Decennial Response Integration System of the Census Bureau, is
intended to integrate paper, Internet, and telephone responses. Its high
development and operating costs are expected to make up a large portion of
the $1.8 billion program to develop, test, and implement decennial census
systems. In March 2006,22 we testified that the component agency has
established baseline requirements for the acquisition, but the bureau has
not yet validated them or implemented a process for managing the
requirements. We concluded that, until these and other basic contract
management activities are fully implemented, this project faced increased
risks that the system would experience cost overruns, schedule delays, and
performance shortfalls.
21 GAO-05-207.
GAO, Census Bureau: Important Activities for Improving Management of Key
2010 Decennial Acquisitions Remain to be Done, GAO-06-444T (Washington,
D.C.: Mar. 1, 2006).
Page 16 GAO-06-1099T
o The National Polar-Orbiting Operational Environmental Satellite
System-an initiative managed by the Department of Commerce, the
Department of Defense, and the National Aeronautics and Space
Administration-is to converge two satellite programs into a single
satellite program capable of satisfying both civilian and military
requirements. In November 2005,23 we reported that the system was a
troubled program because of technical problems on critical sensors,
escalating costs, poor management at multiple levels, and the lack of
a decision on how to proceed with the program. Over the last several
years, this system has experienced continual cost increases to about
$10 billion and schedule delays, requiring difficult decisions about
the program's direction and capabilities. More recently, we
testified24 that the program is still in trouble and that its future
direction is not yet known. While the program office has corrective
actions under way, we concluded that, as the project continues, it
will be critical to ensure that the management issues of the past are
not repeated.
o Rescue 21, is a planned coastal communications system of the
Department of Homeland Security. We recently reported25 that
inadequacies in several areas contributed to Rescue 21 cost overruns
and schedule delays. These inadequacies occurred in requirements
management, project monitoring, risk management, contractor cost and
schedule estimation and delivery, and executive level oversight.
Accordingly, the estimated total acquisition cost has increased from
$250 million in 1999 to $710.5 million in 2005, and the timeline for
achieving full operating capability has been extended from 2006 to
2011.
23
GAO, Polar-Orbiting Operational Environmental Satellites: Technical
Problems, Cost Increases, and Schedule Delays Trigger Need for Difficult
Trade-Off Decisions, GAO-06- 249T (Washington, D.C.: Nov. 16, 2005).
24 GAO, Polar-Orbiting Operational Environmental Satellites: Cost
Increases Trigger Review and Place Program's Direction on Hold,
GAO-06-573T (Washington, D.C.: Mar. 30, 2006).
25
GAO, United States Coast Guard: Improvements Needed in Management and
Oversight of Rescue System Acquisition , GAO-06-632 (Washington, D.C.: May
31, 2006).
For the projects we identified as appearing to meet OMB's criteria for
high risk, the responsible agencies reported that they did not consider
these investments to be high risk projects for such reasons as (1) the
project was not a major investment; (2) agency management is experienced
in overseeing projects; or (3) the project did not have weaknesses in its
business case. In particular, one agency stated that their list does not
include all high risk projects, only those that are the highest priority
of the high risk investments. However, none of the reasons provided are
associated with OMB's high risk definition. Without consistent application
of the criteria, OMB and executives cannot have the assurance that all
projects that require special attention have been identified.
OMB Does Not Use an Aggregate List to Perform Its Oversight of the
Management Watch List or High Risk Projects
While OMB's Management Watch List identified opportunities to strengthen
investments and promote improvements in IT management, OMB did not develop
a single, aggregate list identifying the projects and their weaknesses.
According to OMB officials, they did not construct a single list of
projects meeting their watch list criteria because they did not see such
an activity as necessary in performing OMB's predominant mission: to
assist in overseeing the preparation of the federal budget and to
supervise agency budget administration. Thus, OMB did not exploit the
opportunity to use the list as a tool for analyzing IT investments on a
govermentwide basis, limiting its ability to identify and report on the
full set of IT investments requiring corrective actions.
In addition, while OMB asked agencies to take corrective actions to
address weaknesses associated with projects on the Management Watch List,
it did not develop a structured, consistent process or criteria for
deciding how to follow up on these actions. We also reported that because
it did not consistently monitor the follow-up performed, OMB could not
tell us which of the 621 projects identified on the fiscal year 2005 list
received follow-up attention, and it did not know whether the specific
project risks that it identified through its Management Watch List were
being managed effectively. This approach could leave resources at risk of
being committed to poorly planned and managed projects. Thus, OMB was not
using its Management Watch List as a tool for improving IT investments on
a governmentwide basis and focusing attention where it was most needed.
Similar to the Management Watch List, we reported in June 2006 that while
OMB analysts review the quarterly performance reports on high risk
projects, they did not compile a single aggregate list of high risk
projects. According to OMB staff they did not see such an activity as
necessary in achieving the intent of the guidance-to improve project
planning and execution. Consistent with our Management Watch list
observations and recommendations, we believe that by not having a single
list, OMB is limiting its ability to identity and report on the full set
of IT investments across the federal government that require special
oversight and greater agency management attention.
Implementation of Recommendations Can Lead to Improved Processes to Identify
and Oversee Management Watch List and High Risk Projects
To address our key findings, we made several recommendations to the
Director of OMB. For example, to improve how the Management Watch List
projects are identified, we have made several recommendations to improve
the accuracy and validity of exhibit 300s for major IT investments,
including that the Director require agencies to determine the extent to
which the information contained in each exhibit 300 is accurate and
reliable, and, where weaknesses in accuracy and reliability are
identified, disclose them and explain the agency's approach to mitigating
them. We also recommended that the Director provide for training of agency
personnel responsible for completing exhibit 300s, and specified that, in
developing the training, OMB consult with agencies to identify
deficiencies that the training should address. Likewise, to improve how
high risk projects are identified, we recommended that the Director direct
federal agency CIOs to ensure that they are consistently applying the high
risk criteria defined by OMB.
To improve how the Management Watch List is provided oversight, in our
April 2005 report, we recommended that the Director of OMB develop a
central list of projects and their deficiencies and report to Congress on
progress made in addressing risks of major IT investments and management
areas needing attention. In addition, to fully realize the potential
benefits of using the Management Watch List, we recommended that OMB use
the list as the basis for selecting projects for follow-up, tracking
follow-up activities and analyze the prioritized list to develop
governmentwide and agency assessments of the progress and risks of IT
investments, identifying opportunities for continued improvement. We also
made similar recommendations to the Director of OMB regarding high risk
projects. Specifically, we recommended that OMB develop a single aggregate
list of high risk projects and their deficiencies and use that list to
report to Congress progress made in correcting high risk problems, actions
under way, and further actions that may be needed.
OMB generally disagreed with our recommendations for strengthening the
Management Watch List and high risk projects processes. Specifically,
OMB's Administrator of the Office of E-Government and Information
Technology stated that the ultimate responsibility to improve the accuracy
and reliability of the exhibit 300s lies with the agencies. While this is
true, OMB also has statutory responsibility for providing IT guidance
governmentwide, especially when it involves an OMB-required budget
document. Regarding the consistent application of the high risk criteria,
the Administrator stated that some flexibility in the application of the
criteria is essential. While some flexibility may be appropriate, we
believe that these criteria should be more consistently applied so that
projects that clearly meet them are identified and provided oversight. The
Administrator also disagreed with our recommendations that an aggregated
governmentwide Management Watch List and high risk project list is
necessary to perform adequate oversight. However, we continue to believe
that these lists are needed to facilitate OMB's ability to track progress.
Addressing these recommendations would provide increased assurance that
poorly planned and performing projects are accurately identified and more
effectively provided oversight.
In summary, the Management Watch List and High Risk processes play
important roles in improving the management of federal IT investments by
helping to identify poorly planned and performing projects totaling at
least $12 billion that require management attention. However, the number
of projects identified on both lists is likely understated because the
Management Watch List is derived from budgetary documents that are not
always accurate and reliable and the high risk projects are not always
identified consistently using OMB criteria. In addition, we noted areas
where oversight of both sets of projects could be strengthened primarily
by reporting the results in the aggregate so that governmentwide analyses
can be performed, progress can be tracked, and Congress can be informed.
The recommendations we made to agencies and OMB to address these issues
are aimed at providing greater assurance that poorly planned and
performing projects are more accurately identified and receiving adequate
oversight, and ultimately ensuring that potentially billions of taxpayers
dollars are not wasted.
Contacts and Acknowledgements
If you should have any questions about this testimony, please contact me
at (202) 512-9286 or by e-mail at [email protected]. Other individuals who
made key contributions to this testimony are Sabine Paul and Niti Tandon.
(310828)
*** End of document. ***