Information Technology Management: Governmentwide Strategic
Planning, Performance Measurement, and Investment Management Can
Be Further Improved (12-JAN-04, GAO-04-49).
Over the years, the Congress has promulgated laws and the Office
of Management and Budget and GAO have issued policies and
guidance, respectively, on (1) information technology (IT)
strategic planning/performance measurement (which defines what an
organization seeks to accomplish, identifies the strategies it
will use to achieve desired results, and then determines how well
it is succeeding in reaching resultsoriented goals and achieving
objectives) and (2) investment management (which involves
selecting, controlling, and evaluating investments). To obtain an
understanding of the government's implementation of these key IT
management policies, congressional requesters asked GAO to
determine the extent to which 26 major agencies have in place
practices associated with key legislative and other requirements
for (1) IT strategic planning/ performance measurement and (2) IT
investment management.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-49
ACCNO: A09100
TITLE: Information Technology Management: Governmentwide
Strategic Planning, Performance Measurement, and Investment
Management Can Be Further Improved
DATE: 01/12/2004
SUBJECT: Agency evaluation
Agency missions
Federal agencies
General management reviews
Information technology
Performance measures
Strategic planning
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-49
United States General Accounting Office
GAO
Report to Congressional Requesters
January 2004
INFORMATION
TECHNOLOGY
MANAGEMENT
Governmentwide Strategic Planning, Performance Measurement, and Investment
Management Can Be Further Improved
a
GAO-04-49
Highlights of GAO-04-49, a report to congressional requesters
Over the years, the Congress has promulgated laws and the Office of
Management and Budget and GAO have issued policies and guidance,
respectively, on (1) information technology (IT) strategic
planning/performance measurement (which defines what an organization seeks
to accomplish, identifies the strategies it will use to achieve desired
results, and then determines how well it is succeeding in reaching
resultsoriented goals and achieving objectives) and (2) investment
management (which involves selecting, controlling, and evaluating
investments).
To obtain an understanding of the government's implementation of these key
IT management policies, congressional requesters asked GAO to determine
the extent to which 26 major agencies have in place practices associated
with key legislative and other requirements for (1) IT strategic planning/
performance measurement and (2) IT investment management.
GAO is making a number of recommendations, including that each agency take
action to address IT strategic planning, performance measurement, and
investment management practices that are not fully in place. In commenting
on a draft of the report, most agencies generally agreed with our findings
and recommendations.
www.gao.gov/cgi-bin/getrpt?GAO-04-49.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Powner at (202)
512-9286 or [email protected].
January 2004
INFORMATION TECHNOLOGY MANAGEMENT
Governmentwide Strategic Planning, Performance Measurement, and Investment
Management Can Be Further Improved
Agencies' use of 12 IT strategic planning/performance measurement
practices-identified based on legislation, policy, and guidance-is uneven
(see figure, below left). For example, agencies generally have IT
strategic plans and goals, but these goals are not always linked to
specific performance measures that are tracked. Without enterprisewide
performance measures that are tracked against actual results, agencies
lack critical information about whether their overall IT activities are
achieving expected goals.
Agencies' use of 18 IT investment management practices that GAO identified
is also mixed (see figure, below right). For example, the agencies largely
have IT investment management boards, but no agency had the practices
associated with the control phase fully in place. Executive-level
oversight of project-level management activities provides organizations
with increased assurance that each investment will achieve the desired
cost, benefit, and schedule results.
Agencies cited a variety of reasons for not having practices fully in
place, such as that the chief information officer position had been
vacant, that not including a requirement in guidance was an oversight, and
that the process was being revised, although they could not always provide
an explanation. Regardless of the reason, these practices are important
ingredients for ensuring effective strategic planning, performance
measurement, and investment management, which, in turn, make it more
likely that the billions of dollars in government IT investments are
wisely spent.
Percentage of Agencies' Use of IT Strategic Planning/Performance
Measurement Practices (left) and Investment Management Practices (right)a
aPercentages do not add up to 100 percent due to rounding.
Note: Yes-the practice was in place. Partially-the agency has some, but
not all, aspects of the practice in place. Examples of circumstances in
which the agency would receive this designation include when (1) some, but
not all, of the elements of the practice were in place; (2) the agency
documented that it has the information or process in place but it was not
in the prescribed form (e.g., in a specific document as required by law or
the Office of Management and Budget); (3) the agency's documentation was
in draft form; or (4) the agency had a policy related to the practice but
evidence supported that it had not been completely or consistently
implemented. No- the practice was not in place. Not applicable-the
practice was not relevant to the agency's particular
circumstances.
Contents
Letter
Results in Brief
Background
Agencies' Use of IT Strategic Planning/Performance Measurement
Practices Is Uneven Agencies' Use of IT Investment Management Practices Is
Mixed Conclusions Recommendations Agency Comments and Our Evaluation
1 3 4
15 28 43 44 45
Appendixes
Appendix I:
Appendix II:
Appendix III:
Appendix IV:
Appendix V:
Appendix VI:
Appendix VII:
Appendix VIII:
Appendix IX:
Appendix X: Appendix XI:
Appendix XII:
Appendix XIII: Recommendations to Departments and Agencies 51
Comments from the Department of Agriculture 77
Comments from the Department of Commerce 78
Comments from the Department of Defense (including EURcomments from the
Departments of the Air Force, Army, and EURNavy) 79
GAO Comments 87
Comments from the Department of Education 88
GAO Comments 89
Comments from the Environmental Protection Agency 90
GAO Comments 92
Comments from the General Services Administration 93
GAO Comments 101
Comments from the Department of Health and Human EURServices 104
Comments from the Department of Housing and Urban EURDevelopment 106
Comments from the Department of the Interior 107
Comments from the Department of Justice 109
GAO Comments 111
Comments from the Department of Labor 112
GAO Comments 114
Comments from the National Aeronautics and Space EURAdministration 115
Contents
GAO Comments 119
Appendix XIV: Comments from the Nuclear Regulatory 120
Commission
Comments from the Social Security 121
Appendix XV: Administration
GAO Comments 126
Appendix XVI: Comments from the Department of State 127
GAO Comments 130
Comments from the U.S. Agency for
Appendix XVII: International
Development 131
GAO Comments 134
Comments from the Department of Veterans 135
Appendix XVIII: Affairs
GAO Comments 140
Appendix XIX: GAO Contacts and Staff Acknowledgments 141
GAO Contacts 141
Staff Acknowledgments 141
Table 1: IT Strategic Planning/Performance
Tables Measurement
Practices 23
Table 2: IT Investment Management Practices 35
Figure 1: Percentage of Agencies' Use of IT
Figures Strategic Planning/ Performance Measurement
Practices 16
Figure 2: Percentage of Agencies' Use of IT
Investment
Management Practices 29
Contents
AbbreviationsEUR
CFO chief financial officer
CIO chief information officer
COTS commercial-off-the-shelf
DHS Department of Homeland Security
DOD Department of Defense
EPA Environmental Protection Agency
FISMA Federal Information Security Management Act
GISRA Government Information Security Reform Act
GPRA Government Performance and Results Act
GSA General Services Administration
HHS Department of Health and Human Services
HUD Department of Housing and Urban Development
IRM information resources management
IT information technology
IV&V independent verification and validation
NARA National Archives and Records Administration
NASA National Aeronautics and Space Administration
NRC Nuclear Regulatory Commission
NSF National Science Foundation
OMB Office of Management and Budget
OPM Office of Personnel Management
SBA Small Business Administration
SSA Social Security Administration
USAID U.S. Agency for International Development
VA Department of Veterans Affairs
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States General Accounting Office Washington, D.C. 20548
January 12, 2004
The Honorable Susan M. Collins
Chairman
Committee on Governmental Affairs
United States Senate
The Honorable Tom Davis
Chairman
Committee on Government Reform
House of Representatives
The Honorable Adam H. Putnam
Chairman
Subcommittee on Technology, Information Policy,
Intergovernmental Relations and the Census Committee on Government Reform
House of Representatives
According to the President's most recent budget, the federal government
spends billions of dollars annually on information technology (IT)-
reportedly investing about $50 billion in fiscal year 2002 and expecting
to invest about $60 billion in fiscal year 2004.1 Despite this substantial
investment, the government's management of information resources has
produced mixed results. Although agencies have taken constructive steps to
implement modern strategies, systems, and management policies and
practices, our most recent high-risk and performance and accountability
series identified continuing high-risk system modernization efforts and
governmentwide information and technology management challenges.2
For years, the Congress has been working to increase the effectiveness of
information and technology management in the federal government by passing
legislation and providing oversight. For example, the Paperwork Reduction
Act of 1995 applied life-cycle management principles to
1Office of Management and Budget, Budget of the U.S. Government, Fiscal
Year 2004, Report on IT Spending for the Federal Government for Fiscal
Years 2002, 2003, and 2004. We did not verify these data.
2U.S. General Accounting Office, High-Risk Series: An Update, GAO-03-119
(Washington, D.C.: January 2003) and Major Management Challenges and
Program Risks: A Governmentwide Perspective, GAO-03-95 (Washington, D.C.:
January 2003).
information and technology management and required that agencies indicate
in strategic information resources management (IRM) plans how they are
applying information resources to improve the productivity, efficiency,
and effectiveness of government programs.3 The Clinger-Cohen Act of 1996
amended the Paperwork Reduction Act, establishing agency chief information
officers (CIO) who report directly to the agency head and are responsible
for information resources management activities. Among other things, the
Clinger-Cohen Act also (1) required senior executive involvement in IT
decision making and (2) imposed much-needed discipline in acquiring and
managing technology resources.
To obtain a broad view of the government's implementation of key IT
management, you requested that we determine the extent to which agencies
have in place practices associated with key legislative and other
requirements for (1) IT strategic planning/performance measurement and (2)
IT investment management. To address these objectives, we identified and
reviewed major legislative requirements and executive orders pertaining to
IT strategic planning/performance measurement, which defines what an
organization seeks to accomplish, identifies the strategies it will use to
achieve desired results, and then determines-through measurement-how well
it is succeeding in reaching results-oriented goals and achieving
objectives; and IT investment management, which involves selecting,
controlling, and evaluating investments. Specifically, we identified 30
important IT management practices in these areas using legislative
requirements, such as the Paperwork Reduction Act and the Clinger-Cohen
Act, and policy and guidance issued by the Office of Management and Budget
(OMB)4 and GAO.5 We selected 26 organizations
3The Paperwork Reduction Act of 1995 revised the information resources
management responsibilities established under the Paperwork Reduction Act
of 1980, as amended in 1986.
4Office of Management and Budget, Circular A-11, Preparation, Submission,
and Execution of the Budget (July 27, 2002) and Circular A-130, Management
of Federal Information Resources (Nov. 28, 2000).
5U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
GAO/AIMD-10.1.23, Exposure Draft (Washington, D.C.: May 2000).
for our review (23 major departments and agencies identified in 31 U.S.C.
9016 and the 3 military services).
Results in Brief Agencies' use of IT strategic planning/performance
measurement practices is uneven-46 percent of the practices are in place,
41 percent are partially in place, and 7 percent are not in place.7 The
lack of full implementation of these practices is of concern because
effective strategic planning is important to ensure that agencies' IT
goals are aligned with the strategic goals of the agency. Also important
is having measures in place to monitor whether, or the extent to which, IT
is supporting the agency. The agencies generally have IRM plans or IT
strategic plans, but these plans do not always address important IRM
elements, such as information collection, records management, or privacy.
In addition, although agencies generally have goals associated with IT,
these goals are not always linked to specific performance measures.
Moreover, many agencies do not monitor actualversus-expected performance
against enterprisewide IT performance measures in their IRM plans.
Agencies cited a variety of reasons why the strategic planning/performance
measurement practices are not in place, including that there was a lack of
support from agency leadership, that the agency had not been developing
IRM plans until recently and recognized that the plans needed further
refinement, or that the process is being revised. In addition, the
agencies in our review could not always identify why the practices were
not fully in place. Regardless of the reason, these practices were
generally derived from legislative requirements and governmentwide
policies and are fundamental ingredients to effective IT planning and
performance measurement; therefore, it is important that they be
implemented.
Agencies' use of IT investment management practices is also mixed in that
44 percent of the practices are in place, 37 percent are partially in
place, and 17 percent are not in place.8 Only by effectively and
efficiently
6This section of the United States Code requires 24 departments and
agencies to establish chief financial officers. We did not include the
Federal Emergency Management Agency in our review, even though it is one
of the 24 departments and agencies, because this agency has been
transferred to the Department of Homeland Security.
7Six percent were not applicable.
8One percent were not applicable. Percentages do not add up to 100 percent
due to rounding.
managing their IT resources through a robust investment management process
can agencies gain opportunities to make better allocation decisions among
many investment alternatives and further leverage their investments. As
part of their investment management process, the agencies largely have IT
investment management boards in place that are responsible for making
decisions on selecting investments. However, many of these boards do not
have written policies and procedures covering oversight or control of
projects that cover such critical areas as corrective action plans and the
tracking of such actions to resolution. Having these policies and
procedures is a critical element of the control phase of a comprehensive
IT investment management process, which helps ensure that investments are
on track and are continuing to meet mission needs. As in the strategic
planning/performance measurement area, agencies were not always able to
explain why certain IT investment management practices were not in place.
However, among the reasons cited were that the CIO position had been
vacant, that not including a given requirement in an investment management
guide was an oversight, and that the investment management process was
being revised. Nevertheless, the full implementation of the investment
management practices would bring more rigor and structure to how agencies
select and manage their IT investments.
We are making a number of recommendations, including that each agency take
action to address IT strategic planning, performance measurement, and
investment management practices that are not fully in place.
We received written or oral comments on a draft of this report from 25 of
the agencies9 in our review. Most agencies generally agreed with our
findings and recommendations, and some provided additional documentation
and information that we incorporated into the report, as appropriate.
Background Advances in the use of IT and the Internet are continuing to
change the way that federal agencies communicate, use, and disseminate
information; deliver services; and conduct business. For example,
electronic government (e-government) has the potential to help build
better relationships between government and the public by facilitating
timely and
9DOD submitted a single letter that included comments from the Departments
of the Air Force, Army, and Navy.
efficient interaction with citizens. To help the agencies more effectively
manage IT, the Congress has established a statutory framework of
requirements and roles and responsibilities relating to information and
technology management. Nevertheless, the agencies face significant
challenges in effectively planning for and managing their IT. Such
challenges can be overcome through the use of a systematic and robust
management approach that addresses critical elements, such as IT strategic
planning and investment management.
Federal Government's Statutory Framework for Information and Technology
Management
The Congress established a statutory framework to help address the
information and technology management challenges that agencies face. Under
this framework, agencies are accountable for effectively and efficiently
developing, acquiring, and using IT in their organizations. In particular,
the Paperwork Reduction Act of 1995 and the Clinger-Cohen Act of 1996
require agency heads, acting through agency CIOs, to, among other things,
o better link their IT planning and investment decisions to program
missions and goals;
o develop and maintain a strategic IRM plan that describes how IRM
activities help accomplish agency missions;
o develop and maintain an ongoing process to establish goals for
improving IRM's contribution to program productivity, efficiency, and
effectiveness; methods for measuring progress toward these goals; and
clear roles and responsibilities for achieving these goals;
o develop and implement a sound IT architecture;
o implement and enforce IT management policies, procedures, standards,
and guidelines;
o establish policies and procedures for ensuring that IT systems provide
reliable, consistent, and timely financial or program performance data;
and
o implement and enforce applicable policies, procedures, standards, and
guidelines on privacy, security, disclosure, and information sharing.
Moreover, under the government's current legislative framework, OMB has
important responsibilities for providing direction on governmentwide
information and technology management and overseeing agency activities in
these areas. Among OMB's responsibilities are
o ensuring agency integration of IRM plans, program plans, and budgets
for the acquisition and use of IT and the efficiency and effectiveness of
interagency IT initiatives;
o developing and maintaining a governmentwide strategic IRM plan;
o developing, as part of the budget process, a mechanism for analyzing,
tracking, and evaluating the risks and results of all major capital
investments made by an executive agency for information systems;10
o directing and overseeing the implementation of policy, principles,
standards, and guidelines for the dissemination of and access to public
information;
o encouraging agency heads to develop and use best practices in IT
acquisitions; and
o developing and overseeing the implementation of privacy and security
policies, principles, standards, and guidelines.
Further, in 2002, the Congress passed, and the President signed,
legislation intended to improve the collection, use, and dissemination of
government information and to strengthen information security.
Specifically, Public Law 107-347, the E-Government Act of 2002, which was
enacted in December 2002, includes provisions to promote the use of the
Internet and other information technologies to provide government services
electronically. The E-Government Act also contains the Federal Information
Security Management Act (FISMA) of 2002, which replaced and strengthened
the Government Information Security Reform legislative provisions
(commonly referred to as "GISRA").11 Among other provisions,
10This responsibility is in addition to OMB's role in assisting the
President in reviewing agency budget submissions and compiling the
President's budget, as discussed in 31 U.S.C. chapter 11.
11Government Information Security Reform, Title X, Subtitle G, Floyd D.
Spence National Defense Authorization Act for Fiscal Year 2001, P.L.
106-398, Oct. 30, 2000.
FISMA requires each agency, including national security agencies, to (1)
establish an agencywide risk-based information security program to be
overseen by the agency CIO and ensure that information security is
practiced throughout the life cycle of each agency system; and (2)
develop, maintain, and annually update an inventory of major information
systems (including major national security systems) operated by the agency
or under its control.
Federal IT Challenges Even with the framework laid out by the Congress,
the federal government faces enduring IT challenges. Specifically, in
January 2003, we reported on a variety of challenges facing federal
agencies in continuing to take advantage of the opportunities presented by
IT.12 Unless and until the challenges outlined below are overcome, federal
agencies are unlikely to optimize their use of IT, which can affect an
organization's ability to effectively and efficiently implement its
programs and missions.
o Pursuing opportunities for e-government. E-government offers many
opportunities to better serve the public, make government more efficient
and effective, and reduce costs. Federal agencies have implemented a wide
array of e-government applications, including using the Internet to
collect and disseminate information and forms; buy and pay for goods and
services; submit bids and proposals; and apply for licenses, grants, and
benefits. Although substantial progress has been made, the government has
not yet fully reached its potential in this area. Recognizing this, a key
element of the President's Management Agenda is the expansion of
e-government to enhance access to information and services, particularly
through the Internet. In response, OMB established a task force that
selected a strategic set of initiatives to lead this expansion. Our review
of the initial planning projects associated with these initiatives found
that important aspects-such as collaboration and customer focus-had not
been thought out for all of the projects and that major uncertainties in
funding and milestones were not uncommon. Accordingly, we recommended that
OMB take
12GAO-03-95.
steps as overseer of the e-government initiatives to reduce the risk that
the projects would not meet their objectives.13
o Improving the collection, use, and dissemination of government
information. The rapid evolution of IT is creating challenges in managing
and preserving electronic records. Complex electronic records are
increasingly being created in a decentralized environment and in volumes
that make it difficult to organize them and make them accessible. Further,
storage media themselves are affected by the dual problems of obsolescence
and deterioration. These problems are compounded as computer hardware and
application software become obsolete, since they may leave behind
electronic records that can no longer be read. Overall responsibility for
the government's electronic records lies with the National Archives and
Records Administration (NARA). Our past work has shown that while NARA has
taken some action to respond to the challenges associated with managing
and preserving electronic records, most electronic records remain
unscheduled; that is, their value had not been assessed and their
disposition had not been determined.14 In addition, records of historical
value were not being identified and provided to NARA; as a result, they
were at risk of being lost. We recommended that NARA develop strategies
for raising agency management's awareness of the importance of records
management and for performing systematic inspections. In July 2003 we
testified that although NARA has made progress in addressing these issues,
more work remains to be done.15
The growth of electronic information-as well as the security threats
facing our nation-are also highlighting privacy issues. For example,
online privacy has emerged as one of the key-and most contentious- issues
surrounding the continued evolution of the Internet. In addition, our
survey of 25 departments and agencies about their implementation of the
Privacy Act-which regulates how federal agencies may use the personal
information that individuals supply when obtaining
13U.S. General Accounting Office, Electronic Government: Selection and
Implementation of the Office of Management and Budget's 24 Initiatives,
GAO-03-229 (Washington, D.C.: Nov. 22, 2002).
14U.S. General Accounting Office, Information Management: Challenges in
Managing and Preserving Electronic Records, GAO-02-586 (Washington, D.C.:
June 17, 2002).
15U.S. General Accounting Office, Electronic Records: Management and
Preservation Pose Challenges, GAO-03-936T (Washington, D.C.: July 8,
2003).
government services or fulfilling obligations-found that a key
characteristic of the agencies' 2,400 systems of records is that an
estimated 70 percent contained electronic records.16 Our survey also found
that although compliance with Privacy Act provisions and related OMB
guidance was generally high in many areas, according to agency reports, it
was uneven across the federal government. To improve agency compliance and
address issues reported by the agencies, we made recommendations to OMB,
such as to direct agencies to correct compliance deficiencies, to monitor
agency compliance, and to reassess its guidance.
o Strengthening information security. Since September 1996, we have
reported that poor information security is a high-risk area across the
federal government with potentially devastating consequences.17 Although
agencies have taken steps to redesign and strengthen their information
system security programs, our analyses of information security at major
federal agencies have shown that federal systems were not being adequately
protected from computer-based threats. Our latest analyses of audit
reports published from October 2001 through October 2002 continue to show
significant weaknesses in federal computer systems that put critical
operations and assets at risk.18 In addition, in June 2003 we testified
that agencies' fiscal year 2002 reports and evaluations required by GISRA
found that many agencies have not implemented security requirements for
most of their systems, such as performing risk assessments and testing
controls.19 In addition, the usefulness of agency corrective action plans
may be limited when they do not identify all weaknesses or contain
realistic completion dates.
One of the most serious problems currently facing the government is cyber
critical infrastructure protection, which is protecting the
16U.S. General Accounting Office, Privacy Act: OMB Leadership Needed to
Improve Agency Compliance, GAO-03-304 (Washington, D.C.: June 30, 2003).
17U.S. General Accounting Office, Information Security: Opportunities for
Improved OMB Oversight of Agency Practices, GAO/AIMD-96-110 (Washington,
D.C.: Sept. 24, 1996).
18U.S. General Accounting Office, Computer Security: Progress Made, but
Critical Federal Operations and Assets Remain at Risk, GAO-03-303T
(Washington, D.C.: Nov. 19, 2002).
19U.S. General Accounting Office, Information Security: Continued Efforts
Needed to Fully Implement Statutory Requirements, GAO-03-852T (Washington,
D.C.: June 24, 2003).
information systems that support the nation's critical infrastructures,
such as national defense and power distribution. Since the September 11
attacks, warnings of the potential for terrorist cyber attacks against our
critical infrastructures have increased. In addition, as greater amounts
of money are transferred through computer systems, as more sensitive
economic and commercial information is exchanged electronically, and as
the nation's defense and intelligence communities increasingly rely on
commercially available information technology, the likelihood increases
that information attacks will threaten vital national interests. Among the
critical infrastructure protection challenges the government faces are (1)
developing a national critical infrastructure protection strategy, (2)
improving analysis and warning capabilities, and (3) improving information
sharing on threats and vulnerabilities. For each of the challenges,
improvements have been made and continuing efforts are in progress, but
much more is needed to address them. In particular, we have identified and
made numerous recommendations over the last several years concerning
critical infrastructure challenges that still need to be addressed. As a
result of our concerns in this area, we have expanded our information
security high-risk area to include cyber critical infrastructure
protection.20
o Constructing and enforcing sound enterprise architectures. Our
experience with federal agencies has shown that attempts to modernize IT
environments without blueprints-models simplifying the complexities of how
agencies operate today, how they want to operate in the future, and how
they will get there-often result in unconstrained investment and systems
that are duplicative and ineffective. Enterprise architectures offer such
blueprints. Our reports on the federal government's use of enterprise
architectures in both February 2002 and November 2003 found that agencies'
use of enterprise architectures was a work in progress, with much to be
accomplished.21 Nevertheless, opportunities exist to significantly improve
this outlook if OMB were to adopt a governmentwide, structured, and
systematic approach to
20U.S. General Accounting Office, High-Risk Series: Protecting Information
Systems Supporting the Federal Government and the Nation's Critical
Infrastructures, GAO-03121 (Washington, D.C.: January 2003).
21U.S. General Accounting Office, Information Technology: Leadership
Remains Key to Agencies Making Progress on Enterprise Architecture
Efforts, GAO-04-40 (Washington, D.C.: Nov. 17, 2003) and Information
Technology: Enterprise Architecture Use across the Federal Government Can
Be Improved, GAO-02-6 (Washington, D.C.: Feb. 19, 2002).
promoting enterprise architecture use, measuring agency progress, and
identifying and pursuing governmentwide solutions to common enterprise
architecture challenges that agencies face. Accordingly, we made
recommendations to OMB to address these areas.
o Employing IT system and service management practices. Our work and
other best-practice research have shown that applying rigorous practices
to the acquisition or development of IT systems or the acquisition of IT
services improves the likelihood of success. In other words, the quality
of IT systems and services is governed largely by the quality of the
processes involved in developing or acquiring each. For example, using
models and methods that define and determine organizations'
software-intensive systems process maturity that were developed by
Carnegie Mellon University's Software Engineering Institute, which is
recognized for its expertise in software processes, we evaluated several
agencies' software development or acquisition processes. We found that
agencies are not consistently using rigorous or disciplined system
management practices. We have made numerous recommendations to agencies to
improve their management processes, and they have taken, or plan to take,
actions to improve.22 Regarding IT services acquisition, we identified
leading commercial practices for outsourcing IT services that government
entities could use to enhance their acquisition of IT services.23
o Using effective agency IT investment management practices. Investments
in IT can have a dramatic impact on an organization's performance. If
managed effectively, these investments can vastly improve government
performance and accountability. If not, however, they can result in
wasteful spending and lost opportunities for improving delivery of
services to the public. Using our information
22For example, see U.S. General Accounting Office, Information Technology:
Inconsistent Software Acquisition Processes at the Defense Logistics
Agency Increase Project Risks, GAO-02-9 (Washington, D.C.: Jan. 10, 2002);
and HUD Information Systems: Immature Software Acquisition Capability
Increases Project Risks, GAO-01-962 (Washington, D.C.: Sept. 14, 2001).
23U.S. General Accounting Office, Information Technology: Leading
Commercial Practices for Outsourcing of Services, GAO-02-214 (Washington,
D.C.: Nov. 30, 2001).
technology investment management maturity framework,24 we evaluated
selected agencies and found that while some processes have been put in
place to help them effectively manage their planned and ongoing IT
investments, more work remains.25
IT Challenges Are Interdependent
Complicating the government's ability to overcome these IT management
challenges are these challenges' interdependencies. As a result, the
inability of an organization to successfully address one IT management
area can reduce the effectiveness of its success in addressing another
management function. For example, a critical aspect of implementing
effective e-government solutions and developing and deploying major
systems development projects is ensuring that robust information security
is built into these endeavors early and is periodically revisited.
The government's many IT challenges can be addressed by the use of
effective planning and execution, which can be achieved, in part, through
strategic planning/performance measurement, and investment management. For
example, strong strategic planning is focused on using IT to help
accomplish the highest priority customer needs and mission goals, while
effective performance measurement helps determine the success or failure
of IT activities. Finally, IT investment management provides a systematic
method for minimizing risks while maximizing the return on investments and
involves a process for selecting, controlling, and evaluating investments.
These processes, too, are interdependent. For example, the investment
management process is a principal mechanism to ensure the effective
execution of an agency's IT strategic plan.
24U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
Exposure Draft, GAO/AIMD10.1.23 (Washington, D.C.: May 2000).
25For example, U.S. General Accounting Office, Information Technology:
Departmental Leadership Crucial to Success of Investment Reforms at
Interior, GAO-03-1028 (Washington, D.C.: Sept. 12, 2003); Bureau of Land
Management: Plan Needed to Sustain Progress in Establishing IT Investment
Management Capabilities, GAO-03-1025 (Washington, D.C.: Sept. 12, 2003);
United States Postal Service: Opportunities to Strengthen IT Investment
Management Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002);
Information Technology: DLA Needs to Strengthen Its Investment Management
Capability, GAO-02-314 (Washington, D.C.: Mar. 15, 2002); and Information
Technology: INS Needs to Strengthen Its Investment Management Capability,
GAO-01-146 (Washington, D.C.: Dec. 29, 2000).
Objectives, Scope, and Methodology
Our objectives were to determine the extent to which federal agencies are
following practices associated with key legislative and other requirements
for (1) IT strategic planning/performance measurement and (2) IT
investment management.
To address these objectives, we identified and reviewed major legislative
requirements and executive orders pertaining to IT strategic planning,
performance measurement, and investment management. Specifically, we
reviewed
o the Paperwork Reduction Act of 1995;
o the Clinger-Cohen Act of 1996;
o the E-Government Act of 2002;
o the Federal Information Security Management Act of 2002;
o Executive Order 13011, Federal Information Technology; and
o Executive Order 13103, Computer Software Piracy.
Using these requirements and policy and guidance issued by OMB26 and
GAO,27 we identified 30 IT management practices that (1) can be applied at
the enterprise level and (2) were verifiable through documentation and
interviews. These 30 practices focused on various critical aspects of IT
strategic management, performance measurement, and investment management,
including the development of IRM plans, the identification of goals and
related measures, and the selection and control of IT investments,
respectively.
26Office of Management and Budget, Circular A-11, Preparation, Submission,
and Execution of the Budget (July 2002) and Circular A-130, Management of
Federal Information Resources (Nov. 30, 2000).
27U.S. General Accounting Office, Information Technology Investment
Management: A Framework for Assessing and Improving Process Maturity,
GAO/AIMD-10.1.23, Exposure Draft (Washington, D.C.: May 2000).
We selected 26 major departments and agencies for our review (23 entities
identified in 31 U.S.C. 901 and the 3 military services).28 At our
request, each agency completed a self-assessment on whether and how it had
implemented the 30 IT management practices. We reviewed the completed
agency self-assessments and accompanying documentation, including agency
and IT strategic plans, agency performance plans and reports required by
the Government Performance and Results Act, and IT investment management
policy and guidance, and interviewed applicable agency IT officials to
corroborate whether the practices were in place. We did not evaluate the
effectiveness of agencies' implementation of the practices. For example,
we did not review specific IT investments to determine whether they were
selected, controlled, and reviewed in accordance with agency policy and
guidance. However, we reviewed applicable prior GAO and agency inspector
general reports and discussed whether agency policies had been fully
implemented with applicable agency IT officials.
On the basis of the above information, we assessed whether the practices
were in place, using the following definitions:
o Yes-the practice was in place.
o Partially-the agency has some, but not all, aspects of the practice in
place. Examples of circumstances in which the agency would receive this
designation include when (1) some, but not all, of the elements of the
practice were in place; (2) the agency documented that it has the
information or process in place but it was not in the prescribed form
(e.g., in a specific document as required by law or OMB); (3) the agency's
documentation was in draft form; or (4) the agency had a policy related to
the practice but evidence supported that it had not been completely or
consistently implemented.
o No-the practice was not in place.
28The Departments of Agriculture, the Air Force, the Army, Commerce,
Defense, Education, Energy, Health and Human Services, Housing and Urban
Development, the Interior, Justice, Labor, the Navy, State,
Transportation, the Treasury, and Veterans Affairs; and the Environmental
Protection Agency, General Services Administration, National Aeronautics
and Space Administration, National Science Foundation, Nuclear Regulatory
Commission, Office of Personnel Management, Small Business Administration,
Social Security Administration, and U.S. Agency for International
Development.
o Not applicable-the practice was not relevant to the agency's particular
circumstances.
We also collected information from the Department of Homeland Security
(DHS) but found that since it had been established so recently, it was too
early to judge its IT strategic planning, performance measurement, and
investment management. As a result, although we provided information on
what DHS was doing with respect to these areas, we did not include it in
our assessment.
We also interviewed officials from OMB's Office of Information and
Regulatory Affairs regarding OMB's role in establishing policies and
overseeing agencies' implementation of the identified practices.
We performed our work at the agencies' offices in greater Washington, D.C.
We conducted our review between April and mid-December 2003 in accordance
with generally accepted government auditing standards.
Agencies' Use of IT Strategic Planning/Performance Measurement Practices
Is Uneven
The use of IT strategic planning/performance measurement practices is
uneven (see fig. 1), which is of concern because a well-defined strategic
planning process helps ensure that an agency's IT goals are aligned with
that agency's strategic goals. Moreover, establishing performance measures
and monitoring actual-versus-expected performance of those measures can
help determine whether IT is making a difference in improving performance.
Among the practices or elements of practices that agencies largely have in
place were those pertaining to establishing goals and performance
measures. On the other hand, agencies are less likely to have fully
documented their IT strategic planning processes, developed comprehensive
IRM plans, linked performance measures to their enterprisewide IT goals,
or monitored actual-versus-expected performance for these enterprisewide
goals. Agencies cited various reasons, such as the lack of support from
agency leadership, for not having strategic practices/performance
measurement practices in place. Without strong strategic management
practices, it is less likely that IT is being used to maximize improvement
in mission performance. Moreover, without enterprisewide performance
measures that are being tracked against actual results, agencies lack
critical information about whether their overall IT activities, at a
governmentwide cost of billions of dollars annually, are achieving
expected goals.
Figure 1: Percentage of Agencies' Use of IT Strategic Planning/Performance
Measurement Practices
6%
Critical aspects of the strategic planning/performance measurement area
include documenting the agency's IT strategic planning processes,
developing IRM plans, establishing goals, and measuring performance to
evaluate whether goals are being met. Although the agencies often have
these practices, or elements of these practices, in place, additional work
remains, as demonstrated by the following examples:
o Strategic planning process. Strategic planning defines what an
organization seeks to accomplish and identifies the strategies it will use
to achieve desired results. A defined strategic planning process allows
Yes
Partially
No
Not applicable
Source: GAO.
Note: Yes-the practice was in place. Partially-the agency has some, but
not all, aspects of the practice in place. Examples of circumstances in
which the agency would receive this designation include when (1) some, but
not all, of the elements of the practice were in place; (2) the agency
documented that it has the information or process in place but it was not
in the prescribed form (e.g., in a specific document as required by law or
OMB); (3) the agency's documentation was in draft form; or (4) the agency
had a policy related to the practice but evidence supported that it had
not been completely or consistently implemented. No-the practice was not
in place. Not applicable-the practice was not relevant to the agency's
particular circumstances.
Governmentwide Progress Demonstrated, but More Work Remains
an agency to clearly articulate its strategic direction and to establish
linkages among planning elements such as goals, objectives, and
strategies.
About half of the agencies fully documented their strategic planning
processes. For example, the General Services Administration (GSA)
documented an IT governance structure that addresses the roles and
responsibilities of various organizations in strategic planning and
investment management. In addition, in its IT strategic plan, GSA
describes how it developed the plan, including its vision, businessrelated
priorities, and goals. In contrast, the Department of Agriculture has not
completely documented its IT strategic planning process or integrated its
IT management operations and decisions with other agency processes.
According to Agriculture IT officials, the department's ongoing budget and
performance integration initiative is expected to result in a more clearly
defined and integrated IT strategic management planning process. Such a
process provides the essential foundation for ensuring that IT resources
are effectively managed.
o Strategic IRM plans. The Paperwork Reduction Act requires that agencies
indicate in strategic IRM plans how they are applying information
resources to improve the productivity, efficiency, and effectiveness of
government programs. An important element of a strategic plan is that it
presents an integrated system of high-level decisions that are reached
through a formal, visible process. The plan is thus an effective tool with
which to communicate the mission and direction to stakeholders. In
addition, a strategic IRM plan that communicates a clear and comprehensive
vision for how the agency will use information resources to improve agency
performance is important because IRM encompasses virtually all aspects of
an agency's information activities.
Although the Paperwork Reduction Act also requires agencies to develop IRM
plans in accordance with OMB's guidance, OMB does not provide cohesive
guidance on the specific contents of IRM plans. OMB Circular A-130 directs
that agencies have IRM plans that support agency strategic plans, provide
a description of how IRM helps accomplish agency missions, and ensure that
IRM decisions are integrated with organizational planning, budgets,
procurement, financial management, human resources management, and program
decisions. However, Circular A-130 does not provide overall guidance on
the plan's contents. As a result, although agencies generally
provided OMB with a variety of planning documents to meet its requirement
that they submit an IRM plan, these plans were generally limited to IT
strategic or e-government issues and did not address other elements of
IRM, as defined by the Paperwork Reduction Act. Specifically, these plans
generally include individual IT projects and initiatives, security, and
enterprise architecture elements but do not often address other
information functions, such as information collection, records management,
and privacy, or the coordinated management of all information functions.
OMB IT staff agreed that the agency has not set forth guidance on the
contents of agency IRM plans in a single place, stating that its focus has
been on looking at agencies' cumulative results and not on planning
documents. In addition, these staff also noted that agencies account for
their IRM activities through multiple documents (e.g., Information
Collection Budgets29 and Government Paperwork Elimination Act30 plans).
However, the OMB IT staff stated that they would look at whether more
guidance is needed to help agencies in their development of IRM plans, but
have not yet made a commitment to provide such guidance. Half the agencies
indicated a need for OMB to provide additional guidance on the development
and content of IRM plans.
Strong agency strategic IRM plans could also provide valuable input to a
governmentwide IRM plan, which is also required by the Paperwork Reduction
Act. As we reported last year, although OMB designated the CIO Council's
strategic plan for fiscal years 2001-2002 as the governmentwide strategic
IRM plan, it does not constitute an effective and comprehensive strategic
vision.31 Accordingly, we recommended that OMB develop and implement a
governmentwide strategic IRM plan that articulates a comprehensive federal
vision and plan for all aspects
29Each year, OMB's Office of Information and Regulatory Affairs publishes
an Information Collection Budget by gathering data from executive branch
agencies on the total number of burden hours it approved for collection of
information at the end of the fiscal year and agency estimates of the
burden for the coming fiscal year.
30In fulfilling its responsibilities under this act, OMB requires agencies
to report to OMB on their plans for providing the public with the option
of submitting, maintaining, and disclosing required information
electronically, instead of on paper.
31U.S. General Accounting Office, Information Resources Management:
Comprehensive Strategic Plan Needed to Address Mounting Challenges,
GAO-02-292 (Washington, D.C.: Feb. 22, 2002).
of government information. In April 2003, we testified that OMB had taken
a number of actions that demonstrate progress in fulfilling the Paperwork
Reduction Act's requirement of providing a unifying IRM vision.32 However,
more remains to be done. In particular, we reported that although OMB's
strategies and models are promising, their ability to reduce paperwork
burden and accomplish other objectives depends on how OMB implements them.
One element required by the Clinger-Cohen Act to be included in agency IRM
plans is the identification of a major IT acquisition program(s), or any
phase or increment of that program, that significantly deviated from cost,
performance, or schedule goals established by the program. However, few
agencies met this requirement. In these cases, a common reason cited for
not including this information was that it was not appropriate to have
such detailed information in a strategic plan because such plans should be
forward thinking and may not be developed every year. Agencies also
identified other mechanisms that they use to track and report cost,
schedule, and performance deviations. Because agencies generally do not
address this Clinger-Cohen Act requirement in their IRM plans, they may
benefit from additional guidance from OMB on how to address this
requirement.
o IT goals. The Paperwork Reduction Act and the Clinger-Cohen Act require
agencies to establish goals that address how IT contributes to program
productivity, efficiency, effectiveness, and service delivery to the
public. We have previously reported that leading organizations define
specific goals, objectives, and measures, use a diversity of measure
types, and describe how IT outputs and outcomes impact operational
customer and agency program delivery requirements.33
The agencies generally have the types of goals outlined in the Paperwork
Reduction Act and the Clinger-Cohen Act. For example, the Social Security
Administration (SSA) set a goal of achieving an average of at least a 2
percent per year improvement in productivity, and it
32U.S. General Accounting Office, Paperwork Reduction Act: Record Increase
in Agencies' Burden Estimates, GAO-03-619T (Washington, D.C.: April 11,
2003).
33U.S. General Accounting Office, Executive Guide: Measuring Performance
and Demonstrating Results of Information Technology Investments,
GAO/AIMD-98-89 (Washington, D.C.: March 1998).
expects that advances in automation will be a key to achieving this goal
along with process and regulation changes. In addition, the Department of
Veterans Affairs' (VA) latest departmental strategic plan has a goal that
includes using business process reengineering and technology integration
to speed up delivery of benefit payments, improve the quality of health
care provided in its medical centers, and administer programs more
efficiently. The VA goal includes strategies such as using its enterprise
architecture as a continuous improvement process, implementing
e-government solutions to transform paper-based electronic collections to
electronic-based mechanisms, and establishing a single, high-performance
wide area data network. Five agencies do not have one or more of the goals
required by the Paperwork Reduction Act and the Clinger-Cohen Act. For
example, the Department of Labor's single IT strategic goal-to provide
better and more secure service to citizens, businesses, government, and
Labor employees to improve mission performance-which it included in its
fiscal year 2004 performance plan, does not address all required goals.
Further, in contrast to other agencies, Labor does not have goals in its
IRM plan. It is important that agencies specify clear goals and objectives
to set the focus and direction of IT performance.
o IT performance measures. The Paperwork Reduction Act, the Clinger-Cohen
Act, and Executive Order 13103 require agencies to establish a variety of
IT performance measures, such as those related to how IT contributes to
program productivity, efficiency, and effectiveness, and to monitor the
actual-versus-expected performance of those measures. As we have
previously reported, an effective performance management system offers a
variety of benefits, including serving as an early warning indicator of
problems and the effectiveness of corrective actions, providing input to
resource allocation and planning, and providing periodic feedback to
employees, customers, stakeholders, and the general public about the
quality, quantity, cost, and timeliness of products and services.34
Although the agencies largely have one or more of the required performance
measures, these measures are not always linked to the agencies'
enterprisewide IT goals. For example, the Department of Defense (DOD), Air
Force, and Navy have a variety of enterprisewide IT goals but do not have
performance measures associated with these
34GAO/AIMD-98-89.
goals. Each of these organizations are in the process of developing such
measures. To illustrate, the Air Force's August 2002 information strategy
includes nine goals, such as providing decision makers and all Air Force
personnel with on-demand access to authoritative, relevant, and sufficient
information to perform their duties efficiently and effectively, but does
not have performance measures for these goals. The Air Force recognizes
the importance of linking performance measures to its goals and is
developing such measures, which it expects to complete by the fourth
quarter of fiscal year 2004.
Leading organizations use performance measures to objectively evaluate
mission, business, and project outcomes. Such organizations also focus on
performance measures for gauging service to key management processes and
tailoring performance measures to determine whether IT is making a
difference in improving performance. Few agencies monitored
actual-versus-expected performance for all of their enterprisewide IT
goals. Specifically, although some agencies tracked actual-versus-expected
outcomes for the IT performance measures in their performance plans or
accountability reports and/or for specific IT projects, they generally did
not track the performance measures specified in their IRM plans. For
example, although the Department of Health and Human Services' (HHS) IT
strategic plan identifies enterprisewide goals and performance measures,
these measures generally do not identify quantified outcomes (e.g., the
measures indicate that the outcome will be a percentage transaction
increase or cost decrease in certain areas but do not provide a baseline
or target). In addition, the HHS plan does not describe how the department
will monitor actual-versus-expected performance for these measures. HHS's
Director of Business Operations in its IRM office reported that the
department recognizes the need to develop an integrated program for
monitoring performance against the enterprisewide measures in the IT
strategic plan. He stated that HHS has recently begun an initiative to
establish such a process. By not measuring actual-versus-expected
performance, agencies lack the information to determine where to target
agency resources to improve overall mission accomplishment.
o Benchmarking. The Clinger-Cohen Act requires agencies to quantitatively
benchmark agency process performance against publicand private-sector
organizations, where comparable processes and organizations exist.
Benchmarking is used by entities because there may be external
organizations that have more innovative or more efficient
processes than their own processes. Our previous study of IT performance
measurement at leading organizations found that they had spent
considerable time and effort comparing their performance information with
that of other organizations.35
Seven agencies have mechanisms-such as policies and strategies-in place
related to benchmarking their IT processes. For example, DOD's information
resources and IT directive states that DOD components shall routinely and
systematically benchmark their functional processes against models of
excellence in the public and private sector and use these and other
analyses to develop, simplify, or refine the processes before IT solutions
are applied. In general, however, agencies' benchmarking decisions are ad
hoc. Few agencies have developed a mechanism to identify comparable
external private- or public-sector organizations and processes and/or have
policies related to benchmarking; however, all but 10 of the agencies
provided examples of benchmarking that had been performed. For example,
the Small Business Administration (SBA) does not have benchmarking
policies in place, but the agency provided an example of a benchmarking
study performed by a contractor that compared SBA's IT operations and
processes against industry cost and performance benchmarks and best
practices and resulted in recommendations for improvement.
Practice-Specific Analysis Table 1 provides additional detail on each
strategic planning/performance measurement practice and our evaluation of
whether each agency had the practice in place. The table indicates that
work remains for the agencies to have each of the practices fully in place
as well as that several agencies reported that they were taking, or
planned to take, actions to address the practices or elements of
practices.
35GAO/AIMD-98-89.
Table 1: IT Strategic Planning/Performance Measurement Practicesa
Practice 1.1: The agency has documented its IT strategic management
process, including, at a minimum,
o the responsibilities and accountability for IT resources across the
agency, including the relationship between the chief information officer
(CIO), chief financial officer (CFO), and mission/program officials; and
o the method by which the agency defines program information needs and
develops strategies, systems, and capabilities to meet those needs.
Results Comments
Yes 12 o Yes-the Departments of the Air Force, Army, Commerce, Defense
(DOD), Education, Energy, Labor, Navy, and Veterans Affairs (VA) and the
General Services Administration (GSA), the Office of Personnel Management
(OPM),
Partially 11 and the Social Security Administration (SSA) have this
practice in place.
o Partially-the Departments of Agriculture,c Health and Human Services
(HHS),c Interior, Justice, and No 1 Transportation, and the Environmental
Protection Agency (EPA), the National Aeronautics and Space Administration
(NASA),c and the Small Business Administration (SBA) do not have a
completely documented IT
NA 2 strategic planning process. The Department of Housing and Urban
Development (HUD)c does not clearly describe the roles and
responsibilities of the CFO and program managers in IT strategic planning.
The Nuclear Regulatory Commission's (NRC) roles and responsibilities in
its IT strategic management process are not clearly defined. The
Department of the Treasury'sc documentation supporting this practice is in
draft form.
o No-the National Science Foundation (NSF) does not have this practice in
place.
o NA (not applicable)-the Department of State and the U.S. Agency for
International Development (USAID) are transitioning to a joint strategic
planning process that will support their common policy objectives. The
first step in this process was the August 2003 issuance of a State/USAID
strategic plan. Because a new joint IT strategic planning process is also
being implemented, it is too early to evaluate whether the new process
will address this practice.
Practice 1.2: The agency has documented its process to integrate IT
management operations and decisions with organizational planning, budget,
financial management, human resources management, and program decisions.
Results Comments
Yes 13 o Yes-Air Force, Army, Commerce, DOD, Education, GSA, Labor,
Navy, NSF, OPM, SBA, SSA, and VA have this practice in place.
Partially 10 o Partially-Agriculturec and EPA have not completely
documented the integration of their IT management operations and decisions
with other agency processes. Energy,c HUD, NASA,c and Justice have not
documented how their IT
No 1 management operations and decisions are integrated with human
resources management. HHSc has not documented how its IT management
operations and decisions are integrated with its budget processes. NRC
NA 2 reported that improvement is needed in how IT planning is integrated
with the budget and human resources management. Transportation'sc IT human
capital planning is not yet integrated with the agency's human capital
planning. Treasury'sc documentation pertaining to this practice is in
draft form.
o No-Interior does not have this practice in place.
o NA-this practice is not applicable to State and USAID for reasons
outlined in practice 1.1.
Practice 1.3: The agency requires that information security management
processes be integrated with strategic and operational planning processes.
Results Comments
Yes 24 o Yes-Agriculture, Air Force, Army, Commerce, DOD, Education,
Energy, EPA, GSA, HHS, HUD, Interior, Justice, Labor, NASA, Navy, NSF,
OPM, SBA, SSA, State, Transportation, USAID, and VA have this practice in
place.
Partially 2 o Partially-NRC and Treasury'sc documentation supporting
this practice is in draft form.
No 0
NA 0
(Continued From Previous Page)
Practice 1.4: The agency has a process that involves the CFO, or
comparable official, to develop and maintain a full and accurate
accounting of IT-related expenditures, expenses, and results.
Results Comments
Yes 15 o Yes-Agriculture, Commerce, Energy, GSA, HUD, Interior, Justice,
NASA, NRC, NSF, OPM, SSA, Transportation, Treasury, and VA reported that
they have this practice in place.b
Partially 11 o Partially-prior GAO or inspector general work indicates
that Army, Air Force, DOD, EPA, and Navy do not capture and report on the
full costs of their programs. State and USAID reported that IT internal
costs are not consistently
No 0 captured. HHS reported that not all internal costs are captured and
that the CFO is not involved in the process used to derive its IT costs.
Education and Labor's CFOs are not involved in the process used to derive
their IT
NA 0 costs. SBA reported that not all costs are captured for nonmajor
systems.
Practice 1.5: The agency prepares an enterprisewide strategic information
resources management (IRM) plan that, at a minimum,
o describes how IT activities will be used to help accomplish agency
missions and operations, including related resources; and
o identifies a major IT acquisition program(s) or any phase or increment
of that program that has significantly deviated from the cost,
performance, or schedule goals established for the program.
Results Comments
Yes 2 o Yes-Commerce and NSF have this practice in place.
o Partially-Agriculture, Air Force, Army, EPA, GSA, HHS, HUD, Interior,
Justice, Labor, NASA, OPM, and SBA's IRM Partially 22 plans do not include
resources and major IT acquisition programs that deviated from cost,
schedule, or performance goals. Education, Energy, Navy, SSA, and
Transportation's IRM plans do not include major IT No 0 acquisition
programs that deviated from cost, schedule, or performance goals. DOD and
NRC's draft IRM plans do not include resources and major IT acquisition
programs that deviated from cost, schedule, or performance goals in
NA 2 their IRM plans. Treasury and VA's draft IRM plans do not include
resources or major IT acquisition programs that deviated from cost,
schedule, or performance goals in their IRM plans.
o NA-this practice is not applicable to State and USAID for reasons
outlined in practice 1.1.
Practice 1.6: The agency's performance plan required under Government
Performance and Results Act (GPRA) includes
o a description of how IT supports strategic and program goals,
o the resources and time periods required to implement the information
security program plan required by the Federal Information Security
Management Act (FISMA), and
o a description of major IT acquisitions contained in the capital asset
plan that will bear significantly on the achievement of a performance
goal.
Results Comments
Yes 0 o Partially-no agency's performance plan, except VA's, includes
time periods, and none includes resources required to implement the
information security program plan required by FISMA. In addition,
Agriculture, DOD, HHS, and
Partially 23 Interior's plans also do not include a description of major
IT acquisitions contained in their capital asset plans that bear
significantly on the achievement of a performance goal.
No 0 o NA-this practice is not applicable to Air Force, Army, and Navy
because they are not required to produce such plans.
NA 3
(Continued From Previous Page)
Practice 1.7: The agency has a documented process to
o develop IT goals in support of agency needs,
o measure progress against these goals, and
o assign roles and responsibilities for achieving these goals.
Results Comments
Yes 4 o Yes-Army, GSA, OPM, and SSA have this practice in place.
o Partially-Agriculture,c NRC, and NSF do not have a documented process
for assigning roles and responsibilities Partially 12 for achieving their
enterprisewide IT goals. DODc and HHSc have not established a documented
process for measuring progress against their enterprisewide IT goals.
Energy has this process in place for some, but not all, of No 8 its IT
goals and performance measures. Air Force,c Education, and Navyc do not
have a documented process to measure against their enterprisewide IT goals
or to assign roles and responsibilities for achieving these goals.
NA 2 Treasury'sc documentation in support of this practice is in draft
form. Transportation is piloting a process. VA'sc documentation supporting
this practice does not explicitly address how IT goals are developed and
roles and responsibilities assigned.
o No-Commerce,c EPA, HUD,c Interior, Justice,c Labor, NASA, and SBA do
not have this practice in place.
o NA-this practice is not applicable to State and USAID for reasons
outlined in practice 1.1.
Practice 1.8: The agency has established goals that, at a minimum, address
how IT contributes to
o program productivity,
o efficiency,
o effectiveness, and
o service delivery to the public (if applicable).
Results Comments
Yes 19 o Yes-Agriculture, Air Force, Army, Commerce, DOD, Education,
EPA, GSA, HHS, HUD, Interior, Justice, NASA, NSF, OPM, SBA, SSA, Treasury,
and VA have this practice in place.
Partially 5 o Partially-Navy does not have an IT goal associated with
service delivery to the public. Energy, Labor, and Transportation do not
have a goal associated with how IT contributes to program productivity.
NRC's
No 0 documentation in support of this practice is in draft form.
o NA-this practice is not applicable to State and USAID for reasons
outlined in practice 1.1.
NA 2
(Continued From Previous Page)
Practice 1.9: The agency has established IT performance measures and
monitors actual-versus-expected performance that at least addresses
o how IT contributes to program productivity,
o how IT contributes to the efficiency of agency operations,
o how IT contributes to the effectiveness of agency operations,
o service delivery to the public (if applicable),
o how electronic government initiatives enable progress toward agency
goals and statutory mandates,
o the performance of IT programs (e.g., system development and
acquisition projects), and
o agency compliance with federal software piracy policy.
Results Comments
Yes 0 o Partially-Agriculture,c HHS,c Interior, NASA, OPM, and VAc
generally do not track actual-versus-expected performance for
enterprisewide measures in their IRM plans. Commerce,c EPA, Justice, SBA,
and Treasury have
Partially 23 some enterprisewide IT performance measures in their
performance plans or accountability reports in which
actual-versus-expected performance is tracked but do not have measures for
the enterprisewide IT goals in their
No 1 IRM plans. SBA also does not have performance measures associated
with program productivity, efficiency, effectiveness, and performance of
IT programs. Moreover, Treasury'sc IRM plan is in draft form. Air Forcec
has not
NA 2 developed measures for the enterprisewide goals in its information
strategy and does not have measures associated with program productivity,
electronic government, and service delivery to the public. Armyc has
neither performance measures for all of the objectives related to its
enterprise IT goals nor measures associated with service delivery to the
public. Navyc has not developed measures for the enterprisewide goals in
its IRM plan and does not have measures related to how IT contributes to
the effectiveness and efficiency of agency operations, service delivery to
the public, or e-government. Education does not have measures related to
how IT contributes to program productivity and the effectiveness and
efficiency of agency operations and does not track actual-versusexpected
performance of measures identified in its IRM plan. GSA did not provide
evidence that it tracked actual versus expected performance for one of its
IT goals in its IRM plan. HUDc does not have performance measures related
to how IT contributed to program productivity and does not track
actual-versus-expected performance for enterprisewide measures in its IRM
plan. Labor does not have performance measures associated with program
productivity and efficiency. Energy and NRC's performance measures are not
linked to the enterprisewide IT goals contained in their IRM plans. In
addition, Energy does not have a measure associated with program
productivity. Transportation'sc performance measures are generally not
linked to the goals contained in its IRM plan, and it does not track
actual-versus-expected performance for its enterprisewide measures. SSA
reported that it has performance measures associated with the overall
performance of its IT programs but provided no supporting documentation.
Finally, no agency has performance measures related to the effectiveness
of controls to prevent software piracy. o No-DODc does not have this
practice in place but is working on developing such measures.
o NA-this practice is not applicable to State and USAID for reasons
outlined in practice 1.1.
Practice 1.10: The agency has developed IT performance measures that align
with and support the goals in the GPRA performance plan.
Results Comments
Yes 22 o Yes-Agriculture, Commerce, Education, Energy, EPA, GSA, HHS, HUD,
Interior, Justice, Labor, NASA, NRC, NSF, OPM, SBA, SSA, State, Transportation,
Treasury, USAID, and VA have this practice in place.
Partially 0 o No-DOD does not have this practice in place.
o NA-this practice is not applicable to the Air Force, Army, and Navy
because they are not required to produce such No 1 plans.
NA 3
(Continued From Previous Page)
Practice 1.11: The agency developed an annual report, included as part of
its budget submission, that describes progress in achieving goals for
improving the efficiency and effectiveness of agency operations and, as
appropriate, the delivery of services to the public through the effective
use of IT.
Results Comments
Yes 25 o Yes-Agriculture, Air Force, Army, Commerce, DOD, Education,
Energy, EPA, GSA, HHS, HUD, Interior, Justice, Labor, NASA, Navy, NRC,
NSF, OPM, SSA, State, Transportation, Treasury, USAID, and VA have this
practice in Partially 1 place.
o Partially-SBA has not reported progress on achieving its goals for
improving the efficiency and effectiveness of No 0 agency operations.
NA 0
Practice 1.12: The agency requires that its IT management processes be
benchmarked against appropriate processes and/or organizations from the
public and private sectors in terms of cost, speed, productivity, and
quality of outputs and outcomes where comparable processes and
organizations in the public or private sectors exist.
Results Comments
Yes 7 o Yes-Air Force, Army, DOD, Education, Navy, NRC, and VA have this
practice in place.
o Partially-Agriculture, Commerce, Energy, GSA, Interior, NASA, SBA, SSA,
and Transportation provided an Partially 9 example of a process that they
have benchmarked, but benchmarking is being performed on an ad hoc basis.
o No-EPA, HHS,c HUD,c Justice, Labor, NSF, OPM, State, Treasury,c and
USAID do not have this practice in place. No 10
NA 0
Source: GAO.
aDue to its recent establishment, we did not include DHS as a part of this
analysis.
bWe have previously reported that agencies are making progress to address
financial management system weaknesses but that agency management does not
yet have the full range of information needed for accountability,
performance reporting, and decision making. In addition, for fiscal year
2002, auditors reported that 19 agency systems were not compliant with the
Federal Financial Management Improvement Act, including Agriculture,
Commerce, Education, HUD, Interior, and NASA. (Financial Management:
Sustained Efforts Needed to Achieve FFMIA Accountability, GAO-031062,
Sept. 30, 2002).
cThe agency reported that it was taking, or planned to take, action to
address this practice or elements of the practice.
Note: Yes-the practice was in place. Partially-the agency has some, but
not all, aspects of the practice in place. Examples of circumstances in
which the agency would receive this designation include when (1) some, but
not all, of the elements of the practice were in place; (2) the agency
documented that it has the information or process in place but it was not
in the prescribed form (e.g., in a specific document as required by law or
OMB); (3) the agency's documentation was in draft form; or (4) the agency
had a policy related to the practice, but evidence supported that it had
not been completely or consistently implemented. No-the practice was not
in place. NA (not applicable)-the practice was not relevant to the
agency's particular circumstances.
Agency IT officials could not identify why practices were not in place in
all cases, but in those instances in which reasons were identified, a
variety of explanations were provided. For example, reasons cited by
agency IT officials included that they lacked the support from agency
leadership, that the agency had not been developing IRM plans until
recently and recognized that the plan needed further refinement, that the
process was
being revised (in at least one case because of changes that are needed to
reflect a loss of component organizations to the new DHS), and that
requirements were evolving. In other cases, the agency reported that it
had the information but it was not in the format required by legislation.
For instance, FISMA requires agencies to include in the performance plans
required by the Government Performance and Results Act the resources,
including budget, staffing, and training, and time periods to implement
its information security program. None of the agencies included this
information in their performance plans.36 However, the agencies commonly
reported that they had this information but that it was in another
document. Nevertheless, this does not negate the need for having the
agency report to the Congress in the required form. This is particularly
important since, as in the example of the FISMA requirement, the reporting
requirement involves a public document, whereas other reports may not be
publicly available.
In the case of DHS, while we did not include the department in our
assessment and in table 1, the department is in the process of developing
its first IT strategic plan. According to DHS, it expects to complete this
plan by mid-February 2004.
Agencies' Use of IT Investment Management Practices Is Mixed
The use of IT investment management practices is mixed (as shown in fig.
2), which demonstrates that agencies do not have all the processes in
place to effectively select, control, and evaluate investments. An IT
investment management process is an integrated approach to managing
investments that provides for the continuous identification, selection,
control, life-cycle management, and evaluation of IT investments. Among
the investment management practices that are most frequently in place are
having investment management boards and requiring that projects
demonstrate that they are economically beneficial. Practices less commonly
in place are those requiring that IT investments be performed in a
modular, or incremental, manner and that they be effectively controlled.
Only by effectively and efficiently managing their IT resources through a
robust investment management process can agencies gain opportunities to
make better allocation decisions among many investment alternatives and
further leverage their IT investments.
36VA included the time periods to implement its information security
program in its performance plan.
Figure 2: Percentage of Agencies' Use of IT Investment Management Practicesa
1%
Critical aspects of IT investment management include developing
wellsupported proposals, establishing investment management boards, and
selecting and controlling IT investments. The agencies' use of practices
associated with these aspects of investment management is wide-ranging, as
follows:
o IT investment proposals. Various legislative requirements, an executive
order, and OMB policies provide minimum standards that govern agencies'
consideration of IT investments. In addition, we have issued
Yes
Partially
No
Not applicable
Source: GAO.
aPercentages do not add up to 100 percent due to rounding.
Note: Yes-the practice was in place. Partially-the agency has some, but
not all, aspects of the practice in place. Examples of circumstances in
which the agency would receive this designation include when (1) some, but
not all, of the elements of the practice were in place; (2) the agency
documented that it has the information or process in place but it was not
in the prescribed form (e.g., in a specific document as required by law or
OMB); (3) the agency's documentation was in draft form; or (4) the agency
had a policy related to the practice, but evidence supported that it had
not been completely or consistently implemented. No-the practice was not
in place. Not applicable-the practice was not relevant to the agency's
particular circumstances.
Governmentwide Progress Demonstrated, but More Work Remains
guidance to agencies for selecting, controlling, and evaluating IT
investments.37 Such processes help ensure, for example, that investments
are cost-beneficial and meet mission needs and that the most appropriate
development or acquisition approach is chosen.
The agencies in our review have mixed results when evaluated against these
various criteria. For example, the agencies almost always require that
proposed investments demonstrate that they support the agency's business
needs, are cost-beneficial, address security issues, and consider
alternatives. To demonstrate, the Department of Transportation requires
that proposed projects complete a business case to indicate that the
project (1) will meet basic requirements in areas such as mission need,
affordability, technical standards, and disabled access requirements, (2)
is economically beneficial, and (3) has considered alternatives.
One element in this area that agencies were not as likely to have fully in
place was the Clinger-Cohen Act requirement that agencies follow, to the
maximum extent practicable, a modular, or incremental, approach when
investing in IT projects. Incremental investment helps to mitigate the
risks inherent in large IT acquisitions/developments by breaking apart a
single large project into smaller, independently useful components with
known and defined relationships and dependencies. An example of such an
approach is DOD's policy stating that IT acquisition decisions should be
based on phased, evolutionary segments that are as brief and narrow in
scope as possible and that each segment should solve a specific part of an
overall mission problem and deliver a measurable net benefit independent
of future segments.38 However, 14 agencies do not have a policy that calls
for investments to be done in a modular manner. For example, although the
Environmental Protection Agency (EPA) reported that it worked with program
offices to try to segment work so that the scope and size of each project
is manageable, it does not have a policy that calls for
37GAO/AIMD-10.1.23 and U.S. General Accounting Office, Assessing Risks and
Returns: A Guide for Evaluating Federal Agencies' IT Investment
Decision-making, GAO/AIMD10.1.13 (Washington, D.C.: February 1997).
38We have previously reported that certain DOD system acquisitions were
not utilizing incremental management best practices or were just beginning
to do so. For example, see U.S. General Accounting Office, DOD Systems
Modernization: Continued Investment in Standard Procurement System Has Not
Been Justified, GAO-01-682 (Washington, D.C.: July 31, 2001).
investments to be done in a modular manner. The absence of a policy calls
into question whether EPA is implementing incremental investment in a
consistent and effective manner.
o Investment management boards. Our investment management guide states
that establishing one or more IT investment boards is a key component of
the investment management process. According to our guide, the membership
of this board should include key business executives and should be
responsible for final project funding decisions or should provide
recommendations for the projects under its scope of authority. Such
executive-level boards, made up of business-unit executives, concentrate
management's attention on assessing and managing risks and regulating the
trade-offs between continued funding of existing operations and developing
new performance capabilities.
Almost all of the agencies in our review have one or more enterpriselevel
investment management boards. For example, HUD's Technology Investment
Board Executive Committee and supporting boards have responsibility for
selecting, controlling, and evaluating the department's IT investments.
HUD's contractor-performed maturity audits also have helped the department
validate its board structure and its related investment management
processes. However, the investment management boards for six agencies are
not involved, or the agency did not document the board's involvement, in
the control phase. For example, the National Science Foundation (NSF) has
a CIO advisory group that addresses only the select phase of the IT
investment management process. NSF's CIO explained that the agency reviews
the progress of its major information system projects through other means,
such as meetings with management. In providing comments on a draft of this
report, the CIO stated that he believes that NSF has a comprehensive set
of management processes and review structures to select, control, and
evaluate IT investments and cited various groups and committees used as
part of this process. However, NSF's summary of its investment management
process and memo establishing the CIO advisory group include only general
statements related to the oversight of IT investments, and NSF provided no
additional documentation demonstrating that its investment management
board plays a role in the control and evaluation phases. Our investment
management guidance identifies having an IT investment management board(s)
be responsible for project oversight as a critical process. Maintaining
responsibility for oversight with the same body that selected the
investment is crucial to
fostering a culture of accountability by holding the investment board that
initially selected an investment responsible for its ongoing success.
In addition, 17 agencies do not fully address the practice that calls for
processes to be in place that address the coordination and alignment of
multiple investment review boards. For example, we recently reported that
the Department of the Interior has established three departmentlevel IT
investment boards and begun to take steps to ensure that investment boards
are established at the bureau level.39 However, at the time of our review,
the department (1) could not assert that department-level board members
exhibited core competencies in using Interior's IT investment approach and
(2) had limited ability to oversee investments in its bureaus. We made
recommendations to Interior to strengthen both the activities of the
department-level boards and the department's ability to oversee investment
management activities at the bureaus.
o Selection of IT investments. During the selection phase of an IT
investment management process, the organization (1) selects projects that
will best support its mission needs and (2) identifies and analyzes each
project's risks and returns before committing significant funds. To
achieve desired results, it is important that agencies have a selection
process that, for example, uses selection criteria to choose the IT
investments that best support the organization's mission and prioritizes
proposals.
Twenty-two agencies use selection criteria in choosing their IT
investments. In addition, about half the agencies use scoring models40 to
help choose their investments. For example, the working group and CIO
office officials that support the Department of Education's investment
review board used a scoring model as part of deciding which IT investments
to recommend for the board's consideration and approval. This model
contained two main categories of criteria: (1) value criteria that
measured the impact and significance of the
39GAO-03-1028.
40With a scoring model, the assessment body typically attaches numerical
scores and "relative value" weights to each of the individual selection
criteria. Investments are then assessed relative to these scores and then
against weights associated with each individual criterion. Finally, the
weighted scores are summed to create a numerical value for each
investment.
initiative, given project goals and the strategic objectives of the
department; and (2) health criteria that measured the potential for the
success of the initiative and helped to assess both the performance and
the associated risks that are involved in project and contract management.
In the case of DOD, in February 2003 we reported that it had established
some, and was establishing other IT investment criteria, but these
criteria had not been finalized.41 Accordingly, we recommended, and DOD
concurred, that DOD establish a standard set of criteria. In September we
reported that this recommendation had not been implemented.42 DOD
officials stated that the department was developing the criteria but that
the proposed governance structure had not yet been adopted.
o Control over IT investments. During the control phase of the IT
investment management process, the organization ensures that, as projects
develop and as funds are spent, the project is continuing to meet mission
needs at the expected levels of cost and risk. If the project is not
meeting expectations or if problems have arisen, steps are quickly taken
to address the deficiencies. Executive level oversight of projectlevel
management activities provides the organization with increased assurance
that each investment will achieve the desired cost, benefit, and schedule
results.
Although no agencies had the practices associated with the control phase
fully in place, some have implemented important aspects of this phase. For
example, Labor requires project managers to prepare a control status
report based on a review schedule established during the selection phase,
which is reviewed by the Office of the CIO and its technical review board
as part of determining whether to continue,
41U.S. General Accounting Office, DOD Business Systems Modernization:
Improvements to Enterprise Architecture Development and Implementation
Efforts Needed, GAO-03-458 (Washington, D.C.: Feb. 28, 2003).
42U.S. General Accounting Office, DOD Business Systems Modernization:
Important Progress Made to Develop Business Enterprise Architecture, but
Much Work Remains, GAO-03-1018 (Washington, D.C.: Sept. 19, 2003).
modify, or cancel the initiative.43 For initiatives meeting certain
criteria, the technical review board makes recommendations to the
management council, which serves as the department's top tier executive
investment review council, is chaired by the Assistant Secretary of
Administration and Management, and consists of component agency heads.
Nevertheless, in general, the agencies are weaker in the practices
pertaining to the control phase of the investment management process than
in the selection phase. In particular, the agencies did not always have
important mechanisms in place for agencywide investment management boards
to effectively control investments, including decision-making rules for
project oversight, early warning mechanisms, and/or requirements that
corrective actions for under-performing projects be agreed upon and
tracked. For example, the Department of the Treasury does not have a
department-level control process; instead, each bureau may conduct its own
reviews that address the performance of its IT investments and corrective
actions for underperforming projects. In a multitiered organization like
Treasury, the department is responsible for providing leadership and
oversight for foundational critical processes by ensuring that written
policies and procedures are established, repositories of information are
created that support IT investment decision making, resources are
allocated, responsibilities are assigned, and all of the activities are
properly carried out where they may be most effectively executed. In such
an organization, the CIO is specifically responsible for ensuring that the
organization is effectively managing its IT investments at every level.
Treasury IT officials recognize the department's weaknesses in this area
and informed us that they are working on developing a new capital planning
and investment control process that is expected to address these
weaknesses. Similarly, the Department of Energy is planning on
implementing the investment control process outlined in its September 2003
capital planning and investment control guide in fiscal year 2004, which
addresses important elements such as corrective action plans. However,
this guide does not document the role of Energy's investment management
boards in this process.
43The technical review board, which serves as the department's first-tier
investment review board, is chaired by the deputy CIO, and its members
consist of IRM managers and administrative officers from each component.
Practice-Specific Analysis Table 2 provides additional detail on each
investment management practice and our evaluation of whether each agency
had the practice in place. The table indicates those practices in which
improvement is needed as well as which agencies reported that they were
taking, or planned to take, actions to address the practices or elements
of practices.
Table 2: IT Investment Management Practicesa
Practice 2.1: The agency has a documented IT investment management process
that, at a minimum,
o specifies the roles of key people (including the CIO) and groups within
the IT investment management process, o outlines significant events and
decision points,
o identifies external and environmental factors that influence the
process,
o explains how the IT investment management process is coordinated with
other organizational plans and processes, and
o describes the relationship between the investment management process
and the agency's enterprise architecture.
Results Comments
Yes 12 o Yes-Commerce, Education, Energy, GSA, HUD, Interior, Justice,
OPM, SBA, SSA, State, and USAID have this practice in place.
Partially 14 o Partially-Agriculture and Labor do not describe the
relationship between their investment management processes, and their
enterprise architectures in their IT capital planning and investment
control guide. Air Force,
No 0 EPA, and VA documentation related to this practice is in draft form.
In addition, Air Force'sc draft portfolio management document does not
identify external and environmental factors or describe the relationship
between
NA 0 the investment management process and the enterprise architecture.
DODc is piloting a draft IT portfolio management policy, but this policy
does not address how this process relates to its other organizational
plans and processes and its enterprise architecture or identify external
and environmental factors. HHSc does not address how this process relates
to its other organizational plans and processes and its enterprise
architecture or identify external and environmental factors. NRC's current
and draft capital planning and investment control policies do not address
how this process relates to its other organizational plans and processes
and its enterprise architecture or identify external and environmental
factors. Army and NASA'sc investment management policies and guidance do
not describe the relationship of this process to its enterprise
architecture. Navyc recognizes the need to clarify roles and
responsibilities related to IT investment management, and its IT capital
planning guide does not identify external and environmental factors. NSF
does not have an IT investment management guide, and its summary of its
policy does not address how this process relates to its other
organizational plans and processes and its enterprise architecture or
identify external and environmental factors. Transportation reported that
there was little integration between its capital planning and investment
control process and the budget. Treasuryc does not have a capital planning
and investment control guide, and its documentation supporting this
practice is in draft form.
(Continued From Previous Page)
Practice 2.2: The agency established one or more agencywide IT investment
management boards responsible for selecting, controlling, and evaluating
IT investments that, at a minimum,
o have final project funding decision authority (or provide
recommendations) over projects within their scope of authority, and
o are composed of key business unit executives.
Results Comments
14 o Yes-Agriculture, Commerce, Education, GSA, HHS, HUD, Interior,
Labor, OPM, SBA, SSA, State, Transportation, and VA have this practice in
place. 10 o Partially-Treasuryc and USAID have not completely
implemented this practice. Air Force, Army,c Energy, NASA, NRC, and NSF's
IT investment management boards are not responsible for controlling and
evaluating IT 2 investments, or this role has not been fully documented.
EPA's documentation in support of this practice is in draft form. Navy'sc
IT investment management board governance process is not completely
implemented. 0 o No-DODb,c does not have this practice in place. Justicec
reported that it is piloting an IT investment management board, but did
not provide documentation on the responsibilities, processes, or makeup of
this board.
Practice 2.3: The agencywide board(s) work processes and decision-making
processes are described and documented.
Results Comments
Yes 9 o Yes-Agriculture, Commerce, Education, HUD, Interior, Labor, SBA,
State, and Transportation have this practice in place.
Partially 6 o Partially-Army has not consistently implemented this
practice. GSA did not have policies and procedures for each of its IT
investment management boards. HHS has not established procedures for the
development,
No 11 documentation, and review of IT investments. EPA and VA's
documentation related to this practice is in draft form. USAID has not
completely implemented this practice.
NA 0 o No-Air Force, DOD,b,c Energy, Justice,c NASA, Navy,c NRC, NSF,
OPM, SSA,c and Treasuryc do not have this practice in place.
Practice 2.4: If more than one IT investment management board exists in
the organization (e.g., at the component level), the organization has
o documented policies and procedures that describe the processes for
aligning and coordinating IT investment decision making,
o criteria for determining where in the organization different types of
IT investment decisions are made, and
o processes that describe how cross-functional investments and decisions
(e.g., common applications) are handled.
Results Comments
Yes 2 o Yes-GSA and Labor have this practice in place.
o Partially-Agriculture does not have documented policies and processes
for aligning and coordinating IT Partially 10 investment decision making
or processes for describing how cross-functional investments and decisions
are made. Air Force, Army, Commerce, Education, HHS,c and Transportationc
do not have documented policies and No 7 procedures for aligning and
coordinating investment decision making among their investment management
boards. Interiorb has not fully implemented its governance process for
aligning and coordinating its IT investment
NA 7 decision making. OPM did not describe its criteria for determining
major systems or describe how cross-functional investments and decisions
are handled. SBA did not address whether its enterprisewide board can
invoke final decision-making authority over its program office boards. o
No-DOD,b,c Energy, NASA, Navy,c Treasury,c and VAc do not have this
practice in place. Justicec reported that it is piloting an IT investment
management board but did not provide supporting documentation.
o NA-EPA, HUD, NRC, NSF, SSA, State, and USAID do not have multiple IT
investment management boards.
(Continued From Previous Page)
Practice 2.5: As part of its investment management process, the agency has
available an annually updated comprehensive inventory of its major
information systems that includes major national security systems and
interfaces.
Results Comments
Yes 21 o Yes-Agriculture, Air Force, Commerce, Education, EPA, GSA, HUD,
Interior, Justice, Labor, Navy, NRC, NSF, OPM, SBA, SSA, State,
Transportation, Treasury, USAID, and VA have this practice in place.
Partially 5 o Partially-Army'sc inventory is not complete and does not
include interfaces. A DOD inspector general report stated that DOD's
inventory may not capture the universe of current DOD business management
systems.
No 0 Energy and NASA's inventories do not include interfaces. HHS reported
that its Exhibit 300s fulfill the requirements of this practice but did
not provide supporting documentation.
NA 0
Practice 2.6: A standard, documented procedure is used so that developing
and maintaining the inventory is a repeatable event, which produces
inventory data that are timely, sufficient, complete, and compatible.
Results Comments
Yes 21 o Yes-Agriculture, Air Force, Commerce, DOD, Education, EPA, GSA,
HUD, Interior, Justice, Labor, Navy, NSF, OPM, SBA, SSA, State,
Transportation, Treasury, USAID, and VA have this practice in place.
Partially 1 o Partially-Army's documentation is in draft form.
o No-Energy,c HHS, NASA, and NRC do not have this practice in place. No 4
NA 0
Practice 2.7: The IT asset inventory is used as part of managerial decision
making.
Results Comments
Yes 12 o Yes-Agriculture, Army, Commerce, Education, GSA, HUD, Labor,
Navy, SSA, State, Transportation, and VA have this practice in place.
Partially 11 o Partially-DOD, Energy, EPA, Interior,b NRC,c NSF, OPM,
SBA, and USAID do not explicitly document how their IT asset inventory is
used to identify asset duplication. Air Force reported that its inventory
is not being
No 3 consistently used to identify asset duplication. Justicec reported
that it has begun to use its IT asset inventory to identify asset
duplication as part of a pilot of its new IT investment management
process.
NA 0 o No-HHS, NASA, and Treasuryc do not have this practice in place.
(Continued From Previous Page)
Practice 2.8: Proposed IT investments are required to document that they
have addressed the following items during project planning:
o that the project supports the organization's business and mission needs
and meets users' needs,
o whether the function should be performed by the public or private
sector,
o whether the function or project should be performed or is being
performed by another agency,
o that alternatives have been considered, and
o how security will be addressed.
Results Comments
Yes 25 o Yes-Agriculture, Air Force, Army, Commerce, DOD, Education,
Energy, EPA, GSA, HUD, Interior, Justice, Labor, NASA, Navy, NRC, NSF,
OPM, SBA, SSA, State, Transportation, Treasury, USAID, and VA have this
practice in
Partially 1 place.
o Partially-the HHS policy addressing the element related to whether the
function or project should be performed
No 0 by the private sector or another government agency is in draft form.
This information is normally contained in the Exhibit 300s, but HHS did
not provide us with this documentation.
NA 0
Practice 2.9: In considering a proposed IT project, the agency requires
that the project demonstrate that it is economically beneficial through
the development of a business case that at least addresses costs,
benefits, schedule, and risks.
Results Comments
Yes 25 o Yes-Agriculture, Air Force, Army, Commerce, Education, Energy,
EPA, GSA, HHS, HUD, Interior, Justice, Labor, NASA, Navy, NRC, NSF, OPM,
SBA, SSA, State, Transportation, Treasury, USAID, and VA have this
practice in
Partially 1 place.
o Partially-DOD has not consistently implemented this practice. No 0
NA 0
Practice 2.10: In considering a proposed IT project, the agency requires
that the project demonstrate that it is consistent with federal and agency
enterprise architectures.
Results Comments
Yes 20 o Yes-Air Force, Army, Commerce, Education, Energy, EPA, GSA,
HHS, HUD, Interior, Labor, Navy, NRC, NSF, OPM, SSA, State,
Transportation, Treasury, and VA have this practice in place.
Partially 6 o Partially-the agencies are required to include how major
IT investments align with the agency's enterprise architecture and the
federal enterprise architecture in their budget Exhibit 300s. However, the
following agencies
No 0 do not have policies requiring compliance with the agency enterprise
architectures, which is a core element in our Enterprise Architecture
Management Framework. Agriculture and NASAb,c do not have approved
policies related
NA 0 to this practice but require compliance as part of their IT
investment management reviews for the fiscal year 2005 budget cycle.
Justice and USAID do not have a policy requiring that IT projects comply
with their enterprise architecture. SBA's policy requiring compliance with
its enterprise architecture is in draft form. DODb,c does not have a
policy requiring compliance with its business enterprise architecture.
(Continued From Previous Page)
Practice 2.11: The agency requires that the proposed IT investment, at a
minimum,
o support work processes that it has simplified or redesigned to reduce
costs and improve effectiveness, and
o make maximum use of commercial-off-the-shelf (COTS) software.
Results Comments
Yes 11 o Yes-Air Force, Army, DOD, GSA, Justice, Labor, NASA, Navy, NSF,
SSA, and VA have this practice in place.
o Partially-Education, HHS, Interior, and SBA do not require that
proposed IT investments support work processes Partially 7 that have been
simplified or redesigned. NRC has policies related to this practice but
reported that they have not been fully integrated into its investment
decision making. Energy's business case guidelines address this
practice,No 8 but Energy reported that consideration of these factors is
not required for selection and approval. EPA's policy related to COTS is
in draft form.
NA 0 o No-Agriculture,c Commerce, HUD, OPM, State, Transportation,
Treasury,c and USAID do not have this practice in place.
Practice 2.12: The agency has established project selection criteria
distributed throughout the organization that include, at a minimum,
o cost, benefit, schedule, and risk elements;
o measures such as net benefits, net risks, and risk-adjusted return on
investment; and
o qualitative criteria for comparing and prioritizing alternative
information systems investment projects.
Results Comments
Yes 6 o Yes-Agriculture, GSA, Energy, NASA, Transportation, and VA have
this practice in place.
o Partially-Commerce,c Education, HUD,c Justice, Labor, Navy, SBA,b
State, and USAID have project selection Partially 16 criteria that do not
include net risk and risk-adjusted return on investment. DODb has
established some IT investment criteria, but these criteria are not
finalized or part of an investment review process. EPA has project No 4
selection criteria that do not include net risks, risk-adjusted return on
investment, or qualitative criteria. EPA's documentation in support of
this practice is also in draft form. Interior's project selection criteria
do not include cost
NA 0 and schedule. Air Forcec and Army'sc project selection criteria do
not include cost, benefit, schedule, and risk elements or measures such as
net benefits, net risks, and risk-adjusted return on investment. OPM has
not consistently implemented this practice. SSA'sb criteria is high-level
and not explicit.
o No-HHS,c NRC, NSF, and Treasuryc do not have this practice in place.
(Continued From Previous Page)
Practice 2.13: The agency has established a structured selection process
that, at a minimum,
o selects IT proposals using selection criteria;
o identifies and addresses possible IT investments and proposals that are
conflicting, overlapping, strategically unlinked, or redundant;
o prioritizes proposals; and
o is integrated with budget, financial, and program management decisions.
Results Comments
Yes 8 o Yes-Agriculture, Commerce, Education, GSA, HUD, Labor, SBA, and
State have this practice in place.
o Partially-Air Force'sc documentation in support of this practice is in
draft form and does not include prioritizing Partially 15 proposals across
the enterprise or the use of a scoring model. Army's prioritized list is
limited to investments to address capability shortfalls. DODc is piloting
a draft IT portfolio management policy that includes a selection No 3
process. EPA's documentation of its selection processes is in draft form.
Energy, Interior, and Transportation do not prioritize their IT proposals.
Justicec does not use a scoring model or prioritize or rank its IT
proposals. NASAc
NA 0 does not have a process for identifying possible conflicting,
overlapping, strategically unlinked, or redundant proposals; does not use
a scoring model; and does not prioritize or rank its IT proposals. Navy
generally does not use its IT investment management boards outlined in its
governance process as part of its IT investment selection process and does
not use a scoring model or prioritize or rank its IT proposals. NRC does
not select IT proposals using selection criteria, prioritize proposals, or
document how its selection process is integrated with budget, financial,
and program management decisions. OPM has not consistently implemented
this practice. SSAb does not use a scoring model. USAIDc does not have a
process for identifying possible conflicting, overlapping, strategically
unlinked, or redundant proposals. VA does not have a process to identify
and address possible conflicting, overlapping, strategically unlinked, or
redundant IT investments and does not prioritize IT proposals for
selection.
o No-HHS,c Treasury,c and NSF do not have this practice in place.
Practice 2.14: Agency policy calls for investments to be modularized
(e.g., managed and procured in well-defined useful segments or modules
that are short in duration and small in scope) to the maximum extent
achievable.
Results Comments
Yes 9 o Yes-Air Force, Army, Education, Justice, NASA, Navy, NRC, SBA,
and VA have this practice in place.
o Partially-DOD had not consistently implemented this practice. HHS and
NSF's documentation supporting this Partially 3 practice is in draft form.
o No-Agriculture, Commerce,c Energy, EPA, GSA,c HUD, Interior, Labor,
OPM, SSA, State, Transportation,c No 14 Treasury, and USAID do not have
this practice in place.
NA 0
(Continued From Previous Page)
Practice 2.15: The agencywide investment management board(s) has written
policies and procedures for management oversight of IT projects that
cover, at a minimum,
o decision-making rules for project oversight that allow for terminating
projects, when appropriate;
o current project data, including expected and actual cost, schedule, and
performance data, to be provided to senior management periodically and at
major milestones;
o criteria or thresholds related to deviations in cost, schedule, or
system capability actuals versus expected project performance; and
o the generation of an action plan to address a project's problem(s) and
track resolution.
Results Comments
Yes 0 o Partially-Agriculturec reported that it has not implemented the
corrective action plan element in a consistent manner. Air Force,c NASA,c
and SSAc have control processes but do not explicitly document the role,
responsibility,
Partially 20 and authority of their enterprisewide IT investment
management boards in the control phase. Army,c DOD,b,c and Navy's control
processes do not involve enterprisewide IT investment management boards.
Commercec does not
No 6 have decision-making rules to guide oversight of IT investments and
projects are not required to submit reports of deviations in system
capability. Education has not consistently required corrective actions or
tracked corrective
NA 0 actions related to control phase reviews. GSA does not have clear
decision-making rules, require projects to report on deviations in system
capability, or require that corrective actions be tracked to resolution.
HHSc does not have decision-making rules to guide oversight of IT
investments, review projects at major milestones, or systematically track
corrective actions. HUDc does not require reports of deviations of system
capability or monitor projects at key milestones. Interiorb does not have
decision-making rules for oversight of IT investments, require reports of
deviations of system capability, or require corrective action plans.
Justiceb,c reported that it is piloting an IT investment management board
that includes the control phase but has not provided documentation
supporting that all of the practice elements are addressed. Labor and
Transportation have evaluation criteria to assess investments during the
control phase, but do not have decision-making rules to guide their
investment management boards' decisions. OPM has not consistently
implemented this practice. State's draft documentation does not require
projects to be reviewed at key milestones. USAIDc does not have
decision-making rules, require reports on deviations in system capability,
and review projects at major milestones, and its policy for requiring
action plans is in draft form. VA'sc policies and procedures on
decision-making rules, criteria or thresholds for system capability, and
the generation of action plans have not been fully documented.
o No-SBAb,c and Treasuryc do not have this practice in place. Energy
plans to implement a control process in fiscal year 2004, but its new
capital planning and investment review guide does not address the role of
its investment management boards in the process. EPAc is implementing its
control process in fiscal year 2004. NRC's current and draft capital
planning and investment control documentation do not address the elements
of this practice and do not explicitly document the role, responsibility,
and authority of its enterprisewide IT investment management board in this
process. NSF's investment management board is not responsible for the
control process. NSF reported that it uses other mechanisms to implement
this practice but provided no supporting documentation.
(Continued From Previous Page)
Practice 2.16: The agencywide investment management board(s) established
an oversight mechanism of funded investments that, at a minimum,
o determines whether mission requirements have changed;
o determines whether the investment continues to fulfill ongoing and
anticipated mission requirements;
o determines whether the investment is proceeding in a timely manner
toward agreed-upon milestones;
o employs early warning mechanisms that enable it to take corrective
action at the first sign of cost, schedule, or performance slippages; and
o includes the use of independent verification and validation (IV&V)
reviews of under-performing projects, where appropriate.
Results Comments
Yes 2 o Yes-GSA and VA have this practice in place.
o Partially-Agriculturec reported that its oversight of IT investments
has not been consistently implemented. Air Partially 19 Force,c NASA,c and
SSAc have control processes but did not explicitly document the role,
responsibility, and authority of their enterprisewide IT investment
management boards in this process. Army,c DOD,b,c and Navy's No 5 control
processes do not involve enterprisewide IT investment management boards.
Commerce and Labor do not employ an early warning mechanism. Statec has
procedures for control phase reviews, but they are not fully
NA 0 implemented. Education, HHS, and HUD do not have a process for using
IV&V reviews. Interiorb,c does not have a process to determine whether
investments are proceeding in a timely manner toward agreed-upon
milestones, employ an early warning mechanism, or use IV&V reviews.
Justiceb,c reported that it is piloting an IT investment management board
that includes the control phase but did not provide documentation
supporting that all of the practice elements are addressed. OPM has not
consistently implemented this practice. SBAb did not provide evidence that
it had implemented all of the oversight mechanisms in its investment
management guide and did not use IV&V reviews. Transportation and USAID do
not employ an early warning system or have a process for using IV&V
reviews.
o No-Treasuryc does not have this practice in place. Energy plans to
implement a control process in fiscal year 2004, but its new capital
planning and investment review guide does not address the role of its
investment management boards in the process. EPAc is implementing its
control process in fiscal year 2004. NRC's current and draft capital
planning and investment control documentation does not address the
elements of this practice and does not explicitly document the role,
responsibility, and authority of its enterprisewide IT investment
management board in this process. NSF's investment management board is not
responsible for the control process. NSF reported that it uses other
mechanisms to implement this practice but provided no supporting
documentation.
Practice 2.17: Corrective actions for under-performing projects are agreed
upon, documented, and tracked by the agencywide investment management
board(s).
Results Comments
Yes 5 o Yes-Commerce, HUD, Labor, Transportation, and VA have this
practice in place.
o Partially-Agriculturec and SBAb reported that they have not
consistently implemented this practice. Air Force,c Partially 12 NASA,c
and SSA have control processes but did not explicitly document the role,
responsibility, and authority of their enterprisewide IT investment
management boards in this process. SSAc also did not provide support that
it No 9 was tracking corrective actions. Army,c DOD,b and Navy's control
processes do not involve enterprisewide IT investment management boards.
Education has not consistently required corrective actions or tracked
corrective
NA 0 actions related to control phase reviews. GSA and HHSc do not
systematically track corrective actions. Statec has procedures for control
phase reviews, but they are not fully implemented.
o No-Interior,c Justice, OPM, Treasury,c and USAID do not have this
practice in place. Energy plans to implement a control process in fiscal
year 2004, but its new capital planning and investment review guide does
not address the role of its investment management boards in the process.
EPAc is implementing its control process in fiscal year 2004. NRC's
current and draft capital planning and investment control documentation
does not address the elements of this practice and does not explicitly
document the role, responsibility, and authority of its enterprisewide IT
investment management board in this process. NSF's investment management
board is not responsible for the control process. NSF reported that it
uses other mechanisms to implement this practice, but provided no
supporting documentation.
(Continued From Previous Page)
Practice 2.18: The agencywide investment management board(s) requires that
postimplementation reviews be conducted to
o validate expected benefits and costs, and
o document and disseminate lessons learned.
Results Comments
Yes 6 Partially 17 No 3 NA 3 o Yes-Agriculture, GSA, HUD, Labor, OPM,
and VA have this practice in place.
o Partially-Army, DOD, NASA,c Navy, NRC, NSF, and SSA'sc evaluation
processes do not involve an enterprisewide IT investment management board.
NSF also does not define what is to be included in a postimplementation
review and SSAb reported that such reviews are not done regularly.
Commercec reported that postimplementation reviews have not been
consistently completed and are not required to be reported to its
investment management board. Air Force'sc documentation in support of this
practice is in draft form and does not document the role of its IT
investment management boards in this process. Educationc reported that
postimplementation reviews were not always performed. Energy,c Justice,b,c
Transportation,c and USAID have a policy related to this practice, but it
has not been implemented. Also, Energy's processes do not involve an
enterprisewide IT investment management board. HHS, SBA,b,c and Statec
have a policy related to this practice but did not provide evidence that
it has been completely implemented. In addition, HHS's policy does not
specifically address validating expected benefits and costs.
o No-EPAc is implementing its evaluation process in fiscal year 2004.
Interiorc and Treasuryc do not have this practice in place.
Source: GAO.
aDue to its recent establishment, we did not include DHS as a part of this
analysis.
bWe have an outstanding recommendation related to this practice.
cThe agency reported that it was taking, or planned to take, action to
address this practice, or elements of the practice.
Note: Yes-the practice was in place. Partially-the agency has some, but
not all, aspects of the practice in place. Examples of circumstances in
which the agency would receive this designation include when (1) some, but
not all, of the elements of the practice were in place; (2) the agency
documented that it has the information or process in place but it was not
in the prescribed form (e.g., in a specific document as required by law or
OMB); (3) the agency's documentation was in draft form; or (4) the agency
had a policy related to the practice, but evidence supported that it had
not been completely or consistently implemented. No-the practice was not
in place. Not applicable-the practice was not relevant to the agency's
particular circumstances.
Among the variety of reasons cited for practices not being fully in place
were that the CIO position had been vacant, that not including a
requirement in the IT investment management guide was an oversight, and
that the process was being revised. However, in some cases the agencies
could not identify why certain practices were not in place.
Regarding DHS, although we did not include the department in our
assessment or table 2, the department has investment management processes
that it has put in place or is in the process of putting in place.
Conclusions Federal agencies did not always have in place important
practices associated with IT laws, policies, and guidance. At the
governmentwide level, agencies generally have IT strategic plans or
information resources
management (IRM) plans that address IT elements, such as security and
enterprise architecture, but do not cover other aspects of IRM that are
part of the Paperwork Reduction Act, such as information collection,
records management, and privacy. This may be attributed, in part, to OMB
not establishing comprehensive guidance for the agencies detailing the
elements that should be included in such a plan. There were also numerous
instances of individual agencies that do not have specific IT strategic
planning, performance measurement, or investment management practices
fully in place. Agencies cited a variety of reasons for not having these
practices in place, such as that the CIO position had been vacant, not
including a requirement in guidance was an oversight, or that the process
was being revised. Nevertheless, not only are these practices based on
law, executive orders, OMB policies, and our guidance, but they are also
important ingredients for ensuring effective strategic planning,
performance measurement, and investment management, which, in turn, make
it more likely that the billions of dollars in government IT investments
will be wisely spent. Accordingly, we believe that it is important that
they be expeditiously implemented by individual agencies.
Recommendations To help agencies in developing strategic IRM plans that
fully comply with the Paperwork Reduction Act of 1995, we recommend that
the Director, OMB, develop and disseminate to agencies guidance on
developing such plans. At a minimum, such guidance should address all
elements of IRM, as defined by the Paperwork Reduction Act. As part of
this guidance, OMB should also consider the most effective means for
agencies to communicate information about any major IT acquisition
program(s) or phase or increment of that program that significantly
deviated from cost, performance, or schedule goals established by the
program. One option for communicating this information, for example, could
be through the annual agency performance reports that are required by the
Government Performance and Results Act.
We are also generally making recommendations to the agencies in our review
regarding those practices that are not fully in place unless, for example,
(1) we have outstanding recommendations related to the practice, (2) the
agency has a draft document addressing the practice, or (3) implementation
of the practice was ongoing. Appendix I contains these recommendations.
Agency Comments and Our Evaluation
We received written or oral comments on a draft of this report from OMB
and 25 of the agencies in our review. 44 We also requested comments from
the Department of Homeland Security and the Office of Personnel
Management, but none were provided.
Regarding OMB, in oral comments on a draft of this report, representatives
from OMB's Office of Information and Regulatory Affairs and Office of the
General Counsel questioned the need for additional IRM plan guidance
because they do not want to be prescriptive in terms of what agencies
include in their plans. We continue to believe that agencies need
additional guidance from OMB on the development and content of their IRM
plans because OMB Circular A-130 does not provide overall guidance on the
contents of agency IRM plans and half the agencies indicated a need for
OMB to provide additional guidance on the development and content of IRM
plans. Further, additional guidance would help to ensure that agency plans
address all elements of IRM, as defined by the Paperwork Reduction Act. A
strategic IRM plan that communicates a clear and comprehensive vision for
how the agency will use information resources to improve agency
performance is important because IRM encompasses virtually all aspects of
an agency's information activities.
In commenting on a draft of the report, most of the agencies in our review
generally agreed with our findings and recommendations. The agencies'
specific comments are as follows:
o Agriculture's CIO stated that the department concurred with the
findings in this report and provided information on action it was taking,
or planned to take, to implement the recommendations. Agriculture's
written comments are reproduced in appendix II.
o The Secretary of Commerce concurred with the recommendations in this
report and stated that, in response, the department is updating its
policies and procedures. Commerce's written comments are reproduced in
appendix III.
o DOD's Deputy Assistant Secretary of Defense (Deputy CIO) stated that
the department concurred or partially concurred with the
44DOD submitted a single letter that included comments from the
Departments of the Air Force, Army, and Navy.
recommendations in this report. DOD also provided additional documentation
and information on actions that it is taking, or planned to take, to
address these recommendations. We modified our report based on these
comments and documentation, as appropriate. DOD's written comments, along
with our responses, are reproduced in appendix IV.
o Education's Assistant Secretary for Management/CIO stated that the
agency generally agreed with our assessment of the department's use of IT
strategic planning/performance measurement and investment management
practices. Education provided additional comments and documentation
related to two of our practices. We modified our report on the basis of
these comments and documentation, as appropriate. Education's written
comments, along with our responses, are reproduced in appendix V.
o Energy's Director of Architecture and Standards provided e-mail
comments stating that the department believes that GAO fairly depicted
where the department currently stands in the IT investment management
process. The director also provided other comments that were technical in
nature and that we addressed, as appropriate.
o EPA's Assistant Administrator/CIO generally agreed with our findings
and recommendations on the need to complete work currently under way to
formalize the documentation of IT management practices. However, EPA
questioned our characterization of the agency's IT management and
strategic planning and provided other comments, which we addressed, as
appropriate. EPA's written comments, along with our responses, are
reproduced in appendix VI.
o GSA's CIO stated that the agency generally agreed with the findings and
recommendations in the report. GSA provided suggested changes and
additional information and documentation related to nine of our practices
and two recommendations. We modified our report on the basis of these
comments and documentation, as appropriate. GSA's written comments, along
with our responses, are reproduced in appendix VII.
o HHS's Acting Principal Deputy Inspector General stated that the
department concurred with the findings and recommendations of the report.
HHS's written comments are reproduced in appendix VIII.
o HUD's Assistant Secretary for Administration/CIO stated that the
department was in agreement with the recommendations in this report. HUD's
written comments are reproduced in appendix IX.
o Interior's Acting Assistant Secretary for Policy, Management and Budget
stated that the recommendations in our report would further improve the
department's IT investment management. Interior's written comments are
reproduced in appendix X.
o Justice's CIO stated that, overall, the department concurred with the
findings and recommendations in this report, noting that our
recommendations will assist in further defining IT strategic planning,
performance measurement, and investment management practices. Justice's
written comments, along with our response, are reproduced in appendix XI.
o Labor's Assistant Secretary for Administration and Management/CIO
reported that the department generally concurred with this report and
provided suggested changes in two areas, which we addressed, as
appropriate. Labor's written comments, along with our responses, are
reproduced in appendix XII.
o NASA's Deputy Administrator reported that the agency generally
concurred with the recommendations in this report and provided additional
information on actions that it is taking, or planned to take, to address
these recommendations. NASA's written comments, along with our response,
are reproduced in appendix XIII.
o NSF's CIO provided e-mail comments disagreeing with three areas of this
report. First, NSF did not agree with our assessment of practice 1.1,
stating that the agency has a comprehensive agency-level planning
framework that includes a suite of planning documents and internal and
external oversight activities that it believes addresses IT planning
requirements. However, our review of the planning documents cited by NSF
in its self-assessment found that it did not address the elements of the
practice. In particular, the agency did not describe the responsibility
and accountability for IT resources or the method that it uses to define
program information needs and how such needs will be met. Moreover, in our
exit conference with NSF officials, the CIO indicated agreement with our
assessment. Since NSF provided no additional documentation, we did not
modify the report. Second, the CIO disagreed with our characterization of
the agency's enterprisewide investment management
board. We modified the report to reflect the CIO's comments; however, we
did not change our overall assessment of the role of the board because
NSF's summary of its investment management process and memo establishing
the CIO advisory group include only general statements related to the
oversight of IT investments, and NSF provided no additional documentation
demonstrating that its investment management board plays a role in the
control and evaluation phases. Third, the CIO stated that NSF has
established processes, management, and oversight controls over IT
investments. However, NSF provided limited documentation on the control
phase of its investment management process. In particular, NSF's summary
of its investment management process and memo establishing the CIO
advisory group include only general statements related to the oversight of
IT investments, and NSF provided no additional documentation demonstrating
that its investment management board plays a role in the control and
evaluation phases. Accordingly, we did not modify the report.
o NRC's Executive Director for Operations stated that this report
provides useful information and agreed that the practices are important
for ensuring effective use of government IT investments but had no
specific comments. NRC's written comments are reproduced in appendix XIV.
o SBA's GAO liaison provided e-mail comments questioning the need to have
its enterprise investment management board have final decisionmaking
authority over IT investments. Our IT investment management guidance
states that enterprise-level IT investment boards be capable of reviewing
lower-level board actions and invoking final decision-making authority
over all IT investments.45 In particular, if disputes or disagreements
arise over decision-making jurisdiction about a specific IT investment
project, the enterprise board must be able to resolve the issue.
Accordingly, we did not modify the report. SBA also provided technical
comments that we incorporated, as appropriate.
o SSA's Commissioner generally agreed with the recommendations in the
report and provided comments on each recommendation that we addressed, as
appropriate. SSA's written comments, along with our responses, are
reproduced in appendix XV.
45GAO/AIMD-10.1.23.
o State's Assistant Secretary/Chief Financial Officer stated that the
findings in the report are consistent with discussions held with its IT
staff and provided additional information on four practices. On the basis
of this additional information, we modified our report, as appropriate.
State's written comments, along with our response, are reproduced in
appendix XVI.
o A program analyst in the Department of Transportation's Office of the
CIO provided oral comments that were technical in nature that we
addressed, as appropriate.
o The Acting Director, Budget and Administrative Management in Treasury's
Office of the CIO, provided oral comments stating that the department
concurred with our findings and recommendations. The official further
stated that the department recognized its shortcomings and was working to
correct them.
o USAID's Assistant Administrator, Bureau for Management, did not address
whether the agency agreed or disagreed with our overall findings or
recommendations but commented on our evaluation of two practices, which we
addressed, as appropriate. USAID's written comments, along with our
response, are reproduced in appendix XVII.
o The Secretary of VA stated that the department concurred with the
recommendations in the report and provided comments on actions that it has
taken, or planned to take, in response. We modified the report based on
these comments, as appropriate. VA's written comments, along with our
responses, are reproduced in appendix XVIII.
As agreed with your offices, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the report date. At that time, we will send copies of this report to the
secretaries of the Departments of Agriculture, the Air Force, the Army,
Commerce, Defense, Education, Energy, Health and Human Services, Homeland
Security, Housing and Urban Development, the Interior, Justice, Labor, the
Navy, State, Transportation, the Treasury, and Veterans Affairs; the
administrators of the Environmental Protection Agency, General Services
Administration, National Aeronautics and Space Administration, Small
Business Administration, and U.S. Agency for International Development;
the commissioners of the Nuclear Regulatory Commission and the Social
Security Administration; and the directors of the National
Science Foundation, Office of Management and Budget, and Office of
Personnel Management. We will also make copies available to others upon
request. In addition, this report will be available at no charge on the
GAO Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please
contact me at (202) 512-9286 or Linda J. Lambert, Assistant Director, at
(202) 512-9556. We can also be reached by e-mail at [email protected] and
[email protected], respectively.
Other contacts and key contributors to this report are listed in appendix
XIX.
David A. Powner Director, Information Technology Management Issues
Appendix I
Recommendations to Departments and Agencies
Agriculture To improve the department's information technology (IT)
strategic planning/performance measurement processes, we recommend that
the Secretary of Agriculture take the following six actions:
o document the department's IT strategic management processes and how
they are integrated with other major departmental processes, such as the
budget and human resources management;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by the Federal Information Security Management Act (FISMA) and
include a description of major IT acquisitions contained in its capital
asset plan that bear significantly on its performance goals;
o implement a process for assigning roles and responsibilities for
achieving the department's IT goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for the department's
enterprisewide IT performance measures in its information resources
management (IRM) plan; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Agriculture take the following four
actions:
o include a description of the relationship between the IT investment
management process and the department's enterprise architecture in its IT
capital planning and investment control guide and require that IT
investments be in compliance with the agency's enterprise architecture;
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments, including cross-cutting investments;
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
Appendix I Recommendations to Departments and Agencies
improve effectiveness and that makes maximum use of
commercial-offthe-shelf (COTS) software; and
o establish a policy requiring modularized IT investments.
Air Force To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of the Air Force
take the following two actions:
o establish a documented process for measuring progress against the
department's IT goals and assign roles and responsibilities for achieving
these goals; and
o develop IT performance measures related to the IT goals in the
department's information strategy, including measures such as those
contained in practice 1.9 in our report, and track actual-versus-expected
performance.
To improve the department's IT investment management processes, we
recommend that the Secretary of the Air Force take the following four
actions:
o include a description of the relationship between the IT investment
management process and the department's enterprise architecture, and an
identification of external and environmental factors in its portfolio
management guide;
o include costs, benefits, schedule, and risk elements as well as
measures such as net benefits, net risks, and risk-adjusted
return-on-investment in the department's project selection criteria;
o implement a scoring model and develop a prioritized list of IT
investments as part of its project selection process; and
o document the role, responsibility, and authority of its IT investment
management boards, including work processes, alignment, and coordination
of decision making among its various boards, and document processes for
controlling and evaluating IT investments, such as those outlined in
practices 2.15, 2.16, 2.17, and 2.18.
Appendix I Recommendations to Departments and Agencies
Army To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of the Army take
the following action:
o complete the development of IT performance measures related to the
Army's enterprisewide IT goals, including measures such as those in
practice 1.9 in our report, and track actual-versus-expected performance.
To improve the department's IT investment management processes, we
recommend that the Secretary of the Army take the following four actions:
o include a description of the relationship between the IT investment
management process and the department's enterprise architecture in the
department's IT capital planning and investment control guide;
o document the alignment and coordination of responsibilities of its
various IT investment management boards for decision making related to IT
investments;
o include costs, benefits, schedule, and risk elements as well as
measures such as net benefits, net risks, and risk-adjusted
return-on-investment in the department's project selection criteria; and
o involve the department's IT investment management boards in controlling
and evaluating IT investments, including the development and documentation
of oversight processes such as those in practices 2.15, 2.16, 2.17, and
2.18.
Commerce To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of Commerce take
the following four actions:
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o document its process of developing IT goals in support of agency needs,
measuring progress against these goals, and assigning roles and
responsibilities for achieving these goals;
Appendix I Recommendations to Departments and Agencies
o develop performance measures related to the department's IT goals in
its IRM plan, and track actual-versus-expected performance for these IT
performance measures; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Commerce take the following eight actions:
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments;
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o include net risks and risk-adjusted return-on-investment in the
department's project selection criteria;
o establish a policy requiring modularized IT investments;
o develop decision-making rules to help guide the investment management
board's oversight of IT investments during the control phase;
o require that reports of deviations in systems capability in a project
be submitted to the IT investment management board;
o develop an early warning mechanism that enables the investment
management board to take corrective action at the first sign of cost,
schedule, or performance slippages; and
o require postimplementation reviews be completed and the results
reported to its investment management board.
Defense To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of Defense take the
following three actions:
Appendix I Recommendations to Departments and Agencies
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA, align its performance measures with the goals in the
plan, and include a description of major IT acquisitions contained in its
capital asset plan that bear significantly on its performance goals;
o establish a documented process for measuring progress against the
department's IT goals;
o develop IT performance measures related to its IT goals, including, for
example, the measures contained in practice 1.9 in our report and track
actual-versus-expected performance.
To improve the department's IT investment management processes, we
recommend that the Secretary of Defense take the following action:
o document, as part of its planned IT portfolio management process, how
this process relates to other departmental processes and the department's
enterprise architecture, and document the external and environmental
factors that influence the process.
Education To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of Education take
the following four actions:
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o establish and document a process for measuring progress against the
department's IT goals in its IRM plan and for assigning roles and
responsibilities for achieving these goals;
o develop performance measures related to how IT contributes to program
productivity, the effectiveness and efficiency of agency operations, and
the effectiveness of controls to prevent software piracy; and
o track actual-versus-expected performance for the department's
enterprisewide IT performance measures in its IRM plan.
Appendix I Recommendations to Departments and Agencies
To improve the department's IT investment management processes, we
recommend that the Secretary of Education take the following five actions:
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments;
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs;
o include net risks and risk-adjusted return-on-investment in the
department's project selection criteria;
o develop a process to use independent verification and validation
reviews, when appropriate; and
o track the resolution of corrective actions for under-performing
projects and report the results to the investment management board.
Energy To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of Energy take the
following six actions:
o document how its IT management operations and decisions are integrated
with human resources management;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o develop a goal related to how IT contributes to program productivity;
o develop performance measures related to how IT contributes to program
productivity and the effectiveness of controls to prevent software piracy;
o develop and link performance measures to the department's
enterprisewide goals in its IRM plan and track actual-versus-expected
performance for these measures; and
Appendix I Recommendations to Departments and Agencies
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Energy take the following four actions:
o include interfaces in its inventory of the agency's major information
systems, implement a standard, documented procedure to maintain this
inventory, and develop a mechanism to use the inventory as part of
managerial decision making;
o prioritize the department's IT proposals;
o establish a policy requiring modularized IT investments; and
o document the role, responsibility, and authority of its IT investment
management boards, including work processes, alignment, and coordination
of decision making among its various boards, and document the processes
for controlling and evaluating IT investments, such as those in practices
2.15, 2.16, 2.17, and 2.18.
Environmental Protection Agency
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Administrator of the Environmental
Protection Agency take the following six actions:
o document the agency's IT strategic management processes and how they
are integrated with other major departmental processes, such as the budget
and human resources management;
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
Appendix I Recommendations to Departments and Agencies
o track actual-versus-expected performance for the agency's measures
associated with the IT goals in its IRM plan; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Administrator of the Environmental Protection Agency take the
following three actions:
o include net risks, risk-adjusted return-on-investment, and qualitative
criteria in the agency's project selection criteria;
o establish a policy requiring modularized IT investments; and
o fully implement an IT investment management control phase, including
the elements contained in practices 2.15, 2.16, and 2.17.
General Services Administration
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Administrator of the General Services
Administration take the following four actions:
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for each of the agency's
measures associated with the IT goals in its IRM plan; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Administrator of the General Services Administration take the
following four actions:
o develop work processes and decision-making processes for the agency's
investment management boards;
Appendix I Recommendations to Departments and Agencies
o establish a policy requiring modularized IT investments;
o help guide the oversight of IT investments by developing clear
decisionmaking rules for its IT investment management board and by
requiring that IT projects report on deviations in system capability; and
o track the resolution of corrective actions for under-performing
projects and report the results to the investment management board.
Health and Human Services To improve the department's IT strategic
planning/performance measurement processes, we recommend that the
Secretary of Health and Human Services take the following six actions:
o document the department's IT strategic management processes and how
they are integrated with its budget processes;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA and include a description of major IT acquisitions
contained in its capital asset plan that bear significantly on its
performance goals;
o establish a documented process for measuring progress against the
department's IT goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for its enterprisewide IT
performance measures in its IRM plan; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Health and Human Services take the
following 10 actions:
o revise the department's IT investment management policy to include (1)
how this process relates to other agency processes, (2) an identification
of external and environmental factors, (3) a description of the
Appendix I Recommendations to Departments and Agencies
relationship between the process and the department's enterprise
architecture, and (4) the use of independent verification and validation
reviews, when appropriate.
o develop procedures for the department's enterprisewide investment
management board to document and review IT investments;
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments;
o implement a standard, documented procedure to maintain the department's
inventory of major information systems and develop a mechanism to use the
inventory as part of managerial decision making;
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness;
o implement a structured IT selection process that includes processes and
criteria such as those in practices 2.12 and 2.13;
o develop decision-making rules to help guide the investment management
board's oversight of IT investments during the control phase;
o require the investment management board to review projects at major
milestones;
o track the resolution of corrective actions for under-performing
projects and report the results to the investment management board; and
o revise the department's investment management policy to require
postimplementation reviews to address validating benefits and costs, and
conduct such reviews.
Housing and Urban To improve the department's IT strategic
planning/performance
Development measurement processes, we recommend that the Secretary of
Housing and Urban Development take the following six actions:
Appendix I Recommendations to Departments and Agencies
o document the roles and responsibilities of the chief financial officer
and program managers in IT strategic planning and how the department's IT
management operations and decisions are integrated with human resources
management;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to how IT contributes to program
productivity and the effectiveness of controls to prevent software piracy;
o track actual-versus-expected performance for the department's
enterprisewide IT performance measures in its IRM plan; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Housing and Urban Development take the
following five actions:
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o include net risks and risk-adjusted return-on-investment in the
department's project selection criteria;
o establish a policy requiring modularized IT investments;
o require IT projects to report on deviations in system capability and
monitor IT projects at key milestones; and
o develop a process to use independent verification and validation
reviews, when appropriate.
Appendix I Recommendations to Departments and Agencies
Interior To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of the Interior
take the following six actions:
o document the department's IT strategic management processes and how
they are integrated with other major departmental processes, including
organizational planning, budget, financial management, human resources
management, and program decisions;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA and include a description of major IT acquisitions
contained in its capital asset plan that bear significantly on its
performance goals;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for the department's
enterprisewide IT performance measures in its IRM plan; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of the Interior take the following five
actions:
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness;
o include cost and schedule in the department's project selection
criteria and prioritize its IT proposals;
o establish a policy requiring modularized IT investments;
Appendix I Recommendations to Departments and Agencies
o require that corrective actions be undertaken, tracked, and reported to
the investment management board for under-performing projects; and
o implement an evaluation process for IT investments that addresses the
elements of practice 2.18.
Justice To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Attorney General take the
following six actions:
o document the department's IT strategic management processes;
o document how the department's IT management operations and decisions
are integrated with human resources management processes;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to the department's IT goals in
its IRM plan, and track actual-versus-expected performance for these IT
performance measures; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Attorney General take the following five actions:
o develop work processes and procedures for the department's investment
management boards, including aligning and coordinating IT investment
decision making among its various boards;
o establish a policy requiring that IT investments be in compliance with
the agency's enterprise architecture;
Appendix I Recommendations to Departments and Agencies
o include net risks and risk-adjusted return-on-investment in the
department's project selection criteria;
o implement a scoring model and develop a prioritized list of investments
as part of the department's project selection process; and
o require that corrective actions be undertaken, tracked, and reported to
the investment management board for under-performing projects.
Labor To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of Labor take the
following five actions:
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop a goal related to how IT contributes to program productivity;
o develop performance measures related to how IT contributes to program
productivity, efficiency, and the effectiveness of controls to prevent
software piracy, and track actual-versus-expected performance; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Labor take the following five actions:
o include a description of the relationship between the IT investment
management process and the department's enterprise architecture in the
department's IT capital planning and investment control guide;
o include net risks and risk-adjusted return-on-investment in its project
selection criteria;
Appendix I Recommendations to Departments and Agencies
o establish a policy requiring modularized IT investments;
o develop decision-making rules to help guide the investment management
board's oversight of IT investments during the control phase; and
o develop an early warning mechanism that enables the investment
management board to take corrective action at the first sign of cost,
schedule, or performance slippages.
National Aeronautics and Space Administration
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Administrator of the National Aeronautics
and Space Administration take the following seven actions:
o document the agency's IT strategic management processes;
o document how the agency's IT management operations and decisions are
integrated with human resources management processes;
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for the agency's
enterprisewide IT performance measures in its IRM plan; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Administrator of the National Aeronautics and Space
Administration take the following four actions:
Appendix I Recommendations to Departments and Agencies
o revise the agency's IT investment management policy and guidance to
describe the relationship of this process to the agency's enterprise
architecture;
o include interfaces in its inventory of the agency's major information
systems, implement a standard, documented procedure to maintain this
inventory, and develop a mechanism to use the inventory as part of
managerial decision making;
o within the agency's IT investment selection process, implement a
mechanism to identify possible conflicting, overlapping, strategically
unlinked, or redundant proposals; implement a scoring model; and develop a
prioritized list of investments; and
o document the role, responsibility, and authority of its IT investment
management boards, including work processes, alignment, and coordination
of decision making among its various boards, and document the processes
for controlling and evaluating IT investments, such as those in practices
2.15, 2.16, 2.17, and 2.18.
National Science Foundation
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Director of the National Science
Foundation take the following five actions:
o document the agency's IT strategic management processes;
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o implement a process for assigning roles and responsibilities for
achieving its IT goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
Appendix I Recommendations to Departments and Agencies
To improve the agency's IT investment management processes, we recommend
that the Director of the National Science Foundation take the following
four actions:
o develop an IT investment management guide that includes a description
of the relationship between the IT investment management process and the
agency's other organizational plans and processes and its enterprise
architecture, and identify external and environmental factors that
influence the process in the agency's IT capital planning and investment
control policy;
o implement a structured IT selection process that includes the elements
of practices 2.12 and 2.13;
o involve the department's IT investment management board in controlling
and evaluating IT investments, including the development and documentation
of oversight processes such as those in practices 2.15, 2.16, 2.17, and
2.18; and
o define and document the elements of the agency's postimplementation
reviews.
Navy To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of the Navy take
the following three actions:
o develop a documented process to measure progress against the
department's enterprisewide IT goals and assign roles and responsibilities
for achieving these goals;
o develop an IT goal related to service delivery to the public; and
o develop IT performance measures related to the department's IT goals,
including, at a minimum, measures contained in practice 1.9 in our report,
and track actual-versus-expected performance.
To improve the department's IT investment management processes, we
recommend that the Secretary of the Navy take the following four actions:
o include net risks and risk-adjusted return-on-investment in the
department's project selection criteria;
Appendix I Recommendations to Departments and Agencies
o implement a structured IT selection process that includes the elements
of practice 2.13;
o involve all elements of the department's IT investment management board
governance process in selecting, controlling, and evaluating IT
investments; and
o document the role, responsibility, and authority of its IT investment
management boards, including work processes, alignment, and coordination
of decision making among its various boards, and document the processes
for controlling and evaluating IT investments, such as those outlined in
practices 2.15, 2.16, 2.17, and 2.18.
Nuclear Regulatory Commission
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Commissioner of the Nuclear Regulatory
Commission take the following five actions:
o document the agency's roles and responsibilities for its IT strategic
management processes and how IT planning is integrated with its budget and
human resources planning;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
o develop a documented process to assign roles and responsibilities for
achieving its enterprisewide IT goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy; and
o develop performance measures for the agency's enterprisewide goals in
its IRM plan, and track actual-versus-expected performance for these
measures.
To improve the agency's IT investment management processes, we recommend
that the Commissioner of the Nuclear Regulatory Commission take the
following five actions:
o include a description of the relationship between the IT investment
management process and the department's other organizational plans
Appendix I Recommendations to Departments and Agencies
and processes and its enterprise architecture, and identify external and
environmental factors that influence the process in the agency's IT
capital planning and investment control policy;
o develop work processes and procedures for the agency's investment
management boards;
o implement a standard, documented procedure to maintain its IT asset
inventory, and develop a mechanism to use the inventory as part of
managerial decision making;
o develop a structured IT investment management selection process that
includes project selection criteria, a scoring model, and prioritization
of proposed investments; and
o document the role, responsibility, and authority of its IT investment
management boards, including work processes and control, and evaluate
processes that address the oversight of IT investments, such as what is
outlined in practices 2.15, 2.16, 2.17, and 2.18.
Office of Personnel Management
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Director of the Office of Personnel
Management take the following four actions:
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o track actual-versus-expected performance for the agency's
enterprisewide IT performance measures in its IRM plan; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Director of the Office of Personnel Management take the following
four actions:
Appendix I Recommendations to Departments and Agencies
o develop work processes and procedures for the agency's investment
management board, including establishing criteria for defining major
systems and documenting a process for handling cross-functional
investments;
o implement a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o establish a policy requiring modularized IT investments; and
o require that corrective actions be undertaken, tracked, and reported to
the investment management board for under-performing projects.
Small Business Administration
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Administrator of the Small Business
Administration take the following five actions:
o document the agency's IT strategic management processes;
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA;
o develop a documented process to develop IT goals in support of agency
needs, measure progress against these goals, and assign roles and
responsibilities for achieving these goals;
o develop performance measures related to the agency's IT goals in its
IRM plan, including, at a minimum, measures related to how IT contributes
to program productivity, efficiency, effectiveness, the overall
performance of its IT programs, and the effectiveness of controls to
prevent software piracy, and track actual-versus-expected performance for
these IT performance measures; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
Appendix I Recommendations to Departments and Agencies
To improve the agency's IT investment management processes, we recommend
that the Administrator of the Small Business Administration take the
following two actions:
o document a process that the investment management board can invoke
final decision-making authority over IT investments addressed by
lowerlevel boards; and
o implement a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs.
Social Security Administration
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Commissioner of the Social Security
Administration take the following three actions:
o include in its annual performance plan the resources and time periods
required to implement the information security program plan required by
FISMA;
o develop performance measures related to the performance of the agency's
IT programs and the effectiveness of controls to prevent software piracy;
and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Commissioner of the Social Security Administration take the
following four actions:
o develop work processes and procedures for the agency's investment
management board;
o establish a policy requiring modularized IT investments;
o document the role, responsibility, and authority of its IT investment
management board for the oversight of IT investments, such as what is
outlined in practices 2.15, 2.16, and 2.18; and
o require that corrective actions be tracked and reported to the
investment management board for under-performing projects.
Appendix I Recommendations to Departments and Agencies
State To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of State take the
following two actions:
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of State take the following five actions:
o implement a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o establish a policy requiring modularized IT investments;
o include risk-adjusted return-on-investment in the department's project
selection criteria;
o revise the department's draft IT investment management policy to
include reviewing projects at major milestones; and
o fully implement an IT investment management control phase, including
the elements contained in practices 2.16 and 2.17.
Transportation To improve the department's IT strategic
planning/performance measurement processes, we recommend that the
Secretary of Transportation take the following five actions:
o document its IT strategic planning process;
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
Appendix I Recommendations to Departments and Agencies
o develop a goal related to how IT contributes to program productivity;
o develop performance measures related to the department's IT goals in
its IRM plan, and track actual-versus-expected performance for these IT
performance measures; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of Transportation take the following six
actions:
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments;
o implement a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o prioritize the department's IT proposals;
o establish a policy requiring modularized IT investments;
o develop and document decision-making rules to help guide the investment
management board's oversight of IT investments during the control phase;
and
o as part of the department's control phase, employ an early warning
mechanism, and use independent verification and validation reviews, when
appropriate.
Treasury To improve the department's IT strategic planning/performance
measurement processes, we recommend that the Secretary of the Treasury
take the following four actions:
o include in the department's annual performance plan the resources and
time periods required to implement the information security program plan
required by FISMA;
Appendix I Recommendations to Departments and Agencies
o develop performance measures related to the effectiveness of controls
to prevent software piracy;
o develop performance measures related to the department's IT goals in
its IRM plan, and track actual-versus-expected performance for these IT
performance measures; and
o develop a mechanism for benchmarking the department's IT management
processes, when appropriate.
To improve the department's IT investment management processes, we
recommend that the Secretary of the Treasury take the following eight
actions:
o develop a capital planning and investment control guide that includes,
for example, the elements of practice 2.1;
o develop work processes and procedures for the agency's IT investment
management board, and document the alignment and coordination of
responsibilities of its various boards for decision making related to
investments, including the criteria for which investments-including
cross-cutting investments-will be reviewed by the enterprisewide board;
o use the department's IT asset inventory as part of managerial decision
making, including using it to identify the potential for asset
duplication;
o establish a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o implement a structured IT selection process that includes the elements
of practices 2.12 and 2.13;
o establish a policy requiring modularized IT investments;
o implement an IT investment management process that includes a control
phase that addresses, for example, the elements of practices 2.15, 2.16,
and 2.17; and
Appendix I Recommendations to Departments and Agencies
o implement an IT investment management process that includes an
evaluation phase that addresses, for example, the elements of practice
2.18.
U.S. Agency for International Development
To improve the agency's IT strategic planning/performance measurement
processes, we recommend that the Administrator of the U.S. Agency for
International Development take the following two actions:
o include in the agency's annual performance plan the resources and time
periods required to implement the information security program plan
required by FISMA; and
o develop a mechanism for benchmarking the agency's IT management
processes, when appropriate.
To improve the agency's IT investment management processes, we recommend
that the Administrator of the U.S. Agency for International Development
take the following nine actions:
o develop work processes and procedures for the agency's IT investment
management board;
o establish a policy requiring that IT investments be in compliance with
the agency's enterprise architecture;
o develop a policy requiring that proposed IT investments support work
processes that have been simplified or redesigned to reduce costs and
improve effectiveness and that makes maximum use of COTS software;
o include net risks, risk-adjusted return-on-investment, and qualitative
criteria in the agency's project selection criteria;
o within the agency's IT investment selection process, implement a
mechanism to identify possible conflicting, overlapping, strategically
unlinked, or redundant proposals;
o develop a policy requiring modularized IT investments;
o develop decision-making rules, review projects at major milestones, and
require projects to report on deviations in system capability to help
Appendix I Recommendations to Departments and Agencies
guide the oversight of IT investments by the agency's investment
management board during the control phase;
o as part of the agency's control phase, employ an early warning
mechanism, and use independent verification and validation reviews, when
appropriate; and
o require that corrective actions be undertaken, tracked, and reported to
the investment management board for under-performing projects.
Veterans Affairs To improve the department's IT strategic
planning/performance measurement processes, we recommend that the
Secretary of Veterans Affairs take the following four actions:
o include in the department's annual performance plan the resources
required to implement the information security program plan required by
FISMA;
o develop a documented process to measure progress against the
department's IT goals, and assign roles and responsibilities for achieving
these goals;
o develop performance measures related to the effectiveness of controls
to prevent software piracy; and
o track actual-versus-expected performance for the department's
enterprisewide IT performance measures in its IRM plan.
To improve the department's IT investment management processes, we
recommend that the Secretary of Veterans Affairs take the following two
actions:
o document the alignment and coordination of responsibilities of the
department's various IT investment management boards for decision making
related to IT investments, including cross-cutting investments; and
o within the agency's IT investment selection process, implement a
mechanism to identify possible conflicting, overlapping, strategically
unlinked, or redundant proposals, and prioritize its IT investments.
Appendix II
Comments from the Department of Agriculture
Appendix III
Comments from the Department of Commerce
Appendix IV
Comments from the Department of Defense (including comments from the
Departments of the Air Force, Army, and Navy)
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
See comment 1.
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
See comment 2.
See comment 3.
Appendix IVEURComments from the Department of Defense EUR(including
comments from the Departments EURof the Air Force, Army, and Navy)EUR
Appendix IVEURComments from the Department of Defense EUR(including comments
from the Departments EURof the Air Force, Army, and Navy)EUR
The following are GAO's comments on the Department of Defense's (DOD)
letter dated December 5, 2003.
GAO Comments 1.
2.
3.
DOD provided its annual report to the President and the Congress, which
included its fiscal year 2004 performance plan. Based on a review of this
plan, we modified our report.
We disagree that the cited objective fully addresses this issue.
Specifically, although this objective addresses e-government, the wording
of the objective, its description, and the discussion of related
initiatives do not explicitly address service delivery to the public.
Accordingly, we did not modify our report.
Our review of the acquisition management process documentation provided by
the Navy did not support that the department's selection criteria include
net risks and risk-adjusted return-on-investment. Accordingly, we did not
modify our report.
Appendix V
Comments from the Department of Education
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 1.
See comment 2.
Appendix VEURComments from the Department of EUREducationEUR
The following are GAO's comments on the Department of Education's letter
dated December 10, 2003.
GAO Comments 1. We agree that Education requires IT investments to have
performance measures. However, our practice dealt with enterprise-level
measures, such as those found in the department's IRM plan, not
project-specific measures. Education reported that the performance
measures in its IRM plan do not measure how IT contributes to program
productivity and the efficiency and effectiveness of agency operations.
Accordingly, we did not modify our report.
2. We modified our assessment of practice 2.6 in this report and deleted
the related recommendation based on our evaluation of additional
documentation provided by Education.
Appendix VI
Comments from the Environmental Protection Agency
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix VIEURComments from the Environmental EURProtection AgencyEUR
See comment 1.
See comment 2.
Appendix VIEURComments from the Environmental EURProtection AgencyEUR
The following are GAO's comments on the Environmental Protection Agency's
(EPA) letter dated December 9, 2003.
GAO Comments 1. As we reported and EPA acknowledged, its documentation on
IT strategic planning and investment management was not complete or
finalized. For example, the partial rating we gave EPA for its IT
management and strategic planning practices-practices 1.1 and 1.2- matched
the agency's own self-assessment in these areas. Specifically, our review
of planning documents cited by EPA in its self-assessment found that while
the agency had documented agencywide roles and responsibilities for
planning and managing IT resources and had documented its process to
integrate the IT investment management process with the budget, EPA had
not addressed other key elements of the practices. As an example, EPA had
not fully documented the method by which it defines program information
needs and develops strategies, systems, and capabilities to meet those
needs. Since EPA provided no additional documentation, our practice
assessment and our related recommendations remain unchanged.
2. As stated in our report, practice 1.7 refers to the documentation of
the process used to develop IT goals and measures and the responsibility
for achieving them. As EPA states in its comments, it is currently working
on documenting this process. Accordingly, we did not modify our report.
Appendix VII
Comments from the General Services Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 1.
See comment 2.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 3.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 4.
See comment 5.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 6.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 7.
See comment 8.
See comment 9.
Appendix VIIEURComments from the General Services EURAdministrationEUR
See comment 10.
See comment 11.
See comment 12.
Appendix VIIEURComments from the General Services EURAdministrationEUR
Appendix VIIEURComments from the General Services EURAdministrationEUR
The following are GAO's comments on the General Services Administration's
(GSA) letter dated December 9, 2003.
GAO Comments 1.
2.
3.
4.
5.
6.
We based our evaluation on the agency's self-assessment and comments made
by GSA's Director, Office of Policy and Plans. However, based on GSA's
representation in commenting on our draft, we changed our evaluation of
the referenced practice.
The Clinger-Cohen Act requires agencies to include in its information
resources management (IRM) plan the identification of a major IT
acquisition program(s), or any phase or increment of that program, that
significantly deviated from cost, performance, or schedule goals
established by the program. As we acknowledge in this report, agencies,
which would include GSA, identified other mechanisms that they use to
track and report cost, schedule, and performance deviations. Moreover, we
evaluated agencies as a "partially" instead of a "no" in this practice to
take into account that the agency had the required information, although
it was not in the prescribed format. Accordingly, we did not modify our
report.
The Federal Information Security Management Act of 2002 requires agencies
to include in the performance plans required by the Government Performance
and Results Act the resources and time periods to implement their
information security program. As we noted in this report, agencies, which
would include GSA, commonly stated that they had this information but that
it was in another document. Nevertheless, this does not negate the need
for having the agency report to the Congress in the form that it requires.
This is particularly important since performance plans are public
documents. Accordingly, we did not modify our report.
GSA's new documentation illustrates that it has performance measures for
each of the IT goals in its IRM plan. However, GSA did not provide
evidence that it was tracking actual versus expected performance for
measures associated with one of its goals. We revised our report to
reflect GSA's new documentation and our evaluation.
We revised our report on the basis of this new documentation.
GSA's highest-level IT investment management board is its Executive
Committee. GSA did not provide a charter or any other evidence of
Appendix VIIEURComments from the General Services EURAdministrationEUR
policies and procedures for this committee. We therefore did not modify
our report.
7. The additional documentation provided by GSA (1) does not address
decision-making rules and (2) illustrates that GSA uses a monthly project
control report on cost, schedule, and performance status, but the report
does not explicitly address deviations in system capability. In addition,
according to GSA's capital planning and investment control order, the
format of the report is left to the applicable organization, thereby
making it less likely that the investment management boards are obtaining
consistent information. We therefore did not modify our report.
8. We agree that GSA's capital planning and investment control order
requires that projects that have significant variances are to provide "get
well" plans and that monthly control reports are used to report on project
cost, schedule, and performance status. However, it is not clear that
these status reports can be used to systemically track corrective actions.
Moreover, according to GSA's capital planning and investment control
order, the format of the monthly control report is left to the applicable
organization, thereby making it less likely that the status of corrective
actions is being consistently reported. We therefore did not modify our
report.
9. See comment 8.
10. We modified our recommendations based on our evaluation of GSA's
documentation. See comment 4 for our assessment.
11. Executive Order 13103 requires agencies to use software piracy
performance measures that comply with guidance issued by the federal CIO
Council.1 The Council, in turn, called on the agencies to develop such
measures. The additional documentation that GSA provided was an order
requiring agency employees to use properly licensed software, but it does
not include performance measures that would demonstrate that this
requirement is being honored. Measuring how well agencies are combating
software piracy is important because it can verify that
1The CIO Council is the principal interagency forum for improving agency
practices related to the design, acquisition, development, modernization,
use, operation, sharing, and performance of federal government information
resources.
Appendix VIIEURComments from the General Services EURAdministrationEUR
the controls that they have put in place are working. Accordingly, we did
not change this part of the recommendation.
12. We modified our recommendation to reflect that GSA requires projects
that have significant variances to develop corrective action plans.
However, the other elements of the recommendation pertaining to the
tracking and reporting on corrective actions remain outstanding. See
comment 8 for additional information.
Appendix VIII
Comments from the Department of Health and Human Services
Appendix VIIIEURComments from the Department of Health EURand Human
ServicesEUR
Appendix IX
Comments from the Department of Housing and Urban Development
Appendix X
Comments from the Department of the Interior
Appendix XEURComments from the Department of the EURInteriorEUR
Appendix XI
Comments from the Department of Justice
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XIEURComments from the Department of JusticeEUR
See comment 1.
Appendix XIEURComments from the Department of JusticeEUR
The following are GAO's comments on the Department of Justice's letter
dated December 2, 2003.
GAO Comments 1. GAO has ongoing work looking at OMB's initiative. However,
the Federal Information Security Management Act of 2002 requires agencies
to include in the performance plans required by the Government Performance
and Results Act the resources and time periods to implement its
information security program. Accordingly, we did not change the
recommendation.
Appendix XII
Comments from the Department of Labor
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XIIEURComments from the Department of LaborEUR
See comment 1.
See comment 2.
Appendix XIIEURComments from the Department of LaborEUR
The following are GAO's comments on the Department of Labor's letter dated
December 2, 2003.
GAO Comments 1. Because Labor did not disagree with our characterization
of its IT goal, no changes were made to our report.
2. We agree with Labor's characterization of its IT strategic goal and
order 3-2003. Nevertheless, the recommendation, and related practice 1.7,
refers to the documentation of the process used to develop IT goals and
measures and the responsibility for achieving them. Labor neither provided
documentation of such a process nor took issue with our assessment of
practice 1.7, in which we stated that the agency did not have this
practice in place. Moreover, Labor's self-assessment referenced a draft
performance measurement guidebook and quarterly review process in support
of this practice. However, these mechanisms relate to performance measures
associated with IT projects, not Labor's enterprisewide IT goal. Finally,
as we noted in our report, unlike other agencies in our review, Labor does
not have goals in its IRM plan. Accordingly, we did not change this
recommendation.
Appendix XIII
Comments from the National Aeronautics and Space Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XIIIEURComments from the National Aeronautics EURand Space
AdministrationEUR
Appendix XIIIEURComments from the National Aeronautics EURand Space
AdministrationEUR
See comment 1.
Appendix XIIIEURComments from the National Aeronautics EURand Space
AdministrationEUR
Appendix XIIIEURComments from the National Aeronautics EURand Space
AdministrationEUR
The following are GAO's comments on the National Aeronautics and Space
Administration's (NASA) letter dated December 8, 2003.
GAO Comments 1. Our practice dealt with enterprise-level measures, not
project-specific measures. In addition, although we agree that NASA's IRM
plan included performance measures, the agency generally does not track
actual-versus-expected performance for these enterprisewide measures.
Appendix XIV
Comments from the Nuclear Regulatory Commission
Appendix XV
Comments from the Social Security Administration
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XVEURComments from the Social Security EURAdministrationEUR
See comment 1.
See comment 2.
Appendix XVEURComments from the Social Security EURAdministrationEUR
See comment 3.
See comment 4.
Appendix XVEURComments from the Social Security EURAdministrationEUR
See comment 5.
Appendix XVEURComments from the Social Security EURAdministrationEUR
Appendix XVEURComments from the Social Security EURAdministrationEUR
The following are GAO's comments on the Social Security Administration's
(SSA) letter dated December 3, 2003.
GAO Comments 1.
2.
3.
4.
5.
We agree that SSA needs to consider the level of detail that is
appropriate to include in its performance plans so as not to compromise
security.
We requested documentation to support SSA's assertion that it has
performance measures associated with the performance of IT programs (e.g.,
the percentage of IT projects that are meeting cost, schedule, and
performance goals), but none were provided. Accordingly, we did not modify
our report.
We agree that it is not appropriate to include measures related to the
effectiveness of controls to prevent software piracy in agency performance
plans. Neither our practice nor our recommendation specifies the document
or process that should be used to address software piracy.
As we noted in this report, SSA performs benchmarking in an ad hoc manner.
We believe that taking a more systematic approach is necessary to ensure
that benchmarking is performed at suitable times using an appropriate
methodology. Without a systematic approach, it is not possible to validate
that the agency performs benchmarking "when appropriate." Accordingly, we
did not modify our report.
References to OMB's Circular A-11 in agency policy documentation alone do
not ensure that these practices are met. In particular, we believe that
agency policies related to modularized IT investments should be explicit
and that it is neither prudent nor practical to rely on users of SSA's
documentation of its capital planning and investment control process to
review a secondary source.
Appendix XVI
Comments from the Department of State
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XVIEURComments from the Department of StateEUR
See comment 1.
See comment 2.
Appendix XVIEURComments from the Department of StateEUR
See comment 3.
See comment 4.
Appendix XVIEURComments from the Department of StateEUR
The following are GAO's comments on the Department of State's letter dated
December 9, 2003.
GAO Comments 1.
2.
3.
4.
We based our evaluation on the agency's draft Capital Planning and
Investment Control Program Guide that was provided during our review.
However, based on State's newly finalized Capital Planning and Investment
Control Program Guide, we changed this evaluation in our report.
We based our evaluation on the agency's draft Capital Planning and
Investment Control Program Guide that was provided at the time of our
review. Based on the final version of the Capital Planning and Investment
Control Program Guide provided by State in its response, we modified the
language in our report, as appropriate.
See comment 2.
See comment 2.
Appendix XVII
Comments from the U.S. Agency for International Development
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
See comment 1.
Appendix XVIIEURComments from the U.S. Agency for EURInternational
DevelopmentEUR
Appendix XVIIEURComments from the U.S. Agency for EURInternational
DevelopmentEUR
Appendix XVIIEURComments from the U.S. Agency for EURInternational
DevelopmentEUR
The following are GAO's comments on the U.S. Agency for International
Development's (USAID) letter dated December 9, 2003.
GAO Comments 1. References to OMB's Circular A-11 in agency policy
documentation alone do not ensure that these practices are met. In
particular, we believe that agency policies related to practices 2.11 and
2.14 should be explicit and that it is neither prudent nor practical to
rely on users of USAID's directives to review a secondary source.
Regarding USAID's comments that it uses the criteria in practices 2.11 and
2.14 as part of its evaluation and scoring of investments, we agree that
the agency does ask some questions on the use of commercial-off-the-shelf
software and whether the agency uses "successive chunks" within its
proposed IT investment scoring model. However, addressing these criteria
as part of a scoring model does not address our practice because scoring
projects on the basis of the questions asked does not necessarily preclude
projects from continuing if they do not fully meet the criteria.
Additionally, the questions asked as part of the scoring model do not
fully meet the requirements of the practices. Accordingly, we did not
modify our report.
Appendix XVIII
Comments from the Department of Veterans Affairs
Note: GAO comments supplementing those in the report text appear at the
end of this appendix.
Appendix XVIIIEURComments from the Department of Veterans EURAffairsEUR
See comment 1.
See comment 2.
See comment 3.
Appendix XVIIIEURComments from the Department of Veterans EURAffairsEUR
See comment 4.
See comment 5.
Appendix XVIIIEURComments from the Department of Veterans EURAffairsEUR
See comment 6.
Appendix XVIIIEURComments from the Department of Veterans EURAffairsEUR
See comment 7.
Appendix XVIIIEURComments from the Department of Veterans EURAffairsEUR
The following are GAO's comments on the Department of Veterans Affairs'
(VA) letter dated December 5, 2003.
GAO Comments 1.
2.
3.
4.
5.
6.
7.
VA's response indicates that the department will address this
recommendation in the future and, therefore, we did not remove this
recommendation.
See comment 1.
See comment 1.
VA's monthly performance reports track project-specific measures, not
enterprisewide IT performance measures. VA's draft IRM plan states that it
will establish metrics to measure performance for IT strategic
initiatives. However, progress toward doing so was not addressed by VA in
its comments. Therefore, we do not believe this recommendation has been
fully addressed.
See comment 1.
Although VA describes a process followed for reviewing investment
proposals, it did not provide evidence to support that this practice was
actually followed. In addition, VA did not address the element of our
recommendation related to prioritizing its IT investments. Therefore, we
did not remove this recommendation.
On the basis of the additional information provided, we agree that the
recommendation has been implemented and modified our report accordingly.
Appendix XIX
GAO Contacts and Staff Acknowledgments
GAO Contacts Linda J. Lambert, (202) 512-9556 Mark D. Shaw, (202) 512-6251
Staff Joseph P. Cruz, Lester P. Diamond, Laurence P. Gill, David B.
Hinchman, Robert G. Kershaw, David F. Plocher, Susan S. Sato, and Patricia
D. Slocum
Acknowledgments made key contributions to this report.
GAO's Mission The General Accounting Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO's Web site (www.gao.gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as "Today's Reports," on its
Web site daily. The list contains links to the full-text document files.
To have GAO e-mail this list to you every afternoon, go to www.gao.gov and
select "Subscribe to e-mail alerts" under the "Order GAO Products"
heading.
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. General Accounting Office 441 G Street NW, Room LM Washington, D.C.
20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact: Web site: www.gao.gov/fraudnet/fraudnet.htm
Waste, and Abuse in E-mail: [email protected]
Federal Programs Automated answering system: (800) 424-5454 or (202)
512-7470
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. General Accounting Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted StandardEURPostage & Fees PaidEURGAOEURPermit No. GI00EUR
United StatesEURGeneral Accounting OfficeEURWashington, D.C. 20548-0001EUR
Official BusinessEURPenalty for Private Use $300EUR
Address Service RequestedEUR
*** End of document. ***