Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems
Is Needed (20-AUG-04, GAO-04-822).
The Federal Aviation Administration's (FAA) mission is to promote
the safe, orderly, and expeditious flow of air traffic in the
United States airspace system, commonly referred to as the
National Airspace System (NAS). To maintain its ability to
effectively carry out this mission FAA embarked, in 1981,on a
multi-billion dollar effort to modernize its aging air traffic
control (ATC) system, the principle technology component of the
NAS. Yet the NAS modernization has continued to be plagued by
cost increases, schedule delays, and performance shortfalls. To
gain insight into how FAA is meeting its management challenges,
congressional requesters asked GAO to evaluate FAA's processes
for making IT investment management decisions. The objectives of
this review included (1) evaluating FAA's capabilities for
managing its IT investments and (2) determining what plans, if
any, the agency might have for improving these capabilities.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-04-822
ACCNO: A11881
TITLE: Information Technology: FAA Has Many Investment
Management Capabilities in Place, but More Oversight of
Operational Systems Is Needed
DATE: 08/20/2004
SUBJECT: Air traffic control systems
Information technology
Internal controls
Schedule slippages
Cost overruns
Investment planning
FAA Air Traffic Control System
FAA National Airspace System Plan
GAO Information Technology Investment
Management Framework
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-04-822
United States Government Accountability Office
GAO Report to Congressional Requesters
August 2004
INFORMATION
TECHNOLOGY
FAA Has Many Investment Management Capabilities in Place, but More Oversight of
Operational Systems Is Needed
a
GAO-04-822
August 2004
INFORMATION TECHNOLOGY
FAA Has Many Investment Management Capabilities in Place, but More Oversight of
Operational Systems Is Needed
Judged against the criteria of GAO's framework for information technology
investment management (ITIM), which measures the maturity of an
organization's investment management processes, FAA has established about
80 percent of the basic selection and control practices that it needs to
manage its mission-critical investments (see table below). For example,
business lines actively monitor projects throughout their life cycles.
However, the agency's senior IT investment board does not regularly review
investments that are in the "in-service management," or operational, phase
of their life cycles, and this creates a weakness in FAA's ability to
oversee more than $1 billion of its IT investments. In addition, the
agency has not yet established the key practices that would allow it to
manage all of its investments as one portfolio-an integrated set of
competing options. Until FAA has established the practices that would
enable it to effectively manage its annual IT budget of about $2.5
billion, agency executives lack assurance that they are selecting and
managing the mix of investments that best meets the agency's needs and
priorities.
The agency has initiated efforts to improve its investment management
processes, but it has not yet developed and implemented a comprehensive
plan-supported by management-to guide all of its improvement efforts. Such
a plan is crucial in helping FAA to coordinate and prioritize its
improvement efforts and sustain its commitment to the efforts it already
has under way. Without such a plan-and controls for implementing it-FAA
will be unlikely to develop a mature investment management capability.
Summary of Results for Foundational Critical Processes and Key Practices
Key practices
executed for
Critical process Purpose NAS systems
Highlights of GAO-04-822, a report to congressional requesters
The Federal Aviation Administration's (FAA) mission is to promote the
safe, orderly, and expeditious flow of air traffic in the United States
airspace system, commonly referred to as the National Airspace System
(NAS). To maintain its ability to effectively carry out this mission FAA
embarked, in 1981,on a multibillion dollar effort to modernize its aging
air traffic control (ATC) system, the principle technology component of
the NAS. Yet the NAS modernization has continued to be plagued by cost
increases, schedule delays, and performance shortfalls. To gain insight
into how FAA is meeting its management challenges, congressional
requesters asked GAO to evaluate FAA's processes for making IT investment
management decisions. The objectives of this review included (1)
evaluating FAA's capabilities for managing its IT investments and (2)
determining what plans, if any, the agency might have for improving these
capabilities.
To strengthen FAA's investment management capability, GAO recommends that
FAA develop and implement a plan to address the weaknesses identified in
this report. In commenting on a draft of this report, the Department of
Transportation commented that the report was balanced and fair, showing
where FAA has many capabilities in place and identifying areas that need
improvement.
www.gao.gov/cgi-bin/getrpt?GAO-04-822.
To view the full product, including the scope and methodology, click on
the link above. For more information, contact David Powner, 202-512-9286,
[email protected] or Lester Diamond, 202-512-7957, [email protected].
To define and establish an appropriate IT investment Instituting an
management structure and the processes for selecting, investment board
controlling, and evaluating IT investments. 88% Meeting business To ensure
that IT projects and systems support the needs organization's business
needs and meets users' needs. 86% Selecting an To ensure that a
well-defined and disciplined process is investment used to select new IT
proposals and reselect ongoing
investments. 70%
To review the progress of IT projects and
Providing systems, using
investment pre-defined criteria and checkpoints, in meeting
cost,
oversight schedule, risk, and benefit expectations and to
take
corrective action when these expectations are not
being
met. 57%
To make available to decision makers information
Capturing to
investment evaluate the impacts and opportunities created by
information proposed (or continuing) IT investments. 100%
Overall 79%
Source: GAO.
Contents
Letter
Results in Brief
Background
FAA Has Established an IT Management Structure to Manage Its
NAS Investments FAA Has Initiated Efforts to Improve Its Investment
Management Process DOT is Taking Steps to Integrate Oversight of FAA's IT
Investments Conclusions Recommendations for Executive Action Agency
Comments
1 2 3
19
40
41 42 43 44
Appendixes
Appendix I: Objectives, Scope, and Methodology 45
Appendix II: Investment Management Process Used by Some Organizational
Units to Manage Non-NAS Investments 48
Tables Table 1: Table 2:
Table 3:
Table 4: Table 5: Table 6: Table 7: Table 8: Table 9:
NAS and Non-NAS IT Investments
Stage 2 Critical Processes-Building the Investment
Foundation
Summary of Results for Stage 2 Critical Processes and Key
Practices for NAS Investments
Instituting the Investment Board
Meeting Business Needs
Selecting an Investment
Providing Investment Oversight
Capturing Investment Information
Stage 3 Critical Processes-Developing a Complete
Investment Portfolio
5
18
19 21 25 28 31 34
37 38
Table 10: Status of Stage 3 Critical Processes
FAA's Life Cycle Management Process
Figures Figure 1: Figure 2: Detailed Breakdown of FAA's Life Cycle 8
Management
Process 12
Figure 3: The Five ITIM Stages of Maturity with
Critical
Processes 16
Contents
Abbreviations
ABA Financial Services unit
AHR Human Resource Management unit
AIO Information Services unit
AMS Acquisition Management System
APB acquisition program baseline
ARA Research and Acquisition unit
ARC Region and Center Operations unit
ATC air traffic control
ATO Air Traffic Organization
ATS Air Traffic Services unit
AVR Regulation and Certification unit
CIO Chief Information Officer
DOT Department of Transportation
ECG En Route Communications Gateway
F&E facilities and equipment (predeployment stage of system life
cycle) FAA Federal Aviation Administration FTI FAA Telecommunications
Infrastructure IAT Investment Analysis Team IT information technology ITEB
Information Technology Executive Board ITIM Information Technology
Investment Management framework ITIPS Information Technology Investment
Portfolio System JRC Joint Resources Council NAS National Airspace System
NSIP NAS Support Integration Process OMB Office of Management and Budget
OPS operations (postdeployment stage of system life cycle) PIR
postimplementation review VCSU VSCS Control Subsystem Upgrade VSCS Voice
Switching and Control System
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
August 20, 2004
The Honorable Tom Davis Chairman Committee on Government Reform U.S. House
of Representatives
The Honorable Adam H. Putnam
Chairman
Subcommittee on Technology, Information Policy, Intergovernmental
Relations and the Census Committee on Government Reform U.S. House of
Representatives
The Federal Aviation Administration's (FAA) mission is to promote the
safe, orderly, and expeditious flow of air traffic in the United States
airspace system, commonly referred to as the National Airspace System
(NAS). To maintain its ability to effectively carry out this mission FAA
embarked, in 1981, on a multibillion dollar effort to modernize its aging
air traffic control (ATC) system, the principle technology component of
the NAS. Over the past 2 decades, individual FAA modernization projects
have experienced cost overruns, schedule delays, and performance
shortfalls of large proportions. Because of the size, complexity, cost,
and problemplagued past of FAA's modernization program, we have designated
it a highrisk information technology investment since 1995.1
This report is one in a series of reports responding to your request to
evaluate FAA's efforts to address the information technology (IT)
management challenges it faces as it continues to modernize the ATC
system. It focuses on FAA's processes for making IT investment management
decisions and uses our Information Technology Investment Management (ITIM)
framework,2 which was released at a hearing of the subcommittee on March
3, 2004. The framework provides a method for
1See GAO, High-Risk Series: An Overview, GAO/HR-95-1 (Washington, D.C.:
February 1995); GAO, High-Risk Series: Information Management and
Technology, GAO/HR-97-9 (Washington, D.C.: February 1997); GAO, High-Risk
Series: An Update, GAO/HR-99-1 (Washington, D.C.: January 1999); GAO,
High-Risk Series: An Update, GAO-01-263 (Washington, D.C.: January 2001);
and GAO, High-Risk Series: An Update, GAO-03-119 (Washington, D.C.:
January 2003).
2GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.:
March 2004).
evaluating and assessing how well an agency is selecting and managing its
IT resources. As agreed, our objectives were to (1) evaluate FAA's
capabilities for managing its IT investments, (2) determine what plans the
agency might have for improving these capabilities, and (3) describe the
Department of Transportation's (DOT) oversight of FAA's investments and
investment management process. To address these objectives we analyzed
documents and interviewed agency officials to (1) validate and update
FAA's self-assessments of the key practices in the framework, (2) evaluate
FAA's plans for improving its capabilities, and (3) describe the
department's oversight role. We performed our work from October 2003,
through July 2004, in accordance with generally accepted government
auditing standards. Appendix I contains further details on our objectives,
scope, and methodology.
Results in Brief FAA has established most-about 80 percent-of the basic
practices needed to manage its mission critical investments, including
many of the foundational practices for selecting and controlling IT
investments. These key practices provide additional assurance that the
investments selected will meet organizational needs and will be completed
on time and within budget. The practices also will enable the agency to
manage its IT investments as a portfolio, or integrated set of competing
options.
Even with these many capabilities in place, weaknesses remain in several
areas. Specifically, FAA
o does not involve its senior IT investment board in regular reviews of
investments that have entered the in-service management phase, that is,
those systems that have completed development and become operational;
o does not have standard practices for managing its mission-support and
administrative investments;
o has not developed a process where the senior IT investment board
regularly reviews the full portfolio of investments; and
o has not implemented postimplementation reviews of its major investments
to validate that they are providing the expected benefits after they
become operational.
FAA has begun to act to resolve the weaknesses described above, but until
FAA establishes the practices it needs to effectively manage its IT
investments, executives cannot be assured that they are selecting and
managing the mix of investments that best meets the agency's needs and
priorities. Establishing the capabilities needed to effectively manage
investments requires the development and implementation of a plan,
supported by management, that defines and prioritizes improvements to the
investment process. While FAA has initiated a series of efforts to improve
its investment management processes, it does not have such a plan. Without
this plan-and controls for implementing it-it is unlikely that the agency
will effectively establish mature investment management capabilities.
The Department of Transportation has recently initiated several efforts
that can serve to provide better departmental oversight of FAA
investments. For example, DOT is finalizing capital investment guidance
for all of its operating administrations to follow in implementing their
investment management processes, and it has initiated a process for
reviewing the fiscal year 2006 budget justifications for major programs,
including those of FAA. The department has also identified about a dozen
programs it plans to monitor on a regular basis and has asked FAA to
report cost, schedule, and performance data on some of its programs
quarterly.
To further strengthen FAA's investment management capability, we are
recommending that the agency develop and implement a plan aimed at
addressing the weaknesses identified in this report.
In commenting on a draft of this report, the Department of
Transportation's Director of Audit Relations stated that GAO did a good
job of keeping the report balanced and fair, showing where FAA has many
capabilities in place and identifying areas that need improvement. The
agency also provided a technical comment, which we have incorporated into
the report.
Background
FAA's Mission and As an agency of the Department of Transportation, FAA's
mission is to
Organizational Structure promote the safe, orderly, and expeditious flow
of air traffic in the national airspace. To fulfill its mission requires
the extensive use of technology. The achievement of the agency's mission
is also dependent in large part on the
skills and expertise of its workforce. Its workforce of nearly 50,000
people provides aviation services that include air traffic control;
maintenance of air traffic control equipment; and certification of
aircraft, airline operations, and pilots.
FAA is organized into several staff support offices (examples include the
Office of Information Services and the Office of Human Resource
Management) and five lines of business, which include Airports, Regulation
and Certification, Commercial Space Transportation, the Office of Security
and Hazardous Materials, and the newly formed Air Traffic Organization
(ATO). The ATO was formed on February 8, 2004, to better provide safe,
secure, and cost-effective air traffic services now and into the future.
The Air Traffic Services and the Research and Acquisitions units, which
had been primarily responsible for managing air traffic services within
FAA, were combined into one performance-based organization to create ATO.
ATO is led by FAA's Chief Operating Officer and consists of 10 service
units.3
FAA's Use of IT FAA relies extensively on information technology to carry
out its NAS operations. It constantly depends on the adequacy and
reliability of the nation's ATC system, which comprises a vast network of
radars; automated data processing, navigation, and communications
equipment; and ATC facilities.4 Through this system, FAA provides services
such as controlling takeoffs and landings and managing the flow of traffic
between airports. For example, the Integrated Terminal Weather System is
employed to allow maximum use of airport runways in all kinds of weather
through a variety of weather sensors. The Wide Area Augmentation System is
used to provide
3The 10 service units that make up the ATO include Safety, Communications,
Operations Planning, Finance, Acquisition & Business Services, En Route
and Oceanic Services, Terminal Services, Flight Services, System
Operations Services, and Technical Operations Services.
4FAA uses three types of facilities to control traffic: airport towers,
terminal radar approach control facilities, and en route centers. Airport
towers direct traffic on the ground, before landing, and after takeoff
within 5 nautical miles from the airport and about 3,000 feet above the
airport. Terminal radar approach control facilities sequence and separate
aircraft as they approach and leave airports, beginning about 5 nautical
miles and ending about 50 nautical miles from the airport and generally up
to 10,000 feet above the ground. Air route traffic control centers, called
en route centers, control planes in transit and during approaches to some
airports, generally controlling air space that extends above 18,000 feet
for commercial aircraft.
vertically guided landing to aircraft at thousands of airports and
airstrips where there is currently no vertically guided landing
capability.
FAA also relies on IT to carry out its mission-support and administrative
operations (non-NAS operations). For example, FAA uses IT to support
accident and incident investigations, security inspections, and personnel
and payroll functions.
With an IT budget of about $2.5 billion for fiscal year 2004, FAA accounts
for over 90 percent of the Department of Transportation's IT budget. The
amount of investments in both NAS and non-NAS IT is shown in the table 1
below.
Table 1: NAS and Non-NAS IT Investments
Total IT investment in
Type of fiscal year 2004 in
investment Funding of IT investments billions of dollars
NAS Facilities and Equipment (F&E)
(development through 2 years of 1.464
operations)
Operations (OPS) (through the rest
of the life
cycle) 0.834
Non-NAS Operations 0.350-0.500a
Source: FAA.
aAccording to FAA, these numbers will be verified via baselining by the
end of Fiscal Year 2004 to reflect non-NAS IT assets and their costs.
Prior Reviews Identified Weaknesses in the Agency's IT Investment Management
Process
In 1995, we designated FAA's modernization of its air traffic control
system, the principle technology component of the NAS, as a high-risk area
because of the size and complexity of the program and FAA's many failures
in meeting projects' cost, schedule, and performance goals. In our latest
High-Risk Series, issued in January 2003,5 we addressed the critical need
for FAA to continue to improve its investment management practices-the
management processes the agency uses to select, control and evaluate the
benefits realized from its IT spending-because the agency would be
spending nearly $16 billion more through FY 2007, after having already
spent $35 billion since 1981. Other reports have also noted weaknesses in
5GAO-03-119.
FAA's IT investment management processes and have made a number of
recommendations to address this area.6 For instance, last year we reported
that while FAA had improved its processes, several issues remained
unresolved. We noted, for example, that the agency had not yet implemented
processes for evaluating projects after implementing them, in order to
identify lessons learned and improve the investment management process.
FAA's Current Approach to Investment Management
Process for Managing NAS Investments
FAA's process for managing an IT investment varies depending on the type
of investment-NAS systems in development through the second year of
operation (F&E), NAS systems in operation after the second year (OPS), and
non-NAS systems each follow different processes. NAS investments are
managed through a standardized process, the FAA Acquisition Management
System (AMS), and non-NAS investments are managed through a number of
different processes.
In April 1996, FAA implemented its AMS in response to legislation that
directed the agency to develop a new acquisition management system.7
Because of FAA's contention that some of its modernization problems were
caused by federal acquisition regulations, the Congress enacted
legislation in November 1995 that exempted the agency from most federal
procurement laws and regulations and directed FAA to develop and implement
a new acquisition management system that would address the unique needs of
the agency. AMS was intended to reduce the time and cost for fielding new
products and services by introducing (1) a new investment management
system that spans the entire life cycle of an acquisition, (2) a new
procurement system that provides flexibility in selecting and managing
contractors, and (3) organizational and human capital reforms that support
the new investment and procurement systems.
AMS provides high-level acquisition policy and guidance for selecting and
controlling FAA's NAS investments through all phases of the acquisition
life cycle, which is organized into a series of phases and decision points
that
6GAO, Air Traffic Control: FAA's Modernization Investment Management
Approach Could Be Strengthened, GAO/RCED/AIMD-99-88 (Washington, D.C.:
Apr. 30, 1999); GAO, Air Traffic Control: FAA's Modernization
Efforts-Past, Present, and Future, GAO-04-227T (Washington, D.C.: Oct. 30,
2003).
749 U.S.C. 40110(d).
include (1) mission analysis, (2) investment analysis, (3) solution
implementation, and (4) in-service management. To select investments, FAA
has established two processes-mission analysis and investment
analysis-which together constitute a set of policies, procedures, and
guidance that enhance the agency's ability to screen projects that are
submitted for funding. Also, through these two processes FAA is to assess
and rank each project based on its relative costs, benefits, risks, and
contribution to FAA's mission, and a senior, corporate-level
decisionmaking group selects projects for funding. After a project has
been selected, FAA officials are required to formally establish the life
cycle cost, schedule, benefits, and performance baselines that are used to
monitor the project's status throughout the remaining phases of the
acquisition management life cycle. See figure 1 for a graphic depiction of
FAA's life cycle management process.
Figure 1: FAA's Life Cycle Management Process
Source: GAO based on FAA documents.
Note: During the front end of the life cycle, research and system analysis
activities are undertaken to discover applications of new technology for
FAA's present services, explore new opportunities for service delivery,
solve problems within current operations, and define requirements.
Several groups are involved in managing FAA's NAS investments; they
perform functions from analysis of mission needs and alternative
investments through system development, implementation, operation, and,
ultimately, disposal. The roles and responsibilities of each group are
described below:
Joint Resources Council (JRC)-This board makes corporate-level resource
and investment decisions and establishes investment programs. Members
include Associate Administrators representing FAA's lines of
business, the FAA Acquisition Executive,8 the Chief Financial Officer, the
Chief Information Officer (CIO), and the Assistant Administrators for
System Safety, for Policy, Planning and International Aviation, and for
Region and Center Operations. The board is supported by the JRC
Secretariat Team, a group that facilitates the board's processes by
maintaining the meeting calendar and guidance documents, developing
records of decisions, and providing advisory and liaison support to
programs.
Systems Engineering/Operational Analysis Team-This team performs
affordability assessments for newly proposed investments and prepares
recommendations for the reprogramming of funds from lower priority
programs. It also prepares annual budget submissions for approval by the
JRC. This team is composed of representatives from each line of business
and from other functional disciplines and is chaired by the Director,
System Architecture and Investment Analysis.
Investment Analysis Team (IAT)-This team is assembled for a relatively
short period for each specific investment being considered, to conduct the
detailed analysis of alternatives that will lead to selecting and
recommending a preferred acquisition solution. The team draws experts from
the integrated product teams,9 the organizational unit with the need, the
investment analysis staff,10 and other organizations.
Corporate Mission Analysis Organization-Performs agency-level mission
analysis and coordinates service area analysis, an activity that is
conducted during mission analysis to (1) identify capability shortfalls
for or in
8May be delegated to an associate administrator.
9These teams may operate as entities or be organized into subintegrated
product teams or product teams to develop, procure, and deliver products
and services for users or customers. They are responsible for the
acquisition of new or improved capability for services and products
throughout their life cycles and for developing cost and schedule
baselines for candidate solutions during investment analysis.
10The investment analysis staff assists and oversees the work of all the
investment analysis teams, is responsible for all investment analyses, and
is responsible for developing the tools, techniques, and databases to
ensure quality performance of investment analysis on behalf of the JRC.
conjunction with service organizations,11 (2) ensure alignment with agency
strategic goals, and (3) eliminate redundant activity, duplicate benefits,
service gaps, and service overlaps. It also develops and maintains
standards and tools for conducting service area analysis, and it assists
service organizations in establishing a service area analysis capability.
In addition to identifying the roles and responsibilities of the groups
involved in the management process, AMS provides guidance on the documents
and decisions that result from each of the life cycle phases. For example,
through the mission analysis phase, FAA identifies critical needs that the
agency must meet for improving the safety, capacity, efficiency, and
effectiveness of the NAS. Approval of a mission need statement by the JRC
signifies that the agency agrees that the need is critical enough to
proceed to the next phase-investment analysis. During the investment
analysis phase, the IAT is to analyze and recommend a solution that best
satisfies FAA's performance goals and customer service needs. This team is
then to rank each proposed project based on a number of factors, including
how well it meets mission needs compared to other projects and whether it
has a favorable cost-benefit ratio. As part of the JRC selection process,
the life cycle cost, schedule, benefits, and performance baselines are
established in a formal document called the acquisition program baseline
(APB), which is designed to be used by program offices to monitor a
project's status in achieving those baselines throughout the remaining
phases of the acquisition management life cycle.
The solution implementation phase begins when the JRC approves and funds a
project, establishes its acquisition program baseline, and authorizes the
service organizations to implement and manage the project over its life
cycle. After the project has been implemented and is in operation (FAA's
inservice management phase), the service organizations monitor and assess
operational performance. Also during this phase, the project is monitored
to determine whether the current capability satisfies the demand for
services or whether another solution offers the potential for improving
11A service organization is any organization within FAA that delivers a
service, whether it is a business unit, project office, program
directorate, or integrated product team, or whether it is engaged in air
traffic services, security, regulation, certification, operations,
commercial space transportation, or airport development. These
organizations are responsible and accountable for managing service
delivery throughout the life cycle. Investment decisions are made to
support service delivery. Specifically, after the investment decision has
been made, the service organization assumes responsibility for the
investment program, implements the selected solution, and manages the
product throughout the in-service management phase of its life cycle.
safety or effectiveness or for significantly lowering costs. If the
current capability is lacking, FAA initiates a process whereby the mission
need would be revalidated and the investment analysis process begun again,
possibly leading to a new investment decision. Figure 2 provides detail on
the phases of FAA's IT investment management process and decision points.
The highlighted decision points represent those for which the JRC must
make an approval decision before a project can move forward.
Senior executives have stated that with the reorganization of the ATO in
February 2004, discussions have been held about realigning the investment
management process to make the heads of the service units responsible and
accountable for managing programs' capital investments and operating costs
from inception to retirement. In the past, the business units have been
organized to manage either capital investments or operating costs, but not
both. These discussions have not yet led to specific changes in FAA's
investment management processes and responsibilities.
Figure 2: Detailed Breakdown of FAA's Life Cycle Management Process
Decision point for which the JRC must make an approval decision before a
project can move forward. Source: GAO based on FAA documents.
aCorporate requirements organization takes the lead in planning for the
concept and requirements definition phase of the life cycle management
process.
bIn a February 2004 memorandum, FAA's Chief Operating Officer assumed the
in-service decision authority and stated that he would delegate this
responsibility to the vice presidents of the service organizations, unless
the JRC retained the in-service decision authority. If the JRC retains
this authority, it determines the in-service decision authority at the
time of the final investment decision.
Process for Managing Non-NAS While the AMS was intended to apply to all
FAA investment programs, it
Investments has not been implemented for non-NAS investments. Each of the
agency's business line and staff offices that manage non-NAS investments12
has implemented its own processes for managing these investments. Examples
of these various non-NAS investment processes include the following:
o Regarding an investment management board structure, the Financial
Services staff office has an informal board consisting of the Chief
Financial Officer, Deputy Chief Financial Officer, and heads of offices
within Financial Services. The Financial Services life cycle process guide
directs the board's operations. In the Regulation and Certification unit,
the senior management team makes investment management decisions with
input from the Chief Information management team. This unit is developing
an IT investment management processes guide, which is expected to be
completed by the end of the fiscal year.
o When selecting investments, the Human Resource Management unit uses its
established annual budget formulation process, while the Region and Center
Operations unit is moving toward a new process whereby in order to be
selected investments need to demonstrate, at a minimum, that they (1) are
compliant with FAA's architecture, (2) have a business sponsor, (3) have a
solid business case, and (4) can be funded.
o In controlling investments, Information Services has developed
processes to monitor contract expenditures, and unit managers regularly
perform financial management reviews of the programs under their purview,
but there is no structured process for oversight of projects' performance
against expectations. In the Human Resource Management unit, division
managers hold quarterly reviews to assess projects' progress in meeting
cost and schedule expectations and aligning with strategic goals.
12Non-NAS business units include Information Services (AIO), Region and
Center Operations (ARC), Regulation and Certification (AVR), Financial
Services (ABA), Research and Acquisition (ARA), Air Traffic Services
(ATS), and Human Resource Management (AHR).
Descriptions of the processes used by each of the units responsible for
managing non-NAS investments can be found in appendix II.
In January 2004, the FAA Administrator established the Information
Technology Executive Board (ITEB) to "strengthen FAA's ability to use IT
as an agencywide strategic asset" and "guide fundamental changes in the
governance of IT assets." Its charter calls for the ITEB to assume
responsibility for making investment decisions about non-NAS IT
investments. However, the ITEB has not yet implemented this aspect of its
charter. Therefore, at the current time there is no single board or
investment management process for non-NAS investments that would be
analogous to the JRC board and AMS process that are used for NAS
investments.
ITIM Maturity Framework The ITIM framework is a maturity model composed of
five progressive stages of maturity that an agency can achieve in its
investment management capabilities.13 It was developed on the basis of our
research into the IT investment management practices of leading
private-and publicsector organizations. The framework identifies critical
processes for making successful IT investments, organized into the five
increasingly mature stages. These maturity stages are cumulative; that is,
in order to attain a higher stage of maturity, the agency must have
institutionalized all of the requirements for all of the lower stages, in
addition to those for the higher stage.
The ITIM can be used both to assess the maturity of an agency's investment
management processes and as a tool for organizational improvement. The
overriding purpose of the framework is to encourage investment processes
that increase business value and mission performance, reduce risk, and
increase accountability and transparency in the decision process. We have
used the framework in several of our evaluations,14 and a number of
13GAO-04-394G.
14GAO, Information Technology: DLA Needs to Strengthen Its Investment
Management Capability, GAO-02-314 (Washington, D.C.: Mar. 15, 2002); GAO,
United States Postal Service: Opportunities to Strengthen IT Investment
Management Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002); GAO,
Information Technology: Departmental Leadership Crucial to Success of
Investment Reforms at Interior, GAO-03-1028 (Washington, D.C.: Sept. 12,
2003); and GAO, Bureau of Land Management: Plan Needed to Sustain Progress
in Establishing IT Investment Management Capabilities, GAO-03-1025
(Washington, D.C.: Sept. 12, 2003).
agencies have adopted it. These agencies have used ITIM for purposes
ranging from self-assessment to redesign of their IT investment management
processes.
ITIM's five maturity stages represent steps toward achieving stable and
mature processes for managing IT investments. Each stage builds on the
lower stages; the successful attainment of each stage leads to improvement
in the organization's ability to manage its investments. With the
exception of the first stage, each maturity stage is composed of "critical
processes" that must be implemented and institutionalized in order for the
organization to achieve that stage. These critical processes are further
broken down into key practices that describe the types of activities that
an organization should be performing to successfully implement each
critical process. An organization may be performing key practices from
more than one maturity stage at the same time. This is not unusual, but
efforts to improve investment management capabilities should focus on
becoming compliant with lower-stage practices before addressing
higher-stage practices.
Stage 2 of the ITIM framework encompasses building a sound investment
management process by establishing basic capabilities for selecting new IT
projects. It also involves developing the capability to control projects
so that they finish predictably within established cost and schedule
expectations and the capability to identify potential exposures to risk
and put in place strategies to mitigate that risk. The basic selection
processes established in Stage 2 lays the foundation for more mature
selection capabilities in Stage 3.
Stage 3 requires that an organization continually assess both proposed and
ongoing projects as parts of a complete investment portfolio-an integrated
and competing set of investment options. It focuses on establishing a
consistent, well-defined perspective on the IT investment portfolio and
maintaining mature, integrated selection (and reselection), control, and
evaluation processes, which are to be evaluated during postimplementation
reviews (PIR). This portfolio perspective allows decision makers to
consider the interaction among investments and the contributions to
organizational mission goals and strategies that could be made by
alternative portfolio selections, rather than relying exclusively on the
balance between the costs and benefits of individual investments.
Stages 4 and 5 require the use of evaluation techniques to continuously
improve both the investment portfolio and investment processes in order
to better achieve strategic outcomes. At Stage 4 maturity an organization
has the capacity to conduct IT succession activities and therefore can
plan and implement the deselection of obsolete, high-risk, or low-value IT
investments. An organization with Stage 5 maturity conducts proactive
monitoring for breakthrough information technologies that will enable it
to change and improve its business performance. Organizations implementing
Stages 2 and 3 have in place the selection, control, and evaluation
processes that are required by the Clinger-Cohen Act. Stages 4 and 5
define key attributes that are associated with the most capable
organizations.
Figure 3 shows the five maturity stages and the critical processes
associated with each.
Figure 3: The Five ITIM Stages of Maturity with Critical Processes
Source: GAO.
As defined by the model, each critical process consists of "key practices"
that must be executed to implement the critical process.
FAA Has Established Many Key Practices for Managing NAS Investments but
Lacks Oversight of Operational Systems
In order to have the capabilities to effectively manage IT investments, an
agency should, at a minimum, (1) build an investment foundation by putting
basic, project-level control and selection practices in place (Stage 2
capabilities) and (2) manage its projects as a portfolio of investments,
treating them as an integrated package of competing investment options and
pursuing those that best meet the strategic goals, objectives, and mission
of the agency; and it should also conduct PIRs to maintain mature,
integrated selection, control, and evaluation processes (Stage 3
capabilities). In addition, an agency would be well served by implementing
capabilities for improving its investment process through performance
evaluations of its portfolio and succession management of current
investments (Stage 4 capabilities). In order to develop the capabilities
to effectively manage its investments, FAA would, at minimum, need to
implement Stage 2 capabilities for both its NAS and non-NAS investments
and Stage 3 capabilities for its portfolio of investments.
FAA's investment management capabilities vary depending on whether an
investment is considered to be NAS or non-NAS. Specifically:
o For NAS investments, FAA has executed 30 of the 38 Stage 2 key
practices that are required to establish a foundation for investment
management maturity. For these investments, the agency has in place a
strong set of processes to support investment management, although the JRC
does not regularly review investments that have passed into the in-service
management phase (i.e., operational systems).
o For its non-NAS investments, the agency has not yet adequately
implemented a single management line of responsibility and the standard
processes needed to manage in a consistent manner. Although some
structured processes exist within individual business units, this lack of
consistency undermines the agency's maturity.
o In Stage 3, the lack of regular JRC oversight of operational systems
and the absence of a structured approach to managing non-NAS investments
prevent FAA from managing its investments as a portfolio that includes all
major NAS and non-NAS investments. In addition, the agency is not
conducting PIRs on its major investments.
o FAA has not executed any of the Stage 4 key practices for managing the
succession of its information systems, although the agency has begun to
address this weakness by defining procedures for retiring investments in
the AMS.
When FAA implements all of the key practices associated with building the
investment foundation and managing its investments as a portfolio, the
agency will have greater assurance that it has selected the mix of
investments that best supports its strategic goals and that it will be
able to manage the investments to successful completion.
FAA Has Established Much of the Foundation Needed to Manage Its NAS
Investments
At the ITIM Stage 2 level of maturity, an organization has attained
repeatable, successful IT project-level investment control processes and
basic selection processes. Through these processes, the organization can
identify expectation gaps early and take appropriate steps to address
them. According to ITIM, critical processes at Stage 2 include (1)
defining IT investment board15 operations, (2) identifying the business
needs for each IT investment, (3) developing a basic process for selecting
new IT proposals and reselecting ongoing investments, (4) developing
projectlevel investment control processes, and (5) collecting information
about existing investments. Table 2 describes the purpose of each of the
Stage 2 critical processes.
Table 2: Stage 2 Critical Processes-Building the Investment Foundation Critical
process Purpose
Instituting the investment board To define and establish an appropriate IT
investment management structure and the processes for selecting,
controlling, and evaluating IT investments.
Meeting business needs To ensure that IT projects and systems support the
organization's business needs and meets users' needs.
Selecting an investment To ensure that a well-defined and disciplined
process is used to select new IT proposals and reselect ongoing
investments.
Providing investment oversight To review the progress of IT projects and
systems, using pre-defined criteria and checkpoints, in meeting cost,
schedule, risk, and benefit expectations and to take corrective action
when these expectations are not being met.
Capturing investment information To make available to decision makers
information to evaluate the impacts and opportunities created by proposed
(or continuing) IT investments.
Source: GAO.
15An IT investment board is a decision-making body, made up of senior
program, financial, and information managers, that is responsible for
making decisions about IT projects and systems based on comparisons and
trade-offs among competing projects, with an emphasis on meeting mission
goals.
To its credit, FAA has put in place about 80 percent of the key practices
associated with managing its NAS investments through the Stage 2 critical
processes. The agency has satisfied all of the key practices associated
with capturing investment information and most of those associated with
instituting the investment board, meeting business needs, selecting an
investment, and providing investment oversight. Most of the weaknesses in
these critical processes relate to NAS investments in the in-service
management phase. Table 3 summarizes the status of FAA's critical
processes for Stage 2, showing how many key practices FAA has executed in
managing its NAS investments.
Table 3: Summary of Results for Stage 2 Critical Processes and Key
Practices for NAS Investments
Critical process
Key practices executed
Total required by critical process
Percentage of key practices executed
Instituting the investment
board 7 8
Meeting business needs 6 7
Selecting an investment 7 10
Providing investment
oversight 4 7
Capturing investment
information 6 6
Total 30 38
Source: GAO.
FAA Has Established an IT Management Structure to ManageIts NAS Investments
The establishment of decision-making bodies or boards is a key component
of the IT investment management process. At the Stage 2 level of maturity,
organizations define one or more boards, provide resources to support
their operations, and appoint members who have expertise in both
operational and technical aspects of proposed investments. The boards
operate according to a written IT investment process guide that is
tailored to the organization's unique characteristics, thus ensuring that
consistent and effective management practices are implemented across the
organization. Once board members are selected, the organization ensures
that they are knowledgeable about policies and procedures for managing
investments. Organizations at the Stage 2 level of maturity also take
steps to ensure that executives and line managers support and carry out
the
decisions of the IT investment board. According to ITIM, an IT investment
management process guide should be a key authoritative document that the
organization uses to initiate and manage IT investment processes and
should provide a comprehensive foundation for the policies and procedures
that are developed for all of the other related processes. (The complete
list of key practices is provided in table 4.)
FAA has executed 7 of the 8 key practices for this critical process. For
example, in 1996, Congress directed FAA to develop a new acquisition
management system as part of a broad mandate for acquisition reform at the
agency.16 In response, FAA implemented AMS in April 1996. AMS establishes
policy and guidance for all aspects of the agency's acquisition life cycle
and documents the investment management process used for NAS investments.
The agency established the JRC as its corporate-level investment board for
the NAS investments. The JRC makes select and control decisions, including
corporate decisions on mission needs, acquisition investments, and
acquisition program baseline changes; it also reviews and recommends
approval of the agency's F&E budget submission.
The board is adequately resourced to support its operations. The JRC
Secretariat Team supports the board in such ways as developing and
updating guidance, scheduling meetings, and preparing and executing the
JRC readiness process. In addition, the Mission Analysis Steering Group17
is responsible for assisting the board in prioritizing mission needs,
while the Systems Engineering/Operational Analysis Team is to assist in
addressing budget issues among investments. The JRC consists of senior
officials from both business and IT areas, including the Chief Information
Officer and the associate administrators representing FAA lines of
business. These members are to exhibit the core competencies required by
FAA in selecting executives and in assessing executive training needs. In
addition, the agency offers a 3-day AMS overview course for all employees,
including JRC members. Although the board as an entity does not oversee
the development and maintenance of AMS, it is involved through FAA's
Acquisition System Advisory Group, which evaluates all proposed changes to
AMS. To ensure that the board's decisions are carried out, an acquisition
program baseline document is approved at the JRC final investment
1649 U.S.C. 40110(d).
17An advisory group, composed of representatives from each line of
business, that establishes guidelines for conducting mission analysis and
developing mission need statements as well as resolving agencywide mission
analysis issues.
decision point; this document identifies the capabilities, benefits,
costs, and schedule for the approved investment, which are monitored by
FAA through its variance reporting process.
Despite these strengths, FAA has not yet clearly defined the relationship
between the JRC and the newly formed ITEB. Although the ITEB was
established by the Administrator to function as the central authority
responsible for assuring that FAA IT investments are based on sound
business practices, FAA has not yet clearly delineated the specific roles
the ITEB is to play and the relationship it will have with the JRC. This
task has been assigned to the ITEB as a longer-range initiative.
Table 4 shows the rating for each key practice required to implement the
critical process for instituting the investment board at the Stage 2 level
of maturity. Each of the "Executed" ratings shown below represents
instances where, based on the evidence provided by FAA officials, we
concluded that the specific key practices were executed by the
organization.
Table 4: Instituting the Investment Board Type of practice Key Practice Rating
Summary of evidence
Organizational commitments
1. An enterprisewide IT investment board composed of senior executives
from IT and business units is responsible for defining and implementing
the organization's IT investment governance process.
Executed The JRC, FAA's corporate-level investment board for the NAS
investments, is responsible for defining and implementing the agency's IT
investment governance process. It consists of the agency's most senior
executives, including the CIO, Chief Financial Officer, FAA's Acquisition
Executive, and Associate Administrators from its lines of business.
2. The organization Executed FAA's AMS sets forth acquisition policy
and processes for the JRC.
has a documented Also, the board has established its own
guidance to implement
IT investment AMS core JRC policy.
process directing
each investment
board's operations.
(Continued From Previous Page)
Type of practice Key Practice Rating Summary of evidence
Prerequisites 1. Adequate resources, including people, funding, and tools,
are provided for supporting the operations of each IT investment board.
Executed Adequate resources are provided to support the board's
operations. The JRC Secretariat Team provides such operations support as
developing and updating guidance, scheduling meetings, and preparing and
distributing records of decisions. Two other groups support the JRC
decision-making process. The Mission Analysis Steering Group assists the
board in ranking mission needs, while the Systems Engineering/Operational
Analysis Team provides assistance by performing affordability assessments
for the JRC when it is considering alternatives during investment
analysis.
2. The board members understand the organization's IT investment
management policies and procedures and the tools and techniques used in
the board's decision-making process.
Executed JRC members are senior managers representing all agency lines of
business. They include the CIO, Chief Financial Officer, and Associate
Administrators representing FAA lines of business such as Air Traffic
Services. Core executive competencies based on FAA's Executive Success
Profile are used in selecting executives and in assessing executive
training needs. A 3-day AMS overview course is also available.
3. Each board's span Not executed The JRC is FAA's corporate-level
investment board for making
of authority and decisions related to NAS investments.
In January 2004, the ITEB
responsibility is was established to oversee the
governance of the agency's IT
defined to minimize assets. However, the ITEB has yet to
take significant action on the
charge in its charter to clearly
overlaps or gaps delineate the roles it is to play and
its
among the boards. relationship with the JRC.
Activities 1. The enterprisewide investment board has oversight
responsibilities for the development and maintenance of the organization's
documented IT investment process.
Executed Although the JRC does not directly oversee the development and
maintenance of the FAA's documented investment process, it is involved in
this process through FAA's Acquisition System Advisory Group, which is a
corporate crossfunctional body that evaluates all proposed changes to AMS.
Membership consists of representatives from each line of business as well
as the JRC Secretariat Team. Policy changes that are endorsed by the Group
are presented, via the FAA Acquisition Executive, who is on the JRC, to
the Administrator for approval.
2. Each investment Executed The JRC is operating in accordance with
its assigned authority and
board operates in responsibility as FAA's corporate-level
investment board for making
accordance with its decisions related to NAS investments. The
charter for the ITEB
specifically indicates the ITEB's
assigned authority responsibilities, including for
making decisions for non-NAS IT
and responsibility. acquisitions.
(Continued From Previous Page)
Type of practice Key Practice Rating Summary of evidence
3. The organization Executed FAA has controls for ensuring that the
JRC's investment decisions
has established are carried out as approved. At the JRC
final investment decision
point, an acquisition program baseline
management document is finalized and
controls for approved, which represents the mutual
agreement between the
ensuring that JRC, the provider organization, and the
user organization
investment boards' concerning the expected capability,
benefits, costs, and schedule
decisions are for the investment program. It also
establishes performance
carried out. metrics for assessing the program's
success.
Source: GAO.
FAA Has a Process for Ensuring That its Investments Support Business Needs
and Meet Users' Needs
Defining business needs for each IT project helps to ensure that projects
and systems support the organization's business needs and meet users'
needs. This critical process ensures that a link exists between the
organization's business objectives and its IT management strategy.
According to ITIM, effectively meeting business needs requires, among
other things, (1) documenting business needs with stated goals and
objectives, (2) identifying specific users and other beneficiaries of IT
projects and systems, (3) providing adequate resources to ensure that
projects and systems support the organization's business needs and meet
users' needs, and (4) periodically evaluating the alignment of IT projects
and systems with the organization's strategic goals and objectives. (The
complete list of key practices is provided in table 5.)
FAA has in place 6 of the 7 key practices for meeting business needs. The
agency's AMS and mission analysis guidance calls for business needs for
both proposed and ongoing IT projects and systems to be identified in the
mission need statement developed during the mission analysis phase. FAA
also has detailed procedures for developing this document that call for
identifying business needs. Resources for ensuring that IT projects and
systems support the organization's business needs and meet users' needs
include service organizations, the Corporate Mission Analysis
Organization, the Mission Analysis Steering Group, and detailed procedures
and associated templates for developing mission need statements. FAA's
specific business mission, with stated goals and objectives, is defined in
the Federal Aviation Administration Flight Plan for fiscal years 2004
through 2008.
Further, FAA defines and documents business needs for both proposed and
ongoing IT projects and identifies users and other beneficiaries during
its mission analysis activities. In addition, the AMS policy calls for
users to participate in project management throughout the FAA life cycle
management process. For the three projects we reviewed,18 we verified that
business needs and specific users and other beneficiaries were identified
and documented in mission needs statements as well as in other documents.
In addition, users are involved in project management throughout the life
cycle of the projects. For example, according to project officials, En
Route Communications Gateway (ECG) users participate in project meetings,
weekly integrated product team status meetings, and monthly En Route
domain national deployment teleconferences. FAA Telecommunications
Infrastructure's (FTI) end users are heavily involved in the "operational
test" period, which determines whether the equipment can be safely
implemented in NAS. VSCS Control Subsystem Upgrade19 users are involved in
the project's life cycle via a Web site through which they review and
comment on project documentation.
Despite these strengths, the JRC has no process for evaluating the
organizational alignment of NAS systems through most of their in-service
management phase (and non-NAS investments, which are described separately
in this report). While the JRC does evaluate the alignment of projects and
systems with organizational goals throughout the systems' development and
2 years into their operations as part of the annual budget formulation
process, it does not use any consistent process to review projects and
systems after that point in their life cycles. For NAS systems in the
in-service management phase, these activities are carried out within the
business unit that owns the system, but the JRC does not regularly oversee
these processes and may go for several years without reviewing a system's
alignment with organizational goals. In-service NAS systems only return to
the JRC if they are judged to require additional funds for correction.
Until FAA establishes a process for periodic evaluation of systems
throughout the in-service management phase and takes corrective actions
when misalignment occurs, the agency will not be able to ensure that these
projects, totaling about $1.3 billion per year, are still continuing to
maintain alignment with the FAA's strategic plans and its business goals
and objectives.
18We reviewed the FAA Telecommunications Infrastructure, En Route
Communications Gateway, and Voice Switching and Control System (VSCS)
Control Subsystem Upgrade (VCSU) projects. The projects are described in
appendix I.
19VCSU is a subcomponent of the VSCS project. We decided to review VCSU
because its investment management process was carried out using FAA's AMS,
whereas the VSCS project was funded before the AMS became part of the
FAA's investment management process.
Table 5 shows the rating for each key practice required to implement the
critical process for meeting business needs at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 5: Meeting Business Needs Type of practice Key practice Rating Summary of
evidence
Organizational commitments
1. The organization has documented policies and procedures for ensuring
IT projects or systems that support the organization's ongoing and future
business needs.
Executed AMS and mission analysis guidance contain documented policies and
procedures for identifying the IT projects or systems that support the
organization's ongoing and future business needs.
Prerequisites 2. The organization has a Executed The Federal Aviation
Administration Flight Plan (Strategic
documented business mission with stated goals and objectives.
Plan) for fiscal years 2004 through 2008 defines the agency's mission
goals and objectives.
3. Adequate resources, Executed FAA has adequate resources for
ensuring that its IT
including people, funding, projects and systems support the
organization's business
and tools, are provided needs and meet users' needs. They
include service
for ensuring that IT organizations, the Mission Analysis
Steering Group, and
projects and systems the Corporate Mission Analysis
Organization. FAA also
support the has detailed procedures and
associated templates for
organization's business developing mission need statements.
needs and meet users'
needs.
The organization AMS policy calls for
Activities 1. defines Executed business needs for both
proposed
and documents business and ongoing IT projects and
systems to be specified in
needs for both proposed the mission need statement.
We verified that business
and ongoing IT projects needs were defined and
documented in mission need
statements for the three
and systems. projects we reviewed.
2. The organization Executed FAA policy and procedures call for
specific users and
identifies specific users other beneficiaries of IT projects
and systems to be
and other beneficiaries of identified. We verified that
specific users and other
IT projects and systems. beneficiaries were identified for
the three projects we
reviewed.
3. Users participate in Executed FAA policies and procedures call
for users to participate
project management in project management throughout an
IT project's or
throughout an IT project's system's life cycle. We verified
that users participated in
or system's life cycle. project management throughout the
life cycle of the three
projects we reviewed.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
4. The investment board Not executed The JRC evaluates the alignment
of systems through
periodically evaluates development and 2 years into
the operations with the
alignment of its IT organization's strategic goals
and objectives-through
projects and systems the annual budget formulation
process-and takes
with the organization's corrective actions when
misalignment occurs. However,
strategic goals and there is no process for the JRC
to periodically evaluate
objectives and takes the alignment of investments
later in their life cycles.
corrective actions when
misalignment occurs.
Source: GAO.
FAA Has a Disciplined Process for Selecting New IT Proposals but Lacks a
Similar Process for Reselecting Ongoing Investments
Selecting new IT proposals and reselecting ongoing investments requires a
well-defined and disciplined process to provide the agency's investment
board, business units, and developers with a common understanding of the
process and the cost, benefit, schedule, and risk criteria that will be
used both to select new projects and to reselect ongoing projects for
continued funding. According to ITIM, this critical process requires,
among other things, (1) making funding decisions for new proposals
according to an established process; (2) providing adequate resources for
investment selection activities; (3) using a defined selection process to
select new investments and reselect ongoing investments; (4) establishing
criteria for analyzing, prioritizing, and selecting new IT investments and
for reselecting ongoing investments; and (5) creating a process for
ensuring that the criteria change as organizational objectives change.
(The complete list of key practices is provided in table 6.)
FAA has executed 7 of the 10 key practices associated with selecting an
investment. For example, the AMS establishes two processes-mission
analysis and investment analysis-that together constitute a set of
policies and procedures, as well as guidance that is designed to enhance
the agency's ability to select investments. In addition, FAA has policies
and procedures for its annual F&E budget formulation process to reselect
ongoing IT projects. Also, FAA's AMS sets forth policies and procedures
for reselecting ongoing IT investments by identifying their capability
shortfalls and addressing them as new investments.
The AMS also integrates funding with the process of selecting an
investment by requiring the Systems Engineering/Operational Analysis Team
to perform affordability assessments for new proposed investment programs;
it may recommend funding reallocations from lower priority programs when
an alternative solution cannot be funded within FAA
planning and budgeting baselines. This team also supports the JRC to
ensure that the executives' funding decisions are aligned with selection
decisions during the investment analysis activities.
Resources for proposal selection activities include the program director,
the Integrated Product Team, and the Investment Analysis Team, as well as
detailed procedures and a template that have been defined for developing
investment analysis reports. The investment analysis reports identify the
evaluation criteria used, the alternatives analyzed, and the ranking of
each alternative so that the JRC can select the best overall solution
identified in the mission need statement. The criteria that were
established during the initial investment analysis phase are used by the
Investment Analysis Team to rank each proposed project on the basis of how
well it meets the agency's mission needs compared with other projects.
FAA uses the processes defined in the AMS for selecting new IT
investments. In addition, it uses two processes to reselect ongoing IT
investments. Specifically, the FAA uses its annual budget formulation
process for projects in development or in the first 2 years of operations.
It also uses the AMS process when a system's capability shortfall is
identified, and it treats the correction of the shortfall as a new
investment. The managers of the three projects we reviewed confirmed that
their projects were selected using the AMS process. One project's
officials stated that this included market, alternatives, investment, and
affordability analyses. The program managers also stated that the annual
F&E budget formulation process is used to reselect their projects. These
project officials also noted that if a project is scheduled for a hardware
replacement, a reselection is done. The AMS process is followed to explore
new alternatives and make sure the replacement is in the best interest of
the government.
Despite these strengths, FAA has not developed similarly strong processes
for NAS investments more than 2 years into their operations-those NAS
systems that are in the in-service management phase. For example, while
FAA's F&E budget formulation process establishes criteria for analyzing,
prioritizing, and reselecting IT investments for systems in development or
up until 2 years into operations, neither of the two processes used to
reselect IT investments has established criteria for investments beyond 2
years into operations. In addition, while FAA uses its annual budget
formulation process to reselect projects that are part of the F&E budget,
the agency does not have an analogous reselection process as part of its
operations budget formulation. Until FAA establishes consistent criteria
for reselecting all of its IT investments, it will not be adequately
assured that it
is consistently and objectively continuing to fund ongoing projects that
still meet the needs and priorities of the agency in a cost-effective and
riskinsured manner.
Table 6 shows the rating for each key practice required to implement the
critical process for selecting an investment at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 6: Selecting an Investment
Type of practice Key practice Rating Summary of evidence
1. The organization Executed FAA's AMS policy, mission
Organizational has analysis, and investment
commitments documented policies analysis guidance has
and documented policies and
procedures for procedures for selecting new
selecting IT proposals.
new IT proposals.
2. The organization has Executed FAA has documented policies and
procedures for its
documented policies and annual F&E budget formulation
process, which is used to
procedures for reselecting reselect ongoing IT projects. In
addition, FAA's AMS
ongoing IT investments. policy has documented policies and
procedures for
reselecting ongoing IT investments
throughout the FAA's
acquisition life cycle.
3. The organization has Executed FAA's AMS and investment analysis
guidance have
documented policies and documented policies and procedures
for integrating
procedures for integrating funding with the process of
selecting an investment.
funding with the process of
selecting an investment.
Adequate resources are
Prerequisites 1. Adequate resources, Executed provided for identifying
and
including people, selecting IT projects and
funding, systems. They include the
and tools, are program director,
provided for Integrated Product Teams,
and the
identifying and Investment Analysis Team.
selecting IT FAA also has detailed
projects and systems. procedures and associated
templates for developing
investment analysis
reports.
2. Criteria for analyzing, Executed The Investment Analysis Team has
established criteria
prioritizing, and for analyzing, prioritizing, and
selecting selecting new IT
new IT investment investment opportunities. The
investment analysis
opportunities have been report, which is submitted to the
JRC, identifies the
established. evaluation criteria, the alternatives
analyzed, and the
ranking for each alternative.
3. Criteria for analyzing, Not executed While FAA's F&E budget
formulation process has prioritizing, and established criteria for
analyzing, prioritizing, and reselectinga IT investment reselecting IT
investments that are part of that budget, opportunities have been neither
of the two processes used to reselect IT established. investment
opportunities has established criteria for
investments beyond 2 years into operations.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
4. A mechanism exists to Not executed While FAA ensures that the criteria
continue to reflect ensure that the criteria organizational objectives for
selecting new IT continue to reflect investments, there are no consistent
criteria used by the organizational objectives. JRC to reselect
investments more than 2 years into
operations.
Activities 1. The organization uses Executed FAA uses the policies and
its procedures defined in the
defined selection AMS, mission analysis, and
process, investment analysis
guidance to select new IT
including predefined investments. We verified
that
selection criteria, to the three projects we
select reviewed were selected using
the
mission analysis and
new IT investments. investment analysis
activities
defined in the AMS, mission
analysis and the investment
analysis guidance.
2. The organization uses the Not executed FAA has a process defined in
the AMS to reselect defined selection process, ongoing IT investments. It
also uses its annual budget including predefined formulation process to
reselect projects that are part of selection criteria, to the F&E budget.
According to the project managers of reselecta ongoing IT the three
projects we reviewed, this budget process is investments. used to reselect
their projects. However, FAA does not
consistently use these defined processes to reselect IT
investments more than 2 years into operations.
3. Executives' funding Executed The Systems Engineering/Operational
Analysis Team,
decisions are aligned which is composed of representatives
with from FAA's service
selection decisions. organizations, supports the JRC in
making funding
decisions that are aligned with
selection decisions as
part of FAA's investment analysis
activities.
Source: GAO.
aAccording to the GAO ITIM framework, reselecting is the periodic
reconsideration of an investment's continuing value to the organization
and the decision to continue funding. It is a recurring process that
continues for as long as a project is receiving funding.
FAA Does Not Have a Process for Effectively Overseeing Investments in All
Phases of Their Life Cycles
An organization should provide effective oversight for its IT projects
throughout all phases of their life cycles. Its investment board should
maintain adequate oversight and observe each project's performance and
progress toward predefined cost and schedule expectations as well as each
project's anticipated benefits and risk exposure. The investment board
should also employ early warning systems that enable it to take corrective
action at the first sign of cost, schedule, or performance slippages. This
board has ultimate responsibility for the activities within this critical
process. According to ITIM, effective project oversight requires, among
other things, (1) having written policies and procedures for management
oversight; (2) developing and maintaining an approved management plan for
each IT project; (3) making up-to-date cost and schedule data for each
project available to the oversight boards; (4) having regular reviews by
each investment board of each project's performance against stated
expectations; and (5) ensuring that corrective actions for each
underperforming project are documented, agreed to, implemented, and
tracked until the desired outcome is achieved. (The complete list of key
practices is provided in table 7.)
FAA has in place 4 of the 7 key practices associated with effective
project oversight. The agency has developed written policies and
procedures for management oversight of its investments. These include (1)
AMS; (2) the integrated program plan, which is the detailed planning
document for all aspects of a program's implementation, including program
control; and (3) the Integrated Baseline Establishment and Management
Process document for reporting variances from the performance expectations
approved by the JRC in the acquisition program baseline.
We verified that cost, schedule, benefit, and risk expectations were
documented in the acquisition program baseline and that the integrated
program plan contained details for project execution for En Route
Communications Gateway and FAA Telecommunications Infrastructure. For the
VSCS Control Subsystem Upgrade, performance expectations and details on
project execution were both captured in the integrated program plan.20 In
addition, the JRC Secretariat Team maintains a tracking system for action
items that are assigned during a project's acquisition reviews, including
the action to be taken, the responsible FAA organization, and whether the
underlying problem has been resolved.
FAA has not established processes that bring investments before the JRC
for oversight on a regular basis. There is a process for reporting
variances from the performance expectations that were approved by the JRC
in the investment's acquisition program baseline. However, although this
process is carried out as part of the F&E budget formulation for IT
investments in development or less than 2 years into operations, it is not
being carried out for investments that are part of the operations budget.
Investments that are meeting performance expectations may not return to
the JRC for several years. FAA also conducts acquisition reviews as a
means for program offices to report to agency executives on the status of
investments compared to program baselines. However, since program offices
may select which investments they wish to bring forward for review, many
investments may never come forward. Until FAA develops (1) procedures for
reporting on an investment throughout its entire acquisition life cycle
20According to FAA, no acquisition program baseline was prepared for VCSU.
and (2) mechanisms for ensuring that all investments are reviewed
regularly, the agency is placing itself at risk that underperforming
investments will not be reported to the JRC in order for it to take
appropriate actions.
Table 7 shows the rating for each key practice that is required to
implement the critical process for project oversight at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
Table 7: Providing Investment Oversight Type of practice Key practice Rating
Summary of evidence
FAA has developed written
Organizational 1. The organization has Executed policies and procedures
for
commitment documented policies management oversight of IT
and projects and systems.
These include AMS, the
procedures for integrated program plan,
and
management oversight the Integrated Baseline
of Establishment and
Management
IT projects and Process document for
systems. reporting variances from
the
performance expectations
approved in the
acquisition
program baseline for an
investment program.
Prerequisites 1. Adequate resources, Executed FAA has adequate resources
for providing IT project
including people, oversight. The agency has
funding, staff for compiling monthly
and tools, are variance reports submitted
provided for by the investment program
IT project areas, preparing quarterly
oversight. baseline variance reports
for
the JRC, and preparing
semi-annual baseline
variance
report for the FAA
Administrator. An automated
system
is used to facilitate the
maintenance of information
for
these reports.
2. IT projects and systems, including those in steady state (operations
and maintenance), maintain approved project management plans that include
expected cost and schedule milestones and measurable benefit and risk
expectations.
Executed AMS policy calls for an acquisition program baseline (APB)
document and an integrated program plan to be available at the JRC final
investment decision point. The APB document serves as the AMS
cost/schedule/technical performance/benefits/risks control document. The
integrated program plan specifies how the APB baselines will be controlled
and details the management, contracting, and technical actions and
activities to be performed in executing the acquisition. We verified that
cost, schedule, benefit, risk, and performance expectations were
documented in the APBs for ECG and FTI. For VCSU, these expectations were
documented in an integrated program plan.
Data on actual FAA has established a
Activities 1. performance Not executed process for reporting
variances to
(including cost, the JRC from the
schedule, performance expectations
that have
been approved by the JRC
benefit, and risk in the APB for an
investment.
performance) are This process is carried
provided out for IT investments
that are
part of the F&E budget,
to the appropriate IT but it is not being
carried out for
investments that are
investment board. managed as part of the
operations
budget.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
2. Using verified data, each Not executed FAA does not have a process that
provides an investment board regularly opportunity for the JRC to
regularly review investment reviews the performance of performance. It has
a process for conducting acquisition IT projects and systems reviews where
program offices provide status against stated information to agency
executives on the progress of expectations. investments against their
acquisition program baselines. However, the individual program offices
choose which investments they want to discuss at these reviews. Also,
although the process for reporting variances from the performance
expectations approved by the JRC in the acquisition program baseline is
carried out for IT investments that are part of the F&E budget, this
process is not being carried out for investment programs that are part of
the operations budget, and it only results in investments with variances
to be reviewed.
3. For each underperforming Not executed During acquisition reviews,
action items are identified for IT project or system, investment programs
discussed, an organization appropriate actions are assigned responsibility
to carry them out, and the items taken to correct or are tracked until the
appropriate action is taken, at which terminate the project or time they
are closed out. Similarly, variance reports are system in accordance with
prepared quarterly for the JRC identifying investments defined criteria
and the with a 10 percent or greater variance from the documented policies
and established acquisition program baseline. An investment procedures for
program is to remain on the quarterly variance report management
oversight. until successful corrective action is taken. However, FAA has
no mechanism that provides assurance that every program has an acquisition
review regularly, since it is left to the individual program offices to
decide which programs they want discussed at the reviews. Also, although
variance reports are prepared for IT investments that are part of the F&E
budget, reports are not prepared for investments that are part of the
operations budget.
4. The investment board Executed The JRC's Secretariat Team
maintains a tracking
regularly tracks the system for action items assigned
during a project's
implementation of acquisition reviews. This system
identifies the action to
corrective actions for each be taken, what FAA organization is
to perform it, and
underperforming project whether it is open or closed.
until the actions are
completed.
Source: GAO.
FAA Has a Structured Process for Capturing Investment Information and Using
It to Support Investment Management
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format. During this critical process an
organization identifies its IT assets and creates a comprehensive
repository of investment information. This repository provides information
to investment decision makers to help them evaluate the impacts and
opportunities that would be created by proposed or continuing investments.
It can provide insights and trends about major IT cost and management
drivers. The repository can take many forms and does not have to be
centrally located, but the collection method should identify each IT
investment and its associated components. This critical process may be
satisfied by the information contained in the organization's current
enterprise architecture, augmented by additional information-such as
financial information and information on risk and benefits-that the
investment board may require to ensure that informed decisions are being
made. According to ITIM, effectively managing this repository requires,
among other things, (1) developing written policies and procedures for
identifying and collecting the information, (2) assigning responsibility
for ensuring that the information being collected meets the needs of the
investment management process, (3) identifying IT projects and systems and
collecting relevant information to support decisions about them, and (4)
making the information easily accessible to decision makers and others.
(The complete list of key practices is provided in table 8.)
FAA's AMS guidance identifies specific information that is needed in the
investment management process, including information for its investment
analysis phase. FAA maintains a number of repositories of relevant
information, including its Simplified Program Information Reporting &
Evaluation database, which reports variances in cost, schedule,
performance, or benefits from an investment's approved acquisition program
baseline. The information that is collected is made available to the JRC
in several documents, including program plans and the acquisition program
baseline document. The JRC Secretariat Team ensures that the investment
board has all the relevant information it needs for its decisionmaking
process.
Table 8 shows the rating for each key practice required to implement the
critical process for capturing investment information at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
Table 8: Capturing Investment Information
Type of practice Key practice Rating Summary of evidence
FAA has developed
1. The organization Executed policies and procedures
Organizational has for
documented policies identifying and
commitments and collecting information to
support the
procedures for investment management
identifying and process. For example, AMS
collecting information guidance indicates what
about IT information is needed for
the
projects and systems investment analysis phase
to support of FAA's investment
the investment management process.
management
process.
2. An official is assigned Executed AMS guidance specifies which
officials are responsible responsibility for ensuring that for approving
the completion of reports containing the information collected during
information prepared for the investment management project and systems
identification process. The JRC Secretariat Team ensures through meets the
needs of the its readiness process that all of the necessary investment
management information is available to the JRC for its decision process.
making.
Prerequisite 1. Adequate resources, Executed FAA has adequate resources
including for meeting this key
people, funding, and practice. Several teams,
tools, are including a mission analysis
provided for team and an investment
identifying IT analysis team, collect the
projects and systems relevant investment
and information needed by the
JRC to
collecting relevant make its decisions on which
investment investments to approve.
information about The JRC Secretariat Team
them. ensures that the JRC has
all the relevant information
for its decision making.
Activities 1. The organization's IT Executed FAA maintains data relevant
projects to the investment
and systems are management process in
identified, and several sources, including
its
specific information is financial management
collected to system, its Simplified
Program
support decisions about Information Reporting &
them. Evaluation tool, and its
Capital
Investment Plan. Also, AMS
guidance identifies
information to be collected
for the investment
management
process-including for the
investment
analysis phase-to aid the
JRC in its final investment
decisions.
2. The information that has been Executed Information collected for the
JRC decision-making collected is easily accessible and process is compiled
in documents such as detailed understandable to decision program plans,
acquisition program baselines, and makers and others. investment analysis
reports. The JRC Secretariat Team ensures that the JRC has all the
relevant information for its decision making through its readiness
process.
3. The information Executed The JRC Secretariat Team, through its
repository is JRC readiness
used by investment process, collects information for the
decision JRC to use in its
makers and others to decision-making process. The JRC also
support receives an
investment management. investment analysis report that
contains all of the
information that has been gathered
during investment
analysis activities.
Source: GAO.
FAA Does Not Have Structured Processes to Manage Its Non-NAS Investments
FAA does not have a single set of processes for making consistent basic
selection and control decisions for its non-NAS investments (Stage 2
capabilities). As previously discussed in the background section of this
report, several business units within FAA make decisions about non-NAS
investments. We reviewed the investment management processes of seven of
these units-Information Services, Region and Center Operations, Regulation
and Certification, Financial Services, Research and Acquisition, Air
Traffic Services, and Human Resource Management. Appendix II describes the
investment management processes we found in these units. The extent to
which these processes comply with the ITIM framework for Stage 2 varies
considerably by business unit, and FAA currently does not specify non-NAS
investment management processes in a coordinated manner. Since the ITIM
framework calls for a consistent investment management process, we
assessed FAA's non-NAS investment management capability at an aggregate
level. That is, we assessed FAA's capability to manage its non-NAS
investments, not the capability of each individual business unit. Even
though individual business units may have some of these processes in
place, FAA as a whole has not yet defined
o
an investment management structure that allows the agency to consistently
manage its non-NAS investments,
o
a uniform process for ensuring that non-NAS investments are linked to
business needs and meet users' needs,
o
a process for selecting new IT proposals and reselecting ongoing
investments,
o
a single process for reviewing the progress of investments and taking
corrective action when performance expectations are not being met, or
o
a comprehensive inventory of project and system information to support
investment decisions.
According to FAA officials, the agency has not defined a coherent
investment management structure and a set of processes for non-NAS
investments in the past because many of these investments have not had the
agencywide impact of the NAS investments. However, because there is now
recognition that a disciplined approach to managing non-NAS investments
could help control FAA's IT assets and costs in general, efforts are
currently under way to address this weakness. As previously discussed, an
IT Executive Board (ITEB) has been chartered with responsibility for,
among other things, making decisions about non-NAS IT investments, but it
has not yet taken action on developing a standard process. Until FAA fully
establishes the consistent practices it needs to make basic project
selection and control decisions, executives will be hampered in their
ability to effectively manage non-NAS investments and ultimately to find
the opportunities to achieve the cost savings they are seeking.
FAA Lacks Key Capabilities Needed to Manage All IT Investments as a
Portfolio and Does Not Conduct Postimplementation Reviews
During Stage 3, the investment board enhances the IT investment management
process by developing a complete investment portfolio and carrying out
PIRs. An IT investment portfolio is an integrated, agencywide collection
of investments that are assessed and managed collectively on the basis of
common criteria. Managing investments within the context of such a
portfolio is a conscious, continuous, and proactive approach to expending
limited resources on an organization's competing initiatives in light of
the relative benefits expected from these investments. Taking an
agencywide perspective enables an organization to consider its investments
comprehensively, so that collectively the investments optimally address
the organization's missions, strategic goals, and objectives. Managing IT
investments with a portfolio approach also allows an organization to
determine priorities and make decisions about which projects to fund, and
continue to fund, based on analyses of the relative organizational value
and risks of all projects, including projects that are proposed, under
development, and in operation. For an organization to reap the full
benefits of the portfolio process, it should collect all of its
investments into an enterprise-level portfolio that is overseen by its
senior investment board. Although investments may initially be selected
into subordinate portfolios-based on, for example, lines of business or
life cycle stages-and managed by subordinate investment boards, they
should ultimately be aggregated into this enterprise-level portfolio.
The purpose of a PIR is to evaluate an investment after its development
has been completed (i.e., after its transition from the implementation
phase to the in-service management phase) in order to validate actual
investment results. This review is conducted to (1) examine differences
between estimated and actual investment costs and benefits and their
possible ramifications for unplanned funding needs in the future and (2)
extract "lessons learned" about the investment selection and control
processes that can be used as the basis for management improvements.
Similarly, PIRs should be conducted for investment projects that were
terminated before completion, to help to readily identify potential
management and process improvements.
According to ITIM, critical processes performed by Stage 3 organizations
include (1) defining the portfolio criteria, (2) creating the portfolio,
(3) evaluating the portfolio, and (4) conducting PIRs. Table 9 shows the
purpose of each critical process in Stage 3.
Table 9: Stage 3 Critical Processes-Developing a Complete Investment Portfolio
Critical process Purpose
Defining the portfolio criteria To ensure that the organization develops
and maintains IT portfolio selection criteria that support its mission,
organizational strategies, and business priorities.
Creating the portfolio To ensure that IT investments are analyzed
according to the organization's portfolio selection criteria and that an
optimal IT investment portfolio with manageable risks and returns is
selected and funded.
Evaluating the portfolio To review the performance of the organization's
investment portfolio(s) at agreed-upon intervals and to adjust the
allocation of resources among investments as necessary.
Conducting postimplementation reviews To compare the results of recently
implemented investments with the expectations that were set for them and
to develop a set of lessons learned from these reviews.
Source: GAO.
FAA has executed only 1 of the 27 key practices associated with Stage 3
critical processes: it has a process for distributing portfolio criteria
to project management personnel and other stakeholders. The remaining 26
key practices were not executed-primarily because FAA does not involve the
JRC in the regular oversight of non-NAS investments or in NAS investments
during their in-service management phase, weaknesses that we noted in our
assessment of Stage 2 requirements. Since Stage 3 requires an
enterprisewide perspective, the lack of oversight of these classes of
investments precludes the successful completion of most Stage 3 critical
processes. In addition, Stage 3 requires an enterprisewide perspective
that FAA has not adopted, which would enable the JRC to oversee all major
IT investments, regardless of life cycle phase or business unit. Although
it can be appropriate for FAA to manage its NAS, in-service NAS, and
non-NAS investments as separate subordinate portfolios-depending on the
successful execution of all Stage 2 key practices-its enterprise-level
portfolio should contain all major IT investments regardless of life cycle
stage or business line. In building this enterprise-level portfolio, the
JRC can choose whether to include specific investments based on
predetermined criteria, as described by the ITIM framework. Until FAA
fully implements the critical processes associated with managing its
investments as a complete portfolio, it will not have the data or
enterprisewide perspective it needs to make informed decisions about all
of its major IT investments.
In addition, FAA has not executed the six key practices for conducting
PIRs. In June 2004, in response to a recommendation contained in our 1999
report21 that FAA initiate PIRs for projects or programs within 3 to 12
months of deployment or termination, the NAS Configuration Management and
Evaluation Staff developed a proposed approach to PIRs, but this approach
was not implemented. In November 2003, the life cycle management policy
team proposed a change to the AMS that would require conducting these
reviews, but there has been no action on the proposal. Although the JRC
has recently reaffirmed its commitment to implement PIRs, there is no
policy and no established process to carry them out. If PIRs are not
conducted on a routine basis, then FAA will not be able to effectively
evaluate the results of its IT investments; this will affect the agency's
ability to determine whether to continue, modify, or terminate an IT
investment in order to meet its stated mission objectives.
Table 10 summarizes the status of FAA's critical processes for Stage 3,
showing how many associated key practices it has executed.
Table 10: Status of Stage 3 Critical Processes Critical process
Key practices executed
Total required by critical process
Percentage of key practices executed
Defining the portfolio criteria 1 7
Creating the portfolio 0 7
Evaluating the portfolio 0 7
Conducting
postimplementation reviews 0 6
Totals 1 27
Source: GAO.
21GAO/RCED/AIMD-99-88.
FAA Has Not Established a Process for Managing the Succession of Its
Information Systems
Once an agency has attained Stage 3 maturity, it evaluates its IT
investment processes and portfolios to identify opportunities for
improvement (Stage 4 capabilities). This entails (1) improving the
portfolio's performance and (2) managing systems and technology
succession. We did not assess FAA's capability for improving the
portfolio's performance, because it did not claim to be executing any of
the relevant key practices in its self-assessment.
According to ITIM, regarding system and technology succession management
includes (1) defining policies and procedures for managing the IT
succession process, (2) assigning responsibility for the IT succession
process, (3) developing criteria for identifying IT investments that may
meet succession status, and (4) periodically analyzing IT investments to
determine whether they are ready for succession. This critical process
enables an organization to recognize low-value or high-cost IT investments
and augments the routine replacement of systems at the end of their useful
lives. It also promotes the development of a forward-looking,
solution oriented view of IT investments that anticipates future resource
requirements and allows the organization to plan appropriately. This
process differs from the reselection activity in Stages 2 and 3 in that it
focuses on anticipating and planning for the retirement of legacy systems
and on meeting remaining requirements with other, perhaps new, systems. In
addition, succession management takes place at the end of a system's life
cycle.
FAA has not executed any of the nine key practices required to implement
this critical process. Although the agency has defined procedures in AMS
for retiring investments, it still needs to describe how to regularly
review systems that are in operations in order to identify candidates for
retirement. According to FAA, decisions on succession are made by the
service organizations. However, no individual or group has been assigned
responsibility for managing the succession process from an enterprise
perspective, which would allow the FAA to better anticipate and plan for
future resource requirements. Without an institutionalized process for
succession management, the FAA may not be able to identify those IT
investments that are eligible for succession in enough time to minimize
the effect of the transition on their successors. In addition, by
establishing an effective succession management process, the agency can
identify systems for retirement, freeing resources for other, superior,
investments.
FAA Has Initiated Efforts to Improve Its Investment Management Process
We have previously reported that to effectively implement IT investment
management processes, organizations need to be guided by a plan that (1)
is based on an assessment of strengths and weaknesses; (2) specifies
measurable goals, objectives, and milestones; (3) specifies needed
resources; (4) assigns clear responsibility and accountability for
accomplishing tasks; and (5) is approved by senior management.
FAA has begun to take steps to resolve some of the weaknesses identified
in this report. For example, at a June 10, 2004, meeting, the JRC decided
to incorporate budget justification documents (Exhibit 300s), which are
currently prepared for the Office of Management and Budget (OMB) as part
of the President's Budget formulation process, into the AMS process for
managing NAS investments. The Exhibit 300 will become the board's
decision-making document, and essential information from existing
AMS required documents-the investment management report, the acquisition
strategy paper, the integrated program plan, and the requirements
documents-will be incorporated into the Exhibit 300. The JRC also recently
decided to implement PIRs in order to track metrics during program
implementation. Finally, at that same meeting, the board decided to
collectively determine, at the meeting where the F&E budget is approved,
which F&E and OPS programs should be brought forward for review the
following year. This decision serves to bring certain investments in the
in-service management phase under the JRC's direct purview, although it
does not specify that consistent criteria be established, as the ITIM
framework requires.
FAA has also begun to initiate steps to bring more clarity to the ITEB's
responsibilities, although the specifics have yet to be defined. In its
charter, the ITEB is charged with making investment decisions about
non-NAS IT investments. This action would begin to bring all of the
non-NAS investments under a single authority. The charter suggests that
the ITEB choose among three options: (1) to send major non-NAS investment
decisions to the JRC, (2) to make the decision itself, given an acceptable
review process similar to the JRC processes, or (3) have the CIO, Chief
Financial Officer, and owning assistant/associate administrator make the
decision jointly. This description of the ITEB's roles and
responsibilities further alludes to the senior board's evolving
responsibility toward major non-NAS IT investments, although it falls
short of laying out specific criteria for selecting which investments
should be sent forward to the JRC. The ITEB has been given responsibility
for four short-term initiatives as well, including establishing an
agencywide cost control program for non-
NAS expenditures and ensuring that all OMB Exhibit 300s receive a passing
grade for the 2006 budget year. The ITEB has been charged with the
long term initiative of clearly delineating the roles it plays and its
relationship with the more senior board. The successful completion of this
initiative is likely to satisfy the single key practice that FAA has not
yet executed in the Instituting the Investment Board critical process of
the ITIM.
The Chief Operating Officer's recent reorganization of the ATO is intended
to make the heads of the service units responsible for IT projects from
their inception through the in-service management phase. This new
organization is designed to support his expressed intentions to increase
accountability for systems in operation in order to manage costs more
effectively. According to the Chief Operating Officer, FAA recognizes that
good processes are needed for both NAS and non-NAS to improve the way the
agency manages its investments.
While FAA has initiated these improvement efforts, it has not linked them
together in a plan with the characteristics listed above that would help
coordinate and guide the efforts. Until FAA develops a plan that would
allow for the systematic prioritization, sequencing, and evaluation of
improvement efforts, the agency risks not being able to effectively
establish mature investment management processes.
DOT is Taking Steps to Integrate Oversight of FAA's IT Investments
DOT has recently initiated several efforts that can serve to provide
better departmental oversight of FAA investments. This fiscal year DOT and
FAA reached an agreement by which DOT reviews FAA's Exhibit 300s as part
of the department's annual budget process, in which all departmental
components participate. Under this agreement, DOT conducts a review of all
FAA Exhibit 300s starting in June of each budget year and culminating in
the review of all Exhibit 300s by the Department Investment Review Board
in late August, prior to the submission of the budget to OMB in September.
As part of this agreement, DOT has outlined a process and schedule for
reviewing the fiscal year 2006 budget justifications for major FAA
programs and is monitoring FAA's progress in meeting this schedule. In
addition, the department has identified about a dozen programs that it
plans to monitor regularly and has begun reviewing these programs through
its senior investment management decision-making board, on which the FAA
Administrator is a voting member. DOT has also requested that FAA set
reasonable expectations for cost, schedule, and performance for its major
projects and that it then report quarterly on variances to those
expectations. FAA submitted its first quarterly report as of June 2004.
These regular reports are intended to help DOT maintain oversight of FAA's
processes and ensure that they are appropriate and consistent with OMB's
requirements. Furthermore, the department is currently planning to issue
an investment management guide that specifies minimum expectations that
its operating administrations (including FAA) are to follow in managing
their investments. According to DOT officials, FAA has been complying with
the department's requests for information to facilitate its oversight
process.
Department officials are attributing their increased oversight-and
cooperation from FAA-to the fact that the department has recently
reinstituted its own investment management processes. In addition, DOT
officials said that FAA now understands the role the department can play
in helping it to obtain the funding it needs for its programs.
Conclusions
FAA has established most of the project selection and control capabilities
needed to manage its NAS investments. This should help provide the
executive-level decision-making and oversight capabilities required to
establish accountability and guide major IT investments through most of
their life cycles. However, weaknesses remain. For example, although
business units are involved in the regular review of investments
throughout their life cycles, the JRC may not review the performance of
operations systems for several years unless they require significant
additional funds. Also, FAA has yet to define and implement the practices
it needs to select and control its non-NAS investments. Ultimately,
because the JRC does not regularly review NAS systems during the
in-service management phase and does not regularly review the non-NAS
systems in general, significant portions of FAA's approximately $2.5
billion investment in IT go without top-level executive oversight and are
not viewed as part of an enterprisewide portfolio. FAA has taken some
initial steps to implement PIRs, but it has not yet established a process
to carry them out.
The agency has begun to take some steps to develop improvements to address
some of these weaknesses, such as establishing an Information Technology
Executive Board with relevant responsibilities. In addition, the JRC has
begun integrating some budgeting and oversight processes, and the Chief
Operating Officer has begun to articulate a vision that includes
additional accountability for investments in operations. But FAA has not
developed a comprehensive plan to guide all improvement efforts. Such a
plan would help coordinate and prioritize improvement efforts and help
sustain commitment to the efforts under way. The increasing collaboration
between FAA and DOT further contributes to the likelihood that the
management of FAA's investments will improve as FAA's Exhibit 300s have
the benefit of department-level review and the departmental investment
review board conducts periodic reviews of selected projects.
Recommendations for Executive Action
To strengthen FAA's investment management capability and address the
weaknesses discussed in this report, we recommend that the Secretary of
the Department of Transportation direct the FAA Administrator to develop
and implement a plan for improving FAA's IT investment management
processes. The plan should address the weaknesses described in this
report, beginning with those we identified in our Stage 2 analysis and
continuing with those we identified in our Stage 3. The plan should also
draw together ongoing efforts as well as instituting new initiatives where
called for. The plan should, at a minimum, provide for accomplishing the
following:
In Stage 2
o Define procedures for aligning the JRC and the newly established ITEB.
o
Establish a process for the JRC to periodically reevaluate the alignment
of projects in the in-service management phase with strategic goals and
objectives.
o
Establish a process for the JRC to regularly review the performance of IT
systems throughout their life cycles and take corrective actions when
expected performance is not being met.
o
Define and implement an IT investment management structure, including an
investment management board and a disciplined process for managing all
non-NAS investments.
In Stage 3
o
Define and implement processes for managing major investments as part of
an enterprise-level portfolio, including NAS F&E investments, NAS
investments in the in-service management phase, and non-NAS investments.
o
Define and implement processes for carrying out PIRs on investments as
they enter the in-service management stage.
In developing the plan, the FAA Administrator should ensure that it (1)
specifies measurable goals, objectives, and milestones; (2) specifies
needed resources; (3) assigns clear responsibility and accountability for
accomplishing tasks; and (4) is approved by senior management. In
implementing the plan, the FAA Administrator should ensure that the needed
resources are provided to carry out the plan and that progress is measured
and reported periodically to the Secretary of Transportation.
Agency Comments
In commenting on a draft of this report, DOT's Director of Audit Relations
stated via e-mail that DOT appreciated the opportunity to review and offer
comment on our report and that GAO had done a good job keeping the report
balanced and fair, showing where FAA has many capabilities in place and
identifying areas that need improvement. The Director also provided a
technical comment, which we have incorporated into the report.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
the date of this letter. At that time, we will send copies to other
interested congressional committees, the Director of the Office of
Management and Budget, the Secretary of Transportation, FAA's
Administrator and CIO, and other interested parties. We also will make
copies available to others upon request. In addition, the report will be
available at no charge on the GAO Web site at www.gao.gov.
Should you or your offices have questions on matters discussed in this
report, please contact me at (202) 512-9286 or Lester P. Diamond,
Assistant Director, at (202) 512-7957. We can also be reached by e-mail at
[email protected], or [email protected], respectively. Key contributors to
this report were William G. Barrick, Niti Bery, Joanne Fiorino, Michael
Giannone, Sabine R. Paul, and Nik Rapelje.
David A. Powner Director, IT Management Issues
Appendix I
Objectives, Scope, and Methodology
The objectives of our review were to (1) evaluate FAA's capabilities for
managing its IT investments, (2) determine what plans the agency might
have for improving these capabilities, and (3) describe how DOT oversees
FAA's investments and investment process. Because FAA told us that it
managed its NAS and non-NAS investments differently, we performed separate
assessments for the practices to evaluate FAA's capabilities for managing
IT investments.
To address the first objective, for the NAS investments we reviewed the
results of the agency's self-assessment of Stages 2, 3, and 4 practices
using GAO's ITIM framework1 and validated and updated the results of the
self assessment through document reviews and interviews with officials. We
reviewed written policies, procedures, and guidance and other
documentation providing evidence of executed practices, including FAA's
Acquisition Management System guidance, mission analysis and investment
analysis guidance, and memorandums. We also reviewed JRC guidance and
records of decision, acquisition review guidance and meeting minutes, and
variance reporting procedures and reports. We did not assess FAA's
progress in establishing the capabilities found in one of the two Stage 4
critical processes, entitled Improving the Portfolio's Performance, or in
any of the Stage 5 critical processes, because FAA acknowledged that it
had not executed any of the key practices in these critical processes. For
the non-NAS investments, we reviewed the results of FAA's self assessments
of Stage 2 practices using GAO's ITIM framework and conducted interviews
to clarify and update the results. We did not perform a detailed
assessment of these practices because they most likely will be superseded
by a new process (when it is defined) for managing non-NAS investments,
and non-NAS investments are of lower cost and impact to FAA.
As part of our analysis, we selected three IT projects as case studies to
verify that the critical processes and key practices were being applied.
We selected projects that (1) supported different FAA functional areas,
(2) were in different life cycle phases, and (3) required different levels
of funding. The three projects are described below:
o
FAA Telecommunications Infrastructure (FTI)-FTI is a performance based
telecommunications services contract for voice, video, and data
point-to-point support for telecommunications for the National Airspace
1GAO-04-394G.
Appendix I
Objectives, Scope, and Methodology
System and its support system. It contributes to both the separation of
aircraft (the mission-support network) and other FAA uses (the operational
network, e.g., e-mail and phone). FTI will replace the current telecom
system. FTI will eliminate the need for other subnetworks, of which there
are currently eight or nine, and therefore eliminate the management
overhead associated with operating so many networks. The integration of
multiple networks and subnetworks will provide a single source and single
vehicle for telecom. FTI is in the Technical Operations unit and has
estimated life cycle costs of $2 billion. The contract for FTI was awarded
in June 2002.
o
En Route Communications Gateway (ECG)-ECG is a mission critical gateway,
or interface, for data from radar sites to Air Route Traffic Control
Centers. ECG will serve as a single domain communications gateway and will
provide the path for exchanging flight plan data from outside sources and
transfer data among systems. ECG provides a commercial-off-the-shelf
nondevelopmental item digital gateway using a modern, open and extensible
platform consisting of modular scalable hardware components. ECG will
incorporate interface capability to support legacy and future systems and
will provide the capability to transition to modern network communications
and access more surveillance sources. The flexibility provided by the ECG
system architecture will facilitate the evolution of the En Route domain
modernization. ECG will replace the Peripheral Adapter Module Replacement
Item system and provide a modern domain gateway that will support the
current and future En Route infrastructure. ECG is in the En Route &
Oceanic Service group and has estimated life cycle costs of $442.5 million
through September 2015.
o
Voice Switching and Control System (VSCS)-In our review of the VSCS
program, we focused our review on one of VSCS's subcomponents, the VSCS
Control Subsystem Upgrade (VCSU). The VCSU program, part of the Technical
Operations Communications service group, is designed to maintain overall
supportability of VSCS2 by replacing the hardware for the existing control
subsystem, associated VSCS operational and application software, required
software licenses, and supporting software and hardware documentation.
Deliverables for the VCSU
2VSCS is FAA's highly distributed, computer-controlled communications and
control system for U.S. air traffic management that allows air traffic
controllers to establish all air-to ground and ground-to-ground
communications with pilots and other air traffic controllers.
Appendix I
Objectives, Scope, and Methodology
program include all hardware, spare parts, software, software licenses,
system baseline documentation, training, and other technical documentation
necessary to support the product at 21 locations. According to FAA, the
VCSU program has a funding baseline of over $59 million and is in the
operations and maintenance phase.
For these projects, we reviewed project management documentation, such as
mission needs statements, acquisition program baselines, and integrated
program plans. We also interviewed the project managers for these
projects.
We compared the evidence collected from our document reviews and
interviews to the key practices in ITIM. We rated the key practices as
"executed" on the basis of whether the agency demonstrated (by providing
evidence of performance) that it had met the criteria of the key practice.
A key practice was rated as "not executed" when we found insufficient
evidence of a practice during the review or when we determined that there
were significant weaknesses in FAA's execution of the key practice.
To address our second objective, we obtained and evaluated documents
showing what management actions had been taken and what initiatives had
been planned by the agency. This documentation included JRC records of
decisions, the agency's capital investment guidance, and the recently
formed ITEB charter and meeting minutes. We also interviewed the Chief
Information Officer, other members of the JRC, and the Chief Operating
Officer to determine what efforts FAA had undertaken to improve IT
investment management processes.
To address our third objective, we reviewed documentation on DOT's process
for reviewing FAA's budget proposals and capital planning and investment
control reviews. We also conducted interviews with both FAA and DOT
officials, including DOT's CIO and Director for Capital Planning and
Investment Control to determine DOT's oversight role in FAA's investments
and investment management processes.
We conducted our work at FAA Headquarters in Washington, D.C., from
October 2003 through July 2004, in accordance with generally accepted
government auditing standards.
Appendix II
Investment Management Process Used by Some Organizational Units to Manage
Non-NAS Investments
Financial Services (ABA)
Instituting the investment board ABA has an investment board that conducts
periodic and monthly program reviews for all IT programs to determine
whether a program will be approved as an IT investment. A life cycle
process guide is now in place to direct the activities of the investment
board along with providing oversight of IT projects within ABA.
Meeting business needs The business needs of a project within ABA, along
with the dates for achieving them, need to be aligned with the strategic
goals established in the FAA Flight Plan. Projects or systems that are no
longer aligned with the Flight Plan will be decommissioned.
A project management plan identifies, among other things, the system's
users, customers, and types of services to be provided.
Selecting an investment Selecting and reselecting an IT investment within
ABA involves both the executive management team and ABA's CIO team. The
executive management team reviews the business needs of the investment and
compares them against the ABA's IT budget, while ABA's CIO team is
involved with the selecting and reselecting processes by analyzing the
technical costs associated with the IT investment and comparing those
technical costs against the ABA's IT budget.
Providing investment oversight ABA uses its life cycle process guide to
help manage its $25 million IT budget, which consists of 22 or 23
financial systems, 5 or 6 of them considered major programs under OMB's
definition of a major IT investment. A requirement of the life cycle
process guide is for every critical system in ABA to have a detailed
project management plan that addresses performance measures such as cost,
schedule, benefits, and risks.
The day-to-day progress of IT projects is tracked against critical
milestones that have been already established through weekly summary
reviews with IT staff. For major IT projects, biweekly meetings are
conducted that address any concerns with meeting the performance measures.
Capturing the investment information ABA captures its IT asset information
using its Information Technology Investment Portfolio System (ITIPS),
which is available to all ABA management and system support personnel. The
information in ITIPS is used to manage projects that are in production as
well as ensuring that the life cycle activities are in alignment with
FAA's mission statements.
Research and Acquisitions (ARA)
Instituting the investment board ARA uses its Operations Resource
Management Team guide to select, control, and evaluate ARA IT investments.
The team composed of representatives from ARA service units. ARA
investments are controlled and tracked through quarterly reviews. These
reviews look at the cost, schedule, and overall performance of the
investment.
Meeting business needs The business needs for ARA investments need to be
mapped back to the Flight Plan. A monthly status review report is prepared
in order to ensure that the business needs are tracking back to the Flight
Plan.
Selecting an investment ARA does not have any well-defined selection
criteria since each program uses its own configuration management plan.
ARA Ops build process guides the establishment of new projects.
Appendix II Investment Management Process Used by Some Organizational Units to
Manage Non-NAS Investments
(Continued From Previous Page)
Providing investment oversight A project plan does exist, along with
established expenditures, which the program managers submit to the ARA CIO
on a monthly basis. These monthly status reports occur between the CIO and
the program managers to decide if an investment's resources, such as
funding, need to be reallocated. Once the CIO and program manager decide
that it is necessary for an investment's resources to be reallocated, the
CIO will discuss the need further with the Deputy Associate Administrator
for ARA, who ultimately will determine whether a program will receive
additional resources, such as funding.
With respect to the level of interaction that ARA has had with the JRC in
the past, only one program from ARA, NextGen, has gone before the JRC.
According to the ARA CIO, in order for a program to go to the JRC, there
must be justification made to the council that the program is fully
operational and is considered to be a benefit and a priority to FAA. The
ARA Deputy Associate Administrator will determine if a program should go
before the JRC for approval and funding.
Capturing investment information The configuration control board uses a
database to capture asset inventory data about the systems that are owned
by the ARA CIO. According to the ARA CIO, in order for IT assets to be
effectively managed in ARA, there needs to be vision from AIO about what
programs to invest in over the next 5 years.
Air Traffic Services (ATS)
Instituting the investment board The Information Resource Management
Executive Board is responsible for selecting, controlling, and evaluating
ATS IT investments.
Meeting business needs Not all services within ATS have defined their
business needs. Even though ATS has the NAS Support Integration Process
(NSIP) data repository available for capturing IT asset information,
including business needs, and for defining system users, there is no
consistency in terms of the records being complete because there are
systems within ATS that have not registered with NSIP.
Selecting an investment The ATS CIO manages the selection process, which
begins with the NSIP registration criteria.
Providing investment oversight Each business unit within ATS has its own
project management plan and procedures. The day-to-day tracking of
projects as well as the monitoring of whether corrective actions are being
executed is also the responsibility of the individual business units. Even
though the individual business units are tasked with this level of
responsibility, the ATS CIO does play an oversight role by setting the
criteria and policies for the investments to be made for the projects.
Capturing investment information ATS uses the NSIP meta data repository to
collect any changes to the IT projects and systems by providing a full
declaration of the project or system. This includes providing information
to help ATS avoid unwanted costs due to systems having redundant
functionality and determining whether a system's or a project's functions
match the stated mission goals for ATS. NSIP also handles the technical
rollover for ATS systems or projects.
Information Services (AIO)
Instituting the investment board AIO's investment management process can
be characterized as iterative and well managed, but undocumented. The AIO
Business Plan and IT Strategy are used to ensure that when funds are
appropriated and allocated that they map back to the Flight Plan.
Investments are controlled or tracked by the Deputy CIO on a monthly basis
to get an indication as to where the program is in the process against the
expenditures that have been already established. Weekly meetings are held
with the unit's CIO to discuss any issues regarding AIO's investment
management process.
Appendix II Investment Management Process Used by Some Organizational Units to
Manage Non-NAS Investments
(Continued From Previous Page)
Meeting business needs AIO does not have any written policies or
procedures for identifying business needs for its IT projects. Only one of
its major projects, NAS Adaptation Service and Environment, has documented
its requirements, which includes specific users.
Selecting an investment AIO uses an undocumented process for reviewing new
IT proposals to reach an agreement on selection.
Providing investment oversight There are no AIO-wide policies or
procedures for managing projects or investment oversight. The Information
Technology Executive Board (ITEB)a has been formed to provide a governing
structure for non-NAS programs. One of the targets for ITEB is to look at
cost control and cross-cutting IT initiatives by involving the heads of
the lines of business. The ITEB is also going to be involved with
improving the scores on the Exhibit 300 business cases for OMB.
Capturing the investment information AIO uses ITIPS to track its asset
inventory and IT investments. The Deputy CIO of AIO is responsible for
ensuring that the inventory located in ITIPS meets the needs of AIO's
investment management process. According to AIO, the information within
ITIPS is updated at least twice a year.
Human Resource Management (AHR)
Instituting the investment board AHR does not have an investment board.
Instead, AHR's senior managementb is responsible for selecting,
controlling, and evaluating all IT investments by using established agency
acquisition policies and procedures to conduct investment management
decisions.
Meeting business needs Business needs and specific users for each project
are identified within the project plan and are aligned with the AHR
Strategic Plan, the FAA Flight Plan, and the AIO Plan. AHR is also
aligning its business needs to the ITEB plans. Business needs are
re-evaluated on a quarterly basis to ensure that a project is aligned with
FAA's strategic goals and objectives.
Selecting an investment AHR senior management uses its prioritization
process to evaluate and select investments for funding. The office and
center directors determine their requirements and then a budget request is
submitted for proposal funding. AHR receives an allowance amount from the
budget office. The first priority is to handle personnel payments. The
remaining balance is then redistributed to the business divisions. The
"building blocks" process starts at this point. This is when base funding
is reviewed to decide if a current investment needs continued funding by
asking questions about the importance of continuing the funding of a
particular project by looking at the project activity and what the impact
will be if this project is no longer funded. Each division will submit a
list of prioritized projects with costs to the directorate. This list may
exceed the budget level. The directorate will reprioritize the original
list.
Providing investment oversight AHR has a Human Resource Management
Automation Plan that contains procedures for approving IT projects, and
describes the policies and procedures that AHR uses for project
management. Despite having project management policies and procedures, not
all projects within AHR have a formal project plan. The size and scope of
the project are two factors that help determine whether a project has a
formal project plan. AHR Division Managers ensure that projects are on
time by performing quarterly reviews that assess a project's cost and
schedule. AHR uses a color scheme (red, green, and yellow) to indicate the
schedule status of major milestones.
Capturing the investment information AHR uses the ITIPS as its inventory
for making investment management decisions. AHR projects are listed in
ITIPS, along with business cases.
Appendix II Investment Management Process Used by Some Organizational Units to
Manage Non-NAS Investments
(Continued From Previous Page)
Regions and Center Operations (ARC)
Instituting the investment board The IT Configuration Management Board is
ARC's investment review board. The board's charter has recently been
redone to provide more traceability back to the Flight Plan. The board
functions include evaluating potential IT investment options for ARC,
making recommendations on IT investment, establishing ARC-wide IT
standards, and developing and maintaining investment policies and
procedures. The board is led by the unit's CIO and includes four IT
managers from the regional offices and aeronautical center and two members
from the ARC Management Team. The ARC Management Team makes the
finalselection decisions. The IT investment management decisions are then
incorporated into the ARC Business Plan. The ARC unit is also involved
with cross-organizational investment decisions for FAA through its
membership on the FAA CIO Council.
Meeting business needs Business needs are identified through entries made
in ITIPS, along with documentation from Exhibit 300s and Exhibit 53s.
Selecting an investment ARC does not have its selection criteria
documented. To evaluate and select IT investments, the ARC IT
Configuration Management Board considers such things as benefits to ARC
across the regions, expected return on investment, technical feasibility,
and risk. The ARC business plan and the Flight Plan are the documents that
address these priorities.
Providing investment oversight ARC does not have policies or procedures
for project management. Instead, ARC uses a weekly teleconference to
address expectations and progress of ARC-wide IT initiatives at the IT
manager level across ARC. According to the ARC CIO, a second
teleconference has been added to discuss portfolio management-schedule,
budget, training, and deployment along with whether the project will be
integrated with other lines of business.
Capturing the investment information ARC uses ITIPS as its standardized
repository for collecting asset information that will be useful for ARC's
IT investment management decisions by providing information about what
types of systems and functions are available and how they are supporting a
specific business issue.
Regulation and Certification (AVR)
Instituting the investment board Similar to an IT investment board, AVR
has a two-tiered management structure that is composed of the AVR
management team and the CIO management team. The AVR management team
includes the Associate Administrator and the Service Directors who make
the final decisions based upon recommendations and input from the CIO
management team and its business partners from each of the service units.
According to AVR, its IT investment process guide is still under
development and will be completed at the end of Fiscal Year 2004.
Meeting business needs Each line of business within AVR identifies and
documents its business needs including project requirements and specific
users. Once the business needs have been identified, the IT Management and
Resources section prioritizes them for funding.
Selecting an investment Programs in AVR are reviewed quarterly. For major
projects, meetings are designed to look at project milestones to see if
they are being met. These meetings are carried out biweekly and presented
to the AVR management team.
Providing investment oversight The AVR CIO management team is responsible
for monitoring projects and reporting to the AVR Management team. Biweekly
meetings are held for major projects within AVR.
Capturing the investment information AVR's system inventory is a part of
its enterprise architecture. The system inventory is being used primarily
in developing the Exhibit 300s. The performance of IT projects in AVR is
monitored daily, based upon each project's individual plan, using project
management tools such as MS Project. According to AVR, not all projects
have a project plan in place, but AVR is trying to make it a requirement.
Source: GAO, based on information from FAA.
Appendix II Investment Management Process Used by Some Organizational
Units to Manage Non-NAS Investments
aITEB is a board that can provide a governing structure so that
information technology is used as an agency wide strategic asset.
bComposed of Assistant Administrator; two Deputy Assistant Administrators;
three Office Directors; the Director, Center for Management and
Development; and the AHR Business Officer.
GAO's Mission The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:Waste, and Abuse in Web site:
www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Federal Programs Automated answering system:
(800) 424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125
Relations Washington, D.C. 20548
Public Affairs Jeff Nelligan, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Presorted Standard
Postage & Fees Paid
GAO
Permit No. GI00
United States Government Accountability Office Washington, D.C. 20548-0001
Official Business Penalty for Private Use $300
Address Service Requested
*** End of document. ***