Federal Aviation Administration: Stronger Architecture Program
Needed to Guide Systems Modernization Efforts (29-APR-05,
GAO-05-266).
The Federal Aviation Administration's (FAA) mission is to promote
the safe, orderly, and expeditious flow of air traffic in the
U.S. airspace system. To this end, FAA is modernizing its air
traffic control systems, a multibillion dollar effort that GAO
has designated as a high-risk program. GAO's research into the
practices of successful public- and private-sector organizations
has shown that developing and using an enterprise architecture,
or blueprint, to guide and constrain systems investments is
crucial to the success of such a modernization effort. GAO was
asked to determine whether FAA has established effective
processes for managing the development and implementation of an
enterprise architecture.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-05-266
ACCNO: A22959
TITLE: Federal Aviation Administration: Stronger Architecture
Program Needed to Guide Systems Modernization Efforts
DATE: 04/29/2005
SUBJECT: Agency missions
Air traffic control systems
Best practices
Best practices methodology
Enterprise architecture
Information technology
Internal controls
Strategic planning
Transportation safety
Commercial aviation
Program management
FAA National Airspace System Plan
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-05-266
United States Government Accountability Office
GAO Report to Congressional Requesters
April 2005
FEDERAL AVIATION ADMINISTRATION
Stronger Architecture Program Needed to Guide Systems Modernization Efforts
a
GAO-05-266
[IMG]
April 2005
FEDERAL AVIATION ADMINISTRATION
Stronger Architecture Program Needed to Guide Systems Modernization Efforts
What GAO Found
FAA has two architecture projects-one for its National Airspace System
(NAS) operations and one for its administrative and mission support
activities-that together constitute its enterprise architecture program.
However, it has established only a few of the management capabilities for
effectively developing, maintaining, and implementing an architecture. For
example, the agency reports that it has allocated adequate resources to
the projects, and it has established project offices to be responsible for
developing the architecture, designated a chief architect for each
project, and released Version 5.0 of its NAS architecture. But the agency
has yet to establish other key architecture management capabilities-such
as designating a committee or group that represents the enterprise to
direct, oversee, or approve the architecture, and establishing an
architecture policy. FAA agreed that the agency needs an effective
enterprise architecture program and stated that it plans to improve its
management of both projects. For example, the agency intends to establish
a steering committee; develop a policy that will govern the development,
maintenance, and implementation of the architecture program; and have an
approved architecture project management plan for the non-NAS
architecture.
GAO's experience in reviewing other agencies has shown that not having an
effective enterprise architecture program can be attributed to, among
other things, an absence of senior management understanding and support
and cultural resistance to having and using one. It has also shown that
attempting major systems modernization programs like FAA's without having
and using an enterprise architecture often results in system
implementations that are duplicative, are not well integrated, require
costly rework to interface, and do not effectively optimize mission
performance.
FAA's Mission
Source: Dynamic Graphics.
United States Government Accountability Office
Contents
Letter 1
Results in Brief 2
Background 3
FAA Has Yet to Establish Key Architecture Development,
Maintenance, and Implementation Processes 17
Conclusions 26
Recommendations for Executive Action 26
Agency Comments 27
Appendixes
Appendix I:
Appendix II:
Appendix III:
Appendix IV:
Objective, Scope, and Methodology
Assessment of Architecture Management Efforts for the National Airspace
System
Assessment of Architecture Management Efforts for the Non-National
Airspace System
GAO Staff Acknowledgments
29
31
35 40
Tables Table 1: Responsibilities of FAA's Staff Offices 7
Table 2: Responsibilities of FAA's Lines of Business 8
Table 3: GAO's Framework for Enterprise Architecture
Management Maturity (Version 1.1) 12
Table 4: GAO's Framework for Enterprise Architecture (EA)
Management Maturity (Version 1.1) 21
Figures Figure 1: Summary of the Air Traffic Control System over the
Continental United States and Oceans 4
Figure 2: Simplified Diagram of FAA's Organizational Structure 6
Contents
Abbreviations
CIO chief information officer
COO chief operating officer
EA enterprise architecture
FAA Federal Aviation Administration
FEA Federal Enterprise Architecture
FEAF Federal Enterprise Architecture Framework
IG Office of Inspector General
IT information technology
NAS National Airspace System
OMB Office of Management and Budget
SRA Systems Research and Applications International, Inc.
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
April 29, 2005
The Honorable Tom Davis Chairman, Committee on Government Reform House of
Representatives
The Honorable Adam H. Putnam House of Representatives
The Federal Aviation Administration's (FAA) mission is to promote the
safe, orderly, and expeditious flow of air traffic in the U.S. airspace
system-commonly referred to as the National Airspace System (NAS). To
accomplish this mission, it relies on air traffic control systems to
provide such services as controlling takeoffs and landings and managing
the flow of traffic between airports. It also relies on its administrative
and mission support systems (non-NAS) to perform other activities needed
to achieve its mission, such as accident and incident investigations and
security inspections. To support its mission performance, FAA is
modernizing its air traffic control systems.1
Our research into the practices of successful public-and private-sector
organizations has shown that developing and using a well-defined
modernization blueprint-an enterprise architecture-is essential to an
organization's ability to transform its operations and supporting systems
in a way that eliminates duplication, promotes interoperability, reduces
costs, and optimizes mission performance.
1In 1995 we designated the air traffic control modernization program as
high risk because of the program's size, importance, and complexity and
because of the cost and the numerous problems it had encountered in
systems acquisition. It has remained on our high-risk list since that
time.
You asked that we evaluate whether FAA is following best practices in key
information technology (IT) management areas, such as enterprise
architecture, investment management, and software/system development. This
report is one in a series of reports responding to your request.2 As
agreed, the objective of our review was to determine whether FAA has
established effective processes for managing the development and
implementation of an enterprise architecture. To accomplish this
objective, we analyzed documents, interviewed agency officials, and
compared FAA's architecture development, maintenance, and implementation
practices against our enterprise architecture management maturity
framework.3 We performed our work in accordance with generally accepted
government auditing standards. Details on our objective, scope, and
methodology are in appendix I.
Results in Brief FAA has taken steps to develop an enterprise architecture
through two architecture projects that are intended to cover its two core
business areas-NAS operations and non-NAS administrative and
mission-support operations. However, it has yet to establish most of the
key management structures, processes, and controls that are necessary to
effectively manage either of these two architecture projects. For example,
for the NAS architecture project, the agency has devoted resources,
established a project office, designated a chief architect, and issued
Version 5.0 of its architecture, but it has not implemented other key
management capabilities, such as designating a committee or group
representing the enterprise to direct, oversee, or approve the
architecture. Similarly, for the non-NAS architecture project, the agency
has devoted resources, established a project office, and designated a
chief architect. However, it has not yet implemented other key management
capabilities, such as establishing a written and approved architecture
policy. FAA officials agreed that management improvements are needed for
both projects, and they told us that they are in the early stages of
implementing these improvements, including establishing a steering
committee, developing an
2See also, GAO, Information Technology: FAA Has Many Investment Management
Capabilities in Place, but More Oversight of Operational Systems Is
Needed, GAO-04-822 (Washington, D.C.: Aug. 20, 2004); and GAO, Air Traffic
Control: System Management Capabilities Improved, but More Can Be Done to
Institutionalize Improvements, GAO-04-901 (Washington, D.C.: Aug. 20,
2004).
3GAO, Information Technology: A Framework for Assessing and Improving
Enterprise Architecture Management (Version 1.1), GAO-03-584G (Washington,
D.C.: April 2003).
architecture policy, and having an approved architecture project
management plan for the non-NAS architecture.
Our experience in reviewing other agencies has shown that not having an
effective enterprise architecture program can be attributed to, among
other things, an absence of senior management understanding and support of
an architecture, and cultural resistance to having and using one. Our
experience also shows that attempting major system modernization programs,
like FAA's, without having and using a well-defined enterprise
architecture often results in system implementations that are duplicative,
are not well integrated, require costly rework to interface, and do not
effectively optimize mission performance.
To support FAA in managing its efforts to develop and implement an
enterprise architecture, we are making recommendations to the Secretary of
the Department of Transportation related to establishing an effective
enterprise architecture management program. In comments on a draft of this
report provided by the Department's Director of Audit Relations, the
department neither agreed nor disagreed with our conclusions and
recommendations. The Director provided technical comments, which we have
incorporated as appropriate in the report.
Background
FAA's Mission and Organizational Structure
FAA's primary mission is to provide a safe, secure, and efficient global
airspace system that promotes airspace safety in the United States and
contributes to national security. The agency's roles include regulating
civil aviation, developing and operating a system of air traffic control
and navigation for civil and military aircraft, and researching and
developing the NAS, which consists of more than 19,000 airports, 750 air
traffic control facilities, and about 45,000 pieces of equipment.
FAA's mission performance depends on the adequacy and reliability of the
nation's air traffic control system. The air traffic control system, the
primary component of the NAS, is a vast network of computer hardware,
software, and communications equipment. This system consists of automated
information processing and display, communication, navigation,
surveillance, and weather resources that permit air traffic controllers to
view key information-such as aircraft location, aircraft flight plans, and
prevailing weather conditions-and to communicate with pilots. These
resources reside at, or are associated with, several air traffic control
facilities-towers, terminal radar approach control facilities, air route
traffic control centers (en route centers), flight service stations, and
the System Command Center. Figure 1 shows a visual summary of the air
traffic control system over the continental United States and oceans.
Figure 1: Summary of the Air Traffic Control System over the Continental
United States and Oceans
Source: GAO.
FAA's mission performance also depends on the skills and expertise of its
work force, composed of over 50,000 staff who provide aviation services-
including air traffic control; maintenance of air traffic control
equipment; and certification of aircraft, airline operations, and pilots.
In fiscal year 2005, FAA's budget authority to support its mission was
approximately $14 billion.4 According to FAA officials, approximately 95
percent of the agency's total spending is in support of the NAS. Further,
FAA estimates that it will spend $7.6 billion over the next two years to
complete key modernization projects.
As figure 2 illustrates, FAA has twelve staff offices to accomplish its
mission-including the Office of International Aviation and the Office of
Information Services/Chief Information Officer-and four lines of
business-Air Traffic Organization, Commercial Space Transportation,
Airports, and Regulation and Certification. Tables 1 and 2 provide
additional information about the responsibilities of these offices and
lines of business.
4Federal Aviation Administration: Budget in Brief Fiscal Year 2006
(Washington, D.C.: February 2005).
Figure 2: Simplified Diagram of FAA's Organizational Structure
Source: FAA.
Table 1: Responsibilities of FAA's Staff Officesa
Office Responsibilities
Chief Counsel Provides legal services and representation to FAA's
Administrator and all agency organizations at the headquarters, regional,
and center levels.
Civil Rights Advises, represents, and assists FAA's Administrator on civil
rights and equal opportunity matters such as unlawful discrimination;
program beneficiaries; and valuing, using, and managing the differences
that individuals bring to the workplace.
Government and Industry Affairs Advises and represents FAA's Administrator
on matters concerning the Congress, aviation industry groups, and other
governmental organizations. Works with other FAA offices to develop and
review plans and strategies involving these groups.
System Safety Develops and implements improved tools and processes, such
as hazard identification and risk management tools and processes, in order
to coordinate safety issues and facilitate more effective use of safety
data, both inside and outside the agency.
Public Affairs Provides the public with information about the agency's
mission, policies, activities, and operations.
Human Resource Management Advises and assists FAA's Administrator in
directing, coordinating, communicating, and ensuring the adequacy of
agency plans, programs, and initiatives associated with, among other
things, human capital planning, measurement, and evaluation.
International Aviation Works with key aviation partners and the
International Civil Aviation Organization to support the adoption of
international safety standards and to implement harmonized air traffic
procedures and technologies. Also works through various international
organizations and programs to share civil aviation safety information.
Financial Services Advises the FAA offices about plans and programs for
budget, financial management, and performance management.
Security and Hazardous Materials Ensures the integrity of those
individuals who work in or support the NAS and protects FAA employees and
facilities from criminal and terrorist acts.
Regions and Center Operations Provides corporate shared services,
including financial systems and operations; emergency readiness;
enterprisewide information services and business application development;
and logistics services such as acquisition, real estate, and supply
support.
Aviation Policy, Planning and Environment Leads the agency's strategic
policy and planning efforts, coordinates FAA's reauthorization before the
Congress, and is responsible for national aviation policies and strategies
in the environment and energy arenas.
Information Services/Chief Information Provides policy and direction for
the agency in the areas of IT strategic planning, IT
Officer investment analysis, process engineering, information management,
information security, and enterprise architecture.
Source: FAA.
aThese twelve staff offices will provide input that will be used to
develop the agency's non-NAS architecture.
Table 2: Responsibilities of FAA's Lines of Business
Line of business Responsibilities
Air Traffic Organization/Chief Operating Moves air traffic safely and
efficiently and manages the results of these efforts through
Officera objectives, goals, customer service standards, and targets for
improved cost and performance.
Commercial Space Transportation Ensures the protection of the public,
property, and the national security and foreign policy interests of the
United States during a commercial launch or re-entry activity and
encourages, facilitates, and promotes U.S. commercial space
transportation.
Airports Provides leadership in planning and developing a safe and
efficient national airport system to satisfy the needs of the aviation
interests of the United States.
Regulation and Certification Establishes aviation safety standards,
monitors safety performance, conducts aviation safety education and
research, and issues and maintains aviation certificates and licenses.
Source: FAA.
aThe Air Traffic Organization was formed on February 8, 2004, by combining
the Air Traffic Services and the Research and Acquisitions units, which
had been primarily responsible for managing air traffic services within
FAA. It is headed by a chief operating officer, whose responsibilities
include establishing and maintaining organizational and individual goals,
a 5-year strategic plan that includes the air traffic control system
mission and objectives, and a framework agreement with FAA's Administrator
to establish the organization's relationships with other agency
organizations.
An Enterprise Architecture Is Critical to Successful Systems Modernization
Effective use of enterprise architectures, or modernization blueprints, is
a trademark of successful public and private organizations. For more than
a decade, we have promoted the use of architectures to guide and constrain
systems modernization, recognizing them as a crucial means to a
challenging goal: agency operational structures that are optimally defined
in both business and technological environments. The Congress, the Office
of Management and Budget (OMB), and the federal Chief Information Officer
(CIO) Council have also recognized the importance of an
architecture-centric approach to modernization. The Clinger-Cohen Act of
19965 mandates that an agency's CIO develops, maintains, and facilitates
the implementation of an IT architecture. Further, the E-Government Act of
20026 requires OMB to oversee the development of enterprise architectures
within and across agencies.
5The Clinger-Cohen Act of 1996, 40 U.S.C. 11315(b)(2). 6E-Government Act
of 2002, Pub. L. No. 107-347 (Dec. 17, 2002).
Enterprise Architecture: A Brief Description
Generally speaking, an enterprise architecture connects an organization's
strategic plan with program and system solution implementations by
providing the fundamental business and technology details needed to guide
and constrain investments in a consistent, coordinated, and integrated
fashion. As such, it should provide a clear and comprehensive picture of
an entity, whether it is an organization (e.g., federal agency) or a
functional or mission area that cuts across more than one organization
(e.g., air traffic control). This picture consists of snapshots of both
the enterprise's current or "As Is" environment and its target or "To Be"
environment, as well as a capital investment road map for transitioning
from the current to the target environment. These snapshots further
consist of "views," which are basically one or more architecture products
that provide conceptual or logical representations of the enterprise.
The suite of products and their content that form a given entity's
enterprise architecture are largely governed by the framework used to
develop the architecture. Since the 1980s, various frameworks have emerged
and been applied. For example, John Zachman developed a structure or
"framework" for defining and capturing an architecture.7 This framework
provides for six windows from which to view the enterprise, which Zachman
terms "perspectives" on how a given entity operates: those of (1) the
strategic planner, (2) the system user, (3) the system designer, (4) the
system developer, (5) the subcontractor, and (6) the system itself.
Zachman also proposed six abstractions or models associated with each of
these perspectives: these models cover (1) how the entity operates, (2)
what the entity uses to operate, (3) where the entity operates, (4) who
operates the entity, (5) when entity operations occur, and (6) why the
entity operates.
In September 1999, the federal CIO Council published the Federal
Enterprise Architecture Framework (FEAF), which is intended to provide
federal agencies with a common construct for their respective
architectures, to facilitate the coordination of common business
processes, technology insertion, information flows, and system investments
among federal agencies. FEAF describes an approach, including models and
definitions, for developing and documenting architecture descriptions for
multiorganizational functional segments of the federal government. Similar
to most frameworks, FEAF's proposed models describe an entity's
7J. A. Zachman, "A Framework for Information Systems Architecture," IBM
Systems Journal 26, no. 3 (1987).
business, the data necessary to conduct the business, applications to
manage the data, and technology to support the applications.
More recently, OMB established the Federal Enterprise Architecture (FEA)
Program Management Office to develop a federated enterprise architecture
according to a collection of five "reference models, and a security and
privacy profile overlaying the five models."
o The Performance Reference Model is intended to describe a set of
performance measures for the major IT initiatives and their contribution
to program performance. Version 1.0 of the model was released in September
2003.
o The Business Reference Model is intended to describe the federal
government's businesses, independent of the agencies that perform them. It
serves as the foundation for the FEA. Version 2.0 of the model was
released in June 2003.
o The Service Component Reference Model is intended to identify and
classify IT service (i.e., application) components that support federal
agencies and promote the reuse of components across agencies. Version 1.0
of the model was released in June 2003.
o The Data Reference Model is intended to describe, at an aggregate
level, the types of data and information that support program and business
line operations and the relationships among these types. Version 1.0 of
the model was released in September 2004.
o The Technical Reference Model is intended to describe the standards,
specifications, and technologies that collectively support the secure
delivery, exchange, and construction of service components. Version 1.1 of
the model was released in August 2003.
o The Security and Privacy Profile is intended to provide guidance on
designing and deploying measures that ensure the protection of information
resources. OMB has released Version 1.0 of the profile.
Although these various enterprise architecture frameworks differ in their
nomenclatures and modeling approaches, they consistently provide for
defining an enterprise's operations in both (1) logical terms, such as
interrelated business processes and business rules, information needs and
flows, and work locations and users and (2) technical terms, such as
hardware, software, data, communications, and security attributes and
performance standards. The frameworks also provide for defining these
perspectives for both the enterprise's current or "As Is" environment and
its target or "To Be" environment, as well as a transition plan for moving
from the "As Is" to the "To Be" environment.
The importance of developing, implementing, and maintaining an enterprise
architecture is a basic tenet of both organizational transformation and IT
management. Managed properly, an enterprise architecture can clarify and
help to optimize the interdependencies and relationships among an
organization's business operations and the underlying IT infrastructure
and applications that support these operations. Employed in concert with
other important management controls, such as portfolio-based capital
planning and investment control practices, architectures can greatly
increase the chances that an organization's operational and IT
environments will be configured to optimize its mission performance. Our
experience with federal agencies has shown that making IT investments
without defining these investments in the context of an architecture often
results in systems that are duplicative, not well integrated, and
unnecessarily costly to maintain and interface.8
8See, for example, GAO, Homeland Security: Efforts Under Way to Develop
Enterprise Architecture, but Much Work Remains, GAO-04-777 (Washington,
D.C.: Aug. 6, 2004); DOD Business Systems Modernization: Limited Progress
in Development of Business Enterprise Architecture and Oversight of
Information Technology Investments, GAO-04-731R (Washington, D.C.: May 17,
2004); Information Technology: Architecture Needed to Guide NASA's
Financial Management Modernization, GAO-04-43 (Washington, D.C.: Nov. 21,
2003); DOD Business Systems Modernization: Important Progress Made to
Develop Business Enterprise Architecture, but Much Work Remains,
GAO-03-1018 (Washington, D.C.: Sept. 19, 2003); and Information
Technology: DLA Should Strengthen Business Systems Modernization
Architecture and Investment Activities, GAO-01-631 (Washington, D.C.: June
29, 2001).
Our Prior Work Has Emphasized the Need for FAA to Establish Architecture
Management Capabilities
In November 2003,9 we reported the results of our governmentwide survey of
agencies' progress-including FAA's-in establishing key enterprise
architecture management capabilities as described in Version 1.1 of our
architecture management maturity framework.10 This framework associates
specific architecture management capabilities with five hierarchical
stages of management maturity, starting with creating enterprise
architecture awareness and followed by building the enterprise
architecture management foundation, developing the enterprise
architecture, completing the enterprise architecture, and leveraging the
enterprise architecture to manage change. Table 3 provides a more detailed
description of the stages of Version 1.1 of the framework.
Table 3: GAO's Framework for Enterprise Architecture Management Maturity
(Version 1.1) Maturity stage Description
Stage 1: Creating enterprise architecture Organization does not have plans
to develop and use an architecture, or it has plans that
awareness do not demonstrate an awareness of the value of having and using
an architecture. While stage 1 agencies may have initiated some
architecture activity, these agencies' efforts are ad hoc and
unstructured, lack institutional leadership and direction, and do not
provide the management foundation that is necessary for successful
architecture development.
Stage 2: Building the enterprise Organization recognizes that the
architecture is a corporate asset by vesting
architecture management foundation accountability for it in an executive
body that represents the entire enterprise. At this stage, an organization
assigns architecture management roles and responsibilities and establishes
plans for developing enterprise architecture products and for measuring
program progress and product quality; it also commits the resources
necessary for developing an architecture-people, processes, and tools.
Stage 3: Developing the enterprise Organization focuses on developing
architecture products according to the selected
architecture framework, methodology, tool, and established management
plans. Roles and responsibilities assigned in the previous stage are in
place, and resources are being applied to develop actual enterprise
architecture products. The scope of the architecture has been defined to
encompass the entire enterprise, whether organization-based or
function-based.
Stage 4: Completing the enterprise Organization has completed its
enterprise architecture products, meaning that the
architecture products have been approved by the architecture steering
committee or an investment review board and by the CIO. Further, an
independent agent has assessed the quality (i.e., completeness and
accuracy) of the architecture products. Additionally, evolution of the
approved products is governed by a written architecture maintenance policy
approved by the head of the organization.
9GAO, Information Technology: Leadership Remains Key to Agencies Making
Progress on Enterprise Architecture Efforts, GAO-04-40 (Washington, D.C.:
Nov. 17, 2003).
10GAO-03-584G.
(Continued From Previous Page)
Maturity stage Description
Stage 5: Leveraging the enterprise Organization has secured senior
leadership approval of the enterprise architecture
architecture to manage change products and a written institutional policy
stating that IT investments must comply with the architecture unless they
are granted an explicit compliance waiver. Further, decision makers are
using the architecture to identify and address ongoing and proposed IT
investments that are conflicting, overlapping, not strategically linked,
or redundant. Also, the organization tracks and measures architecture
benefits or return on investment, and adjustments are continuously made to
both the architecture management process and the enterprise architecture
products.
Source: GAO.
Based on information provided by FAA, we reported that the agency had not
established an architecture management foundation; as a result, we rated
the agency to be at stage 1 of our framework. Specifically, we reported
that it had not (1) allocated adequate resources and (2) established a
framework, methodology, and automated tools to build the enterprise
architecture. According to our framework, effective architecture
management is generally not achieved until an enterprise has a completed
and approved architecture that is being effectively maintained and is
being used to leverage organizational change and support investment
decision making. An enterprise with these characteristics would need to
have satisfied all of the stage 2 and 3 core elements and most of the
stage 4 and 5 elements.
Our Prior Work Has Also Emphasized the Need for FAA to Institutionalize
Other Key IT Management Controls
In August 2004,11 we reported that FAA had established most-about 80
percent-of the basic practices needed to manage its mission-critical
investments, including many of the foundational practices for selecting
and controlling IT investments. However, we reported that weaknesses still
existed in the process. For example, FAA had not involved its senior IT
investment board in regular reviews of investments that had completed
development and become operational, and had not implemented standard
practices for managing its mission-support and administrative investments.
Because of these weaknesses, we concluded that agency executives could not
be assured that they were selecting and managing the mix of investments
that best met the agency's needs and priorities. Accordingly, we made
several recommendations, including that the agency develop and implement a
plan aimed at addressing the weaknesses identified in our
11GAO-04-822.
report. FAA generally concurred with our conclusion and recommendations.
In addition, in August 2004,12 we reported that FAA had made progress in
improving its capabilities for acquiring software-intensive systems, but
that there were still areas that needed improvement. Specifically, we
reported that it had recurring weaknesses in the areas of measurement and
analysis,13 quality assurance, and verification. We concluded that these
weaknesses prevented FAA from consistently and effectively managing its
mission-critical systems and increased the risk of cost overruns, schedule
delays, and performance shortfalls. We made several recommendations,
including that FAA address these specific weaknesses and institutionalize
its process improvement initiatives by establishing a policy and plans for
implementing and overseeing process improvement initiatives. FAA generally
concurred with our conclusion and recommendations.
Our Prior Work Has Identified Problems with the Air Traffic Control
Modernization Program
FAA has a long and well-documented history of problems with its air
traffic control modernization program, including cost overruns, schedule
delays, and performance shortfalls. We first identified this program as an
area at high risk in 1995 because of the modernization's size, complexity,
cost, and problem-plagued past. Over the past decade, we have continued to
report on these problems. The program remains on our high-risk list
today.14
In March 1999,15 we testified that FAA had had some success in deploying
new modernization systems over the past two decades, but that the agency
had not delivered most of its major air traffic control systems in
accordance with its cost, schedule, and performance goals, due largely to
its failure to implement established guidelines for acquiring new systems.
Specifically, we testified that the agency had not fully implemented an
effective process for monitoring the cost, schedule, benefits,
performance, and risk of its key projects throughout their life cycles. We
also noted that
12GAO-04-901.
13The purpose of measurement and analysis is to develop and sustain a
measurement capability that is used to support management information
needs.
14GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January
2005).
15GAO, Air Traffic Control: Observations on FAA's Air Traffic Control
Modernization Program, GAO/T-RCED/AIMD-99-137 (Washington, D.C.: March 25,
1999).
FAA lacked an evaluation process for assessing outcomes after projects had
been developed, in order to help improve the selection and monitoring of
future projects. Moreover, we testified that the agency's problems in
modernizing its systems resulted from several root causes, including the
agency's attempt to undertake this modernization without the benefit of a
complete NAS architecture to guide its efforts. We concluded that the
agency would continue to experience problems in deploying new systems
until it had fully implemented solutions that addressed these root causes
of its modernization problems and strengthened controls over its
modernization investments.
In February16 and October 2003,17 we testified that FAA had taken steps to
improve the management of its air traffic control modernization, but that
systemic management issues, including inadequate management controls and
human capital issues, were contributing to the continued cost overruns,
schedule delays, and performance shortfalls that major air traffic
projects have consistently experienced. We stated that to overcome these
problems, FAA would need to, among other things, improve its software
capabilities by requiring that all systems achieve a minimum level of
progress before they would be funded, and improve its cost estimating and
cost accounting practices by incorporating actual costs from related
system development efforts in its processes for estimating the costs of
new projects. We testified that until these issues had been resolved,
resources would not be spent cost-effectively, and improvements in
capacity and efficiency would be delayed.
FAA's Enterprise Architecture Program: A Brief Description
According to FAA, its enterprise architecture initiative is intended to
influence the agency's ongoing initiatives in E-Government, data
management, information systems security, capital planning, investment
analysis, and air traffic control and navigation and is to benefit the
agency by aligning business processes with IT processes; improving flight
safety; reducing the development and maintenance costs of systems;
decreasing airline delays; guiding IT investments; and improving the
security, interoperability, and data usage of these systems. FAA officials
told us that
16GAO, National Airspace System: Reauthorizing FAA Provides Opportunities
and Options to Address Changes, GAO-03-473T (Washington, D.C.: Feb. 12,
2003).
17GAO, Air Traffic Control: FAA's Modernization Efforts-Past, Present, and
Future, GAO-04-227T (Washington, D.C.: Oct. 30, 2003).
the agency plans by April 2006 to have a comprehensive version of its
enterprise architecture to guide and constrain the agency's investment
decisions.
The Assistant Administrator for Information Services, who is the agency's
CIO, has been assigned responsibility for developing and maintaining the
agency's enterprise architecture. The CIO has designated a program
director to oversee this effort. Two project offices are responsible for
developing the NAS and non-NAS segments of the enterprise architecture,
respectively, in coordination with the program director. Brief
descriptions of the NAS and non-NAS architecture projects are provided
below.
NAS Architecture Project According to FAA, the NAS architecture is
intended to be the agency's comprehensive plan for improving NAS
operations through the year 2015 and is to address how FAA will replace
aging equipment and introduce new systems, capabilities, and procedures.
The NAS architecture, which FAA reports is being developed in
collaboration with the aviation community, is intended to achieve several
objectives. For example, it is to (1) ensure that the NAS can handle
future growth in aviation without disrupting critical aviation services,
(2) improve flight safety and the use of airspace, (3) decrease airline
delays, and (4) improve systems integration and investment planning.
The agency is developing the NAS architecture in a series of incremental
versions. It released the first version of the NAS architecture in
September 1995. In 1999, FAA released Version 4.0 of this architecture,
which, according to the agency, was the first version to include a 15 to
20-year view (a "To Be" view) and support budget forecasts. According to
FAA, the current version of the NAS architecture (Version 5.0) shows how
the agency intends to achieve the target system described by 2015.
The chief operating officer (COO) for the Air Traffic Organization is
responsible for developing and implementing the NAS segment of the
architecture. The COO has tasked the Operations Planning/Systems
Engineering group within FAA's Air Traffic Organization with the
day-to-day activities involved in this effort. This group is headed by the
Vice President for Operations Planning, who reports directly to the COO.
The COO has also designated a chief architect, who reports to the Director
of Systems Engineering, to develop and maintain the NAS architecture and
to provide technical leadership and guidance, as necessary, to support
investment decision making. The Operations Planning/Systems Engineering
group
receives input from several FAA organizations, but primarily from business
units within the Air Traffic Organization.
Non-NAS Architecture Project According to FAA, the non-NAS architecture
will cover the agency's administrative services and mission support
activities-the process areas, data, systems, and technology that support
such functions as budget and finance, as well as all of the other
governmental air transportation missions and functions that are unique to
the agency (e.g., certification of aircraft). FAA initiated a project to
develop the non-NAS architecture in March 2002 and, according to FAA, the
agency plans to have, by January 2005, an initial baseline architecture
that will describe the "As Is" and "To Be" environments. According to FAA,
it plans to incrementally build on this baseline and have a version of the
non-NAS architecture by April 2006 that will also include a sequencing
plan.
According to FAA, the Information Management Division within the Office of
Information Services/CIO is responsible for developing and maintaining the
non-NAS architecture. FAA has designated a chief architect, who reports to
the program director, to oversee the day-to-day program activities for
developing and maintaining the non-NAS architecture. To develop the
non-NAS architecture, this division will receive input from the agency's
twelve staff offices and four lines of business.
FAA Has Yet to Establish Key Architecture Development, Maintenance, and
Implementation Processes
FAA recognizes the need for and has begun to develop an enterprise
architecture; however, it has yet to establish key architecture management
capabilities that it will need to effectively develop, maintain, and
implement the architecture. As previously stated, the agency has set up
two separate project offices and tasked each with developing one of the
two architecture segments (NAS and non-NAS) that together are to compose
FAA's enterprise architecture. The agency also reports that it has
allocated adequate resources to these project offices and that chief
architects have been assigned to head the architecture projects. However,
FAA has not established other key architecture management capabilities,
such as designating a committee or group representing the enterprise to
direct, oversee, or approve the architecture effort; having an approved
policy for developing, maintaining, and implementing the architecture; and
fully developing architecture products that meet contemporary guidance and
describe both the "As Is" and "To Be" environments and a sequencing plan
for transitioning between the two.
According to FAA officials, attention to and oversight of the enterprise
architecture program have been limited in the past, and the agency has not
documented its architecture management policies, procedures, and
processes; but this is changing. For example, by the end of this fiscal
year, FAA plans to issue a policy governing its enterprise architecture
efforts and to establish a steering committee to guide and direct the
program. By April 2005, the agency also plans to approve an architecture
project management plan for the non-NAS architecture. In addition, it
plans to have a framework for developing the NAS architecture by September
2005.
Based on our experience in reviewing other agencies, not having an
effective enterprise architecture program is attributable to, among other
things, limited senior management understanding and commitment and
cultural resistance to having and using an architecture. The result is an
inability to implement modernized systems in a way that minimizes overlap
and duplication and maximizes integration and mission support.
FAA Has Yet to Implement Key Best Practices for Managing Its NAS
Architecture Project
As we first reported in 1997, it is critical that FAA have and use a
comprehensive NAS architecture to guide and constrain its air traffic
control system investment decisions. To effectively develop, maintain, and
implement this architecture, FAA will need to employ rigorous and
disciplined architecture management practices. Such practices form the
basis of our architecture management maturity framework; the five maturity
stages of our Version 1.1 framework are described in table 3. Some of
these key practices or core elements associated with each of the stages
are summarized below. For additional information on these key practices or
core elements, see the framework.
For stage 2, our framework specifies nine key practices or core elements
that are necessary to provide the management foundation for successfully
launching and sustaining an architecture effort. Examples of stage 2 core
elements are described below.
o Establish a committee or group, representing the enterprise, that is
responsible for directing, overseeing, or approving the enterprise
architecture. This committee should include executive-level
representatives from each line of business, and these representatives
should have the authority to commit resources and enforce decisions within
their respective organizational units. By establishing this enterprisewide
responsibility and accountability, the agency
demonstrates its commitment to building the management foundation and
obtaining buy-in from across the organization.
o Appoint a chief architect. The chief architect should be responsible
and accountable for the enterprise architecture, supported by the
architecture program office, and overseen by the architecture steering
committee. The chief architect, in collaboration with the CIO, the
architecture steering committee, and the organizational head is
instrumental in obtaining organizational buy-in for the enterprise
architecture, including support from the business units, as well as in
securing resources to support architecture management functions such as
risk management, configuration management, quality assurance, and security
management.
o Use a framework, methodology, and automated tool to develop the
enterprise architecture. These elements are important because they provide
the means for developing the architecture in a consistent and efficient
manner. The framework provides a formal structure for representing the
enterprise architecture, while the methodology is the common set of
procedures that the enterprise is to follow in developing the architecture
products. The automated tool serves as a repository where architectural
products are captured, stored, and maintained.
o Develop an architecture program management plan. This plan specifies
how and when the architecture is to be developed. It includes a detailed
work breakdown structure, resource estimates (e.g., funding, staffing, and
training), performance measures, and management controls for developing
and maintaining the architecture. The plan demonstrates the organization's
commitment to managing architecture development and maintenance as a
formal program.
Our framework similarly identifies key architecture management practices
associated with later stages of architecture management maturity. For
example, at stage 3-the stage at which organizations focus on architecture
development activities-organizations need to satisfy six core elements.
Examples of these core elements are discussed below.
o Issue a written and approved organization policy for development of the
enterprise architecture. The policy defines the scope of the architecture,
including the requirement for a description of the baseline and target
architectures, as well as an investment road map or sequencing plan
specifying the move between the two. This policy is an
important means for ensuring enterprisewide commitment to developing an
enterprise architecture and for clearly assigning responsibility for doing
so.
o Ensure that enterprise architecture products are under configuration
management. This involves ensuring that changes to products are
identified, tracked, monitored, documented, reported, and audited.
Configuration management maintains the integrity and consistency of
products, which is key to enabling effective integration among related
products and for ensuring alignment between architecture artifacts.
At stage 4, during which organizations focus on architecture completion
activities, organizations need to satisfy eight core elements. Examples of
these core elements are described below.
o Ensure that enterprise architecture products and management processes
undergo independent verification and validation. This core element
involves having an independent third party-such as an internal audit
function or a contractor that is not involved with any of the architecture
development activities-verify and validate that the products were
developed in accordance with architecture processes and product standards.
Doing so provides organizations with needed assurance of the quality of
the architecture.
o Ensure that business, performance, information/data,
application/service, and technology descriptions address security. An
organization should explicitly and consistently address security in its
business, performance, information/data, application/service, and
technology architecture products. Because security permeates every aspect
of an organization's operations, the nature and substance of
institutionalized security requirements, controls, and standards should be
captured in the enterprise architecture products.
At stage 5, during which the focus is on architecture maintenance and
implementation activities, organizations need to satisfy eight core
elements. Examples of these core elements are described below.
o Make the enterprise architecture an integral component of the IT
investment management process. Because the road map defines the IT systems
that an organization plans to invest in as it transitions from the "As Is"
to the "To Be" environment, the enterprise architecture is a critical
frame of reference for making IT investment decisions. Using the
architecture when making such decisions is important because organizations
should approve only those investments that move the organization toward
the "To Be" environment, as specified in the road map.
o Measure and report return on enterprise architecture investment. Like
any investment, the enterprise architecture should produce a return on
investment (i.e., a set of benefits), and this return should be measured
and reported in relation to costs. Measuring return on investment is
important in order to ensure that expected benefits from the architecture
are realized and to share this information with executive decision makers,
who can then take corrective action to address deviations from
expectations.
Table 4 summarizes our framework's five stages and all of the associated
core elements for each.
Table 4: GAO's Framework for Enterprise Architecture (EA) Management Maturity
(Version 1.1)
Stage Core elements
Stage 1: Creating EA awareness Agency is aware of EA.
Stage 2: Building the EA management Adequate resources exist. foundation
Committee or group representing the enterprise is responsible for
directing, overseeing, or approving EA.
Program office responsible for EA development and maintenance exists.
Chief architect exists.
EA is being developed using a framework, methodology, and automated tool.
EA plans call for describing the "As Is" and "To Be" environments and a
sequencing plan.
EA plans call for describing the enterprise in terms of business,
information/data, application/service, and technology.
EA plans call for business, performance, information/data,
application/service, and technology descriptions to address security.
EA plans call for developing metrics for measuring EA progress, quality,
compliance, and return on investment.
Stage 3: Developing EA products Written and approved organization policy
exists for EA development. (includes all elements from stage 2)
EA products are under configuration management.
EA products describe or will describe the enterprise's business,
performance, information/data, application/service, and the technology
that supports them.
(Continued From Previous Page)
Stage Core elements
EA products describe or will describe the "As Is" and the "To Be"
environments and a sequencing plan.
Business, performance, information/data, application/service, and
technology descriptions address or will address security.
Progress against EA plans is measured and reported.
Stage 4: Completing EA products Written and approved organization policy
exists for EA maintenance. (includes all elements from stage 3)
EA products and management processes undergo independent verification and
validation.
EA products describe the "As Is" and the "To Be" environments and a
sequencing plan.
EA products describe the enterprise's business, performance,
information/data, application/service, and the technology that supports
them.
Business, performance, information/data, application/service, and
technology descriptions address security.
Organization's chief information officer has approved current version of
EA.
Committee or group representing the enterprise or the investment review
board has approved current version of EA.
Quality of EA products is measured and reported.
Stage 5: Leveraging the EA to manage Written and approved policy exists
for IT investment compliance with EA.
change
(includes all elements from stage 4)
Process exists to formally manage EA change.
EA is integral component of IT investment management process.
EA products are periodically updated.
IT investments comply with EA.
Organization head has approved current version of EA.
Return on EA investment is measured and reported.
Compliance with EA is measured and reported. Source: GAO.
For its NAS architecture project, FAA is currently at stage 1 of our
maturity framework. The NAS project office has satisfied three of the core
elements associated with "building the enterprise architecture management
foundation"-stage 2 of our framework--and three of the elements associated
with "developing enterprise architecture products"-stage 3 of our
framework. It has not satisfied other stage 2 and 3 core elements or any
core elements associated with stages 4 and 5. According to the framework,
effective architecture management is generally not achieved until an
enterprise has a completed and approved architecture that is being
effectively maintained and is being used to leverage organizational change
and support investment decision making; having these characteristics is
equivalent to having satisfied all of the stage 3 core elements and many
of the stage 4 and 5 elements.
For the stage 2 core elements, FAA reports that it has allocated adequate
resources for developing a NAS architecture. Further, it has established a
project office that is responsible for architecture development and
maintenance and has assigned a chief architect to the project. However,
the agency has not satisfied other core elements for stage 2, such as
assigning responsibility for directing, overseeing, or approving the
architecture to a committee or group representing the enterprise. Without
such an entity to lead and be accountable for the architectural effort,
there is increased risk that the architecture will not represent a
corporate decision-making tool and will not be viewed and endorsed as an
agencywide asset.
With respect to stage 3, according to the CIO, FAA plans to build on the
current version of the NAS architecture (Version 5.0) to ensure that
architecture products are developed that meet contemporary guidance and
standards. According to FAA officials, including the CIO and the chief
scientist for the NAS project office, the current NAS architecture does
not conform to contemporary architecture guidance or standards-including
OMB's FEA reference models and GAO's enterprise architecture management
maturity framework-because it predates them and has not been updated to
comply with them. However, the CIO stated that future versions of the
architecture will conform to this guidance. Among other things, this
guidance calls for products that describe the "As Is" and "To Be"
business, performance, information/data, applications/services,
technology, and security environments as well as a sequencing plan for
transitioning from the "As Is" to the "To Be" states. Other stage 3 core
elements nevertheless have not been met, such as having a written and
approved architecture development policy. Further, none of the stage 4 and
5 core elements have been met, although the CIO stated that FAA has
recently begun to take steps associated with meeting some of these core
elements. The detailed results of our assessment of the NAS project
office's progress in implementing the core elements associated with the
five maturity stages are provided in appendix II.
In addition, FAA's senior enterprise architecture officials, including the
program director, stated that attention to and oversight of the enterprise
architecture program have been limited in the past and that the agency has
not documented its architecture management policies, procedures, and
processes. These officials stated that the agency recognizes the need to
establish an effective NAS architecture project and that it intends to do
so.
To this end, FAA currently plans to have, by September 2005, a framework
for developing the architecture and an approved enterprise architecture
policy requiring the development, maintenance, and implementation of an
enterprise architecture. The CIO also stated that the agency plans to
update its NAS architecture to reflect current architecture standards and
guidance.
Our research of successful organizations and our experience in reviewing
other agencies' enterprise architecture efforts show that not having these
controls is, among other things, a function of limited senior management
understanding of and commitment to an enterprise architecture and cultural
resistance to having and using one. Until such barriers are addressed and
effective architecture management structures and processes are
established, it is unlikely that an agency will be able to produce and
maintain a complete and enforceable architecture and thus implement
modernized systems in a way that minimizes overlap and duplication and
maximizes integration and mission support. Given the size and complexity
of FAA's air traffic control systems and their importance to FAA's ability
to achieve its mission, it is critical that FAA develop a well-defined
architecture that can be used to guide and constrain system investment
decisions.
FAA Has Yet to Implement Key Best Practices for Managing Its Non-NAS
Architecture Project
Similar to its NAS architecture effort, FAA's attempt to develop,
maintain, and implement its non-NAS architecture needs to be grounded in
the kind of rigorous and disciplined management practices embodied in
Version 1.1 of our architecture management maturity framework. (Tables 3
and 4 provide a description of the framework's five maturity stages and
the key practices or core elements associated with each stage.)
For its non-NAS architecture project, FAA is currently at stage 1 of our
maturity framework. The non-NAS project office has satisfied three of the
core elements associated with "building the enterprise architecture
management foundation"-stage 2 of our framework-and four of the core
elements associated with stages 3 and 5. According to the framework,
effective architecture management is generally not achieved until an
enterprise has a completed and approved architecture that is being
effectively maintained and is being used to leverage organizational change
and support investment decision making; having these characteristics is
equivalent to having satisfied all of the stage 2 and 3 core elements and
many of the stage 4 and 5 elements.
For stage 2 core elements, FAA reports that it has allocated adequate
resources, and it has established a project office and assigned a chief
architect. However, the agency has not satisfied several of the stage 2
core elements that are critical to effective architecture management. For
example, the agency has not established a committee or group representing
the enterprise to guide, direct, or approve the architecture. Having such
a corporate entity is critical to overcoming cultural resistance to using
an enterprise architecture. As previously stated, the absence of such an
entity increases the risk that the architecture will not represent a
corporate decision-making tool and will not be viewed and endorsed as an
agencywide asset.
Concerning stage 3, FAA has not satisfied three of the six core elements.
For example, although the agency is developing architecture products, it
does not have a written and approved policy for architecture development.
Without such a policy, which, for example, identifies the major players in
the development process and provides for architecture guidance, direction,
and approval, FAA will be challenged in overcoming cultural resistance to
using an enterprise architecture and achieving agencywide commitment and
support for an architecture.
The agency has not implemented any of the stage 4 core elements, but it
has implemented one core element-architecture products are periodically
updated-associated with stage 5 of our framework. For example, FAA has not
(1) documented and approved a policy for architecture implementation, (2)
implemented an independent verification and validation function that
covers architecture products and architecture management processes, and
(3) made the architecture an integral component of its IT investment
management process. The detailed results of our assessment of the non-NAS
project office's progress in implementing the core elements associated
with the five maturity stages are provided in appendix III.
According to FAA's senior enterprise architecture officials, including the
chief architect, the attention to and oversight of the enterprise
architecture program have been limited in the past, and the agency has not
documented its architecture management policies, procedures, and
processes. FAA officials, including the CIO and the chief architect for
the non-NAS project, agreed with our assessment of the project office's
current architecture management capabilities. These officials stated that
the agency recognizes the need to establish an effective non-NAS
architecture project, and it intends to do so. For example, the agency's
strategic plan includes the goal
of having an approved enterprise architecture policy requiring the
development, maintenance, and implementation of an enterprise architecture
by September 2005, and the agency intends to establish a steering
committee. In addition, the chief architect stated that FAA plans to have
an approved architecture project management plan by April 2005, and a
comprehensive version of the non-NAS architecture by April 2006.
As previously stated, our research and our experience show that not having
these controls is, among, other things, attributable to limited senior
management understanding of and commitment to an enterprise architecture
and cultural resistance to having and using one. Until such barriers are
addressed and effective architecture management structures and processes
are established, it is unlikely that any agency will be able to develop
and maintain a complete and enforceable architecture and thus implement
modernized systems in a way that minimizes overlap and duplication and
maximizes integration and mission support.
Conclusions Having a well-defined and enforced enterprise architecture is
critical to FAA's ability to effectively and efficiently modernize its NAS
and non-NAS systems. To accomplish this, it is important for FAA to
establish effective management practices for developing, maintaining, and
implementing an architecture. Currently, FAA does not have these practices
in place. Establishing them begins with agency top management commitment
and support for having and using an architecture to guide and constrain
investment decision making.
Recommendations for Executive Action
To ensure that FAA has the necessary agencywide context within which to
make informed decisions about its air traffic control system and other
systems modernization efforts, we recommend that the Secretary of the
Department of Transportation direct the FAA Administrator to ensure that
the following four actions take place.
o Demonstrate institutional commitment to and support for developing and
using an enterprise architecture by issuing a written and approved
enterprise architecture policy.
o Ensure that the CIO, in collaboration with the COO, implements, for the
NAS architecture project, the best practices involved in stages 2 through
5 of our enterprise architecture management maturity framework.
o Ensure that the CIO focuses first on developing and implementing a NAS
architecture.
o Ensure that the CIO implements, for the non-NAS architecture project,
the best practices involved in stages 2 through 5 of our enterprise
architecture management maturity framework.
Agency Comments In commenting on a draft of this report, the Department of
Transportation's Director of Audit Relations stated via e-mail that FAA is
continuing its NAS architecture efforts. The Director also provided
technical comments, which we have incorporated as appropriate in the
report. The Director's comments did not state whether the department
agreed or disagreed with the report's conclusions and recommendations.
As agreed with your office, unless you publicly announce the contents of
this report earlier, we plan no further distribution until 30 days from
the report date. At that time, we will send copies to interested
congressional committees, the Director of OMB, the Secretary of the
Department of Transportation, the FAA Administrator, FAA's CIO, and FAA's
COO. We will also make copies available to others upon request. In
addition, the report will be available at no charge on the GAO Web site at
http://www.gao.gov.
If you or your staff have any questions on matters discussed in this
report,
please contact Randolph C. Hite at (202) 512-3439 or [email protected], or
David A. Powner at (202) 512-9286 or [email protected]. Major
contributors to this report are acknowledged in appendix IV.
Randolph C. Hite
Director
Information Technology Architecture and Systems Issues
David A. Powner
Director
Information Technology Management Issues
Appendix I
Objective, Scope, and Methodology
Our objective was to determine whether the Federal Aviation Administration
(FAA) has established effective processes for managing the development and
implementation of an enterprise architecture.
To address our objective, we used our enterprise architecture management
maturity framework, Version 1.1,1 which organizes architecture management
best practices into five stages of management maturity. Specifically, we
compared our framework to the ongoing efforts of FAA's two project offices
to develop the National Airspace System (NAS) and non-NAS segments of the
architecture. For example, for the NAS architecture, we reviewed program
documentation, such as the acquisition management system policy, the Joint
Resources Council's investment management guidance for NAS investments,
and FAA's NAS architecture development process. We reviewed, for the
non-NAS architecture, program documentation, such as the methodology FAA
is using to develop this architecture, a Systems Research and Applications
report2 on the agency's efforts to implement management processes and
controls over its architecture development activities, and the Department
of Transportation's Enterprise Architecture Subcommittee and Architecture
Review Board charters. We then compared these documents with the elements
in our framework.
To augment our documentation reviews of FAA's architecture management
efforts, we interviewed various officials, including the chief information
officer, the program director, the chief architects for the NAS and
non-NAS architectures, and the chief scientist for the NAS architecture,
to determine, among other things, the agency's plans to develop an
enterprise architecture. Specifically, we inquired about (1) the agency's
plans for developing an enterprise architecture, including the key
milestones and deliverables for completing the two segments of the
architecture, (2) the content of the NAS and non-NAS architecture segments
(i.e., architecture products that have been developed to date), and (3)
the strategy to be used to align the NAS and non-NAS architectures.
1GAO-03-584G.
2Systems Research and Applications International, Inc., Independent
Validation and Verification Final Report, Version 1.0, Jan. 30, 2004.
Appendix I Objective, Scope, and Methodology
We conducted our work at FAA headquarters in Washington, D.C. We performed
our work from June 2004 to March 2005, in accordance with generally
accepted government auditing standards.
Appendix II
Assessment of Architecture Management
Efforts for the National Airspace System
Stage Core element Satisfied? Comments
Agency is aware The Federal Aviation
Stage 1: Creating of EA. Yes Administration (FAA) strategic
plan
enterprise architecture includes the goal of having an
approved EA policy
(EA) awareness requiring the development,
maintenance, and
implementation of an EA by
September 2005.
Stage 2: Building the Adequate resources exist (funding, EA management
people, tools, and technology). foundation
Yes According to the chief scientist and the chief architect for the NAS
architecture, the agency has adequate project funding. FAA reports that
fiscal year 2005 funding for the National Airspace System (NAS)
architecture is about $2.6 million. In addition, the agency reports that
it has skilled staff, including contractor support, for its NAS
architecture project. Furthermore, FAA is using automated tools and
technology, such as Rational Rose by Rational Software Corporation/IBM
Software Group, CORE by Vitech Corporation, and Dynamic Object Oriented
Requirements System by Telelogic.
Committee or group representing the enterprise is responsible for
directing, overseeing, or approving the EA.
No FAA has not assigned responsibility for directing, overseeing, or
approving a NAS architecture to a group or committee representing the
enterprise.
Program office responsible for EA development and maintenance exists.
Yes In 1997, FAA established a project office that is responsible for
developing and maintaining a NAS architecture.
Chief architect exists. Yes In February 2004, FAA designated the chief
architect for the NAS architecture project.
EA is being developed using a No According to the chief scientist, the NAS
architecture is
framework, methodology, and automated being developed using a framework
that focuses on
tool. strategically supporting FAA's investment management process. The
chief scientist stated that, unlike other architecture frameworks, this
framework is not fully developed or documented. Further, FAA has yet to
provide us with any documentation on this framework and on how it is being
implemented to support the agency's investment management process.
According to the chief information officer (CIO), the agency plans to
select an architecture framework by September 2005. FAA does not have a
methodology that defines the standards, steps, tools, techniques, and
measures that it is following to develop, maintain, and validate a NAS
architecture. As stated above, FAA is using automated tools to build a NAS
architecture.
EA plans call for describing both the "As No FAA has yet to develop
architecture project management
Is" and the "To Be" environments of the plans.
enterprise, as well as a sequencing plan
for transitioning from the "As Is" to the
"To Be."
Appendix II Assessment of Architecture Management Efforts for the National
Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
EA plans call for describing both the "As No FAA has yet to develop
architecture project management
Is" and the "To Be" environments in plans.
terms of business, performance,
information/data, application/service,
and technology.
EA plans call for business, performance, No FAA has yet to develop
architecture project management
information/data, application/service, plans.
and technology descriptions to address
security.
EA plans call for FAA has yet to develop
developing metrics for No architecture project
management
measuring EA progress, plans and metrics for
quality, measuring NAS architecture
compliance, and return on progress, quality, compliance,
investment. and return on investment.
Stage 3: Written/approved No FAA has yet to develop a
Developing organization policy written/approved policy for
EA products exists for EA development. developing a NAS architecture.
(includes However, FAA's strategic
all elements plan includes the goal of
from developing an approved
stage 2) enterprise architecture policy
by September 2005.
EA products are under No FAA has yet to establish a
configuration configuration management
management. process.
According to the chief
EA products describe or Yes information officer (CIO),
will describe future
both the "As Is" and the versions of the NAS
"To Be" architecture will conform to
environments of the contemporary guidance. Such
enterprise, as well guidance describes,
as a sequencing plan for among other things, products
transitioning that describe the "As Is"
from the "As Is" to the and "To Be" environments and a
"To Be." sequencing plan.
Both the "As Is" and the Yes According to the CIO, future
"To Be" versions of the NAS
environments are described architecture will conform to
or will be contemporary guidance.
described in terms of Such guidance describes, among
business, other things, products
performance, that describe the "As Is" and
information/data, "To Be" environments in
application/service, and terms of business,
technology. performance, information/data,
application/service, and
technology.
Business, performance, information/data, application/service, and
technology descriptions address or will address security.
Yes According to the CIO, future versions of the NAS architecture will
conform to contemporary guidance. Such guidance includes, among other
things, business, performance, information/data, application/service, and
technology descriptions that address security in both the "As Is" and "To
Be" environments.
Progress against EA plans is measured No FAA has yet to develop architecture
project management and reported.
plans and metrics; therefore, progress against plans is not measured and
reported.
Stage 4: Written/approved No FAA has yet to develop a
Completing organization policy written/approved policy for
EA products exists for EA maintaining a NAS architecture.
(includes maintenance. However, FAA's
all elements from strategic plan includes the goal
of developing an
stage 3) approved enterprise architecture
policy by September
2005.
EA products and No FAA has yet to establish an
management independent verification and
processes undergo validation process.
independent
verification and
validation.
Appendix II Assessment of Architecture Management Efforts for the National
Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
EA products describe both the "As Is" No FAA has yet to develop NAS
architecture products that
and the "To Be" environments of the describe both the "As Is" and the "To
Be" environments
enterprise, as well as a sequencing plan and a sequencing plan.
for transitioning from the" As Is" to the
"To Be."
Both the "As Is" and the "To Be" No FAA has yet to develop NAS
architecture products that environments are described in terms of describe
both the "As Is" and the "To Be" environments business, performance,
information/data, in terms of business, performance, information/data,
application/service, and technology. application/service, and technology.
Business, performance, No FAA has yet to develop business,
performance,
information/data, information/data, application/service, and
application/service, technology
and technology descriptions descriptions that address security in both
address the "As Is" and
security. "To Be" environments.
Organization CIO has approved current No FAA has yet to develop a version
of the NAS architecture
version of EA. for the CIO to approve that conforms to contemporary
guidance and standards.
Committee or group representing No FAA has yet to develop a version of the
the NAS architecture
enterprise or the investment that conforms to contemporary guidance
review and standards
board has approved current for a committee or investment review
version of board to approve.
EA.
Quality of EA products is measured and No FAA has yet to develop NAS
architecture product
reported. metrics; therefore, product quality is not measured and
reported.
Stage 5: Leveraging Written/approved organization policy No FAA has yet to
develop a written/approved policy the EA for managing exists for IT
investment compliance with requiring IT investments to comply with a NAS
change (includes all EA. architecture. However, FAA's strategic plan
includes the elements from stage 4) goal of developing an approved
enterprise architecture
policy by September 2005.
Process exists to formally manage EA No FAA has yet to establish a formal
process for managing change. changes to a NAS architecture.
EA is integral component of IT No According to the CIO, FAA has recently
begun to
investment management process. consider architecture compliance as part
of its Joint
Resources Council process and the CIO's
approval of
Exhibit 300 budget exhibits for NAS
investments, and he
anticipates that over the next couple of
years the NAS
architecture will become integral to the
investment
process.
EA products are periodically updated. No FAA has yet to complete
development of NAS architecture products.
IT investments comply with EA. No According to the CIO, FAA has recently
begun to consider investment compliance with the architecture, and the CIO
expects this compliance determination to expand and evolve over the next
couple of years.
Organization head has approved current No FAA has yet to complete
development of a NAS version of EA. architecture for the Administrator to
approve.
Appendix II Assessment of Architecture Management Efforts for the National
Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
Return on EA investment is No FAA has yet to develop metrics and
measured processes for
and reported. measuring NAS architecture benefits;
therefore, return
on investment is not measured and
reported.
Compliance with EA is measured No FAA has yet to develop metrics for
and measuring
reported. compliance with the NAS architecture;
therefore,
compliance with an architecture is not
measured and
reported.
Source: GAO analysis of FAA data.
Appendix III
Assessment of Architecture Management Efforts for the Non-National
Airspace System
Stage Core element Satisfied? Comments
Stage 1: Creating Agency is aware Yes The Federal Aviation Administration
of EA. (FAA) strategic
enterprise plan includes the goal of having an
architecture approved EA policy
(EA) awareness requiring the development,
maintenance, and
implementation of an EA by
September 2005.
Stage 2: Building the Adequate resources exist (funding, EA management
people, tools, and technology). foundation
Yes According to the chief architect, the agency has adequate project
funding. FAA reports that fiscal year 2005 funding for the non-National
Airspace System (NAS) architecture is $1.5 million. In addition, the
agency reports that it has skilled staff (two government employees, six
full-time contractors, and additional contractor staff as needed) working
to develop its non-NAS architecture. FAA is also using automated tools,
such as Rational Rose by Rational Software Corporation/IBM Software Group,
Microsoft Visio and an Oracle portal server.
Committee or group representing No FAA has not assigned responsibility
the for directing,
enterprise is responsible for overseeing, or approving the non-NAS
directing, architecture to
overseeing, or approving the EA. any group or committee. According to
the chief
architect, FAA plans to assign
responsibility for
directing the non-NAS architecture to
its Information
Technology Executive Board by April
2005.
Program office responsible for Yes In January 2003, FAA established a
EA project office that is
development and maintenance responsible for developing and
exists. maintaining the non-
NAS architecture.
Chief architect exists. Yes In January 2003, FAA designated a chief
architect for
the non-NAS architecture.
EA is being developed using a framework, methodology, and automated tool.
No According to the chief architect, FAA is using the Federal Enterprise
Architecture Framework and the Office of Management and Budget's Federal
Enterprise Architecture reference models to develop the non-NAS
architecture. FAA also has a methodology for developing the architecture,
but the methodology does not define the standards, steps, tools,
techniques, and measures that it is following to develop, maintain, and
validate the non-NAS architecture. However, according to the chief
architect, FAA will update its methodology to describe management
activities by March 2005. As stated above, FAA is using automated tools to
build the non-NAS architecture.
EA plans call for describing both the "As No FAA has yet to develop
architecture project
Is" and the "To Be" environments of the management plans. However, the
chief architect stated
enterprise, as well as a sequencing plan that the agency intends to have
an approved plan by
for transitioning from the "As Is" to the April 2005 and the plan will
call for describing both the
"To Be." "As Is" and the "To Be" environments of the enterprise,
as well as a sequencing plan.
Appendix III Assessment of Architecture Management Efforts for the
Non-National Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
EA plans call for describing both the "As No FAA has yet to develop
architecture project
Is" and the "To Be" environments in management plans. However, the chief
architect stated
terms of business, performance, that the agency intends to have an
approved plan by
information/data, application/service, April 2005 and the plan will call
for describing both the
and technology. "As Is" and the "To Be" environments in terms of
business, performance, information/data, application/service, and
technology.
EA plans call for business, performance, No FAA has yet to develop
architecture project information/data, application/service, management
plans. However, the chief architect stated and technology descriptions to
address that the agency intends to have an approved plan by security.
April 2005 and the plan will call for the business,
performance, information/data, application/service, and
technology descriptions to address security for both the
"As Is" and "To Be" environments.
EA plans call for developing No FAA has yet to develop architecture
metrics for project
measuring EA progress, quality, management plans and metrics.
According to the chief
compliance, and return on architect, the agency intends to have
investment. an approved plan
by April 2005 and the plan will
include metrics for
measuring non-NAS architecture
progress, quality,
compliance, and return on investment.
Stage 3: Written/approved No FAA has yet to develop a
Developing organization policy written/approved policy for
EA products exists for EA developing the non-NAS
(includes development. architecture. However, FAA's
all elements from strategic plan includes the goal
of developing an
stage 2) approved enterprise architecture
policy by September
2005.
EA products are under No FAA has yet to develop non-NAS architecture
configuration products
and a configuration management process has not
management. been
established. However, according to the chief
architect,
FAA plans to update its methodology to address
how
changes to all architecture products will be
documented
by March 2005.
EA products describe or will Yes The chief architect stated that the
describe agency intends to
both the "As Is" and the "To Be" have an approved plan by April 2005
and that the plan
environments of the enterprise, will call for describing both the "As
as well Is" and the "To Be"
as a sequencing plan for environments of the enterprise, as
transitioning well as a
from the "As Is" to the "To Be." sequencing plan. The chief architect
also stated that
the agency will have a comprehensive
non-NAS
architecture by April 2006.
Both the "As Is" and the "To Be" Yes The chief architect stated that the
agency intends to
environments are described or have an approved plan by April 2005
will be and that the plan
described in terms of business, will call for describing both the
"As Is" and the "To Be"
performance, information/data, environments in terms of business,
performance,
application/service, and information/data,
technology. application/service, and technology.
The chief architect also stated that
the agency will have
a comprehensive non-NAS architecture
by April 2006.
Appendix III Assessment of Architecture Management Efforts for the
Non-National Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
Business, performance, Yes The chief architect stated that the agency
intends to information/data, application/service, have an approved plan by
April 2005 and that the plan and technology descriptions address or will
call for the business, performance, information/data, will address
security. application/service, and technology descriptions to
address security for both the "As Is" and "To Be" environments. The chief
architect also stated that the agency will have a comprehensive non-NAS
architecture by April 2006.
Progress against EA plans is measured No FAA has yet to develop
architecture project
and reported. management plans and metrics; therefore, progress against
plans is not measured and reported. However, according to the chief
architect, the agency intends to have an approved plan by April 2005 and
progress against the plan will be measured and reported.
Stage 4: Written/approved No FAA has yet to develop a
Completing organization policy written/approved policy for
EA products exists for EA maintaining the non-NAS
maintenance. architecture. However, FAA's
(includes all strategic plan includes the goal of
elements developing an
from stage 3) approved enterprise architecture
policy by September
2005.
EA products and management No FAA has yet to establish an independent
verification
processes undergo independent and validation process. a However,
according to the
verification and validation. chief architect, the non-NAS architecture
products and
architecture management processes will
undergo
independent verification and validation
by December
2005.
EA products describe both the "As Is" No The current non-NAS architecture
products do not yet and the "To Be" environments of the fully describe
both the "As Is" and the "To Be" enterprise, as well as a sequencing plan
environments of the enterprise, or a sequencing plan. for transitioning
from the" As Is" to the However, according to the chief architect, FAA
will have "To Be." a comprehensive non-NAS architecture that describes
both the "As Is" and the "To Be" environments of the enterprise, as well
as the sequencing plan by April 2006.
Both the "As Is" and the "To Be" No The current non-NAS architecture
products do not yet
environments are described in fully describe both the "As Is" and
terms of the "To Be"
business, performance, environments in terms of business,
performance,
information/data, information/data,
application/service, application/service, and technology.
and technology. However, according to the chief
architect, FAA will have
a comprehensive non-NAS architecture
that describes
these terms by April 2006.
Business, performance, No According to the chief architect, the
non-NAS
information/data, architecture does not yet contain complete
application/service, business,
and technology descriptions performance, information/data,
address application/service, and
security. technology descriptions that address
security for both
the "As Is" and "To Be" environments.
However, FAA
will have a comprehensive non-NAS
architecture
comprised of these terms by April 2006.
Appendix III Assessment of Architecture Management Efforts for the
Non-National Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
Organization chief information No FAA has yet to develop a non-NAS
officer architecture for the
(CIO) has approved current CIO to approve. According to the chief
version of architect, the
EA. first comprehensive version of the
non-NAS
architecture is scheduled for release in
April 2006.
Committee or group representing No FAA has yet to develop a non-NAS
the architecture for the
enterprise or the investment committee or investment review board to
review approve.
board has approved current According to the chief architect, the
version of first comprehensive
EA. version of the non-NAS architecture is
scheduled for
release in April 2006.
In addition, FAA has yet to establish a
committee or
investment review board that will be
responsible for
approving the non-NAS architecture.
Quality of EA products is measured and No FAA has yet to develop metrics
and assess the quality
reported. of the non-NAS architecture products that it is currently
developing; therefore, product quality is not measured and reported.
However, according to the chief architect, the agency intends to have an
approved plan by April 2005 and the plan will include metrics for
measuring the quality of the non-NAS architecture products.
Stage 5: Written/approved No FAA has yet to develop a
Leveraging organization policy written/approved policy
the EA for exists for IT investment requiring that IT investments
managing compliance with comply with the
change EA. architecture. However, FAA's
strategic plan includes the
(includes all goal of developing an approved
elements enterprise architecture
from stage 4) policy by September 2005.
Process exists to formally No FAA has yet to establish a formal process
manage EA for managing
change. changes to the non-NAS architecture.
However,
according to the chief architect, the agency
intends to
have an approved architecture project
management
plan by April 2005 and the plan will include
a formal
process for managing architecture changes.
EA is integral component of IT No FAA has yet to complete development of
a non-NAS
architecture, and it is not an
investment management process. integral component of the
IT investment management process. b
EA products are periodically Yes FAA updates the non-NAS architecture
updated. products
annually to reflect the agency's
investment decisions. b
IT investments comply with EA. No FAA has yet to complete development of a
non-NAS architecture; therefore, IT investments are not evaluated for
compliance with the architecture. However, the first version of the
non-NAS architecture is scheduled for release in April 2006.b
Organization head has approved current No FAA has yet to complete
development of a
version of EA. non-NAS architecture for the Administrator to approve.
However, the first version of the non-NAS architecture is scheduled for
release in April 2006.b
Appendix III Assessment of Architecture Management Efforts for the Non-National
Airspace System
(Continued From Previous Page)
Stage Core element Satisfied? Comments
Return on EA investment is No FAA has yet to develop metrics and processes
measured for
and reported. measuring non-NAS architecture benefits;
therefore,
return on investment is not measured and
reported.
However, the first version of the non-NAS
architecture
is scheduled for release in April 2006.b
Compliance with EA is measured No FAA has yet to develop metrics for
and measuring
reported. compliance with the non-NAS
architecture. However,
the first version of the non-NAS
architecture is
scheduled for release in April 2006.b
Source: GAO analysis of FAA data.
aIn January 2004, Systems Research and Applications (SRA) International,
Inc. reviewed FAA's architecture management processes. The results of this
review were provided to the chief architect, but a corrective action plan
was not developed. According to the chief architect, the weaknesses in the
SRA report will be addressed as the agency establishes additional
architecture management capabilities and refines its investment management
process to include non-NAS investments.
bAccording to the chief architect, these core elements will be addressed
in the enterprise architecture policy that FAA plans to issue by September
2005.
Appendix IV
GAO Staff Acknowledgments
Staff Staff who made key contributions to this report were Kristina
Badali, Joanne Fiorino, Michael Holland, Anh Le, William Wadsworth, and
Angela
Acknowledgments Watson.
GAO's Mission The Government Accountability Office, the audit, evaluation
and investigative arm of Congress, exists to support Congress in meeting
its constitutional responsibilities and to help improve the performance
and accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of The fastest and easiest way to obtain copies of GAO
documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO postsGAO
Reports and newly released reports, testimony, and correspondence on its
Web site. To Testimony have GAO e-mail you a list of newly posted products
every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Congressional Gloria Jarmon, Managing Director, [email protected] (202)
512-4400 U.S. Government Accountability Office, 441 G Street NW, Room 7125
Relations Washington, D.C. 20548
Public Affairs Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
*** End of document. ***