Information Technology: Centers for Medicare & Medicaid Services
Needs to Establish Critical Investment Management Capabilities
(28-OCT-05, GAO-06-12).
To carry out its mission of ensuring health care security for
beneficiaries, the Centers for Medicare & Medicaid Services (CMS)
relies heavily on information technology (IT) systems. In fiscal
year 2005, CMS's total IT appropriations was about $2.55 billion,
of which about $760 million, or 30 percent, was to support
internal investments, and $1.79 billion was to fund the Medicaid
Management Information Systems (MMIS) that states use to support
their Medicaid programs. (GAO is using the term "internal" to
refer to all of CMS' IT investments excluding state MMISs.) In
light of the size and significance of these investments, GAO's
objectives were to (1) evaluate CMS's capabilities for managing
its internal investments, (2) determine any plans the agency
might have for improving these capabilities, and (3) examine
CMS's process for approving and monitoring state MMISs.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-06-12
ACCNO: A40680
TITLE: Information Technology: Centers for Medicare & Medicaid
Services Needs to Establish Critical Investment Management
Capabilities
DATE: 10/28/2005
SUBJECT: Information resources management
Information technology
IT investment management
IT policies
IT standards
Management information systems
Medicaid
Medicare
Program evaluation
Systems design
Systems evaluation
General management reviews
Policies and procedures
Medicaid Management Information System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-06-12
United States Government Accountability Office
GAO
Report to the Chairman, Committee on
Finance, U.S. Senate
October 2005
INFORMATION
TECHNOLOGY
Centers for Medicare & Medicaid Services Needs to Establish Critical Investment
Management Capabilities
a
GAO-06-12
October 2005
INFORMATION TECHNOLOGY
Centers for Medicare & Medicaid Services Needs to Establish Critical Investment
Management Capabilities
What GAO Found
Judged against GAO's framework for IT investment management, which
measures the maturity of an organization's investment management process,
CMS's capabilities for effectively managing its internal investments are
limited. Specifically, the agency has established a little over half of
the foundational practices it needs to manage individual investments (see
figure below) and has executed 2 of the 27 key practices needed to manage
investments as a portfolio. Until CMS fully establishes foundational and
portfolio-level practices, executives will lack the assurance that they
are managing the agency's collection of investments in a manner that
minimizes risks and maximizes returns.
CMS has initiated steps to improve its investment management process;
however, these steps do not fully address the weaknesses GAO identifies in
this report, nor are they coordinated with other needed improvement
efforts into a plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by
senior-level management. Without such a plan and procedures for
implementing it, CMS will be challenged in sustaining the commitment it
needs to fully establish its investment management process.
The process for approving requests for federal funding of MMIS activities
(including development, operations, and maintenance activities) is
characterized by standard procedures, guidance, and reported information
to CMS's Center for Medicaid and State Operations. In contrast, the
process for monitoring MMIS activities lacks standard procedures,
guidance, and reporting requirements. Without these elements for
monitoring MMIS activities, CMS may not be able to easily determine
whether the state MMISs in which CMS invests close to $1.7 billion
annually are facilitating the delivery of Medicaid benefits in the most
effective and beneficial manner.
Foundational Practices Implemented by CMS
Critical process
Instituting the investment board
Key practices executed
Total required by critical process
Percentage of key practices executed
58 Meeting business needs 5 7
Selecting an investment 4 10
Providing investment oversight 1 7
Capturing investment information 5 6 Total 20 38
United States Government Accountability Office
Contents
Letter
Results in Brief
Background
CMS's Capabilities to Manage Its Internal Investments Are
Limited CMS Does Not Have a Comprehensive Plan to Coordinate and Guide Its
Improvement Efforts Process for Monitoring MMISs Could Benefit from
Standard
Procedures, Guidance, and Reporting Requirements Conclusions
Recommendations for Executive Action Agency Comments and Our Evaluation
1 3 5
17
34
35 38 39 41
Appendixes
Appendix I: Objectives, Scope, and Methodology 44
Appendix II: Comments from the Centers for Medicare & Medicaid Services 47
Appendix III: GAO Contacts and Staff Acknowledgments 51
Tables Table 1: Stage 2 Critical Processes-Building the Investment
Foundation 18
Table 2: Instituting the Investment Board 21
Table 3: Meeting Business Needs 24
Table 4: Selecting an Investment 27
Table 5: Providing Investment Oversight 29
Table 6: Capturing Investment Information 32
Table 7: Stage 3 Critical Processes-Developing a Complete
Investment Portfolio 34
Table 8: Frequency of Oversight Mechanisms Used by the 5
Regional Offices Interviewed 37
Figures Figure 1: Distribution of CMS's Information Technology Budget,
Fiscal Year 2005 6
Figure 2: CMS Selection Process for Internal IT Investments 11
Figure 3: The Five ITIM Stages of Maturity with Critical
Processes 17 Figure 4: Summary of Results for Stage 2 Critical Processes
and Key Practices for Internal IT Investments 19
Contents
Abbreviations
APD advance planning document
CIO Chief Information Officer
CMS Centers for Medicare & Medicaid Services
ESC Executive Steering Committee
HHS Department of Health and Human Services
IT information technology
ITIM Information Technology Investment Management framework
ITIRB Information Technology Investment Review Board
MMA Medicare Prescription Drug, Improvement, and Modernization
Act of 2003 MMIS Medicaid Management Information Systems
This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.
A
United States Government Accountability Office Washington, D.C. 20548
October 28, 2005
The Honorable Charles E. Grassley Chairman, Committee on Finance United
States Senate
Dear Mr. Chairman:
The Centers for Medicare & Medicaid Services (CMS), formerly called the
Health Care Financing Administration, within the Department of Health and
Human Services (HHS), is responsible for overseeing the Medicare and
Medicaid programs. In 1990, we designated the Medicare program as
highrisk, in part, because of its sheer size and complexity. Similarly, in
2003, we placed the Medicaid program on our high-risk list, noting the
growing concerns about the quality of fiscal oversight. In our latest
high-risk series, issued in January 2005,1 we continued to designate both
these programs as high risk. While the Medicare program is financed and
administered by the federal government, the Medicaid program is jointly
financed by the federal government and the states and is administered
directly by the states.2
To carry out its responsibilities, CMS depends on hundreds of information
technology (IT) systems to maintain information on Medicare beneficiaries,
providers, and medical services provided as well as to carry out its
oversight of the states' Medicaid programs for low-income Americans. For
example, IT systems support the Medicare program, which enrolls about 41
million elderly and disabled beneficiaries and, in fiscal year 2004, had
estimated outlays of $297 billion in health care benefits. The agency also
provides funding assistance (through grants) to the states to develop and
operate automated systems, known as Medicaid Management Information
Systems (MMIS), to support their Medicaid programs.3 While
1GAO, High-Risk Series: An Update, GAO-05-207 (Washington, D.C.: January
2005).
2Medicaid consists of 56 distinct state-level programs, including 1 for
each of the 50 states; the District of Columbia; Puerto Rico; and the U.S.
territories of American Samoa, Guam, the Northern Mariana Islands, and the
Virgin Islands. Hereafter, these 56 entities are referred to as states.
Within broad federal guidelines, each program establishes its own
eligibility standards; determines the type, amount, duration, and scope of
covered services; and sets payment rates.
3The Medicaid Management Information System is the primary claims
processing and information retrieval system, which states are required to
have.
the responsibility for managing CMS's internal4 IT investments falls to
its Information Technology Investment Review Board, the responsibility for
approving requests for federal funding of state MMIS activities and for
monitoring these activities5 falls to CMS's Center for Medicaid and State
Operations and the 10 regional offices. For fiscal year 2005, CMS's total
IT appropriations was about $2.55 billion, of which about $1.79 billion,
or 70 percent, was to be used to support Medicaid state IT investments.
This report is one of two we prepared in response to your request that we
review HHS's and CMS's IT management processes.6 It focuses on CMS's
processes for making IT investment management decisions and evaluates how
well these processes compare with the accepted practices presented in our
IT Investment Management framework.7 This framework provides a method for
evaluating and assessing how well an agency is selecting and managing its
IT resources. As we agreed with your office, our objectives were to (1)
evaluate CMS's capabilities for managing its internal IT investments, (2)
determine any plans the agency might have for improving these
capabilities, and (3) examine CMS's processes for approving and monitoring
the state MMISs it funds. To address these objectives, we analyzed
documents and interviewed agency officials to (1) validate and update
CMS's self-assessment of key practices in the framework, (2) evaluate the
agency's plans for improving its capabilities, and (3) examine CMS's
processes for approving and monitoring the state MMISs. We performed our
work from January 2005 through September 2005 in accordance with generally
accepted government auditing standards. Appendix I contains further
details on our objectives, scope, and methodology.
4We are using the term "internal" to refer to all of CMS's IT investments,
excluding the state MMISs. Internal investments include Medicare claims
processing systems used by contractors.
5States request funding for the design, development, and installation of a
new MMIS or for the operations and maintenance of or enhancement to an
existing MMIS.
6Our second report, Information Technology: HHS Has Several Investment
Management Capabilities in Place, but Needs to Address Key Weaknesses,
GAO-06-11 (Washington, D.C.: Oct. 28, 2005), addresses HHS's (1)
capabilities for managing its IT investments and (2) plans for improving
those capabilities.
7GAO, Information Technology Investment Management: A Framework for
Assessing and Improving Process Maturity, GAO-04-394G (Washington, D.C.:
March 2004).
Results in Brief Judged against our framework for information technology
investment management, which measures the maturity of an organization's
investment management process, CMS's capabilities for effectively managing
its internal investments are limited. Specifically, CMS has established a
little over half of the foundational practices needed to manage its
internal investments individually and 2 of the 27 key practices required
to manage its investments as a portfolio-that is, an integrated,
agencywide collection of investments that are assessed and managed
collectively on the basis of common criteria. For example, CMS has
established most of the practices for capturing investment information and
many of the practices associated with instituting an investment board.
However, weaknesses remain in several areas. Specifically:
o the agency's investment management guide does not reflect current
processes;
o procedures for selecting and reselecting investments are not fully
documented;
o procedures for involving the board in efforts to systematically review
the progress of IT projects and systems in meeting cost, schedule, risk,
and benefit expectations have not been defined; and
o critical processes for defining portfolio criteria, creating the
portfolio, evaluating the portfolio, and conducting the postimplementation
reviews-necessary for portfolio management-have not been implemented.
According to CMS officials, the agency's investment management
capabilities are limited because investment management has only recently
become an area of management focus. Until CMS implements all of the key
practices it needs to build the investment foundation and manage its
investments as a portfolio, executives cannot be assured that they are
selecting and managing the mix of investments that best meets the agency's
needs and priorities, or that its investment decisions will result in the
most effective support and minimized risk for the multibillion-dollar
Medicare and Medicaid programs.
CMS has initiated steps to improve its investment management process;
however, these steps do not fully address the weaknesses we identify in
this report, nor are they coordinated with other needed improvement
efforts into a plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by
senior-level management. Without such a plan and procedures for
implementing it, CMS will be challenged in sustaining the commitment it
needs to fully establish its investment management process.
In approving funding for MMISs that CMS jointly funds with the states,
regional office staff use standard procedures, rely on established
guidance, and are required to report on their approval activities to CMS's
Center for Medicaid and State Operations. In contrast, in monitoring MMIS
activities, regional office staff lack standard procedures, guidance, and
reporting requirements. Without these elements for monitoring MMIS
activities, CMS may not be able to easily determine whether the state
MMISs, in which CMS invests close to $1.7 billion annually, are
facilitating the delivery of Medicaid benefits in the most effective and
beneficial manner.
To strengthen CMS's capability to manage its internal IT investments, we
are recommending that the Secretary for Health and Human Services direct
CMS's Administrator to develop and implement a plan aimed at addressing
the weaknesses identified in this report. We also are making
recommendations to improve CMS's process for monitoring the state MMISs
that it funds.
In commenting on a draft of this report, CMS provided information on
actions it is taking or plans to take to address our recommendations. The
agency, however, contended that many of the improvements to its IT
investment management process were not fully reflected in the report. This
is not accurate. The report sections in which we discuss the
implementation of specific key practices associated with critical
processes from our IT investment management framework each describe CMS's
efforts and accomplishments to improve its IT investment management
processes. In its written comments, CMS also took exception with our
recommendation for up-to-date, documented processes to ensure consistency,
and noted that the emphasis should be on strengthening these processes
first, and updating the documentation later. As we note in the report,
documenting processes does not preclude future revisions or improvements
to them, and provides a basis for consistent implementation across the
agency.
Background CMS has become the largest purchaser of health care in the
United States, serving nearly 83 million Medicare and Medicaid
beneficiaries.8 The agency administers the Medicare program, enacted in
1965, which provides health insurance to people who are aged 65 years and
over and to some people with disabilities who are under aged 65 years. The
agency also works with the states to administer the Medicaid program,
enacted in 1965 as a jointly funded program in which the federal
government matches state spending according to a formula to provide
medical and health-related services to low-income Americans.
In fiscal year 2005, CMS will reportedly spend about $519 billion: 63
percent for Medicare, 35 percent for Medicaid and Medicaid administration,
and the remaining 2 percent for the State Children's Health Insurance
Program and other administrative costs. CMS estimates that its total
budget in fiscal year 2006 will be $622 billion.
The agency carries out its responsibilities from its national headquarters
located in Baltimore, Maryland, and its 10 regional offices located
throughout the nation. It is organized around three centers (to support
its key functions): the Center for Medicare Management, the Center for
Beneficiary Choices, and the Center for Medicaid and State Operations.9
Numerous other offices throughout the agency support these centers.
CMS's Use of Information Technology
IT systems play a vital role in helping CMS to fulfill its
responsibilities in carrying out the Medicare and Medicaid programs. These
systems help to maintain Medicare information on the millions of
beneficiaries, providers, and medical services provided. For example,
CMS's Medicare Fee-for-Service claims processing systems process more than
1 billion claims annually and make benefit payments for the 41 million
elderly and disabled beneficiaries. In fiscal year 2004, the Medicare
program had estimated outlays of $297 billion in health care benefits.
8Of these nearly 83 million beneficiaries, more than 6 million are
children covered by the State Children's Health Insurance Program.
9The Center for Medicare Management is responsible for the Medicare
Fee-for-Service program. The Center for Beneficiary Choices is responsible
for Medicare's managed care program and also focuses on beneficiary
educational efforts. The Center for Medicaid and State Operations focuses
on programs administered by the states, such as Medicaid.
Similarly, IT systems are relied on to manage the Medicaid program. In
fiscal year 2003, this program provided benefits totaling about $261
billion to nearly 54 million people. Of this amount, the federal share was
about $153 billion.
To assist the states in developing and operating MMISs used to process
Medicaid claims and administer the program, CMS provides funding
assistance through grants. In fiscal year 2005, about $1.79 billion, or 70
percent, of CMS's nearly $2.55 billion total appropriations for IT went to
support Medicaid state investments. The remaining approximately $0.76
billion, or 30 percent, was used for CMS's internal investments. Figure 1
shows the breakdown of this funding between CMS's internal IT investments
and Medicaid state IT investments.
Figure 1: Distribution of CMS's Information Technology Budget, Fiscal Year
2005
$1.79 billion
IT investments for state MMISs
$0.76 billion
Internal IT investments
Weaknesses Previously Identified in CMS's IT Investment Management Processes
In September 2001,10 we reported that CMS's processes for managing its IT
investments omitted key review, approval, and evaluation steps. We
recognized that the agency was making efforts to strengthen its IT
planning and had developed guidance for an improved management process,
but stated that it would need to make considerable progress in
implementing these changes to ensure that its ongoing modernization
efforts stayed on track. To improve its investment management processes,
we made several recommendations to the CMS Administrator, including
establishing sufficient and written criteria to ensure a consistent
process for funding IT projects agencywide, and establishing a systematic
process for evaluating completed IT projects that included cost,
milestone, and performance data.
CMS's Approach to Investment Management
Several groups and individuals play a role in CMS's process to manage its
internal IT investments, including an investment board for establishing
the IT investment governance principles. However, a different process is
used to oversee the Medicaid IT systems that the agency jointly funds with
the states. This process is carried out by CMS's Center for Medicaid and
State Operations and 10 regional offices. Both of these processes, along
with the roles and responsibilities of the groups and individuals
involved, are described below.
Process for Managing Internal Investments
The groups and individuals who play a role in CMS's internal IT investment
management process include the Information Technology Investment Review
Board, Executive Steering Committees, Enterprise Architecture Group, and
Component Leads.
o Information Technology Investment Review Board (ITIRB). This board was
established in January of 2004 to provide a corporate perspective in
evaluating IT investments against CMS's business priorities. Its members
consist of senior leadership from CMS centers, offices, and regional
offices, and it is chaired by the agency's Chief Information Officer
(CIO). Initially, the primary ITIRB responsibility was overseeing
investments associated with the Medicare Presciption Drug, Improvement,
and Modernization Act of 2003 (MMA) and with CMS's
10GAO, Medicare: Information Systems Modernization Needs Stronger
Management and Support, GAO-01-824 (Washington, D.C.: Sept. 20, 2001).
revitalization initiative.11 These investments made up about one-third of
CMS's fiscal year 2005 Operating Plan for internal systems. In the spring
of 2005, the role of the board was expanded to include all internal IT
investments. To assist the ITIRB in its activities, CMS staff from the
Office of Information Services and the Office of Financial Management
provide administrative support. According to its charter, the board is
responsible for
o establishing the criteria for the selection, control, and evaluation of
CMS's portfolio of IT projects;
o developing the agency's IT operation plan and responding to the
President's budget request;
o reviewing the performance of IT investments using the criteria and
checkpoints in meeting cost, schedule, risk, and benefit expectations and
taking corrective actions when expectations are not being met;
o ensuring that IT investments in operation are periodically evaluated to
determine whether they should be retained, modified, replaced, or
terminated; and
o comparing the results of implemented investments with the expectations
that were set for them and developing a set of lessons learned for future
process improvement.
o Executive Steering Committees (ESC). The ESCs were established to
support the ITIRB in carrying out its responsibilities. Each ESC is
responsible for managing IT projects (or investments) that are grouped
together into a portfolio for each of CMS's business components. This
responsibility includes maintaining the appropriate mix of IT investments
in its portfolio, managing the investments in its portfolio, and providing
funding recommendations to the ITIRB for these investments. The membership
of each ESC depends on the IT investments contained in the portfolio, but,
at a minimum, every CMS
11The Medicare Prescription Drug, Improvement, and Modernization Act of
2003, Pub. L. No. 108-173, is to provide seniors and individuals with
disabilities with prescription drug benefits, more choices, and better
benefits under Medicare. CMS's revitalization initiative is the agency's
effort to address long-term IT issues.
component that sponsors a project is to have a representative on the ESC.
o Enterprise Architecture Group. This group, formally known as the IT
Architecture Planning Staff, supports the IT investment management
process, by, among other things, reviewing business case analyses for new
investments and major enhancements to ensure that they are consistent with
the enterprise architecture, by making recommendations based upon that
review that are aimed at the optimal leveraging of assets.
o Component Leads. These individuals provide support in the IT investment
management process by serving as liaisons between the Office of
Information Services and individual project managers. Component Leads are
to assist project managers in understanding CMS's investment management
process and other operational policies and processes. They can also
provide project managers with key contacts for various IT services that
project owners may require during implementation of a project.
In the spring of 2005, CMS implemented a new budget formulation process
and used it to select its IT investments. This process begins with an
information request from the CIO asking that each component submit
information on all of its investments, both new and ongoing. This
information is to include (1) a score sheet for each investment that shows
how it compared with prescribed criteria, such as alignment with business
drivers and IT strategic goals, and (2) a prioritized list of all
investments for the component. For new investments, the components also
are to submit an IT Fact Sheet (an investment proposal) that the ITIRB
support staff; the Enterprise Architecture Group; and, ultimately, the
board review to determine if the need for the new investment is justified.
If the need is found to be justified, project managers receive funding to
develop a Business Case Analysis (smaller projects may not require such a
document), which goes through the same review process as the IT Fact
Sheet.
The ITIRB support staff review all information submitted in response to
the information request and prepare it for the ESCs' review. The ESCs
reevaluate the investments against the criteria, making adjustments to the
scoring if necessary, and make funding recommendations to the board. The
ITIRB makes strategic and funding recommendations regarding CMS's IT
capital investment portfolio to CMS's Chief Operating Officer who, in
turn,
provides recommendations to the CMS's Office of Financial Management for
integration into the agency's overall budget. Figure 2 illustrates CMS's
process for selecting its internal IT investments.
Figure 2: CMS Selection Process for Internal IT Investments
Source: Centers for Medicare & Medicaid Services.
To date, the ITIRB's role in controlling (overseeing) IT investments has
been primarily limited to those associated with the MMA and revitalization
initiatives. According to CMS officials, efforts to define procedures for
the board to control all internal investments, in accordance with the
responsibilities described in its charter, are currently under way.
Process for Approving and The ITIRB plays no role in approving and
monitoring state Medicaid IT
Monitoring State Medicaid IT investments. Instead, the process for
approving states' requests for
Investments matching funds for MMIS activities-including the design,
development, and installation of new MMISs, and the operations,
maintenance, and enhancement of existing MMISs-is the shared
responsibility of CMS's Center for Medicaid and State Operations
(hereafter referred to as the central office) and its 10 regional offices.
According to regulations,12 the State Medicaid Manual,13 and officials we
interviewed at CMS's central office and 5 regional offices, CMS's process
for approving states' requests generally consists of the activities
discussed below:
o To request federal funds for state MMIS activities, states must prepare
an advance planning document (APD), which identifies, among other things,
the purpose, scope, benefits, and preliminary cost estimates for the
activities they want to undertake. States submit this document to the
regional office, which reviews the APD for completeness and technical
content. Regional office staff generally ensure that requests support the
Medicaid program, are in compliance with federal requirements, and
represent cost-effective solutions. Also, the regional office may have
suggestions for the states to improve their APDs. Some of the officials we
interviewed told us that they work with the states to complete the APDs to
expedite the review and approval process.
12Medicaid regulations are in 42 C.F.R. Ch. IV. Regulations pertaining to
the advance planning document process are set forth at 45 C.F.R. Part 95.
13The State Medicaid Manual provides instructions, regulatory citations,
and information for carrying out the Medicaid program.
o Once regional office staff determine that an APD adequately justifies
the request for funding and the request is approved by that regional
office's Associate Regional Administrator for Medicaid, the CMS central
office and HHS are notified of the approval through a process referred to
as the Office of the Secretary Notice process.14 Once the central office
concurs, the regional office can send an approval letter to the state.
o The states typically hire contractors to perform the MMIS activities.
With the approval of an APD, a state is given the clearance to develop the
request-for-proposals for soliciting contractor proposals. While the APD
is a high-level justification for funding, the request-for-proposals is to
contain the more detailed requirements of the MMIS activities. Before it
is issued, the request-for-proposals must be approved by the CMS regional
office through a process similar to that used for the APD.
o The states review the proposals received and evaluate them in order to
make the final selection. While regional office staff do not formally
approve a state's evaluation process, they do review the process to ensure
that it allows for open and free competition, to the maximum extent
practicable.
o The states draft a contract for the MMIS activities. Prior to its
award, the contract is reviewed by regional office staff and approved by
the Associate Regional Administrator for Medicaid. The state then makes an
award to the contractor whose bid or offer is responsive to the
solicitation and most advantageous to the state-considering price,
quality, and other factors.15
o When the contracted MMIS activities start, regional office staff begin
monitoring the status of these activities through a variety of mechanisms,
including reviews of status reports; on-site visits; and meetings with
external groups, such as industry associations, provider groups, and
vendors.
14The Office of the Secretary Notices are one-page summaries of the
reviews performed by CMS regional office staff of documentation (APDs,
request-for-proposals, and contracts) submitted by a state for MMIS
funding assistance. The summaries are submitted to CMS's Center for
Medicaid and State Operations, which must review and "clear" them before
the regional office can release the official approval letter to the state.
1545 C.F.R. 95.613(b) and 45 C.F.R. 74.43.
o Once MMISs are built and become operational, CMS establishes a team
consisting of headquarters and regional office staff with expertise in
relevant areas to do on-site reviews, referred to as certification
reviews. During these reviews, which are to be conducted about 6 months
after a system has been in operation, the team makes sure that the system
satisfies the terms of the state's APD, meets minimal federal
requirements, and complies with current regulations and policy. CMS has
written guidance for conducting these reviews, which it is in the process
of revising.16
o Regional office staff are to continue monitoring MMIS activities
through the previously mentioned mechanisms.
Information Technology Investment Management Maturity Framework
The Information Technology Investment Management (ITIM) framework is a
maturity model comprising five progressive stages of maturity that an
agency can achieve in its investment management capabilities.17 The ITIM
framework was developed on the basis of our research into the IT
investment management practices of leading private- and public-sector
organizations. It identifies critical processes for making successful IT
investments, organized into the five increasingly mature stages. These
maturity stages are cumulative; that is, in order to attain a higher stage
of maturity, the agency must have institutionalized all of the
requirements for all of the lower stages in addition to the higher stage.
The ITIM framework can be used to assess the maturity of an agency's
investment management processes and as a tool for organizational
improvement. The overriding purpose of the framework is to encourage
investment processes that increase business value and mission performance,
reduce risk, and increase accountability and transparency in the decision
process. We have used the framework in several of our
16CMS's certification guidance is defined in the agency's Medicaid
Management Information System Certification Review Protocol.
17GAO-04-394G.
evaluations,18 and a number of agencies have adopted it. These agencies
have used ITIM for purposes ranging from self-assessment to the redesign
of their IT investment management processes.
The ITIM framework's five maturity stages represent steps toward achieving
stable and mature processes for managing IT investments. Each stage builds
on the lower stages; the successful attainment of each stage leads to
improvement in the organization's ability to manage its investments. With
the exception of the first stage, each maturity stage is composed of
"critical processes" that must be implemented and institutionalized in
order for the organization to achieve that stage. These critical processes
are further broken down into key practices that describe the types of
activities that an organization should be performing to successfully
implement each critical process. An organization may be performing key
practices from more than one maturity stage at the same time. This is not
unusual, but efforts to improve investment management capabilities should
focus on becoming compliant with lower-stage practices before addressing
higher-stage practices.
Stage 2 of the ITIM framework encompasses building a sound investment
management process by establishing basic capabilities for selecting new IT
projects. It also involves developing the capability to control projects
so that they finish predictably within established cost and schedule
expectations and the capability to identify potential exposures to risk
and put in place strategies to mitigate that risk. The basic selection
processes established in Stage 2 lays the foundation for more mature
selection capabilities in Stage 3.
Stage 3 requires that an organization continually assess both proposed and
ongoing projects as parts of a complete investment portfolio-an integrated
and competing set of investment options. It focuses on establishing a
consistent, well-defined perspective on the IT investment
18GAO, Information Technology: DLA Needs to Strengthen Its Investment
Management Capability, GAO-02-314 (Washington, D.C.: Mar. 15, 2002);
United States Postal Service: Opportunities to Strengthen IT Investment
Management Capabilities, GAO-03-3 (Washington D.C.: Oct. 15, 2002);
Information Technology: Departmental Leadership Crucial to Success of
Investment Reforms at Interior, GAO-03-1028 (Washington, D.C.: Sept. 12,
2003); Bureau of Land Management: Plan Needed to Sustain Progress in
Establishing IT Investment Management Capabilities, GAO-03-1025
(Washington, D.C.: Sept. 12, 2003); and Information Technology: FAA Has
Many Investment Management Capabilities in Place, but More Oversight of
Operational Systems Is Needed, GAO-04-822 (Washington, D.C.: Aug. 20,
2004).
portfolio and maintaining mature, integrated selection (and reselection),
control, and evaluation processes that can be evaluated during
postimplementation reviews. This portfolio perspective allows decision
makers to consider the interaction among investments and the contributions
to organizational mission goals and strategies that could be made by
alternative portfolio selections, rather than focusing exclusively on the
balance between the costs and benefits of individual investments.
Organizations implementing Stages 2 and 3 have in place the selection,
control, and evaluation processes that are required by the Clinger-Cohen
Act of 1996.19
Stages 4 and 5 require the use of evaluation techniques to continuously
improve both the investment portfolio and the investment processes in
order to better achieve strategic outcomes. At Stage 4 maturity, an
organization has the capacity to conduct IT succession activities and,
therefore, can plan and implement the deselection of obsolete, high-risk,
or low-value IT investments. An organization with Stage 5 maturity
conducts proactive monitoring for breakthrough information technologies
that will enable it to change and improve its business performance. Stages
4 and 5 define key attributes that are associated with the most capable
organizations.
Figure 3 shows the five ITIM stages of maturity and the critical processes
associated with each stage.
1940 U.S.C. S: 11312(b)(1).
Figure 3: The Five ITIM Stages of Maturity with Critical Processes
In order to have the capabilities to effectively manage IT investments, an
agency, at a minimum, should (1) build an investment foundation by putting
basic, project-level control and selection practices in place (Stage 2
capabilities) and (2) manage its projects as a portfolio of investments,
treating them as an integrated package of competing investment options and
pursuing those that best meet the strategic goals, objectives, and mission
of the agency (Stage 3 capabilities).
CMS has executed 20 of the 38 key practices that are required to build a
foundation for IT investment management. In addition, because CMS has
focused primarily on establishing the Stage 2 practices, it has executed
only 2 of the 27 Stage 3 key practices. Until CMS implements all of the
key practices associated with building the investment foundation and
managing its investments as a portfolio, the agency will not have much
assurance that it has selected the mix of investments that best supports
its strategic goals, or that it will be able to manage the investments to
successful completion.
Source: GAO.
As defined by the model, each critical process consists of "key practices"
that must be executed to implement the critical process.
CMS's Capabilities to Manage Its Internal Investments Are Limited
CMS Has Established about Half of the Foundational Practices for Investment
Management
At the ITIM Stage 2 level of maturity, an organization has attained
repeatable, successful IT project-level investment control processes and
basic selection processes. Through these processes, the organization can
identify expectation gaps early and take the appropriate steps to address
them. According to the ITIM framework, critical processes at Stage 2
include (1) defining IT investment board20 operations, (2) identifying the
business needs for each IT investment, (3) developing a basic process for
selecting new IT proposals and reselecting ongoing investments, (4)
developing project-level investment control processes, and (5) collecting
information about existing investments to inform investment management
decisions. Table 1 describes the purpose of each of these Stage 2 critical
processes.
Table 1: Stage 2 Critical Processes-Building the Investment Foundation Critical
process Purpose
Instituting the investment board To define and establish an appropriate
information technology (IT) investment management structure and the
processes for selecting, controlling, and evaluating IT investments.
Meeting business needs To ensure that IT projects and systems support the
organization's business needs and meet users' needs.
Selecting an investment To ensure that a well-defined and disciplined
process is used to select new IT proposals and reselect ongoing
investments.
Providing investment oversight To review the progress of IT projects and
systems, using predefined criteria and checkpoints, in meeting cost,
schedule, risk, and benefit expectations and to take corrective action
when these expectations are not being met.
Capturing investment information To make available to decision makers
information to evaluate the impacts and opportunities created by proposed
(or continuing) IT investments.
Source: GAO.
Because IT investment management has only recently become an area of
management attention, CMS has put in place 20 of the 38 Stage 2 key
practices required for basic project-level selection and control. The
agency has satisfied the majority of the key practices associated with
establishing an IT investment review board, capturing investment
information, and meeting business needs. CMS also has recently established
a process for
20An IT investment board is a decision-making body-made up of senior
program, financial, and information managers-that is responsible for
making decisions about IT projects and systems on the basis of comparisons
and trade-offs among competing projects, with an emphasis on meeting
mission goals.
selecting investments, but it has not yet established a process for the IT
investment review board to provide investment oversight. Figure 4
summarizes the status of CMS's critical processes for Stage 2, showing how
many key practices CMS has executed in managing its internal IT
investments.
Figure 4: Summary of Results for Stage 2 Critical Processes and Key
Practices for Internal IT Investments
CMS Has an Investment Review Board, but its Investment Management Process
Guide Is Not Current
Source: GAO.
The creation of decision-making bodies or boards is central to the IT
investment management process. At the Stage 2 level of maturity,
organizations define one or more boards, provide resources to support
their operations, and appoint members who have expertise in both
operational and technical aspects of the proposed investments. The boards
operate according to a written IT investment process guide that is
tailored to the organization's unique characteristics, thus ensuring that
consistent and effective management practices are implemented across the
organization. Once board members are selected, the organization ensures
that they are knowledgeable about policies and procedures for managing
investments. Organizations at the Stage 2 level of maturity also take
steps to ensure that executives and line managers support and carry out
the decisions of the investment board. According to the ITIM framework, an
IT investment management process guide should (1) be a key authoritative
document that the organization uses to initiate and manage IT investment
processes and (2) provide a comprehensive foundation for policies and
procedures developed for all other related processes. (The complete list
of key practices is provided in table 2.)
CMS has executed 5 of the 8 key practices for this critical process. For
example, in January 2004, the agency established the ITIRB to manage
internal investments and provide business-driven leadership to its
operations and development. While the ITIRB was initially only responsible
for overseeing MMA and revitalization initiatives, its responsibilities
were expanded this past spring to include management and oversight
responsibilities for all internal investments. ITIRB members are
seniorlevel officials from both business and IT areas who understand board
policies and procedures.
The ITIRB is adequately resourced to maintain its operations. For example,
the Program Management and Support Group within the Office of Information
Services assists the board in such ways as coordinating and integrating
the investment management process. This group serves as the principal
contact and entry point for all new and proposed IT projects. In addition,
nine Executive Steering Committees were recently established to support
the work of the ITIRB by managing a subset of investments grouped together
according to business function. Their responsibilities include, among
other things, scoring and ranking IT investments, and recommending
investments to the ITIRB for funding.
Notwithstanding these strengths, CMS does not have an IT investment
process guide that reflects the agency's current investment management
practices. For example, the agency uses ESCs to work with the board on
specific areas of IT investments, but its process guide does not identify
this critical group. Moreover, the process guide does not mention the
agency's move to classify its IT investments in line with the department's
classification scheme. (The new classification scheme consists of three
levels in which projects are rated as major, supporting, or tactical.)
Instead, the process guide outlines a four-level classification scheme
that identifies investments as A, B, C, or D, depending on the nature and
sensitivity of the project. According to CMS officials, the guide has not
yet been updated because the agency has made a priority of fully defining
its processes before documenting them. Documenting the process, however,
does not preclude it from future revisions or improvements, but does
provide a basis for consistent implementation across the agency. Until
CMS's documented IT investment process guidance is updated, executives are
at risk of inconsistently performing key investment decision-making
activities and inaccurately communicating management practices. Such
updated guidance would also provide a process that could lead to greater
accountability about future IT investment outcomes, which would be helpful
to new members joining the board.
Another key weakness is that CMS's ITIRB has not operated in accordance
with its assigned roles and responsibilities. For example, the ITIRB has
not yet been involved in systematically controlling investments nor has it
actively maintained the documented investment management process. Until
the ITIRB fully carries out its assigned roles and responsibilities,
executives will not have assurance that the whole IT investment management
process is functioning smoothly and effectively as intended.
Table 2 shows the rating for each key practice required to implement the
critical process for instituting the investment board at the Stage 2 level
of maturity. Each of the "executed" ratings shown below represents
instances where, on the basis of the evidence provided by CMS officials,
we concluded that the specific key practices were executed by the agency.
Table 2: Instituting the Investment Board
Type of
practice Key practice Rating Summary of evidence
Organizational commitments
1. An enterprisewide IT investment board composed of senior executives
from IT and business units is responsible for defining and implementing
the organization's IT investment governance process.
Executed CMS has an enterprisewide IT investment board that is responsible
for defining and implementing the organization's IT investment management
process. The board consists of the agency's senior leadership from CMS
centers, offices, and regional offices and is chaired by the Chief
Information Officer.
2. The organization has a Not Although CMS has a documented IT investment
management process documented IT investment executed guide, this guide has
not been updated to reflect current investment process directing each
management processes. For example, CMS has established several investment
board's working groups supporting the investment management process-for
operations example, the Executive Steering Committees-which are not
identified in
the IT investment management process guide. According to officials, CMS
plans to update its process guide in the near future.
(Continued From Previous Page)
Type of
practice Key practice Rating Summary of evidence
Prerequisites 1. Adequate resources, Executed According to CMS officials,
adequate resources are provided to support including people, funding,
board operations. For example, to support the work of the ITIRB, the and
tools, are provided for agency has established Executive Steering
Committees. In addition, the supporting the operations Planning,
Management, and Support Group serves as the principal contact of each IT
investment and entry point for all new and proposed IT projects. CMS also
has an board. ITIRB support group to support the operations of the ITIRB.
2. The board members Executed ITIRB members are senior-level managers who
understand CMS's understand the investment management policies and
procedures as they currently stand. organization's IT Board members have
also undertaken activities that would contribute to investment management
their understanding of board policies and procedures, including attending
a policies and procedures 2-day retreat and monthly meetings. and the
tools and techniques used in the board's decision-making process.
3. Each board's span of Executed CMS's enterprisewide investment board is
responsible for defining and authority and implementing the investment
management process. responsibility is defined to minimize overlaps or gaps
among the boards.
Although the ITIRB has
Activities 1. The enterprisewide Not responsibility for
developing and maintaining
the
documented investment
investment board has executed management process, it has
not been actively
oversight maintaining this process.
responsibilities
for the development and
maintenance of the
organization's
documented IT
investment
process.
2. Each investment board Not CMS's enterprisewide investment board is not
yet fully carrying out the operates in accordance executed scope of its
responsibilities. To date, board members have selected with its assigned
authority investments for inclusion in the fiscal year 2007 budget, but
the board has and responsibility. not yet been involved in systematically
controlling investments. In addition,
the board has not been actively maintaining the organization's documented
IT investment management process.
3. The organization has Executed The ITIRB develops the agency's operating
plan, and, according to officials, established management only investments
listed in the operating plan are funded. controls for ensuring that
investment boards' decisions are carried out.
Source: GAO.
CMS Has a Process for Ensuring Defining business needs for each IT project
helps to ensure that projects That Projects Align with and systems support
the organization's business needs and meet users' Business Needs and Meet
Users' needs. This critical process ensures that a link exists between the
Needs organization's business objectives and its IT management strategy.
According to the ITIM, effectively meeting business needs requires, among
other things, (1) documenting business needs with stated goals and
objectives, (2) identifying specific users and other beneficiaries of IT
projects and systems, (3) providing adequate resources to ensure that
projects and systems support the organization's business needs and meet
users' needs, and (4) periodically evaluating the alignment of IT projects
and systems with the organization's strategic goals and objectives. (The
complete list of key practices is provided in table 3.)
CMS has in place 5 of the 7 key practices for meeting business needs. The
agency's IT Investment Management Process Guide and Business Case Analysis
Development Guide require business needs for both proposed and ongoing IT
projects and systems to be identified in an IT fact sheet and, in some
instances, a business case analysis document. The agency also has detailed
procedures for developing these documents that call for identifying users.
We verified that the four projects we reviewed identified specific users
and also documented how the projects linked back to CMS business needs.21
Resources for ensuring that IT projects and systems support the
organization's business needs and meet users' needs include Component
Leads, the Enterprise Architecture Group, and detailed procedures and
associated templates for developing the IT fact sheet and business case
analysis document.
Although CMS has performed most of the key practices associated with
meeting business needs, a few weaknesses remain. Specifically, officials
told us they rely on the HHS strategic plan to guide their efforts because
CMS's strategic plan documenting the agency's business mission, goals, and
objectives is outdated.22 However, the primary tool used to justify
funding for investments does not tie into the HHS plan but provides
high-level business drivers23 for aligning these investments with business
needs. While, according to agency officials, these business drivers
reflect a common understanding of the agency's goals and objectives, they
are not descriptive enough to drive IT investments. Until CMS develops a
current strategic plan or other detailed statement of business mission
with supporting goals and objectives, the agency is at risk of not being
able to
21The four projects we reviewed-Healthcare Integrated General Ledger
Accounting System, Medicare Claims Processing Redesign, Medicare Managed
Care System, and National Plan and Provider Enumeration System-are
described in appendix I.
22According to the Director of Investment Tracking and Assessment, the
strategic plan has not been updated because of turnover in upper-level
management.
23The tool lists the following four business drivers: (1) beneficiary
health and satisfaction, (2) efficiency and integrity of operations, (3)
health care delivery, and (4) health care quality.
thoroughly communicate critical information on its goals and objectives or
to provide clear and transparent direction for its IT investment
management process.
Finally, CMS's budget formulation process serves as a mechanism to
reevaluate the alignment of projects and systems with the organization's
goals and objectives. However, the ITIRB selected investments for the
first time this past spring and, therefore, has not yet had to reevaluate
projects' and systems' alignment with organizational goals and objectives.
When CMS executes all key practices associated with this critical process,
it will have greater assurance that its projects effectively meet the
agency's business needs.
Table 3 shows the rating for each key practice required to implement the
critical process for meeting business needs at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 3: Meeting Business Needs
Type of practice Key practice Rating Summary of evidence
The IT Investment
Organizational 1. The organization has Executed Management Process Guide
and the
commitments documented policies Business Case Analysis
and Development Guide both
procedures for document procedures for
identifying IT ensuring that IT projects
and
projects or systems systems support the
that organization's business
needs.
support the
organization's
ongoing and future
business
needs.
The organization CMS does not have an
Prerequisites 1. has a Not executed updated strategic plan
or other
documented business detailed statement of
mission business mission with
supporting
with stated goals goals and objectives.
and Instead, the agency uses
a list of
business drivers to
objectives. align IT projects and
systems with
business needs. Although
these business drivers
may
reflect a common
understanding of the
agency's business
drivers, they are not
descriptive enough to
drive IT
investments.
2. Adequate resources, including Executed According to CMS officials,
adequate resources have people, funding, and tools, are been provided for
ensuring that IT investment systems provided for ensuring that IT meet
business and users' needs. These resources projects and systems support
include the Component Leads, the Enterprise the organization's business
Architecture Group, and detailed procedures and needs and meet users'
needs. associated templates for developing the IT fact sheet and
business case analysis.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
The organization CMS requires that all
Activities 1. defines and Executed projects have an IT fact
sheet and,
documents business in some instances, a
needs for business case analysis.
These two
both proposed and documents identify the
ongoing IT business needs for both
proposed
projects and systems. and ongoing IT projects and
systems. We verified that
business needs were
documented for the four
projects we
reviewed.
2. The organization identifies Executed CMS requires that users be
identified in the business specific users and other case analysis and an
IT fact sheet. We verified that beneficiaries of IT projects and specific
users were documented for the four projects we systems. reviewed.
3. Users participate in project Executed CMS has procedures specifying the
involvement of users management throughout an IT in project management
throughout a project's life cycle. project's or system's life cycle. We
verified that for the four projects we reviewed, users
participated in project management throughout the
projects' life cycles.
4. The investment board Not executed CMS's budget formulation process
serves as a periodically evaluates the mechanism to reevaluate the
alignment of projects and alignment of its IT projects and systems with
the organization's goals and objectives. The systems with the ITIRB,
however, selected investments for the first time this organization's
strategic goals past spring and, therefore, has not yet had to reevaluate
and objectives and takes projects' and systems' alignment with
organizational corrective actions when goals and objectives. misalignment
occurs.
CMS Has Processes to Select Investments, but Selection Criteria Do Not
Consider Critical Factors
Source: GAO.
Selecting new IT proposals and reselecting ongoing investments require a
well-defined and disciplined process to provide the agency's investment
board, business units, and developers with a common understanding of the
process and the cost, benefit, schedule, and risk criteria that will be
used both to select new projects and to reselect ongoing projects for
continued funding. According to the ITIM, this critical process requires,
among other things, (1) making funding decisions for new proposals
according to an established process; (2) providing adequate resources for
investment selection activities; (3) using a defined selection process to
select new investments and reselect ongoing investments; (4) establishing
criteria for analyzing, prioritizing, and selecting new IT investments and
for reselecting ongoing investments; and (5) creating a process for
ensuring that the criteria change as organizational objectives change.
(The complete list of key practices is provided in table 4.)
CMS has executed 4 of the 10 key practices associated with selecting an
investment. Specifically, CMS used a process it defined in February
2005-its budget formulation process-to select new investments and reselect
existing investments using a set of limited criteria. We confirmed
that the four projects we reviewed were reselected using this new process.
In addition, by using the budget formulation process to select
investments, executives had assurance that funding decisions were aligned
with selection decisions. Officials indicated that adequate resources were
provided for identifying and selecting investments.
However, weaknesses remain in the selection area. Although CMS has a
number of documents that address investment selection and reselection,
these documents are not linked to provide a clear understanding of the
selection and reselection process. In addition, they do not define (1) the
roles and responsibilities for each participating unit involved in the
project selection process and (2) the decision-making procedures. CMS
officials told us they chose to first implement the selection process and
then go back to document it. Another key weakness in the selection area is
that, although selection and reselection criteria have been defined, they
do not include cost, benefit, schedule, and risk factors. Officials
indicated that because the Executive Steering Committees and the ITIRB had
a short amount of time to perform selection activities this year, they
defined a limited set of criteria to evaluate projects. Further, CMS does
not have a mechanism to ensure that its selection criteria continue to
reflect organizational objectives.
Until CMS implements all of the key practices associated with selecting
investments, executives will not be adequately assured that they are
consistently and objectively selecting projects that meet the needs and
priorities of the agency in a cost-effective and risk-insured manner.
Table 4 shows the rating for each key practice required to implement the
critical process for selecting an investment at the Stage 2 level of
maturity and summarizes the evidence that supports these ratings.
Table 4: Selecting an Investment
Type of practice Key practice Rating Summary of evidence
The organization Although CMS has a
Organizational 1. has Not executed number of documents that
address
documented investment selection,
commitments policies and they are not linked to
provide a clear
procedures for understanding of the
selecting selection process. In
addition, these
documents do not define
new IT proposals. the roles and
responsibilities for
each participating unit
involved in the project
selection
process, nor do they
define the
decision-making
procedures.
2. The organization has Not executed Although CMS has a number of
documents that address documented policies and investment reselection,
they are not linked to provide a procedures for reselecting clear
understanding of the reselection process. In ongoing IT investments.
addition, they do not define the roles and responsibilities for each
participating unit involved in the project reselection process, nor do
they define the decisionmaking procedures.
3. The organization has Not executed Although the process used to
formulate the budget for the documented policies and fiscal year 2006/2007
budget cycle integrated funding with procedures for integrating selection,
there are no policies and procedures funding with the process of
documenting this integration. selecting an investment.
According to the CMS
Prerequisites 1. Adequate resources, Executed Director of Investment
Tracking and
including people, Assessment, there were
funding, adequate resources to
support
and tools, are selection activities this
provided for year. For example, the
Office of
identifying and Financial Management
selecting IT provided some staff
resources, as
did the Office of
projects and systems. Information Services.
2. Criteria for analyzing, Not executed For the fiscal year 2006/2007
budget cycle, CMS's ITIRB
prioritizing, and selecting developed and used criteria,
including alignment with IT
new IT investment strategic goals and primary
business drivers, for the
opportunities have been selection process. However,
these criteria did not include
established. cost, benefit, schedule, and
risk factors.
3. Criteria for analyzing, Not executed For the fiscal year 2006/2007
budget cycle, CMS's ITIRB
prioritizing, and developed and used criteria,
including alignment with IT
reselectinga IT investment strategic goals and primary
business drivers for the
opportunities have been reselection process. However,
these criteria did not
established. include cost, benefit,
schedule, and risk factors.
CMS reported in its
4. A mechanism exists to Not executed self-assessment that there are
no
ensure that the criteria mechanisms to ensure that the
selection criteria continue
continue to reflect to reflect organizational
objectives.
organizational
objectives.
The organization uses This past spring, CMS used
Activities 1. its Executed its defined selection
process,
defined selection including a limited set of
process, predefined selection
criteria, to
including predefined select new IT investments.
selection criteria, to
select
new IT investments.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
2. The organization uses its Executed For the fiscal year 2006/2007
budget cycle, CMS began defined selection process, using a new budget
formulation process, including a including predefined limited set of
predefined criteria, to reselect ongoing IT selection criteria, to
investments. We verified that the four projects we reselecta ongoing IT
reviewed were reselected using this process. investments.
3. Executives' funding Executed Because CMS uses its budget formulation
process to decisions are aligned with select investments, executives'
funding decisions are selection decisions. aligned with selection
decisions.
Source: GAO.
aAccording to the GAO Information Technology Investment Management
framework, "reselecting" is the periodic reconsideration of an
investment's continuing value to the organization and the decision to
continue funding. It is a recurring process that continues for as long as
a project is receiving funding.
CMS Has Not Defined An organization should provide effective oversight for
its IT projects Procedures for Management throughout all phases of their
life cycles. Its investment board should Oversight of IT Projects and
maintain adequate oversight and observe each project's performance and
Systems
progress toward predefined cost and schedule expectations as well as each
project's anticipated benefits and risk exposure. The investment board
should also employ early warning systems that enable it to take corrective
action at the first sign of cost, schedule, or performance slippages. This
board has ultimate responsibility for the activities within this critical
process. According to the ITIM framework, effective project oversight
requires, among other things, (1) having written policies and procedures
for management oversight; (2) developing and maintaining an approved
management plan for each IT project; (3) making up-to-date cost and
schedule data for each project available to the oversight boards; (4)
having regular reviews by each investment board of each project's
performance against stated expectations; and (5) ensuring that corrective
actions for each underperforming project are documented, agreed to,
implemented, and tracked until the desired outcome is achieved. (The
complete list of key practices is provided in table 5.)
CMS has only executed 1 of the 7 key practices associated with effective
project oversight. While CMS's IT Investment Management Process Guide
addresses management oversight of IT projects and systems, it does not
include specific procedures for the ITIRB's oversight of IT projects and
systems. In addition, while the board is receiving performance data for
some investments, including revitalization investments, it is not yet
performing oversight of projects on a systematic basis. CMS officials
indicated that the ITIRB's involvement in overseeing investments to date
has been limited because the board was first focusing on selecting
investments. However, CMS recognizes the importance of the ITIRB's
involvement in oversight of IT investments, and, according to officials,
the agency is currently developing an approach to address this issue.
While CMS is in the process of developing a structured process for the
ITIRB to oversee investments, other entities are involved in the oversight
of projects. For example, performance information for one of the projects
we reviewed was not provided to CMS's ITIRB, but instead was provided to
senior-level management, such as the Chief Technology Officer and
directors from some CMS components. Until the ITIRB systematically
oversees CMS's investments, the oversight process will not benefit from
the corporate perspective that is gained by having an enterprisewide
board. As a result, executives may not be able to easily determine the
impact individual project decisions may have on other projects and on the
attainment of organizational goals and objectives.
Table 5 shows the rating for each key practice that is required to
implement the critical process for project oversight at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
Table 5: Providing Investment Oversight
Type of practice Key practice Rating Summary of evidence
CMS's IT Investment
1. The organization Not executed Management Process
Organizational has Guide
documented policies addresses management
commitment and oversight of IT
projects and
systems, but it does
procedures for not include specific
procedures for
management oversight the ITIRB's oversight
of of IT projects and
systems.
IT projects and According to CMS
systems. officials, these
procedures are
currently being
defined.
According to CMS
1. Adequate Not executed officials, CMS does not
Prerequisites resources, have the
including people, resources it needs to
funding, oversee IT projects and
systems.
For example, they
and tools, are reported that
provided for additional skilled
staff are
IT project needed to perform
oversight. oversight activities.
2. IT projects and systems, including those in steady state (operations
and maintenance), maintain approved project management plans that include
expected cost and schedule milestones and measurable benefit and risk
expectations.
Executed CMS IT projects and systems, including those in operations and
maintenance, maintain approved project management plans that include cost,
schedule, benefit, and risk expectations. We verified that the case-study
projects we reviewed maintained project management plans that include
these expectations.
(Continued From Previous Page)
Type of practice Key practice Rating Summary of evidence
Although data on actual
Activities 1. Data on actual Not executed performance are being
provided
performance, to the ITIRB for some
including projects, there are no
standard
cost, schedule, procedures for
benefit, and involving the ITIRB in
investment
risk performance, oversight. According to
are CMS officials, these
procedures
provided to the are currently being
appropriate determined.
IT investment board.
2. Using verified data, each Not executed The ITIRB has begun to review
the performance of some investment board regularly IT projects and systems
against stated expectations. For reviews the performance of example, the
ITIRB has recently begun to review the IT projects and systems performance
of the National Plan and Provider against stated Enumeration System.
According to CMS officials, expectations. procedures for the ITIRB to do
this on a more systematic
basis are currently being determined.
3. For each underperforming IT project or system, appropriate actions are
taken to correct or terminate the project or system in accordance with
defined criteria and the documented policies and procedures for management
oversight.
Not executed According to CMS officials, procedures for involving the
ITIRB in investment oversight, including procedures for taking corrective
actions, are currently being determined.
4. The investment board Not executed According to CMS officials,
procedures for involving the
regularly tracks the ITIRB in investment oversight,
including procedures for
implementation of tracking the implementation of
corrective actions for
corrective actions for underperforming projects, are
each currently being
underperforming project determined.
until the actions are
completed.
CMS Has a Collection of Information to Support Investment Management
Decisions
Source: GAO.
To make good IT investment decisions, an organization must be able to
acquire pertinent information about each investment and store that
information in a retrievable format. During this critical process, an
organization identifies its IT assets and creates a comprehensive
repository of investment information. This repository provides information
to investment decision makers to help them evaluate the impacts and
opportunities that would be created by proposed or continuing investments.
It can provide insights and trends about major IT cost and management
drivers. The repository can take many forms and does not have to be
centrally located, but the collection method should identify each IT
investment and its associated components. This critical process may be
satisfied by the information contained in the organization's current
enterprise architecture, augmented by additional information-such as
financial information and information on risk and benefits-that the
investment board may require to ensure that informed decisions are being
made. According to the ITIM framework, effectively managing this
repository requires, among other things, (1) developing written policies
and procedures for identifying and collecting the information, (2)
assigning responsibility for ensuring that the information being collected
meets the needs of the investment management process, (3) identifying IT
projects and systems and collecting relevant information to support
decisions about them, and (4) making the information easily accessible to
decision makers and others. (The complete list of key practices is
provided in table 6.)
CMS has in place 5 of the 6 key practices associated with capturing
investment information. For example, CMS's IT Investment Management
Process Guide identifies specific information that is needed in the
investment management process, such as how each IT project relates to the
business needs of CMS. According to officials, adequate resources are
provided to support the collection of investment information, such as the
agency's IT Investment Tracking Database, and an individual assigned
responsibility for ensuring that the necessary information is collected to
meet the needs of the investment management process.
CMS is collecting specific information about IT investments to support
decisions about these investments, including projects' scores against
selection criteria and earned value management24 information. We verified
that this information was collected for the four projects we reviewed.
However, although the ITIRB has used investment information to support
selection decisions, it has not yet used it to systematically oversee
projects. According to CMS officials, specific procedures for the ITIRB's
oversight of IT projects and systems are currently being defined.
Table 6 shows the rating for each key practice required to implement the
critical process for capturing investment information at the Stage 2 level
of maturity and summarizes the evidence that supports these ratings.
24"Earned value management" is a project management tool that integrates
the investment scope of work with schedule and cost elements for
investment planning and control. This method compares the value of work
accomplished during a given period with the value of the work expected in
the period. Differences in expectations are measured in both cost and
schedule variances.
Table 6: Capturing Investment Information
Type of practice Key practice Rating Summary of evidence
Organizational 1. The organization has Executed CMS's IT Investment
Management Process Guide
documented policies defines procedures for
commitments and identifying and
collecting
procedures for information in a
identifying and database to support the
investment
collecting information
about IT management process.
projects and systems
to support
the investment
management
process.
2. An official is assigned Executed The director of CMS's Division of
Investment Analysis responsibility for ensuring that and Budget in the
Planning, Management, and the information collected during Support Group
of the CIO's office is responsible for project and systems ensuring that
the information collected about IT identification meets the needs of
projects and systems meets the needs of the the investment management
investment management process. process.
Adequate resources, According to CMS
Prerequisite 1. including Executed officials, there are
adequate
people, funding, and resources in this area,
tools, are including staff in the
Planning,
provided for Management, and Support
identifying IT Group of CMS's Office of
projects and systems Information Services and
and an IT Investment Tracking
collecting relevant Database.
investment
information about them.
IT projects and systems
Activities 1. The organization's IT Executed are identified and
projects specific
and systems are information is collected
identified, and about them in an IT
specific information is Investment Tracking
collected Database and Excel
to support decisions spreadsheets. We verified
about that information about
our
four case-study projects
them. was collected to support
the
selection and control
processes.
2. The information that has been Executed CMS collects information about
IT projects and collected is easily accessible systems and makes it
available to decision makers and understandable to decision and other
stakeholders in various forms, such as in makers and others. spreadsheets
and graphs. The director of CMS's Division of Investment Analysis and
Budget in the Planning, Management, and Support Group of the CIO's office
ensures that the ITIRB has all the relevant information for IT investment
decision making, and that it is in a format that the ITIRB is able to
easily use.
3. The information Not executed Although the board is using
repository is investment information to
used by investment support selection decisions,
decision procedures have not yet
makers and others to been defined for the board to use
support this information to
investment management. support control decisions.
Source: GAO.
CMS Lacks the Key Capabilities Needed to Manage Its Investments as a
Portfolio
During Stage 3, the investment board enhances the investment management
process by developing a complete investment portfolio. An investment
portfolio is an integrated, agencywide collection of investments that are
assessed and managed collectively on the basis of common criteria.
Managing investments within the context of such a portfolio is a
conscious, continuous, and proactive approach to expending limited
resources on an organization's competing initiatives in light of the
relative benefits expected from these investments. Taking an agencywide
perspective enables an organization to consider its investments
comprehensively, so that, collectively, the investments optimally address
the organization's missions, strategic goals, and objectives. Managing
investments with a portfolio approach also allows an organization to
determine priorities and make decisions about which projects to fund, and
continue to fund, on the basis of analyses of the relative organizational
value and risks of all projects, including projects that are proposed,
under development, and in operation. For an organization to reap the full
benefits of the portfolio process, it should collect all of its
investments into an enterprise-level portfolio that is overseen by its
senior investment board. Although investments may initially be selected
into subordinate portfolios-on the basis of, for example, the lines of
business or life-cycle stages--and managed by subordinate investment
boards, they should ultimately be aggregated into this enterprise-level
portfolio.
According to our ITIM framework, critical processes performed by Stage 3
organizations include (1) defining the portfolio criteria, (2) creating
the portfolio, (3) evaluating the portfolio, and (4) conducting
postimplementation reviews.25 Table 7 shows the purpose of each critical
process in Stage 3.
25The purpose of a postimplementation review is to evaluate an investment
after its development has been completed (i.e., after its transition from
the implementation phase to the in-service management phase) in order to
validate actual investment results. This review is conducted to (1)
examine differences between estimated and actual investment costs and
benefits and their possible ramifications for unplanned funding needs in
the future and (2) extract "lessons learned" about the investment
selection and control processes that can be used as the basis for
management improvements. Similarly, postimplementation reviews should be
conducted for investment projects that were terminated before completion,
to help to readily identify potential management and process improvements.
Table 7: Stage 3 Critical Processes-Developing a Complete Investment Portfolio
Critical process Purpose
Defining the portfolio criteria To ensure that the organization develops
and maintains IT portfolio selection criteria that support its mission,
organizational strategies, and business priorities.
Creating the portfolio To ensure that IT investments are analyzed
according to the organization's portfolio selection criteria, and that an
optimal IT investment portfolio with manageable risks and returns is
selected and funded.
Evaluating the portfolio To review the performance of the organization's
investment portfolio(s) at agreed-upon intervals, and to adjust the
allocation of resources among investments as necessary.
Conducting postimplementation reviews To compare the results of recently
implemented investments with the expectations that were set for them, and
to develop a set of lessons learned from these reviews.
Source: GAO.
CMS has executed very few key practices-2 of 27-associated with Stage 3
critical processes. Specifically, under the critical process for defining
the portfolio criteria, CMS provided evidence that it had designated a
working group to be responsible for developing and modifying portfolio
selection criteria. Under the critical process for creating the portfolio,
CMS provided evidence that it was capturing and maintaining investment
information for future reference. In its self-assessment, the agency
stated that it was not executing any other Stage 3 key practices.
According to officials, CMS has not concentrated on implementing Stage 3
key practices because the agency is first focusing its resources on
establishing the practices associated with Stage 2. Until CMS fully
implements the critical processes associated with managing its investments
as a complete portfolio, it will not have the data or enterprisewide
perspective it needs to make informed decisions about its collection of
investments.
CMS Does Not Have a Comprehensive Plan to Coordinate and Guide Its Improvement
Efforts
CMS has initiated efforts to improve its investment management process.
While these efforts do not fully address any of the weaknesses we identify
in this report, they enhance the agency's ability to perform key
activities. Specifically:
o CMS has begun to implement a tool for capturing project information.
According to officials, the tool will bring together investment
information currently residing in various locations, including project
description information captured in its IT Investment Tracking Database,
information such as project scores collected to support project selection
activities, and earned value management data.
Although information to support investment decisions does not have to be
in one location, doing so will improve accessibility and facilitate its
use by decision makers.
o CMS recently established Executive Steering Committees to support the
ITIRB in carrying out its investment management responsibilities. These
groups played a key role in selecting investments for the fiscal year 2007
budget by reviewing investment information and making recommendations for
funding to the investment board. They are currently determining procedures
for overseeing investments. According to officials, once procedures for
the Executive Steering Committee oversight have been determined, CMS plans
to focus on defining procedures for determining how and when to involve
the investment board in oversight-a key weakness identified in this
report.
Although CMS has initiated these improvement efforts, it has not
coordinated them with the additional efforts needed to address the
weaknesses identified in this report in a comprehensive plan that (1)
specifies measurable goals, objectives, and milestones; (2) specifies
needed resources; (3) assigns clear responsibility and accountability for
accomplishing tasks; and (4) is approved by senior-level management. We
have previously reported that such a plan is instrumental in helping
agencies coordinate and guide improvement efforts.
CMS officials recognize the value of having a comprehensive plan and told
us they have begun to develop one; however, a time frame for completing
the plan has not been established. Until CMS develops this plan, the
agency risks not being able to put in place an effective management
process that will provide appropriate executive-level oversight for
minimizing risks and maximizing returns.
Process for Monitoring MMISs Could Benefit from Standard Procedures, Guidance,
and Reporting Requirements
As we previously noted, the responsibility for approving and monitoring
MMISs that CMS funds jointly with the states falls to CMS's central office
and its 10 regional offices, with the bulk of the activities being
performed by the regional offices. Although the process for approving
states' funding requests for MMIS activities is characterized by (1)
standard procedures performed consistently across the regional offices,
(2) guidance that staff can rely on in carrying out their duties, and (3)
requirements for reporting information to the central office, the process
for monitoring MMIS activities is not.
Standard Procedures, Guidance, and Reporting Requirements Exist for the
Approval Process
The process for approving states' requests for federal funding of MMISs is
characterized by a defined set of activities that are performed
consistently across the regional offices. These activities include
regional office staff review and approval of the standard documentation
(i.e., the APDs, request-for-proposals, and contracts) that the states
prepare to justify their requests. Specifically, as we previously
described:
o States prepare an APD to request funding for MMISs. Regional office
staff review the document to ensure that states' requests support the
Medicaid program, are in compliance with federal requirements, and
represent cost-effective solutions. Once regional office staff determine
that the APD adequately justifies the request, they issue a formal
approval letter to the states (with concurrence from CMS's central
office).
o The request-for-proposals that the states prepare to solicit contractor
bids for MMIS activities, including development and operations, is
reviewed and approved by regional office staff through a process similar
to that used to approve the APDs.
o Regional office staff review the states' process for reviewing
contractors' proposed bids.
o Regional office staff review and approve the contract, after which the
state makes an award to the contractor whose bid or offer is responsive to
the solicitation and is most advantageous to the state-considering price,
quality, and other factors.
Regional office staff told us that they rely on the State Medicaid Manual
and the Code of Federal Regulation for guidance in performing activities
for approving states' requests for federal funding. Regional staff are
also required to inform the CMS central office of all approval actions
through the Office of the Secretary Notice process previously mentioned.
Process for Monitoring In contrast to the approval process, the process
for monitoring MMIS State MMISs Lacks Standard activities lacks (1)
standard procedures regional office staff must perform Procedures,
Guidance, and to carry out their responsibilities, (2) guidance for staff
to rely on, and (3)
requirements for staff to report on the results of their monitoring
efforts toReporting Requirements the central office.
First, regional office staff use a variety of mechanisms to monitor MMIS
activities. These mechanisms include reviews of project status reports;
site visits; telephone calls; and meetings with external groups, such as
industry associations, provider groups, and vendors. In addition, regional
office staff determine if and when to use these mechanisms. Table 8 shows
the different mechanisms used by the regional office staff we interviewed
and the number of regional offices who used them.
Table 8: Frequency of Oversight Mechanisms Used by the 5 Regional Offices
Interviewed
Number of regional offices Mechanism claiming use of mechanism
Reviews of status reports
Site visits
Meetings with provider community
In-process reviews
Meeting with vendors
Participation in status meetings
Contact with state medical society
Assistance from National Account Representativesa
Source: GAO.
aPhiladelphia office regional staff told us their office staff includes
these representatives. They are responsible for staying abreast of state
Medicaid activities. In performing their work, they communicate with the
states' Medicaid director and perform at least two visits a year to each
state.
Second, CMS has no guidance for regional office staff to use in monitoring
MMIS activities. While CMS has a Regional Office Manual that includes
guidance for monitoring MMIS activities, this manual is not used by
regional office staff because, according to officials, it has not been
maintained throughout the years, and it no longer reflects current
processes.
Third, there are no requirements for regional office staff to report to
CMS's central office on their monitoring of states' federally funded MMISs
activities. Monthly teleconferences are conducted between the central
office and regional offices to discuss activities performed by these
offices, including activities to monitor state MMISs. According to CMS
officials, there is some communication outside of the scheduled
teleconferences to discuss any issues that might arise regarding the
status of state MMISs. In
addition, according to officials, the certification reviews performed
about 6 months after the MMISs have become operational provide
opportunities to determine firsthand how systems are performing. Despite
these mechanisms, the central office has no requirements for regional
office staff to regularly report on the results of their efforts to
monitor MMIS activities.
According to CMS officials, the central office has traditionally placed
greater emphasis on the front-end approval of requests for federal
funding. The central office, however, now recognizes the need for and
value of adopting an approach for maintaining the visibility of MMISs from
beginning to end. To address this need, central office staff told us that
they plan to ask the regional offices to provide them with quarterly
reports on the status of MMIS activities in their states as part of a
broader effort that is currently under way to improve the administration
of the Medicaid program.26 Central office staff stated this effort would
also result in standard procedures and guidance to support regional office
staff's monitoring efforts. While these activities would strengthen the
monitoring process, during our review central office staff did not yet
have specific plans or time frames for implementing them.
Until CMS defines standard procedures for monitoring MMIS activities,
guidance for staff to rely on, and reporting requirements, CMS's central
office may not be able to easily determine whether state MMISs are
facilitating the delivery of Medicaid benefits in the most effective and
beneficial manner.
Conclusions Because IT investment management has only recently become an
area of management focus, CMS capabilities to manage its internal
investments are limited. Specifically, the agency has established about
half of the practices for building the investment foundation, but few
practices to manage its investments as a portfolio. Although the
foundational practices have equipped CMS with the capabilities it needs to
improve its management of individual investments, the agency is hampered
in its ability to manage them as a portfolio because it has not
implemented the practices for doing so. Until CMS fully establishes the
key practices required to build the
26This effort, known as the "Medicaid Information Technology Architecture
initiative," involves the development of a framework of enabling
technologies and processes intended to improve the administration of the
Medicaid program. CMS expects to complete this initiative within the next
2 years.
investment foundation and manage its investments as a portfolio, it will
not have the capabilities it needs to ensure that investments supporting
its multibillion-dollar Medicare and Medicaid programs are being managed
to minimize risks and maximize returns.
Critical to CMS's success in going forward will be the development of an
implementation plan that (1) is based on an assessment of strengths and
weaknesses; (2) specifies measurable goals, objectives, and milestones;
(3) specifies needed resources; (4) assigns clear responsibility and
accountability for accomplishing tasks; and (5) is approved by
senior-level management. Although the agency has initiated improvement
efforts, it has not developed a comprehensive plan to guide these and
other efforts needed to improve its investment management process. Without
such a plan and procedures for implementing it, CMS will be challenged in
sustaining the commitment it needs to fully establish its investment
management process.
Finally, the process for approving states' funding requests for MMIS
activities is characterized by standard procedures that are performed
consistently across the regional offices, guidance, and requirements for
informing the central office of regional office staff activities. The
process for monitoring the development and operations of state MMIS, on
the other hand, has no standard procedures for regional office staff, no
guidance, and no requirement to report information to the central office.
Without these elements for monitoring MMIS activities, CMS's central
office may not be able to easily determine whether state MMISs are
facilitating the delivery of Medicaid benefits in the most effective and
beneficial manner.
Recommendations for Executive Action
To strengthen CMS's capability to manage its internal IT investments and
address the related weaknesses addressed in this report, we recommend that
the Secretary of the Department of Health and Human Services direct the
CMS Administrator to develop and implement a plan for improving CMS's IT
investment management processes. The plan should address the weaknesses
described in this report. The plan should (1) first focus on correcting
the weaknesses in Stage 2 critical processes, and next focus on the Stage
3 critical processes, and (2) at a minimum, provide for accomplishing the
following 12 actions:
In Stage 2
o Update the agency's investment management guide to reflect current
investment management processes.
o Establish a process for the board to actively maintain the agency's
documented investment management process.
o Use an updated strategic plan or other detailed statement of business
mission with supporting goals and objectives to align investments with
business needs.
o Ensure that the board periodically evaluates the alignment of IT
projects and systems with strategic goals and objectives and take
corrective actions when misalignment occurs.
o Fully document procedures that address investment selection and
reselection and (1) provide a clear understanding of the selection and
reselection process, (2) define the roles and responsibilities for each
participating unit involved in the project reselection process, and (3)
define the decision-making procedures.
o Document procedures for integrating funding with investment selection.
o Revise the ITIRB's selection and reselection criteria to include cost,
benefit, schedule, and risk factors, and establish a mechanism to ensure
these criteria continue to reflect organizational objectives.
o Define, document, and implement procedures for the ITIRB's oversight of
projects and systems.
o Implement processes to use investment information to fully support
investment management decisions.
In Stage 3
o Implement the Stage 3 critical processes for defining portfolio
criteria, creating the portfolio, evaluating the portfolio, and conducting
postimplementation reviews, which are necessary for portfolio management.
We also recommend that the Secretary for Health and Human Services direct
the CMS Administrator to ensure that the plan draw together ongoing
efforts and additional efforts that are needed to address the weaknesses
identified in this report. The plan should also (1) specify measurable
goals, objectives, and milestones; (2) specify needed resources; (3)
assign clear responsibility and accountability for accomplishing tasks;
and (4) be approved by senior-level management. In implementing the plan,
the Administrator should ensure that progress is measured and reported
periodically to the Secretary of Health and Human Services.
To improve CMS's process for monitoring states' progress in developing and
maintaining Medicaid management information systems, we are recommending
that the Secretary of the Department of Health and Human Services direct
the CMS Administrator to take the following two actions:
o Define standard procedures and supporting guidance for regional offices
to monitor MMIS activities.
o Require regional offices to regularly report on their MMIS monitoring
activities to CMS's central office.
Agency Comments and Our Evaluation
The Administrator of CMS provided written comments on a draft of this
report (reprinted in app. II). In these comments, CMS identified actions
it is taking or plans to take to address our recommendations and stated
that effective management of IT investments is a critical priority at the
agency. CMS contended that many of the agency's improvements to its IT
investment management process were not fully reflected in the report, and
took exception with the need for up-to-date, documented processes to
ensure consistency.
Concerning our description of progress in implementing investment
management processes, CMS commented that the report indicates that the
agency has only established 2 out of 27 key practices needed to manage
investments as a portfolio. CMS stated that this is misleading since the
report also indicates that the agency has accomplished 20 of 38
foundational IT investment management practices. CMS also provided
examples of the practices it has implemented, such as establishing an
investment review board. In our report, we make a distinction between
foundational practices, which are the Stage 2 key practices for
establishing basic project-level selection and control capabilities, and
portfolio-level
practices, which are the Stage 3 key practices for managing investments as
an integrated set of competing options. We also note that both of these
sets of key practices are needed to implement the processes required by
the Clinger-Cohen Act of 1996. On the basis of this, we state that CMS
does not have the full suite of capabilities to manage its internal
investments because it has only established a little over half of the
foundational practices and 2 of 27 portfolio-level key practices, and we
reiterate the need to fully establish both sets of practices to increase
assurance that executives are selecting and managing the mix of
investments that best meets the agency's needs and priorities. In our
report, the sections in which we discuss the implementation of specific
key practices associated with critical processes from our IT investment
management framework each describe CMS's efforts and accomplishments to
improve its IT investment management processes. These include all of the
examples of accomplishments CMS provided in its comments.
In its comments, CMS took issue with our reporting that its IT investment
management guide did not reflect the current process, and that its
procedures for selecting and reselecting IT investments were not fully
documented. Although the agency fully agreed that an up-to-date guide
would constitute a piece of an effective process, it commented that the
emphasis should be on strengthening the process first and updating
documentation later. CMS made three points: (1) it is not practical to
publish an updated guide without having the effective and repeatable
underlying process in place and noted that it is not provided the latitude
to do this; (2) in the section of the report discussing instituting the
investment board, the noted successful execution of key practices appears
to be negated by the statement that the investment management processes
are not documented; and (3) in the same section of the report, we are
implying that an updated guide would improve rather than explain the
process. We disagree with CMS that the process needs to be repeatable and
strengthened before it can be documented. Documented procedures could
actually serve to strengthen and improve the process by ensuring it is
performed consistently. Finally, we are not negating the successful
implementation of key practices to institute the investment board. We are
simply emphasizing the importance of having documentation to drive the
investment management process.
In its comments, CMS also noted actions it is taking to (1) develop a plan
to implement key practices in Stages 2 and 3; (2) revise existing
documentation to reflect processes in place that are not formally
documented; and (3) develop a plan that will be approved by senior
management that will incorporate goals, objectives, and milestones
required to further close the gaps between existing processes and our ITIM
framework. Regarding our recommendation to improve its process for
monitoring state MMIS activities and reporting to the central office, CMS
stated that it is developing standard procedures and supporting guidance
for the regional office(s) for monitoring these systems activities and
reporting to the central office. We agree with CMS that these actions
would address many of the weaknesses we identify in this report.
CMS also provided some technical comments, which we have incorporated into
the report as appropriate.
As agreed with your offices, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days from
the
date of this letter. At that time, we will send copies to other interested
congressional committees, the Secretary of Health and Human Services,
the CMS Administrator, the CMS Chief Information Officer, and other
interested parties. Copies will also be made available at no charge on our
Web site at www.gao.gov. If you have any questions on matters discussed in
this report, please contact David A. Powner at (202) 512-9286, or at
pownerd@gao.gov, or Leslie G. Aronovitz at (312) 220-7600, or at
aronovitzl@gao.gov. Contact points for our Offices of Congressional
Relations and Public Affairs may be found on the last page of this report.
GAO staff who made major contributions to this report are listed in
appendix III.
Sincerely yours,
David A. Powner
Director, Information Technology Management Issues
Leslie G. Aronovitz
Director, Health Care
Appendix I
Objectives, Scope, and Methodology
The objectives of our review were to (1) evaluate Centers for Medicare &
Medicaid Services (CMS) capabilities for managing its internal information
technology (IT) investments, (2) determine any plans the agency might have
for improving these capabilities, and (3) examine CMS's process for
approving and monitoring the state Medicaid management systems it funds.
To address our first objective, we reviewed the results of the agency's
selfassessment of Stages 2 and 3 practices using our Information
Technology Investment Management framework (ITIM) and validated and
updated the results of the self-assessment through document reviews and
interviews with officials. We reviewed written policies, procedures, and
guidance and other documentation providing evidence of executed practices,
including CMS's IT Investment Management Process Guide, CMS's Policy for
IT Investment Management, and CMS's fiscal year 2006/2007 budget process.
We also reviewed the CMS Information Technology Investment Review Board
(ITIRB) meeting minutes. We did not assess CMS's progress in establishing
the capabilities found in Stages 4 and 5 of the ITIM framework because CMS
acknowledged that it had not executed any of the key practices in these
higher maturity stages. In addition, we conducted interviews with
officials from the Office of Information Services who have responsibility
for the development and implementation of CMS's IT investment management
process.
We compared the evidence collected from our document reviews and
interviews with the key practices in our ITIM framework. We rated the key
practices as "executed" on the basis of whether the agency demonstrated
(by providing evidence of performance) that it had met the criteria of the
key practice. A key practice was rated as "not executed" when we found
insufficient evidence of a practice being executed or when we determined
that there were significant weaknesses in CMS's execution of the key
practice. In addition, CMS was provided with the opportunity to produce
evidence for key practices rated as "not executed."
As part of our analysis, we selected four CMS IT projects as case studies
to verify that the critical processes and key practices were being
applied. The projects were selected because they (1) supported different
functional areas, (2) were in various life-cycle phases, and (3) required
different levels of funding. The four projects are described below:
o Healthcare Integrated General Ledger Accounting System-The project is
intended to standardize the collection, recording, and reporting of
Appendix I
Objectives, Scope, and Methodology
Medicare financial information by contractors. It is to replace the
cumbersome ad hoc spreadsheets and "cuff" systems being used by Medicare
contractors to accumulate and report financial information to CMS. The
project's life-cycle cost is estimated at about $567 million.
o Medicare Claims Processing Redesign-This project is intended to
integrate and modernize the Common Working File system and Redesign and
the Medicare Shared Systems enterprise claims processing applications and
data systems. The modernization and unification of these systems is to
allow CMS to significantly enhance program capabilities, integrity,
performance, efficiencies, and maintainability; reduce program change
implementation time frames; improve accuracy, timeliness, and quality of
Medicare transaction processing; reduce system exposure to security risks;
and facilitate use of the Internet. The project's life-cycle cost is
estimated at nearly $494 million.
o Medicare Managed Care System-This project is intended to cover the
redesign of CMS's managed care family of systems, including the legacy
Group Health Plan system. It is to provide the platform for implementing
requirements under the MMA. The project is intended to replace aging
operations and to continue to support the agency's managed care business
needs until all functions are migrated to a new system. Its lifecycle cost
is estimated at about $111 million.
o National Plan and Provider Enumeration System-The project is intended
to implement a Health Insurance Portability and Accountability Act
requirement to issue a unique identifier to each covered health care
provider in the United States. It is expected to result in administrative
savings by simplifying a complicated, multifaceted enumeration scheme,
whereby a provider is issued different identifiers for electronic
transactions by each health plan with which it does business, and
sometimes multiple identifiers from a single plan. It will impact several
million providers and health plans in the nation. The project's life-cycle
cost is estimated at about $38 million.
For these projects, we reviewed project management documentation, such as
project plans, business cases, status reports, and documentation on how
these projects were selected by the ITIRB. We also interviewed the project
managers for these projects.
Appendix I
Objectives, Scope, and Methodology
To address our second objective, we examined documentation on what
management actions had been taken and what initiatives had been planned by
the agency. This documentation included a requirements document for a tool
CMS is currently implementing that is to help the agency with IT
investment management, among other things. We also interviewed officials
from the Office of Information Services to determine what efforts CMS had
undertaken to improve IT investment management processes.
To address our third objective, we reviewed documentation supporting CMS's
implementation of processes for (1) approving states' requests for funding
their Medicaid Management Information Systems (MMIS) and (2) monitoring
these MMISs, including related legislation, policy, and implementing
guidance. We also interviewed officials at CMS headquarters and at the 5
CMS regional offices with the highest fiscal year 2004 expenditures for
administrative services, which includes MMISs.
We conducted our work at CMS headquarters in Washington, D.C., and at 5
CMS regional offices located in New York, New York; Philadelphia,
Pennsylvania; Chicago, Illinois; San Francisco, California; and Atlanta,
Georgia, from January 2005 through September 2005 in accordance with
generally accepted government auditing standards.
Appendix II
Comments from the Centers for Medicare & Medicaid Services
Appendix II
Comments from the Centers for Medicare &
Medicaid Services
Appendix II
Comments from the Centers for Medicare &
Medicaid Services
Appendix II
Comments from the Centers for Medicare &
Medicaid Services
Appendix III
GAO Contacts and Staff Acknowledgments
GAO Contacts David A. Powner, (202) 512-9286, pownerd@gao.gov Leslie G.
Aronovitz, (312) 220-7600, aronovitzl@gao.gov
Staff In addition to the persons named above, William G. Barrick,
Shaunessye Curry, Mary Beth McClanahan, Sabine R. Paul, and Amos Tevelow
made
Acknowledgments key contributions to this report.
GAO's Mission
Obtaining Copies of GAO Reports and Testimony
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061
To Report Fraud, Contact:
Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.govFederal Programs Automated answering system: (800)
424-5454 or (202) 512-7470
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548
Public Affairs Paul Anderson, Managing Director, AndersonP1@gao.gov (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
PRINTED ON RECYCLED PAPER
*** End of document. ***