Medicare Transaction System: Success Depends Upon Correcting Critical
Managerial and Technical Weaknesses (Chapter Report, 05/16/97,
GAO/AIMD-97-78).

Pursuant to a congressional request, GAO reviewed the Health Care
Financing Administration's (HCFA) acquisition of its Medicare
Transaction System (MTS), focusing on the extent to which HCFA is: (1)
effectively managing its interim Medicare processing environment; (2)
using required practices to manage MTS as an investment; and (3)
applying sound system development processes to reduce risk.

GAO noted that: (1) HCFA recognizes that the multiple Medicare
claims-processing systems are difficult to administer, and is looking to
MTS to achieve substantial administrative savings, increase claims
control, and improve customer service; (2) to its credit, HCFA is using
a phased development approach to reduce risks and is beginning to apply
investment practices in its management of the MTS project; (3) however,
the benefits of modernizing Medicare claims processing at an estimated
cost of $1 billion will not succeed unless HCFA overcomes serious
management and technical weaknesses in three major areas that place the
modernization at great risk; (4) HCFA needs to greatly improve its
management of the essential interim Medicare processing environment and
the changes necessary for operating beyond 2000; (5) to successfully
process the claims workload, consolidate existing processing sites,
address year 2000-related systems problems, and convert from nine
systems to two, careful planning is necessary; (6) however, such
planning has not yet been done; (7) HCFA intends to develop these plans,
but has not specified when; (8) the risks associated with concurrently
converting major systems while at the same time managing ongoing
development of MTS is magnified by the fact that changing from the
existing claims processing environment to MTS is a larger, more complex
systems-conversion challenge than anything HCFA has previously faced;
(9) a schedule and estimate of needed resources for each major stage of
the transition to the interim processing environment would ensure the
availability of needed resources and help HCFA better manage and
coordinate transition activities; (10) MTS is not being adequately
managed as an investment; (11) HCFA has not followed practices that are
essential if management is to make informed technology investment
decisions, including preparing a valid cost-benefit analysis,
considering viable alternatives, and fully evaluating how the proposed
technology benefits will contribute to improvements in mission
performance; (12) HCFA has not applied all the sound systems-development
practices necessary to reduce the risk and assist management in
controlling development of system requirements and software; and (13)
along with not developing plans critical to the project's success, HFCA
has not adequately overseen its contractors software development strate*

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  AIMD-97-78
     TITLE:  Medicare Transaction System: Success Depends Upon 
             Correcting Critical Managerial and Technical Weaknesses
      DATE:  05/16/97
   SUBJECT:  Claims processing
             Computer software verification and validation
             Concurrency
             Cost effectiveness analysis
             Strategic information systems planning
             Requirements definition
             Systems design
             Information resources management
             Systems conversions
             Medical information systems
IDENTIFIER:  HCFA Medicare Transaction System
             Software Capability Maturity Model
             Medicare Program
             Florida Shared System
             Medicare Secondary Payer Program
             
**************************************************************************
* This file contains an ASCII representation of the text of a GAO        *
* report.  Delineations within the text indicating chapter titles,       *
* headings, and bullets are preserved.  Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are           *
* identified by double and single lines.  The numbers on the right end   *
* of these lines indicate the position of each of the subsections in the *
* document outline.  These numbers do NOT correspond with the page       *
* numbers of the printed product.                                        *
*                                                                        *
* No attempt has been made to display graphic images, although figure    *
* captions are reproduced. Tables are included, but may not resemble     *
* those in the printed version.                                          *
*                                                                        *
* A printed copy of this report may be obtained from the GAO Document    *
* Distribution Facility by calling (202) 512-6000, by faxing your        *
* request to (301) 258-4066, or by writing to P.O. Box 6015,             *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time.                                    *
**************************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

May 1997

MEDICARE TRANSACTION SYSTEM -
SUCCESS DEPENDS UPON CORRECTING
CRITICAL MANAGERIAL AND TECHNICAL
WEAKNESSES

GAO/AIMD-97-78

Medicare Transaction System

(511212)


Abbreviations
=============================================================== ABBREV

  AIMD - Accounting and Information Management Division
  CCA - Clinger-Cohen Act
  CIO - chief information officer
  CMM - capability maturity model
  FASA - Federal Acquisition and Streamlining Act
  GAO - General Accounting Office
  HCFA - Health Care Financing Administration
  HHS - Department of Health and Human Services
  IRM - information resources management
  IV&V - independent verification and validation
  MTS - Medicare Transaction System
  OMB - Office of Management and Budget
  PRA - Paperwork Reduction Act of 1995
  SA-CMM - Software Acquisition Capability Maturity Model
  SAF - systems assessment framework
  SEI - Software Engineering Institute
  SLIM - software life-cycle management
  SA -
  CCA -
  PRA -

Letter
=============================================================== LETTER


B-275281

May 16, 1997

The Honorable Christopher Shays
Chairman
The Honorable Edolphus Towns
Ranking Minority Member
Subcommittee on Human Resources
Committee on Government Reform and Oversight
House of Representatives

The Honorable Stephen Horn
Chairman
The Honorable Carolyn Maloney
Ranking Minority Member
Subcommittee on Government Management,
 Information and Technology
Committee on Government Reform and Oversight
House of Representatives

This report responds to your request that GAO evaluate the Health
Care Financing Administration's (HCFA) acquisition of its Medicare
Transaction System (MTS).  Specifically, we addressed the extent to
which HCFA is (1) effectively managing its interim Medicare
processing environment, (2) using required practices to manage MTS as
an investment, and (3) applying sound system development processes to
reduce risk.  The report makes several recommendations that could
help HCFA effectively meet its Medicare information technology needs
and decrease the risk of costly technological decisions and wasted
federal expenditures that are often associated with large federal
information technology projects. 

We are sending copies of this report to the Secretary of Health and
Human Services, Administrator of the Health Care Financing
Administration, Director of the Office of Management and Budget, and
appropriate congressional committees.  We will also make copies
available to others upon request. 

Please call me at (202) 512-6253 if you or your staff have any
questions concerning this report.  You can also reach me by e-mail at
[email protected].  Major contributors to this report are
listed in appendix III. 

Joel C.  Willemssen
Director, Information Resources Management


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

Medicare, the nation's largest health insurer, expects to process
over 1 billion claims and pay $288 billion in benefits per year by
2000.  The Health Care Financing Administration (HCFA) is responsible
for administering this program under the Department of Health and
Human Services (HHS).  Nine separate automated information systems
have been used to process Medicare claims.  HCFA plans to spend about
$1 billion to replace these systems with a single, unified
system--the Medicare Transaction System (MTS).  HCFA estimates that
MTS will begin replacing the existing systems in 1998, providing
improved service, reduced operating expenses, better contractor
oversight, and more protection of program funds from waste, fraud,
and abuse, while also accommodating managed care and alternative
payment methodologies. 

At the request of the Subcommittees on Human Resources, and
Government Management, Information, and Technology of the House
Committee on Government Reform and Oversight, GAO reviewed the extent
to which HCFA is (1) effectively managing its interim
claims-processing, including planning for and correcting year
2000-related computer problems, (2) using required practices to
manage MTS as an investment, and (3) applying sound
systems-development practices to reduce risk. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

HCFA manages the Medicare program through about 70 contractors who
process claims and pay benefits at about 45 sites nationwide.  It
considers MTS an important part of its plans to improve the Medicare
claims process.  By replacing Medicare's multiple,
contractor-operated claims-processing systems with a single system,
HCFA believes it will obtain significant benefits.  For example, HCFA
explained that when changes in legislative or administrative policies
require changes in Medicare payments or coverage, each of the
existing claims-processing systems must be individually modified--an
expensive, time-consuming process.  Under MTS, only one system would
need to be modified. 

HCFA has made major changes to the development and implementation
plans for MTS since it was begun.  In previous reviews, GAO
identified the need to reduce MTS acquisition risks through an
incremental deployment approach.\1 HCFA now plans such an approach. 
In an effort to improve its MTS initiative, reduce risk, and achieve
some savings before MTS is implemented, HCFA is simultaneously
undertaking several interim actions while continuing its development
of MTS. 

HCFA's interim efforts consist of selecting a single part A and
single part B system from the nine existing systems and consolidating
the data processing workload, thereby reducing the number of
processing sites from 45 to about 20.  (Medicare part A covers
institutional care, while part B covers physician, supplier, and
other outpatient services.) HCFA plans to move the data processing
workload from the interim consolidated processing sites to two
planned MTS claims-processing sites by mid-1998.  By then, HCFA also
plans to have its contractors revise their systems to accommodate
year-2000 processing.\2

On April 4, 1997, HCFA announced that, as a result of a recent
management review, it was redirecting its MTS contractor to focus
solely on the managed care module--the first of six planned software
releases--while it examines alternative ways to achieve its MTS
goals.  HCFA concluded that its vision of MTS as the best information
technology to take Medicare into the 21st century had not changed. 


--------------------
\1 Medicare:  New Claims Processing System Benefits and Acquisition
Risks (GAO/HEHS/AIMD-94-79, Jan.  25, 1994) and Medicare Transaction
System:  Strengthened Management and Sound Development Approach
Critical to Success (GAO/T-AIMD-96-12, Nov.  16, 1995). 

\2 Due to the use of the two-digit format for dates in many computer
systems, the year 2000 (represented "00") will be indistinguishable
from the year 1900 (also represented "00").  As a result, unless
modifications are made, system or application programs that use dates
to perform calculations, comparisons, or sorting may generate
incorrect results when working with dates after 1999 as a result of
reading "00" as "1900."


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

HCFA recognizes that the multiple Medicare claims-processing systems
are difficult to administer, and is looking to MTS to achieve
substantial administrative savings, increase claims control, and
improve customer service.  To its credit, HCFA is using a phased
development approach to reduce risks and is beginning to apply
investment practices in its management of the MTS project.  However,
the benefits of modernizing Medicare claims processing at an
estimated cost of $1 billion will not be realized unless HCFA
overcomes serious management and technical weaknesses in three major
areas that place the modernization at great risk. 

First, HCFA needs to greatly improve its management of the essential
interim Medicare processing environment and the changes necessary for
operating beyond 2000.  To successfully process the claims workload,
consolidate existing processing sites, address year 2000-related
systems problems, and convert from nine systems to two, careful
planning is necessary.  However, such planning has not yet been
completed.  For example, although HCFA has already begun to convert
its existing systems and consolidate its sites, it has not developed
plans to guide these activities.  Such plans should include a
schedule and estimate of resources required for the transition,
details defining systems-contractor responsibilities, and an approach
for addressing potential year-2000 problems. 

The risks associated with concurrently converting major systems while
at the same time managing ongoing development of MTS is magnified by
the fact that changing from the existing claims processing
environment to MTS is a larger, more complex systems-conversion
challenge than anything HCFA has previously faced.  Further, HCFA is
relying on its Medicare systems contractors to assess, plan, and
implement essential changes for the year-2000 issue, but is not
closely monitoring these critical activities or receiving
certifications or assurances from contractors that the problems will
be corrected. 

A schedule and estimate of needed resources for each major stage of
the transition to the interim processing environment would ensure the
availability of needed resources and help HCFA better manage and
coordinate transition activities.  HCFA agreed that a transition
schedule and resource estimates would be helpful; a contractor is to
assist HCFA in preparing and integrating them into the overall MTS
development schedule, to be completed by late this spring. 

Second, MTS is not being adequately managed as an investment.  HCFA
has not followed practices that are essential if management is to
make informed technology investment decisions, including preparing a
valid cost-benefit analysis, considering viable alternatives, and
fully evaluating how the proposed technology benefits will contribute
to improvements in mission performance.  HCFA estimates that many
programmatic savings will result from automated edits that identify
abusive billing practices and deny related claims.  However, HCFA
stated that because the exact nature of MTS' edits had not been
identified, the resulting savings could be significantly different
from the estimated savings.  Also, since 1992 when the first analysis
was completed, the total cost of this project has increased from $151
million to about $1 billion.\3 The $1 billion includes estimated
costs to transition to the MTS environment and acquire operating
sites. 

Finally, HCFA has not applied all the sound systems-development
practices necessary to reduce risk and assist management in
controlling development of system requirements and software.  Along
with not developing plans critical to the project's success, the
agency has not adequately overseen its contractor's software
development strategy, adequately managed the project's schedule, or
implemented a program to effectively address risk.  More
specifically, deficiencies in several critical systems-development
processes provide early warning of weaknesses in the management
capability of HCFA itself and of its contractors.  Plans are
inadequate or completed too late, the schedule is incomplete and
contains overlap in development phases, HCFA's risk-management
process is inadequate, and its oversight has not prevented an unsound
software-development strategy.  These factors all increase the risk
that MTS cannot be developed into the management information tool
that HCFA needs. 


--------------------
\3 All dollar amounts presented in this report are expressed as
undiscounted current dollars. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      INEFFECTIVE MANAGEMENT OF
      INTERIM PROCESSING
      ENVIRONMENT
-------------------------------------------------------- Chapter 0:4.1

Generally accepted program management practices call for preparing
detailed plans for system transitions and modifications, which for
HCFA should include (1) schedule and resource estimates, (2) defined
responsibilities of the selected Medicare part A and part B systems
contractors, (3) test plans, and (4) performance measures.  These
plans are particularly important for HCFA because the interim
transition is a larger systems-conversion effort than any previously
undertaken by HCFA and is being performed concurrently with HCFA's
management of MTS development. 

HCFA has recently taken some actions and plans others.  It hired a
consultant to help it prepare a schedule and resource estimates for
the transition, and awarded a contract defining responsibilities for
the selected part B system on April 8, 1997.  The part A systems
contractor's statement of responsibilities is to be completed by the
end of May of this year. 

According to HCFA officials, their contractors routinely test and
implement systems changes, and they plan to rely on their contractors
to successfully test and implement the changes occurring during the
transition.  However, the transition differs from routine changes. 
The contractors may not have a particularly high incentive to
properly make these conversions, since HCFA plans to eliminate these
contractors when MTS is fully implemented.  Also, these conversions
involve significantly more data and will require more system capacity
than routine modifications.  Further, HCFA officials said they do not
believe it would be cost-beneficial to use the agency's limited
resources to develop performance measures for interim systems that
will be replaced by MTS.  GAO believes these measures are needed to
ensure that this complex and important interim phase is properly
implemented and delivers the benefits expected. 

HCFA has also not taken enough initial actions to ensure that it can
avoid the systems-related service disruptions that may occur as the
year 2000 approaches.  For example, it has not developed an
assessment of the potential severity of the impact of the century
change or completed a plan for addressing it.  Further, HCFA has not
required systems contractors to submit year-2000 plans for approval. 
It lacks any specific legal agreements with its contractors
addressing year-2000 problems, including how, when, or even if such
problems will be corrected.  The potential risks associated with not
being ready for 2000 are serious, since virtually all Medicare
transactions depend, to some degree, on dates to determine benefits
eligibility--dates of birth, medical procedure, other insurance
coverage, and so forth.  Unless corrected, if a computer system were
to read "00" as 1900 instead of 2000, someone born in 1925 would be
seen as negative 25 years old--not even born yet--rather than the
actual age of 75. 

HCFA is now surveying its contractors about this issue.  The agency
has also asked the contractors to provide estimates showing when
their systems will be year-2000 compliant.  However, HCFA has no
plans to independently validate the contractors' strategies and test
plans.  HCFA likewise has no plans to approve contractors' approaches
for addressing interface and data exchange issues. 


      MTS NOT BEING ADEQUATELY
      MANAGED AS AN INVESTMENT
-------------------------------------------------------- Chapter 0:4.2

Federal legislation and Office of Management and Budget (OMB)
directives require agencies to manage major information technology
acquisitions as investments.  Critical elements of technology
investment decision-making are processes and data that ensure that
(1) the right project proposals are funded on the basis of management
evaluations of costs, risks, and expected benefits to mission
performance and (2) once funded, projects are controlled by examining
costs, the development schedule, and actual versus expected results. 

These goals are accomplished through preparing valid cost-benefit
analyses, considering viable alternatives, having senior management
consistently make key decisions on major projects, and ensuring that
the projects support the agency's mission and goals.  Because HCFA
has not implemented these critical elements, it has no assurance that
its MTS development will reduce risks to the greatest extent
possible, and achieve a maximum return on its MTS investment. 

Since 1992 HCFA has prepared numerous documents showing MTS'
estimated costs and benefits.  However, all of these documents are
flawed in that not all costs were identified, projected savings were
overstated or not adequately supported, and alternative solutions
were not considered.  As a result, reliable cost or benefit estimates
for MTS are not yet available.  In such an information vacuum, it is
impossible to manage a complex technology project such as MTS as an
investment because no basis exists on which to predict likely short-
or long-term results, and compare them against resources spent. 
Furthermore, recent data show that an early key HCFA assumption was
invalid.  HCFA's 1993 analysis assumed that, without MTS, costs per
claim for part A and part B would continually increase from 1993
through 2002.  However, actual contractor cost reports show that
costs per claim for part A and part B decreased for fiscal years 1994
through 1996, from $1.41 to $1.27, and from $0.89 to $0.88,
respectively. 

Beyond monetary uncertainties, decision-making without alternatives
analysis adds risk.  The decision to consolidate a daily
claims-processing workload of about 2.6 million claims at two new
sites yet to be acquired was made without considering other
alternatives, such as using existing processing centers. 

Related to a sound cost-benefit analysis is the level of management
oversight.  While HCFA is making positive changes, such as
designating a chief information officer and establishing an
investment review board, consistent senior-level involvement and
investment-based decision-making are still lacking.  HCFA's executive
decision-making group--the MTS management board--has not made many of
the critical MTS investment decisions, and HCFA has not adequately
linked MTS to its agency goals or mission; likewise, it has not
established performance measures to evaluate how well MTS supports
the goals or programs of the agency. 


      LACK OF SOUND
      SYSTEMS-DEVELOPMENT
      PRACTICES
-------------------------------------------------------- Chapter 0:4.3

Sound systems-development practices are not being followed for MTS. 
HCFA lacks critical project development plans, including plans for
requirements management, software development, and systems
integration.  Without these plans, it is difficult for HCFA to
appropriately manage and monitor the MTS contract and apply sound
systems-development practices.  HCFA's contractor oversight likewise
departs from critical software development best practices, including
an assessment of the software capability level of the MTS development
contractor, use of specific software measures to assess the quality
of software development, and use of sound software-estimation
assumptions to make reasonable project cost and time estimates.  Not
embracing such practices threatens MTS' quality, timeliness, and
cost. 

For example, HCFA's lack of a requirements-management plan
contributed to several redirections that resulted in schedule delays. 
The approach toward defining requirements has been changed, not all
requirements are yet defined, many that have been defined have not
been formally agreed to, and their volatility can affect cost and
development.  For example, during a recent 5-month period, the
requirements for one software release dropped from 1,639 to 1,499,
while the requirements for another release increased from 631 to 868. 

In addition, the MTS development schedule contains serious flaws. 
Resource needs have not been included, a critical path (i.e., the
sequence of dependent tasks that, if delayed, will delay the entire
project) has not been identified, and project development
phases--designed to be sequential--are often concurrent.  As a
result, HCFA cannot rely on the MTS program schedule for management
decision-making. 

Finally, weaknesses in HCFA's MTS risk management process have not
been addressed.  HCFA has no quantitative analysis of the cost,
schedule, or systems performance impact of identified risks to MTS
development; overall responsibility for MTS risk management has not
been assigned.  At the same time, several long-standing critical
risks remain, and serve as a warning that effective risk management
practices have not been institutionalized or uniformly practiced. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

In order to help HCFA effectively manage its interim Medicare
processing environment, GAO recommends that the Secretary of Health
and Human Services direct that the Administrator of the Health Care
Financing Administration take the following steps: 

  Prepare a plan that provides details of the transition to the
     single Medicare part A and part B systems, and defines how
     systems will be converted to address potential year-2000
     problems.  (See chapter 2.)

  Prepare plans for conducting thorough testing before converting
     part A and part B systems.  (See chapter 2.)

  Establish a means of assessing performance in the crucial early
     stages of the transition, and apply any lessons learned to
     planning for MTS.  (See chapter 2.)

  Help ensure the reliable operation of its systems through the year
     2000 by identifying responsibilities for managing and monitoring
     year-2000 actions, preparing an assessment of the severity and
     timing of potential year-2000 impact, developing contingency
     plans for critical systems in the event of failure, and
     regularly reporting to HHS on its progress.  (See chapter 2.)

In addition, in order to help HCFA develop and implement MTS, and
minimize unnecessary spending in the process, GAO recommends the
Secretary of Health and Human Services withhold funding for the MTS
operating site contracts until HCFA cost-justifies them.  (See
chapter 3.) GAO also recommends that the Secretary of Health and
Human Services direct that the Administrator of the Health Care
Financing Administration do the following: 

  Justify the continuation of MTS by producing a valid cost-benefit
     and alternatives analysis that includes goals for reaching
     programmatic savings and links estimated savings to specific
     improvements in Medicare claims processing, and take appropriate
     action based on the results of the analysis.  (See chapter 3.)

  Establish an investment management approach for MTS by explicitly
     linking the roles and responsibilities of the CIO and the
     Investment Review Board to relevant legislative mandates and
     requirements, which include (1) designing and implementing a
     process for maximizing the value and assessing and managing the
     risks of information technology acquisitions, and integrating
     that process with the budget, financial, and program management
     decisions of the agency, (2) providing the means for senior
     management to obtain timely information regarding the progress
     of an investment, including a system of milestones for measuring
     progress, in terms of cost, capability of the system to meet
     specified requirements, timeliness, and quality, and (3)
     ensuring that performance measures are applied to measure how
     well information technology projects support the goals and
     missions of the agency.  (See chapter 3.)

  Complete and implement those plans that are critical to effective
     systems development, including a requirements management plan,
     software development plan, configuration management plan, and
     systems integration plan.  (See chapter 4.)

  Require an independent evaluation of the MTS contractor's software
     development capability prior to beginning the software
     development phase.  To ensure that the contractor's MTS
     development team has the capability required for reasonable
     assurance of success, it should achieve a rating of at least
     level 2.  (See chapter 4.)

  Complete a new, integrated MTS program schedule that includes a
     critical path for the entire initiative, including the interim,
     and resources and costs for each task; it should also minimize
     overlap in the phases of the systems-development process.  (See
     chapter 4.)

  Mitigate critical risks by designating an accountable official for
     risk management and ensuring that this individual implements a
     process that will (1) identify and quantify all significant
     risks, (2) establish time frames for assessing risk status and
     specifying target dates for risk mitigation, (3) develop metrics
     that will compare progress in assessing the effectiveness of
     risk mitigation efforts, (4) provide a mechanism for alerting
     management early of risks that are becoming imminent, (5)
     provide resource estimates of staff, schedule needs, and funding
     to address identified risks, (6) ensure that the MTS risk
     management database incorporates all identified risks, and (7)
     document interdependencies among risks.  (See chapter 4.)

Additional GAO recommendations to improve the transition, management,
development, and implementation of MTS are in chapters 2, 3, and 4. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 0:6

We requested written comments from the Secretary of the Department of
Health and Human Services and the Director of the Office of
Management and Budget.  HHS' Assistant Secretary for Management and
Budget agreed with GAO's recommendations for effectively managing
Medicare's interim claims-processing environment, using required
practices to manage MTS as an investment, and applying sound
systems-development practices.  HHS said that both the Department and
HCFA are committed to complying with these recommendations.  Further,
HHS said that steps have already been initiated to implement several
of GAO's recommendations, including (1) preparing detailed plans for
the transition to the single Medicare part A and part B systems, (2)
requesting implementation plans from its contractors on their
progress in making their systems millennium-compliant, and (3)
reassessing the cost, benefits, and alternative development
strategies to MTS.  HCFA concluded that GAO has been of significant
assistance in offering suggestions for improved management. 

In comments on a draft of this report, OMB's Deputy Director for
Management agreed with GAO's recommendations for improved management
of MTS and concurred that HCFA must take steps to more adequately
plan for the consolidation to the standardized part A and part B
system.  It also concurred that HCFA needs to better manage MTS as an
investment, and apply sound systems-development practices to reduce
risk and assist management in controlling the development of systems
requirements and software. 

GAO believes that by effectively implementing its recommendations
HCFA will improve the management of its modernization effort and
increase its assurance that the approach taken will be
cost-effective, risk-averse, and support the agency's mission and
goals. 

These comments are discussed in chapters 2, 3, and 4 and are
reprinted in appendixes I and II. 


INTRODUCTION
============================================================ Chapter 1

Medicare is the nation's largest health insurer, serving about 38
million Americans by providing federal health insurance to
individuals 65 or older and to many of the nation's disabled.  It now
provides over $200 billion in health care benefits annually. 
Medicare's day-to-day operations are run by the Health Care Financing
Administration (HCFA) under the Department of Health and Human
Services (HHS).  HCFA uses about 70 intermediary and carrier claims
processing contractors to administer the Medicare program. 
Intermediaries are the contractors that handle part A claims
submitted by hospitals, skilled nursing facilities, hospices, home
health agencies, and rehabilitation agencies.  Carrier contractors
handle part B claims submitted by physicians, laboratories, equipment
suppliers, outpatient providers, and other practitioners.  In
December 1996, contractors were using three different systems to
process part A claims and six different systems to process part B
claims. 

Medicare is expected to process over 1 billion claims and pay $288
billion in benefits per year by 2000.  With more claims creating a
rapidly expanding workload, HCFA is planning to replace its current
claims-processing systems in order to be able to handle the expected
increase in numbers of claims and provide better service to its
customers. 


   THE MEDICARE TRANSACTION SYSTEM
---------------------------------------------------------- Chapter 1:1

In January 1994, as part of its plans to improve the efficiency and
effectiveness of Medicare program operations, HCFA awarded a contract
to a software developer to design, develop, and implement a new
government-owned, automated claims-processing information system, the
Medicare Transaction System (MTS).\1 HCFA intends to replace the
claims processing functions being performed by the nine different
systems with a single, unified MTS having improved capabilities to
help achieve significant advances in Medicare management and
operations. 

The specific goals of MTS are to improve service to beneficiaries and
providers; reduce administrative expenses; allow better oversight of
Medicare contractors' operations; better protect program funds from
waste, fraud, and abuse; and accommodate managed care and alternative
payment methodologies. 


--------------------
\1 MTS is used throughout the report to refer to the software
development project as well as other initiatives, such as
interim-phase activities, MTS claims processing sites, and
telecommunications. 


      HCFA REVISED ITS MTS
      TRANSITION AND
      IMPLEMENTATION PLANS
-------------------------------------------------------- Chapter 1:1.1

HCFA initially expected contractors to begin processing claims using
MTS in late 1996 and to implement MTS at all Medicare contractor
locations by December 1998.  However, because of difficulty in
defining system requirements and a revised system development
approach to minimize risk, HCFA now expects to have its first MTS
software module--managed care--implemented by June 1998, and complete
implementing the remaining modules in 2000 and beyond.  It also plans
to award contracts for an MTS data operations and analysis center and
two MTS claims processing sites in late 1997, and move the claims
processing workload to these two processing centers from the existing
claims processing locations as the MTS modules become available. 

The original MTS project schedule was developed on the basis of a
grand design approach, in which the complete system would be
implemented at one time.  HCFA changed its implementation plan to a
phased approach after our January 1994 report, in which we discussed
the reduced financial, schedule, and technical risks associated with
phased implementation strategies.\2 HCFA's new approach includes
deploying MTS in increments and making necessary changes to its
existing systems to allow claims processing past 2000. 

Specifically, HCFA plans to move from its current claims processing
environment to a fully implemented MTS in two phases--first, to an
interim Medicare processing environment and then to a full MTS
environment.  For the interim phase, HCFA (1) will require its
contractors to convert three part A and six part B systems to a
single part A and single part B system,\3 (2) is transferring claims
processing from about 45 contractor sites to about 20 sites
nationwide, and (3) has funded its contractors to modify any existing
software that will still be in use to ensure reliable operations
through the change of century.  (See figure 1.1.) For the final
phase, HCFA will transfer the existing part A and part B systems from
the few remaining processing sites to the two planned MTS processing
centers.  Then, applicable software in these systems will be replaced
by MTS modules as they become available, until these systems are
completely replaced by MTS. 

   Figure 1.1:  Strategy For
   Transition to the Medicare
   Transaction System

   (See figure in printed
   edition.)

HCFA has begun its interim-phase activities.  It selected the Florida
Shared System as the single part A system, converted one of the two
remaining part A systems to the Florida system, and has initiated
action to have the other system's contractors begin converting to the
Florida system.  Second, it has asked system user groups to present
proposals for consolidating the part A processing sites.  HCFA
awarded a contract on April 8, 1997, for the single part B system. 
Also, it has provided limited funding for several system contractors
to begin making millennium-related software changes. 

HCFA's final-phase activities include, in addition to the 1994 MTS
development contract, issuing a request for proposals for two MTS
claims processing sites and one data operations and analysis center. 
The contract for these sites, originally scheduled to be awarded in
March 1997, is now planned for late 1997.  The first MTS software
module is scheduled to be installed at those sites in May 1998. 

On April 4, 1997, HCFA announced that, as a result of a recent
management review, it was redirecting its MTS contractor to focus
solely on the managed care module while it examines alternative ways
to achieve its MTS goals.\4 HCFA concluded that its vision of MTS as
the best information technology to take Medicare into the 21st
century had not changed. 


--------------------
\2 Medicare:  New Claims Processing System Benefits and Acquisition
Risks (GAO/HEHS/AIMD-94-79, Jan.  25, 1994) and Medicare Transaction
System:  Strengthened Management and Sound Development Approach
Critical to Success (GAO/T-AIMD-96-12, Nov.  16, 1995). 

\3 One each of the existing part A and part B systems will become the
single systems used to process all part A and part B claims. 

\4 Managed Care is the first of six planned releases.  The remaining
five releases, in order of planned implementation are the common
working file and the beneficiary insurance file, consolidated
financial server, carrier claims processing, intermediary claims
processing, and encounter data processing. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:2

Our objectives were to determine the extent to which HCFA is (1)
effectively managing its interim Medicare processing environment,
including planning for and correcting year 2000-related computer
problems, (2) using required practices to manage MTS as an
investment, and (3) applying sound systems development processes to
reduce risks. 

To review HCFA's management of its interim environment, we analyzed
documents supporting decisions to select part A and part B systems
and to consolidate existing processing sites.  We also interviewed
HCFA officials responsible for ensuring that claims are properly
processed during this period.  Further, we discussed HCFA's interim
transition decisions with applicable claims system contractors. 

We assessed HCFA's activities to address the millennium change by (1)
assessing millennium-related project plans and directives, (2)
reviewing related budget and funding documents, and (3) discussing
these activities with officials responsible for HCFA's millennium
conversion and with HCFA's system contractors.  In addressing HCFA's
efforts in preparing for the millennium, we drew heavily on
government and private-industry testimony and guidance. 

To assess HCFA's management of MTS as an investment we applied the
following criteria: 

  The Clinger-Cohen Act of 1996 (P.L.  104-106, Division E; Feb.  10,
     1996) (Effective Aug.  8, 1996). 

  The Paperwork Reduction Act of 1995, as amended (P.L.  104-13; May
     22, 1995). 

  The Federal Property and Administrative Services Act of 1949, ï¿½
     313(b), as amended by the Federal Acquisition And Streamlining
     Act of 1994 (FASA), P.L.  103-355; October 13, 1994. 

  The Government Performance and Results Act of 1993 (P.L.  103-62;
     Aug.  3, 1993). 

  Office of Management and Budget Circular A-11, Part 3, "Planning,
     Budgeting, and Acquisition of Fixed Assets," July 1996; Circular
     A-130 Revised, "Management of Federal Information Resources"
     (Feb.  8, 1996); Bulletin 95-03, "Planning and Budgeting for the
     Acquisition of Fixed Assets" (June 27, 1995) (now superseded by
     OMB Circular A-11); Evaluating Information Technology
     Investments, A Practical Guide (Version 1.0, November 1995); and
     executive memorandum M-97-02, Funding Information Systems
     Investments (Oct.  25, 1996). 

(See appendix II for key segments of these investment management
laws, regulations, and guidance.)

In assessing HCFA's management of MTS as an investment, we analyzed
HCFA documentation related to planning and managing information
technology and interviewed members of HCFA's MTS management
committees.  We also obtained and analyzed HCFA's MTS cost models and
discussed them with HCFA officials in assessing how well HCFA had
identified and justified alternatives. 

To assess HCFA's use of effective systems development processes, we
compared them with HCFA's systems development life cycle procedures,
our Systems Assessment Framework (SAF), criteria contained in
Carnegie-Mellon University's Software Engineering Institute's (SEI)
Software Capability Maturity Model and Software Acquisition
Capability Maturity Model, and other generally accepted systems
development practices.\5 We also analyzed HCFA's software and systems
development management documents including HCFA's draft MTS
Requirements Management Plan, risk management reports, systems
development methodology, and MTS Change Management Manual. 

To determine how HCFA is overseeing MTS development, we reviewed
applicable technical plans and MTS software cost and schedule
estimation model results, and compared them to generally accepted
practices.  To assess whether HCFA is managing the MTS schedule, we
obtained and analyzed HCFA's monthly MTS program schedules to
determine whether they were realistic and contained all necessary
schedule elements.  To determine whether HCFA is appropriately
identifying and managing risks associated with the MTS development,
we reviewed HCFA and MTS contractors' risk abatement activities
reported in the risk management reports.  We also obtained copies of
monthly risk reports developed by HCFA and MTS contractors and
assessed how these risks were being mitigated.  Finally, we
interviewed HCFA officials responsible for MTS development, as well
as HCFA's contractors for system development and for independent
verification and validation.  Further, we analyzed a commissioned
study on program controls, interviewed the contractor that performed
the study, and obtained HCFA's response to the recommendations.\6

We performed our work at HCFA headquarters in Baltimore, Maryland,
and its Kansas City, Missouri, regional office; the MTS software
development contractor's offices in Tampa, Florida, and Baltimore,
Maryland; the Independent Verification and Validation (IV&V)
contractor's office in Baltimore Maryland; and several part A and
part B system contractors' offices. 

We performed our work from October 1996 through April 1997, in
accordance with generally accepted government auditing standards. 
HHS and OMB provided written comments on a draft of this report. 
Their comments are presented and evaluated in chapters 2, 3, and 4,
and are included in appendixes I and II. 


--------------------
\5 Systems Assessment Framework:  A Guide for Reviewing Information
Management and Technology Issues in the Federal Government, version
1.0, (GAO, August 1996). 

\6 HCFA MTSI Program Management Final Assessment Report,
(Robbins-Gioia, July 11, 1996). 


INTERIM MEDICARE PROCESSING
ENVIRONMENT NEEDS TO BE MORE
EFFECTIVELY MANAGED
============================================================ Chapter 2

HCFA is not effectively managing its interim Medicare claims
processing environment, increasing the risks of transition delays,
excessive costs, and not achieving the goals of the transition.  The
transition from the current claims processing environment to MTS
represents a major challenge in that it is a larger
systems-conversion effort than any previously undertaken by HCFA, and
is being performed concurrently with HCFA's management of MTS
development.  Although such a major undertaking requires careful
planning and management, HCFA has not followed generally accepted
program management practices, which call for detailed plans for
systems' transitions and modifications.  Also, it has not ensured
that it will avoid the potential systems-related problems that
accompany the year-2000 change. 


   SCHEDULE AND RESOURCE ESTIMATES
   ARE IMPORTANT
---------------------------------------------------------- Chapter 2:1

A schedule and estimate of needed resources for each major stage of
the transition to the MTS processing environment would help HCFA
manage and coordinate its transition activities and ensure that
required resources are available at each stage.  Specifically, this
would include overseeing planning, testing, and implementing the
conversion of the nine part A and part B systems to two, managing the
site preparation and migration from 45 processing centers to about
20, shifting the workloads of local contractors who decide not to
renew their contracts for processing Medicare claims, and converting
systems to address potential year-2000 problems.  The need for such a
transition plan is highlighted in our August 1996 guide for reviewing
information management and technology issues.\1

HCFA agreed that a transition schedule and resource estimates would
be useful.  As part of a consulting contract to help HCFA develop a
complete MTS initiatives program schedule, the consultant is to
assist HCFA in preparing a schedule and resource estimates for HCFA's
transition and then integrating them into the overall MTS program
development schedule work.  These are due to be completed by late
spring 1997. 


--------------------
\1 Systems Assessment Framework (GAO, August 1996). 


   HCFA PLANS TO DEFINE AND
   CONTROL PART A SYSTEM
   RESPONSIBILITIES
---------------------------------------------------------- Chapter 2:2

HCFA selected its single part A system contractor on May 23, 1996,
and as of March 17, 1997, had paid the contractor about $870,000 to
convert one of the two remaining part A systems to the selected
system.  However, HCFA has no legally binding document to define the
responsibilities of the selected part A systems contractor for the
conversion.  Instead, the work is being performed under a 1991
agreement that the contractor will maintain the part A system.  This
agreement does not define the contractor's responsibilities for
consolidating and maintaining a single part A system, supporting the
movement of claims processing from current sites to MTS operating
sites, and developing agreements with users which outlines customer
expectations warranties and guarantees. 

Generally accepted practices for managing and overseeing such work
include (1) preparing a statement to document the requirements of
relevant systems contractors and (2) establishing a change control
mechanism to manage and control changes to these requirements.  HCFA
awarded a contract defining responsibilities for the selected part B
system on April 8, 1997.  According to HCFA officials, they plan to
(1) prepare a statement of responsibilities for the part A system
work by the end of May 1997, and (2) use this statement as a basis
for a legally binding agreement with the part A system contractor. 
According to the MTS project manager, a board to manage changes to
part A began work on April 3, 1997.  HCFA also intends to establish a
similar control board for the part B system. 


   TEST STRATEGY LACKING
---------------------------------------------------------- Chapter 2:3

As addressed in our guide for reviewing information management and
technology issues, a sound test strategy should include testing, such
as the selected part A and part B systems, to ensure that they meet
certain specified requirements.\2 To date, HCFA's involvement in the
ongoing part A conversion has been limited primarily to informal
discussions with contractors and users. 

The testing should measure whether the systems (1) perform required
claims processing and other functions, (2) have the capacity for
processing the total part A and part B workload, (3) can process
claims using data converted from the old systems, and (4) will
operate in the year 2000 and beyond.  A test strategy would ensure
that sufficient testing is conducted and the results evaluated so
that converted systems will be able to reliably process the increased
workload; this is vital to ensuring that Medicare claims will be
processed in an uninterrupted fashion. 

To date HCFA has not taken any of the following steps, each necessary
for a sound testing approach: 

  defining its role in planning or overseeing the testing,

  assigning responsibility for overseeing and approving the part A
     and part B conversion or approving the contractors' acceptance
     testing and results,

  developing criteria for evaluating the contractors' test plans to
     ensure that the systems can adequately handle the combined
     increased workload and that the systems will operate properly in
     the year 2000 and beyond,

  identifying how it will provide resources to manage the testing,

  providing for an independent validation and verification of whether
     test results meet requirements, and

  determining how it will ensure that problems uncovered in testing
     are corrected promptly. 

According to HCFA officials, their contractors routinely test and
implement systems changes in response to legislative mandates,
without a HCFA test plan or strategy.  Consequently, they said, they
plan to rely largely on their contractors to successfully implement
transition-related changes.  They further stated that they expect
system users (local Medicare contractors) to ensure that the testing
is adequate. 

The transition to the MTS system differs in several key ways,
however, from routine changes in response to legislation.  First, the
selected part A and part B contractors may not have a particularly
high incentive to properly make these conversions, since HCFA plans
to eliminate these contractors when MTS is fully implemented. 
Second, converting a system owned by someone else may be more
difficult than making changes to an existing system, and the selected
contractors have no choice over which systems will be converted to
their system--all part A systems will be converted to the selected
part A system, and all part B systems will be converted to the
selected part B system.  Third, these conversions involve
significantly more data and will require more system capacity than
routine modifications.  Finally, as we reported in 1992 and 1994, two
of HCFA's previous system conversions did encounter problems.\3
Specifically, when HCFA shifted an outgoing contractor's claims
processing workload to another contractor's system, serious
disruptions in getting claims processed and payments made to
physicians ensued, as did increases in erroneous payments and
decreases in payment safeguards, possibly resulting in overpayments. 

At the conclusion of our review, the MTS project manager told us that
HCFA is exploring the feasibility of procuring an IV&V contractor for
the transitions to the single part A and part B systems. 


--------------------
\2 Systems Assessment Framework (GAO, August 1996). 

\3 Medicare:  Shared Systems Policy Inadequately Planned and
Implemented (GAO/IMTEC-92- 41, Mar.  18, 1992) and Medicare:  Shared
System Conversion Led to Disruptions in Processing Maryland Claims
(GAO/HEHS-94-66, May 23, 1994). 


   HCFA DOUBTS BENEFITS OF
   PERFORMANCE MEASURES; NONE
   SCHEDULED TO BE USED
---------------------------------------------------------- Chapter 2:4

Measuring the results of the implementation of the interim Medicare
systems is required by legislation and can be useful to HCFA in
understanding and tracking how these systems have altered Medicare
processing.  The Clinger-Cohen Act requires agency heads to implement
a process for managing the risks of information technology
acquisitions, which includes a method of identifying quantifiable
measurements for determining the net benefits and risks of the
investment.  Further the agency head is to ensure that performance
measurements are prescribed for the information technology used by
the agency and that they measure how well the information technology
supports the agency's programs.  The process is to provide the means
for senior management to obtain timely information regarding the
progress of the investment.  The Paperwork Reduction Act also
requires agencies to use effective methods for measuring the progress
of technology in meeting their goals, and OMB guidance emphasizes the
need for such performance measures. 

However, HCFA's plan for evaluating the performance of its transition
systems are inadequate.  Its transition plan does not contain
elements that would allow the agency to determine (1) whether
Medicare systems will continue to provide reliable processing and
adequate service throughout the transition period, (2) whether
expected administrative savings are being achieved, and (3) how MTS
plans might be refined on the basis of results of the transition
systems, such as determining the design and configuration of MTS. 
HCFA officials said they do not believe it would be cost-beneficial
to use the agency's limited resources to develop performance measures
for interim systems that will be replaced by MTS.  We believe
performance measures are needed to ensure that this complex and
important interim phase is properly implemented.  Appendix II cites
legislation that mandates the implementation and use of such
performance measures. 


   MANAGEMENT OVERSIGHT NOT
   ESTABLISHED TO ADDRESS
   POTENTIAL YEAR-2000 PROBLEMS
---------------------------------------------------------- Chapter 2:5

As we approach the year 2000, information systems worldwide could
malfunction or produce incorrect information simply because they have
not been designed to handle dates beyond 1999; Medicare claims
processing systems are no different.  Failure to adjust the systems
for the year 2000 could result in payment delays and in losses due to
bypassed automated controls that flag claims that should be paid by
the beneficiaries' other insurers.  The danger is that, if not
corrected, systems could well read the computer-coded "00" as 1900,
not 2000.  All date-dependent calculations would therefore be
affected, having an obvious impact on age and beneficiary status. 

The timing of HCFA's transition strategy will make addressing the
year-2000 issue even more of a major challenge.  For example, the
single part B system will face converting five other systems to the
selected system, while concurrently modifying the selected system to
make it year-2000 compliant.  Because HCFA now estimates it will not
complete the transition to the single part B system until shortly
before 2000, it has provided initial funding to make four of the six
part B systems year-2000 compliant--the selected single part B system
and three of the remaining five systems.  The Medicare part A systems
contractor has started to modify its software to make it year-2000
compliant, and estimates that it will complete testing this software
and be ready to implement it by July 1997. 

Because HCFA's Medicare contractors routinely make system
modifications in response to legislation, HCFA is relying on them to
make year-2000 changes.  However, the scope of the work required for
contractors to make year-2000 changes is significantly broader than
other systems changes contractors have had to make in the past. 
Specifically, it requires review of all software programs and systems
interfaces, and all systems components that can be affected by date
problems; this includes hardware, operating systems, communications
applications, and databases.  Yet, again, HCFA is not adequately
overseeing this process and further is not requiring contractors to
certify that they will correct the year-2000 problem. 

Adequately addressing the potential year-2000 systems problems for
the Medicare program requires management attention and a wide range
of managerial activities.  As detailed in our February 1997 year-2000
assessment guide,\4 among the most important of these activities are
(1) developing an overall year-2000 plan, (2) identifying
responsibilities for managing and monitoring year-2000 actions, (3)
preparing an assessment of the severity and timing of potential
year-2000 impact, (4) conducting an inventory of the systems on which
Medicare claims processing depends, (5) prioritizing and scheduling
work to convert, replace, or eliminate these systems, (6) developing
validation strategies and test plans for systems, (7) addressing
interface and data exchange issues, and (8) developing contingency
plans for critical systems in the event of failure. 

According to HCFA officials, the agency has prepared an overall
year-2000 plan for its internal systems, intends to include Medicare
claims processing systems in this plan at a future date, and is
collecting information from systems contractors on both their
progress and their planned year-2000 activities.  To date, however,
HCFA has not required systems contractors to submit year-2000 plans
for approval.  Further, it does not have contracts or other specific
legal agreements with any contractors, other than the selected
contractor for the single part B system, which state how or when the
year-2000 problem will be corrected or whether contractors will
certify that they will correct the problem. 

HCFA has also not identified critical areas of responsibility for
year-2000 activities.  Although HCFA's regional offices have a role
in overseeing contractor efforts, their specific year-2000
responsibilities have not been defined, nor has guidance been
prepared on how to monitor or evaluate contractor performance.  While
HCFA has been assessing the impact of the year-2000 on its internal
systems, it has not completed a similar review of Medicare
contractors' claims processing systems.  Further, HCFA has not
required its contractors to prepare an assessment of the severity of
potential year-2000 problems. 

On March 26, 1997, HCFA asked its Medicare contractors to provide an
inventory of the Medicare applications affected by the year-2000
change and their schedules for converting, replacing, or eliminating
these systems.  However, HCFA had no plans to independently validate
the contractors' strategies and test plans.  In addition, while HCFA
has asked the contractors to identify their system interfaces, it had
no plans for approving the contractors' approaches for addressing
interface and data exchange issues. 

HCFA had also not developed contingency plans in the event that
year-2000 systems fail.  HCFA officials are again relying on the
contractors to identify and complete the necessary work in time to
avoid problems.  Yet, the part A and part B contractors not only have
not developed contingency plans, they said they do not intend to do
so because they believe this is HCFA's responsibility. 

On April 22, 1997, at the conclusion of our review, HCFA provided us
with information regarding a technical workgroup, which is to
identify and resolve any year-2000 technical issues.  However, this
workgroup, which was established on January 10, 1997, had not yet
discussed or resolved any technical issues. 


--------------------
\4 Year 2000 Computing Crisis:  An Assessment Guide (GAO, February
1997). 


   CONCLUSIONS
---------------------------------------------------------- Chapter 2:6

HCFA faces a challenging array of tasks as it operates in an interim
Medicare claims processing environment over the next few years.  It
expects this interim phase to better ensure a successful transition
to MTS and ensure reliable claims processing during this period. 
However, HCFA has not prepared the necessary plans to help it manage
this interim period, help ensure that these goals will be met, or
evaluate the performance of its interim claims processing
environment.  Further, unless timely, effective systems changes are
implemented as the year-2000 approaches, HCFA may be unable to
process claims accurately and within required time frames. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 2:7

To better ensure the success of claims processing during the interim
before MTS implementation, we recommend that the Secretary of Health
and Human Services direct that the Administrator, Health Care
Financing Administration, manage and be accountable for the following
actions. 

  Preparing a detailed transition plan, which includes sections that
     (1) provide a schedule and estimate of resources needed for each
     major stage of the transition to the interim processing
     environment (2) define how software changes to the part A and
     part B systems will be controlled and managed, (3) identify how
     HCFA will ensure reliable processing while reducing the number
     of processing centers and shifting the workloads of local
     Medicare contractors who decide not to renew their Medicare
     claims processing contracts, and (4) define how systems will be
     converted to address potential year-2000 problems. 

  Obtaining a legally binding agreement with the part A contractor,
     which identifies all responsibilities for conversion and
     maintenance of the part A system, before providing any
     additional funds for this effort. 

  Preparing plans for conducting thorough testing before converting
     part A and part B systems.  These plans should, at a minimum,
     (1) define HCFA's role in planning or overseeing the testing,
     (2) assign responsibility for overseeing and approving the part
     A and part B conversion or approving the contractors' acceptance
     testing and results, (3) develop criteria for evaluating the
     contractors' test plans to ensure that the systems can
     adequately handle the combined increased workload and that the
     systems will operate properly in the year 2000 and beyond, (4)
     identify how it will provide resources to manage the testing,
     (5) provide for an independent validation and verification of
     whether test results meet requirements, and (6) determine how it
     will ensure that problems uncovered in testing are corrected
     promptly. 

  Establishing a means of assessing performance in the crucial early
     stages of the transition, and applying any lessons learned to
     planning for MTS.  The performance measures should include
     elements that allow HCFA to determine (1) whether Medicare
     systems will continue to provide reliable processing and
     adequate service throughout the transition period, (2) whether
     expected administrative savings are being achieved, and (3) how
     the design and configuration of MTS might be refined on the
     basis of results of the interim systems performance. 

  Helping ensure the reliable operation of its systems through the
     year-2000 by identifying responsibilities for managing and
     monitoring year-2000 actions, preparing an assessment of the
     severity and timing of potential year-2000 impact, and
     developing contingency plans for critical systems in the event
     of failure.  Further, HCFA should require its contractors to
     submit for review and approval (1) plans for identifying and
     correcting potential problems, including a certification that
     their changes will correct the problem, (2) validation
     strategies and test plans for systems, and (3) plans for
     addressing interface and data exchange issues.  Finally, HCFA
     should regularly report to HHS on its progress in addressing the
     year 2000 issue, including the amount of funds spent on this
     effort. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 2:8

HHS agreed with our recommendations to more effectively manage the
interim Medicare processing environment.  HHS stated that it

  has prepared a part A transition plan and plans to complete the
     part B transition plan in early summer 1997;

  agrees with our recommendation to conduct thorough testing before
     converting the part A and part B systems;

  has established performance metrics to monitor the software
     development contract and, as a result, was able to initiate
     corrective action when the work was not progressing
     satisfactorily; and

  has requested implementation plans from its contractors on their
     progress in bringing their systems into millennium compliance,
     and is in the process of analyzing these plans. 

HHS said that both the Department and HCFA are committed to
implementing our recommendations.  It also commented that many of our
recommendations will assist them in better managing the
transformation from antiquated, redundant information and processing
systems to the planned modernized system. 

We believe that the actions HHS has outlined to manage the interim
Medicare processing environment are positive and expect that, if
effectively implemented, they will help the Department and HCFA
achieve a successful transition to MTS.  In addition, just as
software development performance measures are critical to the
software development process, performance measures are also critical
to the success of the transition.  Further, while requesting
year-2000 implementation plans from HCFA contractors can also help by
improving the overall understanding of the status of the year-2000
effort, to ensure success, these contractors must have their
responsibilities specifically defined and their actions reviewed and
approved by HCFA. 

OMB concurred with our recommendations encouraging HCFA to take steps
to more adequately plan for the transition to standardized part A and
part B systems, and said it is continuing to work with HCFA on this
transition.  OMB also said it will look into our recommendations
concerning the year-2000 contractor systems' conversion issue, and
take appropriate action. 


MTS IS NOT BEING MANAGED AS AN
INVESTMENT
============================================================ Chapter 3

HCFA has begun to respond to legislative and OMB requirements for
managing information technology projects as investments; however, it
has not applied effective investment practices in managing MTS.  Such
practices include:  preparing valid cost-benefit analyses,
considering viable alternatives and assessing risks, and having
senior management involved in the critical decision-making process. 
Without these, HCFA has no assurance that its planned system will be
cost-effective, risk-averse, and support the agency's mission and
goals. 


   HCFA HAS NOT PERFORMED AN
   INVESTMENT ANALYSIS FOR CURRENT
   MTS PLAN
---------------------------------------------------------- Chapter 3:1

As early as 1992, when HCFA began planning for MTS, OMB Circular A-11
required that planned information technology acquisitions be based on
a cost-benefit analysis.  Similarly, since 1992, OMB Circular A-94
has required that decisions to initiate government projects be based
on an analysis of expected life-cycle costs and benefits (justified
on economic grounds using net present value calculations), and that
alternative means of achieving program objectives be considered.\1

Over the years, agencies have experienced numerous failures in
acquiring information technology.  Billions of dollars have been
wasted on systems that did not work as planned or cost significantly
more than expected.  Both the Congress and OMB have recognized this
problem and have recently established requirements that more
specifically require agencies to manage their major acquisitions
using sound information technology investment practices.  For
example, the Clinger-Cohen Act requires agencies to focus more on the
results achieved through their investments and to use more rigor and
structure in their processes for selecting and managing their
information technology projects.  The act also provides OMB with
additional authority to ensure that the act's provisions are carried
out.  In an October 1996 policy memorandum, OMB specified that major
investments in information technology should demonstrate a projected
return on the investment that is "clearly equal to or better than
alternative uses of available public resources." (See appendix II for
details.)

Since HCFA began its MTS project in 1992, the estimated cost to
develop MTS software and move to the new system has increased from
about $151 million to about $1 billion.\2 The $1 billion includes
estimated costs to transition to the MTS environment and acquire
operating sites.  In spite of these significant estimated project
cost increases, as discussed below, HCFA's cost analyses have not (1)
identified the specific MTS applications that justify its estimates
of $2.1 billion in reduced program costs over 10 years (stated as a
present value), (2) adequately documented the assumptions used to
estimate total administrative savings of almost $1.5 billion, or (3)
addressed available alternative solutions to MTS, which could provide
much of the estimated program and administrative savings within a
shorter time and at substantially lower costs.  When asked about the
lack of a cost-benefit analysis, HCFA responded that the benefits of
MTS were obvious. 


--------------------
\1 Present value dollars represent the current worth of an amount or
series of amounts payable or receivable (cost-benefits) in the
future, determined by discounting the future amount or amounts at a
predetermined rate of interest. 

\2 All dollar amounts presented in this report are expressed as
undiscounted current dollars unless otherwise noted.  They are not
adjusted for inflation and are not present values. 


      REDUCED PROGRAM COSTS NOT
      LINKED TO SPECIFIC MTS
      PROCESSING IMPROVEMENTS
-------------------------------------------------------- Chapter 3:1.1

HCFA's initial MTS cost-benefit analysis, developed by an outside
contractor in April 1992, and updated in December 1993, compared the
MTS alternative with the status quo claims processing environment at
that time of 80 contractors, 62 processing sites, and 22 claims
processing systems.\3 Since that time, from February 1995 through
November 1996, HCFA has developed, in-house, a series of cost models,
which update the cost and savings estimates of the previous
contractor-developed analyses.  In September 1996, HCFA's Office of
the Actuary provided a report of its estimate of program savings from
MTS, and these estimates were incorporated into HCFA's November 1996
cost model. 

HCFA's Office of the Actuary estimated that MTS would provide annual
programmatic savings of $570 million by fiscal year 2005, resulting
in 10-year life cycle (fiscal years 1997 through 2006) programmatic
savings of about $2.1 billion (stated as a present value).  It
estimated that program cost savings would result from (1) increasing
the use of automated edits to identify abusive billing practices and
deny related claims, (2) improving and standardizing "medical
necessity" review edits, which would result in an increase in the
number of inappropriate claims identified and denied, and (3)
developing a centralized beneficiary insurance file, which would
increase the amount of savings under the Medicare Secondary Payment
program.\4

However, the Office of the Actuary qualified its savings estimate,
stating that too many details of MTS implementation were not
available, especially the exact nature of the edits that will be
built into the software and any requirement for Medicare contractors
to implement those edits.  It concluded that because of this lack of
information about the MTS development, the actual savings associated
with MTS could prove to be significantly different from its estimate. 

As of April 14, 1997, HCFA had not identified the exact nature of the
edits that will be built into MTS.  Further, HCFA's MTS development
strategy is not based on maximizing potential savings early.  The MTS
releases that include new, but as yet undefined edit routines are not
planned for implementation until 1999.  Finally, for several years
both we and the HHS Office of the Inspector General have reported
that HCFA could save hundreds of millions of dollars annually in
program costs by immediately implementing commercially available
automated edits to detect and prevent fraud, waste, and abuse.\5

In May 1995 we reported that HCFA could save billions of dollars by
using commercially available software containing edits to detect and
correct billing abuses.  HCFA is currently evaluating this type of
software. 

We also reported in January 1996 that some contractors were using
several different medical review edits which, if implemented
nationally, could save up to $150 million annually by denying claims
that were not medically necessary.\6 These edits can be implemented
on existing systems without spending hundreds of millions of dollars
to develop MTS and install two MTS claims-processing sites.  HCFA has
not developed a strategy or schedule to implement these edits and is
continuing to allow local contractors to decide whether to use them. 


--------------------
\3 HCFA's 1993 alternative analysis only compares MTS versus
continuing its current operations.  It did not assess viable
alternative solutions to MTS. 

\4 Medical necessity review edits involve identifying claims for
inappropriate or excessive medical services.  Abusive billing
involves such practices as billing for the same procedure twice
although only provided once or billing for an assistant surgeon when
one was not warranted.  Medicare secondary payment savings result
from identifying instances in which beneficiaries have other health
insurance that should be the primary payer, rather than Medicare. 

\5 Commercial Technology Could Save Billions Lost to Billing Abuse
(GAO/AIMD-95-135, May 5, 1995) and Antifraud Technology Offers
Significant Opportunity to Reduce Health Care Fraud (GAO/AIMD-95-77,
Aug.  11, 1995). 

\6 Millions Can Be Saved by Screening Claims for Overused Services
(GAO/HEHS-96-49, Jan.  30, 1996). 


      ESTIMATED ADMINISTRATIVE
      SAVINGS NOT ADEQUATELY
      SUPPORTED
-------------------------------------------------------- Chapter 3:1.2

HCFA's November 1996 MTS cost analysis estimates that the system will
provide net administrative savings of $697 million.  This $697
million estimate is based on two key assumptions.  First, during MTS'
10-year estimated life, expenditures for software improvements will
be substantially less than would have been required for the existing
part A and part B systems.  HCFA projected that $788 million of its
total $1.5 billion administrative savings estimate would result from
these reduced expenditures.  However, according to actual and
estimated system improvements trend data presented in HCFA's budgets
for fiscal years 1995 through 1998, if MTS were not implemented, HCFA
would need only $83.6 million to implement improvements to existing
systems during these 10 years.  In addition, HCFA did not include
in-house costs to develop MTS in its administrative savings analysis,
which it estimates will be about $49 million during fiscal years 1997
through 2000. 

Second, HCFA's current administrative savings estimate assumes that
total costs to process Medicare claims will continue to increase
unless MTS is developed.  Further, HCFA's 1993 analysis, which was
the last analysis to show how much it cost to process individual
claims, assumed that, without MTS, costs per claim for part A and
part B would continually increase from 1993 through 2002.  However,
actual contractor cost reports show that costs per claim for part A
and part B actually decreased for fiscal years 1994 through 1996,
from $1.41 to $1.27 and from $0.89 to $0.88 respectively.  (See
tables 3.1 and 3.2.)



                               Table 3.1
                
                Part A Costs Per Claim for Fiscal Years
                               1994-1996


                                    HCFA         Actual
Fiscal years                  estimate\a        costs\b     Difference
-------------------------  -------------  -------------  -------------
1994                               $1.62          $1.41          $0.21
1995                                1.66           1.33           0.33
1996                                1.70           1.27           0.43
----------------------------------------------------------------------
\a HCFA's 1993 MTS Alternative Analysis. 

\b Medicare contractors' fiscal years 1994-1996 expenditure reports. 



                               Table 3.2
                
                Part B Costs Per Claim for Fiscal Years
                               1994-1996


                                    HCFA         Actual
Fiscal years                  estimate\a        costs\b     Difference
-------------------------  -------------  -------------  -------------
1994                               $1.06          $0.89          $0.17
1995                                1.06           0.94           0.12
1996                                1.07           0.88           0.19
----------------------------------------------------------------------
\a HCFA's 1993 MTS Alternative Analysis. 

\b Medicare contractors' fiscal years 1994-1996 expenditure reports. 

In addition, HCFA's 1993 savings estimates were based on an existing
claims processing environment that included 62 separate contractor
processing sites using 22 different software systems.  HCFA assumed
that it would replace the 22 different systems with MTS and make
substantial reductions in the number of processing sites by
developing MTS-based sites.  However, some of these savings may have
already occurred without MTS as part of HCFA's interim consolidation
effort.  As we mentioned earlier, HCFA has reduced the number of
existing systems from 22 to 9, and is further reducing them to only a
standard part A and part B system prior to implementing MTS.  HCFA
has also consolidated its 62 processing sites to about 45 and plans
further consolidations to about 20 sites. 


      HCFA HAS NOT EVALUATED
      ALTERNATIVE SOLUTIONS
-------------------------------------------------------- Chapter 3:1.3

OMB requires agencies to prepare cost-benefit analyses that consider
alternative means of achieving program objectives.  For example, when
evaluating a decision to acquire a capital asset, OMB requires that
the analysis should consider alternatives such as upgrading,
renovating, sharing or converting existing government property,
leasing, or contracting for services.  Although HCFA has prepared
several estimates of MTS costs and benefits, none has included
assessments of alternative solutions to MTS.  For example, HCFA has
not assessed and compared the costs and benefits of implementing
readily available, commercial medical review edits and abusive
billing software routines to its plans for similar MTS features. 
Likewise, HCFA has not assessed whether the actual and planned
reductions in existing systems and claims processing sites have
achieved or will substantially achieve most of the projected
administrative savings estimated for MTS. 


   OPERATING SITE DECISION MADE
   WITHOUT ANALYZING ALTERNATIVES
   OR RISKS
---------------------------------------------------------- Chapter 3:2

As part of MTS initiatives, HCFA decided to consolidate its daily
claims-processing workload of about 2.6 million claims at two claims
processing sites using MTS software.  It also plans to acquire a data
operations and analysis center.  However, these decisions were made
with inadequate decision criteria or analysis for comparing
alternatives.  Further, technical risk analyses to support the
planned facilities are incomplete. 

In April 1996 HCFA issued a request for proposals for the three MTS
operating sites.  The criteria for determining the number of
processing sites were limited to data storage and disaster recovery
requirement considerations.  More explicit criteria were not used to
evaluate and prioritize alternatives such as HCFA's interim-phase
Medicare processing sites or other processing centers such as the
Department of Veterans' Affairs data center in Austin, Texas. 

Thorough risk assessments have not been conducted to ensure that the
planned MTS processing sites will perform as required and meet system
goals.  Three major steps commonly used in such assessments have not
been completed.  First, a realistic workload analysis, using high
volumes of data as input and output to realistically simulate the
Medicare system, has not been conducted.  Second, a formal capacity
analysis has not been conducted to determine if the commercial
middleware\7 already selected on the basis of a market survey can
handle the high frequency transmission of input and output data
required by MTS.  Finally, a security risk analysis is essential.  In
the MTS case, it is being prepared out of sequence, after the
security engineering analysis, and is not planned for completion
until the summer of 1997.  The risk in this approach is that when the
security risk analysis is complete, any major security-related risks
identified during this analysis will need to be addressed in the
security engineering analysis.  Thus, the security engineering
analysis may have to be modified at added cost to correspond to the
security risk analysis. 


--------------------
\7 The term middleware is used to describe software that resides
between an application program and an operating system and network,
enabling diverse systems to communicate in a common client-server
environment. 


   HCFA RESPONDING TO LEGISLATIVE
   REQUIREMENTS BUT STILL LACKS
   REQUIRED MTS INVESTMENT
   STRATEGY
---------------------------------------------------------- Chapter 3:3

In response to explicit Clinger-Cohen Act requirements and OMB
guidance, HCFA has begun action to follow practices essential for
making informed information technology investment decisions. 
However, it does not yet have consistent senior management
involvement or investment-based decision-making on MTS issues, and
has not explicitly demonstrated how MTS will help the agency meet its
mission, goals, and objectives. 


      HCFA TAKING ACTION BUT MTS
      MANAGEMENT DECISIONS NOT YET
      INVESTMENT-BASED
-------------------------------------------------------- Chapter 3:3.1

The Clinger-Cohen Act requires agencies to designate a chief
information officer (CIO) to develop, maintain, and facilitate the
implementation of a sound and integrated information technology
architecture.\8 Further, this act, and the Federal Acquisition
Streamlining Act, require agencies to provide a means for senior
management to obtain timely information regarding the progress of an
investment in an information system.  In addition, OMB guidance on
investment management practices encourages senior management to (1)
be involved in making decisions in a disciplined and structured
management forum, (2) monitor the progress of ongoing information
technology projects against projected cost, schedule, performance,
and benefits, (3) document all management decisions and supporting
data to avoid replication of effort in analysis, and (4) evaluate how
common problems and their solutions apply to other information
technology projects.\9

In response to the Clinger-Cohen Act, HHS designated a CIO.  The CIO
told us that HHS intends to be more involved in overseeing HCFA's
management of this project than in it has been in prior years. 

Similarly, HCFA has begun to respond by appointing a CIO and
establishing an investment review board.  As envisioned by HCFA, the
board will provide an integrated process for strategic information
technology planning, budget development, and performance-based
management and evaluation of major information technology/system
investments.  HCFA also has recently announced an agencywide
reorganization that recognizes the significant role that the CIO will
have in agency management and planning.  The reorganization is
planned for completion by July 1, 1997. 

Although HCFA is progressing in its overall investment management
approach, it still lacks an MTS investment management strategy.  For
example, HCFA established its most senior MTS decision-making body,
its MTS Initiatives Management Board, on December 14, 1994.  The
Board comprises several senior program and information managers, and
reports directly to the HCFA Administrator.  It is responsible for
reviewing the progress of MTS and keeping decisions on schedule, as
well as reviewing MTS planning to ensure that all future needs are
met and its strategic vision maintained.  Members told us that the
Board (1) acts as the front line MTS decisionmaker, (2) provides
leadership for and facilitates MTS project decisions, and (3)
provides overall strategies and planning for MTS.  The Board meetings
have recently been expanded to include joint sessions with a core
group of technical advisers.  The decision-making and project
management responsibilities that existed with the original Board
remain unchanged with the combined group. 

The Board has not, however, been systematically involved with MTS. 
For example, between 1995 when it was established and January 1997,
the Board was responsible for only one of 34 major MTS project
decisions, that being the decision to locate one print-mail facility
at each MTS claims processing site.  Other critical decisions, such
as the selection of the single part A claims processing system or the
implementation of a new MTS transition strategy, were made by
individual managers or executives, or lower-level MTS groups. 


--------------------
\8 The Paperwork Reduction Act, as amended by the Clinger-Cohen Act,
specifically requires agencies, such as HHS, to designate a CIO. 
Further, the conferees of this legislation also anticipated that
major subcomponents or bureaus, such as HCFA, would also appoint CIOs
responsible for ensuring that the management and acquisition of
information technology is implemented consistent with the law. 

\9 Evaluating Information Technology Investments:  A Practical Guide,
Office of Management and Budget, November 1995. 


      MTS STRATEGIC PLAN
      DEVELOPED, BUT NOT
      ADEQUATELY LINKED TO AGENCY
      MISSION, GOALS
-------------------------------------------------------- Chapter 3:3.2

When organizations manage information systems projects as
investments, they view projects as efforts to improve mission
performance, not simply as actions to implement information
technology.  Since its passage in 1980, the Paperwork Reduction Act
has required agencies to ensure that information technology is
acquired and used in a manner that improves service delivery and
program management, and increases productivity.  In its 1995
revision, the act explicitly required agencies to (1) ensure that
information resources management operations and decisions are
integrated with organizational planning, budget, financial
management, human resources management, and program decisions, and
(2) establish goals for improving information resources management's
contribution to program productivity, efficiency, and effectiveness,
and methods for measuring progress toward achieving those goals. 
More recent legislation supports this requirement and mandates that
performance measures be established to gauge how well information
technology supports agency programs. 

In response to these requirements, in February 1994, HCFA developed
an agency strategic plan, which includes its overall agency mission,
goals to support that mission, and objectives to achieve those goals. 
Annually, the agency also develops 5-year Information Resources
Management (IRM) plans that document its IRM goals, accomplishments,
and major information technology initiatives.  HCFA's September 1996
IRM plan (1) provides a mechanism for incorporating its IRM planning
process with its budget process and its strategic plan, (2)
graphically links the agency's information technology initiatives and
each of the strategic goals, and (3) lists each of its strategic
goals, along with accomplishments that HCFA believes have moved it
along toward achieving each of those goals. 

Although HCFA's IRM plan provides a mechanism for integrating its IRM
planning process with its agencywide strategic planning, as mandated
by the Paperwork Reduction Act of 1995, it does not adequately apply
this integrated approach in its information technology planning.  MTS
and other information technology projects are still ranked on the
basis of budget priorities determined by a HCFA budget review group
rather than on how well the systems will help fulfill HCFA's mission
and meet its program needs.  Using this budget approach, without
cost-benefit analyses and ranked key investment technology projects
as required by legislation and OMB, HCFA has no assurance that it is
funding the most important information technology projects that are
necessary to meet its mission and goals. 

Further, while MTS is HCFA's major information technology project, it
is not highlighted in HCFA's IRM plan.  This plan includes a series
of graphics linking numerous information technology initiatives,
including MTS, with the seven strategic goals that they support.  The
plan also lists each of the agency's strategic goals, along with
accomplishments that HCFA believes has moved it closer to achieving
those goals.  However, MTS has been linked to only one of the seven
strategic goals, which is to "create excellence in the design and
administration of HCFA's programs." According to the plan, other
important goals, such as "be a leader in health care information
resources management" or "provide leadership in the continuing
evolution of the health care system" are not being addressed by MTS. 
Further, the plan shows that MTS supports only 1 of the 30 strategies
that HCFA has developed to achieve its goal of creating excellence in
HCFA's programs.  According to HCFA officials, they are revising
their strategic plan and are arraying their projects in support of
all strategic goals rather than tying them to individual goals. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 3:4

HCFA's MTS initiative has been under development for over 3 years,
however, it still has not been adequately cost-justified, as required
by legislation and OMB directives.  Consequently, by moving forward
with the MTS development contract and planning to award contracts for
MTS operating sites without preparing required analyses, HCFA
continues to put at risk the opportunity for the most cost-effective
and beneficial Medicare claims processing environment possible. 
Further, HCFA is limiting its MTS investment management approach with
its lack of consistent oversight from its senior decision-making
body, as required by statutes and OMB directives.  Until HCFA
officials implement an investment management approach and produce
adequate analyses to justify the cost of MTS and its related
initiatives, HCFA has no assurance that the project will be
cost-effective, delivered within estimated time frames, or result in
a more efficient or effective Medicare claims processing environment. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 3:5

We recommend that the Secretary of Health and Human Services better
ensure the success of MTS by

  withholding funding for the MTS operating site contracts until an
     approach has been selected on the basis of an alternatives
     analysis; alternatives are ranked on the basis of cost, benefit,
     performance, risk, and technical factors, and are justified with
     valid cost-benefit analyses; and supported with a thorough risk
     assessment, which includes a realistic workload analysis, formal
     capacity analysis, and a sound security risk analysis;

  requiring the Administrator of the Health Care Financing
     Administration to justify the continuation of MTS by producing a
     valid cost-benefit and alternatives analysis that includes goals
     for reaching programmatic savings and links estimated savings to
     specific improvements in Medicare claims processing, and take
     appropriate action based on the results of the analysis; and

  requiring the Administrator of the Health Care Financing
     Administration to establish an investment management approach
     for MTS by explicitly linking the roles and responsibilities of
     the CIO and the Investment Review Board to relevant legislative
     mandates and requirements, which include (1) designing and
     implementing a process for maximizing the value and assessing
     and managing the risks of information technology acquisitions,
     and integrating that process with the budget, financial and
     program management decisions of the agency, (2) utilizing
     specific quantitative and qualitative criteria for comparing and
     prioritizing alternative information systems investment
     projects, (3) providing the means for senior management to
     obtain timely information regarding the progress of an
     investment, including a system of milestones for measuring
     progress, in terms of cost, capability of the system to meet
     specified requirements, timeliness, and quality, and (4)
     ensuring that performance measures are applied to measure how
     well the information technology supports the goals and missions
     of the agency. 

Completing these actions by the end of 1997 is essential so that the
Congress, in monitoring HCFA's progress, has assurance that HCFA is
pursuing a course that will efficiently fulfill its goals and
missions. 

We recommend that the Secretary of Health and Human Services assist
HCFA in its modernization effort by providing oversight in accordance
with provisions in the Clinger-Cohen, Paperwork Reduction, and
Federal Acquisition and Streamlining Acts.  This should include
requiring the Department's Chief Information Officer to (1) review
the MTS project at predetermined project milestones to measure its
progress in terms of cost, capability of meeting specified
requirements, timeliness and quality, and (2) identify suitable
actions to be taken, including termination, if the project falls
significantly behind schedule, over budget, or is not in compliance
with performance or capability requirements. 

We also recommend that the Director of the Office of Management and
Budget utilize the enforcement authority provided by section 5113
(b)(5) of the Clinger-Cohen Act to ensure that the Health Care
Finance Administration complies with the act's provisions, including
the requirement to justify major information technology projects such
as MTS with sound cost-benefit and alternatives analyses. 


   AGENCY COMMENTS AND OUR
   EVALUATION
---------------------------------------------------------- Chapter 3:6

HHS stated that improvement is needed in its MTS investment
management activities, and essentially concurred with our
recommendations.  HHS also stated that both the Department and HCFA
agree that no funds will be obligated for the MTS operating site
procurements until a reassessment of the project has been completed
and a revised development strategy is established.  Further, HCFA
stated that (1) it will continue to analyze the cost and savings of
MTS development strategies and evaluate alternatives and (2) it
concurs with our recommendation to link the roles and
responsibilities of the CIO and Investment Review Board.  However,
HCFA commented that the following clarifications of our analyses were
warranted.  First, it said that we failed to recognize the positive
efforts it has made in managing MTS, such as preparing multiple
investment models, hiring consultants, and broadening the agency's
MTS management team.  Second, HCFA said that because the scope of the
costs included in its 1992 estimate differ from those included in the
1996 estimate, it is misleading to state that MTS costs have
increased from $151 million to about $1 billion.  Moreover, HCFA
noted that, while the amount of the estimated MTS investment has
increased, administrative savings have been significant in every
analysis performed.  Third, HCFA stated that while available
commercial software has value, it does not provide the sophistication
necessary to detect and prevent a significant amount of abusive
billing.  HCFA also said that it uses several types of commercial
software to detect inappropriate billing; and is using the Los Alamos
National Laboratory to identify prepayment techniques for detecting
inappropriate billing. 

Throughout the report, we have recognized the positive efforts that
HCFA has made in managing MTS.  For example, we noted that HCFA had
prepared a series of cost models, hired consultants such as its IV&V
contractor, and responded to legislation and OMB regulations.  Yet,
although HCFA has prepared a series of cost models, none of these
models included all costs or evaluations of available alternatives. 
Thus, a complete cost-benefit analysis that includes an assessment of
viable alternatives has not been performed. 

Further, our statement that estimated MTS costs have increased from
$151 million to about $1 billion is accurate.  Whether the scope of
the project has changed is immaterial to the fact this project has
incurred a substantial cost increase, which we described to emphasize
that HHS and HCFA need to manage this project as an investment.  One
concern we have always had is that HCFA has not finalized the
requirements for MTS.  Until these requirements are finalized, no
reliable cost estimate can be prepared.  Further, although all of
HCFA's MTS cost analyses included estimates of substantial
administrative savings, these estimates were based on an invalid
assumption--that Medicare claims processing costs would continue to
increase anyway.  Actual cost data have shown this to be incorrect. 
In addition, the cost-benefit analyses do not consider other
alternatives for administrative savings, such as those offered by
consolidating existing processing sites.  Accordingly, we believe
that HCFA needs to reassess the estimated administrative savings MTS
will provide. 

Finally, although commercially available software may not provide the
sophistication HCFA desires for MTS, we believe it has potential for
HCFA's claims-processing systems.  HCFA does not expect MTS to be
fully implemented for years.  This commercial software for detecting
fraud and abusive billing, along with the consolidation of the
existing claims processing software and claims processing sites,
offers opportunities for substantial program and administrative cost
savings in the interim. 

OMB agreed with our recommendations for HCFA to manage MTS as an
investment, and for OMB to ensure that HCFA justifies its major
information technology projects such as MTS with sound cost-benefit
and alternatives analyses.  OMB said it will request that HCFA
develop benefit, risk, return on investment, and alternatives design
analyses that correspond to the changes in the MTS software design
and that will address methodological concerns raised in our report. 
Such analyses should greatly assist management in making sound
investment management decisions. 


SIGNIFICANT SYSTEMS-DEVELOPMENT
PRACTICES NOT BEING FOLLOWED,
INCREASING MTS RISK
============================================================ Chapter 4

HCFA has never before managed a systems development project the size
and complexity of MTS.  Recognizing its lack of systems development
and risk management experience, HCFA has relied on contractors to (1)
provide the hardware, telecommunications, and software required to
process Medicare claims, (2) review and make recommendations on the
efficiency and effectiveness of the MTS program, including risk
management, (3) assist in developing an integrated program schedule
and identifying the related critical path and risks associated with
it, and (4) assist in identifying the scope and components of MTS
integration, and the risks associated with this integration. 

HCFA's ability to adequately monitor and oversee the work of its
contractors, however, remains a question, given its own inexperience. 
Deficiencies in several critical systems-development practices
provide early signs of weakness in HCFA's system acquisition
management capability and its contractors' software development
practices.  Plans essential to MTS' success are either inadequate,
incomplete, or are being completed too late in the development cycle;
the project schedule is incomplete and contains risky overlap in
development phases; HCFA's risk-management process is inadequate; and
its oversight has not prevented a risky software-development
strategy. 


   REQUIREMENTS MANAGEMENT PLAN
   COMPLETED TOO LATE IN SYSTEMS
   DEVELOPMENT PROCESS
---------------------------------------------------------- Chapter 4:1

A requirements management plan describes the process that will be
used to define, validate, rank, and control systems requirements.\1
This plan is critical because requirements are the key element of any
system design.  Because requirements provide the foundation for
designing, developing, testing, and implementing the system software,
it is essential that they be defined and implemented early in the
systems development life cycle.  In addition, requirements must be
clearly defined to avoid ambiguity, overlap, and duplication, and
should completely and logically describe all features and functions
needed in the planned system.  Using an appropriate requirements
management plan to define and manage requirements significantly
reduces the risk that requirements defects will cause technical
problems such as unacceptable system response times and inadequate
software interfaces.  It also reduces the likelihood of needing to
make time-consuming requirements changes later in the development
when they are more costly and risky to implement. 

Although HCFA officials stated that they are following an interim
requirements management process, they have not yet made final or
implemented an official requirements management plan.  Instead, they
have been relying on a draft MTS Business Requirements Writer's Guide
for developing and managing MTS business requirements.  The
requirements management process discussed in the guide does not
describe how requirements are to be assessed to determine their
impact on the overall project. 

HCFA's lack of a requirements management plan contributed to several
redirections that caused schedule delays.  First, HCFA has twice
redirected the requirements definition approach and, 3 years into the
MTS contract, requirements have yet to be completely defined or
approved.  Second, although requirements for the managed care
module--the first MTS release--were expected to be completed and
documented in a systems requirements document by November 1996, they
have not yet been officially approved by HCFA, nor have they been
baselined.\2 Despite the lack of approved requirements, the MTS
contractor has progressed into the development phase for the managed
care module.  Further, as shown in figure 4.1, the system
requirements have been volatile.  During a recent 5-month period, the
requirements for one software release dropped from 1,639 to 1,499,
while the requirements for another release increased from 631 to 868. 
Such volatility will affect cost, schedule, defects, and overall
quality of the software. 

   Figure 4.1:  Volatility of MTS
   Requirements

   (See figure in printed
   edition.)

HCFA's IV&V contractor has recommended that HCFA develop and
implement a requirements management plan, and has warned that without
such a plan, a high probability exists that a system will be
delivered that does not meet customer needs.  In early January 1997,
HCFA officials said that they expect to complete a requirements
management plan by January 31, 1997.  As of April 1997, no plan had
yet been completed. 


--------------------
\1 Requirements management plans are addressed in the Capability
Maturity Model (CMM) Software Version 1.1 Software Engineering
Institute (Pittsburgh, Pennsylvania:  February 1993); Software
Acquisition Capability Maturity Model (SA-CMM) Version 1.01, Software
Engineering Institute (Pittsburgh, Pennsylvania:  December 1996); and
Guidelines for Successful Acquisition and Management of
Software-Intensive Systems (Air Force Guidelines), Department of the
Air Force, Software Technology Support Center, Volume 1, (February
1995). 

\2 A baseline is a specification or product that has been formally
reviewed and agreed upon, and thereafter serves as the basis for
further development.  It should then only be changed through formal
change-control procedures. 


   SOFTWARE DEVELOPMENT PLAN IS
   INADEQUATE
---------------------------------------------------------- Chapter 4:2

The software development plan is a key document that reflects the
contractor's overall approach to developing software and serves as a
benchmark for monitoring how well the contractor adheres to approved
procedures and activities.\3

HCFA does not have an adequate integrated software development plan. 
It did not require such a software development plan in its January
1994 MTS contract, and did not specifically request one during its
contract renegotiation in May 1996.  HCFA officials explained that as
part of the negotiation process, they provided the MTS contractor
with a template containing the essential requirements of a software
development plan.  They also stated that while the negotiations
document met HCFA's overall requirements, several elements of a
software development plan are contained in various contract
deliverables.  According to HCFA officials, the need for a software
development plan for MTS has been fulfilled.  The IV&V contractor,
who, in May 1995, indicated that the lack of a software development
plan was an area of significant risk, stated that the negotiation
document received from the MTS development contractor incorporates
most of the requirements for such a plan.  The IV&V contractor added
that the negotiation document, along with other internal MTS
contractor practices, are sufficient to satisfy the software
development plan requirement. 

While the MTS negotiation document contains the technical approach
for developing MTS, it lacks critical components of a software
development plan, such as a description of the software development
library standards and metrics.  A software development library is
critical to the software development effort because it provides two
fundamental capabilities:  (1) storage of computer software in
machine readable form for computer operation and (2) storage of
computer software documentation in human-readable form.  Software
development metrics are measures used to indicate progress or
achievement.  Without these elements, the orderly development and
subsequent support of software cannot be supported, and the quality
of software cannot be measured respectively. 

According to HCFA officials, other parts of the MTS software
development plan are contained in numerous documents.  However, this
makes it difficult for effective use by the MTS contractor and for
HCFA to oversee and manage the software development process.  For
example, HCFA provided nine pages of references to various documents
that it considered contained components of a software development
plan.  Such document dispersal precludes effectively managing
software development.  Sound software development practices require
the entire plan to be reviewed and approved by all user groups,
including those responsible for documentation, testing, training,
installation, and quality assurance.  Further, at each phase of
review the plan is to be updated.  These practices are important to
obtaining user agreement on procedures and to holding individuals
accountable for the delivered software products.  Not having these
documents integrated and formally agreed to by all parties
responsible for software development increases the risk that they
will not be adequately reviewed. 


--------------------
\3 Software development plans are addressed in the (1) Software
Engineering Institute's Capability Maturity Model, Software and its
Software Acquisition Capability Maturity Model, and (2) Air Force's
Guidelines for Successful Acquisition and Management of
Software-Intensive Systems. 


   CONFIGURATION MANAGEMENT PLAN
   NOT YET IMPLEMENTED
---------------------------------------------------------- Chapter 4:3

A configuration management plan describes how changes to software and
the total system including key documents will be managed and
controlled throughout the software development process.\4 This
process facilitates communication among development team members
regarding the status of software engineering efforts, and ensures
that proposed changes to software or other system components, and
related documents, are reviewed by a configuration control board to
determine whether the changes should be approved.  While system
development contractors are responsible for developing configuration
management plans, agencies acquiring information systems--such as
HCFA--also need such a plan to control all activities associated with
the software development initiative.  The configuration management
plan should be completed and used for system development. 

HCFA has not yet implemented a configuration management plan for MTS. 
It relied on the MTS development contractor's configuration
management plan to manage changes to MTS development, but this plan
is limited to software related products.  Configuration management
for other MTS components, including those that will be part of the
planned data operations and analysis center and processing sites, has
not yet been addressed.  In response to this weakness, HCFA' s change
management development team developed a plan in February 1997 which
documented the configuration management process for the entire MTS
initiative.  According to HCFA, it will use this process to manage
and control changes across all MTS products.  However, the plan has
not yet been implemented. 

Without such a process, changes to items such as software
requirements, and key documents can not be effectively managed or
controlled.  For example, MTS' managed care requirements are being
defined without a change management process.  As a result, HCFA may
not know whether all managed care requirements that the contractor is
using to develop the first module adequately represent and fulfill
Medicare's functions and MTS' goals.  Also, without a configuration
management process, HCFA will be unable to effectively communicate
hardware changes to the planned operating sites that affect other
members of the MTS development community. 

HCFA's configuration management plan outlines a process for
documenting and reporting all requests for changes to MTS
configuration items and provides change management support for
related MTS documentation.  HCFA intends to integrate the plan with
the MTS design contractor's configuration management plan. 


--------------------
\4 Configuration management plans are addressed in the (1) Software
Engineering Institute's Capability Maturity Model, Software and its
Software Acquisition Capability Maturity Model and (2) Air Force's
Guidelines for Successful Acquisition and Management of
Software-Intensive Systems. 


   SYSTEMS INTEGRATION PLAN NOT
   YET DEVELOPED
---------------------------------------------------------- Chapter 4:4

A systems integration plan is developed and used to ensure that the
hardware, software, and telecommunications standards are adhered to
so that components of the system will interface seamlessly with each
other and with users.\5 A well-defined systems integration plan
identifies related interfaces between hardware and software, and
includes procedures for managing and controlling these interfaces. 
These procedures should include provisions for identifying the
functional and physical characteristics between the applicable
software or hardware units and ensuring that they are compatible. 
Control over the interface structure helps management make
cost-effective, functional allocations among systems and can provide
needed safeguards during systems integration testing. 

In 1995, the IV&V contractor cited that HCFA lacked a systems
integration plan.  However, HCFA has not yet developed a systems
integration plan to help ensure that all required interfaces will be
developed and implemented.  Without a systems integration plan, HCFA
cannot ensure that all of the legacy systems and MTS software and
hardware components will interface with each other as they should. 
In September 1996, HCFA tasked a consultant with (1) defining the
scope of the MTS systems integration effort and identifying its key
systems integration activities and tasks, (2) identifying who will
perform key systems integration activities, and (3) developing an
overall plan and schedule for systems integration activities.  On
April 1, 1997, HCFA received a copy of the consultant's report.  HCFA
plans to use this report as a basis for developing a detailed system
integration plan for MTS.  On April 22, 1997, at the conclusion of
our review, HCFA had not established a date for the final plan. 


--------------------
\5 Systems integration plans are addressed in the Systems Engineering
Management Guide, Defense Systems Management College, January 1990. 


   HCFA OVERSIGHT OF MTS
   CONTRACTOR'S SYSTEM DEVELOPMENT
   IS RISKY
---------------------------------------------------------- Chapter 4:5

Leading software-related organizations enhance their software
development capabilities by applying modern software development
standards and assessing their software engineering processes.  Many
such organizations improve their software development programs by
using a capability maturity model to assess their capability to
produce high-quality software.\6 These organizations also define and
use software measures, or metrics, to assess the quality of software
development, and prepare software development cost and schedule
estimates, as part of the initial planning process. 


--------------------
\6 The Software Engineering Institute has developed two models to
assist organizations in assessing the maturity of their software
development processes.  These models are the Capability Maturity
Model for Software, and the Software Acquisition Capability Maturity
Model.  The Institute provides leadership in advancing the state of
the practice of software engineering to improve the quality of
systems that depend on software. 


      SOFTWARE CAPABILITY MATURITY
      IS KEY TO SUCCESS
-------------------------------------------------------- Chapter 4:5.1

The MTS development contractor estimates that about 2 million lines
of software code will need to be designed and developed for MTS. 
Even though the software was the crucial component of MTS, HCFA did
not require prospective contractors to provide the results of
independent assessments of their software capability maturity, as
recommended by the Software Engineering Institute and other
organizations, including the Department of Defense.\7 Such
assessments help agencies ensure that the selected contractor has the
key software development processes in place to reduce risks. 

According to the MTS development contractor, although its MTS
software development team had not yet been evaluated using the
software capability maturity model, its corporate goal is to have the
entire corporation attain a maturity level three in 3 years.\8

This is but a goal; as yet a plan to achieve it has not been
developed.  Further, the contractor had not established a software
improvement process. 

In addition, the systems development methodology being used was
developed specifically for MTS and, as a result, the contractor's MTS
team has no experience with this methodology.  Further, it is still
incomplete.  For example, a complete systems development methodology
consists of a series of steps and tasks that are used by developers
to provide a structured approach for systems development from systems
planning and design through implementation and support.  The MTS
contractor's methodology only addressed the design, development, and
testing and validation phases.  It does not address other key phases,
which consist of the analysis and implementation phases. 

An inadequate software development methodology greatly increases
development risk because this methodology is used to control the key
software development phases, such as planning, design, development,
testing, and implementation.  In addition, the contractor is not
addressing all phases in the proper sequence.  For example, it has
already moved into the development phase for the Managed Care module
before HCFA has approved these requirements. 


--------------------
\7 Guidelines for Successful Acquisition and Management of Software-
Intensive Systems:  Weapon Systems Command and Control Systems
Management Information Systems, Department of the Air Force, February
1995. 

\8 Level 3 is the "defined" level on a scale ranging from one to
five, with five being the highest rating.  Level 3 means that the
software process for both management and engineering activities is
documented, standardized, and integrated into an organization-wide
software process. 


      MTS CONTRACTOR NOT USING
      COMPLETE SOFTWARE
      DEVELOPMENT METRICS
-------------------------------------------------------- Chapter 4:5.2

Software development metrics are numerical measures used to predict a
dimension of software quality throughout the project.  Early
detection and avoidance of problems and control of software
development projects are possible through the collection, validation,
and analysis of metrics.  Useful metrics include numbers of defects
found at various stages of development, costs to repair defects, and
the extent of test coverage.  Metrics such as the number and
frequency of errors associated with a specific software module are
used to analyze software quality.  Such analysis can identify
questionable or unacceptable situations. 

Despite the importance of software metrics, the MTS contractor has
only included metrics that measure how much of the planned activity
has been completed at each measurement point and metrics that help
refine future cost and schedule estimates.  While these are essential
for assessing the contractor's overall performance, they are
incomplete because they do not contain critical measures to determine
the quality of the software being developed, such as the number of
defects identified and corrected per software module. 


      QUESTIONABLE ASSUMPTIONS
      USED TO DEVELOP SOFTWARE
      COST AND SCHEDULE ESTIMATES
-------------------------------------------------------- Chapter 4:5.3

Software estimating tools, used along with sound assumptions about a
planned project, help developers make reasonable projections about
how much a planned project will cost and how long it will take to
develop.  The MTS software developer applied the widely used software
life-cycle management (SLIM) model for the MTS estimate. 

Since HCFA began MTS in 1994, it has worked with the software
developer in making many software development cost and schedule
projections.  HCFA's original MTS contract called for the software to
be developed by late 1996 for about $18 million and was supposed to
have been completed by October 1996.  Now, following contract
renegotiations, the software development will cost over $90 million
including award fees, and will not be completely implemented until
after 2000.  During the renegotiations, HCFA directed the MTS
contractor to estimate the cost and schedule for developing the
software. 

The processes that should be used for estimating software cost are
illustrated in figure 2.  Planning and cost estimating should be one
of the first set of activities the project team responsible for
producing software performs.  Sound planning helps ensure that
resources will be used effectively to support corporate strategic
objectives.  Good software estimating provides the basis for cost
trade-offs and resource allocation decisions.  Together, planning and
software estimates set the stage for project controls and for
managing progress. 

   Figure 4.2:  Software Cost
   Estimating Process

   (See figure in printed
   edition.)

The MTS software developer, with direction from HCFA, applied to the
SLIM model a series of assumptions and constraints on such factors as
the desired completion date and staff resources and skills to
calculate outputs, including cost, development schedule, and degree
of risk.  Based on this model, the MTS project will require over 1.7
million lines of code. 

However, several assumptions used as input for the SLIM model do not
realistically portray the MTS software development environment.  For
instance, one assumption used in the model rated the team as "very
experienced" with the software infrastructure.  This was based on
plans to use specific, contractor-owned, proprietary software.  Since
these estimates were made, however, the software developer has
decided to use a commercial off-the-shelf product.  The "very
experienced" indicator has not been adjusted to reflect that the
software development team had never before used this commercial
product.  In another assumption, the software developer characterized
the volume of data to be processed as "average" when, in fact, MTS
will be required to process enormous amounts of data.  Furthermore,
requirements have been described as being "defined." Yet, HCFA has
not approved the software requirements specifications or
documentation for any of the software releases.  In addition, the
specifications and requirements for the infrastructure needed to
support the six releases have likewise not yet been approved. 

These types of assumptions cause the SLIM model to inflate a key
estimation variable--the productivity index--above the rating it
would have received if more accurate assumptions had been used.  This
results in unrealistic software cost and schedule estimates. 
According to the SLIM vendor, for every point the productivity index
drops or rises, a project will take 10 percent more or less time, and
cost 30 percent more or less to develop than other typical,
large-scale development projects. 

According to the software developer, the estimated number of lines of
code will be reviewed during the design phase on the basis of further
knowledge about software component size and complexity, better
estimates of anticipated code reuse, and commercial off-the-shelf
software use; this in turn will result in cost and schedule
revisions. 


   MTS SCHEDULE INCOMPLETE,
   DANGEROUSLY COMPRESSED
---------------------------------------------------------- Chapter 4:6

Generally accepted systems development practices require project
managers to continually monitor a project's schedule to ensure that
its activities are completed as planned.  In addition, OMB requires
that agencies establish appropriate controls to help ensure that
information technology acquisitions remain on schedule.  Further, to
avoid technical problems, federal systems acquisition guidance calls
for agencies to minimize overlap of systems development
phases--analysis, design, software development, testing, conversions,
and deployment.\9 In other words, to minimize the risk of major
system development rework, all activities that need to be completed
in one phase should be completed before the next system development
phase begins. 

HCFA has integrated its program schedule with that of its development
contractor, and has added broadly defined transition tasks.  However,
it has not yet identified (1) how the schedule's tasks interrelate or
(2) a critical path showing the sequence of tasks that is longer than
any other.  Such critical paths are used to determine the overall
project completion date.  Also, HCFA has not developed resource
estimates for each task, without which realistic projections about
the time to complete each task or the entire project cannot be made. 
Further, detailed plans for several major MTS initiatives are still
not included in the current program schedule, such as the transition,
year-2000 effort, and the release of the final software module. 
Finally, key critical tasks are inappropriately scheduled.  For
example, according to the program schedule, the first MTS software
release will be installed at the planned MTS processing centers
before the new processing centers are scheduled for implementation. 

HCFA has hired a contractor to assist in developing these missing
program schedule elements.  By mid-summer 1997, HCFA plans to have
(1) project-level resource requirements for each of the approximately
30 projects that constitute the MTS initiative and (2) a
program-level critical path. 

Finally, MTS project phases still overlap considerably for both the
overall MTS project as well as the development phases within each MTS
module release.  As we reported in 1994, if a contractor advances too
far into a succeeding systems development phase before sufficient
progress has been made in previous phases, the risk of technical
problems is significantly increased.\10 HCFA's current program
schedule shows concurrency in all overall project phases and, between
September 1997 and September 1998, HCFA plans to perform all five
systems development phases at one time.  (See figure 4.3.) In
addition, each one of the five scheduled MTS software releases
contains project phases that overlap.  Figure 4.4 shows the
overlapping phases scheduled for the first release--Managed Care. 

   Figure 4.3:  Medicare
   Transaction System Program
   Schedule Revisions

   (See figure in printed
   edition.)

   Figure 4.4:  Medicare
   Transaction System Program
   Schedule for Managed
   Care--Release 1

   (See figure in printed
   edition.)

HCFA's IV&V contractor has also expressed concern about the MTS
schedule.  In HCFA's February 1997 risk tracking report, the
contractor stated that the software development schedule was risky
because it "did not allow any slack time to accommodate slippage or
partial performance." The contractor explained that the schedule has
never contained sufficient time to completely perform tasks leading
to key deliverables such as the systems requirements document and
systems external specifications.\11 The IV&V contractor said that
because the MTS development contractor was 1 month late in delivering
these two products, additional resources would have to be taken from
other tasks to complete this work.  The IV&V contractor concluded
that "This, in turn, will have a direct negative impact on the tasks
being performed with reduced resources."


--------------------
\9 Guidelines for Successful Acquisition and Management of Software
Intensive Systems, Department of the Air Force, Software Technology
Support Center, February 1995. 

\10 GAO/HEHS/AIMD-94-79, Jan.  25, 1994. 

\11 A systems requirements document specifies the requirements that
are to be included in a planned software system.  System external
specifications describe interfaces to the planned system, and include
descriptions of subsystem and related software programs. 


   WEAKNESSES IN MTS RISK
   MANAGEMENT REQUIRE ATTENTION
   BEFORE PROCEEDING WITH
   DEVELOPMENT
---------------------------------------------------------- Chapter 4:7

Federal statutes and OMB directives require effective risk management
as an essential part of successful information technology system
development.  OMB Memorandum 97-02 directed agencies to reduce risks
in major information systems investments by establishing clear
measures and accountability for project progress.  The Paperwork
Reduction Act of 1995 and the Clinger-Cohen Act of 1996 require
agencies to manage risks associated with information technology
investments.  Further, an investment guide signed jointly by us and
OMB recommends the use of key investment control techniques,
including risk assessments, to expose potential technical and
managerial weaknesses that could impair project success.\12

In addition to these criteria, guidance developed by federal
agencies, including the Department of Defense, suggests other key
elements that should be part of an effective risk management program. 
These include quantitatively estimating risk impact, developing
success criteria and measures when the risk can be considered
mitigated, assigning a full-time risk management officer, and
contracting for IV&V services. 

Measuring HCFA and MTS against these criteria reveals a disappointing
picture.  First, HCFA's risk mitigation plans contain no established
time frame for assessing risk status and do not specify target dates
for risk mitigation.  Second, metrics have not been developed to
provide HCFA with the means for comparing progress in assessing the
effectiveness of risk mitigation efforts.  Third, the risk management
process has no mechanism for providing management with early warnings
of risks becoming imminent.  Fourth, resource estimates of staff,
schedule needs, and funding to address risks have not been
identified.  Fifth, the MTS risk management database does not
incorporate all identified risks.  Finally, documents do not identify
interdependencies among risks. 

Overall responsibility for risk management has not been formally
assigned.  The chapter on risk management in HCFA's program
management plan assigns broad responsibility for risk management to
the MTS Initiatives Management Group, Office Leads, and Project
Owners.\13 The MTS project manager serves as the de facto risk
management official.  The recent assignment of a team responsible for
risk management oversight under this official provides support for
risk management activities; however, current risk management policies
and procedures remain unchanged. 

In February 1997, HCFA formed a team responsible for risk oversight
and management; this team reports to the MTS project manager.  The
team's responsibilities include incorporating all identified risks
into the MTS risk database; helping to identify additional risks;
updating the risk management section of the program management plan;
and helping to monitor, and mitigate reported risks to the point
where they are removed from the risk management report. 

Long-standing unmitigated risks indicate that effective risk
management practices have not been institutionalized and uniformly
applied.  Table 4.1 describes critical risks for which the cost,
schedule, and performance impact has not yet been adequately
quantified. 



                               Table 4.1
                
                     Critical Unmitigated MTS Risks

               Date/by whom
Risk           identified     Description          Impact
-------------  -------------  -------------------  -------------------
Lack of a      May 1995 by    A software           HCFA will be unable
software       IV&V           development plan     to assess MTS
development    contractor     describing the       software
plan\a                        methodology and      development. It
                              approach to          will be difficult
                              developing and       for HCFA to move to
                              testing MTS          a new MTS software
                              software has not     development
                              been created.        contractor
                                                   effectively and
                                                   cost-efficiently.

Lack of a      November 1994  A requirements       Difficulty in
requirements   by IV&V        management plan is   tracing
management     contractor     needed to control    requirements to MTS
plan                          new and changing     or Medicare
                              requirements.        functions. Managing
                                                   changing
                                                   requirements may
                                                   result in a system
                                                   that does not meet
                                                   HCFA's needs.

Lack of a      June 1995 by   A well-defined       HCFA cannot ensure
system         IV&V           process is needed    that systems are
integration    contractor     to describe how the  interfacing
plan                          various systems      appropriately and
                              supporting MTS will  producing the
                              be integrated.       correct results.

Lack of a      May 1995 by    HCFA has not         HCFA cannot ensure
configuration  IV&V           developed a          that the integrity
management     contractor     configuration        of MTS products is
plan                          management plan to   maintained and that
                              manage and control   only approved
                              changes to MTS       changes are being
                              products such as     made to MTS
                              requirements,        throughout the
                              software, and other  system development
                              contract             life cycle.
                              deliverables.

Compressed     June -         The MTS software     Any delay in a task
MTS            December 1996  development          on the critical
development    by IV&V and    schedule does not    path will result in
schedule       MTS software   provide time for     a delay in the
               development    any delays.          overall schedule.
               contractors                         Therefore, the
                                                   expected MTS
                                                   completion date
                                                   will not be met,
                                                   and additional
                                                   funds to complete
                                                   the project may be
                                                   required.

Lack of        October 1996   The MTS software     Insufficient
Medicare       by MTS         development          Medicare expertise
subject-       software       contractor lacks     can delay defining
matter         development    sufficient Medicare  requirements, and
experts        contractor     business analysts,   result in needing
                              necessary to         to rework
                              analyze and define   requirements.
                              MTS requirements
                              for releases 2
                              through 5.

Incorrect      October 1996   Software and         Incorrect metrics
metrics        by MTS         financial metrics    may lead to
               software       either understate    negative MTS cost,
               development    or overstate MTS     schedule, and
               contractor     software quality     performance trends
                              and performance.     and unacceptable
                                                   software.
----------------------------------------------------------------------
\a Although this risk item was taken off the risk report by the IV&V
contractor in November 1996, we believe that it should still be
classified as a risk because it has not been mitigated. 


--------------------
\12 Evaluating Information Technology Investments, OMB Office of
Information and Regulatory Affairs, Information Policy and Technology
Branch, November 1995. 

\13 The MTS Management Group is a core advisory group that shares
definition, development and implementation responsibilities for MTS. 
The MTS Office Leads is composed of representatives of HCFA bureaus,
who meet regularly to discuss MTS integration, coordination, program
monitoring, and communications issues.  The Project Owners are
responsible for discrete MTS project segments. 


   CONCLUSIONS
---------------------------------------------------------- Chapter 4:8

HCFA has attempted to address systems requirements, schedule, and
risk management issues that we have been reporting since 1994, yet
many critical problems remain inadequately addressed.  Critical
management plans have not been adequately developed, the contractor
is using a risky software development strategy, the MTS schedule is
incomplete and dangerously compressed, HCFA is not using sound risk
management procedures, and its IV&V contractor is not ensuring that
potential critical risks are routinely assessed.  Unless HCFA solves
these problems before proceeding further with MTS development or
implementation, it risks extensive development rework, substantial
cost increases, and a system that will not fully meet HCFA's needs. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 4:9

To better ensure the success of MTS, we recommend that the Secretary
of Health and Human Services require the Administrator, Health Care
Financing Administration, to direct and remain accountable for the
following actions before proceeding further with MTS development. 

  Complete a requirements management plan to assist in identifying,
     approving, managing, and controlling the requirements
     development process. 

  Support the development of MTS software with an integrated software
     development plan, which includes a description of such critical
     elements as software development library standards and metrics. 
     All critical elements should be in a single document to
     facilitate the review, approval, and use by involved
     individuals. 

  Implement a configuration management process that includes change
     controls for requirements as well as all other related MTS
     issues such as hardware changes to the planned operation sites. 

  Complete a comprehensive systems integration plan to ensure that
     all MTS-related interfaces are identified, developed, managed,
     and controlled. 

  Require an independent evaluation of the MTS contractor's software
     development capability prior to beginning the software
     development phase.  To ensure that the contractor's MTS
     development team has the capability required for reasonable
     assurance of success, it should achieve a rating of at least
     level 2. 

  Improve software development oversight by requiring the MTS
     developer to include measures of the quality of software in the
     software development metrics. 

  Direct the MTS developer to rerun the SLIM model using appropriate
     assumptions and constraints, and use the results in reassessing
     the cost and time required to develop MTS. 

  Complete a new, integrated MTS program schedule that includes a
     critical path for the entire initiative, including the interim
     Medicare processing environment, and resources and costs for
     each task.  The schedule should also minimize overlap in the
     phases of the system development process. 

  Mitigate critical risks by designating an accountable official for
     risk management and ensuring that this individual implements a
     process, which will (1) identify all significant risks, (2)
     quantify the impact of identified risks, (3) establish time
     frames for assessing risk status and specifying target dates for
     risk mitigation, (4) develop metrics that will compare progress
     in assessing the effectiveness of risk mitigation efforts, (5)
     provide a mechanism for alerting management early of risks that
     are becoming imminent, (6) provide resource estimates of staff,
     schedule needs, and funding to address identified risks, (7)
     ensure that the MTS risk management database incorporates all
     identified risks, and (8) document interdependencies among
     risks.  Further, this accountable official should (1) ensure
     that mitigation plans are developed to address identified risks,
     (2) hold individuals in authority accountable for prompt
     completion and implementation of risk mitigation plans, and (3)
     periodically evaluate the adequacy of HCFA's progress in
     mitigating risks and identify new risks. 

  Require the IV&V contractor to assist HCFA in mitigating risks by
     quantifying the impacts of identified risks on program cost and
     schedule.  HCFA should also reflect these in its program status
     reports. 

To help HCFA improve its ability to use effective systems development
practices and improve its software acquisition capability, we
recommend that the Secretary of Health and Human Services direct the
Administrator, Health Care Financing Administration, to (1) obtain an
independent assessment of its software acquisition capabilities using
the Software Engineering Institute's software acquisition capability
maturity model, and implement improvements to correct any identified
weaknesses, and (2) report its findings to both HHS and OMB. 


   AGENCY COMMENTS AND OUR
   EVALUATION
--------------------------------------------------------- Chapter 4:10

HHS agreed with our recommendations in this chapter.  It specifically
concurred with our recommendations regarding the need for plans
critical to effective systems development, a complete and integrated
program schedule, and a designated official accountable for risk
management.  While HHS agreed that the SEI certification of software
development contractors at a level 2 has value, and plans to use this
rating as one of its selection criteria for future software
development contractors, it said that requiring the current software
developer to achieve this rating would have little if any value, and
that using the level-2 rating as a minimum qualification would limit
the range of potential competitors. 

We believe that such a rating is critical to HCFA's assurance that
its software developers have the capability to successfully complete
contract requirements.  Further, we believe that limiting the choice
of contractors to those who have achieved a level-2 rating is not
only appropriate, but necessary to the successful development of MTS. 

OMB agreed with our recommendations for HCFA to apply sound
systems-development practices to reduce risks and assist management
in controlling MTS.  It also commented that it would request an
evaluation of the benefits of performing an independent assessment of
HCFA's software acquisition capability and, if the benefits are
confirmed, would incorporate the results of this evaluation in its
overall discussions with HCFA regarding the next steps for MTS. 

Based on the complexity of this project and the difficulties HCFA has
had in managing it, we are certain that HCFA will benefit from an
independent assessment of its software acquisition capabilities using
SEI's software acquisition capability maturity model. 




(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
HEALTH AND HUMAN SERVICES
============================================================ Chapter 4



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)




(See figure in printed edition.)Appendix II
COMMENTS FROM THE OFFICE OF
MANAGEMENT AND BUDGET
============================================================ Chapter 4



(See figure in printed edition.)


KEY PROVISIONS OF LAWS,
REGULATIONS, AND BEST PRACTICES
RELATING TO INFORMATION TECHNOLOGY
INVESTMENTS
========================================================= Appendix III

The citations presented in this appendix include key laws,
regulations, and guidance pertaining to information technology
investment issues.  They are organized on the basis of the major
sections of this report. 


   INFORMATION TECHNOLOGY PROJECT
   TRANSITIONS AND
   YEAR 2000
------------------------------------------------------- Appendix III:1


      INFORMATION TECHNOLOGY
      TRANSITION ENVIRONMENT
----------------------------------------------------- Appendix III:1.1

The Clinger-Cohen Act of 1996 (CCA), (formerly named the Information
Technology Management Reform Act of 1996), P.L.  104-106, Division E;
February 10, 1996, effective August 8, 1996, 40 USC 1423(3):  Agency
heads shall "ensure that performance measurements are prescribed for
information technology used by, or to be acquired for the executive
agency, and that the performance measurements measure how well the
information technology supports programs of the executive agency."

CCA, 40 USC 1425(c)(2):  The agency CIO shall "monitor the
performance of information technology programs of the agency,
evaluate the performance of those programs on the basis of the
applicable performance measurements, and advise the head of the
agency regarding whether to continue, modify, or terminate a program
or project."

The Federal Property and Administrative Services Act of 1949, ï¿½
313(b), as added by the Federal Acquisition And Streamlining Act of
1994 (FASA), P.L.  103-355; October 13, 1994, 41 USC 263(b):  "The
head of each executive agency shall approve or define the cost,
performance, and schedule goals for major acquisition programs of the
agency."

The Paperwork Reduction Act of 1995 (PRA) as amended (P.L.  104-13;
May 22, 1995), 44 USC 3506(b)(3)(C):  Requires agencies "to develop
and maintain an ongoing process .  .  .  to establish goals for
improving information resources management's contribution to program
productivity, efficiency, and effectiveness; methods for measuring
progress toward those goals, and clear roles and responsibilities for
achieving those goals."

Office of Management and Budget (OMB) Circular No.  A-130,
"Management of Federal Information Resources," February 8, 1996,
8b(3)(f):  "Agencies shall establish information systems management
oversight mechanisms .  .  .  that ensure that major information
systems proceed in a timely fashion towards agreed-upon milestones in
an information system life cycle, meet user requirements, and deliver
intended benefits to the agency and affected publics through
coordinated decision making about the information, human, financial,
and other supporting resources."

OMB Memorandum M-97-02, Funding Information Systems Investments,
October 25, 1996:  Investments in major information systems proposed
for funding in the President's budget should "be implemented in
phased, successive chunks as narrow in scope and brief in duration as
practicable, each of which solves a specific part of an overall
mission problem and delivers a measurable net benefit independent of
future chunks."

OMB, Evaluating Information Technology Investments:  A Practical
Guide, version 1.0, Office of Information and Regulatory Affairs,
Information Policy and Technology Branch, November 1995:  "Senior
managers need to compare the preliminary results being achieved by a
project against its projected costs, benefits, and risks, and to
identify actual or potential managerial, organizational, or technical
problems."

General Accounting Office (GAO), Systems Assessment Framework:  A
Guide for Reviewing Information Management and Technology Issues in
the Federal Government (SAF), version 1.0, August 1996:  During the
design, development, and deployment of systems, agencies are to
prepare products and documents, including (1) a completed work plan
with human resources, scheduling, and funding for each step, (2) a
transition plan, including conversion from the legacy environment to
the replacement system, and (3) performance measures that link to the
users' operations.  Further, agency management responsibilities
include (1) review and approval of the system test and conversion
plans, (2) formal verification and validation of the developed
system, (3) review and approval of the transition plan, including
procedures for site surveys, conversion preparation, and
contingencies, and (4) agreement that acceptance test results meet
management's criteria. 


      INFORMATION TECHNOLOGY
      MEASURES TO ADDRESS
      YEAR 2000
----------------------------------------------------- Appendix III:1.2

GAO, Year 2000 Computing Crisis:  An Assessment Guide, Exposure
Draft, February, 1997:  This guide presents a structured approach and
a checklist to aid federal agencies in planning, managing, and
evaluating their year 2000 programs.  The most important year-2000
activities are (1) developing an overall year-2000 plan, (2)
identifying responsibilities for managing and monitoring the
year-2000 efforts, (3) preparing an assessment of the severity and
timing of potential year-2000 impact, (4) conducting an inventory of
the systems on which Medicare claims processing depend, (5)
prioritizing and scheduling activities to convert, replace, or
eliminate these systems, (6) developing validation strategies and
test plans for systems, (7) addressing interface and data exchange
issues, and (8) developing contingency plans for critical systems in
the event of failure. 


   MANAGING INFORMATION TECHNOLOGY
   PROJECTS AS INVESTMENTS
------------------------------------------------------- Appendix III:2


      REQUIRED INVESTMENT ANALYSES
----------------------------------------------------- Appendix III:2.1

CCA, 40 USC 1427:  The agency head shall identify major information
technology acquisition programs that have significantly deviated from
the cost, performance, or schedule goals established for the program
in the IRM plan required by the PRA. 

FASA, 41 USC 263(a):  "It is the policy of Congress that the head of
each executive agency should achieve, on average, 90 percent of the
cost and schedule goals established for major and nonmajor
acquisition programs of the agency without reducing the performance
or capabilities of the items being acquired."

FASA, 41 USC 263(c):  The agency head shall "(1) determine whether
there is a continuing need for programs that are significantly behind
schedule, over budget, or not in compliance with performance or
capability requirements; and (2) identify suitable actions to be
taken, including termination, with respect to such programs."

OMB Bulletin No.  95-03, Planning and Budgeting for the Acquisition
of Fixed Assets, June 27, 1995 (superseded):  "The planning for fixed
asset acquisitions should be based on a systematic analysis of
expected benefits and costs.  The fundamental method of formal
economic analysis is benefit-cost analysis."

OMB Circular No.  A-11, "Preparation and Submission of Budget
Estimates," Part 3 Planning, Budgeting and Acquisition of Fixed
Assets (July 1996); Appendix 300A(b):  "The planning for fixed asset
acquisitions should be based on a systematic analysis of expected
benefits and costs.  The fundamental method of formal economic
analysis is benefit-cost analysis."

OMB Circular No.  A-94, "Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs," October 29, 1992;
(5)(c)(3):  Benefit-cost analyses should "consider alternative means
of achieving program objectives by examining different program
scales, different methods of provision, and different degrees of
government involvement.  For example, in evaluating a decision to
acquire a capital asset, the analysis should generally consider:  (i)
doing nothing; (ii) direct purchase; (iii) upgrading, renovating,
sharing, or converting existing government property; or
(iv) leasing or contracting for services."

OMB Circular No.  A-130, 8b(1)(c):  Agencies shall "conduct
benefit-cost analyses to support ongoing management oversight
processes that maximize return on investment and minimize financial
and operational risk for investments in major information systems on
an agency-wide basis."

OMB Memorandum M-97-02:  "Investments in major information systems
proposed for funding in the President's budget should .  .  . 
demonstrate a projected return on the investment that is clearly
equal to or better than alternative uses of available public
resources."


      REQUIRED ALTERNATIVES AND
      RISK ANALYSES
----------------------------------------------------- Appendix III:2.2

CCA, 40 USC 1422:  Agency heads are to "design and implement in the
executive agency a process for maximizing the value and assessing and
managing the risks of the information technology acquisitions of the
executive agency." The process is to (among other things) provide for
the selection of information technology investments using minimum
criteria on whether to undertake an investment (including
quantitatively expressed projected net, risk-adjusted return on
investment, and specific quantitative and qualitative criteria for
comparing and ranking alternative information systems investment
projects) and to provide a means for senior management to obtain
timely information regarding progress (at established milestones) in
terms of cost, capability of the system to meet specified
requirements, timeliness, and quality. 

OMB Bulletin No.  95-03, Attachment A (superseded):  "Alternative
fixed assets, information system designs, and other solutions to meet
a mission need should always be explored.  Analyses of fixed asset
acquisitions should include a comprehensive set of options.  If it is
decided to acquire the services of fixed assets, the decision whether
to purchase or lease may be analyzed by comparing the present value
of expected life-cycle costs over the period during which the
services of the asset will be needed."

OMB Circular No.  A-94; (5)(b):  "A program is cost-effective if, on
the basis of life cycle cost analysis of competing alternatives, it
is determined to have the lowest costs expressed in present value
terms for a given amount of benefits." (5)(c)(3) "Analyses should
also consider alternative means of achieving program objectives by
examining different program scales, different methods of provision,
and different degrees of Government involvement.  For example, in
evaluating a decision to acquire a capital asset, the analysis should
generally consider:  (i) doing nothing; (ii) direct purchasing; (iii)
upgrading, renovating, sharing, or converting existing Government
property; or (iv) leasing or contracting for services."


      REQUIRED INVESTMENT
      MANAGEMENT STRATEGIES
----------------------------------------------------- Appendix III:2.3

CCA, 40 USC 1422(b)(2):  The information technology investment
process of executive agencies is to "be integrated with the processes
for making budget, financial, and program management decisions within
the executive agency."

CCA, 40 USC 1423(3):  "The head of an executive agency shall .  .  . 
ensure that performance measurements are prescribed for information
technology used by or to be acquired for the executive agency and
that the performance measurements measure how well the information
technology supports programs of the executive agency."

PRA, 44 USC 3506(b)(2):  Each agency is to develop and maintain a
strategic IRM plan on how IRM activities help accomplish agency
missions. 

PRA, 44 USC 3506(b)(3)(A):  Requires agencies to "ensure that
information resources management operations and decisions are
integrated with organizational planning, budget, financial
management, human resources management and program decisions" 44 USC
3506(b)(3)(C), and to "establish goals for improving information
resources management's contribution to program productivity,
efficiency, and effectiveness, methods for measuring progress towards
those goals, and clear roles and responsibilities for achieving those
goals."

OMB Circular No.  A-130, 8b(2):  "Agencies shall establish and
maintain strategic information resources management planning
processes" that include "[s]trategic IRM planning that addresses how
the management of information resources promotes the fulfillment of
an agency's mission."

OMB's Practical Guide:  Agency processes should include a disciplined
and structured management forum for making information technology
investment decisions, with the authority to approve, cancel, or delay
projects, mitigate risks, and validate expected returns.  Also, the
agency should establish an executive management team that makes
funding decisions based upon comparisons and trade-offs among
competing project proposals, especially for those projects expected
to have agencywide impact.  All management decisions should be
documented along with data supporting the required changes.  Common
problems and their solutions, which are applicable to one information
technology project, should be evaluated as to how they apply to other
information technology projects under management's purview. 


   GENERALLY ACCEPTED SYSTEMS
   DEVELOPMENT BEST PRACTICES
------------------------------------------------------- Appendix III:3


      CRITICAL SYSTEMS DEVELOPMENT
      PLANS:  REQUIREMENTS
      MANAGEMENT
----------------------------------------------------- Appendix III:3.1

Department of the Air Force, Guidelines for Successful Acquisition
and Management of Software-Intensive Systems (Air Force Guidelines)
Volume 1, Version 1.1, February 1995:  Requirements must constantly
be managed because they significantly affect total system development
costs and schedule.  Management of requirements must stress a
commitment to an iterative process that utilizes structured
requirements methods and appropriate tracking and analysis tools.  In
addition, traceability from original, identified needs to their
derived requirements, designs, and implementation must be assured. 

Software Engineering Institute, Capability Maturity Model Software
(CMM),Version 1.1, February 1993:  Requirements management includes
(1) managing and documenting the system requirements and their
allocation throughout the project's life, (2) providing adequate
resources and funding for managing the allocated requirements, and
(3) ensuring that members of the software engineering group and other
software-related groups are trained to perform their requirements
management activities. 

Software Engineering Institute, Software Acquisition Capability
Maturity Model (SA-CMM), Version 1.01, December 1996:  Requirements
management ensures that software requirements are unambiguous,
traceable, testable, documented, and controlled.  A plan describing
requirements management should be developed prior to contractual
actions and should cover:  (1) objectives of the project team's
requirements development and management activities, (2) activities to
be performed, including requirements identification, (3)
identification of the groups, and intergroup coordination, associated
with requirements development and management activities, (4) the
extent of end-user involvement in the acquisition, (5) procedures for
requirements development, including planning, identification,
analysis, and verification, (6) procedures for requirements
management, including baseline establishment, change control, and
status reporting, and (7) procedures for impact analysis of changes
to requirements or introduction of new requirements, including
performance, cost, and schedule.  The plan should also describe a
mechanism for tracing requirements during software development to
ensure that requirements have been included in the implemented work
products and services. 


      CRITICAL SYSTEMS DEVELOPMENT
      PLANS:  SOFTWARE DEVELOPMENT
----------------------------------------------------- Appendix III:3.2

Air Force Guidelines:  The software development plan is the key
software document reflecting the contractor's overall software
development approach.  It includes resources, organization,
schedules, risk identification and management, data rights, metrics,
quality assurance, control of nondeliverable computer resources and
identification of commercial off-the-shelf systems, reuse, and
government-furnished software that the contractor intends to use. 

CMM:  The software development plan provides the basis for performing
and managing the software project's activities and addresses the
commitments to the software project's customer according to the
resources, constraints, and capabilities of the software project. 
The software development plan includes (1) the software project's
purpose, scope, goals, and objectives, (2) selection of a software
life cycle, (3) identification of the selected procedures, methods,
and standards for developing and/or maintaining the software, (4)
identification of software work products to be developed, (5) size
estimates of the software work products and changes to the software
work products, (6) estimates of the software project's effort and
costs, (7) estimated use of critical computer resources, (8) the
software project's schedules, including identification of milestones
and reviews, (9) identification and assessment of the project's
software risks, and (10) the project's software engineering
facilities and support tools. 

SA-CMM:  The project team should track the contractor's development
of the software engineering environment required to support the
software. 


      CRITICAL SYSTEMS DEVELOPMENT
      PLANS:  CONFIGURATION
      MANAGEMENT
----------------------------------------------------- Appendix III:3.3

Air Force Guidelines:  Requests for proposals should require offerers
to provide a configuration management plan that addresses change
control throughout the development process, the offerer's
configuration management organization, the tools to be used,
configuration management personnel experience, and a description of
the offerer's configuration management training program.  The
government should also have a configuration management plan to
control changes to functional and performance requirements.  This
plan is needed when the software and its documentation are released
to the government. 

CMM:  The purpose of software configuration management is to
establish and maintain the integrity of the products of the software
project throughout the project's software life cycle.  This practice
includes the development of a configuration management plan in the
early stages of overall project planning, which is used as the basis
for performing configuration management activities, including
establishing a configuration management library system as a
repository for the software baselines, and identifying the software
products to be placed under configuration management. 

SA-CMM:  The customer's project team should develop and implement the
plans for moving and supporting the acquired software products.  One
goal for moving software from the contractor to the customer is
maintaining configuration management throughout the transition. 
Software acquisition planning documents such as a configuration
management plan should be developed prior to contractual actions. 


      CRITICAL SYSTEMS DEVELOPMENT
      PLANS:  SYSTEMS INTEGRATION
----------------------------------------------------- Appendix III:3.4

Defense Systems Management College:  Systems Engineering Management
Guide, January 1990:  A primary role of systems engineering is to
ensure that the many diverse elements constituting a system are
compatible and ready when needed.  This avoids the situation in which
hardware or software, when integrated into the system, fails to
function as intended as part of the system.  Integration ensures that
all pieces of the system will work together to realize system goals. 


      CONTRACTOR'S SOFTWARE
      DEVELOPMENT STRATEGY AND
      CAPABILITY
----------------------------------------------------- Appendix III:3.5

Air Force Guidelines:  Prior to or during source selection,
contracting agencies should evaluate offerers' software development
capabilities.  This can start with an overall assessment, such as
SEI's software capability evaluation, which focuses on the details of
tools, metrics, personnel facilities, and management control. 
Contracting agencies should require that offerers selected for best
and final offers submit to a capability evaluation with the goal of
achieving a level-3 rating. 

Air Force Guidelines:  A prudent contractor will implement a
measurement process that includes collecting and receiving actual
data and analyzing that data.  For measurement to be effective, a
metrics usage plan should be developed to determine what data should
be collected and analyzed.  Contractors should be required to submit
this plan to the government, since it describes to what extent and at
what frequency offerors will provide metrics to the government and
how they will be used internally to manage the proposed program. 

CMM:  In order for software development organizations to achieve
lasting results from process improvement efforts, it is necessary to
design an evolutionary path that increases an organization's software
process maturity in stages.  The capability maturity model for
software provides software organizations with guidance on how to gain
control of their processes for developing and maintaining software
and how to evolve toward a culture of software engineering and
management excellence.  The CMM guide was designed to assist software
organizations in selecting process improvement strategies by
determining current process maturity and identifying the issues most
critical to software quality and process improvement. 

Measurements are made and used to determine the functionality and
quality of software products.  Examples of these measurements include
the numbers, types, and severity of defects identified in the
software products tracked cumulatively and by stage.  Other
measurements are made and used to determine the status of software
product engineering activities such as the cost to implement and test
software changes.  As part of software project tracking and
oversight, the size of the software products is tracked, actual size
of code is compared with estimates documented in the software
development plan, and the overall projected size of the software is
refined, monitored, and adjusted regularly. 

SAF:  Agencies should ensure that they design, develop, test and
deploy their automated systems in accordance with conventional
management and technical practices.  This activity includes
evaluating the agency's software engineering processes for critical
attributes such as configuration management, software subcontract
management, and software quality assurance.  Also, agencies are
responsible for preparing estimated project cost schedules using
commercially available software packages, such as the constructive
cost model or the software life cycle management (SLIM) model, or
customized in-house techniques. 


      PROJECT SCHEDULE DEVELOPMENT
      AND MANAGEMENT
----------------------------------------------------- Appendix III:3.6

CCA, 40 USC 1427:  The agency head shall identify in the agency's IRM
plan (required by PRA) "any major information technology acquisition
program, or phase or increment of such a program, that has
significantly deviated from the cost, performance, or schedule goals
established for the program."

SA-CMM:  A project management plan is required to manage the critical
dependencies and critical paths of the project's overall software
acquisition schedule.  The project's overall acquisition schedule
typically specifies that milestones, tasks, commitments, critical
dependencies, staffing, costs, and reviews are allocated in the
schedule consistent with the project's defined software acquisition
process.  In addition, critical dependencies and paths defined and
reflected in the schedule should be tracked on a regular basis. 


      PROJECT RISK MANAGEMENT
      PROCESSES
----------------------------------------------------- Appendix III:3.7

CCA, 40 USC 1422:  Requires executive agency heads to design and
implement a process to assess and manage the risks of the information
technology acquisitions. 

PRA, 44 USC 3506(h)(5):  Requires agencies to assume responsibility
for assessing and managing the risks of major information systems
initiatives through a process that is (1) integrated with budget,
financial, and program management decisions, and (2) used to select,
control, and evaluate the results of major information systems
initiatives. 

OMB Memorandum 97-02:  Directs federal agencies to reduce risks
associated with information technology investments "by establishing
clear measures and accountability for project progress."

Air Force Guidelines:  Provides detailed guidance on the development,
components, and operation of an effective risk management process. 

SA-CMM:  Recommends that organizations identify risks as early as
possible, adjust the acquisition strategy to manage those risks, and
develop and implement a risk management process as an integral part
of the organization's software acquisition process. 

Improving Mission Performance Through Strategic Information
Management and Technology, GAO/AIMD-94-115:  Recommends that agencies
include a disciplined risk management process based on explicit
criteria, to assess risk as a component of managing information
systems projects as investments. 

OMB's Practical Guide:  Recommends that senior managers compare the
preliminary results of information technology projects against
projected costs, benefits, and risks, and identify actual or
potential managerial, organizational, or technical problems. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix IV


   ACCOUNTING AND INFORMATION
   MANAGEMENT DIVISION,
   WASHINGTON, D.C. 
-------------------------------------------------------- Appendix IV:1

Mark E.  Heatwole, Assistant Director
Leonard J.  Latham, Technical Assistant Director
Danny R.  Latta, Technical Adviser
Yvette R.  Banks, Senior ADP Telecommunications Analyst
James Houtz, Senior Information Systems Analyst
Elizabeth A.  Roach, Senior Business Process Analyst
J.  Michael Resser, Business Process Analyst
Michael P.  Fruitman, Communications Analyst


   KANSAS CITY REGIONAL OFFICE
-------------------------------------------------------- Appendix IV:2

John Mollet, Senior Evaluator

*** End of document. ***