Information Technology: Greater Use of Best Practices Can Reduce
Risks in Acquiring Defense Health Care System (26-SEP-02,
GAO-02-345).
This report examines the acquisition of the Composite Health Care
System (CHCS) II. It is one in a series of reports reviewing the
Department of Defense's use of best practices in acquiring
information technology systems. CHCS II is expected to cost about
$1 billion to deliver full capability to almost 1,100 health
facilities worldwide by 2008. GAO's objectives were to determine
(1) what progress has been made against project commitments, (2)
whether the system has been economically justified, and (3)
whether effective technical and management controls are in place.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-02-345
ACCNO: A05197
TITLE: Information Technology: Greater Use of Best Practices Can
Reduce Risks in Acquiring Defense Health Care System
DATE: 09/26/2002
SUBJECT: Best practices
Cost control
Information technology
Medical information systems
Schedule slippages
Systems design
Systems management
DOD Composite Health Care System
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-02-345
Report to the Chairman and Ranking Minority Member, Subcommittee on
Readiness and Management Support, Committee on Armed Services, U. S.
Senate
United States General Accounting Office
GAO
September 2002 INFORMATION TECHNOLOGY Greater Use of Best Practices Can
Reduce Risks in Acquiring Defense Health Care System
GAO- 02- 345
Why GAO Did This Study
This report examines the acquisition of the Composite Health Care System
(CHCS) II. It is one in a series of reports reviewing the Department of
Defense*s use of best practices in acquiring information technology
systems. CHCS II is expected to cost about $1 billion to deliver full
capability to almost 1,100 health facilities worldwide by 2008. GAO*s
objectives were to determine (1) what progress has been made against
project commitments, (2) whether the system has been economically
justified, and (3) whether effective technical and management controls are
in place.
September 2002 INFORMATION TECHNOLOGY Greater Use of Best Practices Can
Reduce Risks in Acquiring Defense Health Care System
This is a test for developing highlights for a GAO report. The full
report, including GAO's objectives, scope, methodology, and analysis, is
available at www. gao. gov/ cgi- bin/ getrpt? GAO- 02- 345. For additional
information about the report, contact Randolph C. Hite (202- 512- 3439).
To provide comments on this test highlights, contact Keith Fultz (202-
512- 3200) or email HighlightsTest@ gao. gov.
Highlights of GAO- 02- 345, a report to the Chairman and Ranking Minority
Member, Subcommittee on Readiness and Management Support, Committee on
Armed Services, U. S. Senate
What GAO Recommends
GAO is making several recommendations to the Secretary of Defense,
including ensuring expanded use of best practices in managing CHCS II by
(1) modifying the project*s investment strategy to justify investment in
each system release before beginning development, and measuring return on
investment; and (2) employing performance- based contracting practices
where possible on all future delivery orders. Defense generally agreed
with our recommendations.
United States General Accounting Office
What GAO Found
CHCS II is envisioned as a state- of- the- art automated medical
information system allowing improved health- care decisions and lower
medical and system costs. An expected highlight is computer- based patient
records that doctors and other health service providers will be able to
access from any military treatment facility, irrespective of location (see
figure). While early CHCS II progress was limited, clear improvement has
been evident over the past 2 years, as Defense has begun to embrace
industry best practices. The first incremental release of the system, with
a deployment decision scheduled for September 2002, is set to contain more
capability than was originally planned. The current schedule is
nevertheless 3 years beyond the initial estimate, due in part to major
program changes and a system redesign; benefits are in question since
measurement has not yet begun; and costs to date are about 2 1/2 times the
1998 estimate for deploying the first increment to one region.
Until recently, Defense*s basis for investing in this system has been an
outdated cost/ benefit analysis that did not reflect important changes in
assumptions and, further, justified all system increments beyond the first
solely on an economic analysis of the entire system. Officials now,
however, are finalizing an updated analysis and have stated they plan to
adopt an incremental approach to justifying investment in CHCS II, a best
practice that successful organizations have been following for years.
The department is moving forward in ensuring that effective acquisition
controls are in place, but further progress can be made. Technical and
management controls are largely in effect in several areas, including
testing and risk management. But performance- based contracting is not
being followed, resulting in the risk that CHCS II will take longer to
acquire and cost more than necessary.
Creating a Computer- based Patient Record
Source: GAO based on DOD data.
G A O Accountability Integrity Reliability
Highlights
Page i GAO- 02- 345 Defense Health Care System Letter 1
Results in Brief 2 Background 4 DOD Has Made Recent Progress Since Missing
Early Project
Commitments 10 DOD Has Not Adequately Justified Investments in CHCS II
Releases, but Improvements Are Under Way and Planned 19 Important
Technical and Management Controls Now Being
Followed, but Opportunities Exist For Further Use of Best Practices 23
Conclusions 35 Recommendations for Executive Action 35 Agency Comments and
Our Evaluation 37
Appendix I Objectives, Scope, and Methodology 40
Appendix II Comparison of CHCS and CHCS II Capabilities 44
Appendix III Detailed Explanation of How CHCS II Will Operate 46
Appendix IV CHCS II Release 1 Software Packages 51
Appendix V CHCS II Capabilities Planned for Releases 3- 7, and Expected
Release Dates 53
Appendix VI GAO Contact and Staff Acknowledgments 54 GAO Contact 54
Acknowledgments 54 Contents
Page ii GAO- 02- 345 Defense Health Care System Tables
Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment
Dates 9 Table 2: Division of CHCS II Management Responsibilities and
Functions Among DOD Units 10 Table 3: MHS Enterprise Architecture
Satisfaction of DOD
Architecture Framework Essential Products 28 Table 4: Simplified Part of
Information Exchange Matrix 30 Table 5: Examples of MHS and CHCS II
Mapping 31 Table 6: CHCS II Release 1 Software Packages by Location 51
Figures
Figure 1: Simplified DOD Health Affairs Organization Chart 5 Figure 2:
Release 1 Interfaces with Four Existing Systems 8 Figure 3: Time Line of
CHCS II*s Initial and Current Schedules 13 Figure 4: CHCS II Expenditures
to Date 14 Figure 5: Number of Priority 1, 2, and 3 Defects Opened June
2001
through July 2002 18 Figure 6: Unresolved Priority 1, 2, and 3 Defects
Opened April 2001
through July 2002 19 Figure 7: A Simplified Operational High- Level
Graphic 29 Figure 8: Creating a Computer- based Patient Record 46 Figure
9: CHCS II Tri- level Architecture 48 Figure 10: Example of CHCS II
Regional Communications 50
Page iii GAO- 02- 345 Defense Health Care System Abbreviations
ASD( C31) Assistant Secretary of Defense for Command, Control,
Communications, and Intelligence CIO chief information officer CDR
clinical data repository COTS commercial- off- the- shelf CHCS Composite
Health Care System CPR computer- based patient record DISA Defense
Information Systems Agency DOD Department of Defense GOTS government- off-
the- shelf IT information technology MHS Military Health System MTF
military treatment facilities SEI Software Engineering Institute
Page 1 GAO- 02- 345 Defense Health Care System
September 26, 2002 The Honorable Daniel K. Akaka Chairman The Honorable
James M. Inhofe Ranking Minority Member Subcommittee on Readiness and
Management Support Committee on Armed Services United States Senate
This report responds to your request that we review the Department of
Defense*s (DOD) acquisition of the Composite Health Care System (CHCS) II,
an automated medical information system to be deployed to about 1,100
health facilities at 142 military installations worldwide. According to
the department, CHCS II will facilitate the worldwide delivery of health
care to active- duty service members, their dependents, and other eligible
beneficiaries, when care is received in military facilities. DOD has
invested about $320 million to acquire an initial version of CHCS II, and
expects to invest about $1 billion to acquire and deploy the full system
and an additional $3.2 billion to operate and maintain the system over its
useful life. DOD began the last phase of testing of the initial version of
the system in May 2002, plans to make a deployment decision for this
version in September 2002, and plans to deploy it to all sites by October
2005. Over the next 6 years, DOD plans to acquire and deploy a series of
more capable versions of the system, delivering full capability to all
health facilities at all installations by 2008.
This report is one in a series examining DOD*s use of best practices in
acquiring information technology (IT) systems. 1 As agreed with your
offices, the objectives of our review were to determine (1) what progress
DOD has made against CHCS II project commitments, including required
system capabilities, expected system benefits, and estimated project costs
and milestones; (2) whether DOD has economically justified its investment
in the system; and (3) whether DOD has effective technical and
1 U. S. General Accounting Office, DOD Systems Modernization: Continued
Investment in the Standard Procurement System Has Not Been Justified, GAO-
01- 682 (Washington, D. C.: July 31, 2001); U. S. General Accounting
Office, Information Technology: DLA Should Strengthen Business Systems
Modernization Architecture and Investment Activities,
GAO- 01- 631 (Washington, D. C.: June 29, 2001).
United States General Accounting Office Washington, DC 20548
Page 2 GAO- 02- 345 Defense Health Care System
management controls in place for acquiring the system. The controls
reviewed were requirements management, test management, architecture
development and alignment, risk management, and contract management.
Details of our objectives, scope, and methodology are included in appendix
I.
Concerning progress to date, DOD did not meet its May 1998 commitment to
deliver initial CHCS II system capabilities and associated mission
benefits in April 1999; and, since it did not estimate the cost of
delivering the initial system capabilities (release 1), it does not have a
cost commitment against which to measure progress. Reasons for missing
delivery of the capabilities include initial use of a Web- based
architecture that did not meet system performance requirements, initial
requirements that were ill- defined, a later influx of requirements
changes, and unexpected budget cuts that forced changes in the project*s
scope and approach. By July 2000, 26 months later, the department had
redefined its plans for CHCS II to include adopting a new technical
architecture, establishing a means for controlling changes to
requirements, committing to incremental releases 2 of system capabilities,
and delaying the decision date for deploying release 1 to January 2001,
which was 21 months later than its May 1998 commitment. It did not,
however, similarly redefine benefits, or provide a cost commitment for the
initial version.
Since July 2000, DOD has made progress in delivering release 1*s system
capabilities, although precisely which capabilities will be included in
this release have continued to change, and the deployment decision date
was delayed another 20 months, to September 2002. Progress against CHCS
II*s benefits commitment is not yet known, because the project has yet to
measure the accrual of actual benefits even though versions of release 1
have been used at test sites for about 2 years. Similarly, progress
against a cost commitment has not been measured because DOD did not
develop a cost estimate for release 1 until April 2002, only 5 months
before this release is to be deployed. Available measures of whether
release 1 meets revised commitments 3 indicate that the system is
maturing, but questions still exist as to the system*s operational
efficiency. For example, while
2 DOD plans to divide the CHCS II acquisition into seven releases. Release
1 was scheduled for a deployment decision in September 2002; release 2 in
July 2003; the remaining deployments following at about 1- year intervals.
3 Examples include functional and performance requirements. Results in
Brief
Page 3 GAO- 02- 345 Defense Health Care System
DOD has corrected all of the most serious defects, about 50 defects
affecting system efficiency remain unresolved.
Second, DOD has not economically justified its investment in CHCS II
releases, but it plans to do so from this point forward. It initially
developed a single economic justification for CHCS II in 1998, which has
been used as the basis for its past and ongoing investment in releases 1,
2, and 3. However, this justification had known limitations in the cost
estimate and was not updated to reflect material changes in planned system
capabilities, costs, and milestones. Further, although DOD has adopted an
incremental approach to acquiring and deploying system releases, it did
not treat investment in releases 2 and 3 as separate investment decisions.
This approach is not consistent with best practices and federal
requirements. As such, it has invested hundreds of millions of dollars in
the system over 4 years without knowing whether the cost of a particular
release is outweighed by its benefits. Recently, DOD updated its analysis
and reported that releases 1 and 2 are economically justified. CHCS II
project officials have acknowledged that incremental investment is a best
practice that should be followed, and stated that they will change their
strategy for future releases to include economically justifying each
release before investing in it and verifying each release*s benefits and
costs after deployment.
Finally, DOD has appropriately given attention and priority to employing
certain acquisition best practices, including those related to managing
system requirements and test events, aligning the system architecture with
the DOD military health system enterprise architecture, 4 and proactively
managing project risks. Such practices better position the department for
successfully acquiring CHCS II. Nevertheless, management can be improved
upon. For example, performance- based contracting methods are not being
used to ensure contractor accountability. Until this is changed,
acquisition risks will remain higher than they need to be.
DOD*s recent progress on CHCS II is due in part to its attention and
commitment to adopting certain acquisition management best practices, such
as those governing requirements management. We are making recommendations
to the Secretary of Defense for similarly pursuing
4 An enterprise architecture is the operational and technical blueprint
for DOD- wide military health affairs.
Page 4 GAO- 02- 345 Defense Health Care System
improvements in CHCS II investment and other acquisition management
controls.
DOD provided what it termed *official oral comments* on a draft of this
report. In its comments, DOD generally agreed with the report*s message
and recommendations. Its comments are discussed more fully in the Agency
Comments and Our Evaluation section of this report.
DOD operates a nationwide health care program, including overseas
facilities, for its health care beneficiaries. This program is just one of
the responsibilities of the Office of the Under Secretary of Defense for
Personnel and Readiness, along with recruiting, training, educating, and a
range of morale, welfare, and recreation services. Within the Office of
the Under Secretary is the Office of the Assistant Secretary for Health
Affairs, as well as assistant secretaries or deputy undersecretaries for
force management, reserve affairs, readiness, and program integration.
The Assistant Secretary for Health Affairs is responsible for the military
health system (MHS) program (see fig. 1). The MHS program has two
missions: wartime readiness (maintaining the health of service members and
treating wartime casualties), and peacetime care (providing for the health
care needs of the families of active- duty members, retirees, their
families, and survivors). The Assistant Secretary for Health Affairs
establishes policy regarding health care for all DOD beneficiaries and
also plans and budgets for health care operations and maintenance. At the
same time, each military service has its own medical department, headed by
a surgeon general, which operates medical facilities and recruits and
funds military medical personnel. Health Affairs, through the surgeons
general, provides policy direction, oversight, and resource support to
these medical facilities, referred to as military treatment facilities.
Background
Page 5 GAO- 02- 345 Defense Health Care System
Figure 1: Simplified DOD Health Affairs Organization Chart
Source: GAO based on DOD data.
Today, over 8 million people are eligible to receive care from MHS
facilities* about 80 percent retirees and dependents and about 20 percent
active- duty personnel. Reflecting the magnitude of this patient
population, MHS*s worldwide operations are significant: about 130,000
personnel at about 1,000 Army, Navy, and Air Force medical facilities
divided into 12 domestic plus European, Pacific, and Latin American
regions. The fiscal year 2002 MHS budget is almost $24 billion, with about
$5 billion of this amount funded by the three services* budgets for
military medical personnel. MHS: A Large Health Care
Provider
Page 6 GAO- 02- 345 Defense Health Care System
DOD provides about 75 percent of MHS services through these military
facilities, supplementing this by contracting for health services with
civilian providers. Active- duty members are required to obtain care at
military treatment facilities if such are available; in contrast, retirees
and dependents may obtain care at either military facilities, on a
spaceavailable basis, or through civilian contract providers.
Since 1968, DOD has pursued the goal of providing computer support to its
hospitals and clinics. During fiscal years 1976 to 1984, DOD spent about
$222 million to acquire, implement, and operate various health- care
computer systems. The original CHCS, begun by DOD in 1988 and first
deployed in 1993, is the primary DOD medical information system now used
in all MHS facilities worldwide. This system supports DOD hospitals and
clinics in registering patients, documenting inpatient activity, and
providing laboratory, radiology, pharmacy, drug- interaction, and other
functions. Other medical information systems include the Ambulatory Data
System, Preventive Health Care Application, and the Nutrition Management
Information System. 5 According to a 1996 analysis of the MHS mission, the
existing medical information systems cannot meet DOD*s needs. The
department thus initiated CHCS II in 1997 with the goal of assisting
clinicians in making health- care decisions and lowering medical and
systems costs through, for example, replacement of each of the above-
cited systems. (Appendix II shows a comparison of the capabilities
provided by CHCS and CHCS II.)
CHCS II is designed to be a multi- level, open system based on a
clientserver model. 6 (Appendix III describes the system in greater
detail.) The system is to consist of user 7 workstations and application
and security network computing devices, located at military treatment
facilities. DOD plans to divide the CHCS II acquisition into seven
releases. The decision to
5 Each of these systems provides certain patient- related information. For
example, the Ambulatory Data System captures certain outpatient
information relating to diagnosis and treatment; the Preventive Health
Care Application contains information on preventive health services; and
the Nutrition Management Information System supports therapeutic nutrition
therapy and medical food management.
6 Open systems conform to industry standards so that commercial products
can easily be used and support costs can be minimized. A client is usually
a desktop computing device or program that is *served* by one or more
networked computing devices.
7 CHCS II users are physicians, nurses, other medical personnel, and
technical or administrative support staff. CHCS II Design: A Brief
Description
Page 7 GAO- 02- 345 Defense Health Care System
deploy release 1 is scheduled for September 2002; release 2 for July 2003.
Remaining deployments are scheduled to begin at about 1- year intervals.
DOD*s description of release 1 shows users creating and storing
computerbased patient records (CPRs) using 30 CHCS II workstation- and
computer- based software packages (21 commercial, 9 government- owned; see
app. IV for more information on these packages). DOD plans to connect each
facility*s workstations and servers via each installation*s local or wide
area networks. Each installation is also to be connected through a wide
area network 8 to a defense computing center, where the CPRs will be
stored in a database known as the clinical data repository. As release 1
is deployed to more installations, DOD intends that medical providers will
be able to access a patient*s CPR from any military treatment facility, no
matter where the patient is being or has been treated.
Release 1 is not intended to meet all needs, and thus additional system
capabilities are to be added over several years, beginning in July 2003.
DOD plans for release 1 to provide system security; user and system
interface controls; and various functions such as appointment management,
order- entry/ management, reporting, preventive health care, immunization
tracking, and CPR management. Release 1 will target ambulatory
beneficiaries (out- patients). Other service categories, such as inpatient
support, are planned for future releases. Release 1 will also replace the
Ambulatory Data System and the Preventive Health Care Application.
Release 1 is planned to interface with four systems: CHCS; the Defense
Enrollment Eligibility Reporting System, which receives and responds to
CHCS II requests for eligibility and immunization information; the
Thirdparty Outpatient Collection System, which receives third- party
billing data from CHCS II; and the Corporate Executive Information System,
which receives patient information from CHCS II that is used for executive
reporting (see fig. 2). Two additional interfaces are planned for release
2, one to a new patient eligibility system and one to a new primary care
manager system. 9
8 The network is DOD*s Unclassified but Sensitive Internet Protocol Router
Network. 9 The new eligibility system is intended to replace the existing
Defense Enrollment Eligibility Reporting System and improve MHS sharing of
health care information and eliminate manual benefits determinations. The
new Primary Care Manager System is intended to allow assignment and change
of beneficiaries*s primary care managers.
Page 8 GAO- 02- 345 Defense Health Care System
Figure 2: Release 1 Interfaces with Four Existing Systems
Source: GAO based on DOD data.
The acquisition plan calls for release 1 to rely on certain existing CHCS
capabilities, including patient appointment/ scheduling, radiology,
pharmacy, and laboratory; plans call for future releases to gradually
replace CHCS, allowing it to be shut down by 2008. Table 1 provides
details on releases 1 and 2 deployment dates and capabilities. Appendix V
provides details on capabilities and deployment dates for later releases.
Page 9 GAO- 02- 345 Defense Health Care System
Table 1: CHCS II Releases 1 and 2 Capabilities and Expected Deployment
Dates Release number and expected deployment date Capabilities included
1 September 2002
Automated health evaluation/ assessment questionnaire Beneficiary
population health reports Centralized health record repository CHCS,
eligibility, executive information, and third- party collection system
interfaces Clinical outpatient graphical user interface CPR Enrollment and
eligibility data Immunization data New ambulatory data and preventive
health systems Patient data transcription Patient encounter documentation
Patient encounter results codes Patient health history Patient
satisfaction reports Patient self- assessment data entry Provider profile
data Specialist consults results reports User alerts and reminders User
role- based access to system 2 July 2003
Automated clinical practice guidelines Dental charting and documentation
Enhanced clinical decision support Global health record repository New
eligibility and primary care manager systems interfaces Optometric
documentation and order entry Theater (deployed) functions
Source: GAO based on DOD data.
Several DOD units within the Office of the Assistant Secretary for Health
Affairs, supported by other DOD components, are involved in acquiring and
deploying CHCS II. The MHS chief information officer (CIO) under the
Assistant Secretary is responsible for several MHS IT projects, of which
CHCS II is but one. MHS*s clinical information technology program office
provides direct management of the project. The Assistant Secretary of
Defense for Command, Control, Communications, and Intelligence (ASD( C3I))
is responsible for overseeing the project and deciding whether it can
proceed to the next acquisition cycle milestone. Table 2 shows how
management responsibility for CHCS II is divided among DOD units. CHCS II
Project
Management
Page 10 GAO- 02- 345 Defense Health Care System
Table 2: Division of CHCS II Management Responsibilities and Functions
Among DOD Units Entity Responsibility/ function
MHS CIO Oversees the MHS information management and technology program.
Program Executive Office Directs all centrally managed MHS IT system
acquisitions, including oversight of
procurement, development, implementation, deployment, maintenance, and
operations. Clinical IT Program Office (includes the CHCS II project team)
Acquires and deploys CHCS II; monitors contract performance; performs
testing and
training on use of the product delivered by the contractor. Other DOD
organizations Service military treatment facilities upgrade power needs,
validate data conversions,
and uninstall and remove older system hardware. Another MHS program office
provides communications systems infrastructure. The Office of Program
Analysis and Evaluation is responsible for verifying and validating the
reliability of economic analyses for major projects such as CHCS II. The
Army Test and Evaluation Command is responsible for conducting the
operational test. The Navy Center for Cost Analysis is responsible for
validating the CHCS II life- cycle cost estimate. ASD (C3I) Approves the
project to proceed through its acquisition cycle on the basis of a review
of
key documents * an independently evaluated life- cycle cost/ benefit
estimate, a component cost analysis, and an acquisition strategy and
project baseline. (Is also the milestone decision authority for CHCS II,
which is categorized as a major automated information system initiative.)
a Joint Requirements Oversight Council Approves mission need and
operational requirements for automated information
systems with joint (multiservice) interest. a A project that is either
designated as such by the Assistant Secretary, or estimated to require (a)
project costs in any single year in excess of $30 million; (b) total
project costs in excess of $120 million; or (c) total life- cycle costs in
excess of $360 million, all in fiscal year 1996 constant dollars.
Source: GAO based on DOD data.
Best practices and related federal guidance emphasize the need for
disciplined processes and information to help ensure that projects are
implemented at acceptable costs and within reasonable and expected time
frames. 10 They also recognize that acquisitions should contribute to
tangible, observable mission performance enhancements. In short, projects
are expected to meet the capability, schedule, benefit, and cost
commitments upon which their approval was justified.
For the last 4 years, MHS has invested about $320 million in CHCS II but
either does not know if it is meeting its commitments or has largely not
met them. The exception is in its system capability commitment, where it
is delivering more in release 1 than originally envisioned. The reasons
for missing commitments relate to a flawed initial design, user
dissatisfaction with the system prototype, additional requirements,
technical problems,
10 Clinger- Cohen Act of 1996, Public Law 104- 106 and Office of
Management and Budget (OMB) Circular A- 130, Management of Federal
Information Resources (Nov. 30, 2000). DOD Has Made
Recent Progress Since Missing Early Project Commitments
Page 11 GAO- 02- 345 Defense Health Care System
such as slow network performance, and a funding cut for the 1999 MHS
budget. As a result, the deployment and concomitant benefits have been
delayed, and costs have risen substantially over the initial costs
approved for release 1. Nevertheless, release 1 is now in the last phase
of testing and available data suggest that it is largely performing as
intended, although some issues surrounding system efficiency remain.
During the first 2 years of the project, MHS made little progress against
the CHCS II capability, schedule, benefits, and cost commitments it made
in 1998. Progress has improved since then, however.
Capability Commitment. CHCS II release 1 is intended to meet
foundational system functional, performance, and interface requirements,
allowing medical providers at military treatment facilities to exchange,
display, and place data into beneficiaries*s medical records regardless of
where the patient is being treated or where past treatments occurred.
Release 1 is also expected to deliver more capabilities than the original
commitment due to the requirements added in 2001 in response to user
demands. Examples of added requirements (originally planned for future
releases) include Windows 2000, pathology order entry, patient
selfassessment, and certain patient medical charting features. MHS reports
that these additional capabilities added about $7 million to the cost of
release 1, and about 6 months to the schedule.
Schedule Commitment. MHS plans to make a deployment decision for release
1 in September 2002** over 3 years late. When the project was approved in
May 1998, MHS envisioned deploying a prototype system beginning in October
1998 and beginning deployment of the initial version about April 1999. The
initial deployment has been delayed for several reasons. The prototype
architecture, which employed a Web- based user interface, 11 was found to
cause slow system performance. Further, medical personnel were not
satisfied with other system functions, such as the ability to view and
print patient documentation, causing requirements redefinition and a
system redesign; and the Joint Requirements Oversight Council added more
CHCS II requirements in 2000, which added 15
11 A program used to connect the user to the system. The prototype version
was based on a program used to connect to the World Wide Web. Overall
Progress Against
Project Commitments Has Been Limited, but Recent Events Show Improvement
Page 12 GAO- 02- 345 Defense Health Care System
months to the schedule. DOD also cut the MHS budget for fiscal year 1999
that MHS reports caused the initial deployment to be delayed 6 months.
System users demanded a number of additional requirements in 2001, and
system testing found technical problems with the implementation, such as
slow network performance. This combination of the additional user
requirements and technical problems added another 20 months to the
schedule, for a total delay of 41 months past the original April 1999
estimate. Figure 3 provides a time line of CHCS II*s initial and current
schedules through 2003.
Page 13 GAO- 02- 345 Defense Health Care System
Figure 3: Time Line of CHCS II*s Initial and Current Schedules
Source: GAO based on DOD data.
Benefits Commitment. The 1998 economic analysis estimated CHCS II
benefits at about $5.7 billion in 1998 dollars, including $86 million in
benefits between fiscal years 2000 and 2002. A May 2002 update of this
analysis estimates benefits of about $6. 6 billion in 1998 dollars, but
accrual of benefits is not expected to begin until fiscal year 2003. Thus,
MHS has not met its original benefit commitments. In addition, it has yet
to begin verifying whether its benefit expectations are being met, and
only began
Page 14 GAO- 02- 345 Defense Health Care System
validating its plan for measuring benefit accrual in March 2002, although
versions of release 1, each providing more capabilities, have been
operating at test sites since June 2000. According to project officials,
they did not begin this process earlier because users at the sites would
not yet have been in a position to determine how CHCS II affected their
work.
Cost Commitment. MHS did not prepare a baseline cost estimate for
acquiring and deploying release 1 until April 2002. However, on the basis
of the 1998 economic analysis, DOD did approve project funding of about
$109 million for acquiring and deploying CHCS II from 1997 through 1999.
This included the acquisition cost for release 1, plus costs to deploy the
system to the sites in a single MHS region from April through September
1999. As of April 2002, MHS reports it has spent about $284 million to
acquire release 1, which is about two and one- half times the amount
approved in 1998 to acquire and deploy release 1 to a single MHS region.
As with the schedule delays, MHS reports that these additional costs are
due to the problems with the prototype, additional requirements, and
technical complexity. In addition to the release 1, MHS reports that it
has spent about $35 million to date on release 2 and about $700,000 on
release 3, as shown in figure 4.
Figure 4: CHCS II Expenditures to Date
Source: GAO based on DOD data.
Page 15 GAO- 02- 345 Defense Health Care System
Two indicators of system maturity are the results of system testing and
the number of system defects remaining. One type of testing that both best
practices and federal guidance advocate before a system is deployed is
system acceptance testing, the purpose of which is to verify that the
system meets key technical and system requirements. Defects are system
problems that require resolution. As a system matures, the number of
defects should show a downward trend. The CHCS II project categorizes
defects by severity, ranging from 1 to 5, with 1 being the most serious
and the highest priority. 12 According to the test plan, priority 1 and 2
defects, which jeopardize patient safety or adversely affect mission-
essential capabilities, need to be resolved before the system can continue
to operational testing and deployment. Priority 3, 4, and 5 defects do not
need to be resolved since these defects have less effect on the system.
Both the system acceptance test results and trends in priority 1 and 2
defects show that release 1 is maturing. Trends in priority 3 defects,
which require inefficient workarounds to overcome, do not yet show a
maturing trend.
The project team conducted the acceptance test to validate that the system
is effective and suitable when operating in a productionrepresentative
military treatment facility environment, and that the system meets all
critical operational issues and key performance parameters. An outcome of
acceptance testing is a determination of the system*s maturity and its
readiness to proceed to the final phase of testing* operational testing
and evaluation. 13
The acceptance test was sufficiently scoped to provide a basis for making
each of these determinations. For example, the test was conducted at 4
12 Priority 1 defects prevent the accomplishment of an operational- or
mission- essential capability or jeopardize personnel safety. Priority 2
defects adversely affect the accomplishment of a mission- essential
capability, degrading performance, for which no alternative workaround is
known. Priority 3 defects are similar to priority 2, but workarounds are
available. Priority 4 and 5 defects are less serious.
13 Operational testing is used to independently assess effectiveness and
suitability for end users. Recent Test Results and
System Defect Rates Show Initial Release Is Maturing, But Issues Remain
Acceptance Test Results
Page 16 GAO- 02- 345 Defense Health Care System
sites and covered all 28 categories of release 1 functionality, 14 and all
1,138 release 1 requirements, including performance (e. g., system
availability) and interface requirements. While two functionality
categories* telephone consults and self- assessments* were not exercised
at all test sites, this was mitigated by the fact that *telephone consult*
was exercised at one site using an automated test tool, and *self-
assessment* was tested at one site and also used by the test team on test
patients at another site.
The acceptance test also defined 15 system maturity parameters/
indicators. 15 According to the test results, 10 of these 15 were fully
satisfied (judged to be mature) and 5 were partially satisfied (judged to
be of limited maturity). For example, one parameter called *functional
completeness* required that 90 percent of the system requirements be met.
According to the test results, release 1 exceeded this requirement,
meeting 100 percent of the requirements. Other examples of parameters that
were fully met are *fault closure rate* and *fault severity/ safety,*
meaning that defect density is decreasing and no priority 1 and 2 defects
exist, respectively. According to the test results and our analysis of
defects (discussed in the next section of this report), both were met.
For the five parameters that were not fully satisfied and thus rated as
limited maturity, we analyzed the reasons provided for these
determinations, and believe that the parameters were sufficiently
satisfied to mitigate any concerns about whether the parameter was
sufficiently satisfied. For example, one of the five is *key performance
parameters* which actually consists of nine performance requirements that
must be met. Of these nine, test results show that eight were met and one
was not. However, the one that was not met is actually a performance
requirement for release 2 and not release 1. Similarly, one of the five is
*system
availability,* which for CHCS II is 99 percent system availability.
According to the test results, this parameter was not met because a
14 Alerts, allergies, appointments/ scheduling, assessment plan, clinical
notes, consult tracking, encounter coding, encounter documentation,
external interfaces, forms/ reports, immunizations, medications, order
entry, order sets, patient demographics, patient disposition, patient
search, patient self- assessment tools, problem list, results retrieval,
screening, security, summary of care, telephone consults, template
management, templates, user preferences, and wellness.
15 Configuration management, data recovery and restoration, fault closure
rate, fault severity, functional baseline, functional completeness,
installation, interchangeability, key performance parameters, network
design, reliability, software, system availability, test environment, and
safety.
Page 17 GAO- 02- 345 Defense Health Care System
scheduled equipment replacement at the Defense Information Systems Agency
computing center caused the system to not be operational while the
replacement occurred. However, as we have previously reported, 16 system
availability is generally regarded as the time that a system is operating
satisfactorily, expressed as percentage of the time that the system is
required to be operational (i. e., excluding the time that scheduled
system maintenance is occurring). Thus, the down time associated with the
scheduled equipment replacement should not have been used in measuring
system availability during the acceptance test, and according to the test
report, if the equipment replacement had occurred during nonclinic hours,
the system would have met the 99 percent requirement.
On the basis of the 15 maturity parameter/ indicator ratings, the project
office determined that release 1 was mature, meaning that the testers had
high confidence that the system was ready for the final phase of testing*
operational testing and evaluation.
Another indicator of system maturity and quality is defect density, which
can be measured in a number of ways, including the trend in the number of
defects being reported and the trend in the number of unresolved
(uncorrected) defects. Available data on these measures show a generally
positive maturity trend, although issues of system efficiency and progress
relative to defect expectations exist. For example, after adding
requirements in June 2001, the number of defects opened each month began
to rise, as shown in figure 5. By January 2002, however, this number began
to drop and by the end of July 2002 was about zero for priority 1, 2, and
3 defects combined.
16 See, for example, Weather Forecasting: Radar Ability Requirement Not
Being Met,
GAO/ AIMD- 95- 132 (Washington, D. C.: May 31, 1995) and Air Traffic
Control: FAA Plans to Replace Its Host Computer System Because Future
Availability Cannot Be Assured,
GAO/ AIMD- 98- 138R (Washington, D. C.: May 1, 1998). Trends in System
Defects
Page 18 GAO- 02- 345 Defense Health Care System
Figure 5: Number of Priority 1, 2, and 3 Defects Opened June 2001 through
July 2002
Source: GAO based on DOD data.
The number of unresolved defects (or defects open each month) also began
to rise after last June, as shown in figure 6. Since February, however,
this number, especially for priority 1s and 2s, has dropped. As of the end
of July 2002, all priority 1 and 2 defects had been resolved, but 46
priority 3 defects remained open.
While not as serious as priority 1s and 2s, priority 3 defects are still
detrimental because they require workarounds of some sort, which by
definition decrease system efficiency. For example, one unresolved
priority 3 defect is when CHCS II prevents terminal access because the
user had not employed the workstation within the required time limit. When
this happens, reentering the user password should allow renewed access,
but does not. This means that the workstation is unusable until it can be
restarted, resulting in lost work time. The test plan only requires
priority 1 and 2 defects to be resolved before moving on to operational
testing and deployment. However, according to DOD officials, they plan to
correct all defects identified prior to July 31, 2002, before deploying
release 1, except for 11 that are embedded in vendor commercial- off-
theshelf software packages and thus must be corrected by the respective
vendors.
Page 19 GAO- 02- 345 Defense Health Care System
Figure 6: Unresolved Priority 1, 2, and 3 Defects Opened April 2001
through July 2002
Source: GAO based on DOD data.
Between the project*s start in 1997 and April 2002, MHS invested hundreds
of millions of dollars in CHCS II without following all IT investment best
practices and related federal guidance that call for separating large
systems into smaller increments and making informed return- oninvestment
decisions based on reliable estimates of incremental project costs,
benefits, and risks. Instead, MHS has justified its past and ongoing
investment in releases 1, 2, and 3 using an outdated analysis of costs and
benefits that, until recently, did not reflect material changes to cost
and benefit assumptions. Moreover, MHS did not treat investment in
releases 2 and 3 as separate decisions, instead viewing these releases as
being justified as part of MHS*s economic analysis of the entire CHCS II
system (all releases). Such a monolithic approach to analyzing and
justifying a system*s return on investment has been abandoned by
successful organizations as inherently unreliable because it relies on
predictions of many variables over many years. According to CHCS II
project officials, they will adopt an incremental approach to investing in
releases from this point forward. DOD Has Not
Adequately Justified Investments in CHCS II Releases, but Improvements Are
Under Way and Planned
Page 20 GAO- 02- 345 Defense Health Care System
The Clinger- Cohen Act of 1996 and OMB guidance 17 provide a framework for
IT investment management that comports with recognized best practices.
Together, they set requirements for (1) economically justifying proposed
projects on the basis of reliable analyses of expected life- cycle costs,
benefits, and risks; (2) using these analyses throughout a project*s life
cycle as the basis for investment selection, control, and evaluation
decisionmaking; and (3) doing so for large projects (to the maximum extent
practical) by dividing them into a series of smaller, incremental
subprojects or releases. By doing so, the tremendous risk associated with
investing large sums of money over many years in anticipation of
delivering capabilities and expected business value far into the future
can be spread across project fragments that are smaller, of shorter
duration, and capable of being more reliably justified and more
effectively measured against cost, benefit, and risk expectations. DOD
policy also reflects these investment principles by requiring that
investments be justified by an economic analysis and, more recently, 18
that investment decisions for major projects such as CHCS II be made
incrementally to better ensure that each increment delivers measurable
benefit, independent of future increments.
IT investment management best practices and federal guidance advocate
economically justifying proposed investments on the basis of a net present
value benefit- to- cost ratio that is greater than 1, basing that ratio on
reliable analyses of expected life- cycle benefits, costs, and risks, and
updating the analysis when significant system changes occur. MHS has not
followed this practice. It originally justified its investment in CHCS II
in 1998, with an economic analysis that showed a 1.14 to 1 benefit- to-
cost ratio for the total program and a 1.09 to 1 ratio for the initial
increment (based on present value adjustments to 1998 dollars). However,
in January 1999 the DOD Inspector General reported that the cost estimate
was based on assumptions that could be flawed and result in estimate
inaccuracies, and that the analysis did not disclose this or its return-
on- investment implications. 19 Moreover, since this analysis was
developed, the project has
17 Clinger- Cohen Act of 1996, Public Law 104- 106; OMB Circular A- 130
(Nov. 30, 2000). 18 DOD Interim Regulation 5000. 2- R and DOD Instruction
5000. 2, Change 1, Operation of the Defense Acquisition System (Jan. 4,
2001). 19 Office of the Inspector General (OIG), Department of Defense,
Acquisition Management of the Composite Health Care System II Automated
Information System, report number 99- 068 (Jan. 21, 1999). Reliable
Economic
Analysis and Incremental Investment Are Tenets of Effective Investment
Management
DOD Did Not Economically Justify Past, Ongoing Investments in CHCS II, But
Has Taken Steps to Do So
Page 21 GAO- 02- 345 Defense Health Care System
undergone a number of changes affecting its cost and benefits profile, as
previously discussed.
MHS prepared two draft updates to its 1998 economic analysis (January and
August 2000), but it did not approve an update until May 2002. According
to project officials, the approved version was delayed because system
requirements were in a state of flux and did not become stable until late
2000. Project officials also stated that the draft updates, which had
benefit- to- cost ratios greater than 1, provided them with sufficient
assurance that continued investment in CHCS II (releases 1, 2, and 3) was
justified. However, according to the project office*s quarterly reports
during 2000, CHCS II was still in a state of change during and after the
time of these updates. As stated in the October 2000 quarterly report, the
economic analysis required revision to reflect recent project developments
and a change in the system release strategy.
The May 2002 economic analysis estimates life- cycle benefits and costs
for the entire CHCS II system, including benefits of about $6.5 billion
and costs of about $4.3 billion (both in 1998 dollars). 20 When converted
to present values, these estimates are approximately $4.6 billion and $2.8
billion, respectively, producing a benefit- to- cost ratio of 1.63 to 1
and a net present value of about $1.78 billion. 21 However, this analysis
is still undergoing review by DOD stakeholders. For example, the
analysis*s lifecycle cost estimate, which was developed by MHS, is
currently being validated by the Naval Center for Cost Analysis. Because
this analysis is still being reviewed, we did not evaluate it.
Incremental investment management involves three fundamental components:
(1) acquiring a large system in a series of smaller increments; (2)
individually justifying investment in each separate increment on the basis
of costs, benefits, and risks; and (3) monitoring actual benefits achieved
and costs incurred on ongoing increments. MHS has structured CHCS II into
a series of seven increments (releases). However, until
20 The cost estimate appropriately does not include the about $320 million
already spent on CHCS II because this constitutes a sunk cost and thus is
not relevant to determining current net present value.
21 The life- cycle benefits estimate includes benefits resulting from
improvements in MHS processes ($ 4.4 billion) and benefits from replacing
existing MHS systems with CHCS II ($ 2.1 billion). Incremental
Justification
Planned for Future CHCS II Investments
Page 22 GAO- 02- 345 Defense Health Care System
recently it had not followed best practices in incrementally justifying
investments in all system releases. More specifically, it did not treat
releases 2 and 3 as separate investments. Further, it has yet to determine
whether actual benefit and cost expectations are being met by the first
release. According to project officials, this is because DOD policy did
not require them to do so, and the DOD CIO, who is the project*s milestone
decision authority, approved the 1998 economic analysis and granted
approval to proceed.
Nevertheless, the May 2002 economic analysis, which was developed during
the course of our review and is currently under review within the
department, does define release 1 and 2 costs and benefits. The release 1
analysis reports life- cycle benefits and costs of $3.4 billion and $2.8
billion, respectively (in 1998 dollars). When converted to present values,
these estimates are approximately $2.4 billion and $2.1 billion,
respectively, producing a benefit- to- cost ratio of 1. 12 to 1 and a net
present value of about $255 million. The release 2 analysis reports life-
cycle benefits and costs of $509 million and $103 million, respectively
(in 1998 dollars). When converted to present values, these estimates are
approximately $359 million and $84 million, producing a benefit- to- cost
ratio of 4.25 to 1 and a net present value of about $275 million. The
analysis does not, however, address release 3.
According to project officials, they have recently decided to economically
justify all future system releases as separate investments based on
updates to the CHCS II economic analysis, and they plan to determine
whether return- on- investment expectations for deployed releases are
being met. The officials also stated that they would analyze the benefits
and costs of all future releases during MHS*s annual IT investment
portfolio reviews and decide whether to fund them. Further, they stated
that actual benefits and costs from deployed releases will be measured for
each release, starting in February 2003 for release 1, and the CHCS II
investment strategy will be updated to reflect these changes.
Page 23 GAO- 02- 345 Defense Health Care System
MHS has generally established technical and management control best
practices in key areas: requirements management, test management,
architecture development and alignment, and risk management. This has not
been the case throughout the project*s life, but project officials said
that they have devoted considerable management attention and given
priority to applying lessons learned and acting on the results of our
work, and they have made improvements to apply best practices.
Nevertheless, the CHCS II project office can improve its risk management
efforts, and it is still not following performance- based contracting best
practices. According to project officials, they did not know how to apply
such contracting practices to a system acquisition, but have recently
begun to develop this expertise. By not following these practices, MHS
risks paying more and taking longer to acquire CHCS II than necessary.
Effectively managing requirements involves establishing and maintaining
agreement among the project team** including end users and contractors* on
what the system is to do (functionality), how well it is to do it
(performance), and how it is to interact with other systems (interfaces).
Best practices 22 for managing requirements include (1) adhering to a
documented requirements management plan, (2) establishing a validated set
of requirements that serves as the baseline against which changes are
made, (3) controlling changes to the baseline, (4) maintaining
traceability among requirements and related project deliverables, and (5)
involving end users in developing and changing requirements.
For CHCS II, MHS has defined products and processes that meet each of
these tenets of effective requirements management. First, there is a
documented CHCS II requirements management plan with specific written
steps governing development and maintenance of requirements. Second, the
requirements for release 1 have been validated and approved by CHCS II
users, and a baseline requirements set exists. Third, a formal change
control process has been put in place, which requires change control board
approval for baseline modifications based on the changes*s impact on the
project. Fourth, a procedure exists for tracking requirements to other
work products by allowing changes to the contractor*s requirements
database only from DOD*s requirements database. Last, the process
22 See, for example, Software Acquisition Capability Maturity Model SM
(CMU/ SEI- 99- TR002, April 1999). Important Technical
and Management Controls Now Being Followed, but Opportunities Exist for
Further Use of Best Practices
Key Requirements Management Controls Have Recently Been Established
Page 24 GAO- 02- 345 Defense Health Care System
provides for end user participation in both developing and approving
changes to existing requirements and addition of new requirements.
To confirm that established requirements management controls were being
followed, we reviewed requirements for two CHCS II capabilities* clinical
practice guidelines, which were defined in 2000, the first year the
current management controls were put in place, and patient index, which
were defined in 2001. In both cases, we determined that defined process
controls were being followed. For example, in both cases, users were
involved in developing and changing requirements and requirements were
baselined and put under change control. Also in both cases, changes to
requirements were assessed in terms of costs, benefits, and risks before
being accepted.
According to project officials, they did not effectively manage
requirements until 2000. A 1999 DOD Inspector General report affirms early
problems with requirements management, 23 as does a report in October 1999
from a CHCS II test team concluding that the lack of complete requirements
caused test problems. In response, MHS redefined how requirements would be
managed, adopting the current process during the 2000- 2001 timeframe. By
doing so, the risks associated with defining complete and correct system
requirements, and preventing uncontrolled changes, commonly referred to as
*requirements creep,* have been mitigated.
Complete and thorough testing is essential to providing reasonable
assurance that systems perform as intended. Testing a system is not a one
time event, but rather is a series of test events throughout the systems
development and maintenance cycle, each of which addressing different
levels and aspects of the system and building on the results of previous
tests. Among the types of tests are (1) tests of small units of software,
known as unit testing; (2) tests of whether an integrated version of the
system meets defined requirements (functional, performance, and
interface), referred to as acceptance testing; and (3) tests of whether
the system meets the needs of users in an operational setting, called
operational testing.
23 Office of the Inspector General (OIG), Department of Defense, report
number 99- 068 (Jan. 21, 1999). Key System Acceptance
Testing Management Controls in Place
Page 25 GAO- 02- 345 Defense Health Care System
Best practices including our own guidance 24 recommend that organizations
implement a structured and disciplined approach to managing each type of
tests. DOD policy requires a similar approach. In general, effectively
managing a test event, such as system acceptance testing, entails (1)
developing a test plan that defines, for example, test objectives, scope,
schedules, resource needs, locations, and documentation and reporting
requirements; (2) preparing test procedures, based on the plan, that
specify test cases, steps, data, inputs, and expected outputs, and that
are traceable to requirements; (3) defining test exit criteria, which
establish the requirements that must be met to successfully complete
testing; (4) executing tests in accordance with plans and procedures; (5)
documenting test results in accordance with plans and procedures; (6)
identifying, prioritizing, and correcting defects, and re- testing defect
corrections; (7) comparing test results with exit criteria to ensure that
specified requirements are met; and (8) reporting test results to
management in accordance with plans and procedures.
Between January and April 2002, MHS conducted system acceptance testing.
Our analysis of the management of this test event showed that key tenets
of effective test management were met. For example, MHS prepared test
plans that addressed test objectives, scope, schedules, resource needs,
locations, and documentation and reporting requirements. Also, MHS
developed detailed test procedures that included, among other things, test
steps, cases, data, inputs, and expected outcomes. Further, our analysis
showed that test procedures were directly linked to functional,
performance, and interface requirements. Specifically, we analyzed a
statistically valid sample of 59 requirements against the test procedures
25 and found that 57 were traceable to specific test procedures. With
respect to the two other requirements, one was a duplicate, and the other
was not a CHCS II requirement but rather a requirement for an interfacing
system that was inadvertently associated with CHCS II.
Additionally, the test plan defined acceptance test exit criteria as
closing all priority 1 and 2 defects, developing workarounds for priority
3 defects, freezing the system baseline, and demonstrating system
interoperability and stability. Further, MHS executed the acceptance test
between January and April 2002 and documented the results in an acceptance
report.
24 See, for example, Year 2000 Computing Crisis: A Testing Guide (GAO/
AIMD- 10. 1. 21, November 1998). 25 See appendix I for statistical
sampling methodology.
Page 26 GAO- 02- 345 Defense Health Care System
During and after the test, DOD identified, prioritized, and corrected
defects, closing all priority 1 and 2 defects. Finally, in the acceptance
report, DOD compared test results with exit criteria and provided the test
results to management.
Developing, maintaining, and using architectures, or blueprints, is a best
practice in engineering both individual systems and entire enterprises. 26
Requirements for having and using architectures to guide and constrain IT
investment decisionmaking are also addressed in federal law and guidance,
27 and in DOD policy. 28 When acquiring or developing new IT systems or
maintaining existing systems, these sources recognize that it is very
important to ensure that systems are built and modified (i. e.,
*architected*) within the context of the architecture for the enterprise
that the system supports. To do less risks having systems that are, for
example, duplicative and not integrated. In the case of CHCS II, we found
that the system and the MHS enterprise architecture are generally aligned.
An enterprise architecture is a systematically derived snapshot* in useful
models, diagrams, and narrative* of a given entity*s operations (business
and systems), including how its operations are performed, what information
and technology are used to perform the operations, where the operations
are performed, who performs them, and when and why they are performed. The
architecture describes the entity in both logical terms (e. g.,
interrelated functions, information needs and flows, work locations,
systems, and applications) and technical terms (e. g., hardware, software,
data, communications, and security). Enterprise architectures provide
these perspectives for both the entity*s current, or *as is* environment,
and for its target, or *to be* environment; they also provide a high-
level capital investment roadmap for moving from one environment to the
other.
26 Enterprises can be single organizations or business/ mission areas that
transcend more than one organization (e. g., financial management, combat
system identification, or medical health care).
27 Clinger- Cohen Act of 1996, P. L. 104- 106; OMB Circular A- 130 (Nov.
30, 2000); A Practical Guide to Federal Enterprise Architectures, Version
1. 0, Chief Information Officers Council (February 2001); Federal
Enterprise Architecture Framework, Version 1.1, Chief Information Officers
Council (September 1999).
28 February 28, 1998, memorandum * jointly signed by the Under Secretary
of Defense for Acquisition and Technology; the Acting Assistant Secretary
of Defense for Command, Control, Communications, and Intelligence, ASD(
C3I); and the Director for Command, Control, Communications, and Computer
Systems, Joint Chiefs of Staff. CHCS II and MHS
Architectural Alignment Is Occurring
MHS Enterprise Architecture
Page 27 GAO- 02- 345 Defense Health Care System
Managed properly, an enterprise architecture can clarify and help optimize
the interdependencies and interrelationships among a given entity*s
business operations and the underlying systems and technical
infrastructure that support these operations. 29 Our experience with
federal agencies has shown that attempting to define system- level
architectures (e. g., develop requirements and design specifications) and
use these systems architectures to build systems without having an
enterprise architecture and aligning its systems* architectures with the
enterprise architecture often results in systems that are duplicative, not
well integrated, unnecessarily costly to maintain, and limited in terms of
optimizing mission performance.
MHS has developed an initial version of an enterprise architecture. 30 We
analyzed this architecture against the requirements for an enterprise
architecture as defined by the DOD architecture framework, 31 which
specifies the requirements that all DOD enterprise architectures are to
meet. As we have previously reported, this framework is consistent with
commercial architecture frameworks. 32 According to the DOD framework,
enterprise architectures must have seven essential products, including a
high- level operational concept graphic, information exchange matrix, and
technical architecture profile. Our analysis shows that the current
version of the MHS enterprise architecture has each of these products (see
table 3).
29 For additional information on enterprise architectures, see Information
Technology: Enterprise Architecture Use across the Federal Government Can
Be Improved, GAO- 02- 6 (Feb. 19, 2002).
30 DOD has recently initiated a program to develop and implement a DOD-
wide financial management enterprise architecture, which is to include
each DOD business area that triggers a financial event. MHS is
participating in this DOD- wide architecture effort, including providing
MHS architecture products and staff.
31 DOD*s architecture framework is called the Command, Control,
Communications, Computers, Intelligence, Surveillance, and Reconnaissance
Architecture Framework. 32 GAO- 01- 631.
Page 28 GAO- 02- 345 Defense Health Care System
Table 3: MHS Enterprise Architecture Satisfaction of DOD Architecture
Framework Essential Products Product Description MHS architecture
satisfies?
Enterprise view and summary information Serves as planning guide and
summarizes *who, what, when, why, and how*
for architecture to be developed Yes Integrated dictionary Provides a
central source for definitions of all terms used in all architecture
products Yes High- level operational concept graphic Shows a high- level
graphic description of operational concept, including
organizations, missions, and geographic distribution of assets Yes
Operational node connectivity description Identifies organizational
elements that produce, process, and consume
information; need to exchange information between elements; and
characteristics of information exchanged (content, media, volume
requirements, security classification, timeliness, and interoperability
requirements)
Yes Operational information exchange matrix Provides information exchange
requirements, identifying who exchanges what
information with whom, why information is necessary, and how it is needed
Yes System interface description Links operational and systems
architecture views by depicting information
systems and their interfaces to organizational elements that produce,
process, and consume information
Yes Technical architecture profile Establishes a set of rules governing
system implementation and operation;
references existing technical guidance and discusses how that guidance has
been or needs to be implemented
Yes Source: GAO based on DOD data.
For example, MHS*s operational high- level graphic illustrates four
business areas: access to care, provision of health services, population
health management, and manage the business. (See fig. 7 for a simplified
version of the operational high- level graphic.) These business areas are
then decomposed into activities. The activities, in part or whole, are
carried out by MHS systems.
Page 29 GAO- 02- 345 Defense Health Care System
Figure 7: A Simplified Operational High- Level Graphic
Source: GAO based on DOD data.
As another example, DOD*s framework requires an information exchange
matrix that identifies who exchanges what information with whom, why the
information is necessary, and how it is needed. The MHS enterprise
architecture has this matrix, and for each information exchange, it
identifies the information source and destination, criticality and
timeliness, and frequency and format requirements. (See table 4.)
Page 30 GAO- 02- 345 Defense Health Care System
Table 4: Simplified Part of Information Exchange Matrix Who? Why? How?
Information exchange Source Destination Timeliness Criticality Frequency
Media type
Eligibility determination Enrollment/
eligibility management staff
Enterprise- wide registration staff Seconds- hoursdays Critical Event-
driven Data, text
Customer data release agreement
Enrollment/ eligibility management staff
Primary care manager Seconds- hoursdays High Event- driven Data, text
Encounter (administrative) data
Enterprise- wide registration staff Enterprise- wide
scheduling staff Seconds- minutes Routine Event- driven Data Source: GAO
based on DOD data.
A third example pertains to DOD*s requirement that component architectures
contain a profile of related DOD technical standards. 33 MHS*s enterprise
has such a standards profile, and this profile is aligned with the
relevant DOD technical standards. For example, DOD requires a standard
graphic data interchange, specific Windows management standards, and use
of the federal information processing standard for password usage. The MHS
architecture specifies each of these standard requirements in its
standards profile.
MHS has also developed a CHCS II architecture to define a level of system
detail and specificity that is not found in an enterprise architecture,
but which is needed in order to build the system. The system architecture
is articulated in a number of system engineering documents, such as the
system requirements document, the system design specifications, a data
dictionary and database description, and commercial- off- the- shelf
application and system software descriptions. According to MHS officials,
the CHCS II architecture is fully aligned with the MHS architecture. They
attributed this alignment largely to the fact that the two architectures
were developed at the same time by the same individuals.
To verify this alignment, we compared the two architectures at both the
logical and the technical levels. Our analysis showed that the CHCS II
system architecture is aligned with the MHS enterprise architecture. At
the logical level, we identified 52 capabilities that CHCS II is to
perform in support of the MHS mission. Of these 52 capabilities, we
judgmentally selected 13, or 25 percent, from 3 of the 4 MHS business
areas that relate
33 Department of Defense, Joint Technical Architecture, version 3.1, March
31, 2000. CHCS II System Architecture
Page 31 GAO- 02- 345 Defense Health Care System
to CHCS II, and compared them with activities defined in MHS*s enterprise
architecture. All 13 were aligned with the MHS enterprise architecture.
(See table 5 for three examples.)
Table 5: Examples of MHS and CHCS II Mapping MHS business area MHS
activity CHCS II capability
Access to Care Check- in patient Patient registration Population Health
Management Educate patients concerning
new practices Patient education Provision of Health Services Document care
plans and
delivery Clinical documentation Source: GAO based on DOD data.
At the technical level, we compared the MHS technical standards profile
and the CHCS II system architecture to determine if they specified the
same standards. MHS*s profile identifies 61 technical standards that are
appropriate for CHCS II. Of the 61 technical standards, 59 are specified
in the CHCS II system architecture. For instance, DOD requires that
medical systems conform to the defense information infrastructure common
operating environment. MHS and CHCS II plans show that CHCS II is to be
compliant with this standard in release 2. In addition, DOD requires that
medical systems follow certain file transfer standards, and CHCS II
architecture documentation specifies these standards. Further, DOD
requires that a standard database language be used for medical systems,
and CHCS II architecture documentation specifies this standard language.
The two standards in the MHS standards profile that are not specified in
the CHCS II system architecture relate to document interchange. These
standards are not currently applicable because the interfaces they apply
to are for future releases.
Managing project risk means proactively identifying facts and
circumstances that increase the probability of failing to meet project
commitments, and taking steps to prevent this from occurring. Best
practices and federal guidance 34 advocate risk management. To be
effective, risk management activities should be (1) based on documented
policies and procedures and (2) executed according to a written plan that
34 See, for example, Software Acquisition Capability Maturity Model SM
(CMU/ SEI- 99- TR002, April 1999); OMB Circular A- 130 (Nov. 30, 2000).
Risk Management Controls
in Place, but Application of Controls Can Be Improved
Page 32 GAO- 02- 345 Defense Health Care System
provides for identifying and prioritizing risks, developing and
implementing appropriate risk mitigation strategies, and tracking and
reporting on progress in implementing the strategies. By doing so,
potential problems can be avoided before they manifest themselves into
cost, schedule, and performance shortfalls.
MHS has largely implemented a process for managing CHCS II risks that
meets risk management best practices. Using a documented risk management
policy and associated procedures, project officials have developed and are
implementing a written plan for managing CHCS II risks. Under this plan,
identified risks are captured in a database, and each risk is assigned a
priority of 1 to 5, with 1 being the most serious and the highest
priority. 35 Further, a strategy for mitigating each risk is defined and
its implementation is managed, including assigning accountability for the
risk, tracking status of the risk, and reporting on progress in
implementing the risk mitigation strategy.
Using its risk management approach, MHS has identified 99 risks over the
life of the project, including 27 priority 1 and 2 risks for release 1 and
release 2. To verify that the established risk management approach was
being followed, we reviewed the history and status of these priority 1 and
2 risks and found that they were generally being managed in a manner
consistent with the process. For example, of the five open priority 1 and
2 risks associated with release 1 of the system, all had risk mitigation
strategies that were being implemented and tracked through formal status
reporting. However, of the five priority 1 and 2 risks associated with
release 2, four were being reported as active but no mitigation actions
were being reported over the last year for any of the four. According to
project officials and documentation, this was because the four were
actually inactive but the risk database had not been updated. The project
office has since updated the database.
In addition, our analysis led us to question why one priority 1 risk* user
acceptance of the system* had been closed and was no longer being actively
managed since user acceptance is a common risk of commercialoff- the-
shelf- based system solutions and, given the status of CHCS II
35 In MHS*s risk management plan, priority 1 risks require immediate
project changes to eliminate or reduce the risk and management attention;
priority 2 risks require some project changes, mitigation strategies, and
careful monitoring; priority 3 risks require mitigation strategies and
careful monitoring; and priority 4 and 5 risks do not require mitigation
activities.
Page 33 GAO- 02- 345 Defense Health Care System
deployment, user interaction with the system to date has been extremely
limited (i. e., confined largely to only the approximately 100 medical
personnel who have participated in system acceptance testing). In
response, the project office returned this risk to active status, and it
is now being formally managed using a mitigation strategy and tracked via
formal reporting.
Project officials attributed these instances to lapses in updating the
risk database. The quality of this database is important, as it defines
where limited resources will be focused to thwart the emergence of real
problems with real consequences. Without an accurate and complete
inventory of risks, their status, and the progress of strategies to
address them, the chances of system cost, schedule, and performance
problems occurring are increased.
Performance- based contracting is a recognized best practice that federal
procurement policy reflects. 36 Specifically, this policy requires
agencies to use performance- based contracting methods to the maximum
extent practicable when acquiring services. Moreover, the CHCS II
acquisition strategy calls for performance- based contracting practices.
To qualify as performance- based under federal regulations, an acquisition
should have (1) requirements that define the work to be performed in
measurable terms; (2) standards (quality or timeliness) that are tied to
performance requirements; (3) a quality assurance plan that describes how
the contractor*s performance in meeting requirements will be measured
against standards; and (4) positive and negative incentives.
In the delivery orders that it has executed to date with the CHCS II
integration contractor, the project office has not fully satisfied any of
these tenets. First, the delivery orders contain performance requirements
and the requirements are written in measurable, mission- related terms.
However, the terms are allowed to change with each delivery order
modification without holding the contractor accountable for performance up
to the point of the modification. Instead, both the requirements and
performance prior to the modification are supplanted by the modified
requirements, and contractor performance begins anew. In light of the
number of modifications that have accompanied CHCS II delivery orders,
36 Federal Acquisition Regulation, Part 37. Performance- based
Contract Management Practices Not Being Used
Page 34 GAO- 02- 345 Defense Health Care System
this results in considerable contractor activity not being managed on the
basis of performance. For example, one delivery order was modified 19
times during its 30- month life, and the contractor*s performance was
started anew five times, although work was behind schedule each time.
Second, the CHCS II delivery orders contain timeliness, but not quality,
performance standards for each requirement. More specifically, each order
has a defined delivery date for each deliverable, but does not specify
standards governing the quality of each deliverable. Third, CHCS II
project management does not have a quality assurance plan defining how it
will apply standards to measure contractor performance in meeting
requirements. Fourth, the orders do not specify incentives for either
positive or negative contractor performance.
According to project officials, performance- based contracting practices
have not been employed because these practices have traditionally not been
used for IT services contracts, and they did not know how to successfully
do so for a system acquisition like CHCS II, where the government is
acquiring a product rather than a service. The officials also stated that
the multi- agency contract vehicle they chose to use offered them
flexibility and was immediately available, and they have not viewed
performance- based contracting as a priority.
By not using performance- based contracting practices on CHCS II, the
project office lacks an effective means for ensuring that it pays a fair
price for what it receives. To illustrate, three times in 2001 the project
office agreed to a schedule delay for release 2 deliverables because the
contractor was not able to meet the release 1 schedule. In each instance,
the release 2 work was also behind schedule. Nevertheless, in both
instances, release 2 delivery dates were changed and the contractor*s
performance was shown as being on schedule.
Project officials stated that they recognize the value of performance-
based contracting. To prepare them for applying these best practices on
future CHCS II releases, they stated that they have recently awarded a
performance- based contract for help- desk services for a number of MHS
systems, including CHCS II, and that another MHS project is currently
negotiating a performance- based contract for software development. They
said that they intend to use these experiences to assist them in
negotiating a performance- based contract for CHCS II release 3.
Page 35 GAO- 02- 345 Defense Health Care System
Owing largely to the absence of the kind of management and technical
controls that are hallmark qualities of system investment and acquisition
best practices, CHCS II*s early years produced little more than lessons
learned. Since then, the project*s management team has recognized the need
to change and given priority attention to doing so. As a result, they
introduced key missing best practices and made other improvements to the
project, some of which have occurred during the course of our review.
These needed practices and improvements have contributed to where MHS
stands today: in the latter stages of having an initial version of CHCS II
that shows signs of maturation and operational readiness, although
questions about operational efficiency due to unresolved defects remain a
concern.
A larger concern, however, are unanswered questions about CHCS II*s
investment value. These questions exist because the project*s management
and oversight teams, to include both the MHS and DOD CIOs, have not given
implementation of incremental investment management practices adequate
priority and attention. Consequently, MHS has continued to invest in CHCS
II without sufficient economic justification for doing so. The prospects
for this changing are encouraging, given statements by project officials,
but until defining and implementing processes for incremental investment,
the risk of DOD*s spending more on the system than it will return in
measurable benefits is increased. Opportunities also exist for increasing
the effectiveness of ongoing risk management activities by ensuring that
the risk database is complete and correct, thereby heading off problems
before they occur. Further, opportunities exist for adopting performance-
based contracting best practices to better ensure that DOD pays a fair
price for contractor deliverables.
Greater use of best practices in the areas of investment, risk, and
contract management will better position CHCS II management to ensure that
it will not only be investing in the right vehicle but that it will be
investing the right way, meaning that it will be following the kind of
proven management practices that increase the probability that required
system capabilities and expected benefits will be delivered on time and
within budget.
To strengthen CHCS II investment, risk, and contract management practices,
and thereby increase the chances of the department*s investment in CHCS
II*s producing mission value commensurate with system costs, we recommend
that the Secretary of Defense, through the Assistant Secretary of Defense
for Health Affairs, direct the MHS CIO to give expanded use of best
practices in managing CHCS II the attention and Conclusions
Recommendations for Executive Action
Page 36 GAO- 02- 345 Defense Health Care System
priority they deserve. At a minimum, the Assistant Secretary should direct
the MHS CIO to take the following actions:
As part of CHCS II deployment decisions, including any request to the
DOD CIO for deployment approval, consider the aggregate impact on defense
health affairs mission performance caused by the workarounds needed to
compensate for all unresolved defects affecting the system*s operational
efficiency.
Define and implement incremental investment management processes to
include (1) modifying the CHCS II investment strategy to define how this
approach will be implemented; (2) justifying investment in each system
release before beginning detailed design and development of the release;
(3) requiring that such justification be based on reliable estimates of
costs, benefits, and risks; (4) measuring whether actual return- on-
investment for each deployed release is in line with justification
forecasts; and (5) using actual return- on- investment results in deciding
whether to begin detailed design and development of the next system
release.
Verify that the CHCS II inventory of risks is complete and correct, and
report this to the Assistant Secretary for Health Affairs every 6 months,
along with a report on the status of all top priority risks, including
each risk*s probability of occurrence and impact on mission.
Employ performance- based contracting practices on all future CHCS II
delivery orders to the maximum extent possible, including (1) defining
performance standards against which deliverables can be judged, (2)
developing and using quality assurance plans that describes how contractor
performance against the standards will be measured, and (3) defining and
using contractor incentives and penalties tied to the quality plan.
Additionally, we recommend that the Secretary of Defense direct the
Assistant Secretary of Defense for Command, Control, Communications, and
Intelligence, who is the designated approval authority for CHCS II, to
monitor the project*s use of best practices, including implementation of
each of the above recommendations, and use this information to oversee the
project*s movement through its acquisition cycle. To this end, we also
recommend that the Assistant Secretary, or other designated CHCS II
approval authority, not grant any request for deployment approval of any
CHCS II release that is not justified by reliable analysis of the
release*s costs, benefits, and risks.
Page 37 GAO- 02- 345 Defense Health Care System
DOD provided what it termed *official oral comments* from the Assistant
Secretary of Defense for Health Affairs on a draft of this report. In its
comments, DOD stated that it agreed with our report*s overall message of
expanding the department*s use of acquisition management best practices on
CHCS II. Further, it agreed with two of our three primary recommendations
and associated findings, and it partially agreed with the third, taking
exception only to two of this recommendation*s component elements. Each of
these two elements is discussed below.
First, DOD disagreed with the recommendation calling for using actual
return- on- investment results from an implemented system release in
deciding whether to begin detailed design and development of the next
system release, citing the impact on the product delivery cycle (i. e.,
schedule). Instead, DOD stated that it would use *post- deployment
findings and lessons learned for a given release to minimize risks
associated with future release development.* Assuming that DOD*s reference
to *post- deployment findings* includes having at least preliminary
information on cost and benefit accrual, we do not disagree with DOD*s
proposed alternative and do not believe that it is inconsistent with our
recommendation. If it does not, we do not believe that the proposed
alternative adequately positions the department to make informed
investment decisions on future releases because it does not provide for
knowing, at least preliminarily, whether a prior release*s mission value
is being realized. Given that producing mission value is a paramount
reason for investing in technology, such information about prior system
releases is essential in making prudent decisions about further investment
in the system
Moreover, by not having at least preliminary information about actual
return- on- investment, DOD would not be able to accomplish its stated
goal of using *post- deployment findings . . . to minimize risks
associated with future release development* because it would not have any
indication about whether release return- on- investment expectations are
accruing. On the contrary, DOD*s approach would expose the program to
unnecessary risk in the name of meeting a scheduled milestone. Restated,
it would increase the chances of doing the wrong thing faster. The intent
of our recommendation is to prevent this by ensuring that DOD has an
adequate understanding of returns from prior investments before choosing a
course of action on subsequent investments. Accordingly, we have not
modified our recommendation.
Second, DOD disagreed with the recommendation element calling for
reporting on CHCS II program risks to the Assistant Secretary for Health
Agency Comments
and Our Evaluation
Page 38 GAO- 02- 345 Defense Health Care System
Affairs every 6 months, instead stating that the timing of its reporting
would comply with DOD internal milestone review requirements, which based
on program plans would result in annual reporting. We do not believe that
DOD*s proposed alternative provides for adequate disclosure of those items
that have a high probability of significantly affecting delivery of
promised system capabilities on time and within budget to the executive
leadership sponsoring CHCS II. As our research of leading organization IT
investment management practices shows, awareness of program risks,
particularly the top priority risks covered by this recommendation, are
fundamental to ongoing investment control activities and, among other
things, should be frequently disclosed to investment executives. Further,
over extended periods of time, such as 1 year, these risks can manifest
themselves into material program cost, schedule, and performance
shortfalls. For these reasons, 6- month reporting on these risks should be
viewed as the minimum acceptable frequency. Thus, we have not modified our
recommendation.
DOD also provided a comment on our recommendation for the Assistant
Secretary of Defense for Command, Control, Communications, and
Intelligence to only approve deployment of a CHCS II release if it is
justified by reliable analysis of the release*s cost, benefits, and risks.
Specifically, DOD stated that while the Assistant Secretary was currently
the approval authority for CHCS II deployment and related milestone
decisions, this decision authority could be delegated in the future. We
have adjusted our recommendation to recognize this possibility.
We are sending copies of this report to the Chairman and Ranking Minority
Members of the Senate Committee on Armed Services; House Committee on
Armed Services; Subcommittee on Defense, Senate Committee on
Appropriations; the Subcommittee on Defense, House Committee on
Appropriations; the Subcommittee on Military Readiness, House Committee on
Armed Services; Senate Committee on Government Affairs; and House
Committee on Government Reform. We are also sending copies to the
Secretary of Defense and the Director, Office of Management and Budget.
Copies will be available upon request and will also be available without
charge at our Web site at www. gao. gov.
Page 39 GAO- 02- 345 Defense Health Care System
Should you have any questions on matters discussed in this report, please
contact me at (202) 512- 3439 or by e- mail at HiteR@ gao. gov. Key
contributors to this report are listed in appendix VI.
Randolph C. Hite Director, Information Technology Architecture
and Systems Issues
Appendix I: Objectives, Scope, and Methodology
Page 40 GAO- 02- 345 Defense Health Care System
The objectives of our review were to determine (1) what progress the
Department of Defense (DOD) has made against the Composite Health Care
System (CHCS) II project commitments, (2) whether DOD has economically
justified its investment in CHCS II, and (3) whether DOD has effective
technical and management controls in place for acquiring the system.
Project commitments included in our scope of work were required system
capabilities, promised system benefits, and estimated project costs and
milestones; controls included requirements management, test management,
architecture development and alignment, risk management, and contract
management.
To determine the progress made against commitments, we reviewed relevant
project management documents, such as acquisition strategy and program
baseline documents, acquisition decision memoranda, and quarterly Defense
Acquisition Executive Summary reports, and we interviewed CHCS II project,
military health system (MHS) program, and DOD oversight officials, to
determine original and revised system capabilities, expected benefits,
estimated costs, and project milestones for both system increments and the
entire system. We then reviewed program management reports and briefings
and system documentation (e. g., cost reports, defect reports, and test
results), and interviewed project, program, and oversight officials, to
determine actual (as reported) system capabilities, costs, and schedules,
as well as accrued benefits. Comparing the two, we identified variances
and, through document review and interviews, identified the causes of the
variances. We did not independently validate the status information
obtained.
To determine whether DOD had economically justified CHCS II, we reviewed
best practices governing system investment management, including those
pertaining to incremental investment practices, and we reviewed relevant
legislative requirements and federal guidance. 37 We then analyzed the
available economic justification for the project, an April 1998 economic
analysis. 38 We compared this analysis to the best practices and federal
guidance, and discussed with project, program, and oversight officials, as
well as CHCS II contractor officials to determine how the analysis was
prepared, including the basis for cost and benefit estimates and
assumptions. We also requested updates of this analysis, and reviewed
draft updates provided to determine whether these updates were
37 Clinger- Cohen Act of 1996, Public Law 104- 106, and OMB Circular A-
130 (Nov. 30, 2000). 38 CHCS II Milestone I Economic Analysis, April 15,
1998. Appendix I: Objectives, Scope, and
Methodology
Appendix I: Objectives, Scope, and Methodology
Page 41 GAO- 02- 345 Defense Health Care System
approved. Using data from approved and updated draft analyses, we also
calculated net present values for CHCS II. In addition, we reviewed
available documentation and interviewed CHCS II project officials on its
portfolio- based investment analysis process and how this process was
applied to CHCS II.
To determine whether DOD has effective technical and management controls
in place for acquiring the system, we focused on whether best practices
were being employed in five key areas: requirements management, test
management, architecture development and alignment, risk management, and
contract management. We reviewed these areas because they are critical to
successfully acquiring systems. Among other sources, these practices were
derived from the work and research of Carnegie Mellon University*s
Software Engineering Institute (SEI), 39 as well as our own research. 40
Where appropriate, we also applied relevant legislative requirements and
federal guidance that embody these best practices. 41 We also interviewed
MHS officials, CHCS II project officials, and contractor staff from
Integic, Incorporated, the CHCS II integration contractor. More detailed
description of our scope and methodology in each control area is provided
below.
Requirements Management: To evaluate requirements management controls, we
reviewed the current CHCS II requirements management process and compared
it to best practices. We then evaluated the project*s compliance with its
established process by analyzing selected sets of requirements that were
managed under the current process. Since the current process was not in
place until 2000, we judgmentally selected one of the three requirements
sets that were added in 2000, and one of the five that were added in 2001.
Requirements documentation that we reviewed also included requirements
descriptions, cost and impact estimates, and the related 2000 and 2001
budget analysis documents. We also interviewed project and MHS officials
responsible for requirements management.
39 See, for example, Software Acquisition Capability Maturity Model SM
(CMU/ SEI- 99- TR002, April 1999). 40 See, for example, Year 2000
Computing Crisis: A Testing Guide (GAO/ AIMD- 10. 1. 21, November 1998);
Information Technology Investment Management: A Framework for Assessing
and Improving Process Maturity, Exposure Draft, GAO/ AIMD- 10- 1. 23,
version 1 (Washington, D. C.: May 2000).
41 Clinger- Cohen Act of 1996, Public Law 104- 106; OMB Circular A- 130,
(Nov. 30, 2000); A Practical Guide to Federal Enterprise Architectures,
Version 1. 0, Chief Information Officers Council (October 2000).
Appendix I: Objectives, Scope, and Methodology
Page 42 GAO- 02- 345 Defense Health Care System
Test management: To evaluate test management controls, we reviewed the
CHCS II master test plan to determine the overall test approach and scope,
including the types of testing performed and planned. Because system
acceptance testing was being planned, conducted, and completed during the
course of our review, we focused on this test event. Specifically, we
analyzed acceptance test planning documents, including test procedures,
and the actual test results, and we compared the practices defined and
followed to relevant best practices to identify whether any variances
existed. We also analyzed the scope of the test by selecting a
statistically valid sample of 59 requirements and analyzing test
procedures to determine if each requirement was addressed. This sample
size provided us with a 95 percent confidence level. Additionally, we
analyzed test results to determine whether defined system maturity
parameters were met, and if not, analyzed the variance to determine the
significance of the variance.
Architecture development and alignment: To evaluate these controls, we
first focused on the MHS enterprise architecture, determining what the
architecture framework is based on and whether the framework used is
consistent with recognized commercial frameworks. We then compared the MHS
enterprise architecture to the framework to determine if essential
architectural artifacts had been developed, and where applicable, whether
the MHS architecture was consistent with DOD- wide architecture
requirements, such as whether the technical standards profile in the MHS
architecture was consistent with the DOD Joint Technical Architecture. 42
Next, we obtained relevant CHCS II architectural documents, such as
requirements documents and design specifications, and traced selected CHCS
II architecture characteristics to the MHS architecture to determine
alignment. At the business or logical level of the architecture, we
compared 13 release 1 system capabilities (categories of requirements) to
the business areas and activities in the MHS architecture. We selected 13
of the 52 release 1 capabilities (25 percent) to ensure that we included
capabilities covering all MHS business areas relevant to CHCS II. We did
not plan to select more than the 13 capabilities unless we were unable to
trace any of the 13. At the technical level, we compared all of the CHCS
II technical standards to the MHS technical standards profile. In
conducting our analysis, we also interviewed MHS and project officials and
reviewed architecture development plans to determine how the respective
42 Department of Defense, Joint Technical Architecture, version 3.1, March
31, 2000.
Appendix I: Objectives, Scope, and Methodology
Page 43 GAO- 02- 345 Defense Health Care System
architectures were developed, who developed them, and how they ensured
alignment between the two.
Risk management: To evaluate risk management controls, we reviewed the
CHCS II risk management process and compared it with best practices. To
determine whether the process was being followed, we analyzed all priority
1 and 2 risks for release 1 and release 2, including determining whether
defined steps were performed and criteria for reporting on and closing
risks were met. Additionally, we used the results of review of CHCS II to
determine whether additional risks should be added to the existing risk
inventory. In conducting our analysis, we interviewed project officials
and reviewed relevant documentation, such as the risk management plan,
risk mitigation strategies, and risk status reports.
Contract management: In evaluating contract management controls, we
discussed with project officials their criteria and approach for defining
contract deliverables and measuring contractor performance, and whether
they had plans for changing from past practices. We also reviewed the 11
1999 to 2001 release 1 contract delivery orders with the CHCS II
integration contractor and compared them with tenets of performancebased
contracting, which are defined in federal acquisition regulations. 43
Additionally, we reviewed internal reports, such as earned value
management system summary reports, which described progress against
delivery order schedules and budgets to obtain data on contractor
performance.
We conducted our work at offices of the Military Health System in Falls
Church, Virginia, and at Integic, Incorporated, offices in Chantilly,
Virginia, from August 2001 through July 2002, in accordance with generally
accepted government auditing standards.
43 Federal Acquisition Regulation, Part 37.
Appendix II: Comparison of CHCS and CHCS II Capabilities
Page 44 GAO- 02- 345 Defense Health Care System
Capability Provided by CHCS?
Provided by CHCS II release 1?
Provided by CHCS II when all releases complete? Access to Care
Bed Availability Reporting Yes No Yes Case Management No No Yes
Enrollment/ Eligibility Yes Yes Yes Enterprise Member/ Patient Index No No
Yes Enterprisewide Registration No No Yes Enterprisewide Scheduling No No
Yes Evacuation Requests Yes No Yes Global Clinical Data Repository No No
Yes Referral Management Yes No Yes Triage and Demand Management No No Yes
Provision of Health Services
Alerts and Reminders (including Allergies and Drug Interactions) No Yes
Yes Centralized Health Record Repository No Yes Yes Clinical Decision
Support No No Yes Clinical Documentation No Yes Yes Dental Charting and
Documentation No No Yes Discharge Planning No No Yes Encounter Coding No
Yes Yes Enterprise Health Record No Yes Yes Home- based Monitoring No No
Yes Inpatient Order Entry/ Management (Laboratory, Pharmacy, Radiology)
No No Yes Operating Room Management No No Yes Optometric Documentation/
Order Entry No No Yes Outpatient Order Entry/ Management (Laboratory,
Pharmacy, Radiology)
Yes No Yes Patient Education No No Yes Patient Health History No Yes Yes
Pharmaceutical Profiling No No Yes Results Reporting (Ancillary Services)
Yes Yes Yes Role- Based Security No Yes Yes Telemedicine No No Yes Theater
Integration Related to Provision of Health Service No Yes Yes
Transcription Services Interface No Yes Yes Voice Recognition No No Yes
Population Health Management
Clinical Enterprise Member/ Patient Index No No Yes Clinical Look- back No
No Yes Clinical Outcomes Reporting No No Yes Clinical Registries No No Yes
Appendix II: Comparison of CHCS and CHCS II Capabilities
Appendix II: Comparison of CHCS and CHCS II Capabilities
Page 45 GAO- 02- 345 Defense Health Care System
Capability Provided by CHCS?
Provided by CHCS II release 1?
Provided by CHCS II when all releases complete?
Demand Material No No Yes Health Plan Employer Data and Information Set
Reporting No No Yes Health Risk Assessment No Yes Yes Health Surveillance
Monitoring/ Reporting No No Yes Immunization Tracking No Yes Yes
Individual Medical Readiness Status Reporting No No Yes Interface to VA
Mail Order Pharmacy No No Yes Occupational Health Monitoring Yes No Yes
Patient Satisfaction Reporting No Yes Yes Patient Self- assessment Data
Entry No Yes Yes Population Utilization Management No No Yes Provider
Profiling No Yes Yes Radiation Health Monitoring and Reporting Yes No Yes
Rules- based Clinical Protocols No No Yes Theater Integration Related to
Population Health Material No Yes Yes
Source: GAO based on DOD data.
Appendix III: Detailed Explanation of How CHCS II Will Operate
Page 46 GAO- 02- 345 Defense Health Care System
The initial version (release 1) of CHCS II is a medical information system
that combines existing MHS systems with commercial software to provide a
computer- based patient record (CPR) of treatment provided to DOD health-
care beneficiaries in DOD medical treatment facilities (MTF). CHCS II is
intended to allow a provider (physician, nurse, technician, etc.) to
record information gained from a patient*s visit to the MTF in the
patient*s CPR, as shown in figure 8.
Figure 8: Creating a Computer- based Patient Record
Source: GAO based on DOD data.
CHCS II*s architecture is an open system, client- server design of three
levels: the user (client) workstation at various MTF locations, the MTF
computers* (servers*) operating system and storage hardware and software,
and a clinical data repository at a remote computing center where the CPR
is stored. DOD plans to connect all workstations at an installation*s
hospital or clinics to the servers through the installation*s local or
wide area network. DOD also plans for each installation to be Appendix
III: Detailed Explanation of How
CHCS II Will Operate
Appendix III: Detailed Explanation of How CHCS II Will Operate
Page 47 GAO- 02- 345 Defense Health Care System
connected, via the Defense Information Systems Agency*s (DISA)
unclassified wide area network, 44 to the Montgomery, Alabama, computing
center, where the CPRs will be stored in a database known as the clinical
data repository (CDR). Figure 9 shows how the three CHCS II levels are
connected.
44 The network is the Unclassified but Sensitive Internet Protocol Router
Network.
Appendix III: Detailed Explanation of How CHCS II Will Operate
Page 48 GAO- 02- 345 Defense Health Care System
Figure 9: CHCS II Tri- level Architecture
Source: GAO based on DOD data.
Appendix III: Detailed Explanation of How CHCS II Will Operate
Page 49 GAO- 02- 345 Defense Health Care System
DOD plans to deploy release 1 regionally, starting with all sites in MHS
region 2 (Virginia and North Carolina). Communications between the MTFs
and the CDR go through a single access point in each region (the regional
hub). For region 2, the access point is planned for Portsmouth, Virginia,
site of the Portsmouth Naval Hospital. The Portsmouth regional access
point plan shows a primary and secondary (for redundancy and diversity)
connection to the CDR through two separate DISA computing centers. The
communication plan shows each installation in region 2 with an
installation access point connected to Portsmouth, and a backup connection
to the CDR through a DISA computing center. Figure 10 shows a typical
communications setup for release 1, in this case for region 2. The figure
uses the Ft. Bragg, North Carolina, MTF as an example of the nonregional
hub.
Appendix III: Detailed Explanation of How CHCS II Will Operate
Page 50 GAO- 02- 345 Defense Health Care System
Figure 10: Example of CHCS II Regional Communications
Source: GAO based on DOD data.
Appendix IV: CHCS II Release 1 Software Packages
Page 51 GAO- 02- 345 Defense Health Care System
Release 1 is composed of a number of commercial- off- the- shelf (COTS) 45
and government- off- the- shelf (GOTS) software packages, as shown in
table 7. The core CHCS II functionality is provided by the existing CHCS
application (nine GOTS packages), plus the two 3M software packages,
MEDCIN charting software, data dictionary, and clinical data repository
(five COTS packages). The packages are integrated by software developed by
the CHCS II contractor. In addition to the core functionality, CHCS II is
also composed of 16 other COTS packages that provide operating system,
security, data exchange, and other system functions. The CHCS II software
is located on hardware platforms at the 142 military treatment facilities
(user workstations and application, security, interface, and CHCS servers)
and the DISA computing center (clinical data repository and server).
Table 6: CHCS II Release 1 Software Packages by Location Location Product
Use Type
MEDCIN Charting Commercial Snareworks Security Commercial Adobe Acrobat
Reader Form viewer Freeware Dimension 4 Time service Freeware Windows 2000
Operating system Commercial Problem Knowledge Couplers Education
Commercial McAfee Netshield Virus protection Commercial Client Workstation
PARS II Data Collector Data collection Government Problem Knowledge
Coupler Education Commercial Adobe Acrobat Reader Form viewer Freeware
McAfee Netshield Virus protection Commercial Administrator Workstation
3M Clinical Workstation User interface Commercial 3M Software Data
dictionary, repository Commercial SNOMED Data dictionary Commercial
Netscape Unix User registration Commercial HP Distributed computing
Security Commercial Snareworks Security Commercial Oracle Database
Commercial Clinical Data Repository / Enterprise
Security Server Tuxedo On- line transaction monitor Commercial
45 Some of the COTS packages are described as *freeware,* which can be
used without charge, or *shareware* which, if you use it regularly, you*re
required to register and pay for. Appendix IV: CHCS II Release 1 Software
Packages
Appendix IV: CHCS II Release 1 Software Packages
Page 52 GAO- 02- 345 Defense Health Care System
Location Product Use Type
Enosis Connection Manager Software management Commercial Tardis Time
service Shareware Windows NT 4 Operating system Commercial Gradient
Middleware Commercial Military Treatment Facility Security
Server Snareworks Security Commercial Active X Operating system component
Government Enosis Connection Manager Data exchange Commercial Snareworks
Security Commercial Tardis Time Service Shareware Oracle Database
Commercial Military Treatment Facility Application
Server Business Objects Reports Government Interface Engine Server
Datagate Operating system;
Interface: 3M to CHCS II Commercial CHCS Server CHCS CHCS application
Government
Fileman CHCS application Government GIS Interface software Government HL 7
Message software Government OV MVS Operating system Commercial MUMPS
Programming language Commercial M Adapter Data exchange Government M
Functions Data exchange Government
Source: GAO based on DOD data.
Appendix V: CHCS II Capabilities Planned for Releases 3- 7, and Expected
Release Dates
Page 53 GAO- 02- 345 Defense Health Care System
Release number and expected release date Capabilities
3 July 2004
Enterprise member/ patient index Government computerized patient record
(DOD* Department of Veterans Affairs clinical information exchange) Health
risk assessment Health surveillance monitoring and reporting Individual
medical readiness status reporting New eligibility system interface
Patient self- assessment data entry (additional functionality) Referral
management Replacement ancillary system* pharmacy Rules- based clinical
protocols Theater integration 4 July 2004
Bed availability reports Cost accounting and patient accounting interfaces
Dental imaging Enterprisewide registration Enterprisewide scheduling
(includes operating room) Evacuation requests Operating room management
Outpatient order entry and management Pharmaceutical profiling Replacement
laboratory, pathology, radiology systems Theater integration Triage and
demand management 5 July 2005
Assurance/ safety Case management Inpatient order entry and management
Patient education Patient safety reporting Population utilization
management and quality Theater integration Voice recognition 6 July 2005
Clinical look- back Clinical outcomes reporting Discharge planning
Enterprise health record Health plan employer data reporting Provider
database interface Theater integration 7 July 2006
Home- based monitoring Nonradiation and nonlaboratory ancillary procedures
interface Occupational health monitoring Radiation health monitoring and
reporting Telemedicine Theater integration
Source: GAO based on DOD data.
Appendix V: CHCS II Capabilities Planned for Releases 3- 7, and Expected
Release Dates
Appendix VI: GAO Contact and Staff Acknowledgments
Page 54 GAO- 02- 345 Defense Health Care System
Carl L. Higginbotham, (404) 679- 1824 In addition to the individual named
above, key contributors to this report included Nabajyoti Barkakati,
Harold J. Brumm Jr., Katherine I. ChuHickman, Michael P. Fruitman, Chetna
Lal, and Keith E. Steck. Appendix VI: GAO Contact and Staff
Acknowledgments GAO Contact Acknowledgments
(310217)
The General Accounting Office, the investigative arm of Congress, exists
to support Congress in meeting its constitutional responsibilities and to
help improve the performance and accountability of the federal government
for the American people. GAO examines the use of public funds; evaluates
federal programs and policies; and provides analyses, recommendations, and
other assistance to help Congress make informed oversight, policy, and
funding decisions. GAO*s commitment to good government is reflected in its
core values of accountability, integrity, and reliability.
The fastest and easiest way to obtain copies of GAO documents at no cost
is through the Internet. GAO*s Web site (www. gao. gov) contains abstracts
and fulltext files of current reports and testimony and an expanding
archive of older products. The Web site features a search engine to help
you locate documents using key words and phrases. You can print these
documents in their entirety, including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and
correspondence. GAO posts this list, known as *Today*s Reports,* on its
Web site daily. The list contains links to the full- text document files.
To have GAO e- mail this list to you every afternoon, go to www. gao. gov
and select *Subscribe to daily E- mail alert for newly released products*
under the GAO Reports heading.
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U. S. General Accounting Office 441 G Street NW, Room LM Washington, D. C.
20548
To order by Phone: Voice: (202) 512- 6000 TDD: (202) 512- 2537 Fax: (202)
512- 6061
Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm E- mail:
fraudnet@ gao. gov Automated answering system: (800) 424- 5454 or (202)
512- 7470
Jeff Nelligan, managing director, NelliganJ@ gao. gov (202) 512- 4800 U.
S. General Accounting Office, 441 G Street NW, Room 7149 Washington, D. C.
20548 GAO*s Mission
Obtaining Copies of GAO Reports and Testimony
Order by Mail or Phone To Report Fraud, Waste, and Abuse in Federal
Programs
Public Affairs
*** End of document. ***