Military Personnel: The Navy Has Not Provided Adequate
Justification For Its Decision to Invest in MCTFS (25-JUL-07,
GAO-07-1139R).
In June 2006, the Navy completed a feasibility assessment of both
the Marine Corps Total Force System (MCTFS)--the Marine Corps'
integrated personnel and payroll system--and the Defense
Integrated Military Human Resources System DIMHRS. Based on the
results of this assessment, the Navy expressed a preference to
deploy MCTFS instead of DIMHRS. In August 2006, the Defense
Business Systems Management Committee (DBSMC) made a business
decision to accept the Navy's plan to proceed with the
development and implementation of MCTFS. However, the Department
of Defense (DOD) had already committed to deploy DIMHRS to
provide a joint, integrated, standardized military personnel and
pay system across all military components. According to the
department, as of September 2006, it had spent over $668 million
on the program. The John Warner National Defense Authorization
Act for Fiscal Year 2007, Pub. L. No. 109-364, 324 (2006)
directed the Secretary of the Navy to prepare a report about
MCTFS, including (1) an analysis of alternatives to MCTFS,
including a comparison between the costs of deploying and
operating MCTFS within the Navy and the cost of including the
Navy in DIMHRS; (2) a business case analysis of the costs and
benefits to both the Navy and DOD of the alternatives to MCTFS
considered in the first objective; and (3) an analysis of the
compatibility of MCTFS with the department's business enterprise
architecture. The Navy concluded in its report--which we received
on April 25, 2007--that (1) either MCTFS or DIMHRS could provide
basic personnel and pay capability for the Navy uniformed force
at approximately equivalent cost; (2) the DIMHRS alternative has
substantially higher risks on cost, schedule, and function
because MCTFS is already operational; and (3) MCTFS is fully
compatible and compliant with the department's business
enterprise architecture. The Act directs the DBSMC to
determine--not sooner than 120 days after April 25, 2007--whether
the deployment of MCTFS by the Navy is in the best interest of
DOD. The Act directed us to review the Navy's report and provide
a written assessment to the congressional defense committees and
the Chairman of the DBSMC within 90 days of our receipt of the
Navy's report. Specifically, we determined whether the Navy in
its congressionally mandated report has justified its decision to
invest in MCTFS. As part of our review, we also determined
whether the department's business enterprise architecture was
sufficient to guide and constrain the acquisition of MCTFS.
-------------------------Indexing Terms-------------------------
REPORTNUM: GAO-07-1139R
ACCNO: A73488
TITLE: Military Personnel: The Navy Has Not Provided Adequate
Justification For Its Decision to Invest in MCTFS
DATE: 07/25/2007
SUBJECT: Comparative analysis
Cost analysis
Enterprise architecture
Military forces
Naval procurement
Payroll systems
Program evaluation
Systems conversions
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO Product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
******************************************************************
GAO-07-1139R
* [1]PDF6-Ordering Information.pdf
* [2]Order by Mail or Phone
July 25, 2007
Congressional Committees
Subject: Military Personnel: The Navy Has Not Provided Adequate
Justification For Its Decision to Invest in MCTFS
In June 2006, the Navy completed a feasibility assessment of both the
Marine Corps Total Force System (MCTFS)--the Marine Corps' integrated
personnel and payroll system--and the Defense Integrated Military Human
Resources System DIMHRS. Based on the results of this assessment, the Navy
expressed a preference to deploy MCTFS instead of DIMHRS. In August 2006,
the Defense Business Systems Management Committee (DBSMC)^1 made a
business decision to accept the Navy's plan to proceed with the
development and implementation of MCTFS. However, the Department of
Defense (DOD) had already committed to deploy DIMHRS to provide a joint,
integrated, standardized military personnel and pay system across all
military components. According to the department, as of September 2006, it
had spent over $668 million^2 on the program.
The John Warner National Defense Authorization Act for Fiscal Year 2007,
Pub. L. No. 109-364, S324 (2006) directed the Secretary of the Navy to
prepare a report about MCTFS, including (1) an analysis of alternatives to
MCTFS, including a comparison between the costs of deploying and operating
MCTFS within the Navy and the cost of including the Navy in DIMHRS; (2) a
business case analysis of the costs and benefits to both the Navy and DOD
of the alternatives to MCTFS considered in the first objective; and (3) an
analysis of the compatibility of MCTFS with the department's business
enterprise architecture.^3 The Navy concluded in its report--which we
received on April 25, 2007--that (1) either MCTFS or DIMHRS could provide
basic personnel and pay capability for the Navy uniformed force at
approximately equivalent cost; (2) the DIMHRS alternative has
substantially higher risks on cost, schedule, and function because MCTFS
is already operational; and (3) MCTFS is fully compatible and compliant
with the department's business enterprise architecture. The Act directs
the DBSMC to determine--not sooner than 120 days after April 25,
2007--whether the deployment of MCTFS by the Navy is in the best interest
of DOD.
^1The Deputy Secretary of Defense established the DBSMC, which oversees
DOD's business transformation, and tasked it with responsibility for
overall departmental transformation pursuant to 10 U.S.C. S 186.
^2The cost figure includes costs incurred by the Enterprise Program
Management Office-DIMHRS, the Army, the Air Force, the U. S. Marine Corps,
and the Defense Finance and Accounting Office; it does not include costs
incurred by the Navy since program inception.
^3An enterprise architecture provides a clear and comprehensive picture of
an entity, whether it is an organization (e.g., a federal department) or a
functional or mission area that cuts across more than one organization
(e.g., human resources). It is an essential tool for effectively and
efficiently engineering business processes and for implementing and
evolving systems that support the business processes.
The Act directed us to review the Navy's report and provide a written
assessment to the congressional defense committees and the Chairman of the
DBSMC within 90 days of our receipt of the Navy's report. Specifically, we
determined whether the Navy in its congressionally mandated report has
justified its decision to invest in MCTFS. As part of our review, we also
determined whether the department's business enterprise architecture was
sufficient to guide and constrain the acquisition of MCTFS.
Scope and Methodology
To perform our assessment, we reviewed the Navy report,^4 which summarized
the results of the analysis of alternatives, the business case analysis of
the costs and benefits of the alternatives, and the analysis of the
compatibility of MCTFS with the department's enterprise architecture.
Using our Cost Assessment Guide and relevant Office of Management and
Budget (OMB) criteria,^5 we evaluated the analysis of alternatives and the
business case analysis. According to these criteria, the decision to
invest in any system should be based on reliable analyses of estimated
costs and expected benefits over the life of the system. Best practices
state that agencies should prepare a business case analysis--a comparative
analysis that presents facts and supporting details about competing
alternatives (i.e., analysis of alternatives). A business case analysis
should consider all life-cycle costs and nonquantifiable as well as
quantifiable benefits. It should consider all possible alternatives and
should not be developed solely to support a predetermined solution. A
business case analysis should be rigorous enough that independent auditors
can review it and clearly understand why a particular alternative was
chosen. We evaluated the estimates that the Navy used in the analysis of
alternatives and the business case analysis to determine whether these
estimates were comprehensive, accurate, well-documented, and credible. Our
Cost Assessment Guide considers an estimate to be: (1) `comprehensive' if
it includes all relevant cost elements; (2) `accurate' if it is unbiased
and neither overly conservative nor overly optimistic and is based on an
assessment of the costs most likely to be incurred; (3) `well-documented'
if the it can be traced back to the documentation, which should include
the source of the data, detailed calculations and results, and
explanations of why a particular method or reference was chosen; and (4)
`credible' if limitations of the data and assumptions are documented.
Using applicable guidance,^6 we evaluated the efforts taken by the
Navy to show the extent to which MCTFS is compliant and compatible
with the architecture. This guidance states that agencies should
develop and use enterprise architectures to guide and constrain
their investment decisions. We obtained and analyzed the documents
provided--such as MCTFS architecture products, the results of the
Navy's analysis of MCTFS using the Architecture Compliance and
Requirements Traceability Tool, and a Gartner, Inc. report^7--the
technology standards, and version 4.0 of DOD's business enterprise
architecture.
^4 Department of the Navy, Report from the Secretary of the Navy to the
Congressional Defense Committees and the Comptroller General on the Marine
Corps Total Force System (MCTFS), April 2007.
^5 See GAO, Cost Assessment Guide: Best Practices for Developing and
Managing Program Costs, OMB Circular A-11, and OMB Circular A-94.
^6 Office of Management and Budget (OMB) Capital Programming Guide,
Version 2.0 ; [3]GAO-03-584G ; Department of Defense, DOD IT Business
Systems Investment Review Process, Business Enterprise Architecture (BEA)
Compliance Guidance; and Department of Defense, Department of Defense
Architecture Framework, Version 1.5, Volume 1.
We interviewed officials to discuss the results of our work and to
obtain explanations for the discrepancies we identified.
Specifically, we interviewed officials from the following:
o Several organizations within the Department of the
Navy, including the Office of the Chief of Naval
Operations; the Assistant Secretary of the Navy
Financial Management and Comptroller; the Manpower
and Reserve Affairs, U. S. Marine Corps; and the
Naval Center for Cost Analysis. These officials were
responsible for developing the analysis of
alternatives, the business case analysis, and the
analysis of the compatibility of MCTFS with the
department's business enterprise architecture.
o The Enterprise Program Management Office-DIMHRS,
which is responsible for acquiring and implementing
DIMHRS, including the DIMHRS Enterprise Deputy
Program Manager.
o The Personnel and Readiness Information Management
Office, which is responsible for developing the
DIMHRS architecture, including staff supporting the
investment management process.
o The Business Transformation Agency, which is
responsible for maintaining and updating the
department's business enterprise architecture,
including the chief architect for the Business
Mission Area.
Our work was conducted from April through July 2007 in accordance with
generally accepted government auditing standards.
Summary
In its report, the Navy has not adequately justified its decision to
invest in MCTFS. Specifically, the analysis of alternatives and the
business case analysis of both MCTFS and DIMHRS were unreliable, because
the cost and benefit estimates used in these analyses were not
sufficiently comprehensive, accurate, documented, or credible. The Navy's
analysis shows that MCTFS is sufficiently compatible with DOD's evolving
business enterprise architecture to allow the Navy to begin acquisition.^8
However, we found that the architecture lacks sufficient content to be
used effectively to guide and constrain the acquisition of MCTFS. DOD
officials from the Navy, the Marine Corps, and the Business Transformation
Agency^9 provided oral comments on the facts presented in this report and
we have incorporated these comments as appropriate.
^7 Gartner, Inc., A Report for U. S. Marine Corps and Defense Finance and
Accounting Service: Marine Corps Total Force System Program Evaluation
(June 19, 2006).
^8Our analysis determined the extent to which MCTFS was compatible with
version 4.0 of the department's business enterprise architecture.
^9The Business Transformation Agency is responsible for leading and
coordinating business transformation efforts across the department.
Analysis of Alternatives and Business Case Analysis Were Unreliable
The Navy's report included an analysis of alternatives and a business case
analysis of MCTFS and DIMHRS; however, these analyses were unreliable
because the cost and benefit estimates used for these analyses were not
sufficiently comprehensive, accurate, documented, or credible. For
example, the cost estimates were not comprehensive, because they did not
include all possible alternatives. For example, the Navy's report included
a third alternative that involves deploying MCTFS initially and then
deploying DIMHRS after the Army and Air Force have successfully
implemented it. However, the analysis of alternatives and the business
case analysis did not include cost estimates for this alternative. In
March 2007, DBSMC tasked the Navy to prepare the cost and benefit
estimates for this alternative, after which Navy officials said that they
discussed it with the Secretary of the Navy. These estimates were based on
those that had been used for both the MCTFS and DIMHRS alternatives and we
had concluded that these estimates were unreliable. The cost estimates
were not accurate, because the Navy excluded the third alternative, which
was more costly than the MCTFS and DIMHRS alternatives. This exclusion
skewed the results of the analyses by showing that the MCTFS alternative
was the least costly solution. The cost estimates were not
well-documented, because source data was not provided and the calculations
performed for several cost elements were not explained. The cost estimates
were not credible, because there were neither cross-checks of these
estimates nor a current independent cost estimate; both of these are
necessary in order to determine whether a different estimating method
would have produced similar results.
Moreover, the analyses based on these cost estimates did not take into
account the following risks: (1) the potential impact of moving MCTFS
development operations from Kansas City to Indianapolis as part of the
Base Realignment and Closure 2005 actions and (2) the potential impact of
the contractor discontinuing its support of DIMHRS after 2013, as planned.
A Navy official responsible for MCTFS development operations stated his
intent to delay the closing of the Kansas City site as long as possible in
order to mitigate its impact on MCTFS. However, this official did not
provide documentation of his authority to carry out this intent or of the
Navy's plans for doing so within the time limitations established for the
Base Realignment and Closure 2005 actions. After the Navy issued its
report, the contractor changed its policy and stated its intent to support
DIMHRS indefinitely; however, this support will be limited. For example,
`sustaining support' does not include key services, such as new updates,
product fixes, security alerts, critical patch and regulatory updates, and
certification of DIMHRS with new products developed by the contractor. It
is unclear what the costs, if any, will be for this sustaining support.
However, the contractor's change in policy is not a binding legal
agreement; the contractor could later decide to discontinue its support of
DIMHRS or increase the cost prohibitively for sustaining support. Since
the accuracy of the cost estimates is in question, the benefits or cost
savings to be realized are also in question.
According to best practices,^10 the decision to invest in any system
should be based on reliable analyses of estimated costs and expected
benefits over the life of the system. Best practices state that agencies
should prepare a business case analysis--a comparative analysis that
presents facts and supporting details about competing alternatives. A
business case analysis considers all life-cycle costs and quantifiable and
nonquantifiable benefits. It should consider all possible alternatives and
should not be developed solely to support a predetermined solution. A
business case analysis should be rigorous enough that independent auditors
can review it and clearly understand why a particular alternative was
chosen.
Navy officials said that the report did not include the strategic
alternative showing the Navy's intent to ultimately transition to DIMHRS
because the mandate did not require them to do so. However, best practices
require that an agency consider all possible alternatives when preparing a
business case analysis, so this alternative should have been included in
the report. In November 2006, the Navy had tasked the Naval Center for
Cost Analysis to develop cost estimates for the business case analysis to
be included in the report by February 2007. Naval Center for Cost Analysis
officials said that this deadline give them adequate time to meet best
practices for developing comprehensive, accurate, well-documented, and
credible cost estimates. These officials stated, and we agree, that it
typically takes 6 months to develop reliable cost estimates. DIMHRS
program officials stated that the cost and benefit estimates the Navy used
in its analyses were outdated and did not reflect the current DIMHRS
program. They also stated that they had not been contacted by the Naval
Center for Cost Analysis to provide updated information for the Navy
report. However, officials from the Naval Center for Cost Analysis
provided documentation showing that they had in fact contacted DIMHRS
program officials and that these officials had refused to answer
programmatic and technical questions about the DIMHRS program. Because the
analyses were unreliable, Congress and the DBSMC cannot be assured that
the most cost-effective solution will be selected.
MCTFS Is Compatible with the Business Enterprise Architecture, but the
Architecture Is Insufficient to Guide Investment Decisions
The Navy's analysis of the compatibility of MCTFS with the department's
business enterprise architecture was sufficient for the Navy to begin
acquisition of the system. However, we found that the architecture lacks
sufficient content to be used effectively to guide and constrain
investments for systems such as MCTFS. As a result of this lack of
architectural constraint on systems, the Navy determined the architecture
requirements MCTFS will need to meet to fit within the department's vision
as outlined in the evolving architecture. According to best practices and
DOD guidance, ^11 agencies should develop and use enterprise architectures
to guide and constrain investment decisions. Senior DOD and Navy officials
acknowledged that the architecture cannot be used at this time to
effectively guide and constrain component-level system investments such as
MCTFS. As a result, there is a risk that the department will not be able
to achieve its vision to deploy standard systems such as DIMHRS. By
deploying standard systems, DOD may be able to address the weaknesses in
its current system environment, which is comprised of numerous systems
that are duplicative in functionality, are not interoperable, are
unnecessarily costly to maintain and interface, and do not optimize
mission performance and accountability.
^10GAO, Cost Assessment Guide: Best Practices for Developing and Managing
Program Costs, July 2007; Office of Management and Budget (OMB) Circular
A-11, Preparation, Submission, and Execution of the Budget, Part 7:
Planning, Budgeting, Acquisition, and Management of Capital Assets (June
2006); and OMB Circular A-94, Guidelines and Discount Rates for
Benefit-Cost Analysis of Federal Programs (Revised Oct. 29, 1992).
^11Office of Management and Budget (OMB) Capital Programming Guide,
Version 2.0 (June 2006); GAO, Information Technology: A Framework for
Assessing and Improving Enterprise Architecture Management (Version 1.1),
[4]GAO-03-584G (Washington, D.C.: April 2003); Department of Defense, DOD
IT Business Systems Investment Review Process, Business Enterprise
Architecture (BEA) Compliance Guidance, Version 1.0 (Apr. 10, 2006); and
Department of Defense, Department of Defense Architecture Framework,
Version 1.5, Volume 1 (Apr. 23, 2007).
Conclusion
Our assessment of the Navy's report shows that the analyses are based on
flawed estimates and an insufficient architecture. If the DBSMC relies
solely on the Navy's report in deciding whether or not to deploy MCTFS, it
risks recommending an investment decision that would not be in the best
interest of DOD. As it deliberates, the DBSMC should consider the issues
we have identified.
________________________________________________________________________
We are sending copies of this report to selected congressional committees
and the Chairman of the Defense Business Systems Management Committee. We
will also make copies available to others upon request. In addition, the
report will be available at no charge on GAO's Web site at
[5]http://www.gao.gov .
If your offices or staffs have any questions concerning this report,
please contact me at (202) 512-3604 or by e-mail at [email protected].
Contact points for our Offices of Congressional Relations and Public
Affairs may be found on the last page of this report. GAO staff who made
major contributions to the report are listed in enclosure I.
Brenda S. Farrell
Director, Defense Capabilities and Management
Enclosure - 1
List of Committees
The Honorable Daniel Inouye
Chairman
The Honorable Ted Stevens
Ranking Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Carl Levin
Chairman
The Honorable John McCain
Ranking Member
Committee on Armed Services
United States Senate
The Honorable John P. Murtha
Chairman
The Honorable C.W. Bill Young
Ranking Member
Subcommittee on Defense
Committee on Appropriations
House of Representatives
The Honorable Ike Skelton
Chairman
The Honorable Duncan Hunter
Ranking Member
Committee on Armed Services
House of Representatives
Enclosure I GAO Contact and Staff Acknowledgments
GAO Contact
Brenda S. Farrell (202) 512-3604 or [email protected]
Acknowledgments
In addition to the contact above, Cynthia Jackson, Assistant Director;
Brian Bothwell; Harold J. Brumm, Jr.; Jennifer K. Echard; Joanne
Landesman; Sonya Phillips; Karen A. Richey; Matthew S. Spiers; Randolph
Tekeley; Joseph J. Watkins; and Angela D. Watson made key contributions to
this report.
(351049)
GAO's Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of GAO Reports and Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site ( [6]www.gao.gov ). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
[7]www.gao.gov and select "Subscribe to Updates."
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2
each. A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
copies mailed to a single address are discounted 25 percent. Orders should
be sent to:
U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud, Waste, and Abuse in Federal Programs
Contact:
Web site: [8]www.gao.gov/fraudnet/fraudnet.htm
E-mail: [9][email protected]
Automated answering system: (800) 424-5454 or (202) 512-7470
Congressional Relations
Gloria Jarmon, Managing Director, [10][email protected] (202) 512-4400 U.S.
Government Accountability Office, 441 G Street NW, Room 7125 Washington,
D.C. 20548
Public Affairs
Paul Anderson, Managing Director, [11][email protected] (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
This is a work of the U.S. government and is not subject to copyright
protection in the United States. The published product may be reproduced
and distributed in its entirety without further permission from GAO.
However, because this work may contain copyrighted images or other
material, permission from the copyright holder may be necessary if you
wish to reproduce this material separately.
References
Visible links
3. http://www.gao.gov/cgi-bin/getrpt?GAO-03-584g
4. http://www.gao.gov/cgi-bin/getrpt?GAO-03-584g
5. http://www.gao.gov/
6. http://www.gao.gov/
7. http://www.gao.gov/
8. http://www.gao.gov/fraudnet/fraudnet.htm
9. mailto:[email protected]
10. mailto:[email protected]
11. mailto:[email protected]
*** End of document. ***