Comments on the Office of Personnel Management's February 20,	 
2008 Report to Congress Regarding the Retirement Systems	 
Modernization (28-MAR-08, GAO-08-576R). 			 
                                                                 
The Office of Personnel Management (OPM) is modernizing the	 
paper-intensive processes and antiquated information systems it  
uses to support the retirement of civilian federal employees	 
through the Retirement Systems Modernization (RSM) program. In	 
January 2008, we reported on the agency's management of this	 
program, in which we noted concerns and made recommendations for 
improvement in four key areas: (1) system testing, (2) system	 
defect resolution, (3) program cost estimation, and (4) program  
earned value management. The explanatory statement of the House  
Appropriations Committee regarding the fiscal year 2008 	 
Consolidated Appropriations Act directed OPM to submit to	 
Cpngressional Committees and to GAO not later than February 20,  
2008, a report of its actions on the four areas of concern that  
we identified. Further, the explanatory statement directed that  
GAO provide to Congressional Committees and to OPM our comments  
on the agency's report. 					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-576R					        
    ACCNO:   A81441						        
  TITLE:     Comments on the Office of Personnel Management's February
20, 2008 Report to Congress Regarding the Retirement Systems	 
Modernization							 
     DATE:   03/28/2008 
  SUBJECT:   Civil service retirement system			 
	     Cost analysis					 
	     Cost control					 
	     Federal employee retirement programs		 
	     Federal employees retirement system		 
	     Program evaluation 				 
	     Program management 				 
	     Reporting requirements				 
	     System vulnerabilities				 
	     Systems analysis					 
	     Systems conversions				 
	     Systems evaluation 				 
	     Systems integrity					 
	     Systems testing					 
	     Technology modernization programs			 
	     Cost estimates					 
	     Program costs					 
	     OPM Retirement Systems Modernization		 
	     Program						 
                                                                 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-576R

This is the accessible text file for GAO report number GAO-08-576R 
entitled 'Comments on the Office of Personnel Management's February 20, 
2008 Report to Congress Regarding the Retirement Systems 
Modernization,' which was released on March 28, 2008.

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to [email protected]. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

GAO-08-576R: 

United States Government Accountability Office: 
Washington, DC 20548: 

March 28, 2008: 

The Honorable Richard Durbin:
Chairman:
The Honorable Sam Brownback:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
United States Senate: 

The Honorable Josï¿½ E. Serrano:
Chairman:
The Honorable Ralph Regula:
Ranking Member:
Subcommittee on Financial Services and General Government:
Committee on Appropriations:
House of Representatives: 

Subject: Comments on the Office of Personnel Management's February 20, 
2008 Report to Congress Regarding the Retirement Systems Modernization: 

The Office of Personnel Management (OPM) is modernizing the paper-
intensive processes and antiquated information systems it uses to 
support the retirement of civilian federal employees through the 
Retirement Systems Modernization (RSM) program. In January 2008, we 
reported on the agency's management of this program, in which we noted 
concerns and made recommendations for improvement in four key areas: 
(1) system testing, (2) system defect resolution, (3) program cost 
estimation, and (4) program earned value management.[Footnote 1] 

The explanatory statement of the House Appropriations Committee 
regarding the fiscal year 2008 Consolidated Appropriations Act directed 
OPM to submit to your Committees and to GAO not later than February 20, 
2008, a report of its actions on the four areas of concern that we 
identified.[Footnote 2] Further, the explanatory statement directed 
that GAO provide to your Committees and to OPM our comments on the 
agency's report. In response to this direction, our comments (including 
a summarization of our report findings and recommendations and OPM's 
reported actions for each area of concern) follow. 

System Testing: 

GAO Finding and Recommendation: Effective testing is an essential 
component of any program that includes system development. Generally, 
the purpose of testing is to identify defects or problems in meeting 
defined system requirements or satisfying system user needs[Footnote 
3]. Our study determined that OPM's initial RSM system test results did 
not provide assurance that a major system component--the Defined 
Benefits Technology Solution (DBTS)--would perform as intended, and 
that the agency's compressed and concurrent testing schedule increased 
the risk that it would not have sufficient resources or time to verify 
that the system will work as intended. Accordingly, we recommended that 
OPM ensure that sufficient resources are provided to fully test 
functionality, actions for mitigating the risks inherent in concurrent 
testing are identified, test results verify that all system components 
perform as expected, and test activities and results are subjected to 
independent verification and validation. 

OPM Report to Congress: OPM's report summarized the results of certain 
user acceptance tests (UAT) that the agency has conducted as part of 
verifying that DBTS meets requirements such as accurately calculating 
retirement benefits. The agency reported test results as shown in table 
1. 

Table 1: OPM Reported Test Results in Terms of Scenarios: 

Scenarios Tested: 
Scenarios Passed: 
Percent Passed: 

Test: UAT 5; 
Scenarios Tested: 192; 
Scenarios Passed: 142; 
Percent Passed: 74. 

Test: UAT 6; 
"Discontinued and reconstituted as UAT 6.5 to limit focus to 
high frequency functionality for Go Live". 

Test: UAT 6.5; 
Scenarios Tested: 57; 
Scenarios Passed: 51; 
Percent Passed: 89. 

Source: OPM. 

[End of table] 

GAO Comments: OPM limited its discussion of system testing to the UAT 
results. In this regard, the results shown for UAT 5, indicating that 
74 percent of scenarios had passed testing, falls short of the agency's 
stated goal of 95 percent of scenarios passed. In addition, the 
statement that UAT 6 was "discontinued and reconstituted as UAT 6.5 to 
limit focus to high frequency functionality for Go Live" indicates that 
the scope of UAT 6.5 was reduced from the scope planned for UAT 6. 

Further, OPM's report did not address the results of other critical 
tests that the agency had planned to conduct starting in December 2007 
and ending in February 2008 as illustrated in figure 1. 

Figure 1: RSM Test Schedule: 

[See PDF for image] 

This figure is a timeline of the RSM test schedule. The following items 
are displayed on the timeline: 

Test: UAT 1: 
Date: Mid-March 2007. 

Test: UAT 2 part 1; 
Date: Late May 2007. 

Test: UAT 2 part 2; 
Date: Early June 2007. 

Test: UAT 3 part 1; 
Date: Early August 2007. 

Test: UAT 3 part 2; 
Date: Mid-September 2007. 

Test: UAT 4; 
Date: Mid-October 2007. 

Test: UAT 5; 
Date: Mid-December 2007; 

Test: Parallel test; 
Date: Mid-to-late December 2007. 

Test: UAT 6; 
Date: Mid-January 2008. 

Test: Parallel test; 
Date: Mid-January to Mid-February 2008. 

Test: Integrated product test; 
Date: Late January to Mid-February 2008. 

Test: Performance test; 
Date: Late January to Mid-February 2008. 

Test: Business capacity release test; 
Date: February 2008. 

Increment 1 deployment:
Date: End of February 2008. 

Source: GAO, based on OPM data. 

[End of figure] 

Specifically, the report did not discuss the results of the following 
tests: 

* Parallel test to verify that the new system produces the same results 
as existing systems. 

* Integrated product test to confirm that system components meet 
functional requirements (e.g., accurately calculate benefits). 

* Performance test to confirm that the new system meets performance 
requirements (e.g., processing volume and execution time). 

* Business capability test to confirm the operational readiness of the 
new system for end users. 

As our report noted, these tests are intended to verify that DBTS and 
other system components work together as expected when they are 
combined and that the complete system resulting from the RSM program 
satisfies all requirements (e.g., functional and performance) and is 
acceptable to end users. 

Finally, the report did not address the conduct of independent 
verification and validation of test activities and results. The purpose 
of independent verification and validation is to provide an independent 
review of system processes and products to ensure that quality 
standards are being met. 

System Defect Resolution: 

GAO Finding and Recommendation: Defects are system problems that 
require a resolution and can be due to a failure to meet the system 
specifications. Defects are often identified prior to and during system 
tests. As we have previously reported, having current and accurate 
defect information is necessary to adequately understand system 
maturity and to make informed decisions about how to best allocate 
limited resources to meet competing priorities.[Footnote 4] Our study 
of RSM determined that trends in identifying and resolving system 
defects indicated a growing backlog of problems to be resolved prior to 
deployment of the new system. These include urgent and high priority 
defects, which by OPM's definition are defects that either prevent 
progress or defects that need to be addressed in the current system 
phase prior to deployment. As a result, we recommended that the agency 
monitor and review DBTS defects to ensure that all urgent and high 
priority defects are resolved prior to system deployment and that the 
resolution of urgent and high priority defects is subjected to 
independent verification and validation. 

OPM Report to Congress: OPM's report summarized system defect 
resolution as shown in table 2. 

Table 2: OPM Reported Defects Status: 

Defect Severity Priority: Urgent; 
Number of defects, October 2007: 9; 
Number of defects, January 2008: 14; 
Projected number of defects at February 2008 deployment: 1. 

Defect Severity Priority: High; 
Number of defects, October 2007: 129; 
Number of defects, January 2008: 70; 
Projected number of defects at February 2008 deployment: 12. 

Defect Severity Priority: Medium/low; 
Number of defects, October 2007: 229; 
Number of defects, January 2008: 85; 
Projected number of defects at February 2008 deployment: 2. 

Defect Severity Priority: Total; 
Number of defects, October 2007: 367; 
Number of defects, January 2008: 168; 
Projected number of defects at February 2008 deployment: 15. 

Source: OPM. 

[End of table] 

In its report, OPM stated that the decreasing pattern of defects was 
consistent with the agency's expectation for the identification and 
resolution of defects. The report also stated that the one urgent 
priority defect that was projected to remain at the time of the 
February 2008 deployment was related to functionality referred to as 
"unpaid re-deposit" and is expected to be infrequent and easily 
identifiable. 

GAO Comments: With the exception of an increase in urgent priority 
defects from 9 to 14, OPM reported progress toward resolving defects 
between October 2007 and January 2008. However, the agency did not 
report actual defect resolution data for February 2008; instead it 
reported the projected defects it expected to remain at the time of 
system deployment, scheduled for February 25, 2008. If this projection 
was realized, it would represent continued defect resolution progress. 
Nevertheless, OPM's plan to deploy its system with a total of 13 
unresolved urgent and high priority system defects increased risk and 
contradicted earlier assertions that the agency would resolve urgent 
and high priority defects prior to deployment. In addition, the report 
did not address conducting independent verification and validation of 
system defect resolution results. 

Cost Estimating: 

GAO Finding and Recommendation: A cost estimate is the summation of 
individual program cost elements, using established methods and valid 
data to estimate future costs. Credible cost estimates are produced by 
following rigorous steps and are accompanied by detailed documentation, 
including descriptions of the system under development, estimation 
methodology, ground rules and assumptions, and sensitivity and 
uncertainty analyses.[Footnote 5] Our study determined that the 
reliability of OPM's $421.6 million RSM life-cycle cost estimate was 
questionable because the agency could not support the estimate with a 
description of the system to be developed and a description of the 
methodology used to produce the estimate. Thus, we recommended that the 
agency develop a revised RSM cost estimate that addresses the 
weaknesses identified and task an independent verification and 
validation contractor with reviewing the process used to develop the 
estimate and assessing the reliability of the resulting estimate. 

OPM Report to Congress: Relative to this area of concern, OPM reported 
that 86 percent of program costs are associated with fixed price 
contracts and that remaining costs are predominantly for OPM staff and 
project management activities. Further, the agency asserted that most 
future program costs are known and do not require cost estimating. 
Additionally, the agency provided information on factors that 
contributed to an increase in the RSM life-cycle cost estimate from 
$371.2 million to $421.6 million. 

GAO Comments: The report did not provide new information or describe 
the progress the agency has made to address the weaknesses in the RSM 
cost estimate. Specifically, the agency did not provide documentation 
about the system under development (i.e., a technical baseline 
description), a cost estimating methodology, estimating ground rules 
and assumptions, and sensitivity and uncertainty analyses. Without such 
documentation, an assessment of the reliability of OPM's estimate 
cannot be made. Further, the extent to which OPM's estimate reflects 
changes to the program, including delayed deployment of certain 
functionality from February 2008 to August 2008, is not readily 
apparent. Finally, OPM's report did not address conducting independent 
verification and validation of its cost estimate and the process used 
to develop the estimate. 

Earned Value Management: 

GAO Finding and Recommendation: Earned value management (EVM) is a tool 
for measuring program progress by comparing the value of work 
accomplished with the amount of work expected to be accomplished. Such 
a comparison permits actual performance to be evaluated, based on 
variances from the planned cost and schedule, and future performance to 
be forecasted. Our study determined that OPM's reporting of program 
progress using EVM was unreliable because the agency did not establish 
and validate a meaningful performance measurement baseline. 
Accordingly, we recommended that the agency establish a basis for 
effective use of earned value management by validating the RSM 
performance measurement baseline through a program level integrated 
baseline review, and that it task an independent verification and 
validation contractor with reviewing the process used to develop the 
baseline and assessing the reliability of the performance measurement 
baseline. 

OPM Report to Congress: OPM stated in its report that the agency had 
established a performance measurement baseline, and had implemented EVM 
practices. Specifically, the agency stated that it had measured and 
reported performance on a monthly basis, used an agency-standard EVM 
tool set, and worked with its contractors to improve EVM reporting. 
Further, OPM characterized GAO's concerns about its use of EVM as being 
related to the proper inclusion of work to be performed by government 
personnel and of work related to a delay in deployment of planned 
functionality. 

GAO Comments: OPM's report did not provide new information or describe 
the progress the agency has made toward addressing the three specific 
weaknesses that we identified in its use of EVM. First, the report did 
not state whether OPM has established a performance measurement 
baseline that reflects the full scope of the RSM program. Second, the 
report did not say whether the agency has validated its performance 
measurement baseline in an integrated baseline review. Lastly, the 
report did not address whether the baseline against which the agency 
has measured and reported has been stabilized. In addition to these 
three weaknesses, the report also did not address conducting 
independent verification and validation of the performance measurement 
baseline and the process used to develop the baseline. Without 
addressing these weaknesses, OPM's use of EVM will continue to be 
unreliable. 

Agency Comments: 

In written comments on a draft of this letter, the Director of OPM 
stated that the agency successfully initiated rollout of its new 
retirement system to the approximately 26,000 employees in agencies 
serviced by the General Services Administration payroll processing 
center on February 25, 2008. Further, the Director stated that OPM's 
forthcoming formal response to GAO's January 2008 report on RSM will 
provide detailed information on the agency's progress in developing and 
implementing the new system. Finally, the Director commented that the 
short time between its February 20, 2008 report to Congress and system 
deployment necessitated that the report be brief and that its contents 
(which are summarized in this letter) be limited to official use only. 
However, in subsequent comments e-mailed to us, OPM's liaison to GAO 
stated that the report to Congress did not contain sensitive 
information. The agency's comments are reprinted in the enclosure. 

We are sending copies of this letter to the Director of the Office of 
Personnel Management and other appropriate congressional committees. We 
will make copies available to other interested parties upon request and 
will made it available at no charge on GAO's Web site at [hyperlink, 
http://www.gao.gov]. 

If you or your staff have any questions about this correspondence, 
please contact me at (202) 512-6304 or [email protected]. Key 
contributors to this letter were Mark T. Bird, Rebecca E. LaPaze, and 
Teresa M. Neven. 

Signed by: 

Valerie C. Melvin: 
Director, Human Capital and Management:
Information Systems Issues: 

[End of correspondence] 

Enclosure: 

Comments from the Office of Personnel Management: 

The Director: 
United States Office Of Personnel Management: 
Washington, DC 20415: 
[hyperlink, http://www.opm.gov]: 
[hyperlink, http://www.usaJobs.gov]: 
"Our mission is to ensure the Federal Government has an effective 
civilian workforce."	 

March 18, 2008: 

Mr. Gene L. Dodaro: 
Acting Comptroller General: 
U.S. Government Accountability Office: 
441 G Street, NW: 
Washington, DC 20548: 

Dear Mr. Dodaro: 

Thank you for the opportunity to comment on the Government 
Accountability Office's (GAO) draft response to the Office of Personnel 
Management's (OPM) February 20, 2008, Retirement Systems Modernization 
(RSM) Report to Congress. 

On February 25, 2008, OPM successfully initiated the rollout of its 
landmark strategic program that has modernized the business processes 
and supporting technology used to administer the Federal Government's 
defined retirement benefit plans. Approximately 26,000 employees in 
agencies serviced by the General Services Administration (GSA) payroll 
processing center are now eligible to retire under the new system, 
which is being called RetireEZ. These agencies include OPM, GSA, the 
National Archives and Records Administration, and the Railroad 
Retirement Board. Subsequent rollouts will cover the remainder of the 
Executive Branch, the U.S. Postal Service and the Legislative and 
Judicial Branches. 

OPM was tasked with responding to the Congressional request for a 
status report just three days before the launch of RetireEZ. The 
challenging timing of this request necessitated that our responses were 
briefer than we would have preferred. We believe the brevity of this 
report supports our position that this information ï¿½ while accurate ï¿½ 
should be limited to official use only. We believe the public deserves 
the benefit of a more comprehensive report. 

OPM is currently preparing our sixty day response to GAO's report 
entitled Office of Personnel Management: Improvements Needed to Ensure 
Successful Retirement Systems Modernization (GAO-08-345). This 
information, to be submitted to the United States House of 
Representatives and Senate Committees on Appropriations no later than 
March 31, 2008, will provide greater detail on OPM's progress in 
developing and implementing the new system, in addition to addressing 
the four recommendations in this particular GAO report. 

Thank you again for the opportunity to provide GAO with these comments. 
OPM continues to take the necessary steps to ensure that the Federal 
Government has a state-of-the-art retirement administration system for 
annuitants and employees. 

Sincerely, 

Signed by: 

Linda M. Springer:
Director: 

[End of enclosure] 

Footnotes: 

[1] GAO, Office of Personnel Management: Improvements Needed to Ensure 
Successful Retirement Systems Modernization, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-08-345] (Washington, 
D.C.: January 31, 2008). 

[2] Consolidated Appropriations Act, 2008, P.L. 110-161 (Dec. 26, 
2007); 153 Cong. Rec. H15741, H16056 (daily ed. Dec. 17, 2007). 

[3] GAO, Year 2000 Computing Crisis: A Testing Guide, GAO/AIMD-10.1.21 
(Washington, D.C.: November 1998); Information Technology: Customs 
Automated Commercial Environment Progressing, but Need for Management 
Improvements Continues, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-05-267] 
(Washington, D.C.: March 14, 2005); and Homeland Security: Visitor 
and Immigrant Status Program Operating, but Management Improvements 
Are Still Needed, [hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T] 
(Washington, D.C.: January 25, 2006). 

[4] GAO, Customs Service Modernization: Automated Commercial 
Environment Progressing, but Further Acquisition Management 
Improvements Needed, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-03-406] 
(Washington, D.C.: February 28, 2003) and Homeland Security: 
Visitor and Immigrant Status Program Operating, but Management 
Improvements Are Still Needed, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-06-318T] 
(Washington, D.C.: January 25, 2006). 

[5] GAO, Cost Assessment Guide: Best Practices for Estimating and 
Managing Program Costs, Exposure Draft, [hyperlink, 
http://www.gao.gov/cgi-bin/getrpt?GAO-07-1134SP] (Washington, 
D.C.: July 2007). 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***