Defense Transportation: DOD Has Adequately Addressed		 
Congressional Concerns Regarding the Cost of Implementing the New
Personal Property Program Initiatives (09-JUN-05, GAO-05-715R).  
                                                                 
Military personnel and their families can expect to relocate many
times during a servicemember's career. As the moving industry's  
single largest customer, the Department of Defense (DOD) spends  
more than $1.7 billion annually for its personal property	 
program, which provides household goods transportation and	 
storage services for military personnel and their families when  
they relocate. The program manages more than 600,000 personal	 
property shipments each year. For more than 10 years, DOD has	 
been pursuing various initiatives for improving the quality of	 
its personal property program. In June 2002, the U.S.		 
Transportation Command completed an extensive study that compared
the features of the current personal property program with three 
pilot programs that tested alternative approaches for improving  
the current program. In November 2002, DOD issued a report to	 
Congress that included three recommended program improvement	 
initiatives resulting from this study and estimated that an	 
additional 13 percent increase over current program costs would  
be required to implement two of these initiatives. In April 2003,
we reported on the pilot program evaluation and stated that the  
recommendations contained in DOD's November 2002 report offered  
solutions to long-standing problems in the personal property	 
program and should be implemented within budget constraints.	 
However, we raised concerns about whether the two recommendations
related to the claims and contracting processes could be	 
implemented within the projected 13 percent cost estimate. Our	 
concerns were partially based on DOD not adequately		 
substantiating the expected economies of scale that supported its
assumption of a 5 percent reduction in average prices under the  
pilot program when a full-scale program is implemented. As a	 
result, we recommended that DOD quantify the risk associated with
the cost estimate before DOD's recommended initiatives are	 
implemented. In its May 2004 report on the National Defense	 
Authorization Act for Fiscal Year 2005, the House Armed Services 
Committee directed the Secretary of Defense to reevaluate DOD's  
proposed cost estimate, quantify the risk or likelihood of	 
achieving its goals within the 13 percent projected cost	 
increase, and develop a range of possible cost increases	 
associated with the risk. The committee also directed GAO to	 
review and report on whether DOD adequately performed these	 
tasks.								 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-05-715R					        
    ACCNO:   A26238						        
  TITLE:     Defense Transportation: DOD Has Adequately Addressed     
Congressional Concerns Regarding the Cost of Implementing the New
Personal Property Program Initiatives				 
     DATE:   06/09/2005 
  SUBJECT:   Household goods					 
	     Military cost control				 
	     Military personnel 				 
	     Personal property					 
	     Relocation expense claims				 
	     Transportation costs				 
	     Program management 				 
	     Evaluation methods 				 
	     Cost estimates					 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-05-715R

United States Government Accountability Office Washington, DC 20548

June 9, 2005

The Honorable John Warner
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate

The Honorable Duncan L. Hunter
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on Armed Services
House of Representatives

Subject: Defense Transportation: DOD Has Adequately Addressed
Congressional Concerns Regarding the Cost of Implementing the New Personal
Property Program Initiatives

Military personnel and their families can expect to relocate many times
during a servicemember's career. As the moving industry's single largest
customer, the Department of Defense (DOD) spends more than $1.7 billion
annually for its personal property program, which provides household goods
transportation and storage services for military personnel and their
families when they relocate. The program manages more than 600,000
personal property shipments each year.

For more than 10 years, DOD has been pursuing various initiatives for
improving the quality of its personal property program. In June 2002, the
U.S. Transportation Command completed an extensive study that compared the
features of the current personal property program with three pilot
programs that tested alternative approaches for improving the current
program. In November 2002, DOD issued a report to Congress that included
three recommended program improvement initiatives resulting from this
study and estimated that an additional 13 percent increase over

current program costs would be required to implement two of these
initiatives.1

In April 2003, we reported on the pilot program evaluation and stated that
the recommendations contained in DOD's November 2002 report offered
solutions to long-standing problems in the personal property program and
should be implemented within budget constraints.2 However, we raised
concerns about whether the two recommendations related to the claims and
contracting processes could be implemented within the projected 13 percent
cost estimate. Our concerns were partially based on DOD not adequately
substantiating the expected economies of scale that supported its
assumption of a 5 percent reduction in average prices under the pilot
program when a full-scale program is implemented. As a result, we
recommended that DOD quantify the risk associated with the cost estimate
before DOD's recommended initiatives are implemented.

In its May 2004 report on the National Defense Authorization Act for
Fiscal Year 2005,3 the House Armed Services Committee directed the
Secretary of Defense to reevaluate DOD's proposed cost estimate, quantify
the risk or likelihood of achieving its goals within the 13 percent
projected cost increase, and develop a range of possible cost increases
associated with the risk. The committee also directed GAO to review and
report on whether DOD adequately performed these tasks.

To conduct our work, we reviewed prior DOD and GAO reports on the personal
property program and interviewed DOD officials and officials from DOD's
contractor who were involved in the current cost reevaluation study. We
assessed the methodologies used by DOD to reevaluate the cost estimate and
quantify the risk associated with implementing the proposed initiatives
within the cost estimate, including establishing ranges of possible cost
increases. We also assessed the reasonableness of the methodologies used
by DOD to perform these tasks. We performed our review from November 2004
through May 2005 in accordance with generally accepted government auditing
standards. The scope and

1 Department of Defense, U.S. Transportation Command Personal Property
Pilot Programs Evaluation Report (Washington, D.C.: November 2002).

2 GAO, Defense Transportation: Monitoring Costs and Benefits Needed While
Implementing a New Program for Moving Household Goods, GAO-03-367
(Washington, D.C.: Apr. 18, 2003).

3 H.R. Rep. No. 108-491, at 298 (2004).

                                Results in Brief

Background

methodology we used in our review are described in further detail in
enclosure I.

The methodology used by DOD to address congressional concerns-reevaluate
its cost estimate, quantify the risk associated with achieving the 13
percent cost increase, and establish a range of possible cost increases-
was reasonable for conducting such assessments. In its reevaluation of the
cost estimate, DOD validated the current mix of household goods shipments
to ensure that the mix used in the original assessment was appropriate.
DOD also provided additional support for its assessment of the anticipated
savings that would result from expanding the pilot to all of DOD.
Furthermore, the results of DOD's risk-based simulation analysis, which
included establishing ranges of possible cost increases, indicated that
about 80 percent of the time, DOD could implement the recommended
initiatives to improve the current program with an increase of 15 percent
or less above current program costs.

DOD concurred with the content of this letter.

DOD's personal property program is managed centrally by the Military
Surface Deployment and Distribution Command, formerly known as the
Military Traffic Management Command.4 DOD has experienced longstanding
problems with its current personal property program, including excessive
loss of or damage to property, high claims costs incurred by the
government, and poor quality of service from moving companies. Moreover,
the program's data management system does not provide reliable information
on the status of individual shipments or on the types of shipments and
their costs.

In its November 12, 2002, report to Congress, DOD made three
recommendations aimed at improving its current personal property program.5
The three recommendations were to (1) reengineer the liability and claims
process by adopting commercial practices of minimum

4 The Military Surface Deployment and Distribution Command, a component of
the U.S. Transportation Command, is the DOD executive agent responsible
for managing the relocation process for servicemembers and their families.

5 Department of Defense, U.S. Transportation Command Personal Property
Pilot Programs Evaluation Report.

valuation, simplifying the filing of claims, and providing direct
settlement with the carrier; (2) change the acquisition process to
implement performance-based service contracts; and (3) implement
information technology improvements, which could interface functions
across such areas as personnel, transportation, financial, and claims. DOD
reported that the estimated cost of implementing its information
technology improvements recommendation would be from $4 million to $6
million, and estimated that the cost of implementing the claims process
and performance-based service contract recommendations would require an
additional 13 percent over the current personal property program's costs.

In our April 2003 report, we stated that the three recommendations
contained in DOD's report were supported by the Transportation Command's
evaluation of the pilot programs' findings and offered solutions to the
long-standing problems that had plagued the current program for many
years.6 Our review showed that the soundness of the methodologies DOD used
to develop cost estimates for implementing the three recommendations
varied. We found that the estimates DOD reported to Congress might
understate the total initial cost for implementing the information
technology improvements recommendation and contained a questionable
adjustment for costs associated with the claims and contracting process
recommendations.

In its original cost assessment for the recommendations related to the
claims and contracting processes, DOD made three adjustments to the
average costs for the pilot programs: (1) reducing the average weight of
shipments, (2) reducing costs to adjust for a mix of small and large
businesses, and (3) reducing the pilot programs' costs to reflect
anticipated savings based on "economies of scale." In our April 2003
report, we agreed that the first two adjustments were reasonable; however,
we questioned the extent to which the third adjustment could be achieved.
Part of our concern was based on DOD not adequately substantiating the
expected economies of scale that supported its assumption of a 5 percent
reduction in the average prices under the pilot programs when a full-scale
program is implemented. Since DOD had not quantified the risk associated
with its projected implementation cost estimate of a 13 percent increase
over the current program's cost, the military services and Congress lacked
information needed to develop and review future budget requests for the
program. Therefore, we questioned

6 GAO-03-367.

the extent to which the recommendations could be implemented within DOD's
projected cost estimate.

As a result of our findings, we recommended that DOD take the following
actions to improve its personal property program: (1) initiate actions
that will implement the recommendations contained in DOD's report to
Congress within budget constraints, (2) quantify the risk associated with
achieving implementation of the recommended initiatives within the
projected 13 percent cost estimate, (3) monitor costs during the
implementation phase to ensure that the proposed changes are being
achieved within an acceptable and a predefined range, and (4) assess the
personal property program after the recommendations have been implemented
to determine whether anticipated improvements are being achieved at a
reasonable cost.

DOD partially concurred with our second recommendation--to provide the
military services and Congress with additional information to quantify the
risk associated with achieving implementation of the recommended
initiatives within the projected 13 percent cost increase. DOD believed it
could incorporate the two recommendations into a new program within its
proposed 13 percent cost estimate due to the conservative approach it took
in developing the estimate. However, in May 2004, the House Armed Services
Committee, in its report on the National Defense Authorization Act for
Fiscal Year 2005, directed DOD to reevaluate its proposed cost estimate,
quantify the risk or likelihood of achieving its goals within the 13
percent cost projection, and develop a range of possible cost increases
associated with that risk. DOD issued a report addressing these
congressional concerns on March 29, 2005.7

In addressing congressional concerns, DOD used a reasonable methodological
approach to adequately reevaluate its cost estimate and quantify the risk
associated with implementing the proposed personal property program
initiatives within the 13 percent cost increase, including establishing a
range of potential cost increases. DOD's analysis included three key
components: (1) an assessment of any changes in the distribution of
shipments among the services and the mix of continental and overseas
household goods shipments, (2) new support for its

7 Department of Defense, Reevaluation of Cost Estimate for DOD Families
First Program (Washington, D.C.: March 2005).

                DOD Adequately Addressed Congressional Concerns

assessment of anticipated savings through "full-program efficiencies"
rather than "economies of scale," and (3) a simulation based on differing
estimates of "full-program efficiencies" to determine the likelihood that
DOD could implement its initiatives within its cost estimate. We believe
that each of these components was necessary to reassess the reasonableness
of DOD's previous cost estimate.

DOD first validated the current mix of household goods shipments to ensure
that the mix used in the original assessment was appropriate. DOD's
reevaluation assessed changes, if any, in the distribution of shipments
among the services and the mix of continental and overseas U.S. household
goods shipments. Because the original cost estimate was based on household
goods shipments picked up and delivered during the last half of fiscal
year 2001, DOD needed to determine if the mix of household goods shipments
had changed. Using data from its management information system, the
Transportation Operational Personal Property Standard System, DOD found
some minor differences in shipments among the services from the baseline
year (fiscal year 2001) to fiscal year 2004, but concluded that the
differences were insignificant and would have no effect on the
calculations developed based upon the fiscal year 2001 data. This was a
reasonable methodological approach to validate the mix of shipments.

DOD then used a reasonable methodology to provide additional support for
its assessment of the anticipated savings that would result from expanding
the pilot to all of DOD. In its reassessment of the cost estimate, DOD
reported that the anticipated savings could be better described as
"full-program efficiencies" instead of "economies of scale." For example,
DOD's reevaluation explained that prospective participants in the pilot
programs faced limitations inherent in the pilot programs in a competitive
market because they could not anticipate the longevity of the pilot
program nor accurately predict the volume of potential shipments.
Potential participants in the pilot programs also faced additional
uncertainties that included not knowing their daily or monthly volume for
this new program, having to project rates into option years without
economic price adjustment, and facing possible penalties for withdrawing
from the program. Consequently, the rates submitted by prospective
participants reflected these uncertainties and other risk factors. It
could reasonably be expected that as experience under the new program is
gained and a steady-state program is reached, these risks would be reduced
and would be reflected in lower rates. We believe that with this detailed
description of the full-program efficiencies, DOD has provided

reasonable additional support for the anticipated savings of the program,
thereby reaffirming the 5 percent efficiency gains it previously reported.

Finally, DOD used a risk-based simulation approach to provide a range of
possible cost increases within statistical confidence intervals around the
cost estimate, which is a reasonable methodology. This simulation
quantified the risk associated with implementing the proposed initiatives
within the 13 percent cost estimate. The results of this risk analysis
indicated that about 80 percent of the time, DOD could implement the
recommended initiatives to improve the current program within an increase
of 15 percent or less above current program costs. All simulations used by
DOD involved 10,000 iterations of six different cases of variation in the
full-program efficiencies estimate to assess how different levels of
full-program efficiencies affected the cost estimates.8 Table 1 summarizes
DOD's simulation results for each scenario. As the table shows, only case
two would result in 80 percent of all estimates being 12.5 percent or
less- an amount lower than the original cost estimate. The remaining five
simulations of the random effect of the full-program efficiencies contain
the original 13.4 percent estimate within 90 percent confidence intervals.

Table 1: Simulation Results Showing the Effect of Varying Program
Efficiency Distributions on Achieving the Projected Cost Estimate

90% confidence interval for cost Program efficiency estimate estimate

Case Low value High value Most frequent       80th           Low      High 
                             value            percentile      value     value 
               5%         5%             5%     14.4%           11%       15% 
               0%        20%             5%     12.5%            4%       15% 
               0%        15%             5%     13.8%            7%       16% 
               0%        20%           2.5%     13.7%            4%       16% 
               0%        15%           2.5%     14.9%            8%       17% 
               0%        10%             5%     15.1%           10%       17% 

Source: GAO analysis of DOD data.

We reviewed the results of the simulation to assess its rigor and how well
the simulation reflects the household goods shipment costs. Based on our

8 In its report, DOD noted that the advantage of the probability-based
simulation technique used to conduct its risk assessment is that it can
account for multiple input costs that have variability in risk in
relationship to how they affect the overall cost estimate.

review, a briefing of the methodology described in the report, and our
knowledge of the analytical approaches for conducting such an assessment,
we believe that DOD used a reasonable methodology that is consistent with
professional standards to quantify the risk associated with implementing
the program improvement initiatives within its cost estimate, because DOD
ran a simulation that is a reasonable approach for assessing risk, using a
commercially available program. See enclosure II for a detailed summary of
the results of DOD's risk assessment.

Agency Comments 	DOD concurred with the content of this letter. DOD's
letter is included in enclosure III.

We are sending copies of this report to the appropriate congressional
committees; the Secretary of Defense; the Commander, U.S.
Transportation Command; and the Director, Office of Management and
Budget. We will also make copies available to others upon request. In
addition, the report will be made available at no charge on the GAO Web
site at http://www.gao.gov.

Please contact me at (202) 512-8365 or [email protected] if you or your staff
have any questions concerning this report. Contact points for our Offices
of Congressional Relations and Public Affairs may be found on the last
page of this report. Key contributors to this report were Jacqueline S.
McColl, Arthur L. James, Jr., Charles W. Perdue, Karen N. Harms, Renee S.
Brown, and Ann Borseth.

William M. Solis
Director, Defense Capabilities and

Management

Enclosures

                       Enclosure I: Scope and Methodology

To assess the methods used by the Department of Defense (DOD) to
reevaluate the cost estimate and quantify the risk associated with
implementing the program improvement initiatives within the stated cost
projection, including establishing a range of possible cost increases, we
reviewed the cost projection methodology used by DOD in its report on the
reevaluation of the projected cost estimate. We also met with DOD
officials and its contractor officials to discuss the methodology used to
perform the cost reevaluation and risk assessment.

To assess the reasonableness of the methodology used by DOD to perform
these tasks, we compared DOD's reevaluation methodology to analytical
approaches applicable for this type of evaluation, assessed adjustments
made in the evaluation strategy to address issues that could affect the
validity of the results, and reviewed the evaluation techniques used to
analyze data. We reviewed the results of the simulation DOD used to
conduct its risk assessment to assess its rigor and how well the
simulation reflects the household goods shipment costs. We also reviewed
the contents of the DOD contractor's briefings on the planned approach and
evaluation techniques used to reevaluate the projected 13 percent cost
estimate.

We did not assess whether the anticipated benefits to be derived from
implementing the three recommendations would warrant the additional costs
that DOD estimates will be required to fund these improvements.
Furthermore, we did not independently test the reliability of data DOD
extracted from its data system to develop costs.

During this review of DOD's evaluation efforts, we met with officials and
obtained documents from the Office of the Assistant Deputy Under Secretary
of Defense (Transportation Policy), Washington, D.C.; the U.S.
Transportation Command, Scott Air Force Base, Illinois; the Surface
Deployment and Distribution Command, Alexandria, Virginia; and LMI
(Surface Deployment and Distribution Command contractor), McLean,
Virginia. In addition to these agency meetings and documents, we drew upon
information contained in a previous GAO report resulting from our prior
review of this program.

Our work for this review was performed from November 2004 through May 2005
in accordance with generally accepted government auditing standards.

                 Enclosure II: Summary of DOD's Risk Assessment

DOD's methodology to quantify the risk associated with the cost estimate
included using a commercially available program to run a Monte Carlo
simulation,1 which is a reasonable methodological approach for assessing
risk. The simulation contained six cases of variation in the full-program
efficiencies estimate to assess how these full-program efficiencies
affected the cost estimates. Case one was the baseline case involving the
original 13.4 percent increase resulting from an assumption of a fixed 5
percent reduction for full-program efficiencies. This case represents the
original estimates provided by DOD in its November 2002 report to
Congress. The remaining five cases randomly varied the assumption for the
full-program efficiencies over five different ranges, from a very compact
range between zero and 10 percent to a very diverse range from zero to 20
percent.

Two cases in particular illustrated the effects of the simulation. Case
six, the very compact case, allowed the frequency of the full-program
efficiencies estimate to increase from zero smoothly to 5 percent and
decline smoothly to 10 percent. The result of this simulation was very
similar to the base case (case one, in which the original estimate of
fullprogram efficiencies was held equal to 5 percent). The results
indicate that 90 percent of the time the cost estimate would fall from 10
to 17 percent, with 80 percent of all estimates less than 15.1 percent.

The most "generous" case, case two, was based upon the assumption of the
frequency for full-program efficiencies increasing smoothly to 5 percent
and declining smoothly to 20 percent. This case is most generous because
it assumes that large cost savings (up to 20 percent) are achievable even
though the most likely cost savings would be 5 percent. The results
indicate that 90 percent of the time the cost estimate would fall from 4
to 15 percent, with 80 percent of all estimates less than 12.5 percent.
This was the one case where the original estimate was outside the 80th
percentile and thus a lower cost estimate was determined to be most
likely. However, the case was most generous in its assumption of the
frequency with which "large" full-program efficiencies would occur and

1 A Monte Carlo simulation is a method of evaluating hypotheses by
developing a computer model of a process, defining the parameters of the
process to reflect the "real world" situation, calculating multiple
results of varying parameters through some range of values, and evaluating
the distribution of results obtained from these samples. The Monte Carlo
method assigns a value to the model's parameters from a sequence of random
numbers, runs the model multiple times through randomly differing
parameter values, captures the outcome of each iteration of the model, and
assesses the distribution of outcomes using standard statistical methods.

Enclosure II: Summary of DOD's Risk Assessment

could reasonably be expected to indicate a smaller increase in estimated
costs over the baseline program.

The other three cases, between the two extremes, varied the frequency of
full-program efficiencies over different ranges and the simulation results
contained the original 13.4 percent estimate within the 90 percent
confidence intervals. Based on our review of DOD's analysis, we believe
that DOD used a reasonable approach that was consistent with professional
standards to quantify the risk associated with implementing the program
improvement initiatives within its cost estimate.

             Enclosure III: Comments from the Department of Defense

                              Related GAO Products

Defense Transportation: Monitoring Costs and Benefits Needed While
Implementing a New Program for Moving Household Goods. GAO-03-367.
Washington, D.C.: April 18, 2003.

Defense Transportation: Final Evaluation Plan Is Needed to Assess
Alternatives to the Current Personal Property Program. GAO/NSIAD-00217R.
Washington, D.C.: September 27, 2000.

Defense Transportation: The Army's Hunter Pilot Project Is Inconclusive
but Provides Lessons Learned. GAO/NSIAD-99-129. Washington, D.C.: June 23,
1999.

Defense Transportation: Plan Needed for Evaluating the Navy Personal
Property Pilot. GAO/NSIAD-99-138. Washington, D.C.: June 23, 1999.

Defense Transportation: Efforts to Improve DOD's Personal Property
Program. GAO/T-NSIAD-99-106. Washington, D.C.: March 18, 1999.

Defense Transportation: The Army's Hunter Pilot Project to Outsource
Relocation Services. GAO/NSIAD-98-149. Washington, D.C.: June 10, 1998.

Defense Transportation: Reengineering the DOD Personal Property Program.
GAO/NSIAD-97-49. Washington, D.C.: November 27, 1996.

This is a work of the U.S. government and is not subject to copyright
protection in the United States. It may be reproduced and distributed in
its entirety without further permission from GAO. However, because this
work may contain copyrighted images or other material, permission from the
copyright holder may be necessary if you wish to reproduce this material
separately.

GAO's Mission

Obtaining Copies of GAO Reports and Testimony

The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO's
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO's Web site (www.gao.gov). Each weekday, GAO posts newly
released reports, testimony, and correspondence on its Web site. To have
GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select "Subscribe to Updates."

Order by Mail or Phone 	The first copy of each printed report is free.
Additional copies are $2 each. A check or money order should be made out
to the Superintendent of Documents. GAO also accepts VISA and Mastercard.
Orders for 100 or more copies mailed to a single address are discounted 25
percent. Orders should be sent to:

U.S. Government Accountability Office 441 G Street NW, Room LM Washington,
D.C. 20548

To order by Phone: 	Voice: (202) 512-6000 TDD: (202) 512-2537 Fax: (202)
512-6061

To Report Fraud, Contact:

Waste, and Abuse in Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: [email protected] Programs Automated answering system: (800)
424-5454 or (202) 512-7470

Gloria Jarmon, Managing Director, [email protected] (202)
512-4400Congressional U.S. Government Accountability Office, 441 G Street
NW, Room 7125 Relations Washington, D.C. 20548

Public Affairs 	Paul Anderson, Managing Director, [email protected] (202)
512-4800 U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548

                                 RECYCLED PAPER
*** End of document. ***