Defense Transportation: The Army's Hunter Pilot Project Is Inconclusive
but Provides Lessons Learned (Letter Report, 06/23/99, GAO/NSIAD-99-129).
Pursuant to a legislative requirement, GAO provided information on the
lessons learned from the Army's personal property program, the Hunter
Pilot Project, focusing on the: (1) Army's evaluation methodology of the
Hunter Pilot, including the validity of data and reported results; and
(2) status of all ongoing and planned pilot projects and the adequacy of
the Department of Defense's (DOD) plans to evaluate the pilot projects.
GAO noted that: (1) although the Army reported that the Hunter pilot was
a success, GAO found that most of the results of its evaluation were
inconclusive; (2) while the Hunter pilot provides services and benefits
that were not previously available during the moving process, GAO was
unable to validate all reported results of the Army's evaluation of this
pilot because of weaknesses in the evaluation methodology and data; (3)
specifically, because of the methods used in conducting the customer
surveys, GAO could not confirm that customer satisfaction improved; (4)
also, due to weaknesses in the Army's methodology and data reliability,
GAO could not validate the extent to which pilot costs exceeded baseline
costs; (5) however, GAO was able to confirm that 33 percent of the pilot
shipments were awarded to small business carriers and agents; (6)
further, lessons learned by the Army in developing an evaluation plan,
conducting the pilot test, and evaluating results can provide useful
information to DOD as it conducts and assesses other pilot efforts; (7)
the Navy and the Military Traffic Management Command each have a
personal property pilot project underway, and DOD is proposing a fourth
pilot to test different approaches to improve its personal property
program; (8) as a result, DOD will be running multiple pilots
concurrently, with different goals, objectives, and expected outcomes;
(9) U.S. Transportation Command (USTRANSCOM) is tasked with evaluating
the results of the pilots and using that information to recommend a
redesigned Department-wide relocation program; (10) however, DOD has not
yet determined how many approaches will ultimately be tested and the
milestones for completing the pilots' evaluation and implementing an
improved process, nor has it assured itself that a methodologically
sound evaluation process is in place to execute this process; (11)
improving DOD's personal property program has been a slow, complex
process; (12) DOD and the services have spent a large amount of time and
effort to dramatically change the quality of services military personnel
receive; and (13) GAO supports pilots as a tool to test different
approaches.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-99-129
TITLE: Defense Transportation: The Army's Hunter Pilot Project Is
Inconclusive but Provides Lessons Learned
DATE: 06/23/99
SUBJECT: Program evaluation
Data integrity
Evaluation methods
Personal property
Employee transfers
Household goods
IDENTIFIER: Army Hunter Pilot Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
GAOSEAL.EPS GAO United States General Accounting Office
Report to Congressional Committees
June 1999 DEFENSE TRANSPORTATION
The Army's Hunter Pilot Project Is Inconclusive but
Provides Lessons Learned
GAO/NSIAD-99-129
Page 1 GAO/NSIAD-99-129 Defense Transportation United States
General Accounting Office
Washington, D. C. 20548 Let ter National Security and
International Affairs Division
B-282406 Letter June 23, 1999 Congressional Committees The
Department of the Army began a pilot project in July 1997 to test
an alternative approach for providing relocation services for its
military
personnel stationed at Hunter Army Airfield, Savannah, Georgia.
The Army undertook this effort to address long- standing concerns
and problems associated with the current personal property
program. The Department of Defense (DOD) has two other pilot
programs underway to test different approaches to improving its
personal property program and is proposing a fourth pilot. To
determine which pilot, or portions thereof, could provide better
long- term results, the Deputy Under Secretary of Defense
(Logistics) tasked the U. S. Transportation Command (USTRANSCOM)
to oversee all
personal property pilot tests and recommend the follow- on course
of action. The statement of managers in the conference report on
the 1997 DOD Appropriations Act directed us to validate the
results and savings achieved before DOD expands any of its
personal property pilot projects. 1 In this regard, we testified
in March 1999 on the status of the Army's pilot and DOD's efforts
to improve its personal property program. 2 This report
supplements our testimony and principally addresses the results of
our review of the 12- month Hunter pilot test and lessons for
evaluating other pilots. Our objectives were to (1) assess the
Army's evaluation methodology of the Hunter pilot, including the
validity of data and reported results and (2) determine the status
of all ongoing and planned pilot
projects and the adequacy of DOD's plans to evaluate the pilot
projects. Results in Brief Although the Army reported that the
Hunter pilot was a success, we found that most of the results of
its evaluation were inconclusive. While the Hunter pilot provides
services and benefits that were not previously
available during the moving process, we were unable to validate
all 1 House Report 104- 863 (Sept. 28, 1996) p. 865. 2 Defense
Transportation: Efforts to Improve DOD's Personal Property Program
(GAO/T-NSIAD-99-106, Mar. 18, 1999).
Let t er
B-282406 Page 2 GAO/NSIAD-99-129 Defense Transportation
reported results of the Army's evaluation of this pilot because of
weaknesses in the evaluation methodology and data. Specifically,
because of the methods used in conducting the customer surveys, we
could not confirm that customer satisfaction improved. Also, due
to weaknesses in the Army's methodology and data reliability, we
could not validate the extent to which pilot costs exceeded
baseline costs. However, we were able to confirm that 33 percent
of the pilot shipments were awarded to small business carriers and
agents. Further, lessons learned by the Army in developing an
evaluation plan, conducting the pilot test, and evaluating results
can provide useful information to DOD as it conducts and assesses
other pilot efforts.
The Navy and the Military Traffic Management Command (MTMC) each
have a personal property pilot project underway, and DOD is
proposing a fourth pilot to test different approaches to improve
its personal property program. As a result, DOD will be running
multiple pilots concurrently, with different goals, objectives,
and expected outcomes. USTRANSCOM 3 is tasked with evaluating the
results of the pilots and using that information to recommend a
redesigned Department- wide relocation program. However, DOD has
not yet determined how many approaches will
ultimately be tested and the milestones for completing the pilots'
evaluation and implementing an improved process, nor has it
assured itself that a methodologically sound evaluation process is
in place to execute this process. Improving DOD's personal
property program has been a slow, complex process. DOD and the
services have spent a large amount of time and effort to
dramatically change the quality of services military personnel
receive. We support pilots as a tool to test different approaches.
However, before the Department can make any credible decisions on
changing the current program, we recommend that the Secretary of
Defense direct USTRANSCOM to develop a comprehensive strategy for
evaluating each of the pilots' attributes in a comparable manner.
Further, on the basis of the Army's lessons learned in conducting
the Hunter pilot and of our evaluation of the pilot, we recommend
that the Department seek expert methodological advice to enhance
the quality of its assessment. 3 The mission of USTRANSCOM, which
is DOD's single manager of all Defense transportation services, is
to provide global air, land, and sea transportation to meet
national security needs. USTRANSCOM executes its mission through
three component commands: MTMC for land transportation and port
operations, the Military Sealift Command for sea transportation,
and the Air Mobility Command for air transportation.
Let t er
B-282406 Page 3 GAO/NSIAD-99-129 Defense Transportation
Background DOD has long been concerned about the quality of
service it provides military personnel and their families when
they relocate. DOD spends approximately $3 billion annually to
transport, store, and manage the household goods and unaccompanied
baggage of its servicemembers and families. Past problems included
poor service from movers, excessive incidence of loss or damage to
servicemembers' property, and high claims costs to the government.
All of these problems contributed to poor quality of service for
people using the system.
Most Results of the Army Hunter Pilot Are Inconclusive
Because of weaknesses in the Army's evaluation methodology and
data, we were unable to validate the reported results of the
quality of life and cost factors of the Hunter pilot program.
Moreover, the Army's evaluation plan methodology was not an
effective tool for collecting and analyzing the pilot results.
However, lessons learned by the Army in conducting the 12- month
pilot test do provide useful information to DOD as it conducts and
assesses its pilot efforts. Also, the Hunter pilot is providing
services and benefits that were not previously available,
including point- to- point move management, personal move
counseling and coordination, direct claims settlement, assistance
in buying/ selling a residence, and visibility of the shipment
throughout the move. The Army's Assessment and Evaluation Approach
The Army reported that through its first 12 months of operations,
the Hunter pilot successfully demonstrated that commercial
practices could be applied to the military relocation process.
Specifically, the Army was able
to contract with Cendant Mobility, a move management company, to
provide services similar to those available in the private sector.
In January 1997, the Army began developing a test evaluation plan
that defined roles and responsibilities, test factors, and
processes for capturing data related to each test factor and
stated how test data would be compared to that from the existing
program (i. e., the baseline). The plan stated that the Army Audit
Agency (AAA) would be responsible for validating the baseline and
test data and for providing the Army a report on the test results.
We reported in June 1998 that the Army needed to further define
the various factors and measurements to be included in its
evaluation plan. 4 The Army
clarified how it would measure pilot success and revised its
methodology 4 Defense Transportation: The Army's Hunter Pilot
Project to Outsource Relocation Services (GAO/NSIAD-98-149, June
10, 1998).
B-282406 Page 4 GAO/NSIAD-99-129 Defense Transportation
in the evaluation plan, dated August 12, 1998. This represented
the third such revision, which was not completed until after the
12- month test period. The modified plan indicated that the pilot
results would be reviewed on the basis of three factors quality of
life, cost, and impact on small business and would use a 5- point
scoring system. The Army assigned the greatest weight to the
quality of life factor, which consisted of one point each for
three sub- factors customer satisfaction, average claims
settlement time, and percentage of direct deliveries. Along with
the 3 points for quality of life, the Army assigned 1 point each
for two other factors, total cost and impact on small business,
for a total of 5 points. Appendix I provides information on the
scale and scores used by the Army to evaluate the results of each
factor and sub- factor. In its October 1998 evaluation report and
February and March 1999 supplements, the Army reported to us that
Cendant Mobility managed 1,349 shipments involving household
goods, unaccompanied baggage, do- it- yourself (DITY) moves, and
mobile homes during the 12- month test. (See appendix II for the
number of shipments in each category, including
domestic and international volumes.) The Army also reported that
the pilot project earned a passing score on each of the test
factors that the Army measured and deemed the Hunter pilot a
success because the factor points exceeded the minimum threshold
of 3 based on a 5- point scale. According to the Army's analysis,
the Hunter pilot test earned 3.75 points as follows: customer
satisfaction improved by over 11 percent (0. 25 point);
the initial offer to settle claims averaged 9 days (1 point);
100 percent of eligible shipments were delivered directly to the
servicemember without storage in transit (1 point); pilot project
costs exceeded baseline costs by 18.6 percent (0.50 point); and
33 percent of the shipments were awarded to small businesses (1
point). The Army stated that it also considered other factors
generally related to transportation process improvements and did
not rely on test scores alone to determine the pilot a success.
These factors included simplifying the price structure and claims
process, providing pre- audit services, and minimizing government-
unique requirements. Each servicemember worked with one personal
move coordinator who integrated the member's move, providing
point- to- point move management, coordination, and prompt
resolution of all problems. This was in contrast to the current
process, in
B-282406 Page 5 GAO/NSIAD-99-129 Defense Transportation
which a servicemember may need to contact four or five different
offices over the course of a single move. Neither AAA nor we
reviewed these cited benefits because they were not included as
measurable factors in the evaluation plan. Evaluation Weaknesses
Led to Inconclusive Results
We identified a number of shortfalls in the Army's evaluation
methodology and data. Specifically, the methodology for surveying
customer satisfaction and calculating claims settlement time,
percentage of direct deliveries made, and overhead costs did not
allow the Army to obtain and analyze the
data needed to accurately assess pilot results. Furthermore, in
several instances the data collected to support the measurements
did not demonstrate achievement of the Army's reported results.
Most importantly, we could not confirm that customer satisfaction
improved by 11. 5 percent. Further, we could not confirm that all
eligible shipments were delivered directly without incurring in-
transit storage costs, offer of claims settlement time averaged 9
days, and the pilot program cost 18.6 percent more than the
baseline cost. On the other hand, we were able
to confirm that 33 percent of the shipments were awarded by
Cendant Mobility to small business carriers and agents 10 percent
over the Army's minimum threshold. Inconclusive Results for
Customer Satisfaction
The Army's methodology for evaluating customer satisfaction
results was flawed as it allowed servicemembers to be surveyed
multiple times by Cendant Mobility, two survey contractors, and
carriers. In addition to the use of different populations and
respondent rates, the various survey instruments also contained
different questions used to measure the level of customer
satisfaction. Therefore, we could not validate that customer
satisfaction improved by 11.5 percent, as reported by the Army, on
the basis of the conclusions reached by one of the four survey
instruments. Notwithstanding these concerns, other data gathered
in these surveys but not scored by the Army in accordance with its
evaluation plan, provided indicators of customer satisfaction as
well as lessons learned that may be useful to DOD for conducting
other pilots.
To measure customer satisfaction, the Army directed its contractor
(Battelle's Pacific Northwest National Laboratory) to survey
members at Hunter and those making similar moves at Fort Stewart,
Georgia (the baseline). Responses from servicemembers at both
locations would be compared. Subsequently, the Army decided to use
USTRANSCOM's survey, conducted by Electronic Data Systems (EDS),
to evaluate and score the pilot project.
B-282406 Page 6 GAO/NSIAD-99-129 Defense Transportation
Nonetheless, servicemembers were surveyed multiple times, possibly
by as many as four different entities. EDS and Battelle (the
survey contractors), Cendant Mobility (which was contractually
required to survey a sample of customers), and the carrier (for
its own quality control purposes) each may have contacted the same
servicemembers to query them on their move experiences. As a
result, the customer satisfaction results were compromised and
therefore inconclusive. EDS reported that some servicemembers
refused to respond to the survey and that an undetermined number
who did respond gave perfunctory answers. In a
caveat in its report, Battelle stated that it was aware that some
individuals were interviewed multiple times. The Army informed us
that it did not use Battelle's results because the Army found the
draft report difficult to interpret and inconclusive. The Army
also had concerns about a disclaimer on the survey results. 5
Moreover, the contractors' survey results varied significantly.
While EDS reported that 11.5 percent more of the Hunter
respondents were satisfied with their current relocation
experience than the Fort Stewart respondents, Battelle reported
only a 3- percent difference between the two groups of
respondents. 6 We cannot explain all the reasons for these
differences. We do know that the questions asked by each
contractor to determine customer satisfaction were not identical
and that the population and number of respondents in the two
surveys varied significantly. EDS also had a much lower response
rate (53 percent) than Battelle (89 percent). Further, EDS used a
75- percent significance level, while Battelle based its results
using a 95- percent significance level. The latter is the standard
used in social science research. 7
Other data obtained by the surveys but not scored by the Army, in
accordance with its evaluation plan, provides indicators of
customer 5 The Army later learned that the Department of Energy
requires such a disclaimer in all studies performed for government
entities and that the disclaimer had no relevance to the results
reported by Battelle.
6 Battelle also reported that the 3- percent figure was
statistically insignificant, meaning that the variance in
responses from the two groups (personnel at Hunter and at Fort
Stewart) was not great enough to conclude that the move
experiences differed significantly.
7 Significance level is the likelihood that a true population
value may be rejected by a statistical test. In the case of the
two surveys, Battelle used a 95- percent and EDS used a 75-
percent probability that the differences in relocation
satisfaction levels were due to something other than random
occurrence. Battelle found no significant difference in
satisfaction levels, with a 5- percent chance of reaching an
incorrect conclusion. EDS did find a significant difference but
with a 25- percent chance of reaching an incorrect conclusion.
B-282406 Page 7 GAO/NSIAD-99-129 Defense Transportation
satisfaction and highlight areas that could be addressed in future
evaluations. For example, the Battelle survey found that 61
percent of the Hunter respondents were more satisfied with their
most recent (pilot) moving experience than with their prior move.
Eighty- seven percent of these respondents were satisfied or very
satisfied with the carrier's responsiveness. Both surveys also
indicated that servicemembers relocating within the continental
United States were generally more
satisfied with the pilot process than were personnel moving
overseas. The surveys also showed that servicemembers liked some
unique features the pilot offered such as one- on- one counseling
and the simplified claims process. However, few servicemembers
took advantage of many of the
additional services that Cendant Mobility offered, particularly
assistance in house hunting. Cendant Mobility reported to us that
its referrals to real estate companies resulted in the sale of one
residence in the Hunter area and the purchase of five residences
at new duty stations. Moreover, officials of The Military
Coalition, an association that represents 5 million active duty,
retired, reserve, and National Guard personnel and their families,
told us that more could be done to tailor relocation services to
meet unique military needs. Such services include assistance in
finding rental property, because military personnel tend to rent
rather than purchase, and assistance for spouses seeking
employment.
One of the lessons learned is related to access to survey data by
outside reviewers. AAA was unable to compare respondent results
across surveys because the survey contractors, citing privacy act
restrictions, did not grant access to identifier documentation.
The Army stated that in future pilot projects it would ensure that
servicemembers are not subjected to multiple
surveys and would establish a better survey method for determining
customer satisfaction.
Limited Data on Claims Settlement Time Due to weaknesses in the
Army's data collection and analysis, we could not validate that
the initial offer to settle claims took an average of 9 days after
a completed claim form was received by Cendant Mobility.
Furthermore, the Army's methodology was limited to examining the
time between filing a
completed claim form and making the initial settlement offer. This
measure does not capture the entire claims process and therefore
does not measure the extent to which it represents an improvement
over the existing system. The pilot process, however, does offer
benefits to the servicemember in terms of full replacement value
protection. Household effects are insured for their full
replacement value (up to $75,000) rather than their depreciated
value (which is capped at $40,000 under the existing system).
B-282406 Page 8 GAO/NSIAD-99-129 Defense Transportation
The pilot process also requires less work from the servicemember
because the contractor provides estimates on lost and damaged
household goods and pays the servicemember directly. The claims
settlement time measurement is based on the contractual
requirement that Cendant Mobility offer settlement to a member
within 30 days after receiving a claim for damaged or lost
property. Consequently, the Army developed a threshold for claims
settlement of 30 days. Under the
Hunter pilot, servicemembers could appeal to the contractor for
reconsideration by providing additional information or file a
claim with the Army if resolution with the contractor could not be
achieved a process that could take more than 2 years to complete.
While we found that some
claims remained unpaid and unsettled because the claims were put
on hold by the servicemember, the Army reported that no one in the
Hunter pilot filed a claim with the Army claims office as of
January 1999. The Army reported that Cendant Mobility took an
average of 9 days to offer settlement for lost or damaged
property, but neither AAA nor we could confirm this because the
Army had not collected the necessary data. AAA determined, and we
concur, that Cendant Mobility may have taken as long as an average
of 28 days to offer settlement. This figure was based on the
time between the date the claim was signed and the date the member
was offered settlement. The difference between the two estimates
was primarily due to a 19- day period between the date the claim
was signed and the date the contractor reported entering the claim
into its database. Although Cendant Mobility officials stated that
standard company practice
is to enter a claim into the database the same day it is received,
they said delays could be caused by (1) the form being mailed some
time after it is dated, (2) mail delays, and (3) contractor delays
in entering the claim into the database. Army officials stated
that, due to the difficulties in measuring claims settlement time,
they would change their data collection and analysis of this sub-
factor in future pilot programs. Direct Delivery Results May Be
Overstated
We could not confirm that all eligible direct deliveries were made
without in- transit storage because the necessary data was not
collected. The Army's methodology for this sub- factor also did
not allow an assessment of whether the results represented an
improvement over the current process. Direct delivery is defined
by the Army as delivery of a shipment to a servicemember's
residence without storage in transit. A shipment is eligible for
direct delivery only if it has a destination address before the
shipment is offered for delivery. Direct delivery is a contract
performance
B-282406 Page 9 GAO/NSIAD-99-129 Defense Transportation
measurement, and the contractor is paid an incentive for
maintaining a direct delivery rate of over 60 percent.
Consequently, the Army threshold for this measurement is 60
percent.
Neither AAA nor we could confirm that shipments delivered directly
to servicemembers were the only ones eligible for direct delivery
because the necessary information on all shipments was not
collected. That is, other shipments may have been eligible under
the Army criteria; however, AAA and we could not determine which
shipments had addresses prior to being offered for delivery, and
were thus eligible, but ended up requiring in- transit storage.
The Army acknowledged the difficulty in validating
eligible direct deliveries and stated that, in the future, it
would consider using storage- in- transit rates as a test factor.
In the February 5, 1999, supplemental information provided to us,
the Army stated that 74 percent of Army- wide shipments required
in- transit storage, compared with 66 percent requiring temporary
storage during the Army pilot.
Servicemembers benefit from having household goods delivered
directly to their home because the practice limits additional
handling of their property, reducing opportunities for loss and
damage. Also, the government avoids temporary in- transit storage,
which is costly and hard for the Army to control. Household goods
shipments require temporary storage when servicemembers do not
know their new address at the time of shipment, and/ or they move
to their new duty station after completing vacations or military
requirements en route.
Some Overhead Cost Data Was Inaccurate and Incomplete Due to
weaknesses in the Army's methodology for collecting overhead cost
data and the questionable reliability of some of the overhead cost
data that was collected, we could not validate that the additional
cost of the pilot project was 18.6 percent higher than the
baseline cost. As we reported in June 1998, developing overhead
costs has historically been difficult for the
government, including DOD, because such data is often unreliable
and unavailable. (See appendix III for a detailed breakdown of
pilot and baseline costs as reported by the Army and validated by
AAA). Neither AAA nor we could validate some overhead costs.
Specifically, we could not confirm the accuracy of the overhead
costs attributed to MTMC because the Army used data from fiscal
year 1994. MTMC has changed significantly since then due to
downsizing, but AAA could not obtain updated costs from MTMC
representing the current organization to determine the
reasonableness of this overhead cost. AAA and we also agree
B-282406 Page 10 GAO/NSIAD-99-129 Defense Transportation
that a drop in document processing (for claims, invoices, and
inbound shipments) would not result in cost savings to the
government unless the activities that perform these functions,
such as the Defense Finance and Accounting Service, take
corresponding actions to reduce their costs (for example, by
reducing personnel to reflect the workload reduction). AAA and we
also identified other costs not included in the Army's analysis
such as those related to awarding the Hunter contract and the
contractor's use of foreign- flag vessels. 8 The Army could not
provide an estimated dollar value associated with the cost of
developing the contract solicitation, reviewing offers, and
resolving bid protests. While we recognize that some of these
estimated costs may be infrequent or one- time only, they should
be considered. Also, the Army did not include the contractor's use
of foreign- flag vessels into the pilot project's transportation
costs. While individually, costs such as these are probably of
low- dollar value in relation to other costs in the analysis,
collectively, they
could materially affect the difference between the baseline and
pilot costs. According to the Army, some of the reasons for the
18.6 percent higher cost of the pilot project can be attributed to
(1) difficulty in calculating accurate baseline costs, which the
Army believes were understated; (2) low volume
of moves (1, 349), which did not provide enough leverage to
negotiate better rates and discounts; (3) higher than expected
costs of overseas shipments; (4) relatively high unaccompanied
baggage shipment rates, which could have been lower using
negotiated rates; (5) packing allowance for DITY
moves, which was calculated using commercial rates during part of
the 12- month test period, resulting in a higher rate; and (6)
quality moving services, which cost more. In regard to the high
cost of overseas shipments, the Army provided us with additional
details of pilot project costs that showed a 44- percent increase
for overseas household goods shipments and a 2- percent increase
for domestic shipments over the baseline cost.
Small Business Usage Exceeded Army Goals
Both AAA and we validated that Cendant Mobility awarded 33 percent
of the shipments to small businesses 10 percent to carriers and 23
percent to agents. The Army measured the impact of the pilot on
these two types of providers and established a threshold of 23
percent to demonstrate successful participation by small
businesses. 9 The Army based this factor on section 15( g) of the
Small Business Act, which establishes a 8 Foreign- flag vessels
are ships registered in foreign countries. 9 In the motor freight
and transportation industry, firms with annual gross revenues of
$18. 5 million or less are classified as small businesses. See
Federal Acquisition Regulation 19.102.
B-282406 Page 11 GAO/NSIAD-99-129 Defense Transportation
governmentwide goal for participation by small business concerns
at not less than 23 percent of the value of all prime contracts.
The Army reported that 261 (33 percent) of the 790 Hunter
shipments billed during the 12- month pilot were subcontracted by
Cendant Mobility to small carriers and agents. As we reported in
June 1998, Cendant Mobility encountered problems in attracting
small carriers and agents to participate in the Hunter pilot. When
the pilot was implemented, several local agents and the companies
they represented elected not to sign agreements to participate.
Cendant Mobility reported that historically, over 40 percent of
the contracts it awards go to small and small and disadvantaged
businesses. However, as of December 31, 1997, only 14 percent of
the shipments booked by Cendant Mobility during the first 6 months
had been awarded to small businesses. By June 30, 1998, a number
of national van lines and their local agents had reversed their
decision and had signed contracts with Cendant Mobility.
No Comprehensive Strategy for Evaluating Multiple Pilots
DOD will be running multiple pilots concurrently, each with
different goals and objectives and expected outcomes. MTMC
reengineering efforts began in 1994 followed by the Army's in 1996
and the Navy's in 1997. DOD is proposing a fourth pilot with no
specific start date. While USTRANSCOM is tasked with evaluating
the results of these pilots and using the information to recommend
a redesigned Department- wide relocation program, it does not have
a comprehensive strategy for doing so. Specifically, it has not
yet determined how many approaches it will ultimately test, it has
not set the milestones for completing the pilots' evaluations and
implementing an improved process, and it has not assured itself
that a methodologically sound evaluation process is in place to
execute this process.
MTMC's Reengineering Efforts Began in 1994
DOD first proposed reengineering its personal property program in
1994. Specifically, on June 21, 1994, USTRANSCOM directed MTMC
(the Army component of USTRANSCOM and the program manager for
DOD's Personal Property Shipment and Storage Program) to
reengineer the personal property program. On June 15, 1995, the
House Committee on National Security, 10 concurring that DOD must
pursue a higher level of service, directed that DOD undertake a
pilot program to implement commercial business practices and
standards of service. MTMC planned to 10 Now called the House
Committee on Armed Services.
B-282406 Page 12 GAO/NSIAD-99-129 Defense Transportation
award contracts for the new program pursuant to the competitive
acquisition system provisions (10 U. S. C. ch. 137 and its
implementing regulation in the Federal Acquisition Regulation).
Expressing congressional concerns about the impact the competition
system and any
nonstandard commercial business practice requirements might have
on small businesses, the statement of managers in the conference
report accompanying the 1997 DOD Authorization Act directed that
DOD report on the impact of the pilot program on small businesses.
After reviewing the reports, the House Committee on National
Security remained concerned that MTMC's pilot program did not
satisfactorily address issues raised by small moving companies and
directed that the Secretary of Defense establish a working group
of military and industry representatives to develop an alternative
pilot program. Although the working group reached consensus on
many issues, including a set of program goals, it could not reach
agreement on the approach to take for the pilot test.
Consequently, the two sides presented separate proposals. In
November 1996, we reported that in our assessment, MTMC's proposal
met the goals of reengineering the personal property program to a
greater extent than the industry's plan. 11 The MTMC pilot
program, delayed by numerous bid protests, finally started in
January 1999. 12 The pilot includes 50 percent of the moves
originating in North Carolina, South Carolina, and Florida. The
pilot will run concurrently
with the existing MTMC- managed program at installations
representing all military services and the Coast Guard in the
three states and will involve approximately 18,500 annual moves.
The key features of the pilot program include selecting carriers
based on servicemember satisfaction and past performance rather
than simply price; achieving stronger carrier commitment with
long- term contracts; and offering full replacement value
protection and direct claims settlement to users. MTMC's pilot
will run for a 3- year test period (1 year and two 1- year option
periods) which will end in December 2002.
11 Defense Transportation: Reengineering the DOD Personal Property
Program (GAO/NSIAD-97-49, Nov. 27, 1996). 12 Solicitation disputes
in the following decisions: Aalco Forwarding, Inc., et al, B-
277241.8, B-277241. 9, October 21, 1997; Aalco Forwarding, Inc. et
al., B-277241. 12, B-277241.13, December 29, 1997; Aalco
Forwarding, Inc. et al., B-277241. 15, March 11, 1998; Aalco
Forwarding, Inc. et al., B-277241. 16, March 11, 1998; Aalco
Forwarding, Inc., et al., B-277241, 277241. 14, June 8, 1998; and
Aalco Forwarding, Inc., et al., B-277241, B-277241.20, B-277241.
21, July 1, 1998.
B-282406 Page 13 GAO/NSIAD-99-129 Defense Transportation
Hunter Pilot Effort Began in 1996 Separately from the MTMC pilot
program, the Army decided to determine whether the commercial
business practice of outsourcing a non- core
function could alleviate known problems in its personal property
program while increasing quality of life. The pilot began in
February 1996 as a quality- of- life effort to improve the
relocation process and test commercial business practices in a
military environment at Hunter Army Airfield, Savannah, Georgia.
On January 31, 1997, the Army selected PHH
Relocation, a relocation/ move management company (now Cendant
Mobility, Bethesda, Maryland), as the contractor for the pilot. In
July 1997, Cendant Mobility began relocation operations at Hunter,
offering to distribute and manage shipments to transportation
providers. Additionally, the contractor provided point- to- point
move management, personal move counseling and coordination,
assistance in buying/ selling a residence, 24- hour in- transit
visibility of shipments, direct claims settlement with
servicemembers, full replacement value, and a single point of
contact for servicemembers. These services and benefits are not
available under the existing DOD system. The current contract with
Cendant Mobility will end on September 30, 1999.
A Cendant Mobility official stated that the management fees paid
by the Army to the contractor pay for these services and
entitlements. Cendant Mobility also obtained revenues from two
other sources, including
(1) commissions from carriers, agents, and freight forwarders and
(2) referral fees from real estate companies on the purchase and
sale of six residences. The commissions paid for services provided
to carriers, agents, and freight forwarders, including contract
training, contract administration, invoice auditing, and quality
control services. Navy Pilot Effort Began in 1997
The Navy initiated its plans for a separate pilot program in 1997
to test the option of allowing servicemembers to select their
carrier, giving them more control over the relocation process to
meet their specific needs. The pilot, which began in January 1998,
is intended to provide an alternative to, rather than a
replacement of, the current program. The pilot affects a very
small number of military personnel. Presently, the option is being
offered only for shipments originating in the areas of Puget
Sound, Washington; San Diego, California; Norfolk, Virginia; and
Groton, Connecticut. Participation is limited to Navy military
personnel with permanent change of station orders from the
participating sites. Shipments must exceed 3,000 pounds and cost
between $2,500 and $25, 000 (the pilot excludes shipments from
non- temporary storage or warehouses). Servicemembers
B-282406 Page 14 GAO/NSIAD-99-129 Defense Transportation
can select a mover from a list of carriers that are self-
certified as small businesses and approved by MTMC. As of March
26, 1999, 223 Navy servicemembers had selected this option. This
pilot does not have a
specific end date. DOD Pilot Effort Proposed in 1999
On February 12, 1999, DOD stated that it intended to begin a
fourth pilot program, called the Full Service Moving Project, with
certain modifications based on lessons learned from the Army
Hunter pilot. The new pilot will include the National Capital
Region (17,000 annual moves), Georgia (26,000 annual moves), and
North Dakota (2,000 annual moves), and will involve all military
services and the Coast Guard. Like the Army pilot, it is intended
to outsource a non- core function and alleviate known problems in
the DOD personal property program while improving the quality of
life associated with the relocation of servicemembers and their
families. While it is projected to begin in calendar year 1999,
presently, this pilot does not have an official start date.
Unresolved Issues in Evaluating Pilot Programs
USTRANSCOM is in the process of finalizing its evaluation plan for
the ongoing and planned pilots. It is proposing to evaluate the
pilots on the same three factors that the Army used in its
evaluation of the Hunter pilot, except that the factors would be
defined differently and would include a more expansive point
scale. Unresolved issues, as of February 1999, included the
development of a method that would recognize the unique
characteristics and/ or process improvements of each pilot program
and the validation of the baseline indirect costs that would be
used for each pilot program. The evaluation plan does not
currently include an evaluation of the Hunter pilot, but
USTRANSCOM used lessons learned in the Hunter
pilot to develop the plan. We provided comments to USTRANSCOM in
this process and, among other things, encouraged the command to
seek expert methodological advice before finalizing the evaluation
plan to enhance the quality of its assessment. Conclusions We
support the use of pilots as a means to test new concepts.
Although this process has been ongoing since the early 1990s, DOD
is not yet in a position to determine what changes it needs to
make to the process. Also, thus far, DOD has not decided how many
approaches it will ultimately test and has not developed
evaluation plans that will yield accurate and credible data.
Lessons learned from the Hunter pilot do provide useful
information for
B-282406 Page 15 GAO/NSIAD-99-129 Defense Transportation
evaluating the three other pilots ongoing and planned. While
USTRANSCOM is in the process of developing a strategy to conduct
this evaluation, it has not yet resolved a number of issues,
including timelines for completing the analysis and implementing
an improved process and a
methodology for the evaluation. Recommendations We recommend that
the Secretary of Defense direct the Commander in Chief,
USTRANSCOM, to develop a comprehensive strategy for evaluating
each of the pilot approaches that includes (1) tests of a finite
number of approaches; (2) timelines for completing the analysis
and for implementing a new personal property process; and (3)
milestones for finalizing a methodologically sound evaluation
plan, including specific criteria for assessing each pilot's
attributes and validating the baseline indirect costs to be used
for each pilot program. To facilitate this process and to enhance
the quality of the assessment, we also recommend that DOD seek
expert methodological advice before finalizing the evaluation
plan. Agency Comments and Our Evaluation
In written comments on a draft of this report, DOD stated that it
generally concurred with the report and its recommendations. DOD
tasked USTRANSCOM to evaluate the results of all the pilots and
provide a recommendation for a DOD- wide program to implement for
the long term. DOD stated that USTRANSCOM is in the process of
retaining a contractor to independently validate success criteria
for a reengineered DOD personal property program, develop a
comprehensive strategy for evaluating each of the pilot
approaches, and support the development of recommendations and
alternative courses of action. USTRANSCOM anticipates having the
contractor commence its efforts by June 1999. DOD's comments are
presented in appendix IV. DOD also provided technical comments and
we revised our report to reflect them where appropriate.
Scope and Methodology
To validate the results and savings of the Hunter pilot, we
reviewed the Army's plans to determine how it would evaluate the
pilot's results. We also reviewed the Army's October 1998 report
and the February and March 1999 supplements detailing the 12-
month test results. We visited both Hunter Army Airfield and the
transportation division at Fort Stewart, Georgia. We contacted
Hunter contractor officials and representatives from the two
B-282406 Page 16 GAO/NSIAD-99-129 Defense Transportation
survey firms hired by the Army and USTRANSCOM to measure customer
satisfaction. We also examined the documents and data compiled by
the AAA to validate the accuracy of the Hunter results.
We met with Army, Navy, MTMC, and USTRANSCOM officials and
reviewed documents to determine the status and goals of other DOD
personal property pilot programs. We also obtained statistics on
actual use by servicemembers of the Navy's pilot as of March 1999.
We did not request statistics on other pilots because the MTMC
pilots began in January 1999
and the DOD- proposed Full Service Moving Project is still in the
planning stage. We also met with USTRANSCOM officials and reviewed
evaluation plans to determine how DOD proposes to make changes to
its current personal property program using lessons learned from
the pilots. We met and discussed matters related to DOD pilots and
efforts to improve the existing personal property program with
officials from the Office of the Deputy Under Secretary of Defense
(Logistics- Transportation Policy), Washington, D. C.; USTRANSCOM,
Scott Air Force Base, Illinois; the Office
of the Army Deputy Chief of Staff for Logistics, Washington, D.
C.; MTMC, Falls Church, Virginia; the Naval Supply Systems
Command, Mechanicsburg, Pennsylvania; Hunter Army Airfield,
Savannah, Georgia; and Fort Stewart, Georgia. In addition, we
discussed the survey results with officials from Battelle's
Pacific Northwest National Laboratory and from Electronic Data
Systems.
We conducted our review between August 1998 and April 1999 in
accordance with generally accepted government auditing standards.
We are sending copies of this report to the Honorable William S.
Cohen, Secretary of Defense; the Honorable Louis Caldera,
Secretary of the Army; General Charles T. Robertson, Jr.,
Commander in Chief, USTRANSCOM; Lieutenant General John G. Coburn,
Deputy Chief of Staff, Logistics, Department of the Army; Major
General Mario F. Montero, Jr., Commander, MTMC; and the Honorable
Jacob J. Lew, Director, Office of Management and Budget. We are
also sending copies to other congressional members who expressed
interest to us in DOD's efforts to improve its personal
property program, including the Honorable Fred Thompson, United
States Senate, and the Honorable Neil Abercrombie, Duncan Hunter,
Martin T. Meehan, and James M. Talent, House of Representatives.
We will also make copies available to others upon request.
B-282406 Page 17 GAO/NSIAD-99-129 Defense Transportation
Please contact me on (202) 512- 8412 if you or your staff have
questions concerning this report. Major contributors to this
report are listed in appendix V. David R. Warren, Director Defense
Management Issues
B-282406 Page 18 GAO/NSIAD-99-129 Defense Transportation
List of Congressional Committees The Honorable John W. Warner
Chairman The Honorable Carl Levin Ranking Minority Member
Committee on Armed Services United States Senate
The Honorable Ted Stevens Chairman The Honorable Daniel K. Inouye
Ranking Minority Member Subcommittee on Defense Committee on
Appropriations United States Senate
The Honorable Floyd D. Spence Chairman The Honorable Ike Skelton
Ranking Minority Member Committee on Armed Services House of
Representatives
The Honorable Herbert H. Bateman Chairman The Honorable Solomon P.
Ortiz Ranking Minority Member Subcommittee on Military Readiness
Committee on Armed Services House of Representatives
The Honorable Jerry Lewis Chairman The Honorable John P. Murtha
Ranking Minority Member Subcommittee on Defense Committee on
Appropriations House of Representatives
Page 19 GAO/NSIAD-99-129 Defense Transportation
Page 20 GAO/NSIAD-99-129 Defense Transportation
Contents Letter 1 Appendix I Thresholds and Scores Used by the
Army for
Hunter Pilot Evaluation Factors
22 Appendix II Types and Number of Shipments Made During the
Hunter Pilot Test
23 Appendix III Hunter Pilot and Baseline Costs Calculated by the
Department of the Army (Office of the Deputy Chief of Staff
for Logistics) 24
Appendix IV Comments From the Department of Defense
25 Appendix V Major Contributors to This Report
28
Contents Page 21 GAO/NSIAD-99-129 Defense Transportation
Related GAO Products 32
Abbreviations
AAA Army Audit Agency DITY do- it- yourself DOD Department of
Defense EDS Electronic Data Systems MTMC Military Traffic
Management Command USTRANSCOM U. S. Transportation Command
Page 22 GAO/NSIAD-99-129 Defense Transportation
Appendix I Thresholds and Scores Used by the Army for Hunter Pilot
Evaluation Factors Ap pe ndi x I
Factors/ definitions Thresholds Scores
Quality of life/ customer satisfaction rating: percent of Hunter
respondents who were more satisfied than Fort Stewart respondents
(control group). 25% and above 1.00 point 20- 24% 0.75 points 15-
19% 0.50 points 10- 14% 0.25 points Below 10% Unsuccessful Quality
of life/ claims settlement time: time period between when a claim
was received and when the contractor offered settlement. 0- 10
days 1.00 point
11- 15 days 0.75 points 16- 20 days 0.50 points 20- 30 days 0.25
points Above 30 days Unsuccessful Quality of life/ percentage of
eligible deliveries: percent of shipments delivered to residence
without storage in- transit. 91- 100% 1. 00 point
81- 90% 0.75 points 71- 80% 0.50 points 61- 70% 0.25 points 60%
and below Unsuccessful Total cost: extent total contract cost
exceeded total baseline cost. 0% 1. 00 point
1- 10% 0. 75 points 11- 20% 0.50 points 21- 30% 0.25 points More
than 30 % Unsuccessful Impact on small business: percent of small
agents and carriers used. 20% or more 1.00 point
27- 28% 0.75 points 25- 26% 0.50 points 23- 24% 0.25 points Less
than 23% Unsuccessful
Page 23 GAO/ NSIAD 99- 129 Defense Transportation
Appendix II Types and Number of Shipments Made During the Hunter
Pilot Test Appe n di x I I
Shipment type Shipments initiated and recorded as completed
Household goods (intrastate Georgia 51 Household goods
(interstate continental U. S., including Alaska) 418 Household
goods (overseas Hawaii) 29 Household goods (overseas Germany) 79
Household goods (overseas Korea) 50 Household goods (overseas
other) 47 Unaccompanied baggage (overseas all locations) 378 Do-
it- yourself (DITY) moves 292 Mobile homes 5
Total 1, 349
Page 24 GAO/NSIAD-99-129 Defense Transportation
Appendix III Hunter Pilot and Baseline Costs Calculated by the
Department of the Army (Office of the Deputy Chief of Staff for
Logistics) Ap pe ndi x I I I
a Army Audit Agency validated all costs except those under $5,
000, concluding that such amounts were immaterial. b Military
Traffic Management Command (MTMC). Source: Office of the Deputy
Chief of Staff for Logistics, Department of Army.
Cost elements a Estimated pilot costs Estimated baseline costs
Transportation costs
Transportation $3,005,229 $2,102, 384 Accessorials 25,767 30, 655
Storage 210,706 211, 655
Total transportation costs $3,241,702 $2,344, 694 Claims paid by
Army
Claims paid by Army $0 $66, 268 Overhead costs
Personnel $79,802 $221, 721 Management price 132,700 0 MTMC
headquarters b 0 63, 510 Automation 0 52, 612 Voucher processing
4,614 90, 658 Inbound processing 0 46, 834 Claims processing 0 11,
510 Pay for performance 14,756 0 Building overhead 28,321 43, 286
Telephone and copier 1,281 10, 106 Consumables 3,760 5, 189
Total overhead costs $265,234 $545, 426 Total costs $3,506,936
$2,956, 388
Pilot costs Baseline
costs Difference Percent difference
Percent pilot costs exceeded baseline costs $3,506,936 $2,956,388
$550, 548 18.6
Page 25 GAO/NSIAD-99-129 Defense Transportation
Appendix IV Comments From the Department of Defense App en di x I
V
Note: GAO comment supplementing those in the report text appear at
the end of this appendix.
See comment 1.
Appendix IV Comments From the Department of Defense
Page 26 GAO/NSIAD-99-129 Defense Transportation
Appendix IV Comments From the Department of Defense
Page 27 GAO/NSIAD-99-129 Defense Transportation
The following are GAO's comments on the Department of Defense's
letter dated May 11, 1999.
GAO Comment 1. As stated in the report, we could not confirm the
customer satisfaction survey results. Specifically, we could not
confirm that all survey results indicated higher levels of
customer satisfaction because of weaknesses in the survey
methodology. Methodological weaknesses that compromised survey
results included using different populations and respondent rates,
using various survey instruments with different questions, and
surveying servicemembers multiple times. In addition, one of the
two survey contractors found no statistically significant
difference between responses from Hunter and Fort Stewart. Our
assessment of customer satisfaction results, as indicated in the
report, was based on the Army's criteria set forth in its
evaluation plan. Because the Army used the results of one question
to score the pilot a success, we also used it as the basis for our
assessment. We agree, however, that other
data obtained by the surveys provided indicators of customer
satisfaction and highlighted areas that could be addressed in
future evaluations. We agree that the bulk of cost elements could
be validated and revised our report to show that some overhead
cost data was inaccurate and incomplete.
Finally, regarding our assessment of the claims process, we did
not acknowledge a significant improvement in the process because
the Army measured only one aspect of it, as indicated in the
report. In addition, on the basis of the Army's criteria set forth
in its evaluation plan for assessing this factor, we could not
confirm the time it took to offer claims settlement to
servicemembers. We also agree that the claims process offered
benefits not previously available to the servicemembers.
Page 28 GAO/NSIAD-99-129 Defense Transportation
Appendix V Major Contributors to This Report Appe nd i x V
National Security and International Affairs Division, Washington,
D. C.
Charles I. Patton, Jr., Associate Director Nomi R. Taslitt,
Assistant Director Robert L. Self, Evaluator- in- Charge Marc J.
Schwartz, Senior Evaluator Jacqueline E. Snead, Senior Evaluator
Richard R. Irving, Evaluator Arthur L. James, Jr., Mathematical
Statistician
Office of the General Counsel, Washington, D. C.
John G. Brosnan, Assistant General Counsel Norfolk Field Office
Daniel A. Omahen, Senior Evaluator
John R. Beauchamp, Senior Evaluator
Page 29 GAO/NSIAD-99-129 Defense Transportation
Page 30 GAO/NSIAD-99-129 Defense Transportation
Page 31 GAO/NSIAD-99-129 Defense Transportation
Page 32 GAO/NSIAD-99-129 Defense Transportation
Related GAO Products Defense Transportation: Efforts to Improve
DOD's Personal Property Program (GAO/T-NSIAD-99-106, Mar. 18,
1999).
Defense Transportation: The Army's Hunter Pilot Project to
Outsource Relocation Services (GAO/NSIAD-98-149, June 10, 1998).
Defense Outsourcing: Better Data Needed to Support Overhead Rates
for A- 76 Studies (GAO/NSIAD-98-62, Feb. 27, 1998).
Defense Transportation: Reengineering the DOD Personal Property
Program (GAO/NSIAD-97-49, Nov. 27, 1996).
Household Goods: Administrative Changes Would Improve DOD's Do-
ItYourself Moving Program (GAO/NSIAD-94-226, Sept. 27, 1994).
DOD Commercial Transportation: Savings Possible Through Better
Audit and Negotiation of Rates (GAO/NSIAD-92-61, Dec. 27, 1991).
Household Goods: Competition Among Commercial Movers Serving DOD
Can Be Improved (GAO/NSIAD-90-50, Feb. 12, 1990).
(709365) Let t er
Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary,
VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013
or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists.
For information on how to access GAO reports on the INTERNET, send
an e- mail message with info in the body to: info@ www. gao. gov
or visit GAO's World Wide Web Home Page at: http:// www. gao. gov
United States General Accounting Office Washington, D. C. 20548-
0001
Official Business Penalty for Private Use $300
Address Correction Requested Bulk Rate
Postage & Fees Paid GAO Permit No. GI00
*** End of document. ***