Defense Transportation: The Army's Hunter Pilot Project to Outsource
Relocation Services (Letter Report, 06/10/98, GAO/NSIAD-98-149).
Pursuant to a legislative requirement, GAO provided information on the
Army's Hunter pilot program, an alternative approach for providing
relocation services for its personnel stationed at Hunter Army Airfield,
Savannah, Georgia, focusing on: (1) data collected to date; (2) GAO's
observations on how the Army plans to evaluate the data; and (3) other
pilot tests that are under way or planned in the Department of Defense
(DOD).
GAO noted that: (1) through the first 6 months of operation, the Hunter
pilot contractor had received and processed nearly 800 requests for
transportation and relocation services, arranged for transportation of
nearly 600 personal property shipments, and assisted Army members
initiate another 200 do-it-yourself moves; (2) because many of the
shipments had not been completed at the time of GAO's review, the Army
had paid for only about 200 of these shipments, at a cost of about
$500,000, but it was actively collecting data on all the
factors--quality of life, cost, and impact on small businesses--it was
planning to use in its evaluation of the pilot; (3) however, at the time
GAO completed its work, the Army had not determined what it would
consider successful within each factor or how much weight each factor
would have in determining overall success; (4) specifically, the Army
had not determined criteria to judge the success or failure of the pilot
to help it assess whether the pilot was performing better or worse than
the existing program; (5) making such determinations before pilot
program data are analyzed would be important to enhancing the
credibility of the Army's assessment and for use in making comparisons
with other pilot programs that are under way or planned in DOD; (6)
these include pilots under way by the Navy or planned by the Military
Traffic Management Command; (7) also, DOD and the Army are considering
expanding the pilot now being tested at Hunter Army Airfield to other
DOD sites; (8) subsequently, in response to GAO's draft report, the Army
provided it with new information on how it will determine the results of
the pilot; (9) the information, with some minor exceptions, represents
important steps toward formulating a sound evaluation plan; (10)
however, the evaluation method has not yet been finalized or made a
formal part of the evaluation plan, and questions remain about the
definition of terms and small business group measurements; and (11) GAO
recognized that further issues could be identified as the evaluation
method is refined and finalized.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-98-149
TITLE: Defense Transportation: The Army's Hunter Pilot Project to
Outsource Relocation Services
DATE: 06/10/98
SUBJECT: Defense procurement
Department of Defense contractors
Freight transportation operations
Privatization
Small business contractors
Federal agency reorganization
Program evaluation
Evaluation methods
Household goods
IDENTIFIER: Army Hunter Pilot Program
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved. Major **
** divisions and subdivisions of the text, such as Chapters, **
** Sections, and Appendixes, are identified by double and **
** single lines. The numbers on the right end of these lines **
** indicate the position of each of the subsections in the **
** document outline. These numbers do NOT correspond with the **
** page numbers of the printed product. **
** **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced. Tables are included, but **
** may not resemble those in the printed version. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
Cover
================================================================ COVER
Report to Congressional Committees
June 1998
DEFENSE TRANSPORTATION - THE
ARMY'S HUNTER PILOT PROJECT TO
OUTSOURCE RELOCATION SERVICES
GAO/NSIAD-98-149
Army's Hunter Pilot Project
(709237)
Abbreviations
=============================================================== ABBREV
DOD - Department of Defense
MTMC - Military Traffic Management Command
USTRANSCOM - U.S. Transportation Command
Letter
=============================================================== LETTER
B-278548
June 10, 1998
Congressional Committees
Because of long-standing concerns and problems associated with the
relocation of military personnel, the Department of the Army began a
pilot project in July 1997 to test an alternative approach for
providing relocation services for its personnel stationed at Hunter
Army Airfield, Savannah, Georgia. The statement of managers in the
conference report\1 on the Department of Defense Appropriations Act,
1997, directed us to validate the results and savings achieved from
this and any other personal property pilot program. While the
conference report originally envisioned us reporting and evaluating
the Army's experience under this pilot program during fiscal year
1997, the late start of the pilot in the last quarter of the fiscal
year provided limited data for assessment purposes. Accordingly, we
are providing an interim report. As discussed with your offices,
this report provides (1) information on the data collected to date
and (2) our observations on how the Army plans to evaluate the data.
We are also providing information on other pilot tests that are
underway or planned in the Department of Defense (DOD).
--------------------
\1 House Report 104-863, Sept. 28, 1996.
BACKGROUND
------------------------------------------------------------ Letter :1
DOD has long been concerned about the quality of service it provides
military personnel and their families when they relocate. DOD
reported that in 1996 it paid roughly $2.8 billion to move 800,000
families. Despite the fact that it moves more household effects than
any U.S. corporation, DOD has found that its system has provided
military personnel some of the worst service in the Nation. DOD has
reported that 25 percent of its moves have resulted in damage claims,
compared to 10 percent of the moves undertaken for the private
sector. Further, best-in-class movers have had customer satisfaction
rates of 75 percent, compared with 23 percent for DOD personnel
moves.
The Army recognizes that in today's environment of limited resources,
it cannot continue to move personal property in the usual way.
Currently, the personal property program is run centrally by the
headquarters office of the Military Traffic Management Command (MTMC)
and relies almost exclusively on more than 1,250 motor van carriers
and forwarders to handle its domestic and international moving needs.
The current system is not specifically designed to select carriers on
the basis of quality service; rather, carriers offering a minimally
acceptable level of quality are generally selected based on the
lowest rates. DOD offers carriers meeting the low rate on each route
the opportunity to share in any traffic moving on that route.
Further burdening the current system are "paper companies," which are
affiliates of larger or parent companies established to increase
shares in DOD traffic. These companies increase DOD's administrative
activities but do not add hauling and storage capacity.
To test whether commercial business practices can help alleviate
these problems and to acquire quality service for its military
personnel and families during the relocation process, the Army began
developing a pilot project in February 1996 to outsource
transportation and relocation services. The company selected for
outsourcing was expected to offer Army personnel and their families a
wide range of relocation services. In addition to packing,
transporting, and storing household goods, the company was to provide
other services that included entitlement counseling; point-to-point
move management; help with selling, buying, or renting a home; and
property management and mortgage services.
On January 31, 1997, the Army selected PHH Relocation, a
relocation/move management company, now named Cendant Mobility, of
Bethesda, Maryland, from among 11 offerors as the contractor for its
pilot project.\2 The evaluated price for 3 years--a base year and two
1-year option periods--was $22.5 million.
The pilot program involves Army personnel stationed at Hunter Army
Airfield moving household goods, unaccompanied baggage, mobile homes,
and boats to other authorized locations, including Georgia, other
states in the continental United States, Alaska, Hawaii, and foreign
countries. It excludes privately owned vehicle shipments and
shipments moving into nontemporary storage. Prior to the pilot
project, Hunter was part of the MTMC-managed program at Fort Stewart,
Georgia, although Hunter staff arranged the moves.
--------------------
\2 Although the Army awarded the contract on January 31, 1997, a
protest to our office delayed the start of operations. The protest
was denied and operations began July 1, 1997. (See decision in
Suddath Van lines, Inc; The Pasha Group, B-274285.2, B-274285.3, May
19, 1997.)
RESULTS IN BRIEF
------------------------------------------------------------ Letter :2
Through the first 6 months of operation, Cendant Mobility, the Hunter
pilot contractor, had received and processed nearly 800 requests for
transportation and relocation services, arranged for transportation
of nearly 600 personal property shipments, and assisted Army members
initiate another 200 Do-It-Yourself moves. Because many of the
shipments had not been completed at the time of our review, the Army
had paid for only about 200 of these shipments, at a cost of about
$500,000, but it was actively collecting data on all the
factors--quality of life, cost, and impact on small businesses--it
was planning to use in its evaluation of the pilot. However, at the
time we completed our work, the Army had not determined what it would
consider successful within each factor or how much weight each factor
would have in determining overall success. Specifically, the Army
had not determined criteria to judge the success or failure of the
pilot to help it assess whether the pilot was performing better or
worse than the existing program. Making such determinations before
pilot program data are analyzed would be important to enhancing the
credibility of the Army's assessment and for use in making
comparisons with other pilot programs that are underway or planned in
DOD. These include pilots underway by the Navy or planned by MTMC.
Also, DOD and the Army are considering expanding the pilot now being
tested at Hunter Army Airfield to other DOD sites.
Subsequently, in response to our draft report, the Army provided us
with new information on how it will determine the results of the
pilot. The information, with some minor exceptions, represents
important steps toward formulating a sound evaluation plan. However,
the evaluation method has not yet been finalized or made a formal
part of the evaluation plan, and questions remain about the
definition of terms and small business group measurements. We
recognize that further issues could be identified as the evaluation
method is refined and finalized.
INFORMATION PROVIDED ON THE
CONTRACTOR'S ACTIVITY AT HUNTER
ARMY AIRFIELD THROUGH
DECEMBER 31, 1997
------------------------------------------------------------ Letter :3
Contract operations at Hunter Army Airfield began on July 1, 1997.
The contractor, Cendant Mobility, receives the authorizations for
relocation from Army members and arranges to move household goods,
unaccompanied baggage, and mobile homes from Hunter to worldwide
destinations. It also assists a member who may want to make a
Do-It-Yourself personal property move.
Data provided by the Army on the first 6 months of contractor
operations indicated that Cendant had received and processed 793
relocation authorizations and arranged (booked) 581 moves. These
moves included 244 domestic (within the continental United States)
household goods shipments, 227 international household goods
shipments, 106 international unaccompanied baggage shipments, and 4
mobile home shipments. Cendant also assisted members in initiating
181 Do-It-Yourself moves. (See table 1.)
Table 1
Contractor Activity Through December 31,
1997
Septembe
July August r October November December Tota
1997 1997 1997 1997 1997 1997 l
-------------- -------- -------- -------- -------- -------- -------- ----
Orders 218 102 184 100 100 89 793
received
authorizing
relocation
Shipments 112 91 114 89 71 104 581
booked
(total)
Domestic 53 46 28 42 32 43 244
household
goods
shipments
International 11 26 11 20 15 23 106
household
goods
shipments
International 46 49 44 26 24 38 227
unaccompanied
baggage
shipments
Mobile homes 2 1 0 1 0 0 4
Do-It- 53 19 34 21 21 33 181
Yourself
moves
--------------------------------------------------------------------------------
Source: Department of the Army.
THE ARMY'S PLANS TO EVALUATE
THE PILOT PROJECT NEED TO BE
FURTHER REFINED
------------------------------------------------------------ Letter :4
The Army has developed an evaluation plan to facilitate data
collection on quality of life, total cost, and impact on small
business; define agency responsibilities; and explain which data will
be used for comparative purposes. It has indicated that data will be
collected for a minimum of
12 months. Quality of life is defined primarily in terms of customer
satisfaction, measured principally from customer surveys at both
Hunter and Fort Stewart. Additionally, quality of life will be
assessed in terms of claims for loss and damage based on incidence,
amounts, and settlement times. Total costs include transportation,
storage, administration, and overhead. Transportation and storage
costs under the pilot will be compared with what the Army would have
paid under the MTMC program. Administrative and overhead costs for
the pilot will be compared with the cost of doing business under the
MTMC program. Claims costs under the pilot will be compared with
claims costs on shipments made in calendar year 1995. Impact on
small business will be based on the percentage of moves, tonnage, and
dollar amount awarded to small and disadvantaged businesses under the
pilot compared with calendar year 1995 experience. The Army Audit
Agency has been tasked to validate all the data used to conduct the
Army's evaluation.
Prior to commenting on a draft of this report, the Army had not
explained what constituted success for the pilot, how much weight
each factor (quality of life, total cost, and impact on small
business) should have in determining overall success, and whether
there were thresholds that specific factors should attain to be
considered successful. Such advance determinations are typically
part of a sound study methodology to enhance the credibility of the
results and to avoid any perception of bias. In response to our
draft report, DOD provided us with additional information it had
received from the Army. The information provides thresholds for
determining the success of the overall Hunter pilot and the relative
importance of each evaluation factor. While the Army has not
finalized its evaluation method, the new information clarifies the
description of the three evaluation factors, establishes a
point-scoring system and threshold for each factor and subfactor, and
prescribes the relative weight of each factor for determining the
pilot success. We will continue to evaluate the Army's evaluation
method as it is finalized and have included the new information in
the following sections of this report.\3
--------------------
\3 The full text of the Army's new information is contained in
appendix I.
EVALUATING THE FACTORS
---------------------------------------------------------- Letter :4.1
We recognize that the relationship among the factors--quality of
life, costs, and impact on small business--complicates the process of
setting absolute targets for each factor or for assessing the
trade-offs among the factors. However, the Army may consider certain
levels critical for certain factors (e.g., the budget may limit the
amount of increased costs that can be paid for increased quality of
life levels, measured by customer satisfaction survey and claims
experience). On the other hand, there may also be certain customer
satisfaction levels (e.g., those similar to the commercial market)
that the Army expects to obtain. Therefore, making these kinds of
decisions early in the evaluation process should provide the Army a
more definitive basis for determining whether the pilot project is
successful.
Accordingly, the Army has taken action in response to our draft
report recommendation that it determine what results it requires to
judge its pilot a success. The Army's evaluation plan indicates that
overall, quality of life is the most important of the evaluation
factors. The additional information recently provided suggests that
the Army will use a point-scoring system to determine the relative
weight of each factor and subfactor, as shown in table 2.
Table 2
Evaluation Factors and Subfactors and
Relative Weights Assigned to Each
Subfacto Factor
Evaluation factor/subfactor r weight weight
-------------------------------------------------- -------- --------
Quality of life 4 points
Customer satisfaction rating 1 point
Claims settlement time 1 point
Percentage of direct deliveries 1 point
Congressional/White House inquiries 1 point
Total cost 1 point
Impact on small business 1 point
======================================================================
Total 6 points
----------------------------------------------------------------------
Source: DOD.
If the scores of the pilot total 3.75 or more, the Army will consider
the pilot a success; if below 3.75, or if any of the individual
factor scores falls below its threshold, the Army will consider it
unsuccessful.
QUALITY OF LIFE
-------------------------------------------------------- Letter :4.1.1
Quality of life is the key element the Army is considering in
assessing the Hunter pilot. The Army's most recent information
indicates that quality of life will be based on results of four
subfactors--customer satisfaction, claims settlement time, percentage
of direct deliveries, and number of congressional or White House
inquiries concerning dissatisfaction of Hunter test participants.
Customer satisfaction will be determined by examining responses from
two surveys: (1) Cendant will obtain comments made by all pilot test
members after their moves and (2) an independent contractor will
survey both the pilot test members and those making similar moves at
Fort Stewart under the MTMC program. Responses from both surveys
will be compared. Although the surveys will be quite detailed, the
single question for determining pilot test success will be " How
satisfied are you with your relocation moving experience?"
The Cendant survey consists of 12 questions, the key question being:
"How satisfied are you with your relocation moving experience?"
Responses are used to determine the amount of "performance" pay that
the contractor can earn in accordance with the contract terms. Data
provided to us for the first 6 months of activity, covering 88
responses where the response choices were "excellent," "very good,"
"good," "fair," and "poor," showed that 98 percent of the members had
received "excellent," or "very good," service. None indicated
dissatisfaction with their overall relocation moving experiences.
The Army Audit Agency, however, has not yet completed its validation
of those responses.
The independent contractor survey is more extensive and probing than
the Hunter contractor survey, consisting of about 100 questions. For
example, it includes a number of questions on the members' previous
moves and services used. The key question, however, is: "How
satisfied are you with your relocation moving experience?" When the
Hunter participants are at least 10 percent more satisfied than the
control group, which are the personnel moving from Fort Stewart
during the same time as the Hunter personnel, the customer
satisfaction rating subfactor will be deemed successful. While there
are a number of possible responses to this question ("very
satisfied," "satisfied," "neither satisfied nor dissatisfied,"
"dissatisfied," or "very dissatisfied"), it is not clear how "more
satisfied" will be defined.
Results to date are limited because the independent contractor did
not start until October 5, 1997, and it needed time to test and
revise the survey before proceeding with the full survey process.
The survey results are expected to be reported to the Army in October
1998.
The second subfactor related to quality of life is claims settlement
time. This is based on the contract requirement that the contractor
offer settlement to a member within 30 days of claim receipt for
damage or lost property. Consequently, the threshold is 30 days.
Information on claims data is preliminary and may take some time to
fully accumulate since servicemembers may file claims to the Army up
to
2 years from delivery of property. As of December 31, 1997, however,
over 500 shipments had been arranged by the contractor, and 33 of the
40 claims filed by customers against the contractor had been closed.
The average time it took the contractor to settle claims was 13 days,
the average amount claimed was $919.18, and the average amount
settled was $455.15. To date, none of the Hunter servicemembers has
exercised his or her option to file a claim with the Army.
The third subfactor is direct delivery, which is defined as
delivering a shipment to a servicemember's residence without storage
intransit. Direct delivery is a contract performance measure in
which the contractor is paid an incentive for maintaining a direct
delivery rate above 60 percent. Consequently, the threshold is 60
percent. Of the shipments delivered during the first 6 months of
contractor operations, 35 percent were direct deliveries.
The fourth subfactor under quality of life will be the number of
congressional or White House inquiries that the Army Office of Deputy
Chief of Staff for Logistics will receive concerning dissatisfaction
of a Hunter test participant. The established threshold is one
inquiry.
TOTAL COST
-------------------------------------------------------- Letter :4.1.2
Total cost is another factor that will be used in the pilot
evaluation, and the carrier industry has expressed concerns that the
pilot program could cost the Army more than it would incur under the
MTMC program. According to the Army, total cost is the sum of (1)
transportation, which includes line-haul transportation,
accessorials, and storage prior to delivery; (2) claims, those costs
paid by the Army; and (3) management cost, which includes pay for
performance, management price, and overhead and administrative
expenses. The Army has indicated that the Hunter pilot will be
determined successful if its total cost does not exceed the baseline
cost, which is the cost, including management cost, that the Army
would have incurred had the shipments moved at MTMC program rates, by
more than 30 percent.
Reported pilot costs are still preliminary. To date, the only
validated pilot cost data relate to transportation, storage,
accessorial services, claims, pay for performance, and management
fees. Through late January 1998, the Army paid the contractor
$503,835 for 203 moves--150 full service shipments and 53
Do-It-Yourself moves (see table 3). The costs for these same
shipments and moves, priced at the current MTMC program rates, are
being developed but are still being validated.
Table 3
Test Program Reported Costs (Based on
payments made to the contractor through
January 1998)
Amount
Element of cost paid
---------------------------------------------------------- ----------
Transportation and related services
Transportation $461,372
Accessorial services 5,594
Storage 10,913
Claims
Claims paid by the Army 0
Management cost
Pay for performance 3,231
Management fees 22,725
Overhead/Administration costs \a
======================================================================
Total $503,835
----------------------------------------------------------------------
\a Data are still being collected and validated by the Army.
Source: Department of the Army.
Developing overhead costs has historically been difficult in
government agencies, including DOD, because such data are often
unreliable and unavailable.\4 The Army, however, is collecting data
on administrative or overhead costs, which include costs for
personnel, supplies, claims processing, voucher processing, inbound
shipment processing, electronic automation, excess cost processing,
building overhead, telephone and copier, and MTMC overhead. Even
though overhead and administrative costs for both the pilot and the
comparable MTMC program are being developed, they have not yet been
fully validated by the Army Audit Agency. The Agency's
responsibilities include providing the Army program managers with
guidance in capturing administrative and overhead costs, guidance in
statistical sampling techniques, and the validation of the test and
baseline data.
--------------------
\4 Defense Outsourcing: Better Data Needed to Support Overhead Rates
for A-76 Studies (GAO/NSIAD-98-62, Feb. 27, 1998).
IMPACT ON SMALL
BUSINESSES
-------------------------------------------------------- Letter :4.1.3
The impact of the Army's pilot on small businesses has been a source
of concern to the carrier industry. The Army is collecting data on
the percentage of moves, tonnage, and dollar amounts awarded to small
and disadvantaged businesses, including carriers, forwarders, and
their agents, under the pilot program. It will then compare that
data with calendar
year 1995 Hunter data.
The Hunter contractor is a relocation firm who subcontracts or who
has written agreements with various service providers. Some of these
providers are carriers and some are carriers' agents. The Army has
indicated that it will measure impact on both types of providers and
has established a threshold of 23 percent to consider the pilot small
business participation a success. It is not clear, however, which
small business groups--carriers, agents, or both--on which
measurements will be taken to establish small business participation
in the pilot.
As of December 31, 1997, 74 of 533, or 14 percent, of the shipments
booked by Cendant during the first 6 months were awarded to small
businesses.\5 In comparison, 32 percent of the calendar year 1995
shipments booked by Hunter under the MTMC program were awarded to
small businesses. The data are still being verified by the Army
Audit Agency.
According to Cendant, moving industry opposition to DOD's
reengineering plans and participation of relocation companies, such
as Cendant, in DOD and civilian government household goods business
has had a major impact on the availability to Cendant of small
business subcontractors in the Hunter pilot. According to Cendant, a
number of major van lines have been reluctant to sign contracts with
it for the pilot. Eight of the 15 small business agents who had been
doing business with Hunter are affiliated with van lines that have
not signed Cendant contracts.
In December 1997, Cendant initiated efforts to obtain signed
contracts and broaden the base of potential small business certified
agents who could participate in the pilot. Specifically, on December
8, 1997, Cendant held a meeting in which 20 officials, representing
15 local moving companies, expressed an interest in providing
relocation services. According to Cendant officials, several of the
local carriers will ask to be considered for future business and
Cendant intends to continue encouraging small business participation
and increase its small business supplier base.
Even as efforts are being made to increase small business
participation in the Army's pilot, a long-standing issue remains--the
lack of verifiable data establishing company size and ensuring that
companies qualify as small businesses. Historically, such data have
been based on carrier/forwarder self-certification, which MTMC has
considered questionable. Under the MTMC program, company size is
established when MTMC initially approves companies' participation in
the program. MTMC, however, does not verify the data or require the
data to be updated. In addition, MTMC has informed us that, in its
view, some businesses have certified themselves as small, but, in
fact, they rely exclusively on the assets of their parent companies
to perform the actual transportation services. Although these
so-called "paper companies" have certified themselves as small
businesses, when the revenues of their parent companies are
considered, MTMC believes these companies may exceed the small
business size threshold. Moreover, few of the MTMC-approved carriers
and forwarders, whether paper companies or not, are identified in the
Small Business Administration data files. Consequently,
carrier/forwarder size determination will be difficult, and the
Army's ability to compare test data with baseline data could be
questionable.
--------------------
\5 For the moving and storage industry, small businesses are defined
as those with average annual gross revenues not exceeding
$18,500,000.
OTHER PILOT PROJECTS OR TESTS
UNDERWAY OR PLANNED
------------------------------------------------------------ Letter :5
Other pilots or tests are also underway or are being planned in DOD.
MTMC, as DOD's overall personal property program manager, is
contracting for a pilot covering selected shipments moving from all
military bases and installations throughout North Carolina, South
Carolina, and Florida. This pilot will use one or more companies to
handle moves on specific traffic channels--from one state to a group
of states, excluding Hawaii and Alaska, and to various locations in
Europe--under long-term contracts. Key goals of this pilot are to
simplify DOD's current transportation acquisition process and to
substantially improve and put on par with corporate customer
standards, the quality its military personnel and their families
receive from DOD's contracted movers. This pilot differs from the
Army Hunter pilot in that it is larger, covering a wider geographical
origin area and more shipments, and involves the relocation of
members of all branches of service. The Army Hunter pilot, on the
other hand, is limited to one location, Hunter, and one service, the
Army, although it provides for move management services, including
entitlement counseling and destination relocation assistance, for all
Hunter Army families, relocating to anywhere in the world. Although
the original MTMC solicitation was issued on March 14, 1997, the
start of the MTMC pilot has been delayed--first, by a disagreement
between DOD and the moving industry over the approach to take for the
pilot test and, second, by a series of protests filed with us
concerning the terms and conditions of the solicitation.\6
The Navy began a test in January 1998 to use commercial practices at
its personal property shipping office at the Fleet Industrial Supply
Center-Puget Sound, Bremerton, Washington. The initial test is
limited to moves being made to five U.S. destinations. It allows
sailors to select from a list of eligible moving companies the
company they would like to move their property. Like the ongoing
Hunter and proposed MTMC pilots, sailors receive counseling, full
replacement value protection for any loss or damage, and direct
claims settlement with the contractor or moving company.
The U.S. Transportation Command (USTRANSCOM) plans to evaluate the
Hunter pilot and the two other military personal property pilots or
tests. USTRANSCOM will also maintain oversight of the personal
property program pilots and tests and assure consistent evaluation
criteria and assessment of the pilot program results. Upon
completion of the pilots and tests, and in coordination with each
service, USTRANSCOM will determine which pilot or test, or portions
thereof, if any, could provide better long-term results for DOD. It
will then recommend the follow-on course of action and time-lines for
implementation throughout DOD, if applicable. We have previously
provided informal comments to USTRANSCOM on their evaluation plan and
will continue to monitor the personal property pilot tests.
Notwithstanding this and other pilots and tests planned or underway,
DOD and Army officials briefed us at the end of this review about
plans to expand the Hunter pilot by testing it at selected DOD sites.
Specifically, the expansion would include a higher shipment volume
(between 40,000 and 50,000 shipments per year) and a larger range of
services (such as adding nontemporary and vehicle storage) to
determine if the Hunter pilot results can be successfully duplicated
throughout DOD. DOD would like to begin implementing the test by
October 1, 1998. However, to date, all of the expansion details have
not been worked through, such as determining how the expansion will
be managed and evaluated and identifying all of the installations and
services that would participate.
--------------------
\6 See our report to the congressional defense committees on the
dispute over the approach to take (Defense Transportation:
Reengineering the DOD Personal Property Program, GAO/NSIAD-97-49,
Nov. 27, 1996) and on the solicitation dispute in four decisions
(Aalco Forwarding, Inc. et.al., B-277241.8, B-277241.9, Oct. 21,
1997; Aalco Forwarding, Inc. et.al., B-277241.12, B-277241.13, Dec.
29, 1997; Aalco Forwarding, Inc. et.al., B-277241.15, Mar. 11,
1998; and Aalco Forwarding, Inc. et.al., B-277241.16, Mar. 11,
1998).
CONCLUSIONS
------------------------------------------------------------ Letter :6
The Army is well underway in implementing its pilot project for
outsourcing relocation services at Hunter Army Airfield. It has
identified quality of life, costs, and impact on small businesses as
important factors to be used to determine the success of the pilot.
In response to our draft report recommendation, the Army provided
additional information that states what results it will use to judge
the pilot a success. The information, with some minor exceptions,
represents important steps toward formulating a sound evaluation
plan. However, the evaluation method has not been finalized or made
a formal part of the evaluation plan, and questions remain about the
definition of terms and small business group measurements. We
recognize that further issues could be identified as the evaluation
method is finalized.
In addition, the Hunter pilot is only one initiative among several
that will be tested to determine how DOD can improve the delivery of
relocation services to military personnel. Should DOD decide to
expand the Hunter pilot to include significantly more military
customers and services, this could provide additional information for
assessing the concept and for evaluating results and savings
achieved. However, USTRANSCOM, which has primary responsibility for
analyzing overall results and savings achieved, will ultimately need
to assess the relative strengths and/or limitations of each pilot
project to determine how DOD can best improve the personal property
program.
RECOMMENDATION
------------------------------------------------------------ Letter :7
We recommend that the Secretary of Defense direct the Secretary of
the Army to finalize its method for evaluating the pilot results to
include the new evaluation information provided in response to our
draft report. In doing so, the Secretary of Defense should direct
the Secretary of the Army to strengthen the evaluation plan by better
defining terms and more clearly defining the small business groups on
which measurements will be taken to establish small business
participation in the pilot.
AGENCY COMMENTS AND OUR
EVALUATION
------------------------------------------------------------ Letter :8
DOD stated that it concurred, with comment, with the report and
recommendation. Our draft report contained a recommendation that the
Secretary of Defense direct the Secretary of the Army to determine
what results the Army will require to judge its pilot a success, such
as specifying the relative weight of each factor it is measuring and
determine if certain factors must meet certain thresholds, and if so,
specify those factors and levels. In agreeing with our draft
recommendation, DOD provided additional information regarding the
Army's method for determining what results it plans to use to judge
the pilot a success. While we must reserve conclusions until an
evaluation method is finalized, our initial assessment of the
additional information provided indicates that it begins to clarify
the description of the three evaluation factors (quality of life,
total cost, and impact on small business), establishes a
point-scoring system and threshold for each factor and subfactor, and
prescribes the relative weight of each factor for determining the
pilot's success. To the extent this information is incorporated into
the Army's evaluation plan, it will provide a more complete basis for
the Army's analysis. However, our initial assessment of the
information indicates that the Army could strengthen the evaluation
by better defining terms, such as "more satisfied" in reference to
comparing customer satisfaction ratings and by more clearly defining
the small business groups--carriers, agents, or both--on which
measurements will be taken to establish small business participation
in the pilot. Consequently, we have revised our report to discuss
the additional information and modified our recommendation to
indicate that the evaluation method should be finalized and be made a
formal part of the Army's evaluation plan.
DOD also had comments on three specific areas. First, DOD disagreed
with our references that the Hunter pilot was being expanded
"DOD-wide." DOD indicated that while expansion of the Hunter pilot is
being considered as part of its total reengineering effort, it has
not approved or decided to expand the program "DOD-wide." We revised
the report to reflect the current status of DOD's plans.
Second, DOD stated that characterizing the DOD overhead cost data as
often unreliable and unavailable undermines the entire effort. We
acknowledge DOD's position; however, the Army Audit Agency's
validation of overhead data will not be completed for several months.
Further, while the preliminary data represent a full 6 months of
performance, the Army has not developed the same full 6 months of
transportation cost data. (The data are needed to accurately portray
the pilot's cost.) Until the Army has developed the cost data both
for overhead and transportation for the full
6 months, and the audit validation process is complete, any
conclusions about the total cost of the pilot are premature.
Third, DOD expressed concern that we had not mentioned that greater
small business participation in the Hunter pilot could be realized if
the parent companies of small businesses that previously serviced
Hunter agreed to allow their small business concerns to sign
contracts with the Hunter contractor. We have added that information
in our discussion of the pilot's impact on small business.
DOD's comments, along with those prepared by the Army, are included
as appendix I.
SCOPE AND METHODOLOGY
------------------------------------------------------------ Letter :9
We asked the Army for data and supporting documents concerning the
first 6 months of activity at Hunter, and the Army provided us that
data and documents on February 2, 1998. We reviewed the data and
observed how it was being collected. We did not independently verify
the accuracy of the data; however, the Army Audit Agency will be
validating the data. To date, sufficient data have not been
collected for comparison purposes.
We also met and discussed matters related to the pilot with officials
of the following:
-- Office of Deputy Under Secretary of Defense
(Logistics-Transportation Policy), Washington, D.C.;
-- USTRANSCOM, Scott Air Force Base, Illinois;
-- Office of Army Deputy Chief of Staff for Logistics, Washington,
D.C.;
-- Hunter Army Airfield, Georgia;
-- Army Audit Agency, Hunter Army Airfield, Georgia;
-- Fort Stewart, Georgia; and
-- MTMC, Falls Church, Virginia.
We reviewed the Army's plans for evaluating the pilot project and
reviewed the chronology of the pilot from announcement of plan,
presolicitation, solicitation, award, protest, and implementation.
Regarding the management of the pilot project, we visited both Hunter
Airfield and the personal property shipping office at Fort Stewart
and discussed the information being collected by Army officials. We
also reviewed the data prepared by the Army and the Army Audit
Agency.
We met with Hunter contractor officials; observed their contract
operations, including the actions of their contracted carrier and
agent activities; and observed a relocation counseling session with a
Hunter soldier. We observed similar activities at Fort Stewart and
were briefed or received documentation on the other planned or
ongoing DOD personal property tests. We also discussed matters
related to the Army Hunter pilot with three agents in Savannah who
are on the MTMC-approved list of agents serving the military in the
Fort Stewart/Savannah area.
Our review was conducted between July 1997 and March 1998 in
accordance with generally accepted government auditing standards.
---------------------------------------------------------- Letter :9.1
We are sending copies of this report to the Secretaries of Defense
and the Army; the Commander in Chief, USTRANSCOM; the Department of
Army Deputy Chief of Staff for Logistics; and the Commander, MTMC.
We will also make copies available to others upon request.
Please contact me on (202) 512-8412 if you or your staff have
questions concerning this report. Major contributors to this report
are listed in appendix II.
David R. Warren, Director
Defense Management Issues
List of Congressional Committees
The Honorable Strom Thurmond
Chairman
The Honorable Carl Levin
Ranking Minority Member
Committee on Armed Services
United States Senate
The Honorable Ted Stevens
Chairman
The Honorable Daniel K. Inouye
Ranking Minority Member
Subcommittee on Defense
Committee on Appropriations
United States Senate
The Honorable Floyd D. Spence
Chairman
The Honorable Ike Skelton
Ranking Minority Member
Committee on National Security
House of Representatives
The Honorable C. W. Bill Young
Chairman
The Honorable John P. Murtha
Ranking Minority Member
Subcommittee on National Security
Committee on Appropriations
House of Representatives
(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================== Letter
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
(See figure in printed edition.)
The following are GAO's comments on the Department of Defense's
letter dated May 21, 1998.
GAO COMMENTS
1. We have included Tab C containing new information that the Army
will use to evaluate the pilot. However, we have not included Tab A
and Tab B, which are administrative and technical comments and were
addressed in the report as appropriate.
2. An Army Hunter pilot program official advised us that the wording
in paragraph 1e (Total Cost) was incorrect and should have indicated
that the Hunter pilot will be determined successful if the total
contract-case cost of the pilot does not exceed the baseline-case
cost by more than 30 percent.
MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II
NATIONAL SECURITY AND
INTERNATIONAL AFFAIRS DIVISION,
WASHINGTON, D.C.
Barry W. Holman, Associate Director
Nomi R. Taslitt, Assistant Director
J. Kenneth Brubaker, Evaluator-in-Charge
Barbara L. Wooten, Evaluator
Leo G. Clarke, III, Evaluator
*** End of document. ***