Depot Maintenance: Status of the Navy's Pearl Harbor Pilot Project
(Letter Report, 09/10/1999, GAO/NSIAD-99-199).
Pursuant to a congressional request, GAO examined the Navy's progress in
implementing the Pearl Harbor Pilot project, focusing on: (1) the
preliminary results of the Pearl Harbor Pilot on improving performance
of maintenance activities; (2) the usefulness of the pilot as a model
for future consolidations; and (3) issues related to financial and
organizational structures for such consolidations.
GAO noted that: (1) although the Pearl Harbor Pilot is not complete,
preliminary results have been mixed, showing either improvements, no
improvements, or insufficient data to determine results; (2) where data
are available, overall indications are that the pilot has the potential
to improve maintenance activities in Hawaii; (3) while the Navy's pilot
test plan calls for evaluating performance using nine metrics, data has
been gathered for only five to compare performance under the
consolidated operation with performance under the preconsolidation
organizations; (4) preliminary results for two metrics indicate
improvements that meet or exceed the Navy's expectations, and two others
indicate improvements that fall slightly short of the Navy's
expectations, while one metric indicates no improvement; (5) other
positive results include increasing workforce flexibility by integrating
nearly 4,000 workers from two centers into a single workforce; (6)
consequently, the number of workers assigned daily to the excess labor
shop has dropped from about 200 to below 10; (7) the Navy has reduced
the maintenance infrastructure by 11 buildings (114,131 square feet) and
plans to eliminate another 6 buildings (24,907 square feet); (8) the
Pearl Harbor Pilot is likely to serve only as a general model for future
consolidation efforts because of unique aspects of ship maintenance
activities in Hawaii, such as the close proximity of facilities and the
large portions of fleet-funded work than in other locations; (9)
likewise, because of this uniqueness, the pilot provides only a general
indication that future consolidations elsewhere will result in similar
efficiencies; (10) nevertheless, it does provide general information on
such issues as combining workforces from separate activities and
consolidating equipment and facilities; (11) the Pearl Harbor Pilot has
sharpened the debate over the most appropriate financial and
organizational structures for such consolidated activities; (12)
conflicting views continue to exist within the Department of Defense and
the Navy over whether these consolidated activities should operate under
direct appropriations, the Navy Working Capital Fund (NWCF), or a
combination of the two financial structures, particularly in terms of
maintaining visibility over total operational costs; and (13) similarly,
little progress has been made toward resolving the departments'
differences over the most appropriate organizational structure to manage
these types of activities.
--------------------------- Indexing Terms -----------------------------
REPORTNUM: NSIAD-99-199
TITLE: Depot Maintenance: Status of the Navy's Pearl Harbor Pilot
Project
DATE: 09/10/1999
SUBJECT: Human resources utilization
Cost effectiveness analysis
Military downsizing
Performance measures
Federal agency reorganization
Military vessels
Equipment maintenance
Naval facilities
IDENTIFIER: Navy Regional Maintenance Program
Navy Working Capital Fund
Pearl Harbor (HI)
Navy Pearl Harbor Pilot Project
******************************************************************
** This file contains an ASCII representation of the text of a **
** GAO report. This text was extracted from a PDF file. **
** Delineations within the text indicating chapter titles, **
** headings, and bullets have not been preserved, and in some **
** cases heading text has been incorrectly merged into **
** body text in the adjacent column. Graphic images have **
** not been reproduced, but figure captions are included. **
** Tables are included, but column deliniations have not been **
** preserved. **
** **
** Please see the PDF (Portable Document Format) file, when **
** available, for a complete electronic file of the printed **
** document's contents. **
** **
** A printed copy of this report may be obtained from the GAO **
** Document Distribution Center. For further details, please **
** send an e-mail message to: **
** **
** **
** **
** with the message 'info' in the body. **
******************************************************************
ns99199 A Report to the Chairman, Subcommittee on Readiness and
Management Support, Committee on Armed Services, U. S. Senate
September 1999 DEPOT MAINTENANCE
Status of the Navy's Pearl Harbor Pilot Project
GAO/NSIAD-99-199
GAO/NSIAD-99-199
Letter 3 Appendixes Appendix I Scope and Methodology 26
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses 29 Appendix III GAO Contacts and
Staff Acknowledgments 42
Related GAO Products 43 Tables Table 1: The Navy's Nine Test
Metrics for Evaluating the Pearl Harbor Pilot's Effectiveness 7
Table 2: Preliminary Results, Strengths, and Weaknesses of the
Pearl Harbor Pilot Test Metrics 10 Table 3: Comparison of Ship
Maintenance Activities in the Pearl
Harbor, Puget Sound, Portsmouth, and Norfolk Areas 15 Table 4:
Navy's Initial Two Metrics to Test Whether the Consolidated
Pearl Harbor NSY & IMF Increased Manpower Utilization and Lowered
Unit Cost of Production 29 Table 5: Metrics Three Through Six to
Test Whether the Consolidated
Pearl Harbor NSY & IMF Increased Manpower Utilization and Lowered
Unit Cost of Production 31 Table 6: Metrics Seven Through Nine to
Test Whether the Consolidated Pearl Harbor NSY & IMF Increased
Manpower Utilization and
Lowered Unit Cost of Production 32 Table 7: Baseline, Performance
Expectations, and Preliminary
Results of the Test Plan 33 Figures Figure 1: Breakdown of the
Naval Shipyards' Fiscal Year 1997
Workloads by Major Customer 17 Figure 2: Comparison of Reported
NWCF Revenues by Activity
Group for Fiscal Year 1998 20
Abbreviations
CNO Chief of Naval Operations CSMP Current Ship Maintenance
Program DOD Department of Defense NAS Naval Audit Service NAVSEA
Naval Sea Systems Command NIMF Naval Intermediate Maintenance
Facility NSY & IMF Naval Shipyard & Intermediate Maintenance
Facility NWCF Navy Working Capital Fund PACFLT Pacific Fleet PHNSY
Pearl Harbor Naval Shipyard
National Security and International Affairs Division
Let ter
B-280524 September 10, 1999 The Honorable James M. Inhofe
Chairman, Subcommittee on Readiness
and Management Support Committee on Armed Services United States
Senate
Dear Mr. Chairman: In recent years, the Navy has implemented many
changes aimed at making its fleet support activities more
efficient and effective. An underlying assumption was that
consolidating similar activities within a region could eliminate
the inefficiencies and redundancies. In 1998, as part of this
effort, the Navy implemented a pilot project consolidating the
management, operations, and funding of the Pearl Harbor Naval
Shipyard and the Naval
Intermediate Maintenance Facility in Hawaii. The Navy expects that
the pilot project, commonly called the Pearl Harbor Pilot, will
serve as a model for other potential consolidations by confirming
that integrating shipyard and intermediate activities can result
in greater efficiency and lower overall unit costs. As you
requested, we examined the Navy's progress in implementing this
pilot project. This report discusses (1) the preliminary results
of the Pearl Harbor Pilot on improving performance of maintenance
activities, (2) the
usefulness of the pilot as a model for future consolidations, and
(3) issues related to financial and organizational structures for
such consolidations. Our scope and methodology are described in
appendix I. Results in Brief Although the Pearl Harbor Pilot is
not complete, preliminary results have
been mixed, showing either improvements, no improvements, or
insufficient data to determine results. Where data are available,
overall indications are that the pilot has the potential to
improve maintenance
activities in Hawaii. While the Navy's pilot test plan calls for
evaluating performance using nine metrics, data has been gathered
for only five to compare performance under the consolidated
operation with performance under the preconsolidation
organizations. Preliminary results for two metrics indicate
improvements that meet or exceed the Navy's expectations, and two
others indicate improvements that fall slightly short
of the Navy's expectations, while one metric indicates no
improvement. Other positive results include increasing workforce
flexibility by integrating nearly 4,000 workers from two work
centers into a single workforce. Consequently, the number of
workers assigned daily to the excess labor shop has dropped from
about 200 to below 10. Furthermore, the Navy has reduced the
maintenance infrastructure by 11 buildings (114,131 square feet)
and plans to eliminate another 6 buildings (24,907 square feet).
At the same time, other anticipated efficiencies have yet to be
realized. For example, not all the industrial plant equipment has
been consolidated and the projected number of overhead workers has
not been moved to direct maintenance positions to improve
productivity.
The Pearl Harbor Pilot is likely to serve only as a general model
for future consolidation efforts because of unique aspects of ship
maintenance activities in Hawaii, such as the close proximity of
facilities and the large portion of fleet- funded work than in
other locations. Likewise, because of this uniqueness, the pilot
provides only a general indication that future
consolidations elsewhere will result in similar efficiencies.
Nevertheless, it does provide general information on such issues
as combining workforces from separate activities and consolidating
equipment and facilities. The Pearl Harbor Pilot has sharpened the
debate over the most appropriate financial and organizational
structures for such consolidated activities. Conflicting views
continue to exist within the Departments of Defense and
the Navy over whether these consolidated activities should operate
under direct appropriations, the Navy Working Capital Fund, or a
combination of the two financial structures, particularly in terms
of maintaining visibility over total operational costs. Similarly,
little progress has been made toward resolving the departments'
differences over the most appropriate organizational structure to
manage these types of activities. This report recommends that the
Navy take steps to address unresolved issues related to financial
and organizational structures as it proceeds with similar
consolidations.
Background In March 1994, the Chief of Naval Operations (CNO)
announced a Regional Maintenance Program to streamline the Navy
ship repair and maintenance processes, reduce infrastructure and
costs, and maximize outputs. The program consisted of three
phases: (1) optimizing intermediate- level maintenance through
consolidation of intermediate activities, (2) integrating
intermediate and depot activities with management by the
fleet commanders, and (3) conducting fleet maintenance using a
single process. The first phase, optimizing intermediate- level
maintenance by minimizing redundant intermediate ship maintenance
capacity and
capability, is nearing completion. To ensure the validity of the
second phase of the program and to provide a model for other
potential consolidations, the Navy implemented the Pearl Harbor
Pilot in 1998. Its completion date has not been established. The
third phase, performing a single maintenance process by using
common business and production processes, is under way and
scheduled to be completed in fiscal year 2001.
The Pearl Harbor Pilot Prior to the consolidation, the Navy
recognized that organizational and funding structures inhibited
the ability of the Pearl Harbor Naval Shipyard (PHNSY) and Naval
Intermediate Maintenance Facility (NIMF) to readily share
workloads and resources. The shipyard was managed by the Naval Sea
Systems Command (NAVSEA) and funded through the Navy Working
Capital Fund (NWCF), 1 while the intermediate maintenance facility
was managed by the Pacific Fleet (PACFLT) and financed through
direct appropriations. 2 During the pilot, PACFLT assumed
ownership and overall
management and financial responsibility for the consolidated
activity, while NAVSEA continued to be the technical and operating
authority. The Navy named the combined activity the Pearl Harbor
Naval Shipyard & Intermediate Maintenance Facility (NSY & IMF). To
achieve a fully integrated organization, the Navy decided the
Pearl Harbor Pilot should use a single financial structure and
selected direct appropriations rather than the working capital
fund as the funding approach. This latter decision was based on
several factors, but Navy officials believed that direct
appropriations were better suited to achieve the pilot goals and
matched the financial structure of the largest shipyard customer-
PACFLT. 1 Under the working capital fund arrangement, activities
such as a shipyard sell goods and services to customers based on
predetermined rates designed to recoup the full cost of goods and
services, including any military personnel costs. Customers pay
for the goods and services with their operation and maintenance
funds. The fund's accounting practices and related requirements
for disclosure of information are intended to help managers and
workers focus on production costs and performance. However, the
Department of Defense (DOD) has long- standing problems in
accumulating and reporting the full costs associated with its
working capital fund operations.
2 Direct appropriations are sometimes referred to as mission
funding by DOD and Navy officials. They are funds appropriated
directly by the Congress that authorize agencies to incur
obligations for designated purposes. An agency may not spend more
than the amount appropriated.
Furthermore, in recent years there has been increasing Navy
dissatisfaction with the costs associated with the working capital
fund. Given the nature of the fund and its emphasis on fully
recovering costs of
the services provided, the perception sometimes exists that the
fund included fees and charges that inflated ship maintenance
costs compared to mission funding, which does not reflect the full
cost of operations. In accordance with one of the basic tenets of
a working capital fund, the costs associated with ship maintenance
are more visible when financed using this mechanism. While the
level of resources required to carry out ship maintenance
activities is likely to be similar regardless of whether financed
using mission funding or through a working capital fund, under the
working capital fund the customer is more likely to be directly
responsible for a larger portion of those costs. Use of a working
capital fund better enables DOD components to fully account for
their share of the program costs. In commenting on a draft of this
report, the Navy said that it chose the Shipyard Management
Information System as the cost accounting system for the Pearl
Harbor Pilot. According to the Navy, this system
(which is used by other naval shipyards under the NWCF) provides
the pilot the same degree of cost visibility as a working capital
fund activity. Full accounting for the costs of federal programs
is required by federal accounting standards. Specifically,
Statement of Federal Financial Accounting Standards No. 4 requires
federal agencies to accumulate the full cost of outputs through
appropriate costing methodologies or cost finding techniques (the
full cost of an output is the sum of (1) the cost of resources
consumed that directly or indirectly contributes to the output
and (2) the cost of identifiable supporting services provided by
other entities).
After Expressing Concerns, In 1998, as Navy officials were
formulating plans for the Pearl Harbor Pilot, Department of
Defense
DOD officials expressed concern that the Navy's test plan and
baseline data Officials Approved the Test would not adequately
measure the pilot's impact on productivity and Plan in September
1998 performance. Consequently, the Navy revised the test plan and
tried to minimize the data limitations in the baseline.
Ultimately, as shown in table
1, the Navy selected nine test metrics to evaluate the pilot's
effectiveness.
Table 1: The Navy's Nine Test Metrics for Evaluating the Pearl
Harbor Pilot's Effectiveness
Metric
1. Total cost of a production shop, direct labor hour of work
delivered to the customer (indicator of the pilot's efficiency in
terms of the cost per direct maintenance hour) 2. Total labor
hours expended to deliver a production shop, direct labor hour to
the customer (indicator of the pilot's productivity in terms of
personnel utilization)
3. Total Current Ship Maintenance Program work items completed
(indicator of the pilot's productivity in terms of the number of
work items completed)
4. Total Current Ship Maintenance Program work items in the
backlog (indicator of the material conditions of PACFLT ships in
terms of work items not yet completed)
5. Schedule adherence of CNO maintenance projects (indicator of
customer satisfaction in terms of completing projects on schedule)
6. Rework index for CNO maintenance projects (indicator of the
pilot's quality of work in terms of hours required to correct work
deficiencies) 7. Activity work schedule integrity index (indicator
of customer satisfaction in terms of completing projects on
schedule) 8. Casualty reports caused by activity work (indicator
of the pilot's quality of work in terms of casualty reports ship
reports indicating equipment failure)
9. Earned value (indicator of the pilot's productivity in terms of
hours to complete similar work items) Source: The Pearl Harbor
Pilot Test Plan, as amended, Department of the Navy (June 30,
1998).
In June 1998, the DOD Inspector General had recommended the Navy
delay the pilot and restart the data collection process, applying
the same data collection methodologies and controls for the
baseline that would be used to collect data during the pilot
period. According to DOD Comptroller and Inspector General
officials, the most severe of the metrics' limitations
affected the baseline costs for the first two test metrics shown
in table 1. For example, military personnel costs were not
traditionally collected by intermediate maintenance facilities
because these facilities were largely funded out of operation and
maintenance appropriations while military personnel costs were
already paid for separately out of military personnel
appropriations. Consequently, military personnel costs were also
accounted for separately. In addition, some labor hours appeared
to be inaccurate. One DOD contractor reported the NIMF database
contained zero labor hours for approximately 6 percent of the
maintenance tasks
completed by the facility. Further, some Navy officials expressed
concern that some recorded labor hours appeared to reflect the
expected work standards and not actual production hours.
In response to the DOD Inspector General recommendation, Navy
officials said the pilot was too far along in the integration
process in June 1998 to reestablish separate maintenance
activities without incurring significant cost and operational
turmoil. The expected cost and turmoil of such a reversal,
according to these officials, would outweigh any potential benefit
obtained from developing a better baseline. Consequently, the Navy
revised the test plan to minimize the data limitations. For
example, as CNO and NAVSEA officials requested, the Naval Audit
Service (NAS) agreed to validate the baseline and pilot results
for six of the nine test metrics. 3 In
several instances, baseline data were routinely collected by
preconsolidation activities, and they are available to provide a
basis for comparison with postconsolidation performance data. In
addition, as requested by CNO and NAVSEA officials, the NAS
developed a methodology to estimate baseline values for the first
two metrics. The
methodology is based on the Navy's personnel wages and standards,
as well as, on the best available data from the former NIMF.
Although the resulting baseline estimates for the two metrics may
not be exact, Navy officials believe they are comparable to
historical trends for intermediate maintenance facilities and,
consequently, provide a reasonable baseline on which to indicate
the pilot's impact on productivity and performance. On September
14, 1998, the Principal Deputy Under Secretary of Defense
(Comptroller) approved the Navy's test plan. Appendix II provides
more
detail concerning the evolution of the test plan and the nine
metrics. Congressional
The Conference Report for the DOD Appropriations Act for Fiscal
Year Requirements for the Pearl 1998 4 recognized that the Navy
planned to initiate a pilot to study whether
Harbor Pilot combining fleet intermediate maintenance facilities
with Navy shipyards might yield economies of scale and allow
maintenance managers to better
balance workloads. The conferees concluded that it would take at
least 2 years before the Navy could determine whether this new
arrangement was effective and should be made permanent or expanded
to other locations. Further, the report directed the Navy not to
expand the pilot until 6 months after it reported to the
Committees on Appropriations on its findings, and that such report
be made on or after April 1, 1999. The conferees also directed the
Navy not to make any permanent changes to 3 Previously, the NAS
had agreed to collect and validate the baseline and pilot results
for the
first two test metrics. 4 House Report 105- 265, September 23,
1997.
the workforce in terms of total number of employees or any other
permanent changes until the pilot had been completed and evaluated
by the Congress. Currently, the Navy plans to issue an interim
report on the pilot shortly after October 1999.
While the Pilot Has Preliminary results of the Pearl Harbor Pilot
have been mixed, showing Potential to Improve either improvements,
no improvements, or insufficient data to determine results. For
most of the measures where data are available, indications are
Maintenance Activities, that the pilot has the potential to
improve maintenance activities in Hawaii. Preliminary Data Show
While the test plan calls for using nine metrics, the data has
been gathered Mixed Results
for only five of them. For these five metrics, preliminary results
for two metrics indicate improvements that meet or exceed the
Navy's expectations; two others indicate improvements that fall
slightly short of
the Navy's expectations; and one metric indicates no improvement.
Other positive results include the fact that the Navy has
integrated the workforces from two work centers into a common
pool, increased management flexibility in assigning workers to
maintenance projects based on evolving priorities, and reduced the
maintenance infrastructure at Pearl Harbor. At the same time,
other planned efficiencies are yet to be realized. For example,
the pilot has not resulted in the number of overhead workers
projected to move to direct maintenance positions or in the
consolidation of all industrial plant equipment. 5 According to
Navy officials, these actions would have increased the number of
people working directly on maintenance projects and the
utilization of existing plant equipment.
The Pilot Has the Potential The nine metrics chosen for the Pearl
Harbor Pilot Test Plan represent a
to Improve Maintenance variety of issues and performance
indicators. Although data available for Activities in Hawaii
some of the metrics are limited, it is sufficient to provide a
general, if imprecise, indication of the pilot's impact on
productivity and performance. In this case, as indicated in table
2, while test results have not yet been fully developed, the
preliminary results have been mixed, showing either improvements,
no improvements, or insufficient data to determine 5 In general,
overhead workers are those employees who are not working directly
on maintenance projects and are assigned to administrative and
support positions such as personnel and financial departments.
Direct maintenance workers are those employees who accomplish work
directly identifiable to a maintenance project such as repair,
maintenance, and installation of equipment and components.
results. Where data are available, overall indications are that
the pilot has the potential to improve maintenance activities in
Hawaii.
Table 2: Preliminary Results, Strengths, and Weaknesses of the
Pearl Harbor Pilot Test Metrics Metric (number) and anticipated
goal Preliminary results Strengths (S) and weaknesses (W)
Improvement indicated
Total cost of a production shop, Indicates improvement because
Variant of a traditional indicator of shipyard efficiency (S)
direct labor hour of work delivered preliminary data show that it
has
Shipyard baseline data are considered reliable (S) to the customer
(1)
cost slightly less to deliver one NAS developed baseline estimates
and plans to validate and
Indicator of the pilot's efficiency in direct maintenance hour in
fiscal generate the test data (S)
terms of the cost per direct year 1998, but falls short of the NAS
developed baseline estimates of the NIMF military maintenance hour
Navy's expectation
personnel costs because the facility did not maintain this type of
data (W) NIMF baseline data on production hours have limitations,
but are considered reliable enough by Navy officials to provide a
rough, if imprecise, estimate of the baseline (W)
Total labor hours expended to Indicates improvement because
Variant of a traditional indicator of shipyard productivity (S)
deliver a production shop, direct preliminary data show that it
has Shipyard baseline data are considered reliable (S)
labor hour to the customer (2) taken fewer hours to deliver one
NAS developed baseline estimates and plans to validate and
Indicator of the pilot's productivity direct maintenance hour in
fiscal
generate the test data (S) in terms of personnel utilization
year 1998 NIMF baseline data on production hours have limitations
but are considered reliable enough by Navy officials to provide a
rough, if imprecise, estimate of the baseline (W)
Total Current Ship Maintenance Indicates improvement because
Traditionally considered an indicator of the material condition of
Program work items in the preliminary data show that there
ships (S) backlog (4)
are fewer backlogged work items Fleet and ship commanders maintain
baseline and test data,
Indicator of the material since the consolidation, but falls which
are considered reliable (S)
conditions of PACFLT ships in short of the Navy's expectation NAS
plans to validate the baseline and test data (S)
terms of work items not yet Is an indicator of the NIMF and the
combined facility's completed
productivity but not the former shipyard (W) Schedule adherence of
CNO ship Indicates improvement because Traditionally considered an
indicator of the shipyard customers' maintenance projects (5)
preliminary data show that more satisfaction (S) Indicator of
customer satisfaction CNO projects are being completed
NAVSEA maintains baseline and test data, which are in terms of
completing projects on on time or ahead of schedule since
considered reliable (S) schedule
the consolidation NAS plans to validate the baseline and test data
(S)
Is an indicator of shipyard and the combined facility's
performance but not the former NIMF (W)
No improvement indicated
Total Current Ship Maintenance Indicates no improvement because
Fleet and ship commanders maintain baseline and test data,
Program work items completed preliminary data show that there
which are considered reliable (S) (3)
are fewer work items being NAS plans to validate the baseline and
test data (S) Indicator of the pilot's productivity completed
since the consolidation
Is an indicator of the NIMF and the combined facility's in terms
of the number of work
productivity, but not the former shipyard (W) items completed
Is affected by other factors not related to the pilot (W)
(continued)
Metric (number) and anticipated goal Preliminary results Strengths
(S) and weaknesses (W)
Not yet determined
Rework index for CNO Not yet determined because the Traditionally
considered an indicator of the shipyard quality of maintenance
projects (6) Navy requires additional time to work (S) Indicator
of the pilot's quality of develop the test data
NAS plans to validate the baseline and test data (S) work in terms
of hours required to Is an indicator of shipyard and the combined
facility's correct work deficiencies
performance, but not the former NIMF (W) Activity work schedule
integrity
Not yet determined because the Traditionally considered an
indicator of the shipyard customers' index (7) Navy does not plan
to gather data
satisfaction (S) Indicator of customer satisfaction needed to
develop this metric
Shipyard maintains baseline and test data, which are in terms of
completing projects on
unless the results of the first six considered reliable (S)
schedule metrics are inconclusive
Is an indicator of shipyard and the combined facility's
performance, but not the former NIMF (W) Is similar to the fifth
metric measuring schedules adherence and, as a result, Navy
officials question the usefulness of developing this metric (W)
Casualty reports caused by Not yet determined because the Fleet
and ship commanders maintain baseline and test data,
activity work (8) Navy does not plan to gather data
which are considered reliable (S) Indicator of the pilot's quality
of needed to develop this metric
Is an indicator of shipyard and the combined facility's work in
terms of casualty reports unless the results of the first six
performance, but not the former NIMF (W) metrics are inconclusive
Is similar to the sixth metric measuring rework and, as a result,
Navy officials question the usefulness of developing this metric
(W) Earned value (9)
Not yet determined because the Attempts to measure the pilot's
effect on maintenance outputs Indicator of the pilot's
productivity Navy does not plan to gather data (S) in terms of
hours to complete needed to develop this metric
Should require significant resources to develop results (W)
similar work items
unless the results of the first six Is based on a limited universe
of work items (W)
metrics are inconclusive Note( s): Results as of June 22, 1999.
Appendix II provides more detail concerning the strengths and
weaknesses of the nine metrics. Source: Interviews with Navy
officials and our analysis of the Pearl HarborPilot Test Plan
dated June 30, 1998; NAS capacity evaluation reports for fiscal
years 1997 and 1998; and pilot status reports.
As shown in table 2, two test metrics indicate improvements that
meet or exceed the Navy's expectations and two indicate
improvements that fall slightly short of the Navy's expectations;
only one metric indicates no improvement. Because of the
operational turbulence experienced during the consolidation, we
believe that it is reasonable to expect that some productivity and
performance indicators might show diminished improvement during
the transition year. In addition, as indicated in the table, the
Navy has not gathered performance data for the remaining four
metrics. The Navy is in the process of collecting test data on the
rework index for CNO maintenance projects. However, Navy officials
do not plan
to gather data needed to develop the last three metrics unless the
test results are inconclusive, even though the three metrics were
added to the test plan at the request of the Principle Deputy
Under Secretary of Defense (Comptroller). Other Efficiencies
Identified In August 1997, the Pearl Harbor Regional Maintenance
Pilot Executive During the Planning Process Steering Committee
issued the Pilot Study Report for the integration of the
PHNSY and NIMF. 6 The purpose of the report was to outline the
approach Have Been Achieved
for this integration. Among other improvements, the report
envisioned increasing productivity and cost- effectiveness by
integrating the workers of former PHNSY and NIMF into a common
workforce and reducing the ship maintenance infrastructure in
Hawaii. As planned in the Pilot Study Report, the Navy has
integrated nearly 4,000 workers from two separate work centers
into a common pool, increased management flexibility in assigning
workers to maintenance projects, and reduced the ship maintenance
infrastructure. Prior to the pilot, it was difficult to shift work
or personnel between maintenance activities due to multiple
financial structures. More specifically, the Navy stated that NWCF
rules prohibited assigning workers to a job without a funding
document, which made it difficult to shift work between
maintenance activities. Accordingly, when administrative and
financial requirements restricted the movement of workers,
shipyard production personnel not working on a specific
maintenance project were
sent to the excess labor shop to wait assignment. During this
time, they performed indirect work such as facility maintenance or
grounds- keeping. 7 Before the pilot, the number of workers
assigned daily to this shop ranged between 100 to 200. By
integrating the workforce under a single financial and command
structure, production shops now have more flexibility in assigning
excess workers to other projects, including projects historically
completed by the former intermediate maintenance facility.
Consequently, the number of workers assigned daily to the excess
labor shop has dropped to below 10.
6 Pilot Study Report for an Integration of Pearl Harbor Naval
Shipyard and Naval Intermediate Maintenance Facility, Pearl
Harbor, Pearl Harbor Pilot Executive Steering Committee (Aug. 26,
1997). 7 Limited- duty personnel with medical injuries or health
problems and personnel whose security clearance levels were under
review were also assigned to the excess labor shop.
In addition, as a result of the consolidation, the Navy has
reduced the maintenance infrastructure at Pearl Harbor. For
example, 11 (114,131 square feet) of the 27 buildings previously
used by the former intermediate maintenance facility were vacated
and turned over to Pearl Harbor Naval Station. 8 While the
consolidated facility will retain 7 of the remaining
16 NIMF buildings, it plans to vacate another 6 (24,907 square
feet) and is reviewing the disposition of 3 buildings. 9 As of
June 1999, the Pearl Harbor Naval Station demolished or plans to
demolish eight former NIMF buildings and was able to demolish two
of its buildings after tenants moved into vacated NIMF buildings.
The demolition costs for all 10 buildings is estimated at $1.9
million with a projected annual cost avoidance of $307,000 or a
payback period of a little over 6 years. Some Efficiencies Have
Yet The Pilot Study Report envisioned increasing productivity and
to Be Fully Achieved
cost- effectiveness by moving overhead workers to direct
maintenance positions and consolidating industrial plant
equipment. However, these efficiencies have not yet been fully
achieved.
For example, the consolidation did not result in the projected
number of overhead workers moving to direct maintenance work-an
important element in increasing the combined facility's
productivity. To increase productivity, one of the stated goals of
the pilot was to increase the number of direct production workers
relative to the number of supervisors and overhead personnel
without increasing costs. Logically, increasing the
number of production workers should result in increased
production. To accomplish this goal, the Pilot Study Report
proposed that 95 civilian overhead workers be moved to positions
in direct maintenance work. However, only four workers moved.
According to Pearl Harbor NSY & IMF officials, while they have
tried, it has been impracticable for them to move civilians to
such positions because of (1) the time required for overhead
workers to become skilled, usually several years; (2) regulations
restricting the process for downgrades; and (3) potentially
negative action by workers
8 In general, the Pearl Harbor Naval Station has ownership and is
responsible for the management and maintenance of Navy buildings
in Hawaii. 9 According to Navy officials, no former shipyard
buildings will be vacated as a result of the Pearl Harbor Pilot.
and employee representatives. 10 In addition, several department
directors and overhead supervisors said they were unwilling to
release any personnel because of the increased workload due to
changes in administrative and financial systems since the
consolidation. Furthermore, other Navy officials said that efforts
to move supervisors to direct production resulted in too few
supervisors, which led to problems in planning and coordinating
work on maintenance projects. According to CNO and NAVSEA
officials, more overhead workers will be moved to direct
maintenance positions as the pilot progresses.
In addition, industrial plant equipment has not been completely
consolidated at integrated production shops. The Pilot Study
Report suggested the in- shop industrial equipment be efficiently
consolidated and
redistributed to improve maintenance operations. Of the 271 items
of industrial equipment at the former intermediate maintenance
facility, 114 items were relocated, 132 items were mothballed, and
25 items were kept operational at their original location.
According to officials formerly associated with the Pearl Harbor
NSY & IMF, the 114 related items were equipment that required
little or no removal and installation costs. Although many of the
132 mothballed items are in better condition and newer than the
same type of equipment located currently in the
consolidated shops, the funding required to relocate the NIMF
equipment has not been available. Most mothballed items are
semipermanently attached heavy equipment that requires funding to
remove, transport, and install elsewhere. According to Pearl
Harbor officials, congressional guidance directing the Navy not to
make any permanent changes until after the pilot period has to a
lesser degree influenced the decision not to relocate the
mothballed items. They plan to request funding to move equipment
after the pilot period is completed.
The Pearl Harbor Pilot Will The Pearl Harbor Pilot is likely to
serve only as a general model for future Serve Only as a General
consolidation efforts because of the unique aspects of ship
maintenance Model for Similar activities in Hawaii, such as the
close proximity of facilities and the Consolidations
significant portion of fleet- funded work. Likewise, because of
this 10 Our prior work examining DOD reform initiatives has noted
the importance of top management commitment and sustained support
for reform initiatives, as well as overcoming culture barriers and
resistance to change. See Defense Reform Initiatives:
Organization, Status, and Challenges (GAO/NSIAD-99-87, Apr. 21,
1999).
uniqueness, the pilot provides only a general indication that
future consolidations will result in similar efficiencies.
While there can be clear benefits to regionalization, available
statistical data suggest conditions may have been more favorable
to such efforts in Pearl Harbor than may exist at other Navy
locations. For example, during fiscal years 1996 through 1998, the
Navy reportedly replaced 698 military personnel at the former NIMF
with 504 experienced civilian workers, mostly from the shipyard.
Although civilian workers were periodically transferred between
intermediate facilities and shipyards at other locations, Pearl
Harbor is the only location where a significant number of workers
has recently been transferred. Consequently, according to Navy
officials, this large cadre of civilian workers with recent
shipyard experience should have expedited the integration of the
workforce by requiring less training and fewer trade skill
certifications. This condition does not exist to such a large
extent at other potential consolidation locations, except at the
Puget Sound area. Table 3 compares some additional differences
between Pearl Harbor and other Navy locations.
Table 3: Comparison of Ship Maintenance Activities in the Pearl
Harbor, Puget Sound, Portsmouth, and Norfolk Areas
Dollars in millions
Category Pearl Harbor Puget Sound Portsmouth Norfolk
Time required to travel between the 5 to 10 minutes 20 to 30
minutes to
2 to 3 hours 30 to 45 minutes to
intermediate maintenance facilities the Trident Refit Facility to
Groton, the Shore
and the shipyards 2 to 4 hours to Everett, Connecticut,
Intermediate
Washington, maintenance maintenance facility
Maintenance facility
Activity Reported shipyard revenues $379.4 $1,041.8 $394. 1 $822.1
Number of military personnel Shipyard
43 47
53 64
Intermediate maintenance facility 641 1,105 1,134 2,516 or
activity
Number of civilian personnel Shipyard
2,742 8,823
3,286 7,127
Intermediate maintenance facility 554 967 149 23 or activity
Note: Fiscal year 1998. Source: Our analysis of interviews and
documents obtained from Navy officials.
As indicated in table 3, Pearl Harbor facilities are located
closer to each other than in other Navy locations. Consequently,
consolidation of activities separated by greater distances at
other Navy locations could be more difficult to manage and, after
the consolidation is completed, could
possibly reduce the combined facility's flexibility in the
management of the workforce and workload. For example, given the
close proximity of facilities in Hawaii, workers with multiple and
specialized skills from both the former PHNSY and NIMF can move
between ship maintenance projects and emergent work in a few
minutes compared to other Navy locations where transportation time
could range up to 4 hours. In addition, the consolidated
production shops in Hawaii are able within a few minutes to pick
up and deliver replacement parts for maintenance projects compared
to other locations, where delivery time could range up to 4 hours
once the shops are consolidated. Consequently, the longer
transportation time between facilities at other Navy locations
could extend the time frames for maintenance projects compared to
projects completed in Hawaii. Further, as indicated in table 3,
Pearl Harbor facilities differ in terms of revenues and staffing
than in some of the other areas. For example, except for the
Portsmouth Naval Shipyard, reported revenues and staffing for the
other two shipyards were more than double the amount of the PHNSY.
The smaller mission at Pearl Harbor should have made the
consolidation easier.
In addition during fiscal year 1997, as indicated in figure 1,
more of the PHNSY workload was for the fleet compared with the
other three naval shipyards that completed more work for non-
fleet activities. According to Navy officials, a single major
customer should simplify the integration and management of ship
maintenance activities. They indicated that it is much
easier to schedule and budget for maintenance projects on behalf
of one primary and a few secondary customers compared to meeting
the varying requirements of several different customers with equal
workloads.
Figure 1: Breakdown of the Naval Shipyards' Fiscal Year 1997
Workloads by Major Customer
60 Percentage of total 50 40 30 20 10
0 Norfolk Naval Shipyard Puget Sound
Portsmouth Naval Pearl Harbor Naval Naval Shipyard
Shipyard Shipyard
Pacific Fleet Atlantic Fleet Naval Sea Systems Command Other
activities
Source: Our analysis of NAVSEA data.
In the future, Navy officials estimated that nearly 90 percent of
the Pearl Harbor NSY & IMF workload would be for PACFLT. The
remaining 10 percent will be completed under a reimbursable basis
for other activities. According to Navy officials, the large share
of PACFLT workload in Hawaii should reduce the need for the
combined maintenance activity to broker large workloads between
different Navy activities. At the same time, having one common
financial system should eliminate the need for potentially
unwieldy financial processes and procedures to manage and budget
for
different work and funding sources. In other locations, NAVSEA and
the
respective fleet are estimated to share the ship maintenance
workload on a more equitable basis.
Issues Related to The Pearl Harbor Pilot has sharpened the debate
over the most appropriate Financial and
financial and organizational structures for such consolidated
activities. Conflicting views continue to exist within the
Departments of Defense and Organizational the Navy over whether
these consolidated activities should operate under Structures Need
to Be direct appropriations, the NWCF, or a combination of the two
financial Resolved
structures, particularly in terms of maintaining visibility over
total operational costs. Similarly, little progress has been made
toward resolving the departments' differences over the most
appropriate organizational structure to manage these types of
activities. Consequently, we believe these issues need to be
resolved to facilitate effective and efficient consolidations.
Issues Remain Unresolved The Navy's proposal recommending direct
appropriations for the Pearl Regarding Financial Harbor Pilot
stated that the pilot provided an actual demonstration of using
Structure
direct appropriations for a major maintenance activity, which
would help to determine the ultimate financial structure for the
Navy's regionalization actions. The proposal further stated that a
common financial structure
needed to be tested by means of the pilot. However, the Navy's
test plan does not attempt to measure the impact of using a single
financial structure or the benefit of using direct appropriations
instead of the working capital
fund. At the same time, DOD and Navy officials continue to have
different views on the potential impact of using direct
appropriations for the pilot on the productivity and performance
of the Pearl Harbor Pilot and future regionalization actions. The
same is true about the command structure to be used for such
consolidated operations. Consequently, we believe these issues
need to be resolved to facilitate effective and efficient
consolidations.
Navy Selected Direct Prior to the consolidation, the Navy
recognized that the differences Appropriations
between activity management and financial structure inhibited the
PHNSY's and NIMF's ability to readily share workloads and
resources. According to Navy officials, the multi- financial and
command structures required them to use cumbersome, workaround
procedures to share
workloads and workers between the former PHNSY and NIMF.
Therefore, the shipyard and intermediate maintenance facility did
not readily share workloads and resources because of the time and
cost required to carry- out
the process. To achieve a fully integrated organization, the Navy
decided the Pearl Harbor Pilot should use a single financial
structure and selected direct appropriations instead of the NWCF.
This decision was based on several factors, but primarily because
PACFLT was the larger customer of the two organizations and direct
appropriations matched the fleet's
financial structure. Consequently, Navy officials expected fewer
financial issues under direct appropriations than NWCF because
PACFLT could integrate the pilot into its existing financial
structure without establishing another system. Views on the
Financial Structure
DOD and Navy officials have different views regarding the
financial impact Differ of removing the PHNSY from the NWCF. There
are several centrally managed functions in support of all
shipyards for which the costs are recovered in the shipyard labor
rates under the working capital fund structure, which are not
normally captured under direct appropriations. According to DOD
officials, these functions include NAVSEA personnel
directly supporting shipyards, the Navy's Crane Center that
manages all crane procurements and repair projects, centralized
automated information systems, programming support, and other
centrally managed
items. In effect, removing the PHNSY from the fund without
addressing the contribution (approximately 14 percent) it made to
cover centrally funded costs for naval shipyards would increase
their rates. However, CNO and NAVSEA officials believe the PHNSY's
share of the overhead cost was not large enough to have a
significant effect on the other shipyards' rates.
As shown in figure 2, the former PHNSY generated 1. 7 percent of
the fund's $21.9 billion in total revenues during fiscal year 1998
and the other shipyards contributed another 10.6 percent, or $2.3
billion.
Figure 2: Comparison of Reported NWCF Revenues by Activity Group
for Fiscal Year 1998
1.7%
Pearl Harbor Naval Shipyard
7.8%
Other maintenance
31.7%
Base support
8.7% 10.1%
Other
10.6%
Other shipyards
29.2%
Supply management Research and development
Note: The other maintenance category includes aviation and U. S.
Marine Corps depot maintenance activities and other includes
ordnance, information services, and transportation activities.
Percentages do not add to 100 percent due to rounding. Source: Our
analysis of the DOD Inspector General report entitled Inspector
General, DOD, Oversight of the Naval Audit Service Audit of the
Navy Working Capital Fund Financial Statements for FY 1998 (99-
099, Mar. 1, 1999).
DOD officials are concerned that the Navy has not adequately
addressed a potentially more significant issue regarding the
impact of removing all naval shipyards from the NWCF. They believe
the Navy decision to remove the shipyards from the fund would
cause other Navy activities remaining in the fund to absorb a
larger share of overhead costs. However, CNO and NAVSEA officials
believe that the shipyards' share is not large enough to
have a significant effect on the activities remaining in the fund.
Further, DOD and Navy officials have different views on other
financial issues:
Working capital funds are not directly subject to the annual
appropriation cycle and can continue operations without
interruption between fiscal years. Consequently, the NWCF provided
former shipyard operators with significant financial flexibility
because they were freed from reprogramming limitations and
restrictions applicable
to regular appropriations and allowed to incur costs without
waiting for enactment of an appropriation. Because this financial
flexibility is considered critical to shipyard operations, DOD
officials are concerned that the Navy has not addressed the impact
of eliminating this flexibility on the pilot operations or future
consolidations. The Congress established working capital funds,
among other purposes, to provide a more flexible funding mechanism
that would allow DOD
industrial, commercial, and service activities to operate on the
same basis as the private sector, including the use of standard
cost accounting practices and widely- accepted management
techniques. According to DOD officials, applying these practices
and techniques to naval
shipyards has provided cost visibility and improved the efficiency
of operations. Although we found that the Navy has retained the
cost visibility of the former shipyard under the pilot, the cost
information is not routinely reported to the major commands as it
was for the NWCF. Further, the visibility is maintained by
continuing to operate the former shipyard accounting and
management system, which could be
eliminated after the pilot is completed. For example, DOD
officials are concerned that the visibility will not be maintained
after the pilot period is completed. Navy officials said they plan
to maintain cost visibility of the maintenance operations in
Hawaii even after the pilot period because the visibility is
useful in the management of those operations. Whichever financing
method is adopted after the completion of the
pilot, the Statement of Federal Financial Accounting Standards No.
4 requires DOD and the Navy to continue to measure and report the
full cost of such ship maintenance operations by responsibility
segment. The buyer/ seller relationship is a fundamental aspect
of working capital funds. According to DOD officials, this
relationship established dynamic interplay between the buyer and
the provider that encouraged cost control and responsiveness to
customer requirements. In addition, it provided the opportunity
for shipyard customers to focus their attention on accomplishing
their respective missions and shipyard operators to be accountable
and responsive to customers' needs. Absent reimbursements for most
maintenance transactions at Pearl Harbor
under direct appropriations, CNO and NAVSEA officials said that a
buyer/ seller relationship has largely been eliminated. They
estimated that only 10 percent of the combined facility's future
workload, such as ship alterations and inactivations, will
continue to be on a reimbursable
basis after the pilot period. According to DOD officials, another
important aspect of working capital funds is their capital
investment program. In the case of the NWCF, the former shipyard
depreciated its capital assets and collected this
expense from customers through its rates. Therefore, the fund had
a ready reserve to finance capital investments for the shipyard.
According to CNO and NAVSEA officials, NAVSEA will budget for
future capital improvements for the Pearl Harbor NSY & IMF.
However, DOD officials are concerned that future funding levels
may be insufficient because of uncertainties in the DOD budgeting
and appropriation process.
Discussions of Other Financial Although other similar
regionalization efforts are likely, according to Navy
Structures Continue officials, they may not look like the Pearl
Harbor Pilot with direct
appropriations as the single financial structure. For example,
Navy officials are currently discussing using either direct
appropriations or the working capital fund to finance the combined
activity of future consolidations, or a combination of the two
financial structures. 11 One combination being discussed involves
using direct appropriations to finance the overhead and military
personnel of the combined facility while using the NWCF to finance
direct maintenance work and materials. Because they believe that
the working capital fund is more costly and less flexible than
direct appropriations, several Navy officials are reluctant to use
the fund to finance the cost of intermediate maintenance
activities, which are
currently financed with direct appropriations. Differing Views
Remain The Navy has made little progress toward resolving
questions over the Regarding Organizational appropriate
organizational structure for such consolidations. While the
Structure
proposal recommending the pilot's financial structure stated that
common ownership was believed to be beneficial to the Navy's
regional maintenance initiative, it needed to be tested by the
pilot. However, the Pearl Harbor Pilot Test Plan does not attempt
to measure the impact of providing PACFLT ownership and management
responsibilities for the pilot and Navy officials are uncertain
which command structure should be used for similar consolidations
at other locations.
Although the pilot was expected to streamline the maintenance
process and eliminate stovepipe organizations, DOD and Navy
officials do not know the extent to which the pilot's current
structure is meeting these objectives or how they might have been
met under other command structures. Prior to the pilot, the NIMF
operated as a PACFLT activity while the former shipyard operated
as a NAVSEA activity. Under the pilot, 11 In its comments on a
draft of this report, the Navy said that NAVSEA and the Fleet
Commanders had agreed now to operate future consolidations under
direct appropriations.
PACFLT assumed ownership and overall management and financial
responsibility for the consolidated activity while NAVSEA
continued to be the technical and operating authority. Although
CNO and NAVSEA officials viewed the duality of fleet ownership and
NAVSEA operation as not adding
another management layer, according to several DOD and other Navy
officials, this appeared to add a level of responsibility and
oversight in lieu of streamlining the former maintenance structure
and eliminating organizations. For example, several Pearl Harbor
NSY & IMF officials said that the current ownership and command
structure has resulted in the practice of reporting and
coordinating simultaneously with both
commands. In addition, there has been some confusion on which
command was ultimately responsible for collecting test data and
analyzing the results of the pilot. Navy officials are still
undecided about the appropriate organizational structure for these
types of activities and are considering different structures for
other regionalization efforts.
Conclusions Although the Pearl Harbor Pilot is not complete,
preliminary results have been mixed, showing either performance
improvements, no improvements, or insufficient data to determine
results. For most of the measures where
data are available, indications are that the pilot has the
potential to improve maintenance activities in Hawaii. However,
because of unique aspects of ship maintenance activities in
Hawaii, the pilot is likely to serve only as a general model for
future consolidation efforts and provides only a general
indication that future consolidations will result in similar
efficiencies. At the same time, DOD and Navy officials continue to
have different views on the potential impact of using direct
appropriations to finance the pilot on (1) the Navy shipyards and
activities remaining in the working capital fund, (2) ship
maintenance activities during periods without appropriations,
(3) cost visibility of ship maintenance activities, (4) incentives
inherent under NWCF's buyer/ seller relationship for improving
productivity and performance, and (5) the capital investment
program for ship maintenance activities. Other issues that are
still unresolved include determining whether the pilot's
organization structure under PACFLT ownership has helped
streamline the ship maintenance process and improve operations in
Hawaii.
Recommendations As the Navy proceeds with other similar
consolidations, we recommend that the Secretary of Defense require
the Secretary of the Navy to resolve
issues related to the appropriate mechanism to finance and manage
these
types of activities. Specific financial questions that need to be
resolved include the impact of using the direct appropriations to
finance the pilot and other potential regionalization actions with
regard to (1) the Navy shipyards and activities remaining in the
working capital fund, (2) ship maintenance activities during
periods without appropriations, (3) cost visibility of ship
maintenance activities, (4) incentives inherent under
NWCF's buyer/ seller relationship for improving productivity and
performance, and (5) the capital investment program for ship
maintenance activities. In addition, other questions that need to
be resolved include determining whether the pilot's command
structure under the Fleet's ownership has helped streamline the
ship maintenance process and
improve operations in Hawaii. Agency Comments and
On August 5, 1999, we requested comments on a draft of this report
from Our Evaluation the Secretary of Defense. On August 31, 1999,
officials of the Office of the Assistant Deputy Under Secretary of
Defense for Maintenance Policy, Programs, and Resources said that
they concurred with the intent of our recommendations. However,
DOD Comptroller officials expressed concern that the draft of this
report did not adequately address the Navy's
underlying assertions and assumptions for the pilot or whether the
stated objectives of the pilot were met. By recommending that the
Navy take steps to address unresolved issues related to financial
and organizational structures as it proceeds with similar
consolidations, we believe our report addresses the most
significant issues for such consolidations. Because the pilot is
not yet complete, we did not attempt to determine whether the
stated objectives were fully met but, instead, reported the
preliminary results of the pilot on improving performance and
maintenance activities in Hawaii. We believe that it could take
several years for the full effects of the pilot to develop and, as
requested, we are reporting the preliminary results. In addition,
Navy officials said that they generally concurred with the draft
of this report except for our use of fiscal year 1998 data to
indicate the preliminary results of the Pearl Harbor Pilot, and
they advised that fiscal year 1999 data would provide the most
valid assessment of the pilot. They
considered fiscal year 1998 as a transition year when the
maintenance activities, financial systems, and workforce were not
fully integrated and, consequently, did not indicate the pilot's
full performance. As discussed in this report, we also considered
fiscal year 1998 as a transition year for the pilot and reported
that the pilot was unlikely to result in significant improvements
during the transition year because of the operational turbulence
experienced during the consolidation. However, we used
available data from both the transition year and fiscal year 1999
to provide the preliminary results of the pilot. We continue to
believe that, in either year, overall indications are that the
pilot has the potential to improve maintenance activities in
Hawaii.
DOD and the Navy also provided technical corrections and
clarifications and, where appropriate, we incorporated them in the
report. We are sending copies of this report to Senator Pete V.
Domenici, Senator Daniel K. Inouye, Senator Frank R. Lautenberg,
Senator Carl Levin, Senator Joseph I. Lieberman, Senator Fred
Thompson, Senator Ted Stevens, Senator Charles S. Robb, and
Senator John W. Warner, and to Representative John R. Kasich,
Representative Jerry Lewis, Representative John P. Murtha,
Representative Ike Skelton, Representative Floyd Spence, and
Representative John M. Spratt, Jr., in their capacities as Chair
or Ranking Minority Member of cognizant Senate and House
Committees and Subcommittees. We are also sending copies of this
report to the Honorable William S. Cohen, Secretary of Defense;
the Honorable William J. Lynn, Under Secretary of Defense
(Comptroller); the Honorable Richard Danzig, Secretary of the
Navy; and the Honorable Jacob Lew, Director, Office of Management
and Budget. Copies will also be made available to others upon
request.
GAO contacts and key contributors to this report are listed in
appendix III. Sincerely yours,
David R. Warren, Director Defense Management Issues
Appendi I x Scope and Methodology During our review, we obtained
data and interviewed Departments of Defense (DOD) and Navy
officials, including those from the offices of the Under Secretary
of Defense (Comptroller), Deputy Under Secretary of Defense for
Logistics, DOD Inspector General, Assistant Secretary of the Navy
for Financial Management and Comptroller, Chief of Naval
Operations (CNO), Naval Sea Systems Command (NAVSEA), the Pacific
Fleet (PACFLT), Pearl Harbor Naval Shipyard & Intermediate
Maintenance Facility (NSY & IMF), San Diego Supervisor of Ship
Building Conversion and Repair Pearl Harbor Detachment, and the
Naval Audit Service (NAS). In addition, we interviewed officials
of the Hawaii Federal Employees
Metal Trades Council, the Ship Repair Association of Hawaii, and
contractors involved in the management and assessment of the Pearl
Harbor Pilot. To identify the preliminary results of the Pearl
Harbor Pilot, we obtained and reviewed status reports and
briefings, financial and human resources documents, infrastructure
data, budget documents, NAS capacity evaluation reports and
documents, the Pearl Harbor Pilot Test Plan and related guidance,
and PACFLT regional maintenance and business plans. In addition,
we interviewed DOD and Navy officials to discuss the preliminary
results of the pilot and challenges that must be resolved to
obtain the full benefits of the consolidation. We observed the
operations of the Pearl
Harbor NSY & IMF and interviewed superintendents, first- line
supervisors, and workers to identify operational conditions and
determine the pilot's impacts on their maintenance shops and
activities. We reviewed the Pearl Harbor Pilot Study Report and
other planning documents to identify maintenance procedures and
processes expected to improve under the
pilot and compared these expectations with actual operational
results for fiscal year 1998 and the beginning of fiscal year
1999. When these results did not meet expectations, we discussed
the differences with Navy officials to determine the causes and
identify expectations for future operations. In addition, we
interviewed officials of various PACFLT commands (customers of the
Pearl Harbor NSY & IMF) to determine whether the
combined facility's maintenance activities and services were
comparable to those provided prior to the consolidation.
Additionally, we interviewed Navy officials and collected data on
the apprentice program, safety and quality assurance programs, and
issues related to the morale of Pearl Harbor NSY & IMF.
To assess the adequacy of the Pearl Harbor Pilot Test Plan, we
reviewed the DOD Inspector General's preliminary assessment of the
test plan and baseline data. We also interviewed DOD, Navy, and
contractor officials
about the methodology for the test plan, reliability of the data,
and strengths and weaknesses of each test metric. To obtain an
understanding of the methodology and data reliability, we reviewed
the process and data used to develop the fiscal year 1997
baseline, the fiscal year 1998 transition results, and the fiscal
year 1999 preliminary results and, on a selected basis, we traced
the data to their original source documents. To identify potential
areas for improving the test plan, we discussed the strengths and
weaknesses of the test plan with DOD, Navy, and contractor
officials. We also discussed several key elements of the pilot
that were omitted from the test plan, including the reasons for
omission or the potential need for inclusion.
To identify issues that remain for future consolidations at other
Navy locations, we obtained and reviewed status reports and
briefings, financial and human resources documents, the Pearl
Harbor Pilot Test Plan and related guidance, and PACFLT regional
maintenance and business plans. In addition, we interviewed DOD
and Navy officials to discuss issues that must be resolved to
obtain the full benefits of the Pearl Harbor
consolidation and other similar consolidations at other Navy
locations. In addition, during meetings with DOD and Navy
officials, we discussed the advantages and disadvantages of
whether future consolidations should use direct appropriations,
the Naval Working Capital Fund (NWCF), or a combination of the two
financial systems. To help assess the potential impact of removing
the former Pearl Harbor Naval Shipyard (PHNSY) and other naval
shipyards from the NWCF, we identified and analyzed the percentage
of revenues the former PHNSY and the other shipyards contributed
to the fund during fiscal years 1997 and 1998. We also
discussed the potential impact of their removal from the fund with
DOD and Navy officials. Additionally, during meetings with DOD and
Navy officials, we discussed the advantages and disadvantages of
whether NAVSEA or the fleets should be responsible for the
ownership and management of future consolidations. To determine
whether the pilot could serve as a model for similar
consolidations at other locations, we identified and analyzed
reasons why the Navy selected Pearl Harbor maintenance activities
for the pilot. We compared these conditions with
those found at the Puget Sound, Portsmouth, and Norfolk areas and
discussed the potential impact of these differences on other
consolidations with DOD and Navy officials. Furthermore, we
documented the reasons the Navy selected direct appropriations,
and the pilot's experience of operating under direct
appropriations. We also interviewed DOD and Navy officials to
discuss the
actual level of cost visibility retained under the pilot and the
various levels considered adequate for ship maintenance
activities. In addition, we determined the level of cost
visibility retained for four ongoing
maintenance projects for the U. S. S. Charlotte, U. S. S. Russell,
U. S. S. Tucson, and U. S. S. Crommelin. Specifically, we reviewed
the start date, estimated date of completion, estimated cost,
direct labor hours of military and civilian personnel, overtime
hours, material and supply cost, and contract services for the
projects. In addition, to determine whether the same level of
detail was being maintained by the Pearl Harbor NSY & IMF under
direct appropriations, we reviewed material expenditures and
laborhours for military and civilian personnel by job order number
for the projects.
In performing this review, we used the same budget and accounting
systems, reports, and statistics DOD and the Navy use to manage
and monitor their ship maintenance program. We did not
independently determine the reliability of the reported financial
information. However, our recent audit of the federal government's
financial statements, including
DOD's and the Navy's statements, questioned the reliability of
reported financial information because not all obligations and
expenditures are recorded to specific budgetary accounts. We
conducted our review from May 1998 to July 1999 in accordance with
generally accepted government auditing standards.
Description of the Pearl Harbor Pilot Test Plan Appendi I I x and
Its Strengths and Weaknesses The Navy's test plan for assessing
the effectiveness of the Pearl Harbor Pilot evolved between
December 1997, when the Deputy Secretary of Defense first required
a test plan, and September 1998, when the last three
test metrics were added to the plan. During this evolution, DOD,
Navy, and audit agency officials expressed various concerns with
the scope and validity of the test plan. Ultimately, the Navy
selected nine test metrics to evaluate the productivity and
performance of the pilot. Metrics to Be Used to After several
revisions and additions, the current test plan contains nine
Evaluate the Pilot
metrics that capture unit cost per output, production efficiency,
material readiness of ships home ported at Pearl Harbor, and
customer satisfaction. During the initial planning period in late
1997, Navy officials discussed several metrics to measure the
impact of the pilot and initially agreed upon
the two shown in table 4.
Table 4: Navy's Initial Two Metrics to Test Whether the
Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production
Metric Data elements Calculation
1. Total cost of a production shop, direct Total costs of the
ship maintenance activity Total costs of the ship maintenance
labor hour of work delivered to the Total production shop, direct
labor hours delivered activity or activities divided by the
customer total production shop, direct labor hours delivered
2. Total available labor hours expended Total available labor
hours, including direct and
Total available labor hours divided by to deliver a production
shop, direct labor indirect hours, expended by the ship
maintenance the total production shop, direct labor hour to the
customer activity
hours delivered Total production shop, direct labor hours
delivered
Source: Interviews with Navy officials and our analysis of the
Pearl HarborPilot Test Plan dated June 30, 1998.
In December 1997, recognizing that the Navy had not developed a
plan or established a baseline for measuring success or failure
for the pilot, the Deputy Secretary of Defense required the Navy
to develop a test plan to measure the benefits of the pilot. 1 In
addition, the Deputy Secretary of Defense suggested that
productive man- days delivered per dollar outlay
and number of civilian end strengths and military on- board it
took to 1 Program Budget Decision Number 404, Deputy Secretary of
Defense (Dec. 11, 1997).
produce the productive man- days should be used as key assessment
metrics. Subsequently, the Chief of Naval Operations (CNO) and the
Naval Sea Systems Command (NAVSEA) requested the Naval Audit
Service (NAS) determine the combined cost of operations for the
former Pearl Harbor Naval Shipyard (PHNSY) and the Naval
Intermediate Maintenance Facility (NIMF) for fiscal year 1997. 2
In an April 1998 report, the NAS provided a basis to compare the
cost effectiveness of combining the two maintenance activities and
metric calculations for the two assessment
metrics defined at that time, which are the first two metrics of
the current test plan. 3 In the report, the NAS recognized that
the consolidation was an evolving process and revisions to its
report may be necessary to reflect changes in consolidation
decisions. In May 1999, after making changes to
the baseline data and fine- tuning its methodology, the NAS
revised its baseline cost and productivity estimates. 4 The
revised estimates did not change significantly from the initial
estimates. In April 1998, the Navy issued a draft of the Pearl
Harbor Pilot Test Plan that included five metrics to capture cost
per unit output, production efficiency and resource utilization,
material readiness of PACFLT ships, customer satisfaction, and
quality. However, in June 1998, the DOD Assistant Inspector
General for Auditing stated that the draft test plan would not
provide sufficient or supportable data to draw conclusions about
the effectiveness of consolidating the operations of the former
shipyard and intermediate maintenance facility. Consequently in
June 1998, a joint DOD and Navy team issued a revised Pearl Harbor
Pilot Test Plan that changed the basis for the quality metric from
formal customer surveys to an analysis of rework items and added a
metric measuring the number of
2 The Navy selected fiscal year 1997 as the baseline year because
this was the last full year the former PHNSY and NIMF operated as
independent activities. Fiscal year 1998 was eliminated because of
the operational turbulence expected by the consolidation of
activities during the year. Fiscal year 1996 was eliminated
because there were separate surface ship and submarine
intermediate maintenance facilities operating during the fiscal
year.
3 Capacity Evaluation Report: Baseline Costs of Operating Naval
Intermediate Maintenance Facility, Pearl Harbor and Pearl Harbor
Naval Shipyard, Auditor General of the Navy (98- 0400, Apr. 1998).
4 Capacity Evaluation Report: Revised Baseline Costs of Operations
and Metrics for Naval Intermediate Maintenance Facility, Pearl
Harbor and Pearl Harbor Naval Shipyard Consolidation, Auditor
General of the Navy (99- 0408, May 1999).
work- items completed. 5 Consequently, the Navy included four
metrics in addition to those two metrics listed in the previous
table to its final test plan. (See table 5.)
Table 5: Metrics Three Through Six to Test Whether the
Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production
Metric Data elements Calculation
3. Total Current Ship Maintenance Total number of CSMP work items
completed on
None required, absolute number of Program (CSMP) work items
completed ships home ported at Pearl Harbor during the fiscal
completed CSMP work items used year 4. Total Current Ship
Maintenance
Total number of CSMP work items in the backlog for None
required, absolute number of Program work items in the backlog
ships home ported at Pearl Harbor CSMP work items in the backlog
used 5. Schedule adherence of CNO
Sum of the differences in the actual and scheduled Sum of the
differences in the actual
maintenance projects completion dates for each CNO ship
maintenance and scheduled completion dates project completed
during the fiscal year
divided by the total scheduled Sum of the scheduled duration
(number of days) for duration (number of days) for each each CNO
maintenance project completed during
CNO scheduled ship maintenance the fiscal year
project completed during the fiscal year 6. Rework index for CNO
maintenance
Sum of the labor hours expended to correct work Sum of the labor
hours expended to projects deficiencies for each CNO scheduled
ship correct work deficiencies divided by maintenance project
completed during the fiscal the total number of direct labor hours
year
delivered for each CNO scheduled Total production shop, direct
labor hours delivered ship maintenance project completed for each
CNO scheduled ship maintenance project during the fiscal year
completed during the fiscal year
Source: Interviews with Navy officials and our analysis of the
Pearl Harbor Pilot Test Plan dated June 30, 1998.
In September 1998, the Principal Deputy Under Secretary of Defense
(Comptroller) suggested the Navy incorporate three additional test
metrics developed under contract by the Anteon Corporation into
the revised June 1998 test plan. 6 (See table 6.) In response, the
Acting Deputy Chief of Naval 5 The Pearl Harbor NSY & IMF has
developed three additional internal metrics: production- shop
hours versus total available hours, direct production worker
versus nonproduction worker, and civilian labor cost per labor
hour. While they are not formally part of the test plan, Pearl
Harbor NSY & IMF officials believe the metrics will assist them in
assessing the effectiveness of the pilot. 6 The Pearl Harbor Pilot
Study, Anteon Corporation (Sept. 10, 1998).
Operations (Logistics) attached them in an appendix E to the plan.
At this time, Principal Deputy Under Secretary of Defense approved
the test plan.
Table 6: Metrics Seven Through Nine to Test Whether the
Consolidated Pearl Harbor NSY & IMF Increased Manpower Utilization
and Lowered Unit Cost of Production
Metric Data elements Calculation
7. Activity work schedule Budgeted quantity of work scheduled
(labor days)
Budgeted allowance of work scheduled (labor integrity index
Actual quantity of work performed (labor days) days) divided by
the actual amount of work performed (labor days)
8. Casualty reports a caused by Casualty reports Analysis of
reports occurring within 6 months activity work following the
completion of maintenance 9. Earned value Actual quantity of work
performed (labor days) for Statistical analysis of actual
quantity of work selected Ship Work Line Item Numbers for the
fiscal performed (labor days) for selected Ship Work year
Line Item Numbers for fiscal years 1997 and 1999
a Reports filed by the ship of equipment failures that may
indicate faulty work performed during more recent maintenance
projects.
Source: Interviews with Navy and Anteon Corporation officials and
our analysis of the Pearl Harbor Pilot Test Plan dated June 30,
1998.
In November 1998, the Director, Industrial Capability, Maintenance
Policy and Acquisition Logistics Division, CNO, requested the NAS
to assume the overall lead and control in the test plan assessment
process. The Director requested that NAS involvement be expanded
to include data collection for all of the test plan metrics,
metric calculation and assessment, and pilot
metric assessment reporting. In response, the auditors agreed to
continue to develop and report data for the first two metrics in
the test plan and verify the data for metrics number 3 through 6.
Additionally, NAS considered the responsibility for data
collection and reporting the results of the pilot to be a
management function. It did not assume responsibility for
collecting and validating the data for the contractor- generated
metrics.
Baseline, Performance Given the time and data constraints in
developing the Pearl Harbor Pilot
Expectations, and Test Plan, the test plan and assessment metrics
provide the Navy a
mechanism for evaluating the pilot's productivity and performance.
As Preliminary Values for shown in table 7, the Navy has developed
the values for the fiscal year 1997 the Pearl Harbor Pilot
baseline and performance expectations for six of the nine metrics
and
preliminary values for five metrics. However, the details for
collecting and analyzing the contractor- generated metrics have
not yet been determined.
Table 7: Baseline, Performance Expectations, and Preliminary
Results of the Test Plan Metric Fiscal year 1997 baseline
Expectation Preliminary result
1. Total cost of a production shop, $135.36 in current year 3- 5
percent reduction $135.59 per hour in fiscal year 1998 direct
labor hour of work delivered dollars to the customer a
$138.13 in adjusted dollars b 2. Total available labor hours 3.
15 hours 3- 5 percent reduction 3. 04 hours in fiscal year 1998
expended to deliver a production shop, direct labor hour to the
customer a
3. Total Current Ship Maintenance 19,777 Maintain at least the
14,568 items in fiscal year 1998 Program work items completed c
same completion rate 7, 786 items as of May 1999
4. Total Current Ship Maintenance 17,733 10 percent reduction
16,462 items as of September 30, 1998 Program work items in the
backlog c 16,538 items as of May 1999
5. Schedule adherence of CNO 11.4 percent late To improve 18.6
percent late in fiscal year 1998
maintenance projectsc 3. 7 percent early in fiscal year 1999
through May 1999
6. Rework index for CNO 0. 086 No degradation of the
Not yet determined maintenance projectsc index 7. Activity work
schedule integrity
Not yet determined Unclear Not yet determined index 8. Casualty
reports caused by Not yet determined Unclear Not yet determined
activity work 9. Earned value Varies by work item Unclear Not
yet determined a Reported by the NAS. b Adjusted for increases in
military and civilian pay rates due to inflation. c Reported by
the Pearl Harbor NSY & IMF, but not yet verified by the NAS.
Source: Interviews with Navy and Anteon Corporation officials and
our analysis of the Pearl Harbor Pilot Test Plan dated June 30,
1998; NAS capacity evaluation reports for fiscal years 1997 and
1998; and pilot status reports.
Although the Navy has established nine metrics that provide
indicators of productivity and performance, the metrics have data
limitations and weaknesses that preclude their use for precise
measurements. Nevertheless, they provide a basis for an order of
magnitude comparison. A discussion of the nine metrics follows.
Cost of a Production Shop, Measuring the pilot's efficiency in
terms of the cost per direct maintenance Direct Labor Hour of Work
hour, the Navy's first metric is determined by dividing the total
activity cost Delivered
by the number of production shop, direct labor hours delivered to
the customer. 7 The NAS estimated that it cost $135.36 to deliver
one production shop, direct labor hour in fiscal year 1997 the
baseline year for measuring the pilot's success. For the pilot to
show improvement, this index should decrease, indicating that
since the consolidation, it cost less to deliver a direct
maintenance hour. The NAS estimated that the index was $135.59
during the fiscal year 1998 transition, indicating some
improvement when compared with the adjusted baseline of $138.13,
but falling short of Navy's performance expectation of 3- to 5-
percent reduction in the index. 8 The NAS adjusted the baseline to
account for differences in military and civilian wages and
inflation between fiscal years 1997 and 1998. Because of the
operational turbulence experienced during the consolidation, we
believe this indicator was unlikely to show a significant
improvement during the
transition year. Strengths and Weaknesses In analyzing this
metric, we noted strengths and weaknesses. The strengths of this
metric are that the baseline is partly based on shipyard data,
which
are considered reliable, and the NAS developed the baseline
estimates and plans to validate and generate the test data.
However, DOD and Navy officials have expressed concern with the
accuracy of the intermediate maintenance data collected during the
baseline year, which is considered a weakness. For example,
military personnel costs were not collected because intermediate
maintenance facilities traditionally did not collect these costs.
In addition, some production hours appeared to be inaccurate. For
example, a DOD contractor reported the former NIMF database
contained zero labor hours for approximately 6 percent of the
maintenance
tasks completed by the facility. Consequently, to strengthen the
metric, the NAS developed a methodology that used Navy personnel
wages and standards and the best available NIMF data to estimate a
baseline for this metric. Although the NIMF baseline data on
military personnel costs and 7 The production shops consist of the
crane, ship fitter, sheet metal, welding, inside machine, outside
machine, boilermaker, electrical, pipe fitter, wood/ fabric,
cryptographic shore electronics, electronics, shipwright/ rigging,
and temporary services shops and the
Navy calibration and nuclear regional maintenance departments. 8
Fiscal Year 1998 Interim Costs for the Consolidated Pearl Harbor
Naval Shipyard and Intermediate Maintenance Facility, Pearl
Harbor, Auditor General of the Navy (99- 0400,
June 1, 1999).
production hours have limitations, they are considered reliable
enough by Navy officials to provide a rough, if imprecise,
estimate of the baseline. Labor Hours Expended to Measuring the
pilot's productivity in terms of personnel utilization, the
Deliver a Production Shop,
second metric is determined by dividing the total number of
available labor Direct Labor Hour
hours by the number of production shop, direct labor hours
delivered to the customers. The NAS estimated that it took 3.15
activity labor hours (overhead and direct maintenance hours) to
deliver one production shop, direct labor hour in fiscal year
1997. For the pilot to show improvement, this index should
decrease indicating that it has taken fewer overhead hours to
generate 1 direct maintenance hour since the consolidation. The
NAS reported that the index decreased to 3.04 hours in fiscal year
1998, indicating a 3. 5- percent improvement during the transition
year. This met the low end of the Navy's performance expectation
of 3- to 5- percent reduction in value.
Strengths and Weaknesses In our analysis, we noted the strengths
and weaknesses of this metric. The strengths of this metric are
that the baseline is partly based on shipyard data, which are
considered reliable, and the NAS developed the baseline estimates
and plans to validate and generate the test data. However, as
discussed previously, DOD and Navy officials have expressed
concern with the intermediate maintenance data collected during
the baseline year. Consequently, to strengthen the metric, the NAS
developed a methodology that used best available NIMF data to
estimate a baseline for this metric. Although the NIMF baseline
data on production hours have weaknesses, they are considered
reliable enough by Navy officials to provide a rough, if
imprecise, estimate of the baseline. Current Ship Maintenance
Measuring the pilot's productivity in terms of output, the third
metric is the Program Work Items number of Current Ship
Maintenance Program (CSMP) work items Completed
completed by the Pearl Harbor NSY & IMF. Although this metric is
not an indicator of the former shipyard's productivity, Navy
officials have considered the number of CSMP work items completed
by their intermediate maintenance facilities as a measurement of
their productivity. 9 The CSMP is the central database for the
maintenance history of each ship and contains corrective,
preventative, and alteration 9 Traditionally, naval shipyards
completed a minimal number of CSMP items.
maintenance items requiring work. The Pearl Harbor NSY & IMF
estimated that the former intermediate maintenance facility
completed 19,777 CSMP work items in fiscal year 1997. However, it
further estimated that the consolidated activity completed 14,568
CSMP work items during the fiscal year 1998 transition and 7,786
work items as of May in fiscal year 1999, indicating no
improvement in performance since the consolidation. Although the
combined facility was expected to maintain the same
completion rate as the former NIMF, several Navy officials believe
that this expectation was unreasonable because the number of
military enlisted personnel decreased from 1,275 in October 1996
to 616 in April 1999. Traditionally, enlisted personnel completed
the majority of CSMP work items. Consequently, Navy officials have
discussed lowering the performance expectation for this metric.
Strengths and Weaknesses During our analysis, we found that this
metric had both strengths and weaknesses. The strengths of this
metric are that the fleet and ship commanders maintain the
baseline and test data, which are considered reliable, and the NAS
plans to validate the baseline and pilot results. However, other
factors not related to the pilot affect the metric that weaken its
usefulness. For example, Navy officials have expressed concern
with using CSMP work items to measure the benefits of the Pearl
Harbor Pilot because one work item does not equal another item in
terms of the labor hours, work skills, time frame, and material
cost required. CSMP items range from replacing a light bulb in a
control panel, which may take a worker less than an hour to
replace, to overhauling a pump, which may take several workers
days to complete. In addition, the requirements needed to overhaul
one pump differ from other overhauls depending on the problem and
the type of pump. Although this metric has weaknesses that
preclude its use for precise measurements, Navy officials believe
that it provides a basis for an order- of- magnitude comparison.
Current Ship Maintenance Historically, Navy officials have
measured the material condition of their
Program Work Items in the ships based on the number of backlogged
CSMP items. Fewer backlogged Backlog
items imply that the ships are in better condition. The Navy's
performance expectation for the combined facility was to reduce
the estimated 17,733- baseline backlog by 10 percent to 15,960
work items. Recently, the Pearl Harbor NSY & IMF estimated that
the backlog decreased to 16,538
items as of May 1999. Although this decrease indicates
improvement, it falls short of the Navy's expectation for the
pilot. However, it was unlikely that the CSMP backlog would
decrease significantly during this period because
of the reduction in the number of military enlisted personnel
since October 1996.
Strengths and Weaknesses As discussed previously, using CSMPs as a
test metric had both strengths and weaknesses. The strengths of
this metric are that it traditionally has been considered an
indicator of the material condition of fleet ships and that the
fleet and ship commanders maintain the baseline and test data,
which are considered reliable. In addition, the NAS plans to
validate the baseline and pilot results. For many of the same
reasons that CSMP work items are not a precise measurement of
output, the backlog is not a precise measurement of the ship's
material condition, which makes the backlog a weakness in regard
to measurement. For example, one work item related to the safety
of a nuclear submarine could potentially restrict the
submarine operations, while a similar item on a surface ship could
have little or no effect on its operations. Additionally, other
factors affecting the backlog include non- pilot related items
such as decommissions of PACFLT ships homeported at Pearl Harbor,
which decrease the backlog by the number of CSMP items recorded
for the
decommissioned ship; visits by the U. S. S. McKee (a maintenance
tender), which decrease the
backlog by the number of CSMP items completed by the tender;
maintenance inspections, which increase the backlog by the number
of unrecorded CSMP items identified by the inspection team; and
procedural changes in identifying and recording CSMP items, which
may increase or decrease the backlog depending on whether the
changes weaken or strengthen the process. Although this metric has
weaknesses that preclude its use for precise measurements, Navy
officials believe that it provides a basis for an orderof-
magnitude comparison.
Schedule Adherence of CNO According to Navy officials, adherence
to completing CNO maintenance Maintenance Projects
projects on schedule should provide an indication of customer
satisfaction because, from the fleet and ship commanders'
perspective, these maintenance projects should be completed on
schedule. 10 The performance 10 CNO maintenance projects include
depot- level maintenance that require skills or facilities beyond
those of the ship and intermediate maintenance activities and are
completed by public and private shipyards.
expectation during the pilot test was to reduce the 11. 4 percent
schedule adherence index achieved during fiscal year 1997, since a
lower percentile would indicate that projects were being completed
closer to their scheduled completion dates. 11 According to the
Pearl Harbor NSY & IMF, the schedule adherence index increased to
18.6 percent during fiscal year
1998 indicating a worsening performance during the transition
year. However, Pearl Harbor NSY & IMF data showed that the
schedule adherence index for the CNO maintenance projects
completed in fiscal year 1999 has exceeded the Navy's performance
expectations for the pilot. For example, although the first CNO
project completed in fiscal year 1999
was 20 percent late, the four subsequent CNO projects have been
either on time or early, decreasing the cumulative index to 3.7
percent early. Strengths and Weaknesses The strengths of this
metric are that it traditionally has been considered an indicator
of the shipyard customers' satisfaction and that NAVSEA maintains
the baseline and test data, which are considered reliable. In
addition, the NAS plans to validate the baseline and pilot
results. The weaknesses include that the metric is based solely on
the former shipyard's performance in meeting the completion dates
for CNO maintenance
projects and, consequently, does not include all types of
maintenance projects. Pearl Harbor NSY & IMF officials believe
this metric should include fleet maintenance projects because
there are more fleet
maintenance projects than CNO projects, thus affecting more ship
commanders. However, according to CNO and NAVSEA officials, they
included only CNO projects for this metric because the schedule
adherence data for fleet maintenance projects were considered of
minimal value. For example, the former intermediate maintenance
process was to defer work rather than to extend the duration of
fleet maintenance projects and, consequently, these projects were
generally completed on time. Because the pilot has not changed
this process for fleet maintenance projects, CNO
and NAVSEA officials believe an analysis of schedule adherence
data for these projects would be of minimal value in assessing the
pilot's effectiveness. 11 The schedule adherence index is the sum
of the differences in the actual and scheduled completion dates
divided by the total scheduled duration (number of days) for each
CNO ship maintenance project completed during the fiscal year.
Rework Index for CNO In analyzing this metric, we found strengths
and weaknesses. Measuring Maintenance Projects
the quality of work completed by the Pearl Harbor NSY & IMF, the
sixth metric is determined by dividing the total number of direct
rework hours required to correct work deficiencies by the number
of direct labor hours delivered for CNO maintenance projects
completed during the fiscal year. Rework hours are the hours spent
to correct work deficiencies after the maintenance project is
completed.
Strengths and Weaknesses The strengths of the metric are that it
traditionally has been considered an indicator of the shipyard
quality of work and the NAS plans to validate the baseline and
test data. However, its weaknesses include that the baseline for
this metric is limited to the shipyard's CNO projects because the
former intermediate maintenance facility did not collect rework
data for its projects. In addition, the metric has not been
computed for fiscal years 1998 or 1999 because no one has assumed
responsibility for tracking this information since the inception
of the pilot.
Activity Work Schedule Measuring the pilot's degree of schedule
integrity, the seventh metric is Integrity Index
calculated by dividing the budget quantity of work (direct labor
hours) scheduled by the actual amount of work (direct labor hours)
delivered to complete the maintenance project. According to the
test plan, actual work exceeding budgeted work indicates
inefficiencies in the facility's planning or production processes.
A metric value of more than one indicates the maintenance project
was completed within budget.
Strengths and Weaknesses The strengths of this metric are that it
traditionally has been considered an indicator of the shipyard
customers' satisfaction and the shipyard maintains the baseline
and test data, which are considered reliable. However, one
weakness of metric is that it does not include data from the
former intermediate maintenance facility. In addition, Navy
officials believe this metric is similar to the fifth metric
measuring schedule adherence and, therefore, is not significantly
useful in determining the benefits of the pilot. Consequently,
they have not expended resources to compute the metric
value for the baseline year or established performance
expectations for this metric. Furthermore, the details of
collecting and analyzing the data have not yet been determined.
According to CNO and NAVSEA officials, the Navy does not plan to
gather the data needed to develop and analyze this metric unless
the results of the first six metrics are inconclusive. For
example, if the pilot results for those six metrics are mixed,
then it may be necessary to measure other pilot results, such as
this metric.
Casualty Reports Caused by Measuring the quality of work completed
by the Pearl Harbor NSY & IMF, Activity Work the eighth metric is
based on analysis of casualty reports. These reports are
filed by the ship after any equipment failure and may help
identify work improperly performed by the Pearl Harbor NSY & IMF.
Those reports indicating improperly performed work may offer
insight to potential corrective actions to the facility's methods
and procedures.
Strengths and Weaknesses We noted strengths and weaknesses in our
analysis of this metric. The primary strength of the metric is
that the fleet and ship commanders maintain the baseline and test
data, which are considered reliable. However, considered a
weakness, the metric does not include data from the former
intermediate maintenance facility. In addition, Navy officials
believe this metric is similar to the sixth metric measuring
rework items and, therefore, is not significantly useful in
identifying the benefits of the
pilot. Consequently, they have not expended resources to determine
the details of collecting and analyzing the data for this metric.
Similar to the previous metric, the Navy does not plan to gather
the data needed to develop and analyze this metric unless the
results of the first six metrics are inconclusive.
Earned Value For the earned value metric, the process determines
the value of a completed task by comparing actual labor hours it
took to complete a unit
of ship maintenance work in fiscal year 1997 with actual labor
hours it took to complete the same unit of work in fiscal year
1999 and subsequent test years. This metric attempts to measure
the pilot's effect on maintenance outputs. Based on an analysis of
the former shipyard and NIMF production and labor data for fiscal
year 1997, the Anteon Corporation identified 14 surface ship and
submarine systems that were considered cost drivers, consuming the
most labor hours during the fiscal year. 12 On the basis of the
distribution of labor hours for each of these cost drivers, the
Anteon Corporation determined that, while the distributions are
nearly identical in form, they were not distributed normally at
the 99. 9 percent confidence level. The most common statistics
used to describe data normally
distributed are the mean and standard deviation. In this case,
however, the mean and standard deviations do not adequately
represent the data distribution. Consequently, non- parametric
statistics-which are generally
12 The Anteon Corporation defined a cost driver as a quantity of
ship maintenance work that represented a major or controlling
portion of work on a particular ship maintenance project.
based on the median (as opposed to the mean) -and percentile
ranges (as opposed to the standard deviation) of the distribution
of labor hours will be used to describe the labor hour
distribution for each of the cost drivers. If the distribution of
labor hours for the 14 cost drivers in fiscal year 1999 is less
than the fiscal year 1997 baseline, the earned value metric
indicates that the productivity of the work performed at the
combined facility has improved compared to when the facilities
operated separately. Conversely, if distribution of labor hours in
fiscal year 1999 is greater than the fiscal
year 1997 baseline, the productivity of the combined facility has
decreased compared to when the facilities operated separately.
Although it is expected that the cost drivers from fiscal year
1997 will be similar to those in fiscal year 1999, a statistical
method will be used to verify that the data for the different
fiscal years are similar. If the data are not statistically
similar, additional analysis of data will be conducted to develop
the earned value metric.
Strengths and Weaknesses A strength of this metric is that it
attempts to measure the pilot's effect on maintenance outputs.
However, the primary weakness of the metric, according to Navy
officials, is that it affects such a small population that the
metric is not significantly useful in measuring the efficiency of
the pilot. Another stated weakness is that the metric requires
significant resources to develop the results. Consequently, Navy
officials have not expended resources to determine the details of
collecting and analyzing the data for the metric. Similar to the
previous two metrics, the Navy does not plan to gather the data
needed to develop and analyze this metric unless the results of
the first six metrics are inconclusive.
Appendi x I I I GAO Contacts and Staff Acknowledgments GAO
Contacts David Warren (202) 512- 8412 Julia Denman (202) 512- 4290
Acknowledgments In addition to those named above, Mark Little,
Dennis De Hart, Jean Orland, Barbara Wooten, and John Brosnan made
key contributions to this report.
Related GAO Products Navy Ship Maintenance: Allocation of Ship
Maintenance Work in the Norfolk, Virginia, Area (GAO/NSIAD-99-54,
Feb. 24, 1999).
Defense Depot Maintenance: Public and Private Sector Workload
Distribution Reporting Can Be Further Improved (GAO/NSIAD-98-175,
July 23, 1998). Defense Depot Maintenance: Contracting Approaches
Should Address Workload Characteristics (GAO/NSIAD-98-130, June
15, 1998).
Defense Depot Maintenance: Use of Public- Private Partnering
Arrangements (GAO/NSIAD-98-91, May 7, 1998). Navy Ship
Maintenance: Temporary Duty Assignments of Temporarily Excess
Shipyard Personnel Are Reasonable (GAO/NSIAD-98-93, Apr. 21,
1998).
Public- Private Competitions: DOD's Additional Support for
Combining Depot Workloads Contains Weaknesses (GAO/NSIAD-98-143,
Apr. 17, 1998). Defense Depot Maintenance: DOD Shifting More
Workload for New Weapon Systems to the Private Sector (GAO/NSIAD-
98-8, Mar. 31, 1998). Public- Private Competitions: DOD's
Determination to Combine Depot Workloads Is Not Adequately
Supported (GAO/NSIAD-98-76, Jan. 20, 1998). Defense Depot
Maintenance: Information on Public and Private Sector Workload
Allocations (GAO/NSIAD-98-41, Jan. 20, 1998). Navy Regional
Maintenance: Substantial Opportunities Exist to Build on
Infrastructure Streamlining Progress (GAO/NSIAD-98-4, Nov. 13,
1997). Navy Depot Maintenance: Privatizing Louisville Operations
in Place Is Not Cost- Effective (GAO/NSIAD-97-52, July 31, 1997).
Defense Depot Maintenance: Challenges Facing DOD in Managing
Working Capital Funds (GAO/ T- NSIAD/ AIMD- 97- 152, May 7, 1997).
Defense Depot Maintenance: Uncertainties and Challenges DOD Faces
in Restructuring Its Depot Maintenance Program (GAO/T-NSIAD-97-
112, May 1,1997) and (GAO/T-NSIAD-97-111, Mar. 18, 1997).
Defense Outsourcing: Challenges Facing DOD as It Attempts to Save
Billions in Infrastructure Costs (GAO/T-NSIAD-97-110, Mar. 12,
1997).
High- Risk Series: Defense Infrastructure (GAO/HR-97-7, Feb.
1997).
GAO United States General Accounting Office
Page 1 GAO/NSIAD-99-199 Depot Maintenance
Contents
Contents Page 2 GAO/NSIAD-99-199 Depot Maintenance
Page 3 GAO/NSIAD-99-199 Depot Maintenance United States General
Accounting Office
Washington, D. C. 20548
B-280524 Page 4 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 5 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 6 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 7 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 8 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 9 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 10 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 11 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 12 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 13 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 14 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 15 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 16 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 17 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 18 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 19 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 20 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 21 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 22 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 23 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 24 GAO/NSIAD-99-199 Depot Maintenance
B-280524 Page 25 GAO/NSIAD-99-199 Depot Maintenance
Page 26 GAO/NSIAD-99-199 Depot Maintenance
Appendix I
Appendix I Scope and Methodology
Page 27 GAO/NSIAD-99-199 Depot Maintenance
Appendix I Scope and Methodology
Page 28 GAO/NSIAD-99-199 Depot Maintenance
Page 29 GAO/NSIAD-99-199 Depot Maintenance
Appendix II
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 30 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 31 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 32 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 33 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 34 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 35 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 36 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 37 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 38 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 39 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 40 GAO/NSIAD-99-199 Depot Maintenance
Appendix II Description of the Pearl Harbor Pilot Test Plan and
Its Strengths and Weaknesses
Page 41 GAO/NSIAD-99-199 Depot Maintenance
Page 42 GAO/NSIAD-99-199 Depot Maintenance
Appendix III
Page 43 GAO/NSIAD-99-199 Depot Maintenance
Related GAO Products Page 44 GAO/NSIAD-99-199 Depot Maintenance
(709349) Lett er
Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary,
VISA and
MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.
Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013
or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC
Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.
Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists. For information on how to access GAO
reports on the INTERNET, send an e- mail message with info in the
body to:
info@ www. gao. gov or visit GAO's World Wide Web Home Page at:
http:// www. gao. gov
*** End of document. ***