    Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs (02/11/2003, GAO-03-57}

    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-03-57
        TITLE:   Best Practices: Setting Requirements Differently Could Reduce Weapon Systems' Total Ownership Costs
         DATE:   02/11/2003

    This is the accessible text file for GAO report number GAO-03-57
    entitled 'Best Practices: Setting Requirements Differently Could Reduce
    Weapon Systems' Total Ownership Costs' which was released on February
    11, 2003.

    This text file was formatted by the U.S. General Accounting Office

    (GAO) to be accessible to users with visual impairments, as part of a

    longer term project to improve GAO products� accessibility. Every

    attempt has been made to maintain the structural and data integrity of

    the original printed product. Accessibility features, such as text

    descriptions of tables, consecutively numbered footnotes placed at the

    end of the file, and the text of agency comment letters, are provided

    but may not exactly duplicate the presentation or format of the printed

    version. The portable document format (PDF) file is an exact electronic

    replica of the printed version. We welcome your feedback. Please E-mail

    your comments regarding the contents or accessibility features of this

    document to Webmaster@gao.gov.

    Report to the Subcommittee on Readiness and Management Support,

    Committee on Armed Services, U.S. Senate:

    United States General Accounting Office:

    GAO:

    February 2003:

    BEST PRACTICES:

    Setting Requirements Differently Could Reduce Weapon Systems� Total

    Ownership Costs:

    GAO-03-57:

    GAO Highlights:

    Highlights of GAO-03-57, a report to the Subcommittee on Readiness and

    Management Support, Committee on Armed Services, U.S. Senate:

    February 2003:

    BEST PRACTICES:

    Setting Requirements Differently Could Reduce Weapon Systems� Total

    Ownership Costs:

    Why GAO Did This Study:

    For fiscal year 2003, the Department of Defense (DOD) asked for about

    $185 billion to develop, procure, operate, and maintain its weapon
    systems.

    This request represents an increase of 18 percent since 2001 for the
    total

    ownership costs of DOD weapon systems. Often, DOD systems need
    expensive

    spare parts and support systems after they are fielded to meet required

    readiness levels. DOD has been increasingly concerned that the high
    cost

    of maintaining systems has limited its ability to modernize and invest
    in

    new weapons. This report examines the best practices of leading
    commercial

    firms to manage a product�s total ownership costs and determines if
    those

    practices can be applied to DOD.

    What GAO Found:

    Even though DOD has implemented several initiatives to reduce total

    ownership costs, some systems, such as the Apache helicopter or the
    Abrams

    tank, have experienced costly maintenance problems and low readiness
    rates,

    which persisted even after the systems were fielded. We found several

    reasons for these problems. First, DOD based requirements for weapon
    systems

    in product development almost exclusively on technical performance,
    with

    little attention to operating and support costs and readiness at the
    beginning

    of development when there is the greatest chance of affecting those
    costs

    positively. Second, using immature technologies to meet performance
    goals

    weakened DOD�s ability to design weapon systems with high reliability.
    Finally,

    DOD�s organizational structure is linear and limits collaboration and
    feedback

    among organizations charged with requirements setting, product
    development,

    and maintenance.

    In contrast, commercial companies that we visited considered operating
    and

    support costs to be integral to their new product development
    decisions.

    Studies have shown that by the time a product is ready for development,
    over

    90 percent of the operating and support costs have been determined. As
    a

    result, these companies required their equipment be easy to maintain,
    ready

    when needed, and reliable at a low cost. These requirements were of
    equal

    importance to other performance characteristics. After setting
    requirements,

    product developers then designed products to meet established
    reliability

    rates, using technologies that were proven through past use or testing.
    At

    all of the companies we visited, customers and product developers
    alike, had

    very collaborative processes and practices that draw extensively on
    data from

    past operations to influence the design of new products.

    Figure: Percent of Life Cycle Costs Determined at Various Points in the

    Acquisition Process:

    [See PDF for image]

    [End of figure]

    What GAO Recommends:

    GAO recommends DOD (1) revise its guidance for setting requirements to
    include

    total ownership cost goals and readiness rates for any major weapons
    system as

    performance parameters equal to any others; (2) revise acquisition
    regulations

    to require a firm estimate of component and subsystem reliability by
    the systems

    integration phase and an estimate of system reliability at the
    production

    decision; and (3) structure contracts to ensure proper trade-offs
    between

    reliability and performance.

    www.gao.gov/cgi-bin/getrpt?GAO-03-57.

    To view the full report, including the scope and methodology, click on
    the

    link above. For more information, contact Katherine Schinasi at (202)
    512-4841.

    Contents:

    Letter:

    Executive Summary:

    Purpose:

    Background:

    Results in Brief:

    Principal Findings:

    Recommendations for Executive Action:

    Agency Comments:

    Chapter 1: Introduction:

    Total Ownership Cost Is the Cost to Ensure Readiness:

    Commercial Best Practices:

    Objectives, Scope, and Methodology:

    Chapter 2: DOD�S Requirements-Setting and Development Practices Yield

    Higher Total Ownership Costs:

    DOD�s Weapon System Programs Encounter Cost Growth in Achieving

    Readiness Rates:

    DOD�s Linear Acquisition Approach Makes It Difficult to Control

    Operations and Support Costs:

    Chapter 3: Commercial Companies Deliberately Manage Ownership Costs

    through Product Requirements and Design:

    A Best Practices Model:

    Leading Companies Treat Readiness and Operating and Support Cost as

    Critical Product Requirements:

    Knowledge-Based Product Development Is Critical to Achieving Desired

    Reliability and Managing Operating and Support Costs:

    Leading Commercial Firms Use Feedback from Operations to Better

    Understand Customer Needs, Product Deficiencies, and Operating and

    Support Costs:

    Chapter 4: Stressing Operating and Support Cost at the Outset of an

    Acquisition Could Help DOD Reduce Total Ownership Costs:

    Differences in Practices Explain Different Outcomes for Commercial

    Companies and DOD in Controlling Total Ownership Costs:

    Several DOD Efforts Underway to Reduce Total Ownership Costs:

    DOD�s Current Environment Does Not Provide Incentives to Reduce Total

    Ownership Cost Early:

    Chapter 5: Conclusions and Recommendations:

    Recommendations for Executive Action:

    Agency Comments and Our Response:

    Related GAO Products:

    Appendix I: Comments from the Department of Defense:

    Appendix II: GAO Staff Acknowledgments:

    Tables:

    Table 1: Readiness and Operating and Support Costs for Selected

    Weapons:

    Table 2: DOD and Commercial Practices for Controlling Operating and

    Support Costs:

    Figures:

    Figure 1: Nominal Life-Cycle Cost of Typical DOD Acquisition Program

    with a 30-Year Service Life:

    Figure 2: Percent of Operating and Support Costs Determined at Various

    Points in the Acquisition Process:

    Figure 3: Readiness, Reliability, and Operating and Support Costs:

    Figure 4: DOD�s Linear Acquisition Process:

    Figure 5: System Readiness Comes at High Operating and Support Costs

    When Reliability Is Not Ensured:

    Figure 6: Apache Helicopter:

    Figure 7: Abrams Tank:

    Figure 8: Commercial Model for Reducing Operating and Support Costs:

    Figure 9: Benefits of Ensuring High Reliability Rates During Product

    Development:

    Figure 10: Polar Tanker�s Polar Endeavor:

    Figure 11: United Airlines/Boeing 777:

    Figure 12: FedEx Express Delivery Van:

    Figure 13: Joint Strike Fighter:

    Figure 14: Advanced Amphibious Assault Vehicle:

    This is a work of the U.S. Government and is not subject to copyright

    protection in the United States. It may be reproduced and distributed

    in its entirety without further permission from GAO. It may contain

    copyrighted graphics, images or other materials. Permission from the

    copyright holder may be necessary should you wish to reproduce

    copyrighted materials separately from GAO�s product.

    This is a work of the U.S. Government and is not subject to copyright

    protection in the United States. It may be reproduced and distributed

    in its entirety without further permission from GAO. It may contain

    copyrighted graphics, images, or other materials. Permission from the

    copyright holder may be necessary should you wish to reproduce

    copyrighted materials separately from GAO�s product.

    Letter:

    February 11, 2003:

    The Honorable John Ensign

    Chairman

    The Honorable Daniel Akaka

    Ranking Minority Member

    Subcommittee on Readiness and Management Support

    Committee on Armed Services

    United States Senate:

    As you requested, this report examines how best practices offer

    improvements to the way the Department of Defense develops new weapon

    systems to reduce their total ownership cost, especially the operating

    and support costs, during design. It examines how the department

    currently designs for operating and support costs and how best

    practices could improve outcomes and reduce costs. We make

    recommendations to the Secretary of Defense for improvements to weapon

    system requirements policy, including establishing operating and

    support cost and readiness goals as performance parameters equal to any

    other performance parameters; revising acquisition policies to require

    a firm estimate of reliability during product development; and

    providing contract incentives for product developers to make

    appropriate trades between reliability and performance before

    production.

    We are sending copies of this report to the Secretary of Defense; the

    Secretary of the Army; the Secretary of the Navy; the Secretary of the

    Air Force; the Director of the Office of Management and Budget; the

    Director, Missile Defense Agency; and interested congressional

    committees. We will also make copies available to others upon request.

    In addition, the report will be available at no charge on the GAO Web

    site at http://www.gao.gov.

    If you have any questions regarding this report, please call me at

    (202) 512-4841. Other contacts are listed in appendix II.

    Katherine V. Schinasi

    Director

    Acquisition and Sourcing Management:

    Signed by Katherine V. Schinasi:

    [End of section]

    Executive Summary:

    Purpose:

    For fiscal year 2003, the Department of Defense (DOD) asked for about

    $184.9 billion to develop, procure, operate, and maintain weapon

    systems. With this funding, DOD will have received increases totaling

    about 18 percent since 2001 for what it defines as total ownership

    costs of its equipment. DOD�s budget for operations and maintenance

    increased by about 5.6 percent during the same period--from about $59

    billion to about $62.3 billion. DOD has been increasingly concerned

    that the high cost of maintaining weapon systems to meet required

    readiness levels is depleting modernization accounts and denying the

    department the flexibility to invest in new weapons. In recognition of

    this concern, DOD has established goals to reduce operating and support

    costs of weapon systems already in the field as well as those currently

    in development. In order to provide another perspective on this

    problem, GAO has continued a body of work to identify best practices

    used by leading commercial companies to manage new products� total

    ownership costs to see if these practices can be applied to DOD�s

    weapon system acquisitions.

    This report addresses how DOD can use best practices from commercial

    companies during its acquisition process to reduce total ownership

    costs of its major weapon systems. It presents a model of the process

    commercial companies use to incorporate reasonable and manageable

    operating and support cost into their product development process. In

    response to a request from the Chairman and Ranking Minority Member,

    Subcommittee on Readiness and Management Support, Senate Committee on

    Armed Services, GAO (1) determined the practices, processes and metrics

    DOD has historically used to manage and control operating and support

    costs of its major weapon systems; (2) determined the practices,

    processes and metrics commercial companies use to manage and control

    operating and support costs; and (3) analyzed the extent to which

    opportunities exist to apply best practices to reduce operating and

    support costs during product development.

    Background:

    Readiness is a critical parameter of all DOD�s weapon systems. If a

    system is not ready, its performance characteristics are of no use.

    Each weapon system has an expected readiness rate, usually expressed in

    some percentage of available units, that it is expected to maintain for

    our national security. Readiness can be achieved by building highly

    reliable weapon systems or, if the systems are not highly reliable,

    supporting them with an extensive logistics system that can ensure

    spare parts and other support items are available when needed. In

    essence, the cost of a product�s readiness is the cost to develop,

    produce, and operate and maintain that system.

    DOD recognizes that the total cost of a weapon system includes more

    than development and procurement costs. Traditionally, development and

    procurement have accounted for about 28 percent of a weapon�s total

    ownership cost, while costs to operate, maintain, and dispose of the

    weapon system account for about 72 percent of the total. For a number

    of years, the department�s goal has been to spend less on supporting

    systems and to devote more funds to development and procurement in

    order to modernize weapon systems. But, in fact, growth in operating

    and support costs has limited the department�s buying power. DOD

    officials have cited shortages of spare parts and unreliable equipment

    as reasons for low mission-capable rates for some weapons. As a result,

    some modernization has been postponed in order to pay high and

    unexpected operating and maintenance costs.

    GAO has issued a series of reports on best practices that commercial

    firms use to manage and control the acquisition cost of their products.

    Commercial firms attain knowledge early in the development process

    about technology that they plan to incorporate; they make sure the

    design is mature before production; and they have production processes

    under control before production begins. The building of knowledge in

    these areas can also contribute to the reduction of a product�s

    operating costs over its life cycle, thereby reducing a major portion

    of its total ownership costs. While those reports focused on best

    practices for reducing the cost and cycle time for acquiring weapon

    systems, this report will focus on what DOD can do during the

    acquisition of a weapon system, prior to and during product

    development, to ensure that products are available when needed, more

    cost effective to operate and support and more reliable once fielded,

    thereby reducing their total ownership cost.

    Results in Brief:

    DOD weapons acquisition processes do not consider operations and

    support costs and readiness as key performance requirements for new

    weapon systems, and DOD places less emphasis on establishing operations

    and support cost and readiness as key nontradable goals early in

    product development. Generally, the department settles for lower

    reliability in its new weapon systems� designs. In our review of data

    for five fielded weapon systems, we found none had established

    operating and support cost or readiness as key requirements. Although

    recent readiness levels were acceptable to the services for the most

    part, the five systems had experienced growth in their operating and

    support cost estimates of between 16 and 48 percent within the last 12

    years and problems with reliability once in the field.

    We found several reasons for cost growth. DOD�s acquisition process is

    linear, progressing from one organization to the next with little

    interaction among the groups. Requirements-setting by the war-fighting

    community focuses on system performance. DOD policy does not require

    inclusion of readiness or operating and support cost goals as key

    performance requirements equal in importance to other performance

    requirements. None of the systems we reviewed had a readiness or an

    operating and support cost goal as a key requirement. Further, during

    product development, the use of immature technologies and components to

    meet performance goals worked against designing weapon systems with

    high reliability. Using immature technologies also acts as a barrier to

    manufacturing techniques such as open systems[Footnote 1] or designing

    for fewer parts, practices that typically help reduce maintenance costs

    of the system and increase its reliability. DOD�s systems for

    accumulating data to analyze operating and maintenance actions on

    weapon systems already in the field do not provide adequate or reliable

    information, thus making it difficult for DOD to understand the total

    cost of operations and support. The outcome of these practices at DOD

    has been an inability to stem the continuous growth in total ownership

    cost, with actual operating costs continuing to exceed initial

    estimates. As a result, DOD continues to request more operating and

    support funding to sustain its systems or to reprogram funds from other

    accounts to pay the bills.

    In contrast, commercial companies consider reasonable operating and

    support costs and the readiness or availability of their equipment as

    requirements equal in importance to other performance characteristics,

    thereby ensuring that the developer places proper emphasis on achieving

    reliability and operating and support cost goals during product

    development. Commercial companies have a collaborative process for

    setting requirements, developing the product, and collecting and

    sharing data on maintaining and supporting the product once it is

    delivered. Both the customer and the developer have a voice in the

    process. During product development, especially during the design

    process, the maintainer has an active voice and is armed with

    information about operating and support cost drivers in the previous

    product. Commercial product developers maintain high standards for

    reliability, using proven technologies to achieve critical performance

    requirements. They find an evolutionary development process is critical

    to reducing operating and support costs and achieving high readiness.

    Emphasis is placed on reducing the number of parts in a design so there

    is less to maintain, using standardized parts that are readily

    available in the industrial base and using open systems to maintain

    competition. Once the product is delivered to the customer, maintainers

    keep detailed records on its reliability and the cost of its

    maintenance and support. Importantly, information on the product�s

    performance is communicated back to the developer to be used to improve

    the product.

    DOD has implemented initiatives to reduce the total ownership cost of

    its weapon systems. It has modified acquisition policies, established

    programs to reduce operating and maintenance costs in existing systems,

    and selected several acquisition programs to test different approaches

    to reduce life-cycle costs during development. However, these steps do

    not incorporate many of the practices used by commercial companies

    during requirements determination, product development, and fielding.

    In comparing DOD�s practices to those found at leading commercial

    firms, we have identified several differences. Because companies

    operate in an environment where operating costs and readiness are

    critical to their survival, commercial customers establish low

    operating and support cost and high readiness requirements when

    purchasing a product. This forces product developers to design reliable

    systems that are easy and relatively cheap to operate and maintain. The

    collaborative relationship between the customer and the product

    developer is essential to driving down operating and support costs.

    Further, companies understand that accurate operating and support cost

    data from current products are also necessary to make good decisions

    related to the purchase of a product, facilitate cost/performance

    trade-offs, and provide feedback to the manufacturer for continuous

    improvement.

    GAO is making recommendations to the Secretary of Defense on ways to

    improve DOD�s management of operating and support costs. We are

    recommending that DOD revise its requirements generation process to

    include total ownership cost, especially operating and support cost,

    and weapon system readiness rates as performance parameters equal to

    any others. We also recommend that any revision of the current policy

    governing acquisition processes require a firm estimate of the systems�

    reliability based on demonstrated reliability rates at component and

    subsystem levels no later than the end of the system integration phase

    and at the system level no later than the production decision. Finally,

    we recommend that DOD structure contracts by Milestone B, the start of

    the system development and demonstration phase, to ensure that proper

    trades are made between reliability and performance before the

    production decision.

    Principal Findings:

    DOD�S Current Practices for Setting Requirements and Developing New

    Weapon Systems Continue to Yield Higher Total Ownership Costs:

    DOD is spending more on operating and support costs for its weapon

    systems than it planned. We found three primary reasons for the high

    cost of operating and supporting DOD�s fielded weapon systems. These

    were (1) little or no attention to the trade-offs between readiness

    goals and the cost of achieving them when setting the key parameters

    for weapon systems;[Footnote 2] (2) the use of immature technologies

    during product development and delays in acquiring knowledge about the

    design and its reliability until late in development, or in some cases,

    production; and (3) insufficient data on the operations and maintenance

    costs and actions for fielded systems that would allow improvements in

    products currently in development. The outcome of these practices in

    DOD has been an inability to stem continuous growth in total ownership

    cost, with actual operating costs continuing to exceed initial

    estimates. As a result, DOD continues to request more operating and

    support funding to sustain its systems or reprogram funds from other

    accounts to pay the bills.

    Even though operating and support costs are the largest factor in a

    weapon system�s total ownership cost, they do not receive the same

    attention when requirements are set for a weapon system as other

    performance characteristics. In our review of data for five fielded

    weapon systems, we found that none had established an operating and

    support cost or a readiness goal as a key requirement prior to product

    development. In fact, operating and support cost estimates were not

    available in the Selected Acquisition Reports until at least 5 years

    into product development on these programs. Most of the fielded systems

    we reviewed were near or achieving readiness goals, but had experienced

    significant cost growth in operations and support cost estimates within

    the last 12 years to do so. Two of these systems, the Apache and the

    Abrams, were designated as the Army�s most expensive weapons to

    support. The C-17 reported a cost increase of almost 25 percent, and

    program officials stated that they would not have a firm estimate of

    operating and support cost until 2010--more than 25 years after the

    start of development.

    We found practices in three areas--requirements-setting, product

    development, and operations and maintenance--that contributed to this

    condition. DOD� s acquisition process is linear and serially involves

    several organizations whose responsibilities in the process have

    differing objectives. Communication among the different organizations

    is fragmented. Requirements focused on the weapon system�s performance

    characteristics. Once the weapon system�s requirements were set and the

    development of the system began, product development focused on

    achieving the program�s acquisition cost, schedule and performance

    goals, rather than on increasing reliability in order to reduce its

    total life-cycle costs. We found that the maintainers had limited

    involvement in making design trades for lower operating and support

    costs during development and that best practices such as designing for

    open systems or ease of maintenance were not used by the developer. We

    also found that once a system is fielded, the services� systems for

    tracking operating and support costs were suspect, providing inadequate

    feedback to suppliers and requirements setters.

    Commercial Companies Deliberately Manage Ownership Costs through

    Product Requirements and Design Process:

    We found that commercial companies that use capital equipment

    considered operating and support costs integral to their new product

    development decisions. Companies such as United Airlines, FedEx

    Express, and Polar Tanker employ practices to maintain the readiness of

    their fleets at as low an operating cost as possible. Reducing these

    costs translates into revenues, profits, and market growth. Increases

    to these costs can mean market failure. As customers, they have

    established operating and support costs and product readiness as key

    system requirements before development begins for a new product that

    are equal in importance to requirements for its performance and the

    cost to develop and produce the product. For example, United Airlines

    requires that new aircraft maintain a readiness rate of 98.5 percent or

    the manufacturer must reimburse it for lost revenue. Polar Tanker

    established a requirement that its Endeavor Class tanker operate at

    least 330 days a year at a reduced operating cost per tanker. These

    dual requirements drove trades during design, sometimes increasing

    development costs to achieve lower operating costs. Before FedEx

    Express agreed to a new design for its delivery trucks, it required

    that the new design last for at least 300,000 miles over a specified

    number of years and at a specific cost per mile. In gaining agreement

    with product developers on these requirements prior to product

    development, these companies sometimes had to trade performance or

    spend more in development, but they received more reliable products,

    reduced total ownership costs, and made those costs more predictable.

    To meet their customers� supportability requirements, we found that

    commercial product developers focused on designing a product that was

    easy to maintain, would be ready when needed, and reliable at low cost.

    They used an evolutionary development process. Consequently, they did

    not allow components or subsystems into a product�s design unless the

    technology had been proven reliable through past use or testing. For

    example, Boeing told us that it defers use of immature technology to

    later evolutions of design and makes the reliability ratings of its

    components available to the airlines before it begins product

    development. Maytag completed reliability testing on every new product

    prior to going to production. These companies also emphasized product

    designs with fewer parts and open systems. Maytag has established a

    parts reduction program as part of its development process, and Boeing

    built its 777 so that any of three engines--GE, Pratt & Whitney, or

    Rolls Royce--would fit. Developers also gained insight into design

    features their customers valued through regular communication with

    them. For example, the design for Boeing�s latest generation 737--

    geared toward reducing operating costs--was inspired by the airlines.

    Boeing emphasized open systems, standardized parts, and reduced parts

    from one generation to the next to satisfy the airlines� need for

    reduced operating costs.

    All of the companies we visited, customers and product developers

    alike, had very collaborative processes and practices for drawing

    extensively on data from past operations to influence the design of new

    products. This information was used as a baseline for new product

    designs and was used to estimate the operating costs of new products.

    United Airlines officials told us that the airlines and the

    manufacturers both keep meticulous reliability and cost records at all

    levels of an airplane--components, subsystems, and at the system level.

    Major operating costs drivers are tracked on a daily basis by the

    airlines, and the manufacturers usually have personnel residing with

    the airlines� maintenance crew to help solve problems on the spot and,

    just as importantly, to feed information back to the manufacturer so

    that the next product can be improved. FedEx Express and Polar Tanker

    both emphasized extensive data collection from current operations. In

    fact, FedEx Express sets annual targets for operating and support cost

    reductions based on data gathered on the road. Polar Tanker gathered

    maintenance data from past operations and established a team made up of

    its own maintenance personnel and outside consultants to determine

    areas that could result in higher reliability and lower maintenance

    costs in designing the new Endeavor Class tanker.

    Greater Emphasis on Operating and Support Cost at the Outset and during

    an Acquisition Program Could Help DOD Reduce Total Ownership Costs:

    DOD and the commercial companies we visited have policy goals of

    developing products that will meet customers� needs at the lowest

    possible cost to build and operate. The difference between them is in

    how each implements its policies. Leading commercial companies follow

    an integrated, collaborative process of setting requirements,

    developing the product, and ensuring that the product can be supported

    at an acceptable cost. DOD�s process is composed of disparate practices

    carried out by separate organizations with differing objectives and

    little communication among them about how to support fielded systems.

    While commercial firms focus on total ownership costs at the outset,

    DOD focuses mostly on technical performance. One cause of this is that

    in DOD the accountability and responsibility for total ownership costs

    are spread across many organizations with separate goals. Another cause

    lies in motivation for low costs. The commercial companies we visited

    are driven by the need to be as profitable as possible to survive, and

    low total ownership costs translate to higher profitability. DOD�s

    environment does not provide such incentives, and the organizations

    charged with acquiring and operating weapon systems are unconstrained

    by a need to lower costs since they can request additional operations

    and maintenance funding to keep systems working.

    Some of the practices used by commercial companies to reduce a new

    product�s operating costs during its development may be helpful to DOD.

    In setting requirements, commercial customers make readiness and

    operating cost requirements and collaborate directly with the product

    developer. Product developers establish sound cost estimates early;

    designs are simplified by using open systems and reducing parts;

    reliability testing is done early; and a reliability growth curve is

    established before production begins. Once a product is fielded,

    operating costs are managed to established targets; operating cost data

    is collected, analyzed, and used by the developer and the customer to

    develop more reliable products in the future; and continuous

    improvements are made to future products. The commercial practice of

    establishing readiness and operating cost as key requirements for a new

    product necessitates substantive input from operators and developers

    before and during product development. The commercial practices used

    during product development to design reliable systems that are easy and

    less costly to operate and maintain depend on the use of good product

    development practices including the use of mature technologies to meet

    requirements. Commercial firms use incremental product development

    processes and depend on a strong relationship between the manufacturer

    and the customer�s operators and maintainers to continue throughout

    product development.

    DOD does not focus on operating and support costs to the degree

    commercial companies do. In setting requirements, DOD does not make

    readiness and operating cost key parameters, performance is rarely

    reduced in favor of reliability or reduced operating cost, and there is

    no direct relationship between the requirement setters and the product

    developer. During product development, firm estimates of operating

    costs are not required, little attention is paid to reliability rates,

    and open systems or design for manufacturing techniques are rarely

    used. Once a weapon system is fielded, there is a lack of complete and

    reliable data available from the field, and there is little

    collaboration between maintainers and product developers to improve new

    systems. DOD�s acquisitions usually begin with critical technologies

    that are immature, with unproven reliability. This makes it difficult

    to implement best practices such as design for manufacturing during

    product development. Accurate operating and support cost data are not

    available for helping management make good decisions, facilitating

    cost/performance trade-off decisions, and providing feedback to the

    manufacturer for continuous improvement. On the weapon systems we

    reviewed, we found that the programs had poor initial estimates of

    operating and support costs for weapon systems, partly because they do

    not have reliable systems in place to track those costs per weapon

    system.

    DOD has taken some steps to lower its weapon systems� total ownership

    costs. Those actions include concurring with and implementing

    recommendations concerning the use of technology readiness levels,

    indicators of design maturity, and controlled production processes.

    Further, the department initiated pilot programs with 30 acquisition

    programs to develop methods for reducing total ownership costs.

    However, DOD�s current environment--both culturally and

    organizationally--is not presently conducive to applying them.

    Currently, its acquisition policies do not provide specific guidance

    for controlling total ownership cost and its requirements-generation

    policies provide no guidance for establishing readiness or cost goals

    for weapon systems once they are fielded.

    Recommendations for Executive Action:

    GAO recommends that the Secretary of Defense:

    * revise the Chairman of the Joint Chiefs of Staff Instruction 3170.01B

    on the requirements generation process to include total ownership cost,

    especially operating and support cost, and weapon system readiness

    rates as performance parameters equal in priority to any other

    performance parameters for any major weapon system prior to beginning

    the acquisition program;

    * revise the current policy governing the operation of the defense

    acquisition system (currently under revision) to require that the

    product developer establish a firm estimate of a weapon system�s

    reliability based on demonstrated reliability rates at the component

    and subsystem level no later than the end of the system integration

    phase, coinciding with the system-level critical design review, before

    proceeding into the system demonstration phase of product development;

    and at the system level no later than the full-rate production

    decision; and:

    * structure DOD contracts for major systems acquisitions so that at

    Milestone B the product developer has incentives to ensure that proper

    trades are made between reliability and performance prior to the

    production decision. One option is to provide specific clauses in the

    development contract to address reliability growth.

    Agency Comments:

    DOD partially concurred with all of our recommendations but, for the

    most part, found no further action was needed to lower total ownership

    cost. We disagree. We believe that if DOD takes no further action in

    implementing these recommendations, it would ignore significant

    opportunities to improve readiness by lowering total ownership cost in

    a budget environment that demands more effort to reduce these costs.

    A detailed discussion of DOD�s comments appears in Chapter 5 and the

    full text of DOD�s comments is in appendix I.

    [End of section]

    Chapter 1: Introduction:

    For fiscal year 2003, DOD asked for $184.9 billion to fund research and

    development, procurement and direct operations and maintenance costs of

    its weapon systems. These elements along with disposal costs are

    defined as the total ownership cost of a weapon system. The budget has

    increased by about 18 percent for these activities since 2001--with

    direct cost for operations and maintenance of weapon systems increasing

    by about 5.6 percent from $59 billion to about $62.3 billion. Since the

    late 1990�s, DOD has been increasingly concerned that the cost of

    operating and supporting weapon systems to meet required readiness

    levels is depleting its modernization accounts and denying the

    department the flexibility to invest in new weapons.

    Total Ownership Cost Is the Cost to Ensure Readiness:

    Commercial companies and DOD both use readiness as a key indicator of a

    product�s success. Readiness is usually expressed as the percentage of

    total units available and capable of performing a mission at any given

    time. If a weapon system is not ready when it is needed, its

    performance characteristics are of no use. In general, readiness can be

    achieved either by building highly reliable weapon systems or, if the

    systems are not highly reliable, supporting them with an extensive

    logistics system that can ensure spare parts and other support items

    are available when needed. In essence, the cost to ensure a product�s

    readiness is the cost to develop, produce, operate, and maintain that

    product through its life cycle.

    The development and production cost of a weapon system, also known

    as acquisition cost, usually represents about 28 percent of the weapon

    system�s total ownership costs. The acquisition cost is funded through

    DOD�s research, development, test and evaluation, and procurement

    accounts. These funds are used to mature new technology and design and

    manufacture new weapon systems. Operating and support costs[Footnote 3]

    are typically the highest portion of a weapon system�s total ownership

    cost because they represent the cost to operate the system and keep it

    ready for action over many years, sometimes more than 30 years. These

    costs are about 72 percent of the total ownership cost of a weapon

    system and are funded through DOD�s operations and maintenance account.

    Operating and support costs reflect the purchases of fuel, lubricants,

    and repair parts and their associated maintenance as well as

    modification kit procurement and installation. Figure 1 depicts the

    typical distribution of total ownership costs of DOD weapon systems

    over a 30-year life cycle.

    Figure 1: Nominal Life-Cycle Cost of Typical DOD Acquisition Program

    with a 30-Year Service Life:

    [See PDF for image]

    [End of figure]

    Figure 1 shows that the greater part of a weapon system�s total

    ownership cost is made up of its operating and support cost. While 72

    percent of the life-cycle cost of a weapon system is realized only

    after it is fielded, the decisions made during its acquisition--when

    its performance requirements are being established and its design is

    being matured--will dictate operating and support costs very early. In

    fact, studies show that about 85 percent of the operating and support

    costs of a weapon system will be determined as soon as requirements are

    set, while less than 10 percent of the life-cycle cost have been spent.

    By the time a product is ready for production, over 90 percent of the

    operating and support costs have been determined, and about 28 percent

    of the total life-cycle costs have been spent. Figure 2 illustrates

    this phenomenon.

    Figure 2: Percent of Operating and Support Costs Determined at Various

    Points in the Acquisition Process:

    [See PDF for image]

    [End of figure]

    Because so much of the eventual cost to support and maintain a weapon

    system is decided very early, it makes sense that more attention should

    be paid to supportability when the product�s requirements are being set

    and its design is being finalized. World-class commercial companies

    that either use or develop high-performing products know this and set

    requirements and designs accordingly.

    Commercial Best Practices:

    GAO has undertaken an extensive body of work that examines DOD�s

    acquisition issues from a different, more cross-cutting perspective--

    one which draws upon the lessons learned from the best commercial

    practices to see if they are applicable for DOD�s acquisition

    processes. Previous GAO best practices reports focused on what DOD

    could do to control product development costs that represent about 28

    percent of total ownership costs. This report will focus on best

    practices for reducing the largest segment of total ownership costs--

    the operating and support costs. The concepts discussed build on our

    previous reports that looked at earlier phases of an acquisition,

    including matching customer needs with available resources and

    designing and manufacturing products within cost, schedule, and

    performance goals. A complete list of best practices reports is at the

    end of this report:

    Leading commercial companies expect to obtain high-quality products

    that meet their expectations in terms of performance, price, and

    reliability. To ensure they make prudent buying decisions, they use a

    structured product development process that ensures a high level of

    knowledge exists about a product at key junctures during its

    development. Such a knowledge-based process enables decision makers to

    be reasonably certain that product quality, reliability, and timeliness

    are assured.

    * Knowledge point 1 occurs when a match is made between the customer�s

    needs and the available resources--technology, design, time, and

    funding. Commercial companies use this knowledge to meet essential

    product requirements, such as low operating and support costs. To

    ensure that the knowledge is attained, private companies require the

    product be demonstrated in its intended environment. In addition, the

    product developer must complete a preliminary product design, using

    systems engineering to balance customer desires with available

    resources.

    * Knowledge point 2 occurs when the product�s design demonstrates its

    ability to meet performance requirements. Program officials are

    confident that the design is stable and will perform acceptably when at

    least 90 percent of engineering drawings are complete. To obtain this

    knowledge, commercial companies use simulations and testing to fully

    understand how the product should be built.

    * Knowledge point 3 occurs when the product can be manufactured within

    cost, schedule, and quality targets and is reliable. Important

    indicators of this are when critical manufacturing processes are in

    control and consistently producing items within quality standards and

    tolerances. Private companies demand these critical manufacturing

    processes be in control because they could affect the product

    reliability.

    Objectives, Scope, and Methodology:

    The Chairman and the Ranking Minority Member, Subcommittee on Readiness

    and Management Support, Senate Committee on Armed Services, requested

    that we examine best practices for reducing total ownership cost of

    DOD�s weapon systems. This report primarily covers the operating and

    support cost portion of total ownership costs. Our overall objective

    was to determine whether commercial best practices for reducing total

    ownership costs, particularly operating and support costs, prior to and

    during the acquisition of weapon systems offer opportunities to improve

    outcomes in DOD�s acquisitions and its efforts to manage and reduce

    total ownership costs. Specifically, we (1) determined the practices,

    processes, and metrics DOD uses to manage and control total ownership

    costs of its major weapon systems; (2) determined the practices,

    processes, and metrics commercial companies use to manage and control

    total ownership cost; and (3) analyzed the extent to which

    opportunities exist for DOD to apply best practices to reduce operating

    and maintenance costs during product development.

    We used case studies of the following six DOD weapons, chosen to

    reflect all of the services across time, to examine DOD�s practices,

    processes and metrics:

    The Advanced Amphibious Assault Vehicle: This vehicle is the U.S.

    Marine Corps� replacement for its presently fielded amphibious assault

    vehicle. The new development vehicle is equipped with a 30mm automatic

    cannon and provides the capability to transport a Marine rifle squad at

    a speed of 20-25 knots in the water, and cross country with the agility

    and mobility equal to or greater than that of the M-1 tank. The

    contract for the Product Definition/Risk Reduction Phase was awarded in

    1996. The Marine Corps expects to buy 1,013 of these vehicles.

    Acquisition costs total $9.6 billion; operating and support costs,

    $16.0 billion.

    The Joint Strike Fighter program: This aircraft is the next-generation

    aircraft for the Navy, Air Force, Marine Corps, and Allies. There are

    three variants: a carrier variant will provide the Navy a multi-role,

    stealthy aircraft to complement the F/A-18E/F. The Air Force variant

    will be a multi-role aircraft, but primarily used in an air-to-ground

    role to replace the F-16 and the A-10 and complement the F-22. The

    Marine Corps variant will be a short-takeoff and vertical-landing

    aircraft to replace the Sea Harrier and GR-7 for the United Kingdom

    Royal Navy and Royal Air Force. The program is currently in system

    development and demonstration. Acquisition costs total $226.5 billion;

    operating and support costs, $387.6 billion.

    The Landing Platform Dock 17 ship program: These ships are used for

    transporting amphibious assault vehicles and other cargo. They

    incorporate both a flight deck for helicopters and a well deck to

    support landing craft. The contract for the detail design, integration

    and construction of the ship was awarded in 1996. The Navy expects to

    buy 12 ships to replace 27 older amphibious ships. Acquisition costs

    total $15.4 billion; operating and support costs, $56.5 billion.

    The Apache helicopter program: This Army helicopter�s mission is to

    find tanks and other armored targets and destroy them with its laser-

    guided Hellfire missiles, its 2.75-inch rockets, or its 30-millimeter

    gun. Apache development began in 1973 and the helicopter entered

    production in 1982 and was fielded in 1984. The Longbow Apache is a

    remanufactured and upgraded version of the Apache, which includes

    improved radar, engine and Hellfire missiles. The Army currently fields

    741 Apache and Longbow Apache helicopters. Acquisition costs total $ 19

    billion; operating and support costs for the Longbow Apache are $11.1

    billion.[Footnote 4]

    The C-17 cargo aircraft: The C-17 is a multi-engine, cargo aircraft

    expected to improve Air Force capability to rapidly deploy, reinforce,

    and sustain combat forces worldwide. The C-17 is capable of carrying

    outsized cargo over extended distances into unimproved airfields. The

    C-17 introduces a direct deployment capability that significantly

    improves airlift responsiveness. Development began in 1982 and the

    aircraft entered production in 1988 and was fielded in 1993. As of

    December 2002, the Air Force fielded 96 of 180 aircraft. Acquisition

    costs total $58.2 billion; operating and support costs, $144.9 billion.

    The M-1 series Abrams tank program: This tank is the Army�s main battle

    tank. The M-1�s development began in November 1972, entered production

    in 1979, and was fielded in 1986. The M1A2 is an improved version of

    the M-1, with improved armor, digital electronics, and an improved

    commander�s weapon station. The Army has fielded over 8,800 M-1 and

    M1A1 tanks, with about 1,000 upgraded to the M1A2 versions. Acquisition

    costs total $29 billion; operating and support costs, $16

    billion.[Footnote 5]

    For each of the six programs, we interviewed key managers and logistics

    representatives to discuss how operating and support costs were being

    managed and controlled during design. For the Joint Strike Fighter

    program we also visited the prime contractor, Lockheed Martin Aircraft

    Company, in Ft. Worth, Texas, and interviewed key designers and

    engineers. We analyzed operating and support cost data contained in

    Selected Acquisition Reports for the Apache Longbow helicopter, the

    C-17 cargo aircraft, the Abrams tank, the Advanced Amphibious Assault

    vehicle, Joint Strike Fighter and Landing Platform Dock 17. Information

    obtained from the Selected Acquisition Reports was not always

    consistent because three of the systems--the Advanced Amphibious

    Assault Vehicle, Joint Strike Fighter and Landing Platform Dock 17--are

    still in development. We supplemented information for the development

    systems with other program cost estimates. We also conducted limited

    analysis of the B-1, B-2, and F-22 operating and support cost and

    readiness data, based on information provided by their respective

    program offices and previous GAO reports.

    In addition to the case studies, we reviewed DOD policy, describing the

    roles and responsibilities of various organizations in the requirements

    development process. We discussed the implementation of these policies,

    particularly the role of the logistics community in the requirements

    determination process, with officials from each of the six program

    offices listed above, as well as appropriate officials from the:

    Under Secretary of Defense (Acquisition, Technology, and Logistics);

    Air Mobility Command, Scott Air Force Base, Ill.;

    437th Airlift Wing, Charleston Air Force Base, S.C.;

    Office of the Assistant Secretary of the Army, Washington, D.C.; and

    U.S. Army Aviation Center, Ft. Rucker, Ala.

    To determine the best practices, processes, and metrics commercial

    companies use to manage and control operating and support costs, we

    used a case study methodology by judgmentally selecting companies based

    upon general literature searches and discussions with experts. On this

    basis, we identified a number of commercial companies that have a

    structured and defined process for managing and controlling their

    operating and support costs. The following are descriptions of the six

    commercial companies and one quasi-governmental agency we visited:

    Boeing Commercial Aircraft designs and manufactures commercial

    airplanes. In 2001, it reported revenues totaling $58.2 billion. We

    visited its offices in Seattle, Washington, and discussed the

    development of the 737, the 767, and the 777 aircraft.

    Chicago Transit Authority is a quasi-governmental agency that operates

    the nation�s second largest public transportation system and covers the

    City of Chicago and 38 surrounding suburbs. In 2002, it reported an

    operating budget totaling $915 million. We visited its offices in

    Chicago, Illinois, and discussed the requirements determination

    process for acquiring buses.

    Allison Transmission, a division of General Motors, designs and

    manufactures transmissions for medium and large vehicles as well as

    military vehicles. In 2001, General Motors reported sales and revenues

    totaling $177.3 billion. We visited its offices in Indianapolis,

    Indiana, and discussed the development process for new transmissions.

    FedEx Express delivers packages, freight, and information to its

    customers worldwide. In 2001, it reported sales and revenues totaling

    $15.5 billion. We visited its offices in Memphis, Tennessee and

    discussed their process for setting requirements for a new package

    delivery vehicle.

    Maytag designs and manufactures major home appliances. In 2001, it

    reported sales and revenues totaling $4.1 billion. We visited its

    offices in Newton, Iowa, and discussed the development process for new

    appliances, particularly the Neptune washer and the Wide-By-Side

    refrigerator.

    Polar Tanker, a shipping division of ConocoPhillips Marine, manages the

    marine transportation of ConocoPhillips� Alaska North Slope crude oil

    production. In 2001, it reported sales and revenues totaling $26.9

    billion. We visited its offices in Avondale, Louisiana, and discussed

    the development process for the new Endeavor Class tanker, the Polar

    Endeavor.

    United Airlines, a commercial airline division for the UAL Corporation,

    is a major commercial air transportation company, engaged in the

    transportation of persons, property, and mail throughout the U.S. and

    abroad. In 2001, it reported sales and revenues totaling $16.1 billion.

    We visited its offices in Seattle, Washington; and San Francisco,

    California; and discussed the product development process and

    maintenance activities for the Boeing 777 aircraft.

    At each of these organizations, we conducted structured interviews with

    representatives to gather uniform and consistent information about

    their processes, practices, and metrics for controlling operating and

    support costs. During these meetings, we obtained a detailed

    description of the practices and processes they believe are necessary

    and vital to control and reduce operating and support costs. We met

    with managers of reliability, maintainability, and new aircraft

    development; general directors of operations; controllers; directors of

    configuration and integration; and principal engineers. We did not use

    examples from Chicago Transit or from Allison Transmissions, but

    discussions with those firms helped to refine the commercial model and

    inform our analysis of commercial best practices.

    During the past 5 years, we have also gathered information on operating

    and support costs from such companies as 3M, Chrysler Corporation,

    Caterpillar, Bombardier Aerospace, Ford Motor Company, Hughes Space and

    Communications, and Motorola Corporation. This information enabled us

    to develop an overall model to describe the general practices,

    processes, and metrics leading commercial companies take to control

    operating and support costs.

    This report highlights several best practices in controlling operating

    and support costs based on our fieldwork. As such, they are not

    intended to describe all practices or suggest that commercial companies

    are without flaws. Representatives from the commercial companies

    visited told us that the practices and processes, which we considered

    best practices, evolved over many years and that the practices continue

    to be improved based on lessons learned and new ideas and information.

    They admit that the application and the use of these practices have not

    always been consistent or without error. However, they strongly

    suggested that the probability of success in controlling operating and

    support costs is greatly enhanced by the use of these practices and

    processes.

    We conducted our review between August 2001 and February 2003 in

    accordance with generally accepted government auditing standards.

    [End of section]

    Chapter 2: DOD�S Requirements-Setting and Development Practices Yield

    Higher Total Ownership Costs:

    DOD is spending more on operating and support costs for its weapon

    systems than it planned. We found three primary reasons for the high

    cost of operating and supporting DOD�s fielded weapon systems. These

    were (1) little or no attention to the trade-offs between readiness

    goals and the cost of achieving them when setting the key parameters

    for weapon systems; (2) the use of immature technologies during product

    development and delays in acquiring knowledge about the design and its

    reliability until late in development, or in some cases, production;

    and (3) insufficient data on the operations and maintenance costs and

    actions for fielded systems that would allow improvements in products

    currently in development. The outcome of these practices in DOD has

    been an inability to stem continuous growth in total ownership cost,

    with actual operating costs continuing to exceed initial estimates. As

    a result, DOD continues to request more operating and support funding

    to sustain its systems or reprogram funds from other accounts to pay

    the bills.

    Three key groups are involved in DOD�s process to get a weapon system

    to the war fighter. First, the war fighter�s service-based requirements

    community establishes requirements for a new system. Second, the

    service-based acquisition organizations design and produce a product.

    Finally, after the product is developed and produced, it is turned over

    to the war fighter�s operating and maintenance communities, who have

    the responsibility to operate and maintain it. Decisions made in

    setting requirements very early in product development have the most

    impact on the subsequent costs of supporting a system. Trade-offs

    during the design process can also be significant. In DOD, the focus in

    the requirements and development process is to establish and meet

    technical war-fighting performance capabilities, and when trade-offs

    are made, they are usually to optimize those capabilities. The

    maintainers often come into this process very late and have little

    influence. At the end, DOD has no alternative but to pay the operating

    and support bills that accrue in order to maintain readiness.

    DOD�s Weapon System Programs Encounter Cost Growth in Achieving

    Readiness Rates:

    A weapon system�s operating and support cost will depend to a great

    extent on its performance characteristics, expected readiness rate, and

    the overall reliability of its design. If a weapon system has a very

    high expected-readiness rate but its design is not reliable, its

    operating and support costs will be high and unpredictable. Conversely,

    if the design has been thoroughly tested for reliability and is robust,

    the cost to operate and support it will be lower and more predictable.

    Ideally, there is a balance that customers and product developers can

    strike between readiness and operating cost. Figure 3 illustrates this

    balance.

    Figure 3: Readiness, Reliability, and Operating and Support Costs:

    [See PDF for image]

    [End of figure]

    A product developer can opt to drive higher reliability into the

    product during its development by reducing technical requirements,

    using highly reliable and proven components, or investing more in early

    testing. Those decisions would increase the product�s reliability and

    consequently improve prospects for readiness and reduce operating costs

    across the life cycle of the product.

    We reviewed five weapon system programs currently in the field and

    found that most had experienced significant growth in their estimated

    operating and support cost. We also found that none of the programs

    established goals for readiness or operating and support cost as key

    parameters for the weapon system prior to launching the acquisition

    program. In addition, we found that once fielded, some systems were not

    achieving the readiness rates that program officials thought were

    possible during development. Table 1 shows whether systems had

    specified readiness and operating and support cost goals as key

    requirements as well as the growth in operating and support costs that

    the systems have experienced within the last 12 years.

    Table 1: Readiness and Operating and Support Costs for Selected

    Weapons:

    Weapon: Apache; System readiness

    as a key parameter: No; Operating and support cost

    goals as a key parameter: No; Readiness: planned/actual: 62/73 %;

    Percentage growth for

    operating and support cost: 48 %.

    Weapon: Abrams; System readiness

    as a key parameter: No; Operating and support cost

    goals as a key parameter: No; Readiness: planned/actual: 90/93    ;

    Percentage growth for

    operating and support cost: 24   .

    Weapon: C-17; System readiness

    as a key parameter: No; Operating and support cost

    goals as a key parameter: No; Readiness: planned/actual: 92/84    ;

    Percentage growth for

    operating and support cost: 25   .

    Weapon: B-2; System readiness

    as a key parameter: No; Operating and support cost

    goals as a key parameter: No; Readiness: planned/actual: 70/42    ;

    Percentage growth for

    operating and support cost: 35   .

    Weapon: B-1; System readiness

    as a key parameter: No; Operating and support cost

    goals as a key parameter: No; Readiness: planned/actual: 67/64    ;

    Percentage growth for

    operating and support cost: 16   .

    Source: DOD (data), GAO (analysis).

    [End of table]

    While some systems have maintained their expected readiness rates,

    they have experienced between 16 and 48 percent growth in estimated

    operating and support cost. It is reasonable to conclude that the

    systems have not achieved the reliability rates that were needed to

    meet their expected readiness goals and, as a result, had to expend

    more funds on parts and labor in the field than were planned for

    maintenance. Two of the systems, the Apache and the Abrams, were

    designated as the Army�s most expensive weapons to support. The C-17

    has already experienced a cost growth of 25 percent in its operating

    costs, but program officials stated that they would not have a firm

    estimate of operating and support costs until 2010--more than 25 years

    after the start of development.

    DOD�s Linear Acquisition Approach Makes It Difficult to Control

    Operations and Support Costs:

    Traditionally, DOD has used a linear approach to setting requirements

    and developing a product. It focuses attention during product

    development on achieving revolutionary performance goals while trying

    to keep acquisition costs for a program as low as possible. Often, it

    is not until the system is fielded and responsibility shifts to other

    agencies or the services that the operating and support costs become an

    overriding concern. By this time, there is no alternative but to pay

    the bills that accrue to maintain readiness, no matter the cost.

    Three key groups are involved in DOD�s process to get a weapon system

    to the war fighter. First, requirements representatives from the war-

    fighting community establish a need for a new system, and the combat

    developers formulate a set of operating performance requirements to

    address the need. Requirements concerning how available the system must

    be and the cost to operate and support it are not considered key

    performance parameters. Second, DOD establishes an acquisition program

    office to begin product development and coordinate design development

    and production with the defense contractor. Often, the technologies and

    components needed to achieve tough performance requirements are new and

    unreliable; however, the program manager is responsible for developing

    and producing the weapon system within certain acquisition costs and

    schedule guidelines. Third, after the product is developed and

    produced, the war fighter has the responsibility to operate and

    maintain it. Although decisions made in setting requirements very early

    in product development have the most impact on the cost to support the

    system, the personnel who maintain it have less influence on the

    product development process because the focus is on achieving difficult

    performance requirements.

    Figure 4 briefly describes DOD�s process for managing a new weapon

    system�s requirements development, acquisition, and fielding. The most

    notable aspect of the model is that there is little communication or

    input from the maintenance community early in the process--during

    requirement setting and product development--when decisions will be

    made that will significantly influence the cost to maintain and support

    the weapon system. The model shows that, traditionally, DOD�s processes

    are separate and independent of each other. First, requirements are set

    independently of the maintainers and the product developers. Second,

    once the product development process begins, the focus is on maturing

    technologies and achieving a producible design that will meet the

    technical performance requirements for the weapon system. Finally, the

    operator is tasked to use and maintain the system that has been

    developed and produced.

    Figure 4: DOD�s Linear Acquisition Process:

    [See PDF for image]

    [End of figure]

    The goal of this process is to field a high-performing weapon system

    that will satisfy the needs of the war fighter better than any other

    weapon system available. We previously reported that competition for

    limited funding both within and among the services leads to performance

    requirements that will make the particular weapon system stand out from

    existing or alternative systems. [Footnote 6] Those that provide the

    greatest leap forward in promised performance stand the best chance of

    winning the funding. As a result, the design for the weapon system is

    usually based on undemonstrated components and subsystems that, when

    integrated into a weapon system, have low reliability and, ultimately,

    high operating and maintenance costs over their lifetime. Figure 5

    illustrates this phenomenon. The demonstrated reliability of the new

    system is lower, causing an imbalance in the relationship between

    readiness and operating cost toward the need for higher costs to

    maintain readiness.

    Figure 5: System Readiness Comes at High Operating and Support Costs

    When Reliability Is Not Ensured:

    [See PDF for image]

    [End of figure]

    Logistics officials at Air Mobility Command told us that even though

    they are represented in the requirements determination process for new

    weapon systems, they view their role as providing input on how the

    logistics community could support performance requirements of a new

    system, not on forcing trade-offs that would reduce operating and

    support costs. The officials also said that the logistics community

    leads many reliability improvement and cost reduction initiatives once

    aircraft are fielded. However, this approach is harder and more costly

    to implement than if reliability and operating and support costs had

    been considered when requirements were set and product development

    began. Further, the program must compete against other programs for

    operating and support funds to implement the upgrades and improvement

    initiatives.

    Operating and Support Cost Estimates Are Provided Later in DOD�s

    Process:

    Supportability aspects of system performance, such as operating the

    system at the lowest possible cost and the percentage of time the

    system has to be available for operations, are not given the same high

    priority as enhanced performance. Operating and support costs are not

    estimated until much later in development. For example, the B-2 bomber,

    Apache helicopter, and Abrams tank programs did not publish an initial

    estimate for the operating and support costs for those weapon systems

    in the Selected Acquisition Reports until more than 15 years after the

    start of development. The C-17 program did not publish an estimate

    until more than 7 years after development began. These estimates were

    not in the form of goals or key performance parameters. Throughout

    product development, then, design goals for reliability are not

    supported by the war fighters� need to meet operating cost and

    availability goals.

    Maturing the Technology Is the Focus of Development:

    In DOD, requirements for new weapon systems are usually based on the

    enhanced performance deemed necessary to achieve a certain war-fighting

    mission with little hesitancy in using new technology or what the cost

    to support it may be. The user representatives define system

    performance with limited input from the product developers and

    maintainers.

    A case in point is the B-2 bomber. The low-observable requirement for

    the B-2 bomber could only be met with technology that was immature at

    program launch, and this technology continued to cause problems

    throughout development, production, and fielding. The B-2 program began

    full-scale development in 1981, and the Air Force began low-rate

    initial production concurrently with development in November 1987. By

    1991, problems with the B-2�s low-observable material were still being

    reported, causing delays in delivery and cost increases because the

    material was difficult to manufacture. Once fielded, the low-observable

    materials required very high maintenance. The processes to repair them

    were time consuming and required an environmentally controlled repair

    facility. Poor durability and extensive maintenance kept the aircraft

    from achieving its planned availability. All of these factors are due

    to decisions to proceed with product development without understanding

    this technology.

    A more recent example is the F-22 fighter. The requirements for the

    F-22 were very demanding. Performance characteristics included low

    observability, super cruise speed, and fused avionics. These

    requirements caused the product design to include many new and unproven

    technologies. During product development, the program planned to

    achieve a system-level reliability rate for the F-22 of 3 flying hours

    mean time between maintenance actions when fully mature. The Air Force

    had estimated that in late 2001, when the F-22 entered limited

    production, it should have been able to demonstrate almost 2 flying

    hours between maintenance actions. However, when the F-22 actually

    began limited production, it could only fly an average of .44 hours

    between maintenance actions. In other words, the F-22 is requiring

    significantly more maintenance actions than planned. To date, the

    program has identified about 260 types of failures, such as main

    landing gears wearing out more quickly than planned, fasteners being

    damaged, and canopy delaminating, all leading to unanticipated

    operating and support costs.

    Product Development in DOD Goes Forward in Spite of Poor Prospects for

    Reliability:

    Our previous work identifying best practices during product development

    concluded that during this phase, the tasks are to ensure the stability

    of the design and to ensure that the product can be produced.[Footnote

    7] In DOD, the product developer frequently is trying to catch up to

    design and production tasks because product developments begin with

    immature technology. Schedule concerns override the need to capture

    knowledge about the design and production processes, and programs often

    proceed through development and into production before the reliability

    of the subsystems and systems has been demonstrated. Design features

    such as open architectures that allow systems to receive upgrades as

    technology advances or reductions to the number of parts in a design

    that reduce the need for spare parts and maintenance time are not given

    due consideration, even though they could lower operating and support

    costs of the system.

    As the schedule tightens, the lack of knowledge becomes more

    acceptable, even preferred. Reliability testing is often pushed closer

    to fielding, resulting in supportability problems being identified

    during operational testing. The Office of the Director, Operational

    Test and Evaluation has commented that operational testers identified

    reliability as a problem in almost every program, because the product

    developer places more emphasis on performance requirements than

    suitability. According to the operational testers, many systems enter

    operational test and evaluation with known, but unresolved reliability

    problems.

    Once a system is fielded, unless the reliability and ease of

    maintenance have been incorporated into the design already, there are

    limited opportunities to improve these metrics without costly redesign

    or retrofit. As the product�s design becomes firm and the system is

    produced and fielded, the opportunities to influence these costs

    diminish. For example, the Army is currently attempting to reduce

    operating and support costs on its Apache helicopter and its Abrams

    tank. These efforts should have a favorable impact on the systems�

    operating and support costs. However, the Army is retrofitting and

    replacing components that were identified as problems much earlier in

    the programs. Examples from the Apache and Abrams development

    illustrate the problems.

    Figure 6: Apache Helicopter:

    [See PDF for image]

    [End of figure]

    The Apache program focused on acquisition costs, schedule, and

    performance during product development, even when problems were

    identified that would impact reliability and maintainability. Today,

    the Apache helicopter is the Army�s most costly system in terms of

    operating and support costs with those costs rising over the years. For

    example, the target acquisition and designation system enables the

    Apache to find targets and guide its weapons. It is the helicopter�s

    most sophisticated system, involving 26 major electrical, optical and

    mechanical components. However, the sight requires frequent

    maintenance, and its complexity reduces its reliability. The pilot�s

    night vision system is also a highly complex system that experienced

    problems in development. Because its target acquisition and designation

    system and the pilot�s night vision system proved unreliable, the

    Apache incurred higher than expected costs to maintain the helicopter�s

    availability.

    During development, problems identified with the Apache foreshadowed

    future support problems. These problems included excessive vibration

    and excessive aircraft weight in addition to difficulty in achieving

    reliability rates for the aircraft�s target acquisition and designation

    system. During development, problems with the Apache�s fault detection

    system led to questions concerning whether it could operate safely and

    reliably during operations. Further, Army test and evaluation agencies

    warned that these problems could cause serious supportability issues

    since they would result in frequent need for maintenance and repair. By

    1990, the majority of the Apache helicopters had been produced and

    fielded, but Apache could achieve only 50 percent availability rates,

    well short of their 70 percent goal. Tests showed that the Apache

    required maintenance actions to correct significant problems every 2.5

    flying hours while the Army�s goal was 4 hours between failures.

    Maintenance units were physically unable to handle the repairs required

    to keep the helicopter flying. Subsequently, the Army added 18

    maintainers to the Apache battalion, constructed more contractor repair

    facilities, and hired more contractors. During Operation Desert Storm,

    although the system was effective in destroying tanks and other

    targets, it continued to experience reliability and logistical support

    problems that grounded some aircraft.

    Figure 7: Abrams Tank:

    [See PDF for image]

    [End of figure]

    Likewise, the Army produced the Abrams tank without first resolving

    reliability problems. The tank provided a major improvement in speed,

    agility and lethality over the older M-60 tank and was also supposed to

    decrease the operating and support cost burden. However, during

    development, the tank experienced serious failures with the track, the

    engine, fuel filters, the fuel and water separator, and the fuel pump.

    In particular, the durability and reliability of the Army�s turbine

    engine was a major concern. Tests performed up to the time of the

    initial production decision showed that the tank generally met its

    performance requirements. However, frequent breakdowns and component

    failures raised serious questions about approving the tank�s

    production. In a 1993 operational test report from the Office of the

    Director, Operational Test and Evaluation, the reviewer concluded that

    the tank was operationally effective, but not operationally suitable

    because of its many support problems such as those mentioned above.

    Today, the Abrams is the second most costly Army system in terms of

    operating and support costs and accounts for half the repair parts

    costs by the entire ground combat fleet. The tank engine is a major

    contributor to the tank�s high support costs. In recent years, the

    operating and support costs for the Abrams have remained steady at

    about $2 billion per year, although the Army has reduced the number of

    tanks to be supported by more than 300.

    Data on Operating and Support Costs Is Not Reliable:

    When estimating operating and support costs for a new weapon system or

    trying to establish maintenance trends, maintenance data on the current

    system are an important source of information. However, the three

    services� operating and support data collection systems do not provide

    accurate and reliable information about the cost to operate and

    maintain the systems because they do not collect and maintain data on

    all elements of the weapon system�s operating and support cost. Without

    reliable information, the services cannot assess trends or identify top

    cost drivers and take corrective action.

    The Army�s Operating and Support Management Information System provides

    historical cost data on Army weapon systems and is the primary source

    of operating and support data used by the program managers to project

    costs of new systems, forecast spare parts budgets, and generally

    manage their programs. The Army uses this data system to develop its

    operating and support cost budget for weapon systems for consumable

    items such as repair parts, petroleum, oil, lubricant, fuel, and

    ammunition as well as for some maintenance. However, the system does

    not provide a complete and reliable basis for developing and reporting

    the costs of weapon system support. The data are often incomplete and

    12 to 18 months out-of-date. Several important cost elements used to

    establish the Army�s operating and support budget estimates--such as

    contractor logistics support, supply maintenance and software support-

    -are not included in the database.

    The Navy�s central tracking system for operating and support costs,

    like the Army data collection system just described, provides

    historical cost data on weapon systems. Navy program officials told us

    that this data is often inaccurate and cannot be relied upon to

    pinpoint causes of failure or maintenance actions.

    Until recently, tracking Air Force operating and support costs was

    difficult because the cost data were unavailable in a usable format or

    were of poor quality. Poor cost data weakened operating and support

    cost estimates. In 1998, the Air Force set up a total ownership cost

    database, with the objective of providing accurate and reliable data to

    identify cost drivers and support decision makers in making

    improvements to fielded aircraft. However, the new system is not

    available at all aircraft maintenance locations, and therefore data may

    not be complete. In any case, the Air Force will need several years of

    cost data before it can evaluate the effectiveness of the system.

    [End of section]

    Chapter 3: Commercial Companies Deliberately Manage Ownership Costs

    through Product Requirements and Design:

    Leading commercial companies consider the total ownership cost of a

    new product, including its operating and support costs, integral to

    their new product development decisions. Companies that use airplanes,

    trucks, and ships to deliver goods and people such as United Airlines,

    FedEx Express, and Polar Tanker understand the importance of

    maintaining the readiness of their fleets at as low an operating cost

    as possible. Reducing these costs results in increased revenues,

    profits, and market growth. Increases in these costs, on the other

    hand, can result in failure. The companies also understand that unless

    reliability is designed into a product, there are limited opportunities

    to improve readiness and reduce cost without costly retrofit or

    redesign. They have been successful in reducing these costs because

    they developed a collaborative process with companies that develop

    those products, such as Boeing, for setting the product�s requirements,

    developing the product with operating and support costs in mind, and

    capturing accurate operations and support data once it is delivered.

    The companies we visited that bought products all established the

    product�s operating cost and its availability as requirements equal to

    its performance characteristics prior to product development, thereby

    ensuring that the developer placed priority on those goals during

    product development. They were amenable to reducing the product�s

    performance features to reduce its operating cost, as long as

    performance was within acceptable limits for achieving market

    objectives. They also considered bearing additional cost for the

    product�s design if it resulted in a net benefit from reduced operating

    costs. Once product development began, maintainers had a strong voice

    in the product�s design and leading product developers set high

    reliability standards for components they chose, using proven

    technologies to achieve performance requirements. Companies, such as

    Boeing and Maytag, chose an evolutionary approach to product

    development to achieve goals for life-cycle costs, testing extensively

    and early for reliability. These companies also emphasized other

    practices to reduce operating costs such as reducing the number of

    parts in a product�s design, using standardized parts, and using open

    systems to ease maintenance and maintain competition. Once the product

    is delivered, maintainers keep detailed records on its reliability and

    maintenance and provide that information to the developer to improve

    future products.

    A Best Practices Model:

    Figure 8 represents a model of the best practices that were most

    helpful to the companies we visited in achieving high reliability and

    reducing a product�s operating and support costs. Notably, the most

    critical events--those that have the most impact on a product�s

    operating and support costs--take place very early, either prior to

    product development when the product�s key requirements are

    established, or very early in product development before the design is

    finalized. Another notable aspect of the model is that each of the

    activities--requirements-setting, product development, and operations-

    -depend on clear and constant communication and collaboration among the

    customer, the product developer, and the maintainer from the time a

    product is conceived until the operator disposes of it.

    Figure 8: Commercial Model for Reducing Operating and Support Costs:

    [See PDF for image]

    [End of figure]

    The goal of this process is to develop and field a product that will

    perform in accordance with the customer�s needs and will be ready when

    needed within cost targets. As illustrated above, decisions about the

    product�s performance and cost are finalized prior to beginning the

    product�s development, and the costs to operate and support the product

    are key considerations. Once the requirements are set, the product

    developer can focus as much on achieving a reliable and robust design

    for the product as on achieving its performance goals. Finally, once

    the product is fielded, those responsible for its operation and

    maintenance continue to feed information back to the developer to

    improve future designs. Figure 9 illustrates what happens when the

    commercial firms we visited set requirements that force the product

    developer to consider reliability rates during design. When both

    operating cost and readiness are key requirements for the product

    developer, the developer focuses on using as many components and

    subsystems as possible that have demonstrated reliability rates. The

    customer, by demanding readiness at a certain cost as a hard

    requirement upfront has raised the importance of achieving it from the

    very start.

    Figure 9: Benefits of Ensuring High Reliability Rates During Product

    Development:

    [See PDF for image]

    [End of figure]

    Leading Companies Treat Readiness and Operating and Support Cost as

    Critical Product Requirements:

    We visited three companies--Polar Tanker, an operator of large oil

    tankers; United Airlines; and FedEx Express--to determine their

    practices for ensuring low operating and support costs from the

    equipment they purchased from product developers. All three companies

    believe that understanding and controlling the cost to operate and

    support a product while it is being designed is essential to driving

    down the total ownership cost of a product. To do this, they set

    requirements for a new product�s availability and its operating and

    support costs equal in importance to requirements for its performance

    and acquisition cost. We found these companies set requirements before

    development begins that their products be ready almost 100 percent of

    the time at the lowest operating cost possible. They typically set

    maintenance goals that drive operating and support cost decisions, such

    as maintenance cost per mile over a product�s lifetime. The following

    summaries illustrate the commercial processes.

    Figure 10: Polar Tanker�s Polar Endeavor:

    [See PDF for image]

    [End of figure]

    Polar Tanker:

    Polar Tanker is a commercial oil-transporting firm that recently

    decided that a new oil tanker was necessary to haul oil between Prince

    William Sound and Puget Sound. The company�s critical requirements for

    the new Endeavor tanker, which Polar Tanker believed would reduce the

    cost of delivering oil, were:

    * less expensive operations and maintenance over a 30-year life cycle

    (versus the industry standard of a 20-year life cycle); and:

    * availability for operations at least 330 days a year.

    Polar Tanker teamed its maintenance engineers with industry consultants

    to ensure these requirements were met. The procurement team relied on

    its archived maintenance data from previous Alaskan operations to

    develop its double-hulled tankers. It documented locations, lengths and

    types of fractures, and stresses in the structures of its existing

    inventory of ships. This documentation from past operations was used to

    determine structural changes required to reduce maintenance on the new

    Endeavor class. As a result of their record keeping and the constant

    communication with the product developer, Northrop Grumman�s Avondale

    Shipyard, Polar Tanker�s owners were successful in redesigning their

    ships and meeting their requirements for improved performance,

    reliability, and lower operations and maintenance costs.

    The dual requirement of reduced operating and support cost coupled with

    high readiness rates drove design trades that increased development

    costs but improved reliability. For example, once Polar Tanker�s

    procurement team identified ballast tank maintenance as one of the most

    significant maintenance burdens, it directed Northrop Grumman to use

    the best and most expensive epoxy coatings and specialized paints to

    protect the tanks from corrosion. Another design trade--adding

    additional structure to the ship�s hull to reduce the impacts of

    fatigue cracking--increased the acquisition cost of the tanker but

    improved reliability and reduced the need for maintenance. Polar Tanker

    accumulated data from its current fleet and conducted extensive

    modeling of the hull design to understand where the most critical

    cracking occurred and to identify operating and support cost drivers.

    It hired an engineering consulting company to conduct further analysis.

    As a result, Polar Tanker and Northrop Grumman utilized the most

    current design tools to optimize the ship�s structure. Polar Tanker

    also developed a list of equipment and suppliers based on reliability

    analysis and incorporated that list into the design contract with

    Northrop Grumman. Polar Tanker estimated that these design trades cost

    about $25 million in additional design costs, but they believe the

    changes will ensure its tankers will be able to operate more reliably

    over 30 years.

    Polar Tanker�s procurement team also required that the new Endeavor

    Class design use open systems when possible. It worked closely with

    Northrop Grumman, its contractor, to ensure requirements for an

    integrated bridge system that consisted of commercial-off-the shelf

    components with open systems to provide the capability to modernize the

    system much less expensively as technology improved. For example,

    systems in the tanker�s wheelhouse including the autopilot, marine

    radars, bridge control console, and satellite communication equipment

    used the open systems concept. The design also included easily

    accessible decks to minimize delays and disruption during maintenance.

    Polar Tanker has already upgraded its electronics since putting its

    first ship, the Polar Endeavor, into service and experienced minimal

    disruption in operations.

    Figure 11: United Airlines/Boeing 777:

    [See PDF for image]

    [End of figure]

    United Airlines:

    As the launch customer for the new Boeing 777, United Airlines

    established stringent requirements for aircraft readiness and operating

    costs, thereby ensuring that reliability would be an important design

    element. When United and Boeing negotiated the requirements, United

    stated that it wanted a twin-engine airplane that could fly extended

    ranges from any airport in the United States. In addition, United

    required that the 777 be available at the gate for departure within 15

    minutes of scheduled departure 98.5 percent of the time. Boeing

    guaranteed United that the 777 would meet the departure requirement by

    the third year of operation or Boeing would pay a financial penalty.

    According to United officials, the 98.5 percent rate was achieved by

    the third year. United also specified that operating and support costs

    for the 777 be no higher than on past airplanes. The agreement reached

    with Boeing was that Boeing would reimburse United for revenues lost as

    a result of airplanes being unavailable. By setting requirements for

    operating cost and readiness, United ensured that Boeing would build

    reliability into the design of the 777.

    Boeing brought together a working group of customers--the leading

    commercial airlines--to discuss requirements for the new design. Boeing

    officials told us that during those requirements meetings the

    participating airlines defined major design initiatives for the Boeing

    777 based on estimates of life-cycle costs. Initially there were

    differences among the airlines as to what exactly was needed on the new

    aircraft--from wider fuselages to additional electronics--but from the

    very first meeting, the airlines were all equally concerned with

    operating and maintenance costs. They were focused on designing an

    aircraft that would be easy for mechanics to repair. The airlines

    unanimously agreed that an airline maintenance representative was

    needed to adequately address operations and maintenance requirements.

    Boeing named a chief mechanic who had previously worked for United to

    the working group that was influential in defining the maintenance

    requirements. Although Boeing provided its customers engineering

    estimates for the operating cost of its new aircraft in comparison to

    the older model, United officials said they developed their own

    estimates based on its historical experience with Boeing aircraft.

    United officials said that having two perspectives from which to

    consider its purchase was helpful.

    Figure 12: FedEx Express Delivery Van:

    [See PDF for image]

    [End of figure]

    FedEx Express:

    The FedEx Express mission is to provide global air and ground

    transportation of high-priority goods and documents that require rapid,

    time-certain delivery. This mission demands high availability and

    reliability from its delivery equipment. In purchasing its newly

    designed fleet of delivery trucks, FedEx Express considered

    reliability, maintainability, and low operating and support costs to be

    critical measures of a successful acquisition. For example, improving

    availability and reliability were the key drivers in its acquisition of

    the new 700 cubic foot truck. FedEx Express collaborated with a product

    developer, Freightliner, to set the requirements for a newly designed

    vehicle with high reliability and endurance to withstand frequent

    stops, short travel distances between stops, and demanding use of the

    brakes. During the requirements-setting process, they also established

    cost and reliability requirements for the new truck that estimated an

    assumed number of stops per day, a certain number of miles per year at

    an assumed cost per mile.

    To make sure all costs, particularly operating and support costs were

    considered during product development, FedEx Express had a logistics

    manager lead their discussions with Freightliner for the development of

    the new truck. The FedEx Express logistics managers told us that they

    are required to reduce the company�s operating and support costs, and

    the company believes that it is essential to give these managers an

    integral role during these discussions. The new delivery truck has been

    successful in meeting its reliability and maintenance goals. The truck

    currently averages 70,000 miles between breakdowns and is operating

    within the cost per mile of service that was set as a requirement by

    FedEx Express at the beginning of development.

    Knowledge-Based Product Development Is Critical to Achieving Desired

    Reliability and Managing Operating and Support Costs:

    To meet the readiness and operating and support requirements that their

    customers demanded, we found that product developers focused on

    designing a product that was durable, easy to maintain, ready when

    needed, and reliable at low cost. They used an evolutionary development

    process to meet these requirements and did not allow components or

    subsystems into a product�s design unless their reliability had been

    proven through past use or testing. Boeing told us that it makes the

    reliability ratings of its components available to the airlines before

    it begins product development, and Maytag did extensive reliability

    testing on every new product prior to going to production. The

    companies also emphasized product designs with fewer parts and used

    open systems architectures as much as possible. Developers also gained

    valuable insight into design features their customers valued that drove

    down operating and support costs. For example, the design for Boeing�s

    latest generation of the 737--geared toward reducing operating costs--

    was inspired by the airlines� request for more affordable operations

    and maintenance.

    Once defined, the functional and operating and support requirements are

    tightly managed and controlled to minimize scope increase during

    product development. Companies work within a common framework to

    provide management oversight and control. To move a product

    successfully from concept to operations, the companies we visited used

    a �gated� product development process and firm criteria to dictate when

    a product is ready to exit a stage. The design reviews address all

    operating and support cost requirements of the products. Senior

    management review teams grant product approval at each gate only after

    determining that business cases adequately address major drivers of

    operating and support cost as well as reliability goals.

    Boeing:

    Boeing uses a structured, gated product development process to define,

    evaluate, and approve projects and to integrate new technology into its

    aircraft. This process separates technology development from product

    development programs. In fact, Boeing keeps immature or untested

    technologies in a research and development phase until they have been

    tested for reliability in a realistic environment. The process forces

    the company to obtain purchase agreements from customers and build a

    business case that shows the expected profitability of a product line

    before detailed designs are developed and a large dollar investment is

    made in manufacturing. In order to get these purchase agreements,

    Boeing product teams work collaboratively with potential customers to

    set requirements for readiness, reliability, performance, acquisition

    costs, and operating and support costs. Because aircraft are large

    capital investments, customers are working with other developers to

    make sure that the competitive forces of the market will help them get

    the best aircraft at the lowest cost.

    Boeing officials told us that to remain competitive in this

    environment, they focus on meeting customer requirements at the highest

    reliability leading to the lowest operating and support cost possible.

    For example, when it was designing the new 737, Boeing used maintenance

    records to prove to the airline that they could redesign the 737 with

    high reliability and reduced operating and support costs, a key market

    requirement for that aircraft. According to Boeing officials, they

    prepared a comparative analysis of the acquisition and operating and

    support costs for older 737 aircraft with the estimated costs for the

    737 Next Generation. This analysis showed that operating and support

    costs were significantly lower for the proposed next generation 737 and

    resulted in big savings to the customer. Boeing was able to develop

    good operating and support cost estimates for potential customers

    because it used an evolutionary approach for developing new aircraft.

    This approach allowed Boeing to improve performance and to insert new,

    reliable, and mature technology.

    Once a new product line is approved, Boeing continues to work with its

    customers to identify improvements that can be made to an aircraft in

    terms of parts reduction, parts standardization, and ease of

    maintenance. For example, Boeing formed four airline working groups

    with representatives from 21 airlines around the world to focus on

    maintainability, interiors, power plant, auxiliary power units, and

    common display system issues on the 737. Their objective was to collect

    feedback from operators on design changes that would reduce maintenance

    costs. Examples of airline-driven design changes include:

    * simplifying the wing flaps by eliminating one third of the parts,

    designing a simpler flap mechanism, and making parts removable with

    common grip length fasteners;

    * reducing engine removal and installation time by increasing the on-

    wing life of the engine by 40 percent and reducing the predicted change

    time from 12 hours to 6 hours; and:

    * improving the reliability and standardizing parts of the fuel system

    by using a fuel shutoff valve that is common with other aircraft.

    To facilitate and improve communication with its customers, Boeing

    oftentimes collocates customer representatives at its production

    facilities. Boeing has found that involving the customer in early

    design decisions improves their ability to design a reliable aircraft

    that is easy and less costly to maintain. For example, when it began

    developing the 777 aircraft, airline maintenance workers offered over

    5,000 suggestions for changes to the design based upon their

    experiences with other Boeing aircraft. These suggestions helped reduce

    parts and improve reliability of the 777, resulting in increased time

    between maintenance actions. Boeing also utilized the concept of open

    systems to reduce the total ownership cost of the 777 by allowing

    customers to choose from three different types of engines--GE, Pratt &

    Whitney, or Rolls Royce--depending on their needs.

    Maytag:

    Maytag follows a similar product development process. For Maytag, the

    most critical phase is when design specifications for product

    reliability and manufacturing feasibility are fully defined and

    understood. Early in its product development, an integrated product

    team takes all new product features through rigorous reliability growth

    analysis to determine what can be expected from the design and whether

    it will be able to meet the requirements. Maytag uses failure analyses,

    mock-ups, and other simulation tools to focus testing on the most

    critical elements of the new design and reduce the number of design/

    build/test iterations of the product. Even though these analyses

    require more upfront planning, Maytag officials stated that they still

    cut testing times in half and yet improved reliability and lower costs.

    During this phase, Maytag employs two project leaders, both a

    marketing manager and technical systems engineer, to conduct cost

    performance trades and co-chair subsystem design reviews. As a result

    of this co-leadership, most design issues are resolved immediately, and

    80 percent of all reliability testing and cost reductions take place

    early in the process. Also, during this phase the project leaders

    present their strongest business case, fully disclosing feasibility

    data, product definitions, and estimates of life-cycle costs for team

    approval. The team uses this information to make the critical product

    development decision to commit further to product development or kill a

    product idea. Maytag officials stated that their company has a quality

    image to uphold and that consumers expect the highest reliability and

    quality from their products.

    Maytag officials were also conservative in their use of new

    technologies during product development. Decisions to incorporate new

    technology were made in the earliest phases of product development and

    were based on assessments of the adaptability and maturity of the

    technology and associated risks to achieving established reliability

    targets. Even though its Neptune washer incorporated over 90 percent

    new technologies in its development, Maytag officials stated that they

    spent time with suppliers and developers, maturing technologies to an

    acceptable level of reliability before launching the new development.

    Leading Commercial Firms Use Feedback from Operations to Better

    Understand Customer Needs, Product Deficiencies, and Operating and

    Support Costs:

    The collection and analysis of the operating and support costs for

    delivered products was considered essential by leading commercial

    companies. Once a product is fielded, leading companies track actual

    operating cost, reliability of parts, and readiness of the product

    against what was estimated during product development to make sure the

    company is getting what it paid for. We found companies are always

    identifying the top drivers of operating and support cost and working

    with the manufacturers to reduce these costs. When there are part

    failures, the companies can quickly identify whether or not they are

    under warranty and get the part replaced.

    All of the companies we visited, customers and product developers

    alike, had very collaborative processes and practices for drawing

    extensively on data from current and past operations to improve the

    reliability of existing products or influence the design of new

    products. United Airlines officials told us that the airlines and

    Boeing both keep meticulous reliability and cost records at all levels

    of the 777: components, subsystems, and the system level. Major

    operating costs drivers are tracked on a daily basis by the airlines.

    The manufacturers usually have personnel residing with the airlines�

    maintenance crews to help solve problems on the spot and, just as

    importantly, feed information back to the manufacturer so that the next

    product can be improved. FedEx Express and Polar Tanker both emphasized

    extensive collection of data from current operations. In fact, FedEx

    Express sets annual targets for operating and support cost reductions

    based on reliability data gathered on the road. Polar Tanker gathered

    maintenance data from past operations to determine areas that could

    result in higher reliability and lower maintenance costs in designing

    the new tanker.

    United conducts quarterly meetings for each of its fleets to discuss

    open issues and short-term and long-term solutions to current problems

    with operational aircraft. Attendees at the meeting include maintenance

    mangers, financial representatives, representatives of the

    manufacturers, and executives with authority to resolve issues.

    United�s practice is to resolve problems as expeditiously as possible,

    no matter how small. United also monitors flight movements on a real

    time basis through a computer system that tracks each aircraft by tail

    number. The monitoring system provides the maintenance history of the

    aircraft, reports problems on a current flight that require maintenance

    upon landing, and alerts for other required maintenance based upon the

    number of flying hours on the aircraft. If a specific part is broken,

    the system also indicates whether or not it is under warranty. United

    archives information on parts that break, when they break, and whether

    they are still under warranties.

    Manufacturers like Boeing and other major suppliers like this type of

    feedback because it provides useful information to them on how to

    improve the product for future iterations. They also believe that quick

    responses to customer problems will help them get repeat business. They

    use this feedback to develop preventive maintenance schedules, better

    estimate operating and support costs, and refine reliability

    requirements to be used in preparing budgets and cost estimates for

    future products. Feedback mechanisms also accumulated operations and

    maintenance data and �lessons learned� that highlighted reliability

    problems and other maintenance issues. United officials stated that

    taking lessons learned from data gathered from current products is an

    effective tool for improving product reliability and maintainability or

    developing requirements for new products.

    FedEx Express takes any failure to meet its on-time delivery goal very

    seriously and holds daily failure analysis meetings every morning to

    analyze and review each delivery failure from the night before. This

    constant feedback allows them to take immediate corrective action on

    individual vehicles and to identify trends that may result in larger

    problems and costlier maintenance requirements. FedEx Express

    maintained a metric for vehicle miles between road calls and

    maintenance costs per asset. In addition to tracking these costs per

    asset, FedEx Express has established a performance goal with its

    managers to reduce the overall costs of maintenance each year. FedEx

    Express managers set their annual targets for reductions in operating

    and support cost based on reliability data gathered on the road.

    [End of section]

    Chapter 4: Stressing Operating and Support Cost at the Outset of an

    Acquisition Could Help DOD Reduce Total Ownership Costs:

    DOD and the commercial companies we visited have policy goals of

    developing products that will meet customers� needs at the lowest

    possible cost to build and operate. The difference between them is in

    how each implements its policies. Leading commercial companies follow

    an integrated, collaborative process of setting requirements,

    developing the product, and ensuring that the product can be supported

    at an acceptable cost. DOD�s process is composed of disparate practices

    carried out by separate organizations with differing objectives and

    little communication between them about how to support fielded systems.

    While commercial firms focus on total ownership costs at the outset,

    DOD focuses mostly on technical performance. One cause of this is that

    in DOD the accountability and responsibility for total ownership costs

    are spread across many organizations with separate goals. Another cause

    lies in motivation for low costs. The commercial companies we visited

    are driven by the need to be as profitable as possible to survive, and

    low total ownership costs translate to higher profitability. DOD�s

    environment does not provide such incentives. The organizations charged

    with acquiring and operating weapon systems are unconstrained by a need

    to lower costs since they can request additional operations and

    maintenance funding to keep systems working.

    From time to time, DOD stated the need to lower its total ownership

    costs in policy documents and in annual budget statements; however, it

    has not been successful because it does not have an environment that

    demands collaboration and accountability in setting requirements and

    developing products with operating costs in mind. DOD has some efforts

    underway to improve. First, it has rewritten its acquisition and

    requirements generation policies. Second, a few programs now in

    development established an early estimate of operating and support

    costs and are working to gain knowledge of the impacts of requirements

    and design on those costs. Third, information from pilots on fielded

    systems, if disseminated throughout the acquisition community, could be

    used to lower costs. Results are pending.

    What DOD�s efforts do not do is provide incentives to make investments

    for more reliable, less-costly-to-maintain systems at the beginning of

    an acquisition. Instead, DOD provides incentives to field systems with

    unknown reliability by allowing whatever funding necessary to operate

    and maintain the systems once they are fielded.

    Differences in Practices Explain Different Outcomes for Commercial

    Companies and DOD in Controlling Total Ownership Costs:

    In Chapter 3, we discussed our findings that leading commercial

    companies set specific requirements for readiness and operating and

    support costs prior to initiating product development that forced

    developers to design products with a high degree of reliability. In

    Chapter 2, we noted that DOD, on the other hand, typically focused its

    requirements on revolutionary performance that often forced developers

    to mature technologies at the same time they were completing detailed

    design work. As a consequence, system reliability often suffered,

    forcing the department to spend a great deal of money to maintain and

    repair fielded systems in an effort to achieve desired readiness

    levels. The following table provides a comparison of the specific

    practices used by commercial companies we visited and DOD programs we

    reviewed to address operating and support costs early in a new

    product�s life cycle.

    Table 2: DOD and Commercial Practices for Controlling Operating and

    Support Costs:

    Commercial prevailing practice: Practices used to set initial product

    requirements.

    Commercial prevailing practice: Operating and support cost goals as a

    key requirement.; DOD prevailing practice: Practices used to set

    initial product requirements: Operating and support cost goals are not

    established as key parameters..

    Commercial prevailing practice: Readiness a key requirement.; DOD

    prevailing practice: Practices used to set initial product

    requirements: Readiness is not a key parameter..

    Commercial prevailing practice: Trade performance for reduced operating

    and support costs, if appropriate; sometimes results in increased

    costs.; DOD prevailing practice: Practices used to set initial product

    requirements: Technical performance is sometimes traded using cost as

    an independent variable, but cost is usually production cost or

    development cost, and the trades occur during the design phase..

    Commercial prevailing practice: Direct relationship during

    requirements-setting between the user and the product developer.; DOD

    prevailing practice: Practices used to set initial product

    requirements: User and product developer separated by user

    representative and government program office..

    Commercial prevailing practice: Practices used during product

    development.

    Commercial prevailing practice: Provide detailed operating and support

    cost estimates early in product development.; DOD prevailing practice:

    Practices used to set initial product requirements: Operating and

    support cost estimates not required until product development launch..

    Commercial prevailing practice: User and developer focus on ways to

    reduce product parts and standardize parts across product lines.; DOD

    prevailing practice: Practices used to set initial product

    requirements: Product developer has responsibility of focusing on ways

    to reduce parts counts or use standardized parts with little input from

    the user (operators or maintainers)..

    Commercial prevailing practice: Use open systems architecture approach

    to improve the cost effectiveness and installation efficiency of future

    upgrades to the product.; DOD prevailing practice: Practices used to

    set initial product requirements: Open systems approach is mandated but

    implementation is limited..

    Commercial prevailing practice: Set realistic reliability growth goals

    for the product.; DOD prevailing practice: Practices used to set

    initial product requirements: Reliability goals set, but they are

    tradable or not met..

    Commercial prevailing practice: Conduct reliability testing early.; DOD

    prevailing practice: Practices used to set initial product

    requirements: Reliability testing sporadically performed..

    Commercial prevailing practice: Practices used during operations.

    Commercial prevailing practice: Collect and analyze operations and

    support data.; DOD prevailing practice: Practices used to set initial

    product requirements: Data is often incomplete or unreliable..

    Commercial prevailing practice: Manage operations and support costs to

    targets.; DOD prevailing practice: Practices used to set initial

    product requirements: Do not manage to operations and support targets..

    Commercial prevailing practice: Identify areas for continuous

    improvement.; DOD prevailing practice: Practices used to set initial

    product requirements: Lack of complete and reliable data makes

    identifying areas for improvement difficult; some areas that are

    identified are not funded for improvement..

    Commercial prevailing practice: Feedback to developer on product

    performance.; DOD prevailing practice: Practices used to set initial

    product requirements: Limited feedback to the developer. The maintainer

    does not have a direct relationship with the product developer..

    [End of table]

    Source: GAO.

    Clearly, the practices used by the commercial companies we visited

    before product development when requirements are set, early in product

    development when the design is finalized, and during the new product�s

    operating life focus as much on providing a reliable product as on

    providing a high-performance product. The companies make operating cost

    and readiness key requirements, they perform extensive reliability

    testing, and they aim toward continuous improvement once the product is

    in the field. In DOD, because performance is the overriding concern of

    the requirement setters, none of these practices are in place.

    Several DOD Efforts Underway to Reduce Total Ownership Costs:

    The changes to policies and the investment in improving systems�

    reliability are encouraging indicators that DOD has focused its

    attention on reducing costs to support weapons. DOD has revised

    acquisition policies, tested new approaches for reducing costs in a few

    systems, explored differing approaches, and created initiatives to

    reduce costs of legacy systems. Each of these efforts had some initial

    success, but most are aimed at reducing costs after fielding when over

    90 percent of the costs have been determined.

    DOD�s Requirements Generation and Acquisition Policies Could Be More

    Specific in Addressing Supportability Issues:

    DOD has revised its 5000 series acquisition policies several times over

    the past 10 years with the intent of defining an acquisition

    environment that makes DOD a smart and responsive buyer. During this

    time, the policy has not substantively changed with regard to how

    acquisition programs can best control total ownership costs. The

    department is striving for an integrated acquisition and logistics

    process that is characterized by, among other things, a stronger focus

    on using supportability as a key design and performance factor.

    However, rules for total ownership cost goals at the outset of an

    acquisition program are defined by the Chairman of the Joint Chiefs of

    Staff�s Instruction 3170.01B[Footnote 8] on requirements generation.

    This guidance states that cost should be addressed in the operational

    requirements document for a new weapon system, if an estimate is

    available at that time. However, policy does not require the services

    to set requirements for operating and support costs or readiness.

    Instead, it allows them to identify system capabilities or

    characteristics they consider essential for successfully completing the

    mission. It states that the DOD sponsor may make cost a key requirement

    if it desires and identify the cost it wishes to evaluate.

    We previously reported that DOD officials believe they must promise

    new, revolutionary weapon systems with significantly better performance

    capabilities than the ones they are replacing in order to obtain

    funding.[Footnote 9] Therefore, key parameters are usually focused on

    performance rather than supportability.

    In order to effectively minimize total ownership costs of its systems,

    the Department of the Navy recently issued its own guidance that

    establishes specific supportability and affordability thresholds and

    objectives[Footnote 10] for all requirements documents. The Navy

    believes that by establishing readiness and operating and support cost

    as required parameters, there is assurance that major drivers of total

    ownership costs will be addressed and minimized throughout the

    acquisition process. Specifically, the new guidance states that

    requirements documents must include goals for operating and support

    costs. It also states that operational availability be included as a

    key performance parameter, except when logistics delays are not an

    issue or if the requirements are for a major aircraft or ship platform.

    In those cases, mission capable rates or full mission capable rates

    focused on the platform�s primary mission areas will be used as key

    requirements. The Army is discussing a similar change in its guidance.

    Three New Acquisition Programs Are Placing Greater Emphasis on

    Readiness and Operating Cost Goals:

    We found three DOD programs still in development--the Joint Strike

    Fighter, the Advanced Amphibious Assault Vehicle, and the Landing

    Platform Dock 17--that appear to be using good practices to reduce

    operating and support costs during product development. Each, in its

    own way, has had a powerful internal incentive to establish more

    collaborative practices or to focus attention on operating and support

    costs and product reliability.

    Figure 13: Joint Strike Fighter:

    [See PDF for image]

    [End of figure]

    The Joint Strike Fighter program is intended to produce an affordable

    next-generation aircraft to replace DOD�s aging aircraft inventory. The

    program is structured to use a common production line to produce three

    aircraft variants that meet conventional flight requirements for the

    Air Force, short take-off and vertical landing characteristics for the

    Marine Corps, and carrier operation suitability needs for the Navy. The

    program will also provide aircraft to the British Royal Navy and Air

    Force. A key objective of the acquisition strategy is affordability--

    reducing the development, production, and operating costs of the

    program relative to prior fighter aircraft it will replace. The

    program�s latest stated estimate for operating and support cost savings

    compared to legacy systems is $135 billion, or a 56 percent reduction

    in cost.

    To achieve this affordability objective, the program office has

    incorporated various DOD and commercial initiatives into the

    acquisition strategy. For example, two key provisions in its

    operational requirements document--mission reliability and logistics

    footprint--will have a direct impact on operating and support costs.

    Specifically, all variants of the fighter are expected to achieve a

    mission reliability rate of over 90 percent and meet numeric goals of

    cargo aircraft or ship space needed to support a 30-day self-sustained

    deployment. These two requirements, along with other reliability and

    maintainability goals, demonstrate DOD�s desire to reduce total

    ownership costs. The product developer currently estimates the Joint

    Strike Fighter will be able to reduce operating and support cost

    primarily through efforts to improve:

    * reliability and durability of materials,

    * accessibility of parts or systems that need to be inspected or

    replaced,

    * supportability of low observable materials,

    * ability of on-board systems to predict impending flight critical

    failures, and:

    * training materials and systems.

    For example, DOD expects to save about $39 billion over the life of the

    fighter through reduced maintenance on low observable materials. The

    developer estimates that 99 percent of the maintenance actions will

    require no low observable restoration because they are using high

    durability materials, parts, or systems that are easier to access and

    harder to damage. In order to reach this level of savings, the

    developer spent a great deal of time evaluating previous DOD

    maintenance experience with the B-2A bomber and the F-117 fighter

    aircraft and used an evolutionary approach for upgrading these

    materials. Operating and support costs for the B-2A bomber, for

    example, were significantly increased by the decision to use an

    immature technology for low observability.

    However, the Joint Strike Fighter program must be careful not to

    overestimate the total ownership cost savings it can achieve over

    legacy systems it will be replacing because the program is also

    depending on new technology for on-board systems to predict failure--

    prognostics and health management technology--that is not yet ready for

    product development. In October 2001, GAO reported that this technology

    was not at an acceptable readiness level for inclusion in product

    development, but DOD and the contractor decided to include it in order

    to meet total ownership cost objectives.[Footnote 11] Program officials

    stated that about $16 billion--12 percent of the estimated $135 billion

    in total ownership cost savings--is expected to come from that

    technology. Since then, the officials have allowed for the possibility

    that the technology may not be included on initial production lots for

    the Joint Strike Fighter if it is not ready.

    Figure 14: Advanced Amphibious Assault Vehicle:

    [See PDF for image]

    [End of figure]

    The Advanced Amphibious Assault Vehicle is a Marine Corps program to

    improve its amphibious landing vehicle. The new development promises

    faster sea and land speeds, better protection, and more lethality. The

    development program has focused on maturing technology and paying

    attention to operating and support costs early in development. The

    program has used some of the best practices of commercial companies

    during development. Some of those include collocating the program

    office at the contractor�s facility and making extensive use of Marine

    Corps war fighters and maintainers to provide a �hands on� assessment

    of how effective the vehicle would be in operations as well as how

    supportable it would be during operations. The Marines developed an

    early estimate of total ownership costs and included a reliability

    metric as one of its key requirements. The Marines developed three

    vehicle prototypes to mature the design and have conducted extensive

    reliability testing. The vehicle will have parts that are common with

    other weapons such as a gun that will be common with the Landing

    Platform Dock 17. Advanced Amphibious Assault Vehicle officials

    estimated that they will save $29 million in operating and support

    costs.

    The Navy program office for the Landing Platform Dock 17 has adopted a

    total ownership cost approach. The program office established a process

    for suggesting and evaluating design trades that could reduce operating

    and support costs. Some of the design changes that the Navy made

    include enclosing the mast to reduce exposure to weather and salt

    water, and investing in high performance covering for the deck and well

    deck to mitigate corrosion. Other practices include involving users to

    complete tasks using the virtual software to test special design

    elements in the ship, making greater use of sensors and automated

    processes to reduce maintenance and to reduce crew requirements.

    Both the Landing Platform Dock -17�s and the Advanced Amphibious

    Assault Vehicle�s estimate for operating and support costs have

    recently increased. The Navy raised its estimate of Landing Platform

    Dock 17�s requirements for spare parts, fuel, and software maintenance.

    The Advanced Amphibious Assault Vehicle program office attributed its

    increase in the cost estimate to funding additional prototypes to

    improve reliability. Information like this allows options for decision

    makers, while the system is still in development, to accept the costs

    or re-examine the performance characteristics to see if they can be

    relaxed in order to improve reliability and, thereby, reduce operating

    and support costs.

    DOD Pilot Programs Attempt to Reduce Total Ownership Costs:

    In 1999, the Defense Systems Affordability Council implemented a

    program to explore ways to reduce the total ownership cost of its

    weapon systems. The Council--chaired by the Undersecretary of Defense,

    Acquisition, Technology, and Logistics--set a goal of reducing

    logistics costs for selected fielded weapon systems by 20 percent by

    fiscal year 2005. The Council also set a goal for selected systems

    still in development to achieve total ownership costs targets that are

    20 percent to 50 percent below historical norms.

    DOD selected 30 programs (10 from each military department) to test

    various approaches for reducing total ownership costs, such as using

    commercial items or technology to reduce costs of legacy systems and

    using industry standards when developing systems to make upgrades

    easier and less costly to complete.

    The Air Force�s C-17 program is one of the pilot programs. It is a

    fielded system whose operating and support costs increased by about 25

    percent between 1995 and 1999. One C-17 initiative is an engine upgrade

    to extend the time between removals, reduce unexpected shop visits and

    spares purchases, and reduce the number of engine overhauls. The Air

    Force believes the C-17 could avoid $724.5 million in support costs if

    the upgrade is completed.

    The Apache recapitalization program, another of DOD�s pilots,

    integrates a number of selected upgrades that taken together are

    expected to achieve a 30 percent reduction in operating and support

    costs by 2010. Most of the Apache helicopters will be refurbished and

    modified to the Apache Longbow configuration. The target acquisition

    and designation system, the top cost driver, is a focus of these

    improvements along with improvements in the drive train, the rotor, and

    the propulsion system. The current average cost per flying hour for the

    Apache fleet is $3,348. The Army�s projected cost per flight hour after

    the modifications is $2,230.

    The Abrams tank is also undergoing a major upgrade estimated to cost

    about $5 billion. The top cost driver on the tank is the power pack,

    which includes the engine and transmission, followed by the auxiliary

    automotive system, hull and frame, fire control system, armament, and

    track. Army officials believe upgrading and replacing the engine is the

    most effective way to reduce operating and support costs for the tank.

    The current cost per mile for the Abrams fleet is $181 per mile,

    including repair parts and fuel but excluding most personnel cost. The

    recapitalization program�s goal is to reduce the cost to $107 per mile

    and to improve reliability.

    Other Initiatives Show Promise, but Implementation Is Slow:

    We found three other initiatives--the change in acquisition policy

    toward evolutionary acquisition, an open systems approach for weapon

    systems and the Commercial Operations and Support Savings Initiative--

    that could help DOD reduce total ownership cost. However,

    implementation has been limited in the latter two initiatives because

    consistent high-level support is lacking.

    DOD defines evolutionary acquisition as an approach for delivering

    capability in increments, recognizing the need for future capability

    improvements. DOD allows two processes to achieve an evolutionary

    acquisition, both of which include requirements for collaboration

    between the user, the tester, and the developer. The first process is

    referred to as incremental development. In an incremental development

    process, a desired capability is identified, an end-state requirement

    is established, and the requirement is met over time by the development

    of several increments of the product, each dependent on available,

    mature technology. The second process is referred to as spiral

    development. In a spiral development process, the end-state requirement

    is not known, and each increment of the product is based on feedback

    from the user. Each increment yields the best possible capability for

    the user. The movement toward evolutionary acquisition and time-phased

    requirements bodes well for the potential to understand reliability,

    readiness, and implications for total ownership cost early, because an

    evolutionary process allows an acquisition program to design a weapon

    system to requirements based only on demonstrated technologies. This is

    very similar to commercial practices.

    DOD chartered an open-systems joint task force to implement an open

    systems approach in weapon systems acquisitions. Open systems can

    reduce cost through use of widely accepted standard products from

    multiple suppliers, allowing DOD to benefit from the commercial market

    place and take advantage of the competitive pressures that motivate

    commercial companies to improve products and reduce prices. DOD

    expected open systems to reduce the cost of ownership of weapon

    systems, delay system obsolescence, and allow fielding superior war-

    fighting capability more quickly. The DOD Inspector General recently

    reported that the DOD acquisition community has not fully applied the

    use of open systems objectives in the acquisition planning and review

    process.[Footnote 12] The report recommended DOD enforce the use of an

    open systems approach as part of the acquisition milestone review

    process.

    Another initiative that showed promise but lacks high level support is

    the Commercial Operations and Support Savings Initiative introduced to

    improve weapon system readiness and reduce operating and support costs

    by inserting existing commercial items or technology into military

    legacy systems. It emphasizes the rapid development of prototypes and

    fielding of production items based on current commercial technology.

    According to a 2001 report by an independent assessment team, the

    initiative�s three objectives of reducing operations and support costs

    for legacy systems, simplifying prototype development, and attracting

    commercial firms to the defense marketplace are being met. But, the

    initiative lacks high-level support. The Under Secretary of Defense

    (Acquisition, Technology, and Logistics) recently directed that funding

    for the program be terminated.

    DOD�s Current Environment Does Not Provide Incentives to Reduce Total

    Ownership Cost Early:

    While initiatives for acquisition programs and potential reductions for

    the fielded systems are welcomed, the department has not

    institutionalized the practices used in the initiatives by demanding

    them on all acquisition programs. As we discussed in Chapter 1, 90

    percent of the operating and support costs are determined before

    fielding, and these initiatives do not attack the causes of higher

    operating and support costs. Those are: the division of responsibility

    among the requirements community, the acquisition community, and the

    maintenance community for controlling costs; the lack of focused

    attention on reliability early in development; and the lack of

    accountability for total ownership cost when setting requirements that

    is caused by the division of responsibilities across these communities.

    Companies we visited have incentives to make operating cost and product

    readiness equal to technical performance when setting requirements for

    new products because these factors largely determine their

    profitability and, therefore, survival in the market place. Lower

    operating costs translate to higher profits and increased sales.

    Customers cannot afford to have large amounts of capital tied up in

    extra equipment, spare parts, or personnel to ensure their equipment is

    ready to perform when needed. They cannot afford to have equipment fail

    during operations, because failure precludes accomplishment of the

    company�s mission and loss of revenue. These companies are constrained

    by a finite amount of funding to acquire and operate their equipment,

    and, therefore, they hold the people setting the product requirements

    accountable for total ownership cost. Many of the companies we visited

    use one integrated product team to identify needs, set requirements,

    and monitor product development. Most importantly, the organization

    that will be responsible for supporting the equipment in the field sets

    requirements for new products. There is also a direct relationship

    between the requirements-setting team and the product developer while

    the product�s requirements are being set, during development, and after

    products are put into service. Information flows throughout the

    integrated process, with each new phase in the process being informed

    by knowledge from the phase just ending.

    DOD�s current acquisition environment does not provide the same

    incentives or practices. Traditionally, DOD does not constrain its

    requirement setters in the same way. Requirement setters in DOD have

    demanded weapon systems that, due to their performance features,

    consistently cost more to operate and support than anticipated to

    achieve necessary readiness levels. This has been accepted because a

    large logistics organization--separate from the requirement setting

    organization--is charged with supporting these weapon systems and uses

    monies from a different budget to do so. In essence, DOD�s environment

    frees the requirements community to insist on technical requirements

    that cannot be made into reliable products, are costly to support, and

    cannot be maintained cost effectively. Accountability for operating and

    support costs does not rest with the requirement setters, or, for that

    matter, with the acquisition community. Eventually, maintenance

    organizations have no choice but to request sufficient funding to keep

    weapon systems operating once they are fielded. DOD has identified this

    division of responsibility as a key cause of higher weapon system

    operating and support costs. In this current environment, there is no

    incentive for collaboration and accountability in setting requirements

    and developing products with operating costs in mind. Instead, it

    provides incentives to field systems with unknown reliability by

    allowing whatever funding necessary to operate and maintain the systems

    once they are fielded.

    [End of section]

    Chapter 5: Conclusions and Recommendations:

    Acceptable readiness levels are a function of having platforms

    available when required. Such levels can be achieved by having highly

    reliable platforms, by spending whatever is necessary on ongoing

    maintenance, or by having excess capacity. The high cost of maintaining

    weapon systems to meet required readiness levels is depleting DOD�s

    modernization accounts and denying DOD the flexibility to invest in new

    weapons. DOD must find ways to reduce total ownership cost while

    maintaining needed readiness rates. Readiness is a critical component

    of all DOD weapons systems. If a system is not ready, its performance

    capabilities are of no use. The decision on whether readiness will be

    achieved by spending additional funds on operations or by designing

    high reliability into the weapon system must be made while requirements

    are being set and early in product development.

    DOD�s prevailing practices run counter to achieving high reliability.

    Often, DOD does not make readiness or operating cost performance

    parameters equal in importance to others when it establishes

    requirements for weapons systems. Further, reliability growth during

    product development is hampered by immature technologies and delays in

    gaining knowledge about the product�s design. Finally, DOD does not

    have sufficient knowledge about its fielded systems to inform its

    product development process for new systems. DOD is at a crossroad in

    this regard. It has made positive changes to acquisition policy in

    order to change its environment. Requiring higher readiness at lower

    cost will enable DOD to take the next step, ensuring lower total

    ownership cost.

    In contrast, commercial companies that are in the market for new

    capital equipment understand that they must specify and control the

    readiness and total ownership cost of a product, especially the

    operating and support costs early in development. Therefore, they

    specify how available or ready the products must be in order to carry

    out the company�s mission. Further, they set goals for operating costs

    when acquiring new equipment; they make sure they understand their own

    operating costs from data they have collected and analyzed on equipment

    they are now using. Those two goals--how available the product must be

    and how much the customer wants to spend per operating unit to support

    the equipment--are key requirements equal in importance to other

    performance characteristics that the commercial customer demands from

    the companies that develop the products. Bounded by the twin

    requirements of specific operating costs and availability, the product

    developer sets reliability goals for the components, subsystems, and

    the full system once it is integrated into a product that will satisfy

    the customer�s requirements. Product developers remain focused on good

    product development practices with mature technologies, stable designs,

    and production processes that are in control. During operations they

    collect data from their customers on reliability and performance and

    use that data to predict operating and support costs for subsequent

    developments or upgrades.

    DOD has initiatives underway that partially address the issue of

    controlling operating and support costs. However, without significant

    emphasis on providing a better framework for decision-making, these

    initiatives will not yield sufficient improvements. The department has

    encouraged the services to include key performance parameters in its

    newer developments such as the Joint Strike Fighter that indicate how

    long a system must perform between maintenance actions. It has moved to

    follow best practices for reducing risk from technology and achieving

    more stable designs in the Advanced Amphibious Assault Vehicle and the

    Joint Strike Fighter. However, these programs are early in development

    and it will take some time to see how reliably they perform. We believe

    that practices found at the commercial companies we visited to make

    operating and support costs and product readiness requirements equal in

    priority to other performance characteristics forces developers to

    focus on achieving high reliability and that adopting these practices

    will help DOD achieve high readiness and control total ownership cost.

    Recommendations for Executive Action:

    DOD should take steps to make the cost to operate and support weapon

    systems at required readiness rates a priority when setting weapon

    system requirements for an affordable weapon system and finalizing the

    design of the selected system. To do this, its requirements and

    acquisition communities must collaborate to fully understand and

    control the costs to operate and support a weapon system prior to and

    early in product development, when it is possible to have significant

    impact on those costs. In establishing requirements for a weapon

    system, the requirements community should include the costs to operate

    and support the weapon system over its life cycle and the readiness

    rate for the weapon system. To establish an affordable design for the

    weapon system, the acquisition community and acquisition programs

    should be required to accurately estimate--based on demonstrated

    component and subsystem reliability testing--that portion of the costs

    that DOD plans to spend on operations and support of the weapon system

    throughout its life cycle before the design is finalized.

    With this in mind, to ensure that the user�s requirements for a weapon

    system can be met with a reliable design, we recommend that the

    Secretary of Defense:

    * revise the Chairman of the Joint Chiefs of Staff Instruction 3170.01B

    on the requirements generation process to include total ownership cost,

    especially operating and support cost, and weapon system readiness

    rates as performance parameters equal in priority to any other

    performance parameters for any major weapon system before beginning the

    acquisition program;

    * revise the current policy governing the operation of the defense

    acquisition system (currently under revision) to require that the

    product developer establish a firm estimate of a weapon system�s

    reliability based on demonstrated reliability rates at the component

    and subsystem level no later than the end of the system integration

    phase, coinciding with the system-level critical design review, before

    proceeding into the system demonstration phase of product development;

    and at the system level no later than the full-rate production

    decision; and:

    * structure its contracts for major systems acquisitions so that at

    Milestone B the product developer has incentives to ensure that proper

    trades are made between reliability and performance prior to the

    production decision. One option is to provide specific clauses in the

    development contract to address reliability growth.

    Agency Comments and Our Response:

    DOD partially concurred with all of our recommendations; however, for

    the most part, it found no further action was needed to lower total

    ownership cost. We disagree. We believe that if DOD takes no further

    action in implementing these recommendations, it ignores significant

    opportunities to improve readiness and lower the total ownership cost

    of its major weapon systems. The current budget environment demands

    more effort in reducing these costs.

    The performance of weapon systems as described in this report is

    evidence that they demand much more money than planned to remain ready.

    DOD should consider each of these recommendations as parts of a whole

    solution for its �death spiral�--that is, the inability to modernize

    its forces because the cost to operate and maintain unreliable weapon

    systems at needed readiness rates constantly impinges on its

    modernization budget. Taken as a whole, our recommendations encourage

    DOD�s requirement setters to demand readiness at an affordable cost as

    a part of a system�s performance, provide a mechanism to hold the

    product developer accountable for determining the reliability needed to

    satisfy DOD�s requirements, and provide contractual incentives for the

    product developer to build reliability into a weapon system very early

    in its development. The details of DOD�s response to each

    recommendation are summarized below along with our rebuttal.

    In a response prepared by the Office of the Joint Chiefs of Staff, DOD

    partially concurred with our recommendation to include readiness and

    total ownership cost as performance parameters equal in priority to any

    others before beginning an acquisition program, commenting that they

    are currently equal in priority to �non-key performance parameters.� We

    are concerned that DOD does not recognize the importance of requiring

    targets for a system�s readiness and its total ownership cost before

    beginning product development. These targets are critical to providing

    a realistic goal for the product developer to deliver reliable, cost-

    efficient weapon systems.

    We examined five deployed weapon systems for this report. None had key

    requirements for readiness rates or operating costs. All had

    significant problems with reliability and, therefore, readiness and

    total ownership cost. We also reviewed several commercial products that

    were developed with readiness and total ownership cost as critical

    requirements. In each case, the products were ready to perform when

    needed at affordable and predictable cost. Unless these requirements

    are equal in importance to any others, they will not withstand the

    pressures of an acquisition program. The Joint Chiefs stated that its

    requirements generation policy is currently under revision and that no

    decision has been made about the priority readiness and cost should

    have as requirements. We believe DOD has an excellent opportunity to

    finally lower the total ownership cost of current and future weapon

    acquisitions, thereby freeing significant funds for modernization if it

    implements this recommendation.

    In a response prepared by the Office of the Undersecretary of Defense

    for Acquisition, Technology, and Logistics, DOD partially concurred

    with our second recommendation to establish estimates of a weapon

    system�s reliability--first, based on demonstrated reliability rates at

    the component and subsystem level by the end of system integration,

    coincident with the critical design review and next, at the system

    level at the time the production decision is made--however, DOD found

    no need to revise the policy governing the defense acquisition system

    to achieve this. We disagree. Demonstrating reliability during product

    development has not received the priority it requires if DOD is to have

    a realistic opportunity to reduce the total ownership cost of its

    weapons while maintaining required readiness levels. The current policy

    does not provide a mechanism to ensure consistent application of

    reliability estimates based on demonstrated performance. We believe

    that the disparity between the actual costs to operate and maintain

    weapon systems and what DOD had estimated those costs to be during the

    weapon systems� development, as described in the report, provides

    strong evidence that product developers do not understand reliability

    under DOD�s current process. Implementation of this recommendation will

    assist DOD in requiring reliability estimates on a consistent basis.

    In a response prepared by Undersecretary of Defense, Acquisition,

    Technology, and Logistics to our recommendation that DOD structure its

    development contracts to include requirements to provide incentives for

    product developers to trade performance for reliability when it makes

    sense, DOD found no need for additional incentives to contractors

    beyond giving them total system performance responsibility. We have yet

    to see evidence that total system performance responsibility has

    provided an incentive for any product developer to trade performance

    for reliability in order to reduce total ownership cost. Further, we

    believe a contractual agreement similar to those we found in commercial

    cases--such as financial penalties for readiness below certain

    specified rates--would provide an excellent incentive for product

    developers to gain the knowledge required to meet reliability rates

    early in a weapon system�s design before committing to production.

    DOD provided some technical comments in attachment 2. We have addressed

    those in the report as necessary. The full text of the department�s

    response to the recommendations is provided in appendix I.

    [End of section]

    Related GAO Products:

    Best Practices: Capturing Design and Manufacturing Knowledge

    Early Improves Acquisition Outcomes. GAO-02-701. Washington, D.C.:

    July 15, 2002.

    Defense Acquisitions: DOD Faces Challenges in Implementing Best

    Practices. GAO-02-469T. Washington, D.C.: February 27, 2002.

    Best Practices: Better Matching of Needs and Resources Will Lead

    to Better Weapon System Outcomes. GAO-01-288. Washington, D.C.:

    March 8, 2001.

    Defense Acquisitions: Higher Priority Needed for Army Operating and

    Support Cost Reduction Efforts. GAO/NSIAD-00-197. Washington, D. C.:

    September 29, 2000.

    Defense Acquisitions: Air Force Operating and Support Costs Reductions

    Need Higher Priority. GAO/NSIAD-00-165. Washington, D. C.: August 29,

    2000.

    Best Practices: A More Constructive Test Approach Is Key to

    Better Weapon System Outcomes. GAO/NSIAD-00-199. Washington, D.C.:

    July 31, 2000.

    Defense Logistics: Actions Needed to Enhance Success of Reengineering

    Initiatives. GAO/NSIAD-00-89. Washington, D. C.: June 23, 2000.

    Defense Acquisition: Employing Best Practices Can Shape Better Weapon

    System Decisions. GAO/T-NSIAD-00-137. Washington, D.C.: April 26,

    2000.

    Best Practices: DOD Training Can Do More to Help Weapon System Programs

    Implement Best Practices. GAO/NSIAD-99-206. Washington, D.C.:

    August16, 1999.

    Best Practices: Better Management of Technology Development Can Improve

    Weapon System Outcomes. GAO/NSIAD-99-162. Washington, D.C.: July 30,

    1999.

    Defense Acquisitions: Best Commercial Practices Can Improve Program

    Outcomes. GAO/T-NSIAD-99-116. Washington, D.C.: March 17, 1999.

    Defense Acquisition: Improved Program Outcomes Are Possible. GAO/

    T-NSIAD-98-123. Washington, D.C.: March 18, 1998.

    Best Practices: DOD Can Help Suppliers Contribute More to Weapon System

    Programs. GAO/NSIAD-98-87. Washington, D.C.: March 17, 1998.

    Best Practices: Successful Application to Weapon Acquisition Requires

    Changes in DOD�s Environment. GAO/NSIAD-98-56. Washington, D.C.:

    February 24, 1998.

    Major Acquisitions: Significant Changes Underway in DOD�s Earned Value

    Management Process. GAO/NSIAD-97-108. Washington, D.C.: May 5, 1997.

    Best Practices: Commercial Quality Assurance Practices Offer

    Improvements for DOD. GAO/NSIAD-96-162. Washington, D.C.:

    August 26, 1996.

    [End of section]

    Appendix I: Comments from the Department of Defense:

    ACQUISITION, TECHNOLOGY AND LOGISTICS:

    OFFICE OF THE UNDER SECRETARY OF DEFENSE:

    3000 DEFENSE PENTAGON WASHINGTON, DC 20301-3000:

    DEC 2002:

    Ms. Katherine V. Schinasi:

    Director, Acqusition and Sourcing Management U.S. General Accounting

    Office:

    441 G Street, N. W. Washington, D.C. 20548:

    Dear Ms. Schinasi:

    This is the Department of Defense (DoD) response to the GAO draft

    report, �BEST PRACTICES: Setting Requirements for Cost and Readiness

    Could Reduce Weapon Systems� Total Ownership Costs,� dated November 26,

    2002 (GAO Code 120092/GAO-03-057). There are two enclosures. The first

    contains the DoD detailed comments to the recommendations. The second

    are additional comments on the report. The DoD response to

    Recommendation 1 was provided by the Joint Chiefs of Staff.

    Sincerely,

    Spiros G. Pallas, Principal Deputy, Strategic and Tactical Systems:

    Signed by Spiros G. Pallas:

    Enclosure(s): As stated:

    ENCLOSURE (1):

    GAO DRAFT REPORT - DATED NOVEMBER 26, 2002 GAO CODE 120092/GAO-03-057:

    �BEST PRACTICES: Setting Requirements for Cost and Readiness Could

    Reduce Weapon Systems� Total Ownership Costs�:

    DEPARTMENT OF DEFENSE COMMENTS TO THE RECOMMENDATIONS:

    RECOMMENDATION 1: The GAO recommended that the Secretary of Defense

    revise the Chairman of the Joint Chiefs of Staff Instruction 3170.01 on

    the requirements generation process to include total ownership cost,

    especially operating and support cost, and weapon system readiness

    rates as performance parameters equal in priority to any other

    performance parameters for any major weapon system prior to beginning

    the acquisition program. (p. 66/GAO Draft Report):

    DoD RESPONSE: Partially Concur. The Operational Requirements Document

    currently addresses program affordability, stated in terms of a

    performance parameter threshold and objective, and system performance

    parameters, including mission reliability; they are equal in priority

    to any other �non-KPP� performance parameter. The supporting guidance

    in CJCSI 3170.01 is currently under revision-the Department has not

    decided whether to clarify it with a more specific description towards

    weapon system readiness rates.

    RECOMMENDATION 2: The GAO recommended that the Secretary of Defense

    revise the current policy governing the operation of the defense

    acquisition system to require that the product developer establish a

    firm estimate of a weapon system�s reliability based on demonstrated

    reliability rates at the component and subsystem level no later than

    the end of the system integration phase, coinciding with the system-

    level critical design review, before proceeding into the system

    demonstration phase of product development; and at the system level no

    later than the full-rate production decision. This policy is currently

    under revision, to be finalized in early 2003. (p. 66/GAO Draft

    Report):

    DoD RESPONSE: Partially Concur: While there is no formal direction to

    provide a firm estimate of reliability during critical design review,

    program system performance is reviewed for the final design

    configuration, including compliance to any reliability requirement or

    specification included in the contract. Because this is already

    accomplished, there is no need to further direct the recommendation.

    RECOMMENDATION 3: The GAO recommended that the Secretary of Defense

    structure its contracts for major systems acquisitions at Milestone B

    to provide incentives for the product developer to ensure that proper

    trades are made between reliability and performance prior to the

    production decision. One option is to provide specific clauses in the

    development contract to address reliability growth. (p. 66/GAO Draft

    Report):

    DoD RESPONSE: Partially Concur: DoD views reliability as part of system

    performance. Also, DoD already encourages system design trades

    throughout development. Incentivizing reliability growth is one of many

    possible approaches already used in some cases. There is no need for

    additional emphasis by DoD. Other approaches used by DoD include total

    system support responsibility (TSSR) which automatically incentivizes

    contractors to provide reliable systems while increasing their profits.

    [End of section]

    Appendix II GAO Staff Acknowledgments:

    Acknowledgments:

    Cheryl Andrew, Beverly Breen, Belinda LaValle, Carol Mebane, Gary

    Middleton, Michael Sullivan, Adam Vodraska, and Earl C. Woodard:

    FOOTNOTES

    [1] An open system is one that is designed with interfaces to accept

    upgrades easily without redesign of the total unit. Replacements in an

    open system only have to meet interface requirements to be accepted.

    [2] A key performance parameter represents a capability that is so

    significant that failure to meet the minimum value could be a reason

    for DOD or the services to reevaluate the concept or system or

    terminate the program.

    [3] Operations and support of weapons systems is a part of the

    Operations and Maintenance budget, which also includes amounts for

    health care, base and facilities support, and other activities for the

    well-being and operations of the military forces. Costs for operations

    and support of weapon systems were about 48 percent of the Operations

    and Maintenance budget in fiscal year 2002.

    [4] The operating and support cost for the Apache is not available

    before 1993.

    [5] The operating and support cost for the M-1 series Abram is not

    available before 1993.

    [6] U.S. General Accounting Office, Best Practices: Better Matching of

    Needs and Resources Will Lead to Better Weapon System Outcomes, GAO-01-

    288 (Washington, D.C.: Mar. 8, 2001).

    [7] U.S. General Accounting Office, Best Practices: Capturing Design

    and Manufacturing Knowledge Early Improves Acquisition Outcomes, GAO-

    02-701 (Washington, D.C.: July 15, 2002).

    [8] Chairman of the Joint Chiefs of Staff Instruction, Requirements

    Generation System, (Washington, D.C.: Apr. 15, 2001).

    [9] GAO-01-288.

    [10] A threshold is the minimum acceptable operational combat
    capability

    required to meet war-fighter minimum requirements. An objective is the

    capability desired of the system beyond minimum requirements.

    [11] U.S. General Accounting Office, Joint Strike Fighter Acquisition:

    Mature Critical Technologies Needed to Reduce Risks, GAO-02-39

    (Washington, D.C.: Oct. 19, 2001).

    [12] Department of Defense, Office of the Inspector General, Use of An

    Open Systems Approach for Weapon Systems, Report No. D-2000-149

    (Washington, D.C.: June 14, 2000).

    GAO�s Mission:

    The General Accounting Office, the investigative arm of Congress,

    exists to support Congress in meeting its constitutional

    responsibilities and to help improve the performance and accountability

    of the federal government for the American people. GAO examines the use

    of public funds; evaluates federal programs and policies; and provides

    analyses, recommendations, and other assistance to help Congress make

    informed oversight, policy, and funding decisions. GAO�s commitment to

    good government is reflected in its core values of accountability,

    integrity, and reliability.

    Obtaining Copies of GAO Reports and Testimony:

    The fastest and easiest way to obtain copies of GAO documents at no

    cost is through the Internet. GAO�s Web site ( www.gao.gov ) contains

    abstracts and full-text files of current reports and testimony and an

    expanding archive of older products. The Web site features a search

    engine to help you locate documents using key words and phrases. You

    can print these documents in their entirety, including charts and other

    graphics.

    Each day, GAO issues a list of newly released reports, testimony, and

    correspondence. GAO posts this list, known as �Today�s Reports,� on its

    Web site daily. The list contains links to the full-text document

    files. To have GAO e-mail this list to you every afternoon, go to

    www.gao.gov and select �Subscribe to daily E-mail alert for newly

    released products� under the GAO Reports heading.

    Order by Mail or Phone:

    The first copy of each printed report is free. Additional copies are $2

    each. A check or money order should be made out to the Superintendent

    of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or

    more copies mailed to a single address are discounted 25 percent.

    Orders should be sent to:

    U.S. General Accounting Office

    441 G Street NW,

    Room LM Washington,

    D.C. 20548:

    To order by Phone:

    	Voice: (202) 512-6000:

    	TDD: (202) 512-2537:

    	Fax: (202) 512-6061:

    To Report Fraud, Waste, and Abuse in Federal Programs:

    Contact:

    Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

    Automated answering system: (800) 424-5454 or (202) 512-7470:

    Public Affairs:

    Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

    General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

    20548: