    Joint Strike Fighter Acquisition: Mature Critical Technologies
    Needed to Reduce Risks (19-OCT-01, GAO-02-39).

    The Joint Strike Fighter Program (JSFP) is the most expensive
    aircraft program in the Department of Defense (DOD). It is
    intended to produce affordable, next-generation aircraft to
    replace aging aircraft in the military services' inventories.
    Although JSFP has made good progress in some technology areas,
    the program is at risk of not meeting its affordability objective
    because critical technologies are not projected to be matured to
    levels GAO believes would indicate a low risk program at the
    planned start of engineering and manufacturing development in
    October 2001.
    -------------------------Indexing Terms-------------------------
    REPORTNUM:   GAO-02-39
        ACCNO:   A02347
      TITLE:     Joint Strike Fighter Acquisition: Mature Critical
    Technologies Needed to Reduce Risks
         DATE:   10/19/2001
      SUBJECT:   Fighter aircraft
    	     Procurement planning
    	     Defense cost control
    	     Defense procurement
    	     Joint Direct Attack Munition
    	     Joint Strike Fighter
    	     Best Practices

    ******************************************************************
    ** This file contains an ASCII representation of the text of a  **
    ** GAO Testimony.                                               **
    **                                                              **
    ** No attempt has been made to display graphic images, although **
    ** figure captions are reproduced.  Tables are included, but    **
    ** may not resemble those in the printed version.               **
    **                                                              **
    ** Please see the PDF (Portable Document Format) file, when     **
    ** available, for a complete electronic file of the printed     **
    ** document's contents.                                         **
    **                                                              **
    ******************************************************************
    GAO-02-39

    Report to the Chairman, Subcommittee on National Security, Veterans'
    Affairs, and International Relations, Committee on Government Reform, House
    of Representatives

    United States General Accounting Office

    GAO

    October 2001 JOINT STRIKE FIGHTER ACQUISITION

    Mature Critical Technologies Needed to Reduce Risks

    GAO- 02- 39

    Page i GAO- 02- 39 Joint Strike Fighter Acquisition Letter 1

    Results in Brief 2 Background 4 Technology Readiness Assessments Provide
    Opportunities to

    Improve Outcomes 6 Critical Technologies Are Not Projected to Be at Low Risk
    8 Conclusions 11 Recommendations for Executive Action 11 Agency Comments and
    Our Evaluation 12 Scope and Methodology 13

    Appendix I Technology Readiness Levels and Their Descriptions 15

    Appendix II Critical Technologies and Their Descriptions 16

    Appendix III Comments From the Department of Defense 17

    Table

    Table 1: Military Services? Planned Use for the Joint Strike Fighter 4

    Figures

    Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft Design
    Concepts 6 Figure 2: Actual and Projected Joint Strike Fighter Critical

    Technology Readiness Levels 9

    Abbreviations

    DOD Department of Defense TRL technology readiness level Contents

    Page 1 GAO- 02- 39 Joint Strike Fighter Acquisition

    October 19, 2001 The Honorable Christopher Shays Chairman, Subcommittee on
    National Security,

    Veterans? Affairs, and International Relations Committee on Government
    Reform House of Representatives

    Dear Mr. Chairman: The Joint Strike Fighter Program is the most expensive
    aircraft program in the Department of Defense (DOD). It is intended to
    produce affordable, next- generation aircraft to replace aging aircraft in
    the military services? inventories. As currently planned, the program will
    cost about $200 billion to develop and procure about 3,000 aircraft and
    related support equipment. Two contractor teams, led by the Boeing Company
    and by the Lockheed Martin Aeronautics Company, are competing for the
    engineering and manufacturing development phase. This phase of the program
    is projected to last about 8 years and cost about $20 billion and typically
    involves large, fixed investments in human capital, facilities, and
    materials.

    Last year, we testified and reported that a key objective of the program?s
    acquisition strategy is affordability and that a part of that strategy-
    entering into engineering and manufacturing development with low technical
    risk- would not be achieved because technologies critical to meeting the
    program?s cost and requirement objectives were projected to be at low levels
    of technical maturity in April 2001, the date then scheduled for awarding
    the engineering and manufacturing development contract. 1 We stated that the
    program?s approach was not consistent with best practices in which
    technologies are more fully developed before proceeding into product
    development. Organizations that use best practices recognize that delaying
    the resolution of technology problems until product development- analogous
    to the engineering and manufacturing development phase- can result in at
    least a ten- fold cost

    1 Joint Strike Fighter Acquisition: Development Schedule Should Be Changed
    to Reduce Risks (GAO/ T- NSIAD- 00- 132, Mar. 16, 2000); Joint Strike
    Fighter Acquisition: Development Schedule Should Be Changed to Reduce Risks
    (GAO/ NSIAD- 00- 74, May 9, 2000); and Defense Acquisitions: Decisions on
    the Joint Strike Fighter Will Be Critical for Acquisition Reform (GAO/ T-
    NSIAD- 00- 173, May 10, 2000).

    United States General Accounting Office Washington, DC 20548

    Page 2 GAO- 02- 39 Joint Strike Fighter Acquisition

    increase; delaying the resolution until after the start of production could
    increase costs by a hundred- fold.

    Because of concerns about the adequacy of the Joint Strike Fighter?s short
    take- off and vertical landing flight test program, the maturity of its
    critical technologies, and other factors, the Fiscal Year 2001 National
    Defense Authorization Act directed that the contract for the aircraft?s
    engineering and manufacturing development not be awarded until certain
    criteria were met. 2 For example, the act required that the program?s short
    take- off and vertical landing demonstration aircraft accumulate at least 20
    hours of flight test time before the program could enter the engineering and
    manufacturing development phase. The engineering and manufacturing
    development contract award is now scheduled for October 2001.

    At your request, we assessed whether the Joint Strike Fighter?s critical
    technologies will have matured to a low technical risk at the start of the
    upcoming engineering and manufacturing development phase. 3 We assessed
    technical maturity based on technology readiness levels- measures pioneered
    by the National Aeronautics and Space Administration and adopted by the Air
    Force Research Laboratory to determine the readiness of technologies to be
    incorporated into a weapon or other type of system. 4 The research
    laboratory considers a technology to be low risk for the engineering and
    manufacturing development stage when a prototype of that technology has been
    developed that includes all of its critical components in approximately the
    same size, weight, and configuration of the end product and that prototype
    has been demonstrated to work in an environment similar to the planned
    operational system.

    Although the Joint Strike Fighter program has made good progress in some
    technology areas, the program is at risk of not meeting its affordability
    objective because critical technologies are not projected to

    2 Section 212, P. L. 106- 398, Oct. 30, 2000. 3 A technology is considered
    to be mature when it has been developed to a point that it can be readily
    integrated into a new product and counted on to meet product requirements. 4
    The Air Force Research Laboratory is a science and technology organization
    that matures advanced technologies to the point that they can be included in
    weapon system programs and be expected to perform as required. Results in
    Brief

    Page 3 GAO- 02- 39 Joint Strike Fighter Acquisition

    be matured to levels that we believe would indicate a low risk program at
    the planned start of engineering and manufacturing development in October
    2001. In fact, the critical technologies are projected to be at the same
    level of maturity in October 2001 as they were projected to be in April
    2001, the previous scheduled date for the engineering and manufacturing
    development contract. Several of these technologies under development are
    critical to making the long- term ownership costs of the program affordable.
    These technologies remain at higher risk levels for engineering and
    manufacturing development because (1) they have not been developed to
    approximately the same size, weight, and configuration of the end product
    and/ or (2) they have not been demonstrated to work in an environment
    similar to the planned operational system. For example, the contractors have
    demonstrated certain key component technologies in flight environments, but
    the tested hardware was not always the same size and weight required for the
    Joint Strike Fighter aircraft. In other instances, components built to the
    required size and weight were only demonstrated in ground- tests. 5

    Based on our work on best practices in product development, and the
    importance of the Joint Strike Fighter to DOD and the industrial base, we
    believe that DOD needs to ensure that the program?s critical technologies
    are at demonstrated levels of maturity before making engineering and
    manufacturing investments in the program. Failure to do so could result in
    increases in both the production and long- term ownership costs, schedule
    delays, and compromised performance as problems arise in product
    development. Moreover, the impact of failing to mature one critical
    technology could ripple throughout the program. Therefore, we are
    recommending that DOD ensure that critical technologies are mature before
    proceeding into engineering and manufacturing to improve the likelihood of
    meeting program expectations or to take additional actions if it chooses to
    accept the risk of immature technologies.

    In commenting on our report, DOD said that it assessed the technology
    maturity of the Joint Strike Fighter to be sufficient for the next phase of
    the program. DOD also said that the Joint Strike Fighter Program Office has
    implemented a rigorous risk management program that will continue to monitor
    and address technology risks, as well as other risks, throughout the
    program?s life. We disagree with DOD?s assertion that technology is

    5 Due to the current Joint Strike Fighter competition, the technologies are
    not specified so as to not associate them with either Boeing or Lockheed
    Martin, respectively.

    Page 4 GAO- 02- 39 Joint Strike Fighter Acquisition

    mature enough to move forward. The technology readiness level assessment
    conducted as part of our review of the Joint Strike Fighter showed that
    critical technologies are not projected to be matured to levels that would
    stem risks at the start of engineering and manufacturing development. Our
    previous work has shown that when programs proceed in this fashion, they
    experience delays, rework, and substantial cost increases that could force
    the Department to divert much- needed funds from other important weapon
    system programs.

    The Joint Strike Fighter Program is structured to use a common production
    line to produce three versions of a single aircraft. These aircraft will be
    tailored to meet conventional flight requirements for the U. S. Air Force,
    short take- off and vertical landing characteristics for the U. S. Marine
    Corps, and carrier operation suitability needs for the U. S. Navy. The
    program will also provide aircraft to the British Royal Navy and Air Force.
    Table 1 shows the services? planned use for the Joint Strike Fighter.

    Table 1: Military Services? Planned Use for the Joint Strike Fighter Service
    Quantity Planned use

    U. S. Air Force 1, 763 Replacement for F- 16 and A- 10; complement the F- 22
    U. S. Marine Corps 609 Replacement for the AV- 8B and F/ A- 18 C/ D U. S.
    Navy 480 Complement the F/ A- 18 E/ F Great Britain 150 Replacement for the
    Sea Harrier and GR. 7 Source: Joint Strike Fighter program office.

    A key objective of the Joint Strike Fighter acquisition strategy is
    affordability- reducing the development, production, and ownership costs of
    the program relative to prior fighter aircraft programs. To achieve its
    affordability objective, the Joint Strike Fighter program has incorporated
    various acquisition initiatives into the program?s acquisition strategy and
    various technological advances into the fighter. Among the acquisition
    initiatives planned was to develop critical technologies to a level where
    they represent low technical risk before the engineering and manufacturing
    contract is awarded. The expectation was that incorporating these
    initiatives into the acquisition strategy would avoid cost growth, schedule
    slippage, and performance shortfalls that have been experienced in other
    weapon acquisition programs. Background

    Page 5 GAO- 02- 39 Joint Strike Fighter Acquisition

    To date, the Joint Strike Fighter Program has awarded contracts totaling
    over $2 billion to Boeing and Lockheed Martin for the current concept
    demonstration phase. During this phase, DOD required each contractor to
    design and build two aircraft to demonstrate the following:

     commonality/ modularity to validate the contractors? ability to produce
    three aircraft versions on the same production line;

     the aircraft?s ability to do a short take- off and vertical landing,
    hover, and transition to forward flight; and

     satisfactory low airspeed, carrier approach flying and handling qualities.
    Each contractor was required to submit a Preferred Weapon System Concept,
    which outlines its final design concept for developing a Joint Strike
    Fighter aircraft that is affordable and meets performance requirements. The
    Preferred Weapon System Concept includes results from the flight and ground
    demonstrations and is being used by DOD to select the winning aircraft
    design and to award the engineering and manufacturing development contract.
    During engineering and manufacturing development, the Joint Strike Fighter
    will be fully developed, engineered, designed, fabricated, tested, and
    evaluated to demonstrate that the production aircraft will meet stated
    requirements. Critical junctures in engineering and manufacturing
    development are the preliminary and critical design reviews and commitments;
    testing of aircraft; and commitments to production hardware, including the
    purchase of long lead production items. It is at the critical design review
    that decisions are made toward finalizing the aircraft design and begin
    building test aircraft. About two- thirds of engineering and manufacturing
    development funding will be spent after this review. Figure 1 shows planned
    Joint Strike Fighter aircraft designs by contractor.

    Page 6 GAO- 02- 39 Joint Strike Fighter Acquisition

    Figure 1: Boeing and Lockheed Martin Joint Strike Fighter Aircraft Design
    Concepts

    Source: Boeing and Lockheed Martin.

    In our previous work on best business practices, commercial firms have told
    us that a key part of product development is getting the technology into the
    right size, weight, and configuration needed for the intended product- in
    this case, the final Joint Strike Fighter design. Once this has been
    demonstrated, the technology is at an acceptable level for product
    development. Technology readiness levels (TRL) can be used to assess the
    maturity of technology and can reveal whether a gap exists between a
    technology?s maturity and the maturity demanded for successful inclusion in
    the intended product. Defining this gap for the Joint Strike Fighter
    technologies is important for determining whether they can be expected to
    demonstrate required capabilities before being integrated into the aircraft
    design. Readiness levels are measured along a scale of one to nine, starting
    with paper studies of the basic concept, proceeding with laboratory
    demonstrations, and ending with a technology that has proven itself on the
    intended product. (See app. I for a detailed description of TRLs.) The Air
    Force Research Laboratory considers TRL 7 an acceptable risk for starting
    the engineering and manufacturing development phase. The readiness
    Technology Readiness

    Assessments Provide Opportunities to Improve Outcomes

    Page 7 GAO- 02- 39 Joint Strike Fighter Acquisition

    level definitions state that for a technology to be rated at TRL 7, it must
    be demonstrated using prototype hardware (such as a complete radar
    subsystem) that is the same size, weight, and configuration as that called
    for in the final aircraft design and that prototype has to be demonstrated
    to work in an environment similar to the planned operational system.

    We have previously reviewed the impact of incorporating technologies into
    new product and weapon system designs. 6 The results showed that programs
    met product objectives when the technologies were matured to higher levels
    and conversely showed that cost and schedule problems arose when programs
    started when technologies were at low readiness levels. For example, the
    Joint Direct Attack Munition (JDAM) used modified variants of proven
    components for guidance and global positioning. It also used mature,
    existing components from other proven manufacturing processes for its own
    system for controlling tail fin movements. The munition was touted for its
    performance in Kosovo and was purchased for less than half of its expected
    unit cost. However, the Comanche helicopter program began with critical
    technologies such as the engine, rotor, and integrated avionics at TRL
    levels of 5 or below. That program has seen 101 percent cost growth and 120-
    percent schedule slippage as a result of these low maturity levels and other
    factors.

    In commenting on our report concerning better management of technology
    development, DOD agreed that TRLs are important and necessary in assisting
    decision makers in deciding on when and where to insert new technologies
    into weapons system programs and that it is desirable to mature technologies
    to TRL 7 prior to entering the engineering and manufacturing development
    phase of a weapon system program. 7 Since that time, DOD has adopted the
    technology readiness levels as a means of assessing the technological
    maturity of new major programs. In a July 5, 2001, memorandum, the Deputy
    Under Secretary of Defense (Science and Technology) stated that new DOD
    regulations require that the military services? science and technology
    executives conduct a technology readiness level assessment for critical
    technologies identified in major weapon systems programs prior to the start
    of engineering and

    6 Best Practices: Successful Application to Weapon Acquisitions Requires
    Changes in DOD?s Environment (GAO/ NSIAD- 98- 56, Feb. 24, 1998) and Best
    Practices: Better Management of Technology Development Can Improve Weapon
    System Outcomes

    (GAO/ NSIAD- 99- 162, July 30, 1999). 7 GAO/ NSIAD- 99- 162, July 30, 1999.

    Page 8 GAO- 02- 39 Joint Strike Fighter Acquisition

    manufacturing development and production. The memorandum notes that
    technology readiness levels are the preferred approach for all new major
    programs unless the Deputy Under Secretary approves an equivalent assessment
    method.

    The Joint Strike Fighter Program, like many other DOD programs, has used
    risk management plans and engineering judgment as a way of assessing
    technological maturity. The Principal Deputy Under Secretary of Defense
    (Acquisition and Technology) has determined that these means will continue
    to be used by DOD and the Joint Strike Fighter contractors to assess the
    program?s technological risk. Risk management plans and judgment are
    necessary to managing any major development effort like the Joint Strike
    Fighter. However, without an underpinning such as technology readiness
    levels that allow transparency into program decisions, these methods allow
    significant technical unknowns to be judged acceptable risks because a plan
    exists for resolving the unknowns in the future. Experience on previous
    programs has shown that such methods have rarely assessed technical unknowns
    as a high or unacceptable risk; consequently, they failed to guide programs
    to meet promised outcomes. Technology readiness levels are based on actual
    demonstrations of how well technologies actually perform. Their strength
    lies in the fact that they characterize knowledge that exists rather than
    plans to gain knowledge in the future; they are, thus, less susceptible to
    optimism.

    In May 2000 we reported that all of the eight technologies identified by the
    Joint Strike Fighter program office as critical to the program were expected
    to be at maturity levels below that considered acceptable for low risk when
    entering engineering and manufacturing development (TRL 7). The eight
    critical technologies are: prognostics and health management, integrated
    flight propulsion control, subsystems, integrated support system, integrated
    core processor, radar, manufacturing, and mission systems integration. (See
    app. II for a description of these technologies.) During our review last
    year, we worked with the two competing contractors and the program office to
    arrive at the applicable TRLs for the critical technologies. Specifically,
    on separate visits to the contractors, with program office personnel
    present, we asked the contractors? relevant technology managers to score the
    technologies they considered critical to enable their Joint Strike Fighter
    design to meet DOD requirements for the aircraft. At that time, we also
    asked them to describe their plans to mature the technologies to the planned
    start of the engineering and manufacturing development phase, then scheduled
    for April 2001. Upon reviewing these Critical Technologies

    Are Not Projected to Be at Low Risk

    Page 9 GAO- 02- 39 Joint Strike Fighter Acquisition

    scores with the program office and in order to gain an overall Joint Strike
    Fighter Program perspective on technical maturity, the Joint Strike Fighter
    office agreed to provide us with TRL scores for the eight technologies they
    considered critical for meeting program cost and performance requirements.
    Figure 2 reflects the program office scores at the time of our last review.
    Due to the current Joint Strike Fighter competition, the specific
    technologies mentioned previously are not linked to scores so as not to
    divulge competition sensitive information.

    Figure 2: Actual and Projected Joint Strike Fighter Critical Technology
    Readiness Levels

    Source: Joint Strike Fighter program office.

    As the figure shows, all eight technologies were projected to be below the
    level of maturity (TRL 7) considered acceptable for low risk when entering
    the engineering and manufacturing development phase and six of the

    Page 10 GAO- 02- 39 Joint Strike Fighter Acquisition

    technologies were projected to be below the level of maturity (TRL 6) that
    is considered low risk for entering the demonstration phase, which the Joint
    Strike Fighter Program began in 1996.

    During our current review, we again visited the two competing contractors to
    discuss the status of the eight technologies. We learned that they have
    essentially accomplished, or plan to accomplish by October 2001, the
    technology development and demonstrations that they planned to accomplish as
    of April 2001. Thus figure 2 represents the current assessment of technical
    maturity. While two of these areas are very close to appropriate maturity
    levels, the Joint Strike Fighter?s critical technologies are not projected
    to be matured to levels that we believe would indicate a low risk program at
    the planned start of the engineering and manufacturing development phase.
    Key component technologies remain at higher risk levels for engineering and
    manufacturing development because (1) they have not been developed to
    approximately the same size, weight, and configuration called for in the
    final aircraft design and/ or (2) they have not been demonstrated to work in
    an environment similar to the planned operational system.

    The Joint Strike Fighter Program has made good progress in some technology
    areas. For example, contractor and program officials told us that because of
    concerns about propulsion technology, both contractors focused considerable
    attention on that area. Both contractors flew aircraft that demonstrated the
    capability for short take- off and vertical landing and accumulated at least
    20 hours of flight time on those aircraft, which should satisfy the
    requirement in the Fiscal Year 2001 National Defense Authorization Act. In
    some other areas, the technology maturation has not been uniform across all
    critical components of a technology. For example, the radar has a number of
    critical components that must work together as a system. Both contractors
    have made considerable progress on one or more of those components, but the
    other critical components have not been matured to an acceptable level of
    risk. In order for this technology to achieve a TRL level of 7, all
    components had to be (1) demonstrated in the size and weight required to
    meet aircraft capabilities, (2) integrated together as they would be in the
    final aircraft design, and (3) flown in an environment similar to what the
    Joint Strike Fighter will be subjected. To demonstrate some critical
    technologies, both contractors flew key electronic and other components in
    flying avionics test beds (commercial aircraft reconfigured as flying
    laboratories). While these tests occurred in a relevant environment (e. g.,
    in flight), the tested hardware was not always the same size and weight
    required for the Joint Strike Fighter aircraft. Conversely, some components
    were built to the required size and weight,

    Page 11 GAO- 02- 39 Joint Strike Fighter Acquisition

    but were demonstrated only in ground- testing environments. By not having
    matured all critical technology areas to appropriate maturity levels, the
    program remains at risk for achieving cost and performance goals upon
    entering product development.

    Moving into engineering and manufacturing development creates an expectation
    that the Joint Strike Fighter can be delivered for a stated time and dollar
    investment and with a given set of capabilities. The decisions the
    Department of Defense makes now and over the next 2 years will largely
    determine whether those expectations can be met.

    A key component of the Joint Strike Fighter Program?s acquisition strategy
    is to enter the engineering and manufacturing development phase with low
    technical risk. The program will not have achieved that point by October
    2001 because technologies, which the Joint Strike Fighter Program Office
    identified as critical to meeting the program?s cost and requirements
    objectives, will not have been matured to an acceptable risk level. By
    entering the engineering and manufacturing development phase with immature
    critical technologies, the program will need to continue to develop those
    technologies at the same time it will be concentrating on production issues
    and the integration of subsystems into a Joint Strike Fighter. This approach
    would not be consistent with best practices. In fact, it would more closely
    follow DOD?s traditional practices in weapon system programs that have often
    resulted in cost increases, schedule delays, and compromised performance.

    To eliminate one of the major sources of cost and schedule risk, we
    recommend that the Secretary of Defense delay the start of engineering and
    manufacturing development until critical technologies are matured to
    acceptable levels.

    Alternatively, if the Secretary of Defense decides to accept these risks and
    move the program into engineering and manufacturing development as
    scheduled, we recommend that the Secretary dedicate the resources to
    ensuring that maturity of the critical technologies is demonstrated by the
    critical design review or defer the inclusion of immature technologies from
    the approved design. Conclusions

    Recommendations for Executive Action

    Page 12 GAO- 02- 39 Joint Strike Fighter Acquisition

    In written comments on a draft of this report, the Director of Strategic and
    Tactical Systems, within the Office of the Under Secretary of Defense for
    Acquisition, Technology, and Logistics, partially concurred with our
    recommendation. DOD contended that an independent technology readiness
    assessment it carried out on the program showed that technology has
    extensively matured and the program is now ready to enter into systems
    development and demonstration. DOD also stated that the Joint Strike Fighter
    Program Office has implemented a risk management program that will continue
    to monitor and address technology risks, as well as other risks, throughout
    the program?s life. The full text of DOD?s comments is included in appendix
    III.

    We disagree with the Department?s assessment of technological maturity. The
    TRL assessment conducted as part of our review showed that technologies
    critical to the Joint Strike Fighter Program are not projected to be matured
    to levels that we believe would indicate a low risk program at the planned
    start of the engineering and manufacturing development phase. Many of the
    technologies have not been demonstrated in their appropriate size and
    weight, nor have they been demonstated to function in an environment in
    which they will be used. For example, many of the technologies are still in
    the laboratory and will require considerable maturation before they can be
    incorporated into the final design. By entering the engineering and
    manufacturing development phase with immature critical technologies, the
    program will need to continue to develop those technologies at the same time
    it will be concentrating on engineering, designing, and fabricating the
    product. As it has with many other DOD programs, this approach increases the
    likelihood of schedule delays and program cost increases. This is primarily
    why DOD?s new acquisition regulations emphasize separating technology
    development from product development. In fact, experience has shown that
    resolving technology problems in product development can result in at least
    a tenfold cost increase.

    Moreover, DOD incorrectly states that the tools it used to assess its
    technology and the TRLs used for our review are equivalent methodologies for
    assessing technological maturity. The Willoughby Templates used by DOD are a
    risk management tool. They can be an excellent way to manage program risks,
    but in practice they have not been used to identify risk. Identifying risk
    is the first step to managing it. By contrast, by focusing specifically on
    assessing technology maturity against objective standards, TRLs have proven
    successful at identifying risks. A more appropriate approach for DOD to take
    is to use technology readiness levels in conjunction with a management tool
    such as the Willoughby Agency Comments

    and Our Evaluation

    Page 13 GAO- 02- 39 Joint Strike Fighter Acquisition

    templates since this can result in more informed decision making and fewer
    unanticipated problems in an acquisition program. In fact, the Joint Strike
    Fighter program provides DOD with an excellent opportunity to apply these
    concepts in tandem.

    To assess whether the Joint Strike Fighter?s critical technologies are
    projected to mature to low technical risk at the start of the engineering
    and manufacturing development phase, we used the technology readiness level
    tool and information provided by Joint Strike Fighter program officials and
    contractor officials at the Boeing Company, Seattle, Washington; Lockheed
    Martin Aeronautics Company, Fort Worth, Texas; and Pratt & Whitney, East
    Hartford, Connecticut. During our previous review, we had obtained detailed
    briefings from Boeing and Lockheed Martin officials on their plans to mature
    critical technologies prior to the date for awarding the engineering and
    manufacturing development contract, then scheduled for April 2001. We had
    also obtained program office and contractor assessments of the expected
    technology readiness levels for the critical technologies at April 2001.
    During our current review, we obtained detailed briefings from program
    office personnel on the status of critical technologies. We also obtained
    detailed briefings from Boeing, Lockheed Martin, and Pratt & Whitney
    officials on the contractors? progress in maturing critical technologies and
    any further maturation plans through October 2001. We compared the latest
    information from the program office and the contractors to the information
    obtained during our prior review to determine if the critical technologies
    had been matured to higher technology readiness levels and the levels
    achieved.

    We conducted our review from April through September 2001 in accordance with
    generally accepted government auditing standards.

    We are sending copies of this report to the congressional defense
    committees; the Honorable Donald H. Rumsfeld, Secretary of Defense; the
    Honorable James G. Roche, Secretary of the Air Force; the Honorable Gordon
    R. England, Secretary of the Navy; General James L. Jones, Commandant of the
    Marine Corps; and the Honorable Mitchell E. Daniels, Jr., Director, Office
    of Management and Budget. We will also make copies available to other
    interested parties on request. Scope and

    Methodology

    Page 14 GAO- 02- 39 Joint Strike Fighter Acquisition

    Please contact me at (202) 512- 4841 if you or your staff have any questions
    concerning this report. Key contributors to this report were Robert
    Pelletier and Brian Mullins.

    Sincerely yours, Katherine V. Schinasi Director, Acquisition and

    Sourcing Management

    Appendix I: Technology Readiness Levels and Their Descriptions

    Page 15 GAO- 02- 39 Joint Strike Fighter Acquisition

    Technology readiness level Description

    1. Basic principles observed and reported. Lowest level of technology
    readiness. Scientific research begins to be translated into

    applied research and development. Examples might include paper studies of a
    technology?s basic properties. 2. Technology concept and/ or application
    formulated. Invention begins. Once basic principles are observed, practical
    applications can be

    invented. The application is speculative and there is no proof or detailed
    analysis to support the assumption. Examples are still limited to paper
    studies. 3. Analytical and experimental critical function and/ or
    characteristic proof of concept.

    Active research and development is initiated. This includes analytical
    studies and laboratory studies to physically validate analytical predictions
    of separate elements of the technology. Examples include components that are
    not yet integrated or representative. 4. Component and/ or breadboard
    validation in laboratory environment. Basic technological components are
    integrated to establish that the pieces will work

    together. This is relatively ?low fidelity? compared to the eventual system.
    Examples include integration of ?ad hoc? hardware in a laboratory. 5.
    Component and/ or breadboard validation in relevant environment. Fidelity of
    breadboard technology increases significantly. The basic technological

    components are integrated with reasonably realistic supporting elements so
    that the technology can be tested in a simulated environment. Examples
    include ?high fidelity? laboratory integration of components. 6. System/
    subsystem model or prototype demonstration in a relevant environment.
    Representative model or prototype system, which is well beyond the
    breadboard tested

    for TRL 5, is tested in a relevant environment. Represents a major step up
    in a technology?s demonstrated readiness. Examples include testing a
    prototype in a high fidelity laboratory environment or in simulated
    operational environment. 7. System prototype demonstration in an operational
    environment. Prototype near or at planned operational system. Represents a
    major step up from TRL 6,

    requiring the demonstration of an actual system prototype in an operational
    environment, such as in an aircraft, vehicle, or space. Examples include
    testing the prototype in a test bed aircraft. 8. Actual system completed and
    ?flight qualified? through test and demonstration.

    Technology has been proven to work in its final form and under expected
    conditions. In almost all cases, this TRL represents the end of true system
    development. Examples include developmental test and evaluation of the
    system in its intended weapon system to determine if it meets design
    specifications. 9. Actual system ?flight proven? through successful mission
    operations. Actual application of the technology in its final form and under
    mission conditions, such as

    those encountered in operational test and evaluation. In almost all cases,
    this is the end of the last ?bug fixing? aspects of true system development.
    Examples include using the system under operational mission conditions.

    Source: Best Practices: Better Management of Technology Development Can
    Improve Weapon System Outcomes (GAO/ NSIAD- 99- 162, July 30, 1999).

    Appendix I: Technology Readiness Levels and Their Descriptions

    Appendix II: Critical Technologies and Their Descriptions

    Page 16 GAO- 02- 39 Joint Strike Fighter Acquisition

    Critical technologies Description

    Prognostics and health management Involves the ability to detect and isolate
    the cause of aircraft problems and then predict when maintenance activity
    will have to occur on systems with pending failures. Life- cycle cost
    savings are dependent on prognostics and health management through improved
    sortie generation rate, reduced logistics and manpower requirements, and
    more efficient inventory control. Integrated flight propulsion control
    Includes integration of propulsion, vehicle management system, and other
    subsystems

    as they affect aircraft stability, control, and flying qualities (especially
    short take- off and vertical landing). Aircraft improvements are to reduce
    pilot workload and increase flight safety. Subsystems Includes areas of
    electrical power, electrical wiring, environmental control systems, fire

    protection, fuel systems, hydraulics, landing gear systems, mechanisms, and
    secondary power. Important for reducing aircraft weight, decreasing
    maintenance cost, and improving reliability. Integrated support systems
    Involves designing an integrated support concept that includes an aircraft
    with

    supportable stealth characteristics and improved logistics and maintenance
    functions. Life- cycle cost savings are expected from improved low
    observable maintenance techniques and streamlined logistics and inventory
    systems. Integrated core processor Includes the ability to use commercial-
    based processors in an open architecture design

    to provide processing capability for radar, information management,
    communications, etc. Use of commercial processors reduces development and
    production costs and an open architecture design reduces future development
    and upgrade costs. Radar Includes advanced integration of communication,
    navigation, and identification functions

    and electronic warfare functions through improved apertures, antennas,
    modules, radomes, etc. Important for reducing avionics cost and weight, and
    decreasing maintenance cost through improved reliability. Manufacturing
    Involves lean, automated, highly efficient aircraft fabrication and assembly
    techniques.

    Manufacturing costs should be less through improved flow time, lower
    manpower requirements, and reduced tooling cost. Mission systems integration
    Involves decreasing pilot workload by providing information for targeting,
    situational

    awareness, and survivability through fusion of radar, electronic warfare,
    and communication, navigation, and identification data. Improvements are
    achieved through highly integrated concept of shared and managed resources,
    which reduces production costs, aircraft weight, and volume requirements, in
    addition to improved reliability.

    Source: Joint Strike Fighter program office.

    Appendix II: Critical Technologies and Their Descriptions

    Appendix III: Comments From the Department of Defense

    Page 17 GAO- 02- 39 Joint Strike Fighter Acquisition

    Appendix III: Comments From the Department of Defense

    Appendix III: Comments From the Department of Defense

    Page 18 GAO- 02- 39 Joint Strike Fighter Acquisition

    Appendix III: Comments From the Department of Defense

    Page 19 GAO- 02- 39 Joint Strike Fighter Acquisition (120050)

    The General Accounting Office, the investigative arm of Congress, exists to
    support Congress in meeting its constitutional responsibilities and to help
    improve the performance and accountability of the federal government for the
    American people. GAO examines the use of public funds; evaluates federal
    programs and policies; and provides analyses, recommendations, and other
    assistance to help Congress make informed oversight, policy, and funding
    decisions. GAO?s commitment to good government is reflected in its core
    values of accountability, integrity, and reliability.

    The fastest and easiest way to obtain copies of GAO documents is through the
    Internet. GAO?s Web site (www. gao. gov) contains abstracts and full- text
    files of current reports and testimony and an expanding archive of older
    products. The Web site features a search engine to help you locate documents
    using key words and phrases. You can print these documents in their
    entirety, including charts and other graphics.

    Each day, GAO issues a list of newly released reports, testimony, and
    correspondence. GAO posts this list, known as ?Today?s Reports,? on its Web
    site daily. The list contains links to the full- text document files. To
    have GAO E- mail this list to you every afternoon, go to our home page and
    complete the easy- to- use electronic order form found under ?To Order GAO
    Products.?

    The first copy of each printed report is free. Additional copies are $2
    each. A check or money order should be made out to the Superintendent of
    Documents. GAO also accepts VISA and Mastercard. Orders for 100 or more
    copies mailed to a single address are discounted 25 percent. Orders should
    be sent to:

    U. S. General Accounting Office P. O. Box 37050 Washington, D. C. 20013

    To order by phone: Voice: (202) 512- 6000 TDD: (301) 413- 0006 Fax: (202)
    258- 4066

    GAO Building Room 1100, 700 4th Street, NW (corner of 4th and G Streets, NW)
    Washington, D. C. 20013

    Contact: Web site: www. gao. gov/ fraudnet/ fraudnet. htm, E- mail:
    fraudnet@ gao. gov, or 1- 800- 424- 5454 (automated answering system).

    Jeff Nelligan, Managing Director, NelliganJ@ gao. gov (202) 512- 4800 U. S.
    General Accounting Office, 441 G. Street NW, Room 7149, Washington, D. C.
    20548 GAO?s Mission

    Obtaining Copies of GAO Reports and Testimony

    Order by Mail or Phone Visit GAO?s Document Distribution Center

    To Report Fraud, Waste, and Abuse in Federal Programs

    Public Affairs
    *** End of document. ***