Weapons Acquisition: Low-Rate Initial Production Used to Buy Weapon
Systems Prematurely (Chapter Report, 11/21/94, GAO/NSIAD-95-18).

The Pentagon plans to spend more than $79 billion in fiscal year 1995
for the acquisition of weapon systems.  Because the Defense Department
(DOD) allows low-rate initial production to begin before operational
testing and evaluation is completed, the military has amassed large
inventories of unsatisfactory weapons requiring modification; meanwhile,
combat troops have been outfitted with substandard equipment.  Once
begun, low-rate initial production significantly limits the options
available to DOD and Congress when a system proves deficient.  In GAO's
view, the key decision as to whether to proceed with production should
be made at the start of low-rate initial production because, in many
cases, it is also the de-facto full-rate production decision.
Therefore, decisionmakers need good independent information on the
system's performance and suitability at that point.  In today's national
security environment, there should be very few cases in which an urgent
need dictates that DOD will start low-rate initial production without
being certain that the system will work as intended.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-95-18
     TITLE:  Weapons Acquisition: Low-Rate Initial Production Used to 
             Buy Weapon Systems Prematurely
      DATE:  11/21/94
   SUBJECT:  Military procurement
             Testing
             Proposed legislation
             Product performance evaluation
             Advanced weapons systems
             Procurement evaluation
             Defense cost control
             Procurement procedures
             Federal procurement policies
             Defense capabilities
IDENTIFIER:  C-17 Aircraft
             T-45A Aircraft
             Apache Helicopter
             Advanced Medium Range Air-to-Air Missile
             ALR-56C Radar Warning Receiver
             ALR-67 Radar Warning Receiver
             APR-39 Radar Warning Receiver
             ALQ-135 Jammer
             ALQ-131 Block I Jammer
             ALQ-131 Block II Jammer
             ALQ-135 Preplanned Product Improvement Jammer
             ALQ-184 Jammer
             Airborne Self-Protection Jammer
             USM-464 Electronic Warfare Test Set
             F-14D Aircraft
             Pioneer Unmanned Aerial Vehicle
             Family of Medium Tactical Vehicles
             Army Palletized Load System
             Secure Mobile Anti-Jam Reliable Tactical Terminal
             AN/AVR2 Laser Warning System
             F-22 Aircraft
             SLQ-32 Electronic Warfare System
             MK-50 Torpedo
             MH-53E Helicopter
             DOD Multifunctional Information Distribution System
             AMRAAM
             ASPJ
             FMTV
             
**************************************************************************
* This file contains an ASCII representation of the text of a GAO        *
* report.  Delineations within the text indicating chapter titles,       *
* headings, and bullets are preserved.  Major divisions and subdivisions *
* of the text, such as Chapters, Sections, and Appendixes, are           *
* identified by double and single lines.  The numbers on the right end   *
* of these lines indicate the position of each of the subsections in the *
* document outline.  These numbers do NOT correspond with the page       *
* numbers of the printed product.                                        *
*                                                                        *
* No attempt has been made to display graphic images, although figure    *
* captions are reproduced. Tables are included, but may not resemble     *
* those in the printed version.                                          *
*                                                                        *
* A printed copy of this report may be obtained from the GAO Document    *
* Distribution Facility by calling (202) 512-6000, by faxing your        *
* request to (301) 258-4066, or by writing to P.O. Box 6015,             *
* Gaithersburg, MD 20884-6015. We are unable to accept electronic orders *
* for printed documents at this time.                                    *
**************************************************************************


Cover
================================================================ COVER


Report to Congressional Requesters

November 1994

WEAPONS ACQUISITION - LOW-RATE
INITIAL PRODUCTION USED TO BUY
WEAPON SYSTEMS PREMATURELY

GAO/NSIAD-95-18

Weapons Acquisition


Abbreviations
=============================================================== ABBREV

  DOD - Department of Defense
  DOD-IG - DOD Inspector General
  DOT&E - Director, Operational Test and Evaluation
  GAO - General Accounting Office
  LRIP - low-rate initial production
  OT&E - operational test and evaluation

Letter
=============================================================== LETTER


B-257304

November 21, 1994

The Honorable William V.  Roth, Jr.
The Honorable David Pryor
United States Senate

This report, which was prepared at your request, addresses the
Department of Defense's policies and practices on the use of low-rate
initial production in relation to operational test and evaluation. 
It also examines the impact of the current legislative requirements
in each area.  We make recommendations to the Secretary of Defense
and present matters for congressional consideration, both of which
are intended to minimize the risk of prematurely committing to
production and later encountering costly technical problems. 

As agreed with your office, unless you publicly announce its contents
earlier, we plan no further distribution of this report until 30 days
from its issue date.  At that time, we will send copies to the
Chairmen and Ranking Minority Members of the House Committee on
Government Operations, Senate Committee on Governmental Affairs,
House and Senate Committees on Armed Services, and House and Senate
Committees on Appropriations; the Secretaries of Defense, the Army,
the Air Force, and the Navy; and the Director of the Office of
Management and Budget.  We will also make copies available to others
upon request. 

This report was prepared under the direction of Louis J.  Rodrigues,
Director of Systems Development and Production Issues, who may be
reached on (202) 512-4841 if you or your staff have any questions. 
Other major contributors are listed in appendix II. 

Henry L.  Hinton, Jr.
Assistant Comptroller General


EXECUTIVE SUMMARY
============================================================ Chapter 0


   PURPOSE
---------------------------------------------------------- Chapter 0:1

The Department of Defense (DOD) plans to spend over $79 billion in
fiscal year 1995 for the acquisition of weapon systems.  DOD has
acquired some of the most technologically advanced and effective
weapon systems in the world.  However, DOD has often been criticized
for not acquiring the systems in the most efficient manner. 

Because of concern over how DOD initiates the production of weapon
systems, Senators David Pryor and William V.  Roth, Jr., requested
that GAO review DOD's use of a practice called low-rate initial
production (LRIP) in system acquisition programs.  Specifically, GAO
was requested to determine whether LRIP practices were resulting in
production of systems with adequate performance and whether the
legislation underlying LRIP policies was adequate. 


   BACKGROUND
---------------------------------------------------------- Chapter 0:2

The Congress has been concerned about the sometimes significant
quantity of systems produced during LRIP and before system
performance is operationally demonstrated.  As a result, in 1989, the
Congress enacted legislation intended to limit LRIP quantities for
major systems.  The law,
10 U.S.C.  2400, provides that, except for ships and satellites, LRIP
is production in the minimum quantity necessary to (1) provide
production-configured or representative articles for operational test
and evaluation (OT&E), (2) establish an initial production base for
the system, and (3) permit an orderly increase in the production rate
for the system sufficient to lead to full-rate production upon the
successful completion of OT&E. 

OT&E is the primary means of assessing weapon system performance in a
combat-representative environment.  It is defined as (1) the field
test, conducted under realistic conditions, to determine an item's
effectiveness and suitability for use in combat by typical military
users and (2) the evaluation of the results of such a test.  If used
effectively, OT&E can be a key internal control to ensure that
decisionmakers have objective information available on a weapon
system's performance, thereby minimizing risks of procuring costly
and ineffective systems. 

As weapon system programs move through the phases of the acquisition
process, they are subject to review at major decision points called
milestones.  The milestone review process is predicated on the
principle that systems advance to higher acquisition phases by
demonstrating that they have met prescribed technical specifications
and performance thresholds.  Before systems advance to the milestone
that authorizes full-rate production, DOD is statutorily required to
conduct OT&E for all major systems. 


   RESULTS IN BRIEF
---------------------------------------------------------- Chapter 0:3

Current legislation and DOD's acquisition policies permit LRIP to
start before any OT&E is conducted.  The consequences have included
procurement of substantial inventories of unsatisfactory weapons
requiring costly modifications to achieve satisfactory performance
and, in some cases, deployment of substandard systems to combat
forces.  Once started, LRIP significantly limits the options
available to DOD decisionmakers and the Congress when a system is
found to be deficient.  In GAO's view, the key decision as to whether
to proceed with production should be made at the start of LRIP
because, in many cases, it is also the de-facto full-rate production
decision.  Therefore, decisionmakers need good, independent
information on the system's performance and suitability at that
point.  In today's national security environment, there should be
very few cases in which an urgent need dictates that DOD start LRIP
without a demonstrated level of confidence that the system will work
as intended. 


   PRINCIPAL FINDINGS
---------------------------------------------------------- Chapter 0:4


      CURRENT LEGISLATION AND DOD
      ACQUISITION POLICIES PERMIT
      LRIP TO BEGIN WITHOUT ANY
      OT&E
-------------------------------------------------------- Chapter 0:4.1

There are no specific principles or guidelines--in legislation or DOD
policy--on when and how programs should begin LRIP, on the type and
amount of testing to be done before LRIP, on how much LRIP can or
should be done, or under what type of circumstances LRIP should be
curtailed or stopped.  As a result, programs are often permitted to
begin LRIP with little or no scrutiny and before any OT&E has been
conducted.  Although programs are sometimes delayed in getting
approval for full-rate production, LRIP is not usually stopped or
slowed down significantly. 


      CONTROLS NEEDED OVER LRIP
-------------------------------------------------------- Chapter 0:4.2

Over the years, GAO has found numerous instances in which production
of both major and nonmajor systems from all of the services was
optimistically permitted to begin under LRIP and continue based on
factors other than the systems' technical maturity.  Many of the
weapon systems that start production prematurely later experience
significant effectiveness and/or suitability problems.  For example,
the reliability of the Air Force's C-17 aircraft has shown to be
significantly less than expected, and it cannot meet its minimum
payload and range specifications.  Major design changes are often
needed to correct the problems and additional testing is needed to
verify that the corrective action was effective.  For example, major
design changes to correct deficiencies in the Navy's T-45A aircraft
included a new engine and new wings.  When problems are uncovered
after production starts, costly retrofits are often needed for any
delivered units.  Also, problems have sometimes not been fixed, and
substandard systems have been deployed to field units.  For example,
although considered an effective weapon system, the Army's Apache
helicopter has proven to be a difficult and costly system to support. 
Although the thrust of the LRIP legislation is to authorize only
minimum necessary quantities, the continuance of LRIP on an
indefinite basis has resulted in major production commitments.  For
example, while the original plans called for the production of 810
Advanced Medium Range Air-to-Air Missiles during 2 years of LRIP,
over 4,100 were eventually produced during its 6 years of LRIP. 

In a 1993 report, the DOD Inspector General stated that major
acquisition programs were entering LRIP without meeting development,
testing, and production readiness prerequisites.\1 The Inspector
General recommended that DOD (1) provide guidance for all programs on
the specific minimum required program accomplishments for entry into
and continuation of LRIP and (2) require that program-specific exit
criteria be established for entry into and continuation of LRIP.  To
reduce the risk of finding major operational effectiveness and
suitability problems after production starts, GAO has often
recommended as much OT&E as possible before production starts.  GAO
recognizes that, in addition to ship and satellite programs, it may
be very costly and disruptive to suspend certain unique programs
while OT&E is underway.  These programs would typically involve
inherent fabrication complexity, small procurement quantities, high
unit cost, and long unit production periods.  However, one means to
reduce technical risks on these programs would be to conduct OT&E of
key subsystems on surrogate platforms before production starts. 


--------------------
\1 Low-Rate Initial Production in Major Defense Acquisition Programs
(DOD Inspector General Report No.  94-014, Nov.  9, 1993). 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 0:5

GAO recommends that the Secretary of Defense revise DOD system
acquisition policies in the following ways: 

Require that, before entry into LRIP, programs (with the exception of
ships, satellites, and programs that involve inherent fabrication
complexity, small procurement quantities, high unit costs, and long
unit production periods) plan, buy prototypes for, and conduct enough
realistic testing for the services' independent testing agencies
and/or the Director, Operational Test and Evaluation (DOT&E), to be
able to certify to the decision authority that (1) the system's
developmental testing is essentially complete and the basic results
have been validated in an operational environment; (2) the system has
clearly shown that it can meet the key parameters among its minimum
acceptable performance requirements;\2 (3) the system has clearly
demonstrated the potential to fully meet all of its minimum
acceptable requirements for performance and suitability without major
or costly design changes; and (4) the system should be able to
readily complete its remaining OT&E in time to support the planned
full-rate production decision.\3

Require that those programs excluded from the requirement to test
prototypes instead test all key subsystems in an operational
environment before entry into LRIP. 

Adopt the recommendations made by the DOD Inspector General regarding
controls over the start and continuation of LRIP. 

GAO also recommends that the Secretary of Defense work with the
service secretaries to ensure that these policies are implemented for
the acquisition of both major and nonmajor programs. 


--------------------
\2 Minimum acceptable requirements are the values for the particular
parameters that are required to provide a system capability that will
satisfy the validated mission need.  These requirements are also
known as thresholds and are established in the Operational
Requirements Document at each milestone.  Key parameters are those
for which the decision authority would require a reevaluation of
alternative concepts or design approaches if the thresholds are not
met. 

\3 This certification should be made by DOT&E for all major defense
acquisition programs and by the services' independent testing
agencies for all other systems. 


   MATTERS FOR CONGRESSIONAL
   CONSIDERATION
---------------------------------------------------------- Chapter 0:6

GAO recommends that the Congress legislatively mandate (1) that
certain OT&E requirements be met before LRIP may start and (2)
specific limits on the number of units allowed to be produced during
LRIP.  Specifically, the Congress may wish to require that all
defense acquisition programs (major and nonmajor) conduct enough
realistic testing on the entire system or key subsystems to ensure
that key performance parameters are met before LRIP is permitted to
start. 


   AGENCY COMMENTS AND GAO
   EVALUATION
---------------------------------------------------------- Chapter 0:7

In commenting on a draft of this report, DOD stated that while much
of the information reported by GAO is factual, some of the
implications to be drawn from the information are misleading.  DOD
questions the magnitude of the problem and believes that the current
legislation provides sufficient guidance to limit the production of
LRIP units without potentially causing costly and disruptive gaps in
production.  DOD opposes the mandating of OT&E requirements that must
be met before the start of production and fixed limits on the number
of units to be produced in LRIP because they could cause production
stretchouts and production stoppages. 

On numerous occasions, GAO has reported on the problems with
premature commitments to production, the inherent risks in highly
concurrent acquisition strategies, and the benefits of early
operational testing.  However, DOD continues to downplay or deny the
value of testing as a means of ensuring that weapons will work as
expected.  Such a position contradicts the commonsense notion that
quantities of products should be bought only after a clear
demonstration that the product actually works.  In this report, GAO
is suggesting additional legislation because DOD, on its own, has
been unwilling or unable to appropriately control the start and
continuation of production, despite the clear evidence that such
control is needed.  The objective of GAO's recommendations is to
avoid the premature commitment to production and thereby avoid
fielding systems that do not meet requirements and need costly and
time-consuming retrofits. 


INTRODUCTION
============================================================ Chapter 1

In fiscal year 1995, the Department of Defense (DOD) plans to spend
over $79 billion for research, development, test, evaluation, and
production of weapon systems.  While DOD has acquired some of the
most technologically advanced and effective weapon systems, DOD has
often been criticized for not acquiring the systems in the most
efficient manner. 


   SYSTEM ACQUISITION PROCESS AND
   DOD'S POLICY
---------------------------------------------------------- Chapter 1:1

As weapon system programs progress through the phases of the
acquisition process, they are subject to review at major decision
points called milestones.  The milestone review process is predicated
on the principle that systems advance to higher acquisition phases by
demonstrating that they meet prescribed technical specifications and
performance thresholds.  Figure 1.1 illustrates the DOD's weapon
system acquisition process. 

   Figure 1.1:  DOD's Weapon
   System Acquisition Process

   (See figure in printed
   edition.)

At milestone 0, a determination is made about whether an identified
mission need warrants a study of alternative concepts to satisfy the
need.  If warranted, the program is approved to begin the concept
exploration and definition phase.  At milestone I, a determination is
made about whether a new acquisition program is warranted.  If
warranted, initial cost, schedule, and performance goals are
established for the program, and authorization is given to start the
demonstration and validation phase.  At milestone II, a determination
is made about whether continuation of development, testing, and
preparation for production is warranted.  If warranted, authorization
is given to start the engineering and manufacturing development
phase.  Also, approval of this phase will often involve a commitment
to low-rate initial production (LRIP).  At
milestone III, a determination is made about whether the program
warrants a commitment to build, deploy, and support the system. 

DOD acquisition policy states that program risks shall be assessed at
each milestone decision point before approval is granted for the next
phase.  The policy adds that test and evaluation shall be used to
determine system maturity and identify areas of technical risk. 
Operational test and evaluation (OT&E) is a key internal control to
ensure that decisionmakers have objective information available on a
weapon system's performance, to minimize risks of procuring costly
and ineffective systems.  OT&E has been defined as (1) the field
test, under realistic conditions, of any item of (or key component
of) weapons, equipment, or munitions for the purpose of determining
its effectiveness and suitability for use in combat by typical
military users and (2) the evaluation of the results of such a test. 


   LEGISLATIVE REQUIREMENTS ON
   LRIP AND OT&E OF WEAPON SYSTEMS
---------------------------------------------------------- Chapter 1:2

Over a period of many years, the Congress has been concerned about
the performance of weapon systems being acquired by DOD.  As early as
1972, the Congress required DOD to provide it with information on the
OT&E results of major weapon systems before committing them to
production.  However, the Congress continued to receive reports from
the DOD Inspector General (DOD-IG), us, and others that (1) weapon
systems were not being adequately tested before beginning production,
(2) fielded systems were failing to meet their performance
requirements, and (3) OT&E being conducted on weapon systems was of
poor quality. 

In the late 1970s and early 1980s, the Congress enacted a series of
laws to ensure that U.S.  military personnel receive the best weapon
systems possible and that the U.S.  government receives best value
for the defense procurement dollar.  Among other things, these laws

specified that independent OT&E be conducted;

established the Office of the Director, Operational Test and
Evaluation (DOT&E), and assigned it specific oversight duties and
responsibilities;

specified that OT&E of a major defense acquisition program may not be
conducted until DOT&E approves the adequacy of the plans for that
OT&E\1

required that a major system may not proceed beyond LRIP until its
initial OT&E is completed;\2 and

required that DOT&E analyze the results of OT&E conducted for each
major defense acquisition program and, prior to a final decision to
proceed beyond LRIP, report on the adequacy of the testing and
whether the results confirm that the items tested are operationally
effective\3 and suitable\4

for combat. 

In the late 1980s, the Congress found that DOD was acquiring a large
portion of the total program quantities, using the LRIP concept,
without successfully completing OT&E.  In the National Defense
Authorization Act for Fiscal Years 1990 and 1991 (P.L.  101-189), the
Congress addressed this situation by including a definition of LRIP
and a requirement that the determination of the LRIP quantities to be
procured be made when a decision is made to enter engineering and
manufacturing development.  According to the act, LRIP was defined as
the minimum quantity needed to (a) provide production-representative
articles for OT&E, (b) establish an initial production base, and (c)
permit orderly ramp-up to full-rate production upon completion of
OT&E. 

In the conference report for the act, the conferees indicated that
they did not condone the continuous reapproval of LRIP quantities
that eventually total a significant percentage of the total planned
procurement.  Also, the conferees granted an exception to the LRIP
legislation for ship and satellite programs because of their inherent
production complexity, small number, high unit cost, and long unit
production periods.  However, they directed the Secretary of Defense
to develop regulations that capture the spirit of the LRIP
legislation as it applies to these programs.  This special
consideration for ships and satellites carries with it additional
reporting requirements to improve the oversight of these programs. 

Finally, in the National Defense Authorization Act for Fiscal Year
1994, the Congress required that the Secretary of Defense ensure that
appropriate, rigorous, and structured testing be completed prior to
LRIP of any electronic combat or command, control, and communications
countermeasure system. 


--------------------
\1 A "major defense acquisition program" is defined as a system whose
research and development cost is expected to exceed $300 million in
fiscal year 1990 dollars or whose procurement cost is expected to
exceed $1.8 billion in fiscal year 1990 dollars. 

\2 According to DOD regulation, a "major system" is defined as a
system whose research, development, test, and evaluation cost is
estimated to exceed $115 million in fiscal year 1990 dollars or whose
procurement cost is estimated to exceed $540 million in fiscal year
1990 dollars. 

\3 DOD defines "operational effectiveness" as the overall degree of
mission accomplishment of a system when used by representative
personnel in the environment planned or expected for operational
employment of the system considering organization, doctrine, tactics,
survivability, vulnerability, and threat. 

\4 DOD defines "operational suitability" as the degree to which a
system can be placed satisfactorily in field use with consideration
given to such factors as availability, compatibility,
transportability, interoperability, reliability, wartime usage rates,
maintainability, safety, and supportability. 


   OBJECTIVES, SCOPE, AND
   METHODOLOGY
---------------------------------------------------------- Chapter 1:3

Senators David Pryor and William V.  Roth, Jr., requested that we
review DOD's use of LRIP in the acquisition of major defense
programs.  Specifically, the Senators asked that we determine whether

LRIP policies were resulting in the production of systems with
adequate performance capabilities and

the legislation underlying the LRIP policies was adequate. 

We analyzed the legislation and DOD policies governing the production
and testing of weapon systems, particularly those dealing with (1)
the purposes of LRIP, (2) the criteria or requirements for entering
LRIP and full-rate production, and (3) the testing requirements
related to this process.  We used the results of our extensive body
of work from the past decade or so on defense acquisition programs
and the acquisition process\5 to determine whether the LRIP concept,
as currently authorized and practiced by DOD, has resulted in a
premature commitment to production of both major and nonmajor
systems.  We reviewed the 1993 report of the DOD-IG on LRIP and held
discussions with the DOD-IG staff.  We gathered and summarized data
on numerous ongoing system acquisition programs (both major and
nonmajor programs) and supplemented that information with discussions
with officials from the Office of the Secretary of Defense and the
military services. 

In addition, we gathered and analyzed information on the advantages
and disadvantages of conducting OT&E before LRIP (for both major and
nonmajor systems).  We also held discussions with those officials on
DOD's current acquisition strategies and OT&E policies and practices. 

This review was conducted from April 1993 to May 1994 in accordance
with generally accepted government auditing standards. 


--------------------
\5 See the Related GAO Products section of this report for a partial
listing of GAO reports on these topics since late 1990.  Well in
excess of 100 GAO reports, covering a very broad range of programs,
were used in preparing this report. 


DOD LACKS CONTROLS OVER LRIP
============================================================ Chapter 2

Our extensive body of work over the years has amply demonstrated that
improper usage of LRIP has been widespread.  Many major and nonmajor
systems from each of the services have been prematurely committed to
production, which often results in problems being found after a
substantial number of units have been produced and a significant
commitment made to the entire procurement program.  In addition,
contrary to the statutory emphasis on minimum LRIP quantities and
conferee statements, many programs continue in LRIP for prolonged
periods.  DOD's continuing reluctance to employ the discipline of
early OT&E is evident in each of the services and in many major and
nonmajor programs.  Adequate controls have not been established over
the start and continuation of LRIP.  A requirement to successfully
complete enough independent testing in an operational environment to
ensure that the item meets its key performance parameters before LRIP
starts would be feasible in most cases and would be an effective
management control over the premature start of production. 


   PREMATURE COMMITMENTS TO
   PRODUCTION
---------------------------------------------------------- Chapter 2:1

Over the years, we have found numerous instances from all three
services in which production of major and nonmajor systems was
permitted to begin and continue based not on the systems' technical
maturity, but on schedule or other considerations.  DOD has
frequently committed programs to production without assurance that
the systems would perform satisfactorily.  Many of the weapon systems
that start production prematurely later experience significant
operational effectiveness and/or suitability problems.  As a result,
major design changes were often needed to correct the problems,
additional testing was needed to verify that the corrective action
was effective, and costly retrofits were needed for any delivered
units.  A few of the many examples of premature and extensive
commitments to production of major and nonmajor systems are shown in
the following tables.  Table 2.1 shows systems that entered LRIP
before any operational tests were conducted and later experienced
significant problems during the tests.  Table 2.2 shows systems that
were subjected to early operational tests but were allowed to enter
LRIP even though the performance deficiencies were not corrected. 



                                    Table 2.1
                     
                      Examples of Systems That Entered LRIP
                     Before Operational Tests Were Conducted
                      and Later Experienced Problems During
                                   These Tests


                                    Inadequate
                                    system       Percent
                        Program     deployed    procured
System                  category    to field     in LRIP  Comments
----------------------  ----------  ----------  --------  ----------------------
Air Force C-17          Major       To be             33  The C-17's reliability
Aircraft                            determined            is significantly less
                                                          than expected, and the
                                                          system cannot meet
                                                          current payload/range
                                                          specifications. Also,
                                                          while known problems
                                                          with the wings, flaps,
                                                          and slats are being
                                                          fixed, other problems
                                                          continue to emerge.
                                                          (GAO/T-NSIAD-94-166,
                                                          Apr. 19, 1994.)

Air Force AN/ALR-56C    Nonmajor    Yes              8\a  Despite the poor OT&E
Radar Warning Receiver                                    results, the Air Force
                                                          continued full-rate
                                                          production and had
                                                          acquired about 750
                                                          systems at a cost of
                                                          over $570 million, as
                                                          discussed in a
                                                          classified GAO report.

Air Force AN/ALQ-135    Nonmajor    Yes              100  All 65 systems were
Quick Reaction                                            produced under LRIP at
Capability Jammer                                         a cost of $256
                                                          million, before any
                                                          OT&E was conducted.
                                                          Because of performance
                                                          problems, most of the
                                                          jammers were placed in
                                                          storage and only 24
                                                          were installed on
                                                          aircraft. One year
                                                          later, the 24 jammers
                                                          were deactivated
                                                          because of poor
                                                          performance. (GAO/
                                                          NSIAD-90-168, July 11,
                                                          1990.)

Air Force AN/ALQ-135    Nonmajor    Yes             64\b  Through 1993, 331 of
Improved Jammer                                           the 514 planned units
                                                          were acquired under
                                                          LRIP. However, the
                                                          system has encountered
                                                          significant software
                                                          problems, which have
                                                          delayed completion of
                                                          development testing by
                                                          about 2 years. OT&E
                                                          has not yet started.

Air Force AN/ALQ-131    Nonmajor    Yes              100  After the Air Force
Block II Jammer                                           bought most of the
                                                          total quantity of
                                                          units under LRIP,
                                                          tests found serious
                                                          performance problems.
                                                          As a result, the
                                                          system was deployed
                                                          with the receiver/
                                                          processor inoperative
                                                          due to a lack of
                                                          software. Other
                                                          deficiencies were also
                                                          present. (GAO/NSIAD-
                                                          90-168, July 11,
                                                          1990.)

Air Force AN/USM-464    Nonmajor    Yes              100  Before the Air Force
Electronic Warfare                                        conducted OT&E, 72
Test Set                                                  test sets were
                                                          procured under LRIP at
                                                          a cost of $272
                                                          million. Later testing
                                                          showed that the
                                                          equipment would not
                                                          meet requirements, and
                                                          the units were put in
                                                          storage.

Air Force AN/ALQ-184    Nonmajor    Yes              8\c  DOT&E recommended that
Jammer                                                    jammer production be
                                                          stopped because of
                                                          poor OT&E results.
                                                          However, the system
                                                          had already entered
                                                          and continued full-
                                                          rate production
                                                          anyway. We later found
                                                          that most of the 24
                                                          jammers deployed to a
                                                          tactical fighter wing
                                                          had been placed in
                                                          storage. (GAO/NSIAD-
                                                          90-168, July 11,
                                                          1990.)

Navy F-14D Aircraft     Major       Yes              100  OT&E showed that the
                                                          F-14D was not
                                                          sufficiently developed
                                                          and lacked critical
                                                          hardware and software
                                                          capabilities. The
                                                          program was terminated
                                                          after 55 units were
                                                          produced. (GAO/IMTEC-
                                                          92-21, Apr. 2, 1992.)

Navy T-45A Aircraft     Major       Yes               33  One year into LRIP,
                                                          OT&E found that the T-
                                                          45A was not effective
                                                          in a carrier
                                                          environment and was
                                                          not operationally
                                                          suitable because of
                                                          safety deficiencies.
                                                          Subsequent major
                                                          design changes have
                                                          included a new engine,
                                                          new wings, and a
                                                          modified rudder. (GAO/
                                                          NSIAD-91-46, Dec. 14,
                                                          1990.)

Navy Pioneer Unmanned   Nonmajor    Yes            n/a\d  The Navy procured and
Aerial Vehicle                                            deployed Pioneer as a
                                                          nondevelopmental item
                                                          and without testing
                                                          it. Numerous problems
                                                          ensued, including
                                                          engine failures,
                                                          landing difficulties,
                                                          and a cumbersome
                                                          recovery system. Many
                                                          modifications were
                                                          required to bring
                                                          Pioneer up to a
                                                          minimum essential
                                                          level of performance.

Army Family of Medium   Major       To be            4\e  Before the Army did
Tactical Vehicles                   determined            any OT&E, a multiyear
                                                          production contract
                                                          was awarded for up to
                                                          10,843 trucks.
                                                          Subsequent OT&E was
                                                          suspended because the
                                                          vehicles were found to
                                                          be unreliable and not
                                                          operationally
                                                          effective. However,
                                                          production continues.
                                                          (GAO/NSIAD-93-232,
                                                          Aug. 5, 1993.)

Army Palletized Load    Major       Yes               29  OT&E showed the system
System/Family of Heavy                                    to be not
Tactical Vehicles                                         operationally
                                                          suitable. Despite the
                                                          need for design
                                                          modifications to
                                                          correct reliability
                                                          and maintainability
                                                          problems, full-rate
                                                          production was
                                                          approved.
--------------------------------------------------------------------------------
\a Proceeded beyond LRIP before OT&E was conducted. 

\b Because of the quantity already procured in LRIP and the lack of
OT&E to date, additional units are likely to be procured in LRIP. 

\c Proceeded beyond LRIP beyond OT&E was conducted. 

\d Production was not separated into LRIP and full-rate production
phases. 

\e At least 3,800 trucks are expected to be produced in LRIP, or
about



                                    Table 2.2
                     
                      Examples of Systems That Entered LRIP
                     Despite Problems Identified During Early
                               Operational Testing

                                    Inadequate
                                    system       Percent
                        Program     deployed    procured
System                  category    to field     in LRIP  Comments
----------------------  ----------  ----------  --------  ----------------------
Army AN/AVR-2, AN/      Nonmajor    Yes             53\a  Even though over 900
AVR-2A Laser Warning                                      units have been
System                                                    procured, neither the
                                                          redesigned AN/AVR-2
                                                          nor the further
                                                          redesigned AN/AVR-2A
                                                          had completed OT&E.
                                                          (GAO/NSIAD-93-14,
                                                          Jan. 25, 1993.)

Army AN/APR-39A(V)1     Nonmajor    Yes             43\b  In 1989, the Army's
Radar Warning Receiver                                    independent testers
                                                          recommended curtailing
                                                          production because of
                                                          poor results. However,
                                                          production continues.

Navy Airborne Self-     Major       N/A            100\c  This $2 billion
Protection Jammer                                         program was terminated
                                                          after procurement of
                                                          95 systems because of
                                                          failure to pass
                                                          required OT&E. (GAO/
                                                          NSIAD-92-103, Mar. 23,
                                                          1992.)

Navy AN/SLQ-32          Major       Yes               73  Numerous modifications
Shipboard Electronic                                      have been made at a
Countermeasures System                                    cost of over $300
                                                          million, but the AN/
                                                          SLQ-32's effectiveness
                                                          remains questionable.
                                                          (GAO/NSIAD-93-272,
                                                          Aug. 19, 1993.)

Navy MK-50 Torpedo      Major       Yes            100\d  OT&E, conducted after
                                                          LRIP began, was halted
                                                          until technical
                                                          problems were
                                                          addressed. However,
                                                          later OT&E continued
                                                          to show performance
                                                          problems. Due to
                                                          drastic reductions in
                                                          total quantities, all
                                                          1,073 torpedoes were
                                                          procured in LRIP.

Navy MH-53E Helicopter  Major       Yes               38  OT&E conducted after
                                                          LRIP started revealed
                                                          several major
                                                          mechanical
                                                          deficiencies. Later,
                                                          OT&E found the MH-53E
                                                          to be marginally
                                                          operationally
                                                          effective and not
                                                          operationally
                                                          suitable.

Navy AN/ALR-67(V)2      Nonmajor    Yes               67  OT&E conducted after
Radar Warning Receiver                                    the start of LRIP
                                                          pointed out several
                                                          serious problems.
                                                          However, by that time,
                                                          the Navy had bought
                                                          and deployed over 700
                                                          systems at a cost of
                                                          $467 million. The Navy
                                                          later spent
                                                          $96 million to correct
                                                          the deficiencies and
                                                          to upgrade the
                                                          systems. However,
                                                          production of the
                                                          upgrades started
                                                          before OT&E was
                                                          conducted, and many
                                                          units were placed in
                                                          storage pending
                                                          completion of testing,
                                                          as discussed in a
                                                          classified GAO report.

Air Force AN/ALQ-131    Nonmajor    Yes               78  The Block I jammer was
Block 1 Jammer                                            produced and deployed
                                                          although it failed to
                                                          pass various
                                                          reliability and
                                                          maintainability tests.
                                                          As a result, many of
                                                          the jammers required
                                                          major part
                                                          replacements and
                                                          technical adjustments,
                                                          as discussed in a
                                                          classified GAO report.
--------------------------------------------------------------------------------
\a Quantity includes some AN/AVR-2 full-rate production units. 

\b Production was not separated into LRIP and full-rate production
phases.  However, 2,000 units (or 43 percent of the total program
quantity of 4,604 units) were on contract at the time of the
independent tester's recommendation. 

\c Program terminated while in LRIP. 

\d Due to drastic reductions in total procurement quantities of the
MK 50, the Navy decided not to get approval for full-rate production. 


   MAJOR PRODUCTION COMMITMENTS
   ARE MADE DURING LRIP
---------------------------------------------------------- Chapter 2:2

Programs that enter production prematurely often require more time
and resources than originally planned to correct problems and to meet
the requirements for full-rate production.  LRIP is often continued,
despite the evidence of technical problems, well beyond that needed
to provide test articles and to establish an initial production
capability.  As a result, major production commitments are often made
during LRIP. 

In the conference report for the LRIP legislation, the conferees
stated that they did not intend to authorize the continuance of LRIP
on an indefinite basis.  Nevertheless, the existing LRIP legislation
does not include any specific principles or guidelines on when and
how programs should begin LRIP, on the type and amount of testing to
be done before LRIP, on how much LRIP can or should be done, or under
what circumstances LRIP should be curtailed or stopped.  Instead, the
emphasis has been placed almost entirely on the full-rate production
decision, at which point the law requires, among other things, that a
report be provided on the adequacy of the testing conducted and an
assessment be made of the system's operational effectiveness and
suitability.  Although programs are delayed getting approval for
full-rate production, LRIP is rarely stopped or slowed significantly. 
As a result, the decision to start LRIP, in many cases, is also the
de-facto full-rate production decision. 


   DOD FAILS TO EMPHASIZE EARLY
   OT&E
---------------------------------------------------------- Chapter 2:3

DOD's written policies provide that acquisition strategies be
event-driven and link major contractual commitments and milestone
decisions to demonstrated accomplishments in development, test, and
initial production.  However, DOD policies state that a primary goal
in developing an acquisition strategy shall be to minimize the time
and cost of satisfying a need consistent with common sense and sound
business practices. 

In addition, DOD's policies state, but without detailed requirements,
that OT&E should be conducted throughout the acquisition process. 
However, while DOD is statutorily required to conduct OT&E before
full-rate production is approved, DOD's policies permit LRIP to begin
before any OT&E is conducted.  The point at which LRIP begins is not
a required milestone under DOD policy. 

As a result, for many major defense acquisition programs, the
services do not plan to conduct any OT&E prior to the start of LRIP. 
It has been and continues to be the exception, rather than the rule,
for programs to include OT&E before LRIP starts.  In some instances,
the services plan to start LRIP even though they plan to use
developmental or prototype units for their initial OT&E, not LRIP
units.  Although not required by written DOD or Navy policy, the Navy
now performs a limited phase of OT&E before LRIP to prepare for later
phases of OT&E on some of its programs.  However, these programs are
not required to meet specific testing-related criteria before
entering LRIP.  As shown in table 2.2, even when some OT&E was
conducted prior to the start-up of production, identified problems
were not verified as corrected, and significant performance problems
emerged later in the program. 

Over the past several years, DOD has stated that it planned to
reemphasize the need for OT&E as early as possible in the acquisition
process.  However, we have not detected any reemphasis on early OT&E,
and DOD's 1991 revision of its key acquisition directives did not
address this issue.  DOD acquisition and testing officials concede
that there has not been any major reemphasis on early OT&E.  In fact,
DOD has recently supported legislative proposals that would reduce
the current overall requirements to conduct OT&E.\1

DOD has recognized that reducing the amount of production prior to
completing development provides for greater design maturity, which
increases the likelihood of meeting system requirements and avoiding
retrofit costs.  In commenting on our 1992 report,\2 DOD officials
said they were lessening the amount of concurrent development and
production in weapon programs due to the end of the Cold War.  In
1992, the Under Secretary of Defense for Acquisition also stated that
the need to replace existing weapon systems in order to maintain a
significant technological advantage was no longer as urgent. 
However, acquisition strategies of many current programs do not
reflect these positions.  DOD's acquisition practices continue to
stress the importance of minimizing the time to deploy new or
improved weapon systems. 

Highly concurrent acquisition strategies continue to be featured in
many current major and nonmajor programs,\3 with little, if any, OT&E
expected until well after the start of production and a significant
commitment is made to the procurement of the system.  Our analysis of
the current selected acquisition reports shows that many programs
continue to postpone initial OT&E until well after the start of
production. 

LRIP is expected to be approved in February 1996 for the Army's
Secure Mobile Anti-Jam Reliable Tactical Terminal.  Initial OT&E will
not be completed until July 1998, by which time a total of 125 units,
or 3 years of LRIP, is planned to be approved out of a total program
quantity of 367 units. 

The LRIP decision for the Air Force's F-22 aircraft program is
expected in June 1998, and initial OT&E is to be conducted from March
to November 2001.  Thus, 1 year of preproduction verification and 4
years of LRIP--80 aircraft out of a total quantity of 442 units--are
planned to be approved before completion of OT&E. 

The Navy plans to procure 106 of the 630 planned Multifunctional
Information Distribution Systems before OT&E is completed in December
2000 and a full-rate production decision is made in June 2001.  In
addition, 42 prototype systems are to be built as part of the system
development effort. 

These programs feature major commitments to LRIP before development
is completed and before any OT&E is completed, even though
developmental prototypes are expected to be available for testing in
these programs.  Accordingly, a substantial and frequently
irreversible commitment to production will have been made before the
results of independent testing are available to decisionmakers. 


--------------------
\1 Acquisition Reform:  Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar.  22,
1994). 

\2 Weapons Acquisition:  A Rare Opportunity for Lasting Change
(GAO/NSIAD-93-15, Dec.  1992). 

\3 A program with high levels of concurrency typically proceeds into
production before a significant amount of initial OT&E is conducted. 


   CONTROLS RECOMMENDED OVER THE
   START AND CONTINUATION OF LRIP
---------------------------------------------------------- Chapter 2:4

In its 1993 report,\4 the DOD-IG found that major defense acquisition
programs were entering LRIP without meeting development, testing, and
production readiness prerequisites.  As a result, the DOD-IG
concluded that DOD incurred excessive program risk of overcommitment
to production of systems without obtaining assurance that the design
is stable, potentially operationally acceptable, and capable of being
produced efficiently.  Among other things, the DOD-IG recommended
that DOD (1) provide guidance on the specific minimum required
program accomplishments for entry into and continuation of LRIP and
(2) require that program-specific exit criteria be established for
entry into and continuation of LRIP.  DOD is currently considering
what, if any, actions will be taken in light of the DOD-IG's
recommendations. 


--------------------
\4 Low-Rate Initial Production in Major Defense Acquisition Programs
(DOD Inspector General Report No.  94-014, Nov.  9, 1993). 


CONCLUSIONS AND RECOMMENDATIONS
============================================================ Chapter 3


   CONCLUSIONS
---------------------------------------------------------- Chapter 3:1

The decision to begin LRIP should be given much more attention
because decisionmakers find it very difficult to stop or slow down
programs once they are in production.  Given the cost risks involved
and DOD's inability or unwillingness to curtail production after it
starts, we agree with the DOD-IG that controls are urgently needed
over the start and continuation of LRIP. 

A key criterion for all programs beginning LRIP should be the
completion of a phase of independent testing in an operational
environment.  During such testing, some problems should be expected. 
However, enough realistic testing should be conducted for the
services' independent testing agencies and/or DOT&E to be able to
certify to the decision authority that (1) the system's developmental
testing is essentially complete and the basic results have been
validated in an operational environment, (2) the system has clearly
shown that it can meet the key parameters among its minimum
acceptable performance requirements,\1 (3) the system has clearly
demonstrated the potential to fully meet all of its minimum
acceptable requirements for performance and suitability without major
or costly design changes, and (4) the system should be able to
readily complete its remaining OT&E in time to support the planned
full-rate production decision.\2

Comprehensive testing of a system's operational suitability features,
such as supportability, may not be possible during early independent
testing.  However, the testing should be sufficient to reveal major
suitability problems. 

Conducting OT&E before LRIP will not, by itself, result in a better
weapon system, but it is the best means available to guard against
the premature start of production.  Decisionmakers need verifiable
information on system design maturity and where corrective actions
are needed before production start-up.  Every effort should be made
to correct problems in development, not in production, because early
fixes are less expensive, easier to implement, and less disruptive. 
In today's national security environment, there should be very few
cases in which an urgent need dictates that DOD start production
without assurance that the system will work as intended. 

We realize that, for some programs, a significant effort (personnel
and facilities) may be needed to produce one or more prototypes for a
phase of early OT&E.  These programs would typically involve inherent
fabrication complexity, small procurement quantities, high unit cost,
and long unit production periods.  To suspend that type of effort
while OT&E is underway could be costly and disruptive. 
Alternatively, key subsystems should be independently tested on
surrogate platforms before production.  Once underway, production
should be limited until acceptable OT&E results are obtained on the
entire system. 

We believe that LRIP should be used to focus on (1) addressing
producibility and product quality issues; (2) producing just enough
systems to support initial OT&E, to prove out the production process,
and to sustain the production line; and (3) testing those systems and
correcting any deficiencies.  A limit on the quantity that can be
produced under LRIP would provide an opportunity to correct problems
that are identified during initial OT&E, without incurring the risk
of overproducing under the LRIP phase. 


--------------------
\1 Minimum acceptable requirements are the values for the particular
parameters that are required to provide a system capability that will
satisfy the validated mission need.  These requirements are also
known as thresholds and are established in the Operational
Requirements Document at each milestone.  Key parameters are those
for which the decision authority would require a reevaluation of
alternative concepts or design approaches if the thresholds are not
met. 

\2 This certification should be made by DOT&E for all major defense
acquisition programs and by the services' independent testing
agencies for all other systems. 


   RECOMMENDATIONS
---------------------------------------------------------- Chapter 3:2

We recommend that the Secretary of Defense revise DOD's acquisition
policies in the following ways: 

Require that, before entry into LRIP, programs (with the exception of
ships, satellites, and those other programs that involve inherent
fabrication complexity, small procurement quantities, high unit
costs, and long unit production periods) plan, buy prototypes for,
and conduct enough realistic testing for the service's independent
testing agency and/or DOT&E to be able to certify to the decision
authority that (1) the system's developmental testing is essentially
complete and the basic results of that testing have been validated in
an operational environment; (2) the system has clearly shown that it
can meet the key parameters among its minimum acceptable performance
requirements; (3) the system has clearly demonstrated the potential
to fully meet all of its minimum acceptable requirements for
performance and suitability without major or costly design changes;
and (4) the system should be able to readily complete its remaining
OT&E in time to support the planned full-rate production decision. 

Require that those programs excluded from the requirement to test
prototypes instead test all key subsystems in an operational
environment before entry into LRIP. 

Adopt the recommendations made by the DOD-IG regarding controls over
the start and continuation of LRIP such as (1) providing guidance on
the specific minimum required program accomplishments for entry into
and continuation of LRIP and (2) requiring that program-specific exit
criteria be established for entry into and continuation of LRIP. 

We also recommend that the Secretary of Defense work with the service
secretaries to ensure that these policies are implemented for the
acquisition of both major and nonmajor systems. 


   MATTERS FOR CONGRESSIONAL
   CONSIDERATION
---------------------------------------------------------- Chapter 3:3

The legislation defining LRIP has not been effective in accomplishing
its purpose, which was to limit the commitment to major production
quantities pending satisfactory completion of OT&E.  Therefore, we
recommend that the Congress legislatively mandate (1) that certain
OT&E requirements be met before LRIP may start and (2) specific
limits on the number of units allowed to be produced during LRIP. 
Specifically, the Congress may wish to require that all defense
acquisition programs (major and nonmajor) conduct enough realistic
testing on the entire system or key subsystems to ensure that its key
performance parameters are met before LRIP is permitted to start.  In
addition, the Congress may wish to (1) specify a percentage (10
percent, for example) of a system's total procurement beyond which a
program may not proceed during LRIP and/or (2) amend 10 U.S.C.  2400
(by deleting subsection (b)(3)) to preclude the use of LRIP authority
to ramp-up the production rate prior to the successful completion of
OT&E. 




(See figure in printed edition.)Appendix I
COMMENTS FROM THE DEPARTMENT OF
DEFENSE
============================================================ Chapter 3

See comment 1. 

See comment 2. 

See comment 3. 

See comment 4. 



(See figure in printed edition.)

See comment 3. 

See comment 5. 

See comment 6. 

See comment 7. 

See comment 8. 



(See figure in printed edition.)

See comment 9. 

See comment 10. 



(See figure in printed edition.)

See comment 11. 

See comment 12. 

See comment 13. 

See comment 14. 



(See figure in printed edition.)

See comment 15. 

See comment 8. 

See comment 7. 



(See figure in printed edition.)

See comments 5 and 6. 

See comment 6. 



(See figure in printed edition.)

See comments 6 and 16. 



   GAO COMMENTS
---------------------------------------------------------- Chapter 3:4

1.  The total procurement quantity of C-17s was reduced for several
reasons, including affordability, but we strongly dispute the
Department of Defense's (DOD) assertion here that the reduction was
as a result of problems encountered in testing. 

2.  After careful review, we have concluded that the percentages in
table 2.1 are correct as stated and that no changes are needed.  Our
report shows that 73 percent of the AN/SLQ-32 program quantity was
bought during LRIP, not 100 percent as stated by DOD.  DOD may have
confused the AN/SLQ-32 with the AN/ALQ-135 Quick Reaction Capability
Jammer, of which 100 percent of the total program quantity was
procured during low-rate initial production (LRIP).  Also, on the
Pioneer system, the Navy made a major commitment to
procurement--whether it was under special circumstances or
LRIP--without any assurance of the system's adequacy.  Nevertheless,
in our view, the cited systems illustrate one of the more severe
adverse impacts of beginning LRIP prematurely, that is, the unplanned
termination of a program after procurement of a few costly but
unsatisfactory systems.  Our proposed solution is to obtain at least
some confidence in the system's adequacy in an operational
environment before any commitment is made to production. 

3.  We have clarified the description of our review methodology and
have clearly pointed out that the examples used in the report are
only a sampling of what we consider to be the widespread problem of
premature commitment to production.  Our conclusions and
recommendations are based on our large body of weapon system reviews
over the past decade or so plus additional analyses of the current
selected acquisition report systems and recent Defense Acquisition
Board decisions.  The DOD Inspector General (DOD-IG) has arrived at
similar conclusions based on its independent reviews.  The problems
discussed in this report are not new and continue to regularly occur
in small and large programs and programs from all the services.  The
recommendations are consistent with our previous recommendations as
well as those made by several "blue ribbon" panels on defense
acquisition matters, including the Packard Commission.  While DOD's
reluctance to adequately address the problem is not new, we are very
concerned that DOD now appears to question the significance of the
problem. 

4.  The statements in our report that (a) DOD acquisition policies
permit LRIP to start without doing any operational, test and
evaluation (OT&E) and (b) there are no specific principles or
guidelines on when or how programs should begin LRIP are accurate and
are not sweeping generalizations.  We agree that the regulations do
mention LRIP in many places; however, the guidance is neither clear
nor specific, and implementation has not been effective in
controlling the commitment to production. 

5.  The current legislation contains no specific controls on the
start or continuation of LRIP; 10 U.S.C.  2400 defines the purposes
of LRIP but does not specify when and under what circumstances it may
begin or continue. 

6.  Our position would not cause any production stretchouts or
production stoppages and, for systems that can readily demonstrate
their key performance parameters, only a brief delay in starting
production.  In fact, production stretchouts and/or stoppages are
usually the result of problems identified during operational testing
performed after, rather than before, the start of production.  We
believe that it is reasonable to delay the start of production to
obtain assurance that the system performs as intended, thereby
avoiding the potential for stretchouts or delays when subsequent
operational testing identifies system shortcomings. 

7.  Although DOD may want to deemphasize realistic operational
testing during the acquisition of defense systems, it has provided no
factual basis to state that test and evaluation is a cause or even a
contributing factor to the continuing problems in the acquisition of
defense systems.  Streamlining of the acquisition process should not
be used as an excuse for buying defective weapon systems. 

8.  Enclosures 2 and 3 are not reproduced here because they are
lengthy and address many issues not addressed in this report. 

9.  Developmental test agencies are elements of the DOD acquisition
community and, in that sense, are not independent.  In fact, their
lack of independence was a key factor in creating the office of the
Director, Operational Test and Evaluation (DOT&E).  While the
independence of testers is a key point, we believe that early testing
in a realistic operational environment is crucial to the
identification of weaknesses in the system.  Developmental testing is
important in itself, but it does not expose the system to the rigors
of the operational environment. 

10.  Operational assessments are not typically based on the results
of realistic field testing and, therefore, do not provide the
quantity and quality of input to decisionmakers that we believe is
necessary. 

11.  We believe the cited DOD policy is flawed in that a system could
be approved to begin LRIP and yet not be approved as ready for
operational testing.  In our view, if a system is not ready to do
well in operational testing, it is not ready to begin LRIP.  Our
recommendation would require operational testing prior to LRIP, which
could add some cost and time up to that point.  However, we believe
that additional investment would be well spent in terms of improved
performance and reliability when the system is deployed and could
help avoid the costly modifications and retrofits that often result
from DOD committing to production of systems without any knowledge
about their operational performance. 

12.  It is important to have a milestone review at the LRIP decision
point because the decisionmakers have more flexibility to change the
program at that point.  As we state in this report, the decision to
start LRIP, in many cases, is also the de-facto full-rate production
decision. 

13.  We recognize that our recommendations are at odds with current
acquisition practices for many programs.  However, DOD should
recognize that, in today's environment, highly concurrent acquisition
strategies are no longer necessary and that it is technically
feasible for prototypes to be sufficiently mature to be operationally
tested and to meet their key parameters before LRIP.  We believe
that, in most cases, production representative articles can be
assembled and realistic operational testing conducted before LRIP
begins.  If a system is not ready for operational testing, it should
not be ready for production. 

14.  This recommendation is aimed at those very large programs for
which it may not be feasible or cost-effective to produce prototypes
without starting LRIP.  While we recognize the potential for
integration and interface problems among subsystems, we question the
need to begin production of the primary system before sufficient
confidence is available that the subsystems perform adequately and
are reliable. 

15.  In response to the DOD-IG report, DOD conceded that more
attention may be needed to the LRIP decision and that some sort of
controls may be needed over the start and continuation of LRIP.  We
believe that, to be effective, such controls need to be specific
rather than a generic call for more emphasis. 

16.  Our positions would cause the stetchout of only those production
programs in which the system was not able to complete its required
operational testing.  In numerous cases, LRIP was continued
indefinitely because of the inability to complete operational
testing, resulting in costly retrofits of produced units and/or the
fielding of unsatisfactory systems. 


MAJOR CONTRIBUTORS TO THIS REPORT
========================================================== Appendix II


   NATIONAL SECURITY AND
   INTERNATIONAL AFFAIRS DIVISION,
   WASHINGTON, D.C. 
-------------------------------------------------------- Appendix II:1

William Graveline
David Hand
Stacy Edwards
Jose Ramos


   ATLANTA REGIONAL OFFICE
-------------------------------------------------------- Appendix II:2

Jack Guin



RELATED GAO PRODUCTS
============================================================ Chapter 1

Military Airlift:  The C-17 Program Update and Proposed Settlement
(GAO/T-NSIAD-94-166, Apr.  19, 1994). 

Acquisition Reform:  Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar.  22,
1994). 

Army Acquisition:  Information on the Status and Performance of the
Javelin Antitank Weapon (GAO/NSIAD-94-122BR, Mar.  9, 1994). 

Battlefield Automation:  Premature Acquisition of the Army's Combat
Service Support Control System (GAO/NSIAD-94-51, Feb.  4, 1994). 

Navy Aviation:  V-22 Development--Schedule Extended, Performance
Reduced, and Costs Increased (GAO/NSIAD-94-44, Jan.  13, 1994). 

Strategic Bomber:  Issues Related to the B-1B's Availability and
Ability to Perform Conventional Missions (GAO/NSIAD-94-81, Jan.  10,
1994). 

Unmanned Aerial Vehicles:  Performance of Short-Range System Still in
Question (GAO/NSIAD-94-65, Dec.  15, 1993). 

Army Acquisition:  Problems With the Sense and Destroy Armor Munition
(GAO/NSIAD-94-59, Nov.  23, 1993). 

Missile Development:  TSSAM Production Should Not Be Started as
Planned (GAO/NSIAD-94-52, Oct.  8, 1993). 

Test and Evaluation:  DOD Has Been Slow in Improving Testing of
Software-Intensive Systems (GAO/NSIAD-93-1989; Sept.  29, 1993). 

Electronic Warfare:  Inadequate Testing Led to Faulty SLQ-32s on
Ships (GAO/NSIAD-93-272, Aug.  19, 1993). 

Army Acquisition:  Medium Truck Program Is Not Practical and Needs
Reassessment (GAO/NSIAD-93-232, Aug.  5, 1993). 

Army Acquisition:  More Testing Needed to Solve Heavy Equipment
Transporter System Problems (GAO/NSIAD-93-228, July 16, 1993). 

Battlefield Automation:  More Testing and Analysis Needed Before
Production of Air Defense Radar (GAO/NSIAD-93-175, July 30, 1993). 

Tactical Intelligence:  Joint STARS Needs Current Cost and
Operational Effectiveness Analysis (GAO/NSIAD-93-117, Apr.  28,
1993). 

Electronic Warfare:  Laser Warning System Production Should Be
Limited (GAO/NSIAD-93-14, Jan.  25, 1993). 

Weapons Acquisition:  A Rare Opportunity for Lasting Change
(GAO/NSIAD-93-15, Dec.  1992). 

Military Communications:  Joint Tactical Information Distribution
System Issues (GAO/NSIAD-93-16, Nov.  12, 1992). 

Battlefield Automation:  Planned Production Decision for Army Control
System Is Premature (GAO/NSIAD-92-151, Aug.  10, 1992). 

Strategic Bombers:  Need to Redefine Requirements for B-1B Defensive
Avionics System (GAO/NSIAD-92-272, July 17, 1992). 

Comanche Helicopter:  Program Needs Reassessment Due to Increased
Unit Cost and Other Factors (GAO/NSIAD-92-204, May 27, 1992). 

Embedded Computer Systems:  Software Development Problems Delay the
Army's Fire Direction Data Manager (GAO/IMTEC-92-32, May 11, 1992). 

Embedded Computer Systems:  Significant Software Problems on C-17
Must Be Addressed (GAO/IMTEC-92-48, May 7, 1992). 

Strategic Missiles:  ACM Program, Opportunity for Additional Savings
(GAO/NSIAD-92-154, May 6, 1992). 

Operation Desert Storm:  Apache Helicopter Was Considered Effective
in Combat, but Reliability Problems Persist (GAO/NSIAD-92-146, Apr. 
20, 1992). 

Electronic Warfare:  Faulty Test Equipment Impairs Readiness of Army
Helicopters (GAO/NSIAD-92-128, Apr.  17, 1992). 

Embedded Computer Systems:  F-14D Aircraft Software Is Not Reliable
(GAO/IMTEC-92-21, Apr.  2, 1992). 

Strategic Defense Initiative:  Estimates of Brilliant Pebbles'
Effectiveness Are Based on Many Unproven Assumptions
(GAO/NSIAD-92-91, Mar.  27, 1992). 

Electronic Warfare:  Established Criteria Not Met for Airborne
Self-Protection Jammer Production (GAO/NSIAD-92-103, Mar.  23, 1992). 

Army's M109 Howitzer:  Required Testing Should Be Completed Before
Full-Rate Production (NSIAD-92-44, Jan.  23, 1992). 

Longbow Apache Helicopter:  Key Factors Used to Measure Progress in
Development Need to Be Changed (NSIAD-92-43, Nov.  21, 1991). 

Submarine Combat System:  BSY-2 Development Risks Must Be Addressed
and Production Schedule Reassessed (GAO/IMTEC-91-30, Aug.  22, 1991). 

Munitions Procurement:  Resolve Questions Before Proceeding With
Sensor Fuzed Weapon Production (GAO/NSIAD-91-235, Aug.  16, 1991). 

Battlefield Automation:  Army Tactical Command and Control System
Development Problems (GAO/NSIAD-91-172, July 31, 1991). 

Defense Management:  Stronger Oversight of Joint Service Imagery
Processing System Needed (GAO/NSIAD-91-164, July 26, 1991). 

The B-2 Program:  Procurement Decisions Should Be Based on
Demonstrated Performance (GAO/T-NSIAD-91-45, July 17, 1991). 

Missile Procurement:  AMRAAM's Reliability Is Improving, but
Production Challenges Remain (GAO/NSIAD-91-209, June 20, 1991). 

Battlefield Automation:  Army Needs to Reevaluate Air Defense Radar
Acquisition Programs (GAO/NSIAD-91-91, May 3, 1991). 

Global Positioning System:  Production Should Be Limited Until
Receiver Reliability Problems Are Resolved (GAO/NSIAD-91-74, Mar. 
20, 1991). 

Electronic Warfare:  Early Production of Tacit Rainbow Missile Not
Warranted (GAO/NSIAD-91-71, Mar.  8, 1991). 

Letter Report to Senator Roth on F-117A Acquisition Management
Practices (B-238891, Jan.  11, 1991). 

Antisubmarine Warfare:  Tactical Surveillance Sonobuoy and Related
Software Need to Be Tested Together (GAO/NSIAD-91-41, Jan.  9, 1991). 

T-45 Training System:  Navy Should Reduce Risks Before Procuring More
Aircraft (GAO/NSIAD-91-46, Dec.  14, 1990). 

Army Acquisition:  Air Defense Antitank System Did Not Meet
Operational Test Objectives (GAO/NSIAD-91-51, Dec.  10, 1990). 

