Test and Evaluation: Impact of DOD's Office of the Director of
Operational Test and Evaluation (Letter Report, 10/24/97,
GAO/NSIAD-98-22).

Pursuant to a congressional request, GAO reviewed the Department of
Defense's (DOD) Office of the Director of Operational Test and
Evaluation's (DOT&E) operations and organizational structure for
overseeing operational testing, focusing on: (1) DOT&E's efforts and
their impact on the quality of operational testing and evaluation in
DOD; and (2) the strengths and weaknesses of the current organizational
framework in DOD for operational testing.

GAO noted that: (1) GAO's review of 13 case studies indicated that DOT&E
oversight of operational testing and evaluation increased the
probability that testing would be more realistic and more thorough; (2)
specifically, DOT&E was influential in advocating increasing the
reliability of the observed performance and reducing the risk of
unknowns through more thorough testing, conducting more realistic
testing, enhancing data collection and analysis, reporting independent
findings, and recommending follow-on operational test and evaluation
when suitability or effectiveness was not fully demonstrated prior to
initiating full-rate production; (3) the independence of DOT&E--and its
resulting authority to report directly to Congress--is the foundation of
its effectiveness; (4) that independence, along with its legislative
mandate, provides sufficient freedom and authority to exercise effective
oversight of the operational testing and evaluation of new systems
before a decision is made to begin full-rate production; (5) DOT&E can
reduce the risk that systems are not adequately tested prior to the
full-rate production decision but DOT&E cannot ensure that: (a) only
systems whose operational effectiveness and suitability have been
demonstrated through operational testing will proceed to the full-rate
production decision; or (b) new fielded systems will accomplish their
missions as intended or that the fielded systems are safe, survivable,
and effective; (6) DOT&E management must balance its oversight
responsibilities for operational testing with the broader acquisition
priorities of program managers and service test agencies; (7) though
supportive of DOT&E's mission and independence, program and service
representatives frequently considered the time, expense, and resources
expended to accommodate DOT&E concerns to be ill-advised; (8) several
current trends may challenge DOT&E's ability to manage its workload and
its ability to impact operational test and evaluation: (a) service
challenges to DOT&E's authority to require and oversee follow-on
operational testing and evaluation; (b) a decline in resources available
for oversight; (c) an expansion of DOT&E involvement in activities other
than oversight of major acquisition programs; (d) participation of DOT&E
in the acquisition process as a member of working-level integrated
product teams; and (e) greater integration of developmental and
operational testing; and (9) these trends make it imperative that DOT&E
prioritize its workload to achieve a balance between the oversight of
major defense acquisition programs and other initiatives important to
the quality of operational test and evaluation.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-98-22
     TITLE:  Test and Evaluation: Impact of DOD's Office of the Director 
             of Operational Test and Evaluation
      DATE:  10/24/97
   SUBJECT:  Testing
             Weapons systems
             Product performance evaluation
             Military systems analysis
             Cost effectiveness analysis
             Interagency relations
             Prioritizing
             Military cost control
             Defense procurement

             
******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  Delineations within the text indicating chapter **
** titles, headings, and bullets are preserved.  Major          **
** divisions and subdivisions of the text, such as Chapters,    **
** Sections, and Appendixes, are identified by double and       **
** single lines.  The numbers on the right end of these lines   **
** indicate the position of each of the subsections in the      **
** document outline.  These numbers do NOT correspond with the  **
** page numbers of the printed product.                         **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                                            **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************


Cover
================================================================ COVER


Report to the Honorable
William V.  Roth and the Honorable
Charles E.  Grassley, U.S.  Senate

October 1997

TEST AND EVALUATION - IMPACT OF
DOD'S OFFICE OF THE DIRECTOR OF
OPERATIONAL TEST AND EVALUATION

GAO/NSIAD-98-22

DOT&E Impact

(973444)


Abbreviations
=============================================================== ABBREV

  AWACS - airborne warning and control system
  DOD - Department of Defense
  DODIG - Department of Defense Inspector General
  DOT&E - Office of the Director of Operational Test and Evaluation
  IDA - Institute for Defense Analyses
  JointSTARS - Joint Surveillance Target Attack Radar System
  LCAC - landing craft air cushion
  LRIP - low-rate initial production
  OSD - Office of the Secretary of Defense
  RSIP - radar system improvement program
  USD(A&T) - Under Secretary of Defense (Acquisition and Technology)

Letter
=============================================================== LETTER


B-276799

October 24, 1997

The Honorable William V.  Roth
The Honorable Charles E.  Grassley
United States Senate

The Department of Defense (DOD) has proposed that the practices and
policies of the Office of the Director of Operational Test and
Evaluation (DOT&E) be modified to reduce the time and cost of
developing and fielding new weapon systems.  To help focus
deliberations on DOD's proposal, you asked us to review DOT&E's
operations and organizational structure for overseeing operational
testing.  Specifically, you asked us to assess (1) DOT&E's efforts
and their impact on the quality of operational testing and
evaluation\1 in DOD and (2) the strengths and weaknesses of the
current organizational framework in DOD for operational testing.  As
part of our review, we conducted 13 case studies of the testing of
individual weapon systems.  (Our scope and methodology are described
in app.  I, and brief descriptions of the 13 weapon systems are
provided in app.  II.)


--------------------
\1 The term "operational test and evaluation" means (1) the field
test, under realistic conditions, of any item or key component of a
weapon system, equipment, or munition for the purpose of determining
the effectiveness and suitability of the weapon, equipment, or
munition for use in combat by typical military users and (2) the
evaluation of the results of the test. 


   BACKGROUND
------------------------------------------------------------ Letter :1

In 1983, Congress established DOT&E to coordinate, monitor, and
evaluate operational testing of major weapon systems.\2 As part of
the Office of the Secretary of Defense (OSD), DOT&E is separate from
the acquisition community that conducts developmental and operational
testing and therefore is in a position to provide the Secretary and
Congress with an independent view.  Congress created DOT&E in
response to reports of conflicts of interest in the acquisition
community's oversight of operational testing leading to inadequate
testing of operational suitability\3 and effectiveness\4 and the
fielding of new systems that performed poorly.  (DOD's system
acquisition process is described in app.  III.)

By law, DOT&E serves as the principal adviser on operational test and
evaluation in DOD and bears several key responsibilities, including

  -- monitoring and reviewing all operational test and evaluation in
     DOD,

  -- reporting to the Secretary of Defense and congressional
     committees whether the tests and evaluations of weapon systems
     were adequate and whether the results confirmed that the system
     is operationally suitable and effective for combat before a
     decision is made to proceed to full-rate production, and

  -- submitting to the Secretary of Defense and congressional
     decisionmakers an annual report summarizing operational test and
     evaluation activities during the preceding fiscal year. 

In 1993, DOD's advisory panel on streamlining and codifying
acquisition laws\5 concluded that DOT&E was impeding the goals of
acquisition reform by (1) promoting unnecessary oversight, (2)
requiring excessive reporting detail, (3) inhibiting the services'
discretion in testing, and (4) limiting participation of system
contractors in operational tests where such involvement is deemed
necessary by the services.  The following year, DOD proposed
legislative changes that would have reduced the scope and authority
of DOT&E.  In testimony, we opposed these changes because they were
directed at perceived rather than documented problems and would
undermine a key management control over the acquisition
process--independent oversight of operational test and evaluation.\6

Although the legislative proposals were not adopted, in 1995 the
Secretary of Defense implemented several operational test and
evaluation initiatives in the Department to (1) involve operational
testers earlier in the acquisition process, (2) use models and
simulations effectively, (3) combine tests where possible, and (4)
combine tests and training.  The goals of these initiatives included
saving time and money by identifying and addressing testing issues
earlier in the acquisition process; merging or closely coordinating
historically distinct phases, such as developmental and operational
testing to avoid duplication; and using existing technologies and
training exercises to create realistic and affordable test
conditions. 


--------------------
\2 P.L.  98-94 sec.  1211(a)(1), 97 Stat.  684.  DOT&E's legislation
is now codified at 10 U.S.C.  139. 

\3 DOD defines "operationally suitable" as the degree to which a
system can be placed satisfactorily in field use, with consideration
given to such factors as availability, compatibility,
transportability, interoperability, reliability, wartime usage rates,
maintainability, safety, and supportability. 

\4 DOD defines "operationally effective" as the overall degree of
mission accomplishment of a system when used by representative
personnel in the environment planned or expected for operational
employment of the system, considering organization, doctrine,
tactics, survivability, vulnerability, and threat. 

\5 Established under section 800 of the National Defense
Authorization Act for Fiscal Year 1991
(P.L.  101-510, 1990). 

\6 Acquisition Reform:  Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar.  22,
1994). 


   RESULTS IN BRIEF
------------------------------------------------------------ Letter :2

Our review of 13 case studies indicated that DOT&E oversight of
operational testing and evaluation increased the probability that
testing would be more realistic and more thorough.\7 Specifically,
DOT&E was influential in advocating increasing the reliability of the
observed performance and reducing the risk of unknowns through more
thorough testing; conducting more realistic testing; enhancing data
collection and analysis; reporting independent findings; and
recommending follow-on operational test and evaluation when
suitability or effectiveness was not fully demonstrated prior to
initiating full-rate production. 

The independence of DOT&E--and its resulting authority to report
directly to Congress--is the foundation of its effectiveness.  That
independence, along with its legislative mandate, provides sufficient
freedom and authority to exercise effective oversight of the
operational testing and evaluation of new systems before a decision
is made to begin full-rate production.  In the conduct of its
oversight, DOT&E (1) executes its approval authority over test and
evaluation master plans and operational test plans and (2) provides
independent annual and summary reports on the test and evaluation of
individual weapon systems to the Secretary of Defense and Congress. 

DOT&E can reduce the risk that systems are not adequately tested
prior to the full-rate production decision.  But DOT&E cannot ensure
that (1) only systems whose operational effectiveness and suitability
have been demonstrated through operational testing will proceed to
the full-rate production decision or (2) new fielded systems will
accomplish their missions as intended or that the fielded systems are
safe, survivable, and effective.  Moreover, service and acquisition
officials have argued that DOT&E does not have the independent
authority to require and approve service-conducted follow-on
operational test and evaluation after full-rate production begins. 
In addition, the Office is not currently required to report on
whether new systems are both operationally suitable and effective
before they are fielded. 

DOT&E management must balance its oversight responsibilities for
operational testing with the broader acquisition priorities of
program managers and service test agencies.  Though supportive of the
Office's mission and independence, program and service
representatives frequently considered the time, expense, and
resources expended to accommodate DOT&E concerns to be ill-advised. 
Service officials contended that the additional testing requested by
DOT&E was either unnecessary for determining the operational
effectiveness or suitability of a program or unrealistic in light of
the limitations in the services' testing resources. 

DOT&E must manage multiple oversight, advisory, and coordination
responsibilities.  Several current trends may challenge DOT&E's
ability to manage its workload and its ability to impact operational
test and evaluation.  These trends include (1) service challenges to
DOT&E's authority to require and oversee follow-on operational
testing and evaluation, (2) a decline in resources available for
oversight, (3) an expansion of DOT&E involvement in activities other
than oversight of major acquisition programs, (4) participation of
DOT&E in the acquisition process as a member of working-level
integrated product teams, and (5) greater integration of
developmental and operational testing.  These trends make it
imperative that DOT&E prioritize its workload to achieve a balance
between the oversight of major defense acquisition programs and other
initiatives important to the quality of operational test and
evaluation. 


--------------------
\7 Aspects of realism can include (1) equipment and personnel placed
under realistic stress and operational tempo, (2)
threat-representative forces, (3) end-to-end testing, (4) realistic
combat tactics, (5) operationally realistic environments and targets,
(6) countermeasured environments, (7) interfacing systems, (8)
terrain and environmental conditions, and (9) contractor involvement. 


   DOT&E ADVOCATES MORE THOROUGH
   TESTING THAN THE SERVICES
------------------------------------------------------------ Letter :3

A frequent complaint among representatives of the services'
operational testing agencies was that DOT&E frequently demanded more
tests than were proposed by the operational test agencies in draft
master plans or test plans.  Operational test agency representatives
contended that the additional testing was either unnecessary for
determining the operational effectiveness or suitability of a program
or unrealistic in light of the limitations in the services' testing
resources.  However, our review indicated that DOT&E urged more
testing to reduce the level of risk and number of unknowns prior to
the decision to begin full production, while program and service
officials typically sought less testing and were willing to accept
greater risk when making production decisions.  The additional
testing DOT&E advocated, often over the objections of service
testers, served to meet the underlying objectives of operational
testing--to reduce the uncertainty and risk that systems entering
full-rate production would not fulfill their requirements. 

The impact of DOT&E oversight varies with the system under
development.  Table 1 summarizes the types of impacts that DOT&E
advocated or facilitated in operational testing among the 13 cases we
studied.  While the impacts vary, one consistent pattern in our case
studies was a reduction in uncertainty regarding the weapon systems'
suitability or effectiveness prior to the full-rate production
decision.  Each of the impacts are discussed in more detail in tables
2-6 and in subsequent sections. 



                                     Table 1
                     
                       Types of Impacts on the Operational
                        Testing of 13 Systems Due to DOT&E
                                    Oversight

                                                                 Follow-on
                                                                 operational
             More                      Enhancement  DOT&E's      test and
             testing      More         s made in    conclusion   evaluation
             advocated    realism      data         deviated     advocated and
             and          included in  collection   from the     planned or
System       conducted    test design  or analysis  service's    conducted
-----------  -----------  -----------  -----------  -----------  ---------------
AH-64D       X            X            X            X            X
Longbow
Apache
helicopter

ASPJ\a       X                         X
jammer

C-17A        X                                                   X
aircraft

E-3 AWACS\b               X            X
(RSIP\c)

F-22         X            X                                      X
fighter

Javelin      X            X                         X
missile

Joint                     X            X            X            X
STARS\d

LPD-17       X                         X
assault
ship

M1A2 tank                              X            X            X

Sensor       X            X                         X            X
fuzed
weapon

Standard     X            X
missile

Tomahawk                  X            X
Weapon
System

V-22                      X
aircraft
--------------------------------------------------------------------------------
Note:  The absence of an "X" does not necessarily indicate the
absence of DOT&E impact.  For example, blanks may occur where DOT&E
and the service agreed on issues; however, the deterrent effect of
DOT&E oversight is unquantifiable.  In addition, blanks may occur
because the system has not yet progressed through the entire
acquisition process. 

\a Airborne Self-Protection Jammer. 

\b Airborne Warning and Control System. 

\c Radar System Improvement Program. 

\d Surveillance Target Attack Radar System. 


      DOT&E OVERSIGHT LED TO MORE
      TESTING THAN PROPOSED BY THE
      OPERATIONAL TEST AGENCIES
---------------------------------------------------------- Letter :3.1

Two of DOT&E's typical concerns in reviewing service test plans are
that the proposed test methodologies enable (1) comparisons of a
system's effectiveness through side-by-side testing between the
existing and modified systems and (2) assessments of a system's
reliability through a sufficient number of test repetitions.  Table 2
illustrates examples of cases where additional testing was conducted
at DOT&E's insistence or with DOT&E's support to alleviate these and
other types of effectiveness and suitability concerns. 



                                     Table 2
                     
                        Examples of Programs that Expanded
                          Testing Due to DOT&E Oversight

System              Expanded testing               Impact
------------------  -----------------------------  -----------------------------
AH-64D Longbow      DOT&E insisted that the Army   Testers were able to
Apache helicopter   include a baseline AH-64A      demonstrate the gunnery
                    company in gunnery and force-  performance improvements of
                    on-force exercises to ensure   the AH-64D. These
                    direct comparability with the  improvements included that
                    Longbow.                       (1) the AH-64D had 300
                                                   instances of lethality
                                                   compared to 75 for the AH-
                                                   64A, (2) the AH-64D was
                                                   approximately 8 times more
                                                   survivable than the AH-64A,
                                                   and (3) the AH-64D had zero
                                                   fratricide instances compared
                                                   to 34 for the AH-64A.

ASPJ jammer         In follow-on operational test  The revised test plan enabled
                    and evaluation of the F-14D    testers to address the
                    begun in 1995, DOT&E insisted  critical operating issue--
                    that the scope of the test     that the F-14D is more
                    plan address the ASPJ's        survivable with the ASPJ as
                    contribution to the            part of its electronic
                    aircraft's survivability--     warfare suite than without
                    not merely the jammer's        it.
                    compatibility with the
                    aircraft's avionics. This
                    expansion of the scope
                    necessitated an additional 18
                    open air flight tests to
                    measure the ASPJ's
                    effectiveness against air-
                    to-air threats and a
                    requirement to gather
                    suitability data pertaining
                    to ASPJ, including its built-
                    in test equipment.\a

C-17 aircraft       The ability to safely perform  DOT&E forced testing that
                    a mass personnel airdrop       confirmed operational
                    while flying in close          limitations, and the Army has
                    formation is a key Air Force   yet to approve mass airdrops
                    capability needed to conduct   of personnel from C-17s
                    a strategic brigade airdrop.   flying in close formation.
                    DOT&E insisted that an         Operational tests identified
                    airdrop of a brigade slice of  specific problems with the C-
                    personnel and equipment be     17's airdrop capability--
                    done. The Air Force's          that with the air turbulence
                    position was that the airdrop  created in the wake of the
                    was unnecessary before the     aircraft, flying in close
                    full-rate production decision  formation can cause the
                    and that the use of the        parachutes dropping from
                    aircraft in airdrops would be  aircraft to oscillate,
                    determined after the full-     partially deflate, or
                    rate production decision.      collapse. These conditions
                                                   could result in serious
                                                   injury or death to
                                                   paratroopers.

F-22 fighter        DOT&E and the Air Force        The use of multiple testing
                    agreed to a balanced approach  and evaluation techniques
                    of open-air testing, full      will reduce uncertainty in
                    mission simulation, and        system effectiveness more
                    digital models against then-   than the Air Force's initial
                    current and future threats in  preference to use test
                    an overall F-22 and F-15       results to support evaluation
                    effectiveness analysis.        by modeling.

Javelin missile     DOT&E insisted that the        The test provided additional
                    system undergo additional      confidence that the weapon
                    operational testing prior to   system's modifications had
                    the full-rate production       not affected Javelin's
                    decision in 1997 because over  suitability for combat.
                    50 design changes had been
                    made to the system since
                    initial operational test and
                    evaluation in 1993. The Army
                    claimed that successful
                    passage of technical tests
                    was adequate assurance of
                    suitability for combat and
                    did not originally intend to
                    conduct operational tests
                    until 1998, over a year after
                    the start of full-rate
                    production.\b

Javelin missile     Based on data collected from   Before the additional test
(con't)             initial operational testing,   was conducted, the Army
                    DOT&E disagreed with the       modified components of the
                    Army's conclusion that the     command launch unit to
                    Javelin was suitable for       increase its reliability. The
                    combat and supported the       subsequent test demonstrated
                    Army's operational test        that the modifications were
                    agency in requiring the        successful. The test also
                    program manager to conduct an  provided two additional
                    operational test to confirm    benefits. First, missile
                    the unit's reliability.        failures during the test led
                                                   to discovery and correction
                                                   of a design flaw that
                                                   prevented the missiles from
                                                   leaving the launch tube when
                                                   the gunner pulled the
                                                   trigger. Second, while
                                                   developing the test plan,
                                                   DOT&E discovered that the
                                                   Army had no Javelin-specific
                                                   tactical doctrine and
                                                   recommended the Army study
                                                   this deficiency. As a result,
                                                   the Army developed
                                                   operational tactics to guide
                                                   officers in integrating
                                                   Javelin with other antitank
                                                   systems.

LPD-17 assault      The originally proposed        The expanded scope of the
ship                operational test for the LPD-  test plan will more closely
                    17 consisted of at-sea         encompass the range of system
                    steaming and some landing      requirements as well as
                    craft air cushion (LCAC)       enhance the realism of the
                    operations. DOT&E forced the   test scenario.
                    incorporation of full-scale
                    assault operations with
                    LCACs, aircraft, ground
                    assault equipment, and
                    personnel.

Sensor fuzed        DOT&E insisted on a second     The additional testing of
weapon              phase of operational test and  system issues not fully
                    evaluation before the full-    tested in the first phase
                    rate production decision that  (such as additional
                    the Air Force did not want to  countermeasures, multiple
                    conduct.                       releases, and an alternate
                                                   target formation) reduced
                                                   uncertainty in system
                                                   effectiveness and
                                                   reliability.

Standard missile    DOT&E insisted on and          DOT&E's insistence on
SM-2                obtained five flight tests of  additional testing has
                    the User Operational           lowered the technical risk of
                    Evaluation System SM-2 block   the program by providing for
                    IVA missile, a theater         a series of tests to
                    ballistic missile defense      establish system level
                    system. The Navy planned only  validation. These tests will
                    two at-sea safety flights      help to demonstrate the level
                    against nonthreat-             of reliability and
                    representative targets. Some   effectiveness of the SM-2
                    of the new flight tests will   block IVA missile.
                    be conducted against threat-
                    representative targets from
                    the integrated AEGIS system.
--------------------------------------------------------------------------------
\a See Electronic Warfare (GAO/NSIAD-96-109R, Mar.  1, 1996). 

\b See Army Acquisition:  Javelin Is Not Ready for Multiyear
Procurement (GAO/NSIAD-96-199, Sept.  26, 1996). 


      DOT&E OVERSIGHT LED TO MORE
      REALISTIC TESTING THAN
      PROPOSED BY THE OPERATIONAL
      TEST AGENCIES
---------------------------------------------------------- Letter :3.2

Table 3 illustrates examples where the design or conduct of
operational testing was modified at DOT&E's insistence or with
DOT&E's support to increase the realism of test conditions and reduce
the uncertainty of system suitability or effectiveness. 



                                     Table 3
                     
                     Examples of Programs That Conducted More
                     Realistic Testing Due to DOT&E Oversight

                           Enhanced realism in tests  Impact
System                     -------------------------  --------------------------
AH-64D Longbow             DOT&E required a           The testing revealed
Apache helicopter          demanding air defense      operational limitations of
                           network, directly          the AH-64D variant without
                           intervening to ensure      the fire control radar and
                           that a specific threat     thereby raised the issue
                           would be present in the    of the appropriate mix of
                           force-on-force trials.     variants to procure. The
                                                      AH-64D variant with the
                                                      fire control radar was
                                                      unable to reduce the air
                                                      defense threat
                                                      sufficiently to allow the
                                                      variant without the fire
                                                      control radar to move into
                                                      battle positions without
                                                      significant possibility of
                                                      being engaged by those air
                                                      defense units.

E-3 AWACS (RSIP)           DOT&E insisted that (1)    Reduced uncertainty of
                           mission crews comprise a   system effectiveness
                           cross section of typical   because (1) AWACS
                           AWACS aircrew members,     personnel from the
                           (2) RSIP be employed       engineering and
                           against an array of        developmental test sorties
                           actual Soviet and other    were excluded, resulting
                           threats, and (3) the       in the use of two test
                           system be used in eight    crews comprised of a
                           different terrain          typical ratio of U.S. and
                           combinations in both the   Canadian deployment
                           United States and Europe.  personnel and (2) actual
                                                      threats and realistic
                                                      environments were
                                                      incorporated.

F-22 fighter               DOT&E was instrumental in  The credibility of the
                           ensuring that a full       full mission simulator
                           mission simulator was      (used to compare relative
                           developed for comparison   mission effectiveness of
                           testing using validated    the F-15 and F-22) will be
                           software and hardware,     enhanced.
                           insisting that the
                           functionality and
                           fidelity of the
                           simulation be validated
                           by open air flight data.

                           DOT&E insisted that the    The confidence level of
                           test and evaluation        the model's prediction is
                           master plan include high   enhanced by introducing
                           tempo demonstrations to    surge data from actual
                           test the required sortie   operations.
                           generation rate.

Javelin missile            DOT&E required Army        The Army found that
                           troops to carry the        missiles carried during
                           missile a representative   the test failed to leave
                           distance during missions   the launch tube because of
                           and prior to actual        a faulty design of the
                           firings to ensure that     external restraining pin-
                           the missile's reliability  wiring harness. This
                           would not be affected by   finding led the Army to
                           field handling.            redesign the assembly,
                                                      which prevented potential
                                                      missile malfunctions in
                                                      combat situations.

Joint STARS                In the development of the  Deployment of the system
                           test plan, DOT&E           to Bosnia precluded
                           encouraged participation   testing at the National
                           of Air Force and Army      Training Center, but the
                           testers in training        test design precedent was
                           exercises at the National  established.
                           Training Center as a way
                           to enhance test realism.

Sensor fuzed weapon        During the second phase    The enhanced realism of
                           of initial operational     testing reduced
                           test and evaluation,       uncertainty of system
                           DOT&E required an          effectiveness at low
                           extensive validation of    altitudes and confirmed
                           the infrared signature     decreased effectiveness as
                           and the use of             altitude, dive angle, and
                           countermeasures, insisted  time of flight increase.
                           on all-weather and all-
                           altitude testing at
                           numerous test sites;
                           insisted on realistic and
                           comprehensive
                           countermeasures testing;
                           and ensured realistic
                           targets were made
                           available for testing.

Standard missile SM-2      During the review of the   The need for realistic
                           Navy's draft test and      aerial targets is a
                           evaluation master plan     significant issue cutting
                           for the SM-2 block IV,     across all Navy surface
                           DOT&E identified           antiair warfare programs
                           inadequacies in aerial     such as the Phalanx Close-
                           target programs and        In Weapon System and the
                           required that threat-      Rolling Airframe Missile,
                           representative targets be  as well as the various SM-
                           available for operational  2 blocks.
                           testing.

Tomahawk Weapon System     DOT&E was instrumental in  The use of realistic
                           ensuring that only ship    operators reduced
                           crews were used during     uncertainty in system
                           the testing of the all-    reliability and
                           up-rounds\a and the        effectiveness.
                           Tomahawk Weapon Control
                           System. Support personnel
                           conducted testing, while
                           contract personnel
                           maintained the equipment
                           as they do in actual
                           operations.

V-22 aircraft              DOT&E has emphasized the   The test program has been
                           effects of the V-22        revised to conduct
                           downwash on personnel and  downwash testing in 1997
                           material in the vicinity   rather than 1999 to
                           of the hovering aircraft   address the concerns of
                           and the need to test in    DOT&E and others.
                           more realistic ship and
                           landing zone
                           environments.
--------------------------------------------------------------------------------
\a Each Tomahawk missile variant is contained within a pressurized
canister to form an all-up-round. 


      DOT&E OVERSIGHT LED TO
      CHANGES IN THE DATA ANALYSIS
      PLAN
---------------------------------------------------------- Letter :3.3

DOT&E can insist on or support changes in data analysis plans that
provide more meaningful analyses for decisionmakers.  Table 4
illustrates instances in which DOT&E altered the proposed data
collection or analysis plans to enhance the reliability or utility of
the test data. 



                                     Table 4
                     
                      Examples of Programs in Which Changes
                     Were Made in the Data Analysis Plan Due
                                to DOT&E Oversight

                           Changes in data analysis
                           plan                       Impact
System                     -------------------------  --------------------------
AH-64D Longbow             DOT&E insisted on          DOT&E input allowed
Apache helicopter          performance criteria to    testers to more accurately
                           assess the superiority of  compare the AH-64D to the
                           the AH-64D over the AH-    AH-64A in quantifiable
                           64A. The criteria--a 20-   categories of lethality,
                           percent improvement--had   survivability, and
                           not formally been          fratricide.
                           included in the test and
                           evaluation master plan.
                           DOT&E required measures
                           that addressed the number
                           of targets killed and
                           helicopters lost.

ASPJ jammer                DOT&E required the Navy    The Navy determined that
                           to test the ASPJ against   the ASPJ was effective
                           the type of missile that   against that threat.
                           shot down an F-16 over
                           Bosnia in June 1995.\a

                           DOT&E was instrumental in  Independent oversight of
                           establishing a             ASPJ's suitability
                           requirement to gather      assessment confirmed
                           suitability data on its    ongoing concerns with
                           built-in test equipment.   system reliability.
                           While the contractor
                           reported improvement in
                           previously unreliable
                           built-in test equipment,
                           DOT&E questioned the data
                           collection and
                           interpretation.

E-3 AWACS (RSIP)           DOT&E insisted that        Reduced uncertainty of
                           service personnel be       system effectiveness
                           trained to operate         because the contractor was
                           contractor data            removed from data
                           extraction systems,        processing ensuring test
                           thereby removing the       integrity.
                           contractor from the
                           process and ensuring data
                           integrity. DOT&E reviewed
                           a major radar failure and
                           discovered an error in
                           the technical path
                           described by the service.

Joint STARS                DOT&E insisted that the    The change in test
                           Air Force modify its       measures resulted in test
                           original technical         data that were more
                           requirements to include    operationally relevant to
                           measures of effectiveness  system effectiveness.
                           that directly addressed
                           the missions of
                           surveillance, targeting,
                           and battlement
                           management. DOT&E
                           stressed differentiation
                           between user and system
                           requirements.

LPD-17 assault ship        DOT&E insisted on          The change in test
                           measures of effectiveness  measures will result in
                           that addressed the         test data that are more
                           movement of men and        operationally relevant to
                           equipment ashore rather    system effectiveness.
                           than the Navy's original
                           requirements that focused
                           on technical
                           specifications.

M1A2 tank                  DOT&E required that the    Reduced uncertainty of
                           Army use credible data     improved effectiveness and
                           for the determination of   suitability of the M1A2
                           reliability in follow-on   compared with the M1A1.
                           operational test and
                           evaluation. The Army
                           proposed the use of
                           failures and other
                           secondary measures that
                           would not provide a
                           credible basis for
                           reversing the results of
                           initial operational test
                           and evaluation. DOT&E
                           insisted that the
                           operational testing be
                           conducted to compare the
                           M1A2 with the M1A1.
                           Several improvements in
                           the M1A2 addressed
                           command and control that
                           could not be directly
                           measured. By conducting
                           several operations with
                           both tanks, the
                           difference in movements
                           and coordination could be
                           examined to determine the
                           value of the command and
                           control improvements. By
                           adding uncertainty to
                           test scenarios, DOT&E
                           enabled the Army
                           operational test agency a
                           means to identify
                           differences between the
                           M1A1 and M1A2 models.

Tomahawk Weapon System     DOT&E was instrumental in  More rigorous data
                           ensuring that the          collection and validation
                           effectiveness of mission   reduced uncertainty of
                           planning systems was       system effectiveness.
                           validated using high-
                           fidelity models and
                           simulations and that bit-
                           by-bit checks were
                           conducted to validate the
                           effectiveness of
                           functional operations of
                           the planning system.
--------------------------------------------------------------------------------
\a See Airborne Self-Protection Jammer (GAO/NSIAD-97-46R, Jan.  29,
1997). 


      DOT&E INTERPRETED THE
      RESULTS OF SOME TESTING LESS
      FAVORABLY THAN THE
      OPERATIONAL TEST AGENCIES
---------------------------------------------------------- Letter :3.4

DOT&E's independent analysis of service test data may confirm or
dispute the results and conclusions reported by the service.  In the
cases described in table 5, DOT&E's analysis of service operational
test and evaluation data resulted in divergent, often less favorable
conclusions than those reached by the service. 



                                     Table 5
                     
                     Examples of Programs in Which DOT&E and
                           Service Conclusions Differed

System              Conflicting test results       Impact
------------------  -----------------------------  -----------------------------
AH-64D Longbow      DOT&E's independent analysis   The Army will conduct a
Apache helicopter   of the test data identified a  series of simulations and
                    predominant firing technique   additional missile firings to
                    that had not previously been   determine the factors
                    identified as useful. Though   affecting the overall
                    the technique was never        effectiveness of the
                    anticipated to be used so      technique and its relative
                    extensively and had not been   effectiveness to the primary
                    considered in the development  modes of engagement, thereby
                    of the Longbow's tactics,      increasing certainty in
                    techniques, and procedures,    system effectiveness.
                    DOT&E determined that over
                    half of the operational test
                    engagements were conducted
                    using this technique.
                    Nonetheless, this revelation
                    was not in the Army test
                    report.

Javelin missile     DOT&E did not use reliability  The Army made numerous design
                    data from the pre-initial      changes to the launch unit
                    operational test and           and round before the
                    evaluation period because the  contractor initiated low-
                    data were not realistic; as a  rate production.
                    result, DOT&E found the
                    command launch unit failed to
                    meet its reliability
                    criteria, differing from the
                    Army's report.

Joint STARS         DOT&E disagreed with the Air   DOT&E's Beyond-LRIP report
                    Force operational test         indicated not only the Joint
                    agency's positive assessment   STARS' disappointing test
                    of the operational             results but also the need for
                    suitability and effectiveness  extensive follow-on
                    of Joint STARS following its   operational test and
                    deployment to Operation Joint  evaluation. Subsequently, the
                    Endeavor. DOT&E concluded      Joint STARS acquisition
                    that Joint STARS met one of    decision memorandum required
                    three critical operational     that the test and evaluation
                    effectiveness issues--with     master plan be updated and
                    limitations, while the other   that follow-on operational
                    two effectiveness issues       test and evaluation address
                    could not be determined.       the deficiencies identified
                    Overall, the Air Force's       in initial operational test
                    conclusion was "suitable with  and evaluation by DOT&E.
                    deficiencies"; DOT&E's
                    conclusion was "as tested is
                    unsuitable."\a DOT&E and the
                    Air Force operational test
                    agency also disagreed on how
                    to report data when terrain
                    masking occurred. DOT&E
                    objected to the Air Force's
                    phrasing "nothing significant
                    to report," when in fact
                    nothing could be seen.

M1A2 tank           DOT&E evaluated the tank as    Follow-on operational test
                    not operationally suitable--   and evaluation was conducted
                    a finding at odds with Army    to determine if the Army's
                    testers. DOT&E determined      design changes had improved
                    that the tank was unreliable   the system. The suitability
                    and unsafe due to uncommanded  problems persisted and the
                    turret movements, hot          follow-on operational test
                    surfaces that caused contact   and evaluation was suspended.
                    burns, and inadvertent firing  New design changes were made
                    of the .50 caliber machine     and a second follow-on
                    gun.                           operational test and
                                                   evaluation was conducted,
                                                   which determined that the
                                                   safety issues were resolved
                                                   and that the tank is now
                                                   operationally suitable.

Sensor fuzed        Based on the results of the    As a result of the unresolved
weapon              first phase of operational     issues in 1992, a second
                    test and evaluation ending in  phase of operational test and
                    1992, the Air Force concluded  evaluation was planned and
                    that the sensor fuzed weapon   executed, leading DOT&E to
                    was "suitable and effective    conclude in 1996 that the
                    for combat." In contrast,      system was operationally
                    DOT&E concluded from the same  suitable and effective--when
                    tests that the system was      employed at low altitude
                    only "potentially              using level or shallow angle
                    operationally effective and    dive deliveries.
                    suitable."
--------------------------------------------------------------------------------
\a See Tactical Intelligence:  Joint STARS Full-Rate Production
Decision Was Premature and Risky (GAO/NSIAD-97-68, Apr.  25, 1997). 


      DOT&E RECOMMENDED FOLLOW-ON
      OPERATIONAL TEST AND
      EVALUATION
---------------------------------------------------------- Letter :3.5

When DOT&E concludes that a weapon system has not fully demonstrated
operational suitability or effectiveness, or if new testing issues
arise during initial operational test and evaluation, it may
recommend that follow-on operational test and evaluation be done
after the full-rate production decision.  Table 6 identifies
follow-on operational test and evaluation that DOT&E supported. 



                                     Table 6
                     
                       Examples of Programs in Which DOT&E
                      Called for Follow-On Operational Test
                                  and Evaluation

                    Advocated follow-on
                    operational test and
System              evaluation                     Impact
------------------  -----------------------------  -----------------------------
AH-64D Longbow      DOT&E sought follow-on         The use of this technique was
Apache helicopter   operational test and           not fully anticipated prior
                    evaluation to characterize     to initial operational test
                    the Hellfire missile's         and evaluation. Its use
                    performance when using lock-   provided an unexpected level
                    on before launch-inhibit       of survivability for the AH-
                    technique. This method of      64D crews. This technique had
                    engagement enables crews to    been subjected to little, if
                    immediately take cover after   any, developmental testing.
                    target detection and fire at   Further testing will
                    moving targets from those      establish its probability of
                    covered locations. This        hit. The Army operational
                    method was used in over half   test agency plans to fire 8
                    of the operational test        to 10 missiles in August
                    engagements, though it had     1998.
                    not been considered
                    sufficiently significant to
                    incorporate in the Longbow's
                    tactics, techniques, and
                    procedures.

C-17A aircraft      DOT&E urged follow-on          The Air Force has undertaken
                    operational test and           further testing with the Army
                    evaluation to demonstrate the  to overcome system
                    system's ability to meet       deficiencies and demonstrate
                    operational readiness          effectiveness. The Army is
                    objectives, including          formulating a time
                    combination and brigade        requirement of about 30
                    airdrops, and software         minutes for completing a
                    maturity.                      strategic airdrop. The C-17
                                                   currently has a 5.5 minute
                                                   aircraft separation
                                                   restriction that essentially
                                                   prohibits formation flying
                                                   and therefore requires 2.5
                                                   hours to complete a strategic
                                                   airdrop. This resulted in
                                                   continuing efforts to resolve
                                                   these operational
                                                   limitations.

F-22 aircraft       DOT&E insisted that the test   The commitment to test these
                    and evaluation master plan     capabilities is formally
                    require follow-on operational  acknowledged.
                    test and evaluation on two
                    capabilities that will not be
                    released until after initial
                    operational test and
                    evaluation: employment of the
                    Joint Direct Attack Munition
                    and Cruise Missile Defense.

Joint STARS         DOT&E stated in its Joint      The Joint STARS acquisition
                    STARS Beyond-LRIP report that  decision memorandum directed
                    only 18 of 71 performance      additional testing to address
                    criteria tested were           suitability deficiencies in
                    demonstrated by the system     logistics and software.
                    and that further testing was
                    required for the remaining
                    53.

M1A2 tank           The M1A2, during initial       The Army executed a program
                    operational test and           to correct the deficiencies
                    evaluation in 1993, failed to  identified during initial
                    meet the combat mission        operational test and
                    reliability threshold,         evaluation and conducted
                    encountered an excessive       follow-on operational test
                    number of battery failures,    and evaluation in 1995.
                    consumed                       Suitability issues, such as
                    15 percent more fuel,          uncommanded turret movement
                    exhibited uncommanded main     and power loss, were again
                    gun/turret movements and       experienced. The follow-on
                    inadvertent .50 caliber        operational test and
                    machine-gun firing that made   evaluation was put on hold
                    the tank unsafe. DOT&E,        until additional corrective
                    through a Secretary of         actions could be applied.
                    Defense letter accompanying    Follow-on operational test
                    the M1A2 Beyond-LRIP report    and evaluation resumed in
                    to Congress, required follow-  July 1996. The safety
                    on operational test and        problems were found to have
                    evaluation on M1A2             been addressed by the design
                    suitability issues when the    changes, and there were no
                    Army claimed it was            observed instances of the
                    unnecessary.                   problems experienced during
                                                   initial or beginning follow-
                                                   on operational test and
                                                   evaluation.

Sensor fuzed        The test and evaluation        Follow-on operational test
weapon              master plan for the second     and evaluation ensures
                    phase of operational test and  further investigation of
                    evaluation specified a series  system limitations known at
                    of follow-on operational test  the time of the full-rate
                    and evaluations that would     production decision.
                    address how well the addition
                    of the Wind Compensated
                    Munition Dispenser and the
                    preplanned product
                    improvements will rectify
                    system limitations.
--------------------------------------------------------------------------------

   STRENGTHS AND WEAKNESSES OF
   CURRENT ORGANIZATIONAL
   FRAMEWORK
------------------------------------------------------------ Letter :4

The existence of a healthy difference of opinion between DOT&E and
the acquisition community is a viable sign of robust oversight.  In
nearly all of the cases we reviewed, the services and DOT&E cited at
least one testing controversy.  For example, services differ on how
they view the relationship between operational testing and their
development of tactics, techniques, and procedures.  In addition,
DOT&E's ability to independently view the development and testing of
new systems across the services brings value to the context of
testing.  However, several current trends have the potential to
adversely affect DOT&E's independence and its ability to affect
operational test and evaluation, including (1) service challenges to
DOT&E's authority to require and oversee follow-on operational
testing and evaluation, (2) declining resources available for
oversight, (3) the management of limited resources to address
competing priorities, (4) DOT&E's participation in the acquisition
process as a member of the program manager's working-level integrated
product teams, and (5) greater integration of developmental and
operational testing.  DOT&E's impact on operational testing is
dependent upon its ability to manage these divergent forces while
maintaining its independence. 



      INDEPENDENCE IS THE KEY TO
      DOT&E'S EFFECTIVENESS
---------------------------------------------------------- Letter :4.1

Although the acquisition community has three central
objectives--performance, cost, and schedule--DOT&E has but one: 
operational testing of performance.  These distinct priorities lead
to testing disputes.  Characteristically, the disputes for each
system we reviewed revolved around questions of how, how much, and
when to conduct operational testing, not whether to conduct
operational testing.  Conflicts encompassed issues such as (1) how
many and what types of tests to conduct; (2) when testing should
occur; (3) what data to collect, how to collect it, and how best to
analyze it; and (4) what conclusions were supportable, given the
analysis and limitations of the test program.  The foundation of most
disputes lay in different notions of the costs and benefits of
testing and the levels of risk that were acceptable when making
full-rate production decisions.  DOT&E consistently urged more
testing (and consequently more time, resources, and cost) to reduce
the level of risk and number of unknowns before the decision to
proceed to full-rate production, while the services consistently
sought less testing and accepted more risk when making production
decisions.  Among our case studies, these divergent dispositions
frequently led to healthy debates about the optimal test program, and
in a small number of cases, the differences led to contentious
working relations. 

In reviews of individual weapon systems, we have consistently found
that testing and evaluation is generally viewed by the acquisition
community as a requirement imposed by outsiders rather than a
management tool to identify, evaluate, and reduce risks, and
therefore a means to more successful programs.  Developers are
frustrated by the delays and expense imposed on their programs by
what they perceive as overzealous testers.  The program office
strives to get the program into production despite uncertainties that
the system will work as promised or intended.  Therefore, reducing
troublesome parts of the acquisition process--such as operational
testing--is viewed as a means to reduce the time required to enter
production. 

Nonetheless, the commanders and action officers within the service
operational test agencies were nearly unanimous in their support for
an independent test and evaluation office within OSD.  For example,
the Commander of the Army's Operational Test and Evaluation Command
commended the style and orientation of the current DOT&E Director and
affirmed the long-term importance of the office and its independent
reporting responsibilities to Congress.  The Commander of the Navy's
Operational Test and Evaluation Force stated that the independence of
both DOT&E and the operational test agency was an essential element
in achieving their common goal of ensuring that new programs pass
sufficiently rigorous and realistic operational testing prior to
fielding.  The Commander of the Air Force's Operational Test and
Evaluation, while critical of DOT&E oversight of several major weapon
systems, said that the services were well served by DOT&E's potential
to independently report to Congress.  Moreover, nearly all the
operational test agency action officers we interviewed participate in
the integrated product teams with the DOT&E action officers and
recognized the value of the Office's independent oversight role.  The
action officers within the service testing organizations also have a
degree of independence that enables them to represent the future
users of systems developed in the acquisition community.  These
action officers stated that their ability to voice positions
unpopular with the acquisition community was strengthened when DOT&E
separately supported their views. 

In discussions with over three dozen action officers and analysts
responsible for the 13 cases we reviewed, the independence of DOT&E
emerged as the fundamental condition to enable effective and
efficient oversight.  The foundation of interagency (i.e., DOT&E and
service operational test agencies) relations is based on the
independence of DOT&E, its legislative mandate, and its independent
reporting to Congress.  DOT&E is outside the chain of command of
those responsible for developing and testing new systems.  The
services need to cooperate with DOT&E primarily because the Office
must approve all test and evaluation master plans and operational
test plans.  Moreover, DOT&E independently reports on the operational
suitability and effectiveness at a system's full-rate production
milestone, a report that is sent separately to Congress. 


      UNFAVORABLE REPORTS ON
      OPERATIONAL TESTING DO NOT
      ALWAYS INHIBIT FULL-RATE
      PRODUCTION
---------------------------------------------------------- Letter :4.2

DOT&E's report on a system's operational suitability and
effectiveness is only one of several inputs considered before the
full-rate production decision is made.  An unfavorable DOT&E report
does not necessarily prevent full-rate production.  In each of the
cases cited below, an affirmative full-rate production decision was
made despite a DOT&E report concluding that the system had not
demonstrated during operational test and evaluation that it was both
operationally suitable and operationally effective: 

  -- Full-rate production of the M1A2 tank was approved despite
     DOT&E's report that found the system unsuitable. 

  -- Full-rate production of Joint STARS was approved, though the
     system demonstrated only limited effectiveness for "operations
     other than war" and found "as tested is unsuitable." Only 18 of
     the 71 performance criteria were met; 53 others required more
     testing. 

  -- Full-rate production of the C-17 Airlifter was approved despite
     a number of operational test and evaluation deficiencies,
     including immature software and failure to meet combination and
     brigade airdrop objectives. 


      SERVICES CONTEST DOT&E
      OVERSIGHT OF FOLLOW-ON
      OPERATIONAL TEST AND
      EVALUATION
---------------------------------------------------------- Letter :4.3

The services contend that DOT&E does not have authority to insist on,
or independently approve the conduct of, follow-on operational test
and evaluation.  However, in several of the systems we reviewed,
DOT&E overcame service opposition and monitored follow-on operational
test and evaluation.  It used several means to achieve success, such
as (1) incorporating follow-on operational test and evaluation in
test and evaluation master plans developed and approved prior to the
full-rate production decision milestone; (2) persuading the Secretary
of Defense to specify follow-on operational test and evaluation, and
DOT&E's oversight role, in the full-rate production acquisition
decision memorandum; and (3) citing policy, based on title 10, that
entitles DOT&E to oversee operational test and evaluation whenever it
occurs in the acquisition process.\8

Nonetheless, DOT&E action officers stated that the service's
acceptance of DOT&E's role in follow-on operational test and
evaluation varies over time, by service and acquisition system, and
is largely dependent upon the convictions of executives in both the
services and DOT&E.  Among the cases reviewed in this report, the
services offered a variety of arguments against DOT&E's having a role
in follow-on operational test and evaluation.  They specifically
asserted the following: 

  -- DOT&E need not be involved because the scope of follow-on
     operational test and evaluation is frequently less encompassing
     than initial operational test and evaluation.  Follow-on
     operational test and evaluation has been characterized as
     testing by the user to determine the strengths and weaknesses of
     the system and to determine ways to compensate for, or fix,
     shortcomings observed in initial operational test and
     evaluation.\9

  -- Title 10 provides DOT&E with the authority to monitor and
     review--but not necessarily approve--service follow-on
     operational test and evaluation plans.\10

  -- Follow-on operational test and evaluation is unnecessary when a
     system is found to be operationally effective and suitable
     during initial operational test and evaluation--even though
     DOT&E does not concur.\11

A clear distinction between DOT&E oversight in follow-on operational
test and evaluation versus initial operational test and evaluation is
that DOT&E is not required to report follow-on operational test and
evaluation results to Congress in the detailed manner of the
Beyond-LRIP report.  Therefore, even if follow-on operational test
and evaluation is conducted to assess modifications to correct
effectiveness or suitability shortcomings reported to Congress in the
Beyond-LRIP report, there is no requirement that Congress receive a
detailed accounting of the impact of these modifications. 


--------------------
\8 In March 1997 DOT&E issued the "Policy on DOT&E Oversight of
Systems in Follow-on Operational Test and Evaluation (FOT&E)." The
Director stated that 10 U.S.C.  section 139 provides DOT&E with the
authority to oversee follow-on operational test and evaluation. 
Specifically, DOT&E shall oversee follow-on operational test and
evaluation to (1) refine estimates made during operational test and
evaluation, (2) complete initial operational test and evaluation
activity, (3) verify correction of deficiencies, (4) evaluate
significant changes to design or employment, and (5) evaluate the
system to ensure that it continues to meet operational needs and
retains effectiveness in a substantially new environment or against a
new threat.  The Director elaborated by specifying that normal DOD
5000.2R documental and approval requirements apply. 

\9 In the case of Joint STARS, the acquisition decision memorandum
required the Air Force and the Army to update a test and evaluation
master plan for OSD approval--but did not require DOT&E approval. 
Moreover, the Director of Air Force Test and Evaluation termed
post-milestone III testing as "regression testing" and emphasized
that DOT&E had no oversight role. 

\10 In two case study systems, the C-17 and Joint STARS, the Air
Force provided DOT&E with a copy of its follow-on operational test
and evaluation test plans for review but did not allow sufficient
time and had no expectation that DOT&E would approve the plans prior
to the initiation of testing. 

\11 The acquisition decision memorandum for the M1A2 tank required
the Army to conduct follow-on operational test and evaluation (with
DOT&E oversight) on safety and suitability shortcomings identified by
DOT&E in initial operational test and evaluation, though the Army had
already determined that the system was operationally suitable as
tested. 


      DOT&E'S RESOURCES ARE
      DECLINING
---------------------------------------------------------- Letter :4.4

DOT&E's primary asset to conduct oversight--its cadre of action
officers--has decreased in size throughout the decade.  This creates
a management challenge for the Office because at the same time staff
has decreased, the number of programs overseen by DOT&E has
increased.  As illustrated in table 7, authorized staffing declined
from 48 in fiscal
year 1990 to 41 in fiscal year 1997, as did funding (in constant
dollars) from $12,725,000 in fiscal year 1990 to $11,437,000 in
fiscal year 1997.  The decline in DOT&E funding is consistent with
the general decline in DOD appropriations during this period. 
However, since fiscal year 1990, while the authorized staffing to
oversee operational test and evaluation has declined by 14.6 percent,
the number of systems on the oversight list has increased by 17.7
percent. 



                                     Table 7
                     
                            DOT&E Staffing and Funding

                              (Dollars in thousands)

                                           Fiscal year
                  --------------------------------------------------------------
                    1990    1991    1992    1993    1994    1995    1996    1997
----------------  ------  ------  ------  ------  ------  ------  ------  ------
Funding\a         $12,72  $13,55  $12,83  $12,33  $11,45  $12,50  $12,18  $11,43
                       5       0       6       3       0       1       3       7
Authorized            48      46      44      44      43    43\b      42      41
 staffing
Oversight            186     207     204     191     199     202     219     219
 programs
--------------------------------------------------------------------------------
\a Funding for operational test and evaluation program element only;
funding provided for the live fire test and evaluation program
element assumed by DOT&E beginning in fiscal year 1995 is not
reflected in the funding data for fiscal years 1995-97. 

\b The authorized end strength for DOT&E beginning in fiscal year
1995 increased by four a result of the congressionally directed
(Federal Acquisition Streamlining Act of 1994, P.L.  103-355) move of
live fire test and evaluation responsibilities to DOT&E.  Since these
positions are dedicated to live fire testing and not operational
testing, their numbers are not reflected in this table. 


      DOT&E'S LIMITED RESOURCES
      MUST ADDRESS COMPETING
      PRIORITIES
---------------------------------------------------------- Letter :4.5

With declining resources, DOT&E must manage competing priorities
related to its oversight, advisory, and coordination
responsibilities.  DOT&E must balance the continuing need to allocate
resources to these different priorities while not being perceived as
having lost any independence.  DOT&E management has flexibility in
defining some portion of the scope of its oversight and has continued
to electively oversee a substantial number of nonmajor defense
acquisition programs and assumed a leading role in advocating an
examination of the modernization needs of the test and evaluation
infrastructure. 


         DOT&E CONTINUES TO
         OVERSEE A SUBSTANTIAL
         NUMBER OF NONMAJOR
         PROGRAMS
-------------------------------------------------------- Letter :4.5.1

Between fiscal year 1990 and 1996, the number of nonmajor acquisition
programs overseen annually by DOT&E ranged between 19 and 43.  In
fiscal year 1996, when the oversight list reached a peak of 219, 1 of
every 8 programs was listed at the discretion of DOT&E.  Thus, during
this period when the resources to oversee operational testing
declined and acquisition reforms have placed additional burdens on
oversight staff, the directors of DOT&E continued to place extra
responsibility on their staff by augmenting the required oversight of
major acquisition programs with a substantial number of optional
systems. 

Despite a relative decline in resources for oversight, DOT&E
management has also elected to assume "a larger role in test resource
management planning and leadership in an attempt to achieve
much-needed resource modernization."\12 Although the Director is
designated as the principal adviser to the Secretary of Defense and
the Under Secretary of Defense for Acquisition and Technology on
operational test and evaluation, including operational test
facilities and equipment,\13 assuming the larger role defined by
DOT&E may be at the expense of its testing oversight mission and
perception of independence.  The DOT&E Director is now an adviser to
the Central Test and Evaluation Investment Program and previously
served as Chairman of the Test and Evaluation Committee.  The
Committee is responsible for the investment program and presides over
the planning, programming, and budgeting for development and
operational test resources.  When the Director served as chairman, we
questioned whether these ties created the perception that the
Director was not independent from developmental testing.\14 This
issue may resurface as DOT&E seeks a larger role in test resource
management planning.  Also, as the emphasis, cost, and time for
operational test and evaluation are increasingly questioned in the
drive to streamline acquisition, and as oversight assets are
stretched, new DOT&E initiatives may stress the Office's capacity to
manage oversight effectively. 


--------------------
\12 Director, Operational Test and Evaluation, FY'95 Report, March
1996, p.  I-6. 

\13 10 U.S.C.  section 139 assigns six responsibilities to the
Director, the fifth of which is to "review and make recommendations
to the Secretary of Defense on all budgetary and financial matters
relating to operational test and evaluation, including operational
test facilities and equipment {emphasis added}, in the Department of
Defense. 

\14 In Test and Evaluation:  The Director, Operational Test and
Evaluation's Role in Test Resources (GAO/NSIAD-90-128, Aug.  27,
1990), we found that the Director's independence was jeopardized
because the Director had influence over the types of development test
assets used by the services.  Responsibility for developmental test
resources rests with the services.  In 1987 Congress amended DOT&E's
statute to emphasize the separation of operational testing from
functions associated with developmental testing by stating that "the
Director may not be assigned any responsibility for developmental
test and evaluation, other than the provision of advice to officials
responsible for such testing."


         DOT&E PARTICIPATION IN
         WORKING-LEVEL INTEGRATED
         PRODUCT TEAMS HAS THE
         POTENTIAL TO COMPLICATE
         INDEPENDENCE
-------------------------------------------------------- Letter :4.5.2

In May 1995, the Secretary of Defense directed DOD to apply the
integrated product and process development concept--using integrated
product teams--throughout the acquisition process.  The revised DOD
acquisition regulations (DOD 5000.2-R March 1996) also addressed the
use of empowered integrated product teams at the program office
level.  DOT&E action officers participate as members of the
working-level integrated product teams, and the DOT&E Director is a
member of the overarching team.  One objective of integrated product
teams, and DOT&E participation in particular, is to expedite the
approval process of test documents by reaching agreement on the
strategy and plan through the identification and resolution of issues
early, understanding the issues, and documenting a quality test and
evaluation master plan that is acceptable to all organizational
levels the first time.  Integrated product teams are designed to
replace a previously sequential test and evaluation master plan
development and approval process and therefore enhance timeliness. 
While this management tool could increase communication between
testers and the program managers, it also poses a challenge to DOT&E
independence.  The challenge was recognized by the Department of
Defense Inspector General (DODIG) when after reviewing the conduct
of operational testing it subsequently recommended that "to meet the
intent of 10 U.S.C.  139, DOT&E should be a nonvoting member [of the
working-level integrated product team] so as to maintain his
independence."\15 {emphasis added} Though integrated product teams
were not used throughout the entire time period covered by this
report, several action officers noted that this management tool
created threats to their effectiveness other than having their
positions out-voted.  One DOT&E action officer reported having the
lone dissenting opinion in a meeting of 30 participants seeking to
reach consensus and resolve issues early.  The pressure of
maintaining independent, contrary positions in large working groups
can be a test.  Several DOT&E representatives also noted that the
frequency of integrated product team meetings to cover the multiple
systems for which they were responsible made it impossible for them
to attend all, thereby lessening the possibility that testing issues
can be identified and resolved as early as possible. 

Moreover, program managers and DOT&E pursue different objectives
through integrated product teams.  The services and program managers
view the teams as a way to facilitate their program objectives for
cost, schedule, and performance; DOT&E's objective is oversight of
performance through operational testing.  The program managers and
DOT&E share a desire to identify testing issues as early as possible. 
However, the priority of the program manager to resolve these issues
as early as possible through the teams may conflict with DOT&E's
mission.  DOT&E must remain flexible and react to unknowns as they
are disclosed during developmental testing, operational assessments,
and initial operational test and evaluation.  Thus, DOT&E's
participation on the teams is a natural source of tension and a
potential impediment to the team's decision-making.  The challenge
for DOT&E action officers is to maintain an independent and
potentially contrary position in an ongoing working group during the
life of a program, which may extend over several years. 


--------------------
\15 See Department of Defense Office of the Inspector General,
Operational Testing Performed on Weapons Systems, Report No.  96-107,
May 6, 1996, p.  11. 


      INCREASED INTEGRATION OF
      DEVELOPMENTAL AND
      OPERATIONAL TESTING MAY
      ATTENUATE INDEPENDENT
      OVERSIGHT
---------------------------------------------------------- Letter :4.6

The objectives of developmental and operational testing are distinct. 
Developmental testing determines whether a system meets its
functional requirements and contractual technical performance
criteria sufficiently to proceed with operational testing. 
Operational testing determines whether the system meets the
operational requirements and will contribute to mission effectiveness
in relevant operational environments sufficiently to justify
proceeding with production.  The integration of these two disparate
test activities is proposed to save the time and resources required
for testing and evaluation.  The sentiment to more closely link
developmental and operational testing dates from at least the 1986
Blue Ribbon Commission on Defense Management (Packard Commission),
which found that "developmental and operational testing have been too
divorced, the latter has been undertaken too late in the cycle, and
prototypes have been used and tested far too little."\16 However,
both we and the DODIG have found that systems were regularly tested
before they were ready for testing.  In its 1996 report, the DODIG
reported that "4 of 15 systems we examined for operational testing
were not ready for testing.  This situation occurred because a
calendar schedule rather than system readiness often drove the start
of testing."\17 Similarly, we have observed numerous systems that
have been pushed into low-rate initial production without sufficient
testing to demonstrate that the system will work as promised or
intended.  Our reviews of major system development in recent years
have found that because insufficient time was dedicated to initial
testing, systems were produced that later experienced problems during
operational testing and systems entered initial production despite
experiencing problems during early operational testing.\18

In 1996 the Secretary of Defense also urged the closer integration of
developmental and operational testing, and combined tests where
possible, in part to enhance the objectives of acquisition reform. 
Combined developmental and operational testing is only one of many
sources of test data that DOT&E has used to foster more timely and
thorough operational test and evaluation.  Other sources of
information include contractor developmental testing, builder's
trials, component testing, production lot testing, stockpile
reliability testing, and operational deployments.  While DOT&E has
some influence over the quality of operational testing, by
independently reviewing the design, execution, analysis, and
reporting of such tests, it has no direct involvement or oversight of
these other sources of testing information.  The use of alternative
sources of test data as substitutes for operational test and
evaluation will limit DOT&E's oversight mission, which was created to
improve the conduct and quality of testing. 


--------------------
\16 A Quest for Excellence:  Final Report by the President's Blue
Ribbon Commission on Defense Management, June 1986, p.  xxiii. 

\17 Office of the Inspector General, Department of Defense,
Operational Testing Performed on Weapons Systems, Report No.  96-107,
May 6, 1996, p.  16. 

\18 See Weapons Acquisition:  Low-Rate Initial Production Used to Buy
Weapon Systems Prematurely (GAO/NSIAD-95-18, Nov.  21, 1994). 


   CONCLUSIONS AND RECOMMENDATIONS
------------------------------------------------------------ Letter :5

DOT&E's challenge is to manage an expansion in independent oversight
while satisfying the efficiency goals of acquisition reform and
undergoing the economic pressures of downsizing.  DOT&E oversight is
clearly affecting the operational testing of new defense systems. 
DOT&E actions (such as the insistence on additional testing, more
realistic testing, more rigorous data analysis, and independent
assessments) are resulting in more assurance that new systems fielded
to our armed forces are safe, suitable, and effective.  However,
DOT&E is not, by design or practice, the guarantor of effective and
suitable acquisitions.  DOT&E oversight reduces, but does not
eliminate, the risk that new systems will not be operationally
effective and suitable.  Affirmative full-rate production decisions
are made for systems that have yet to demonstrate their operational
effectiveness or suitability.  Moreover, the services question
DOT&E's authority regarding follow-on test and evaluation of
subsequent corrective actions by the program office. 

We recommend that the Secretary of Defense revise DOD's operational
test and evaluation policies in the following ways: 

  -- Require the Under Secretary of Defense for Acquisition and
     Technology, in those cases where affirmative full-rate
     production decisions are made for major systems that have yet to
     demonstrate their operational effectiveness or suitability, to
     (1) take corrective actions to eliminate deficiencies in
     effectiveness or suitability and (2) conduct follow-on test and
     evaluation of corrective actions until the systems are
     determined to be operationally effective and suitable by the
     Director, Operational Test and Evaluation. 

  -- Require the Director, Operational Test and Evaluation, to (1)
     review and approve follow-on test and evaluation master plans
     and specific operational test plans for major systems before
     operational testing related to suitability and effectiveness
     issues left unresolved at the full-rate production decision and
     (2) upon the completion of follow-on operational test and
     evaluation, report to Congress, the Secretary of Defense, and
     the Under Secretary of Defense for Acquisition and Technology
     whether the testing was adequate and whether the results
     confirmed the system is operationally suitable and effective. 

Further, in light of increasing operational testing oversight
commitments and to accommodate oversight of follow-on operational
testing and evaluation, we recommend that the Director, Operational
Test and Evaluation, prioritize his Office's workload to ensure
sufficient attention is given to major defense acquisition programs. 


   AGENCY COMMENTS AND OUR
   EVALUATION
------------------------------------------------------------ Letter :6

In commenting on a draft of this report, DOD concurred with our first
and third recommendations and partially concurred with our second
recommendation.  Concerning the recommendation with which it
partially concurred, DOD stated that system specific reports to the
Secretary of Defense and Congress are not warranted for every system
that requires follow-on operational test and evaluation.  DOD pointed
out that for specific programs designated for follow-on oversight,
test plans are prepared to correct previously identified deficiencies
by milestone III, and DOT&E includes the results of follow-on testing
in its next annual report. 

We continue to believe our recommendation has merit.  We recommended
that the Secretary require DOT&E approval of follow-on test and
evaluation of corrective actions because during our review we found
no consensus within the defense acquisition community concerning
DOT&E's role in follow-on operational test and evaluation.  In its
comments DOD did not indicate whether it intended to give DOT&E a
role in follow-on operational test and evaluation that is comparable
to its role in initial operational test and evaluation.  Moreover, we
continue to believe that if a major system goes into full-rate
production (even though it was deemed by DOT&E not to be
operationally suitable and effective) based on the premise that
corrections will be made and some follow-on operational test and
evaluation will be performed, DOT&E should report, as promptly as
possible, whether or not the follow-on operational test and
evaluation results show that the system in question had improved
sufficiently to be characterized as both operationally suitable and
effective. 

DOD's comments are reprinted in their entirety in appendix IV, along
with our specific evaluation. 


As agreed with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 15 days after its date of issue.  We will then send copies to
other congressional committees and the Secretary of Defense.  We will
also make copies available to others upon request. 

If you have any questions or would like additional information,
please do not hesitate to call me at (202) 512-3092 or the
Evaluator-in-Charge, Jeff Harris, at (202) 512-3583. 

Kwai-Cheung Chan
Director of Special Studies and Evaluation


SCOPE AND METHODOLOGY
=========================================================== Appendix I

To develop information for this report, we selected a case study
methodology--evaluating the conduct and practices of DOT&E through an
analysis of 13 weapon systems.  Recognizing that many test and
evaluation issues are unique to individual systems, we determined
that a case study methodology would offer the greatest probability of
illuminating the variety of factors that impact the value or
effectiveness of oversight at the level of the Office of the
Secretary of Defense (OSD).  Moreover, with nearly 200 systems
subject to review of the Director, Operational Test and Evaluation
(DOT&E) at any one time, we sought a sample that would enable us to
determine if the Office had any impact as well as the ability to
examine the variety of programs overseen.  Therefore, we selected a
judgmental sample of cases reflecting the breadth of program types. 
As illustrated in table I.1,\1 we selected systems (1) from each of
the primary services, (2) categorized as major defense systems, and
(3) representing a wide array of acquisition and testing phases--from
early operational assessments through and beyond the full-rate
production decision.  We studied both new and modified systems. 



                                    Table I.1
                     
                      Characteristics of Weapon Systems Used
                                 for Case Studies

                                                Estimated or
                                                actual            New or
                                                year of selected  modification
                              Acquisition       development       of existing
System            Service(s)  category\a        phase             system
----------------  ----------  ----------------  ----------------  --------------
AH-64D Longbow    Army        1D                MS III (1995);    Modification
Apache                                          IOT&E (1995)
helicopter

Airborne Self-    Navy        1D                FOT&E (1995-      New
Protection                                      96);
Jammer                                          Bosnia (1995)

C-17A Airlifter   Air Force   1D                FOT&E (1996-      New
                                                98);
                                                MS IIIB (1995)

E-3 AWACS Radar   Air Force   1C                AFSARC III        Modification
System                                          (1997);
Improvement                                     IOT&E (1995-96)
Program

F-22 fighter      Air Force   1D                MS III (2003);    New
aircraft                                        IOT&E (2002);
                                                LRIP (1999)

Javelin missile   Army        1D                MS III (1997);    New
                                                LUT (1996);
                                                UE (1996)

Joint
Surveillance
Target Attack
Radar System

E-8 aircraft      Air Force   1D                FOT&E (1997);     New
                                                MS III (1996);
                                                Bosnia (1996)

Common ground     Army        1D                MS III (1998);    New
station                                         IOT&E (1997-
                                                98);
                                                Bosnia (1995)

LPD-17            Navy        1D                MS II (1996);     New
Amphibious                                      EOA-2 (1996);
Assault Ship                                    EOA-1 (1994-95)

M1A2 tank         Army        1D                FOT&E (1995-      Modification
                                                96);
                                                MS III (1994)

Sensor Fuzed      Air Force   1D                FOT&E-1 (1997-    New
Weapon                                          98);
                                                MS III (1996);
                                                IOT&E-2 (1995-
                                                96)

Standard Missile
SM-2

Block IIIB        Navy        II                FOT&E (1997);     Modification
version                                         MS III (1996);
                                                OPEVAL (1996)

Block IV version  Navy        1D                MS III (1997);    Modification
                                                DT/IOT&E (1994)

Tomahawk Weapon
System

Baseline III      Navy        1C                MS III (1998);    Modification
                                                OPEVAL (1998);
                                                IOT&E (1997)

Baseline IV       Navy        1C                MS III (2000);    Modification
                                                OPEVAL (1999-
                                                00);
                                                IOT&E (1999)

V-22              Navy        1D                OPEVAL (1999);    New
                                                OT-IIC (1996)
--------------------------------------------------------------------------------
Legend

AFSARC = Air Force Systems Acquisition Review Council
DT = developmental testing
EOA = early operational assessment
FOT&E = follow-on operational test and evaluation
IOT&E = initial operational test and evaluation
LRIP = low-rate initial production
LUT = limited user test
MS = milestone
OA = operational assessment
OPEVAL = operational evaluation
OT = operational testing
UE = user evaluation

\a The Under Secretary of Defense for Acquisition and Technology (USD
(A&T)) designates major defense acquisition programs as either
acquisition category 1D or 1C.  The milestone decision authority for
category 1D programs is USD (A&T).  The milestone decision authority
for category 1C programs is the Department of Defense (DOD) component
head or, if delegated, the DOD component acquisition executive. 
Category I programs are major defense acquisition programs estimated
to require more than $355 million (fiscal year 1996 constant dollars)
for expenditures in research, development, test, and evaluation, or
more than $2.135 billion (fiscal year 1996 constant dollars) for
procurement.  Category II programs are those that do not meet the
criteria for category I but do meet the criteria for a major system. 
A major system is estimated to require more than $75 million in
fiscal year 1980 constant dollars (approximately $140 million in
fiscal year 1996 constant dollars) for expenditures in research,
development, test, and evaluation, or more than $300 million in
fiscal year 1980 constant dollars (approximately $645 million in
fiscal year 1996 constant dollars) for procurement. 

DOT&E, the service operational test agencies, and the Institute for
Defense Analyses (IDA) personnel agreed that DOT&E was influential in
the testing done on these 13 systems.  In several cases, the
participating agencies vehemently differed on the value of DOT&E's
actions; however, whether DOT&E had an impact on testing (be it
perceived as positive or negative) was not in dispute. 

In conducting our 13 case studies, we assessed the strengths and
weaknesses of the organizational framework in DOD for operational
testing via test agency representatives, an assessment on the origins
and implementation (exemplified by the 13 cases) of the title 10
amendments creating and empowering DOT&E, and a review of the
literature. 

To compile case study data, we interviewed current action officers in
both DOT&E and the appropriate operational test agency and reviewed
documentation provided by the operational test agencies, DOT&E, and
IDA.  Using structured questionnaires, we interviewed 12 DOT&E and 27
operational test agency action officers responsible for the 13
selected systems as well as managers and technical support personnel
in each organization.  In addition, we interviewed the commanders of
each of the service testing agencies and DOT&E.  When possible, we
corroborated information obtained from interviews with documentation,
including test and evaluation master plans, beyond low-rate initial
production reports, defense acquisition executive summary status
reports, defense acquisition memoranda, and interagency
correspondence. 

In Washington, D.C., we obtained data from or performed work at the
Office of the Director of Operational Test and Evaluation, OSD;
Deputy Under Secretary of Defense for Acquisition Reform; Directorate
of Navy Test and Evaluation and Technology Requirements, Office of
the Chief of Naval Operations; Test and Evaluation Management Agency,
Director of Army Staff; Air Force Test and Evaluation Directorate;
and the DOD Office of the Inspector General.  We also reviewed data
and interviewed officials from the Army Operational Test and
Evaluation Command and the Institute for Defense Analyses,
Alexandria, Virginia; the Navy Commander, Operational Test and
Evaluation Force, Norfolk, Virginia; and the Air Force Operational
Test and Evaluation Command, Kirtland Air Force Base, New Mexico. 

The use of a systematic case study framework enabled us to identify
and categorize the types of impacts attributable to DOT&E among the
systems studied.  In addition, this framework enabled us to identify
trends among factors that correlate with DOT&E effectiveness. 
However, we were unable to generalize to all systems subject to
OSD-level oversight.  In light of this limitation, we included only
major (high-cost) systems and systems identified by DOT&E and the
lead operational test agency as having been affected by DOT&E
initiatives.  Moreover, while our methodology and data collection
enabled us to qualitatively assess the impact of DOT&E, it was not
sufficiently rigorous either to evaluate the cost-effectiveness of
DOT&E actions or to determine the deterrent effects, if any, the
Office exerts over the acquisition and testing process.  Finally, our
methodology did not enable an assessment of whether the additional
testing requested by DOT&E was necessary to provide full-rate
production decisionmakers the essential information on a system's
operational effectiveness and suitability or whether the additional
data was worth the time, expense, and resources necessary to obtain
it. 

Our review was performed from June 1996 through March 1997 in
accordance with generally accepted government auditing standards. 


--------------------
\1 Table I.1 lists the lead service, program size, and acquisition or
testing phase for each of the case study systems, as well as whether
the program is a development effort or a modification of an existing
system. 


DESCRIPTION OF 13 CASE STUDY
SYSTEMS
========================================================== Appendix II


   AH-64D LONGBOW APACHE
   HELICOPTER
-------------------------------------------------------- Appendix II:1

The AH-64D Longbow Apache is a remanufactured and upgraded version of
the AH-64A Apache helicopter.  This Army system is equipped with a
mast-mounted fire control radar, fire-and-forget radio frequency
Hellfire missile, and airframe improvements (i.e., integrated
cockpit, improved engines, and global positioning system navigation). 


   AIRBORNE SELF-PROTECTION JAMMER
-------------------------------------------------------- Appendix II:2

The Airborne Self-Protection Jammer is a defensive electronic
countermeasures system using reprogrammable deceptive jamming
techniques to protect tactical aircraft from radar-guided weapons. 
This Navy system is intended to protect Navy and Marine Corps F-18
and F-14 aircraft. 


   C-17A AIRLIFTER
-------------------------------------------------------- Appendix II:3

The C-17A Airlifter provides strategic/tactical transport of all
cargo, including outsized cargo, mostly to main operational bases or
to small, austere airfields, if needed.  Its four-engine turbofan
design enables the transport of large payloads over intercontinental
ranges without refueling.  This Air Force aircraft will replace the
retiring C-141 aircraft and augment the C-130 and C-5 transport
fleets. 


   E-3 AWACS RADAR SYSTEM
   IMPROVEMENT PROGRAM
-------------------------------------------------------- Appendix II:4

The Air Force's E-3 AWACS consists of a Boeing 707 airframe modified
to carry a radome housing a pulse-Doppler radar capable of detecting
aircraft and cruise missiles, particularly at low altitudes.  The
Radar System Improvement Program replaces several components of the
radar to improve detection capability and electronic countermeasures
as well as reliability, availability, and maintainability. 


   F-22 AIR SUPERIORITY FIGHTER
-------------------------------------------------------- Appendix II:5

The F-22 is an air superiority aircraft with a capability to deliver
air-to-ground weapons.  The most significant features include
supercruise, the ability to fly efficiently at supersonic speeds
without using fuel-consuming afterburners, low observability to
adversary systems with the goal to locate and shoot down the F-22,
and integrated avionics to significantly improve the pilot's
battlefield awareness. 


   JAVELIN MISSILE
-------------------------------------------------------- Appendix II:6

The Javelin is a man-portable, antiarmor weapon developed for the
Army and the Marine Corp to replace the aging Dragon system.  It is
designed as a fire-and-forget system comprised of a missile and
reusable command launch unit. 


   JOINT SURVEILLANCE TARGET
   ATTACK RADAR SYSTEM
-------------------------------------------------------- Appendix II:7

The Joint Surveillance Target Attack Radar System is designed to
provide intelligence on moving and stationary targets to Air Force
and Army command nodes in near real time.  The system comprises a
modified Boeing 707 aircraft frame equipped with radar,
communications equipment, and the air component of the data link,
computer workstations, and self-defense suite as well as ground
station modules mounted on Army vehicles. 


   LPD-17 AMPHIBIOUS ASSAULT SHIP
-------------------------------------------------------- Appendix II:8

The LPD-17 will be an amphibious assault ship capable of launching
(1) amphibious assault craft from a well deck and (2) helicopters or
vertical takeoff and landing aircraft from an aft flight deck.  It is
intended to transport and deploy combat and support elements of
Marine expeditionary brigades as a key component of amphibious task
forces. 


   M1A2 ABRAMS MAIN BATTLE TANK
-------------------------------------------------------- Appendix II:9

The M1A2 Abrams main battle tank is an upgrade of the M1A1 and is
intended to improve target acquisition and engagement rates and
survivability while sustaining equivalent operational suitability. 
Specifically, the modified tank incorporates a commander's
independent thermal viewer, a position navigation system, and an
intervehicle command and control system. 


   SENSOR FUZED WEAPON
------------------------------------------------------- Appendix II:10

The Sensor Fuzed Weapon is an antiarmor cluster munition to be
employed by fighter, attack, or bomber aircraft to achieve multiple
kills per pass against armored and support combat formations.  Each
munition contains a tactical munitions dispenser comprising 10
submunitions containing a total of 40 infrared sensing projectiles. 
High-altitude accuracy is to be improved through the incorporation of
a wind-compensated munition dispenser upgrade. 


   STANDARD MISSILE-2
------------------------------------------------------- Appendix II:11

The Standard Missile-2 is a solid propellant-fueled, tail-controlled,
surface-to-air missile fired by surface ships.  It was originally
designed to counter high-speed, high-altitude antiship missiles in an
advanced electronic countermeasures environment.  The block IIIA
version provides improved capacity against low-altitude targets with
an improved warhead.  The block IIIB adds an infrared seeker to the
block IIIA to enhance the missile's capabilities against specific
threats.  These improvements are being made to provide capability
against theater ballistic missiles while retaining its capabilities
against antiair warfare threats. 


   TOMAHAWK WEAPON SYSTEM
------------------------------------------------------- Appendix II:12

The Tomahawk Weapon System is a long-range subsonic cruise missile
for land and sea targets.  The baseline IV upgrade is fitted with a
terminal seeker, video data link, and two-way digital data link.  The
primary
baseline IV configuration is the Tomahawk multimission missile; a
second variant is the Tomahawk hard target penetrator. 


   V-22 OSPREY
------------------------------------------------------- Appendix II:13

The V-22 is a tilt rotor vertical/short takeoff and landing,
multimission aircraft developed to fulfill operational combat
requirements in the Marine Corps and Special Operations Forces. 


DOT&E'S AND DOD'S SYSTEM
ACQUISITION PROCESS
========================================================= Appendix III

DOT&E's role in the system acquisition process does not become
prominent until the latter stages.  As weapon system programs
progress through successive phases of the acquisition process, they
are subject to major decision points called milestones.  The
milestone review process is predicated on the principle that systems
advance to higher acquisition phases by demonstrating that they meet
prescribed technical and performance thresholds.  Figure III.1
illustrates DOD's weapon system acquisition process. 

   Figure III.1:  DOD's Weapon
   System Acquisition Process

   (See figure in printed
   edition.)

Per DOD directive, test and evaluation planning begins in phase 0,
Concept Exploration.  Operational testers are to be involved early to
ensure that the test program for the most promising alternative can
support the acquisition strategy and to ensure the harmonization of
objectives, thresholds, and measures of effectiveness in the
operational readiness document and the test and evaluation master
plan.  Early testing of prototypes in phase I, Program Definition and
Risk Reduction, and early operational assessments are to be
emphasized to assist in identifying risks.  A combined developmental
and operational test approach is encouraged to save time and costs. 
Initial operational test and evaluation is to occur during phase II
to evaluate operational effectiveness and suitability before the
full-rate production decision, milestone III, on all acquisition
category I and II programs.  For all acquisition category I programs
and other programs designated for OSD test and evaluation oversight,
a test and evaluation master plan is prepared and submitted for
approval prior to first milestone review (excluding milestone 0).\1
The master plan is to be updated at milestones when the program has
changed significantly.  DOT&E must approve the test and evaluation
master plan and the more specific operational test plans prior to
their execution.  This process and the required plan approvals
provide DOT&E opportunities to affect the design and execution of
operational testing throughout the acquisition process. 



(See figure in printed edition.)Appendix IV

--------------------
\1 Master plans for acquisition category I programs are to be
submitted to the Director, Test Systems Engineering and Evaluation,
30 days prior to the first milestone.  For all other programs
designated for OSD oversight, the plans must be submitted 90 days
prior to the first milestone. 


COMMENTS FROM THE DEPARTMENT OF
DEFENSE
========================================================= Appendix III



(See figure in printed edition.)



(See figure in printed edition.)



(See figure in printed edition.)


The following are GAO's comments on the September 19, 1997, letter
from the Department of Defense. 

GAO COMMENTS

1.  In prior reviews of individual weapon systems, we have found that
operational testing and evaluation is generally viewed by the
acquisition community as a costly and time-consuming requirement
imposed by outsiders rather than a management tool for more
successful programs.  Efforts to enhance the efficiency of
acquisition, in general--and in operational testing, in
particular--need to be well balanced with the requirement to
realistically and thoroughly test operational suitability and
effectiveness prior to the full-rate production decision.  We
attempted to take a broader view of acquisition reform efficiency
initiatives to anticipate how these departures from past ways of
doing business could impact both the quality of operational testing
and the independence of DOT&E. 

2.  We were asked to assess the impact of DOT&E on the quality and
impact of testing and reported on the Secretary of Defense
initiatives only to the extent they may pose a potential impact on
DOT&E's independence or effectiveness.  Moreover, we did not
recommend or suggest that testers wait until milestone III to
discover problems that could have been learned and corrected earlier. 
Since its inception, DOT&E has been active in test integration and
planning working groups and test and evaluation master plan
development during the earliest phases of the acquisition process. 
In fact, we have long advocated more early testing to demonstrate
positive system performance prior to the low-rate initial production
decision.  DOT&E's early involvement in test planning is appropriate,
necessary, and required by DOD regulations.  In this report we do not
advocate the elimination of DOT&E participation during the early
stages of the acquisition process; rather, we merely observe that
DOT&E participation through the vehicle of working-level program
manager integrated product teams has the potential to complicate
independence and may be increasingly difficult to implement with
declining resources and increasing oversight responsibilities
following milestone III. 

3.  We did not recommend or suggest that DOT&E ignore its statutory
responsibility to review and make recommendations to the Secretary of
Defense on budgetary and financial matters related to operational
test facilities and equipment.  We only observed that in an era of
declining resources, earlier participation, and extended oversight
responsibilities, a decision to assume a larger role in test resource
management planning and leadership is likely to result in tradeoffs
in other responsibilities--the largest being oversight. 

4.  We made this recommendation because DOT&E, the services, and the
program offices did not necessarily agree on the degree to which
system performance requirements have been met in initial operational
test and evaluation.  Furthermore, there was no consensus within the
acquisition community concerning DOT&E's authority to oversee
follow-on operational test and evaluation conducted to ensure that
proposed corrections to previously identified deficiencies were
thoroughly tested and evaluated. 

5.  Under 10 U.S.C.  2399, DOT&E is required to independently report
to Congress whether a major acquisition system has proven to be
operationally suitable and effective prior to the full-rate
production decision.  When follow-on operational test and evaluation
is necessary to test measures intended to correct deficiencies
identified in initial operational test and evaluation, Congress does
not receive an equivalent independent report from DOT&E that
concludes, based on required follow-on operational test and
evaluation, whether or not a major system has improved sufficiently
to be considered both operationally suitable and effective. 



RELATED GAO PRODUCTS
============================================================ Chapter 0

Tactical Intelligence:  Joint STARS Full-Rate Production Decision Was
Premature and Risky (GAO/NSIAD-97-68, Apr.  25, 1997). 

Weapons Acquisition:  Better Use of Limited DOD Acquisition Funding
Would Reduce Costs (GAO/NSIAD-97-23, Feb.  13, 1997). 

Airborne Self-Protection Jammer (GAO/NSIAD-97-46R, Jan.  29, 1997). 

Army Acquisition:  Javelin Is Not Ready for Multiyear Procurement
(GAO/NSIAD-96-199, Sept.  26, 1996). 

Tactical Intelligence:  Accelerated Joint STARS Ground Station
Acquisition Strategy Is Risky (GAO/NSIAD-96-71, May 23, 1996). 

Electronic Warfare (GAO/NSIAD-96-109R, Mar.  1, 1996). 

Longbow Apache Helicopter:  System Procurement Issues Need to Be
Resolved (GAO/NSIAD-95-159, Aug.  24, 1995). 

Electronic Warfare:  Most Air Force ALQ-135 Jammers Procured Without
Operational Testing (GAO/NSIAD-95-47, Nov.  22, 1994). 

Weapons Acquisition:  Low-Rate Initial Production Used to Buy Weapon
Systems Prematurely (GAO/NSIAD-95-18, Nov.  21, 1994). 

Acquisition Reform:  Role of Test and Evaluation in System
Acquisition Should Not Be Weakened (GAO/T-NSIAD-94-124, Mar.  22,
1994). 

Test and Evaluation:  The Director, Operational Test and Evaluation's
Role in Test Resources (GAO/NSIAD-90-128, Aug.  27, 1990). 

Adequacy of Department of Defense Operational Test and Evaluation
(GAO/T-NSIAD-89-39, June 16, 1989). 

Weapons Testing:  Quality of DOD Operational Testing and Reporting
(GAO/PEMD-88-32BR, July 26, 1988). 


*** End of document. ***