Chemical and Biological Defense: Program Planning and Evaluation Should
Follow Results Act Framework (Testimony, 05/24/2000, GAO/T-NSIAD-00-180).

During the last decade, concerns about the possible use of chemical and
biological weapons in both military and civilian settings have led
Congress to boost funding for new and expanded initiatives to counter
these threats. For example, the Chemical and Biological Defense Program
appropriation has more than doubled from $388 million in fiscal year
1996 to $791 million. This testimony discusses whether a framework
exists to monitor and evaluate the impact of increased funding on
protecting soldiers from the effects of chemical and biological weapons.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  T-NSIAD-00-180
     TITLE:  Chemical and Biological Defense: Program Planning and
	     Evaluation Should Follow Results Act Framework
      DATE:  05/24/2000
   SUBJECT:  Chemical warfare
	     Biological warfare
	     Defense capabilities
	     Military research and development
	     Performance measures
	     Strategic planning
	     Program evaluation
	     Combat readiness
IDENTIFIER:  DOD Chemical and Biological Defense Program
	     GPRA
	     Government Performance and Results Act

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Testimony.                                               **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO/T-NSIAD-00-180

We are pleased to be here to discuss our August 1999 report on the
Department of Defense's application of the Government Performance and
Results Act in its Chemical and Biological Defense Program. In the last
decade, concerns about the possible use of chemical and biological weapons
in both military and civilian settings led Congress to increase funding for
new and expanded initiatives to counter these threats. For example, the
Chemical and Biological Defense Program appropriation has more than doubled
from $388 million in fiscal year 1996 to $791 million. Today we will address
whether a framework exists to monitor and evaluate the impacts of the
increased funding on protecting service members from the effects of chemical
and biological warfare agents.

Since the Persian Gulf War, members of Congress have raised concerns
regarding the adequacy of technology used by the Department of Defense (DOD)
to detect, identify, prepare for, and protect troops against chemical and
biological weapons. In 1993, the National Defense Authorization Act for
Fiscal Year 1994 (P.L. 103-160) directed the Secretary of Defense to take
actions to improve the Department's chemical and biological defense
capabilities, including coordination and integration of all chemical and
biological defense programs into what is now the Chemical and Biological
Defense Program. More recently, concerns that terrorists might use chemical
or biological devices led Congress to authorize the federal government to
improve domestic capabilities to respond to such incidents. With the
initiation of these domestic preparedness programs in fiscal year 1997,
federal research and development efforts to develop nonmedical chemical and
biological defense technology expanded considerably, and they continue to
grow.

In 1993 Congress enacted the Government Performance and Results Act
(commonly referred to as the Results Act). The legislation was designed to
have agencies focus on the performance and results of their programs, rather
than on program activities and resources, as they had traditionally done.
Congress sought to shift federal management and oversight from its
preoccupation with program staffing, activity levels, and tasks completed to
program results-that is, to the real differences that federal programs make
in people's lives. The outcome-oriented principles of the Results Act, which
Congress anticipated would be institutionalized and practiced at all
organizational levels in federal agencies, include (1) establishing general
goals and quantifiable, measurable, outcome-oriented performance goals and
related measures; (2) developing strategies for achieving the goals,
including strategies for overcoming or mitigating major impediments to goal
achievement; (3) ensuring that goals at lower organizational levels align
with and support the general goals; and (4) identifying the resources that
will be required to achieve the goals.

We examined the extent to which DOD has applied the Results Act's
outcome-oriented principles to the Chemical and Biological Defense Program,
focusing in particular on research and development, testing, and evaluation
(henceforth referred to as R&D) activities that lead to new defense
technologies and capabilities. Specifically, we assessed whether (1) Results
Act principles can and should be applied to the Chemical and Biological
Defense Program's R&D activities, (2) current Chemical and Biological
Defense Program planning and evaluation practices follow the Results Act
framework, and (3) organizations executing the R&D activities have
incorporated Results Act principles in their program planning and evaluation
practices. Moreover, we examined whether DOD has implemented our
recommendation to development of a performance plan for the Chemical and
Biological Defense Program based on the outcome-oriented management
principles embodied in the Results Act.

SUMMARY

Congressional reports and administrative guidance indicate that DOD programs
such as the Chemical and Biological Defense Program should follow the
Results Act's outcome-oriented principles, including the establishment of
general goals; quantifiable, measurable, outcome-oriented performance goals;
and related measures. Moreover, research organizations such as the Research
Roundtable, the National Academy of Sciences, the National Academy of
Engineering, and the Institute of Medicine have concluded that both applied
and basic research programs supported by the federal government could be
evaluated meaningfully in accordance with the Results Act framework.

DOD's Chemical and Biological Defense Program in general, and its R&D
activities in particular, have not incorporated key Results Act principles.
Program goals are vague and unmeasurable and the performance measures
emphasize activities rather than impacts. In the absence of explicit and
measurable goals, it is difficult to assess the impact of the Program on
warfighters' ability to survive, fight, and win in a chemical and biological
environment.

Chemical and Biological Defense Program research and development
organizations have incorporated Results Act principles inconsistently. Only
one of three DOD organizations that engage in R&D activities in support of
the Chemical and Biological Defense Program has adopted the Results Act
planning and evaluation tools. The remaining two cited either the
utilization of equivalent planning tools or the unique challenges of
evaluating research and development activities as reasons for not adopting
the Results Act processes.

Our August 1999 report recommended that the Secretary of Defense direct that
actions be taken to develop a performance plan for the Chemical and
Biological Defense Program based on the outcome-oriented management
principles embodied in the Results Act. DOD concurred with the
recommendation and agreed to develop a full detailed and coordinated plan
for inclusion in its next DOD Chemical and Biological Defense Program Annual
Report to Congress. Nevertheless, the next Report to Congress in March 2000
did not contain a plan containing the elements outlined in our
recommendation. In the March 2000 Report to Congress, DOD established a new
set of program goals and stated specific technology and systems goals will
be included in a performance plan to be completed during calendar year 2000
and included in the next annual report to Congress.

BACKGROUND

The DOD's Chemical and Biological Defense Program addresses three nonmedical
defensive capabilities: contamination avoidance, protection, and
decontamination. These areas comprise the DOD's framework for developing
nonmedical program requirements. When changes in doctrine, training, or
organizational structure cannot satisfy warfighters' needs in these areas,
DOD seeks new equipment through the research, development, and acquisition
cycle. Chemical and biological defense funding is divided between the
program's two primary activities: R&D and procurement. Of the Chemical and
Biological Defense Program appropriation of $791 million in fiscal year
2000, $410 million (52 percent) is for R&D and the remaining $381 million
(48 percent) for procurement.

Consistent with the National Defense Authorization Act for Fiscal Year 1994,
the Secretary of Defense assigned responsibility for the overall
coordination and integration of the Chemical and Biological Defense Program
to a single office headed by the Deputy Assistant to the Secretary of
Defense for Chemical and Biological Defense. The office is responsible for
approving all planning, programming, and budgeting documents; ensuring
coordination between the medical and nonmedical chemical and biological
defense efforts; and overseeing management oversight in accordance with the
law. The Deputy Assistant to the Secretary of Defense manages program
research, development, and acquisition efforts for Chemical and Biological
Defense. The Deputy Assistant Secretary is also Executive Secretary of a
Steering Committee that is responsible for oversight of the program. In
August 1999 the Steering Committee was comprised of the Directors of the
Defense Threat Reduction Agency and Defense Research and Engineering as well
as their top officials responsible for chemical and biological defense.
Since our report was issued, the membership of the Committee has been
expanded to include representation for the joint Chiefs of Staff, the
Assistant Secretary of Defense for Strategy and Threat Reduction, and the
Assistant Secretary for Health Affairs, as depicted in figure 1.

Figure 1: DOD Chemical and Biological Defense Program Oversight Structure
and Selected DOD Organizations Executing Research and Development

Source DOD

As illustrated in figure 1, the program's DOD research and development
organizations in the execution of the program include the Soldier and
Biological Chemical Command, the Joint Program Office for Biological
Defense, and the Defense Advanced Research Projects Agency.

THE CHEMICAL AND BIOLOGICAL DEFENSE PROGRAM SHOULD FOLLOW THE RESULTS ACT'S
OUTCOME-ORIENTED PRINCIPLES

Congressional and administrative guidance indicate that DOD programs such as
the Chemical and Biological Defense Program should follow the
outcome-oriented principles of the Results Act. The 1997 Quadrennial Defense
Review, which serves as DOD's overall strategic planning document, directs
DOD organizations at all levels to review their objectives to ensure that
they link to the goals and objectives of the Quadrennial Defense Review and
to ensure that Results Act performance plans indicate progress toward
meeting Quadrennial Defense Review goals. DOD guidance for implementing the
Results Act states that the goals, objectives, measures of success,
quantifiable performance measures, and program outcome evaluations of
subordinate organizations should be linked to the DOD goals articulated in
the Quadrennial Defense Review and made operational in DOD's annual
performance plan. Chemical and Biological Defense Program R&D activities
support DOD's second goal to "prepare now for an uncertain future by
pursuing a focused modernization effort that maintains U.S. qualitative
superiority in key warfighting capabilities."

Congress has recognized that successful implementation on the Results Act in
science agencies would not come quickly or easily. Nonetheless, several
professional science organizations have concluded that the Results Act
principles can or should be applied to R&D. The Research Roundtable, a group
of federal researchers and managers representing a cross section of
departments and agencies, concluded in 1995 that the results of a research
program's performance could be measured. The Army Research Laboratory was
designated as a pilot project for performance measurement under the act and
ultimately outlined an evaluation approach that made use of three pillars:
metrics, peer review, and customer feedback. In 1999, the Committee on
Science, Engineering, and Public Policy of the National Academy of Sciences,
the National Academy of Engineering, and the Institute of Medicine concluded
that both applied and basic research programs supported by the federal
government could be evaluated meaningfully on a regular basis.

DOD'S CHEMICAL AND BIOLOGICAL DEFENSE PROGRAM IN GENERAL, AND ITS R&D
ACTIVITIES IN PARTICULAR, HAVE NOT INCORPORATED KEY RESULTS ACT PRINCIPLES

Results Act outcome-oriented principles have not been widely applied by
either Chemical and Biological Defense Program planners or executing
organizations. Chemical and biological defense research and development
outcomes and impacts are not being systematically measured because the
Program lacks both quantifiable performance goals and measurable objectives.

Although DOD has taken the initial and necessary step of articulating
Chemical and Biological Defense Program goals, the goals are not articulated
in a manner consistent with Results Act principles. The stated goals are
vague and unmeasurable, and they fail to articulate specific desired
impacts. A Results Act framework requires that managers define a related set
of long-term strategic goals, annual agency goals, and measurable
performance goals for each program. In 1999, the five Chemical and
Biological Defense Program goals were to

-- deter chemical and biological weapon use by denying military advantage to
an enemy through a combination of avoidance, protection, decontamination,
and medical support capabilities, allowing U.S. forces to operate largely
unimpeded by chemical and biological attacks and their subsequent effects;

-- address the most probable chemical and biological weapon threats that
could be encountered in regional conflicts and field capabilities to the
forces required for two major theater wars;

-- ensure the chemical and biological weapon threat drives chemical and
biological defense research, development, and acquisition programs;

-- emphasize a joint service approach to chemical and biological defense
R&D, and acquisition; and

-- complete critical R&D and acquisition of improved chemical and biological
detection, identification, and warning systems; individual and collective
protection systems; and medical support and decontamination systems.

Measuring the first goal is unachievable, determining a deterrence effect is
problematic, and attributing the specific rationale for the deterrence is
unrealistic. The second, third, and fourth goals address the size, focus,
and coordination of the program-not program outcomes. Together, these goals
direct that the program be sufficiently large to address the needs resulting
from two major theater wars; sufficiently focused to address the likely
validated threats; and sufficiently coordinated to capitalize on
efficiencies and other benefits of joint requirements determination,
research, development, and procurement. The objective of the fifth goal is
measurable but addresses program outputs without discussing program outcomes
or impacts (such as decreased defensive vulnerabilities or increased
operational capabilities). The completion of R&D or procurement cannot be
assumed to result in a positive impact on the defensive posture or
operational flexibility of U.S. forces. While the completion of these
activities may generate benefits for U.S. troops, in the absence of valid,
reliable measures, the contributions of R&D or procurement cannot be
determined.

Program planners cite the execution of technology development plans, and the
completion of defense technology objectives and advanced concept technology
demonstrations as measures of progress toward program goals. Program
planners cited a number of supporting plans as being "in the spirit of the
Results Act," even though not specifically assessing outcomes and impacts.
For example, DOD's Chemical and Biological Defense Annual Report to Congress
and the Joint Service Nuclear, Biological, and Chemical Defense Research,
Development, and Acquisition Plan are updated annually and include detailed
metrics and time lines reflecting the program performance in developing new
defense technologies. Technology development plans track progress toward
defense technology objectives and advanced concept technology demonstrations
that, when achieved, DOD claims will create new operational capabilities. In
addition, Chemical and Biological Defense Program planners cited ongoing
programmatic peer reviews, such as Technology Area Review Assessments, as
additional means to measure progress toward meeting program goals.

We do not agree that the conduct of an advanced concept technology
demonstration measures the impact of the Chemical and Biological Defense
Program on the warfighter. Advanced concept technology demonstrations
represent a means for rapidly introducing new technologies and reducing the
time from the start of a program to the system's initial operational
capability. However, the demonstration of a new technology may not by itself
result in the effective and safe deployment of a military capability in
support of the warfighter. Moreover, as we previously reported, DOD has not
always emphasized the need to complete concept and product development or
testing before production, thus increasing the risk of approving advanced
concept technology demonstrations in support of chemical and biological
defense that include immature technologies and then prematurely starting
production.

We also do not agree that peer reviews measure the impact of the program on
the warfighter. Technology Area Review Assessments are peer reviews
conducted by the Director, Defense Research and Engineering on each of DOD's
12 science and technology programs-one being, chemical and biological
defense. These peer reviews address progress toward achieving defense
technology objectives and form the basis of DOD's performance in science and
technology. However, the application of the assessments to generate
performance measures of DOD's science and technology programs-such as
chemical and biological defense-is limited by several factors. First, the
measure is limited because these peer reviews only address defense
technology objectives. Funding for these objectives, however, comprises less
than 50 percent of total funding for applied and advanced technology
development research and development. Thus, the Results Act ratings do not
capture the majority of the Chemical and Biological Defense Program's R&D
activities. Second, the focus of Technology Area Review Assessments is on
budgets, schedules, and technical performance. The reviews do not measure
technology transition from the laboratory to the battlefield. Lastly, the
peer reviews do not measure improvements in the ability of U.S. troops to
survive, fight, and win in a chemical and biological environment.

THE PROGRAM'S RESEARCH AND DEVELOPMENT ORGANIZATIONS HAVE INCORPORATED
RESULTS ACT PRINCIPLES INCONSISTENTLY

The three DOD organizations that execute or contribute to the research and
development goals of the Chemical and Biological Defense Program vary in
their use of the Results Act principles to plan and assess their activities.
The Soldier and Biological Chemical Command is the only R&D organization to
systematically apply results act principles. The Soldier and Biological
Chemical Command has demonstrated that the Results Act principles can be
integrated into the planning and evaluation process of an organization
conducting research and development for the Chemical and Biological Defense
Program. The Command's strategic plan for fiscal years 1998 - 2004 is driven
by and linked with the strategic plans of DOD, the Army, and the Army
Materiel Command. Its strategic planning model directly links the attainment
of its vision with the development of goals and enabling strategies-followed
by the execution of the strategies and measurement of performance. Separate
measures were developed to assess goal achievement as well as progress
toward goal achievement.

The performance plan for fiscal years 1998 - 2004 identifies performance
measures for each Command goal and performance goals for each strategy. The
performance measures address both accomplishments and progress toward
accomplishments. Examples of quantitative measures of research and
development accomplishments include (1) the percentage of new chemical and
biological systems that meet survivability requirements, (2) the percentage
of nonexempt acquisitions receiving waivers from performance specifications,
and (3) the percentage of Command science and technology programs
transitioning to joint service and Army development programs with user
validation through modeling, wargames, or similar methods. Command officials
noted that identification of measures in the research and development has
been an ongoing challenge and continues to evolve.

In contrast, neither the Defense Advanced Research Projects Agency, nor the
Joint Program Office for Biological Defense has developed a performance
plan. The reasons cited for not incorporating the Results Act's principles
into their program planning or evaluation systems were that current DOD
planning processes were equivalent to those of the act, resulting in plans
that were "in the spirit" of the Results Act and that the unique nature of
R&D activities did not lend itself to the act's performance measurement and
evaluation. The Joint Program Office cites the conduct of advanced concept
technology demonstrations as measures of its performance. Defense Advanced
Research Projects Agency officials maintained that the nature of the
Agency's mission -- to pursue long-term, far-reaching, and
high-risk/high-payoff technology and systems for military systems in the
distant future - does not lend itself to the application of performance
measurement. In December 1998, the Defense Management Council agreed and
notified the Agency that it was exempt from the Results Act requirements.

CONCLUSIONS

Chemical and biological defense research and development outcomes and
impacts are not being systematically measured. The Chemical and Biological
Defense Program lacks both quantifiable performance measures and measurable
objectives. In the absence of measures of program impacts and measurable
objectives, progress toward achieving program goals cannot be determined.
Program planning consists of a series of technology development plans
leading to specific equipment items. Managers cite activity measures and
technology demonstrations as measures of the program's contribution. These
planning and programming steps are appropriate and necessary, but they are
insufficient for quantifying outcomes and impacts. Current measures do not
assess the incremental changes attributable, in whole or in part, to the
Chemical and Biological Defense Program that improve warfighters' ability to
survive, fight, and win in a chemical and biological environment.

Results Act outcome-oriented principles have not been widely applied by
either Chemical and Biological Defense Program planners or executing
organizations. The use of these principles can enable managers and those
overseeing the program to quantify the relative success of the program and
of component projects in satisfying requirements across different activities
(e.g., point detection, early warning, warning and reporting, modeling).
Impact measures can provide a planning tool to allocate finite Chemical and
Biological Defense Program resources among competing sets of unmet
requirements.

RECOMMENDATION

In August 1999, we recommended that the Secretary of Defense direct the
development of a performance plan for the Chemical and Biological Defense
Program based on the outcome-oriented management principles embodied in the
Results Act. We specified that the plan should be agreed to and supported by
the relevant R&D organizations and incorporated in DOD's Chemical and
Biological Defense Annual Report to Congress. Specifically, the plan should
(1) establish explicit and outcome-oriented goals linked to warfighters'
ability to survive, fight, and win in a chemical and biological environment;
(2) identify quantitative or qualitative performance measures that can be
used to assess progress toward goal achievement; (3) describe how
performance data would be validated; (4) describe how R&D activities of
participating DOD and non-DOD organizations are coordinated to achieve
program goals; and (5) identify human capital, financial, and resource
challenges or external factors that limit the ability of the program to
achieve its goals.

DOD RESPONSE

DOD agreed with our recommendation to develop a performance plan and stated
it would develop a strategic plan more closely aligned with the tenets of
the Results Act and publish that plan in the next DOD Chemical and
Biological Defense Annual Report to Congress. Nevertheless, the March 2000
Report to Congress does not contain a performance plan. DOD has defined
seven new program goals and stated that more specific technology and systems
goals will be included in a performance plan under development. The steps
taken and promised in the March 2000 Report to the Congress still reflect
only partial compliance with the first of the four outcome-oriented
principles by failing even to identify quantifiable, measurable,
outcome-oriented performance goals. DOD states that specific technology and
systems goals will be included in a performance plan to be completed during
calendar year 2000 and included in the next annual report to Congress.

Thus concludes our formal statement. If you or other members of the
committee have any questions, we will be pleased to answer them. For future
contacts regarding this testimony, please contact Kwai-Cheung Chan at (202)
512-3652. Individuals making key contributors to this assignment were Sushil
Sharma, Jeffrey Harris and Weihsueh Chiu.

(713064)

ORDERS BY INTERNET

For information on how to access GAO reports on the INTERNET, send an e-mail
message with "info" in the body to

HYPERLINK mailto:[email protected] [email protected]

or visit GAO's World Wide Web Home Page at

HYPERLINK http://www.gao.gov http://www.gao.gov

TO REPORT FRAUD, WASTE, AND ABUSE

IN FEDERAL PROGRAMS

Contact one:
*** End of document ***