Combating Nuclear Smuggling: Additional Actions Needed to
Ensure Adequate Testing of Next Generation Radiation Detection
Equipment (18-SEP-07, GAO-07-1247T).						 
                                                                 
The Department of Homeland Security's (DHS) Domestic Nuclear
Detection Office (DNDO) is responsible for addressing the threat
of nuclear smuggling. Radiation detection portal monitors are
key elements in our national defenses against such threats. DHS
has sponsored testing to develop new monitors, known as advanced
spectroscopic portal (ASP) monitors. In March 2006, GAO
recommended that DNDO conduct a cost-benefit analysis to
determine whether the new portal monitors were worth the
additional cost. In June 2006, DNDO issued its analysis. In
October 2006, GAO concluded that DNDO did not provide a sound
analytical basis for its decision to purchase and deploy ASP
technology and recommended further testing of ASPs. DNDO
conducted this ASP testing at the Nevada Test Site (NTS) between
February and March 2007. GAO's statement addresses the test
methods DNDO used to demonstrate the performance capabilities of
the ASPs and whether the NTS test results should be relied upon
to make a full-scale production decision.	 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-07-1247T						        
    ACCNO:   A76428						        
  TITLE:     Combating Nuclear Smuggling: Additional Actions
Needed to Ensure Adequate Testing of Next Generation Radiation
Detection Equipment							 
     DATE:   09/18/2007 
  SUBJECT:   Cost analysis
             Data collection
             Hazardous substances
             Homeland security
             Inspection
             Nevada Test Site
             Operational testing
             Product evaluation
             Radiation monitoring
             Smuggling
             Standards
             Testing

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-07-1247T



This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to [email protected]. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Testimony: 

Before the Subcommittee on Oversight and Investigations, Committee on 
Energy and Commerce, House of Representatives: 

United States Government Accountability Office: 

GAO: 

For Release on Delivery Expected at 10:00 a.m. EDT: 

Tuesday, September 18, 2007: 

Combating Nuclear Smuggling: 

Additional Actions Needed to Ensure Adequate Testing of Next Generation 
Radiation Detection Equipment: 

Statement of Gene Aloise, Director: 

Natural Resources and Environment: 

Keith Rhodes, Chief Technologist: 

Combating Nuclear Smuggling: 

GAO-07-1247T: 

GAO Highlights: 

Highlights of GAO-07-1247T, a testimony before the Subcommittee on 
Oversight and Investigations, Committee on Energy and Commerce, House 
of Representatives. 

Why GAO Did This Study: 

The Department of Homeland Securityï¿½s (DHS) Domestic Nuclear Detection 
Office (DNDO) is responsible for addressing the threat of nuclear 
smuggling. Radiation detection portal monitors are key elements in our 
national defenses against such threats. DHS has sponsored testing to 
develop new monitors, known as advanced spectroscopic portal (ASP) 
monitors. In March 2006, GAO recommended that DNDO conduct a cost-
benefit analysis to determine whether the new portal monitors were 
worth the additional cost. In June 2006, DNDO issued its analysis. In 
October 2006, GAO concluded that DNDO did not provide a sound 
analytical basis for its decision to purchase and deploy ASP technology 
and recommended further testing of ASPs. DNDO conducted this ASP 
testing at the Nevada Test Site (NTS) between February and March 2007. 

GAOï¿½s statement addresses the test methods DNDO used to demonstrate the 
performance capabilities of the ASPs and whether the NTS test results 
should be relied upon to make a full-scale production decision. 

What GAO Found: 

Based on our analysis of DNDOï¿½s test plan, the test results, and 
discussions with experts from four national laboratories, we are 
concerned that DNDOï¿½s tests were not an objective and rigorous 
assessment of the ASPsï¿½ capabilities. Our concerns with the DNDOï¿½s test 
methods include the following: 

* DNDO used biased test methods that enhanced the performance of the 
ASPs. Specifically, DNDO conducted numerous preliminary runs of almost 
all of the materials, and combinations of materials, that were used in 
the formal tests and then allowed ASP contractors to collect test data 
and adjust their systems to identify these materials. It is highly 
unlikely that such favorable circumstances would present themselves 
under real world conditions. 

* DNDOï¿½s NTS tests were not designed to test the limitations of the 
ASPsï¿½ detection capabilitiesï¿½a critical oversight in DNDOï¿½s original 
test plan. DNDO did not use a sufficient amount of the type of 
materials that would mask or hide dangerous sources and that ASPs would 
likely encounter at ports of entry. DOE and national laboratory 
officials raised these concerns to DNDO in November 2006. However, DNDO 
officials rejected their suggestion of including additional and more 
challenging masking materials because, according to DNDO, there would 
not be sufficient time to obtain them based on the deadline imposed by 
obtaining Secretarial Certification by June 26. 2007. By not 
collaborating with DOE until late in the test planning process, DNDO 
missed an important opportunity to procure a broader, more 
representative set of well-vetted and characterized masking materials. 

* DNDO did not objectively test the performance of handheld detectors 
because they did not use a critical CBP standard operating procedure 
that is fundamental to this equipmentï¿½s performance in the field. 

Because of concerns raised that DNDO did not sufficiently test the 
limitations of ASPs, DNDO is attempting to compensate for weaknesses in 
the original test plan by conducting additional studiesï¿½essentially 
computer simulations. While DNDO, CBP, and DOE have now reached an 
agreement to wait and see whether the results of these studies will 
provide useful data regarding the ASPsï¿½ capabilities, in our view and 
those of other experts, computer simulations are not as good as actual 
testing with nuclear and masking materials. 

What GAO Recommends: 

GAO recommends that, among other things, the Secretary of Homeland 
Security delay a full-scale production decision of ASPs until all 
relevant studies and tests have been completed, and determine in 
cooperation with U.S. Customs and Border Protection(CBP), the 
Department of Energy (DOE), and independent reviewers, whether 
additional testing is needed. 

[hyperlink, http://www.gao.gov/cgi-bin/getrpt?GAO-07-1247T]. 

To view the full product, including the scope and methodology, click on 
the link above. For more information, contact Gene Aloise, at (202) 512-
3841 or [email protected]. 

[End of section] 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss our assessment of the testing 
of advanced spectroscopic portal (ASP) monitors conducted by the 
Domestic Nuclear Detection Office (DNDO). The results of these tests, 
including the methods by which they were conducted, are critically 
important because they will serve as the primary support for a required 
Department of Homeland Security (DHS) Secretarial Certification of the 
performance of this equipment and, in turn, authorization to spend up 
to $1.2 billion for the full-scale production of the next generation of 
radiation detection technology to be deployed to U.S. ports of entry. 

The radiation portal monitors in use today can detect the presence of 
radiation, but they cannot distinguish between types of radiological 
material. For example, they cannot tell the difference between harmless 
products that emit radiation, such as ceramic tile, and dangerous 
materials, such as highly enriched uranium that could be used to 
construct a nuclear weapon. DNDO is primarily responsible for 
preventing unauthorized nuclear or radiological materials from entering 
the United States. U.S. Customs and Border Protection (CBP) is 
responsible for screening cargo as it enters the nation at our borders, 
which includes operating radiation detection equipment to intercept 
dangerous nuclear and radiological materials. The Department of Energy 
(DOE) is the primary agency responsible for the international 
deployment of radiation detection equipment. 

CBP's standard procedures require incoming cargo to pass through a 
radiation portal monitor to screen for the presence of radiation. This 
"primary inspection" alerts CBP officers when a radioactive threat 
might be present. If there is a potential threat, CBP procedures 
require a "secondary inspection." To confirm the presence of radiation, 
this secondary inspection usually includes a second screening by a 
radiation portal monitor as well as a screening by CBP officers using 
handheld radioactive isotope identification devices (RIIDs). These 
devices are used to differentiate between types of radioactive material 
to determine if the radiation being detected is dangerous. Both the 
radiation portal monitors and handheld devices are limited in their 
abilities to detect and identify nuclear material. 

DNDO asserts that false alarms, or "nuisance alerts," result in large 
numbers of secondary inspections--especially at high-volume ports of 
entry. CBP officials believe that the number of secondary inspections 
required by the currently deployed system are resource intensive and 
could be reduced, allowing CBP officers to focus on other border 
enforcement responsibilities such as illegal immigration and drug 
interdiction. Importantly, however, these officials acknowledged that 
the current system provides the best possible radiological and nuclear 
screening coverage available and that it does not have a significant 
impact on commerce. 

DHS would like to improve the capabilities of its portal monitors so 
that they can perform the dual roles of detecting radiation and 
identifying radiological materials. In this regard, DHS has sponsored 
research, development, and testing activities designed to create ASP 
portal monitors capable of performing both functions. In July 2006, DHS 
awarded contracts to three vendors to develop the ASPs' capabilities. 
These awards were based mainly on performance tests conducted at DHS's 
Nevada Test Site in 2005, where ten competing ASP vendors' monitors 
were evaluated. At the same time, three currently deployed portal 
monitors that use polyvinyl toluene plastic scintillators, known as 
PVTs, were also tested. 

To ensure that DHS's substantial investment in radiation detection 
technology yields the greatest possible level of detection capability 
at the lowest possible cost, in a March 2006 GAO report,[Footnote 1] we 
recommended that once the costs and capabilities of ASPs were well 
understood, and before any of the new equipment was purchased for 
deployment, the Secretary of DHS work with the Director of DNDO to 
analyze the costs and benefits of deploying ASPs. Further, we 
recommended that this analysis focus on determining whether any 
additional detection capability provided by the ASPs was worth the 
considerable additional costs. In response to our recommendation, DNDO 
issued its cost-benefit analysis in May 2006[Footnote 2] and an 
updated, revised version in June 2006. According to senior agency 
officials, DNDO believes that the basic conclusions of its cost-benefit 
analysis showed that the new ASP monitors are a sound investment for 
the U.S. government. 

In an October 2006 GAO report[Footnote 3], we concluded that DNDO's 
cost benefit analysis did not provide a sound basis for DNDO's decision 
to purchase and deploy ASP technology because it relied on assumptions 
of the anticipated performance level of ASPs instead of actual test 
data, and that it did not justify DHS's plan to spend $1.2 billion to 
purchase and deploy ASPs. We also reported that DNDO did not assess the 
likelihood that ASPs would either misidentify or fail to detect nuclear 
or radiological material. Rather, it focused its analysis on reducing 
the time necessary to screen traffic at border check points and reduce 
the impact of any delays on commerce. We recommended that DNDO conduct 
further testing of ASPs and the currently deployed PVTs before spending 
additional funds to purchase ASPs. 

Mr. Chairman, my remarks today focus on the tests conducted by DNDO at 
the Nevada Test Site between February and March of this year and the 
test methods DNDO used to demonstrate the performance capabilities of 
the ASPs. Specifically, I will discuss how the tests were conducted at 
the Nevada Test Site, and whether these test results should be relied 
upon to support Secretarial Certification or to make a full-scale 
production decision. I will also discuss current DNDO testing efforts 
and how they may impact future decision making. 

In conducting our review, we analyzed DNDO's test plans and procedures 
and interviewed senior DNDO officials responsible for managing the ASP 
program, including the development and testing of ASP monitors. We 
observed DNDO's testing conducted at the Nevada Test Site and the New 
York Container Terminal. We obtained information on DNDO's test methods 
from National Institute of Standards and Technology (NIST) officials 
and discussed the efficacy of DNDO's test methods with experts from 
NIST, DOE, the private sector, and four national laboratories. We also 
met with senior CBP and DOE officials as the main end users of portal 
monitor equipment. We conducted our review in Washington, D.C. from 
March to September 2007 in accordance with generally accepted 
government auditing standards. 

In Summary: 

Based on our analysis of DNDO's test plan, the test results, and 
discussions with experts from four national laboratories, we are 
concerned that DNDO's tests were not an objective and rigorous 
assessment of the ASPs capabilities. Our concerns with DNDO's test 
methods include the following: 

* DNDO used biased test methods that enhanced the performance of the 
ASPs. Specifically, DNDO conducted numerous preliminary runs of almost 
all of the materials, and combinations of materials, that were used in 
the formal tests and then allowed ASP contractors to collect test data 
and adjust their systems to identify these materials. It is highly 
unlikely that such favorable circumstances would present themselves 
under real world conditions. 

* DNDO's tests conducted at the Nevada Test Site were not designed to 
test the limitations of the ASPs' detection capabilities--a critical 
oversight in DNDO's original test plan. DNDO did not use a sufficient 
amount of the type of materials that would mask or hide dangerous 
sources and that ASPs would likely encounter at ports of entry. DOE and 
national laboratory[Footnote 4] officials raised these concerns to DNDO 
in November 2006. However, DNDO officials rejected their suggestion of 
including additional and more challenging masking materials because, 
according to DNDO, it would not be able to obtain such materials in 
time to meet the Secretarial Certification deadline. By not 
collaborating with DOE until late in the test planning process, DNDO 
missed an important opportunity to procure a broader, more 
representative set of well-vetted and characterized masking materials. 

* DNDO did not objectively test the performance of handheld detectors 
because it did not use a critical CBP standard operating procedure that 
is fundamental to this equipment's performance in the field. 

As a result of concerns raised that DNDO's NTS tests did not 
sufficiently test the limitations of ASPs, DNDO is now attempting to 
compensate for weaknesses in the original test plan by conducting 
additional testing known as injection studies--essentially computer 
simulations. While DNDO, CBP, and DOE have now reached an agreement to 
wait and see whether the results of these studies will provide useful 
data regarding the ASPs' capabilities, in our view and those of other 
experts, computer simulations are not as good as actual testing with 
nuclear and masking materials. 

We are recommending that the Secretary of DHS delay certification until 
all tests and studies have been completed and validated, and all test 
results have been provided to relevant parties, including CBP and DOE. 
If DNDO, CBP and DOE determine that additional testing is needed, then 
an independent group within DHS should be formed to conduct this 
testing. In addition, the results of the tests and analyses should be 
reported to the appropriate congressional committees before large scale 
purchases are made. 

Background: 

The Department of Homeland Security Appropriations Act for Fiscal Year 
2007 states that "none of the funds appropriatedï¿½shall be obligated for 
full scale procurement of [ASP] monitors until the Secretary of 
Homeland Security has certifiedï¿½that a significant increase in 
operational effectiveness will be achieved."[Footnote 5] DNDO noted 
that certification would meet DHS guidelines for the review and 
approval of complex acquisitions. Specifically, DNDO stated that the 
Secretary's decision would be made in the context of DHS "Key Decision 
Point 3," which details the review and approval necessary for DHS 
acquisition programs to move from the "Capability Development and 
Demonstration" phase to the "Production and Deployment Phase." 

To meet the statutory requirement to certify the ASPs will provide a 
"significant increase in operational effectiveness," and requirements 
outlined in DHS Management Directive 1400, DNDO, with input from 
subject matter experts, developed a series of tests intended to 
demonstrate, among other things, ASP performance and deployment 
readiness. The tests were conducted at several venues, including the 
Nevada Test Site, the New York Container Terminal, the Pacific 
Northwest National Laboratory, and five ports of entry. DNDO stated 
that its request for full-scale production approval would be based upon 
completed and documented results of these tests. To meet the 
Secretary's goal of deploying 225 ASPs by the end of calendar year 
2008, Secretarial Certification was scheduled for June 26, 2007. 

To guide the test operations, DNDO defined a set of Critical 
Operational Issues that outlined the tests' technical objectives and 
provided the baseline to measure demonstrated effectiveness. The 
purpose of the Critical Operational Issue 1 is to "verify operational 
effectiveness" of ASPs and determine whether "ASP systems significantly 
increase operational effectiveness relative to the current generation 
detection and identification system." DNDO conducted a series of tests 
at the Nevada Test Site, the single focus of which, according to DNDO, 
was to resolve Critical Operational Issue 1. According to DNDO, these 
tests began in February 2007 and concluded in March 2007. DNDO's Nevada 
Test Site test plan, dated January 12, 2007, identified three primary 
test objectives comparing the operational effectiveness of the ASP 
systems with existing detection and identification systems at current 
high-volume operational thresholds. Specifically, DNDO sought to 
determine the ASPs' probability to (1) detect and identify nuclear and 
radiological threats (2) discriminate threat and non-threat 
radionuclides in primary [screening positions], and (3) detect and 
identify threat radionuclides in the presence of non-threat 
radionuclides. 

How the Tests at the Nevada Test Site Were Conducted: 

The Nevada Test Site test plan had two key components. First, DNDO 
developed guidelines for basic test operations and procedures, 
including test goals and expectations, test tasks and requirements, and 
roles and responsibilities of personnel involved in the testing, 
including the ASP contractors. The second component involved the 
National Institute of Standards and Technology developing test 
protocols that defined, among other things, how many times a container 
carrying test materials would need to be driven through portal monitors 
in order to obtain statistically relevant results. 

DNDO's tests at the Nevada Test Site were designed to compare the 
current system--using PVTs in primary inspections and a PVT and RIID 
combination in secondary inspections--to other configurations including 
PVTs in primary and ASPs in secondary, and ASPs in both primary and 
secondary inspection positions. DNDO tested three ASPs and four PVTs. 
The ASP vendors included Thermo, Raytheon, and Canberra. The PVT 
vendors included SAIC, TSA, and Ludlum. According to the test plan, to 
the greatest extent possible, PVT, ASP, and RIID handheld devices would 
be operated consistent with approved CBP standard operating procedures. 

Prior to "formal" collection of the data that would be used to support 
the resolution of Critical Operational Issue 1, DNDO conducted a series 
of tests it referred to as "dry runs" and "dress rehearsals." The 
purpose of the dry runs was to, among other things, verify ASP systems' 
software performance against representative test materials and allow 
test teams and system contractors to identify and implement software 
and hardware improvements to ASP systems. The purpose of the dress 
rehearsals was to observe the ASPs in operation against representative 
test scenarios and allow the test team to, among other things: 

* develop confidence in the reliability of the ASP system so that 
operators and data analysts would know what to expect and what data to 
collect during the formal test, 

* collect sample test data, and: 

* determine what errors were likely to occur in the data collection 
process and eliminate opportunities for error. 

In addition to improving ASP performance through dry runs and dress 
rehearsals conducted prior to formal data collection, ASP contractors 
were also significantly involved in the Nevada Test Site test 
processes. Specifically, the test plan stated that "[ASP] contractor 
involvement was an integral part of the NTS test events to ensure the 
systems performed as designed for the duration of the test." 
Furthermore, ASP contractors were available on site to repair their 
system at the request of the test director and to provide quality 
control support of the test data through real time monitoring of 
available data. DNDO stated that Pacific Northwest National Laboratory 
representatives were also on site to provide the same services for the 
PVT systems. 

DNDO conducted its formal tests in two phases. The first, called Phase 
1, was designed to support resolution of Critical Operational Issue 1 
with high statistical confidence. DNDO told us on multiple occasions 
and in a written response that only data collected during Phase 1 would 
be included in the final report presented to the Secretary to request 
ASP certification. According to DNDO, the second, called Phase 3, 
provided data for algorithm development which targeted specific and 
known areas in need of work and data to aid in the development of 
secondary screening operations and procedures. According to DNDO 
documentation, Phase 3 testing was not in support of the full-scale 
production decision. Further, DNDO stated that Phase 3 testing 
consisted of relatively small sample sizes since the data would not 
support estimating the probability of detection with a high confidence 
level. 

On May 30, 2007, following the formal tests and the scoring of their 
results, DNDO told GAO that it had conducted additional tests that DNDO 
termed "Special Testing." The details of these tests were not outlined 
in the Nevada Test Site test plan. On June 20, 2007, DNDO provided GAO 
with a test plan document entitled "ASP Special Testing" which 
described the test sources used to conduct the tests but did not say 
when the tests took place. According to DNDO, special testing was 
conducted throughout the formal Phase 1 testing process and included 12 
combinations of threat, masking, and shielding materials that differed 
from "dry run," "dress rehearsal," and formal tests. DNDO also stated 
that the tests were "blind," meaning that neither DNDO testing 
officials nor the ASP vendors knew what sources would be included in 
the tests. According to DNDO, these special tests were recommended by 
subject matter experts outside the ASP program to address the 
limitations of the original NTS test plan, including: 

* available time and funding resources, 

* special nuclear material sources, and: 

* the number of test configurations that could be incorporated in the 
test plan, including source isotope and activity, shielding materials 
and thicknesses, masking materials, vehicle types, and measurement 
conditions. 

Unlike the formal tests, National Institute of Standards and Technology 
officials were not involved in determining the number of test runs 
necessary to obtain statistically relevant results for the special 
tests. 

DNDO's Test Methods Raise Concerns Regarding the Reliability of Test 
Results: 

Based on our analysis of DNDO's test plan, the test results, and 
discussions with experts from four national laboratories, we are 
concerned that DNDO used biased test methods that enhanced the 
performance of the ASPs. In the dry runs and dress rehearsals, DNDO 
conducted many preliminary runs of radiological, nuclear, masking, and 
shielding materials so that ASP contractors could collect data on the 
radiation being emitted, and modify their software accordingly. 
Specifically, we are concerned because almost all of the materials, and 
most combinations of materials, DNDO used in the formal tests were 
identical to those that the ASP contractors had specifically set their 
ASPs to identify during the dry runs and dress rehearsals. It is highly 
unlikely that such favorable circumstances would present themselves 
under real world conditions. 

A key component of the NTS tests was to test the ASPs' ability to 
detect and identify dangerous materials, specifically when that 
material was masked or "hidden" by benign radioactive materials. Based 
on our analysis, the masking materials DNDO used at NTS did not 
sufficiently test the performance limits of the ASPs. DOE national 
laboratory officials raised similar concerns to DNDO after reviewing a 
draft of the test plan in November 2006. These officials stated that 
the masking materials DNDO planned to use in its tests did not emit 
enough radiation to mask the presence of nuclear materials in a 
shipping container and noted that many of the materials that DOE 
program officials regularly observe passing through international ports 
emit significantly higher levels of radiation than the masking 
materials DNDO used for its tests. 

DNDO officials told us that the masking materials used at the Nevada 
Test Site represented the average emissions seen in the stream of 
commerce at the New York Container Terminal. However, according to data 
accumulated as part of DOE's program to secure international ports (the 
Megaports program), a significant percentage of cargo passing through 
one European port potentially on its way to the United States has 
emission levels greater than the average radiation level for cargo that 
typically sets off radiation detection alarms. Importantly, DNDO 
officials told us that the masking materials used at the Nevada Test 
Site were not intended to provide insight into the limits of ASP 
detection capabilities. Yet, DNDO's own test plan for "ASP Special 
Testing" states, "The DNDO ASP NTS Test Plan was designed toï¿½ measure 
capabilities and limitations in current ASP systems." 

In addition, the NTS tests did not objectively test the ASPs against 
the currently deployed radiation detection system. DNDO's test plan 
stated that, to the greatest extent possible, PVT, ASP, and RIID 
handheld devices would be operated consistent with approved CBP 
standard operating procedures. However, after analyzing test results 
and procedures used at the Nevada Test Site, CBP officials determined 
that DNDO had, in fact, not followed a key CBP procedure. In 
particular, if a threat is identified during a secondary screening, or 
if the result of the RIID screening isn't definitive, CBP procedures 
require officers to send the data to CBP's Laboratories and Scientific 
Services for further guidance. DNDO did not include this critical step 
in its formal tests. CBP officials also expressed concern with DNDO's 
preliminary test results when we met with them in May 2007. 

In regards to the special tests DNDO conducted, based on what DNDO has 
told us and our own evaluation of the special test plan, we note that: 

* because DNDO did not consult NIST on the design of the blind tests, 
we do not know the statistical significance of the results, and: 

* the tests were not entirely blind because some of the nuclear 
materials used in the blind tests were also used to calibrate the ASPs 
on a daily basis. 

During the course of our work, CBP, DOE, and national laboratory 
officials we spoke to voiced concern about their lack of involvement in 
the planning and execution of the Nevada Test Site tests. We raised our 
concerns about this issue and those of DOE and CBP to DNDO's attention 
on multiple occasions. In response to these concerns, specifically 
those posed by DOE, DNDO convened a conference on June 27, 2007, of 
technical experts to discuss the Nevada test results and the methods 
DNDO used to test the effects of masking materials on what the ASPs are 
able to detect. As a result of discussions held during that meeting, 
subject matter experts agreed that computer-simulated injection studies 
could help determine the ASPs' ability to detect threats in the 
presence of highly radioactive masking material. 

According to a Pacific Northwest National Laboratory report submitted 
to DNDO in December 2006[Footnote 6], injection studies are 
particularly useful for measuring the relative performance of 
algorithms, but their results should not be construed as a measure of 
(system) vulnerability. To assess the limits of portal monitors' 
capabilities, the Pacific Northwest National Laboratory report states 
that actual testing should be conducted using threat objects immersed 
in containers with various masking agents, shielding, and cargo. DNDO 
officials stated at the meeting that further testing could be 
scheduled, if necessary, to fully satisfy DOE concerns. 

On July 20, 2007, DHS Secretary Chertoff notified certain members of 
the Congress that he planned to convene an independent expert panel to 
review DNDO's test procedures, test results, associated technology 
assessments, and cost-benefit analyses to support the final decision to 
deploy ASPs. In making this announcement, Secretary Chertoff noted the 
national importance of developing highly effective radiation detection 
and identification capabilities as one of the main reasons for seeking 
an independent review of DNDO's actions. On August 30, 2007, the DHS 
Undersecretary for Management recommended that the Secretary of 
Homeland Security delay Secretarial Certification of ASPs for an 
additional two months. According to DHS, the current delay is in order 
to provide CBP more time to field ASP systems, a concern CBP had raised 
early in our review. 

Conclusions: 

Effectively detecting and identifying radiological or nuclear threats 
at U.S. borders and ports of entry is a vital matter of national 
security, and developing new and advanced technology is critical to 
U.S. efforts to prevent a potential attack. However, it is also 
critical to fully understand the strengths and weaknesses of any next 
generation radiation detection technology before it is deployed in the 
field and to know, to the greatest extent possible, when or how that 
equipment may fail. 

In our view, the tests conducted by DNDO at the Nevada Test Site 
between February and March 2007 used biased test methods and were not 
an objective assessment of the ASPs' performance capabilities. We 
believe that DNDO's test methods--specifically, conducting dry runs and 
dress rehearsals with contractors prior to formal testing--enhanced the 
performance of the ASPs beyond what they are likely to achieve in 
actual use. Furthermore, the tests were not a rigorous evaluation of 
the ASPs' capabilities, but rather a developmental demonstration of ASP 
performance under controlled conditions which did not test the 
limitations of the ASP systems. 

As a result of DNDO's test methods and the limits of the tests-- 
including a need to meet a secretarial certification deadline and the 
limited configurations of special nuclear material sources, masking, 
and shielding materials used--we believe that the results of the tests 
conducted at the Nevada Test Site do not demonstrate a "significant 
increase in operational effectiveness" relative to the current 
detection system, and cannot be relied upon to make a full-scale 
production decision. 

Recommendations: 

We recommend that the Secretary of Homeland Security take the following 
actions: 

* Delay Secretarial Certification and full-scale production decisions 
of the ASPs until all relevant tests and studies have been completed 
and limitations to these tests and studies have been identified and 
addressed. Furthermore, results of these tests and studies should be 
validated and made fully transparent to DOE, CBP, and other relevant 
parties. 

* Once the tests and studies have been completed, evaluated, and 
validated, DHS should determine in cooperation with CBP, DOE, and other 
stakeholders including independent reviewers, if additional testing is 
needed. 

* If additional testing is needed, the Secretary should appoint an 
independent group within DHS, not aligned with the ASP acquisition 
process, to conduct objective, comprehensive, and transparent testing 
that realistically demonstrates the capabilities and limitations of the 
ASP system. This independent group would be separate from the recently 
appointed independent review panel. 

* Finally, the results of the tests and analyses should be reported to 
the appropriate congressional committees before large scale purchases 
of ASP's are made. 

Mr. Chairman, this concludes our prepared statement. We would be happy 
to respond to any questions you or other members of the subcommittee 
may have. 

GAO Contact and Staff Acknowledgements: 

For further information about this testimony, please contact me, Gene 
Aloise, at (202) 512-3841 or at [email protected]. Contact points for our 
Offices of Congressional Relations and Public Affairs may be found on 
the last page of this statement. Erika D. Carter, Alison O'Neill, Jim 
Shafer, Daren Sweeney, and Eugene Wisnoski made key contributions to 
this statement. 

Related GAO Products: 

Combating Nuclear Smuggling: DHS's Decision to Procure and Deploy the 
Next Generation of Radiation Detection Equipment Is Not Supported by 
Its Cost-Benefit Analysis. GAO-07-581T. Washington, D.C.: March.14, 
2007. 

Nuclear Nonproliferation: Focusing on the highest Priority Radiological 
Sources Could Improve DOE's Efforts to Secure Sources in Foreign 
Countries. GAO-07-580T. Washington, D.C.: March. 13, 2007. 

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the 
National Laboratories' Test Results on Radiation Portal Monitors in 
Support of DNDO's Testing and Development Program. GAO-07-347R. 
Washington, D.C.: March 9, 2007. 

Technology Assessment: Securing the Transport of Cargo Containers. GAO- 
06-68SU. Washington, D.C.: January 25, 2006. 

Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the 
Purchase of New Radiation Detection Portal Monitors Was Not Based on 
Available Performance Data and Did Not Fully Evaluate All the Monitors' 
Costs and Benefits. GAO-07-133R. Washington, D.C.: October 17, 2006. 

Combating Nuclear Terrorism: Federal Efforts to Respond to Nuclear and 
Radiological Threats and to Protect Emergency Response Capabilities 
Could Be Strengthened. GAO-06-1015. Washington, D.C.: September 21, 
2006. 

Border Security: Investigators Transported Radioactive Sources Across 
Our Nation's Borders at Two Locations. GAO-06-940T. Washington, D.C.: 
July 7, 2006. 

Combating Nuclear Smuggling: Challenges Facing U.S. Efforts to Deploy 
Radiation Detection Equipment in Other Countries and in the United 
States. GAO-06-558T. Washington, D.C.: March 28, 2006. 

Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation 
Detection Equipment at U.S. Ports-of-Entry, but Concerns Remain. GAO- 
06-389. Washington, D.C.: March 22, 2006. 

Combating Nuclear Smuggling: Corruption, Maintenance, and Coordination 
Problems Challenge U.S. Efforts to Provide Radiation Detection 
Equipment to Other Countries. GAO-06-311. Washington, D.C.: March 14, 
2006. 

Combating Nuclear Smuggling: Efforts to Deploy Radiation Detection 
Equipment in the United States and in Other Countries. GAO-05-840T. 
Washington, D.C.: June 21, 2005. 

Preventing Nuclear Smuggling: DOE Has Made Limited Progress in 
Installing Radiation Detection Equipment at Highest Priority Foreign 
Seaports. GAO-05-375. Washington, D.C.: March 31, 2005. 

Homeland Security: DHS Needs a Strategy to Use DOE's Laboratories for 
Research on Nuclear, Biological, and Chemical Detection and Response 
Technologies. GAO-04-653. Washington, D.C.: May 24, 2004. 

Homeland Security: Summary of Challenges Faced in Targeting Oceangoing 
Cargo Containers for Inspection. GAO-04-557T. Washington, D.C.: March 
31, 2004). 

Homeland Security: Preliminary Observations on Efforts to Target 
Security Inspections of Cargo Containers. GAO-04-325T. Washington, 
D.C.: December 16, 2003. 

Homeland Security: Radiation Detection Equipment at U.S. Ports of 
Entry. GAO-03-1153TNI. Washington, D.C.: September 30, 2003. 

Homeland Security: Limited Progress in Deploying Radiation Detection 
Equipment at U.S. Ports of Entry. GAO-03-963. Washington, D.C.: 
September 4, 2003). 

Container Security: Current Efforts to Detect Nuclear Materials, New 
Initiatives, and Challenges. GAO-03-297T. Washington, D.C.: November 
18, 2002. 

Customs Service: Acquisition and Deployment of Radiation Detection 
Equipment. GAO-03-235T. Washington, D.C.: October 17, 2002. 

Nuclear Nonproliferation: U.S. Efforts to Combat Nuclear Smuggling. GAO-
02-989T. Washington, D.C.: July 30, 2002. 

Nuclear Nonproliferation: U.S. Efforts to Help Other Countries Combat 
Nuclear Smuggling Need Strengthened Coordination and Planning. GAO-02- 
426. Washington, D.C.: May 16, 2002. 

Footnotes: 

[1] GAO, Combating Nuclear Smuggling: DHS Has Made Progress Deploying 
Radiation Detection Equipment at U.S. Ports of Entry, but Concerns 
Remain, GAO-06-389 (Washington, D.C.: Mar.22, 2006). 

[2] DNDO, Cost Benefit Analysis for Next Generation Passive Radiation 
Detection of Cargo at the Nation's Border Crossings, May 30, 2006. 

[3] GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to 
Support the Purchase of New Radiation Detection Portal Monitors Was Not 
Based on Available Performance Data and Did Not Fully Evaluate All the 
Monitors' Cost and Benefits, GAO-07-133R (Washington, D.C.: Oct. 17, 
2006). 

[4] DOE manages the largest laboratory system of its kind in the world. 
The mission of DOE's 22 laboratories has evolved. Originally created to 
design and build atomic weapons, these laboratories have since expanded 
to conduct research in many disciplines--from high-energy physics to 
advanced computing. 

[5] Department of Homeland Security Appropriations Act for Fiscal Year 
2007, Pub. L. No. 109-295, tit. IV, 120 Stat. 1355, 1376 (October 4, 
2006). 

[6] PNNL, Energy Window Ratio Algorithms For Plastic Scintillator 
Portal Monitors: Development, Deployment and Performance, PNNL-16283 
(Richland, WA: December, 2006). 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Gloria Jarmon, Managing Director, [email protected] (202) 512-4400: 

U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Susan Becker, Acting Manager, [email protected] (202) 512-4800: 

U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***