Elections: Federal Programs for Accrediting Laboratories That	 
Test Voting Systems Need to Be Better Defined and Implemented	 
(09-SEP-08, GAO-08-770).					 
                                                                 
The 2002 Help America Vote Act (HAVA) created the Election	 
Assistance Commission (EAC) and assigned both it and the National
Institute of Standards and Technology (NIST) responsibilities for
accrediting laboratories that test voting systems. NIST assesses 
a laboratory's technical qualifications and makes recommendations
to EAC, which makes a final accreditation decision. In view of	 
the continuing concerns about voting systems and the important	 
roles that NIST and EAC play in accrediting the laboratories that
test these systems, GAO was asked to determine whether each	 
organization has defined an effective approach for accrediting	 
laboratories that test voting systems and whether each is	 
following its defined approach. To accomplish this, GAO compared 
NIST and EAC policies, guidelines, and procedures against	 
applicable legislation and guidance, and reviewed both agencies' 
efforts to implement them.					 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-770 					        
    ACCNO:   A84074						        
  TITLE:     Elections: Federal Programs for Accrediting Laboratories 
That Test Voting Systems Need to Be Better Defined and		 
Implemented							 
     DATE:   09/09/2008 
  SUBJECT:   Documentation					 
	     Elections						 
	     Eligibility determinations 			 
	     Evaluation criteria				 
	     Federal regulations				 
	     Institution accreditation				 
	     Laboratories					 
	     Operational testing				 
	     Program evaluation 				 
	     Program management 				 
	     Standards						 
	     Systems analysis					 
	     Systems evaluation 				 
	     Systems integrity					 
	     Systems testing					 
	     Technology 					 
	     Testing						 
	     Voting						 
	     NIST National Voluntary Laboratory 		 
	     Accreditation Program				 
                                                                 
	     Electronic Voting System				 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-770

   

This text file was formatted by the U.S. Government Accountability 
Office (GAO) to be accessible to users with visual impairments, as part 
of a longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to [email protected]. 

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

Report to the Chairman, Committee on House Administration, House of 
Representatives: 

United States Government Accountability Office: 

GAO: 

September 2008: 

Elections: 

Federal Programs for Accrediting Laboratories That Test Voting Systems 
Need to Be Better Defined and Implemented: 

Accreditation of Voting System Testing Laboratories: 

GAO-08-770: 

GAO Highlights: 

Highlights of GAO-08-770, a report to the Chairman, Committee on House 
Administration, House of Representatives. 

Why GAO Did This Study: 

The 2002 Help America Vote Act (HAVA) created the Election Assistance 
Commission (EAC) and assigned both it and the National Institute of 
Standards and Technology (NIST) responsibilities for accrediting 
laboratories that test voting systems. NIST assesses a laboratoryï¿½s 
technical qualifications and makes recommendations to EAC, which makes 
a final accreditation decision. In view of the continuing concerns 
about voting systems and the important roles that NIST and EAC play in 
accrediting the laboratories that test these systems, GAO was asked to 
determine whether each organization has defined an effective approach 
for accrediting laboratories that test voting systems and whether each 
is following its defined approach. To accomplish this, GAO compared 
NIST and EAC policies, guidelines, and procedures against applicable 
legislation and guidance, and reviewed both agenciesï¿½ efforts to 
implement them. 

What GAO Found: 

NIST has largely defined and implemented an approach for accrediting 
voting system testing laboratories that incorporates many aspects of an 
effective program. In particular, its approach addresses relevant HAVA 
requirements and reflects relevant laboratory accreditation guidance, 
including standards accepted by the international standards community. 
However, NISTï¿½s defined approach does not, for example, cite explicit 
qualifications for the persons who conduct accreditation technical 
assessments, as called for in federal accreditation program guidance. 
Instead, NIST officials said that they rely on individuals who have 
prior experience in reviewing such laboratories. Further, even though 
the EAC requires that laboratory accreditation be based on demonstrated 
capabilities to test against the latest voting system standards, NISTï¿½s 
defined approach has not always cited these current standards. As a 
result, two of the four laboratories accredited to date were assessed 
using assessment tools that were not linked to the latest standards. 
Moreover, available documentation for the four laboratory assessments 
was not sufficient to determine how the checklists were applied and how 
decisions were reached. According to NIST officials, the four 
laboratories were consistently assessed. Moreover, they said that they 
intend to evolve NISTï¿½s accreditation approach to, for example, clearly 
provide for sufficient documentation of how accreditation reviews are 
conducted and decisions are reached. However, they had yet to develop 
specific plans for accomplishing this. EAC recently developed a draft 
laboratory accreditation program manual, but this draft manual does not 
adequately define all aspects of an effective approach, and it was not 
used in the four laboratory accreditations performed to date. 
Specifically, while this draft manual addresses relevant HAVA 
requirements, such as the requirement for the commissioners to vote on 
the accreditation of any laboratory that NIST recommends for 
accreditation, it does not include a methodology governing how 
laboratories are to be evaluated or criteria for granting 
accreditation. Because the manual was not approved at the time EAC 
accredited four laboratories, these accreditations were governed by a 
more broadly defined accreditation review process that was described in 
correspondence sent to each laboratory and a related document receipt 
checklist. As a result, these accreditations were based on review steps 
that were not sufficiently defined to permit them to be executed in a 
repeatable manner. According to EAC officials, including the official 
who conducted the accreditation reviews for the four laboratories, 
using the same person to conduct the reviews ensured that the steps 
performed on the first laboratory were repeated on the other three. 
However, given that both the steps and the results were not documented, 
GAO could not verify this. EAC officials stated that they intend to 
evolve the program manual over time and apply it to future 
accreditations and reaccreditations. However, they did not have 
specific plans for accomplishing this. Further, although EAC very 
recently approved an initial version of its program manual, this did 
not occur until after EAC provided comments, and GAO had finalized, 
this report. 

What GAO Recommends: 

GAO is making recommendations to NIST and EAC aimed at further defining 
and implementing their respective accreditation programs in a way that 
better ensures that voting system laboratory accreditations are 
performed consistently and are verifiable. NIST and EAC generally 
agreed with the need for their respective programs to continuously 
improve and both sought clarification on the reportï¿½s recommendations, 
which GAO has added. 

To view the full product, including the scope and methodology, click on 
[hyperlink,http://www.gao.gov/cgi-bin/getrpt?]. For more information, 
contact Randolph C. Hite at (202) 512-3439 or [email protected]. 

[End of section] 

Contents: 

Letter: 

Results in Brief: 

Background: 

NIST Has Defined and Implemented an Accreditation Approach That 
Reflects Relevant Standards but Is Missing Details Needed for 
Consistent and Verifiable Implementation: 

EAC's Recently Drafted Accreditation Approach and Its Earlier Performed 
Laboratory Accreditations Lack Key Effectiveness Factors and Features: 

Conclusions: 

Recommendations for Executive Action: 

Agency Comments and Our Evaluation: 

Appendix I: Objectives, Scope, and Methodology: 

Appendix II: Comments from the National Institute of Standards and 
Technology: 

Appendix III: Comments from the Election Assistance Commission: 

Appendix IV: GAO Contact and Staff Acknowledgments: 

Tables: 

Table 1: EAC and NIST Responsibilities under HAVA: 

Table 2: Summary of Extent to Which EAC Draft Approach Addresses NIST- 
Identified Accreditation Factors: 

Table 3: Summary of Extent to Which EAC Satisfies Features Common to 
Federal Accreditation Programs: 

Figures: 

Figure 1: A Voting System Life Cycle Model: 

Figure 2: Overall NIST and EAC Accreditation Processes: 

Figure 3: VSTL Accreditation Program Activities: 

Abbreviations: 

EAC: Election Assistance Commission: 

HAVA: Help America Vote Act: 

ISO: International Organization for Standardization: 

NIST: National Institute of Standards and Technology: 

NVLAP: National Voluntary Laboratory Accreditation Program: 

VSTL: voting system testing laboratory: 

VSS: Voting Systems Standards: 

VVSG: Voluntary Voting System Guidelines: 

United States Government Accountability Office: 

Washington, DC 20548: 

September 9, 2008: 

The Honorable Robert A. Brady: 
Chairman: 
Committee on House Administration: 
House of Representatives: 

Dear Mr. Chairman: 

In the wake of the 2000 and 2004 general elections, we issued a series 
of reports and testified[Footnote 1] on virtually every aspect of our 
nation's overall election system, including the many challenges and 
opportunities associated with various types of voting systems. In this 
regard, we emphasized that voting systems alone were neither the sole 
contributor nor solution to the problems that were experienced during 
the 2000 and 2004 elections, and that the overall election system 
depended on the effective interplay of people, process, and technology 
and involved all levels of government. Among many things, we 
specifically reported in 2001[Footnote 2] that no federal entity was 
responsible for accrediting the laboratories that tested voting 
systems, and we raised the establishment of such an entity as a matter 
for congressional consideration. 

Subsequently, Congress passed the Help America Vote Act (HAVA), which 
created the Election Assistance Commission (EAC) and assigned both it 
and the National Institute of Standards and Technology (NIST) separate 
but related responsibilities for accrediting laboratories that test 
voting systems.[Footnote 3] In general, NIST is responsible for 
assessing a laboratory's technical qualifications and making an 
accreditation recommendation to EAC, while EAC is to use the assessment 
results and recommendation, along with its own review of related 
laboratory capabilities, to reach an accreditation decision. In 2004 
and 2007, NIST and EAC established voting system testing laboratory 
accreditation programs, respectively. To date, EAC has accredited four 
laboratories. In view of the continuing concerns about voting systems 
and the important roles that both NIST and EAC play in accrediting the 
laboratories that test these systems, you asked us to determine whether 
NIST and EAC have each defined an effective laboratory accreditation 
approach and whether each is following its defined approach. 

To accomplish this, we reviewed NIST and EAC policies, guidelines, and 
procedures governing voting system testing laboratory accreditation, 
deaccreditation, and reaccreditation and compared them, as appropriate, 
to applicable statute, such as HAVA, and guidance published by NIST, 
the International Organization for Standardization, and us. We then 
compared NIST and EAC actions and artifacts that were used for 
accrediting four voting system testing laboratories to their respective 
policies, guidelines, and procedures. We did not review a fifth 
laboratory because NIST was in the process of assessing it when we 
started our review, and had yet to recommend the laboratory to EAC for 
final accreditation. In addition, we interviewed officials from NIST, 
EAC, and the four laboratories to understand and clarify approaches 
taken, documentation provided, and decisions reached. 

We conducted this performance audit at EAC and NIST offices in 
Washington, D.C., and Gaithersburg, Maryland, respectively, from 
September 2007 to September 2008 in accordance with generally accepted 
government auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to provide 
a reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a reasonable 
basis for our findings and conclusions based on our audit objectives. 
Further details of our objective, scope, and methodology are included 
in appendix I. 

Results in Brief: 

NIST has largely defined and implemented an approach for accrediting 
voting system testing laboratories that incorporates many aspects of an 
effective program. In particular, its approach addresses relevant HAVA 
requirements and reflects relevant laboratory accreditation standards 
that have been accepted by the international community. However, NIST's 
defined approach does not cite explicit qualifications or training 
requirements for accreditation technical assessors, which, according to 
NIST, is a characteristic of an effective accreditation program. 
Instead, the program has relied, in part, on a small and specialized 
field of potential assessors in the voting system arena. Further, until 
recently, NIST's laboratory accreditation guidance cited different 
versions of required standards. As a result, two of the four accredited 
laboratories were assessed using checklists linked to the current 
standards and two were not. Moreover, NIST's documentation of these 
assessments was not sufficient to determine how the checklists were 
applied and how decisions were reached. According to NIST officials, 
the four laboratories were consistently assessed, and they intend to 
ensure that sufficient documentation is produced to show how 
assessments are conducted and decisions are reached. However, they said 
that they do not have a documented plan for accomplishing this. 

EAC recently developed a draft of a voting system testing laboratory 
accreditation program manual, but the draft manual does not adequately 
define all aspects of an effective approach and was not used in 
accrediting the four laboratories. EAC's draft manual addresses 
applicable HAVA requirements, but does not include either a methodology 
governing how laboratories are to be reviewed or certain criteria 
relevant to granting accreditation. Because the draft manual was not 
available for laboratory assessments until recently, EAC instead used a 
more broadly defined accreditation review process contained in 
correspondence with the laboratories and a related checklist that were 
not specific enough to ensure that review steps were executed in a 
repeatable manner. According to EAC officials, using the same person to 
conduct the reviews ensured that the reviews were consistently 
performed. However, because both the steps and the results were not 
documented, we could not verify this. In addition, EAC officials told 
us that they intend to evolve the program manual over time and apply it 
to future accreditations and reaccreditations; however, they do not 
have a documented plan for accomplishing this.[Footnote 4] 

To assist NIST and EAC in evolving their respective voting system 
testing laboratory accreditation programs, we are making 
recommendations to NIST and EAC to develop and execute plans, with 
specific tasks, milestones, resources, and measures, which are aimed at 
adding consistency and specificity to their defined approaches and 
ensuring that the approaches are fully implemented and documented. 

NIST and EAC provided written comments on a draft of this report, 
signed by the Deputy Director of NIST and the Executive Director of 
EAC, respectively. More specifically, NIST stated that it appreciated 
our careful review of its voting system testing laboratory 
accreditation program and added that it generally concurs with our 
findings that this program must continue to evolve and improve. NIST 
also provided comments that were intended to clarify the current status 
of the program relative to three of our findings. For various reasons 
discussed in the agency comments section of this report, we do not 
believe that these comments affect any of the three findings. 
Therefore, we have not modified the report's presentation of them. In 
light of NIST's recent actions to address one of the findings, we 
updated our report to reflect these actions and have removed the 
associated recommendation from our final report. Further, in order to 
avoid the possibility of any misunderstanding about the actions needed 
to address one other finding, we have slightly modified the 
recommendation associated with it. 

With respect to EAC's comments, the commission described our review and 
report as helpful to the commission as it works to fully develop and 
implement its voting system testing laboratory accreditation program. 
It also stated that it agrees with the report's conclusions that 
additional written internal procedures, standards, and documentation 
are needed to ensure more consistent and repeatable implementation of 
the program. Further, it stated that it generally accepts our 
recommendations, adding that it will work hard to implement them. 
However, it sought clarification about two of the recommendations. In 
response, we have slightly modified both recommendations to avoid any 
confusion as to their intent. Both the NIST and EAC comments are 
discussed in detail in the agency comments section of this report, and 
are reprinted in their entirety in appendixes II and III, respectively. 

Background: 

All levels of government share responsibility in the overall U.S. 
election system. At the federal level, Congress has authority under the 
Constitution to regulate presidential and congressional elections and 
to enforce prohibitions against specific discriminatory practices in 
all federal, state, and local elections. Congress has passed 
legislation that addresses voter registration, absentee voting, 
accessibility provisions for the elderly and persons with disabilities, 
and prohibitions against discriminatory practices.[Footnote 5] At the 
state level, individual states are responsible for the administration 
of both federal elections and their own elections. States regulate the 
election process, including, for example, the adoption of voluntary 
voting system guidelines, the state certification and acceptance 
testing of voting systems, ballot access, registration procedures, 
absentee voting requirements, the establishment of voting places, the 
provision of election day workers, and the counting and certification 
of the vote. 

In total, the overall U.S. election system can be seen as an assemblage 
of 55 distinct election systems--those of the 50 states, 4 U.S. 
territories, and the District of Columbia. Further, although election 
policy and procedures are legislated primarily at the state level, 
states typically have decentralized election systems, so that the 
details of administering elections are carried out at the city or 
county levels, and voting is done at the local level. As we reported in 
2001,[Footnote 6] local election jurisdictions number more than 10,000, 
and their sizes vary enormously--from a rural county with about 200 
voters to a large urban county, such as Los Angeles County, where the 
total number of registered voters for the 2000 elections exceeded the 
registered voter totals in 41 states. Further, these thousands of 
jurisdictions rely on many different types of voting methods that 
employ a wide range of voting system makes, models, and versions. 
Because of the prominent role played by electronic voting systems, 
testing these systems against national standards is critical to 
ensuring their security and reliability. Equally critical is ensuring 
that the laboratories that perform these tests are competent to carry 
out testing activities. 

The Overall U.S. Election System Depends on Effective Interactions 
among People, Processes, and Technology: 

In the United States today, most votes are cast and counted by 
electronic voting systems, and many states require use of systems that 
have been certified nationally or by state authorities. However, voting 
systems are but one facet of a multifaceted, continuous overall 
election system that involves the interplay of people, processes, and 
technology during the entire life of a system. All levels of 
government, as well as commercial voting system manufacturers and 
system testing laboratories, play key roles in ensuring that voting 
systems perform as intended. 

Electronic voting systems are typically developed by manufacturers, 
then purchased as commercial, off-the-shelf products and operated by 
state and local election administrators. Viewed at a high level, these 
activities make up three phases of a system life cycle: product 
development, acquisition, and operations. (See fig. 1.) Key processes 
that span these life cycle phases include managing the people, 
processes, and technologies within each phase and across phases, and 
testing the systems and components during and at the end of each phase. 
Additionally, voting system standards are important through all of the 
phases because they provide criteria for developing, testing, and 
acquiring voting systems, and they specify the necessary documentation 
for operating the systems. 

Figure 1: A Voting System Life Cycle Model: 

This figure is a horizontal flow chart showing a voting system life 
cycle model. 

[See PDF for image] 

Source: GAO analysis of EAC, NIST, and Institute of Electrical and 
Electronics Engineers (IEEE) publications. 

[End of figure] 

* The product development phase includes activities such as 
establishing requirements for the system, designing a system 
architecture, developing software, and integrating components. 
Activities in this phase are performed by the system vendor. 

* The acquisition phase includes activities such as publishing a 
solicitation, evaluating offers, choosing a voting technology and a 
vendor, and awarding and administering contracts. For voting systems, 
activities in this phase are primarily the responsibility of state and 
local governments but entail some responsibilities that are shared with 
the system vendor (e.g., entering into the contract). 

* The operations phase consists of activities such as ballot design and 
programming, setup of systems before voting, pre-election testing, vote 
capture and counting during elections, recounts and system audits after 
elections, and storage of systems between elections. Responsibility for 
activities in this phase typically resides with local jurisdictions, 
whose officials may, in turn, rely on or obtain assistance from system 
vendors for aspects of these activities. 

* Standards for voting systems, as will be discussed in a later 
section, were developed at the national level by the Federal Election 
Commission in 1990 and 2002 and were updated by EAC in 2005. In the 
product development phase, voting system standards serve as 
requirements to meet for developers to build systems. In the 
acquisition phase, they also provide a framework that state and local 
governments can use to evaluate systems. In the operations phase, they 
specify the necessary documentation for operating the systems. 

* Testing processes are conducted throughout the life cycle of a voting 
system. Voting system vendors conduct product testing during 
development of the system and its components. Federal certification 
testing of products submitted by system vendors is conducted by 
national voting system testing laboratories (VSTL). States may conduct 
evaluation testing before acquiring a system to determine how well 
products meet their state-specific specifications, or they may conduct 
certification testing to ensure that a system performs its functions as 
specified by state laws and requirements. Once a voting system is 
delivered by the system vendor, states and local jurisdictions may 
conduct acceptance testing to ensure that the system satisfies 
functional requirements. Finally, local jurisdictions typically conduct 
logic and accuracy tests related to each election and sometimes subject 
portions of the system to parallel testing during each election to 
ensure that the system components perform accurately. 

* Management processes ensure that each life cycle phase produces a 
desirable outcome. Typical management activities that span the system 
life cycle include planning, configuration management, system 
performance review and evaluation, problem tracking and correction, 
human capital management, and user training. These activities are 
conducted by the responsible parties in each life cycle phase. 

In 2004, we reported[Footnote 7] that the performance of electronic 
voting systems, like any type of automated information system, can be 
judged on several bases, including their security, accuracy, ease of 
use, efficiency, and cost. We also reported that voting system 
performance depends on how the system was designed, developed, and 
implemented. 

Laboratory Accreditation Plays an Important Role in Ensuring Accurate, 
Reliable, and Secure Voting Systems: 

Since the passage of HAVA, the use of electronic voting systems has 
increased and become the predominant method of voting. However, 
concerns have been raised about the security and reliability of these 
systems. As we have previously reported,[Footnote 8] testing and 
certifying voting systems is one critical step in acquiring, deploying, 
operating, and administering voting systems, which better ensures that 
they perform securely and reliably. Among other things, rigorous 
execution and careful documentation of system testing is a proven way 
to help ensure that system problems are found before the systems are 
deployed and used in an election. To accomplish this, it is vital that 
the organizations that test the systems be qualified and competent to 
do so. For voting systems, a key testing organization is a federally 
accredited, national VSTL. 

In general, accreditation is the formal recognition that a laboratory 
is competent to carry out specific types of tests or calibrations. 
Federally accredited laboratories perform many different types of 
testing and related activities on various products, ranging from 
inspecting grain to certifying maritime cargo gear. The genesis of 
laboratory accreditation programs owes largely to agencies' need to 
assure themselves of the competency of the organizations responsible 
for testing products or services that involve the use of federal funds. 

To provide national recognition for competent laboratories, the NIST 
Director established the National Voluntary Laboratory Accreditation 
Program (NVLAP) in 1976 at the request of the private sector. Under 
this program, which is based on internationally accepted standards, 
NIST accredits laboratories that it finds competent to perform specific 
types of tests or calibrations. In June 2004, NVLAP announced the 
establishment, in accordance with HAVA, of an accreditation program for 
laboratories that test voting systems using standards determined by 
EAC. 

HAVA Assigned EAC and NIST Responsibility for Accrediting VSTLs: 

Enacted in October 2002, HAVA affected nearly every aspect of the 
voting process, from voting technology to provisional ballots and from 
voter registration to poll worker training.[Footnote 9] In particular, 
the act authorized $3.86 billion in funding over several fiscal years 
to replace punch card and mechanical lever voting equipment, improve 
election administration and accessibility, train poll workers, and 
perform research and pilot studies. HAVA also established EAC, provided 
for the appointment of four commissioners, and specified the process 
for selecting an executive director. Generally speaking, EAC is to 
assist in the administration of federal elections and provide 
assistance in administering certain federal election laws and programs. 

Since the passage of HAVA in 2002, the federal government has taken 
steps to implement the act's provisions. For example, after beginning 
operations in January 2004, EAC updated the existing federal voluntary 
standards for voting systems, including strengthening provisions 
related to security and reliability. Additionally, EAC established an 
interim VSTL accreditation program that leveraged a predecessor program 
run by the National Association of State Elections Directors, and EAC 
and NIST then established companion accreditation programs that 
replaced the interim program. 

EAC Updated the Federal Voluntary Standards for Voting Systems: 

Federal standards for voting systems were first issued in 1990 when the 
Federal Election Commission published standards.[Footnote 10] These 
federal standards identified minimum functional and performance 
requirements, which states were free to adopt in whole, in part, or not 
at all, for electronic voting equipment, and specified test procedures 
to ensure that the equipment met those requirements. In 2002, the 
Federal Election Commission issued its Voting System Standards (VSS), 
which updated the 1990 standards to reflect more modern voting system 
technologies. In 2005, we reported[Footnote 11] that these standards 
identified minimum functional and performance requirements for voting 
systems but were not sufficient to ensure secure and reliable voting 
systems. As a result, we recommended that EAC work to define specific 
tasks, measurable outcomes, milestones, and resource needs to improve 
the voting system standards. Until then, election administrators were 
at risk of relying on voting systems that were not developed, acquired, 
tested, operated, or managed in accordance with rigorous security and 
reliability standards--potentially affecting the reliability of future 
elections and voter confidence in the accuracy of the vote count. 

Following the enactment of HAVA in 2002 and the establishment of EAC in 
2004, EAC adopted the Voluntary Voting System Guidelines (VVSG) in 
2005.[Footnote 12] The VVSG specify the functional requirements, 
performance characteristics, documentation requirements, and test 
evaluation criteria for the national certification of voting systems. 
Accredited testing laboratories are to use the VVSG to develop test 
plans and procedures for the analysis and testing of systems in support 
of EAC's voting system certification program.[Footnote 13] The VVSG are 
also used by voting system manufacturers as the basis for designing and 
deploying systems that can be federally certified. 

EAC Established an Interim Accreditation Program: 

We reported in 2001[Footnote 14] that the National Association of State 
Elections Directors was accrediting independent test authorities to 
test voting equipment against the Federal Election Commission 
standards. Under this program, three laboratories were accredited. 
Under HAVA, NIST is to recommend laboratories for EAC accreditation. In 
2006, NIST notified EAC that its initial recommendations might not be 
available until sometime in 2007. As a result, EAC initiated an interim 
accreditation program and invited the three laboratories accredited by 
the state elections directors to apply.[Footnote 15] As part of the 
interim program, laboratories were required to attest to a set of EAC- 
required conditions and practices, including certifying the integrity 
of personnel, the absence of conflicts of interest, and the financial 
stability of the laboratory. In August and September 2006, EAC granted 
interim accreditation to two of the three laboratories invited to 
apply. EAC terminated its interim program in March 2007. 

EAC and NIST Have Established Separate but Related Laboratory 
Accreditation Programs: 

HAVA assigned responsibilities for laboratory accreditation to both EAC 
and NIST. In general, to reach an accreditation decision, NIST is to 
focus on assessing laboratory technical qualifications, while EAC is to 
use those assessment results and recommendations and augment them with 
its own review of related laboratory capabilities. See table 1 for the 
two agencies' HAVA responsibilities. 

Table 1: EAC and NIST Responsibilities under HAVA: 

HAVA responsibility: Provide for the testing, certification, 
decertification, and recertification of voting system hardware and 
software by accredited laboratories; 
Responsible entity: EAC: X; 
Responsible entity: NIST: [Empty]. 

HAVA responsibility: Conduct evaluations of independent, nonfederal 
laboratories and submit to EAC a list of those laboratories proposed 
for accreditation; 
Responsible entity: EAC: [Empty]; 
Responsible entity: NIST: X. 

HAVA responsibility: Vote on the accreditation of any laboratory, 
taking into consideration the NIST recommendation for it. No laboratory 
may be accredited unless its accreditation is approved by a vote of 
EAC; 
Responsible entity: EAC: X; 
Responsible entity: NIST: [Empty]. 

HAVA responsibility: Publish an explanation for the accreditation of 
any laboratory not included on the list submitted for recommendation by 
NIST; 
Responsible entity: EAC: X; 
Responsible entity: NIST: [Empty]. 

HAVA responsibility: Monitor and review the performance of laboratories 
accredited by EAC and make recommendations to EAC with respect to the 
continuing accreditation of laboratories, including recommendations to 
revoke the accreditation of laboratories; 
Responsible entity: EAC: [Empty]; 
Responsible entity: NIST: X. 

HAVA responsibility: Revoke the accreditation of a laboratory only when 
approved by a vote of EAC; 
Responsible entity: EAC: X; 
Responsible entity: NIST: [Empty]. 

Source: GAO analysis of United States Code. 

[End of table] 

The tasks that NIST is to perform in meeting HAVA's requirements are 
addressed in an annual interagency agreement executed between the 
institute and EAC each year. For example, the 2008 interagency 
agreement states that NVLAP will continue to assess VSTLs and will 
coordinate with EAC to continually monitor and review the performance 
of the laboratories. Additionally, the agreement states that the two 
agencies will coordinate to maintain continuity between their 
respective accreditation programs. 

The NIST and EAC accreditation programs can be viewed together as 
forming a federal VSTL accreditation process that consists of a series 
of 12 complementary steps. These steps are depicted in figure 2, where 
the numbers correspond to a detailed narrative description below. 

Figure 2: Overall NIST and EAC Accreditation Processes: 

This figure is a flowchart of overall NIST and EAC accreditation 
processes. 

[See PDF for image] 

Source: GAO analysis. 

[End of figure] 

1. Laboratory Application to NIST: 

The accreditation process begins when a laboratory submits a completed 
application to NIST, along with administrative information about the 
laboratory, the scope of accreditation being applied for, and an 
agreement to the conditions of accreditation (i.e. practices that must 
be followed to obtain and maintain accreditation). In addition, the 
laboratory submits documentation that supports the application, 
including the laboratory's quality control manual. 

2. NIST Preassessment: 

Using the application and supporting documentation, NIST conducts a 
preassessment review. Among other things, this review includes 
comparing the quality control manual against the requirements in NIST 
accreditation program guidance. If deficiencies in the documentation 
are found, NIST requests corrections to satisfy program requirements. 

3. NIST On-site Assessment: 

Following satisfactory completion of the preassessment review, a NIST 
team visits the laboratory facilities to conduct an on-site assessment. 
This assessment includes staff interviews, reviews of laboratory 
records and audit reports, and demonstrations of staff competence to 
execute planned test methods and procedures. It concludes with the NIST 
team presenting its findings to laboratory management. While conducting 
the on-site assessment, the team records its observations and comments. 

4. NIST Nonconformity Resolution: 

NIST prepares a final report, including a list of any nonconformities, 
and provides it to the laboratory. The laboratory has 30 days to 
respond as to how it will address the areas of nonconformity. NIST 
evaluates the laboratory's response and determines whether the 
nonconformities have been sufficiently addressed. If so, NIST renders 
an accreditation decision. If not, NIST may contact the laboratory for 
additional information or may deny accreditation. If a laboratory is 
denied accreditation, it may reapply to NIST. 

5. NIST Accreditation Decision: 

When a laboratory has no areas of nonconformity, the voting systems 
program manager makes an accreditation recommendation to the Chief of 
NVLAP, who is responsible for all NVLAP accreditation decisions and 
issues all NVLAP accreditation certificates. 

6. NIST Recommendation to EAC: 

In addition to granting the NVLAP accreditation, the Chief provides a 
recommendation to the Director of NIST. The recommendation is reviewed 
by NIST's general counsel, and then a letter is sent to EAC that 
recommends the laboratory for accreditation as a VSTL in accordance 
with HAVA. 

7. Accreditation Application to EAC: 

After receiving the NIST recommendation, EAC sends the recommended 
laboratory an invitation to apply to the EAC accreditation program. In 
the letter, EAC specifies a list of information and documentation that 
the laboratory must provide. 

8. EAC Accreditation Review: 

The laboratory submits an application and supporting information to 
EAC. EAC staff review the application package for completeness. In 
addition, staff review the supporting materials vis-a-vis accreditation 
program requirements. During the course of the review, staff may 
contact the laboratory to clarify the information provided or to inform 
the laboratory of requirements that are not sufficiently addressed. 

9. EAC Nonconformity Resolution: 

EAC submits correspondence (generally through e-mail) to the laboratory 
identifying areas of nonconformity. The laboratory then provides the 
missing and/or clarifying documentation. EAC staff determines if the 
provided information adequately addresses the nonconformity issues, 
contacting the laboratory as needed. 

10. Recommendation to EAC Commissioners: 

The EAC accreditation program director, through the EAC executive 
director, makes a recommendation to the EAC commissioners as to whether 
the laboratory should be accredited by EAC. Along with the 
recommendation, the program director provides the review results, as 
well as laboratory-provided materials. 

11. Vote by EAC Commissioners: 

The commissioners review the material provided and may request 
additional clarification, as needed. At a public meeting, the 
commissioners vote on whether to accredit the laboratory. Should the 
EAC commissioners vote to deny the accreditation, the laboratory must 
wait for EAC to invite the laboratory to reapply. 

12. EAC Accreditation Granted: 

When the commissioners vote to accredit a laboratory, EAC's executive 
director issues an accreditation certificate identifying the scope and 
effective dates of the VSTL accreditation. In addition, the program 
director makes information about the laboratory's accreditation 
publicly available via the EAC Web site. At this point, the laboratory 
is authorized to operate as a VSTL under EAC's testing and 
certification program. 

Postaccreditation Monitoring Activities: 

Once a laboratory has been accredited, both NIST and EAC are to monitor 
its compliance with the terms of its accreditation. In doing so, NVLAP 
staff may visit a laboratory at any time, whether for cause or on a 
random selection basis, and these visits can be either scheduled in 
advance with the laboratory or unannounced. If a laboratory is found to 
not be in compliance, the accreditation may either be 
suspended[Footnote 16] or revoked, depending on the nature of the 
issues involved. A suspension provides the opportunity for the 
laboratory to address the identified issues. EAC also monitors the 
procedures and practices of accredited laboratories through 
documentation reviews and visits. If a VSTL is unable to remedy 
identified compliance issues, the EAC program director can propose that 
the accreditation be suspended and ultimately revoked. As provided for 
under HAVA, the EAC commissioners would vote on any proposed 
revocation. 

Status of Completed VSTL Accreditation Activities: 

As of May 2008, EAC has accredited four laboratories. These 
laboratories are SysTest Labs, LLC; Wyle Laboratories, Inc; iBeta 
Quality Assurance; and InfoGard Laboratories, Inc. A fifth laboratory, 
CIBER Inc., has been granted NVLAP accreditation and has been 
recommended to, but not yet accredited by, EAC. InfoGard Laboratories, 
Inc., whose NVLAP accreditation expires in June 2008, has recently 
notified NIST and EAC that it would not apply to renew its 
accreditation, citing the volatility of the voting system environment 
as one reason. The timeline for each of these accreditations, and other 
accreditation program activities, is found in figure 3. 

Figure 3: VSTL Accreditation Program Activities: 

This figure is a chart depicting VSTL accreditation program activities. 

[See PDF for image] 

Source: GAO based on NIST: and EAC-provided data. 

[End of figure] 

NIST Has Defined and Implemented an Accreditation Approach That 
Reflects Relevant Standards but Is Missing Details Needed for 
Consistent and Verifiable Implementation: 

NIST's defined approach to accrediting voting system laboratories 
largely reflects applicable HAVA requirements and relevant 
international standards, both of which are necessary to an effective 
program. However, this approach is continuing to evolve based on issues 
realized during NIST's implementation experience to date. In 
particular, because NIST's defined program does not, for example, 
specify the nature and extent of assessment documentation to generate 
or retain or specify the version of the voting system standards to be 
used, our analysis of NIST's efforts in accrediting four laboratories 
could not confirm that the agency has consistently followed its defined 
accreditation program. NIST officials stated that these limitations are 
due in part to the relative newness of the program and that they will 
be addressed by updating the accreditation program handbook. However, 
they said that they do not have documented plans to accomplish this. 
Until these limitations are addressed, NIST will be challenged in 
accrediting voting system laboratories in a consistent and verifiable 
manner. 

NIST Voting System Accreditation Program Reflects HAVA Requirements: 

NIST has defined its voting system accreditation program to address 
relevant HAVA requirements. According to HAVA, NIST is to: 

* conduct reviews of independent, nonfederal voting system testing 
laboratories and submit to EAC a list of proposed voting system testing 
laboratories and: 

* monitor and review the performance of those proposed laboratories 
that EAC accredits, including making recommendations to EAC regarding 
accreditation continuance and revocation. 

NIST's defined voting system accreditation program satisfies both of 
these requirements. With respect to the first, NIST announced in June 
2004 the establishment of its voting system testing laboratory 
accreditation program as part of NVLAP, a statutorily created program 
for unbiased, third parties to establish the competence of national 
independent laboratories. As such, NIST adopted its NVLAP 
handbook[Footnote 17] as the basis for its defined approach to 
reviewing VSTLs and has supplemented it with a handbook that is 
specific to voting system testing.[Footnote 18] 

With respect to the second HAVA requirement, the supplemental handbook 
cited above states that the NIST Director will recommend NVLAP- 
accredited VSTLs to EAC for subsequent commission accreditation. 
Additionally, NIST's handbooks provide for both monitoring accredited 
laboratories and for making recommendations regarding a laboratory's 
continued accreditation. For example, the handbook states that a 
monitoring visit may occur at both scheduled and unscheduled times and 
the scope may be limited to a few items or include a full review. It 
also states that a reaccreditation review shall be conducted in 
accordance with the procedures used to initially accredit laboratories. 
Further, the handbook also identifies accreditation or reaccreditation 
decision options, including granting, denying, or modifying the scope 
of an accreditation. 

According to NIST officials, these HAVA requirements are relevant and 
important to defining an effective voting system testing laboratory 
accreditation program. By incorporating them, NIST has reflected one 
key aspect of an effectively defined program. 

NIST Has Incorporated Relevant International Accreditation Standards 
into Its VSTL Accreditation Program: 

NIST's VSTL accreditation program reflects internationally recognized 
standards for establishing and conducting accreditation activities. 
These standards are published by the International Organization for 
Standardization (ISO), and the two that are germane to this 
accreditation program are (1) ISO/IEC 17011,[Footnote 19] which 
establishes general requirements for accreditation bodies and (2) ISO/ 
IEC 17025,[Footnote 20] which establishes the general requirements for 
reviewing the competence of laboratories. According to NIST program 
documentation, this allows NVLAP to both operate as an unbiased, third 
party accreditation body and to utilize a quality management system 
compliant with international standards. As a result, NIST has 
incorporated key aspects of an effective accreditation body into its 
voting system accreditation program. 

NIST Program Meets ISO Accreditation Body Requirements: 

ISO/IEC 17011 requires that an accrediting body have, among other 
things, (1) a management system for accreditation activities, (2) a 
policy defining the types of records to be retained and how those 
records will be maintained, (3) a clear description of the 
accreditation process that covers the rights and responsibilities of 
those seeking accreditation, and (4) a clear description of the 
accreditation activities to be performed. 

NIST VSTL accreditation program-related documentation, including its 
program handbooks, satisfies each of these requirements. In fact, NIST 
has cross-referenced its documentation to each ISO/IEC 17011 
requirement. Specifically, the first requirement is cross-referenced to 
the NVLAP Management System Manual,[Footnote 21] which describes the 
overall accreditation program's management policies and control 
structure, and the second is cross-referenced to the program's record 
keeping policy, which specifies what types of records should be 
maintained and how they should be maintained. The third and fourth 
requirements are cross-referenced to the accreditation process 
descriptions in both the Management System Manual and the general 
handbook. Together, these documents contain, for example, (1) the 
rights of laboratories applying for accreditation and (2) the scope of 
accreditation activities to be performed, including a preassessment 
review, an on-site review, and a final on-site assessment report. 

NIST Program Meets ISO Laboratory Accreditation Review Requirements: 

ISO/IEC 17025 requires that accreditation reviews cover specific 
topics. These include (1) laboratory personnel independence and 
conflicts of interest; (2) a laboratory system for quality control 
(i.e., a framework for producing reliable results and continuous 
improvement to laboratory procedures); and (3) a laboratory mechanism 
for collecting and responding to customer complaints. Additionally, the 
standard establishes basic technical requirements that a laboratory has 
to meet, and thus that reviews are to cover, including (1) competent 
laboratory personnel who are capable of executing the planned tests, 
(2) appropriate tests and test methods, and (3) clear and accurate test 
result documentation. 

NIST voting system testing laboratory accreditation program-related 
documents, including its program handbooks, satisfy these requirements. 
First, the general handbook defines the requirement for a laboratory to 
have personnel that are independent and free of any conflict of 
interest. Second, the handbook requires that a laboratory have a 
management quality control system and that this system provide for 
reliable results and continuous improvement to laboratory procedures. 
Third, the handbook requires that a laboratory have a mechanism for 
receiving and responding to customer complaints. Last, the handbook 
establishes certain technical requirements that a laboratory must meet, 
such as having competent laboratory personnel capable of executing the 
planned tests, using appropriate tests and test methods, and 
documenting test results in a clear and accurate manner. 

For several of these requirements, NIST's voting-specific supplemental 
handbook augments the general handbook. For example, this supplemental 
handbook requires laboratories to submit a quality control manual, as 
well as information to demonstrate the competence of laboratory 
administrative and technical staff. Further, it requires that a 
laboratory's training program be updated so that staff can be retrained 
as new versions of voting system standards are issued. 

NIST Voting System Accreditation Program Does Not Reflect Its Own 
Findings on the Need for Assessor Qualifications and Training and Key 
Assessment Criteria: 

NIST has reported on the importance of ensuring that those persons who 
perform accreditation assessments are sufficiently qualified and that 
the assessments themselves are based on explicitly defined criteria and 
are adequately documented. Nevertheless, NIST has not fully reflected 
key aspects of these findings in its defined approach to accrediting 
voting system testing laboratories. For example, it has not specified 
the basis for determining the qualifications of its accreditation 
assessors, and while a draft update to its handbook now includes the 
specific voting system standards to be used when performing an 
accreditation assessment, this handbook was only recently approved. 
According to NIST officials, these gaps are due to the newness of the 
accreditation program and will be addressed in the near future. Because 
these gaps have confused laboratories as to what standards they were to 
meet, and may have resulted in differences in how accreditations have 
been performed to date, it is important that the gaps be addressed. 

NIST Has Not Specified Requirements for Assessor Qualifications and 
Training: 

NIST has reported[Footnote 22] on the importance of having competent 
and qualified human resources to support accreditation programs. 
According to these findings, an accreditation program should, among 
other things, provide for: 

* having experienced and qualified assessors to perform accreditation 
activities; 

* demonstrating an assessors' qualifications using defined 
documentation and explicit criteria that encompass the person's 
education, experience, and training; and: 

* training (initial and continuing) for assessors. 

NIST's defined approach to VSTL accreditation does not provide for all 
these requirements. To its credit, its program handbook identifies the 
need for experienced and qualified assessors in the execution of 
accreditation activities and provides for each assessor's 
qualifications to be documented. Further, it has defined generic 
training that applies to all of its accreditation assessors. For 
example, the NVLAP Assessor Training Syllabus includes training on ISO/ 
IEC 17011 and 17025, as well as training on the NVLAP general handbook. 
In addition, the VSTL accreditation program manager stated that new 
assessors receive training on the 2002 VSS and 2005 VVSG and that 
periodic training seminars are provided to assessors on changes to 
either the general handbook or the 2005 VVSG. 

In addition, the program manager told us that candidate assessors must 
submit some form of documentation (e.g., a resume), and that this 
documentation is used to evaluate, rank, and select candidates that are 
best qualified. The NIST VSTL assessors that we interviewed confirmed 
that they were required to submit such documentation at NIST's request. 

However, NIST's defined approach does not cite the explicit 
capabilities and qualifications that an assessor must meet or the 
associated documentation needed to demonstrate these capabilities and 
qualifications. According to the program manager, this is because the 
field of potential assessors in the voting system arena is small and 
specialized and because they focused on defining other aspects of the 
program that were higher priorities. Further, NIST has not defined and 
documented the specific training requirements needed to be a VSTL lead 
assessor or a technical assessor for the VSTL program. According to the 
program manager, this is because these assessors receive all the 
training they need by working on the job with more experienced 
assessors. Not specifying criteria governing assessor qualifications 
and training is of concern because differences in assessors' 
capabilities could cause inconsistencies in how assessments are 
performed. 

NIST's Approach Does Not Fully Specify Criteria for Evaluating and 
Documenting VSTL Capabilities: 

NIST recognizes the importance of specifying explicit criteria against 
which all candidate laboratories will be assessed and fully documenting 
the assessments that are performed. Specifically, the general handbook 
provides the criteria and requirements that will be used to evaluate 
basic laboratory capabilities. It also states that technical 
requirements specific to a given field of accreditation are published 
in program-specific handbooks. To that end, NIST published a 
supplemental program-specific handbook in December 2005 that provided 
the voting-specific requirements to be used to evaluate VSTLs, 
additional guidance, and related interpretive information.[Footnote 23] 

NIST's 2005 supplemental handbook does not contain sufficient criteria 
against which to evaluate VSTLs. It identifies specific requirements 
that laboratories are to demonstrate relative to the 2002 VSS but not 
the 2005 VVSG. For example, the handbook states that laboratories are 
expected to develop, validate, and document test methods that meet the 
2002 VSS. However, it does not refer to the 2005 VVSG. In addition, the 
program-specific checklist that accompanies this version of the 
handbook does not identify all the 2005 VVSG standards against which 
laboratories are evaluated. Specifically, this checklist makes 
reference to the VVSG in relation to just a few checklist requirements. 

According to the NIST program manager, the 2005 handbook did not refer 
to the 2005 VVSG requirements because only the 2002 VSS requirements 
were mandatory at the time it was published. He further stated that, 
despite the fact that the 2005 VVSG requirements were not included in 
that handbook, NIST assessors were expected to use them when performing 
the first laboratory assessments. Representatives for two laboratories 
stated that because these requirements were not documented or 
identified in the NIST handbooks, they did not learn that they would be 
required to demonstrate 2005 VVSG-based capabilities until the NIST on- 
site assessment teams arrived. 

In December 2007, NIST released draft revisions of the voting program- 
specific handbook and checklist, stating that labs are expected to meet 
both 2002 VSS and 2005 VVSG. In addition, the 2007 draft handbook 
clearly specifies that laboratories must demonstrate how developed test 
methods and planned tests trace back to and satisfy both the 2002 VSS 
and the 2005 VVSG. Taken together, the new handbook and checklist 
should better identify the requirements and criteria used to evaluate a 
laboratory and document the results. According to NIST, the new 
handbook and checklist have recently been finalized, and both are now 
in use.[Footnote 24] 

Available Documentation Does Not Show That NIST Has Consistently 
Followed All Aspects of Its Defined Accreditation Approach: 

NIST has found that reliable and accurate documentation provides 
assurance that laboratory accreditation activities have been 
effectively fulfilled.[Footnote 25] However, in its efforts to date in 
accrediting four VSTLs, documentation of the assessments does not show 
that NIST has fully followed its defined accreditation approach. While 
we could not determine whether this is due to incomplete documentation 
of the steps performed and the decisions made during an assessment or 
due to steps not being performed as defined, this absence of verifiable 
evidence raises questions about the consistency of the assessments and 
the resultant accreditations. Without adequately documenting each 
assessment, including all steps performed and the basis for any steps 
not performed, such questions may continue to be raised. 

To NIST's credit, available documentation shows that it consistently 
followed some aspects of its defined approach in accrediting the four 
laboratories. For example, we verified that NIST received an 
application from each of the laboratories as required, and our review 
of completed checklists and summary reports shows that preassessment 
reviews and on-site assessments were performed for each laboratory, as 
was required. According to a lead assessor, this review usually focused 
on the laboratories' quality assurance manuals. Moreover, the completed 
checklists identified whether the requirement was met or not for each 
listed requirement, and included comments, in some cases, as to how a 
laboratory addressed a requirement. Also as required, NIST received 
laboratory responses describing how unmet requirements were addressed 
within specified time frames, used the responses in making 
accreditation decisions, and notified EAC of its decisions via letters 
of recommendation. Furthermore, NIST has recently begun reaccreditation 
reviews at two laboratories, as required. 

However, documentation does not show that NIST has consistently 
followed other aspects of its defined approach. Our analysis of the 
checklists that are to be used to both guide and document a given 
assessment, including identifying unmet requirements and capturing 
assessor comments and observations, shows some differences. For 
example: 

* One type of checklist (the supplemental handbook checklist) was 
prepared for only two of the four laboratory assessments. According to 
the program manager, this is because even though a draft revision of 
this checklist was actually used to assess the other two laboratories, 
the assessment results were recorded on a different checklist (the 
general handbook checklist). While this is indicated on one of the two 
checklists, it is not indicated on the other. 

* On the checklist used for one laboratory, an assessor marked several 
sections as "TA" with no explanation as to what this means. Also, the 
checklist used for another laboratory did not identify whether most of 
the requirements were met or not met. Further, the checklist for a 
third laboratory had one section marked as "not applicable" but 
included no explanation as to why that section did not apply, while the 
checklist for a different laboratory marked the same section as "not 
applicable" but included a reason for doing so. 

Notwithstanding these differences, the program manager told us that 
each laboratory was assessed using the same requirements and all 
assessments to date were performed in a consistent manner. On the basis 
of available documentation, however, we could not verify that this is 
the case. As a result, it is not clear that NIST has consistently 
followed its defined approach. 

Available documentation also does not show that NIST followed other 
aspects of its approach. For example: 

* The program handbook states that each laboratory is to identify the 
requested scope of accreditation in its application package. However, 
our analysis of the four application packages shows that two 
laboratories did not specify a requested scope of accreditation. 
According to the program manager, the scope of accreditation for all 
laboratories was the 2002 VSS and 2005 VVSG because, even though the 
latter standards were not yet in effect at the time, they were 
anticipated to be in effect in the near future.[Footnote 26] However, 
NIST did not have documentation that notified the laboratories of this 
scope of accreditation or that indicated whether this scope was 
established by EAC, NIST, or the laboratories. 

* The program handbook states that after receiving a laboratory's 
application package, NIST will acknowledge its receipt in writing and 
will inform the laboratory of the next steps in the accreditation 
process. However, NIST did not have documentation demonstrating that 
this was done. According to the program manager, this was handled via 
telephone conversations. However, representatives for several 
laboratories noted that these calls did not clearly establish 
expectations, adding that some expectations were not communicated until 
the NIST team assessors arrived to conduct the on-site assessment. 

The program manager stated that these deviations from the defined 
approach are attributable to the relative newness of the program, but 
despite these discrepancies, each laboratory was assessed consistently. 
However, we could not verify this, and thus it is not clear that NIST 
has consistently followed its defined approach. According to this 
official, future versions of the program handbook would address these 
limitations. However, documented plans for doing so have not been 
developed. 

EAC's Recently Drafted Accreditation Approach and Its Earlier Performed 
Laboratory Accreditations Lack Key Effectiveness Factors and Features: 

EAC has recently defined its voting system laboratory accreditation 
approach in a draft program manual. However, this draft manual omits 
important content. While addressing relevant HAVA requirements, the 
draft manual does not adequately define key accreditation factors that 
NIST has identified, and a key accreditation feature that we have 
previously reported as being integral to an effective accreditation 
program. Moreover, not all factors and features that the draft manual 
does include have been defined to a level that would ensure thorough, 
consistent, and verifiable implementation. Because this manual was not 
available for EAC to use on the four laboratory accreditations that it 
has completed, the accreditations were performed using a largely 
undocumented series of steps. As a result, the thoroughness and 
consistency of these accreditations is not clear. According to EAC 
officials, these gaps are due to the agency's limited resources being 
focused on other issues, and will be addressed as its accreditation 
program evolves. However, they said that they do not yet have 
documented plans to accomplish this. Until EAC fully defines a 
repeatable VSTL accreditation approach, it will be challenged in its 
ability to treat all laboratories consistently and produce verifiable 
results. 

EAC Has Defined a Draft Accreditation Approach that Meets HAVA 
Requirements: 

In February 2008, EAC issued a draft version of a VSTL accreditation 
program manual[Footnote 27] for public comment. According to HAVA, 
EAC's accreditation program is to meet certain requirements. 
Specifically, it is to provide for voting system hardware and software 
testing, certification, decertification, and recertification by 
accredited laboratories. Additionally, it is to base laboratory 
accreditation decisions, including decisions to revoke an 
accreditation, on a vote of the commissioners, and it is to provide for 
a published explanation of any commission decision to accredit any 
laboratory that was not first recommended for accreditation by NIST. 

To EAC's credit, its draft accreditation program manual addresses each 
of these requirements. First, the manual defines the role that the 
laboratories are to play relative to voting system testing, 
certification, recertification and decertification, and it incorporates 
by reference an EAC companion voting system certification 
manual[Footnote 28] that defines requirements and process steps for 
voting system testing and certification-related activities. 

With respect to the remaining three HAVA requirements, the draft EAC 
accreditation manual also requires (1) that the commissioners vote on 
the accreditation of laboratories recommended by NIST for 
accreditation, (2) that EAC publish an explanation for the 
accreditation of any laboratory not recommended by NIST for 
accreditation, and (3) that the commissioners vote on the proposed 
revocation of a laboratory's accreditation. 

According to EAC officials, its draft approach incorporates HAVA 
requirements because the commission is focused on meeting its legal 
obligations in all aspects of its operations, including VSTL 
accreditation. In doing so, EAC has addressed one important aspect of 
having an effective accreditation program. 

EAC Draft Approach Does Not Adequately Define Key Accreditation-Related 
Steps and Decision Criteria: 

Beyond addressing relevant HAVA requirements, EAC's draft accreditation 
manual defines an accreditation process, including program phases, 
requirements, and certain evaluation criteria. However, it does not do 
so in a manner that fully satisfies factors that NIST has reported can 
affect the effectiveness of accreditation programs.[Footnote 29] 
Moreover, it does not adequately address a set of features that our 
research shows are common to federal accreditation programs[Footnote 
30] and that can influence a program's effectiveness. According to EAC 
officials, these factors and features are not fully addressed in the 
draft program manual because its accreditation program is still in its 
early stages of development and is still evolving. Until they are fully 
addressed, EAC's accreditation program's effectiveness will be limited. 

Key NIST Accreditation Factors Not Fully Addressed: 

According to NIST, having confidence in and ensuring appropriate use of 
an accredited testing laboratory requires that accreditation 
stakeholders have an adequate understanding of the accreditation 
process, scope, and related criteria. NIST further reports that 
confidence in the accreditation process can be traced to a number of 
factors that will influence the thoroughness and competence of 
accreditation programs, and thus these factors can be viewed as 
essential accreditation program characteristics. They include having: 

* published procedures governing how the accreditation program is to be 
executed, such as procedures for granting, maintaining, modifying, 
suspending, and withdrawing accreditation; 

* specific instructions, steps, and criteria for those who conduct an 
accreditation assessment (assessors) to follow, such as a test 
methodology that is acceptable to the accreditation program; 

* knowledgeable and experienced assessors to execute the instructions 
and steps and apply the related criteria; and: 

* complete records on the data collected, results found, and reports 
prepared relative to each assessment performed. 

EAC's draft accreditation program manual addresses one of these factors 
but it does not fully address the other three. (See table 2.) For 
example, while the manual requires that EAC maintain records, it only 
addresses the retention of records associated with the testing of 
voting systems and not those associated with the accreditation of 
laboratories. EAC officials told us that testing records are meant to 
include accreditation records, although they added that this is not 
explicit in the manual and needs to be clarified. Further, the manual 
is silent on the steps to be followed and criteria to be applied in 
reviewing a laboratory's application and the qualifications required 
for accreditation reviewers. By not fully addressing these factors, EAC 
increases the risk that its accreditation reviews will not be performed 
consistently and comprehensively. 

Table 2: Summary of Extent to Which EAC Draft Approach Addresses NIST- 
Identified Accreditation Factors: 

Accreditation program factor: Published accreditation program 
procedures; Addressed by EAC?: Yes. 

Accreditation program factor: Specific accreditation instructions; 
Addressed by EAC?: No. 

Accreditation program factor: Established accreditation personnel 
qualifications; Addressed by EAC?: No. 

Accreditation program factor: Adequate maintenance of records; 
Addressed by EAC?: Partially. 

Source: GAO analysis of EAC data. 

[End of table] 

All but One Key GAO-Reported Accreditation Program Feature Has Been 
Addressed: 

As we have previously reported,[Footnote 31] the nature and focus of 
federal programs for accrediting laboratories vary, but nevertheless 
include certain common features. In particular, these programs require 
laboratories to provide certain information to the accrediting body, 
and they provide for evaluation of this information by the accrediting 
body in making an accreditation determination. As we reported, the 
required information is to include, among other things, the 
laboratory's (1) organizational information, (2) records and record- 
keeping policy, (3) test methods and procedures, (4) conflict of 
interest policy, and (5) financial stability. 

To its credit, EAC's draft accreditation manual provides for 
laboratories to submit information relative to each of these features 
that are common to federal accreditation programs. For example, it 
provides for laboratories to submit organizational information, such as 
location(s), ownership, and organizational chart; a written policy for 
maintaining accreditation-related records for 5 years; conflict of 
interest policies and procedures; test-related polices and procedures, 
as well as system-specific test plans; and financial information needed 
to demonstrate stability. Moreover, for four of the five features, the 
manual identifies the specific types of information needed for 
accreditation and how the information is to be evaluated, including the 
criteria that are to be used in evaluating it. However, for the 
financial stability feature, the manual does not describe what specific 
documents are required from the laboratory to satisfy this requirement, 
nor does the manual indicate how information provided by a laboratory 
will be evaluated. 

Table 3: Summary of Extent to Which EAC Satisfies Features Common to 
Federal Accreditation Programs: 

Accreditation program feature: Organizational information; 
EAC requires information?: Yes; 
EAC specifies information scope and level of detail?: Yes; 
EAC specifies how information is to be evaluated and criteria to be 
used?: Yes. 

Accreditation program feature: Records and record-keeping; 
EAC requires information?: Yes; 
EAC specifies information scope and level of detail?: Yes; 
EAC specifies how information is to be evaluated and criteria to be 
used?: Yes. 

Accreditation program feature: Test methods and procedures; 
EAC requires information?: Yes; 
EAC specifies information scope and level of detail?: Yes; 
EAC specifies how information is to be evaluated and criteria to be 
used?: Yes. 

Accreditation program feature: Conflict of interest policy; 
EAC requires information?: Yes; 
EAC specifies information scope and level of detail?: Yes; 
EAC specifies how information is to be evaluated and criteria to be 
used?: Yes. 

Accreditation program feature: Assurance of financial stability; 
EAC requires information?: Yes; 
EAC specifies information scope and level of detail?: No; 
EAC specifies how information is to be evaluated and criteria to be 
used?: No. 

Source: GAO analysis of EAC data. 

[End of table] 

At the time of our review, EAC's Director of Voting System Testing and 
Certification[Footnote 32] told us that the draft accreditation manual 
was to be submitted for approval and that this draft did not address 
all of the limitations cited above.[Footnote 33] For example, it would 
not contain the information needed and the evaluation approach and 
criteria to be used in making determinations about financial stability 
because this decision is to be based on what the director referred to 
as a "reasonableness" test that involves EAC evaluation of the 
information relative to that provided by other laboratories. Further, 
while EAC officials said that they plan to evolve their approach to 
VSTL accreditation and to address these gaps, EAC does not have 
documented plans for accomplishing this. Without clearly defining 
information to be used and how it is to be used, EAC increases the risk 
that financial stability determinations will not be consistently and 
thoroughly made. 

Available Documentation Does Not Demonstrate EAC's Basis for 
Accrediting Laboratories to Date: 

As of May 2008, EAC has accredited four laboratories,[Footnote 34] but 
the documentation associated with each of these accreditations is not 
sufficient to recreate a meaningful understanding of how each 
evaluation was performed and how decisions were made, and thus, the 
bases for each accreditation were not clear. Specifically, each of the 
accreditations occurred before EAC had defined its approach for 
conducting them. Because of this, EAC performed each one using a 
broadly defined process outlined in a letter to each laboratory and an 
associated checklist that only indicated whether certain documents were 
received. Our analysis of these letters showed that the correspondence 
sent to each laboratory was all the same, identifying three basic 
review steps to be performed and citing a list of documents that the 
laboratories were to provide as part of their applications.[Footnote 
35] However, the letters did not describe in any manner how EAC would 
review the submitted material, including the criteria to be used. 

According to EAC officials, the review steps were not documented. 
Instead, they were derived by a single reviewer using (1) the 
applications and accompanying documents submitted by the laboratories, 
(2) familiarity with the materials used by the state election directors-
sponsored accreditation program, and (3) the judgment of each reviewer. 
Further, while the reviews were supported by a checklist that covered 
each of the items that was to be included in the laboratory 
applications and provided space for the reviewer(s) to make notes 
relative to each of these items, the checklists did not include any 
guidance or methodology, including criteria, for evaluating the 
submitted items. Rather, the EAC accreditation program director told us 
that he was the reviewer on all the accreditations and he applied his 
own, but undocumented, tests for reasonableness in deciding on the 
submissions' adequacy and acceptability. 

Our analysis of the checklists for each laboratory accreditation showed 
that while the same checklist was used for each laboratory, the 
checklists did not provide a basis for evaluating and documenting the 
basis for the sufficiency of those documents. In some cases, additional 
communications occurred between the reviewer and the laboratory to 
obtain additional documents. However, no documentation was available to 
demonstrate what standards or other criteria the laboratories were held 
to or how their submissions were otherwise reviewed. For example, each 
of the checklists indicated that each laboratory provided "a copy of 
the laboratory's conflict of interest policy." However, they did not 
specify, for example, whether the policy adequately addressed 
particular requirements. Nevertheless, for three of the four accredited 
laboratories, documentation shows that EAC sought clarification on or 
modification to the policies provided, thus suggesting that some form 
of review was performed against more detailed requirements. Similarly, 
while the checklists indicate that the laboratories disclosed their 
respective coverage limits for general liability insurance policies, 
and in one case EAC communicated to the laboratory that the limits 
appeared to be low, no documentation specifies the expected coverage 
limits. According to the EAC Director of Voting System Testing and 
Certification, this determination was made after comparing limits among 
the laboratories and was not based on any predetermined threshold. 
Further, while the checklists indicate that each laboratory provided 
audited financial statements, there is no documentation indicating how 
these statements were reviewed. 

According to the EAC program director, the lack of documentation 
demonstrating the basis for EAC's laboratory accreditations is due to 
the need at the time to move quickly in accrediting the laboratories 
and the fact that use of the same individual to review the 
accreditation evaluation negated the need for greater documentation. 
Without such documentation, however, we could not fully establish how 
the accreditations were performed, including whether there was an 
adequate basis for the: 

accreditation decisions reached and whether they were performed 
consistently. 

Conclusions: 

The effectiveness of our nation's overall election system depends on 
many interrelated and interdependent variables, including the security 
and reliability of voting systems. Both NIST and EAC play critical 
roles in ensuring that the laboratories that test these two variables 
have the capability, experience, and competence necessary to test a 
voting system against the relevant standards. NIST has recently 
established an accreditation program that largely accomplishes this, 
and while EAC is not as far along, it has a foundation upon which it 
can build. 

However, important elements are still missing from both programs. 
Specifically, the current NIST approach does not define requirements 
for assessor qualifications and training or ensure that assessments are 
fully documented. Additionally, EAC has not developed program 
management practices that are fully consistent with what NIST has found 
to be hallmarks of an effective accreditation program, nor has the 
agency adequately specified how evaluations are to be performed and 
documented. As a result, opportunities exist for NIST and EAC to 
further define and implement their respective programs in ways that 
promote greater consistency, repeatability, and transparency--and thus 
improve the results achieved. It is also important for NIST and EAC to 
follow through on their stated intentions to evolve their respective 
programs, building on what they have already accomplished through the 
development and execution of well-defined plans of action. If they do 
not, both will be challenged in their ability to consistently provide 
the American people with adequate assurance that accredited 
laboratories are qualified to test the voting systems that will 
eventually be used in U.S. elections. 

Recommendations for Executive Action: 

To help NIST in evolving its VSTL accreditation program, we recommend 
that the Director of NIST ensure that the accreditation program manager 
develops and executes plans that specify tasks, milestones, resources, 
and performance measures that provide for the following two actions: 

* Establish and implement transparent requirements for the technical 
qualifications and training of accreditation assessors. 

* Ensure that each laboratory accreditation review is fully and 
consistently documented in accordance with NIST program requirements. 

To help EAC in evolving its VSTL accreditation program, we recommend 
that the Chair of the EAC ensure that the EAC Executive Director 
develops and executes plans that specify tasks, milestones, resources, 
and performance measures that provide for the following action: 

* Establish and implement practices for the VSTL accreditation program 
consistent with accreditation program management guidance published by 
NIST and GAO, including: 

- documentation of specific accreditation steps and criteria to guide 
assessors in conducting each laboratory review; 

- transparent requirements for the qualifications of accreditation 
reviewers; 

- requirements for the adequate maintenance of records related to the 
VSTL accreditation program; and: 

- requirements for determining laboratory financial stability. 

Agency Comments and Our Evaluation: 

Both NIST and EAC provided written comments on a draft of this report, 
signed by the Deputy Director of NIST and the Executive Director of 
EAC, respectively. These comments are described below along with our 
response to them. 

In its comments, NIST stated that it appreciates our careful review of 
its VSTL program and generally concurs with our conclusions that its 
program must continue to evolve and improve. However, NIST also 
provided comments to clarify the current status of the program relative 
to three of our findings. 

* With respect to our finding that NIST's defined approach for 
accrediting VSTLs does not cite explicit qualifications for the persons 
who conduct the technical assessments, the institute stated that it 
does explicitly cite assessor qualifications for its overall national 
laboratory accreditation program, adding that this approach to 
specifying assessor qualifications has a proven record of success. It 
also stated that the overall program's management manual requires all 
assessors to meet defined criteria in such areas as laboratory 
experience, assessment skills, and technical knowledge, and that 
candidate assessors must submit information addressing each of these 
areas as well as factors addressing technical competence in a given 
laboratory's focus area (e.g., voting systems). Further, it stated that 
candidate assessors' qualification ratings and rankings are captured in 
work sheets. 

In response, we do not disagree with any of these statements. However, 
our finding is that NIST's defined approach for VSTL accreditation does 
not specify requirements for persons who assess those laboratories that 
specifically test voting systems. In this regard, NIST's own written 
comments confirm this, stating that specific requirements for assessors 
are not separately documented for each of its national laboratory 
accreditation programs, such as the VSTL program. Therefore, we have 
not modified this finding or the related recommendation. 

* Regarding our finding that NIST's defined approach for accrediting 
VSTLs has not always cited the current voting system standards, the 
institute affirmed this in its comments by stating that the VSTL 
program handbook that it provided to us only cites the 2002 system 
standards, as these were the only standards in place when the handbook 
was published. However, NIST also noted that when the 2005 system 
guidelines were adopted in December 2005, it began the process of 
updating the handbook and associated assessment checklist, and that the 
handbook update was recently finalized for publication and is now being 
used. 

In response, we stand by our finding that NIST's defined approach has 
not always cited the current voting system standards, which NIST 
acknowledges in its comments. However, we also recognize that NIST has 
recently addressed this inconsistency by finalizing its new handbook 
and the associated assessment checklist. In light of NIST's recent 
actions, we have updated the report to acknowledge the finalization of 
the handbook and checklist, and removed the associated recommendation 
that was contained in our draft report for NIST to ensure that its 
defined approach addresses all required voting system standards. 

* Regarding our finding that available documentation from completed 
accreditations does not show that NIST has consistently followed all 
aspects of its defined approach, the institute stated that, among other 
things, all required documents for its VSTL accreditation program are 
currently in use and reflect the recent update to its handbook and 
checklist, and that all these documents are securely maintained. 

In response, we do not question these statements; however, they are not 
pertinent to our finding. Specifically, our finding is that the four 
completed accreditations that we reviewed were not consistently 
documented. As we state in our report, we reviewed the documentation 
associated with the accreditation assessments for these four 
laboratories, and we found that all four were not documented in a 
similar manner, even though they were based on the same version of the 
program handbook. For example, neither the laboratory notifications of 
the scope of the assessment nor the next steps in the accreditation 
process were consistently documented. Therefore, we have not modified 
our finding, but have slightly modified our recommendation to make it 
clear that its intent is to ensure that all phases of the accreditation 
review are fully and consistently documented. 

In its comments, EAC described our review and report as being helpful 
to the commission as it works to fully develop and implement its VSTL 
program. It also stated that it agrees with the report's conclusions 
that additional written internal procedures, standards, and 
documentation are needed to ensure more consistent and repeatable 
implementation of the program. The commission added that it generally 
accepts our recommendations and will work hard to implement them. To 
assist it in doing so, it sought clarification about two of our 
recommendations, as discussed below. 

* EAC stated that the recommendation in our draft report for the 
commission to develop specific accreditation steps and criteria was 
broadly worded, and thus the recommendation's intent was not clear. EAC 
also stated that it interpreted the recommendation to mean that it 
should define internal instructions to guide assessors in performing an 
accreditation, and that the recommendation was not intended to have any 
impact on its published requirements and procedures governing, for 
example, granting, suspending, or withdrawing an accreditation. We 
agree with EAC's interpretation, as it is in line with the intent of 
our recommendation. To avoid the potential for any future 
misunderstanding, we have modified the wording of the recommendation to 
clarify its intent. 

* EAC stated that the recommendation in our draft report for the 
commission to develop transparent technical requirements for the 
qualifications of its assessors may be confusing because, as we state 
in our report, only NIST performs a technical accreditation review, as 
EAC's review is administrative, non-technical in nature. To avoid the 
potential for any confusion, we have modified the wording of the 
recommendation to eliminate any reference to technical qualification 
requirements. 

We are sending copies of this report to the Ranking Member of the House 
Committee on House Administration, the Chairman and Ranking Member of 
the Senate Committee on Rules and Administration, the Chairmen and 
Ranking Members of the Subcommittees on Financial Services and General 
Government, Senate and House Committees on Appropriations, and the 
Chairman and Ranking Member of the House Committee on Oversight and 
Government Reform. We are also sending copies to the Chair and 
Executive Director of EAC, the Secretary of Commerce, the Deputy 
Director of NIST, and other interested parties. We will also make 
copies available to others on request. In addition, this report will be 
available at no charge on the GAO Website at [hyperlink, 
http://www.gao.gov]. 

Should you or your staffs have any questions on matters discussed in 
this report, please contact me at (202) 512-3439 or at [email protected]. 
Contact points for our Offices of Congressional Relations and Public 
Affairs may be found on the last page of this report. Key contributors 
to this report are listed in appendix IV. 

Sincerely yours, 

Randolph C. Hite: 
Director, Information Technology Architecture and System Issues: 

[End of section] 

Appendix I: Objectives, Scope, and Methodology: 

Our objectives were to determine whether the National Institute of 
Standards and Technology (NIST) and the Election Assistance Commission 
(EAC) have defined effective voting system testing laboratory (VSTL) 
accreditation approaches, and whether each is following its defined 
approach. 

To determine whether NIST has defined an effective accreditation 
approach, we reviewed documentation from its VSTL accreditation 
program, such as handbooks and program manuals for the National 
Voluntary Laboratory Accreditation Program (NVLAP), of which the VSTL 
accreditation program is a part. In doing so, we compared these 
documents with applicable statute, guidance, and best practices, 
primarily the Help America Vote Act of 2002 (HAVA), internationally 
recognized standards from the International Organization for 
Standardization (ISO), and federal accreditation program management 
guidance published by NIST. We compared program documentation with 
HAVA's NIST-specific accreditation requirements to determine the extent 
to which the agency was fulfilling its HAVA responsibilities. We also 
reviewed program documentation against ISO/IEC 17011,[Footnote 36] 
which establishes general requirements for accreditation bodies, and 
ISO/IEC 17025,[Footnote 37] which establishes the general requirements 
for assessing the competence of laboratories, to determine the extent 
to which NIST's accreditation program was based on internationally 
recognized standards. We also compared the documentation against NIST 
publication NISTIR 6014,[Footnote 38] which contains sections that 
provide guidance for laboratory accreditation programs, to determine 
whether the VSTL accreditation program had defined other elements of 
effective accreditation programs. We also interviewed the voting 
accreditation program manager to determine how these documents were 
used to guide the program. 

To determine whether NIST has followed its defined approach, we 
examined artifacts from the accreditation assessments of five VSTLs, 
including one laboratory accredited by NVLAP, but not yet recommended 
to EAC. This material included completed assessment checklists derived 
from the accreditation program handbooks, additional documents 
supporting the assessments, and laboratory accreditation applications 
and supporting documentation. We compared artifacts from these 
assessments to program guidance to determine the extent to which the 
defined process was followed. In addition, we interviewed officials 
from NIST and NIST contract assessors and officials from EAC and the 
four EAC-accredited VSTLs to understand how the NIST process was 
implemented and how it related to the process managed by EAC. 

To determine whether EAC has defined an effective accreditation 
approach, we reviewed documentation from its VSTL accreditation 
program, such as the draft Voting System Test Laboratory Accreditation 
Program Manual.[Footnote 39] In doing so, we compared this document 
with applicable statute and best practices, primarily HAVA and federal 
accreditation program management guidance published by NIST. We 
compared the draft program manual with HAVA's EAC-specific 
accreditation requirements to determine the extent to which the agency 
was fulfilling its HAVA responsibilities. We also compared the 
documentation against the accreditation guidance in NISTIR 6014 to 
determine whether the accreditation program had defined other elements 
of effective accreditation programs. We also interviewed the EAC voting 
program director and executive director to determine how these 
documents were used to guide the program and to understand EAC's 
defined accreditation approach prior to the development of the draft 
manual. 

To determine whether EAC has followed its defined approach, we compared 
artifacts from the accreditation reviews of four VSTLs. We did not 
review a fifth laboratory, which had been accredited by NVLAP, but not 
yet recommended to EAC. The materials reviewed included checklists 
completed by EAC in the absence of an approved program manual. In doing 
so, we compared the review artifacts to accreditation program 
requirements, as communicated to the laboratories, to determine the 
extent to which the agency followed its process, as verbally described 
to us. We did not compare accreditation submissions or EAC review 
artifacts with the draft accreditation manual because agency officials 
stated that the draft manual had not been used in the review of any 
laboratory. In addition, we interviewed officials from NIST, EAC, and 
the four EAC-accredited VSTLs[Footnote 40] to understand how the EAC 
process was implemented and how it related to the process managed by 
NIST. 

To assess data reliability, we reviewed program documentation to 
substantiate data provided in interviews with knowledgeable agency 
officials. We have also made appropriate attribution indicating the 
data's sources. 

We conducted this performance audit at EAC and NIST offices in 
Washington, D.C., and Gaithersburg, Maryland, respectively, from 
September 2007 to September 2008 in accordance with generally accepted 
government auditing standards. Those standards require that we plan and 
perform the audit to obtain sufficient, appropriate evidence to provide 
a reasonable basis for our findings and conclusions based on our audit 
objectives. We believe that the evidence obtained provides a reasonable 
basis for our findings and conclusions based on our audit objectives. 

[End of section] 

Appendix II: Comments from the National Institute of Standards and 
Technology: 

United States Department Of Commerce: 
National Institute of Standards and Technology: 
Gaithersburg, Maryland 20899-0001: 
Office Of The Director: 

Memorandum For: 
Randolph C. Hite: 
Director, Information Technology Architecture and System Issues: 
Government Accountability Office: 

From: 
Signed by: 
James M. Turner, Ph.D.: 
Deputy Director:  

Subject: Comments on Government Accountability Office (GAO) Draft 
Report Entitled "Elections: Federal Programs for Accrediting 
Laboratories that Test Voting Systems Need to Be Better Defined and 
Implemented" (GAO-08-770): 

This is in response to your draft report dated March 2008, entitled 
"Elections: Federal Programs for Accrediting Laboratories that Test 
Voting Systems Need to Be Better Defined and Implemented." Thank you 
for the opportunity to review and comment on this draft. 

I appreciate the GAO team's careful review of the NIST/NVLAP program to 
accredit voting system testing laboratories under the EAC program. I am 
pleased that the GAO review found that the NIST voting system 
accreditation program reflects HAVA requirements, incorporates relevant 
international accreditation standards and meets International 
Organization for Standardization (ISO) accreditation body requirements, 
and laboratory accreditation review requirements. 

The draft report also states, however, that the NIST program is missing 
details needed for consistent and verifiable implementation. While NIST 
generally concurs with the GAO findings that the Voting System Testing 
Laboratory (VSTL) program must continue to evolve and be enhanced over 
time, we think it is important to clarify the current state of the 
program, particularly the state of the documentation associated with 
the program's consistent and verifiable implementation. In the 
attachment to this memorandum, we respond to the three areas of 
specific concern identified in the draft report regarding NIST's role 
in the voting system accreditation process provided for under HAVA. 

I am confident that the accreditation program implemented by NVLAP 
currently fully meets the requirements of HAVA and will lead to 
increased confidence in the electronic voting process. NIST remains 
committed to improving the operation of the program as we move forward. 

We are looking forward to receiving your final report. Please contact 
Steve Willett on (301) 975-8707 should you have any questions regarding 
this response. 

Attachment: 

Below is the NIST response to the three areas of specific concern 
identified in the draft report regarding NIST's role in the voting 
system accreditation process provided for under HAVA. 

(1) First, the draft findings state that NIST has not specified 
requirements for assessor qualifications and training. Specifically, 
"NIST's defined approach does not, for example, cite explicit 
qualifications for the persons who conduct accreditation technical 
assessments, as called for in federal accreditation program guidance." 

Response ï¿½ The NVLAP program does cite explicit qualifications for 
technical assessors, as required under the provisions of ISO/IEC 
17011:2004, the international standard that establishes general 
requirements for accreditation bodies. This standard is incorporated in 
the NVLAP Management System Manual. Section 6.3.2.1 of the manual, 
which addresses assessor selection criteria, states, "NVLAP uses the 
services of assessors and technical experts who meet defined criteria 
in the areas of laboratory experience, communication, assessment 
skills, technical knowledge, quality management, professional 
activities, and education and training." Candidate assessors must 
submit biographies and supporting information that address each of the 
above criteria, in each case specifically addressing factors that 
demonstrate technical competence relevant to a specific NVLAP 
laboratory accreditation program or programs, e.g., voting system 
testing. 

To evaluate a potential assessor, NVLAP relies on the assessor 
biography, interviews, and recommendations to prepare an assessor 
summary sheet that documents the basis for selection. Rating work 
sheets are used to rank the assessor's knowledge and ability in the 
areas of: education, lab experience, communication skills, assessment 
experience, lab management experience, and professional association 
affiliations. While specific requirements for assessors are not 
separately documented for each NVLAP program, the procedure currently 
in use has a proven record of success within NVLAP and other 
accreditation bodies. NVLAP also monitors its assessors and solicits 
written comments from customer laboratories covering many issues, 
including assessor technical competence. Initial qualification, 
monitoring and laboratory feedback information is documented and 
maintained in the assessor files in a secured file room. 

The GAO report also states that, "NIST officials said that they rely on 
individuals who have prior experience in reviewing such laboratories." 
All NVLAP programs rely on experts that have prior experience; this is 
viewed as an asset. Assessors are selected for their technical 
expertise and trained in assessment techniques and NIST Handbook 150 
(NVLAP's version of ISO/IEC 17025). This is accomplished by NVLAP 
through workshops, on the job training, and/or one-on-one training. The 
NVLAP technical assessor supporting the voting system testing 
laboratory program has both extensive prior experience in voting system 
testing and evaluating technical competence through the National 
Association of State Election Directors (NASED) assessment program and 
current experience as an EAC evaluator. This is documented in his 
assessor file. The voting system lead assessor has over 10 years of 
experience as a NVLAP lead management system assessor and as a 
technical assessor in the area of electromagnetic compatibility and 
telecommunications. He has attended courses offered periodically by 
NVLAP to its assessors and has been qualified as a NVLAP assessor with 
the appropriate certificate. This is documented in his assessor file. 

(2) Second, the draft findings state that the current NIST approach 
does not fully specify criteria for evaluating and documenting VSTL 
capabilities. Specifically, "Even though the EAC requires that 
laboratory accreditation be based on demonstrated capabilities to test 
against the latest voting system standards, NIST's defined approach has 
not always cited these current standards." The draft report cites 
instances where the technical (program specific) checklists were not 
always consistent for the on-site assessments and notes that some 
laboratories stated they did not know what was expected of them. This 
finding refers to the development and implementation of the program 
specific handbook and checklist. 

Response - When the NVLAP voting system accreditation program was first 
made available to interested laboratories in June 2005, the 2002 VVS 
was the only approved standard in existence. Therefore, NVLAP's initial 
program specific handbook took only this standard into account. The EAC 
formally adopted the 2005 VVSG in December 2005, with an effective date 
of December 2007. NVLAP informed the candidate VSTLs in early 2006 that 
their scope of accreditation would include both the 2002 VVS and 2005 
VVSG. The process was begun at that time to revise the initial program 
specific handbook to reflect both standards. 

Candidate VSTLs were clearly told by NIST that the accreditation would 
be to both the 2002 VVS and 2005 VVSG. The labs were informed that in 
order to be accredited as Voting System Testing Laboratories by NVLAP 
they would have to be accredited to the core set of test methods 
contained in both standards. This was communicated to the labs during 
the pre- assessments, in e-mail and phone communications, and at the on-
site assessment visit. In addition the candidate VSTLs should have 
known this since the same information was available on the EAC website. 

NVLAP conducted no-cost pre-assessments of each candidate VSTL, 
beginning in February 2006. In parallel, a program specific checklist 
was being developed to encompass both standards. During these pre-
assessments (lasting an average of two days), the laboratory's 
technical and quality systems were reviewed, personnel interviewed and 
issues discussed. The program specific handbook was updated and 
approved at the NVLAP level in December 2007. It was denoted as being 
in draft form at that time because it had not been through a NIST 
internal editorial review process needed for hardcopy publication. This 
editorial process has been completed and the handbook has been 
finalized for publication. Final versions of the program specific 
handbook and checklist are now in use. Each VSTL has been assessed to 
both the 2002 VVS and 2005 VVSG. 

NVLAP's general requirements are found in NIST Handbook 150. Program 
specific handbooks and checklists are provided by NVLAP as a tool for 
identifying and documenting assessment findings and nonconformities 
relative to the test methods being assessed. They incorporate the same 
numbering system and section headings as Handbook 150. The difference 
between the general and program specific handbooks is that the program 
specific handbook (Handbook 150-22 in the case of the voting program) 
provides additional guidance for coverage of specific test methods 
without repeating the general criterion in Handbook 150. In the case of 
the voting system testing program, the specific test methods are found 
in 2002 VVS and 2005 VVSG. There are roughly one thousand test methods 
in the 2005 VVSG alone. The goal of Handbook 150-22 is not to reiterate 
every requirement in the technical standards, since these are already 
stated in the standards themselves, but to provide a framework so the 
laboratory will know where elements of the standard will be considered 
and recorded in our documentation. NVLAP's function is not to educate 
the laboratories in the standard but to document their competency in 
applying voting system standards to equipment under test. 

(3) The third finding was that available documentation does not show 
that NIST has consistently followed all aspects of its defined 
accreditation approach. Specifically, the GAO report recommends that 
the NIST/NVLAP program manager needs to provide actions for, "Ensuring 
that each laboratory accreditation review is fully documented." 

Response: All required documents for the voting system testing program 
are currently in use and include the most recent, up-to-date Handbook 
150-22 with its checklist. The documentation used for accreditation 
review by NVLAP includes: the lab application, laboratory management 
system manual, on-site report, lab responses to nonconformities, and 
documentation to support the laboratory's response. The on-site report 
consists of the general operations checklist, program specific 
checklist, signature sheet, narrative summary, and assessment summary. 
All these documents remain on file in a secure NVLAP file room. 

Every NVLAP program documents the evaluation using the NVLAP On-site 
Assessment Review form. Additionally, as part of the on-site assessment 
report, the assessor completes a narrative summary addressing the 
overall findings of each section of the standard as well as the 
assessment checklists for both the NVLAP general and program specific 
handbooks. The assessment process requires that the assessor leave 
copies of the assessment report with the lab on the last day of the on-
site assessment. 

Unlike NVLAP's other programs, where strict confidentiality is 
observed, the voting program must exhibit a level of transparency in 
order to remain open to the public. To accomplish this, the on-site 
report, lab responses, and evaluation form are made available on the 
NIST voting website. Proprietary information in the nonconformity 
response documentation is not posted. 

Over the past several years NIST has worked to provide the American 
public with a comprehensive accreditation program capable of assessing 
the competency of voting system testing laboratories to current 
standards while also maintaining a process which will allow the program 
to accommodate the development of new standards. NIST recognizes the 
need for an unbiased assessment of its process and commends the GAO for 
providing this service. The NVLAP voting system testing laboratory 
program has gone through a maturation process since 2004. NIST is 
confident that the accreditation program implemented by NVLAP currently 
fully meets the requirements of HAVA and will lead to increased 
confidence in the electronic voting process. We remain committed to 
improving the operation of the program as we move forward. In 
particular, as recommended by the GAO, we will continue to enhance the 
transparency of NVLAP requirements for the technical qualifications and 
training of assessors, ensure that all required voting system standards 
are addressed, and ensure that each laboratory accreditation review is 
fully documented. 

[End of section] 

Appendix III: Comments from the Election Assistance Commission: 

U.S. Election Assistance Commission: 
Office Of The Executive Director: 

1225 New York Avenue, NW, Suite 1100: 
Washington, DC. 20005: 

Mr. Randolph C. Hite, Director: 
Information Technology Architecture and Systems: 
United States Government Accountability Office:  

Re: Draft Report GAO 08-770, Elections: Federal Programs for 
Accrediting Laboratories that Test Voting Systems Need to Be Better 
Defined and Implemented: 

Thank you for the opportunity to provide comments on GAO Report 08-770 
regarding Federal voting system test laboratory accreditation programs. 
The U.S. Election Assistance Commission (EAC) appreciates the time and 
effort put forth by GAO during the preparation of this document and the 
willingness of GAO staff to discuss pertinent issues at length with the 
EAC, NIST and other key stakeholders in the voting system testing 
community. The EAC has found both the review process and report helpful 
as it works to fully develop and implement its new Laboratory 
Accreditation Program. 

GAO recognized that EAC has successfully "published procedures 
governing how the accreditation program is to be executed" (pgs 31-32) 
and that its program includes requirements for all five recognized 
accreditation program features (pgs 33-34). The EAC agrees with the 
report's conclusion that while EAC's accreditation program has 
effectively laid out its program requirements, additional written 
internal procedures, standards and documentation are needed to ensure a 
more consistent and repeatable review process. The EAC is currently 
developing written, internal procedures to better implement and 
document its Laboratory Accreditation Manual. As the EAC implements the 
various aspects of its accreditation program, it will incorporate the 
recommendations provided in the GAO report. 

The report provides four recommendations for the EAC to better conform 
to accreditation program management guidance published by NIST and GAO. 
Generally, the EAC accepts the recommendations provided with little 
comment. However, in one instance the phrasing of a recommendation is 
questioned in light of the report's findings. Additionally, the EAC 
expresses concern that certain technical responsibilities under the 
purview of NIST are placed upon the EAC. The following are EAC's 
comments in response to each recommendation. 

1. Documentation of specific accreditation steps and criteria. This 
recommendation is stated broadly and, as a result, is unclear in light 
of the content of the report. Despite its broad language, the EAC does 
not read this recommendation as impacting the program's laboratory 
accreditation requirements, as the report has found that the EAC has 
properly "published procedures governing how the accreditation program 
is to be executed, such as procedures for granting, maintaining, 
modifying, suspending, and withdrawing accreditation." (pg 31 ï¿½ 32). 
Rather, based upon the discussion on page 31-32 of the report, the EAC 
reads this recommendation as advising that EAC develop additional 
"instructions, steps, and criteria for those who conduct an 
accreditation assessment." 

Based upon this reading, the EAC concurs with GAO's recommendation to 
develop a more formalized internal process for reviewing accreditation 
applications. The EAC has a checklist in place to guide the process; 
however we will review the document with an eye toward improving it by 
developing additional internal procedures and criteria for reviewing 
applications. 

2. Transparent technical requirements for the qualifications of 
accreditation reviewers. EAC is concerned that this recommendation may 
confuse readers of this report regarding the distinct roles of EAC and 
NIST in the laboratory accreditation process. According to HAVA, NIST 
is responsible for the technical review of voting system test 
laboratories; the EAC performs a "nontechnical" review of laboratories 
after the technical review by NIST is completed. As the GAO report 
notes, "NIST is to focus on assessing laboratory technical 
qualifications, while the EAC is to use those assessment results and 
recommendations, and augment them with its own review of related 
laboratory capabilities, to reach an accreditation decision." (pg 13). 
EAC's review is administrative, and it is initiated only after 
receiving a laboratory accreditation recommendation from NIST. 

After it has received technical recommendation from NIST, the EAC 
conducts an administrative review of the applicant laboratory. This 
includes collecting: (1) laboratory information required by Section 
3.4.1. of EAC Draft Voting System Test Laboratory Program Manual 
(Accreditation Manual); (2) a signed agreement affirming that the 
laboratory will meet all elements of the EAC Accreditation Program 
(Chapter 2 of the Accreditation Manual); and (3) a Certification of 
Laboratory Conditions and Practices (Attachment A of the Accreditation 
Manual) documenting that the laboratory has certain conditions and 
required policies in place (specifically: a conflict of interest 
policy, a personnel policy, a recordkeeping policy and evidence of 
sufficient resources and financial stability). Thus, the EAC's 
nontechnical review is limited to (1) determining whether required 
information and the signed agreement and certification are, in fact, 
provided and (2) determining whether the certifications of the 
laboratory (concerning its conflict of interest policy, personnel 
policy, recordkeeping policy and resources and financial stability) are 
acceptable by comparing the policies provided with the EAC program 
requirements. 

EAC's staff has the appropriate credentials for nontechnical reviews of 
laboratory applicants in accordance with the accreditation process 
contained in HAVA. These qualifications include a detailed 
understanding of EAC's Accreditation Program and its requirements. The 
EAC will place this qualification and any others deemed necessary in 
the internal procedures it is creating (as noted above). 

3. Requirements for the adequate maintenance of records related to the 
VSTL accreditation program. This recommendation is similar to 1 and 2, 
above. EAC agrees that more formalized review procedures would benefit 
its accreditation program. Documentation is a key element of such 
action. The adoption of EAC's Accreditation Manual (anticipated date of 
approval, July, 2008) will go a long way to assist with improving the 
laboratory accreditation process. The manual prescribes an application 
package which includes a certification form, an agreement document and 
specific information. As noted above, EAC is already using an 
application checklist which it plans to amend consistent with GAO 
recommendations. Moreover, the Commission will ensure that its internal 
procedures require formal documentation of each EAC action and basis 
for it. The EAC is making documentation a priority throughout its 
Laboratory Accreditation Program. 

The EAC has made it a priority to provide the public with program 
information, including correspondence and many other program-related 
materials. This information is readily available at [hyperlink, 
http://www.eac.gov], and stakeholders including State and local 
election officials, voting system manufacturers, test laboratories and 
the voting public are notified frequently about program updates. The 
EAC also provides links to program material generated by NIST's 
technical reviews of the laboratories. 

4. Requirements for determining laboratory financial stability. The GAO 
Report specifies five program features (or areas of required 
information) that an accreditation program should include. As the GAO 
report notes, EAC's Accreditation Program Manual contains requirements 
for each of these features, including: organizational information, 
records and record keeping, test methods and procedures, conflict of 
interest policy and assurance of financial stability. However, for the 
last requirement, GAO has recommended that the EAC provide additional 
criteria for review and obtain additional laboratory information. 

EAC's Draft Accreditation Manual provides a concise requirement for 
laboratory financial stability. "As a condition of accreditation, all 
VSTLs shall allocate sufficient resources to enable the laboratory to 
properly use and maintain its test equipment, personnel, and facility 
and to satisfactorily perform all required laboratory functions. The 
laboratory shall maintain insurance policies sufficient to indemnify, 
itself against financial liabilities or penalties that may result from 
its operations." (Accreditation Manual ï¿½2.14). The EAC agrees that this 
requirement should look more like its other four program features and 
provide more specific criteria. The EAC will amend its draft manual to 
reflect GAO's recommendations before the document's final publication 
in July. In addition, the EAC will require additional information with 
respect to the issue of financial stability. Presently the draft manual 
requires the submission of insurance, workman's compensation, staffing, 
facilities, and financial reporting information. (Accreditation Manual 
ï¿½3.4.1). 

The EAC recognizes that the Federal government's first voting system 
test laboratory accreditation program must have a solid foundation. The 
EAC continues to evaluate its progress and identify issues of concern 
to ensure that this new program is robust and thorough. For instance, 
the EAC recently asked NIST (Attachment; EAC letter to NIST Standards 
Services Division Chief Mary Saunders, March 13, 2008. Also available 
at [hyperlink, http://www.eac.gov] under the Voting Systems Center.) to 
address and report back to the EAC on some areas of concern noted by 
the EAC in its review of VSTL work products. We also requested NIST to 
review the laboratories' management process for assigning appropriate 
levels of staff, the qualifications of laboratory employees and how 
federal testing assignments are prioritized. Further, we requested a 
review of the laboratories' test methods and the validation process 
used to ensure that "NIST remains confident of each VSTL's ability to 
test voting systems." 

The EAC will continue to work with NIST to ensure that the laboratory 
accreditation process is monitored very closely, and after NIST produce 
this additional information, we will provide it to GAO, Congress, and 
the public. 

Keeping stakeholders and the general public informed about program 
activities, milestones and challenges continues to be a top priority. A 
May 2, 2008 memo (attached and available at [hyperlink, 
http://www.eac.gov] under Voting System Center), to election officials 
from Program Director Brian Hancock indicated that challenges remain in 
the certification process, such as the length of time it takes 
accredited laboratories to evaluate voting systems. Some participants 
in the certification program are experiencing challenges meeting the 
requirements of a more robust and thorough testing process, as 
evidenced by the submission of nine test plans from a single testing 
engagement. All draft and final test plans as well as test reports are 
available at [hyperlink, http://www.eac.gov] under the Voting System 
Center. 

The EAC thanks GAO for its work in assisting the Commission in its 
efforts to improve and develop the Voting System Laboratory 
Accreditation Program. The EAC is committed to continuous improvement 
in all of it programs and will work hard to implement the 
recommendations made in this report. The EAC is committed to making its 
accreditation process part of a world class testing and certification 
program. 

Sincerely, 

Signed by: 

Thomas R. Wilkey: 
Executive Director: 

[End of section] 

Appendix IV: GAO Contact and Staff Acknowledgments: 

GAO Contact: 

Randolph C. Hite, (202) 512-3439 or [email protected]: 

Staff Acknowledgments: 

Paula Moore, Assistant Director; Justin Booth; Timothy Case; Neil 
Doherty; Timothy Eagle; Nancy Glover; Dave Hinchman; Rebecca LaPaze; 
Freda Paintsil; Nik Rapelje; and Jeffrey Woodward made key 
contributions to this report. 

[End of section] 

Footnotes: 

[1] See, for example, GAO, Elections: Perspectives on Activities and 
Challenges Across the Nation, GAO-02-3 (Washington, D.C.: Oct. 15, 
2001); Elections: Status and Use of Federal Voting Equipment Standards, 
GAO-02-52 (Washington, D.C.: Oct. 15, 2001); Elections: A Framework for 
Evaluating Reform Proposals, GAO-02-90 (Washington, D.C.: Oct. 15, 
2001); Elections: Federal Efforts to Improve Security and Reliability 
of Electronic Voting Systems Are Under Way, but Key Activities Need to 
Be Completed, GAO-05-956 (Washington, D.C.: Sept. 21, 2005); and 
Elections: All Levels of Government Are Needed to Address Electronic 
Voting System Challenges, GAO-07-576T (Washington, D.C.: Mar. 7, 2007). 

[2] GAO-02-52. 

[3] 42 U.S.C. ï¿½ 15371. 

[4] In August 2008, after EAC provided comments on, and we had 
finalized, this report, the commission announced that it had approved 
its draft accreditation program manual. As a result, we did not review 
the approved manual. 

[5] GAO-02-3. 

[6] GAO-02-3. 

[7] GAO, Elections: Electronic Voting Offers Opportunities and Presents 
Challenges, GAO-04-975T (Washington, D.C.: July 20, 2004). 

[8] GAO-05-956. 

[9] Help America Vote Act, Pub. L. No. 107-252 (Oct. 29. 2002). 

[10] Federal Election Commission, Performance and Test Standards for 
Punchcard, Marksense, and Direct Recording Electronic Voting Systems 
(January 1990). 

[11] GAO-05-956. 

[12] The VVSG did not take effect until December 2007. 

[13] We have ongoing work to review EAC's certification program for the 
House Committee on House Administration. 

[14] GAO-02-90. 

[15] The state elections directors' accreditation program was 
discontinued in July 2006. 

[16] A suspension provides the laboratory an opportunity to address the 
identified issues. 

[17] NIST, NIST Handbook 150: National Voluntary Laboratory 
Accreditation Program Procedures and General Requirements 
(Gaithersburg, Md.: February 2006). 

[18] NIST, NIST Handbook 150-22: National Voluntary Laboratory 
Accreditation Program: Voting System Testing (Gaithersburg, Md.: 
December 2005). 

[19] ISO, ISO/IEC 17011: Conformity Assessment: General Requirements 
for Accreditation Bodies Accrediting Conformity Assessment Bodies 
(Geneva, Switzerland: Feb. 15, 2005). 

[20] ISO, ISO/IEC 17025: General Requirements for the Competence of 
Testing and Calibration Laboratories (Geneva, Switzerland: May 15, 
2005). 

[21] NIST, National Voluntary Laboratory Accreditation Program: 
Management System Manual, Revision 2 (Gaithersburg, Md.: Mar. 25, 
2008). 

[22] NIST, The ABC's of the U.S. Conformity Assessment System, NISTIR 
6014 (Gaithersburg, Md.: April 1997). 

[23] NIST, NIST Handbook 150-22. 

[24] NIST, Handbook 150-22 2008 Edition: National Voluntary Laboratory 
Accreditation Program: Voting System Testing (Gaithersburg, Md.: May 
2008); NIST Handbook 150-22 Checklist: Voting System Testing Program 
(Rev. 2008-06-25). 

[25] NIST, National Voluntary Laboratory Accreditation Program: 
Management System Manual, Revision 2. 

[26] As mentioned earlier, these guidelines became effective in 
December 2007. 

[27] EAC, Voting System Test Laboratory Accreditation Program Manual 
(draft, Washington, D.C., February 2008). 

[28] EAC, Testing and Certification Program Manual (Washington, D.C.: 
January 2007). 

[29] NIST, The ABC's of the U.S. Conformity Assessment System, NISTIR 
6014. 

[30] GAO, Laboratory Accreditation: Requirements Vary Throughout the 
Federal Government, GAO/RCED-89-102 (Washington, D.C.: Mar. 28, 1989). 

[31] GAO/RCED-89-102. 

[32] The EAC Director of Voting System Testing and Certification also 
manages EAC's VSTL accreditation program. 

[33] On August 4, 2008, EAC reported that it approved an initial 
version of its program manual. However, we did not evaluate this 
initial version to determine the extent to which it addresses 
limitations that we found in the draft manual because it was approved 
after EAC provided comments on a draft of, and we had finalized, this 
report. 

[34] According to the NIST program manager, a fifth laboratory has been 
accredited by NVLAP but not yet recommended to EAC. 

[35] The steps were to: 1) provide information, such as the 
laboratory's conflict of interest policy, evidence of insurance 
coverage limits, and audited financial statements; 2) provide a signed, 
standardized letter of agreement to abide by the EAC program terms; and 
3) provide a signed certification of laboratory practices and 
conditions, such as having policies in place with respect to personnel 
practices, record-keeping requirements, and financial stability. 

[36] ISO, ISO/IEC 17011: Conformity Assessment: General Requirements 
for Accreditation Bodies Accrediting Conformity Assessment Bodies 
(Geneva, Switzerland: Feb. 15, 2005). 

[37] ISO, ISO/IEC 17025: General Requirements for the Competence of 
Testing and Calibration Laboratories (Geneva, Switzerland: May 15, 
2005). 

[38] NIST, The ABC's of the U.S. Conformity Assessment System, NISTIR 
6014 (Gaithersburg, Md.: April 1997). 

[39] EAC, Voting System Test Laboratory Accreditation Program Manual 
(draft, Washington, D.C.: February 2008). EAC approved the program 
manual for publication in July 2008; however this was accomplished too 
late for GAO to review the manual's contents for this report. 

[40] In June 2008, NIST granted NVLAP accreditation to a fifth 
laboratory, CIBER Inc., and recommended it for EAC accreditation. As of 
August 2008, that accreditation had yet to be granted. 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability.  

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "E-mail Updates."  

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:  

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548:  

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061:  

To Report Fraud, Waste, and Abuse in Federal Programs:  

Contact:  

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470:  

Congressional Relations:  

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548:  

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***