Elections: Results of GAO's Testing of Voting Systems Used in	 
Sarasota County in Florida's 13th Congressional District	 
(08-FEB-08, GAO-08-425T).					 
                                                                 
In November 2006, about 18,000 undervotes were reported in	 
Sarasota County in the race for Florida's 13th Congressional	 
District (Florida-13). After the election results were contested 
in the House of Representatives, the task force unanimously voted
to seek GAO's assistance in determining whether the voting	 
systems contributed to the large undervote in Sarasota County. In
October 2007, GAO presented its findings on the review of the	 
voting systems and concluded that while prior tests and reviews  
provided some assurance that the voting systems performed	 
correctly, they were not enough to provide reasonable assurance  
that the voting systems in Sarasota County did not contribute to 
the undervote. GAO proposed that a firmware verification test, a 
ballot test, and a calibration test be conducted. The task force 
requested that GAO proceed with the proposed additional tests.	 
GAO also verified whether source code escrowed by Florida could  
be rebuilt into the firmware used in Sarasota County. To conduct 
its work, GAO conducted tests on a sample of voting systems used 
in Sarasota County during the 2006 general election. GAO	 
witnessed the rebuild of the firmware from the escrowed source	 
code at the manufacturer's development facility. GAO reviewed	 
test documentation from Florida, Sarasota County, and the voting 
system manufacturer and met with election officials to prepare	 
the test protocols and detailed test procedures.		 
-------------------------Indexing Terms------------------------- 
REPORTNUM:   GAO-08-425T					        
    ACCNO:   A80609						        
  TITLE:     Elections: Results of GAO's Testing of Voting Systems    
Used in Sarasota County in Florida's 13th Congressional District 
     DATE:   02/08/2008 
  SUBJECT:   Calibration					 
	     Elections						 
	     Electronic equipment				 
	     System vulnerabilities				 
	     Systems analysis					 
	     Systems evaluation 				 
	     Systems management 				 
	     Systems monitoring 				 
	     Systems testing					 
	     Voting						 
	     Voting records					 
	     Florida						 

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO Product.                                                 **
**                                                              **
** No attempt has been made to display graphic images, although **
** figure captions are reproduced.  Tables are included, but    **
** may not resemble those in the printed version.               **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
******************************************************************
GAO-08-425T

   

This is a work of the U.S. government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. Because this work 
may contain copyrighted images or other material, permission from the 
copyright holder may be necessary if you wish to reproduce this 
material separately. 

United States Government Accountability Office: 
GAO: 

Statement: 

Before the Task Force for the Contested Election in the 13th 
Congressional District of Florida, Committee on House Administration, 
House of Representatives: 

For Release on Delivery: 
Expected at 10:00 a.m. EST:
Friday, February 8, 2008: 

Elections: 

Results of GAO's Testing of Voting Systems Used in Sarasota County in 
Florida's 13th Congressional District: 

Statement of Nabajyoti Barkakati, Ph.D: Acting Chief Technologist: 
Applied Research and Methods: 

GAO-08-425T: 

GAO Highlights: 

Highlights of GAO-08-425T, a statement before the Task Force for the 
Contested Election in the 13th Congressional District of Florida, 
Committee on House Administration, House of Representatives. 

Why GAO Did This Study: 

In November 2006, about 18,000 undervotes were reported in Sarasota 
County in the race for Floridaï¿½s 13th Congressional District (Florida-
13). After the election results were contested in the House of 
Representatives, the task force unanimously voted to seek GAOï¿½s 
assistance in determining whether the voting systems contributed to the 
large undervote in Sarasota County. In October 2007, GAO presented its 
findings on the review of the voting systems and concluded that while 
prior tests and reviews provided some assurance that the voting systems 
performed correctly, they were not enough to provide reasonable 
assurance that the voting systems in Sarasota County did not contribute 
to the undervote. GAO proposed that a firmware verification test, a 
ballot test, and a calibration test be conducted. The task force 
requested that GAO proceed with the proposed additional tests. GAO also 
verified whether source code escrowed by Florida could be rebuilt into 
the firmware used in Sarasota County. 

To conduct its work, GAO conducted tests on a sample of voting systems 
used in Sarasota County during the 2006 general election. GAO witnessed 
the rebuild of the firmware from the escrowed source code at the 
manufacturerï¿½s development facility. GAO reviewed test documentation 
from Florida, Sarasota County, and the voting system manufacturer and 
met with election officials to prepare the test protocols and detailed 
test procedures. 

What GAO Found: 

GAO conducted three tests on the iVotronic Direct Recording Electronic 
(DRE) voting systems in Sarasota County and these tests did not 
identify any problems. Based on its testing, GAO obtained increased 
assurance that the iVotronic DREs used in Sarasota County during the 
2006 general election did not contribute to the large undervote in the 
Florida-13 contest. Although the test results cannot be used to provide 
absolute assurance, GAO believes that these test results, combined with 
the other reviews that have been conducted by the State of Florida, 
GAO, and others, have significantly reduced the possibility that the 
iVotronic DREs were the cause of the undervote. 

GAOï¿½s firmware verification test showed that the firmware installed in 
a statistically selected sample of 115 machines used by Sarasota County 
during the 2006 general election matched the firmware certified by the 
Florida Division of Elections. The statistical approach used in 
selecting these machines lets GAO estimate with a 99 percent confidence 
level that no more than 60 of the 1,499 iVotronic DREs that recorded 
votes in the 2006 general election were using different firmware. 
Consequently, GAO is able to place more confidence in the results of 
other tests conducted on a small number of machines by GAO and by 
others, which indicated that the iVotronic DREs did not cause the 
undervote. GAO also confirmed that when the manufacturer rebuilt the 
iVotronic DRE firmware from the source code that was held in escrow by 
the Florida Division of Elections and previously reviewed by GAO and 
others, the resulting firmware matched the version certified by the 
Florida Division of Elections. 

For the ballot test, GAO cast predefined test ballots on 10 iVotronic 
DREs and confirmed that each ballot was displayed and recorded 
accurately. GAO conducted the calibration test by miscalibrating two 
iVotronic DREs and casting ballots on them to validate that the 
machines recorded the information that was displayed on the touch 
screen. Based on the results of the ballot and calibration tests, GAO 
found that (1) the machines properly displayed, recorded, and counted 
the selections for all test ballots cast during ballot testing 
involving 112 common ways a voter may have interacted with the system, 
and (2) the deliberately miscalibrated machines, though difficult to 
use, accurately recorded the ballot selections as displayed on screen. 

At this point, GAO believes that adequate testing has been performed on 
the voting machine software and does not recommend further testing in 
this area. Given the complex interaction of people, processes, and 
technology that must work effectively together to achieve a successful 
election, GAO acknowledges the possibility that the large undervote in 
Floridaï¿½s 13th Congressional District race could have been caused by 
factors such as voters who intentionally undervoted, or voters who did 
not properly cast their ballots on the iVotronic DRE, potentially 
because of issues relating to interaction between voters and the 
ballot. 

To view the full product, including the scope and methodology, click on 
[hyperlink, http://www.GAO-08-425T]. For more information, contact 
Nabajyoti Barkakati at (202) 512-6412 or [email protected]. 

[End of section] 

Mr. Chairman and Members of the Task Force: 

I am pleased to appear before the task force today to present the 
findings on our testing of the voting equipment used in the 2006 
general election in Florida's 13th Congressional District (Florida-13). 
I would like to thank the task force for its overall support of our 
efforts and specifically for the assistance provided in obtaining 
resources from the House Recording Studio that were critical to 
successfully completing our testing efforts. 

In November 2006, about 18,000 undervotes were reported in Sarasota 
County in the race for Florida's 13th Congressional District.[Footnote 
1] After the election results were contested in the House of 
Representatives, the task force met and unanimously voted to seek GAO's 
assistance in determining whether the voting systems contributed to the 
large undervote in Sarasota County. In our October 2, 2007, statement 
for the task force, we presented the findings of our review of the 
voting systems and stated that while prior tests and reviews provided 
some level of assurance that the voting systems in Sarasota County-- 
iVotronic direct recording electronic (DRE) voting systems manufactured 
by Election Systems and Software (ES&S)--functioned correctly, they 
were not enough to provide reasonable assurance that the iVotronic DRE 
voting systems did not contribute to the undervote.[Footnote 2] 
Specifically, we found that assurance was lacking in three areas and 
proposed to the task force that additional tests--firmware 
verification, ballot, and calibration--be conducted to address these 
areas. We stated that successful accomplishment of these tests would 
provide increased, but not absolute, assurance that the iVotronic DREs 
used in Sarasota County during the 2006 general election did not cause 
the undervote. The task force requested that we proceed with the 
proposed additional tests. Our objectives were to (1) verify that 
firmware installed in a statistical sample of iVotronic DREs was 
identical to the firmware certified by the State of Florida, (2) 
perform ballot testing using 112 ways to cast a ballot for the Florida- 
13 contest to ensure that the voting machines would properly record and 
count the ballots, and (3) deliberately miscalibrate voting machines 
and then cast ballots on those machines to ensure that the voting 
machines would properly record the ballots. As part of the first 
objective, we also validated that the source code, which was held in 
escrow by the Florida Division of Elections, would produce the firmware 
used by Sarasota County during the 2006 general election. 

To conduct our tests, we developed test protocols and detailed test 
procedures. We met with officials from the Sarasota County Supervisor 
of Elections, the Florida Department of State and Division of 
Elections, and ES&S to obtain the necessary details about the voting 
systems and prior tests to document our test procedures. We also 
reviewed voting system documentation to develop a testing approach and 
the test procedures. To ensure that the certified firmware held in 
escrow by the Florida Division of Elections corresponded to the source 
code that was reviewed by a team from Florida State University and us, 
on November 19, 2007, we visited ES&S's development facility in 
Rockford, Illinois, and witnessed the rebuild of the firmware from the 
escrowed source code. 

Further details on our test methodology are included in the following 
sections on each of the three tests. Appendix I outlines the process 
used to select machines for testing, and appendix II lists the 
iVotronic DREs that we tested. We coordinated with the Florida Division 
of Elections and the Sarasota County Supervisor of Elections to obtain 
access to the iVotronic DREs and other necessary test equipment to 
conduct our testing. We conducted the firmware verification, ballot, 
and calibration tests at the Sarasota County Voting Equipment Facility 
(VEF) in Sarasota, Florida. We established the test environment on 
November 26, 2007, and conducted the tests from November 27, 2007, to 
December 4, 2007. During this time, we completed the steps necessary to 
conduct the tests and collected the test data. In addition, we video 
recorded the tests. One camera was used to capture a wide angle shot of 
the test room. Other cameras recorded the conduct of the firmware 
verification, ballot, and calibration tests. 

We provided a draft of this statement to the Florida Department of 
State and ES&S for their review and comments. We briefed the Sarasota 
County Supervisor of Elections on the contents of our statement. The 
Florida Department of State and ES&S also conducted a sensitivity 
review to ensure that business proprietary information is not disclosed 
in this statement. We conducted our work from October 2007 to February 
2008 in Washington, D.C.; Tallahassee and Sarasota, Florida; and at 
ES&S facilities in Rockford, Illinois, and Omaha, Nebraska. 

Results in Brief: 

We conducted three tests on the iVotronic DRE voting systems used in 
Sarasota County and these tests did not identify any problems that 
would indicate that the machines were responsible for the undervote in 
the Florida-13 race in the 2006 general election. In our firmware 
verification test, we extracted the firmware from a random probability 
sample of 115 iVotronic DREs out of the 1,499 iVotronic DREs used in 
Sarasota County's 2006 general election and found that each machine's 
firmware matched the certified version of firmware held in escrow by 
the Florida Division of Elections. The statistical approach used in 
selecting these machines enables us to estimate with a 99 percent 
confidence level that at least 1,439 of the 1,499 machines used the 
same firmware that was certified by the Florida Division of Elections. 
Consequently, we have more confidence in the results of other tests 
conducted on a small number of machines by GAO and by others, which 
indicated that the iVotronic DREs were not the cause of the undervote. 
We witnessed the rebuild of the iVotronic DRE's firmware from the 
source code that was held in escrow by the Florida Division of 
Elections and that was previously reviewed by Florida State University 
and by us. At ES&S's software development facility, we observed that 
rebuilding the firmware from the escrowed source code resulted in the 
same firmware that was certified and held in escrow by the Florida 
Division of Elections. This validation provides greater confidence in 
the results of prior source code reviews by Florida State University 
and us. 

For the ballot test, we cast predefined test ballots on 10 iVotronic 
DREs and confirmed that each ballot was displayed and recorded 
accurately. The test ballots represented 112 common ways a voter may 
have interacted with the iVotronic DRE to select a candidate in the 
Florida-13 race and cast a ballot. These tests were performed on nine 
machines configured as election day machines and then repeated on one 
machine configured as an early voting machine. 

Finally, we conducted the calibration test by miscalibrating two 
iVotronic DREs and casting ballots on them to validate that the 
machines recorded the information that was displayed on the touch 
screen. Our tests, involving a total of 10 different miscalibration 
patterns and capturing 39 ballots, found that the machines correctly 
displayed the selection in the Florida-13 race on the review screen and 
correctly recorded the ballot. Although the machines were more 
difficult to use, the selections shown on the screen were the same 
selections captured by the machine when the ballot was cast. 

Based on the results of these tests, we have obtained increased 
assurance, but not absolute assurance that the iVotronic DREs used in 
Sarasota County's 2006 general election did not contribute to the large 
undervote in the Florida-13 contest. Absolute assurance is impossible 
to achieve because we are unable to recreate the conditions of the 
election in which the undervote occurred. Although the test results 
cannot be used to provide absolute assurance, we believe that these 
test results, combined with the other reviews that have been conducted 
by the State of Florida, GAO, and others, have significantly reduced 
the possibility that the iVotronic DREs were the cause of the 
undervote. At this point, we believe that adequate testing has been 
performed on the voting machine software to reach this conclusion and 
do not recommend further testing in this area. Given the complex 
interaction of people, processes, and technology that must work 
effectively together to achieve a successful election, we acknowledge 
the possibility that the large undervote in Florida's 13th 
Congressional District race could have been caused by factors such as 
voters who intentionally undervoted, or voters who did not properly 
cast their ballots on the iVotronic DRE, potentially because of issues 
relating to interaction between voters and the ballot. 

Background: 

The 13th Congressional District of Florida comprises DeSoto, Hardee, 
Sarasota, and parts of Charlotte and Manatee Counties. In the November 
2006 general election, there were two candidates in the race to 
represent the 13th Congressional District: Vern Buchanan, the 
Republican candidate, and Christine Jennings, the Democratic candidate. 
The State of Florida certified Vern Buchanan the winner of the 
election. The margin of victory was 369 votes out of a total of 238,249 
votes counted. Table 1 summarizes the results of the election and shows 
that the results from Sarasota County exhibited a significantly higher 
undervote rate than in the other counties in the congressional 
district. 

Table 1: Results from 2006 General Election for Florida Congressional 
District 13: 

County: Charlotte; 
Buchanan: 4,460; 
Jennings: 4,277; 
Undervotes: 225; 
Total ballots cast: 8,962; 
Percentage undervote: 2.51. 

County: DeSoto; 
Buchanan: 3,471; 
Jennings: 3,058; 
Undervotes: 142; 
Total ballots cast: 6,672; 
Percentage undervote: 2.13. 


County: Hardee; 
Buchanan: 2,629; 
Jennings: 1,686; 
Undervotes: 269; 
Total ballots cast: 4,584; 
Percentage undervote: 5.87. 

County: Manatee; 
Buchanan: 50,117; 
Jennings: 44,432; 
Undervotes: 2,274; 
Total ballots cast: 96,828; 
Percentage undervote: 2.35. 

County: Sarasota; 
Buchanan: 58,632; 
Jennings: 65,487; 
Undervotes: 18,412; 
Total ballots cast: 142,532; 
Percentage undervote: 12.92. 

County: Total; 
Buchanan: 119,309; 
Jennings: 118,940; 
Undervotes: 21,322; 
Total ballots cast: 259,578; 
Percentage undervote: [Empty]. 

Source: GAO analysis of Florida Division of Elections, Charlotte 
County, DeSoto County, Hardee County, Manatee County, and Sarasota 
County data. 

Note: Numbers do not add up because of overvotes-where voters select 
more than the maximum number of candidates allowed in a race; in this 
case, an overvote was a ballot that had votes for both Buchanan and 
Jennings. 

[End of table] 

As seen in table 1, about 18,000 undervotes were reported in Sarasota 
County in the race for Florida's 13th Congressional District. After the 
election results were contested in the House of Representatives, the 
task force met and unanimously voted to seek GAO's assistance in 
determining whether the voting systems contributed to the large 
undervote in Sarasota County. On June 14, 2007, we met with the task 
force and agreed upon an engagement plan. We reported on the status of 
our review at an interim meeting held by the task force on August 3, 
2007.[Footnote 3] 

On October 2, 2007, we reported that our analysis of election data did 
not identify any particular voting machines or machine characteristics 
that could have caused the large undervote in the Florida-13 
race.[Footnote 4] The undervotes in Sarasota County were generally 
distributed across all machines and precincts. We found that some of 
the prior tests and reviews conducted by the State of Florida and 
Sarasota County provided assurance that certain components of the 
voting system in Sarasota County functioned correctly, but they were 
not enough to provide reasonable assurance that the iVotronic DREs did 
not contribute to the undervote. We proposed three tests--firmware 
verification, ballot, and calibration--to provide increased assurance, 
but not absolute assurance, that the iVotronic DREs did not cause the 
large undervote in Sarasota County. We stated that the successful 
conduct of the tests could reduce the possibility that the voting 
systems caused the undervote and shift attention to the possibilities 
that voters intentionally undervoted or voters did not properly cast 
their ballots on the iVotronic DRE, potentially because of issues 
relating to interaction between voters and the ballot. 

Overview of the Voting Systems Used in Sarasota County in the 2006 
General Elections: 

In the 2006 general election, Sarasota County used voting systems 
manufactured by ES&S. The State of Florida has certified different 
versions of ES&S voting systems. The version used in Sarasota County 
was designated ES&S Voting System Release 4.5, Version 2, Revision 2, 
and consisted of iVotronic DREs, a Model 650 central count optical scan 
tabulator for absentee ballots, and the Unity election management 
system. It was certified by the State of Florida on July 17, 2006. The 
certified system includes different configurations and optional 
elements, several of which were not used in Sarasota County.[Footnote 
5] 

The election management part of the voting system is called Unity; the 
version that was used was 2.4.4.2. Figure 1 shows the overall election 
operation using the Unity election management system and the iVotronic 
DRE. 

Figure 1: Overview of Election Operation Using the Unity Election 
Management System and iVotronic DRE: 

[See PDF for image] 

This figure is an illustration of the Overview of Election Operation 
Using the Unity Election Management System and iVotronic DRE. The 
following data is depicted: 

Unity: 

Election Data Manager: 
Program, store, and format election data for a jurisdiction; 

Ballot Image Manager: 
Create paper ballot layouts and create information for personalized 
electronic ballot (PEB) to be used in iVotronic DREs; 

Hardware Programming Manager: 
Copy election definitions for iVotronic DREs into PEBs; 

Precinct Count, iVotronic DRE: 
iVotronic DREs are used to capture votes and tabulate votes for a 
precinct; 
Copy precinct-level votes from PEBs into Election Reporting Manager and 
ballot images and event logs from compact flash cards. 

Upload via modem to: Data Acquisition Manager; 

Data Acquisition Manager: sends to: Election Reporting Manager. 

Election Reporting Manager: 
Outputs Election reports. 

Source: GAO. 

[End of figure] 

Sarasota County used iVotronic DREs for early and election day voting. 
Specifically, Sarasota County used the 12-inch iVotronic DRE, hardware 
version 1.1 with firmware version 8.0.1.2.[Footnote 6] Some of the 
iVotronic DREs are configured to use audio ballots, which are often 
referred to as Americans with Disabilities Act (ADA) machines. The 
iVotronic DRE uses a touch screen--a pressure-sensitive graphics 
display panel--to display and record votes (see fig. 2). 

Figure 2: The iVotronic DRE Voting System and Its Components: 

[See PDF for image] 

This figure is a photograph of the iVotronic DRE Voting System and its 
components. Four components are labeled in the photograph: 

Touch screen; 
Personalized electronic ballot; 
Vote button; 
Connection to communications pack and printer (removed during voting). 

Source: GAO. 

[End of figure] 

The machine has a storage case that also serves as the voting booth. 
The operation of the iVotronic DRE requires the use of a personalized 
electronic ballot (PEB), which is a storage device with an infrared 
window used for transmission of ballot data to and from the iVotronic 
DRE. The iVotronic DRE has four independent flash memory modules, one 
of which contains the program code--firmware--that runs the machine; 
the remaining three flash memory modules store redundant copies of 
ballot definitions, machine configuration information, ballots cast by 
voters, and event logs (see fig. 3). The iVotronic DRE includes a VOTE 
button that the voter has to press to cast a ballot and record the 
information in the flash memory. The iVotronic DRE also includes a 
compact flash card that can be used to load sound files onto iVotronic 
DREs with ADA functionality. The iVotronic DRE's firmware can be 
updated through the compact flash card. Additionally, at the end of 
polling, the ballots and audit information are to be copied from the 
internal flash memory module to the compact flash card. 

Figure 3: Inside View of the iVotronic DRE Showing the Flash Memory 
Modules: 

[See PDF for image] 

This figure is a photograph of an Inside View of the iVotronic DRE 
Showing the Flash Memory Modules. Two components are labeled in the 
photograph: 

Firmware flash memory module; 
Redundant flash memory modules with ballot data and event log. 

Source: GAO. 

[End of figure] 

To use the iVotronic DRE for voting, a poll worker activates the 
iVotronic DRE by inserting a PEB into the PEB slot after the voter has 
signed in at the polling place. After the poll worker makes selections 
so that the appropriate ballot will appear, the PEB is removed and the 
voter is ready to begin using the system. The ballot is presented to 
the voter in a series of display screens, with candidate information on 
the left side of the screen and selection boxes on the right side (see 
fig. 4). 

Figure 4: Second Ballot Page Showing the Congressional and 
Gubernatorial Races in Sarasota County's 2006 General Election: 

[See PDF for image] 

This figure is a screen capture of the Second Ballot Page Showing the 
Congressional and Gubernatorial Races in Sarasota County's 2006 General 
Election. The following data is depicted on the page: 

U.S. Representative in Congress, 13th Congressional District (Vote for 
One): 

Vern Buchanan, Republican; 
Christine Jennings, Democrat. 

State: Governor and Lieutenant Governor (Vote for One): 

Charlie Crist, Jeff Kottkamp, Republican; 
Jim Davis, Daryl L. Jones, Democrat; 
Max Linn, Tom Macklin, Ref; 
Richard Paul Dembinsky, Dr. Joe Smith, NPA; 
John Wayne Smith, James J. Kearney, NPA; 
Karl C.C. Behm, Carol Castagnero, NPA; 
Write-in. 

Previous Page; 
Page 2 of 15 (Public Count: 0); 
Next Page. 

Source: Sarasota County Supervisor of Elections. 

[End of figure] 

The voter can make a selection by touching anywhere on the line, and 
the iVotronic DRE responds by highlighting the entire line and 
displaying an X in the box next to the candidate's name. The voter can 
also change his or her selection by touching the line corresponding to 
another candidate or by deselecting his or her choice. "Previous Page" 
and "Next Page" buttons are used to navigate the multipage ballot. 
After completing all selections, the voter is presented with a summary 
screen with all of his or her selections (see fig. 5). From the summary 
screen, the voter can change any selection by selecting the race. The 
race will be displayed to the voter on its own ballot page. When the 
voter is satisfied with the selections and has reached the final 
summary screen, the red VOTE button is illuminated, indicating the 
voter can now cast his or her ballot. When the VOTE button is pressed, 
the voting session is complete and the ballot is recorded on the 
iVotronic DRE. In Sarasota County's 2006 general election, there were 
nine different ballot styles with between 28 and 40 races, which 
required between 15 and 21 electronic ballot pages to display, and 3 to 
4 summary pages for review purposes. 

Figure 5: First Summary Page in Sarasota County's 2006 General 
Election: 

[See PDF for image] 

This figure is a screen capture of the First Summary Page in Sarasota 
County's 2006 General Election. The following data is depicted on the 
page: 

Instructions: 
Return to any contest by touching the contest title. To cast your 
ballot now, press the Vote button. 

United States Senator: 
No selection made. 

U.S. Representative in Congress: 
No selection made. 

Governor and Lieutenant Governor: 
No selection made. 

Attorney General: 
No selection made. 

Chief Financial Officer: 
No selection made. 

Commissioner of Agriculture: 
No selection made. 

State Representative: 
No selection made. 

Charter Review Board District: 
No selection made. 

Charter Review Board District: 
No selection made. 

Charter Review Board District: 
No selection made. 

Charter Review Board District: 
No selection made. 

Charter Review Board District: 
No selection made. 

Previous Page. 
Summary Ballot, Page 1 of 3. 
Next Page. 

Source: Sarasota County Supervisor of Elections. 

[End of figure] 

Election Systems Involve People, Processes, and Technology: 

An election system is based upon a complex interaction of people 
(voters, election officials, and poll workers), processes (controls), 
and technology that must work effectively together to achieve a 
successful election. The particular technology used to cast and count 
votes is a critical part of how elections are conducted, but it is only 
one facet of a multifaceted election process that involves the 
interplay of people, processes, and technology. 

As we have previously reported, every stage of the election process-- 
registration, absentee and early voting, preparing for and conducting 
Election Day activities, provisional voting, and vote counting--is 
affected by the interaction of people, processes, and 
technology.[Footnote 7] Breakdowns in the interaction of people, 
processes, and technology may, at any stage of an election, impair an 
accurate vote count. For example, if the voter registration process is 
flawed, ineligible voters may be allowed to cast votes. Poll worker 
training deficiencies may contribute to discrepancies in the number of 
votes credited and cast, if voter information was not entered properly 
into poll books. Mistakes in using the DRE systems could result from 
inadequate understanding of the equipment on the part of those using 
it. 

As noted in our October statement, we recognize that human interaction 
with the ballot layout could be a potential cause of the undervote, and 
we noted that several suggestions have been offered as possible ways to 
establish that voters are intentionally undervoting and to provide some 
assurance that the voting systems did not cause the undervote.[Footnote 
8] For instance, 

* A voter-verified paper trail could provide an independent 
confirmation that the touch screen voting systems did not malfunction 
in recording and counting the votes from the election. The paper trail 
would reflect the voter's selections and, if necessary, could be used 
in the counting or recounting of votes. This issue was also recognized 
in the source code review performed by the Security and Assurance in 
Information Technology (SAIT) laboratory at Florida State University as 
well as the 2005 and draft 2007 Voluntary Voting Systems Guidelines 
prepared for the Election Assistance Commission. We have previously 
reported on the need to implement such a function properly.[Footnote 9] 

* Explicit feedback to voters that a race has been undervoted and a 
prompt for voters to affirm their intent to undervote might help 
prevent many voters from unintentionally not casting a vote in a race. 
On the iVotronic DREs, such feedback and prompts are provided only when 
the voter attempts to cast a completely blank ballot, but not when a 
voter fails to vote in individual races. 

* Offering a "none of the above" option in a race would provide voters 
with the opportunity to indicate that they are intentionally 
undervoting. For example, the State of Nevada provides this option in 
certain races in its elections. 

We reported that decisions about these or other suggestions about 
ballot layout or voting system functions should be informed by human 
factors studies that assess such measures' effectiveness in accurately 
recording voters' preferences, making voting systems easier to use, and 
preventing unintentional undervotes. 

Tests Confirm Sarasota County iVotronic DREs Used Same Firmware 
Certified by Florida: 

We previously reported that having reasonable assurance that all 
iVotronic DREs that recorded votes in the 2006 general election were 
running the same certified firmware would allow us to have more 
confidence that the iVotronic DREs will behave similarly when 
tested.[Footnote 10] Consequently, if we are reasonably confident that 
the same firmware was running in all 1,499 machines, then we are more 
confident that the results of other tests, conducted both by GAO and by 
others, on a small number of machines can be used to obtain increased 
assurance that the iVotronic DREs did not cause the undervote. We also 
reported that there was a lack of assurance that the source code that 
was held in escrow by the Florida Division of Elections and that was 
previously reviewed by Florida State University and by us, if rebuilt, 
would corresponded to the firmware that was certified and held in 
escrow by the Florida Division of Elections. We found that the firmware 
on a statistically selected sample of 115 iVotronic DREs was the same 
as that certified by the Florida Division of Elections. We also found 
that the escrowed source code, when rebuilt into executable firmware, 
corresponded to the 8.0.1.2 firmware that was certified by the Florida 
Division of Elections. 

Methodology for Firmware Verification Testing: 

Our methodology to obtain reasonable assurance that the firmware used 
on Sarasota County's iVotronic DREs during the 2006 general election 
was the same as that certified by the State of Florida was broken down 
into two basic steps: (1) selecting a representative sample of 
machines, and (2) verifying that the firmware extracted from the voting 
machines was the same as the escrowed firmware that had been certified 
by the Florida Division of Elections. Appendix I details the 
methodology for selecting the representative sample of machines. 
Appendix II contains a list of the serial numbers of the tested 
iVotronic DREs. 

To ensure that we would be testing with the iVotronic firmware 
certified by the Florida Division of Elections, on October 18, 2007, we 
and officials from the Florida Division of Elections made two copies of 
the escrowed iVotronic 8.0.1.2 firmware on compact discs (CD) and 
placed them in two tamper-evident bags with serial numbers. The bags 
were subsequently hand-delivered by a Florida Division of Elections 
official for our use in the firmware verification test and for the 
rebuilding of the firmware from the source code. 

In order to extract the firmware from an iVotronic DRE, the machine was 
placed on an anti-static mat and the case was opened using a special 
screwdriver. After lifting the case, a special extraction tool was used 
to remove the flash memory module that contains the firmware. The flash 
memory module was then inserted in the socket of a Needham Electronics' 
EMP-300 device that was connected to the universal serial bus (USB) 
port of a personal computer (PC). The EMPWin application running on 
that PC was used to read the firmware from the flash memory module and 
save the extracted firmware on the PC. The Florida Division of 
Elections loaned us the EMP-300 and EMPWin application for use in 
extracting firmware from the flash memory module. 

To compare the extracted firmware with the escrowed version, we relied 
on two commercially available software programs. First, we acquired a 
license for PrestoSoft's ExamDiff Pro software that enables comparison 
of files. The ExamDiff Pro software is a commercially available program 
designed to highlight the differences between two files. For each 
selected iVotronic DRE, the extracted firmware was compared with the 
escrowed version with any differences highlighted by the program. 

Second, to further ensure that the extracted firmware matched the 
escrowed firmware, we compared the SHA-1 hash value of the extracted 
firmware to the hash value of the comparable certified 
firmware.[Footnote 11] We computed the SHA-1 hash by using the 
Maresware hash software that was provided by the Florida Division of 
Elections. In order to ensure that the commercial Maresware hash 
software properly calculated the SHA-1 hash value, we (1) created four 
files and obtained a fifth file that contained executable code, (2) 
obtained hash values for each file by either using an external program 
that generated the hash values using the same hashing algorithm as the 
commercial product or using known hash values,[Footnote 12] and (3) 
used the commercial program acquired for testing the firmware to ensure 
that the hash values it generated for these five files were identical 
to the expected hash values for those files. In each case, the hash 
values generated by the commercial program were identical to the 
expected values. Accordingly, reasonable assurance for the purposes of 
our review was obtained that the commercial program produced its hash 
values in accordance with the NIST algorithm. 

At the end of each day, we (1) used the commercial Maresware software 
to compute hash values for each of the firmware programs that had been 
unloaded during that day and all previous days, and (2) compared each 
hash created by this program to the expected value that was calculated 
from the firmware that had been escrowed by the Florida Division of 
Elections. This comparison provided further assurance that the 
extracted firmware was (1) identical to the version escrowed by the 
Florida Division of Elections when the hashes agreed, or (2) different 
if the hashes did not agree. 

We also verified that sequestered machines were not used since the 2006 
general election. For each of these sequestered machines, we used an 
audit PEB to copy the audit logs onto a compact flash card and then 
used the Unity election reporting manager to generate event log 
reports. We examined the event logs for the date and time of occurrence 
of activities that would indicate whether the machine had been used. 
Lack of such activities since the 2006 general election provided 
reasonable assurance that the machines had not been used since they 
were sequestered.[Footnote 13] 

In addition, to verify that the source code for iVotronic DRE firmware 
version 8.0.1.2 previously examined by the Florida State University 
SAIT source code review team and by GAO corresponded with the version 
certified by the Florida Division of Elections, ES&S officials stated 
that it still had the development environment that could be used to 
compile, or rebuild, the certified firmware from the source code 
retained in escrow by the Florida Division of Elections.[Footnote 14] 
As we previously noted, a software review and security analysis of the 
iVotronic DRE firmware was conducted by a team led by Florida State 
University's SAIT laboratory.[Footnote 15] The software review team 
attempted to confirm or refute many different hypotheses that, if true, 
might explain the undervote in the race for the 13th Congressional 
District. In doing so, they made several observations about the source 
code, which we were able to independently verify. 

The rebuilding of the firmware was conducted by ES&S at its Rockford, 
Illinois, facility on November 19, 2007, and witnessed by us. Prior to 
the rebuild, the Florida Division of Elections provided an unofficial 
copy of the source code to ES&S so that ES&S could prepare the 
development environment and test the rebuild steps. Using the official 
sealed copy of the source code CD, ES&S rebuilt the firmware in front 
of GAO representatives. ES&S described the development environment and 
we inspected it to satisfy ourselves that the firmware was faithfully 
rebuilt using the escrowed source code. After the rebuilding of the 
firmware, the certified version of 8.0.1.2 firmware was compared with 
the rebuilt version using PrestoSoft's ExamDiff Pro. 

Results of Firmware Verification Testing: 

While the Florida audit team had previously confirmed that the firmware 
running on six iVotronic DREs matched the certified version held in 
escrow by the Florida Division of Elections, we found that the sample 
size was too small to support generalization to all 1,499 iVotronic 
DREs that recorded votes during the 2006 general election. Accordingly, 
we conducted a firmware verification test on a statistically valid 
sample of 115 iVotronic DRE machines used by Sarasota County during the 
2006 general election. The selected machines fell into two groups--
machines that had not been used since the 2006 general election 
(referred to as sequestered machines) and machines that had been used 
in subsequent elections. For each machine, we extracted the firmware 
from a flash memory module in that machine and then compared the 
extracted firmware with the escrowed version using commercially 
available file comparison tools to determine whether they agreed. We 
found that the firmware installed in the flash memory module of each 
machine matched the escrowed firmware that had been certified by 
Florida. The statistical approach used to select these machines lets us 
estimate with a 99 percent confidence level that at least 1,439, or 96 
percent, of the 1,499 machines used in the 2006 general election used 
the firmware that was certified by the State of Florida. 

We witnessed the rebuild of the iVotronic DRE's firmware from the 
source code that was held in escrow by the Florida Division of 
Elections and that was previously reviewed by Florida State University 
and by us. At ES&S's software development facility, we observed that 
rebuilding the firmware from the escrowed source code resulted in the 
same firmware that was certified and held in escrow by the Florida 
Division of Elections. The comparison of the escrowed firmware to the 
version that was rebuilt by the vendor identified no differences and 
provides us reasonable assurance that the escrowed firmware 
corresponded to the escrowed source code. The successful rebuilding of 
the firmware from the escrowed source code enables us to have greater 
confidence in the conclusions derived from prior source code reviews by 
Florida State University and us. 

Ballot Testing Showed That Machines Accurately Recorded and Counted 
Ballots: 

In our October 2007 statement, we noted that there were 112 common ways 
a voter may interact with the system to select a candidate in the 
Florida-13 race and cast the ballot, and that prior testing of the 
iVotronic DREs covered only 13 of these 112 possible ways. We developed 
224 test ballots to verify that the iVotronic DRE could accurately 
capture ballots using each of these 112 common ways a voter may 
interact with the system; 112 test ballots were cast on one machine 
configured for early voting, and another 112 ballots were cast on nine 
machines configured for election day voting. Our tests showed that for 
each of the 224 test ballots, the iVotronic DRE correctly captured each 
vote as cast for the Florida-13 race. We also conducted firmware 
verification tests on these machines and verified that they were 
running the certified firmware. 

Methodology for Ballot Testing: 

The methodology for ballot testing can be broken into two major areas-
-development of the test ballots and execution of the test using those 
ballots. The following sections discuss these areas. 

Development of Test Ballots: 

In examining how the system allowed voters to make a selection in the 
Florida-13 race, we found at least 112 different ways a voter could 
make his or her selection and cast the ballot in the Florida-13 race, 
assuming that it was the only race on the ballot. Specifically, a voter 
could (1) initially select either candidate or neither candidate (i.e., 
undervote), (2) change the vote on the initial screen, and (3) use a 
combination of features to change or verify his or her selection by 
using the page back and review screen options. Accordingly, we tested 
these 112 ways to select a candidate on the early voting machine and on 
the election day machines (224 test ballots in total). 

The 112 standard test ballots cover all combinations of the following 
types of voter behavior: 

* Voter makes selection on the initial ballot screen and makes no 
changes or takes any other action to return to the contest to review or 
change selection. 

* Voter makes selection on the initial ballot screen and decides before 
leaving that screen to change the selection because of an error in 
selecting the candidate or for some other reason. 

* Voter makes selection on the initial ballot screen and then decides 
to use the page back option to review or change selection. 

* Voter makes selection on the initial ballot screen and continues to 
the review screen and then decides to use the review screen option to 
review or change selection. 

* Voter makes selection on the initial ballot screen and uses a 
combination of page back and review screen options to review or change 
selection. 

In each instance where a selection could be made, three choices were 
possible for the Florida-13 race: a selection for one of the two 
candidates, or no selection (i.e., an undervote). 

In developing the standard test ballots, we did not consider all 
combinations of some other types of voter behavior that would have 
significantly increased the number of test cases without providing 
significant benefits. In most cases, such behavior are variants of the 
primary voter behavior that we examined. The following are examples of 
voter behavior that were not included in the standard test set in order 
to reduce the number of test cases to practicable levels: 

* Using a one-touch or two-touch method to make changes on a ballot 
page.[Footnote 16] 

* Varying the number of pages a voter may go back ("page backs") to 
return to the page containing the Florida-13 race to change or review a 
selection. 

* Casting a ballot from the review screen selection. The VOTE button is 
not activated until the voter reaches the last review screen. However, 
once the VOTE button has been activated, a ballot may be cast from any 
screen. For example, a voter may activate the VOTE button and then 
return to a contest to review or change the selection using the review 
screen option. Once the voter goes to the contest from the review 
screen and makes any desired changes, the voter can then cast the 
ballot from that screen rather than going back to the last page of the 
review screen or even the review screen that was used to return to the 
selection. 

Although we did not consider all combinations of these types of voter 
behavior when developing the standard test ballots, we included some of 
these user interactions in the execution of applicable test ballots to 
provide increased assurance that the system would handle these voter 
behaviors. For each applicable test ballot, we randomly determined the 
test procedure that should be used for the following attributes: 

* Initial change method - The standard test ballots address voters 
making changes on the initial ballot screen. Where possible, the method 
used to change (one-touch or two-touch) the selection was randomly 
selected. 

* Number of page backs - The ballots used by Sarasota County included 
the page back function. After reviewing the ballots, it appeared 
reasonable to expect that voters who may have used the page back option 
would probably decide that they had missed the race by the time they 
went one or two pages beyond the page with the Florida-13 race. 
Therefore, when a standard test ballot contained a page back 
requirement, the number of page backs was randomly selected to 
determine whether one or two page backs should be used. 

* Page back change method - Some test ballots required a change after 
the page back option was selected. As with the initial change method, 
where possible, the method of changing (one-touch or two-touch) the 
selection was randomly assigned. 

* Review screen change method - The system displays a review screen 
that shows the voter's selections (or lack of selections) after the 
voter has progressed through all contests. On the review screen, the 
voter can select a race to go directly to that contest and (1) review 
the selection made, and (2) make any desired corrections. The standard 
test ballots were designed to cover this type of event. Where possible, 
the method used to make the change (one-touch or two-touch) was 
randomly selected. 

* Activate VOTE button and cast ballots from the review screen - In 
order to test casting ballots from locations other than the last review 
screen, the VOTE button must be activated prior to going to a screen 
where the ballot is cast.[Footnote 17] In order to determine which test 
ballots should be used for this test, a two-step approach was adopted. 
First, a random selection of the ballots that use the review screen 
option was made to determine which test ballots should have the VOTE 
button activated. Then a random selection of these test ballots was 
made to determine whether the ballot should be cast from the review 
screen selection. 

Besides those attributes that directly affect the selection in the 
Florida-13 race, we varied the other attributes on the ballot in order 
to complete the ballot test. For each of the 224 test ballots, we used 
random values for other attributes, including the following: 

* Ballot style - Each ballot was randomly assigned one of the nine 
ballot styles used in the election. 

* Write-in candidate - All ballot styles includes write-in options in 
at least 2 races --United States Senate and State Governor/Lieutenant 
Governor. To verify that the iVotronic DRE accurately recorded the 
selection in the Florida-13 race for each test ballot, we needed a way 
to identify each test ballot in the ballot image log. To accomplish 
this, we randomly selected one of these two races, selected the write- 
in candidate for the race, and entered a unique value (i.e., the test 
ballot number) in the write-in field. 

* Candidates and selections in other races on the ballot - Each ballot 
style had between 28 and 40 contests on the ballot. The values for the 
contests besides the Florida-13 race and the write-in field were also 
randomly selected. For example, most items had three possible choices-
-candidate 1 (or Yes), candidate 2 (or No), and undervote. Which of 
these three values was used for a given contest was randomly 
determined. 

The values used for these attributes were independently determined for 
the election day and early voting test ballots. For example, Test 
Ballot 2 (election day) and Test Ballot 202 (early voting) were 
designed to test the same standard condition described by one of the 
112 standard test ballots.[Footnote 18] Table 2 illustrates some of the 
similarities and differences between the two test ballots that result 
from the random selection process used to determine the other aspects 
of the ballot. 

Figure 8: Examples of Differences between Test Ballot 2 and Test Ballot 
202: 

Test item: Precinct; 
Test Ballot 2: 142; 
Test Ballot 202: 143. 

Test item: Ballot style; 
Test Ballot 2: 6; 
Test Ballot 202: 7. 

Test item: Contest used to contain unique value used to identify the 
test ballot during the review process.; 
Test Ballot 2: Governor/Lieutenant Governor; 
Test Ballot 202: U.S. Senate. 

Test item: Method used to make change on initial screen for contest; 
Test Ballot 2: Two-touch; 
Test Ballot 202: One-touch. 

Test item: Number of page backs to return to contest; 
Test Ballot 2: 2; 
Test Ballot 202: 1. 

Test item: Method used to make change after paging back to contest; 
Test Ballot 2: Two-touch; 
Test Ballot 202: Two-touch. 

Test item: Activate Vote button prior to using the review screen to 
return to the contest; 
Test Ballot 2: No; 
Test Ballot 202: Yes. 

Test item: Selection for Attorney General; 
Test Ballot 2: McCollum; 
Test Ballot 202: Campbell. 

Test item: Selection for Constitutional Amendment 1; 
Test Ballot 2: No; 
Test Ballot 202: Undervote. 

Test item: Selection for Constitutional Amendment 8; 
Test Ballot 2: No; 
Test Ballot 202: No. 

Test item: Method used to make change using the review screen approach; 
Test Ballot 2: Two-touch; 
Test Ballot 202: Two-touch. 

Test item: Cast ballot from contest selection; 
Test Ballot 2: No; 
Test Ballot 202: Yes. 

Test item: Return to review screen and then cast ballot; 
Test Ballot 2: Yes; 
Test Ballot 202: No. 

Source: GAO. 

[End of table] 

Finally, we selected 10 random machines to be used for the ballot 
testing.[Footnote 19] One machine was selected from those that were 
used in early voting in the 2006 general election. The other nine were 
selected from those that used each of the ballot styles on election day 
in the 2006 general election.[Footnote 20] For each election day 
machine, the assigned precinct was the same as the precinct where the 
machine was used during the 2006 general election. For the early voting 
machine, we needed to assign precincts for each ballot style. We used 
the precinct associated with the back-up machine used for election day 
testing as the precinct for that ballot style.[Footnote 21] If the 
first back-up machine was assigned the same precinct number as the 
primary election day machine, then we used the precinct associated with 
the second back-up machine. This approach was taken to maximize the 
number of precincts used in the testing efforts. 

Process Used in Executing the Ballot Test: 

A two-person test team conducted the ballot testing. One tester read 
out aloud the steps called for in the test ballot while the other 
tester performed those actions. In order to ensure that all of the 
actions relating to the Florida-13 congressional race were performed as 
laid out in the test ballots, a two-person review team observed a video 
display of the test and compared the actions taken by the tester to 
those called for in the test ballot. Furthermore, after the testing was 
completed, another team reviewed the video recording of these tests to 
validate that the actions relating to the Florida-13 contest taken by 
the tester were consistent with those called for by the test 
ballots.[Footnote 22] 

The criteria used to determine whether the test produced the expected 
result was derived from the Florida Voting System Standards.[Footnote 
23] Specifically, among other things, these standards require the 
system to allow the voter to (1) determine whether the inputs given to 
the system have selected the candidates that he or she intended to 
select, (2) review the candidate selections made by the voter, and (3) 
change any selection previously made and confirm the new selection 
prior to the act of casting the ballot. Furthermore, the system must 
communicate to the voter the fact that the voter has failed to vote in 
a race (undervote) and require the voter to confirm his or her intent 
to undervote before casting the ballot. During the ballot test, the 
actual system response was compared to the expected results by a review 
team and after the testing was completed another review team compared 
the video records to the test ballots to validate that the tests had 
been performed in accordance with test scripts for the Florida-13 
contest. 

At the beginning of testing on each iVotronic DRE, the machine was 
opened for voting and a zero tape was printed. After the casting of all 
test ballots on the machine, the machine was closed and a results tape 
was printed. The closing of the machine also writes the audit data to 
the compact flash card, including event data and ballot images. We 
examined the results tapes and compared the total votes cast for the 
Florida-13 contest against what was expected from the test ballots. We 
also kept track of the total number of ballots handled by the machine, 
called the "protective count" of an iVotronic DRE, before and after the 
test and confirmed that the increase in protective count matched the 
number of test ballots cast on that machine.[Footnote 24] 

Using the Unity election reporting manager, we read the compact flash 
cards and processed the audit data on each ballot test machine. We 
generated the ballot image log and examined the individual test ballots 
in the ballot image log. We looked for the unique identifier that was 
used for each test ballot and then confirmed that the ballot image 
reflected the correct selection for the Florida-13 race as called for 
by the test ballot. For example, the test script for Test Ballot 1 
required the tester to (1) select a write-in candidate for U.S. Senate 
and (2) enter the value of "TB1" in the write-in field. Because only 
this test ballot used this value, we could review the ballot image log 
to determine what selection the voting machine recorded for the Florida-
13 contest for the ballot showing "TB1" as the write-in candidate for 
U.S. Senate.[Footnote 25] 

Finally, using the process discussed previously for firmware testing, 
the firmware on all machines used for ballot testing was validated to 
ensure these machines used the same firmware that had been certified by 
the Florida Division of Elections. 

Results of Ballot Testing: 

After executing the ballot tests on the election day and early voting 
machines, we found that all 10 iVotronic DREs captured the votes for 
the Florida-13 race on the test ballots accurately. We used a unique 
identifier in a write-in field in each test ballot and verified that 
the iVotronic DRE accurately captured the tester's final selections in 
the Florida-13 race for each test ballot. 

Testing 112 ways to select a candidate on a single machine also 
provided us some additional assurance that the volume of ballots cast 
on election day did not contribute to the undervote. We noted that 
casting 112 ballots on a single machine was more than the number of 
ballots cast on over 99 percent of the 1,415 machines used on election 
day. 

Deliberately Miscalibrated iVotronic DREs Accurately Recorded Displayed 
Ballots: 

Because little was known about the effect of a miscalibrated machine on 
the behavior of an iVotronic DRE, we deliberately miscalibrated two 
iVotronic DREs using 10 different miscalibration methods to verify the 
functioning of the machine. Although the miscalibration made the 
machine more difficult to use, the 39 ballots used in this test 
confirmed that the system correctly recorded the displayed vote for the 
Florida-13 contest and did not appear to contribute to the undervote. 

Methodology for Calibration Testing: 

For the calibration testing, we judgmentally selected five different 
miscalibration patterns and repeated each pattern twice--once with a 
small amount of miscalibration and the second time with a large amount 
of miscalibration. The amount of miscalibration was also subjective-- 
roughly 0.25 to 0.5 inch for a small amount and about 0.7 to 1 inch for 
a large miscalibration. 

The miscalibration patterns are shown in the following figures. 

Figure 6: Miscalibration Pattern 1: For Each Calibration Point, the 
Tester Touches a Point Shifted Diagonally Inward: 

[See PDF for image] 

This figure is an illustration of Miscalibration Pattern 1. 

Source: GAO. 

[End of figure] 

Figure 7: Miscalibration Pattern 2: For Each Calibration Point, the 
Tester Touches a Point Shifted Horizontally Inward: 

[See PDF for image] 

This figure is an illustration of Miscalibration Pattern 2. 

Source: GAO. 

[End of figure] 

Figure 8: Miscalibration Pattern 3: For Each Calibration Point, the 
Tester Touches a Point Shifted Vertically Inward: 

[See PDF for image] 

This figure is an illustration of Miscalibration Pattern 3. 

Source: GAO. 

[End of figure] 

Figure 9: Miscalibration Pattern 4: For Each Calibration Point, the 
Tester Touches a Point Shifted Horizontally to the Right: 

[See PDF for image] 

This figure is an illustration of Miscalibration Pattern 4. 

Source: GAO. 

[End of figure] 

Figure 10: Miscalibration Pattern 5: For Each Calibration Point, the 
Tester Touches a Point Shifted Horizontally to the Left: 

[See PDF for image] 

This figure is an illustration of Miscalibration Pattern 5. 

Source: GAO. 

[End of figure] 

We conducted calibration testing on two different machines that were 
used for ballot testing.[Footnote 26] As with ballot testing, at the 
beginning of testing of each machine, we opened the machine for voting 
and printed a zero tape. During the opening process, we calibrated the 
machine with one of the miscalibration patterns. After the machine was 
miscalibrated, we then executed at least three of the test ballots that 
were used during ballot testing on that machine for each test.[Footnote 
27] The test ballots were rotated among the miscalibration patterns. 
For example, one of the machines had eight different ballot test 
scripts. The first three were used on one miscalibration pattern, the 
next three on another miscalibration pattern, and the final two plus 
the first one would be used on another miscalibration pattern. After 
the ballots were cast for one miscalibration pattern, the machine would 
be miscalibrated with another pattern. After the needed miscalibration 
patterns were tested on a machine, the iVotronic DRE was closed and a 
results tape was printed. The closing of the iVotronic DRE also wrote 
the audit data to the compact flash card. 

During the testing, the tester was instructed to take whatever actions 
were necessary to achieve the desired result. For example, if the 
script called for the selection of Candidate A, then the tester would 
keep touching the screen until Candidate A was selected. A review team 
monitored the testing to ensure that (1) the proper candidate for the 
Florida-13 congressional race was ultimately selected and (2) the 
review screen showed this candidate selection when it was first 
presented. 

As with the ballot test, we used the Unity election reporting manager 
to read the compact flash cards and processed the audit data or each 
ballot test machine. We generated the ballot image log and examined the 
individual test ballots in the ballot image log. We looked for the 
unique identifier that was used for each test ballot and then confirmed 
that the ballot image reflected the correct selection for the Florida- 
13 race as called for by the test ballot. After the testing had been 
completed, the expected results shown in the test ballot scripts were 
compared to the actual results contained in the ballot image log and 
the results tape using the same process discussed in the ballot testing 
methodology. 

Results of Calibration Testing: 

The 39 ballots used in this test confirmed that the system correctly 
recorded the displayed vote for the Florida-13 contest. We also noted 
that the miscalibration clearly made the machines harder to use and 
during an actual election these machines would have probably been 
either recalibrated or removed from service once the voter brought the 
problem to the precinct's attention, according to a Sarasota County 
official who observed the tests.[Footnote 28] 

Figure 11 shows an example of effects of our miscalibration efforts on 
the screen that is used to confirm the calibration results. 
Specifically, the stylus points to where the tester is touching the 
screen while the "X" on the screen shows where the machine indicated 
the stylus was touching the screen.[Footnote 29] In a properly 
calibrated machine, the stylus and the "X" are basically at the same 
point. 

Figure 11: Example of the Effects of a Miscalibrated Machine on the 
Calibration Screen: 

[See PDF for image] 

This figure is an illustration of the effects of a Miscalibrated 
Machine on the Calibration Screen. 

Source: GAO. 

[End of figure] 

Figure 12 shows an example of where the tester is touching the screen 
to make a selection and how this "touch" is translated into a 
selection. As can be seen, the finger making the selection is touching 
a position that in a properly calibrated machine would not result in 
the selection shown. However, the machine clearly shows the candidate 
selected and our tests confirmed that for the 39 ballots tested, the 
candidate actually shown by the system as selected (in this example, 
the shaded line) was the candidate shown on the review screen, as well 
as the candidate that received the vote when the ballot was cast. 

Figure 12: Example of the Effects of a Miscalibrated Machine on a 
Candidate Selection: 

[See PDF for image] 

This figure is an illustration of the effects of a Miscalibrated 
Machine on a Candidate Selection. 

Source: GAO. 

[End of figure] 

Conclusions: 

Our tests showed that (1) the firmware installed in a statistically 
selected sample of machines used by Sarasota County during the 2006 
general election matched the firmware certified by the Florida Division 
of Elections, and we confirmed that when the manufacturer rebuilt the 
iVotronic 8.0.1.2 firmware from the escrowed source code, the resulting 
firmware matched the certified version of firmware held in escrow, (2) 
the machines properly displayed, recorded, and counted the selections 
for all test ballots cast during the ballot testing involving the 112 
common ways a voter may interact with the system to cast a ballot for 
the Florida-13 race, and (3) the machines accurately recorded the test 
ballots displayed on deliberately miscalibrated machines. The results 
of these tests did not identify any problems that would indicate that 
the iVotronic DREs were responsible for the undervote in the Florida-13 
race in the 2006 general election. 

As we noted when we proposed these tests, even after completing these 
tests, we do not have absolute assurance that the iVotronic DREs did 
not play any role in the large undervote. Absolute assurance is 
impossible to achieve because we are unable to recreate the conditions 
of the election in which the undervote occurred. Although the test 
results cannot be used to provide absolute assurance, we believe that 
these test results, combined with the other reviews that have been 
conducted by Florida, GAO, and others, have significantly reduced the 
possibility that the iVotronic DREs were the cause of the undervote. At 
this point, we believe that adequate testing has been performed on the 
voting machine software to reach this conclusion and do not recommend 
further testing in this area. Given the complex interaction of people, 
processes, and technology that must work effectively together to 
achieve a successful election, we acknowledge the possibility that the 
large undervote in Florida's 13th Congressional District race could 
have been caused by factors such as voters who intentionally 
undervoted, or voters who did not properly cast their ballots on the 
iVotronic DRE, potentially because of issues relating to interaction 
between voters and the ballot. 

Comments: 

We provided draft copies of this statement to the Secretary of State of 
Florida and ES&S for their review and comment. We briefed the Sarasota 
County Supervisor of Elections on the contents of this statement and 
asked for their comments. The Florida Department of State provided 
technical comments, which we incorporated. ES&S and the Sarasota County 
Supervisor of Elections provided no comments. 

Mr. Chairman, this completes my prepared statement. I would be happy to 
respond to any questions you or other Members of the Task Force may 
have at this time. 

Contact and Acknowledgments: 

For further information about this statement, please contact Naba 
Barkakati at (202) 512-6412 or [email protected]. Contact points for 
our Office of Congressional Relations and Public Affairs may be found 
on the last page of this statement. Other key contributors to this 
statement include James Ashley, Stephen Brown, Francine Delvecchio, 
Cynthia Grant, Geoffrey Hamilton, Richard Hung, Douglas Manor, John C. 
Martin, Jan Montgomery, Daniel Novillo, Deborah Ortega, Keith Rhodes, 
Sidney Schwartz, Patrick Tobo, George Warnock, and Elizabeth Wood. We 
also appreciate the assistance of the House Recording Studio in the 
video recording of the tests. 

[End of section] 

Appendix I: Methodology for Selecting IVotronic DREs for GAO Testing: 

Each of the three tests--firmware verification, ballot, and 
calibration--was conducted on a sample of the 1,499 iVotronic DREs that 
recorded votes during the 2006 general election in Sarasota County, 
Florida. We selected 115 iVotronic DREs for the firmware test, 10 for 
the ballot test, and 2 for the calibration test. Appendix II contains 
the serial numbers of the iVotronic DREs that were tested. 

Firmware Test Sample: 

We selected a stratified random probability sample of iVotronic DREs 
from the population of 1,499. The sample was designed to allow us to 
generalize the results of the firmware sample to the population of 
iVotronic DREs used in this election. We stratified the population into 
two strata based on whether the machines had been sequestered since the 
2006 general election. There were a total of 818 machines that were 
sequestered and 681 machines that had been used in subsequent 
elections. The population and sample are described in table 3. 

We calculated the sample size in each stratum using the hypergeometric 
distribution to account for the relatively small populations in each 
stratum. We determined each sample size to be the minimum number of 
machines necessary to yield an upper bound of 7.5 percent, at the 99 
percent confidence level, if we observed zero failures in the firmware 
test. Assuming that we found no machines using an uncertified firmware 
version, these sample sizes allowed us to conclude with 99 percent 
confidence that no more than 7.5 percent of the machines in each 
stratum were using uncertified firmware. Further, this sample allowed 
us to conclude that no more than 4 percent of the 1,499 iVotronic DREs 
were using uncertified firmware, at the 99 percent confidence level. 

Table 2: Description of the Stratified Population and Sample Sizes for 
the Firmware Test: 

Stratum: Sequestered machines; 
Population size: 818; 
Sample size: 58. 

Stratum: Non-sequestered machines; 
Population size: 681; 
Sample size: 57. 

Stratum: Total; 
Population size: 1,499; 
Sample size: 115. 

Source: GAO based on analysis of Sarasota County voting data. 

[End of table] 

An additional five sequestered machines and five non-sequestered 
machines were selected as back-up machines should there be problems in 
locating the selected machines or some other problem that prevented 
testing them. 

Ballot Test Sample: 

We randomly selected a total of 10 machines from the population of 
1,384 machines that were not selected in the firmware test sample. This 
sample size is not sufficient to allow us to make direct 
generalizations to the population. However, if we are reasonably 
confident that the same software is used in all 1,499 machines, then we 
are more confident that the results of the other tests on a small 
number of machines can be used to obtain increased assurance that the 
iVotronic DREs did not cause the undervote. We randomly selected one 
machine from each of the nine ballot styles used during the general 
election and one machine from the machines used for early 
voting.[Footnote 30] In case of problems in operating or locating the 
machines, we also selected randomly selected two additional machines 
for each ballot style and for early voting. 

Calibration Test Sample: 

The two iVotronic DREs selected for calibration testing were selected 
from those tested in the ballot test. Because the machines used for the 
ballot tests included an ADA machine and "standard" machines, we 
selected one of each for calibration testing. Although we did not test 
the ADA capabilities of the ADA machine (e.g., the audio ballots), we 
found that the on-screen appearance of selections on the ADA machine 
differed slightly from that on non-ADA machines. For example, the 
standard non-ADA machine displayed a blue bar across the screen and an 
X in the box next to the candidate's name when a selection was made, 
while an ADA machine only showed an X in the box next to the 
candidate's name. 

[End of section] 

Appendix II: List of Machines Tested by GAO: 

[End of section] 

Table 4 table lists the iVotronic DREs that were tested by GAO. For 
each machine, the table shows whether the machine was sequestered and 
what type of testing was conducted on the machine. 

Table 4: List of iVotronic DREs Tested by GAO: 

Serial number: V0105178; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105203; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105222; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105255; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105305; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105351; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105379; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105390; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105396; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105422; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105481; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105499; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105500; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105524; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105526; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105563; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105573; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105607; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105613; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105623; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105651; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105656; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105661; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105664; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105743; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105848; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0105873;
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105874; 
Machine sequestered: No;
Type of testing conducted: 
Firmware testing. 

Serial number: V0105894; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105903; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105906; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105923; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105964; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105971; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105992; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106001; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106016; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106024; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106025; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106034; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106064; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106068; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106069; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106084; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106087; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106126; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106156; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106191; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106203; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106254; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106264; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106265; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106274; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106282; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106343; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106368; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106377; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106396; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106445; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106461; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106475; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106478; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106486; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106507; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106522; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106525; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106531; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106552; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106585; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106586; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106588; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106602;
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106615; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106656; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106658; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106661; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106667; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106681; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106711; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106718;
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106740; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106744; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106833; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106840; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106864; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106865; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106878; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106881; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106883; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106907; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0106933; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106936; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106949; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0106965; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107000; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107011; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107020; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107042; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107045; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107053; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107077; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107082; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107094; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107108; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107138; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0107143; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0107147; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0110355; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0111064; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0113816; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0114087; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0114415; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0117658; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0118183; 
Machine sequestered: No; 
Type of testing conducted: Firmware testing. 

Serial number: V0118293; 
Machine sequestered: Yes; 
Type of testing conducted: Firmware testing. 

Serial number: V0105386; 
Machine sequestered: Yes; 
Type of testing conducted: Early voting ballot testing. 

Serial number: V0105266; 
Machine sequestered: Yes; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0105694; 
Machine sequestered: No; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0106082; 
Machine sequestered: Yes; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0106145; 
Machine sequestered: Yes; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0106247; 
Machine sequestered: Yes; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0106509; 
Machine sequestered: No; 
Type of testing conducted: Election day ballot testing and calibration 
testing. 

Serial number: V0106671; 
Machine sequestered: Yes; 
Type of testing conducted: Election day ballot testing. 

Serial number: V0117861; 
Machine sequestered: No; 
Type of testing conducted: Election day ballot testing and calibration 
testing. 

Serial number: V0117951; 
Machine sequestered: No; 
Type of testing conducted: Election day ballot testing. 

Source: GAO. 

[End of table] 

[End of section] 

Footnotes: 

[1] Undervotes occur when the number of choices selected by the voter 
is fewer than the maximum allowed for that contest. In this case, it 
means ballots that did not record a selection for either candidate in 
the congressional contest. 

[2] GAO, Elections: Further Testing Could Provide Increased but Not 
Absolute Assurance That Voting Systems Did Not Cause Undervotes in 
Florida's 13th Congressional District, GAO-08-97T (Washington, D.C.: 
Oct. 2, 2007). 

[3] GAO, Elections: Status of GAO's Review of Voting Equipment Used in 
Florida's 13th Congressional District, GAO-07-1167T (Washington, D.C.: 
Aug. 3, 2007). 

[4] GAO-08-97T. 

[5] In May 2007, the State of Florida enacted legislation requiring, in 
general, the use of optical scan voting equipment that provides a paper 
trail. These requirements are effective July 1, 2008. There is an 
exemption from these requirements for voting by persons with 
disabilities. 

[6] The certified version of ES&S Voting System Release 4.5, Version 2, 
Revision 2, specifies the use of iVotronic hardware version 1.0. 
According to Florida Division of Election officials, hardware version 
1.1 of the iVotronic DRE has been available since at least 2004 and 
should have been included as a part of the certification for ES&S 
Voting System Release 4.5, Version 2, Revision 2. According to ES&S 
officials, iVotronic firmware version 8.0.1.2 runs in exactly the same 
manner on hardware versions 1.0 and 1.1. 

[7] GAO, Elections: The Nation's Evolving Election System as Reflected 
in the November 2004 General Election, GAO-06-450 (Washington, D.C.: 
June 6, 2006). 

[8] GAO-08-97T. 

[9] GAO, Elections: Federal Efforts to Improve Security and Reliability 
of Electronic Voting Systems Are Under Way, but Key Activities Need to 
Be Completed, GAO-05-956 (Washington, D.C.: Sept. 21, 2005). 

[10] GAO-08-97T. 

[11] The National Institute of Standards and Technology (NIST) has 
issued a Federal Information Processing Standard (FIPS) that describes 
four hashing algorithms that are iterative, one-way hash functions that 
can process a file and produce a condensed representation called a 
message digest or "hash." These algorithms enable the user to validate 
a file's integrity since any change to the file will, with a very high 
probability, result in a different message digest. The technical 
details of this process are contained in FIPS 180-2. The algorithm 
selected for this testing effort is commonly referred to as SHA-1 and 
is the same algorithm used by the Florida Division of Elections during 
its audit. 

[12] Two of the files and the expected values used came from FIPS 180- 
2. 

[13] We verified that sequestered machines were not used since the 2006 
general election by (1) verifying that the seals placed on these 
machines agreed with Sarasota County's records, and (2) checking the 
event logs maintained on the machine to determine whether the machines 
had been used since the machine had been sequestered. In every case, we 
found that the seal numbers agreed with Sarasota County's records. We 
were able to check the event log for 57 of the 58 sequestered iVotronic 
DREs. We were unable to power up the remaining iVotronic DRE and were 
consequently unable to extract the needed audit data. 

[14] In our October 2007 statement, we reported that according to ES&S, 
firmware compiled from the Florida escrowed source code may not be 
exactly identical to the firmware certified by the Florida Division of 
Elections because the embedded date and time stamp in the firmware 
would be different. We found that the date and time was not embedded in 
the firmware and that an identical version could be created. 

[15] Security and Assurance in Information Technology Laboratory, 
Florida State University, Software Review and Security Analysis of the 
ES&S iVotronic 8.0.1.2 Voting Machine Firmware (Tallahassee, Florida: 
Feb. 23, 2007). 

[16] The iVotronic DREs used in Sarasota County allow the user to make 
changes using two methods. The first method allows the user to simply 
touch the other candidate; e.g., Candidate A is initially selected and 
the voter decides to select Candidate B by touching the name of 
Candidate B. We referred to this as the "one-touch method." The other 
method, referred to as the "two-touch method," involves the user first 
deselecting the initial choice and then making another selection; e.g., 
Candidate A is initially selected and the voter decides to select 
Candidate B by (1) touching the name of Candidate A, which deselects 
Candidate A, and then (2) touching the name of Candidate B to select 
it. 

[17] The actual procedure is to (1) go to the last review screen, which 
activates the VOTE button, (2) page back to the contest (normally 2 or 
3 page backs depending on the ballot style), and (3) selecting the 
contest on the review screen that should be revisited. We assumed that 
voters would cast such ballots using this procedure instead of using 
the page back option because it did not appear reasonable that a voter 
would page back at least 17 screens to reach the Florida-13 race, which 
was the focus of the testing. 

[18] The standard actions taken in these two test ballots called for 
the tester to (1) make a selection on the initial screen and then 
change the selection, (2) page back to the initial selection screen and 
change the selection, and (3) use the review screen option to change 
the selection again. 

[19] Details on the random selection can be found in appendix I. 

[20] We excluded machines from one precinct that used two ballot styles 
instead of one. 

[21] In order to ensure that we could complete our tests even if a 
machine selected for testing failed to operate, our statistical 
sampling methodology generated a list of machines that could be used as 
replacements and still maintain the integrity of the testing process. 
These are referred to as "back-up" machines. 

[22] These two reviews identified two early voting and seven election 
day test ballots where the specified scripts were not followed exactly 
for the Florida-13 contest. Because these test ballots had not followed 
the test script for the Florida-13 contest exactly, they were retested. 
Accordingly, the testing efforts resulted in 233 actual ballots being 
cast. 

[23] Florida Department of State, Florida Voting System Standards, Form 
DS-DE 101 (Jan. 12, 2005). 

[24] The iVotronic DRE is designed to maintain a count of all ballots 
cast on a given machine and functions much like an automobile's 
odometer. The protective count can be used to help ensure that the 
election process did not lose any votes. For example, before a machine 
is sent to a precinct, the protective count is recorded. Accordingly, 
if the precinct's voting register show that 100 individuals voted, then 
the increase in the protective counts for all machines assigned to that 
precinct should increase by 100. This value can then be compared to the 
actual votes recorded in the election to ensure that the values are 
consistent; i.e., the results tape for the election shows that 100 
votes have been accounted for during this election using this example 
precinct. 

[25] In some cases, a test ballot had to be reentered because the 
original test did not follow all of the desired actions associated with 
the Florida-13 contest. In these cases, the value entered was made 
unique by adding a letter to the value, e.g., "TB1A". 

[26] The approach used to select these machines is described in 
appendix I. 

[27] In the testing of the first two miscalibration patterns for the 
first machine, all the test ballots used in the ballot testing for that 
machine were repeated. However, the individual performing the testing 
soon recognized the changes that were needed to compensate for the 
miscalibration. Accordingly, the tester did not make as many attempts 
to perform the desired function in the later cases as with the first 
three cases. Therefore, for the remaining eight miscalibration test 
patterns, we executed three test ballots per pattern because these 
cases produced the greatest likelihood of generating spurious touches 
before obtaining the desired selection. 

[28] Our review of the election day records identified two reported 
cases on election day where the miscalibration of the iVotronic DRE led 
to its closure and discontinued use for the rest of the day. 

[29] While votes are normally cast using fingers on the touch screen, a 
stylus is normally used during the calibration process. 

[30] We also excluded those election day machines from one precinct 
that supported two different ballot styles. 

[End of section] 

GAO's Mission: 

The Government Accountability Office, the audit, evaluation and 
investigative arm of Congress, exists to support Congress in meeting 
its constitutional responsibilities and to help improve the performance 
and accountability of the federal government for the American people. 
GAO examines the use of public funds; evaluates federal programs and 
policies; and provides analyses, recommendations, and other assistance 
to help Congress make informed oversight, policy, and funding 
decisions. GAO's commitment to good government is reflected in its core 
values of accountability, integrity, and reliability. 

Obtaining Copies of GAO Reports and Testimony: 

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through GAO's Web site [hyperlink, http://www.gao.gov]. Each 
weekday, GAO posts newly released reports, testimony, and 
correspondence on its Web site. To have GAO e-mail you a list of newly 
posted products every afternoon, go to [hyperlink, http://www.gao.gov] 
and select "Subscribe to Updates." 

Order by Mail or Phone: 

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to: 

U.S. Government Accountability Office: 
441 G Street NW, Room LM: 
Washington, D.C. 20548: 

To order by Phone: 
Voice: (202) 512-6000: 
TDD: (202) 512-2537: 
Fax: (202) 512-6061: 

To Report Fraud, Waste, and Abuse in Federal Programs: 

Contact: 

Web site: [hyperlink, http://www.gao.gov/fraudnet/fraudnet.htm]: 
E-mail: [email protected]: 
Automated answering system: (800) 424-5454 or (202) 512-7470: 

Congressional Relations: 

Ralph Dawn, Managing Director, [email protected]: 
(202) 512-4400: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7125: 
Washington, D.C. 20548: 

Public Affairs: 

Chuck Young, Managing Director, [email protected]: 
(202) 512-4800: 
U.S. Government Accountability Office: 
441 G Street NW, Room 7149: 
Washington, D.C. 20548: 

*** End of document. ***