Executive Summary

Three voting systems, the Diebold AccuVote TSx, Hart eSlate and Sequoia Edge I and II, were evaluated for usability and accessibility for voters with disabilities and voters with alternate language needs, using both heuristic and user testing techniques. Although each of the tested voting systems included some accessibility accommodations, none met the accessibility requirements of current law and none performed satisfactorily in test voting by persons with a range of disabilities and alternate language needs. In some cases the accessibility or usability deficits could be partially or wholly mitigated. Some of these mitigations would not require new federal and state certification testing.

Introduction

Voting system accessibility surveys and reviews by Cook County (Illinois), Access World and others have shown that the voting systems previously used in California have significant limitations in accessibility for many voters with disabilities and alternative language needs.

Recently, the designs of some of the voting systems have been changed to try to improve both their accessibility and their security. In some cases, the changes made to add voter verified paper audit trail (VVPAT) printers and other security and privacy improvements have had a negative impact on the accessibility of the systems for voters with disabilities.

Because it is impossible to affirm overall accessibility and usability conformance merely by examining documentation for voting products, and because there have not been in-depth accessibility studies performed for these voting systems, rigorous testing was required to assess the accessibility and usability of California's voting systems. However there are currently no rigorous methodologies or standards defined, in the 2005 Voluntary Voting System Guidelines (VVSG) or elsewhere, for testing accessibility of voting systems. The Federal Election Assistance Commission (EAC) and the National Institute of Standards and Technology (NIST) are still in the early stages of development of standards for voting system accessibility testing.

Purpose of the Review

This review was undertaken primarily to identify whether the three systems were sufficiently accessible for voters with a range of different disabilities, and whether they were generally usable as well. Alternative language accessibility was also assessed.

Additionally, this access review was tasked with identifying specific accessibility and usability concerns and reporting options for near-term mitigations appropriate for the 2008 elections, as well as longer-term mitigations including voting system design changes.

The results may be used to guide future actions by the Secretary of State’s office regarding certification. In addition, vendors and local election officials may find useful information in this report that will improve the usability and accessibility of voting technologies in both the near and long terms.

Finally, the authors hope that the larger community of stakeholders concerned with voting technologies will find both the methodology and results useful in developing advanced practices of design, testing, and implementation.

VVSG Provisions on Usability and Accessibility, Including A Brief History

The Help America Vote Act of 2002 (HAVA) established the United States Election Assistance Commission (EAC), the agency responsible for federal funding and technical assistance in voting technologies. HAVA section 301(a)(3) sets forth accessibility requirements. EAC inherited previous guidelines, the 1990 Performance and Test Standards for Punchcard, Marksense and Direct Recording Electronic Voting Systems, and the 2002 Voting Systems Standards (VSS). EAC empanelled the Technical Guidelines Development Committee (TGDC) to develop further standards, supported by the National Institute for Standards and Technology (NIST). The work of the TGDC, comments from the public and other experts at public hearings, and the EAC itself as well as other authoritative reviewers, resulted in the release of new Voluntary Voting System Guidelines (VVSG) in December 2005. The VVSG supersedes the 2002 VSS effective in December 2007.

The VVSG contain a completely new section on usability and accessibility requirements. These reflect the HAVA 301(a)(3) accessibility requirements.

As already indicated, there is not nor should there be an absolute line drawn between accessibility and usability. The Section 3 requirements in the VVSG contain references to both, and provide specific guidance regarding the goals of accuracy, efficiency, and satisfaction. Using a framework from the domain of accessible technology, they address the needs of users with functional limitations in vision, hearing, mobility, dexterity, speech, and cognition. For all of these dimensions of functional limitation, VVSG addresses both perception and interaction where relevant. Additional issues are also included: privacy, protection of voters who use alternate formats or methods for voting, and the use of alternate languages.

In this report we will use the VVSG requirements in two ways. First, they form the framework in which we identify overarching issues we found in our testing. Second, they are the reporting method we use in the appended results of the review for each of the voting systems tested.

Usability and Accessibility

The scope of this accessibility review is primarily limited to human factors issues, meaning we are concerned with the entire process of the voter casting a ballot as they intended.

This process involves not only the voting system interface directly experienced by the voter, but also includes usability issues pertaining to ballot design, the influence of the polling place environment on accessibility and usability, as well as the setup, operation, and support of accessible voting systems by election administrators and pollworkers.

Definitions of Usability and Accessibility

The International Standards Organization defines usability as:

“The extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.”

It could be said that accessibility consists in defining those “specified users” as inclusively as possible. That is, the goals of effectiveness, efficiency, and satisfaction are unchanged, but equal attention is paid to making sure that those goals are met for people with disabilities. Accessibility can be further clarified by categorizing the accommodations used to include these “additional” users: some are extensions to usability features (such as magnification) with significant benefit to non-disabled users, and some provide compatibility with assistive technologies rarely useful to people without disabilities (such as sip and puff switches).

The Goal of Good Usability and Accessibility

The goal of good usability in voting equipment for voters and pollworkers is to easily, accurately, and intuitively use the equipment with minimal instruction and training and to successfully complete the voting process.

Accessibility can be seen as the additional accommodations, used primarily by people with disabilities, that help them to successfully use the voting equipment. For this reason, this report most often treats accessibility as a subset of usability.

Testing voting equipment for both usability and accessibility is potentially complex, but necessary for two key reasons:

1. Any usability problems that a piece of voting equipment has is likely to impact voters with disabilities more seriously than voters without disabilities.

2. A piece of voting equipment might have the legally required accessibility accommodations, but still may not be very usable by voters with disabilities, if the accommodations were poorly designed or improperly implemented.

Scope and Limits of the Review

This study was undertaken over an extremely brief period of time from its approval to its completion, due to the rescheduling of the California primary and the resulting compression of the election preparation schedule. This, as well as other factors have unfortunately limited the scope of this access review, and we wish to be explicit about those limitations.

Only three systems were included, although several more, using different designs, have been certified for use in California.

The versions of these three systems may not have been the only versions used in California, and new models of one or more of the systems may already be undergoing testing for approval by the federal Election System or the Secretary of State.

The ballot definitions used on the three machines were not identical, and were not based on a ballot design optimized for usability testing.

We were not able to test a multiple member race in which the voter may choose more than one candidate.

The alternate language ballot definitions supplied by the vendors for the machines we tested may not have been as complete or optimal as those used in a normal election.

The 45 volunteers who served as test voters (“voters” or “users”) were selected to represent a broad range of disabilities. They cannot be assumed, however, to be perfectly representative of all possible voters with disabilities in degree and type of functional limitation, experience with voting, or pre-existing attitude toward voting technologies.

Finally, because successful accessibility of the voting system by the voter depends, in many ways, on the ability of the pollworkers and elections administrators to set up, operate, and support the voting systems properly, it is important to also review and address the usability of the voting systems for pollworkers. This was outside the scope of the current access review.

General Methodology

Heuristic Analysis

The two authors of this report, assisted by a volunteer with expertise in accessible technology and voting systems, served as expert reviewers of the three systems. The goal of their analysis was to identify as many potential accessibility and usability issues for voters as possible in advance of the user testing, and afterwards to confirm and clarify issues identified during the user testing.

These expert analyses took three forms:

Group Walkthroughs

All three experts worked together in a single session, engaging in a dialogue as one expert navigated through the scenario. These were recorded by note taking, video/audio recording, or both.

Individual Walkthroughs

A single expert exercised a specific interface, with his verbalized interactions and findings recorded on audio and video.

Review of User Videos

After the user testing was complete, we reviewed a subset of the sessions for several purposes. First, we wanted to confirm the accuracy of the session timing results. Second, we wanted to be sure that particular issues noted in the session data forms appeared similarly in the recording. Third, we wanted to identify any additional information that would help explain usability or accessibility issues, or deepen our analysis.

Methodology for User Testing

Physical Layout

The testing took place in a large multifunction room on the ground floor of the office of the Secretary of State in downtown Sacramento. Within this room we installed 3 10-foot-square cubicles with sound abating walls. One voting system was installed in each cubicle. Each voting machine was loaded with a ballot definition based on an actual 2004 California general election ballot. As indicated in the Scope section, the ballot definitions for the three systems tested were similar but not identical.

Cameras were positioned in the cubicles to record the contents of the screen, the user’s actions at the controls, and the user’s face to capture any emotional expressions. A microphone recorded any speech by the user and “pollworker”, and an additional sound track captured any speech from the voting system.

Testing Protocol

We developed a brief intake instrument for the study, focused on basic demographic information, disabilities, and voting experience. Project staff assisted the users in completing this form and the consent forms as needed.

For privacy protection, the users were not referred to by their names during the testing. To better simulate a real polling place experience, the test subjects were referred to as "voter" and the experimenters were referred to as "pollworker". To limit user bias for or against particular vendors or product names, the voting systems were only referred to as 'A', 'B', or 'C', and the cubicles were so labeled.

Users were assigned to the three systems in a randomized order. For each system, users were assigned a pollworker who provided a standardized amount of orientation and assistance in getting started. For users using the audio interface this often meant adjusting the volume. For users who used wheelchairs it was usually necessary to re-arrange the equipment.

Users were encouraged at all points to perform as many tasks as possible independently. This included entering the four-digit voter access code used by the Hart eSlate and inserting the voter access cards into the slots on the Diebold TSX and Sequoia Edge.

There were five timed segments to each trial. The first was the orientation provided by and with the pollworker. Next, the voter was allowed to complete the ballot however he/she wished, moving through the ballot "freestyle" as if in an actual election, and making choices according to his/her own preferences. Once the voter reached the end of the ballot, we asked him/her to go back to a specific race and change his/her selection. Then we asked the user to enter a write-in name for another specific race. Then we began a sequence leading up to actually casting the vote, which included reviewing the ballot, printing a verification, rejecting the ballot and making a change, and finally printing another verification and casting. We collected the elapsed time for each of these five segments.

We asked the users to state out loud what selections they were making for each contest, as they were making their selections. This was to assist us in determining their intended choice, so we could later determine their voting accuracy.

We also encouraged the users to verbalize their thoughts as they were working with the system. Several of the users were able to give us extremely useful verbal stream-of-consciousness observations.

After each user completed voting on all three systems we interviewed the user to collect specific reactions on a data collection sheet. We asked the user to rate the system on several factors, and asked whether he/she would be willing to use that system in a real election. Additionally, we encouraged a full discussion of the user’s reaction to the system, including specific features he/she thought were important as well as any suggestions. When possible we conducted these discussions while the user was still in front of the voting system, so they were able to look at and/or touch the system while commenting about it.

Major Findings

Physical Access to the Voting Machines

Physical access concerns arose in four contexts:

• Positioning for wheelchair users

• Interference with wheelchair controls and armrests, both frontal and lateral

• Standing and seated use

• Machine support stand stability

Positioning for Wheelchair Users

The VVSG requires a minimum of 30 inches of toe and knee clearance. No machine provided that clearance. This deficit posed a problem to almost every wheelchair-using voter in this study.

Sequoia Edge II Diebold TSX

Hart eSlate

Interference with Wheelchair Controls and Armrests – Frontal Approach

The bottom edge of the eSlate interferes with the wheelchair joystick and even with the armrest.

The same joystick barely cleared the VVPAT on the Diebold after adjusting the angle of the machine.

The front part of this wheelchair control barely makes it under the bottom edge of the eSlate; that edge is too low for the joystick itself.

Side view of the same wheelchair control approaching the Diebold; the VVPAT interferes with the joystick.

The user’s knees meet the bottom of the eSlate while the user is still out of range of the controls.

This wheelchair control juts forward of the right armrest. The Diebold VVPAT prevents enough forward progress for the user to reach the touch screen.

The narrow clearance between the two front legs of this machine prevents the user from approaching from the front, as the wheelchair footplates are wider than the legs would allow.

Interference with Wheelchair Controls and Armrests – Lateral Approach

The Diebold VVPAT interferes with armrest when user approaches from the left.

The lower edge of the machine is too close to the wheelchair control when this user approaches from the right.

Adjustability for Standing and Seated Use

The systems we tested had little or no height adjustment capability; two were able to adjust the angle at which the screen was presented. This lack of adjustability posed a problem principally for voters who would prefer to sit, but also posed a problem for voters who wanted to stand but were limited in their ability to bend over to read the screen and/or VVPAT.

The range of angle adjustment was occasionally insufficient. In some cases, we needed to place blocks under the back legs of the voting machine stand, in order to tilt the unit forward enough to give a voter in a wheelchair access to the touch screen.

Machine Support Stand Stability Concerns

One machine’s support structure seemed prone to tipping or breaking if it was subjected to uncontrolled movements of an individual or a wheelchair.

The lack of adjustment flexibility in the support stands of the current systems severely limits physical access to the voting machines for many voters with motor impairments.

Manual Dexterity Accommodation Concerns

Users with impaired dexterity and reach had some difficulty using these systems. Concerns arose in three specific areas regarding dexterity: voter access cards, physical controls, and touch screen controls.

Voter Access Cards

Voters with impaired grip strength had difficulty picking up the cards (if they were placed on a table), holding the cards, orienting them properly, reaching the card slot (especially on the Diebold TSX), applying enough force to engage the card in the card reader, and retrieving the card after voting. The Hart eSlate, which uses an access code instead of a card, did not pose any accessibility concerns here for voters with dexterity limitations.

Physical Controls

All physical controls pose some degree of difficulty for people with impaired dexterity. The Hart eSlate controls are fairly well designed, with a good dish on the Select wheel that would aid use by finger or head- or mouth-stick. Note, however, that the circular motion required by the Select wheel is difficult for most stick users. The other buttons are large enough, and placed far enough apart (except for the Previous and Next buttons) for use by all but the most severely physically disabled voter, who would probably prefer the external switches. However the lack of relief on the built-in controls means that head- or mouth-sticks might pop out before activating the intended control.

The other two systems are intended to be controlled by touch screen, not by their physical controls (which are intended for blind users). We will briefly mention the possible difficulties these controls would pose to users with both dexterity and vision impairments. The Sequoia Edge controls are too stiff for some users, with a large degree of travel and sideways displacement. The controls protrude more than a quarter-inch from the shell of the control box, making them especially difficult for voters using head- or mouth-sticks. This is also the case for the Diebold TSX. The Diebold keypad keys are more stiff than necessary, and do not offer any friction surface.

Touch Screen Controls

The two touch screen systems pose a reach/range problem for voters with impaired range of motion, and reduced strength near the limits of their range. Some users had difficulty reaching the very top of the touch screen – the distance from top to bottom is quite large, by design. If the touch screens were oriented in landscape mode this range problem would be much reduced.

The other problem lies in the size and shape of the screen targets, and the separation between them. In most cases the active targets are low, long rectangles with little or no space between them in the vertical dimension. The optimal design for people with even moderately reduced accuracy would be square or circular targets with large, evident dead spaces between them.

Access User Interface Connectors

On the Sequoia Edge, the supplied sip and puff head-mounted control plugs into the keypad unit properly. However, the jelly switches also supplied by Sequoia are not compatible, as they have a right angle plug and the Sequoia Edge keypad needs straight-in plugs. Because the dual switch jacks on the tactile keypad are too close together and tightly surrounded by the case housing, it is not easy to find extension cables or extension adapters that can plug into the two jacks, side-by-side.

Additionally, the opening around the Sequoia Edge audio headphone jack is in a channel that is too narrow for some 1/8 inch headphone plugs to fit, making it difficult or impossible for voters with assistive listening devices or voter-supplied personal headphones to plug in.

Both of the Sequoia Edge I and II units we were testing required an on-site visit from factory representatives, due to the units freezing up or “hanging” without helpful diagnostic message when keypads were plugged into an unused improper jack on the back of the units identical and immediately adjacent to the jack intended for connection of the keypad. Instead of being labeled "Keypad", the correct jack was labeled "Audio" and no color coding was used to help pollworkers match the plug and jack. When the keypad was plugged into the incorrect jack and the voting machine was loaded with a voter card, the unit would hang, and would not report that it could not find a keypad plugged in to support the audio voting specified on the voter card.

Another time, the Sequoia Edge II test unit began making random sounds that could be described as an “angry chipmunk chattering". Sequoia engineers advised us that the noises were probably the result of the same keypad plug coming loose, according to Sequoia engineers. Subsequent daily checks by the access team leader found this same keypad plug often was not plugged in firmly and could be removed with only the slightest pulling on the cable near the plug.

The Hart eSlate tablet can be removed from its stand for curb-side voting, and when installed on the stand, the clearances between the inside wall of the stand and the jacks on the side of the tablet unit are very tight and only allow space for right angle plugs in those jacks. Therefore, the jacks will not accept a straight-in headphone plug. If headphones supplied by a user do not have a right angle plug they must be connected through an extension cable with a right angle plug.

Additionally, the headphone cable must be routed up through the stand and out the flap openings at the top edge of the tablet. This routing may make the remaining cable on a user's headphones too short.

During our testing, the eSlate unit reported printer failures several times. In each case, the failure appeared to be caused by pulling the tablet out of the stand and then not getting it settled and plugged back into the stand properly. These printer failures exacerbated the inconvenience of having to remove the tablet from the stand every time the headphone or dual-switch input jacks need to be accessed.

Visual Display Concerns

Touch Screen Parallax

Touch screen systems are typically calibrated for use by an average height voter in a standing position. Voters positioned lower in chairs may experience an optical parallax effect that makes the screen's visible buttons appear to be slightly higher than the touch sensing area for that visible button. Touching near the boarder of a visible screen button may cause selection of the adjacent button.

Screen Rendering Time

We measured the time it took both Edge I and Edge II in magnified mode to completely render a screen – the “screen write time”.

• On the Edge I, 14 seconds to set up the next page and 24 seconds to set up a horizontal scroll.

• On the Edge II, 7 seconds to set up the next page and 8 seconds to set up a horizontal scroll.

The screen write times of both systems are long; such individual delays may confuse or frustrate a voter.

Touch Screen Controls

Our testing revealed that touch screen controls on all three tested systems pose insuperable barriers for sighted voters who cannot see the screen well enough to identify where the active surfaces are. All three systems provided alternative input devices for these voters.

Non-Touch Screen Input Controls

Several voters in our testing found that some of the voting systems require so many repeated button presses or wheel movements that they were feeling exhausted and sore. Some of the input control methods take several times as many keystrokes or actions as the other systems.

The three systems have very different physical controls

Hart eSlate

The controls of the Hart eSlate are built into the tablet, all along the lower edge directly beneath the screen. At the extreme right is a rotating Select wheel (radius = 31 mm). The wheel has radial ridges and a single dished depression for a finger or mouthstick. To its left is the Enter button, 50 mm tall by 32 mm wide, with a scooped right edge echoing the Select wheel’s circumference. To its left are three buttons: a rounded rectangular Help button (11 mm tall by 32 mm wide) above a pair of triangular buttons Previous and Next. These latter buttons are roughly equilateral, 20 mm tall and 20 mm wide, oriented as arrowheads pointing in opposite directions. At the extreme left is the Cast Ballot button, round (radius = 17 mm) with a truncated top. Distances between the controls range from 11 mm to 25 mm. The Cast Ballot button is red with white letters; the others are off-white with black letters. Braille appears below the buttons. There are no volume or speech rate controls.

The Select wheel is used as a scrolling device within and between races, with approximately 20 tactile click points per rotation.

All buttons have a very slight bevel, but there is no relief or reveal. That is, aside from the bevel, the buttons are flush with the shell of the tablet. The key tops are not dished or recessed, to make it easier to position and keep a mouthstick on them. Some users found the buttons difficult to navigate by touch; one commented, "The keys should be raised to make them more obvious."

The braille on the Previous and Next keys is not horizontal like the others, but follows the diagonal line of the bottom of the arrowhead alignment. This was disorienting to some braille users.

Some users liked the wheel, but some found it too easy to turn inadvertently. Some found it at least initially confusing. One voter said she started off thinking she would not like the wheel and would prefer arrow keys, but liked the wheel for write-ins. One voter commented, "[the] wheel makes it more intuitive". At least one user was confused about which direction to turn the wheel.

Some users found that there was not enough mechanical feedback from the buttons; they were concerned that their input had not been accepted.

Having a Cast Ballot button made that function clear and evident, but it did cause some user error when voters pressed it before they were really ready to vote. Some confused the Cast Ballot button with merely activating their current choice in a race.

Voters did not use the Next and Previous keys very often.

Diebold TSX

The controls are laid out like a conventional telephone keypad: 4 rows of 3 columns, with the digits, ‘*’, and ‘#’. There is a nib on the ‘5’ key. The keys are 13 mm high by 21 mm wide, and are separated by 5 mm horizontally and 9 mm vertically. Print on the keys is white on a black background. There is no braille.

The control keypad is tethered to the machine and nests above the VVPAT. The keypad can be moved and repositioned without difficulty. However, it was difficult for voters to replace the keypad properly. Its undersurface has rubbery pads for stability when used on a flat surface such as a table or wheelchair lap tray. The laterally dished underside does not facilitate connection to a voter’s leg or armrest. One voter asked for “the keypad mount [to] be a horizontal slide to let you position it left, middle, or right." Because of its width and lack of an underside grip, it was awkward and tiring for users to hold the keypad in one hand, for the long voting process, while pressing keys with the other.

In theory a familiar keypad is an advantage. However, this may have been offset by the fact that there was no direct mapping between the keys and many of the functions the voter wanted to perform, and no braille, color, or shape indication of their functions.

Frustration was expressed by several voters during the write-in task. Letters were entered via the ‘2’ through ‘9’ keys. Editing and selection functions were arbitrarily assigned to the other keys. Key assignments changed from one function to another, further confusing some users. One user asked for a “summary of which letters go on which keys”; another complained that “Key '5' is sometimes used for cancel and sometimes confirm; that seems contradictory.”

Sequoia Edge

At the top of the control box are small (8 mm) buttons for volume control and speed of speech. The volume buttons are separate and identical; the speed controls are connected to each other. Below these, near the right edge, is a blue square (21 mm) Help button. Below this is a pair of arrow buttons, Back and Next, yellow and green respectively, 21 mm high by 24 mm wide. Below these, near the right edge, is a round (radius = 12 mm) red Select button. The distances between controls range from 16 to 18 mm.

Although there are braille labels for the keys, the labels are shallow and too close to key caps to facilitate reading by some voters' fingers.

The Sequoia Edge ATI keypad is tethered to the voting machine by a heavy data cable. With a Velcro cloth adapter added to its back, the keypad can be stored by sticking it to the Velcro claw patches on the outer and bottom edge of the left privacy panel. However, because of the keypad size and weight, it does not appear to be stored securely when it is stuck onto any of the Velcro patches of the privacy panel. Voters with manual dexterity impairments can encounter problems attempting to independently remove a Velcro-attached keypad from or reattaching it to the system's privacy panels. "Parking" the keypad on the privacy panel for operation puts the keypad at a height and angle that makes its operation extremely uncomfortable for visually impaired voters who need to keep their fingers on the keys, for touch typing rather than operation by single finger poking. The force required to press the keys of the keypad would make the flexible privacy panel sway or wobble back and forth substantially, if voters attempted to use it mounted there. It appears that the Velcro cloth roll adapter on the back of the keypad supplied for testing is a new feature that has not been generally available on currently fielded Sequoia Edge systems in California. Although it may be of some use for storing the keypad between voting sessions, sticking the keypad to the privacy shield isn't a reasonable solution for a parking place to support the keypad for single handed operation.

Several voters commented that the controls were easy to use, but might have better instructions.

One user found that pressing the Select button off center could catch the key cap under the shell of the control box, at least temporarily.

Counting of Keystrokes or Other User Control Input Actions

Note: This is for the non-touch-screen input control alternatives such as keypad or selector wheel.

Minimum keystrokes needed for a write-in of "John Smith":

• 103 for system A, Hart eSlate, wheel clicks and Enter keystrokes. Multi click wheel spinning might decrease the effort.

• 32 keystrokes for system B, Diebold TSX, using telephone-style text messaging entry.

• 130 keystrokes for system C, Sequoia Edge, using back and forth selection in a circular alphabetic list.

Note: The Sequoia write-in character selection list has numbers 0-9 and several other punctuation characters that make it a much longer list to navigate than that of the shorter eSlate character list, and takes about 4 times more keystrokes than the phone text-messaging approach of the Diebold TSX.

In the case of write-ins, the cognitive load of phone-style text entry may cause many voters to prefer the lower cognitive load of the selector wheel for write-in tasks.

Also, because sip and puff or other dual-switch input controls are not supported with a Previous Selection control, write-in with dual-switch control could take several hundred strokes or sip/puffs.

<