Washblog

This is my response to the King County Executive's plan to use automatic signature verification (ASV) during mail ballot processing. I claim that ASV will result in more errors, more work, more expense, less transparency, and less confidence in our election results.

With mail balloting, voters sign the return ballot envelope. Those signatures are then compared with the signature on the voter's registration form. Currently, the process is semi-automated, where humans do the signature verification. King County intends to fully computerize this process with automatic signature verification (ASV) software.

Real World Testing

Whatcom County tested Diebold VoteRemote's ASV system April 24th, 2007. The King Co Executive's ballot accountability business case has chosen ParaScript's solution, which is the same idea. (And may be the same underlying software used by Diebold.)

Whatcom County conducted a controlled test using batches of known matching and mismatching signatures. These batches were taken from a recent election. Their findings are very disturbing.  (See attached spreadsheet for source data.)

They determined the false rejection rate (FRR) is between 26% and 63%. This means that matching signatures are incorrectly rejected (challenged) by the system. The false rejection rate is dependent on the settings of the system, from "very tolerant" to "most rigid".

They also determined the false acceptance rate (FAR) is between 29% and 7%. This means that mismatching signatures are incorrectly accepted by the system. Again, the false acceptance rate is dependent on the settings of the system from "very tolerant" to "most rigid".

ASV systems have a no-win tradeoff between false acceptance and false rejection. As Whatcom County's testing shows, attempting to reduce false acceptance results in many, many more false rejections. Conversely, attempting to reduce false rejections results in more false acceptances.

Automated Disenfranchisement

As reported last year, half of the ~24,000 voters who had their signatures challenged (rejected) during the general election November 2006 did not respond in time to have their votes counted. (citation needed)

In their business case, King County Elections states that they'll have humans review all the challenged signatures. This is lunacy. The computers actively injects errors into our elections, to be cleaned up by humans. Wasn't the goal to reduce human processing?

When "automation" creates such a high challenge (rejection) rate, it's inevitable that more voters will be required to revalidate their signatures, leading to higher rates of disenfranchisement.

Risk of Undetectable Voter Fraud

[I know this is hot button issue for both the left and right, but please hear me out. I'm only talking about the technology King County Elections intends to use, not recent events.]

As Whatcom County's testing shows, using ASV results in false acceptance of mismatching signatures. Once these are accepted, there is no way to remove those ballots from the election. So the whole election is spoiled.

If It Works, Smash It

As proudly reported by King County Elections, our current mail ballot processing system is nationally recognized. Humans compare a voter's signature from the voter registration card with the actual signature on the return ballot envelope. If there's a suspected mismatch, the voter registration card is pulled and a side-by-side comparison is done.

Our human verifiers are trained by Washington State Patrol. The computers probably haven't benefitted from this training.

Many kinds of fraud are only detectable by reviewing the actual physical ballot envelope. Tracing of a signature, obviously different handwriting, and so forth.

Since our current system works so well, as a result of continuous improvement over the years, it's now painfully obvious to me the only choice is to throw it all away and start over. (sarcasm)

Inappropriate Use of Technology

This section is a grab bag of remaining issues. When I say "inappropriate", know that I'm the Original Technophile, not some modern day Luddite.

Banking Industry

ASV proponents point at the banking industry and say "See! Good enough for finance, good enough for elections." This is a false comparison. Much like prior attempts to compare computerized voting to automatic teller machines.

The banking industry uses ASV very differently. First, only flagged transactions are examined. For instance, for too much account activity or unusually large amounts. Second, they require that any fully automated system have a FRR and FAR of under 1%. That's just not possible with today's technology.

Secretary of State Approval

The Sec of State recently used its rulemaking authority to address the use of ASV. WAC 434-335-625 "Signature verification system acceptance testing" states in part:

"(3) The county auditor must perform a series of tests to verify that the software is not accepting signatures that the county auditor's trained signature verification personnel would not accept. ..."

No mention of the false rejection rate. No standards for FAR and FRR. No procedures for humans re-examining challenges. Hardly seems adequate. It's a start, I guess.

False Optimization

It's hard to understand why we'd even need ASV. Signature verification is not a bottleneck in mail ballot processing. With our current system and procedures, verifying a batch takes about 30 minutes for about 250 ballots.

Overall, the two largest bottlenecks are the USPS (most ballots are mailed on election day) and opening the return envelopes (about 2 hours per batch). Curiously, neither of the Executive's business cases address either of these actual bottlenecks. Meaning we'll spend $5m for no actual benefits and plenty of downside.

False Savings

As the cover page for the ballot accountability business case notes, there is no positive return on investment (ROI).

One clever bit of the two business cases are that we a) buy the equipment and then b) pay to use the equipment. The usage costs for the sorting and ASV systems chosen by KC Elections aren't included in the plan, so  use the costs for the Diebold VoteRemote, assuming they're comparable.

Diebold charges 10 cents per envelope (to scan the barcode) and then 20 cents per signature successfully verified. I estimate that King County will receive 2m mail ballots every year (general, primary, and special elections). Using the VoteRemote would cost us $600,000 per year. Even if we eliminated all human signature verification jobs (tasks), it's hard to imagine saving any money.

Automation and Quality Assurance

Automation is removing all human decision making from a process. This is often confused with computerization (digitization).

Signature verification requires human judgement. It is therefore not suitable for automation.

Quality assurance is achieved by removing all sources of errors (waste) from a process. Other names are total quality management, continuous improvement, The Toyota Way, etc.

Preventing errors is quality assurance. Detecting or correcting errors is also not quality assurance.

Automatic signature verification introduces errors into mail ballot processing. Its use will therefore reduce quality.

Conclusion

King County's current semi-automated system, where humans do the signature verification is the best available system. Do not change.

Using automatic signature verification for mail ballot processing is a horrible idea. It is more expensive. It is very error prone. Using ASV will undermine our democracy.

Representatives from King County Elections as well as Paul Miller (Sec of State's Office) attended the testing in Whatcom County. I've already asked Miller for a copy of his report. I'll be submitting open records requests shortly.

I do not believe that the Citizens Election Oversight Committee, the Council or the Executive have been given accurate or timely information about ASV. This lack of full disclosure is a very troubling.

I strongly encourage both the CEOC and Council to be more vigorous in their oversight efforts.

Gratitude

Many thanks to Marian Beddill, member of Whatcom Fair Voting. She is a tireless champion of election integrity. I'm citing her testing data.

Also thanks to Whatcom County Auditor Shirley Forslof. She had the wisdom and initiative to actually test the Diebold VoteRemote, despite their relative lack of resources. Forslof also had a public demo of the VoteRemote last year, which I attended.

References

Ballot tracking and accountability recommendations sent to Council: Voters to be able to follow ballot online (May 17, 2007)
http://www.metrokc.gov/exec/news/2007/0517ballot.aspx

King County receives top honors for excellence in mail ballot processing (July 25, 2006)
http://www.metrokc.gov/elections/news/2006_07_25.html

2007 Cycle 2 Rules: Signature Verification, Recounts, Motor Voter, Voter Intent, and Election Reports

Data From Whatcom County Testing April 24th, 2007
(attached)