Text of my
letter to the California Secretary of State office with public commentary on the proposed certification of the LA County VSAP 2.0 system's universal-use Ballot Marking Devices.
Dear Secretary of State Padilla,
I write to provide my comments on the VSAP 2.0 evaluation and certification process for LA County. I have reviewed the written reports and corresponded with Dean Logan, RR/CC for LA County, to try to resolve open questions about the system and the process. I have several open questions remaining, but with the deadline upon us I will give the feedback I can with the information I have. If I have interpreted anything incorrectly, please let me know, and please understand that I have done everything I can with the time allowed to gather information and form opinions
Background
I am a software engineer (B.S. Computer Science, M.S., Computer and Information Sciences). I have worked as a software engineer in the industry for 30 years. I have ordinary skill in the art with various standards and technologies for data storage, distributed computing, and security for online and offline systems. I am not a security specialist but have worked extensively with such specialists in building large scale consumer systems. I have also worked as an election clerk in Santa Clara County and am familiar with the processes and systems used in that county.
General
I have been happy to hear that LA County has been working to create a system that is fully owned by the public, not by a vendor. The flexibility and potential ability to re-use this system in other counties is very promising. At the same time, I have concerns about this system that I ask to be considered both in certifying it for 2020 and going forward in LA County and elsewhere. They primarily involve the security of ballot marking devices and how those impact the assurances of Risk Limiting Audits (RLAs).
Universal Ballot Marking Devices Are an Unnecessary Security Risk
For security reasons, and because it is not a necessary requirement for the improvements incorporated in VSAP 2.0, I strongly object to the requirement for universal use of ballot marking devices (BMDs) for in-person voting. I am basing this on the reporting that “Starting with the presidential primary, every in-person L.A. voter must use a ballot marking device” While voters still have the option of hand marked paper ballots for mail-in ballots, this is not available to in-person voters. The only option for voters wishing to opt for a hand marked paper ballot is to request a pre-printed ballot by Feb 25:
"Pre-printed ballots will not be available at vote centers," Logan said, adding that voters who want to use pen and paper should request a mail-in ballot by Feb. 25.
Mr. Logan claims that denying hand marked paper ballot options to in-person voters is because having that option “creates a separate but equal type of scenario.”. He provides no evidence or argument to support that claim. In addition, the fact that California does allow hand-marked paper ballots for voters who request them before Feb 25 undermines this argument -- if this were truly a “separate but equal” issue, why is it a non-issue for vote by mail voters?
On the other hand, there are strong security arguments for allowing voters to hand mark paper ballots. Specifically, hand marked ballots are not vulnerable to an entire class of electronic attacks against ballot marking devices (which the VSAP 2.0 BMD clearly belongs). We have recent experimental evidence that voters do not effectively verify BMD printed ballots without very specific training and real time prompts in the polling place -- none of which is in place for the March 2020 elections.
In addition, the VSAP 2.0 system may potentially be vulnerable to a variant of the attack Andrew Appel termed “permission to cheat”, because it includes a print head in the paper scanning path for ballots, potentially allowing marks to be added to ballots even if the voter visually verified them first. (This was not evaluated by the security report below).
Allowing voters to use hand marked paper ballots, in conjunction with Risk Limiting Audits, mitigates this entire class of attacks and is a recommended practice among a large group of security experts. Matthew Blaze, testifying before the House Administration Committee earlier this month, made this point clearly:
BMD-based voting systems are controversial, since, by virtue of their design, the correctness of their behavior cannot be effectively audited except by individual voters carefully verifying their machine-printed ballots before they are cast. A maliciously compromised BMD could subtly mismark candidate selections on ballots in a way that might not be noticed by most voters and that could undetectably change election outcomes. Furthermore, if BMDs fail or must be rebooted at a polling place, there may be no alternative method for voters to create marked ballots, making BMDs a potential bottleneck or single point of failure on election day.
As a relatively new technology, BMD-based systems have not yet been widely examined by independent researchers and have been largely absent from practical election security research studies. However, even with relatively little scrutiny, exploitable weaknesses and usability flaws have been found in these systems. This underscores the need for more comprehensive studies and for caution before these systems are purchased by local jurisdictions or widely deployed.
LA County, disregarding this advice, proposes to widely deploy VSAP 2.0 and to force all in-person voters to use the system. It should not do this.
Specific Issues with the Certification Process
The Security Report Has Unresolved Findings
The “Security and Telecommunications Testing of the LA County VSAP 2.0 Voting System” (“Security report”) dated Dec 24, 2019 includes several detailed findings. I will reproduce some of the key findings of concern here, as they are not reproduced fully in the Staff Report and recommendations.
The easily defeated locks and seals on all of the VSAP devices resulted in the system not conforming to CVSS 2.1.1.a, which provides that all systems shall “Provide security access controls that limit or detect access to critical system components to guard against loss of system integrity, availability, confidentiality, and accountability.” It also degrades the ability of the system to meet CVSS 7.3.a. which states, “Any unauthorized physical access shall leave physical evidence that an unauthorized event has taken place.” [Security report, page 17].
Compounding the above, “Booting from a USB drive was not disabled on any of the systems. As such, gaining physical access to the machines allowed access to both the operating and application files for VBL, Tally and FormatOS.“ In addition, “The cryptographic key material used to protect the integrity of elections was not encrypted. All cryptographic keys present were accessible in plaintext.“ and “This allowed secrets used to ensure election integrity to be recovered with only physical access to the system’s storage device.“ [Security report, page 18].
Mitigating this somewhat: “This attack could be conducted by an elections official insider or a vendor insider. A voter would not have sufficient access to the system to successfully complete the prerequisite defeat of physical security without leaving evidence of the attack.“ Even granting this (difficult to evaluate) it does indicate that the system is not secure against insider attacks without additional precautions.
The finding “High Dependency on Root Access” is also concerning: “Root access is required for many regular operations in the VSAP system. These include, but are not limited to, updating cryptographic keys used to protect and verify the integrity of elections and voting information and performing regular system maintenance, including regular system shutdown and startup. This situation invariably leads to poor control of access to the root password which enables subsequent unauthorized access.” [Security report, page 20.] Not conforming with CVSS 2.1.4.f and 7.2.1.b.
The Source Code Review Report Supports The Security Findings
The VSAP Source Code Review Report is relevant here as well, stating on page 91: “The system is airgapped—that is, not connected to the internet or connected to any other system that is connected to the internet. Air gap systems include Ballot Marking Device Manager (BMG) Ballot Marking Device (BMD) VSAP Ballot Layout (VBL) Tally … Note: Unused hardware ports (i.e. USB ports) are protected by port locks and/or tamper evident seals with signaling residue to reveal modification and/or removal. The serialized tamper evident seals are manually logged with an operator signature, seal number, location, date and time. This is to prevent removal of authorized connections when the port is in use and to prevent the insertion of unauthorized connections when the port is not in use. This prevents any infected USB flash drive from crossing any air gap.” However, per the Security report, this does not defend against an insider attack given the seals can be bypassed without detection with an insider’s access.
The Smartmatic response to the concerns involving an infected USB flash drive was (page 91): “A malicious trusted insider would likely attempt other avenues by which to subvert the voting system… At this late time in the Certification campaign, we do not see the ability to remediate the listed software vulnerabilities assuming any could be exploited and would serve as a valuable target.“
On page 93, highly secure key material is left open to all users of the operating system: “The CA certificate and key are stored in tmp and set to 777 file permissions. …” The response indicates that this data is not necessary to the operation of the system and should have been removed as part of installation: “The documentation will be updated to instruct the installer user to delete all data from temp once the install is finished.”. This does not inspire confidence.
The Usability of the System Is Not Yet Proven
While voters clearly enjoyed using the new system, and it may well be an improvement over the old electronic system, that does not mean it produces the correct results.
The “Usability, Accessibility and Privacy Testing” report may also have detected some reliability problems, though it is difficult to tell given the available data. On page 7: “Long periods of silence made it seem as if the voting session was over” and on page 25, “Some voters noted that there were some long delays/pauses in the audio in varying parts of the ballot. This was confusing for the voter. and is also not in conformance with section 3.2.8.b, CVSS standards”. This might indicate a problem that could become worse under load, making the system unusable for some voters and/or confusing for others.
There are other usability problems that are worth calling out because they are key to the claimed superiority of BMDs over hand marked paper ballots: The system does not warn about overvotes.
On page 26, there is the following finding re: CVSS 3.2.2.1 (emphasis added):
CVSS 3.2.2.1: Notification of Effect of Over voting - If the voter attempts to select more than the allowable number of choices within a contest on a VEBD or PCOS, the voting system shall notify the voter of the effect of this action before the ballot is cast and counted.
When a voter attempts to over vote a race the BMD automatically cancels the first choice and accepts the second.
While this is a standard computer list selection UX and might be more familiar than other designs, it is definitely more susceptible to inadvertent touches changing the voter selection by accident than a system which warns and requires confirmation. Without full usability testing it is impossible to say either way, but this clearly appears to violate CVSS 3.2.2.1.
Similarly, on page 26, we find a technical noncompliance that may lead to real election concerns and lawsuits:
3.2.7.a.: No page scrolling - Voting systems shall not require page scrolling by the voter.
Long candidate lists require the voter to scroll on BMDs.
From other reporting it appears that “long” in this context might be “more than 3 candidates” (I am interpreting this finding as equating “scrolling” with the “next” button used to move to the next page of results, where the page size appears to be set to 3 at a time, leading to breaking up even rather small lists of candidates into small sub-sections.) This also appears to violate CVSS 2.3.3.3.f per the Functional Test Report. Finally, this UI also appears to violate CVSS 3.2.5.e.i “The voting system shall visually present a single contest on a single page or column except where the number of choices in a contest makes it impossible.”
The Functional Test Report’s Findings Remain Unaddressed
The “Functional Test Report” reflects a large number of findings, some of which are not noted in the staff report.
On page 12, findings related to CVSS 2.1.1.a, CVSS 2.1.4.f:, CVSS 7.2.1.a.i, CVSS 7.2.1.c “The excessive root access and the ability to boot the system from a USB port give access to the system by unauthorized individuals. Either scenario can result in undetected changes to files and data.”
The Red Team was able to gain access regardless of mitigations:
CVSS 7.5.4.b: “Threat model: failure - Voting systems shall fail open ended vulnerability testing if the manufacturer’s model of the system along with associated use procedures and security controls does not adequately mitigate all significant threats as described in the threat model. The OEVT team may use a threat model that has been amended based on their findings in accordance with 7.5.4.3.c.”
The testers were able to gain access to the system regardless of mitigations.
The Staff Report Does Not Address the Findings Yet Recommends Adoption of a Non Compliant System
Finally, the “Staff Report” summarizes the findings and address them. However, it fails to address all of the findings. I will focus only on some that appear to be unaddressed for no apparent reason.
On Page 15, Table 4a, it lists some but not all of the non-conformance findings I detailed above:
The problems with this table are:
There is no information on how updated “processes and procedures” are going to address physical design issues with tamper evident seals. How?
“The county will apply port protectors” does not appear to be a new mitigation at all, since the ports were theoretically already protected by tamper-evident port blockers per the Source Code Review Report (“Unused hardware ports (i.e. USB ports) are protected by port locks and/or tamper evident seals…”) Thus, this is non responsive and the non conformance is unaddressed.
The finding "Shared/Static Secrets" not conforming w/ CVSS 7.2.4.a is simply ignored.
The finding "High Dependency on Root Access", not conforming w/CVSS 2.1.4.f, CVSS 7.2.1.b, CVSS 7.2.4.a is also simply ignored.
The section “ Accessibility, Usability and Privacy Testing Summary” (page 16) similarly ignores a set of non conformance issues which appear to remain unaddressed anywhere; in particular, the only non conformance issue mentioned is the long period of silence (delay in audio output) with no mitigation or plan to address noted; none of the other non conformance issues listed earlier are noted or addressed.
Three California Elections Code Requirements Are Not Met by VSAP 2.0
Finally the Staff Report lists the sections of the Elections Code and claims that VSAP 2.0 meets all of the requirements. I would take exception to the following:
§19101 (b) (3): The system shall be safe from fraud or manipulation.
The system has unaddressed conformance issues which show it has not yet met this requirement.
§19204.5: The Secretary of State shall not certify or conditionally approve a voting system that cannot facilitate the conduct of a ballot level comparison risk-limiting audit.
§19270 (a): The Secretary of State shall not certify or conditionally approve a direct recording electronic voting system unless the system includes an accessible voter verified paper audit trail.
The system produces ballots that cannot be said to be voter verified and therefore the fundamental requirement for a ballot level comparison risk-limiting audit (per Stark’s definition) cannot yet be met.
Per the above, I do not believe that the Staff Report should be accepted and the system certified for use in elections. With modification, I believe it can be -- if we accept that it should not be used as a universal forced-BMD solution but as an optional mechanism for casting ballots for voters who prefer to use it. That mode does not pose nearly as great a danger and mitigates the non conformance with §19101 (b) (3), §19204.5, and §19270 (a) of the Election Code of California.
