The Annals of Applied Statistics

Conservative statistical post-election audits

Philip B. Stark

Full-text: Open access


There are many sources of error in counting votes: the apparent winner might not be the rightful winner. Hand tallies of the votes in a random sample of precincts can be used to test the hypothesis that a full manual recount would find a different outcome. This paper develops a conservative sequential test based on the vote-counting errors found in a hand tally of a simple or stratified random sample of precincts. The procedure includes a natural escalation: If the hypothesis that the apparent outcome is incorrect is not rejected at stage s, more precincts are audited. Eventually, either the hypothesis is rejected—and the apparent outcome is confirmed—or all precincts have been audited and the true outcome is known. The test uses a priori bounds on the overstatement of the margin that could result from error in each precinct. Such bounds can be derived from the reported counts in each precinct and upper bounds on the number of votes cast in each precinct. The test allows errors in different precincts to be treated differently to reflect voting technology or precinct sizes. It is not optimal, but it is conservative: the chance of erroneously confirming the outcome of a contest if a full manual recount would show a different outcome is no larger than the nominal significance level. The approach also gives a conservative P-value for the hypothesis that a full manual recount would find a different outcome, given the errors found in a fixed size sample. This is illustrated with two contests from November, 2006: the U.S. Senate race in Minnesota and a school board race for the Sausalito Marin City School District in California, a small contest in which voters could vote for up to three candidates.

Article information

Ann. Appl. Stat., Volume 2, Number 2 (2008), 550-581.

First available in Project Euclid: 3 July 2008

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Hypothesis test sequential test auditing elections


Stark, Philip B. Conservative statistical post-election audits. Ann. Appl. Stat. 2 (2008), no. 2, 550--581. doi:10.1214/08-AOAS161.

Export citation


  • Aslam, J. A., Popa, R. A. and Rivest, R. L. (2007). On auditing elections when precincts have different sizes. Available at,
  • Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing., J. Roy. Statist. Soc. Ser. B 57 289–300.
  • Bjornlund, E. C. (2004)., Beyond Free and Fair: Monitoring Elections and Building Democracy. Woodrow Wilson Center Press, Washington, DC.
  • Dopp, K. and Stenger, F. (2006). The election integrity audit. Available at,
  • Elections: Federal efforts to improve security and reliability of electronic voting systems are under way, but key activities need to be completed (2005). Technical Report GAO-05-956, U.S. Government Accountability Office, Washington, DC.
  • Elections: The nation’s evolving election system as reflected in the November 2004 general election (2006). Technical Report GAO-06-450, U.S. Government Accountability Office, Washington, DC.
  • Estok, M., Nevitte, N. and Cowan, G. (2002)., The Quick Count and Election Observation. National Democratic Institute for International Affairs, Washington, DC.
  • Halvorson, M. and Wolff, L. (2007). Report and analysis of the 2006 post-election audit of Minnesotas voting systems. Available at,
  • Hite, R. C. (2007). Elections: All levels of government are needed to address electronic voting system challenges. Technical Report GAO-07-714T, U.S. Government Accountability Office, Washington, DC.
  • Hoeffding, W. (1956). On the distribution of the number of successes in independent trials., Ann. Math. Statist. 27 713–721.
  • Jefferson, D., Alexander, K., Ginnold, E., Lehmkuhl, A., Midstokke, K. and Stark, P. B. (2007). Post election audit standards report–evaluation of audit sampling models and options for strengthening Californias manual count. Available at,
  • McCarthy, J., Stanislevic, H., Lindeman, M., Ash, A., Addona, V. and Batcher, M. (2008). Percentage based vs. statistical-power-based vote tabulation auditing., The American Statistician 62 11–16.
  • Norden, L., Burstein, A., Hall, J. L. and Chen, M. (2007). Post-election audits: Restoring trust in elections. Technical report, Brennan Center for Justice, New York University and Samuelson Law, Technology & Public Policy Clinic at University of California, Berkeley School of Law (Boalt Hall), New, York.
  • National Association of Secretaries of State (2007). Post election audit procedures by state. Available at,
  • Panel on Nonstandard Mixtures of Distributions (1989). Statistical models and analysis in auditing: Panel on nonstandard mixtures of distributions., Statist. Sci. 4 2–33.
  • Rivest, R. L. (2006). On estimating the size of a statistical audit. Available at,
  • Saltman, R. G. (1975). Effective use of computing technology in vote-tallying. Technical Report NBSIR 75-687, National Bureau of Standards, Washington, DC.
  • Siegmund, D. (1985)., Sequential Analysis: Tests and Confidence Intervals. Springer, New York.
  • Stanislevic, H. (2006). Random auditing of e-voting systems: How much is enough? Available at,
  • Stark, P. B. (2008a). Election audits by sampling with probability proportional to an error bound: Dealing with discrepancies. Available at,
  • Stark, P. B. (2008b). A sharper discrepancy measure for post-election audits., Ann. Appl. Statist. To appear.
  • Verified Voting Foundation (2007). Manual audit requirements. Available at,
  • Wald, A. (2004)., Sequential Analysis. Dover, Mineola, NY.