The Annals of Statistics

Optimal and fast detection of spatial clusters with scan statistics

Guenther Walther

Full-text: Open access

Abstract

We consider the detection of multivariate spatial clusters in the Bernoulli model with N locations, where the design distribution has weakly dependent marginals. The locations are scanned with a rectangular window with sides parallel to the axes and with varying sizes and aspect ratios. Multivariate scan statistics pose a statistical problem due to the multiple testing over many scan windows, as well as a computational problem because statistics have to be evaluated on many windows. This paper introduces methodology that leads to both statistically optimal inference and computationally efficient algorithms. The main difference to the traditional calibration of scan statistics is the concept of grouping scan windows according to their sizes, and then applying different critical values to different groups. It is shown that this calibration of the scan statistic results in optimal inference for spatial clusters on both small scales and on large scales, as well as in the case where the cluster lives on one of the marginals. Methodology is introduced that allows for an efficient approximation of the set of all rectangles while still guaranteeing the statistical optimality results described above. It is shown that the resulting scan statistic has a computational complexity that is almost linear in N.

Article information

Source
Ann. Statist., Volume 38, Number 2 (2010), 1010-1033.

Dates
First available in Project Euclid: 19 February 2010

Permanent link to this document
https://projecteuclid.org/euclid.aos/1266586621

Digital Object Identifier
doi:10.1214/09-AOS732

Mathematical Reviews number (MathSciNet)
MR2604703

Zentralblatt MATH identifier
1183.62076

Subjects
Primary: 62G10: Hypothesis testing
Secondary: 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20]

Keywords
Scan statistic Bernoulli model optimal detection multiscale inference fast algorithm concentration inequality

Citation

Walther, Guenther. Optimal and fast detection of spatial clusters with scan statistics. Ann. Statist. 38 (2010), no. 2, 1010--1033. doi:10.1214/09-AOS732. https://projecteuclid.org/euclid.aos/1266586621


Export citation

References

  • Alm, S. E. (1997). On the distribution of the scan statistic of a two-dimensional Poisson process. Adv. in Appl. Probab. 29 1–16.
  • Anderson, N. H. and Titterington, D. M. (1997). Some methods for investigating spatial clustering, with epidemiological applications. J. Roy. Statist. Soc. Ser. A 160 87–105.
  • Chen, J. and Glaz, J. (1996). Two-dimensional discrete scan statistics. Statist. Probab. Lett. 31 59–68.
  • Derbeko, P., El-Yaniv, R. and Meir, R. (2004). Explicit learning curves for transduction and applications to clustering and compression algorithms. J. Artificial Intelligence Res. 22 117–142.
  • Dümbgen, L. and Spokoiny, V. G. (2001). Multiscale testing of qualitative hypotheses. Ann. Statist. 29 124–152.
  • Dümbgen, L. and Walther, G. (2008). Multiscale inference about a density. Ann. Statist. 36 1758–1785.
  • Feller, W. (1968). An Introduction to Probability Theory and Its Applications, 3rd ed. I. Wiley, New York.
  • Glaz, J. and Balakrishnan, N. (eds.) (1999). Scan Statistics and Applications. Birkhäuser, Boston.
  • Glaz, J., Naus, J. and Wallenstein, S. (2001). Scan Statistics. Springer, New York.
  • Hoeffding, W. (1963). Probability inequalities for sums of bounded random. J. Amer. Statist. Assoc. 58 13–30.
  • Kulldorff, M. (1997). A spatial scan statistic. Comm. Statist. Theory Methods 26 1481–1496.
  • Kulldorff, M. (1999). Spatial scan statistics: Models, calculations, and applications. In: Scan Statistics and Applications (J. Glaz and N. Balakrishnan, eds.). Birkhäuser, Boston.
  • Lepski, O. V. and Tsybakov, A. B. (2000). Asymptotically exact nonparametric hypothesis testing in sup-norm and at a fixed point. Probab. Theory Related Fields 117 17–48.
  • Loader, C. R. (1991). Large-deviation approximations to the distribution of scan statistics. Adv. in Appl. Probab. 23 751–771.
  • Naiman, D. Q. and Priebe, C. E. (2001). Computing scan statistic p-values using importance sampling, with applications to genetics and medical image analysis. J. Comput. Graph. Statist. 26 296–328.
  • Naus, J. (1965). Clustering of random points in two dimensions. Biometrika 52 263–267.
  • Neill, D. and Moore, A. (2004a). A fast multi-resolution method for detection of significant spatial disease clusters. Adv. Neural Inf. Process. Syst. 10 651–658.
  • Neill, D. and Moore, A. (2004b). Rapid detection of significant spatial disease clusters. In Proc. Tenth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 256–265. ACM, New York.
  • Roederer, M. and Hardy, R. R. (2001). Frequency difference gating: A multivariate method for identifying subsets that differ between samples. Cytometry 45 56–64.
  • Roederer, M., Moore, W., Treister, A., Hardy, R. R. and Herzenberg, L. (2001). Probability binning comparison: A metric for quantitating multivariate distribution differences. Cytometry 45 47–55.
  • Rohde, A. (2009). Sharp-optimal adjustment for multiple testing in the multivariate two-sample problem. Manuscript. Preprint No. 1356, Weirstrass Institute, Berlin, Germany.
  • Rufibach, K. and Walther, G. (2009). The block criterion for multiscale inference about a density, with applications to other multiscale problems. J. Comput. Graph. Statist. In press.