The Annals of Applied Statistics

The interlocking world of surveys and experiments

Stephen E. Fienberg and Judith M. Tanur

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Random sampling and randomized experimentation are inextricably linked. Beginning with their common origins in the work of Fisher and Neyman from the 1920s and the 1930s, one can trace the development of parallel concepts and structures in the two areas (see Fienberg and Tanur [Bull. Int. Stat. Inst. 51 (1985) Art. ID 10.1; Int. Stat. Rev. 55 (1987) 75–96]). One of the more important lessons to be learned from the parallel concepts and structures is that they can profitably be linked and intertwined, with sampling embedded in experiments and formal experimental structures embedded in sampling designs.

In this paper, we trace some of parallels between sampling theory and theory of experimental design. We then explore some of the ways that experimental and sampling structures have been combined in statistical practice and the principles that underlie their combination; we also make some suggestions toward the improvement of practice.

Article information

Ann. Appl. Stat., Volume 12, Number 2 (2018), 1157-1179.

Received: May 2018
First available in Project Euclid: 28 July 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

External validity internal validity interviewer effects randomized experiments sample surveys control experimental design embedding randomization sampling design


Fienberg, Stephen E.; Tanur, Judith M. The interlocking world of surveys and experiments. Ann. Appl. Stat. 12 (2018), no. 2, 1157--1179. doi:10.1214/18-AOAS1184.

Export citation


  • Aronson, E., Brewer, M. and Carlsmith, J. M. (1985). Experimentation in social psychology. In Handbook of Social Psychology, Vol. I, 3rd ed. (G. Lindzey and E. Aronson, eds.). Random House, New York.
  • Bailar, B. A. (1983). Interpenetrating subsamples. In Encyclopedia of Statistical Sciences, Vol. 4 (S. Kotz and N. Johnson, eds.) 197–201. Wiley, New York.
  • Bailar, B. and Biemer, P. (1984). Some methods for evaluating nonsampling error in household censuses and surveys. In W. G. Cochran’s Impact on Statistics (P. S. R. S. Rao and J. Sedransk, eds.) 253–275. Wiley, New York.
  • Bartlett, M. S. (1978). Fisher, R. A. In International Encyclopedia of Statisitcs (W. H. Kruskal and J. M. Tanur, eds.) 352–358. Free Press, New York.
  • Box, J. F. (1978). R. A. Fisher: The Life of a Scientist. Wiley, New York.
  • Campbell, D. P. (1957). Factors relevant to the validity of experiments in social settings. Psychol. Bull. 54 297–312.
  • Campbell, D. P. and Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. In Handbook of Research on Teaching (N. L. Gage, ed.) 171–246. Rand McNally, Chicago, IL.
  • Cochran, W. G. (1968). Errors of measurement in statistics. Technometrics 10 637–666.
  • Cochran, W. G. (1977). Sampling Techniques, 3rd ed. Wiley, New York.
  • Cochran, W. G. and Cox, G. M. (1957). Experimental Designs, 2nd ed. Wiley, New York; Chapman & Hall, London.
  • Cook, T. D. and Campbell, D. P. (1979). Quasi-Experiments: Design and Analysis Issues for Field Settings. Rand McNally, Skokie, IL.
  • Wood (A. J.) Research Corporation (1959). Woodchips, 4, No. 1.
  • Cox, D. R. (1958). Planning of Experiments. Wiley, New York; Chapman & Hall, London.
  • Durbin, J. and Stuart, A. (1951). Differences in response rates of experienced and inexperienced interviewers. J. Roy. Statist. Soc. Ser. A 114 163–206.
  • Fathi, D. C., Schooler, J. and Loftus, E. E. (1984). Moving survey problems into the cognitive psychology laboratory. In Proceedings of the American Statistical Association Section on Survey Research Methods 19–21. Amer. Statist. Assoc., Washington, DC.
  • Federer, W. T. (1977). Sampling, blocking, and model considerations for split plot and split block designs. Biom. J. 19 181–200.
  • Fellegi, I. P. (1964). Response variance and its estimation. J. Amer. Statist. Assoc. 59 1016–1041.
  • Fienberg, S. E. (1971). Randomization and social affairs: The 1970 draft lottery. Science 171 255–261.
  • Fienberg, S. E., Loftus, E. E. and Tanur, J. M. (1985). Cognitive aspects of health survey methodology: An overview. Milbank Mem. Fund Q. 63 547–564.
  • Fienberg, S. E., Singer, B. and Tanur, J. M., (1985). Large scale social experimentation in the U.S.A. In A Celebration of Statistics (A. C. Atkinson and S. E. Fienberg, eds.) 287–326. Springer, New York.
  • Fienberg, S. E. and Tanur, J. M. (1985). A long and honorable tradition: Intertwining concepts and constructs in experimental design and sample surveys. Bull. Int. Stat. Inst. 51 Art. ID 10.1.
  • Fienberg, S. E. and Tanur, J. M. (1987). Experimental and sampling structures: Parallels diverging and meeting. Int. Stat. Rev. 55 75–96.
  • Fienberg, S. E. and Tanur, J. M. (1989). Combining cognitive and statistical approaches to survey design. Science 243 1017–1022.
  • Fisher, R. A. (1925). Statistical Methods for Research Workers. Oliver & Boyd, Edinburgh.
  • Fisher, R. A. (1926). The arrangement of field experiments. J. Minist. Agric. 33 503–513.
  • Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd, Edinburgh.
  • Frankel, L. R. and Stock, J. S. (1942). On the sample survey of unemployment. J. Amer. Statist. Assoc. 37 77–80.
  • Gilbert, J. P., Light, R. J. and Mosteller, F. (1975). Assessing social innovations: An empirical base for policy. In Evaluation and Experiment: Some Critical Issues in Assessing Social Programs (C. A. Bennett and A. A. Lumsdaine, eds.) 39–193. Academic Press, New York.
  • Greenhouse, J. B., Kaizar, E. E., Kelleher, K., Seltman, H. and Gardner, W. (2008). Generalizing from clinical trial data: A case study. The risk of suicidality among pediatric antidepressant users. Stat. Med. 27 1801–1813.
  • Groves, R. M. and Magilavy, L. J. (1986). Measuring and explaining interviewer effects in centralized telephone surveys. Public Opin. Q. 50 251–266.
  • Hansen, M. H., Hurwitz, W. N. and Bershad, M. A. (1961). Measurement errors in censuses and surveys. Bull. Int. Stat. Inst. 38 359–374.
  • Hansen, M. H., Hurwitz, W. N. and Madow, W. G. (1953a). Sample Survey Methods and Theory, Vol. I: Methods and Applications. Wiley, New York; Chapman & Hall, London.
  • Hansen, M. H., Hurwitz, W. N. and Madow, W. G. (1953b). Sample Survey Methods and Theory, Vol. II: Theory. Wiley, New York; Chapman & Hall, London.
  • Hansen, M. H., Hurwitz, W. N., Marks, E. S. and Mauldin, W. P. (1951). Response errors in surveys. J. Amer. Statist. Assoc. 46 147–190.
  • Hartley, H. O. and Rao, J. N. K. (1978). Estimation of nonsampling variance components in sample surveys. In Survey Sampling and Measurement (N. K. Namboodiri, ed.) 35–43. Academic Press, New York.
  • Jabine, T. B. and Rothwell, N. D. (1970). Split-panel tests of census and survey questionnaires. In Proceedings of the American Statistical Association Social Statistics Section 4–13. Amer. Statist. Assoc., Washington, DC.
  • Jabine, T. B., Straf, M., Tanur, J. M. and Torangeau, R., eds. (1984). Cognitive Aspects of Survey Methodology: Building a Bridge Between Disciplines. National Academy Press, Washington, DC.
  • Kaizar, E. E. (2011). Estimating treatment effect via simple cross design synthesis. Stat. Med. 30 2986–3009.
  • Kempthorne, O. (1952). The Design and Analysis of Experiments. Wiley, New York; Chapman & Hall, London.
  • Kempthorne, O. (1955). The randomization theory of experimental inference. J. Amer. Statist. Assoc. 50 946–967.
  • Kish, L. (1962). Studies of interviewer variance for attitudinal variables. J. Amer. Statist. Assoc. 57 92–115.
  • Kish, L. (1965). Survey Sampling. Wiley, New York.
  • Kish, L. and Frankel, M. R. (1974). Inference from complex samples. J. Roy. Statist. Soc. Ser. B 36 1–37.
  • Kruskal, W. and Mosteller, F. (1980). Representative sampling. IV. The history of the concept in statistics, 1895–1939. Int. Stat. Rev. 48 169–195.
  • Lavrakas, P. J., Traugott, M., Kennedy, C., de Leeuw, E., Holbrook, A. and West, B., eds. (2018). Experimental Methods in Survey Research: Techniques That Combine Random Assignment with Random Probability Sampling. Wiley, New York. In press.
  • Loftus, E. E. and Fathi, D. (1985). Retrieving multiple autobiographical memories. Social Cogn. 3 280–295.
  • Mahalanobis, P. C. (1946). Recent experiments in statistical sampling in the Indian Statistical Institute. J. Roy. Statist. Soc. 109 325–378.
  • Mosteller, F. (1978). Nonsampling errors. In International Encyclopedia of Statisitcs, Vol. I (Kruskal, W. H. and Tanur, J. M., eds.) 208–229. Free Press, New York.
  • Neyman, J. (1934). On the two different aspects of the representative method: The method of stratified sampling and the method of purposive selection. J. Roy. Statist. Soc. Ser. A 109 558–606.
  • Panel on Privacy and Confidentiality as Factors in Survey Response, Committee on National Statistics (1979). Privacy and Confidentiality as Factors in Survey Response. National Academy of Sciences, Washington, DC.
  • Platek, R., Rao, J. N. K., Smrndal, C. E. and Singh, M. B. (1986). Small Area Statistics: An International Symposium. Wiley, New York.
  • Reid, C. (1982). Neyman—From Life. Springer, New York.
  • Rosenthal, R. and Rubin, D. B. (1979). Issues in summarizing the first 345 studies of interpersonal expectancy effects. Behav. Brain Sci. 3 410–415.
  • Scheffé, H. (1959). The Analysis of Variance. Wiley, New York; Chapman & Hall, London.
  • Schuman, H., Steeh, C. and Bobo, L. (1985). Racial Attitudes in America: Trends and Interpretations. Harvard Univ. Press, Cambridge, MA.
  • Scott, C. (1961). Research on mail surveys. J. Roy. Statist. Soc. Ser. A 124 143–205.
  • Scott, C. (1973). Experiments on recall error in African household budget surveys. Unpublished paper presented at meeting of International Association of Survey Statisticians, Vienna, Austria (August 1973).
  • Seng, Y. P. (1951). Historical survey of the development of sampling theories and practice. J. Roy. Statist. Soc. Ser. A 114 214–231.
  • Smith, T. M. F. and Sugden, R. A. (1985). Inference and the ignorability of selection for experiments and surveys. Bull. Int. Stat. Inst. 51 10.2-1–10.2-12.
  • Stephan, F. F. (1948). History of the uses of modern sampling procedures. J. Amer. Statist. Assoc. 43 12–39.
  • Stokes, S. L. (1986). Estimation of interviewer effects in complex surveys with application to random digit dialing. In Proceedings of Second Annual Research Conference 21–31. U.S. Bureau of the Census, Washington, DC.
  • Tchuprov, A. A. (1923). On the mathematical expectation of the moments of frequency distributions in the case of correlated observations. Metron 2 646–680.
  • Tepping, B. J., Hurwitz, W. N. and Deming, W. E. (1943). On the efficiency of deep stratification in block sampling. J. Amer. Statist. Assoc. 38 93–100.
  • Tourangeau, R. (1986). Personal communication.
  • Tourangeau, R. and Rasinski, K. A. (1986). Context effects in attitude surveys. Unpublished manuscript.
  • Waksberg, J. and Pearl, R. B. (1965). New methodological research on labor force measurement. In Proceedings of the Social Statistics Section 227–237. Amer. Statist. Assoc., Washington, DC.
  • Wilk, M. B. and Kempthorne, O. (1955). Fixed, mixed, and random models. J. Amer. Statist. Assoc. 50 1144–1167.
  • Wilk, M. B. and Kempthorne, O. (1956). Some aspects of the analysis of factorial experiments in a completely randomized design. Ann. Math. Stat. 27 950–985.
  • Woltman, H. I., Turner, A. G. and Bushery, J. M. (1980). Comparison of three mixed-mode interviewing procedures in the National Crime Survey. J. Amer. Statist. Assoc. 75 534–543.
  • Yates, E. (1981). Sampling Methods for Censuses and Surveys, 4th ed. Macmillan, New York.
  • Yates, F. (1985). Book review of “W.G. Cochran’s Impact on Statistics”, Ed. P.S.R.S. Rao and J. Sedransk. Biometrics 41 591–592.
  • Yates, E. and Cochran, W. G. (1938). The analysis of groups of experiments. J. Agric. Sci. 28 556–580.
  • Zarkovich, S. S. (1956). Note on the history of sampling methods in Russia. J. Roy. Statist. Soc. Ser. A 119 336–338.
  • Zarkovich, S. S. (1962). A supplement to “Note on the history of sampling methods in Russia”. J. Roy. Statist. Soc. Ser. A 125 580–582.