Until recently, the survey mode of the household panel study Understanding Society was mainly face-to-face interview, but it has now adopted a mixed-mode design where individuals can self-complete the questionnaire via the web. As mode is known to affect survey data, a randomized mixed-mode experiment was implemented during the first year of the two-year Wave 8 fieldwork period to assess the impact of this change. The experiment involved a sequential design that permits the identification of mode effects in the presence of nonignorable nonrandom mode selection. While previous studies have used instrumental variables regression to estimate the effects of mode on the means of the survey variables, we describe a more general methodology based on novel structural moment models that characterizes the overall effect of mode on a survey by its effects on the moments of the survey variables’ joint distribution. We adapt our estimation procedure to account for nonresponse and complex sampling designs and to include suitable auxiliary data to improve inference and relax key assumptions. Finally, we demonstrate how to estimate the effects of mode on the parameter estimates of generalized linear models and other exponential family models when both outcomes and predictors are subject to mode effects. This methodology is used to investigate the impact of the move to web mode on Wave 8 of Understanding Society.
Both authors gratefully acknowledge funding by the U.K. Economic & Social Research Council (ESRC) through grants ES/N00812X/1 (Understanding Society: The U.K. Household Longitudinal Study, Waves 9-11) and ES/S012486/1 (MiSoC: The ESRC Research Centre on Micro-Social Change). Understanding Society is an initiative funded by the ESRC and various government departments with scientific leadership by the Institute for Social and Economic Research, University of Essex and survey delivery by NatCen Social Research and Kantar Public. The research data are distributed by the U.K. Data Service.
The authors would like to thank Peter Lynn and Tarek Al Baghal for helpful discussions about the design of UKHLS Wave 8, Fiona Steele and members of the Understanding Society Scientific Group for their comments and criticisms and two anonymous referees for helpful comments which greatly improved the presentation of the manuscript.
"Estimating mode effects from a sequential mixed-mode experiment using structural moment models." Ann. Appl. Stat. 16 (3) 1563 - 1585, September 2022. https://doi.org/10.1214/21-AOAS1557