Bayesian hidden Markov model for natural history of colorectal cancer: Handling misclassified observations, varying observation schemes and unobserved data

Abstract

Statistical modelling of individual-level event history data arising from varying observation schemes is a challenging problem, particularly due to unobserved and possibly misclassified individual states. Commonly used approaches rely on the hidden Markov models (HMM) to incorporate true underlying states. Each approach needs to account for the underlying data generating process and related external information and requires assumptions for estimation. This article develops a Bayesian HMM for natural history of colorectal cancer (CRC), combining data on latent disease states from randomised screening study and on observed clinical cancers from the population-based cancer registry. With our modelling approach and study design, we are able to provide estimates for latent state occupancy probabilities not only for screening-attenders but also for the control group and those who never attended screening—despite data on latent states only existing for the attenders. We use simulation-based calibration to ensure that posterior distributions can be reliably estimated despite the challenges brought in by the sampling scheme. We apply Bayesian computation to obtain posterior estimates of the quantities of interest. Two algorithms, Hamiltonian Monte Carlo (HMC) and Automatic Differentiation Variational Inference (ADVI), are applied and compared, first by using simulated data and then with a real data set. The modelling workflow can be applied for different cancer screening programmes and datasets which typically have similar challenges.