The Annals of Applied Statistics
- Ann. Appl. Stat.
- Volume 10, Number 1 (2016), 286-304.
Evaluating risk-prediction models using data from electronic health records
The availability of data from electronic health records facilitates the development and evaluation of risk-prediction models, but estimation of prediction accuracy could be limited by outcome misclassification, which can arise if events are not captured. We evaluate the robustness of prediction accuracy summaries, obtained from receiver operating characteristic curves and risk-reclassification methods, if events are not captured (i.e., “false negatives”). We derive estimators for sensitivity and specificity if misclassification is independent of marker values. In simulation studies, we quantify the potential for bias in prediction accuracy summaries if misclassification depends on marker values. We compare the accuracy of alternative prognostic models for 30-day all-cause hospital readmission among 4548 patients discharged from the University of Pennsylvania Health System with a primary diagnosis of heart failure. Simulation studies indicate that if misclassification depends on marker values, then the estimated accuracy improvement is also biased, but the direction of the bias depends on the direction of the association between markers and the probability of misclassification. In our application, 29% of the 1143 readmitted patients were readmitted to a hospital elsewhere in Pennsylvania, which reduced prediction accuracy. Outcome misclassification can result in erroneous conclusions regarding the accuracy of risk-prediction models.
Ann. Appl. Stat., Volume 10, Number 1 (2016), 286-304.
Received: December 2014
Revised: July 2015
First available in Project Euclid: 25 March 2016
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Wang, Le; Shaw, Pamela A.; Mathelier, Hansie M.; Kimmel, Stephen E.; French, Benjamin. Evaluating risk-prediction models using data from electronic health records. Ann. Appl. Stat. 10 (2016), no. 1, 286--304. doi:10.1214/15-AOAS891. https://projecteuclid.org/euclid.aoas/1458909917
- Supplement to “Evaluating risk-prediction models using data from electronic health records”. The supplement provides additional simulation results by summarizing the distribution of percent bias across simulated datasets.