Statistical Science

Statistical and Clinical Aspects of Hospital Outcomes Profiling

Sharon-Lise T. Normand and David M. Shahian

Full-text: Open access


Hospital profiling involves a comparison of a health care provider’s structure, processes of care, or outcomes to a standard, often in the form of a report card. Given the ubiquity of report cards and similar consumer ratings in contemporary American culture, it is notable that these are a relatively recent phenomenon in health care. Prior to the 1986 release of Medicare hospital outcome data, little such information was publicly available. We review the historical evolution of hospital profiling with special emphasis on outcomes; present a detailed history of cardiac surgery report cards, the paradigm for modern provider profiling; discuss the potential unintended negative consequences of public report cards; and describe various statistical methodologies for quantifying the relative performance of cardiac surgery programs. Outstanding statistical issues are also described.

Article information

Statist. Sci., Volume 22, Number 2 (2007), 206-226.

First available in Project Euclid: 27 September 2007

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Mortality report cards quality of care variations risk adjustment hierarchical models profiling evidence-based medicine selection bias


Normand, Sharon-Lise T.; Shahian, David M. Statistical and Clinical Aspects of Hospital Outcomes Profiling. Statist. Sci. 22 (2007), no. 2, 206--226. doi:10.1214/088342307000000096.

Export citation


  • Aitkin, M. and Longford, N. T. (1986). Statistical modelling issues in school effectiveness studies. J. Roy. Statist. Soc. Ser. A 149 1--43.
  • Austin, P. C., Tu, J. V., Alter, D. A. and Naylor, C. D. (2005). The impact of under coding of cardiac severity and comorbid diseases on the accuracy of hospital report cards. Medical Care 43 801--809.
  • Berwick, D. M. and Wald, D. L. (1990). Hospital leaders' opinions of the HCFA mortality data. J. Amer. Medical Assoc. 263 247--249.
  • Birkmeyer, J. D., Dimick, J. B. and Birkmeyer, N. J. (2004). Measuring the quality of surgical care: Structure, process or outcomes? J. Amer. College of Surgeons 198 626--632.
  • Blendon, R. J., Brodie, M., Benson, J. M., Altman, D. E., Levitt, L., Hoff, T. and Hugick, L. (1998). Understanding the managed care backlash. Health Affairs 17(4) 80--94.
  • Bradley, E. H., Herrin, J., Elbel, B., McNamara, R. L., Magid, D. J., Nallamothu, B. K., Wang, Y., Normand, S.-L., Spertus, J. A. and Krumholz, H. M. (2006). Hospital quality for acute myocardial infarction: Correlation among process measures and relationship with short-term mortality. J. Amer. Medical Assoc. 296 72--78.
  • Bradlow, E. T., Wainer, H. and Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika 64 153--168.
  • Burack, J. H., Impellizzeri, P., Homel, P. and Cunningham, J. N., Jr. (1999). Public reporting of surgical mortality: A survey of New York State cardiothoracic surgeons. Ann. Thoracic Surgery 68 1195--1200.
  • Burton, T. M. (1999). Operation that rated hospitals was success, but the patience died. The Wall Street J. Aug. 23, A1, A8.
  • Chassin, M. R., Brook, R. H., Park, R. E., Keesey, J. et al. (1986). Variations in the use of medical and surgical services by the Medicare population. New England J. Medicine 314 285--290.
  • Chassin, M. R., Hannan, E. L. and DeBuono, B. A. (1996). Benefits and hazards of reporting medical outcomes publicly. New England J. Medicine 334 394--398.
  • Christiansen, C. and Morris, C. N. (1997). Hierarchical Poisson regression modeling. J. Amer. Statist. Assoc. 92 618--632.
  • Cohen, I. B. (1984). Florence Nightingale. Scientific American 250 128--136.
  • Daley, J., Forbes, M. G., Young, G. J., Charns, M. P., Gibbs, J. O., Hur, K., Henderson, W. and Khuri, S. F. (1997). Validating risk-adjusted surgical outcomes: Site visit assessment of process and structure. J. Amer. College of Surgeons 185 341--351.
  • Daniels, M. J. and Normand, S.-L. T. (2006). Longitudinal profiling of health care units based on continuous and discrete patient outcomes. Biostatistics 7 1--15.
  • Diehr, P., Cain, K., Connell, F. and Volinn, E. (1990). What is too much variation? The null hypothesis in small-area analysis. Health Services Research 24 741--771.
  • Dimick, J. B., Welch, H. G. and Birkmeyer, J. D. (2004). Surgical mortality as an indicator of hospital quality: The problem with small sample size. J. Amer. Medical Assoc. 292 847--851.
  • Donabedian, A. (1980). Explorations in Quality Assessment and Monitoring 1. The Definition of Quality and Approaches to its Assessment. Health Administration Press, Ann Arbor, MI.
  • Dranove, D., Kessler, D., McClellan, M. and Satterthwaite, M. (2003). Is more information better? The effects of ``report cards'' on health care providers. J. Political Economy 111 555--588.
  • Draper, D. and Gittoes, M. (2004). Statistical analysis of performance indicators in UK higher education. J. Roy. Statist. Soc. Ser. A 167 449--474.
  • Dunson, D. B. (2000). Bayesian latent variable models for clustered mixed outcomes. J. R. Stat. Soc. Ser. B Stat. Methodol. 62 355--366.
  • Epstein, A. M. (1998). Rolling down the runway. The challenges ahead for quality report cards. J. Amer. Medical Assoc. 279 1691--1696.
  • Erickson, L. C., Torchiana, D. F., Schneider, E. C., Newburger, J. W. and Hannan, E. L. (2000). The relationship between managed care insurance and use of lower-mortality hospitals for CABG surgery. J. Amer. Medical Assoc. 283 1976--1982.
  • Finlayson, S. R., Birkmeyer, J. D., Tosteson, A. N. and Nease, R. F., Jr. (1999). Patient preferences for location of care: Implications for regionalization. Medical Care 37 204--209.
  • Fronstin, P. and Collins, S. R. (2005). Early experience with high-deductible and consumer-driven health plans: Findings from the EBRI/Commonwealth Fund Consumerism in Health Care Survey. The Commonwealth Fund.
  • Galvin, R. and Milstein, A. (2002). Large employers' new strategies in health care. New England J. Medicine 347 939--942.
  • Gatsonis, C. A. (1998). Profiling providers of medical care. Encyclopedia of Biostatistics 3 3536--3536. Wiley, New York.
  • Gelman, A. (2002). Prior distribution. Encyclopedia of Environmetrics 3 1634--1637. Wiley, Chichester.
  • Gelman, A. (2006). Prior distributions for variance parameters in hierarchical models. Bayesian Anal. 1 515--533.
  • Gelman, A., Carlin, J. B., Stern, H. S. and Rubin, D. B. (2004). Bayesian Data Analysis, 2nd ed. Chapman and Hall/CRC, Boca Raton, FL.
  • Ghali, W. A., Ash, A. S., Hall, R. E. and Moskowitz, M. A. (1997). Statewide quality improvement initiatives and mortality after cardiac surgery. J. Amer. Medical Assoc. 277 379--382.
  • Goldstein, H. and Spiegelhalter, D. J. (1996). League tables and their limitations: Statistical issues in comparisons of institutional performance (with discussion). J. Roy. Statist. Soc. Ser. A 159 385--443.
  • Green, J. and Wintfeld, N. (1995). Report cards on cardiac surgeons---Assessing New York State's approach. New England J. Medicine 332 1229--1233.
  • Hannan, E. L., Kilburn, H., Jr., Lindsey, M. L. and Lewis, R. (1992). Clinical versus administrative data bases for CABG surgery. Does it matter? Medical Care 30 892--907.
  • Hannan, E. L., Kilburn, H., Jr., O'Donnell, J. F., Lukacik, G. and Shields, E. P. (1990). Adult open heart surgery in New York State. An analysis of risk factors and hospital mortality rates. J. Amer. Medical Assoc. 264 2768--2774.
  • Hannan, E. L., Racz, M. J., Jollis, J. G. and Peterson, E. D. (1997a). Using Medicare claims data to assess provider quality for CABG surgery: Does it work well enough? Health Services Research 31 659--678.
  • Hannan, E. L., Siu, A. L., Kumar, D., Racz, M., Pryor, D. B. and Chassin, M. R. (1997b). Assessment of coronary artery bypass graft surgery performance in New York. Is there a bias against taking high-risk patients? Medical Care 35 49--56.
  • Hannan, E. L., Stone, C. C., Biddle, T. L. and DeBuono, B. A. (1997c). Public release of cardiac surgery outcomes data in New York: What do New York state cardiologists think of it? Amer. Heart J. 134 55--61.
  • Harrell, F. (2001). Regression Modeling Strategies. Springer, New York.
  • Horton, N. J. and Fitzmaurice, G. M. (2004). Tutorial in biostatistics: Regression analysis of multiple source and multiple informant data from complex survey samples. Stat. Med. 23 2911--2933.
  • Huang, I.-C., Frangakis, C., Dominici, F., Diette, G. B. and Wu, A. W. (2005). Application of a propensity score approach for risk adjustment in profiling multiple physician groups on asthma care. Health Services Research 40 253--278.
  • Iezzoni, L. I., ed. (1997a). Risk Adjustment for Measuring Healthcare Outcomes, 2nd ed. Health Administration Press, Chicago.
  • Iezzoni, L. I. (1997b). The risks of risk adjustment. J. Amer. Medical Assoc. 278 1600--1607.
  • Iezzoni, L. I., ed. (2003). Risk Adjustment for Measuring Health Care Outcomes, 3rd ed. Health Administration Press, Chicago.
  • Institute of Medicine (2000). To Err is Human: Building a Safer Health System. National Academies Press, Washington.
  • Institute of Medicine (2001). Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press, Washington.
  • Institute of Medicine (2003). Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care. National Academies Press, Washington.
  • Jaeger, R. M. (1989). Certification of student competence. In Educational Measurement, 3rd ed. (R. L. Linn, ed.) 485--514. MacMillan, New York.
  • Jha, A. K. and Epstein, A. E. (2006). The predictive accuracy of the New York State Coronary Bypass Surgery Report-Card System. Health Affairs 25(3) 844--855.
  • Jha, A. K., Li, Z., Orav, E. J. and Epstein, A. M. (2005). Care in US hospitals---the Hospital Quality Alliance program. New England J. Medicine 353 265--274.
  • Jones, R. H. (1989). In search of the optimal surgical mortality. Circulation 79 I132--I136.
  • Jones, R. H., Hannan, E. L., Hammermeister, K. E., Delong, E. R., O'Connor, G. T., Luepker, R. V. et al. (1996). Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The Working Group Panel on the Cooperative CABG Database Project. J. Amer. College of Cardiology 28 1478--1487.
  • Kassirer, J. P. (1994). The use and abuse of practice profiles. New England J. Medicine 330 634--636.
  • Krumholz, H. M., Brindis, R. G., Brush, J. E., Cohen, D. J., Epstein, A. J. et al. (2006a). Standards for statistical models used for reporting of health outcomes: An American Heart Association scientific statement from the Quality of Care and Outcomes Research Interdisciplinary Writing Group. Circulation 113 456--462.
  • Krumholz, H. M., Normand, S.-L., Spertus, J. A., Shahian, D. M. and Bradley, E. H. (2007). Measuring performance for heart attacks and heart failure: The case for outcomes measurement. Health Affairs 26(1) 75--85.
  • Krumholz, H. M., Wang, Y., Mattera, J. A., Wang, Y., Han, L. F., Ingber, M. J., Roman, S. and Normand, S.-L. (2006b). An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with heart failure. Circulation 113 1693--1701.
  • Krumholz, H. M., Wang, Y., Mattera, J. A., Wang, Y., Han, L. F., Ingber, M. J., Roman, S. and Normand, S.-L. (2006c). An administrative claims model suitable for profiling hospital performance based on 30-day mortality rates among patients with an acute myocardial infarction. Circulation 113 1683--1692.
  • Landon, B. E., Normand, S.-L., Blumenthal, D. and Daley, J. (2003). Physician clinical performance assessment: Prospects and barriers. J. Amer. Medical Assoc. 290 1183--1189.
  • Landrum, M. B., Bronskill, S. E. and Normand, S.-L. (2000). Analytic methods for constructing cross-sectional profiles of health care providers. Health Services and Outcomes Research Methodology 1 23--47.
  • Landrum, M. B., Normand, S.-L. and Rosenheck, R. A. (2003). Selection of related multivariate means: Monitoring psychiatric care in the Department of Veterans Affairs. J. Amer. Statist. Assoc. 98 7--16.
  • Lee, S.-Y. and Shi, J.-Q. (2001). Maximum likelihood estimation of two-level latent variable models with mixed continuous and polytomous data. Biometrics 57 787--794.
  • Mack, M. J., Herbert, M., Prince, S., Dewey, T. M., Magee, M. J. and Edgerton, J. R. (2005). Does reporting of coronary artery bypass grafting from administrative databases accurately reflect actual clinical outcomes? J. Thoracic and Cardiovascular Surgery 129 1309--1317.
  • Mallon, W. J. (2000). Ernest Amory Codman: The End Result of a Life in Medicine. Saunders, Philadelphia.
  • McPherson, K., Wennberg, J., Hovind, O. B. and Clifford, P. (1982). Small-area variations in the use of common surgical procedures: An international comparison of New England, England and Norway. New England J. Medicine 307 1310--1314.
  • Milstein, A., Galvin, R., Delbanco, S. F., Salber, P. and Buck, C. R., Jr. (2000). Improving the safety of health care: The Leapfrog initiative. Effective Clinical Practice 3 313--316. [Correction 4 (2001) 94.]
  • Mood, A. M., Graybill, F. A. and Boes, D. C. (1973). Introduction to the Theory of Statistics, 3rd ed. Appendix A. McGraw-Hill, New York.
  • Nolan, T. and Berwick, B. M. (2006). All-or-none measurement raises the bar on performance. J. Amer. Medical Assoc. 295 1168--1170.
  • Normand, S.-L. T. (2005). Quality of care. Encyclopedia of Biostatistics, 2nd ed. 6 4348--4352. Wiley, Chichester.
  • Normand, S.-L. T., Glickman, M. E. and Gatsonis, C. A. (1997). Statistical methods for profiling providers of medical care: Issues and applications. J. Amer. Statist. Assoc. 92 803--814.
  • Normand, S.-L. T., Wolf, R. E., Ayanian, J. Z. and McNeil, B. J. (2007). Assessing the accuracy of hospital clinical performance measures. Medical Decision Making 27 9--20.
  • O'Connor, G. T., Plume, S. K., Olmstead, E. M., Coffin, L. H., Morton, J. R., Maloney, C. T. et al. (1991). A regional prospective study of in-hospital mortality associated with coronary artery bypass grafting. The Northern New England Cardiovascular Disease Study Group. J. Amer. Medical Assoc. 266 803--809.
  • O'Connor, G. T., Plume, S. K., Olmstead, E. M., Morton, J. R., Maloney, C. T., Nugent, C. T. et al. (1996). A regional intervention to improve the hospital mortality associated with coronary artery bypass graft surgery. The Northern New England Cardiovascular Disease Study Group. J. Amer. Medical Assoc. 275 841--846.
  • Omoigui, N. A., Miller, D. P., Brown, K. J., Annan, K., Cosgrove, D., III, Lytle, B. et al. (1996). Outmigration for coronary bypass surgery in an era of public dissemination of clinical outcomes. Circulation 93 27--33.
  • Osswald, B. R., Blackstone, E. H., Tochtermann, U., Thomas, G., Vahl, C. F. and Hagl, S. (1999). The meaning of early mortality after CABG. European J. Cardio-Thoracic Surgery 15 401--407.
  • Parsonnet, V. (1995). Risk stratification in cardiac surgery: Is it worthwhile? J. Cardiac Surgery 10 690--698.
  • Passaro, E., Jr. and Organ, C. H., Jr. (1999). Ernest A. Codman: The improper Bostonian. Bull. Amer. College of Surgeons 84 16--22.
  • Peterson, E. D., Coombs, L. P., DeLong, E. R., Haan, C. K. and Ferguson, T. B. (2004). Procedural volume as a marker of quality for CABG surgery. J. Amer. Medical Assoc. 291 195--201.
  • Peterson, E. D., DeLong, E. R., Jollis, J. G., Muhlbaier, L. H. and Mark, D. B. (1998). The effects of New York's bypass surgery provider profiling on access to care and patient outcomes in the elderly. J. Amer. College of Cardiology 32 993--999.
  • Peterson, E. D., Roe, M. T., Mulgund, J., DeLong, E. R., Lytle, B. L., Brindis, R. G., Smith, S. C., Pollack, C. V., Newby, L. K., Harrington, R. A., Gibler, W. B. and Ohman, E. M. (2006). Association between hospital process performance and outcomes among patients with acute coronary syndromes. J. Amer. Medical Assoc. 295 1912--1920.
  • Rasch, G. (1960). Probabilistic Models for Some Intelligence and Attainment Tests. Danmarks Paedagogiske Institute, Copenhagen.
  • Rathore, S. S., Wang, Y., Radford, M. J., Ordin, D. L. and Krumholz, H. M. (2003). Quality of care of Medicare beneficiaries with acute myocardial infarction: Who is included in quality improvement measurement? J. Amer. Geriatric Soc. 51 466--475.
  • Sammel, M. D., Ryan, L. M. and Legler, J. M. (1997). Latent variable models for mixed discrete and continuous outcomes. J. Roy. Statist. Soc. Ser. B 59 667--678.
  • Schneider, E. C. and Epstein, A. M. (1996). Influence of cardiac-surgery performance reports on referral practices and access to care---a survey of cardiovascular specialists. New England J. Medicine 335 251--256.
  • Schneider, E. C. and Epstein, A. M. (1998). Use of public performance reports: A survey of patients undergoing cardiac surgery. J. Amer. Medical Assoc. 279 1638--1642.
  • Scott, S. L. and Ip, E. H. (2002). Empirical Bayes and item-clustering effects in a latent variable hierarchical model: A case study from the National Assessment of Education Progress. J. Amer. Statist. Assoc. 97 409--419.
  • Shahian, D. M., Blackstone, E. H., Edwards, F. H., Grover, F. L., Grunkemeier, G. L., Naftel, D. C. et al. (2004). Cardiac surgery risk models: A position article. Ann. Thoracic Surgery 78 1868--1877.
  • Shahian, D. M. and Normand, S.-L. (2003). The volume-outcome relationship: From Luft to Leapfrog. Ann. Thoracic Surgery 75 1048--1058.
  • Shahian, D. M., Normand, S.-L., Torchiana, D. F., Lewis, S. M., Pastore, J. O., Kuntz, R. E. and Dreyer, P. I. (2001). Cardiac surgery report cards: Comprehensive review and statistical critique. Ann. Thoracic Surgery 72 2155--2168.
  • Shahian, D. M., Silverstein, T., Lovett, A., Wolf, R. E. and Norman, S.-L. (2007). Comparison of clinical and administrative data sources for hospital coronary artery bypass graft surgery report cards. Circulation 115 1518--1527.
  • Shahian, D. M., Torchiana, D. F. and Normand, S.-L. (2005). Implementation of a cardiac surgery report card: Lessons from the Massachusetts experience. Ann. Thoracic Surgery 80 1146--1150.
  • Shahian, D. M., Yip, W., Westcott, G. and Jacobson, J. (2000). Selection of a cardiac surgery provider in the managed care era. J. Thoracic and Cardiovascular Surgery 120 978--989.
  • Shroyer, A. L., Marshall, G., Warner, B. A., Johnson, R. R., Guo, W., Grover, F. L. and Hammermeister, K. E. (1996). No continuous relationship between Veterans Affairs hospital coronary artery bypass grafting surgical volume and operative mortality. Ann. Thoracic Surgery 61 17--20.
  • Skrondal, A. and Rabe-Hesketch, S. (2004). Generalized Latent Variable Modeling: Multilevel, Longitudinal and Structural Equation Models. Chapman and Hall/CRC, London.
  • Spiegelhalter, D. J. (1999). Surgical audit: Statistical lessons from Nightingale and Codman. J. Roy. Statist. Soc. Ser. A 162 45--58.
  • Spiegelhalter, D. J., Abrams, K. R. and Myles, J. P. (2004). Bayesian Approaches to Clinical Trials and Health Care Evaluation. Wiley, Chichester.
  • Steinbrook, R. (2004). The cost of admission---tiered copayments for hospital use. New England J. Medicine 350 2539--2542.
  • Stinnett, S. et al. (1990). Women in statistics: Sesquicentennial activities. Amer. Statist. 44 74--80.
  • Thomas, N., Longford, N. and Rolph, J. (1994). Empirical Bayes methods for estimating hospital-specific mortality rates. Stat. Med. 13 889--903.
  • Tu, J. V., Sykora, K. and Naylor, C. D. (1997). Assessing the outcomes of coronary artery bypass graft surgery: How many risk factors are enough? Steering Committee of the Cardiac Care Network of Ontario. J. Amer. College of Cardiology 30 1317--1323.
  • Vladeck, B. C., Goodwin, E. J., Myers, L. P. and Sinisi, M. (1988). Consumers and hospital use: The HCFA ```death list.'' Health Affairs 7(1) 122--125.
  • Wennberg, J. and Gittelsohn, A. (1973). Small area variations in health care delivery. Science 182 1102--1108.
  • Werner, R. M. and Asch, D. A. (2005). The unintended consequences of publicly reporting quality information. J. Amer. Medical Assoc. 293 1239--1244.
  • Werner, R. M. and Bradlow, E. T. (2006). Relationship between Medicare's hospital compare performance measures and mortality rates. J. Amer. Medical Assoc. 296 2694--2702.
  • Williams, S. V., Nash, D. B. and Goldfarb, N. (1991). Differences in mortality from coronary artery bypass graft surgery at five teaching hospitals. J. Amer. Medical Assoc. 266 810--815.