Abstract
BACKGROUND: The pneumonia 30-d readmission rate has been endorsed by the National Quality Forum as a quality metric. Hospital readmissions can potentially be lowered by improving in-hospital care, transitions of care, and post-discharge disease management programs. The purpose of this study was to create an accurate prediction model for determining the risk of 30-d readmission at the point of discharge.
METHODS: The model was created using a data set of 1,295 hospitalizations at the Cleveland Clinic Main Campus with pneumonia over 3 y. Candidate variables were limited to structured variables available in the electronic health record. The final model was compared with the Centers for Medicare and Medicaid Services (CMS) model among subjects 65 y of age and older (n = 628) and was externally validated.
RESULTS: Three hundred thirty subjects (25%) were readmitted within 30 d. The final model contained 13 variables and had a bias-corrected C statistic of 0.74 (95% CI 0.71–0.77). Number of admissions in the prior 6 months, opioid prescription, serum albumin during the first 24 h, international normalized ratio and blood urea nitrogen during the last 24 h were the predictor variables with the greatest weight in the model. In terms of discriminative performance, the Cleveland Clinic model outperformed the CMS model on the validation cohort (C statistic 0.69 vs 0.60, P = .042).
CONCLUSIONS: The proposed risk prediction model performed better than the CMS model. Accurate readmission risk prediction at the point of discharge is feasible and can potentially be used to focus post-acute care interventions in a high-risk group of patients.
Introduction
An unplanned hospital readmission within 30 d of discharge after an acute hospitalization has been endorsed as an important quality indicator for public health reporting purposes.1 Previous studies indicate that many readmissions are associated with suboptimal transitions of care and may be preventable.2,3 The Medicare Payment Advisory Commission has estimated that readmissions cost $15 billion annually.4 The Centers for Medicare and Medicaid Services (CMS) launched the Hospital Readmissions Reduction Program, which requires CMS to reduce payment to participating hospitals with readmission rates in excess of the national average. The Hospital Readmissions Reduction Program has focused on diseases with the highest expenditures and readmission rates and for which adherence to management guidelines improves outcomes. One target condition, pneumonia, had a 30-d readmission rate of 15.7% in 20105 and cost $10.6 billion for Medicare in 2011.6
There are a number of ways hospitals might reduce readmissions. Improvements in in-patient care, optimizing transitions, and appropriate discharge disposition are all associated with reduced readmission rates.7,8 However, comprehensive post-acute discharge programs are costly to implement for all patients. Identifying patients at the highest risk for readmission would allow for more efficient allocation of limited resources at the time of discharge. Amarasingham et al9 leveraged an electronic health record-based risk stratification algorithm to allocate limited resources to successfully prevent readmissions among patients with congestive heart failure in real time. The claims-based model used by the CMS to calculate hospital specific, risk-standardized pneumonia readmission rates was created to compare hospitals, not to identify high-risk patients.10 The CMS model does not include important clinical details, such as vital signs, medications, laboratory results, and in-patient treatments, such as oxygen therapy. Other studies have found that clinical predictor variables add to the accurate prediction of readmission risk.11,12 Finally, the CMS model was created using data from a wide variety of clinical institutions and has poor discrimination (C statistic 0.63). In a specific hospital, it may not perform as well as a model created using local data.
The purpose of this study was to create a practical, local model for predicting the risk of 30-d all-cause readmission following admission for pneumonia. The model was not intended to determine causal inference but rather to create the most accurate risk assessment based on structured variables available in the electronic health record and therefore ready for use at the point of care.
QUICK LOOK
Current knowledge
Pneumonia readmissions represent a significant burden to health-care expenditure and patient morbidity. Assessing risk of readmission during in-patient care may allow focusing intensive interventions to a select group of individuals who may benefit the most while optimizing use of limited resources.
What this paper contributes to our knowledge
This work describes a pneumonia readmission risk prediction model based on data readily available in the electronic health record at the point of care. The model had good discriminative performance and predicted risk accurately along a broad range. The model enables calculation of readmission risk at the point of discharge, potentially allowing intensive interventions to focus on high-risk patients.
Methods
Setting and Subjects
The study was approved by the institutional review board of the Cleveland Clinic (approval 13-382). The study was performed on a retrospective cohort of 1,421 adult hospitalizations age ≥18 y who were admitted to the Cleveland Clinic Main Campus, a tertiary care academic medical center with 1,400 beds. The cohort consisted of subjects admitted with a primary diagnosis of pneumonia, identified by the International Classification of Diseases, 9th Revision (ICD-9) codes utilized in the CMS readmission metric (480.XX, 481, 482.XX, 483.X, 485, 486, 487.0), between January 2010 and December 2012.10 Manual chart review of the entire population was conducted to ensure validity of the outcome data (ie, readmission status). Patients who did not have a documented clinical follow-up in the Cleveland Clinic Health System between 30 d and 6 months after discharge were excluded. The final model was then validated on subjects admitted with a diagnosis of pneumonia to the Cleveland Clinic in 2013 (validation cohort, n = 393). Figure 1 summarizes the inclusion and exclusion criteria for the derivation and validation cohorts.
Flow chart for the creation and validation of the Cleveland Clinic pneumonia model. * Twenty-seven of these patients died within 30 d of discharge according to the Ohio Death Index.
Risk Factors for Readmission
Candidate variables were determined based on literature review and included age,12 sex,11,13,14 race,15 vital signs,16 immunization status,17 medications,17 and comorbidities.18 Variables had to be discretely and reliably available within the electronic health record. The complete list of 134 variables considered as candidates for the model building are displayed in supplementary materials at http://www.rcjournal.com. Univariate analyses were used to screen the complete list of candidate variables. Variables were removed if not significantly associated with the outcome (P ≤ .05) or if they displayed an SE around the estimated coefficient >0.3. A complete logistic model was then fit using the remaining variables with a clustering penalization for the repeating measures within each subject.19 The clustering term was added to the final model to account for repeated admissions by the same subject. Variables that appeared to be causing multicollinearity (as assessed by variable inflation factor ≥2) were removed.
For variables with repeated measures, such as vital signs and laboratory results, the model focused on the periods of admission and discharge. Specifically, only the minimum (lowest) and maximum (highest) measurements recorded within the first and last 24 h of the hospital stay were included. This approach is similar to that of the APACHE III calculator.20 Continuous variables were modeled using natural log transformations to create a linear relationship with the outcome. If a linear relationship could not be established, then the variable was modeled using restricted cubic splines with 3 knots.
Missing data were imputed using the MICE (Multiple Imputation by Chained Equations) package, version 2.22, for R. The imputation process included all other predictor and outcome variables to build the regression equations used to replace the missing data.21 The imputation process was repeated 5 times with replacement. Harrell's22 model approximation process for variable selection was used to rank the variables in order of importance, per the model's R2. Discrimination was calculated using the concordance statistic, and random sampling with replacement was performed 1,000 times to obtain the optimized corrected concordance statistic at each ranking cutoff of the model selection process. The optimized corrected concordance statistic was plotted against model size, and the final model was chosen at the apex of this curve (ie, the final model included variables that maximize the discrimination, as measured by the concordance statistic).
The model was compared head-to-head with the Medicare administrative model.10 The Medicare prediction was calculated using a “cross-walk” file (available at https://www.cms.gov/Medicare/Health-Plans/MedicareAdvtgSpecRateStats/Risk-Adjustors.html) that converts ICD-9 codes to the hierarchical conditional categories used in the Medicare calculation. The comparison with CMS was performed among the subset of subjects 65 y of age and older during both the internal validation (n = 628) and the external validation (n = 170).
The models' discriminative performances were assessed by plotting receiver operating characteristic curves. In addition to discrimination, the accuracy of the final model was evaluated using a calibration curve. All statistical analyses were performed using R 2.15.2.
Results
Of the 1,421 hospitalizations, 2 were excluded because >50% of the data points were missing, and 124 were excluded because their readmission status could not be determined from the electronic health record due to a lack of follow-up information. The investigators speculated that patient deaths occurring outside the hospital could partially explain the lack of follow-up data for these subjects. A search of the Ohio Death Index indicated that 27 of 124 subjects without follow-up electronic health record data had died within 30 d of discharge (22%). Of the remaining 1,295 subjects, 299 were readmitted to a Cleveland Clinic hospital within 30 d. The chart review identified 31 additional subjects who were admitted to other hospitals within 30 d (overall readmission rate 330/1,295 = 25%). There were 1,191 unique subjects in the cohort, with 104 subjects experiencing 2 or more hospitalizations and 3 subjects experiencing 4 hospitalizations. Twenty-two of the 330 readmitted subjects (6.6%) died during the readmission. All of the non-readmitted subjects were alive at the end of the 30-d period.
Univariate associations between risk factors and readmission appear in supplementary materials at http://www.rcjournal.com. Readmitted subjects were older (66 y vs 62 y, P < .001), had a longer stay (7.2 d vs 5.5 d, P < .001); were more likely to have cancer (all types combined), coronary heart disease, chronic heart failure, and chronic kidney disease (P = .001, P < .001, P < .001, and P < .001, respectively); and were more likely to receive prescriptions for warfarin or an opioid upon discharge (both P < .001). Among the laboratory values, lower values of hemoglobin and albumin within the first and last 24 h of hospitalization were also associated with higher readmission rates. Prior health-care utilization indices, such as prior hospital admissions, were also higher in those subjects with 30-d readmissions (2.96 vs 1.46 prior admissions over the past year, P < .001). The final model contained 13 variables (Table 1) and had an internally validated, bias-corrected C statistic of 0.74 (95% CI 0.71–0.77). In the external validation cohort from 2,013 subjects (n = 393), the C statistic was 0.71.
Final Model
The calibration curve suggests good calibration (Fig. 2). The model tended to overestimate risks in the highest risk subjects, and half of the risk probabilities fell between 0.08 and 0.39. In the external validation cohort, the Cleveland Clinic model displayed better discrimination among patients 65 y of age and older compared with the CMS model (C statistic 0.69 vs 0.60, P = .042) (Fig. 3). Although both models had good calibration, the Cleveland Clinic model had a wider range of predicted probability (0–0.8) than the Medicare administrative model (0.1–0.5). This allows for better discrimination of the higher- and lower-risk subjects. The upper 25th percentile of the predicted risk according to the Cleveland Clinic Model is >0.4, with a sensitivity and specificity of 48 and 83%, and the upper 25th percentile of the CMS predicted risk is >0.29, with a sensitivity and specificity of 31 and 81%.
Calibration curve for the final Cleveland Clinic pneumonia model. The 45° line indicates perfect calibration, and any deviation from the ideal line indicates a difference between the predicted probability and the actual probability. Histograms indicate the number of subjects in each risk stratum. C statistic = 0.74 (95% CI 0.71–0.77).
Receiver operating characteristic curves for measuring the sensitivity and specificity of the Cleveland Clinic (CC) model and the Centers for Medicare and Medicaid Services (CMS) model's ability to predict the risk of readmission in a subset of subjects age ≥ 65 y admitted with pneumonia in the 2013 validation set. P = .042. Area under the curve = 0.69 for the CC model and 0.60 for the CMS model.
Discussion
In this retrospective study, we derived and validated a model for predicting 30-d readmission for patients hospitalized with pneumonia. Our model contained 13 variables, all of which are readily available in the electronic health record, and was better at identifying high-risk subjects than the CMS model. One reason for the model's superior performance was the inclusion of clinical variables that are not available in administrative data. In fact, 2 of the 3 most predictive variables (opioid prescription and serum albumin during the first 24 h) were not administrative in nature. Overall, 9 of the 13 variables that contributed to the final model are not included in the CMS model. One reason that CMS uses administrative data for its model is the ease of collection. However, for real-time decision-making, discrete data points that can be extracted from the electronic health record may be easier to use, in addition to being more accurate. For the purpose of discriminating individual patients at high risk for readmission at the point of care, inclusion of clinical data improved model performance.
Readmissions following an index pneumonia admission are common, with approximately 16% of patients readmitted within 30 d.5 The rate of readmission in the current study (25%) is substantially higher than the national average, probably due in part to the complexity of patients seen at tertiary care centers like the Cleveland Clinic. Other factors that account for the difference may include a patient population with greater sociodemographic risk factors and a higher proportion of subjects susceptible to pneumonia complications, such as transplant recipients and cancer patients. Regardless, the differences between hospitals highlights the potential benefits of locally created risk models.
Various studies have assessed factors associated with readmission. Halm et al16 reported that having one or more clinical criteria for instability (abnormal heart rate, breathing frequency, systolic blood pressure, temperature, low oxygen saturation, inability to swallow, and neurological impairment) during the 24 h before discharge was associated with a significantly higher risk of death and readmission. Some studies have identified modifiable risk factors, such as administration of appropriate antibiotics,7 vaccination status,17 and appropriate discharge planning.8 Other studies have noted the importance of non-modifiable factors, such as race15 and socioeconomic status.23,24 In building a high performance predictive model for pneumonia readmissions, one should be inclusive of all possible variables that may contribute to the risk of readmission. The model for pneumonia readmission developed in this paper considered a wide range of variables, including vital signs, laboratory findings, comorbid conditions, medications, and demographic data located within the electronic health record. The resulting model had a moderate to high concordance statistic and good calibration.
In a systematic review of clinical prediction models for various disease states, Kansagara et al25 identified 26 unique models for various diseases that used hospital readmission as the primary outcome. Nine used administrative data from multiple centers and displayed poor discriminative ability, raising questions about the validity of applying predicted risk models based on heterogeneous data for estimation of risk in individual hospitals. To that end, compared with the predictive model used to calculate the risk of readmission following hospitalization for pneumonia among Medicare beneficiaries,10 the Cleveland Clinic model performed better in discriminating between subjects with low and high risk for readmission. It is important to note that the model was built on the entire population of pneumonia subjects and was not limited to the elderly.
In addition to the CMS model, there are 7 other pneumonia readmission prediction models in the literature.10–12,26–29 These prediction models and publications that report on the CMS model10,30 were recently reviewed.31 In general, predictive models were critiqued for not including severity-of-illness measures, data on in-patient course, and stability on discharge.31 Moreover, the models were not compared directly with each other, so it is impossible to know how they would perform in other institutions. Our model outperformed the CMS model, highlighting the importance of a locally derived model for the purposes of identifying high-risk patients. In comparison with the Hartford Hospital model developed by Mather et al,11 our model included all pneumonia hospitalizations rather than only CMS beneficiaries. Recently, Makam et al29 published a pneumonia-specific readmission risk prediction model utilizing electronic health record data from the full hospital stay. Their model displayed good discrimination and predicted a broad range of risk similar to our model, but there are some notable differences in the choice of predictor variables. Specifically, Makam et al29 included median income of subjects as a predictor variable while excluding medications. Although socioeconomic status has been shown to be associated with readmission risk,23,24 inclusion of income may hinder point-of-care application, since this information may not be available through the electronic health record. In contrast, medications are readily available in most electronic health record systems and may provide important clues about severity of illness and potential adverse events.
Although interventions to specifically prevent pneumonia readmissions are scarce,32–34 focusing effective interventions to patients at high risk for readmission is a promising approach to reduce readmissions.9 Wasfy et al35 recently reported an accelerated reduction in readmission rates for pneumonia discharges after the passage of the Patient Protection and Affordable Care Act. This real-world report confirms findings from randomized control trials and instills further confidence in the efficacy of interventions to reduce readmissions. We propose that the next step is focusing resource-intensive interventions to high-risk patients by leveraging accurate risk prediction models. Interventions aimed at preventing readmissions may be improved if the reasons for readmissions are considered in the design of the intervention. An analysis of >200,000 readmissions within 30 d of an index hospitalization for pneumonia using Medicare claims also found a broad range of diagnoses, including pneumonia, heart failure, COPD, septicemia, renal disorders, cardiorespiratory failure, arrhythmias/conduction disorders, Clostridium difficile infections, urinary tract infections, and gastrointestinal bleeding.36 Given the significant heterogeneity in the reasons for readmission, interventions to prevent readmissions should include diverse elements to address the numerous readmission reasons.
Although causality should not be concluded from risk prediction paradigms, we note that opioid prescription on discharge was identified as a predictor variable for 30-d readmission in our cohort. Opioid-related adverse events and overdose are prevalent and pose a serious population health problem.37,38 In other settings, such as major abdominal surgery39 and liver transplantation,40 opioid prescription has been associated with increased 30-d readmissions. To our knowledge, this is the first report of an association between opioid prescription on discharge and increased hospital readmission in the setting of an index pneumonia admission. Inclusion of opioid prescription as a predictor variable may improve performance among existing readmission prediction models for pneumonia.
Our study has limitations. First, we excluded patients whose readmission status could not be ascertained because they had no follow-up at the Cleveland Clinic after hospital discharge. This subgroup represented <10% of the study population and was unlikely to affect overall model performance significantly. Additional sensitivity analysis revealed a similar predicted readmission risk for the cohort of subjects whose outcome was unknown compared with the subjects whose outcome was available to construct our model (data not shown). Second, we recognize that, although the model performed well in a validation cohort, continuous training may be necessary to maintain acceptable discriminative accuracy.27 Third, this model lacks validation beyond our hospital and may not be applicable to all settings. The users of the model can choose a probability threshold a priori, depending on desired sensitivity and specificity in predicting readmissions (ie, a higher probability threshold would identify fewer patients, but each would have a higher probability of readmission for more efficient use of resources). Finally, it should be noted that the model was created in adult subjects ≥18 y of age, whereas the head-to-head validation with the CMS model was restricted to subjects ≥65 y of age. Despite the differences in the composition of these cohorts in terms of age, the model still performed well.
Conclusions
We report a well-calibrated risk prediction model for pneumonia readmissions to the hospital with good discriminative capability, which could potentially be available at the point of discharge. The model was built on all hospitalized pneumonia subjects, not exclusively seniors. By utilizing a clustering term, we included all hospitalizations, including repeat hospitalizations of the same subjects. Discriminative performance was superior to the CMS risk prediction model among seniors. The model includes variables that are accessible and available in real time in the electronic health record. Therefore, the risk of readmission can be calculated at the point of discharge. We foresee such a calculator incorporated into the electronic health record and the calculated risk made available to care managers. The information could be used to focus post-acute care interventions in a high-risk group of patients. Further, the risk threshold could be adjusted to efficiently allocate available resources.
Footnotes
- Correspondence: Umur Hatipoğlu MD, Respiratory Institute-Cleveland Clinic, Desk A.90, 9500 Euclid Avenue, Cleveland, OH 44195. E-mail: hatipou{at}ccf.org.
The authors have disclosed no conflicts of interest.
Dr Hatipoğlu presented a version of this work at the Chest 2015 International Conference, held October 25–28, 2015, in Montréal, Canada.
Supplementary material related to this paper is available at http://www.rcjournal.com.
- Copyright © 2018 by Daedalus Enterprises
