Abstract
BACKGROUND: Existing models developed to predict 30 days readmissions for pneumonia lack discriminative ability. We attempted to increase model performance with the addition of variables found to be of benefit in other studies.
METHODS: From 133,368 admissions to a tertiary-care hospital from January 2009 to March 2012, the study cohort consisted of 956 index admissions for pneumonia, using the Centers for Medicare and Medicaid Services definition. We collected variables previously reported to be associated with 30-day all-cause readmission, including vital signs, comorbidities, laboratory values, demographics, socioeconomic indicators, and indicators of hospital utilization. Separate logistic regression models were developed to identify the predictors of all-cause hospital readmission 30 days after discharge from the index pneumonia admission for pneumonia-related readmissions, and for pneumonia-unrelated readmissions.
RESULTS: Of the 965 index admissions for pneumonia, 148 (15.5%) subjects were readmitted within 30 days. The variables in the multivariate-model that were significantly associated with 30-day all-cause readmission were male sex (odds ratio 1.59, 95% CI 1.03–2.45), 3 or more previous admissions (odds ratio 1.84, 95% CI 1.22–2.78), chronic lung disease (odds ratio 1.63, 95% CI 1.07–2.48), cancer (odds ratio 2.18, 95% CI 1.24–3.84), median income < $43,000 (odds ratio 1.82, 95% CI 1.18–2.81), history of anxiety or depression (odds ratio 1.62, 95% CI 1.04–2.52), and hematocrit < 30% (odds ratio 1.86, 95% CI 1.07–3.22). The model performance, as measured by the C statistic, was 0.71 (0.66–0.75), with minimal optimism according to bootstrap re-sampling (optimism corrected C statistic 0.67).
CONCLUSIONS: The addition of socioeconomic status and healthcare utilization variables significantly improved model performance, compared to the model using only the Centers for Medicare and Medicaid Services variables.
- readmission
- pneumonia
- Centers for Medicare and Medicaid Services
- CMS
- socioeconomic status
- healthcare utilization
Introduction
Reduction in pneumonia-related readmission has been identified as a marker for quality care. The Centers for Medicare and Medicaid Services (CMS) Readmission Reduction Program1 has challenged the healthcare system in the United States to uncover novel ways to reduce readmission rates and maximize reimbursement by CMS. Over 2,000 hospitals have been levied penalties, ranging from 0.01% to 1% of their CMS revenue in fiscal year 2013 for subpar readmission rates for heart failure, pneumonia, and heart attack, totaling approximately $280 million in penalties paid back to CMS. There will be a substantial increase in the penalties in the next 2 years, to 2% in fiscal year 2014, and to 3% in fiscal year 2015, so there is strong incentive to develop predictive algorithms to determine which patients are at highest risk for readmission.
Despite this need to identify patients at risk for readmission, the applicability of available models is circumspect. A recent systematic review2 of 26 models uncovered: most models perform poorly, including the 3 CMS models (pneumonia, heart failure, and myocardial infarction); only one clinical model successfully met the goals set forth for clinical application and had what is considered acceptable discriminative ability when detecting readmission for heart failure3; the successful addition of social or functional variables offer opportunities for improvement in model performance.4,5
There are few readmission models centered on pneumonia 30 day readmissions,6–12 and only a few reported on discriminative ability.6,11,12 In this single-center, retrospective analysis our intention was to leverage variables shown to be significant in previously reported predictive models and supplement the CMS medical record model6 with additional variables shown to be of benefit in other models, in an effort to produce a model to identify before discharge patients at high risk for readmission.
QUICK LOOK
Current knowledge
Hospital readmission for pneumonia is a common and expensive complication. The Centers for Medicare and Medicaid Services (CMS) Readmission Reduction Program identified readmission for pneumonia as a quality indicator for hospitals, with proposed financial penalties for subpar performance.
What this paper contributes to our knowledge
The addition of variables including prior hospitalization, median income, and depression significantly improved the ability of the current CMS guidelines to predict readmission. Readmission factors that are outside of the hospital's control include timely post-discharge visit to a primary care provider, medication adherence, and health literacy.
Methods
This retrospective, observational study was carried out at Hartford Hospital, an 800-bed teaching hospital in Hartford, Connecticut. The Hartford Hospital institutional review board approved the study (assurance FWA000000601) and certified that it met the criteria for a waiver of the informed consent requirement.
Subjects
The subjects were derived from an electronic database collected at Hartford Hospital. The CMS6,12 inclusion criteria were: CMS patients (enrolled in fee-for-service CMS parts A and B) admitted to Hartford Hospital from January 2009 to March 2012 with principal diagnosis of pneumonia (International Classification of Diseases, 9th Revision, Clinical Modification codes 480.XX, 481, 482.XX, 483.X, 485, 486, and 487.0) as potential index pneumonia admissions. We excluded patients < 65 year of age, who died in the hospital, were transferred to another acute care facility, and/or for whom data were incomplete. If a subject was admitted more than once over a 30 day period, only the first admission was counted as an index admission.
Outcomes
The primary outcome was 30-day all-cause readmission.12 Hospital administrative data sources were used to assess readmissions. A pneumonia-related readmission was defined using the CMS13,14 definition: pneumonia (480.XX, 481, 482.XX, 483.X, 485, 486, 487.0) as the primary diagnosis, regardless of secondary diagnosis; or septicemia (038.0–038.99) or respiratory failure (518.81 or 518.84) as the primary diagnosis, with a secondary diagnosis of pneumonia.
Variable Selection
The selection of candidate variables was based on the CMS-developed Hierarchical Condition Category clinical classification system selection algorithm,6 and we used the 35 variables in the final CMS medical record model. Twenty-six of the CMS variables are listed in Table 1 (see column CMS). The 9 remaining variables were defined as missing (yes/no) for: sodium, glucose, hematocrit, creatinine, white blood cell count, blood urea nitrogen, heart rate, systolic blood pressure, and breathing frequency. Briefly, using the Hierarchical Condition Category algorithm, the > 15,000 International Classification of Diseases, 9th Revision diagnosis codes are mapped to one of 189 clinically coherent condition categories. Of the 189, 35 were selected as potentially related to readmission outcome based on: review of a list of variables used for validating the National Quality Forum approved pneumonia mortality measure; reviewing a list of variables from the National Pneumonia Project set; and variables chosen based on a systematic review of the literature. For each subject the presence of the condition was assessed from the secondary diagnosis at the index admission.
Univariate Predictors of 30-Day Readmission for Pneumonia
The laboratory and vital signs were extracted from the hospital electronic health record, where the first value after admission was used. Age, sex, diagnosis, readmission from a skilled nursing facility, marital status, prior admission, and stay were extracted from administrative registries.
Additional variables used, not present in the CMS model, included marital status, stay, anxiety/depression (as coded by the condition categories as described previously), number of hospitalizations prior to the index admission (as a measure of healthcare utilization, and defined with a cutoff of ≥ 3 hospitalizations in the 3 years prior to the index admission for pneumonia), and year 2000 census data15 (to obtain data on median household income for Connecticut zip codes, and used as a measure of socioeconomic status).
Statistical Analysis
Descriptive statistics include frequency, mean ± SD, and median. The chi-square test was used to compare categorical variables, and the Student t test or nonparametric Wilcoxon test was used for continuous variables in the univariate analysis. When converting continuous data to binary variables (number of visits prior to the index visit, median income), we plotted receiver operating characteristic curves to estimate the optimum combination of sensitivity and specificity to arrive at an end point.16 Differences between the areas under the receiver operating characteristic curves were detected as described.17
We used multivariate logistic regression analyses to identify baseline subject characteristics that were independently associated with all-cause 30-day readmission. All predictors that were found to be statistically significant at P < .15 in the univariate analysis were entered into the logistic regression model. We repeated the logistic regression procedure as described in separate models using only significant CMS medical record model6 variables with an end-point of all-cause 30-day readmission; all significant univariate predictors with pneumonia-related 30-day readmissions as the outcome variable; and all significant univariate predictors with pneumonia-unrelated 30-day readmissions as the outcome variable.
The predictive accuracy of the multivariate models was determined by calculating the area under the receiver operating characteristic curve (or C statistic),17 and calculated observed readmission rates in the lowest and highest deciles on the basis of predicted readmission probabilities. We used the method described by Hanley and McNeil17 to determine the differences between the receiver operating characteristic curves. Overall model performance was measured using Nagelkerke's R2, a measure of explained variance,18 and the Hosmer-Lemeshow19 test.
To ensure that the model's predictions are generalizable, it is important to evaluate the model's performance more realistically than simply by calculating its accuracy on the training sample. To do this we used bootstrapping as an internal validation technique.20,21 Briefly, this method involves drawing repeated samples from the original sample, with some randomly excluded, and others included more than once, resulting in a bootstrap sample. This procedure was repeated 200 times, resulting in an average C statistic for the bootstrap sample. This is then subtracted from the C statistic developed from the original sample. The result is termed the optimism of the apparent performance of the model on the training data set. The observed performance is moderated by subtracting the degree of optimism from the apparent performance. One of the benefits of bootstrapping is that it allows all of the available subject data to be included in the data set. It has been shown to estimate model performance more accurately than other approaches, such as those that involve setting aside data for a separate validation sample.
All effects were considered significant at P ≤ .05. The statistical analyses were performed with SPSS 16.0 (SPSS, Chicago, Illinois), with the exception of the bootstrapping, which was conducted with Stata 11 (StataCorp, College Station, Texas).
Results
A total of 1,249 in-patients met the CMS inclusion criteria12 for a pneumonia index admission. Of the 1,249 subjects, CMS exclusion criteria eliminated: 14.1% < 65 years of age, 5.7% in-patient deaths, 0.2% transferred to another acute care facility, 1.3% who had > 1 admission in the 30-day readmission period, and 2.1% due to missing data. This resulted in a final population of 956 (Fig. 1), of which 148 were readmitted within 30 days, for an unadjusted readmission rate of 15.5% (Fig. 2), which was uniformly distributed over the 30-day period. Pneumonia-related hospital readmissions accounted for 16.9% of total 30-day readmissions (Table 2). Five (20%) of the subjects readmitted for pneumonia-related causes died in the hospital, and 13 (52%) were transferred to a skilled nursing facility. The most frequent diagnosis for pneumonia-unrelated readmissions included aspiration pneumonitis, congestive heart failure, and cardiorespiratory failure and shock.
Flow chart.
Kaplan-Meier curve of 30-day hospital readmission.
Pneumonia 30-Day Readmission Diagnosis
Variables Statistically Associated With All-Cause 30-Day Readmission
Table 1 provides the univariate analysis of all study subjects, with comparisons between readmitted and non-readmitted subjects. There were significant differences in stay, discharge to a skilled nursing facility, ≥ 3 previous admissions, median income ≤ $43,000, chronic lung disease, renal disease, cancer, creatinine ≥ 2.5 mg/dL, hematocrit < 30%, and history of anxiety or depression.
When defining readmissions as pneumonia-related (Table 3), ≥ 3 previous admissions, cancer, hematocrit < 30% and white blood cell count of 6–12 b/μL showed statistically significant differences. When the outcome was defined as pneumonia-unrelated, in accord with all readmissions, there were significant differences in stay, ≥ 3 previous admissions, median income ≤ $43,000, chronic lung disease, renal disease, creatinine ≥ 2.5 mg/dL, and hematocrit < 30%.
Univariate Predictors of 30-Day Readmission for Pneumonia-Related and Pneumonia-Unrelated Readmissions
Pneumonia-related readmissions included a higher percent with a history of cancer (36.0% vs 12.2%, P < .001), and a lower percent with heart failure (20.0% vs 48.0%, P = .01) and white blood cell count 6–12 b/μL (28.0% vs 54.9%, P = .01), when compared to pneumonia-unrelated 30-day readmissions.
Multivariate Analysis
As shown in Table 4, variables significantly associated with a risk for 30-day all-cause readmission include male sex, ≥ 3 previous admissions, chronic lung disease, cancer, median income ≤ $43,000, history of anxiety or depression, and hematocrit < 30%. In the logistic regression model, using pneumonia-related readmissions as the dependent variable, factors associated with a higher risk of readmission include ≥ 3 previous admissions, cancer, and history of anxiety or depression. The model using pneumonia-unrelated readmissions found median income ≤ $43,000, ≤ 3 previous admissions, chronic lung disease, and heart rate ≥ 125 beats/min to be significant risk factors.
Pneumonia 30-Day Readmission Logistic Regression Results
Model Performance
As shown in Table 5, the logistic regression model showed reasonable discrimination for all readmission causes (C statistic [area under the curve] 0.71, 95% CI 0.66–0.75). Validation via 300 bootstrap samples resulted in an optimism-corrected C statistic of 0.67. The predicted readmission rate ranged from 7.5% in the lowest to 43.0% in the highest predicted decile, and the model was well calibrated: the Hosmer-Lemeshow goodness of fit statistic was chi-square = 5.92 (P = .66). The model predicting pneumonia-unrelated readmission gave a Hosmer-Lemeshow goodness of fit statistic of chi-square = 2.47, P = .96, and demonstrated fair discrimination (C statistic 0.68, 95% CI 0.64–0.73), but the model's predictive ability was less as the observed readmission rate ranged from 3.3% in the lowest predicted decile to only 36.6% in the highest. When predicting pneumonia-related readmission, the model was well calibrated: the Hosmer-Lemeshow goodness of fit statistic was chi-square = 3.44, P = .90. The model demonstrated poor to fair discrimination (C statistic 0.65, 95% CI 0.60–0.70), and the model's predictive ability was poor, with the observed readmission rate ranging from only 9.1% in the lowest predicted decile to only 34.0% in the highest.
Model Performance
Looking at the relative contribution of each risk factor's predictive ability in the multivariable model for all-cause 30-day readmissions (Fig. 3) we found that low income and high hospital utilization accounted for nearly 40% of the variance in the model.
Relative contribution of risk factors for pneumonia 30-day readmission prediction model.
We compared our model to a model that used only factors used in the CMS medical record sample (excludes stay, marital status, health care utilization, and socioeconomic status, and a measure of anxiety/depression), and found a significant improvement in the C statistic (Fig. 4): 0.71 versus 0.67, z = 2.14, P = .03.
Receiver operating characteristic curve for Hartford Hospital's model and the Centers for Medicare and Medicaid Services (CMS) model for pneumonia-related readmission. The area under the receiver operating characteristic curve demonstrates significantly better discrimination with Hartford Hospital's all significant univariate predictors model. The diagonal line represents an area under the curve of 0.5, which indicates no discrimination.
Discussion
We present an in-patient 30-day risk-adjusted all-cause readmission model for pneumonia patients that can be used in performance measurement and quality improvement at a similar tertiary care hospital. We included 956 subjects hospitalized for pneumonia, who showed a 30-day readmission rate of 15.5%. Our model identified 7 variables as significant risk factors for pneumonia 30-day all-cause readmissions. Some of these factors confirm previous findings, and other are newly reported. In addition, 3 significant risk factors associated were revealed in the pneumonia-related 30-day readmission model, and 4 risk factors were associated with pneumonia-unrelated readmissions. The measure of hospital utilization was significant, regardless of the model outcome.
We chose to select baseline variables for univariate analysis from the CMS Medical Record risk-adjusted model for pneumonia readmissions6 because we were using a similar patient population with identical inclusion/exclusion criteria, were measuring the same outcome, and because of the robustness of the CMS Medical Record risk dataset (47,429 subjects). Despite the size of the population and the number of variables initially screened for use, the CMS models perform poorly,2 demonstrating that there is substantial variance not accounted for by the CMS models. The reason for the lack of discrimination becomes clear when we consider the multitude of factors influencing the likelihood of readmission after hospital discharge9,10,22–24 that are not included in the CMS model. With the inclusion of some of these variables in our model we increased the discriminatory ability of the model significantly over the CMS model.
Our finding of prior healthcare utilization as a significant risk factor is in agreement with others,25–30 and it remained significant across all 3 models (all readmissions, pneumonia-related, or pneumonia-unrelated). The risk factor prior healthcare utilization (eg, prior hospitalizations and emergency room visits) is likely an indicator of patients with unstable or more severe disease. Our inclusion of history of anxiety or de-pression was based on evidence from other studies, in which the mental domain score of the Medical Out-comes Study Short Form 36-item questionnaire was a significant multivariate predictor of non-elective readmission.28
Our finding of lower income is in agreement with others who have shown socioeconomic status indicators as predictors of higher readmission rates. Billings et al4 studied the effect of household income on hospital readmission for the non-CMS population in New York City and discovered that the admission rate was higher for the low-income population than for the high-income population. Amarasingham et al3 found median income a significant univariate predictor of heart failure readmission, with borderline significance, and Philbin et al31 showed that income is significantly associated with increased readmission in heart failure.
On the contrary, Arbaje et al10 failed to include lower income as a significant multivariate predictor of 60-day non-elective readmissions; however, that study lacked power to detect differences in readmission in the designated levels of income. They acknowledged that if the readmission rate had been higher, then low income might have had a significant association with early readmission.
A systemic review of risk prediction models for readmission concluded that the vast majority, including the 3 CMS models (heart failure, acute myocardial infarction, and pneumonia), performed poorly in predicting which patients were at high or low risk for readmission.2 Indeed, only a handful have reported a C statistic above 0.7. Their finding showed that most models were limited to medical comorbidity, prior healthcare utilization, and basic sociodemographic data. Only a few considered variables such as functional status, overall health, social determinants of health (income, employment, socioeconomic status, access to care, social support), or illness severity in their models. Two of the studies3,5 that did include these variables in their models demonstrated increased predictive ability.
An important limitation to this study was that it was conducted in a single, inner-city tertiary care hospital, so its external validity may be limited to the geographic area and practice type. However, many of the variables shown as significant risk factors in other studies in different environments were replicated as significant variables in this study. A further limitation is that the census file we used for median income was from the year 2000, and we recognize that there may have been regional changes in income in the last decade. Finally, the power of the multivariate analysis is limited when segmenting the outcome as pneumonia-related/unrelated, though we did not observe large parameter estimates or standard errors, which can be diagnostic of too few events relative to variables.
Conclusions
We successfully produced a model that can identify, before discharge, patients at high-risk for readmission. We sought to build on previously published predictive models for pneumonia readmission, many of which displayed relatively weak performance with an attempt to enhance the predictive ability with the addition of variables related to marital status, prior hospitalizations, median income, diagnosis of anxiety or depression, and index hospital stay. Of these additions, prior hospitalization, median income, and depression/anxiety were all significant predictors in the all-cause readmission model, and made substantial contributions to model performance.
Despite the significant improvement over previously published models of pneumonia 30 day readmission,6 there is much need for improvement, as the maximum predictive ability remains inadequate. It is apparent that much of the influence on the risk of readmission lies outside of the hospital. Factors such as confirmed and timely post-discharge visit to the patient's primary care provider, medication and diet adherence, social support, health literacy, and communication between providers may be significant contributors to a predictive algorithm for readmission. Further research is needed to develop a more comprehensive model that incorporates potential influential variables such as these.
Footnotes
- Correspondence: Jeffrey F Mather MSc, Department of Research Administration, Hartford Hospital, 80 Seymour Street, Hartford CT 06106. E-mail: jmather{at}harthosp.org.
The authors have disclosed no conflicts of interest.
- Copyright © 2014 by Daedalus Enterprises
