Abstract
OBJECTIVE: To assess the diagnostic accuracy of the clinical pulmonary infection score in the diagnosis of ventilator-associated pneumonia in mechanically ventilated patients.
METHODS: We searched PubMed and the Cochrane database, and included only studies that compared clinical pulmonary infection score with quantitative microbiological analysis of samples for diagnosing ventilator-associated pneumonia. We constructed 2-by-2 tables of diagnostic accuracy from each article, and meta-analyzed the results by pooling estimates of sensitivity, specificity, likelihood ratio for positive index test, likelihood ratio for negative index test, diagnostic odds ratio, and 95% confidence intervals.
RESULTS: Thirteen studies met the inclusion criteria. The pooled estimates for sensitivity and specificity for clinical pulmonary infection score were 65% (95% CI 61–69%) and 64% (95% CI 60–67%), respectively. The combined diagnostic odds ratio was 4.85 (95% CI 2.42–9.71) and the area under the curve was 0.748 (95% CI 0.65–0.85).
CONCLUSIONS: The diagnostic performance of the clinical pulmonary infection score for ventilator-associated pneumonia is moderate. However, the clinical pulmonary infection score is simple and easy to perform, and may still be useful in diagnosing ventilator-associated pneumonia.
Introduction
Ventilator-associated pneumonia (VAP) is the most frequent nosocomial infection in the intensive care unit; VAP prolongs the duration of ventilation and hospital stay, and increases healthcare costs, morbidity, and mortality.1 VAP is a common complication that affects 8–20% of intensive care unit patients.2 A review reported that the prevalence of VAP ranged from 9–68%, with associated mortality of 20–50%.3
VAP is defined as pneumonia that develops more than 48 hours after tracheal intubation or tracheotomy. The challenges of managing VAP include the requirement for appropriate antimicrobial therapy, and the need to avoid administering unnecessary antibiotics.4,5 Overestimating the probability of VAP can lead to inappropriate antibiotic use and consequently to the emergence of multiple-drug-resistant organisms and invasive fungal infection. Conversely, underestimating the probability of VAP can lead to under-treatment of a serious nosocomial infection and increased mortality,6,7 so strategies that contribute to early accurate diagnosis of VAP are clinically important. The diagnostic challenge has multiple implications for therapy. However, at present, the fundamental obstacle to diagnosing VAP is the absence of a uniform accepted standard.8 There are 2 diagnostic approaches for VAP. The first is quantitative microbiological analysis of bronchoalveolar lavage fluid (BALF) or protected specimen brush sample. The other approach relies on clinical criteria, including the new or worsening radiographic infiltrates and leukocytosis or leukopenia, fever, or purulent sputum.9 However, the latter approach has an unacceptably low sensitivity compared to BALF or protected specimen brush sample.10
The invasive diagnostic method (BALF) was postulated to improve identification of patients with true VAP and facilitate the decision whether to treat for VAP, and thus improve clinical outcomes. However, the invasive approach requires rigorous adherence to bronchoscopic and microbiologic techniques, and the culture results only become available substantially after the clinical suspicion of VAP. The use of the invasive technique in everyday practice remains controversial.5 A simple, noninvasive, early VAP diagnosis technique would be useful.
Pugin et al introduced the clinical pulmonary infection score (CPIS) to simplify the diagnosis of VAP, and they found that CPIS had a sensitivity of 93% and specificity of 100% for diagnosing VAP.11 The CPIS incorporates readily available clinical information and has became a popular VAP diagnosis method. A subsequent study found that the CPIS has a sensitivity of 72–77% and a specificity of 42–85% for diagnosing VAP.12 Wood et al found that a modified version of the CPIS, which excluded culture results, had a sensitivity of 60% and a specificity of 43% for diagnosing VAP.13
At present, the diagnostic tools for VAP have not been sufficient and complete, and the early VAP prediction or diagnosis is difficult. The overall diagnostic value of the CPIS has yet to be confirmed. In addition, the clinical utility of such a score would be higher if it helped clinicians to the early and accurate diagnosis of VAP and then in their decision to initiate or withhold antibiotic therapy in patients clinically suspected of VAP.
We systematically reviewed the accuracy of CPIS for diagnosing VAP by pooling estimates of its accuracies and presenting combined diagnostic accuracies, to provide useful evidence for the development of an accepted standard.
Methods
Search Strategy and Study Selection Criteria
We searched the PubMed and Cochrane databases up to July 2010 for studies that evaluated the diagnostic performance of CPIS in VAP. We used no age, language, or publication restrictions in the searches. Our PubMed search terms were: (“pneumonia, ventilator-associated” [MESH terms] AND (“clinical pulmonary infection score” [title/abstract] OR “clinical pulmonary infection score” [text] OR “CPIS” [title/abstract] OR “CPIS” [text])) AND diagnosis/broad [filter]. Our Cochrane search terms were “ventilator-associated pneumonia” (title, abstract or key words) AND “clinical pulmonary infection score” (title, abstract or key words). We supplemented our searches by manually reviewing the references of all relevant studies. Two of us independently performed the process. Any disagreements were adjudicated by a third investigator.
We included prospective and retrospective studies that met all the following criteria:
Diagnosed VAP as the target condition
Enrolled at least 10 human participants
Used CPIS as the diagnostic tool for VAP
Provided enough information to allow us to derive the numbers of true positive, true negative, false negative, and false positive VAP diagnoses
The exclusion criteria were
Review article or letter to the editor
Case report
Conference abstract
Did not provide the numbers of true positive, true negative, false negative, and false positive VAP diagnoses
Data Extraction
We used a standard form to extract data on author, year of publication, type of study, country in which the study was carried out, number of participants, mean participant age, study design, VAP prevalence, cutoff value used for the index test, and the number of true positive, true negative, false negative, and false positive VAP diagnoses, from which we constructed 2-by-2 contingency tables.
Two of us independently assessed the studies' quality, with the Quality Assessment of Diagnostic Accuracy Studies (QUADAS) tool, in which each assessed item receives a “yes”, “no”, or “unclear.”14 Disagreements were resolved by a third reader.
Statistical Analysis
We conducted the meta-analysis with meta-analysis software (MetaDisc 1.4, BMC Medical Research Methodology, http://www.hrc.es/investigacion/metadisc_en.htm).15 We assessed the performance of CPIS with summary receiver operating characteristic curve analysis, diagnostic odds ratio, and pooled sensitivity and specificity and their 95% confidence intervals. The likelihood ratios express the relative odds for occurrence of a specific test combination in a patient with VAP as opposed to patients without VAP; a test that performs well has a high likelihood ratio for a positive index test (ie, > 1) and a low likelihood ratio for a negative index test (ie, < 1). The diagnostic odds ratio is the ratio of the odds of a positive result in a patient with VAP compared to a patient without VAP. Thus, combining both likelihood ratios in a single ratio, the higher the ratio, the better the test. The diagnostic odds ratio is a measure of overall accuracy and has the advantage of allowing the inclusion of covariates to examine heterogeneity in a regression model.15 All summary estimates were obtained by the method of a random effect model (the DerSimonian and Laird procedure).
We characterized the performance of a diagnostic test based on the results of multiple studies with the summary receiver operating characteristic curve. Each point represents a single study; the size of the point represents the weight given to the study (depending on the number of patients). The area under the curve represents the test's ability accurately to distinguish disease from no disease; an area under the curve of 1.0 would represent perfect discriminatory ability, whereas area under the curve of 0.5 would indicate no discriminatory power. The Q index is the intersection between a symmetrical summary receiver operating characteristic curve and the antidiagonal line, at which sensitivity equals specificity.16
To determine the heterogeneity, the likelihood ratios and diagnostic odds ratios are graphed as forest plots and analyzed with the Cochran Q test. A P value of < .05 via the Cochran Q test, was taken as significant heterogeneity. The I2 statistic was used to assess the extent of heterogeneity among the diagnostic odds. I2 values of > 75% were considered high heterogeneity.17 We also conducted threshold analyses (Spearman rank correlation, r) to assess for a threshold effect among the studies. We explored the possible influence of clinical subgroups with stratified analysis in the regression analysis.
Results
Study Selection
Our literature search identified 19 papers for review; the 2 articles in the Cochrane database were redundant. We excluded 5 studies based on the title and/or abstract. Fourteen articles were obtained for full review, and 4 met the inclusion criteria. One study was excluded because it did not provide the information required to generate the 2×2 table, and 9 others were excluded because they were irrelevant to the subject. Our manual search of the references lists of the studies identified in the electronic search identified 11 additional studies, 2 of which had insufficient data to evaluate the study quality. Thus, 13 studies, which included 1,264 patients, were included in the meta-analysis (Fig. 1).18–30
Study selection process. QUADAS = Quality Assessment of Diagnostic Accuracy Studies.
All the studies identified were published between 1999 and 2010. The overall prevalence and mortality of VAP in the included studies were 48% and 37%, respectively. Ten of the 13 included studies used BALF cultures as the VAP diagnosis reference standard, one used protected telescoping catheter cultures, one used autopsy findings, and one used lung biopsy cultures (Table 1).
Included Studies
Figure 2 shows the study quality analysis. Quality factors that were poorly reported include selection criteria, diagnostic review bias, test review bias, and explanation of withdrawals. These are potentially important sources of bias.
Methodological quality assessment.
Diagnostic Accuracy of CPIS for Diagnosis of VAP
The overall pooled sensitivity of CPIS for diagnosing VAP was 65% (95% CI 61–69%, I2 = 89.6%). Overall CPIS specificity for diagnosing VAP was 64% (95% CI 60–67%, I2 = 89.5%). Figure 3 shows the pooled sensitivity and specificity forest plots. The combined likelihood ratios for positive index test and negative index test were 1.94 (95% CI 1.44–2.61) and 0.46 (95% CI 0.32–0.66), respectively, which yielded a diagnostic odds ratio of 4.85. There was no evidence of a threshold effect (r = −0.082, P = .79).
Sensitivity (A), specificity (B), and diagnostic odds ratio (C) of the clinical pulmonary infection score (CPIS) for diagnosing ventilator-associated pneumonia. The circle sizes are proportional to the sample sizes. D: Summary receiver operator characteristic curve for all the studies. The circles represent the individual study estimates of sensitivity and 1–specificity, and the circle sizes are proportional to the inverse variance of each study.
Table 2 shows the CPIS diagnostic performance for VAP in the included studies. The sensitivity range was 30–97% and the specificity range was 42–100%. Figure 3C shows the forest plot of the diagnostic odds ratio. Twelve of the 13 studies are plotted as the confidence intervals of the odds ratio; the 2008 study by Pelosi et al19 had an infinite odds ratio value.
Decision Analysis Data for Clinical Pulmonary Infection Score
Figure 3D shows the summary receiver operating characteristic curve. The curve lies to the left of the diagonal, which signifies that the CPIS has diagnostic value. The area under the curve is 0.748, and the Q* point is 0.69, which indicates that the CPIS has modest diagnostic accuracy for VAP.
Meta-Regression
As expected, pooling all the studies introduced significant heterogeneity (Cochran Q = 67.85, P < .001). We included factors (prevalence, cutoff value, study design, study population, blind, age, study sample, and reference standard) that could be extracted from the included studies in the meta-regression to identify the source of this heterogeneity. Prevalence was a significant source of heterogeneity (meta-regression coefficient −6.397, P = .02). Other variables were not statistically significant sources of variability: cutoff value P = .24, age P = .49, study design P = .49, study population P = .86, blind P = .96, study sample P = .75, reference standard P = .86.
The prevalence-based subgroup analysis showed a change to the shape of the summary receiver-operating characteristic curve when we included only the studies in which prevalence was < 50%. The area under the curve was 0.868 (95% CI 0.75–0.98).
Discussion
There is poor agreement among clinicians on diagnosing VAP, whether clinical, pathologic or histologic plus culture of the lung tissue. To our knowledge, this is the first meta-analysis on the accuracy of CPIS in diagnosing VAP. The area under the summary receiver operating characteristic curve was 0.748, which indicates that CPIS has a modest ability to diagnose VAP.
The diagnostic tests for VAP were recently reviewed by Rea-Neto and colleagues,31 but they could draw no conclusion about CPIS because the study did not assess the pooled estimates and heterogeneity between studies, thus making it difficult to interpret its findings. A similar meta-analysis32 assessed the invasive approaches to diagnosing VAP, but the aim of that study was to evaluate the impact of invasively obtained cultures on VAP outcomes. Invasive lower-airway sampling changed antibiotic management but did not alter mortality. In view of the existing studies and their limitations, we systematically explored the pooled estimates of CPIS and heterogeneity via meta-regression and subgroup analysis to determine whether CPIS performed differently in the low-prevalence (< 50%) studies. The subgroup analysis yielded significantly different estimates of test performance from the overall results. All enhanced the validity and applicability of our findings.
There is a need for an accurate, easy, and inexpensive reference standard diagnostic test for VAP. Though our results indicate that CPIS has only moderate diagnostic performance for VAP, the overall pooled sensitivity (65%) and specificity (64%) is only slightly lower than that of quantitative sample culture (sensitivity 64%, specificity 83%).33 And the use of quantitative culture data as the sole criterion to diagnose VAP should not be routine, because that strategy is not adequately validated. These data may help to refine the clinical suspicion. The importance of microbiological diagnosis of VAP rests not in determining whether the patient has VAP, but rather in optimizing antimicrobial treatment. They should be part of—rather than in preference to—a clinical assessment.1
Limitations
Our comprehensive search strategy, strict selection criteria, and rigorous statistical methods ensured the good quality of this study, but there are limitations to our meta-analysis. The populations studied were from western countries. A more detailed breakdown of the races of the study populations would have provided us more information on a source of heterogeneity. The meta-regression was probably seriously underpowered because of the small number of studies included, and might be a reason why only one factor was positive. In addition, the studies examined were few and had wide differences and small sample sizes.
A major limitation of the literature on CPIS for diagnosing VAP is that BALF culture is not a true accepted standard.19–20,22,24–30 Moreover, the calculation of CPIS was modified by some authors, and different cutoff points were used to diagnose VAP. Importantly, the inter-observer variability affected the sensitivity and specificity.34
A potential limitation of any meta-analysis is the possibility of publication bias. Although we cannot be certain that we retrieved all published studies, we are confident that we obtained additional studies from as many databases as possible. Because our reviewers can read only English literature, we excluded some studies published in other languages, which could have been a source of bias. Currently available statistical approaches for publication bias (eg, funnel plots and regression tests) are not recommended for diagnostic meta-analysis,35 so we cannot rule out potential publication bias in our meta-analysis.
Besides the value of the CPIS as a diagnostic tool, some researchers have proposed that it offers value as a marker of prognosis and a means for preventing antibiotic overuse. Huang et al36 studied early predictors of poor outcomes in patients with VAP and found that the Acute Physiology and Chronic Health Evaluation II score and CPIS at the time of VAP diagnosis were significantly higher in the mortality group. Serial determinations of the CPIS during the course of VAP could be a prognostic factor as early as day 3 of therapy, but it may not predict the outcome of VAP at VAP onset. The CPIS has been most successfully used in guiding treatment decisions for patients with a low likelihood of VAP, for whom CPIS-guided therapy lowered costs and reduced the development of antimicrobial resistance.8,37
Conclusions
The moderate diagnostic performance of CPIS in diagnosing VAP suggests the need to improve its diagnostic efficacy, perhaps by combining CPIS with other VAP diagnosis methods and biological markers. This may be the next step of research and needs well designed, high-powered studies. CPIS has been helpful in guiding appropriate antibiotic treatment. The value of the CPIS is definite, but its usefulness for predicting VAP outcomes also needs further study.
Footnotes
- Correspondence: Jian-Hua Zhu MD, Department of Internal Medicine, Affiliated Hospital Of Nantong University, Xi Si Road 20# Nantong City, Jiangsu Province, 226001 People's Republic of China. E-mail: shjun_2008{at}163.com.
-
This research was supported by the social development fund of Nantong science and technology commission, and Nantong University's fund for technology innovation project for post-graduate students.
-
The authors have disclosed no conflicts of interest.
-
See the Related Editorial on Page 1209
- Copyright © 2011 by Daedalus Enterprises Inc.
