Abstract
BACKGROUND: Twenty-five to 40% of patients pass a spontaneous breathing trial (SBT) but fail to wean from mechanical ventilation. There is no single appropriate and convenient predictor or method that can help clinicians to accurately predict weaning outcomes. This study designed an artificial neural network (ANN) model for predicting successful extubation in mechanically ventilated patients.
METHODS: Ready-to-wean subjects (N = 121) hospitalized in medical ICUs were recruited and randomly divided into training (n = 76) and test (n = 45) sets. Eight variables consisting of age, reasons for intubation, duration of mechanical ventilation, Acute Physiology and Chronic Health Evaluation II score, mean inspiratory and expiratory times, mean breathing frequency, and mean expiratory tidal volume in a 30-min SBT (pressure support ventilation of 5 cm H2O and PEEP of 5 cm H2O) were selected as the ANN input variables. The prediction performance of the ANN model was compared with the rapid shallow breathing index (RSBI), maximum inspiratory pressure, RSBI at 1 min (RSBI1) and 30 min (RSBI30) in an SBT, and absolute percentage change in RSBI from 1 to 30 min in an SBT (ΔRSBI30) using a confusion matrix and receiver operating characteristic curves.
RESULTS: The area under the receiver operating characteristic curves in the test set of the ANN model was 0.83 (95% CI 0.69–0.92, P < .001), which is better than any one of the following predictors: 0.66 (95% CI 0.50–0.80, P = .04) for RSBI, 0.52 (95% CI 0.37–0.67, P = .86) for maximum inspiratory pressure, 0.53 (95% CI 0.37–0.68, P = .79) for RSBI1, 0.60 (95% CI 0.44–0.74, P = .34) for RSBI30, and 0.51 (95% CI 0.36–0.66, P = .91) for ΔRSBI30. Predicting successful extubation based on the ANN model of the test set had a sensitivity of 82%, a specificity of 73%, and an accuracy rate of 80%, with an optimal threshold of ≥0.5 selected from the training set.
CONCLUSIONS: The ANN model improved the accuracy of predicting successful extubation. By applying it clinically, clinicians can select the earliest appropriate weaning time.
- artificial neural network
- weaning prediction
- rapid shallow breathing index
- spontaneous breathing trial
- airway extubation
- receiver operating characteristic curve
Introduction
More than 39% of ICU patients require endotracheal intubation with ventilatory support.1 Choosing the appropriate time for weaning a patient from mechanical ventilation and extubation is crucial for reducing the risks of prolonged ventilatory support and premature weaning.2–4 Various weaning predictors based on breathing pattern parameters have been used to assess weaning preparedness.5–9 The rapid shallow breathing index (RSBI)3,10 and the outcomes of a spontaneous breathing trial (SBT) with either a T-piece or low-level pressure support ventilation (PSV)11 are commonly used clinical methods. However, various thresholds and sensitivities of the RSBI exist among different patient populations and measurement conditions,12–14 and 25–40% of patients who pass an SBT develop failure signs after being weaned from mechanical ventilation.10,15,16 Weaning patients from ventilatory support is a complex issue.17 Until now, no single appropriate and convenient predictor or method could be used satisfactorily to help clinicians predict weaning outcomes.6,18
Medical decision-support systems based on an artificial neural network (ANN) have been developed in different medical areas.19 ANNs are computer-based algorithms that mimic the habits and structures of neurons. The most common application type of ANNs is a multilayer perceptron, which includes an input layer, an output layer, and at least one hidden layer.20 Each layer consists of several perceptrons (or neurons), and each perceptron between layers is connected by weights that are capable of autonomous adjustments and learning directly from experience during the training phases. The perceptrons then output an optimal result for solving nonlinear problems in the manner of the human brain.20,21 Because the training behavior of ANNs has yet to be satisfactorily explained, ANNs are often called black boxes. Therefore, sensitivity analysis or cross-validation has been used to explain how ANN models work.20 Gottschalk et al22 were the first to report that an ANN model that had been trained by using 4 variables (ie, tidal volume [VT], minute ventilation, breathing frequency, and maximum inspiratory pressure [PImax]) recorded during an SBT in ICU subjects could be as effective as experts in predicting whether patients could be successfully weaned from mechanical ventilation. Mueller et al23 demonstrated that their ANN model outperformed clinical expertise and multiple logistic regression in predicting extubation outcomes in premature newborns. However, other studies presented poor results using ANN models to predict extubation outcomes in different respiratory care facilities.7 The main reasons for poor outcome prediction may be due to the differences in clinical input data and patient populations.
The purpose of this study was to develop a medical decision-support ANN model for clinicians in making extubation decisions. The model contains variables selected from a literature review with consideration for the convenience of data collection and involves subjects' characteristics and respiratory pattern parameters during SBTs. The optimal number of perceptrons in the hidden layer was determined by using cross-validation. The ability to predict successful extubation in ICU subjects with the final ANN model was compared with RSBI, PImax, RSBI at 1 min (RSBI1) and 30 min (RSBI30) in an SBT, and absolute percentage change in RSBI from 1 to 30 min in an SBT (ΔRSBI30) by analyzing the receiver operating characteristic (ROC) curves.
QUICK LOOK
Current knowledge
Daily spontaneous breathing trials (SBTs) along with interruption of sedation is the current evidence-based standard of care for facilitating ventilator discontinuation. However, extubation failure remains between 10 and 20% in most trials. Methods to predict success remain an important research focus.
What this paper contributes to our knowledge
In a small group of subjects, an artificial neural network (ANN) developed based on subjects' characteristics and breathing pattern variables improved the accuracy of predicting successful extubation compared with the rapid shallow breathing index or maximum inspiratory pressure. The ANN used subjects' age, reasons for intubation, duration of mechanical ventilation, APACHE II scores, and breathing patterns during a 30-min SBT. The ANN had a sensitivity of 82%, a specificity of 73%, and an accuracy rate of 80%.
Methods
Study Population
All protocols were approved by the institutional review board of the 2 hospitals that we investigated in Taipei, Taiwan, and written informed consent was obtained from the study subjects or their next of kin. A total of 136 intubated, mechanically ventilated, and ready-to-wean subjects3,10 hospitalized in a medical ICU over a 13-month period were included in this study. The subjects exhibited no acute pulmonary or neuromuscular problems or signs of increased intracranial pressure before the study, and their SpO2 was ≥95%. In addition, subjects had to meet the following routine measurements of weaning profiles24–27: bed position elevated to 30°, PImax ≤ −18 cm H2O, spontaneous breathing frequency ≤ 38 breaths/min, and RSBI ≤ 110 breaths/min/L. All subjects were mechanically ventilated with a Evita 4 (Dräger, Lübeck, Germany) or Galileo Gold (Hamilton Medical, Reno, Nevada) ventilator in the PSV mode to maintain a VT of ∼10 mL/kg of ideal body weight. Other ventilator settings were FIO2 ≤ 0.4, PEEP = 5 cm H2O, flow sensitivity setting = 2–3 L/min, and rise time = 0.1- 0.2 s. Sedatives, hypnotics, and narcotics were discontinued for at least 8 h.
When the subject's primary physician made the extubation decision, an arterial blood gas or SpO2 was collected, and the ventilator mode was changed to 30 min of PSV at 5 cm H2O with PEEP at 5 cm H2O in an SBT within 1 h. A computerized pulmonary mechanics monitoring system (CO2SMO+ Model 8100 [Novametrix Medical Systems, Wallingford, Connecticut] or Ventilator Datalogger 3.27.1 [Hamilton Medical, Reno, Nevada]) was used to continuously record the flow, pressure, and volume signals during the trials. The trial was terminated, and the ventilator was returned to the previous level of pressure support if the subject had one or more signs of cardiopulmonary distress.3,10 When the subject passed the 30-min SBT, an arterial blood gas or SpO2 was collected, and the subject was then extubated and provided with a nasal cannula or air-entrainment mask for oxygen therapy.
Fifteen subjects were excluded from the final analysis: 2 experienced respiratory failure during an SBT, 6 had incomplete data collection, and 7 were not extubated within 24 h after the SBT. Therefore, the data of 121 subjects were analyzed (Fig. 1). Extubation success was defined as a subject free from mechanical ventilation for >48 h after extubation. If a subject exhibited one or more signs of cardiopulmonary distress within 48 h after extubation,10 the subject was reconnected to the ventilator with either noninvasive or invasive ventilation and was considered an extubation failure.
Feature Selections
Through the literature review, we found several potential parameters that can help clinicians predict the outcome of extubation.5,6,8,28,29 Cohen30 observed the clinical data of subjects who had tolerated a 1-h SBT with CPAP at 5 cm H2O and reported that the RSBI measured at the end of the SBT with automatic tube compensation was more sensitive than that measured without compensation. Furthermore, airway pressure and breathing frequency were also significantly different between the extubation failure and success groups at the end of the SBT. Frutos-Vivar et al15 collected clinical data from a multi-center study on subjects who had successfully passed an SBT and analyzed their data from the 24-h period before extubation, comparing the difference between the extubation and failure groups. They concluded that the RSBI, positive fluid balance 24 h before extubation, and diagnosis of pneumonia upon initiation of ventilation were most accurate in predicting extubation failure despite subjects completing the SBT.
Several physiological indexes have been developed to predict weaning outcome, such as RSBI, CROP (compliance, breathing frequency, oxygenation, and pressure) index, PImax, minute ventilation,25 and airway-occlusion pressure 0.1 s after the start of inspiration against an occluded airway.31 Meade6 performed a meta-analysis to compare these physiological indexes in 65 observational studies by using a likelihood ratio, determining that the RSBI is one of the most powerful indexes of successful weaning. To improve the prediction of weaning outcomes, more physiological parameters may be considered. However, studies have shown that it is still controversial to use an arterial blood gas to facilitate elevating the accuracy rate of prediction after subjects have passed an SBT.28,29
In breathing pattern-related research, Casaseca-de-la-Higuera et al5 reported that breathing patterns during SBTs were more irregular in subjects who were eventually successfully extubated than subjects who failed extubation. The sample entropy of VT and the mean breathing frequency during SBTs can help clinicians determine whether patients can be successfully weaned from the ventilator. Other parameters of respiratory signals obtained during SBTs, such as frequency and VT, can also facilitate achieving this purpose.32
After reviewing the potential parameters that may predict extubation outcomes, we considered the convenience of data collection for future application and discarded variables that were not available at the bedside at all times. Eight variables among 3 categories were selected for the input of the ANN: (1) demographic data (age and duration of mechanical ventilation), (2) physiological data (APACHE II [Acute Physiology and Chronic Health Evaluation II] score at the time of ICU admission and reasons for respiratory failure), and (3) breathing pattern obtained during a 30-min SBT (mean frequency, mean inspiratory time, mean expiratory time, and mean expiratory VT) (Table 1). The reasons for respiratory failure were classified into 5 categories: COPD exacerbation, pneumonia, heart failure, neuromuscular diseases, and others.
Structure of the ANN
The ANN prediction model was designed using STATISTICA 8.0 (StatSoft, Tulsa, Oklahoma). The applied architecture was a multilayer perceptron combined with a back-propagation algorithm. The designed ANN model consisted of an input layer containing 17 perceptrons defined according to the dummy variables of the abovementioned 8 input parameters. Two perceptrons in the output layer were used to export the prediction results. Because there was no standard method for determining a fixed perceptron number in the hidden layer,33 the number of perceptrons in the hidden layer was set empirically. To obtain an unbiased estimate of the generalization error and to determine the optimal number of perceptrons in the hidden layer, we used the leave-5-out-cross-validation (L5OCV) method.20 The L5OCV method generated 5 ANN models that had the same number of perceptrons in the hidden layer. For each model and its number of perceptrons in the hidden layer, the median of the area under the ROC curve was used to assess its performance. The model with the largest median of the area under the ROC curve was selected as the final ANN model. Figure 2 shows the architecture of the final ANN model designed in this study.
Statistical Analysis
The Kolmogorov-Smirnov or Shapiro-Wilk test was used to verify the normality of the data. The chi-square or Fisher exact test was used to analyze categorical variables, which were expressed as a frequency or a percentage. The continuous variables were analyzed using a t test or Wilcoxon 2-sample test and expressed as the mean ± SD. The area under the ROC curve was used to compare the final ANN model with that of each RSBI, PImax, RSBI1, RSBI30, and ΔRSBI30 in predicting successful extubation.34 The optimal threshold value for each prediction model was established by analyzing the subjects in the training set, and all of the models were compared using only the test sets. Sensitivity represented subjects who had successful extubations and who had been predicted to be successful, whereas specificity stood for subjects who had unsuccessful extubations and who had been predicted to be unsuccessful. Diagnostic accuracy indicated the ability of the model to predict subject extubation outcome accurately. Data were analyzed using statistical software (SAS 9.4 [SAS Institute, Cary, North Carolina] and MedCalc 14.8 [MedCalc Software, Mariakerke, Belgium]), and P < .05 was considered to be statistically significant.
Results
Subjects' Characteristics
Table 1 presents the physical and clinical characteristics of all subjects studied (N = 121). The diagnosis for initiating mechanical ventilation of the study subjects included COPD exacerbations (16.5%), pneumonia (52.9%), heart failure (18.2%), neuromuscular disease (9.1%), and miscellaneous (eg, upper gastrointestinal bleeding and influenza, 34.7%). All subjects passed a 30-min SBT, but 31 subjects exhibited signs of respiratory failure and had to be reconnected to mechanical ventilation within 48 h after extubation. Compared with the extubation success group (n = 90), subjects in the extubation failure group (n = 31) had significantly lower body mass indexes, longer duration of ventilatory support, longer ICU or hospital stays, and higher pulse rates. In addition, for initiating mechanical ventilation, fewer subjects in the extubation failure group were classified into the miscellaneous category. In the routine measurement of weaning profiles, PImax, RSBI, and RSBI30 were significantly lower in the extubation success group, and VT was significantly lower in the extubation failure group. Regarding breathing pattern parameters during the SBT, subjects in the extubation failure group had significantly lower mean inspiratory times, mean expiratory times, and mean expiratory VT and faster mean breathing frequencies. Percentage change in the RSBI after the SBT yielded no significant difference.
Weaning Prediction of the ANN Model
All recruited subjects were randomly divided into a training set (n = 76) for training the final ANN model and a test set (n = 45) for validating the performance of the model. To prevent over-fitting during model training, a comparison between the training and test data sets regarding ANN variables was performed. As shown in Table 2, there was no significant difference between the training and test sets regarding the ANN input variables. To determine the optimal number of perceptrons in the hidden layer of the ANN prediction model, we developed a series of 5 cross-validated models and calculated the area under the ROC curves for both the training and test data sets based on the hidden-layer perceptron numbers, which ranged from 10 to 39. The median of the area under the ROC curve represented the performance of each ANN model. The ANN model that contained 19 hidden perceptrons had the largest median of the area under the ROC curve (0.82) and was used as the final ANN model, which consisted of 17 perceptrons in the input layer, 19 perceptrons in the hidden layer, and 2 perceptrons in the output layer (multilayer perceptron 17-19-2) (Fig. 3).
The area under the ROC curves in the test set of the ANN model was 0.83 (95% CI 0.69–0.92, P < .001), which is better than any one of the following predictors: 0.66 (95% CI 0.50–0.80, P = .04) for RSBI, 0.52 (95% CI 0.37–0.67, P = .86) for PImax, 0.53(95% CI 0.37–0.68, P = .79) for RSBI1, 0.60 (95% CI 0.44–0.74, P = .34) for RSBI30, and 0.51(95% CI 0.36–0.66, P = .91) for ΔRSBI30 (Fig. 4). Predicting successful extubation based on the ANN model of the test set had a sensitivity of 82%, a specificity of 73%, and an accuracy rate of 80% with the optimal threshold of ≥0.5 selected from the training set. The optimal threshold values selected from the ROC curves of the training set for RSBI, PImax, RSBI1, RSBI30, and ΔRSBI30 and their sensitivity, specificity, and diagnostic accuracy in predicting successful extubation from the test set are listed in Table 3.
Discussion
The challenges of minimally complicated measurement methods for weaning prediction have traditionally involved accuracy. Existing SBTs10,15,16 and weaning parameters,6,25,35,36 such as VT, frequency, minute ventilation, PImax, and RSBI, do not demonstrate a high degree of accuracy in predicting extubation outcomes. In this study, the ANN model was designed according to 8 input variables consisting of subjects' age, reasons for intubation, duration of mechanical ventilation, APACHE II scores, and breathing patterns obtained during a 30-min SBT. The proposed ANN model had better discrimination than existing predictors, such as RSBI, PImax, RSBI1, RSBI30, and ΔRSBI30, in predicting successful extubation.
The SBT has become a routine diagnostic test for determining whether patients can be successfully liberated from a ventilator, and patients' breathing patterns during SBTs have the potential to become weaning predictors.37,38 Because there is no difference in re-intubation or mortality rates for 30- and 120-min SBTs,39 we chose a 30-min SBT for data collection rather than a 120-min trial. Based on our literature review, breathing pattern parameters in an SBT have never been used as inputs of an ANN model. In comparing breathing pattern variables during an SBT with those of other studies,5,40 we observed that mean breathing frequency, mean inspiratory time, mean expiratory time, and mean VT showed statistically significant differences between the extubation success and failure groups, which indicated that the subjects developed a shallow breathing pattern during SBTs. Other studies have reported different results.9 The possible reasons for the difference may be attributed to the definition of extubation failure or the different level of pressure support during SBTs when collecting the breathing pattern parameters. Because of ethical concerns, the subjects in this study had to pass routine weaning profiles before the SBT and be extubated. This may have introduced a selection bias and affected the results when we compared the breathing pattern variables between the 2 groups.
Breathing patterns and their variability could also be influenced by different SBT methods.24 We collected subjects' breathing pattern parameters by conducting an SBT with a low level of pressure support, which could help specific medical ICU patients pass the SBT and also improve the success rate of extubation.41 Moreover, the efficiency and convenience of collecting data directly from a microprocessor ventilator met one of the criteria for an ideal weaning parameter. This facilitated designing a model by using the machine learning method, which may help clinicians to assess whether patients can be weaned from the ventilator quickly in a daily trial.
In this study, subjects in the extubation failure group all passed the routine weaning predictors, SBTs, and the primary physician's assessment. However, 25.6% of the subjects still experienced extubation failure. PImax, VT, and RSBI were found to be significantly different between the extubation failure and success groups. These results indicate that the threshold values of these routinely used predictors were not suitable for the subjects in this study. Therefore, their threshold values must be redefined. No significant difference was found in the percentage change in RSBI after a 30-min SBT, which was different from a previous study.42 The possible reasons for this difference may be attributed to different SBT methods used when collecting the breathing pattern parameters. Breathing pattern during SBTs under minimal ventilatory support (eg, CPAP or PSV) may reduce the predictive performance of breathing pattern variability.24,43
Although PImax, RSBI, RSBI1, RSBI30, and ΔRSBI30 were not used as the input parameters for the ANN model, the prediction performance of the ANN model was more precise than any one of them. This result may indicate that more clinical parameters from the subject should be taken into consideration during an SBT before extubation. The ANN model can help clinicians make the final decision by learning from past experience like the human brain.
Previous studies reported that breathing pattern during SBTs may be a potential marker to predict weaning outcome.5–9 The prediction accuracy rate and area under the ROC curve of the ANN model in this study were similar to those in other studies even when we took breathing pattern variables into consideration.7,22 Different patient populations may have different key factors that determine who may be liberated from the mechanical ventilator. Mueller et al23 observed that gestational age, arterial blood gas, and ventilator settings were critical factors when deciding whether newborns could be successfully extubated. APACHE II, the Glasgow coma scale, RSBI, duration of mechanical ventilation, and other underlying diseases may affect the success rate of extubation in patients who are ventilated for >21 d.7 In this study, we applied 8 variables that were different from those mentioned previously. In addition, breathing patterns in the medical ICU were included as input variables of the ANN model. Based on these differences and the lack of a satisfactory explanation for the training behavior of the ANN model, comparison of the performance of the ANN model with that of previous studies is difficult.
This study has certain limitations. First, no subjects or key factors related to laryngeal edema after extubation were trained in the ANN model. Therefore, the ANN model may not be able to predict extubation failure caused by postextubation laryngeal edema. Second, selection bias could not be avoided because of ethical concerns, which may have caused an underestimation of the specificity in predicting the possibility of weaning failure of the ANN model. Finally, model-generated outcomes that do not provide explanations (eg, black box) constitute a major theoretical concern of the ANN model when applied to clinical studies. However, the success of ANNs in the medical decision-making system, such as high performance in predicting clozapine response44 and pain management,45 suggests that ANNs may be the solution to making complex decisions in clinical practice. Similar to a previous study,23 the L5OCV method was used in this study in an attempt to optimize the ANN model to prevent over-learning and obtain an unbiased estimate of the generalization error. We believe that our study is limited and do not know whether the proposed ANN model developed will perform similarly well in other institutions. However, because the ANN model can learn based on previous data, as a predictive model, it can be easily reconstructed in other institutions for local application by adopting the processes used in this study. When electronic medical records are fully implemented in the future, the proposed ANN model could become an efficient decision-support system.
Conclusions
The proposed ANN model, developed based on subjects' characteristics and breathing pattern variables, improved the accuracy of predicting successful extubation. By applying this ANN model clinically, clinicians can select the appropriate earliest weaning time, which could decrease the risks of unnecessarily prolonged ventilatory support and premature weaning. Therefore, the incidence of patients' complication rates and medical costs related to ventilatory support will decrease.
Acknowledgments
We are grateful to Ms Chao-Yi Wu for help in preparing the manuscript.
Footnotes
- Correspondence: Mauo-Ying Bien PhD PT CRT, School of Respiratory Therapy, College of Medicine, Taipei Medical University, 250 Wu-Hsing Street, Taipei 11031, Taiwan. E-mail: mybien{at}tmu.edu.tw.
This study was supported by grant 96TMU-WFH-16 from the Wan Fang Hospital, Taipei, Taiwan. The authors have disclosed no conflicts of interest.
Mr Kuo presented a version of this paper at the AARC Congress 2012, held November 10–13, 2012, in New Orleans, Louisiana.
- Copyright © 2015 by Daedalus Enterprises