Abstract
BACKGROUND: Pediatric Asthma Assessment tools used to guide the weaning of inhaled therapies during inpatient hospitalization require further evaluation and validation. This study aimed to compare 2 asthma assessment tools: an asthma scale versus an asthma score.
METHODS: A prospective, physician-blinded, comparison study was conducted in 2 separate 6-week phases of patients > 2 y old admitted to a tertiary care children's hospital with status asthmaticus between July and November 2014. The asthma scale categorized 5 components (oxygen, auscultation, dyspnea, breathing frequency, and pulse oximetry) into 1 of 3 respiratory assessments: mild, moderate, or severe. The asthma score used a sum of the components, resulting in a score of 1–15. Study tool predictability was measured using a metric based on hours on continuous albuterol, with area under the curve ≥ 0.8 indicating good predictability. Agreement between clinicians was measured using the Cohen kappa statistic. Study tool clinical correlation was measured using Spearman coefficient. Usability was evaluated using web-based surveys.
RESULTS: Phase 1 included 1,971 assessments (97 unique subjects), whereas phase 2 included 607 assessments (69 unique subjects). Using the continuous albuterol metric, predictability of the asthma scale had an area under the curve of 0.62 versus the asthma score area under the curve of 0.80. Agreement early in hospitalization for the asthma scale was kappa = 0.34 (95% CI 0.18–0.5; n = 84) versus kappa = 0.55 (95% CI 0.35–0.76; n = 44) for the asthma score. Agreement late in hospitalization for the asthma scale was kappa = 0.38 (95% CI 0.17–0.59; n = 66) versus kappa = 0.41 (95% CI 0.13–0.69; n = 33) for the asthma score. Clinical correlation for the asthma scale (no. = 1,908) was r = 0.57 (P < .001) versus r = 0.80 (P < .001) for the asthma score (no. = 558). Mean asthma scale usability was 3.38 versus 3.68 for the asthma score.
CONCLUSIONS: The asthma score showed better clinical predictability and clinical correlation compared to the asthma scale. Numerical scores provided more objective assessments compared to categorical scores. Validated scoring tools such as the asthma score are crucial to the success of management of inpatient asthma care.
Introduction
Over the past 2 decades, both the prevalence and the hospital admission rates for asthma in children have increased.1 In the United States, medical costs of asthma are estimated to be $50 billion annually, with hospitalization accounting for nearly 50% of costs.1,2 In addition, variation in care delivery is common with regard to methods of reliever medication delivery,3–9 the frequency of albuterol dosing in the emergency department and in-patient settings,10–14 and asthma education.13
Given this variation, numerous studies have evaluated treatment protocols that streamline care for patients with asthma receiving albuterol.10,14,15 Asthma treatment protocols have also shown significant decrease in stay and total costs.11,12,16 Key to the success of these protocols is the ability of respiratory therapists and nurses to wean a patient's albuterol use based on objective asthma assessment tools, thereby avoiding treatment delay if physicians are not immediately present. Standardized assessment tools and scoring systems are crucial to decreasing the frequency of albuterol administration and effectively moving patients along their treatment regimen.17–25 However, in their analysis of 10 different asthma scoring tools in 2004, Birken et al18 reported that each tool used different assessment markers for asthma severity (eg, breathing frequency, accessory muscle use) and different descriptors of severity. For example, current National, Heart, Blood, and Lung Institute (NHLBI) guidelines recommend assessing a patient as mild, moderate, or severe and adjusting therapy accordingly, whereas many of the most commonly used asthma tools provide numerical scores of severity.13 In addition, while many of these assessment tools were being used in practice, few have been clinically validated. Prior to developing an asthma weaning protocol driven by respiratory therapists or nurses, we needed to determine the asthma assessment tool we would use.
The aims of this study were to compare and evaluate, in a head-to-head manner, 2 tools for assessing pediatric patients with asthma in an in-patient setting, an asthma scale and an asthma score. This prospective study also examined (1) the ability of initial assessment using each tool to predict total length of hospital stay and hours on continuous albuterol; (2) the agreement for each tool (between respiratory therapists and nurses); (3) the clinical correlation between assessment tools and nursing and respiratory therapists' clinical judgment; (4) the usability of each tool.
QUICK LOOK
Current knowledge
Respiratory therapist- or nurse-driven asthma treatment protocols have resulted in significant decreases in length of stay and total costs for patients hospitalized with asthma. Key to the success of these protocols is the use of objective, validated asthma assessment tools.
What this paper contributes to our knowledge
For evaluation of in-patient subjects with asthma, the asthma score was superior to the asthma scale in terms of predictive ability, agreement, clinical correlation, and usability, likely due to its use of a numbering system rather than a categorical scale. The objectivity of using a numerical scale is vital for implementation of severity assessment tools within a protocol driven by respiratory therapists or nurses to effectively manage in-patients with asthma.
Methods
Study Design
This prospective, provider-blinded study evaluated 2 asthma severity assessment tools, an asthma scale and asthma score, with in-patients at an urban, tertiary, free-standing children's hospital, from July to November 2014. We included all children over the age of 2 y who were admitted to the observation unit, the in-patient floor, or the ICU with a primary diagnosis of asthma exacerbation or presenting with their third lifetime episode of wheezing and requiring treatment with an inhaled bronchodilator. The exclusion criteria were patients with any of the following comorbid conditions: chronic lung disease, bronchopulmonary dysplasia, cystic fibrosis, airway anomalies (eg, tracheomalacia), bronchiolitis, pneumonia, stridor on exam, croup, cardiac disease, tracheostomy or ventilator dependence, respiratory failure requiring bi-level positive airway pressure or mechanical ventilation, neuromuscular disease, or metabolic disease. Prior to data analysis, study investigators reviewed all patient charts and removed patients who were not eligible. Demographic data were recorded for all subjects. Asthma severity status, when documented, was collected from the medical record with a plan to analyze this subset separately.
Staff recruited to participate in the study included all respiratory therapists and nurses who administered inhaled bronchodilators to these subjects. Staff received an information letter regarding the study, which outlined their involvement and how to opt out. Clinical care was not affected by the study. Additionally, there was no difference in the type of nebulizer or inhaler device used throughout the study period. Instead, subject status was evaluated with the 2 asthma severity assessment tools without provider involvement. Given that there was no change in patient care, along with the widespread use of bronchodilators and the number of admissions for status asthmaticus, and to be able to capture an adequate sample size, informed consent was waived for participating staff and pediatric subjects. The Institutional Review Board certified this study.
Asthma Severity Assessment Tool Development
In May 2014, the Asthma Improvement in Metrics Team at the children's hospital developed 2 asthma severity tools (an asthma scale and an asthma score) based on a review of the literature and existing tools from peer institutions. These tools were designed for use by bedside respiratory therapists or bedside nurses to manage albuterol therapy for in-patients. Both tools evaluate patients with asthma based on the same 5 parameters: breathing frequency, dyspnea, auscultation, oxygen requirement, and accessory muscle use. Further refinement of these tools prior to use included updating the breathing frequency norms26 and using oxygen requirement as a clinical parameter due to the concerns of variation in interpreting blood oxygen saturation. These refinements were not found in any existing tool in the literature, which necessitated the development of the tools studied here. The asthma scale categorized these 5 parameters into 1 of 3 respiratory assessments: mild, moderate, or severe. The asthma score assigned a numeric value to each parameter, which added up to a cumulative score of 1–15 (Table 1, Table 2).
Subject Assessment
Each asthma severity assessment tool was evaluated independently in a 6-week phase. The asthma scale was evaluated during phase 1 and the asthma score during phase 2. All participating nursing and respiratory therapy staff were educated regarding the asthma severity assessment tool prior to the start of each phase. In each phase, all subjects who met inclusion criteria underwent assessments of asthma severity hourly for subjects on continuous albuterol or every 2, 3, or 4 hours for subjects on intermittent albuterol according to their albuterol schedule. The staff used the designated tool for these assessments, which were printed with 5 instructional points: (1) Circle assessments in the scale or score table. (2) Circle if assessment was pre- or post-albuterol administration, or if it is was during a continuous albuterol administration. (3) Write the date and time of the assessment. (4) Assessment severity: for the scale, circle “mild, moderate, or severe” based on the highest level assessed in the tool; for the score, add up the 5 categories circled in the tool and write the total score. (5) Clinical judgment: for the scale, circle clinical agreement or disagreement with the tool's assessment; for the score, circle the level of respiratory distress the subject clinically had: none, mild, moderate, or severe. Assessments could be conducted on admission, before scheduled albuterol treatments, or hourly for subjects on continuous albuterol. Staff members were encouraged to complete tool assessments as often as possible during a subject's hospital course. To evaluate agreement between each tool, staff were encouraged to have 2 providers document their results using the same asthma severity assessment tool on the same subject and at the same time as often as possible. Respiratory therapists and nurses were blinded to each other's assessment tool results by placing the completed tools into a designated sealed box. To evaluate clinical correlation with the severity assessment tool results, each staff member was asked to indicate their clinical judgment in addition to completing the asthma severity assessment tools (instructional point 5 above). At the end of each phase, all nurses and respiratory therapists who participated in the study were asked to complete an anonymous web-based survey regarding the usability of each asthma severity assessment tool.
Subjects in the study received standard of care in all aspects of their management. Physicians continued to prescribe medications and therapies based on their clinical judgment and were blinded to the asthma severity assessment tool results documented by the respiratory therapists and nurses.
Statistical Analysis
For each asthma severity assessment tool, we evaluated predictive ability, agreement between nursing and respiratory staff, clinical correlation, and usability. To evaluate predictive ability, we generated receiver operating characteristic curves to determine the ability of the first assessment using the tools to predict length of stay and hours on continuous albuterol because hours on continuous albuterol is a common local clinical parameter used to assess severity. Length of stay was stratified by the median value in phase 1 (median = 2 d). Hours on continuous albuterol was stratified by < 6 h and ≥ 6 h because 6 h was a frequent time parameter that patients remained on continuous albuterol prior to the first wean attempt at our institution. The first assessment was defined as the assessment with the earliest date and timestamp noted on the asthma severity assessment tool. Assessments without date or time stamps were excluded. An area under the curve ≥ 0.80 was defined as good predictive ability. Phase 1 and phase 2 groups were compared using chi-square or Fisher exact tests (as appropriate) for categorical variables. Age was compared using a 2-sample t test.
To analyze agreement between paired assessments, only assessments by 2 staff members for the same subject at the same time were evaluated. This was done both early in the hospitalization (ie, Time 1) and closer to discharge (ie, Time 2). Time 1 (early) was defined as the first pair of assessments for a subject within 20 min of each other that were both marked as either continuous or pre-albuterol administration. Time 2 (late) was defined as the last pair of assessments close to hospital discharge for a subject within 20 min of each other that were both marked as either pre-albuterol administration or post-albuterol administration. The Cohen kappa statistic was used to assess agreement between pairs of assessments.
Spearman correlation coefficients were used to evaluate clinical correlation between the assessment tools and nursing and respiratory therapists' clinical judgment at an early time point and a late time point. For the asthma score, the study investigators defined mild, moderate, and severe prior to the study as follows: 0–3 (mild), 4–7 (moderate), and ≥ 8 (severe) via a modified Delphi method of potential patient cases. Respiratory therapists and nurses were blinded to this categorization.
Usability was evaluated with 5-point Likert scale surveys for each assessment tool. Questions were related to the overall tool usability, anticipated need for assistance to use the tool, time needed to complete the tool, consistency within the tool, confidence in the tool's correlation to clinical status, and need for education prior to using the tool.
Feedback from clinical staff was collected. Initial results were reviewed and further subanalysis was conducted to further evaluate 2 metrics of the asthma score (ie, dyspnea and breathing frequency) by removing them from the scoring tool and reassessing predictability, agreement, and clinical correlation. Statistical analysis was performed with SAS 9.4 (SAS Institute, Cary, North Carolina).
Results
Subject Demographics
During phase 1, 1,971 asthma scale assessments were completed on 97 unique subjects (mean age 6.6 ± 3.7 y). During phase 2, 607 asthma score assessments were conducted on 69 unique subjects (mean age 6.7 ± 3.2 y). There were no significant differences between the 2 groups based on age, sex, race, or ethnicity (Table 3). Asthma severity designation via data extraction was available for 40 subjects in phase 1 and for 26 subjects in phase 2, with 18% versus 23% of subjects having moderate-severe asthma in phase 1 and phase 2, respectively (P = .80). The lengths of stay in phase 1 and phase 2 were both 2.2 d (P = .92).
Tool Evaluation
The predictive ability of each assessment tool to predict hours of continuous albuterol resulted in an area under the curve of 0.62 for the asthma scale versus 0.80 for the asthma score (Table 4). Agreement was assessed early in hospitalization (Time 1) for 84 paired assessments available for the asthma scale with a Cohen kappa of 0.34 (95% CI 0.18–0.5) versus 44 paired assessments for the asthma score with a Cohen kappa of 0.55 (95% CI 0.35–0.76). Agreement was assessed late in hospitalization (Time 2) for 66 paired assessments for the asthma scale with a Cohen kappa of 0.38 (95% CI 0.17–0.59) versus 33 paired assessments for the asthma score with a Cohen kappa of 0.41 (95% CI 0.13–0.69).
Clinical correlation for the asthma scale was r = 0.57 (no. = 1,908, P < .001) and r = 0.80 (no. = 558, P < .001) for the asthma score. Early in hospitalization (Time 1), severity correlation with clinical status for the asthma scale was r = 0.50 (n = 92, P < .001) and r = 0.83 (n = 62, P < .001) for the asthma score. Late in hospitalization (Time 2), severity correlation analysis for the asthma scale was r = 0.66 (n = 84, P < .001) and r = 0.69 (n = 47, P < .001) for the asthma score.
After the completion of phase 1, 76 staff members filled out the usability survey for the asthma scale, with mean survey score of 3.38 (68%). For the asthma score, 42 staff members completed surveys, with a mean survey score of 3.68 (74%).
Subanalysis of the Asthma Score
The asthma score demonstrated higher predictive ability, agreement, and clinical correlation compared with the asthma scale. As a result, a subanalysis of the individual metrics of the asthma score was conducted. Predictive ability of the asthma score using hours on continuous albuterol showed an area under the curve of 0.80 when the dyspnea metric was removed, 0.78 when the breathing frequency metric was removed, and 0.79 when both the dyspnea and breathing frequency metric were removed (Table 5). Removing dyspnea resulted in an improved agreement, with a Cohen kappa of 0.64 (95% CI 0.44–0.84) early in the hospitalization and a Cohen kappa of 0.63 (95% CI 0.50–0.75) late in hospitalization. The final assessment tool was modified to a 4-metric score (Table 6) without dyspnea.
Discussion
The aim of this study was to compare and evaluate 2 tools for assessing pediatric subjects with asthma in an in-patient setting: an asthma scale versus an asthma score. Our analysis indicates better predictive ability, agreement, clinical correlation, and usability with the asthma score. Additional subanalyses allowed for further tool refinement with the removal of the dyspnea metric by enhancing agreement for the asthma score without negatively affecting the predictive ability or clinical correlation metrics. While the current NHLBI guidelines recommend assessing a patient as “mild,” “moderate,” or “severe” to drive in-patient management,13 our results suggest that numerical scores provide a more accurate and usable means of assessing patients and guiding management.
Prior studies have demonstrated variability in how asthma assessment tools correlate with severity of illness.20,22,24 With this in mind, we evaluated these tools at 2 time points, early and late in hospitalization, to assess variation throughout the asthma exacerbation. We analyzed predictive ability of the assessment tool with 2 metrics, length of stay and continuous albuterol use for ≥ 6 h. While past studies have used length of stay as a clinical indicator of severity,19,23 this metric is highly influenced by multiple factors at our institution, as noted in other studies.10 Therefore, we evaluated continuous albuterol use as ≥ 6 h because clinical practice suggests it is a better indicator of asthma exacerbation severity.19 Our analysis demonstrated poor predictive ability across both metrics for the asthma scale. The asthma score demonstrated poor predictability for length of stay but good predictability regarding hours on continuous albuterol (area under the curve = 0.80). Both tools showed significant correlation with clinical status at the early and later hospitalization time points.
The agreement analysis suggested poor agreement for the asthma scale and fair to moderate agreement for the asthma score. Of note, in comparing the assessments made at the early and late time points, the 95% CIs overlap, indicating that, while there is some variation in the kappa statistics across time points, these differences may not be statistically significant.
There were several limitations to this study. First, this was a single-center study in an urban setting, so our findings may not be generalizable to other institutions. Second, because the asthma score was evaluated second and the scale first, it is possible that the improved results of the asthma score were to some extent secondary to learning and practicing with the asthma scale. Also, due to incomplete documentation, we were able to obtain asthma severity data from the medical record for less than half of the subjects included in this study. Therefore, we cannot comment on overall asthma severity of our patient population or if there were significant differences between asthma severity for the 2 phases of this study. This limits generalizability given that asthma severity at our institution may vary from other institutions and may impact overall findings if asthma severity was significantly different between phases. However, because our phases were 6 weeks apart during the same calendar year, there is no clinical reason to assume there would be differences in subject's baseline asthma severity. Additionally, we did not specifically analyze the bronchodilator doses and frequency between the 2 phases. However, because clinical care was not affected by the study, there is no reason to assume there were any differences between the 2 back-to-back study periods. In addition, while our study relied on a different clinical indicator of severity compared to past studies, we believe the hours on continuous albuterol variable it is a better and more clinically relevant indicator of asthma exacerbation severity.
Moreover, we chose to evaluate our tools at 2 time points, early and late in the hospitalization, excluding assessments without timestamps. By excluding assessments without dates or timestamps, we risk sampling on assessments that were not representative of the earliest or latest assessments in the hospitalization, which may have impacted overall severity of illness for the 2 time points. Finally, because the assessments were collected voluntarily by nurses and respiratory therapists filling out the asthma tool within 20 min of each other, the number for our paired analysis was smaller than would be expected given our total patient volumes during the study. Despite these limitations, our sample size19 and subject age range18 were larger than many other published studies evaluating asthma severity tools.23
Conclusions
Overall, the asthma score was better than the asthma scale in terms of predictive ability, agreement, clinical correlation, and usability. We conclude that the use of a numbering system in a score rather than a categorical scale provides for better objective evaluation of in-patients with asthma. The objectivity from a numerical scale is vital for implementation of severity assessment tools within a respiratory therapist– or nurse-driven clinical protocol to effectively manage in-patients with asthma. Future studies should explore how asthma severity tools correlate with and impact physician decision-making and how agreement changes over time, in varying phases of clinical illness, and with more extensive use of and familiarity with asthma severity assessment tools.
Footnotes
- Correspondence: Waheeda Samady MD MSCI. E-mail: wsamady{at}luriechildrens.org
- Copyright © 2021 by Daedalus Enterprises