Abstract
BACKGROUND: A respiratory therapy consult service (RTCS) may reduce misallocation of treatments. Misallocation consists of over-ordering (ie, therapy not indicated but ordered) or under-ordering (ie, therapy indicated but not ordered). The rate of agreement with RTCS-based orders is defined as the percentage of patients with no misallocation. This study was undertaken to compare current misallocation and agreement with historical benchmark rates at a hospital with an RTCS (ie, the Main Campus, or “on-MC”) and 2 hospitals that did not have an RTCS (ie, off-Main Campus, or “off-MC”).
METHODS: After approval by the institutional review board, data were collected during normal rounds. A respiratory therapist (RT) determined if the patient had an order for RTCS, what their treatments were, and whether treatment indications were present. RTCS treatments included aerosol therapy, bronchopulmonary hygiene, re-inflation, supplemental oxygen, oxygen monitoring, and suctioning. Agreement and misallocation were compared with chi-square or z-tests with P < .05 indicating significance.
RESULTS: The agreement rate for the RTCS on-MC was less than the benchmark rate established 20 years ago (63% vs 86%, P = .004), ascribed to misallocation of a single therapy, bronchopulmonary hygiene. The agreement rate with the RTCS on-MC was higher than that with off-MC RTCS (63% vs 33%, P < .001). Non-RTCS-based orders on-MC also had higher rates of agreement than orders with off-MC RTCS.
CONCLUSIONS: While the overall rate of agreement was lower with the RTCS currently than in the past, the decline seems solely attributable to a decline in the appropriateness of orders for bronchopulmonary hygiene. In addition, the rate of agreement for non-RTCS-based orders on-MC (71%), where the RTCS has been available for over 20 years, was higher than agreement rate for non-RTCS-based therapies off-MC (20%), where the RTCS has not yet been available. These findings suggest continued efficacy of the RTCS with the need for ongoing vigilance to assure optimal RTCS performance.
- respiratory care protocols
- respiratory therapy consult service
- misallocation
- licensed independent practitioner
Introduction
Clinical practice guidelines (CPGs) are a set of evidence-based practice recommendations intended to ensure optimal, uniform practice across a discipline. On this basis, respiratory care CPGs have been developed and endorsed.1–4 At Cleveland Clinic, a respiratory therapy consult service (RTCS) was developed to enhance the allocation of respiratory care services by licensed independent practitioners, including physicians, nurse practitioners, and physician assistants. The RTCS uses sign- and symptom-based branched logic algorithms to improve the agreement between the licensed independent practitioners' orders and evidence-based indications for each treatment. In non-ICU settings, use of such a consult service led to shorter lengths of stay and lower costs.5 In a randomized controlled trial of the RTCS for in-patient care at Cleveland Clinic in 1998, Stoller et al6 showed that, compared with physician-directed respiratory care, an RTCS was associated with a similar number and duration of respiratory care therapies at a slight savings and without any increase in adverse events, and that, compared with physician-directed respiratory care, RTCS-directed respiratory care showed greater agreement with CPG-based algorithms. Appropriateness of respiratory care orders in that study was assessed as agreement between the ordered respiratory care plan and the plan suggested by a blinded observer who determined a respiratory care plan based on CPGs from the American Association for Respiratory Care. Liberal agreement was achieved if the compared respiratory care plans were identical regarding the categories of therapy (Table 1). For example, 2 plans were in liberal agreement if they included aerosol therapy and bronchopulmonary hygiene because at least 1 indication for each therapy was observed, but excluded re-inflation, supplemental oxygen, oxygen monitoring, and suctioning because no indications for these therapies were present. The study by Stoller et al6 reported results for agreement as percentages for each of the 6 categories of respiratory care therapies (Table 1) and compared agreement between RTCS-directed orders and physician-directed orders, showing benefit for use of the RTCS.
List of Respiratory Care Treatments and the Associated Indications
Twenty years after the initial study by Stoller et al,6 the RTCS is still in use at Cleveland Clinic, essentially unchanged. However, both the organization and the delivery of respiratory therapy services have changed significantly over the ensuing decades, including system growth of the institution and changes in some respiratory care leadership roles. In addition, although respiratory departments across the Cleveland Clinic Healthcare System have been recently unified under central leadership, the RTCS has not yet been adopted in all hospital locations within the system. Finally, the Cleveland Clinic has seen a large growth in the use of nurse practitioners and physician assistants who currently write orders for respiratory care in addition to physicians (all of whom will be referred to as licensed independent practitioners).
Given these changes over the past 20 years and with interest in exploring the current impact of the RTCS, this study was undertaken to simply describe (without certainty about attribution) whether agreement rate and misallocation rates have changed compared to historical benchmark rates (ie, those rates described in the 1998 publication from the Cleveland Clinic).6 A secondary goal was to compare agreement and misallocation at the Main Campus hospital of Cleveland Clinic with 2 other hospitals in the Cleveland Clinic Healthcare System that have not yet implemented the RTCS.
QUICK LOOK
Current knowledge
Clinical practice guidelines are a set of evidence-based practice recommendations intended to ensure optimal, uniform practice across a discipline. On this basis, respiratory therapy consult services (RTCS) have been developed, and studies have suggested that they reduce misallocation of respiratory therapy services.
What this paper contributes to our knowledge
This study extends current understanding by demonstrating for the first time the durability of the favorable impact of an RTCS on misallocation. Furthermore, the finding that non-RTCS-based respiratory care orders were more concordant with guidelines in a hospital where the RTCS was available than where no RTCS was available. This raises the possibility of a diffusion effect, ie, a favorable impact on allocation based simply on the availability of the service in the hospital even when it is not used in ordering respiratory treatments.
Methods
The study was deemed exempt by the Cleveland Clinic institutional review board. Data were collected at 3 hospitals within the Cleveland Clinic Healthcare System. The Main Campus hospital (designated as “on-MC”) was the site of the original study by Stoller et al.6 In the context that all respiratory therapists (RTs) at Cleveland Clinic have recently been unified under a single leadership structure, the 2 other hospitals (designated as “off-MC”) have not yet implemented the RTCS and thus represent an observational control group.
The RTCS was implemented to minimize misallocation of respiratory care orders by using RT-directed, sign- and symptom-based algorithms that were designed on evidence-based standards. The RTCS functions under the medical direction of the Section Heads of Respiratory Therapy in the Respiratory and Anesthesia Institutes. Briefly, the service covers adult in-patients who are ordered by a licensed independent practitioner to receive a respiratory consult for aerosol, bronchopulmonary hygiene, re-inflation, supplemental oxygen, noninvasive monitoring by pulse oximetry, and suctioning (Table 1). Within the RTCS, the RT assesses the patient by obtaining a baseline oxygen saturation percent and forced vital capacity, and creates a written care plan using the specific sign- and symptom-based algorithm. This information is delivered electronically to the ordering licensed independent practitioner through the electronic medical record. Thereafter, non-medication therapies (eg, chest physiotherapy) are initiated, modified, and completed by RTs using the established algorithms, whereas orders for medications are co-signed by the managing licensed independent practitioner. A handbook for the RTCS is freely available as a PDF download (https://app.box.com/s/tofjclhp08k4xa1d6bzzo4x1xybyev71. Accessed March 13, 2019).
Study data were collected over a period of 6 weeks in the summer of 2018 (June 18 to July 27) by one of the authors (AD) accompanying RTs on their normal rounds on in-patient care divisions outside the ICUs. The patients comprised a convenience sample and included all the patients assigned to a particular RT on a particular day-shift schedule. No individual patient information was collected. The RT determined whether the patients had an order for RTCS, what their treatments were, and whether those treatments were indicated (Table 1).
Agreement was defined as the degree to which sets of respiratory therapy orders for patients matched guidelines as expressed by the RTCS. An agreement index was defined as a metric, calculated for an individual patient, with possible values of 0 (no misallocation of therapies) or 1 (≥ 1 instance of a misallocated therapy). Agreement rate was calculated as the percent of patients for whom there were no misallocated therapies. Misallocation was defined as the degree to which the indications for an individual treatment matched patient needs. A misallocation index was defined as a metric with 3 possible scores: over-ordering (ie, errors of commission, or therapy not indicated but ordered), which was scored as +1; under-ordering (ie, errors of omission, or therapy indicated but not ordered), which was scored as −1; and therapy ordered and indicated or not ordered and not indicated, which were scored as 0 (Fig. 1). Misallocation rate was then calculated as a percent of all observations with non-zero scores for each individual therapy. The over-orders metric was calculated as the percentages of observations with +1 scores, and the under-orders metric was calculated as the percentage of observations with −1 scores. Table 2 shows an example of how data were analyzed (some rows of data are hidden).
Misallocation is defined as having 3 possible values: 0, −1, and +1. Values of 0 indicate that treatment was ordered when indicated or not ordered when not indicated (ie, appropriate allocation). A value of +1 indicates a misallocation due to ordering in the absence of indications (ie, over-ordering). A value of −1 indicates a misallocation due to not ordering when indicated (under-ordering).
Example Data Analysis
Five specific hypotheses were tested:
On the Main Campus, current agreement rates for patients receiving RTCS-based orders are not different than historical benchmark rates.6
On the Main Campus, current agreement rates for patients receiving RTCS-based orders versus Non-RTCS are not different.
On the Main Campus, the current misallocation rates of treatments under the RTCS are no different from historical benchmark rates.
Current agreement rates for patients receiving RTCS-based orders on Main Campus are not different from rates for patients receiving non-RTCS-based orders off the Main Campus.
Current misallocation rates for specific treatments on MC for patients receiving RTCS-based orders are not different from patients off the Main Campus receiving non-RTCS based orders.
To further understand how misallocation of specific treatments affects the overall agreement rate for patients, a sensitivity analysis was performed. For this analysis, one treatment at a time was eliminated (by setting misallocation to zero for all patients), and the rate of agreement was recalculated. This was performed for the set of data representing patients receiving RTCS-based orders on Main Campus and for patients receiving non–RTCS-based orders off Main Campus. Data were entered into the online database REDCap and analyzed using chi-square or z-tests when appropriate.
Results
The results of the analyses for the 5 hypothesis tests are summarized in Table 3. The main finding of this study (regarding hypothesis 1) is that the rate of agreement for the RTCS on-MC is significantly less than the benchmark rate established 20 years ago (63% vs 86%, P = .004). In other words, there has been a decay in the rate at which use of the RTCS in the Main Campus hospital resulted in appropriately ordered respiratory care services over this 20-y period. Indeed, the current rate of agreement for patients receiving RTCS-based orders on the Main Campus is no different from the baseline agreement rate for patients not receiving RTCS-based orders (in the same Main Campus hospital) in the former study (current rate 63% vs historical rate 72%, P = .36).
Results of Hypothesis Testing
Further sensitivity analysis (Table 4) to assess the reason for this decrement in the agreement rate indicates that the decay relates singly to bronchopulmonary hygiene. This is shown by the increase in overall agreement rate from 63% (Table 3) to 81% (Table 4) after eliminating bronchopulmonary hygiene from the data set. In this regard, the rate of agreement is highly sensitive to misallocation results for any single respiratory therapy. In contrast, off the Main Campus, no single intervention was very effective in improving the low baseline agreement rate of 20%, as indicated by the range of new agreement rates from only 23% to 29%.
Results of Sensitivity Analysis
Table 3 also shows that, in contrast to findings in the prior study,6 there was no overall difference between the rate of agreement for patients on the Main Campus with RTCS-based orders versus the rate of agreement for patients whose therapies were ordered without the RTCS (hypothesis 2). Still, 3 of the 6 individual current therapies examined (hypothesis 3) did show significantly lower misallocation rates than their historical benchmark counterparts (ie, re-inflation, oxygen monitoring, and suctioning). Despite the overall drop in agreement rate for RTCS-based orders on-MC, the rate of agreement using the RTCS (hypothesis 4) was still significantly higher than the rate observed off-MC without the RTCS (63% vs 20%, P < .001), suggesting that use of the RTCS still confers a benefit in enhancing allocation of respiratory care services (Fig. 2).
Misallocation rates for treatment orders not on the respiratory care consult service (RTCS). BPH = bronchopulmonary hygiene, on-MC = on Main Campus, off-MC = 2 hospitals off the Main Campus with no prior use of the RTCS.
Comparing misallocation rates by therapy on-MC for patients receiving RTCS-based orders versus off-MC (hypothesis 5), the results for RTCS-based orders on-MC were significantly better than non-RTCS-based orders off-MC for all but 1 therapy (ie, bronchopulmonary hygiene). Table 5 shows that misallocation tended to be in the form of over-ordering, which was highest for aerosol therapy off-MC, bronchopulmonary hygiene on-MC, and supplemental oxygen off-MC. On the other hand, the highest frequency of under-ordering was for oxygen monitoring and suctioning off-MC (21% and 20%, respectively).
Results for Misallocation Rates of Specific Therapies
Discussion
The practice of respiratory care, like other aspects of medicine, involves 3 basic processes: assessment of patient need and formation of therapeutic goals, creation of potentially effective treatments, and appropriate matching or allocation of treatment to serve the goals. CPGs are intended to improve allocation. At Cleveland Clinic, an RTCS has been available at the Main Campus hospital for over 20 years as a quality-improvement measure based on the observation that allowing respiratory therapists to start and stop specific treatments using sign- and symptom-based algorithms improves the overall agreement of patient care orders with pre-established, evidence-based indications.1 Our experience indicates that attention must be given to all operational aspects to implement an RTCS successfully, ie, the mechanics of scheduling personnel, supervising quality, and providing the ongoing education of RT staff and other health care providers likely to be involved.7 Inattention to any of these aspects can lead to degraded performance of the RTCS as indicated by observation of lower agreement on periodic quality-assurance evaluations.
The findings of the sensitivity analysis in this study suggest that misallocation for even a single therapy can exert a large effect on the overall rate of agreement of patient care with established CPGs in an RTCS. The finding that the decay in the rate of agreement with current RTCS use compared with historical benchmark rates could be ascribed to slippage in a single treatment type (ie, bronchopulmonary hygiene) creates the opportunity to enhance RTCS performance by focusing on that respiratory treatment specifically. In addition to finding an overall decay in the rate of agreement using the RTCS in the same Main Campus hospital where the RTCS was used and studied 20 years ago, we also observed that the rate of agreement with evidence-based guidelines was higher for patients receiving respiratory care orders without the RTCS on the Main Campus than for patients receiving respiratory care orders (also without an RTCS) on non-Main Campus hospitals, where the RTCS has not yet been available. While our study does not permit firm explanations as to the reasons for these changes over time and concurrent differences across hospitals, this study is, to our knowledge, the only one available that examines longitudinal rates of agreement over RTCS use over 2 decades and examines cross-hospital differences.
The changes in protocol use that were described more than 20 years ago also invite a brief discussion of changes in the environment of respiratory care over that time, which could impact protocol use and adherence. Potential influences on RTCS use and rates of agreement might include changes in leadership personnel in the Section of Respiratory Care, changes in staffing, and changes in institutional priority regarding the provision of respiratory care services. At the Cleveland Clinic over the last 20 years, the commitment to the primacy of high-quality, safe patient care is stronger than ever, so the priority of respiratory care as part of the organization's clinical mission has only been strengthened. In addition, leadership changes in respiratory care have occurred, including new medical directorship and new respiratory care leads in the Section of Respiratory Therapy. Another “mega-trend” that might impact protocol use is the evolution of the Cleveland Clinic Health System since 1998, largely reflecting system growth and sharing of best practices across a currently multi-hospital system within Northeast Ohio (with other hospital components of the health system in Las Vegas, Nevada; Abu Dhabi, United Arab Emirates; and multiple sites in Florida). As noted above, attribution of the described changes in protocol use and rates of agreement to these or any other environmental changes is not possible from the available study data, but we submit that these findings reflect the real world of consolidation and evolution that are common within large health care systems in the United States.
The current rate of agreement for non–RTCS-based orders on the Main Campus, where the RTCS has been available for more than 20 years, was nominally higher than the rate of agreement for non–RTCS-based therapies off the Main Campus, where the RTCS is not yet available. The latter observation suggests the possibility that ordering of respiratory care services by licensed independent practitioners on the Main Campus outside the RTCS has been influenced by availability of the RTCS on the Main Campus. This observation is consistent with a learning effect, which might be called diffusion, related to RTCS availability on the Main Campus hospital, such that even patients receiving care from health care providers outside the RTCS on the Main Campus benefited from a higher rate of agreement with guideline-based care than patients not receiving RTCS-based care off the Main Campus.
Specifically, akin to the well-recognized Hawthorne effect, in which subjects' behavior changes by virtue of participating in a research study,8 we surmise in this hypothesis-generating study that this proposed diffusion effect relates to health care providers' behavior changing by virtue of the availability of protocols that have been the object of study in their hospital. The lower rate of agreement in hospitals where no such protocols have been available (ie, off-MC) supports the concept. This diffusion effect may complement more deliberate learning practiced by health care organizations, in which “creative problem solving benefits from ‘absorptive’ capacity similar to most forms of learning.”9
This study extends current understanding by examining for the first time, to our knowledge, the durability of the impact over 2 decades of an RTCS. While “acceptable” agreement rates are difficult to define, this study provides benchmark agreement values for basic respiratory therapy treatments that can be used as the basis for further quality-improvement studies. Furthermore, our conjecture regarding the so-called diffusion effect on care patterns provides an opportunity to explore this putative effect in other clinical contexts. For example, does the availability of care paths or protocols for care, as are increasingly commonly available, confer benefit for clinical outcomes, and, if so, does this benefit extend to settings in health care systems beyond the immediate locus of protocol or care path use?
Several limitations of this study warrant discussion. First, because the study was conducted in a single, albeit large, health care system, the generalizability of these findings is uncertain. Data were collected from a convenience sample of patients, which may have been biased in unknown ways. As with earlier studies,2,6 the prevalence of under-ordering in this study is likely underestimated because the study sample included only patients for whom some respiratory care order was placed.
We might also question the degree to which the specific respiratory therapies comprising the current RTCS match those of the same service 20 years ago. Bronchopulmonary hygiene, in particular, started out at the Main Campus hospital as positioning, manual clapping, and vibration augmented with the use of mechanical precursors. Today, therapists select from a much wider range of treatments including oscillatory positive expiratory pressure10 and cough-assist devices. Conceivably, this wealth of options may have contributed to the observed over-ordering of bronchopulmonary hygiene.
Another important limitation is the lack of patient-level detail (eg, demographic or clinical details about participants' diagnoses and findings), which precludes any conclusion about why rates of appropriateness of respiratory care differ in the ways that were observed. In this context, the study is simply hypothesis-generating about the proposed diffusion effect, ie, we cannot be certain that such diffusion completely explains why rates of appropriateness in respiratory care orders were higher on the Main Campus in non-RTCS patients than in non-RTCS patients off the Main Campus, where the RTCS has not been available. Confirmation of a diffusion effect will require validation in other studies.
Another limitation of the study is that the data are observational, which creates the possibility of bias. For example, the benefit of the RTCS is suggested by the enhanced level of agreement on the Main Campus of the RTCS compared to non–RTCS-driven orders elsewhere. On the other hand, there is no current difference in appropriateness of orders on the Main Campus between RTCS-directed and non–RTCS-directed care. On balance, we believe that these results are best explained by the observation that the RTCS can, at its best, enhance the allocation of respiratory care, as is supported by several concordant randomized controlled trials.6,11 However, preservation of the benefits of the RTCS requires ongoing vigilance and maintenance of protocols, training, and organizational commitment to fully realize the opportunities of a RTCS. This provides an important lesson for all who intend to adopt this approach to prescribing respiratory care, as is increasingly the case.
Conclusions
RTCS-based ordering still appears to confer benefit in the same hospital where it was introduced, even 20 years later. In addition, in the context that RTCS-based care has been previously shown to lessen misallocation,6,11 we surmise that the non-difference between agreement using RTCS-based versus non-RTCS-based care in that same hospital currently reflects the impact of a so-called diffusion effect of protocol availability. This proposed phenomenon warrants further study in other settings, as does the robustness of these findings about the durable effects of an RTCS in other health care settings.
Acknowledgments
The authors thank summer science interns Angela M Demchuck, David Tan, and John Burwinkel for their contributions. The authors also thank John Burkhart, Carla Encarnacion, Dan Sutton, and Susan Brant for their support for the summer science program.
Footnotes
- Correspondence: Robert L Chatburn MHHS RRT RRT-NPS FAARC, Department of Respiratory Therapy, Respiratory Institute, Cleveland Clinic, Mail Stop M-56, 9500 Euclid Ave., Cleveland, OH. E-mail: chatbur{at}ccf.org.
Mr Chatburn has disclosed relationships with IngMar Medical, imtmedical, and Drive-Devilbiss. The other authors have disclosed no conflicts of interest.
See the Related Editorial on Page 1014
- Copyright © 2019 by Daedalus Enterprises
