Abstract
BACKGROUND: Use of respiratory therapist (RT)-guided protocols enhances allocation of respiratory care. In the context that optimal protocol use requires a system for auditing respiratory care plans to assure adherence to protocols and expertise of the RTs generating the care plan, a live audit system has been in longstanding use in our Respiratory Therapy Consult Service. Growth in the number of RT positions and the need to audit more frequently has prompted development of a new, computer-aided audit system.
METHODS: The number and results of audits using the old and new systems were compared (for the periods May 30, 2009 through May 30, 2011 and January 1, 2012 through May 30, 2012, respectively). In contrast to the original, live system requiring a patient visit by the auditor, the new system involves completion of a respiratory therapy care plan using patient information in the electronic medical record, both by the RT generating the care plan and the auditor. Completing audits in the new system also uses an electronic respiratory therapy management system.
RESULTS: The degrees of concordance between the audited RT's care plans and the “gold standard” care plans using the old and new audit systems were similar. Use of the new system was associated with an almost doubling of the rate of audits (ie, 11 per month vs 6.1 per month).
CONCLUSIONS: The new, computer-aided audit system increased capacity to audit more RTs performing RT-guided consults while preserving accuracy as an audit tool. Ensuring that RTs adhere to the audit process remains the challenge for the new system, and is the rate-limiting step.
Introduction
Allocating respiratory therapy with respiratory therapist (RT)-guided protocols has been shown to confer benefits of enhanced allocation and cost savings with equal clinical efficacy, compared with physician-directed respiratory therapy.1,2 Successful implementation of a service to administer RT-guided protocols, called a Respiratory Therapy Consult Service (RTCS3–5) at the Cleveland Clinic, requires rigorous protocol development and training for RTs. Such a system has been used and described previously.6–8 Specifically, in this original audit system, 8 RTs were chosen randomly each month, and their RTCS-based care plans were compared with that instituted by an expert RTCS auditor, who saw the patient and developed a “gold standard” care plan. The gold standard care plan was then compared to the actual RT-generated plan, and discordance was measured and fed back to the RT who was being assessed. While effective, this live audit system has proven labor- and time-intensive, which has limited the number of RTs who could be trained and reviewed. Recent expansion of the RTCS and the desire to extend the audit's function more broadly among RTs doing consults (from 30 to ∼70 RTs) prompted interest in revamping the audit system. The goals of the revised audit system were to allow a greater number of RTs to be audited and to simplify and enhance record-keeping capability, while maintaining the teaching and quality monitoring functions of the audit system.
The current report presents our early experience with a new, computer-aided audit process for the RTCS. Specifically, in the context that we have developed a system in which all RTs performing consults (called Therapist Evaluators [TEs]) can be audited 3 times a year, we describe the elements of the new audit system, present early experience with its use, and compare audit results using the original, live audit system with the new computer-aided audit system.
QUICK LOOK
Current knowledge
Respiratory therapist (RT)-guided protocols improve the allocation of respiratory care services. Auditing respiratory care plans, to assure adherence to protocols and the expertise of the RTs generating the care plans, is an important part of a respiratory therapy consult service. Traditional live audits are time consuming, so frequent auditing is impractical in large departments.
What this paper contributes to our knowledge
A computer-aided audit system increased the capacity to audit more respiratory care plans, while preserving audit accuracy. RT adherence to the audit process remains a challenge and is the major hurdle to success.
Methods
The Computer-Aided Audit System
As was described in the original report regarding the audit process,7 an audit involves comparing the selected TE's respiratory therapy care plan with that of a gold standard care plan (which is generated by an RTCS protocol expert who is the Education Coordinator [EC] in the Respiratory Therapy Section [author LK]). Generating the respiratory therapy care plan is guided by protocols that are based on the patient's signs and symptoms.1,3,4 The protocols are now embedded within an electronic respiratory therapy management system (MediServe, Tempe, Arizona), which prompts the TE to choose among available respiratory therapies that may be indicated, based on the patient's signs and symptoms. The choices of respiratory treatments then drive construction of the respiratory therapy care plan. The actual audit consists of comparing the treatment choices on the care plan generated by the TE with the gold standard care plan and assessing the degree of discordance, if any.1,6,7
New aspects of the computer-aided audit process regard the ways in which the TE performs the assessment and generates a respiratory therapy care plan, and the EC constructs the gold standard care plan and assesses the degree of discordance. Specifically, in contrast to the original process, which involved actual patient visits by the TE and the EC (at different but ideally close points in time) and submission of written respiratory therapy care plans for direct comparison by the EC, the new system requires assessment of patients using data within the electronic medical record and construction of care plans using an electronic respiratory therapy management system. The TE extracts information from the patient's actual clinical record. The specific steps involved in ordering a respiratory therapy consult and generating a protocol-guided respiratory therapy care plan6,7 are presented in Table 1.
Steps in Ordering a Respiratory Therapy Consult and Generating a Protocol-Guided Respiratory Therapy Care Plan
Principles and processes for scheduling audits under the new audit system include:
Each RT performing consult evaluations (called the TE) should be audited 3 times per year. Thus, with approximately 70 RTs to audit, the goal is to perform a total of approximately 210 audits annually.
Each month, the EC sends to the assigned TE (via the hospital's e-mail system) the names of the patient for whom an assessment is to be performed and a respiratory therapy care plan written (for audit purposes).
To assure assessment over a range of clinical respiratory therapy issues, an effort is made to assess TEs on both ICU and non-ICU in-patients.
A due date, usually the last day of the month, is specified for completion of the TE's care plan.
Finally, steps for completing the actual audit with the new system are summarized in Table 2.
Steps for Completing an Audit With the New, Computer-Aided System
Results
The new computer-aided audit system was implemented on January 1, 2012. For purposes of comparing the old and new systems, results of older, “manual” audits conducted between May 30, 2009, and May 30, 2011, were compared with audits conducted using the new system between January 1, 2012 and April 30, 2012.
The degrees of concordance between TEs' care plans and the gold standard care plans using the old and new audit systems were similar (Table 3). However, more audits per month were conducted with the new system than with the old system. Specifically, in practice, the new audit system was associated with a near doubling of the rate of audits in the first 4 months since its introduction (ie, 11 audits per month in 2012 vs 6.1 audits per month in 2009–2011). This increase in capacity for audits relates to time savings in conducting an audit, compared with the older manual audit system. In addition to sparing the auditor the need to physically visit the patient, the time savings also relates to eliminating the need for generating a unique gold standard care plan. Specifically, in contrast to the older, manual system, where each patient visited required the EC to generate a gold standard respiratory care plan, the newer computer-aided system allows the same patient record to be used for each audit. Estimating that generating a gold standard respiratory care plan requires at least 30 min per patient (and more for complex patients), the monthly time savings associated with conducting 20 audits per month with the new computer-aided audit system is:
Percent Concordance Between the Therapist Evaluator's Respiratory Therapy Care Plan and the Gold Standard Respiratory Therapy Care Plan Using the Old and New Audit Systems
With regard to the new audit system's ability to sample TEs, given that an average of 30 different evaluators assess patients each month, 36.6% of the TEs were audited with the new system versus 20% with the old audit system over the course of this study. When the system is fully implemented, and the target number of 20 TE audits per month is achieved, the potential exists to audit 66.6% of all TEs monthly and to audit all TEs 3 times per year, as planned.
Discussion
This report describes a new, computer-aided system for conducting quality control audits of respiratory therapy care plans generated by TEs in the Cleveland Clinic RTCS. Comparison of this new, streamlined audit system with the original “manual” system shows that the new system is more time-efficient, by allowing a higher rate of audits to be conducted. The number of audits that can be conducted with the new computer-aided system is limited only by the availability of the TE to complete the assigned audit. In contrast, with the old system, the number of audits was limited by the availability of the single auditor. The observation of similar degrees of concordance between TEs' plans and the auditor's gold standard care plans suggests that the new system preserves its ability to assess TEs' accuracy in implementing RT-guided protocols, while achieving greater capacity to audit more RTs. As we are unaware of any earlier reports regarding audit systems in a respiratory therapy protocol service other than our initial description of the manual system,7 we suggest that these early findings extend available knowledge about optimal ways of conducting audits of RTs who are implementing respiratory care protocols.
This greater efficiency of the new audit system relates to its eliminating the need for the auditor to actually see the patient being assessed. Instead, by having both the TE and the auditor generate a respiratory therapy care plan based on data extracted from the online respiratory therapy management system and the electronic medical record, the new audit system permits comparison without actual patient visits. At the same time, the new system assesses not only the TE's ability to accurately select from indicated menus of possible respiratory therapy treatments, but also to select the appropriate data elements from the chart upon which to base the construction of the respiratory therapy care plan. The latter is a key skill in assessment and constructing a care plan. Another advantage of the new system is that it may enhance the accuracy of the audit, because it avoids the bias introduced in the older system by metachronous patient visits by the TE and the auditor. More specifically, in the old system, one possible source of discordance between the TE and the auditor was a change in the patient's signs and symptoms that may have occurred in the time between visits by the TE and the auditor. In the new system, changing clinical status would not be expected to cause discordance, because the TE and the auditor must extract relevant signs and symptoms that are already recorded within the patient's medical record.
A major benefit of the new system is its greater capacity for TE audits, thereby assuring that all TEs will be audited regularly, including those who perform evaluations less frequently. To the extent that these audits are an important quality assessment and improvement tool for the RTCS,6,7 the ability to assess a larger number of RTs broadens quality control of the process by which respiratory therapies are determined in our hospital.
Part of the enhanced efficiency of this new audit system relates to the automated scoring of concordance using the database program (see Fig. 3), replacing the prior need to manually compare and score the TE's and EC's care plans regarding concordance. Storage of results in the database also facilitates preparation of longitudinal score sheets for the TEs, thereby allowing their assessment of personal progress over time (see Fig. 4). Additionally, quarterly or yearly audit summary reports (see Fig. 5) can easily be generated.
Standardized patient assessment form in the Respiratory Therapy Consult Service.
MediServe branching logic consult charting template.
Therapist Evaluator audit feedback form.
An individual respiratory therapist's longitudinal performance report.
Audit summary report.
In the context of these clear advantages of the new audit system, several shortcomings of the new system and of this analysis warrant mention. Regarding the system itself, a limitation is the continued difficulty of assuring that the TEs complete their assignments for audit and do so on time. Vigilance is still needed to assure awareness of and adherence to the audit schedule.
Regarding limitations of the study, the conclusion that the degrees of concordance between the TE and the EC on audits using the new and old systems are equivalent (see Table 3) is challenged by the low power of the comparison. Specifically, the study had only 49% power to find a difference between the compared groups on “overall percent correct,” where a clinically important change was deemed to be 9% (ie, to 100% using the new audit system). The power is obviously lower to find smaller differences between the compared rates for the old and new audit systems. Also, the dependence of the new audit system on use of an electronic medical record and an electronic respiratory therapy management system limits the applicability and generalizability of this proposed solution only to those respiratory therapy departments that share such resources. Departments that lack these systems may still use the old audit system.6–8 Finally, to the extent that the current audit process requires the TE to extract information from the patient's medical record rather than from direct physical examination of the patient, it is possible that the computer-aided audit is based on erroneous physical findings. Another result is that the computer-aided system does not audit the TE's physical examination skills. At the same time, a potential shortcoming of the older manual audit system was that the TE and the auditor examined the patient at different times, such that the physical examination for dynamic findings (eg, wheezing) could have changed. Much like the new computer-aided audit system, any change in physical findings between the 2 examinations would compromise the ability of the older manual audit system to confirm the TE's physical findings. The new and older systems are also similar in requiring the TE to extract laboratory information from the medical record.
Conclusions
In conclusion, this report describes a streamlined, computer-aided system for conducting respiratory therapy care plan audits in a protocol-based environment. Development of this system both invites and permits future studies regarding the impact of the new audit system on the quality of respiratory care provided by the RTCS and on respiratory care outcomes.
Footnotes
- Correspondence: James K Stoller MD MSc FAARC, Education Institute, Cleveland Clinic, 9500 Euclid Avenue, Cleveland OH 44195. E mail: stollej{at}ccf.org.
The authors have disclosed no conflicts of interest.
- Copyright © 2013 by Daedalus Enterprises