Abstract
BACKGROUND: Few published data exist for adherence rates to spirometry acceptability and repeatability criteria in clinical respiratory laboratories. This study quantified adherence levels in this setting and observed changes in adherence levels as a result of feedback and ongoing training.
METHODS: Two tertiary hospital-based, lung function laboratories (L1 and L2) participated. Approximately 100 consecutive, FVC spirometry sessions were reviewed for each year from 2004 to 2008 at L1 and for years 2004 and 2008 at L2. Each spirometric effort and session was interrogated for adherence to the acceptability and repeatability criteria of international spirometry standards of the time. Feedback of audit results and refresher training were provided at L1 throughout the study; in addition, a quality rating scale was implemented in 2006. No formal feedback or follow-up training was provided at L2.
RESULTS: We reviewed 707 test sessions over the 5 years. There was no difference in adherence rates to acceptability and repeatability criteria between sites in 2004 (L1 61%, L2 59%, P = .89). There was, however, a significant difference between sites in 2008 (L1 92%, L2 65%, P < .001). No difference was seen at L2 between 2004 and 2008 (P = .26), while L1 experienced a significant increase in adherence levels between 2004 and 2008 (61% to 92% P < .001).
CONCLUSIONS: Clinical respiratory laboratories met published spirometry acceptability and repeatability criteria only 60% of the time in the first audit period. This improved with regular review, feedback, and implementation of a rating scale. Auditing of spirometry quality, feedback, and implementation of test rating scales need to be incorporated as an integral component of laboratory quality assurance programs to improve adherence to international acceptability and repeatability criteria.
Introduction
In recent years, much focus has been placed on the ability of primary care facilities to achieve valid spirometry results. Schermer et al1 have shown that the quality of routine spirometry tests in general practice is low, with only 39% of tests meeting internationally accepted acceptability and repeatability criteria. We2 and others3 have demonstrated that primary healthcare providers exposed to training with follow-up adhere to internationally accepted criteria4 in 58% of spirometry tests reviewed. The question that remains to be answered is whether or not this level of adherence is comparable to that of clinical respiratory laboratories in large tertiary institutions.
Auditing and feedback of the quality of spirometry is considered essential in research, but appears to receive little attention in the clinical setting. While there are multiple studies scrutinizing the adherence to spirometry acceptability and repeatability criteria in research trials,5–9 few published data exist for clinical respiratory laboratories. Swanney et al10 found only 67% of tests met acceptability criteria, while Enright et al11 found 90% subjects should be able to meet repeatability criteria using clinical data. A study quoting adherence to both acceptability and repeatability criteria in the clinical environment could not be found.
The purpose of this study was to review the level of adherence to acceptability and repeatability spirometry criteria in dedicated complex lung function laboratories located in tertiary hospitals. We also investigated whether adherence levels improved in response to feedback of adherence results and re-training.
QUICK LOOK
Current knowledge
The quality of routine spirometry in general practice is low. Less than half of tests meet internationally accepted acceptability and repeatability criteria. The quality of spirometry in dedicated clinical respiratory laboratories is not known.
What this paper contributes to our knowledge
Clinical respiratory laboratories met published spirometry acceptability and repeatability criteria only 60% of the time. This can be improved with regular review, feedback, and implementation of a rating scale. Auditing of spirometry quality, feedback, and implementation of test rating scales need to be incorporated as integral components of a laboratory quality assurance program.
Methods
Following ethics review and approval, 2 tertiary hospital-based, lung function laboratories (L1 and L2) were recruited to participate in the study. Approximately 100 consecutive FVC spirometry sessions were reviewed for each year from 2004 to 2008 at L1. Approximately 100 consecutive spirometry sessions were also reviewed at L2 for 2004 and 2008. The sample periods were chosen retrospectively, to limit bias. In addition, staff at L1 and L2 were blinded to the sample periods. In 2004 and 2008 the starting date of the sample period was the same at both sites.
To best reflect standard practice in the laboratories studied, test sessions were included regardless of the equipment used, provided that the device provided both flow-volume and volume-time curves for inspection. The devices utilized at L1 included the Elite and Profiler systems (Medical Graphics, St Paul, Minnesota), a Micro Medical portable spirometer (CareFusion, San Diego, California), a Jaeger MasterScreen Spirometer (CareFusion, San Diego, California), and a SensorMedics Vmax rolling seal spirometer (CareFusion, San Diego, California). The devices utilized at L2 were all SensorMedics Vmax mass flow sensor systems (CareFusion, San Diego, California). All devices in the study had the ability to provide automated messages relating to acceptability and repeatability criteria, though neither site encouraged staff to rely solely on the automated quality messages to assess acceptability or repeatability adherence.
Both laboratories undertook daily calibration/calibration checks of devices. Accuracy and precision checks across a range of flows, using calibrated 3 L syringes and biological controls, were conducted on a 4-weekly and fortnightly basis at L1 and L2, respectively.
Each spirometric effort and session was interrogated for adherence to the acceptability and repeatability criteria of the accepted spirometry standard at time of testing,4,12 and recorded in a purpose-built database. Each effort was assessed for adherence to the acceptability criteria using the flow-volume curve, the volume-time curve, the back-extrapolated volume, and the forced expiratory time. In addition, FEV1 and FVC were documented. For each spirometry session, age, height, and sex were documented, as was adherence to the repeatability criteria. Detailed definitions for acceptability and repeatability are outlined below. Where both baseline and post bronchodilator testing were performed within a test session, only baseline spirometry results were inspected.
L1 was considered to be the “intervention” site. Feedback was provided to staff in a number of formats. Results of each audit were presented at a service meeting following analysis of the audit results. Review of the acceptability and repeatability criteria was undertaken in a group setting at a service meeting each year. Finally, individual feedback was provided on an ad hoc basis as issues with acceptability and repeatability were identified. New staff undertook supervised training for at least 10 working days, with consistent reinforcement of acceptability and repeatability criteria during the training period. In 2006 a spirometry performance rating scale, based on that of Ferguson et al,13 was introduced into the laboratory (Table 1). The rating scale was placed at each workstation for reference during testing. Staff were encouraged to aim for a rating of good, and to accept a lesser rating only if 8 spirometric efforts had been attempted or the subject was unable to continue with testing.4,12
Test Quality Rating Scale
L2 was considered to be the “control” site. L2 was not provided with any instruction as to how to manage the quality of spirometry performed in their service by the investigators. No feedback of audit results was provided to the site. In 2008 the requirement for technical comments for each test was introduced by L2's laboratory manager, but a defined scale, like that of L1, was not used.
Staff working in the laboratories included scientists (minimum of bachelor degree), technicians (> 10 years spirometry testing experience), and registered nurses trained in the performance of spirometry. All staff performing spirometry were expected to be able to test unsupervised and to meet international acceptability and repeatability standards for spirometry, regardless of their years of experience in the field.
Definitions
Acceptability Criteria:
Start of Test Criteria: A maximal inhalation with a sharp fast take-off, no excessive hesitation or false start, and no back extrapolation error
Artifact: There could be no artifact, such as cough in the first second, glottic interference, or early termination, that might affect measured parameters
End of Test Criteria: Defined by one of the following: a plateau in the volume time curve for at least 1 second, subject blew for at least 6 seconds, or the subject could not or should not continue to blow.
Repeatability Criteria:
For tests reviewed from 2004 and 2005, having met the acceptability criteria for a minimum of 3 efforts, the highest and second highest FVC values and the highest and second highest FEV1 values from these acceptable efforts were within 200 mL of each other.4
For tests reviewed from 2006 to 2008, having met the acceptability criteria for a minimum of 3 efforts, the highest and second highest FVC values and the highest and second highest FEV1 values from these acceptable efforts were within 150 mL of each other.12
The differences in repeatability criteria over time were due to publication of a new spirometry standard in 2005.12
Repeatability Alone Criteria:
The repeatability criteria of published standards4,12 apply only to acceptable efforts. Repeatability alone criteria was defined as: Whether or not acceptability criteria were met, the highest and second highest FVC values, and the highest and second highest FEV1 values from all efforts were within 200 mL of each other (for years 2004 and 2005) or 150 mL of each other (for years 2006 to 2008).4,12
Statistical Analysis
Analysis was conducted using statistics software (SAS 9.2, SAS Institute, Cary, North Carolina). Comparisons were performed using chi-square tests for equal proportion and the Student t test for continuously normally distributed variables, with results reported as mean ± SD. Changes in the proportion of patients meeting adherence guidelines were assessed using multiple logistic regression, fitting main effects for site and year and an interaction between site and year. To adjust for the potential confounders, age and FEV1 Z score were included in the model as covariates. A 2-sided P value of .05 was considered to be statistically significant.
Results
A total of 707 test sessions were reviewed over the 5 year period. Table 2 outlines the subject demographics of the tests reviewed.14 At L1 the overall population studied had lower spirometry results than L2 (Z scores for FEV1, FVC, and FEV1/FVC, t test, P < .001), and the L2 population was older than the population of L1 (t test, P < .001).
Subject Demographics
Multivariate analysis of the entire data set showed that the ability to meet acceptability and repeatability criteria decreased with age (odds ratio 0.98, 95% CI 0.97–0.99, P < .001). The FEV1 Z score did not affect the ability to meet acceptability and repeatability criteria (odds ratio 1.00, 95% CI 0.92–1.09, P = .93).
Of 3,011 individual efforts that were interrogated for adherence to acceptability criteria, 2,598 (86%) met start of test criteria, 2,279 (76%) had no evidence of artifact, 2,395 (80%) met end of test criteria, and 2,003 (67%) met all acceptability criteria.
Comparisons Between and Within Sites L1 and L2 Over Time
Table 3 shows the number of tests meeting spirometry acceptability and repeatability criteria. There was no difference in the proportion of test sessions that adhered to acceptability and repeatability criteria between sites in 2004 (P = .89). However, there was a significant difference between sites in 2008 (P < .001). Likewise, there were significant improvements in adherence at L1 between 2004 and 2008 (61% to 92%, P < .001). L2 had no statistically significant increase in adherence between 2004 and 2008 (59% to 65%, P = .26). This difference in improvement remained statistically significant between sites after adjustment for patient age and FEV1 Z score (P < .001).
Number of Tests Meeting ATS or ATS/ERS Acceptability and Repeatability Criteria for Spirometry
Table 4 shows the number of spirometry sessions meeting the repeatability criteria alone. Across the study period and both sites, repeatability alone criteria were met on 93% of occasions.
Number of Tests Meeting ATS or ATS/ERS Repeatability Criteria Alone
Staffing Over Study Period
Specific data regarding each individual test operator's ability to meet acceptability and repeatability criteria were not collected due to local ethics committee requirements. Staff turnover occurred throughout the study period at both sites, with more experienced staff generally being replaced with less experienced staff. Between 2004 and 2008 at L1, 7 staff with an average of 6 years of experience in spirometry testing departed, and 8 staff with 1.5 years of experience in spirometry testing on average were gained. Similarly, at L2, 2 staff with an average of 22 years experience departed, and 4 staff with an average of 8 years experience were gained.
Discussion
After taking into account the effects of age and FEV1 on the ability to meet acceptability and repeatability criteria, this study found that, although adherence to repeatability criteria alone was achieved in 93% of the samples reviewed in 2004, adherence to acceptability and repeatability criteria was poor (∼ 60%) at both centers. Feedback to staff and introducing rating scales improved adherence to acceptability and repeatability criteria at L1 over time (61% in 2004 to 92% in 2008), while no differences were seen over time at L2, which had no interventions during the study period. These results suggest that auditing, re-education of acceptability and repeatability criteria, and implementation of quality rating scales are vital components of laboratory quality assurance programs. These findings are clinically important because improved quality of spirometry tests should result in better information for clinicians to use in planning and providing care for their patients.
When criteria were not met, it was most commonly due to the acceptability criteria not being met, with end of test criteria and artifact being the more regular causes for failing to meet the acceptability criteria. Repeatability criteria alone were met in 93% of tests overall, perhaps reflecting that test operators are focused on meeting repeatability criteria rather than acceptability criteria. Similar findings were observed in a previous study investigating adherence in a primary care setting.2
Spirometry standards state that acceptability adherence is more important than repeatability adherence in the performance of spirometry.4,12 Moving a spirometry operator's primary focus from repeatability criteria adherence alone to acceptability and repeatability criteria adherence is vital to improving overall adherence rates. The conspicuous placement of a quality rating scale (see Table 1) at work stations at the beginning of 2006 provided an avenue to improve adherence, as the scale included acceptability and repeatability criteria. Despite the implementation of a rating scale, results from 2006 dropped in comparison to 2005 results, and were no better than the results of 2004 (see Table 3). It is not clear why this occurred; perhaps acceptability criteria were overlooked as staff focus moved to the updated repeatability criteria published in late 2005 but not implemented until 2006.12 Feedback from the 2006 audit and re-education led to increases in adherence of acceptability criteria over subsequent yearly audits. Further, audit results from L1 for 2009 and 2010, which are not included in this study, are on par with 2007 and 2008 data (88% and 92% vs 88% and 92%, respectively) showing sustained improved adherence. While the contribution of each of the methods used to attempt to improve adherence rates cannot be apportioned, as a group, all (feedback of audit results, review of acceptability and repeatability criteria, and implementation of a quality rating scale) are likely to have contributed markedly to the improvements seen in adherence rates over time as these interventions were the main differences between the 2 sites over the study interval. Actively engaging staff in a process to improve adherence, by increasing awareness of acceptability and repeatability criteria and reducing the complacency of staff, may have also contributed to the improvement in adherence rates.
Table 5 summarizes the findings of the current study and previously published studies investigating the adherence to spirometry acceptability and repeatability criteria. The table includes studies from primary care, primary care following spirometry training, and research and clinical settings. These data demonstrate that adherence to spirometry acceptability and repeatability criteria is superior in the research setting, where regular review and feedback of test quality are considered to be essential to practice. Data from clinical laboratories, where regular review is unlikely to be included in the quality control program, sat below research data for adherence to acceptability criteria, but were equivalent for repeatability criteria alone. Adherence to acceptability and repeatability criteria is lowest in the primary care setting.
Comparison of Adherence to Spirometry Acceptability and Repeatability Criteria in Primary Care, Research, and Clinical Laboratories
The current study was able to demonstrate that equivalent levels of acceptability criteria adherence seen in the research setting could be achieved in a clinical setting when regular review and feedback of test quality are undertaken. It should be noted that the study was conducted in busy, publicly funded, clinical respiratory laboratories in tertiary referral centers (6,900 and 3,127 spirometry sessions per annum on average during the study period at L1 and L2, respectively). How translatable the findings are to smaller public or privately operated laboratories, where added pressures such as time and income generation may affect adherence levels, is not known.
The results of this study now mean that comparisons of adherence to spirometry acceptability and repeatability criteria between primary care and clinical pulmonary function laboratories can be made (see Table 5). The adherence results from 2004 in this study are higher, in comparison to results from studies investigating the quality of routine spirometry in general practice.1,15 The quality of spirometry in primary care after initial training and further follow-up training2,3,16 is comparable to 2004 data from this study. This suggests that spirometry in primary care, even after follow-up training, achieves only the bare minimum expected adherence to spirometry acceptability and repeatability criteria. These findings support the importance of regular monitoring of quality and feedback in primary care to improve adherence to acceptability criteria.
It should be noted that in 2010 Miller et al, in a letter to the editor,17 noted that an error had been made in their 2005 table addressing the end of test acceptability criteria (Table 5 in the 2005 paper by Miller et al12). The end of test criteria should have been an exhalation of at least 6 seconds duration and a plateau in the volume-time curve for at least 1 second or the subject could not or should not continue to blow. We reanalyzed our data with this end of test criteria and noted minimal changes in absolute numbers of subjects meeting acceptability and repeatability criteria. The difference in the results did not make any impact on the overall improvement seen with feedback and introducing a quality rating scale, however.
Limitations
In comparison to the total number of tests performed annually at each site, the audit sample size may have introduced bias. The sample size was chosen because of the time it takes to scrutinize each individual effort for adherence to acceptability criteria. As mean age, height, and FEV1 did not differ between audit periods for each site, we are reasonably confident a representative sample of the laboratory populations was chosen for each audit.
Staff turnover was experienced during the study period. While perhaps seen as a limitation, it reflects practice in “real life” and demonstrates that, despite staff turnover and changes in experience of staff, improvements are possible.
Conclusions
In conclusion, without regular quality auditing, tertiary hospital-based, clinical laboratories, dedicated to the performance of lung function, meet published spirometry acceptability and repeatability criteria only 60% of the time. However, this can be improved and maintained with regular review, feedback, and implementation of a quality rating scale. Auditing of spirometry quality, feedback, and implementation of test rating scales need to be incorporated as an integral component of laboratory quality assurance programs to improve adherence to international acceptability and repeatability criteria.
Footnotes
- Correspondence: Brigitte M Borg CRFS, Department of Allergy, Immunology, and Respiratory Medicine, The Alfred Hospital, 55 Commercial Road, Melbourne, Victoria 3004 Australia. E-mail: b.borg{at}alfred.org.au.
Mr Hartley presented a version of this paper at the meeting of the Australia and New Zealand Society of Respiratory Sciences, held April 3–6, 2009, in Darwin, Australia.
The authors have disclosed no conflicts of interest.
- Copyright © 2012 by Daedalus Enterprises Inc.