Abstract
BACKGROUND: Carbon monoxide (CO) exposure causes roughly 40,000 emergency department (ED) visits annually and is commonly misdiagnosed. Whereas the standard method of carboxyhemoglobin (HbCO) measurement utilizes blood gas analysis, a noninvasive, FDA-cleared alternative exists. We evaluated the performance of pulse oximetry (SpCO) for identification of CO exposure in ED patients.
METHODS: We compared pulse oximetry to blood HbCO levels in a prospective observational study of adult and pediatric subjects recruited from the ED. Nurses screened a convenience sample of patients and referred those with SpCO ≥ 10% to research staff. Researchers also approached individuals who presented with signs and symptoms of CO toxicity. We determined diagnostic performance with a Bland-Altman analysis and calculated sensitivity and specificity for detection of elevated HbCO at thresholds of ≥ 10% and ≥ 15%. To optimize the potential sensitivity of SpCO for detection of CO toxicity, research technicians performed 3 SpCO readings within 5 min of the blood draw for laboratory measurement. A positive SpCO test was defined as any SpCO ≥ 10%.
RESULTS: 42,000 patients were screened, 212 were evaluated, and 126 subjects were enrolled. Median HbCO level was 6% (range 1.6–21.9%). Limits of agreement were −10.3% and 8.1%. Of 23 individuals with elevated HbCO ≥ 10%, 13 were not suspected based on clinical assessment. Critically elevated HbCO was present in 6 individuals. Based on our a priori threshold of 10% for a positive test, pulse oximetry identified 14 of 23 subjects with HbCO ≥ 10%, with a sensitivity of 61% (95% CI 39–80%) and a specificity of 86% (95% CI 78–92%), and 5 of 6 subjects with HbCO ≥ 15%, with a sensitivity of 83% (95% CI 36–100%) and a specificity of 81% (95% CI 73–87%).
CONCLUSIONS: Pulse oximetry underestimated HbCO and produced false negative results (ie, SpCO < 10% for all three measurements) in 17% of ED subjects with elevated HbCO ≥ 15%. Triage screening with pulse oximetry detected cases of elevated HbCO that were not suspected by the clinical provider.
Introduction
Carbon monoxide (CO) exposure causes 40,000 emergency department (ED) visits and 3,000 deaths in the United States each year.1,2 Diagnosis of CO toxicity is complicated by non-specific symptoms, and misdiagnosis of CO toxicity is believed to be common.3 The standard method of carboxyhemoglobin (HbCO) measurement is arterial or venous blood gas analysis.3,4 However, a noninvasive point-of-care device (Radical-7 Pulse CO-Oximeter, MX board version 7.5.0.3, Masimo, Irvine, California) has been cleared by the FDA for CO measurement via pulse oximetry (SpCO). The Radical-7 pulse oximeter measures the refraction of 8 wavelengths of light through the nail bed using signal-extraction technology pulse oximetry to noninvasively measure HbCO, methemoglobin (Hbmet), oxyhemoglobin, and pulse rate. The objective of this study was to address the limitations of prior studies and to determine the diagnostic performance of this device in the detection of elevated HbCO under optimal conditions in the ED setting.
Despite the promise of multi-wavelength pulse-oximetry technology, controversy exists regarding the agreement of the device with standard laboratory testing.5–10 Two studies reported relatively wide limitations of agreement with laboratory measurement but concluded that the device has acceptable bias and correlation for screening purposes.5,7 Two additional studies reported satisfactory accuracy and precision for screening subjects in the ED.6–8 Touger et al8 found a bias of 1.4% but unacceptably wide limits of agreement (−11.6% to 14.4%), leading them to conclude that SpCO could not be used as a substitute for laboratory testing. Both Roth et al9 and Sebbane et al,10 on the other hand, found similar bias but much narrower limits of agreement, prompting them to recommend the technology for screening patients in the ED. Limitations of prior studies included times between SpCO measurement and blood draws for laboratory analysis of up to 1 h. Furthermore, the performance of diagnostic tests may vary depending on the magnitude of the abnormal value, and previous studies defined elevated CO at relatively low levels (eg, HbCO > 5%). These studies may therefore have failed to adequately assess the utility of pulse oximetry for detecting higher HbCO levels that are more likely to be toxic.9 We hypothesized that the sensitivity of pulse oximetry for detecting blood HbCO levels ≥ 10% in patients in the ED setting, under optimal conditions and when performed within a short time interval from the blood test, would be improved relative to previous reports.
We elected to use a multipronged recruitment method to enroll as large a number of true positive (ie, CO-poisoned) patients as possible to determine the performance characteristics of the device for detecting individuals with clinically meaningful CO levels in their blood. We identified patients who presented with either report of a CO exposure or clinical symptoms consistent with CO poisoning. We also screened a convenience sample of non-acute ED patients in triage for elevated SpCO at a 10% threshold. We selected this threshold for elevated HbCO ≥ 10% not because it represents CO toxicity, per se, but because it is above the upper limit for the expected physiological range of smokers, even with recent tobacco use.4 Therefore, it represents a clinically meaningful CO elevation in any patient, regardless of the source of CO exposure. To optimize the potential sensitivity of pulse oximetry to detect elevated HbCO, 3 measurements were obtained by trained research technicians. We limited the elapsed time between blood collection and SpCO measurement to < 5 min, thereby eliminating the confounding variable of CO washout. We prospectively compared the experimental measurement of HbCO with SpCO to the criterion standard, which is laboratory measurement of HbCO in the blood sample. We compared concurrent SpCO and HbCO measurements with a Bland-Altman plot and report the limits of agreement. We also report diagnostic performance characteristics for this cohort as sensitivity and specificity of a positive test at the ≥ 10% threshold for the detection of true positive, elevated HbCO at levels of both ≥ 10% and ≥ 15%.
QUICK LOOK
Current knowledge
Despite causing roughly 40,000 emergency department visits annually, misdiagnosis of carbon monoxide (CO) exposure is believed to be common due to its non-specific symptoms. Whereas the standard measurement of carboxyhemoglobin (HbCO) utilizes blood gas analysis, there is a noninvasive, alternative. However, despite noninvasive CO measurement with pulse oximetry, controversy exists regarding the agreement of the device with standard laboratory testing.
What this paper contributes to our knowledge
Despite efforts to optimize the performance of pulse oximetry in this study, we found the limits of agreement between pulse oximetry and laboratory measurement comparable to prior studies in subjects recruited from the emergency department. Pulse oximetry underestimated HbCO overall. Importantly, approximately 17% of subjects with critically elevated blood HbCO levels (ie, HbCO ≥ 15%) were missed with the use of pulse oximetry.
Methods
Study Design and Setting
We performed a prospective, observational study to determine the diagnostic performance of the SpCO sensor in the ED setting under optimal experimental conditions. We enrolled a convenience sample of ED patients suspected of having CO toxicity and measured SpCO and HbCO concurrently. Our academic ED at the University of Vermont Medical Center in Burlington, Vermont, sees approximately 60,000 patient visits each year. The study was conducted in accordance with the Standards for the Reporting of Diagnostic Accuracy studies, and we obtained local institutional review board approval.11 This was an investigator-initiated study supported by the manufacturer of the pulse oximeter (Masimo). The authors designed the study, executed the protocol, performed the analysis, and wrote the manuscript. None of the authors received any salary or travel funds or were otherwise supported or employed by the sponsor. The role of the sponsor included funding for the technicians and the administrative costs of the study; loaning of the SpCO sensors and equipment; and data safety monitoring (ie, review of the institutional review board protocol, training of technicians in use of the sensor, and deployment of study monitors to our site to observe enrollment and data collection). The sponsor did not design the study, collect data, analyze results, or write the manuscript.
Selection of Participants
A convenience sample of ED patients was screened for this study. Screening was conducted from June 2011 through March 2014 during times when research staff were available. The screening process was designed to identify patients who were most likely to have elevated CO levels. Potential subjects were referred to research staff by ED clinicians. Referring health care providers were instructed to look for patients with a documented CO exposure or who exhibited signs and symptoms of CO toxicity. These signs and symptoms included headache, dizziness, nausea, generalized weakness, impaired judgment, confusion, or loss of consciousness.4 In addition, triage nurses screened patients with pulse oximetry to identify patients with SpCO ≥ 10% and referred these individuals to research staff for potential enrollment. This method of recruitment was utilized to enrich the study population with additional subjects with potentially elevated CO levels on the basis of SpCO, who did not have signs and symptoms of CO toxicity. Both adult and pediatric patients were eligible for enrollment. Research staff evaluated subject eligibility and obtained written informed consent. Exclusion criteria included patients with Hbmet > 1.6% as determined with pulse oximetry (SpHbmet) because Hbmet interferes with measurement of SpCO. We also excluded patients with acrylic nails, painted fingernails, or fingernail deformities to avoid interference with SpCO measurement.
Interventions
Research staff were trained by the manufacturer and investigators in proper use of the device. Research nurses were also trained to measure SpCO as part of routine triage of ambulatory patients, when time and staffing permitted this assessment. A representative of the manufacturer visited the ED on a quarterly basis to monitor device use and re-train research personnel.
HbCO Measurements
Upon enrollment of a subject, venous blood was drawn and measurement of HbCO was performed by an accredited, in-hospital laboratory using a Sysmex XN9000 hematology analyzer (Sysmex America, Lincolnshire, Illinois). This blood HbCO measurement is considered the criterion standard measure. The laboratory was blind to SpCO levels, and research staff were blind to HbCO results at the time of SpCO measurement. Trained research staff performed SpCO measurement as close to the time of blood draw as possible, from the contralateral arm, and never during tourniquet or blood pressure cuff use. For SpCO measurements, a Rainbow R 1 25 adult adhesive sensor connected to a Radical-7 Pulse CO-Oximeter was placed on the middle finger of the non-dominant hand and covered with an opaque probe cover to eliminate interference from ambient light. The manufacturer recommends 3 SpCO readings to improve accuracy in cases of unexpectedly high SpCO (Radical-7 Signal Extraction Pulse CO-Oximeter Operator's Manual, pp. 6–1, Available at http://www.infiniti.no/upload/Bruksanvisningar/Masimo/MAS_UM_EN_Radical-7%20Color%20Display%20Operators%20Manual.pdf). We performed 3 readings in all cases for consistency, regardless of the SpCO level. We only included measurements taken within 5 min before or after the blood draw for analysis. Subject demographic data were collected from the electronic medical record. Study data were collected and managed using REDCap (Research Electronic Data Capture), a secure, web-based application designed to support data capture for research studies.12
Outcomes
To determine performance characteristics across the range of HbCO values, we calculated the average of the 3 SpCO values for each individual subject and compared this to the criterion standard of the blood HbCO with a Bland-Altman analysis. For sensitivity and specificity calculations, we defined a positive pulse-oximeter signal as any of 3 SpCO readings ≥ 10%. While some trials have utilized different thresholds for smokers and non-smokers, we used the criterion standard of elevated HbCO as blood HbCO ≥ 10% for all subjects because this is outside of the expected physiological range for smokers.4 As a secondary analysis, we also evaluated the sensitivity and specificity of the pulse oximeter to detect the outcome of critically elevated HbCO, defined as levels of blood HbCO ≥ 15%. This threshold was based on a study in which 10 healthy controls exhibited no unpleasant symptoms or significant changes in vital signs with ≤ 15% HbCO.13
Statistical Analysis
Demographic data were summarized with proportions and with medians and interquartile ranges. We plotted mean versus difference and calculated limits of agreement between the measurements as described by Bland and Altman, showing the mean of the differences between methods ± 1.96 SD above and below the mean.14,15 Other measures, including repeatability and confidence intervals for bias and limits of agreement, are reported as recommended by Chhapola et al16 We determined the diagnostic performance for detection of elevated and critically elevated HbCO according to our a priori outcome criteria. Sensitivity and specificity were calculated with the 95% CIs for the noninvasive hemoglobin measurement (SpCO) to detect the criterion laboratory measurement (HbCO) at levels of ≥ 10% and ≥ 15%. Statistical analyses were performed with Microsoft Excel (Microsoft, Redmond, Washington), and graphs were produced with GraphPad Prism 7 (GraphPad Software, La Jolla, California).
Results
Characteristics of Study Subjects
Of approximately 42,000 patients screened by clinical staff during those times when research staff were present, 212 patients were identified by either clinical signs and symptoms or SpCO as potential subjects likely to have elevated HbCO (Fig. 1). Of these, 207 provided informed consent. Twelve were excluded for acrylic or polished nails, 13 for SpHbmet > 1.6, and 47 because > 5 min had elapsed between measurements. Nine subjects were missing either HbCO or SpCO data and were not analyzed. Thus, a total of 126 subjects were included in the analysis. Subjects ranged in age from 15 y to 89 y and reported a wide variety of chief complaints, such as fever, chest or abdominal pain, or trauma. Included subjects and excluded patients had similar demographic and clinical characteristics except for sex: 54.8% of included subjects and 34.7% of excluded patients were male (Table 1).
Statistical Analysis
Of the 126 subjects analyzed, the range of HbCO levels was 1.6–21.9% with a median of 6.0% (interquartile range 4.2–9.2%). The SpCO readings for these subjects ranged from 0 to 23.6% with a median of 5.3% (interquartile range 2.9–8.0%). For quantitative comparison of SpCO to HbCO, we used the average of the 3 SpCO readings for each individual. The scatter plot of HbCO versus SpCO showed a weak positive relationship (r = 0.292, P < .001) (Fig. 2). The mean difference plot (Fig. 3) showed no evidence of heteroscedasticity (differences in magnitude proportional to the size of the measurements). SpCO was most often an underestimation of HbCO level, as shown by the calculated bias of −1.1% for the mean difference (HbCO − SpCO). The limits of agreement from Bland-Altman analysis were −10.3% to 8.1% (Table 2).
We determined the sensitivity and specificity of positive SpCO (defined a priori as any of 3 measurements ≥ 10%) for detection of elevated HbCO. On the basis of the diagnostic criterion of elevated HbCO ≥10%, our cohort of 126 individuals consisted of 23 true positives and 103 true negatives (Table 3). SpCO was positive in 14 of the 23 true positive subjects, with 9 false negatives (sensitivity 61% [95% CI 39–80%]); SpCO was negative in 89 of 103 true negative subjects, with 14 false positives (specificity 86% [95% CI 78–92%]). Of these 23 individuals with blood levels of HbCO ≥ 10%, 10 were suspected by the treating clinicians to have CO toxicity (4 had a known exposure and 6 had symptoms indicating possible CO toxicity). On the other hand, 13 individuals had neither symptoms nor a known exposure and might therefore be considered “occult” CO toxicity. We also determined the sensitivity and specificity for detection of critically elevated HbCO defined as levels ≥ 15%. At this threshold, the sensitivity of SpCO was 83% (95% CI 36–100%) and the specificity was 81% (73–87%) (Table 3).
We performed a pre-planned sensitivity analysis to determine the extent of bias that might have been introduced by dropping subjects in whom SpCO could not be obtained or who were excluded for other reasons. Thus, we compared the demographic characteristics and HbCO levels in the 72 subjects who were excluded to the 126 who were analyzed (Table 1). The HbCO in the excluded subjects averaged 5.4% (SD 5.2%), which was not significantly different from those subjects who were included. Of note, one dropped subject with a known CO exposure and critically elevated HbCO level of 20.5% was missed by the device, which incorrectly reported elevated SpHbmet and normal SpCO. Another subject with a HbCO level of 15.3% also escaped detection, likely due to acrylic nail polish. Elevated SpHbmet and nail polish were a priori exclusion criteria, so these individuals were not included in the primary analysis.
We performed 3 readings in all cases, regardless of SpCO level. However, the manufacturer's operations manual recommends taking the average of 3 readings to improve accuracy only in cases of unexpectedly high SpCO. An SpCO reading was considered “unexpectedly high” if the subject was not previously suspected of exposure to CO. In normal clinical use, it is therefore more likely that only a single measurement would be performed. We attempted to simulate this by re-analyzing the sensitivity and specificity for the first reading only. In a mean difference analysis of first measurements only, bias was slightly smaller (−0.8%) and limits of agreement widened slightly to −10.4% to 8.7%. Because 2 subjects with blood levels ≥ 10% were only identified on the second or third SpCO reading, use of the first reading only produced a lower sensitivity of 52% (31–73%) with a specificity of 88% (81–94%).
Discussion
Venous blood oximetry is accurate and precise, but phlebotomy is invasive and blood analysis requires time and a credentialed laboratory. Pulse oximetry offers a point-of-care alternative to rapidly screen for elevated HbCO without drawing blood. The diagnostic performance of pulse oximetry for the detection of elevated HbCO in the ED is controversial. We identified 3 studies that evaluated pulse oximetry technology by comparing SpCO to the accepted standard of laboratory measurement and which met Standards for Reporting of Diagnostic Accuracy and reported agreement using Bland and Altman's approach.8–11 Touger et al8 studied 120 ED subjects with a suspected diagnosis of CO toxicity and reported limits of agreement from a Bland-Altman analysis of −11.6% and 14.4% HbCO. The authors defined the a priori threshold as agreement < 5% HbCO and concluded that SpCO measurement may not be used interchangeably with laboratory measurement. Later, Roth et al9 studied a larger sample of 1,500 unselected subjects in the ED and reported slightly better limits of agreement of −3.6 to 9.5 HbCO. They concluded that SpCO measured HbCO with acceptable bias and precision. However, they identified only 17 subjects with CO poisoning, which was defined clinically and included non-smokers with levels of HbCO that were < 10%. In addition, their SpCO measurements were made within 60 min of the HbCO measurement. Sebbane et al10 reported on 93 subjects with suspected CO toxicity, including 26 with CO poisoning (defined as blood HbCO > 5% for non-smokers and > 10% for smokers), and found limits of agreement of −6.7% and 6.3%. Measurements were made within a mean of 19 min (95% CI 10–29 min).
We attempted to improve on previous studies in several ways. We utilized a dedicated research team to recruit ED subjects who were suspected of having CO toxicity. We identified clinically meaningful levels of HbCO ≥ 10% for elevated and ≥ 15% for critically elevated for all CO exposures, including tobacco and other causes. We excluded measurements made > 5 min apart to reduce the effect of CO washout between measurements. We worked with representatives from the manufacturer to ensure ideal training, technique, and equipment function. We also conducted 3 SpCO measurements in each case, using the average of the 3 for Bland-Altman analysis, and we used any reading > 10% as our definition of a positive test for sensitivity and specificity analysis. These efforts produced a rigorous set of experimental data, including subjects with a range of CO levels, but the performance characteristics of pulse oximetry were not better than in previous studies.
Our sensitivity analysis resulted in only slight changes in the main results. Re-analysis using only 1 SpCO reading yielded slightly lower sensitivity and slightly higher specificity. Bias slightly decreased in magnitude, but at the cost of wider limits of agreement. Including all excluded subjects (eg, artificial nails, nail polish, SpHbmet >1.6%, and > 5 min elapsed between methods) again revealed a slightly smaller bias but wider limits of agreement that did not appear substantially different from the restricted dataset. One subject who was excluded due to elevated SpHbmet and had a HbCO level of 20.5% warrants discussion. This individual was symptomatic, and blood analysis confirmed elevated HbCO without Hbmet elevation, but the device missed this diagnosis, reporting SpHbmet elevation with normal SpCO on multiple measurements. Thus, clinicians should be aware of the potential for oximetry to incorrectly suggest methemoglobinemia in some CO toxicity cases.
The relative infrequency of elevated HbCO levels among the large number of potentially eligible subjects is a limitation. It is difficult to identify subjects while they still have high HbCO levels, in part because the half-life of carbon monoxide, even with room air, is < 3 h. It often takes minutes to hours to capture and consent ED patients with CO toxicity, and by this time their blood levels may start to decline. Furthermore, although SpCO measurements occurred within 5 min of the blood draw, the elapsed time between the draw and the measurement of HbCO in the blood sample by the hospital's accredited laboratory was not recorded. It is possible that HbCO levels within the sample could have changed during this time period.
Although we conducted 3 SpCO measurements for all subjects, it is unlikely that this would occur in most clinical applications. Our sensitivity analysis found that when we used only the first of 3 SpCO measurements, sensitivity dropped to only 52% (95% CI 31–73%). Relatively small numbers of true positives result in wide confidence intervals for sensitivity and specificity of the device. We therefore intentionally used a strategy to enrich our sample for subjects most likely to include true positives, enrolling specifically those subjects who were suspected to have CO toxicity due to report of an exposure, clinical symptoms, or SpCO screening levels. This sampling strategy itself has limitations. Among these is the possibility that the subjects recruited from triage may be intrinsically different than those recruited via clinician referral.
We used a threshold of HbCO ≥10% as the upper limit of normal cutoff for screening for elevated HbCO in all subjects with or without smoking history. In clinical practice, HbCO levels > 4% in non-smokers and > 10% in smokers can be considered outside the normal physiological range.4 Two previous studies, one with a cutoff of 6.6% HbCO for all subjects and the other with cutoffs of 9% for smokers and 6% for non-smokers, reported the best combination of sensitivity and specificity for use as a screening tool for CO poisoning.9,10 We settled on a single-criterion standard for elevated HbCO of blood HbCO ≥ 10% because it falls outside of the physiological range for both smokers and non-smokers and because the source of CO exposure is important in clinical management but irrelevant to the performance of the assay.4 Regardless of the threshold that may be applied, the Bland-Altman analysis provides an estimate of performance characteristics and allows comparison between the different studies, independent of thresholds used for the criterion diagnosis. We did not define a priori accepted Bland-Altman limits of agreements, however our limits did fall outside the previously defined < 5% HbCO a priori limits established by Touger et al.8
It should be noted that our study was not designed to test the utility of the device for triage screening of all patients presenting to the ED triage. To assess utility for triage screening, blood testing would need to be performed on all of the 42,000 potentially eligible patients who were screened. This was beyond the scope of our study. That being said, our results offer conflicting clues as to the efficacy of the device in such an application. On the one hand, pulse oximetry returned false negatives for 39% of subjects with HbCO ≥ 10%. This figure fell to 17% of subjects above the critically elevated threshold of HbCO ≥15%. On the other hand, we did identify 13 individuals with elevated CO levels without symptoms or a known exposure. This suggests ED triage screening with pulse oximetry has utility in identification of some individuals with occult CO poisoning.
Conclusion
Despite efforts to optimize the performance of pulse oximetry in this study, we found the limits of agreement between pulse oximetry and laboratory measurement of −10.3% and 8.1%, which were comparable to prior studies in subjects recruited from the ED. SpCO slightly underestimated HbCO overall with a bias of −1.1%.
Although our study was not specifically designed to evaluate the role of pulse oximetry in triage, we noted that triage screening of ED patients with pulse oximetry identified CO exposure in approximately 0.05% of patients, about half of whom might have been missed by clinical assessment alone. Future studies that include blood measurements in all eligible subjects are needed to determine the utility of this device as a screening tool. The sensitivity and specificity results we provide for detection of HbCO at clinically important thresholds should aid clinicians seeking to use SpCO for medical decision-making in the ED setting.
Footnotes
- Correspondence: Kalev Freeman MD PhD, University of Vermont, 89 Beaumont Ave., Given Building, Room E301, Burlington, VT 05405. E-mail: kalev.freeman{at}uvm.edu.
The authors have disclosed a relationship with Masimo.
See the Related Editorial on Page 1450
- Copyright © 2019 by Daedalus Enterprises