Noninvasive Carbon Monoxide Detection: Insufficient Evidence for Broad Clinical Use

Carbon monoxide (CO) poisoning is an important public health issue, as this colorless, odorless gas is a common cause of unintentional poisoning and leads to over 400 deaths each year in the United States.¹ Detection of CO poisoned patients may be difficult, as even classic symptoms, including headache, nausea, dizziness, and fatigue, are vague and nonspecific, and poisonings may occur without an obvious precipitant.² A rapid, noninvasive means of accurately measuring blood carboxyhemoglobin (COHb) levels could offer numerous benefits, and a broad screening program could ideally detect previously unsuspected cases of poisonings presenting to the emergency department (ED). Additionally, out-of-hospital screening could facilitate assessing large numbers of people after disasters or mass casualty events, when exposure to and toxicity from CO may occur.^3–5 Screening for elevated noninvasive COHb levels (S_pCO) in the field by first-responders could also prompt more rapid treatment⁶ and facilitate transport of patients with elevated COHb levels to appropriate facilities. In addition, not all hospitals are capable of performing on-site blood COHb testing. One recent survey of acute care hospitals in the Pacific Northwest revealed that only 44% of the responding hospitals had the capacity to measure COHb levels.⁷

The Masimo Corporation (Irvine, California) developed a noninvasive, portable pulse CO-oximeter that is marketed as the Rad-57 CO-oximeter.⁸ Understanding the accuracy of the device is essential, as false positive readings (ie, readings of elevated CO levels in a patient without CO toxicity) could result in over-triage with associated increased healthcare costs, since these patients could be unnecessarily transported to tertiary care centers for possible hyperbaric treatment or subjected to needless additional diagnostic evaluations. However, in appraising a screening exam, false positives can be tolerated, whereas false negative readings (ie, readings indicating normal COHb levels in patients with CO toxicity) can be disastrous, potentially resulting in adverse patient outcomes, including death. While no screening test can be 100% sensitive, ideally, a screen should be a sensitive as possible.

The company's specifications for the Rad-57 CO-oximeter indicate that 68%, or one standard deviation, of the device's measurements fall within an absolute ± 3% of the value of COHb for a blood level of < 40%.⁸ Therefore, approximately 95% of measurements, or the limitations of agreement, are expected to fall within 2 SD of the COHb, or ± 5.9%.^8,9 Masimo has previously cited internal test data as a rationalization for the ± 3% accuracy specifications; however, these data are not publically available and have not been peer-reviewed.^9,10

In this issue of Respiratory Care, Weaver and colleagues measured S_pCO levels using the Massimo Rad-57 noninvasive CO-oximeter and compared them to COHb levels.¹¹ Numerous prior studies have assessed the accuracy and precision of the Rad-57 CO-oximeter in clinical and research settings, specifically comparing the S_pCO obtained from the device with the gold standard measurement of COHb. The methodologies and settings of the studies evaluating the performance of the Rad-57 are variable. Several small studies have found a good correlation between the 2 values, although, notably, 2 of these studies involved only 9 and 10 healthy volunteers,^12,13 and other studies have only tested correlation among subjects with low COHb levels.^9,14

A retrospective review of 10,856 patients screened with the Rad-57 upon presentation to the ED found 11 cases of occult CO poisoning,¹⁵ and a case series of 7 patients out of 74,880 patients screened¹⁶ reported on the utility of using screening to detect occult poisonings. While these papers support the value of the device for finding some occult cases, they do not provide data regarding the device's accuracy. Of those nearly 11,000 patients in the retrospective review, only 64 had blood COHb levels checked, and those values demonstrated a bias of −4.2, with limitations of agreement −16 to 7.5%.¹⁵ No information regarding accuracy or precision was reported in the case series.¹⁶

Other studies indicate that the Rad-57 is less accurate and precise when used in clinical practice.^17–19 In a prospective study by Touger et al, performed in the ED of a level 1 trauma and burn center, the authors enrolled 120 patients with suspected CO toxicity,¹⁷ measuring S_pCO by the Rad-57 while COHb levels were simultaneously drawn. The accuracy of the Rad-57 was 1.4% in this study, but the limits of agreement were −11.6 to 14.4% with a precision of 6.6%. Of concern, the Rad-57 detected only 11 of 23 patients with levels > 15%. Such imprecise S_pCO readings could result in an inordinate number of false positive and negative results and the associated clinical consequences of misdiagnosis.

A recent prospective study by Roth was reported as finding a positive outcome for noninvasive CO-oximetry, but this study had some important limitations.¹⁹ Of 16,108 patients screened in the ED with a portable CO-oximeter, only 1,578 patients had a blood gas drawn during their routine clinical care within 60 min of the screening, and 17 were ultimately diagnosed with CO poisoning. They found a bias of 2.32% and precision of 4.01%. This study may have suffered from selection bias, as the authors did not report why these patients had blood gases sent from the ED, and 714 patients were excluded for having a blood gas sent > 60 min after S_pCO screening. Furthermore, the limits of agreement were −5.7 to 10.37%, and the Bland-Altman plot showed that error increased with increasing COHb concentrations. Although the authors reported their findings as being within an acceptable range for clinical use, the variable precision is concerning.²⁰

Pre-hospital screening has previously been demonstrated to be feasible, although clinical outcomes of broad screening have not been described.^21–22 One retrospective review of over 1,700 cases of CO toxicity treated with hyperbaric oxygen demonstrated that patients who were first assessed by pulse CO-oximeter (n = 105 out of 1,711) had a statistically significantly shorter time to blood COHb measurement, by approximately 30 min, and were treated with hyperbaric oxygen an average of one hour earlier than patients who were first assessed by blood COHb.⁶ Differences in clinical outcomes were not assessed in this study, and this study did not assess the accuracy or precision of S_pCO measurements, as only patients with an elevated S_pCO and confirmatory COHb were evaluated.

The current study by Weaver et al is thus a valuable addition to the literature. Weaver and colleagues compared S_pCO to COHb levels in a convenience sample of 1,363 patients presenting to the ED of a single medical center in Utah.¹¹ Weaver and colleagues define 2 distinct false positive results: an accuracy false positive, and a screening false positive. The accuracy false positive is an S_pCO that is > 3% higher than the COHb; this outcome assesses the correctness of the manufacturer's claim that the Rad-57's accuracy is ± 3% of the actual COHb. The screening false positive is a clinical outcome, finding an S_pCO > 6% in nonsmokers when the actual COHb is < 6%. False negatives were similarly defined, but were analyzed descriptively, as the authors anticipated that they would not have sufficient numbers of CO poisoned patients to report sensitivity.

Although the device performed within the manufacturer's specifications, the false positive rates, as defined as accuracy or screening false positives, were 9% (122 patients) and 7% (77 patients), respectively, similar to other studies. These results contrast sharply with Touger et al, who found few false positives but several false negatives.¹⁷ Although Weaver did not report on the rate of screening false negatives, a concerning finding in this study is the nature of these false negatives. Even when the device detected a case of occult CO toxicity, the S_pCO reading was 13% while the COHb was 19%. In 3 other occult cases found by COHb levels, the Rad-57 reported an S_pCO of 0% when the COHb was > 10%. In addition, there was discrepancy in the S_pCO and COHb readings for 4 other patients for whom there was clinical concern for CO toxicity. Specifically, the S_pCO under-estimated COHb in all cases, with 2 S_pCO values falling into the range that would not have supported a diagnosis of CO poisoning (2% and 4%). Moreover, the precision of the Rad-57 appears to decrease as the COHb increases,^12,17,19 despite the manufacturer's assertion that the accuracy is ± 3% up to a COHb of 40%.^8–10 While several studies have found that the Rad-57 is likely to systematically overestimate the COHb, as compared to blood levels,^{17,19,23–25} Weaver found repeated underestimation by the device.

This study describes the performance of the Rad-57 CO-oximeter in clinical circumstances, as a tool with which to screen patients presenting to the ED to assess for occult CO toxicity. A strength of the study design is that a large random sample of all ED patients having blood drawn into a lithium heparinized tube were eligible for enrollment, not just patients having blood gases checked or patients with a high pre-test probability of CO poisoning. This reduces the risk of selection bias found in other studies.

This study also attempted to determine the importance of factors that may have impacted the correlation of the blood and screening values. The manufacturer has suggested that the device may perform poorly due to numerous user- or patient-level issues: inappropriate finger positioning in the sensor, inappropriately sized sensor for the subject's finger, timing of S_pCO and COHb measurements not being simultaneous, increased methemoglobin level, patient motion, external light interference, device or sensor malfunction.⁹ Given these considerations, Weaver examined several of these variables by multivariate analyses, using dichotomous logistic regression models. Weaver found female sex, lower perfusion index, higher methemoglobin saturation measurements, and the use of an individual device (“monitor C”) were associated with false positive results. While sex, perfusion index, and methemoglobin saturation levels are identifiable factors clinicians could take into account when considering S_pCO readings, the possibility of device-level variations in accuracy and precision may impact the fidelity of widespread screening.

The limitations of the current study are similar to other studies assessing the accuracy of S_pCO measurements. First, the values for clinically important cutoffs are debatable. Guidelines have recommended that the CO poisoning be defined by clinical symptoms in the setting of CO exposure, rather than absolute COHb levels.²⁴ As smokers have higher baseline CO levels than nonsmokers, normal ranges are generally identified differently for smokers and nonsmokers. Normal ranges for smokers have been reported to be < 2% or < 3%, but ranges for smokers may be as high as 10–20%.²⁰ Unexpectedly, in Weaver's study the mean COHb was 2.67% ± 1.49 percentage points for the 1,053 patients who reported to be non-smokers, and 25% had a COHb > 3%. The large number of non-smokers with COHb > 2% suggested that this upper laboratory limit should be reexamined. For comparison, 47% of non-smokers at another hospital had COHb levels that exceeded the reference range of < 2%. Roth et al used a receiver operating curve characteristic to define a cutoff value of < 6.6% to screen for CO poisoning.¹⁹ Evidently, appropriate clinical levels have yet to be defined.

Additionally, all of the available studies reflected near optimal use of the S_pCO device. Researchers were instructed on proper use prior to data collection, and in several studies, including Weaver's study, the Masimo Corporation provided training and technical support for the study personnel and the devices. Whether or not these devices would work as well in a “real-world” application, where those using the devices may receive minimal training and minimal technical support, is untested.

It is clear that the available literature provides heterogeneous results with regard to the accuracy, precision, and limits of agreement of S_pCO measured by the Rad-57, as compared to actual COHb levels.^{11–13,15,17,19,25} The most powerful use of a noninvasive COHb monitor would be to rapidly, accurately, and precisely determine if patients have occult CO toxicity: a simple screening test that reliably determines whether verification of CO toxicity with a blood test is warranted. The performance of the Rad-57 in this pragmatic study by Weaver and colleagues raises doubts about the device's ability to reliably serve that purpose. Although expecting 100% sensitivity in a screening test is unrealistic, the missed cases, along with the systematic underestimation, are concerning for a screening device. Although Weaver did not have sufficient CO toxicity cases to calculate sensitivity, the variation in performance, as characterized by the frequent underestimation of CO levels and cases of false negatives, are concerning when added to the existing literature. If clinicians are aware of the tendency of this device to have false positives, some may erroneously assume that the oximeter is overestimating levels, which is not always the case, as seen in this study and others. This perception may further exacerbate the risk of false negative readings. While this study found that the device operated within the specifications of the manufacturer, given the discordance of results reported in the available literature, broad reliance on the Rad-57 CO-oximeter for general S_pCO screening is premature. Clinicians must continue to have a high index of suspicion for CO poisoning and be aware of the limitations of CO oximetry, while all patients considered to be at risk must have confirmatory blood levels checked.

• Copyright © 2013 by Daedalus Enterprises