Abstract
BACKGROUND: Symptoms of carbon monoxide (CO) poisoning are non-specific. Diagnosis requires suspicion of exposure, confirmed by measuring ambient CO levels or carboxyhemoglobin (COHb). An FDA-approved pulse oximeter (Rad-57) can measure CO saturation (SpCO). The device accuracy has implications for clinical decision-making.
METHODS: From April 1 to August 15, 2008, study personnel measured SpCO and documented demographic factors at time of clinical blood draw, in a convenience sample of 1,363 subjects presenting to the emergency department at Intermountain Medical Center, Murray, Utah. The technician then assayed COHb. COHb and SpCO values were compared by subject; false positive or negative values were defined as SpCO at least 3 percentage points greater or less than COHb level, reported by the manufacturer to be ± 1 SD in performance.
RESULTS: In 1,363 subjects, 613 (45%) were male, 1,141 (84%) were light-skinned, 14 in shock, 4 with CO poisoning, and 122 (9%) met the criteria for a false positive value (range 3–19 percentage points), while 247 (18%) met the criteria for a false negative value (−13 to −3 percentage points). Risks for a false positive SpCO reading included being female and having a lower perfusion index. Methemoglobin, body temperature, and blood pressure also appear to influence the SpCO accuracy. There was variability among monitors, possibly related to technician technique, as rotation of monitors among technicians was not enforced.
CONCLUSIONS: While the Rad-57 pulse oximeter functioned within the manufacturer's specifications, clinicians using the Rad-57 should expect some SpCO readings to be significantly higher or lower than COHb measurements, and should not use SpCO to direct triage or patient management. An elevated SpCO could broaden the diagnosis of CO poisoning in patients with non-specific symptoms. However, a negative SpCO level in patients suspected of having CO poisoning should never rule out CO poisoning, and should always be confirmed by COHb.
Background
Carbon monoxide (CO) is a common source of accidental poisoning and results in more than 50,000 emergency department visits per year in the United States.1 Because the symptoms of CO poisoning are non-specific,2 diagnosis requires clinical suspicion of CO exposure, with confirmation by measurement of ambient levels of CO where the exposure occurred, or measurement of carboxyhemoglobin (COHb) levels. COHb can be measured from either arterial or venous blood3 using CO-oximetry techniques.4
Traditional noninvasive pulse oximeter devices do not distinguish between oxyhemoglobin and COHb.5 The United States FDA has cleared a pulse oximeter (Rad-57, Masimo, Irvine, California) that measures the saturation of blood with CO (SpCO), in addition to oxyhemoglobin saturation. Later models of this device also measure methemoglobin (SpMet) and hemoglobin (SpHb). This device has been used in emergency departments to screen for occult CO poisoning6,7 and in the pre-hospital environment by first responders.8–10
The device manufacturer reports that approximately 68% of SpCO measurements fall within ± 3% of COHb measurements up to 39.9%.11 In this case, “3%” is not 3% of the value: rather, 3 percentage points on a percent scale. For example, if the SpCO equals 10%, the expected “true” value of COHb would be 7% to 13%. In this paper the term percentage points describes device tolerance or accuracy.
A validation study in healthy volunteers reported a precision for this device of 2.2 percentage points, compared to COHb levels up to 15%.12 In a larger study of unselected emergency department patients, the precision was reported as 3.27 percentage points, with a bias between SpCO and COHb of 2.99 percentage points.13 Studies in patients presenting at a burn center14 and emergency department12 with suspected CO poisoning report good correlation between the 2 measurements, with both studies finding a slight overestimation with SpCO. However, in another study of emergency department patients with suspected CO poisoning, the Rad-57 correctly identified only half of patients with COHb > 15%.14
The published literature lacks information about the false positive rate of this oximeter.12–17 The false positive rate is important for clinical decision-making because if the monitor overestimates the actual COHb value, first responders might endorse transport of a non-poisoned patient, or clinicians could be prompted to verify the elevated SpCO value with an invasive confirmatory test (ie, COHb). In addition, if a large study were ever conducted to determine the incidence of occult CO poisoning in patients presenting to emergency departments, the false positive rate of noninvasive measures would be necessary in order to determine how many COHb levels would have to be measured to confirm CO exposure. To answer this question and provide this important piece of information for clinical decision-making, we conducted a prospective study to determine the false positive rate of SpCO measurements in patients presenting to a level one trauma center emergency department.
QUICK LOOK
Current knowledge
Carbon monoxide (CO) is a common source of accidental poisoning, resulting in over 50,000 emergency room visits per year in the United States. The symptoms of CO poisoning are nonspecific, and diagnosis requires clinical suspicion of CO exposure, along with measurement of carboxyhemoglobin (COHb) levels.
What this paper contributes to our knowledge
The noninvasive measurement of CO by pulse oximetry (SpCO) is not sufficiently accurate to direct triage or patient management. Clinical suspicion of CO poisoning and an elevated SpCO should be confirmed with blood CO measurement by CO-oximetry. A negative SpCO level in patients suspected of having CO poisoning should never rule out CO poisoning, and should always be confirmed by COHb.
Methods
The institutional review board of the Urban Central Region of Intermountain Health Care approved this research. Informed consent was waived by the institutional review board.
A convenience sample of patients presenting to the emergency department from April 1 to August 15, 2008, at Intermountain Medical Center, Murray, Utah, were eligible to participate in this prospective study. Of subjects having a lithium heparin tube of blood drawn for clinical purposes, study personnel measured the SpCO with the Rad-57 pulse oximeter (Masimo, Irvine, California) at the time of the blood draw. The technician first attempted to obtain a measurement using the ring finger. If the technician could not obtain a pulse oximetry measurement using the ring finger, the probe was moved to the middle finger, and lastly to the index finger if no measurement could be obtained from the middle finger. If the initial SpCO measurement on the ring finger read greater than 10, the technician performed a second pulse oximetry measurement on the middle finger, and the lower SpCO measurement was recorded.
After obtaining the pulse oximetry measurement, the technician withdrew 1 mL of blood from the lithium heparin tube, with a blood gas syringe. This sample was taken to the blood gas laboratory adjacent to the emergency department and assayed by CO-oximetry (ABL 825, Radiometer, Copenhagen, Denmark).
The following de-identified information was collected for each subject: date of encounter, age, sex, chief complaint, blood pressure, breathing frequency, temperature, supplemental oxygen delivery rate and method, nail polish color, capillary refill time, smoking status, skin color, and whether or not the finger where the oximeter was placed was cold to touch.
Data recorded from each pulse oximetry measurement were the pulse oximeter oxygen saturation (SpO2), heart rate, SpCO, SpMet, perfusion index, and whether or not the finger probe had to be changed to another digit because it would not display a value. Data recorded from the clinical blood gas instrumentation were the source of the sample (arterial or venous blood), the time of the blood draw, hemoglobin, oxyhemoglobin, COHb, methemoglobin (MetHb), and which technician obtained the sample.
Three emergency department phlebotomists, dedicated to the emergency department, and 2 senior blood gas technicians obtained the measurements and recorded the clinical information into an electronic database. Data were collected when the phlebotomists and technicians were able to perform this activity, generally not when assigned to clinical duties. A representative from the pulse oximeter manufacturer trained, on-site, the phlebotomists and blood gas technicians in proper use of the oximeters, including the proper location of the probe on the finger. Four pulse oximeter devices and 3 CO-oximeters were used in this study. The pulse oximeter devices were new from the manufacturer, and the blood gas instrumentation was maintained by the hospital blood gas department with laboratory accreditation through the College of American Pathologists, and following all Clinical Laboratory Improvement Amendments and College of American Pathologists guidelines for proficiency. The pulse oximeter devices were stored in a secure locker, and the technicians selected a monitor based on convenience (monitor selection was not randomized or controlled).
At the conclusion of the study, the Rad-57 devices were returned to the manufacturer for performance validation and were found to be in good working order.
Definition of False Positive
For this analysis, the blood COHb test was considered the gold standard, and the SpCO values and blood COHb levels were compared by subject. False positives were defined by 2 methods. The first method, accuracy false positive, defined a false positive event when a subject's SpCO was at least 3 percentage points greater than his or her COHb level, in accordance with the manufacturer's allowed limits of precision.11,12 While this method can be used to provide information about the accuracy of the device, it does not provide much useful information for clinical decision-making, as it does not account for where in the overall range the measurements lie: just the difference between them. For example, an SpCO measurement of 4% and a COHb measurement of 1% would be a false positive event, but given the stated precision of the pulse oximeter as well as the reference range for the CO-oximeter, an SpCO measurement of 4% in a patient would not necessarily indicate exogenous CO exposure or prompt further evaluation for CO poisoning on the part of the clinician.
The second method used to define a false positive event attempted to address this clinical perspective: a screening false positive. The reference range for COHb in non-smokers at the study site is 0–2%, based on work to establish normal arterial blood gas values at an elevation of 1,400 m,18 but other accepted reference ranges have included values ≤ 3% in non-smokers.19 Given the commonly accepted reference ranges and the precision of the pulse oximeter, the investigators reasoned that an SpCO level > 6% in a non-smoker would prompt verification by COHb. For this scenario, a false positive event was defined as an SpCO level > 6% with a COHb level ≤ 6% in a non-smoker.
False Negatives
With the projected sample size of this cohort (1,700 subjects), it was unlikely that this study could determine the false negative rate, since CO poisoning would be uncommon in a sample of this size. If patients with CO poisoning and elevated COHb levels were enrolled in this study, the comparison of the SpCO to the COHb level would be made descriptively.
Statistical Methods
False positives were identified using the definitions above. Using the collected demographic elements and blood gas results, statistical analyses were performed to assess potential association between the false positive readings and potential explanatory variables. Initially, frequency tables were created to examine potential relationships between false positive events and variables known to affect oximetry accuracy, including elevated MetHb,11,12 as well as skin color, oxygen saturation, and sex.20 A bivariate analysis (Spearman correlation coefficient) was performed to determine correlation among the independent variables.
For multivariate analyses, dichotomous logistic regression models were fit to the data. False positives (SpCO > 6% where COHb ≤ 6%) were modeled against similar negatives (SpCO and COHb both ≤ 6%). Each analytical variable was run separately in the model to produce unadjusted odds ratios, then the model was fit with all the variables to produce adjusted odds ratios. A model was also examined using the accuracy false positive scenario (SpCO – COHb ≥ 3 percentage points) versus negatives (SpCO – COHb < 3 percentage points) and fit using methods similar to the above model. A variable was considered statistically significant at the P < .05 level.
Because of the small set of false positive events for some variables, the logistic regression models were closely examined, and forward selection methods were used to arrive at reduced models. Forward selection was chosen instead of backward selection, as forward selection tends to keep only the strongest terms and sometimes slightly under-fits a model, where backward selection is more likely to over-fit a model. This analysis began with a base model with intercept, and the strongest terms by score test (at a .05 level for iterative entry) were added sequentially until successive terms did not result in significant change to the model. Residual chi-square tests were examined to determine if the remaining variables as a whole would significantly improve the model. Overall fit was examined using the Hosmer and Lemeshow statistic.
The continuous variables were categorized by quartiles to check for any nonlinear patterns in estimates and odds ratios. The few continuous variables that appeared to have nonlinearity were all insignificant in the final model, and many had too few events to test for true nonlinearity. Therefore, the continuous variables were fit linearly to the final model instead of being fit as quartile or tertile categories.
Interaction effects were considered, but were not selected as part of the forward selection process, and seemed to over-fit the variation in the data. The interaction terms also added too many potential variables to be realistically fit to our sample size, so interaction effects were not used in the final model. Statistical analysis was performed using statistics software (SAS 9.1.3, SAS Institute, Cary, North Carolina).
Results
From April 1 to August 15, 2008, study personnel collected complete data (SpCO, venous blood gas, and demographic information) in 1,363 subjects presenting to the emergency department. Of these, 45% (613) were male and 84% (1,141) were light-skinned. By blood pressure measurement, 1% (14) were in shock, and an additional 4 subjects carried an emergency department diagnosis of CO poisoning. Baseline characteristics are described in Table 1. Descriptive blood CO-oximetry and Rad-57 pulse oximetry data are presented in Table 2.
Baseline Characteristics
Blood Oximetry and Pulse Oximetry Data
The Rad-57 device performed within the parameters reported by the manufacturer11: 73% of SpCO values fell within 3 percentage points (1 SD) of COHb measurements, and 95% fell within 6 percentage points (2 SD) of COHb measurements (Fig. 1).
Carboxyhemoglobin measured noninvasively (SpCO) versus measured by blood (COHb). The limit lines around the line of identity represent the accuracy range of ± 3 percentage points, reported by the manufacturer to be ± 1 standard deviation, which encompasses 68% of the sample. In this study, 73% of subjects fell within this range. Twenty-three individuals had SpCO levels > 10%, and 4 had COHb > 15%. By chance, there were 2 subjects with a COHb value of 19.1% and an SpCO of 13%.
By the accuracy false positive definition (SpCO − COHb ≥ 3 percentage points), 122 subjects (9%) met the criteria for accuracy false positivity (range 3–19 percentage points). The device was more likely to underestimate COHb: by a similar definition for accuracy false negatives (SpCO − COHb ≤ –3 percentage points), 247 subjects (18%) met the criteria for accuracy false negative (range −13 to −3 percentage points).
By the screening false positive definition (SpCO level > 6% with a COHb level ≤ 6%), 77 of 1,169 subjects (7%) met the criteria for a screening false positive event. Of these subjects, 52 were non-smokers by self-report. Of the 77 screening false positives, 71 also met the definition for accuracy false positive (difference between SpCO and COHb ≥ 3 percentage points). False positive events are described in Table 3. COHb and SpCO measurement comparisons, by individual, are shown in Figure 1. The distribution of SpCO versus COHb is depicted in Figure 2.
Frequency of False Positive Events
Distribution of the differences between carboxyhemoglobin measured noninvasively (SpCO) versus via blood (COHb), rounded to the nearest full percentage point. The dashed lines around zero represent the accuracy range of ± 3 percentage points, reported by the manufacturer to be ± 1 standard deviation. There are 3 individuals with SpCO − COHb ≤ −11, and 8 individuals with SpCO − COHb ≥ 11. As presented in this figure, the data support that these monitors more frequently underestimated COHb.
Due to low sample sizes for false positive events, some variables were merged to provide sufficient events for regression analysis. Skin color was re-categorized by merging medium and dark, and nail polish was also categorized by merging none and clear, then merging all colors into one group, causing nail polish to become a yes/no variable. In addition, results for monitor D (3 events) were combined with monitor B, identified as behaving most similarly, with respect to false positives, to monitor D by Pearson chi-square tests, and verified by the Fisher exact tests.
From the frequency tables, the data appeared to exhibit potential relationships between false positive events and variables previously reported to affect oximetry accuracy (elevated MetHb, skin color, and sex) (see Table 3).
Risk factors for false positive events were identified through multivariate analysis (logistic regression after forward selection) (Table 4). Risk factors for false positivity using both definitions include being female, use of monitor C, lower perfusion index, and higher SpMet measurement. The analysis showed a possible interaction between monitor C and 2 technicians; however, those 2 technicians also used monitors A and B, though less often, where no technician/monitor interaction was found.
Adjusted Odds Ratios for Variables Significantly Increasing Risk for False Positivity
By analysis of accuracy false positive events (SpCO –COHb ≥ 3 percentage points), higher body temperature also increased risk for false positivity, while for screening false positive events (SpCO level > 6% with a COHb level ≤ 6%), reported smoking, lower blood pressure, and higher MetHb increased risk. The Homer and Lemeshow test statistics showed good fit for both models (P = .42 for accuracy false positive, P = .53 for screening false positive). Skin color, finger temperature, age, capillary refill time, and breathing frequency were not significant risk factors for false positivity.
Of the 4 individuals who presented to the emergency department for CO poisoning, the SpCO significantly underestimated the COHb. In subjects with COHb levels of 35% and 27%, the SpCO was 31% and 17%, respectively. In 2 subjects seen for CO poisoning with lower COHb levels (8.7% and 8.4%), the SpCO by Rad-57 would not have supported a diagnosis of CO poisoning (4% and 2%, respectively). The Rad-57 device would likely have identified one case of occult CO poisoning (a non-smoker evaluated for shoulder pain with SpCO 13% and COHb 19.1%), but missed 3 potential cases of occult CO poisoning (non-smokers with SpCO 0% but COHb > 10%, evaluated for abdominal pain, psychiatric complaint, and fall). There was only one smoker whose COHb measurement suggested CO poisoning (19.1%), who may or may not have been identified through SpCO measurement (13%).
Fifteen non-smokers had SpCO measurements ≥ 10%. While 3 of these subjects had COHb levels indicating CO poisoning, 7 had COHb ≤ 3%, indicating a spurious SpCO reading, and the other 5 had COHb measurements ranging from 3.8% to 6.4%, suggesting exogenous CO exposure.
Unexpectedly, of the 1,053 subjects who reported to be non-smokers, the mean COHb was 2.67% ± 1.49 percentage points. Six hundred fifty-eight (62%) had a COHb > 2%, the upper reference value used by the institutional laboratory, and 262 (25%) had a COHb > 3%. While some patients might misrepresent smoking status, and secondhand smoke or occult CO exposure could explain some subjects with elevated COHb, the large number of non-smokers with COHb > 2% suggested that this upper laboratory limit should be re-examined. For quality purposes, our medical informatics department queried COHb measurements in non-smoking non-neonate patients at the institution where this work was performed and another network hospital (where CO-oximetry, including COHb, is measured on every blood gas performed). From January 1, 2008, to December 31, 2009, 5,267 of 39,479 (13%) of COHb measurements in all non-smokers exceeded the laboratory reference range of ≤ 2%,18 while 372 of 785 (47%) of non-smokers presenting to the emergency department had COHb levels that exceeded the reference range.
In addition, 20 subject blood samples were analyzed on each of the 4 ABL 825 blood gas analyzers at the research site, as well as an OSM3 (Radiometer, Copenhagen, Denmark), the machine on which the institution's reference ranges were established.18 The mean COHb by the OSM3 was 0.9% (range 0.1–2.5%), while the mean by the ABL 825 analyzers was 1.5% (range 0–3.6%). In only 2 of the 80 head-to-head comparisons were the COHb measurements by OSM3 greater than those by an ABL 825, both on a single machine (difference range −1.2% to 1.6%).
Discussion
In this prospective study, the Rad-57 device performed within the manufacturer's specifications. The accuracy and screening false positive rates were 9% and 7%, respectively. The device was more likely to underestimate COHb than overestimate this parameter (see Fig. 2). Risks for a false positive SpCO reading included being female and having a lower perfusion index. MetHb, body temperature, and blood pressure also appear to influence the SpCO accuracy. There was variability among monitors, possibly related to technician technique, as rotation of monitors among technicians was not enforced. Our study technicians were trained by the manufacturer and taught one-on-one proper use of the Rad-57 device. Over the interval of the study, we interacted frequently to verify that they understood proper measurement technique. We do not know what training the manufacturer provides to clinicians, including pre-hospital care providers, but we suspect the training provided in this study might have been more thorough than to the majority of clinical end-users of the Rad-57. Therefore, if some error in SpCO measurement in this study is due to improper technician technique, one would expect this problem to be present in the clinical setting as well, perhaps to a greater degree.
This study was underpowered to determine the false negative rate of the Rad-57 device, and had too few subjects with elevated COHb to provide meaningful data for sensitivity and specificity. However, the limited data collected in this study suggest that the Rad-57 will miss some subjects with clinically important CO poisoning. Other limitations of this study include the predominantly white Salt Lake City population, which may limit generalizability of these results to areas with a different racial demographic. In addition, the study population includes only subjects presenting at the emergency department during hours of convenience, in whom blood was drawn for other clinical reasons, and may not represent the emergency department population at large. Follow-up information about possible occult CO poisoning is not available due to the de-identified nature of data collection. Neither SpCO nor COHb was included in the subject's medical record, so patient care was not influenced by either result. In this study, the SpCO measurement was recorded at the time of blood draw. The technicians attempted to analyze the COHb promptly, but even if the COHb analysis was performed minutes to hours after obtaining the blood, we expect no change in COHb across this brief period of time. Hampson has shown COHb to be stable in heparinized blood up to 28 days after blood draw.21
Conclusions
In this study the COHb of non-smokers was mildly higher than expected. Validation work supported this finding on a larger scale, especially in emergency department patients. The clinical impact of this finding is negligible. Possible explanations for COHb levels in non-smokers greater than the laboratory reference range include occult exogenous CO exposure, an inaccurate reference range for our present blood gas instrumentation, and the possibility that the subjects failed to accurately report their smoking history.
While the Rad-57 pulse oximeter functioned within the manufacturer's specification of ± 3 percentage points (representing 1 SD, or 68% of measurements that fall into this range), its operating range is not sufficiently accurate to direct triage or patient management. Clinicians using the Rad-57 should expect some SpCO readings to be significantly higher or lower than COHb measurements, and should consider the probability of CO exposure when utilizing this device. Symptoms such as headache or flu-like symptoms, similar illness among family or co-workers, or illness that seems to resolve during the day or at night are consistent with CO exposure, and an elevated SpCO could help broaden the diagnosis of CO poisoning in patients with non-specific symptoms. However, a negative SpCO level in patients suspected of having CO poisoning should never rule out CO poisoning, and should always be confirmed by COHb.
In the event that the SpCO value is unexpectedly elevated, including in the absence of symptoms, we advise confirmation with COHb; a missed diagnosis of CO exposure can result in CO poisoning with associated morbidity and mortality.
Acknowledgments
The authors thank Masimo Corporation for loaning the Rad-57 oximeters and for educating the research staff in their proper use. We thank Brian Spruell of SciMetrika, a contract research organization, for assistance with the statistical analysis. We also thank the emergency department and blood gas technicians who participated in data collection, as well as Dr Scott Evans and Kathy Clark for their assistance in evaluating normal COHb levels of non-smokers at our institutions.
Footnotes
- Correspondence: Lindell K Weaver MD, Hyperbaric Medicine, LDS Hospital, 8th Avenue and C Street, Salt Lake City, UT 84143. E-mail: lindell.weaver{at}imail.org.
The authors have disclosed relationships with SciMetrika and Masimo. This work was supported by a grant from the Centers for Disease Control, through and with additional support by SciMetrika. Masimo provided the oximeters for research use.
Dr Weaver presented a version of this paper at the annual scientific meeting of the Undersea and Hyperbaric Medical Society, held June 4, 2010, in St Pete Beach, Florida.
See the Related Editorial on Page 376
- Copyright © 2013 by Daedalus Enterprises
