You are here

  • Home
  • Archive
  • Volume 32, Issue 4
  • An application of the learning curve–cumulative summation test to evaluate training for endotracheal intubation in emergency medicine

Article Text

Article menu
Original article
An application of the learning curve–cumulative summation test to evaluate training for endotracheal intubation in emergency medicine
  1. Sangmo Je1,
  2. Youngsuk Cho2,
  3. Hyuk Joong Choi3,
  4. Boseung Kang3,
  5. Taeho Lim3,
  6. Hyunggoo Kang3
  1. 1Department of Paediatrics, Paediatric Emergency Centre, Cha University Bundang Hospital, Gyunggi-do, Republic of Korea
  2. 2Department of Emergency Medicine, School of Medicine, The Catholic University of Korea, Seoul, Republic of Korea
  3. 3Department of Emergency Medicine, College of Medicine, Hanyang University, Seoul, Republic of Korea
  1. Correspondence to Professor H Kang, Department of Emergency Medicine, Hanyang University College of Medicine, 222 Wangsimni-ro, Seongdong-ku, Seoul 135-270, Republic of Korea; emer0905{at}gmail.com

Abstract

Objective The learning curve–cumulative summation (LC-CUSUM) test allows for quantitative and individual assessments of the learning process. In this study, we evaluated the process of skill acquisition for performing endotracheal intubation (ETI) in three emergency medicine (EM) residents over a 2 year period in their first 2 years of their EM residency.

Methods We evaluated 342 ETI cases performed by three EM residents using the LC-CUSUM test according to their rate of success or failure of ETI. A 90% success rate (SR) was chosen to define adequate performance and an SR of 80% was considered inadequate. After the learning phase, the standard CUSUM test was applied to ensure that performance was maintained.

Results The mean number of ETI cases required to reach the predefined level of performance was 74.7 (95% CI 62.0 to 87.3). CUSUM tests confirmed that performance was maintained after the learning phase.

Conclusions By using the LC-CUSUM test, we were able to quantitatively monitor the acquisition of the skill of ETI by EM residents. The LC-CUSUM could be useful for monitoring the learning process for the training of airway management in the practice of EM.

  • Airway
  • Education

https://doi.org/10.1136/emermed-2013-202470

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    While the Accreditation Council for Graduate Medical Education recommends that emergency medicine (EM) residents perform at least 35 endotracheal intubations (ETIs) prior to graduation,1 there is little evidence regarding the training required to successfully acquire ETI skills in EM. Several studies have made recommendations regarding the number of cases needed to learn ETI skills for different medical specialties. For example, it is recommended that anaesthesia residents perform 27–57 ETIs to reach 90% success.2–4 Student paramedics required exposure to more than 15–25 live ETI encounters across a range of clinical settings to achieve success rates greater than 90%.5 In the emergency department (ED) setting, the patient population for airway management includes critically ill and injured patients with some degree of instability.6 EM residents may have very different learning experiences given the different patient populations, clinical techniques and teaching conditions they encounter.

    Performing a recommended number of procedures under supervision is the standard method for procedure training in EM residents but there is a lack of objective evidence about individual competency. Recently, application of statistical methods have been used to analyse the learning curve (LC) of surgical and interventional procedures. Biau et al7 reviewed the literature from 1991 to 2006 and noted 22 papers that used the cumulative summation (CUSUM) test to determine the LC. Biau et al8 introduced a modification of the CUSUM method, called the LC-cumulative summation (LC-CUSUM) test, that was developed to signal when an individual’s performance has achieved a predefined level of competence. This new method has been applied to the analysis of the LC and quality control for monitoring clinical performance.9–11 The LC-CUSUM test, to the best of our knowledge, has not yet been well represented in the EM literature. The goal of this study was to evaluate the process of skill acquisition for performing ETI in three EM residents in their first 2 years of EM residency using the LC-CUSUM test. The primary outcome was the number of ETI cases required to pass the LC-CUSUM test.

    Materials and methods

    Study design

    The study was retrospective using prospectively collected data over a 24 month period from an airway management registry of two EDs. The institutional review board of the institution approved the study.

    Study setting and population

    The study was conducted at two tertiary urban university hospitals with annual ED visits of approximately 35 000 patients. The two EDs run 4 year EM residency (PGY 2–5) training programmes jointly, and accept three residents per year. Intubations are usually performed by attending faculty emergency physicians or EM residents. When the first year EM residents were allowed to intubate, the devices and methods used were chosen by supervising faculty members or senior residents.

    We reviewed all 1191 registry forms completed between March 2008 and February 2010, and found 342 ETI cases that had been attempted by all of the three EM residents who started their first year of EM residency in March 2008. All of the residents had completed 1 year internships before starting their EM residencies, and had little prior airway management experience (table 1). They received formal training on ETIs by an attending faculty emergency physician. They also scheduled a rotation in anaesthesia for at least 3 weeks before the start of their EM residency. The exclusion criteria included ETI cases that were intubated by means other than direct or video laryngoscopy, such as use of an elastic bougie, retrograde intubation, cricothyroidotomy, tracheostomy or nasotracheal intubation. In fact, in all of the 342 ETI cases that had been attempted by the three EM residents, intubated was achieved using the technique of direct or video laryngoscopic ETI. The successes or failures of the 342 ETI cases were included in the LC-CUSUM test.

    View this table:
    Table 1

    Baseline characteristics of the study participants and endotracheal intubation results for three emergency medicine residents over a 2 year period

    Methods and measurements

    Registry data were written on a standard single page data sheet for all ED patients that had been intubated by the physician who performed the intubation. During each morning round, the site investigator reviewed completed forms to prevent data omission. The main data collected included patient demographics, assessment of difficulty, intubation devices and methods used in each attempt, success or failure of each attempt, and immediate complications. ETI success was defined as the placement of an endotracheal tube within two attempts without immediate complications, such as oesophageal intubation or dental trauma. Successful intubation was confirmed by chest movement, auscultation and capnography. If intubation failed after two attempts, the faculty members or supervising senior residents took over the procedure and secured the airway. An ETI attempt was defined as a single pass of a blade into the mouth.

    Data analysis

    The LC-CUSUM test was developed to signal when performance was sufficiently far away from an inadequate performance level to consider the performance as adequate.8 It assumes that the process being studied is not being performed successfully (ie, the trainee is not proficient) at the start of monitoring and signals when the process can be considered to be competently performed. LC-CUSUM scores (Slc) were calculated from the successive outcomes and usually plotted below the x axis. The score Slc will fall with an ETI success and rise with an ETI failure. When Slc reaches a certain predefined limit, the LC-CUSUM test is said to emit an alarm indicating the performance has reached an adequate level.12 The LC-CUSUM test has a holding barrier of zero, which prevents the score from drifting too far away from the decision limit. When the graph hits this limit, it stays at zero and monitoring continues. In this way, the trainee will not have to compensate for all the accumulated failures to show acceptable performance.9

    In comparison, the CUSUM is designed to detect a shift from an adequate to an inadequate performance level. The successive outcome is recorded above the x axis. The CUSUM score (Sc) increases with an ETI failure and decreases with an ETI success. If the score Sc crosses the predefined limit, the CUSUM test is said to emit an alarm demonstrating inadequate performance. The CUSUM can be used after the LC-CUSUM has shown that the resident reached proficiency to ensure that the performance is being kept at an acceptable level. We established acceptable and unacceptable success rates as 90% and 80% based on previous studies2–5 of ETI skill acquisition and consensus among EM physicians in the department. For the LC-CUSUM, a limit of h=−3.15 was chosen on the basis of numerical simulations so that the risk of declaring a trainee proficient when his or her performance is inadequate (type I error) was limited to 10.0% over 100 ETI cases, and the risk of not declaring a trainee proficient although his or her performance is adequate (type II error) was limited to 16.2%. For the CUSUM, a limit of h=3.2 was chosen to yield type I and II errors of 9.4% and 10.2%, respectively, in 100 ETI cases. Detailed explanations with regard to calculation and formulation of the LC-CUSUM and CUSUM tests are given in the online supplementary appendix I.

    LC-CUSUM calculations were performed using Microsoft Excel 2010 (Microsoft, Washington, USA). Continuous variables are expressed as means with 95% CIs, while categorical variations are expressed in numbers and percentages. A p value of p<0.05 was considered statistically significant.

    Results

    A total of 342 ETI cases were performed by the three EM residents from March 2008 to February 2010. The mean number of ETI cases over a 2 year period per resident was 114.0 (95% CI 100.5 to 127.5), and the mean success rate of the ETI cases was 89.1% (95% CI 85.0 to 93.1) (table 1). All failed intubations by the residents were subsequently successfully completed by faculty members or senior residents.

    The mean number of ETI cases required to pass the LC-CUSUM test was 74.7 (95% CI 62.0 to 87.3). The LC-CUSUM graph for the residents is presented in figure 1. The LC of the residents showed the improvement of ETI skill (figure 2), with an 80% success rate after a mean of 30 cases. However, the pooled success rate did not reach 90% within a mean of 114 cases.

    Figure 1
    Figure 1

    The learning curve–cumulative summation (LC-CUSUM) and standard cumulative summation (CUSUM) graph regarding endotracheal intubation (ETI) of three residents of emergency medicine (EM). The ETI cases performed by the residents over a 2 year period in their first 2 years of EM residency were monitored. LC-CUSUM scores start at zero and scores decrease with each success by −0.118 and increase with each failure by +0.693. For as long as the LC-CUSUM score remains between the decision limit (h=−3.15, lower broken line) and the holding barrier (zero), the performance of the resident is regarded as inadequate. Resident Nos 1 (A), 2 (B) and 3 (C) passed the LC-CUSUM test at the 53th, 89th and 82th ETI case, respectively. The standard CUSUM test was applied thereafter to ensure that the level of performance was adequate. The upper broken line at +3.2 represents the decision limit for the standard CUSUM test.

    Figure 2
    Figure 2

    Mean success rate of endotracheal intubation (ETI) cases performed by the three emergency medicine residents.

    Discussion

    The LC-CUSUM and related statistical process control charts represent a more customised approach to individual LCs than the usual approach, which relies on uniformly applied measures of competency, such as performance of a specific number of procedures. We assessed the learning process of ETI skill for the three EM residents over a 2 year period in their first 2 years of EM residency. In this study, the LC-CUSUM signalled that the residents were competent for the ETI procedure after a mean of 74.7 ETI cases. In studies of emergencies outside of the operating room, whether taking place in the ED or in the intensive care unit, the risk of a difficult intubation was almost twice as likely.13 While rapid sequence intubation is widely used, EM residents may be allowed fewer attempts to achieve successful ETI on unstable patients in the ED than in the operating room. Less chances of attempting ETI might be the reason that the three EM residents needed more ETI cases to reach 90% success.

    The LC for most airway devices is biphasic: reasonable competency can be achieved within 30 cases but performance continues to improve even after 100 cases.14 A patient with a difficult airway demands a physician who has the expertise to modify his or her technique as necessary. Without deliberate practice, many practitioners will not advance beyond the stage of automaticity and will perform at mediocre levels for the rest of their careers.15 Individualised monitoring and training might counter arrested development by providing challenges outside a physician’s comfort zone as opportunities for deliberate practice. New standard guidelines for procedure training, such as ETI, would also be necessary and should be targeted to different stages of training. Acceptable failure rates are different not only for different medical specialties but also at different stages of training.16

    In the LC-CUSUM, introduced by Biau et al,8 the two initial parameters (acceptable (p0) and unacceptable failure rate (p1)) define the weight value; a weight for success and a weight for failure. An additional variable that influences the outcome of LC-CUSUM tests is the practical limit in number of cases available to each trainee (Nmax).17 If provided with a large enough number of cases, all trainees would eventually be considered competent by chance.18 However, for smaller values of Nmax, in the case of critical care, the true type I error rate, α, tends to increase and the type II error rate, β, tends to decrease. The value of h0 is the fourth parameter, which defines a terminating barrier. Biau et al12 recommended using simulation in order to achieve a false positive rate (type I error) equal to or less than 10% within a given Nmax.

    The LC-CUSUM test allowed us to quantitatively monitor competency among EM residents for ETI performance. After individualised assessment has been accomplished, competency based education against fixed achievement standards should be adopted for uniform educational outcomes.19 If the guideline on what is an acceptable success rate was set, we could adapt the LC-CUSUM test for different stages of training in ETI procedures. An easy tool for measuring performance quantitatively would aid the objective evaluation of resident performance, offering better training opportunities. Further studies should be performed with the goal of individualising training of EM residents, with an understanding of the diversity of the LC among EM residents.

    Limitations

    Some limitations of our study should be mentioned. First, the LC-CUSUM test was applied to a very limited sample. Thus ED data with different numbers of residents and different training environments might show different results. With this limitation, the LC-CUSUM test is valued as a potentially salient methodology to the task of individualised assessment in the high stakes environments of EM. Second, there are no agreed guidelines for the success rate of adequate and inadequate ETI skills acquisition. Different target rates at different stages of training could produce different results. Third, all outcomes in this study were self-reported and therefore may have been subject to self-reporting bias. Future studies should consider a prospective design with an attending EM physician present. Fourth, we explored a restricted competence in emergency airway management. Other important skills, such as rapid sequence intubation, assessing the environmental and external variables, and advanced intubation techniques, were not evaluated in this study.

    Conclusions

    We explored the usefulness of applying the LC-CUSUM test, relatively new to EM, to the task of establishing minimum competency criteria for a high risk procedure. We were allowed to quantitatively monitor the acquisition of ETI skill by EM residents using the LC-CUSUM test. The numbers of ETI cases needed to pass the LC-CUSUM test were variable. The LC-CUSUM test could be useful for monitoring a learning process for the training of airway management in the practice of EM.

    References

    1. ↵
      Accreditation Council for Graduate Medical Education: Emergency Medicine—Guidelines for Procedures and Resuscitations. http://www.acgme.org/acWebsite/RRC_110/110_guidelines.asp (accessed 17 Jan 2012).
    2. ↵
      1. Konrad C,
      2. Schupfer G,
      3. Wietlisbach M,
      4. et al
      . Learning manual skills in anesthesiology: is there a recommended number of cases for anesthetic procedures? Anesth Analg 1998;86:635–9.
      OpenUrlCrossRefPubMedWeb of Science
    3. ↵
      1. de Oliveira Filho GR
      . The construction of learning curves for basic skills in anesthetic procedures: an application for the cumulative sum method. Anesth Analg 2002;95:411–16.
      OpenUrlCrossRefPubMedWeb of Science
    4. ↵
      1. Mulcaster JT,
      2. Mills J,
      3. Hung OR,
      4. et al
      . Laryngoscopic intubation: learning and performance. Anesthesiology 2003;98:23–7.
      OpenUrlCrossRefPubMedWeb of Science
    5. ↵
      1. Wang HE,
      2. Seitz SR,
      3. Hostler D,
      4. et al
      . Defining the learning curve for paramedic student endotracheal intubation. Prehosp Emerg Care 2005;9:156–62.
      OpenUrlCrossRefPubMed
    6. ↵
      1. Sakles JC,
      2. Laurin EG,
      3. Rantapaa AA,
      4. et al
      . Airway management in the emergency department: a one-year study of 610 tracheal intubations. Ann Emerg Med 1998;31:325–32.
      OpenUrlCrossRefPubMedWeb of Science
    7. ↵
      1. Biau DJ,
      2. Resche-Rigon M,
      3. Godiris-Petit G,
      4. et al
      . Quality control of surgical and interventional procedures: a review of the CUSUM. Qual Saf Health Care 2007;16:203–7.
      OpenUrlAbstract/FREE Full Text
    8. ↵
      1. Biau DJ,
      2. Williams SM,
      3. Schlup MM,
      4. et al
      . Quantitative and individualized assessment of the learning curve using LC-CUSUM. Br J Surg 2008;95:925–9.
      OpenUrlCrossRefPubMedWeb of Science
    9. ↵
      1. Dessolle L,
      2. Fréour T,
      3. Barrière P,
      4. et al
      . How soon can I be proficient in embryo transfer? Lessons from the cumulative summation test for learning curve (LC-CUSUM). Hum Reprod 2010;25:380–6.
      OpenUrlAbstract/FREE Full Text
    10. ↵
      1. Papanna R,
      2. Biau DJ,
      3. Mann LK,
      4. et al
      . Use of the learning curve-cumulative summation test for quantitative and individualized assessment of competency of a surgical procedure in obstetrics and gynecology: fetoscopic laser ablation as a model. Am J Obstet Gynecol 2011;204:218.e1–9.
      OpenUrlCrossRefPubMed
    11. ↵
      1. Biau DJ,
      2. Milet A,
      3. Thévenin F,
      4. et al
      . Monitoring surgical performance: an application to total hip replacement. J Eval Clin Pract 2009;15:420–4.
      OpenUrlCrossRefPubMed
    12. ↵
      1. Biau DJ,
      2. Porcher R
      . A method for monitoring a process from an out of control to an in control state: application to the learning curve. Stat Med 2010;29:1900–9.
      OpenUrlCrossRefPubMed
    13. ↵
      1. Nolan JP,
      2. Kelly FE
      . Airway challenges in critical care. Anaesthesia 2011;66:81–92.
      OpenUrlCrossRefPubMedWeb of Science
    14. ↵
      1. Greaves JD
      . Training time and consultant practice. Br J Anaesth 2005;95:581–3.
      OpenUrlFREE Full Text
    15. ↵
      1. Baker PA,
      2. Weller JM,
      3. Greenland KB,
      4. et al
      . Education in airway management. Anaesthesia 2011;66:101–11.
      OpenUrlCrossRefPubMedWeb of Science
    16. ↵
      1. Sivaprakasam J,
      2. Purva M
      . CUSUM analysis to assess competence: what failure rate is acceptable? Clin Teach 2010;7:257–61.
      OpenUrlCrossRefPubMed
    17. ↵
      1. Sims AJ,
      2. Keltie K,
      3. Burn J,
      4. et al
      . Assessment of competency in clinical measurement: comparison of two forms of sequential test and sensitivity of test error rates to parameter choice. Int J Qual Health Care 2013;25:322–30.
      OpenUrlAbstract/FREE Full Text
    18. ↵
      1. Grigg OA,
      2. Farewell VT,
      3. Spiegelhalter DJ
      . Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Meth Med Res 2003;12:147–70.
      OpenUrlAbstract/FREE Full Text
    19. ↵
      1. McGaghie WC,
      2. Issenberg SB,
      3. Cohen ER,
      4. et al
      . Medical education featuring mastery learning with deliberate practice can lead to better health for individuals and populations. Acad Med 2011;86:e8–9.
      OpenUrlCrossRefPubMedWeb of Science

    Supplementary materials

    • Supplementary Data

      This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

      Files in this Data Supplement:

    Footnotes

    • SJ and YC contributed equally to this study.

    • Contributors SJ and YC: drafting of the manuscript, acquisition of the data and analysis of the data. HC, BK and TL: critical revision of the manuscript for important intellectual content. HK: study concept and design, interpretation of the data, critical revision of the manuscript for important intellectual content and final approval of the version to be published.

    • Competing interests None.

    • Ethics approval The study was approved by the institutional review board of Hanyang University Seoul Hospital.

    • Provenance and peer review Not commissioned; externally peer reviewed.