Assessment of the learning curve of peribulbar blocks using the Learning-Curve Cumulative Sum Method (LC-CUSUM): an observational study

Abstract

Introduction:  This study aimed to assess the learning curves of peribulbar anesthesia and estimate the number of blocks needed to attain proficiency.

Methods:  Anonymized records of sequential peribulbar blocks performed by first-year anesthesia residents were analyzed. The block sequential number and the outcomes were extracted from each record. Success was defined as a complete sensory and motor block of the eye, and failure was defined as an incomplete block requiring supplemental local anesthetic injections or general anesthesia. Learning curves using the LC-CUSUM method were constructed, aiming for acceptable and unacceptable failure rates of 10% and 20%, and 10% probability of type I and II errors. Simulations were used to determine the proficiency limit h₀. Residents whose curves reached h₀ were considered proficient. The Sequential Probability Ratio Test Cumulative Sum Method (SPRT-CUSUM) was used for follow-up.

Results:  Thirty-nine residents performed 2076 blocks (median = 52 blocks per resident; Interquartile Range (IQR) [range] = 27–78 [4–132]). Thirty residents (77%) achieved proficiency after a median of 13 blocks (13–24 [13–24]).

Conclusions:  The LC-CUSUM is a robust method for detecting resident proficiency at peribulbar anesthesia, defined as success rates exceeding 90%. Accordingly, 13 to 24 supervised double-injection peribulbar blocks are needed to attain competence at peribulbar anesthesia.

KEYWORDS Nerve Block/methods; Internship and residency/methods; Learning; Models, Theoretical

Introduction

Cumulative sum methods have been used to follow the performance of individual residents to define the point where proficiency has been attained, according to pre-established criteria.¹, ² Cumulative sum methods are based on sequences of successes and failures at the target procedure. A sequential probability test added to the CUSUM method tests the stability of process performance and has been used to follow the learning curve of physicians in several anesthetic and surgical procedures.¹, ² More recently, the Learning Curve CUSUM (LC-CUSUM) has been specifically designed for monitoring procedural learning curves. The LC-CUSUM method departs from the premise that instability is expected during learning and proficiency is characterized by performance stability.³ Thus, the LC-CUSUM method fits better performance monitoring during the learning phase, while the SPRT-CUSUM method is more appropriate for following performance after proficiency. The two methods have been used to monitor the learning phase of medical procedures or to follow performance after proficiency has been achieved.³, ⁴, ⁵ Based on published data or expert consensus, success and failure rates are defined a priori.

Performance monitoring methods help tailor resident supervision needs. Thus, a high accuracy of the monitoring method is desirable. The parameters of choice have been demonstrated to strongly influence the performance of cumulative sum methods.⁶ Therefore, testing the method’s accuracy in distinct scenarios is needed to help define its usefulness and applicability to specific procedures.

This study’s primary hypothesis was that the LC-CUSUM method might detect when anesthesia residents attain proficiency at peribulbar anesthesia, defined as a higher than 90% success rate. Secondarily, we hypothesized that the median number of blocks needed to attain proficiency could be estimated from the study population. This study was designed to test the hypotheses mentioned above.

Methods

With the approval of the Research Ethics Committee of the Hospital Governador Celso Ramos (protocol identification number: 42224821.2.0000.5360), data of 2076 peribulbar blocks performed from March 2008 to October 2013 by 39 first to third-year anesthesia residents were retrieved from a database primarily designed to collect data on in-training resident performance at interventional anesthesia procedures. Original data were collected and used to follow resident in-training performance based on formative feedback based on learning curves. The web-based system did not contain any information capable of identifying patients. Residents’ written informed consent was obtained for using data on their learning curves for this study. This manuscript complies with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement.

Standardized anesthesia technique

Blocks were performed with the patient lying supine. Routine monitoring included Pulse Oximetry (SpO₂), continuous electrocardiography, and noninvasive blood pressure measurement. A peripheral intravenous line was secured before block placement, and 5% glucose or Ringer’s solution was started to keep the line patent. Oxygen was started at 2 L. min⁻¹ through a nasal catheter. Fentanyl 0.5–1 μg.kg⁻¹, and midazolam (0.01–0.02 mg.kg-1 or propofol (0.2 mg. kg⁻¹) incremental boluses were administered to obtain analgesia, amnesia, and sedation for the block. After disinfection with povidone-iodine 10% or chlorhexidine 2% topical solution, small intradermal wheals of lidocaine 0.5 ml were placed at the junction of the middle two-thirds and outer one-third of the eye just above the infraorbital rim, and inferior and medial to the supraorbital notch. The inferior injection was performed first, with the patient looking straight ahead. The needle was inserted at a 90-degree angle to the skin at the inferior orbital margin and advanced 1–2 cm along the orbital floor, aiming at the equator of the eye globe. After negative aspiration for blood or cerebrospinal fluid, 4–6 mL of local anesthetic solution was deposited as the needle was withdrawn into the pre-septal space. The delivered volume depended on the size of the orbit and the manual assessment of orbital tension. Manual compression was applied to the eyeball for 3–5 minutes before the second injection, for which patients were asked to direct their gaze slightly downwards. The second injection was made below and medial to the supraorbital notch, with the needle directed perpendicularly for 1 cm, using 2–3 mL of the same local anesthetic solution. Further manual compression was applied to the eyeball for 3–5 minutes.

Data collection

Our residency program exposes trainees to ophthalmic anesthesia throughout the three-year training period. Anesthesia for ophthalmic surgery is performed in a surgical suite staffed by a dedicated team of attending anesthesiologists. Novices usually observe attendings perform various ophthalmic blocks while being instructed about the technical aspects of the block. Residents must acquire theoretical knowledge on the anatomy, technique, local anesthetic and adjuvant choices, identification, and management of complications through self-study and are probed by the attending. Residents demonstrating sufficient knowledge (according to the attending’s subjective assessment) can perform blocks under regressive supervision.

At the beginning of the data collection, the residents were instructed as to success and failure criteria. The resident who performed the block registered the outcome in a dedicated database immediately after the procedure. A successful block was defined as complete akinesia and anesthesia within 10–15 minutes from block placement. In case of failure, the assistant anesthesiologist took over the procedure and performed additional local anesthetic injections or induced general anesthesia, and the procedure was considered a failure. Procedures were also considered failures if complications related to the peribulbar block caused the postponement of the surgical procedure.

Determination of the proficiency limit for the LC-CUSUM curve

Before constructing the learning curves, the proficiency limit (h₀) for the LC-CUSUM was determined by mathematical simulations. Simulated samples of 10,000 residents performing 50 procedures each were created using a Visual Basic for Applications (VBA) routine on Microsoft Excel (Microsoft Corporation, Redmond, VA, USA).⁷ Runs consisted of a Bernoulli sequence generated for each simulated resident using the success rates from 10 procedure success/failure sequences observed during the first 50 procedures performed by actual residents in the study sample to simulate their performance. Accordingly, success rates were 90% for the first ten procedures, 97% for procedures 11–20, 98% for procedures 21–30, 97% for procedures 31–40, and 100% for procedures 41–50. For estimating h₀, simulation runs were iteratively repeated, adjusting the probabilities of acceptable failure rates (p₁) at 1% increments starting at p₁ = 0.02 with fixed p₀ = 2p₁ and adjusting h₀ values at 0.05 increments starting from h₀ = −2. After each run, type I and II error rates were estimated by comparing the actual success rate when the individual simulated resident’s LC-CUSUM curve crossed the h₀ limit from above. Accordingly, the LC-CUSUM sequence was assigned type I error if the cumulative success rate was lower than 1 – p₀ (inadequate performance). Type II error was assigned when the cumulative success rate was equal to or higher than 1 – p₀ at the end of the Bernoulli sequence (adequate performance), but the LC-CUSUM curve did not reach the h₀ decision limit. Type I and II error rates were calculated as the average number of LC-CUSUM sequences assigned as type I or II error cases. The final value of h₀ and the average number of procedures until proficiency for constructing LC-CUSUM curves for residents in the study sample were defined after obtaining type I and II error rates at or below 10% within an average run length between 20 and 30 procedures, the number of procedures needed for a decision.

LC-CUSUM and SPRT-CUSUM calculations

A downward-running LC-CUSUM curve was constructed for each resident in the study sample by sequentially computing S_t = min (0, S_t-1-W_t) scores based on the resident’s individual success/failure sequences (X_t), being X_t = 0 for success or X_t = 1 for a failure. Accordingly, S_t started at 0 and decreased if X_t = 0 or increased X_t = 1 by a log-likelihood ratio score (W_t), with p₀ = 20% the acceptable failure rate under the null hypothesis (inadequate performance) and p₁ = 10% the expected failure rate at proficiency (the alternative hypothesis).

For residents whose LC-CUSUM curves reached the proficiency barrier h₀ = −1.650, an SPRT CUSUM control chart was initiated, from zero, according to the following formula: S_t = max (0, S_t−1 – s), for success or St = max (0, S_t−1 + [1 – s]), for a success. Parameters for the SPRT-CUSUM curves were: p₀ (now, the acceptable failure rate) = 10%, and p₁ (now, the unacceptable failure rate) = 20%, type I (α) = type II (β) error rate = 10%. These parameters were used to estimate weight s and h₁. The curve was kept flat by setting an absorbing barrier at zero for success so that only performance deterioration was detected by SPRT-CUSUM monitoring after proficiency had been attained according to the LC-CUSUM method. Upward shifts would detect deteriorating performance in the SPRT-CUSUM curve, reaching a resetting unproficiency limit (h₁) set at 2.71. Formulae for LC-CUSUM and SPRT-CUSUM calculations are available in the Appendix.

Results

Resident performance monitoring started after a median of 2.5 months after admission (25^th–75^th percentiles = 1–3; range = 1–13). The upper limit corresponds to residents admitted the year before adopting the learning curve monitoring. Monitoring was terminated at a median of 16 (25^th–75^th percentiles = 12–24; range 8–33) months in training, corresponding to a median duration of monitoring of 12 months (25^th–75^th percentiles = 9–21; range 5–31 months). Most blocks were performed during the first training year 1856 (89.4%), followed by 189 (9.1%) during the second, and 31 (1.5%) in the third training year.

LC-CUSUM parameters obtained by simulation

Assuming type I (α) = type II (β) statistical error rates equal to 10%, an acceptable failure rate (p₁) = 10%, and an unacceptable failure rate (p₀) = 20%, the h₀ limit was set at −1.625. This value was chosen because within an average sequence of 21 procedures, the risk of declaring trainees proficient when their performance was inadequate (Type I error) was limited to 9% (α = 0.09), and the risk of not declaring a trainee proficient although his or her performance was adequate (Type II error) was 1% (β = 0.01).

Number of blocks to attain proficiency

Thirty-nine residents provided data on 2076 peribulbar blocks (median = 52; Interquartile Range (IQR) = 27–78; range = 4–132) blocks per resident). The overall failure rate did not differ among first year (1856 blocks, failure rate = 2.9%), second-year (189 blocks; failure rate = 0.5%), or third-year residents (31 blocks, failure rate = 0%), Chi-Square (2 d.f.) = 4.62; p = 0.09.

Except for one case of local anesthetic systemic toxicity, 12 cases of block suspension because of severe pain during injection, eight cases of orbital hematoma, one globe perforation, and one corneal laceration, all other failures that occurred during uncomplicated cases were assumed as technical failures associated with the learning process.

According to the LC-CUSUM curves, 30 residents (77%) attained proficiency after a median of 13 blocks (IQR = 13 −19 [13–24] blocks). The median success rate at proficiency was 100% (IQR = 95%–100% [75–100%]). Of these, 26 (86.6%) were considered true positive cases because the cumulative success rate at reaching the h₀ proficiency limit was ≥90%. Three LC-CURVES signaled proficiency with a cumulative success rate of 89%, and one curve signaled proficiency at a success rate of 75%. These four cases (13%) were considered false positives. Success rates of residents who attained proficiency were maintained (median = 100%; IQR = 98–100% [91–100%]), after their learning curves crossed h₀ from above. No performance deterioration was observed in the post-proficiency SPRT-CUSUM monitoring.

Curves of nine residents remained within the not-yet proficient area of the LC-CUSUM plot at the end of the respective block sequences (median = 8; IQR = 7–10; range = 4–44 blocks per resident). One resident whose curve did not reach h₀ after 44 blocks with a cumulative success rate of 75% was considered a true-negative case (true-negative rate = 11%). The curves of the remaining eight residents (88%) did not reach the proficiency limit after a median of 8 (IQR = 7–9; range = 4–11) blocks despite 100% success rates. In these cases, no conclusion about the performance of the LC-CUSUM method was deemed possible because none of the residents performed the pre-set average number of procedures found necessary to reach the h₀ decision limit.

Discussion

This study aimed to test whether the LC-CUSUM method might be used to detect when anesthesia residents attain proficiency at peribulbar anesthesia. Accordingly, the LC-CUSUM method could detect proficiency when the cumulative failure rate was at or below the pre-established acceptable failure rate of 10% in most cases (86.6% of true positives) with a type I error probability of 13%, a figure close to the admissible pre-set alpha of 10%. The ability of LC-CUSUM to detect proficiency was further confirmed by the maintenance of individual success rates above 90% after proficiency was signaled, according to the SPRT-CUSUM follow-up performance monitoring. Type I and II error rates depend on the chosen parameters for the LC-CUSUM method.⁶ Our results suggest that under the parameters used in this study, the LC-CUSUM method performed satisfactorily in detecting competence, with a high true-positive rate and a false-positive rate close to the pre-established probability of type I statistical error rate (Fig. 1).

Figure 1
Example LC-CUSUM and SPRT-CUSUM curves: (A) The curve of one resident who attained proficiency after 18 procedures. The three failures detected in the post-proficiency SPRT trace (upward shifts) after proficiency did not characterize deteriorating performance because the upper control limit was not reached. Thus, one may infer that the acceptable failure rate was maintained after the LC-CUSUM curve signaled proficiency. (B) Curve for a resident who, at the end of data collection, had not attained proficiency, probably given the insufficient number of procedures. No SPRT-CUSUM curve was constructed for this resident, who had not yet attained proficiency at the end of the data collection period.

The LC-CUSUM test terminates when the proficiency barrier is crossed. Thus, the available number of procedures influences the outcomes of the test. When the number of cases is finite, as occurs during the residency training period, each trainee may be considered “proficient” if the cumulative score crosses the proficiency barrier, or “not yet proficient” if the cumulative score does not cross h₀ after the available number of procedures. More trainees will be correctly diagnosed as proficient if an infinite number of procedures are available. However, some non-proficient trainees might be considered proficient by chance. Thus, as the number of available procedures increases, the actual type I error rate tends to zero, while the actual type II error rate tends to 100%.

For this reason, assessing proficiency within a pre-set average run length is necessary to use LC-CUSUM for monitoring learners’ performance. In this series, residents whose curves remained within the non-definition space did not reach the estimated average run length despite the high individual success rates. However, because these residents did not attain the average number of procedures, they were not labeled false-negative cases.

Secondarily, the median number of blocks needed to attain proficiency could be estimated based on the performance of 77% of the residents who attained proficiency after a median of 13 peribulbar blocks and maintained high success rates.

The peribulbar block is associated with high success rates. However, reported success rates vary according to the success parameter. In studies that used globe akinesia as the success parameter, 95%⁸ and 96.6%⁹ success have been reported. When the sensory block was the success parameter, a 98% overall success rate was reported.¹⁰ Based on the reported success rates, a minimum 90% cumulative success rate was set as the proficiency parameter for this study, considering that all participants were novices in ophthalmic anesthesia.

Data on the learning curve of eye blocks is scarce. In the only study retrieved from a comprehensive literature search, Clarke and colleagues described the sub-Tenon block learning curves of two anesthesiologists experienced in peribulbar anesthesia using the success rates at every 20 blocks in a sequence of 100 blocks performed by each anesthesiologist to assess learning, who stabilized performance after 60 blocks.¹¹ Thus, to the authors’ best knowledge, this may be the first study describing novice anesthesia residents’ peribulbar anesthesia learning curve.

Several limitations must be considered when interpreting or applying the results of this study: (a) The results are based on a cohort of residents from a single institution; (b) Resident readiness to perform was based on faculty’s subjective assessment of resident theoretical knowledge about peribulbar anesthesia and may have been influenced by leniency/severity biases, halo effect, or any other type of bias caused by resident/faculty interactions; (c) Although self-reported outcomes were common practice among the residents participating in the study, who used personal logbooks to follow their procedural skills in most anesthetic techniques, that is, honest self-reporting was part of the institutional culture, self-reporting bias may have eventually contaminated data collection; (d) The peribulbar technique used in this study was standardized, which may have limited the generalizability of this study’s results. (e) Some eye characteristics may impose more difficulty on peribulbar anesthesia, e.g., the space between the eye and the orbit or myopia.¹² Finally, the type of surgery may affect the success rates of peribulbar anesthesia.⁸ Information on these factors was not collected and could not be used to adjust the learning curves to their possible influence on resident performance.³

On the positive side, this study included residents who performed large numbers of blocks using a standardized technique under close supervision starting early in the anesthesia residency. The number of blocks per resident exceeded the number needed to attain proficiency, thus allowing post-proficiency follow-up.

Conclusions

We conclude that the LC-CUSUM is a robust method for detecting resident proficiency at peribulbar anesthesia, defined as success rates exceeding 90%. Accordingly, 13 to 24 supervised double-injection peribulbar blocks are needed to attain competence at peribulbar anesthesia.

• Research Ethics Committee registration number
  42224821.2.0000.5360.
• Research Ethics Committee Id
  5360 – Hospital Governador Celso Ramos/ SMS – Florianopolis - SC
• Funding
  The authors have no sources of funding to declare for this manuscript.

Supplementary materials

Supplementary material associated with this article can be found in the online version at doi:10.1016/j.bjane.2023.11.003.

References

• ¹ de Oliveira Filho GR. The Construction of Learning Curves for Basic Skills in Anesthetic Procedures: An Application for the Cumulative Sum Method. Anesth Analg. 2002;95:411–6.
• ² Kestin IG. A statistical approach to measuring the competence of anesthetic trainees at practical procedures. Br J Anaesth. 1995;75:805–9.
• ³ Biau DJ, Porcher R. A method for monitoring a process from an out of control to an in control state: Application to the learning curve. Stat Med. 2010;29:1900–9.
• ⁴ Starkie T, Drake EJ. Assessment of procedural skills training and performance in anesthesia using cumulative sum analysis (cusum). Can Anesth. 2013;60:1228–39.
• ⁵ Correa JBB, Dellazzana JEF, Sturm A, Leite DMA, de Oliveira Filho GR, Xavier RG. Using the Cusum curve to evaluate the training of orotracheal intubation with the Truview EVO2 laryngoscope. Rev Bras Anestesiol. 2009;59:321–31.
• ⁶ Sims AJ, Keltie K, Burn J, Robson SC. Assessment of competency in clinical measurement: comparison of two forms of sequential test and sensitivity of test error rates to parameter choice. Int J Qual Health Care J Int Soc Qual Health Care. 2013;25:322–30.
• ⁷ Burton A, Altman DG, Royston P, Holder RL. The design of simulation studies in medical statistics. Stat Med. 2006;25:4279–92.
• ⁸ Davis DB, Mandel MR. Efficacy and complication rate of 16,224 consecutive peribulbar blocks. A prospective multicenter study. J Cataract Refract Surg. 1994;20:327–37.
• ⁹ Crawford M, Kerr WJ. The effect of hyaluronidase on peribulbar block. Anaesthesia. 1994;49:907–8.
• ¹⁰ Alves DR, Baltazar I, Alves J, Almeida T, Santos D, Abreu H. Peribulbar Blocks: The Experience of a Specialized Ophthalmologic Surgery Centre. Glob J Med Res. 2017. at <https://medicalresearchjournal.org/index.php/GJMR/article/view/1381>.
  » https://medicalresearchjournal.org/index.php/GJMR/article/view/1381
• ¹¹ Clarke JP, Roberton G, Plummer J. Sub-Tenon Block: A Learning Curve of 100 Cases. Anaesth Intensive Care. 2006;34:450–2.
• ¹² Kallio H, Rosenberg PH. Advances in ophthalmic regional anesthesia. Best Pract Res Clin Anaesthesiol. 2005;19:215–27.

Publication Dates

• Publication in this collection
  22 Apr 2024
• Date of issue
  2024

History

• Received
  20 Oct 2022
• Accepted
  07 Nov 2023
• Published
  13 Nov 2023