Anesthesiology Journal club assessment by means of semantic changes

Abstracts

BACKGROUND AND OBJECTIVES:  the interactive approach of a journal club has been described in the medical education literature. The aim of this investigation is to present an assessment of journal club as a tool to address the question whether residents read more and critically.

METHODS:  this study reports the performance of medical residents in anesthesiology from the Clinics Hospital - University of São Paulo Medical School. All medical residents were invited to answer five questions derived from discussed papers. The answer sheet consisted of an affirmative statement with a Likert type scale (totally disagree-disagree-not sure-agree-totally agree), each related to one of the chosen articles. The results were evaluated by means of item analysis - difficulty index and discrimination power.

RESULTS:  residents filled one hundred and seventy three evaluations in the months of December 2011 (n = 51), July 2012 (n = 66) and December 2012 (n = 56). The first exam presented all items with straight statement, second and third exams presented mixed items. Separating "totally agree" from "agree" increased the difficulty indices, but did not improve the discrimination power.

CONCLUSIONS:  the use of a journal club assessment with straight and inverted statements and by means of five points scale for agreement has been shown to increase its item difficulty and discrimination power. This may reflect involvement either with the reading or the discussion during the journal meeting.

Education; Medical; Graduate; Anesthesiology; Self-assessment; Program evaluation; Educational measurement; Periodicals as topic

JUSTIFICATIVA E OBJETIVOS:  a abordagem interativa de um clube de revista foi descrita na literatura de ensino médico. O objetivo desta investigação é apresentar uma avaliação do clube de revista como uma ferramenta para discutir a questão da leitura crítica entre os residentes.

MÉTODOS:  este estudo relata o desempenho dos médicos residentes de anestesiologia do Hospital das Clínicas - Faculdade de Medicina da Universidade de São Paulo. Todos os médicos residentes foram convidados a responder a cinco perguntas derivadas de artigos discutidos. A folha de resposta consistia em uma declaração afirmativa com uma escala do tipo Likert (discordo totalmente, discordo, não tenho certeza, concordo, concordo totalmente), cada declaração relacionada a um dos artigos escolhidos. Os resultados foram avaliados por meio da análise de itens - índice de dificuldade e poder de discriminação.

RESULTADOS:  os residentes completaram 173 avaliações nos meses de dezembro de 2011 (n = 51), julho de 2012 (n = 66) e dezembro de 2012 (n = 56). O primeiro teste apresentou todos os itens com declaração direta e o segundo e terceiro testes apresentaram itens mistos. Separar "concordo totalmente" de "concordo" aumentou os índices de dificuldade, mas não melhorou o poder de discriminação.

CONCLUSÕES:  o uso de uma avaliação do clube revista com declarações diretas e inversas e com o uso de uma escala de cinco pontos para acordo mostrou aumentar a dificuldade do item e o poder de discriminação. Isso pode refletir o envolvimento com a leitura ou com a discussão durante as reuniões do clube.

Educação; Médico; Graduação; Anestesiologia; Autoavaliação; Avaliação do programa; Avaliação educacional; Periódicos como tema

JUSTIFICACIÓN Y OBJETIVOS:  El abordaje interactivo de un club de revista ha sido descrito en la literatura de enseñanza médica. El objetivo de esta investigación es presentar una evaluación del club de revista como una herramienta para discutir la cuestión de la lectura crítica entre los residentes.

MÉTODOS:  Este estudio relata el desempeño de los médicos residentes de anestesiología del Hospital das Clínicas de la Facultad de Medicina de la Universidad de São Paulo. Todos los médicos residentes fueron invitados a responder a 5 preguntas derivadas de artículos discutidos. La hoja de respuesta consistía en una declaración afirmativa con una escala del tipo Likert ("estoy totalmente en desacuerdo", "en desacuerdo", "no estoy seguro", "estoy de acuerdo", "estoy totalmente de acuerdo"), cada declaración relacionada con uno de los artículos escogidos. Los resultados se evaluaron por medio del análisis de los ítems: índice de dificultad y poder de discriminación.

RESULTADOS:  Los residentes completaron 173 evaluaciones en los meses de diciembre de 2011 (n = 51), julio de 2012 (n = 66) y diciembre de 2012 (n = 56). El primer test incluyó todos los títulos con declaración directa y el segundo y el tercero tuvieron títulos mezclados. Separar "estoy totalmente de acuerdo" de "estoy de acuerdo" aumentó los índices de dificultad pero no mejoró el poder de discriminación.

CONCLUSIONES:  El uso de una evaluación del club de revista con declaraciones directas e inversas y con el uso de una escala de 5 puntos para "estoy de acuerdo", mostró el aumento de la dificultad del ítem y del poder de discriminación. Eso puede reflejar el nivel de implicación en la lectura o en la discusión durante las reuniones del club.

Educación; Médico; Pregrado; Anestesiología; Autoevaluación; Evaluación del programa; Evaluación educacional; Periódicos como tema

Introduction

The interactive approach of a journal club (JC) has been described in the medical education literature. As stated before, the major goal for most clubs is to teach critical appraisal skills. In addition, early characteristics still work for high attendance: mandatory attendance, availability of food, and perceived importance by the program director.¹ In addition, providing credits for attendance has been associated with increased participation.² It has been described as powerful motivator of critical house-staff reading behavior,³ and indeed as a means to develop a curriculum in epidemiology, biostatistics and experimental design.⁴

Journal clubs have been an integral part of health care education, medical residency in general and in general surgery training and it is an educational resource that can help a residency program to meet core competency requirements.^{5 and 6} A survey with the Association of Program Directors in Surgery showed that the typical journal club meets once in a month and discuss three to six articles. It is interesting to notice, though, that few programs reported performing any systematic or formal evaluation.⁷ Notwithstanding, those who do so used objective assessment with pretest and posttest examinations,⁸ measurement of JC satisfaction with a structured checklist for article review,⁹ and the use of critical appraisal tests.

Lately, the use of internet did not surpass the JC model with in person moderation for surgical residents.¹⁰ Considering the successful JC, it seems reasonable to use checklists, make explicit the learning objectives and provide a formal meeting structure and process, not much different from the features pointed out previously by Alguire.^{1 and 11} It is also noticeable that a journal club should focus on current, real patient's problem of most interest to the group,¹² as well as to provide experiences like the use of the 'Critical Appraisal Tool' (CAT) software to help residents streamline their presentations leaving the majority of journal club time for discussion.¹³

Considering that the medical literature continues to expand and physicians must keep up with the amount of information available, the JC provides a place where specialists openly review with residents the literature while also promote an ambience where to teach to evaluate the scientific methods critically.¹⁴

The aim of this investigation is to present an assessment of JC as a tool to address the question whether residents read more and critically, beyond the JC presentation.

Methods

This study reports the performance of medical residents in anesthesiology from the Clinics Hospital - University of São Paulo Medical School (HCFMUSP), São Paulo, Brazil, during three evaluations of Journal Club activities, December, 2011 through December 2012. These assessments are part of their formal evaluation system to achieve anesthesiology professional certification from HCFMUSP Residency Program. The study is part of a continuing project that surveys the quality of life of medical residents in the HCFMUSP and approved by the Ethical Review Board of University of São Paulo Medical School (protocol 9017).

Medical residents of 1st, 2nd and 3rd years enrolled in the Anesthesiology Residency Program receive their formal graduate training in the various institutes of HCFMUSP that includes Heart Institute, Children Hospital, Orthopedic Surgery Institute, Cancer Institute, University Hospital (normal Obstetric care) and the Central Institute that comprises medical specialties surgeries like Plastic Surgery, Vascular Surgery, Thoracic Surgery, Head and Neck Surgery, Urology, Gynecology and Obstetric (pathologic), Neurosurgery, Ear, Nose, and Throat Surgery, and Ophthalmology and Emergency Surgery. This complex of institutes performs 30,000 surgeries yearly. Besides activities in surgical centers, residents stay on duty daily at the postoperative Anesthesia Intensive Care Unit. They receive theoretical information based on two regular weekly activities: lectures and rotational meetings - complications, intensive care and published articles discussion (journal clubs).

The journal club comprises a monthly meeting where residents previously prepare and present an article in the anesthesiology field recently published, chosen by an instructor of anesthesiology (JEV). The way to choose an article relied mostly on two decisions: either a new guideline or an issue related to the practice of anesthesia. The majority of papers were original investigation and one or two at every semester was a guideline. A second resident colleague is responsible for criticizing and highlighting the key points of the presented investigation.

After five formal meetings and at least 10 articles presented, all medical residents were invited to answer 5 questions derived from some of those read and discussed papers. The answer sheet consisted of an affirmative statement with a Likert type scale (totally disagree-disagree-not sure-agree-totally agree), each related to one of chosen articles. An item was assembled as a straight answer (to be agreed upon) or as an inverted one (to be disagreed upon).

The results were evaluated by means of item analysis in order to achieve difficulty index and discrimination power of every item/question. It can tell whether the item was too easy or too hard and how well it discriminated between high and low scorers on the test. Briefly, to the item difficulty, select one third of exams with the higher scores and the same number with the lower scores. Sum the number of these exams and sum the number among these exams that selected the right answer for each item. To divide the sum of right answers by the total of those exams with higher and lower scores will provide the item difficulty. Smaller this percentage, more difficult is the item. The item discrimination is the result of subtracting the number of right answers in the lower score group from the right answers in the higher group. This result is divided by the average of those one third of exams with the higher scores and the same number with the lower scores.¹⁵

Results

Residents filled out one hundred and seventy three evaluations in the months of December 2011 (n = 51), July 2012 (n = 66) and December 2012 (n = 56), when they received their end of semester assessments. The first exam (December/2011) presented all items with straight statements. Considering separately "totally agree" made the discrimination and difficulty indices to improve (only one became less discriminant, item 4) ( Table 1).

Table 1
The first exam presented items/questions with straight statements.

Indices	Agree + totally agree		Totally agree
	Discrimination	Difficulty	Discrimination	Difficulty
Roquilly et al. on hydrocortisone after trauma (JAMA 2011;305:1201-9)	-0.24	0.82	0.12	0.06
Fast - ASA Committee on Standards and Practice Parameter (Anesthesiology 2011;114:495-511)	0.35	0.65	0.71	0.35
Ingrande et al. on body weight and propofol (Anesth Analg 2011;113:57-62)	0.29	0.79	0.82	0.41
Kahokehr et al. on ropivacaine intraperitoneal (Ann Surg 2011;254:28-38)	0.65	0.44	0.24	0.12
Saal et al. on anesthesia satisfaction (Br J Anaesth 2011;107:703-9)	0.24	0.65	0.29	0.15

• ^{Index Difficulty}
  : smaller the percentage, more difficult is the item
• ^{Index Discrimination}
  : higher means better discrimination.

The second exam (July/2012) was presented in three forms: all items with straight statements to be chosen, two first items with straight plus two final with inverted statements, and the third form with the first two items with inverted statements plus two final with straight ones. The presentation of these mixed up items resulted in higher difficulty index, although have reduced the discrimination power, even slightly. To separate "totally agree" from "agree" increased the difficulty indices, but did not improve the discrimination power to all the questions (Table 2). The third exam (December/2012) was presented in two forms: all items with straight or all items with inverted statements. This presentation resulted in higher difficulty index, and better discrimination power. To separate "totally agree" from "agree" increased the difficulty indices, but did not improve the discrimination power to all the questions (Table 3).

Table 2
The second exam used all items straight, two first items straight plus two final inverted, and first two items inverted plus two final straight.

	Agree + Totally agree		Totally agree
Indices	Discrimination	Difficulty	Discrimination	Difficulty
Gaszynski et al. on TOF with sugammadex (Brit J Anaesth 2012; 108 (2): 236)	0.50	0.63	0.88	0.44
Moon et al on Anti-emetic effect of ondansetron (Brit J Anaesth 2012; 108 (3): 417-22)	0.50	0.50	0.50	0.25
Stumpner et al. on Desflurane-induced post-conditioning (Brit J Anaesth 2012; 108 (4): 594-601)	0.75	0.63	0.63	0.31
Nakasuji et al. on Hypotension from spinal anesthesia in patients more than 80 years (J Clin Anesth 2012; 24: 201-6)	0.63	0.69	0.50	0.25
Radke et al. on Spontaneous breathing during general anesthesia (Anesthesiology 2012; 116 (6): 1227-34)	0.38	0.44	0.13	0.06

• ^{Index Difficulty}
  : Smaller this percentage, more difficult is the item
• ^{Index Discrimination}
  : Higher means better discrimination.

Table 3
The third exam presented either all items/questions with straight statements or with inverted statements.

Indices	Agree + totally agree		Totally agree
	Discrimination	Difficulty	Discrimination	Difficulty
Angst et al. on alfentanil and hereditary (Anesthesiology 2012;117:22)	0.56	0.50	0.33	0.17
van Gulik et al. on remifentanil on chronic pain (Br J Anaesth 2012;109:616)	-0.11	0.72	0.67	0.33
Aldenkortt et al. on PEEP and obesity (Br J Anaesth 2012;109:493)	0.22	0.67	0.56	0.28
Jacob et al. on cerebral metabolism (Anesthesiology 2012;117:1062)	0.67	0.67	0.56	0.28
Walker et al. on spinal anesthesia on neonates (Anesth Analg 2012;115)	0.67	0.67	0.33	0.17

• ^{Index Difficulty}
  : smaller the percentage, more difficult is the item
• ^{Index Discrimination}
  : higher means better discrimination

Discussion

The use of a journal club assessment with straight and inverted statements and by means of five points scale for agreement has been shown to increase its item difficulty and discrimination power. This may reflect involvement either with the reading or the discussion during the journal meeting.

Journal clubs (JC) have played an active role in medical education for over a century and, according to Linzer, should be incorporated into the medical educational curriculum.¹⁶ The organization of these meetings followed previously pointed characteristics associated with long and high attendance: mandatory attendance and fewer house staff, although there was no provision of food.¹⁷

Although this learning activity has been applied worldwide and arranged according to previously described characteristics, regular surveys usually assess goals and monitor satisfaction of attendees. As much positive and reassuring as they can be, such evaluations did not address the objectives, among others, of critical appraisal of the presented literature.¹⁸ The present report suggests the use of exams with specific questions retrieving the literature presented and with a five-point agreement scale as a strategy to measure residents' participation and the retrieval of meaningful learning. This approach is distinct from multiple choice tests or even written essays, considering the interest in preserving authorship - citing the authors' findings, and providing response choices that should be agreed-disagreed upon.

Multiple choice examinations are comprised of questions which usually, but not always have correct answers. These are snapshots at one point in time of a small subset of memory, from which evaluators attempt to make predictions about the overall body of knowledge that a student or resident has achieved at the time they take the examination. The expedient in this investigation of citing the main content of a published literature, and eventually reversing its findings, although eliciting memory, mediates attention as the major general variable through which motivational factors influence meaningful learning. ¹⁹

The exams were also arranged with all items presenting straight or reversing statements, without the use of negatively worded items. This approach works as the stem remains as directly worded item and the response options remain the same but are in a different order. This strategy could guard against medical residents acquiescence and response set behaviors, which means agreeing/disagreeing without reasoning.²⁰ The results computerized "totally agree" separately from a previous degree in a Likert scale and showed improvements of difficulty and discrimination indices. Taken together, the use of mixed items and a Likert scale for agreement, the strategies of reading, participating in the discussion, or both can be differentiated from only watching the JC presentation. Those more active in JC may be prompted to choose the highest Likert degree in contrast to those who decided for a lower engagement.

In conclusion, this investigation proposes to assess the retrieval of JC sessions citing the original article, directly worded or in a different order, by means of a Likert scale of agreement. This approach increased item difficulty and discrimination indices.

References

• ¹ Alguire PC. A review of journal clubs in postgraduate medical education. J Gen Intern Med. 1998;13:347-53.
• ² Hinkson CR, Kaur N, Sipes MW, et al. Impact of offering continu- ing respiratory care education credit hours on staff participation in a respiratory care journal club. Respir Care. 2011;56:303-5.
• ³ Linzer M, Brown JT, Frazier LM, et al. Impact of a medi- cal journal club on house-staff reading habits, knowledge, and critical appraisal skills. A randomized control trial. JAMA. 1988;260:2537-41.
• ⁴ Letterie GS, Morgenstern LS. The journal club teaching critical evaluation of clinical literature in an evidence-based environ- ment. J Reprod Med. 2000;45:299-304.
• ⁵ Harris J, Kearley K, Heneghan C, et al. Are journal clubs effec- tive in supporting evidence-based decision making? A systematic review BEME Guide No. 16. Med Teach. 2011;33:9-23.
• ⁶ Shifflette V, Mitchell C, Mangram A, et al. Current approaches to journal club by general surgery programs within the South- western surgical congress. J Surg Educ. 2012;69:162-6.
• ⁷ Crank-Patton A, Fisher JB, Toedter LJ. The role of the journal club in surgical residency programs: a survey of APDS program directors. Curr Surg. 2001;58:101-4.
• ⁸ Kelly AM, Cronin P. Setting up, maintaining and evaluating an evidence based radiology journal club: the University of Michi- gan experience. Acad Radiol. 2010;17:1073-8.
• ⁹ Burstein JL, Hollander JE, Barlas D. Enhancing the value of jour- nal club: use of a structured review instrument. Am J Emerg Med. 1996;14:561-3.
• ¹⁰ McLeod RS, MacRae HM, McKenzie ME, et al. Evidence Based Reviews in Surgery Steering Committee. A moderated journal club is more effective than an Internet journal club in teach- ing critical appraisal skills: results of a multicenter randomized controlled trial. J Am Coll Surg. 2010;211:769-76.
• ¹¹ Lee AG, Boldt HC, Golnik KC, et al. Using the Journal Club to teach and assess competence in practice-based learning and improvement: a literature review and recommendation for implementation. Surv Ophthalmol. 2005;50:542-8.
• ¹² Misra UK, Kalita J, Nair PP. Traditional journal club: a continuing problem. J Assoc Physicians India. 2007;55:343-6.
• ¹³ Hartzell JD, Veerappan GR, Posley K, et al. Resident run journal club: a model based on the adult learning theory. Med Teach. 2009;31:e156-61.
• ¹⁴ Valentini RP, Daniels SR. The journal club. Postgrad Med J. 1997;73:81-5.
• ¹⁵ Gronlund NE. Constructing achievement tests prentice-hall. New Jersey: Englewood Cliffs; 1968.
• ¹⁶ Linzer M. The journal club medical education: over one hundred years of unrecorded history. Postgrad Med J. 1987;63:475-8.
• ¹⁷ Sidorov J. How are internal medicine residency journal clubs organized and what makes them successful? Arch Intern Med. 1995;155:1193-7.
• ¹⁸ Dirschl DR, Tornetta 3rd P, Bhandari M. Designing, conducting, and evaluating journal clubs in orthopaedic surgery. Clin Orthop Relat Res. 2003;413:146-57.
• ¹⁹ Ausubel DP. The acquisition and retention of knowledge: a cognitive view. Dordrecht, The Netherlands: Kluwer Academic Publishers; 2000.
• ²⁰ Barnette JJ. Effects of stem and Likert response option reversals on survey internal consistency: if you feel the need there is a better alternative to using those negatively worded stems. Educ Psychol Meas. 2000;60:3361-70.

Publication Dates

• Publication in this collection
  Jul-Aug 2014

History

• Received
  09 Sept 2013
• Accepted
  20 Nov 2014