How are we assessing competencies? Experience report of an intervention project in the evaluation instruments of an intensive care medicine residency program

ABSTRACT

Introduction:

Competency-based teaching in medical residencies has evidenced the mismatch between traditional assessment processes and the educational objectives of pedagogical projects aligned with the competency matrices of each specialty. The competency matrix for the Intensive Care Medicine Residency Program (3-year direct admission) was approved in 2021. The objective of this article is to describe an experience report of an intervention project in the performance assessment instruments of residents attending the Intensive Care Medicine Residency Program at a university hospital in São Luís-Maranhão.

Experience Report:

After organizing the study and working group for the intervention, the object “skills assessment tools” was chosen and the Intensive Care Medicine Residency Program was selected. Initially, a questionnaire was applied to all preceptors and residents working in the intensive care unit (ICU) setting with the aim of evaluating their perceptions regarding the current assessment tool, following the guiding question: does the current assessment meet the conception of the program translated by the competency matrix of the National Medical Residency Commission?

Discussion:

Although the majority of the preceptors and residents considered that the evaluation methods met the Program design, there were weaknesses in relation to feedback and evaluation of the residents’ performance. As an intervention, we proposed adapting the existing tool, making it adequate to the performance predicted in the specialty competency matrix with formalization of feedback and introduction of performance assessment in a real scenario using the Mini Clinical Assessment Exercise (Mini-CEX).

Conclusion:

The boundaries between assessment and learning are blurred. Based on indicators on the perception of preceptors and residents of weaknesses in the assessment used for a long time, an intervention was proposed to modify the assessment instruments with the aim of adapting/improving the assessment of skills.

Keywords:
Assessment Methods; Medical Residency; Medical Education; Clinical Competence; Intensive Care Unit

RESUMO

Introdução:

O ensino baseado em competências no âmbito das residências médicas tornou evidente o descompasso dos processos de avaliação tradicionais com os objetivos educacionais dos projetos pedagógicos alinhados às matrizes de competências de cada especialidade. A matriz de competência para o Programa de Residência em Medicina Intensiva (acesso direto em três anos) foi aprovada em 2021. O objetivo deste artigo é descrever o relato de experiência de um projeto de intervenção nos instrumentos de avaliação de desempenho dos residentes no Programa em Residência em Medicina Intensiva de um hospital público universitário em São Luís, no Maranhão.

Relato de experiência:

Após a organização do grupo de estudo e de trabalho para a intervenção, houve a escolha do objeto “ferramentas de avaliação de competências” e a seleção do Programa de Residência de Medicina Intensiva. Inicialmente, foi aplicado um questionário a todos os médicos preceptores e residentes, com atuação no cenário da unidade de terapia intensiva (UTI), com o objetivo de aferir as percepções deles acerca do instrumento avaliativo vigente, seguindo a pergunta norteadora: “A avaliação atual atende à concepção do programa traduzido pela matriz de competência da Comissão Nacional de Residência Médica?”.

Discussão:

Embora a maioria dos preceptores e residentes tenha considerado que os métodos de avaliação atendiam à concepção do programa, havia pontos frágeis em relação ao feedback e à avaliação de desempenho dos residentes. Como intervenção, propusemos adaptação da ferramenta existente, adequando-a aos desempenhos previstos na matriz de competências da especialidade com formalização do feedback e introdução de avaliação de desempenho em cenário real utilizando o Miniexercício Clínico Avaliativo (Mini-Cex).

Conclusão:

Os limites entre a avaliação e a aprendizagem são tênues. Com base em indicadores sobre a percepção de preceptores e residentes de fragilidades na avaliação utilizada de longa data, foi proposta uma intervenção de modificação dos instrumentos avaliativos com o intuito de adequar/melhorar a avaliação de competências.

Palavras-chave:
Métodos de Avaliação; Residência Médica; Educação Médica; Competência Clínica; Unidade de Terapia Intensiva

INTRODUCTION

In medical education, assessments traditionally sought to assess/measure knowledge. The worldwide change from the Flexnerian paradigm to competency-oriented training in recent decades has meant that professional education must meet health challenges at a time of rapid demographic and epidemiological transition, since a fragmented and static curriculum would generate poor professionals, ill-prepared to deal with the current needs of patients and the population^1)-(2. This change was expressed in Brazil in the Curricular Guidelines for Medical Graduation of 2001, ratified in 2014^3)-(4, and by the National Medical Residency Commission (CNRM, Comissão Nacional de Residência Médica), which has approved the competency matrices for Residency Programs since 2018⁵. For Intensive Medicine, there was a last modification in 2021, with the approval of three-year direct admission⁶.

Competence is defined as the inseparable interaction of knowledge, skills and attitudes; understanding the use of knowledge, communication, clinical reasoning, values ​​and reflections in daily practice for the benefit of the individual and the community. Thus, competency-oriented learning should enable the mobilization of these attributes, generating initiative and responsibility in the presence of a professional situation^7)-(8. In this context, there is a gap between traditional evaluation processes and the objectives of the current Pedagogical Projects of Residency programs built in line with the respective competency matrix. From this perspective, the evaluation methods have been studied ^(9)-(10, highlighting tests that measure not only knowledge but also understanding, analysis and application in real-life situations¹¹, using Miller’s Pyramid as the theoretical reference basis^12)-(13.

The aim of this study is to describe, as an experience report, an intervention project in instruments for evaluating the performance of residents in an Intensive Medicine Residency Program at a university hospital in São Luís, Maranhão.

EXPERIENCE REPORT

Setting

This is a reference hospital complex for high complexity cases, with a total of 550 beds distributed in two admission units and its purpose is to “educate to transform care”, being a teaching hospital certified by the Ministry of Education and the Ministry of Health, which has 28 Medical Residency programs and 12 multi- and uniprofessional residency programs.

Creation of a Study and Work Group

Initially, with the intention of advancing and improving residency programs and with the perspective of adding new skills/strategies/management tools to optimize the performance of health residency programs within the scope of the University Hospital (HU), the institution appointed 1 teaching manager representing the Medical Residency Commission (COREME, Comissão de Residência Médica), 1 teaching manager representing the Multiprofessional Residency and Health Commission (COREMU, Comissão de Residência Multiprofissional e Saúde), 2 supervisors of Medical Residency Programs and 1 tutor of the Multiprofessional Residency to carry out a postgraduate program in Management of SUS Residency Programs, through the Support Program for Institutional Development of the Unified Health System (PROADI_SUS, Programa de Apoio ao Desenvolvimento Institucional do Sistema Único de Saúde). This group, advised by a learning facilitator from the aforementioned postgraduate course, constituted a study and work group in March 2022 aiming to build an intervention project in one of the HU Residency Programs.

Problem Selection, Targeted Cause, Preparation for the Intervention, Choice of Program

In a first moment of scenario assessment, we started by reviewing the literature on the relevant legislation^14)-(15 and reading about pedagogical principles and models used in teaching-learning in health^16)-(17. The starting point was the context in which the HU was inserted, both from a structural and teaching point of view.

After evaluating the cartography of the scenarios and listing the main challenges to be faced in residency programs, an attempt was made to identify, on a prioritization scale, problems of high relevance and in line with high governability for a future intervention project. Based on this principle, the chosen problem was narrowed down to the targeted cause defined by the group: “evaluation instruments are merely bureaucratic, non-formative institutional documents that do not allow change in the trajectory of the educational process”. These instruments within the scope of the HU were constructed prior to the current discussions on competency-based teaching and are applied to all residents of the Hospital. These comprise global assessment sheets used monthly (sometimes retroactively) containing the following 10 items: attendance, punctuality, technical-scientific knowledge, discipline, interest, solidarity, ethics, initiative, fulfillment of duties and personal presentation. Each of these 10 items is given a score ranging from 0 to 10, with in the final sum ranging from 0 to 100, without formal provision for feedback.

Empirically, the working group identified that the resident assessment instruments were in disagreement with the assessment of the skills to be acquired during this training period and did not reflect the assessment of the resident’s performance in practice. When developing the project, the Residency Program in Intensive Medicine was chosen as the intervention setting. This is a Program that offers five direct access places per year, lasting three years. Supervision is carried out by an intensive care physician, supported by the ICU coordinator, together with 22 other doctors from the service who act as preceptors. This is a level III care ICU, which assists adult patients, and constitutes the setting for the training and practices not only for the Residency Program in Intensive Medicine, but also for Internal Medicine, General Surgery, Gynecology and Obstetrics, Anesthesiology and for the Multiprofessional Residency Programs in Nursing, Physical therapy, Psychology and Speech Therapy.

The project “construction and implementation of competency-based assessment instruments” was publicly presented in October 2022, being approved by the institution’s managers. Concomitantly, once a month, some assessment instruments were presented in seminars and discussed in group meetings in clinical practice environments, which were validated in the literature and different from those already practiced at HU: feedback, portfolio, mini-evaluative clinical exercise (Mini-CEX), Direct observation of procedures (DOPs), Objective Structured Clinical Examination (OSCE), 360º assessment, Entrustable professional activities (EPAs)^18)-(23.

The research project for publication of this movement as a case report was approved by the ethics committee under number CAAE 70302723.2.0000.5086.

Baseline Indicators

To construct the baseline, a questionnaire was created adapted from the INEP²⁴ undergraduate course evaluation instrument and an instrument applied to questions about feedback from a Chinese study²⁵. The first guiding question aimed to start the process was the following: does the current assessment meet the Program’s design translated by the CNRM competency matrix?

The questionnaire was applied to all medical preceptors and medical residents on rotation in the ICU in November 2022 with the aim of evaluating their perceptions regarding the current assessment instrument.

This questionnaire consisted of three distinct parts: demographic characteristics; perception of the evaluation procedure based on the competency matrix (whether or not it meets the program design defined by the matrix, whether it allows the development of resident autonomy and whether it results from information made available to the residents); and finally, questions about the perception of the instrument purpose, about the feedback and its suitability for evaluating the residents’ performance on a Likert scale. To make the measurement coherent, the current assessment instrument for residents and the Intensive Medicine Program competency matrix were attached to the questionnaire.

To analyze the results, the answers about the adequacy of the current instrument were separated for the group of preceptors and group of residents. The categorical variables were expressed as an absolute number and percentage and the continuous variables, as the mean and their respective standard deviation, if normally distributed; or by the median and its respective interquartile range, 25%-75%, if it showed a non-Gaussian distribution. To evaluate the statistical difference between the group of preceptors and the group of residents in relation to general characteristics, the Fisher’s Exact or Mann-Whitney tests were used, with values of p < 0.05 being considered statistically significant. As for the Likert scale, the terms “agree and strongly agree” were replaced by “yes”, “disagree or strongly disagree” by “no” and “neutral” remained as “neutral”.

DISCUSSION

Baseline Indicator Results

A total of 27 doctors answered the evaluation questionnaire: 19 Program preceptors and 8 residents (4 Intensive Care Medicine residents at the end of the first year and 4 residents from other specialties on rotation in the ICU). There were no refusals to fill out the questionnaire. An exploratory analysis of the data was carried out to obtain the characteristics of the studied population.

Of these 27 doctors who answered the questionnaire, 15 (55.6%) were female, with no difference between the preceptor and resident groups (p=0.48). The group of preceptors was significantly older (p=0.01), with a mean age ranging from 33.6±6.0 years, when compared to the residents, who had a mean age of 27.4±3.2 years. As expected, the time since graduation for preceptors was also significantly longer (p<0.001), with a median and interquartile range [25%-75%] of 7 [6-9] years and for the residents, 2 [1.25-3] years. The 19 preceptors had carried out their in-service education activities for a median of 2 [1-4] years, exactly the same time as they had been involved in care work within the HU and 11 of them (57.9%) had completed their second residency in some clinical specialty, with 6 of them (31.6%) having completed their residency in Clinical Medicine and 2 (10.5%) having completed their Master’s degree. All 8 residents were attending their first residency when they answered the questionnaire.

Regarding the perception of the current instrument used to evaluate residents, among the 27 doctors, 3 preceptors stated that the evaluation instrument being used did not meet the course design defined by the Competency Matrix (11.1%), which was not corroborated by any of the residents. Therefore, 24 doctors (88.9%) pointed out that there is some correlation between what is required by the Intensive Medicine Residency Competency Matrix and what is stated/answered in the current assessment instrument.

Of the 27 doctors, 7 (3 preceptors and 2 residents, 18.5%) rated the current instrument with the maximum score on the scale, noting that it “meets the concepts defined in the competency matrix and allows the development of autonomy and results in systematized information that allow concrete actions to improve learning and are adopted”.

Also among the 27 doctors involved in the process, 10 (3 residents and 7 preceptors, 37.0%) answered that the current instrument “meets the concept defined in the Matrix, but does not allow the development of resident autonomy or does not generate systematized information aimed at improvement”, although 9 doctors (3 residents and 6 preceptors, 33.3%) perceived that the instrument “meets the concepts and allows the development of autonomy as it results in information aimed at improving the process”.

When asked about the purpose of the assessment instrument, the majority of physicians (22, 81.5%) answered that they precisely understood the purpose of the current instrument. All 5 doctors who showed disagreement or were neutral regarding this item were preceptors.

The feedback after applying the current assessment instrument was not perceived by 13 doctors (3 residents and 10 preceptors, 48.1%) and 2 preceptors (7.4%) were neutral in relation to this question. Regarding feedback on daily activities, this was perceived by 16 doctors (2 residents and 14 preceptors, 59.3%), 7 doctors (1 resident and 6 preceptors, 25.9%) were neutral in relation to this item, and 4 (14.8%) did not perceive it.

Finally, when asked whether the score at the end of the current assessment instrument reflects the quality of the residents’ performance, 10 physicians answered “no” (1 resident and 9 preceptors, 37.0%), 10 physicians answered “yes” (4 residents and 6 preceptors, 37.0%) and 7 doctors were neutral in relation to this statement (25.9%).

The graph panel describes the obtained results regarding the perception of the adequacy of the current assessment instrument (Figure 1).

Figure 1
Graph panel: perception of preceptors and residents regarding the adequacy of the current assessment instrument.

Discussion of Indicators and Intervention Proposal

The Intervention Project carried out was based on the premise that residents from all HU residency programs were evaluated using institutional instruments constructed prior to the current discussions of competency-based teaching, which did not reflect the evaluation of the resident’s performance in practice.

The results of the baseline indicators surprisingly showed that the current assessment instrument is considered by preceptors and residents to be better/more adequate than we thought.

Both preceptors and residents are young physicians, most aged between the third and fourth decades of life and with less than 8 years of medical training. The preceptors carry out their teaching activities at the HU at the same time as the assistance, demonstrating that these activities are in fact entwined. It can be inferred, therefore, that they completed the medical course in light of the 2014 National Curricular Guidelines⁴.

Preceptors and residents considered the current assessment instrument to be relevant, since around 9 of 10 of these doctors answered that it met the concepts required by the Competency Matrix. It is noteworthy that 1 in 4 preceptors does not understand precisely what its purpose is, which can be considered a matter of concern, since the educator’s intentionality is a pillar of the assessment aimed to maintain the line of curricular coherence; where the “how” (how to do it) cannot be more important than the “for what” (why to do it)²⁴.

This divergence in the obtained responses between considering it pertinent and not precisely understanding its purpose can be explained by the lack of theoretical basis on preceptorship itself and its respective evaluation processes²⁷. Around 40% of the doctors understand that, although relevant, the assessment instrument does not allow the resident to develop autonomy or does not generate systematized information aimed at improvement. Autonomy is an important step in building competence, translating knowledge, skills and attitudes and in the ability to perform a function or perform an action at the desired level of quality²⁸. Once again, the answers differ between what is described as the theoretical basis for assessing competence¹² and what is practiced.

Medical Residency is a conducive environment for seeking theoretical knowledge based on practical experiences. Getting the resident involved and guiding them to develop autonomy and a sense of belonging to keep them motivated is both a stimulus and a challenge, and must be based on openness, trust, dialogue, while valuing knowledge and previous experiences, feedback, in the horizontality between those who teach and those who learn; responsibilities/commitments of the institution that offers the program and of the preceptors who share this journey²⁹. In particular, in the medical environment, caring for acute illnesses is a rich source for learning. The patients have undifferentiated problems, and adequate and timely management requires competence in clinical assessment, reasoning, investigation and procedures³⁰ and assessment in these practice scenarios has been effective and well accepted by preceptors and residents^30),(31. The new educational paradigm seeks to be more effective, more integrated into the health system and focused on the applicability of knowledge into practice. The new model, based on the gradual acquisition of previously defined, observable and measurable competences, is currently a national⁶ and international reference for residency in Intensive Medicine³².

It is also noteworthy that in the baseline of this intervention project, half of the doctors claimed they did not understand the feedback on the assessment. As for the routine activities, 6 of 10, both preceptors and residents, noticed this feedback. In formative health assessment, where the boundaries between assessment and learning are blurred, the feedback is a crucial point and the impact of assessment as a driving force for learning is increasingly gaining acceptance as one of the principles of good assessment³³. In this sense, daily learning translated into training, practice and feedback must be present in a linear and coherent way throughout the continuum of the teaching-learning process. It is also worth highlighting that in the current instrument, a score is assigned to each evaluated item, generating a final grade with the sum of these items. Assessment associated with grades is considered one of the poorest types of feedback³⁴.

Regarding the residents’ performance assessment, there were both positive and negative responses regarding the adequacy of the current instrument. Since the acquisition of competence requires consolidated knowledge in action and the training of residents is more than training³⁵, the assessment tools must be adapted to evaluate this result, that is, the know-how/performance - one of the upper strata of Miller’s pyramid¹².

When researching performance assessment for residents in intensive medicine, studies carried out in Spain on implementing a competency-based curriculum was based on the training of tutors in feedback, in addition to structured objective assessments based on direct observation with the involvement of tutors, residents and specialists in educational psychology, with high satisfaction related to the proposed methodology, as it allows a fairer evaluation, reflection on areas where further progress is needed, strengths and weaknesses^36),(37.

As an intervention proposal, from the perspective that no single method is capable of covering all the evaluation criteria ^12),(33),(34 and aiming to establish something feasible within the reality and the identified baseline, we proposed, in addition to the quarterly assessment of knowledge:

1. Adapt the current instrument, excluding some items and introducing others, in accordance with the competency matrix: fulfillment of duties, scientific knowledge, interest, attendance and punctuality, ethics and discipline, leadership, teamwork. For this modified instrument, feedback is now provided for each of these items on a monthly basis. Since the current instrument was better evaluated than what was expected, the best strategy was not to exclude it, but to adapt it by proposing feedback, since the latter was considered unsatisfactory in the baseline and is currently considered the core of the formative assessment in health^34),(35.

2. Moreover, propose the assessment in a real scenario (Mini-CEX) at the time of admission, evolution and discharge of the patient from the ICU. We chose this tool to assess competency since, although its educational impact is still being studied^11),(38),(39, it has the following advantages: real-life scenarios, being carried out in a manageable time (around 20-30 minutes), being low cost, evaluating six core competencies (clinical history, physical examination, professionalism, clinical reasoning, communication, organization) and having little interference in day-to-day activities⁴⁰, even where there are intense activities such as in the emergency and Intensive Care Unit ^41)-(42.

During the period of training and testing of the above instruments, we received informal feedback, both from preceptors and residents, that there was an initial improvement perception regarding the residents’ performance assessment. A subsequent perception analysis is planned after this intervention project is widely implemented.

CONCLUSION

Evaluation is an indispensable and inseparable step in the training process. Based on indicators on the perception of preceptors and residents, with the identification of weaknesses in the long-used assessment method, an intervention was proposed to modify the assessment instruments with the aim of adapting/improving the assessment of competencies. The long-term results of the perception of this change in evaluation processes will be subsequently verified after full implementation.

ACKNOWLEDGEMENTS

To Dr. Isis Aparecida Cunácia Massaro, from Instituto de Ensino e Pesquisa/Hospital Sírio Libanês for the encouragement and assistance during the Specialization and for the suggestions for this manuscript.

REFERÊNCIAS

• ¹
  Frenk J, et al. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010 Dec 4;376(9756):1923-58. doi: https://doi.org/10.1016/S0140-6736(10)61854-5.
• ²
  Carracio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competences. Acad Med. 2002 May;77(5):361-7.
• ³
  Brasil. Resolução CNE/CES nº 4, de 7 de novembro de 2001. Institui Diretrizes Curriculares Nacionais do curso de graduação em Medicina. Diário Oficial da União; 9 nov 2001 [acesso em 3 jun 2019]. Disponível em: Disponível em: http://portal.mec.gov.br/cne/arquivos/pdf/CES04.pdf
  » http://portal.mec.gov.br/cne/arquivos/pdf/CES04.pdf
• ⁴
  Brasil. Resolução nº 3 de 20 de junho de 2014. Institui Diretrizes Curriculares Nacionais do curso de graduação em Medicina e dá outras providências. Diário Oficial da União ; 23 jun 2014. Seção 1, p. 8-11 [acesso em]. Disponível em: http://portal.mec.gov.br/index.php?option=com_docman&view=download&alias=15874-rces003-14&category_slug=junho-2014-pdf&Itemid=30192
  » http://portal.mec.gov.br/index.php?option=com_docman&view=download&alias=15874-rces003-14&category_slug=junho-2014-pdf&Itemid=30192
• ⁵
  Brasil. Matrizes de Competências aprovadas pela CNRM. Brasília: Ministério da Educação; 2018 [acesso em 7 jan 2019]. Disponível em: Disponível em: http://portal.mec.gov.br/component/content/article?id=71531
  » http://portal.mec.gov.br/component/content/article?id=71531
• ⁶
  Brasil. Matrizes de Competências Aprovadas pela CNRM. Medicina Intensiva Acesso Direto 03 anos [acesso em]. Disponível em: https://revisamed.com.br/residencia-medica/wp-content/uploads/2021/06/Matriz_de_Competencias_da_Medicina_Intensiva_ID_FINAL.pdf
  » https://revisamed.com.br/residencia-medica/wp-content/uploads/2021/06/Matriz_de_Competencias_da_Medicina_Intensiva_ID_FINAL.pdf
• ⁷
  Epstein RM, Hundert EM. Defining ans assessing professional competence. JAMA. 2002; 287(2):226-32.
• ⁸
  Santos WS. Organização curricular baseada em competência na educação médica. Rev Bras Educ Med. 2001;35(1):86-92
• ⁹
  Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, et al. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 2011;33:206-14.
• ¹⁰
  Norcini J, et al. 2018 Consensus framework for good assessment. Med Teach . 2018. doi: https://doi.org/10.1080/0142159X.2018.1500016
  » https://doi.org/https://doi.org/10.1080/0142159X.2018.1500016
• ¹¹
  Anderson HL, Kurtz J, West D. Implementation and use os workplace-based assessment in clinical learning enviroments: a scope review. Acad Med . 2021;96:S164-74.
• ¹²
  Miller GE. The assessmet of clinical skills/competence/performance. Acad Med . 1990; 65(9):S63-7.
• ¹³
  Cruess RL, Cruess SR, Steinert Y. Amending Miller’s pyramid to include professional identy formation. Acad Med . 2016;91(2):180-5.
• ¹⁴
  Brasil. Decreto 80.281, de 5 de setembro de 1977. Regulamenta a Residência Médica, cria a Comissão Nacional de Residência Médica e dá outras providências. Brasília: Presidência da República, 1977 [acesso em]. Disponível em: https://www2.camara.leg.br/legin/fed/decret/1970-1979/decreto-80281-5-setembro-1977-429283-normaatualizada-pe.pdf
  » https://www2.camara.leg.br/legin/fed/decret/1970-1979/decreto-80281-5-setembro-1977-429283-normaatualizada-pe.pdf
• ¹⁵
  Brasil. Lei nº 6.932, de 7 de julho de 1981. Dispõe sobre as atividades do médico residente e dá outras providências. Brasília: Presidência da República , 1981 [acesso em]. Disponível em: https://www2.camara.leg.br/legin/fed/lei/1980-1987/lei-6932-7-julho-1981-357276-publicacaooriginal-1-pl.html
  » https://www2.camara.leg.br/legin/fed/lei/1980-1987/lei-6932-7-julho-1981-357276-publicacaooriginal-1-pl.html
• ¹⁶
  Simon E, Jezine E, Vasconcelos EM, Ribeiro KS. Active teaching-learning methodologies and popular education: agreements and disagreements in the context of health professionals’ education. Interface. 2018;18(supl 2):1355-64. doi: https://doi.org/10.1590/1807-57622013.0477.
  » https://doi.org/https://doi.org/10.1590/1807-57622013.0477
• ¹⁷
  Taylor DCM, Hamdy H. Adult learning theories: Implications for learning and teching in medical education. AMEE Guide n. 83. Med Teach . 35(11):e1561-72. doi: https://doi.org/10.3109/0142159X.2013.828153.
  » https://doi.org/https://doi.org/10.3109/0142159X.2013.828153
• ¹⁸
  Colbert CY, Ownvy AR, Butler PM. A review of portfolio use in residency programs and considerations before Implementation. Teach Learn Med. 2008:20(4):340-5. doi: https://doi.org/10.1080/10401330802384912.
  » https://doi.org/https://doi.org/10.1080/10401330802384912
• ¹⁹
  Burgess A, Diggele C, Roberts C, Mellis C. Feddback in the clinical setting. BMC Med Educ. 2020;20(Suppl 2):460. doi: https://doi.org/10.1186/s12909-020-02280-5.
  » https://doi.org/https://doi.org/10.1186/s12909-020-02280-5
• ²⁰
  Domingues RCL, Amaral E, Bicudo-Zeferino AM. Global rating: a method for assessing clinical competence. Rev Bras Educ Med . 2009;33(1):148-51.
• ²¹
  Bastos CAH, Botelho NM, Portella MB, Nascimento CL do. Application of the 360º evaluation method in medical residents of gynecology and obstetrics. Electronic Journal Collection Health. 2019;(34):e1423. doi: https://doi.org/10.25248/reas.e1423.2019.
  » https://doi.org/https://doi.org/10.25248/reas.e1423.2019
• ²²
  Franco CAGS, et al. OSCE for communication skills and professionalism: case report and meta-analysis. Rev Bras Educ Med . 2015;39(3):433-41.
• ²³
  Cate O. An updated primer on Entrustable Professional Activities (EPAs). Rev Bras Educ Med . 2019:43(1 supl. 1):712-20.
• ²⁴
  Brasil. Instrumento de avaliação de cursos de graduação: presencial e a distância. Brasília; Ministério da Educação, Inep; 2017 [acesso em 18 set 2022]. Disponível em: efaidnbmnnnibpcajpcglclefindmkaj/Disponível em: efaidnbmnnnibpcajpcglclefindmkaj/https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_reconhecimento.pdf
  » https://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2017/curso_reconhecimento.pdf
• ²⁵
  Chen Y et al. Is the rating result reliable? A new approach to respond to a medical trainee’s concerns about the reliability of Mini-CEx assessment. J Formos Med Assoc. 2022 May;121(5):943-9. doi: https://doi.org/10.1016/j.jfma.2021.07.005.
  » https://doi.org/https://doi.org/10.1016/j.jfma.2021.07.005
• ²⁶
  Simon E, Jezine E, Vasconcelos EM, Ribeiro KS. Active teaching-learning methodologies and popular education: agreements and disagreements in the context of health professionals’ education. Interface (Botucatu). 2018;18(supl2):1355-64. doi: https://doi.org/10.1590/1807-57622013.0477.
  » https://doi.org/https://doi.org/10.1590/1807-57622013.0477
• ²⁷
  Botti SHO, Rego S. Preceptor, supervisor, tutor and mentor: what are their roles? Rev Bras Educ Med . 2008;32(3):363-73.
• ²⁸
  Panúncio-Pinto MP, Troncon LEA. Student assessment-general aspects. Medicina (Ribeirão Preto). 2014;47(3):314-23.
• ²⁹
  Botti SHO, Rego S. Clinical teacher: the complex role of the preceptor in medical residency. Physis. 2011;21(1): 65-85.
• ³⁰
  Johnson G, Wade W, Barrett J, Jones M. Acute care assessment tool: a new assessment in acute medicine. Clin Teach. 2009;6:105-9.
• ³¹
  She J, Hong Q, Zheng Y, Yu Q, Ye L, Song Y. Exploration on the MiniCEX-DOPS evaluation in standardized residency training of respiratory and critical care medicine. Chin J Med Res. 2021;12:826-9.
• ³²
  European Society of Inyensive Care Medicine. Cobatrice: Competecny-based training in Intensive Care Medicine in Europe. Disponível em: Cobatrice/org/em/index.asp
  » Cobatrice/org/em/index.asp
• ³³
  Van der Vleuten CPM, Schuwirth LWT. Assessment of professional competence: from methods to programmes. Med Educ. 2005;39:309-17.
• ³⁴
  Van der Vleuten COM, et al. A model for programmatic assessment fit for purpose. Med Teach . 2012;34(3):205-14.
• ³⁵
  Bok H, et al. Programmatic assessment of competency-based workplace learning: when theory meets practice. BMC Med Educ . 2013:13:123.
• ³⁶
  Castellanos-Ortega A, et al. Acceptance and validity of the methods used to implement a competency based medical education programme in an Intensive Care Department of a teaching referral center. Med Intensiva (Engl Ed). 2021 Oct;45(7):411-20.
• ³⁷
  Castellanos-Ortega A, et al. Competency assessment of residents of Intensive Care Medicine through a simulation-based objective structured clinical evaluation (OSCE). A multicenter observational study. Med Intensiva (Engl Ed) . 2022 Sept;46(9):491-500.
• ³⁸
  Lorwald AC, et al. The educational impacto f Mini-Clinical Evaluation Exercice (Mini-Cex) and Direct Observation Exercice of Procedural skills (DOPS) and its association with implementation: a systematic review and meta-analysis. PLoS One. 2018;13(6):e0198009.
• ³⁹
  Martinsen et al. Examinig the educacional impacto f the Mini-Cex: a randomised contolled study. BMC Med Educ . 2021;21:228.
• ⁴⁰
  Megale L, Gontijo ED, Motta JAC. Evaluation of medical students’ clinical skills using the Mini-Clinical Evaluation Exercise (Mini-CEX). Rev Bras Educ Med . 2009;33(2):166-75. doi: https://doi.org/10.1590/S0100-55022009000200002.
  » https://doi.org/https://doi.org/10.1590/S0100-55022009000200002
• ⁴¹
  Bashir K, et al. Acceptability and feasibility of Mini Clinical Evaluation Exercice (Mini-Cex) in the busy emergency department. Open Acess Emergency Medicine. 2021;13:481-6.
• ⁴²
  She J, Hong Q, Zeng Y, Yu Q, Ye L, Song Y. Exploration on the MiniCEX-DOPS evaluation in standardized residency training of respiratory and critical care medicine. Chin J Med Edu Res. 2021;20(07):826-9. doi: https://doi.org/10.3760/cma.j.cn116021-20200325-00556.
  » https://doi.org/https://doi.org/10.3760/cma.j.cn116021-20200325-00556

• 6
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Jorge Guedes.

Publication Dates

• Publication in this collection
  09 Aug 2024
• Date of issue
  2024

History

• Received
  30 Aug 2023
• Accepted
  02 May 2024