System for Evaluation of Teaching Qualities (SETQ) Smart for use in Brazil: resident’s version

ABSTRACT

Introduction:

It is commonly recognized that the enhancement of students’ learning potential is directly related to preceptors’ qualities. In this sense, the assessment of teachers’ practice becomes an essential element in ensuring the quality in the training of new specialists. However, in the context of medical education in Brazil, there are currently few studies on the evaluation of preceptors and a shortage of instruments for this purpose. One of the currently available instruments is the System for Evaluation of Teaching Qualities (SETQ) Smart.

Objective:

To conduct a cross-cultural adaptation and content validity of SETQ for use in Brazil.

Methods:

This instrument comprises two versions and, in this study, the version for residents was adapted for preceptor assessment. The cultural adaptation followed five steps: two initial English to Brazilian Portuguese translations; two synthesis translations; two back-translations; an expert committee assessment of conceptual analysis, semantic, idiomatic and cultural equivalences; and a pre-test. In addition, a panel of expert judges conducted the content validation.

Results:

Forty resident physicians, with a median age of 30 (IQR = 6.25), participated in the pre-test. Eighty percent of the participants rated the components that make up the SETQ Smart as clear and culturally appropriate, except for the title statement. The expert panel of judges comprised ten residents, 70% female. The agreement rates ranged from 80 to 100% regarding clarity, cultural adequacy, item representativeness within their respective domains and each item permanence in the instrument assessments.

Conclusion:

This study culturally adapted one of the two versions of the SETQ Smart for use in Brazil and provided preliminary evidence sources of validity of the versions through content validation.

Keywords:
Medical Education; Medical Residency; Preceptor Qualities; Evaluation

RESUMO

Introdução:

Já se reconhece que o incremento do potencial de aprendizagem dos alunos está diretamente relacionado às qualidades do preceptor. Nesse sentido, a avaliação da prática dos docentes impõe-se como um elemento essencial da garantia de qualidade na formação de novos especialistas. Todavia, no contexto da educação médica no Brasil, existem atualmente poucas pesquisas sobre avaliação de preceptores e escassez de instrumentos para essa finalidade. Um dos instrumentos atualmente disponíveis é o System for Evaluation of Teaching Qualities (SETQ).

Objetivo:

Este estudo teve como objetivo executar a adaptação transcultural e a validade de conteúdo do SETQ para uso no Brasil.

Método:

Esse instrumento é composto por duas versões, e neste estudo foi adaptada a versão dos residentes para avaliação do preceptor. A adaptação cultural seguiu cinco etapas: duas traduções iniciais do inglês para o português brasileiro; duas traduções de síntese; duas retrotraduções; uma avaliação do comitê de especialistas em termos de análise conceitual, equivalência semântica, idiomática e cultural; e um pré-teste. Além disso, um painel de juízes especialistas conduziu a validação de conteúdo.

Resultado:

Quarenta médicos residentes, com idade mediana de 30 anos (IQR = 6,25), participaram do pré-teste. Oitenta por cento dos participantes classificaram os componentes que compõem o SETQ Smart como claros e culturalmente apropriados, exceto pelo enunciado do questionário. O painel de juízes especialistas incluiu dez residentes, 70% do sexo feminino. As taxas de concordância variaram de 80% a 100% em relação à clareza, adequação cultural, representatividade dos itens dentro de seus respectivos domínios e permanência de cada item nas avaliações do instrumento.

Conclusão:

Este estudo adaptou culturalmente uma das duas versões do SETQ Smart para uso no Brasil e forneceu evidências preliminares de validade dessa versão por meio da validação de conteúdo.

Palavras-chave:
Educação Médica; Residência Médica; Qualidades do Preceptor; Avaliação

INTRODUCTION

Medical residency is a well-established and fundamental step towards the specialization of trained physicians, used by different countries to standardize the development and deepen the knowledge acquired by physicians during the undergraduate medical course^1),(2. In Brazil, the concept of medical residency was established in 1981, when it was first institutionalized by the National Medical Residency Commission (CNRM) as an in-service educational program of a medical institution that requires periodic evaluation³.

After the implementation of a medical residency program, continuous evaluation is required to ensure active monitoring, development, and quality control. According to the CNRM recommendations, these educational evaluations should include not only the residents’ assessment, but also the qualification of preceptors, supervisors, and coordinators of resident physicians⁴. Despite the formal requirement of preceptors’ continuous evaluation, this task lacks definition, guidelines, and standardization, as well as tools with evidence of validity, to enable a reliable assessment of desirable skills set to be developed by groups of medical preceptors⁵.

Residents play a central role in Medical Residency Programs, in which preceptors are fundamentally expected to act as facilitators to offer an enriching learning environment. Thus, the need for preceptors’ continuous development in terms of updating knowledge, skills and attitudes in their specialty and in the medical education field^6),(7.

However, there is a lack of guidelines and tools to enable reliable assessment of the range of desirable skills to be encouraged in groups of medical preceptors⁵.

Preceptors’ and students’ regular evaluation is, therefore, a fundamental prerequisite to guarantee the quality of residency programs, in addition to contributing to improving the selection process of preceptors/teachers based on several requirements, including training profile⁸. The opportunity to provide and receive feedback for improvement is another important benefit to be considered⁹. Nonetheless, despite available evidence, our country still suffers from the absence of a structured evaluation system, lacking standardization, periodicity and monitoring, which is reflected in the variety of institutional assessments that have been carried out, resulting in heterogeneous approaches, making it difficult to monitor performance over time, impairing adjustment to the needs of each service, impinging on preceptors’ development^10)-(12.

Fluit et al.¹³, in a systematic review about evaluation instruments for clinical faculty members during 1976 to 2010, concluded that most studies did not offer a satisfactory approach to crucial issues to be considered. Furthermore, of a total of 32 instruments, those most frequently used lacked a clear and well-articulated theoretical framework, hindering their application in routine practices^13)-(15.

Subsequently, Van Der Meulen et al¹⁶. conducted a systematic literature search for articles on questionnaire-based tools to assess physicians’ professional performance during 1966 to 2016. However, only 12 (23%) of the 53 articles contained evidence supporting all four components of Kane’s proposed validity criteria - scoring, generalizability, extrapolation, and implications. Among them, only 2 sought more evidence through a confirmatory analysis factor, highlighting the System for Evaluation of Teaching Qualities (SETQ) instrument.

The SETQ is the most widely used clinical teaching assessment tool in the Netherlands, having been translated and validated in several other countries^17)-(20. Its construction was grounded in theoretical constructs related to workplace learning and teaching, encompassing the set of competencies adopted by the Canadian Medical Education Directions for Specialists - CanMEDS, an entity that defines indispensable parameters for the success of specialized medical training^17)-(19. Furthermore, the SETQ has been continuously validated and enhanced by the international scientific community over the past twelve years and now has a virtual platform, which facilitates assessment and feedback for preceptors^17)-(19. Initially based on the Stanford Faculty Development Program (SFDP), the SETQ has evolved over time into SETQ Smart, adopting a broader and more discriminative format.

Considering the importance of preceptor assessment and providing feedback to enhance their skills, along with validity evidence of SETQ Smart and the absence of a national instrument composed of a system of preceptor self-assessment interfaced with resident evaluation of the preceptor, this article was written. It aimed to culturally adapt the System for Evaluating Teaching Qualities (SETQ) Smart version for medical residents in Brazil and to assess the content validity of this translated version of SETQ Smart in the Brazilian Portuguese language.

METHOD

Study design and ethical principles

This constituted a methodological and cross-sectional study, approved by the Human Research Ethics Committee (CAAE:60372322.6.1001.0121).

Instrument

The SETQ Smart¹⁷⁽¹⁷ is a system that employs two questionnaires^19),(23, one for medical preceptors’ self-assessment and the other for the residents’ evaluation of preceptors. It has shown strong validity evidence through Exploratory and Confirmatory Factor Analysis. The instrument is multidimensional and encompasses the seven categories for effective clinical teaching, comprising a total of 30 items and 6 domains, including aspects of professional demeanor, disposed on a seven-point Likert scale (completely disagree - completely agree), and an additional item to assess overall performance on a continuous 10-point scale¹⁷. Moreover, the version intended for residents incorporates two open-ended questions concerning the strengths and areas of improvement in the assessed preceptors’ performance.

Cross-cultural Adaptation of SETQ Smart for Medical Residents

Study Phases and Participants

The cross-cultural adaptation followed international recommendations^24)-(26. The process began by translating SETQ Smart into Brazilian Portuguese independently by two Brazilian translators fluent in English, one with expertise in the field and the other without, resulting in translation versions T1 and T2. Two of the authors and the two translators convened to compare the original version with the produced translations, resolving discrepancies and, through a consensus, generated a common translation into Brazilian Portuguese. This translation was then translated back into English (backtranslation) independently by two native English speakers proficient in Portuguese without access to the original version. Two English translations of SETQ Smart were generated (RT1 and RT2). The original version and the translations from the previous phases were compared and evaluated by a panel of expert judges, consisting of three healthcare professionals experienced in research and three professionals with expertise in cross-cultural adaptation research, including one licensed professional in Portuguese-English translation. Following their evaluation, the pre-final version of the instrument was formulated and then submitted to a pre-test, evaluated by medical residents, who were selected by convenience. Brazilian medical residents of both sexes, affiliated with any medical specialty, were considered eligible for this stage. For medical residents, proof of engagement in the program for at least the past six months was required.

Following the pre-test stage, the content validity and form of each generated version were analyzed by a panel of judges, intentionally selected according to the same eligibility criteria as the pre-test phase.

Participants from the Medical Residency Program of two public institutions located in Santa Catarina were selected for the pre-test and the panel of judges. All participants were invited via email and/or institutional phone.

Throughout all stages, the author of the original instrument was contacted to address any doubts concerning specific terms used.

Data Collection

The variables assessed by participants in the pre-test included clarity and cultural adequacy of the SETQ Smart components, answered in a three-point Likert scale for clarity (“not clear at all,” “partially clear,” and “completely clear”) and cultural adequacy (“not suitable for Brazilian culture,” “partially suitable for Brazilian culture,” and “completely suitable for Brazilian culture”). Furthermore, an open section was provided below each domain for participants to offer suggestions or comments.

The evaluation of content validity by the panel of expert judges ensued based on the pre-test results. The participants provided responses indicating their agreement or disagreement with the formulation of each component in terms of clarity, cultural adequacy, relevance of each item included in the respective domains, and the need for reordering. Beneath each evaluated component, there was an open section for participants to leave suggestions for refining the instrument, if necessary.

The data collection instrument utilized was the electronic form Google Forms for both pre-test and content validity evaluation.

For this type of assessment, a threshold of 80% of responses as “completely clear” and “completely suitable for Brazilian culture” was established as sufficient.

Content validity analysis was conducted through the percentage of agreement among the judges, calculated by adding the total number of agreements divided by the total number of participants and multiplying the result by 100. Typically, the minimum acceptable standard is an 80% agreement rate²⁷.

All analyses were performed using the R programming language, version 4.2.1²⁸.

RESULTS

Cross-Cultural Adaptation of SETQ Smart

Considering that some elements of SETQ Smart referred to residents affiliated with the anesthesiology specialty, authorized by the instrument’s main author, the formulation of the pre-final version was rephrased in items 6 “Teach residents the full spectrum of perioperative care,” 23 “Adhere to professional practice standards in the field of anesthesiology,” and 26 “Teach residents organizational aspects of perioperative practice” to meet the needs of the entire range of medical specialties - both clinical and surgical.

A total of 40 medical residents participated in the SETQ pre-test, nine males (22.5%) and 31 females (77.5%), with a median age of 30 years old (IQR=6.25).

The medical residents were engaged in the following specialties: Anesthesiology (2), General Surgery (2), Internal Medicine (1), Endocrinology (1), Obstetrics and Gynecology (2), Pediatric Hematology and Hemotherapy (2), Family and Community Medicine (14), Pediatric Nephrology (1), Pediatrics (13), Psychiatry (1), and Intensive Care (1). Among them, 65% were affiliated with the University Hospital, and 35% were affiliated with the Public Health School (ESP).

Figure 1 shows the participants’ assessment in the pre-test stage regarding clarity and cultural adequacy of all domains comprising SETQ Smart. It can be observed that, except for one statement, all others were deemed clear and culturally suitable by over 80% of the participants.

Figure 1
Illustrative graph of the SETQ Smart for medical residents, pre-test stage result.

Content Validity of the SETQ Smart

The panel of judges for the resident physicians’ version included 10 individuals, with seven (70%) female and three (30%) male individuals. The median age was 29.0 years old (IQR=2.5). In terms of residency completion, 30% were in the first year, 60% in the second year, and 10% in the third year. Concerning the medical specialty, 30% practiced Family and Community Medicine, 10% Anesthesiology, 10% General Surgery, 10% Internal Medicine, 10% Dermatology, 10% Gynecology and Obstetrics, 10% Pediatrics, and 10% Psychiatry. Of these, 70% were affiliated with the HU/UFSC institution, while 30% were affiliated with ESP.

Agreement percentage reached appropriate values as per literature standards²⁹, rendering a comprehensive reevaluation of the instrument unnecessary. Apart from five items - 6, 12, 17, 24, 26 - all other evaluated elements achieved 100% agreement among the expert judges.

Regarding the dimension “Cultural appropriateness and clarity for the target audience,” items numbered 6, 17, and 24 attained 90% agreement. Concerning the statement “The layout and spelling of this element are appropriately structured,” item 12 garnered 80% agreement, so it undergone adaptation and subsequent review by the panel of experts. This led to 100% agreement after adjustments in form and content. Concerning the proposition “This component should be retained within the scope of the instrument,” item 26 achieved 90% agreement.

Table 1 illustrates alterations made to the components comprising SETQ Smart, based on results from the pre-test and content validation phases. As evident, the reformulated items include those numbered 6, 9, 17, 19, 20, and 24.

Table 1
Reformulation of elements comprising SETQ Smart based on results obtained in the pre-test and content validity stages

Elementos reformulados	Versão pré-final - médico residente	Versão final - médico residente
Enunciado	Durante minha residência, esse médico(a)/preceptor (a) geralmente [...]	Por favor, avalie o desempenho do preceptor do(a) preceptor (a)/médico(a) que o(a) supervisiona, assinalando com um “x” o quadrado que condiz com seu grau de concordância com as afirmativas a seguir, em uma escala de […]
No enunciado de cada domínio foram acrescentadas as expressões “quanto a/ao” e “eu” “ele/a”	Ambiente de Aprendizagem	Quanto ao ambiente de aprendizagem ele/ela
Item 1	Estimula os residentes a apresentarem problemas.	Estimula os residentes a falarem sobre problemas
Item 6	Ensina todos os aspectos da atenção médica, em seus diversos contextos, aos residentes.	Ensina todos os aspectos da atividade/cuidado médica, relativos à especialidade, aos residentes
Item 9	É facilmente acessível durante o sobreaviso.	É facilmente acessível durante o sobreaviso e em supervisão à distância
Item 17	Avalia a aplicação do conhecimento dos residentes na prática diária.	Avalia com regularidade a aplicação do conhecimento dos residentes para cada tipo de paciente
Item 19	Fornece feedback positivo aos residentes.	Realça comportamentos desejáveis/adequados já alcançados
Item 20	Fornece feedback corretivo aos residentes.	Aponta comportamentos profissionais ainda não alcançados e os indesejáveis que precisam ser modificados
Item 24	Demonstra compaixão e integridade em suas relações com pacientes e familiares	Demonstra empatia e integridade em suas relações com pacientes e familiares e age com compaixão para aliviar o sofrimento do paciente
Modelo como Profissional	O modelo que os médicos(as)/preceptores(as) representam abrange diferentes componentes que podem ser expressos em 3 tipologias de modelos. Por favor, avalie os seguintes tipos de modelo caso você considere que esse(a) médico(a)/ preceptor(a) é um modelo para você. Durante minha residência, esse(a) preceptor(a) geralmente...	O papel do (a) preceptor (a)/professor (a) é exemplo em diferentes aspectos, que podem ser expressos em um modelo como professor, médico e pessoa. Por favor, indique seu grau de concordância com o desempenho do (a) preceptor (a) em cada uma dessas funções

• Abbreviation: SETQ Smart = System for Evaluation of Teaching Qualities.14

The section regarding the title and response scale showed the lowest level of clarity in the evaluation. Thirty-seven-point-five percent of the residents considered this section partially clear, emphasizing the lack of connection between the introduction and the subsequent evaluated items. This highlighted the need for a more precise introductory text regarding the instrument. While an alternative statement was formulated, consultation with the instrument designer led to the retention of the seven-point Likert scale, rather than adopting a five-point scale as suggested by some participants. Given our limited sample size in comparison to conducted studies and guided by the internationally validated SETQ instrument, after reviewing relevant literature³⁰, it seemed more appropriate to maintain a higher-score scale, adjusted to the SETQ Smart protocol.

Additionally, regarding the statement of each domain, expressions such as “regarding”, and “he/she” were added. For instance, the domain “Assessment” was reformulated as follows: “Regarding Assessment, he/she.” This reformulation is presented in Table 1.

Furthermore, considering the limitations imposed by the pandemic context and the growing appeal for distance learning technologies, we opted to complement item 9 with “under remote supervision.”

Regarding items 19 and 20, it became imperative to provide examples to optimize the understanding of the concepts of “positive feedback” and “corrective feedback” within the context of performance evaluation processes. The term “corrective feedback” was identified as lacking clarity, allowing for alternative interpretations. The need for more precise definitions for “positive feedback” and “corrective feedback” emerged as a pressing concern and was further addressed by considering the residents’ suggestions for additional clarifications. This expanded discussion took place within the context of the “Feedback to Residents” domain. A viable alternative was constructed based on these considerations and is presented in Table 1.

In our study, we also chose to evaluate data related to the respondents’ identification and professional information. After considerations, we added the word gender to the previous sex (female and male) identification item, allowing for inclusion and encompassing other interpretations of this concept.

A preference emerged for an alternative phrasing of item 24, replacing the term “compassion” with “empathy.” This required consultation with the original version author. Recognizing the breadth of these concepts, item 24 was made clearer and exemplified, now being formulated as follows: “Demonstrates empathy and integrity in relationships with patients and family members and acts with compassion to alleviate patient suffering.” This reformulation is also presented in Table 1.

Regarding item 17 within the Assessment domain, due to its similarity to item 15 within the same domain, as reported by the participants in the pre-test, and considering its connection to the study conducted on the preceptor’s self-assessment version, it was decided to adjust it according to the corresponding phrase in the preceptor’s self-assessment version. This decision was influenced by the observation that all other items in the different versions of the SETQ tool are evaluated from two distinct perspectives (preceptor’s and resident’s), forming a mirrored relationship between the questionnaires. This approach of parallelism between the items enhances understanding, as the same item is evaluated by two distinct groups of evaluators in the various presented versions.

In addition to the abovementioned modifications , other changes were limited to structural and grammatical issues, as observed in item 1 and “Model as a Professional,” as listed in Table 1. This was carried out with the intention of enhancing the Brazilian version, while simultaneously preserving the maximum originality of the instrument. The instrument was altered to its definitive formatting, and the alterations are presented in the following table.

Thus, the final version of the SETQ Smart for the residents’ evaluation of preceptors was retranslated into English and submitted to the original author’s approval, which was endorsed. The final version for medical residents is available at the following link: https://github.com/NaaraiCamboim/Questionario_residente/blob/main/Questionario_Residente.pdf

DISCUSSION

This study has produced a version of the SETQ Smart for medical residents to evaluate their preceptors, yielding findings similar to the version designed for medical preceptors’ self-assessment for use in Brazil³¹. Both in the pre-test phase and during the evaluation by a panel of expert judges, it was observed that all items of the instrument were considered clear and culturally appropriate for the Brazilian context, as well as relevant to the construct and representation of their respective domains.

Regarding the adaptation of three items related to the anesthesiology specialty, it was carried out taking into consideration studies with the original version of SETQ. This adaptation was based on studies involving the original SETQ with more than 1,500 participant medical residents and preceptors across more than 29 residency programs encompassing various specialties - both clinical and surgical^32),(33.

Divergences were found in the interpretation of the concepts of compassion and empathy, consistent with controversies surrounding their meanings in the literature³⁴. Despite considerations about the use of these terminologies, the significance of the evaluation utilizing these concepts within preceptor competencies was acknowledged. Therefore, this topic was thoroughly addressed and discussed by the medical preceptors that adapted the medical preceptors’ self-assessment version of SETQ Smart³¹. Although distinct, the concepts describe significant dimensions of human interactions and hold particular relevance within the context of medical education.

“Empathy” refers to a person’s ability to place themselves in another’s place, whereas “compassion” involves the expression of care. While these terms are related, they possess distinct fundamental nuances for ethical and effective medical practice, contributing to a patient-centered and humane approach³⁵.

Concerning feedback, our research unveiled diverse perceptions and considerations related to this domain. Such discrepancies are often noted in the literature, given the variability in feedback denominations and functions, which can be applied in various ways in daily practice^36),(37.

It is important to recognize the lack of consistent comparisons with other Brazilian studies focusing on the evaluation of preceptors by medical residents in the available literature. This gap can be justified considering that even among the few adapted and validated questionnaires in specific areas of Brazilian medical specialization, none of them incorporate the evaluation system interface that integrates resident evaluation with preceptor self-assessment. Furthermore, some of these studies adapted their tools from the perspective of medical undergraduate students rather than medical residents, in the context of performance evaluation processes in postgraduate medical services. These studies remain below the comprehensive scope of SETQ since, as previously emphasized, instruments with this profile have not yet been created and/or validated in Brazil.

Our findings are aligned with the research by Ansari et. al,⁴⁰. concerning the applicability of SETQ in diverse clinical settings. However, in addition to the aspect of cross-cultural adaptability, these researchers also assessed the reliability of the instrument.

A limitation of this study is its sample, which is restricted to a specific geographic area in southern Brazil and it does not encompass all specialties in the medical field. This geographical delimitation implies, in turn, a limited representation of cultural differences and possibly pedagogical approaches, which can vary significantly in different regions of the country. We therefore recognize that the results obtained in this research have restricted generalization due to the specific nature of its sample. In future research, we will consider expanding the geographic scope, covering the entire national territory and including several medical specialties, aiming to provide a more comprehensive and enlightening analysis of the variables under study.

Given these findings, we suggest that medical institutions outline a path towards the effective implementation of ongoing professional development programs aimed at preceptors in medical teaching institutions. Such programs should emphasize clinical reviews, advanced pedagogical techniques and effective communication training, consistent with assigned residents’ responsibilities. Hence, it is relevant that educational institutions adopt previously validated tools for the evaluation and establishment of a preceptor development system. Alternatively, institutions can validate instruments already in use, since the preceptor’s assessment is part of active monitoring, development and quality control, as recommended by the National Medical Residency Commission (CNRM).

Furthermore, the results obtained in this study promote the propagation of an institutional culture that values medical education and the implementation of constructive feedback, thus contributing to improving the teaching process. In this sense, it is proposed to apply this approach as the basis for introducing a system of regular preceptor performance assessments, providing continuous feedback and identifying specific areas for improvement. The systematic incorporation of these practices has been shown to strengthen the residents’ training and, ultimately, raise medical care standards.

Additional research involving SETQ Smart is currently underway to provide further evidence of its validity. This research takes form as a multicenter study in Brazil, aiming to offer a tool with a broad sample size to gather multiple sources of validity evidence for use across various fields of postgraduate medical education in the country.

CONCLUSION

This study has successfully generated a version of the SETQ Smart tailored for use in Brazil, aiming to evaluate the performance of medical preceptors from the perspective of medical residents. Both during the pre-test phase and within the framework of the expert panel review, all items that constitute the instrument were deemed clear, valid, and suitable for the unique aspects of Brazilian culture. As a result, this version presents preliminary evidence of validity from various sources, contributing to the overall validation process.

ACKNOWLEDGEMENTS

We extend our heartfelt gratitude to esteemed Professor Paulo Henrique Freire Vieira, who dedicated his life to education. His teachings have left indelible marks of wisdom on my journey. We are indebted to his insightful readings of this work, as the inspirational legacy he imparted resonates throughout this article. The pursuit of a comprehensive understanding of the human essence, furthered by introspection and continuous evaluation, critiques the confines of both system and soul. This endeavor seeks to embrace transdisciplinary ecoformation, fostered through collaborative and cooperative approaches. Recognizing the value of intellectual and healthcare resources in advancing a more just and sustainable society, we strive to construct a genuine “Commons.”

REFERÊNCIAS

• ¹
  Weisz G. Divide and conquer: a comparative history of medical specialization. Cary, NC: Oxford University Press; 2005.
• ²
  Wright JR, Schachar, NS. Necessity is the mother of invention: William Stewart Halsted’s addiction and its influence on the development of residency training in North America. Can J Surg. 2020;63(1):13-9. doi: https://doi.org/10.1503/cjs.003319.
  » https://doi.org/https://doi.org/10.1503/cjs.003319
• ³
  Brasil. Lei nº 6.932/81, de 7 de julho de 1981. Dispõe sobre as atividades do médico residente e dá outras providências. Diário Oficial da União; 9 jul 1981. Seção 1, p. 12789-90.
• ⁴
  Brasil. Decreto nº 7.562, de 15 de setembro de 2011. Dispõe sobre a Comissão Nacional de Residência Médica - CNRM e o exercício das funções de regulação, supervisão e avaliação de instituições que ofertam residência médica e de programas de residência médica. Diário Oficial da União ; 16 set 2011. p. 3.
• ⁵
  Sam AH, Fung CY, Barth J, Raupach T. A weighted evaluation study of clinical teacher performance at five hospitals in the UK. Adv Med Educ Pract. 2021;12:957-63. doi: https://doi.org/10.2147/AMEP.S322105.
  » https://doi.org/https://doi.org/10.2147/AMEP.S322105
• ⁶
  Johnson CE, Keating JL, Leech M, et al. Development of the feedback quality instrument: a guide for health professional educators in fostering learner-centred discussions. BMC Med Educ. 2021;21(382):1-17. doi: https://doi.org/10.1186/s12909-021-02722-8.
  » https://doi.org/https://doi.org/10.1186/s12909-021-02722-8
• ⁷
  Ferreira IG, Cazella SC, Costa MR da. Medical preceptorship: perceptions and perspectives of new family doctors in Brazil. Rev Bras Educ Med. 2023;47(2):1-11. doi: https://doi.org/10.1590/1981-5271v47.2-20220299.ING.
  » https://doi.org/https://doi.org/10.1590/1981-5271v47.2-20220299.ING
• ⁸
  Heath JK, Dine CJ, Andolsek KM, Burke AE. Teaching the teachers with milestones: using the ACGME Milestones Model for Professional Development. J Gr Med Educ. 13:124-6. doi: https://doi.org/10.4300/JGME-D-20-00891.1.
  » https://doi.org/https://doi.org/10.4300/JGME-D-20-00891.1
• ⁹
  Johnson CE, Keating JL, Boud DJ, et al. Identifying educator behaviours for high quality verbal feedback in health professions education: literature review and expert refinement. BMC Med Educ . 2016;16(96):1-12. doi: https://doi.org/10.1186/s12909-016-0613-5.
  » https://doi.org/https://doi.org/10.1186/s12909-016-0613-5
• ¹⁰
  Lawall PZM. A formação do preceptor médico em medicina de família e comunidade: uma proposta de diálogo com a andragogia [dissertação]. Brasília: Fundação Oswaldo Cruz; 2019.
• ¹¹
  Costa LB, Castro SS, Dourado DXC, Praxedes BS, Mota TC, Tavares TMRL. Adaptação transcultural do questionário EFFECT para português brasileiro. Rev Bras Educ Med . 2021;45(3):1-11. doi: https://doi.org/10.1590/1981-5271v45.3-20200527.
  » https://doi.org/https://doi.org/10.1590/1981-5271v45.3-20200527
• ¹²
  de Oliveira Filho GR , Mago AJD, Garcia JHS, Goldschmidt R. An instrument designed for faculty supervision evaluation by anesthesia residents and its psychometric properties. Anesth Analg. 2008;107(4):1316-22. doi: https://doi.org/10.1213/ane.0b013e318182fbdd.
  » https://doi.org/https://doi.org/10.1213/ane.0b013e318182fbdd
• ¹³
  Fluit CRMG, Bolhuis S, Grol R, Laan R, Wensing M. Assessing the quality of clinical teachers identification of studies: a systematic review of content and quality of questionnaires for assessing clinical teachers. J Gen Intern Med. 2010;25(12):1337-45. doi: https://doi.org/10.1007/s11606-010-1458-y.
  » https://doi.org/https://doi.org/10.1007/s11606-010-1458-y
• ¹⁴
  Tigelaar DE, Dolmans DH, Wolfhagen IH, et al. The development and validation of a framework for teaching competencies in higher education. High Educ. 2004;48(2):253-68. doi: https://doi.org/10.1023/B:HIGH.0000034318.74275.e4.
  » https://doi.org/https://doi.org/10.1023/B:HIGH.0000034318.74275.e4
• ¹⁵
  Holmboe ES, Ward DS, Reznick RK, et al. Faculty development in assessment : the missing link in competency-based medical education. Acad Med. 2011;86(4):460-7. doi: https://doi.org/10.1097/ACM.0b013e31820cb2a7.
  » https://doi.org/https://doi.org/10.1097/ACM.0b013e31820cb2a7
• ¹⁶
  Van Der Meulen MW, Smirnova A, Heeneman S, Oude Egbrink MGA, Van Der Vleuten CPM, Lombarts KMJMH. Exploring validity evidence associated with Questionnaire-Based Tools for Assessing the Professional Performance of Physicians: a systematic review. Acad Med . 2019;94(9):1384-97. doi: https://doi.org/10.1097/ACM.0000000000002767.
  » https://doi.org/https://doi.org/10.1097/ACM.0000000000002767
• ¹⁷
  Lombarts KMJMH, Ferguson A, Hollmann MW, Malling B, Arah OA. Redesign of the System for Evaluation of Teaching Qualities in anesthesiology residency training. Anesthesiology. 2016;125(5):1056-65. doi: https://doi.org/10.1097/ALN.0000000000001341.
  » https://doi.org/https://doi.org/10.1097/ALN.0000000000001341
• ¹⁸
  CanMEDS 2000: Extract from the CanMEDS 2000 Project Societal Needs Working Group Report. Med Teach. 2000;22(6):549-4. doi: https://doi.org/10.1080/01421590050175505.
  » https://doi.org/https://doi.org/10.1080/01421590050175505
• ¹⁹
  Lombarts KMJMH, Bucx MJL, Arah OA. Development of a system for the evaluation of the teaching qualities of anesthesiology faculty. Anesthesiology . 2009;111(4):709-16. doi: https://doi.org/10.1097/ALN.0b013e3181b76516.
  » https://doi.org/https://doi.org/10.1097/ALN.0b013e3181b76516
• ²⁰
  Litzelman DK, Stratos GA, Marriott DJ, Skeff KM. Factorial validation of a widely disseminated educational framework for evaluating clinical teachers. Acad Med . 1998;73(6):688-95. doi: https://doi.org/10.1097/00001888-199806000-00016.
  » https://doi.org/https://doi.org/10.1097/00001888-199806000-00016
• ²¹
  Litzelman DK, Westmoreland GR, Skeff KM, Stratos GA. Factorial validation of an educational framework using residents’ evaluations of clinician-educators. Acad Med . 1999;74(10):S25-S27. doi: https://doi.org/10.1097/00001888-199910000-00030.
  » https://doi.org/https://doi.org/10.1097/00001888-199910000-00030
• ²²
  Lombarts MJ, Bucx MJ, Rupp I, Keijzers PJ, Kokke SI, Schlack W. An instrument for the assessment of the training qualities of clinician-educators. Ned Tijdschr Geneeskd. 2007;151(36):2004-8.
• ²³
  Van der Leeuw R, Lombarts K, Heineman MJ, Arah O. Systematic evaluation of the teaching qualities of obstetrics and gynecology faculty: reliability and validity of the SETQ tools. PLoS One. 2011;6(5):1-7. doi: https://doi.org/10.1371/journal.pone.0019142.
  » https://doi.org/https://doi.org/10.1371/journal.pone.0019142
• ²⁴
  Beaton DE, Bombardier C, Guillemin F, Ferraz MB. Guidelines for the process of cross-cultural adaptation of self-report measures. Spine (Phila Pa 1976). 2000;25(24):3186-91. doi: https://doi.org/10.1097/00007632-200012150-00014.
  » https://doi.org/https://doi.org/10.1097/00007632-200012150-00014
• ²⁵
  Anunciação L. Conceitos e análises estatísticas com R e JASP. São Paulo: Nila Press; 2021.
• ²⁶
  Terwee CB, Prinsen CAC, Chiarotto A, Westerman MJ, Patrick DL, Alonso J, et al. COSMIN methodology for evaluating the content validity of patient-reported outcome measures: a Delphi study. Qual Life Res. 2018;27(5):1159-70. doi: https://doi.org/10.1007/s11136-018-1829-0.
  » https://doi.org/https://doi.org/10.1007/s11136-018-1829-0
• ²⁷
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Cienc Saude Colet. 2011;16(7):3061-8. doi: https://doi.org/10.1590/S1413-81232011000800006.
  » https://doi.org/https://doi.org/10.1590/S1413-81232011000800006
• ²⁸
  R Core Team. A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2020.
• ²⁹
  Hyrkäs K, Appelqvist-Schmidlechner K, Oksa L. Validating an instrument for clinical supervision using an expert panel. Int J Nurs Stud. 2003;40(6):619-25. doi: https://doi.org/10.1016/s0020-7489(03)00036-1.
  » https://doi.org/https://doi.org/10.1016/s0020-7489(03)00036-1
• ³⁰
  Debets MPM, Scheepers RA, Boerebach BCM, Arah OA, Lombarts KMJMH. Variability of residents’ ratings of faculty’s teaching performance measured by five- and seven-point response scales. BMC Med. Educ. 2020;20(1):325. doi: https://doi.org/10.1186/s12909-020-02244-9.
  » https://doi.org/https://doi.org/10.1186/s12909-020-02244-9
• ³¹
  Bezerra CN. Adaptação transcultural e validade de conteúdo do System for Evaluation of Teaching Qualities (SETQ) Smart para o uso no Brasil [dissertação]. Florianópolis: Universidade Federal de Santa Catarina; 2021.
• ³²
  Arah OA, Heineman MJ, Lombarts KMJMH. Factors influencing residents’ evaluations of clinical faculty member teaching qualities and role model status. Med Educ. 2012;46(4):381-9. doi: https://doi.org/10.1111/j.1365-2923.2011.04176.x.
  » https://doi.org/https://doi.org/10.1111/j.1365-2923.2011.04176.x
• ³³
  Boerebach BCM, Arah OA, Busch ORC, Lombarts KMJMH. Reliable and valid tools for measuring surgeons’ teaching performance: residents’ vs. self evaluation. J Surg Educ. 2012;69(4):511-20. doi: https://doi.org/10.1016/j.jsurg.2012.04.003.
  » https://doi.org/https://doi.org/10.1016/j.jsurg.2012.04.003
• ³⁴
  Perez-Bret E, Altisent R, Rocafort J. Definition of compassion in healthcare: a systematic literature review. Int J Palliat Nurs. 2016;22(12):599-606. doi: https://doi.org/10.12968/ijpn.2016.22.12.599.
  » https://doi.org/https://doi.org/10.12968/ijpn.2016.22.12.599
• ³⁵
  Sinclair S, Norris JM, McConnell SJ, Chochinov HM, Hack TF, Hagen NA, et al. Compassion: a scoping review of the healthcare literature. BMC Palliat Care. 2016;15(6):1-16. doi: https://doi.org/10.1186/s12904-016-0080-0.
  » https://doi.org/https://doi.org/10.1186/s12904-016-0080-0
• ³⁶
  Jamtvedt G, Young JM, Kristoffersen DT, Thomson O’Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2003;(3). doi: https://doi.org/10.1002/14651858.CD000259.
  » https://doi.org/https://doi.org/10.1002/14651858.CD000259
• ³⁷
  Paterson C, Paterson N, Jackson W, Work F. What are students’ needs and preferences for academic feedback in higher education? A systematic review. Nurse Educ Today. 2020 Fev;85:104236. doi: https://doi.org/10.1016/j.nedt.2019.104236.
  » https://doi.org/https://doi.org/10.1016/j.nedt.2019.104236
• ³⁸
  Miranda PR, Romano VF. Uma proposta de instrumento de avaliação pedagógica da preceptoria para residências em Medicina de Família e Comunidade. Rev Bras Med Fam Comunidade. 2021;16(43):1-9. doi: https://doi.org/10.5712/rbmfc16(43)2680.
  » https://doi.org/https://doi.org/10.5712/rbmfc16(43)2680
• ³⁹
  Gomes OV, Morais RJL, Schwigel PA, Vasconcelos PT, Lima TRM, Andreto LM, et al. Validade e confiabilidade do Maastricht Clinical Teaching Questionnaire para língua portuguesa. Rev Bras Educ Med . 2019;43(2):15-24. doi: https://doi.org/10.1590/1981-52712015v43n2RB20180061.
  » https://doi.org/https://doi.org/10.1590/1981-52712015v43n2RB20180061
• ⁴⁰
  Al Ansari A, Strachan K, Hashim S, Otoom S. Analysis of psychometric properties of the modified SETQ tool in undergraduate medical education. BMC Med Educ. 2017;17(56):1-9. doi: https://doi.org/10.1186/s12909-017-0893-4
  » https://doi.org/10.1186/s12909-017-0893-4

• 7
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Carlos Collares.

Publication Dates

• Publication in this collection
  08 July 2024
• Date of issue
  2024

History

• Received
  13 Sept 2023
• Accepted
  06 May 2024