Praxis teaching in the ambit of learning assessment of endodontics in a Chilean university

Abstract

The practice of assessing must ensure that the outcome of the process be a reflection of the learning achieved by students. The aim of this study was to describe the essential attributes of the praxis teacher in the area of Endodontics Learning Evaluation, in the School of Dentistry at the Universidad de Concepción, Chile. This study is designed to diagnose the reliability and objectivity of assessing the learning process, as a framework for innovation, and with a focus on evaluating endodontics skills. This hermeneutic study has a qualitative methodology. It was based on in-depth semi-structured interviews applied to 14 students and 5 teachers, and two focus groups consisting of eight students each. When the study was conducted, the research findings indicated that the evaluation process was not objective and lacked established criteria, and especially a guide to determining the skills. The theoretical evaluation was only summative. The formative role was not formally established. The subjects answered mostly psychometric instruments by multiple choice and with short or extended answers. It was concluded that teacher practice held meaning only if it was backed by academic expertise in the area of endodontics, according to no clear criteria or validated instruments. On the other hand, the groundwork was in place for using an epistemological style in endodontics. This provided a basis for the actual improvements, and allowed the tools developed to be dialectically interconnected with teacher experience. A quantitative analysis was not considered, but could be supplemented later to enhance the data analysis in a future study.

Endodontics; Dental Research; Competency-based Education; Educational Measurement

Introduction

For years, the teaching of endodontics in the dental schools of Chile has been presented in many ways, especially by the continual introduction of new technology involving new learning requirements, and has seen the emergence of dental schools staffed by teachers with diverse clinical training, but almost no teacher training.¹

Educational measurement is an essential component of education. The acquisition of skills, knowledge, affective processes and professional values defines the competent practice of dentistry.² Evaluation processes also use formative and summative logic. The formative role adds to the learning process. The summative role refers to the degree or level of performance achieved by students in order to progress.³ The self-evaluation of one’s needs, as a tool among other sources of feedback, can provide a more complete assessment of skills in clinical practice.⁴ A growing number freshmen are unable to accurately assess their results; for this reason, self-assessment skills should be taught. Self-assessment is an integral component of learning and of developing critical thinking skills.⁵

Sequential evaluation must also be developed. It enables cross assessment between skills and the integration of tools. For example, in order for students to assess problem-based learning, they must acquire their own set of critical thinking skills and develop interpersonal skills.⁶ The Spanish Society for Medical Education concluded that the evaluation of skills, like any other evaluative process, should be valid, reliable and feasible.⁷ Assessment is considered reliable and consistent when faculty members understand designated criteria and apply them in the same way, repeatedly and under similar conditions. An assessment calibrated with standard instruments is directly related to the performance of students in final exams, and improves the problem of inconsistency.^8,9

A curriculum based on skills emphasizes the analysis and resolution of problems, and relates to developing people with flexibility, autonomy and creativity. Education is a complex learning process that integrates knowledge, abilities, skills, values and attitudes, and becomes a methodological device for constructing job and disciplinary profiles.¹⁰

Students must be given an increasingly more objective evaluation process, not as a punitive and threatening or summative element to sanction their flaws, but rather, as a tool that will encourage ongoing improvement.¹¹ Preliminary observations made in the School of Dentistry, Universidad de Concepción, identified more traditional assessment practices that emphasized a summative score rather than promoting meaningful learning among students.¹²

Educational research can be based on a hermeneutic paradigm expressed by qualitative methodology. It entails primary research, such as data or facts, and is obtained firsthand. It is empirical investigation, i.e., working with facts studied in their natural environment, without any manipulation by the researchers.¹³ This methodology offers an opportunity to explore an in-depth scope in educational areas, not easily found in quantitative research.¹⁴

In this study, we set out to describe the essential features of the praxis teacher in the field of Learning Assessment of Endodontics, at the Universidad de Concepción.

Methodology

Endodontics is taught in the fourth year of the Dentistry course (Endodontics I) and then again in the fifth year (Endodontics II). The educational research performed in our study involved qualitative methodology¹⁵ and included subjects selected from the following study groups:

1. Teachers’ Cluster: 5 teachers of Endodontics, with over 7 years of teaching experience, and at least 22 hours focused on clinical education. All of the teachers were between 40 and 55 years old. At the time of the investigation, none had a master’s or doctorate degree. All 5 teachers of the Endodontics discipline were included;

2. Undergraduate Students’ Cluster: 15 undergraduate students who studied Endodontics I in their last year;

3. Specialty Graduates’ Cluster: 15 graduates who studied Endodontics II in their last year.

In demographical terms, the dentistry students at the Universidad de Concepción, have similar cultural and socioeconomic traits. These include variables such as age (21 to 25 years old) and sex, in that the number of men and women is proportionally the same. A high empathy level and positive educational climate were observed.¹² All of these antecedents constituted a uniform framework for research. It excluded undergraduates or graduates who had ever failed a course, and who failed the course because of illness or other extracurricular reason, and had to take remedial courses, where the assessment process is not the same as a normal curricular year. The subjects participated in the study after signing an informed consent, where the protocol met all the requirements and ethical standards set by the University of Bio-Bio for research work with people (Protocol # 01/11). All personal information was kept confidential.

In the first stage, data collection was conducted. One of the instruments used was an in-depth semi-structured interview applied in private. Each interview lasted about 45 minutes, and was applied by one of researchers to each of the five teachers, seven graduates and seven undergraduates taking part in the study. The interviews were very similar for each cluster; only a few questions were modified to be more understandable in meaning. For example, one question was, “Who performs the assessments in your course? Do you make self-evaluations or co-evaluations?” This question was changed to “Did you participate in your own assessment or in the evaluation of your classmates?” Three categories were established: “Evaluation Criteria” (Table 1), “Evaluation Procedures” (Table 2) and “Other items from the scope of learning assessment” (Table 3). The questions were based on the subcategories. The other instrument used to collect data consisted of focus groups. A semi-structured interview was applied by the same researcher to a group of eight graduates and another group of eight undergraduates. The responses expressed and the information collected have a high social significance that enhances validity procedures. The research interviewer acted as a facilitator, without making suggestions or expressing opinions, and the study became a thesis for a master’s degree in Educational Sciences. Another researcher had a doctorate degree in Educational Sciences. A methodological expert and discipline expert validated the questionnaires. Audio records of all the information were transcribed for analysis.¹⁶

Table 1
Example of Instrument: Questions made by the interviewer to teachers in Category A.

Categories	Subcategories	Questions
Category A Evaluation Criteria	A.1 Cognitive, procedural and attitudinal skills	1. What criteria are used to assess your students? (Criteria declared and/or undeclared)
		2. What skills are assessed in endodontics subjects, in the conceptual, procedural and attitudinal domains?
	A.2 Role of the Evaluation	3. What role does the assessment of students play in the endodontics subjects of the dental school?
		4. How do you make summative assessments in assessment practices? How do you obtain the score?
		5. How do you make formative assessments in assessment practices?
	A.3 Evaluation Agent	6. Who makes the assessments in your course? Do you perform self-evaluations or co-evaluations?
		7. If there is more than one evaluator for an evaluation process, how is the final grade decided?
	A.4 Normotypes	8. Do you consider a student’s individual assessment, as compared with other students?
		9. Do you apply any kind of proposed criteria for each student, according to his/her individuality? Why?

Table 2
Example of Instrument: Questions made by the interviewer to teachers in Category B.

Categories	Subcategories	Questions
Category B Evaluation Procedures	B.1 Assessment instruments	10. What instruments are most frequently used in assessing skills in endodontics subjects?
		11. How often are each of the instruments applied?
	B.2 Regulatory elements of the evaluation	12. Do you use patterns of correction?
		13. How do you construct a score scale?
		14. What level of requirements is applied in making the assessment?
		15. How do you calculate the time that students take to solve a test?

Table 3
Example of Instrument: Questions made by the Interviewer to teachers in Category C.

Categories	Subcategories	Questions
Category C Other items from the scope of Learning Assessment	C.1 Evaluation Training	16. From your experience in teaching, what is the importance of evaluating the endodontic practice of your students?
		17. Have you received training in performing evaluations?
	C.2 Experience in Performing Evaluations	18. What positive and/or negative experiences do you have in evaluating the theoretical aspects of the subject that you teach?
		19. What positive and/or negative experiences do you have in evaluating the practical aspects of the subject that you teach?
		20. What do you consider to be the strengths and weaknesses of teachers when it comes to evaluation processes?

In the second stage, the information was analyzed by a process of hermeneutical triangulation, in which all the relevant information to be studied was obtained. The information gathered in the study was crossed with a literature review of the evaluation criteria established in the curriculum.

There was a tabulation of an inductive nature, used to develop interpretive syntheses according to qualitative matrices of the three categories created for the study. These ultimately formed a representative body of research results. In triangulating the interviews, we used the method of contingency tables with an interpretative synthesis.¹⁴ The systemic networks method was employed for the triangulation of the focus group.¹⁵ Accordingly, we were able to group different ideas into subcategories within the dimensions established, to gain a more in-depth understanding of the subject under study. The operational definition of the categories was established based on reports of learning evaluations aimed at developing skills in dentistry.¹⁴ A methodological expert and a discipline expert validated the qualitative matrices.¹⁶

Results

The most relevant aspects of the categories studied are presented below, divided in their respective subcategories.

Category A – Evaluation Criteria

• Cognitive, procedural and attitudinal skills: The evaluation criteria were not clear, and were not mentioned as a skill, because the course subject was based on content. Seven procedural levels were not used; however, perception (interpretation), provision or condition (preparation) and guided response (learning) were evaluated. Levels, as shown by complex response (performance), adaptation (modification) or originality (creation) were not evaluated. Teachers reported that the attitudinal aspect was assessed based on the student-patient relationship. There was a mismatch, insofar as graduates and undergraduates said they did not make attitudinal assessments;

• Role of the Evaluation: It has been mentioned that the main role of the evaluation was the fulfillment of the course subject objectives and qualification of the students. Only the summative evaluation was formally established;

• Evaluation Agent: Only the teachers were assessor agents. The undergraduates and graduates applied self-evaluation as an informal process, without guidelines. There was no evaluation among undergraduates or among graduates. Both indicated a constant but informal co-evaluation;

• Normotypes: Although teachers recognize that evaluation-oriented criteria are better, they apply a standard evaluation. Reports show that only continuous assessment can be criterial.

Category B – Evaluation Procedures

• Assessment Instruments: Study subjects mainly used psychometric instruments with multiple choice, and short or extended questions. Graduates also exposed endodontic issues orally, and did not know how they were evaluated. Teachers without calibration carried out practical assessments with an anecdotal record that only some used, and did so in their own way. Both graduates and students recognized the subjectivity of evaluation;

• Regulatory elements of the evaluation:Some teachers used correction patterns. All used the scale scores and notes provided by the university, and applied the requirement level suggested by it. There was no agreement among teachers in regard to using some of the basic regulatory elements of the assessment. Graduates and students did not know the level of difficulty during the assessment stage.

Category C – Other items related to learning assessment

• Evaluation Training: At the time of the study, only two of the five teachers were Master’s students. Both graduates and undergraduates agree that the differences in the assessment process depend on personal characteristics and not on teacher training. The lack of training in performing assessments was recognized as a weakness;

• Experience in Evaluation: All the teachers agreed that evaluation is difficult because the guidelines are not clear. They recognized the need for a consensus and for changes to be made in order to ensure objective evaluation. The students ascribed importance to the feedback that some teachers give in the assessment process, as positive reinforcement. Finally, teachers recognized that the team should consist of specialists, in order to carry weight.

Discussion

Considering the purpose for describing the essential features of teaching practice in assessing endodontics, this study determined no clear criteria for evaluation. In the syllabi, criteria are expressed as learning outcomes. There was only hetero-evaluation in this study. This assesses the attitudinal area informally. Reports say that self-assessment is a tool that must be used in higher education.¹⁷

The main role of theoretical evaluation is summative. Practical assessment is subjective, and there is no consensus on how to perform the rating. Formative evaluation is not established, but it seems that continuous assessment would be a process evaluation. Evaluations are meant to be measured according to the standard, and only a continuous assessment type could be criterial. If we contrast this study with other studies, we must agree that assessment has fundamental implications in shaping future careers, and calls for urgent changes to be made in how results are obtained, so that our students may build the skills needed to integrate and apply the different areas of learning, which, together, define competent practice.¹⁸

The instruments used to assess the theory in this study were mainly psychometric. Practical assessment measures constitute only psychomotor aspects. Contrasting this study with another report, this study agrees that learning assessments are often vague and unreliable, and lead dentistry teachers to devise strategies and guidelines that will enable more effective evaluation to ultimately ensure the validity and reliability of the methodology of assessing students.¹⁹ It was also found that it is at each teacher’s discretion to use the regulatory elements that he/she best sees fit. This means that feedback and assessment are clearly lost as learning tools. Studies suggest that a consensus should be arrived at to develop evaluation systems that include valid criteria, rating scales and evaluator training as regulatory elements of student assessment.²⁰

The level of training of teachers in the area of basic assessment was empirical. However, although students do not relate the level of training to differences in the assessment process, other results indicate a correlation of this level with the flaws in the evaluation processes.²¹ The differences in how to perform assessments depend more on personal traits and less on teacher training. This agrees with reports in which students mentioned “character” as the principal teacher quality over “skill” and “communication.”¹³ Teachers recognized that team strength lies in a body of specialists that establish a uniform assessment framework. In 2006, the National Acceditation Commision (Comisión Nacional de Acreditación, CNA) was created in Chile to ensure the quality of university education, and the Chilean Education Reform called for universities to seek ongoing improvement in the level of teachers in the educational field.²²

Lastly, it has been reported that social studies should not use just statistical research tools. It is important to combine and complement qualitative information together with quantitative methods. This is characteristic of the framework that has been used in recent years to acquire knowledge about society. It stresses the importance of obtaining valid information that gives a complete account the real nature of the phenomena under study, and surpasses traditional paradigmatic dichotomies.²³ It would be an interesting avenue of investigation for future research to apply a complementation of both methodologies.

Conclusion

It was discovered that teaching practice in the field of evaluation was backed only by academic expertise in the area of endodontics, without clear criteria and validated instruments. Whereas specialist training provided only a framework to start the evaluation processes, it has been the basis for the actual innovations that guide our work today. Lastly, we would like to emphasize that qualitative methodology gives us the opportunity to gather in-depth information, which could be supplemented by a quantitative analysis to enhance the data analysis process.

References

• ¹
  Monsalves MJ. The Dentistry we have and don`t want: criticism of the current situation. Rev Chil Salud Publica. 2012;16(3):241-6. Spanish.
• ²
  Kramer GA, Albino J, Andrieu SC, Hendricson WD, Henson L, Horn B, et al. Dental student assessment toolbox. J Dent Educ. 2009 Jan;73(1):12-35.
• ³
  Laluna MCMC, Ferraz CA. Finalities and function of the evaluation in nursing education. Rev Bras Enferm (Brasília). 2007 Nov-Dez;60(6):641-5. Portuguese.
• ⁴
  Colthart I, Bagnall G, Evans A, Allburtt H, Haig A, Illing J, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach. 2008 Feb;30(2):124-45.
• ⁵
  Mould MR, Bray KK, Gadbury-Amyot CC. Student self-assessment in dental hygiene education: a cornerstone of critical thinking and problem-solving. J Dent Educ. 2011 Aug;75(8):1061-72.
• ⁶
  Allareddy V, Havens A, Karimbux N. Evaluation of a new assessment tool in problem-based learning tutorials in dental education. J Dent Educ. 2011 May;75(5):665-71.
• ⁷
  The Spanish Society for Medical Education. Declaration of the Lazareto de Mahón: evaluation of professional competencies in undergraduate medical education. Educ Med. 2004 Oct;7(4):103-5.
• ⁸
  Garland KV, Newell KJ. Dental hygiene faculty calibration in the evaluation of calculus detection. J Dent Educ. 2009 Mar;73(3):383-9.
• ⁹
  Priya M, Muthu MS, Amarlal D, Thomas E. Continuous assessment of undergraduate students at a dental college in India. J Dent Educ. 2012 Apr;76(4):501-8.
• ¹⁰
  Lipp MJ. A process for developing assessments and instruction in competency-based dental education. J Dent Educ. 2010 May;74(5):499-509.
• ¹¹
  Akthar A, Riaz H, Topping J. Students’ perception of university teaching behaviors. Teach Higher Educ. 2009 Dec;14(6):631-47.
• ¹²
  Morales S. Study of empathy level and achievement motivation in students of dentistry in University of Concepción. Rev Educ Cienc Salud. 2012 May;9(2):121-5. Spanish.
• ¹³
  Jahangiri L, McAndrew M, Muzaffar A, Mucciolo TW. Characteristics of effective clinical teachers identified by dental students: a qualitative study. Eur J Dent Educ. 2013 Feb;17(1):10-8.
• ¹⁴
  Sánchez Sanhueza GA, Cisterna Cabrera F. Learning assessment oriented towards the competencies development in dentistry. Educ Med Super. 2014 Jan-Mar;28(1):104-14. Spanish.
• ¹⁵
  Bliss J, Monk M, Ogborn J. Qualitative data analysis for educational research. 6th ed. London: Croom Helm; 1983. 215 p.
• ¹⁶
  Alcota M, Ruiz de Gauna P. González FE. Development of ethical practices and social responsibility in dental education at the university of Chile: student and faculty perceptions. Eur J Dent Educ. 2013 Feb;17(1):e70–6.
• ¹⁷
  Sullivan K, Hall C. Introducing students to self- assessment. Assess Eval Higher Educ. 1997 Jul;22(3)289-303.
• ¹⁸
  Chambers DW. Faculty ratings as part of a competency-based evaluation clinic grading system. Eval Health Prof. 1999 Mar;22(1):86-106.
• ¹⁹
  Beltrán RJ, Ikeda MC. Taxonomy of competencies in dental education. Rev Estomatol Herediana. 2004 Jan-Dec;14(1-2):107-11. Spanish.
• ²⁰
  Licari F, Knight GW, Guenzel PJ. Designing evaluation forms to facilitate student learning. J Dent Educ. 2008 Jan;72(1):48-58.
• ²¹
  Manogue M, Kelly M, Bartakova Masaryk S, Brown G, Catalanotto F, Choo-Soo T, et al. Evolving methods of assessment. Eur J Dent Educ. 2002 Oct;6(Suppl 3):53-66.
• ²²
  Cartes-Velásquez RA. Exponential growth of dental schools in Chile: effects on academic, economic and workforce issues. Braz Oral Res. 2013 Nov-Dec;27(6):471-7.
• ²³
  Bericat E. La integración de los métodos cuantitativo y cualitativo en la investigación social. 1st ed. Barcelona: Ariel; 1998. Chapter 56; p 103-46.

Publication Dates

• Publication in this collection
  2015

History

• Received
  26 May 2014
• Accepted
  13 Aug 2014
• Received
  13 Oct 2014