The evaluation of higher dental education in Brazil: A cross-sectional study

Avaliação da educação superior de Odontologia no Brasil: um estudo transversal

ORIGINAL ARTICLE

The evaluation of higher dental education in Brazil: A cross-sectional study

Avaliação da educação superior de Odontologia no Brasil: um estudo transversal

Renata Grazziotin-Soares; Leandro de Azambuja Reichert; Adair Luiz Stefanello Busato; Airton Pozo de Mattos; Alcebíades Nunes Barbosa; Vania Regina Camargo Fontanella

School of Dentistry, Lutheran University of Brazil (ULBRA), Canoas, RS, Brazil

^{Correspondence}

ABSTRACT

PURPOSE: In Brazil, the SINAES is a process that is currently used to assess higher education institutions, undergraduate programs and students' performance on the ENADE national exam. This study aimed at assessing the Brazilian undergraduate programs in Dentistry, inquiring if the evaluation results induced to changes to improve the teaching-learning process.

METHODS: This cross-sectional study utilized a questionnaire that was e-mailed to 165 professors/coordinators of undergraduate programs in Dentistry in Brazil in 2003/2004. The data analysis was based on the themes of the various open- and closed-ended questions: questions 1, 2, 3 and 4 referred to the evaluation of each institution and its undergraduate program, whereas questions 5, 6 and 7 referred to the ENADE.

RESULTS: Changes to improve the quality of teaching in Dentistry and the education of new professionals were driven by the results of the evaluation. These changes were implemented at all institutions that returned the completed questionnaires.

CONCLUSION: Improvements in the quality of Brazilian Dentistry will only be possible if the education of the students and the professors is improved.

Key words: Dental education; educational measurement; cross-sectional studies

RESUMO

OBJETIVO: No Brasil, o SINAES é a proposta de avaliação da educação superior atualmente em vigência e que avalia as instituições de educação superior, os cursos de graduação e o desempenho dos estudantes (ENADE). Este estudo abordou a avaliação dos cursos de Odontologia em 2003 e 2004 para observar se os resultados da avaliação induziram melhora no processo ensino-aprendizagem.

METODOLOGIA: Este estudo transversal foi realizado por meio de questionários com questões abertas e fechadas enviados por e-mail a 165 professores/coordenadores dos programas de graduação em Odontologia do Brasil em 2003/2004. As questões 1, 2 e 3 se referiram à avaliação das Instituições e dos cursos de Odontologia. As questões 5, 6 e 7 se referiram ao ENADE.

RESULTADOS: As mudanças com objetivo de melhora na qualidade do ensino em Odontologia e na educação de novos profissionais são impulsionadas pelos resultados do processo de avaliação. Tais mudanças foram implementadas em todas as Instituições de ensino que retornaram os questionários preenchidos.

CONCLUSÃO: Mudanças na qualidade da Odontologia brasileira somente serão possíveis se ocorrer melhora no processo educativo de estudantes e professores.

Palavras-chave: Educação em odontologia; avaliação educacional; estudo transversal

Introduction

Several models of evaluation have been proposed for classifying institutions of higher education all over the world (1,2). The first worldwide global university ranking was published in 2003 by the Institute of Higher Education at Shanghai Jiao Tong University (SJTU) (3). In Brazil, the first model of higher education evaluation for undergraduate programs was introduced in 1983. Since 2004, the National Higher Education Evaluation System (SINAES) has been used in Brazil (1,4).

The SINAES has two characteristics: (1) an emancipator evaluation, which makes it possible for institutions to participate in their own evaluation processes and (2) a regulatory evaluation, through which, according to the received concept, there is control over the opening of new undergraduate programs and permission for the ones already open to continue operating (4,5). The SINAES is a system of global evaluation and is composed of three main elements: 1) Evaluation of Higher Education Institutions (AVALIES), compiled from internal and external evaluations; 2) Evaluation of Graduate Programs (ACG), implemented by the commissions of professors; and 3) Evaluation of Students' Performance, based on the performance of a sample of freshmen (students in their first year) and a sample of senior students (students in their last year) on a theoretical test, called the National Examination of Students' Performance (ENADE) (6).

These three components are part of the same evaluation system; however, each of these processes occurs at different times and in different situations. Each component indicates the strong and weak aspects of the programs and institutions, and this information should help improve the education of the students in addition to providing the society with information about Brazilian higher education (6). The SINAES has been used to emphasize and stimulate the expected changes in dental education that were proposed by the National Curricular Guidelines (DCN).

The DCN for undergraduate programs in Dentistry, which was approved in November, 2001, provided new guidance for programs in Dentistry. These guidelines replaced the minimum curriculum (based on the organization of programs and specializations) with a more flexible model. The main guiding principle is a more humanized practice of Dentistry that focuses on human beings (7,8).

Using questionnaires that were e-mailed to 165 professors/coordinators at undergraduate dental programs in Brazil in 2003/2004, we sought to evaluate dental programs that were operating during that period; the questionnaire addressed whether the results induced improvements to the teaching-learning process.

Methods

After approval by the local Ethics Committee in Research, a questionnaire with ten questions was sent by e-mail (between September 2007 and May 2008) to the professors/coordinators of undergraduate programs in Dentistry in Brazil during the period of 2003-2004.

In 2003, there were 165 programs in Dentistry across the 27 states and federal districts of the country. (9). An e-mail was sent to all current professors/coordinators. Thus, the e-mail addresses of the professors/coordinators in 2003-2004 were received. Then, 27 questionnaires were e-mailed based on the state or federal district. Because of the limited number of responses and because the questionnaires were e-mailed immediately following a vacation period at the institutions, between March 16 and 19, 2008, at the beginning of the school year in Brazil, 27 e-mails were distributed again to participants in research, except for those participants who had already returned a completed questionnaire. In the cases in which e-mails were unsuccessfully delivered, the addresses were researched further or the questionnaires were sent to the dental program secretary.

The questionnaire contained ten questions. Seven questions were analyzed in this article (Table 1).

The process of data analysis was undertaken through the aggregation of different themes approached in the questions. Questions 1, 2, 3 and 4 referred to the evaluation of institutions and undergraduate programs in Dentistry, whereas questions 5, 6 and 7 referred to the ENADE.

Questions 1, 2, 3, 5 and 7 were closed-ended questions, whereas questions 4 and 6 were open-ended questions. The responses to the closed-ended questions were separated into affirmative or negative responses, and then they were discussed. The responses to open questions were categorized into various reported themes and then were discussed.

Results

The population of the study consisted of 165 subjects. Only 14 professors/coordinators returned completed questionnaires. The responses were obtained from 9 of the 27 states and federal districts of the country. The numbers of professors/coordinators who responded in relation to their state or federal district were as follows: 4 in Rio Grande do Sul, 1 in Santa Catarina, 1 in Paraná, 1 in Minas Gerais, 3 in São Paulo, 1 in Bahia, 1 in Espírito Santo, 1 in Pernambuco and 1 in Ceará. Table 2 shows the results of the closed and open questions.

Discussion

Any educational institution is a conglomerate of schools, departments, teams, and individual scientists. An aggregate ranking may not accurately reflect the constituent parts of an institution. None of the existing international ranking systems aims to quantify intra-institutional diversity in their evaluations. There has therefore been a lack of relevant information (2,10,11). The SINAES, in Brazil, is an advanced higher education evaluation because its results are separated according to the programs and the students' performance.

A limitation of the present study was the presence of questions characterized as closed, demonstrating a pre-conceived idea. For example, question 1 may have biased the participants toward a positive response. In face-to-face interviews, this problem may be minimized, as the interviewer may ask for clarification of the answers to the closed ended questions.

Despite of index of responses being low, the collected data were of very high quality and can be extrapolated to other Brazilian Institutions.

Our decision to work with the directors of programs as co-authors (12-14) was based on their political power and on their direct involvement in the creation of curricula and the representation of colleges.

Based on the responses to the first question in our research, we noted that all of the interviewees answered affirmatively. In this sense, a respondent emphasized, "... evaluations highlight the strengths and weaknesses of an institution and encourage the professors to profoundly reflect on this information".

Proposed changes, with the goal of improving programs, were implemented at all institutions. We may perceive that changes occurred not only because of the proposals of the evaluation but also, and perhaps most significantly, because of the reforms that addressed the curricula.

Beyond the three factors that were evaluated with the SINAES process, in the opinion of most of the interviewees, great importance was ascribed to the learning-teaching process in practice scenarios, as was interdisciplinarity, which referred to the insertion of faculty members into the pedagogical projects of the program.

The pedagogical projects established at most of Brazil's higher dental education institutions have consisted of certain scientific characteristics without a vision of the whole. The curriculum should promote an approximation between basic scientific concepts and clinical practice. This type of curricular model has already been applied in some countries (15-17).

Some changes are necessary to the quality of dental professors so that the professors, in educating future dentists, may contribute toward producing new professionals who are more humane and are able to contribute to more democratic and egalitarian social developments (18). The professors themselves are mentioned as one of the greatest obstacles standing in the way of the implementation of the DCN. This fact is observed in the following statement extracted from Carli (19), which discusses professors and coordinators of programs in Dentistry: "The professor is the great problem. We currently have an advanced curriculum that is being implemented by professors who are not following the evolution of the curriculum. So unexpectedly, professors go to the clinics (General Dentistry) and teach each other".

With regard to adherence to the recommendations on the part of the programs, 6 (42.85%) respondents did not agree with the concepts provided by the Ministry of Education (MEC) and thought that the final recommendations in 2003-2004 did not reflect the reality of their programs. As one participant said, "I disagree with the opinions presented by the evaluators in their reports. The impression I had was that they used a method by which they compared institutions with their own institutions of origin or with the institutions that they believed were ideal. The time designated for the evaluation in loco was short, and the quality of the documents and information that they highlighted during the visit certainly exceeded the evaluators' abilities to evaluate".

The ENADE aims to assess students' productivity in their undergraduate programs (20). Four (28.57%) professors/coordinators reported that their institutions provided students with preparation for the ENADE test, such as theoretical classes and preparatory seminars for the interpretation and discussion of previous exams. In addition, they explained that there are fundamental activities that stimulate students, as well as the implementation of all suggestions (written and practical exams), during the dental course in the ENADE model, with the concepts addressed on the ENADE as a reference.

However, some interviewers disagreed: "... in my opinion, all the proposed pedagogical actions or the improvement of physical conditions would reverse the improvements at the education level. I do not find it interesting that the institution works only for the purpose of improving performance on the ENADE, even if there are various factors that influence in this performance as, for instance, students' interest in participating".

When we asked about students' positioning in relation to the ENADE test, most of the responses demonstrated expressions of discouragement. According to a professor, "This generation consists of disrespectful and unwilling young people using any regulatory mechanism". Previous studies (21-23) have analyzed students' self-images. A few students have reported that patients were more concerned with technical aspects and professional status.

In relation to the students' quality, changes have been expected, stimulated by the DCN, e.g., changes in the surgical repair paradigm, focusing only on disease, and on invasive practices, toward a global paradigm of health promotion (24). The synthesis report of the 2007 ENADE for programs in Dentistry (25), published by MEC, has already shown a tendency toward changing the dental students' quality.

Ten (71.42%) participants thought that the ENADE test, when evaluating not only freshmen but also senior students, might reflect a change in the students' quality, as shown in the following passage: "I believe that the comparison is important for analyzing the levels of philosophical, technical and scientific learning".

Conclusions

Changes to improve the quality of undergraduate programs in Dentistry have been stimulated by the results of the evaluation process. These changes occurred, not only because of the suggestions made as a result the evaluation, but also because of the curricular reform needed to understand the DCN.

A change in the quality of Brazilian Dentistry, i.e., leaving the traditional technical-scientific model for a newer model that values humanization, will only be possible if the students' quality is changed, in addition to the professors'.

The interviewers have already incorporated into their programs a methodology similar to the one proved effective by the ENADE, i.e., seeking to develop content through process and not only to improve their programs. Despite this new methodology, students have not shown interest in relation to the ENADE.

The expectation of professors/coordinators is that changes may occur in the students' quality, and that these changes may be reflected on the ENADE.

5. Available from: http://www.scielo.br/pdf/aval/v12n4/a04v12n4.pdf [Article in Portuguese].

Received: March 18, 2011

Accepted: May 12, 2011

Conflict of Interest Statement: The authors state that there are no financial and personal conflicts of interest that could have inappropriately influenced their work.

• ¹
  SINAES. Sistema Nacional de Avaliação da Educação Superior. Bases para uma nova proposta de avaliação da educação superior. [Internet]. Brasília, DF: INEP. 2003 - [Cited 2010 Jul 9]. Available from: http://www.unifesp.br/reitoria/orgaos/comissoes/avaliacao/sinaes.pdf
• 2. Ioannidis JP, Patsopoulos NA, Kavvoura FK, Tatsioni A, Evangelou E, Kouri et al. International ranking systems for universities and institutions: a critical appraisal. BMC Med 2007;25;5:30.
• ³
  SJTU. Academic ranking of World Universities. [Internet]. Shanghai, China: Shanghai Jiao Tong University, Institute of Higher Education. 2010 - [Cited 2010 Jul 9]. Available from: http://ed.sjtu.edu.cn/ranking.htm http://ed
• 4. Augusto R, Balzan NC. The voice of the Campinas Higher Education institutions' internal evaluation commission. Avaliação. [Internet]. 2007 - [Cited 2010 Jul 9]; 12:597-622.
• 6. Schlickmann R, Melo PA, Alperstedt GD. Focus of the institutional theory on the Br azilian institutional evaluation models. Avaliação. [Internet]. 2008 - [Cited 2010 Jul 9]; 13:153-168. Available from: http://www.scielo.br/pdf/aval/v13n1/a09v13n1.pdf [Article in Portuguese]
• 7. CONAES. Comissão Nacional de avaliação da Educação Superior. Diretrizes para a avaliação das instituições de educação superior. [Internet]. Brasília, DF: Ministério da Educação. 2004 - [Cited 2010 Jul 9]. Available from: http://www.avaliacao.ufrn.br/files/documentos/diretrizes_conaes_1_.pdf [in Portuguese]
• 8. Conselho Nacional de Educação. Diretrizes Curriculares Nacionais. [Internet]. Brasil: Diário Oficial Mar. 4;42 Seção 1:10-1. 2002 - [Cited 2010 Jul 9]. Available from: http://portal.mec.gov.br/cne/arquivos/pdf/CES032002.pdf
• 9. Lemos CLS. A implantação das diretrizes curriculares dos cursos de graduação em odontologia no Brasil: algumas reflexões. Rev ABENO 2005 Jan/Jun - [Cited 2010 Jul 9]; 5:80-5. Available from: http://adm.abeno.org.br/antigo/revista/arquivos_pdf/2005/Abeno_5-1.pdf [Article in Portuguese]
• 10. CFO. Conselho Federal de Odontologia. [Internet]. DF, Brasil: CFO. 2008 - [Cited 2010 Jun 30]. Available from: http://www.cfo.org.br/
  » link
• 11. Bjorn G. Dissension in the ranks. Nature 2008;454:1024-5.
• 12. Ghabili K, Shoja MM, Ghabili S.. The neglected role of students in international university rankings. Nephrol Dial Transplant 2008;23:1774-5. Epub 2008 Jan 26.
• 13. Secco LG, Pereira MLT. A profissionalização docente e os desafios político-estruturais dos formadores em Odontologia. Rev ABENO 2004 - [Cited 2010 Jul 9]; 4:22-8. Available from: http://www.abeno.org.br/index.php?option=com_content&view=article&id=85:revistas-abeno-em-pdf&catid=40:revista-da-abeno-online&Itemid=59f [Article in Portuguese]
• 14. Dederich DN, Lloyd PM, Dixon-Farmer C, Geurink KV, Nadershahi NA, Robinson FG et al. Perceptions of dental schools from within and outside the university. J Dent Educ 2004;68:1163-71.
• 15. Hillenburg KL, Cederberg RA, Gray SA, Hurst CL, Johnson GK, Potter BJ. E-learning and the future of dental education: opinions of administrators and information technology specialists. Eur J Dent Educ 2006;10:169-77.
• 16. Chiang PS. Problem-based learning definition, implementation and opportunities for collaboration. NY State Dent J 2004;70:44-6.
• 17. Pérez-Rico C, Montes-Mollón MA. The new role of the professor and student in the European Space of Higher Arch Soc Esp Oftalmol 2007;82:265.
• 18. Iacopino AM. The influence of "new science" on dental education: current concepts, trends, and models for the future. J Dent Educ 2007;71:450-62.
• 19. Peret ACA. As políticas públicas em educação superior e saúde e a formação do professor de odontologia numa dimensão crítica [Thesis]. Belo Horizonte: Universidade Federal de Minas Gerais; 2005. Available from: http://webpergamum.admserv.ufmg.br/pergamum/biblioteca/index.php?resolution2 =800#posicao_dados_acervo [in Portuguese]
• 20. Carli G de. Formação docente e educação odontológica: uma perspectiva em construção. [Dissertation]. Canoas: Universidade Luterana do Brasil; 2007. Available from: javascript:open_window(https://memphis.ulbranet.com.br/ALEPH/QIE49SKBIGA7I RDM74XEQSFLSBC7I27US11D3PS9GRFB95GNU9 00263?func=service&doc_library=ULB01&doc_number=000394360&line_number= 0001&func_code=WEBBRIEF&service_type=MEDIA at 09 July 2010 [in Portuguese]
• 21. Avaliação dos cursos de graduação. [Internet]. Brasil: INEP, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. 2008a - [Cited 2010 Jul 9]. Available from: http://www.inep.gov.br/superior/condicoesdeensino/
• 22. Pacca S, Corrêa L, Motta M. Auto-imagem do cirurgião-dentista: um estudo baseado em desenhos de alunos de graduação. Rev. ABENO 2003 - [Cited 2010 Jul 9];3:82-5. Available from: http://www.abeno.org.br/revista/arquivos_pdf/2003/pacc.pdf [Article in Portuguese]
• 23. Chambers DW. The role of dentists in Dentistry. J Dent Educ 2001;65:1430-40.
• 24. Skelly AM, Fleming GJ. Perceptions of a dental career among successful applicants for Dentistry compared with those of fifth-year dental students. Prim Dent Care 2002; 9:41-6.
• 25. Shanley DB. Convergence towards higher standards in international dental education. N Y State Dent J 2004;70:35-9.
• 26. Relatório Síntese Odontologia. [Internet]. Brasil: INEP, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. 2008b - [Cited 2010 Jul 9]. Available from: http://www.inep.gov.br/download/enade/2007/relatorio_sintese/2007_REL_ SINT_ODONTOLOGIA.pdf at 09 July 2010.

Publication Dates

• Publication in this collection
  02 Sept 2011
• Date of issue
  2011

History

• Received
  18 Mar 2011
• Accepted
  12 May 2011