Exploring Data Analysis Techniques in Undergraduate Student Evaluation Exams in the Brazilian Context

Explorando Técnicas de Análisis de Datos en Exámenes de Evaluación de Estudiantes de Grado en el Contexto Brasileño

Abstract

Accreditation and continuous assessments are crucial for ensuring quality and standards in higher education. In Brazil, the federal government also conducts an annual student assessment called Enade. This paper presents a scoping review that identifies and discusses the techniques employed in analyzing Enade data and implementing diagnostic actions to monitor the necessary competencies of graduates. The research encompassed 32 articles covering machine learning (ML), statistical techniques, and system development dedicated to the Enade exam. ML articles primarily focused on analyzing factors that impact student scores, utilizing classification and clustering approaches. Descriptive statistics emerged as the most commonly used technique in articles focusing on statistical techniques. The identified systems primarily revolved around exam administration and result analysis, with only one article exploring the implementation of gamification.

Keywords: accreditation assessments; Enade; scoping review; statistical techniques; machine learning techniques

Resumen

Acreditación y evaluaciones continuas son fundamentales para garantizar la calidad y los estándares en la educación superior. En Brasil, el gobierno federal también lleva a cabo una evaluación anual de los estudiantes llamada Enade. Este artículo presenta una revisión de alcance que identifica y discute las técnicas utilizadas en el análisis de los datos de Enade y en la implementación de acciones de diagnóstico para monitorear las competencias necesarias de los graduados. La investigación abarcó 32 artículos que abordan el aprendizaje automático (ML), técnicas estadísticas y el desarrollo de sistemas. Los artículos de ML se centraron principalmente en el análisis de los factores que afectan las calificaciones de los estudiantes, utilizando enfoques de clasificación y agrupamiento. Las estadísticas descriptivas surgieron como la técnica más comúnmente utilizada en los artículos que se centran en técnicas estadísticas. Los sistemas identificados se centraron principalmente en la administración de exámenes y el análisis de resultados, con solo un artículo explorando la implementación de la gamificación.

Palabras clave: acreditación; Enade; evaluaciones; revisión de alcance; técnicas estadísticas; técnicas de aprendizaje automático

Resumo

Acreditação e avaliações contínuas são cruciais para garantir a qualidade e os padrões no ensino superior. No Brasil, o governo federal também realiza uma avaliação anual dos estudantes chamada Enade. Este artigo apresenta uma revisão de abrangência que identifica e discute as técnicas empregadas na análise dos dados do Enade e na implementação de ações de diagnóstico para monitorar as competências necessárias dos formandos. A pesquisa abrangeu 32 artigos que cobrem aprendizado de máquina (ML), técnicas estatísticas e desenvolvimento de sistemas com foco no Enade. Os artigos de ML se concentraram principalmente na análise dos fatores que afetam as notas dos estudantes, utilizando abordagens de classificação e agrupamento. Estatísticas descritivas surgiram como a técnica mais comumente utilizada nos artigos que se concentraram em técnicas estatísticas. Os sistemas identificados se dedicam principalmente na administração de exames e na análise de resultados, com apenas um artigo explorando a implementação de gamificação.

Palavras-chave: acreditação; Enade; avaliações; revisão de escopo; técnicas estatísticas; técnicas de aprendizado de máquina

1 Introduction

Accreditation in higher education plays a crucial role in ensuring the quality, accountability, and continuous improvement of educational institutions worldwide. It serves as a means of assuring various stakeholders, including students, parents, employers, and the public, that a particular institution meets recognized standards of excellence across a range of critical aspects, including its mission, programs, support services, faculty, and facilities. There are various models and experiences around the world for this type of assessment.

In the United States, for example, accreditation is carried out by private, nongovernmental organizations specifically established for the purpose of evaluating higher education institutions (HEIs) and their programs to ensure quality. These accrediting bodies are recognized by authoritative entities such as the Council for Higher Education Accreditation (CHEA) and the United States Department of Education (USDE).

In Europe, Poland follows a similar path, where university accreditation reports are published by the accreditation agency Polska Komisja Akredytacyjna (PKA) (Rybinski, 2020), which is an accredited member of the European Association for Quality Assurance in Higher Education (ENQA) and adheres to European standards and guidelines (ENQA, 2015). In the United Kingdom, the independent body responsible for verifying standards and quality in higher education institutions is the Quality Assurance Agency (QAA). The QAA introduced the Quality Code for Higher Education, which became effective starting in the academic year 2011-2012 (Rybinski, 2022). These agencies serve as watchdogs, conducting assessments of colleges and universities to ensure they are in compliance with established standards and are providing high-quality education.

In Brazil, the evaluation and accreditation of universities, faculties, and courses in Higher Education Institutions (HEIs) fall under the purview of the National Institute for Educational Studies and Research Anisio Teixeira (Inep). Inep is a federal agency that operates under the Ministry of Education (MEC) and is responsible for overseeing this crucial process. An integral part of this evaluation is the National Student Performance Exam (Enade), which distinguishes Brazil's evaluation system.

In the year 2003, during the government of Luiz Inácio Lula da Silva, in response to criticisms directed at the former "Provão," the higher education evaluation system in Brazil underwent significant transformations (Dias Sobrinho, 2010). These changes were guided by two fundamental goals of the academic community: promoting democratic participation and establishing a more comprehensive evaluation system. As a result, the National Higher Education Assessment System (SINAES) was conceived and subsequently implemented in 2004. This initiative was designed as a state policy, aiming to encompass all higher education institutions, whether public or private, while simultaneously respecting their diverse and distinct identities (Dias Sobrinho, 2010). The Enade is an integral part of SINAES.

Enade is an annual assessment that rigorously scrutinizes higher education courses in Brazil, encompassing both in-person and distance learning programs. This assessment has been in existence and managed by Inep since 2004 (Inep, 2023). Enade functions by evaluating students' performance based on the curriculum content outlined in the National Curriculum Guidelines (DCN) relevant to their specific undergraduate programs, as established by the Brazilian government in 2004 (Brasil, 2004). This comprehensive evaluation is conducted periodically, with no more than a three-year gap between assessments, and it is a mandatory component of the undergraduate curriculum.

Enade employs a range of assessment tools, including questionnaires and examinations, to gauge students' academic performance. The results obtained from these assessments provide valuable insights for educators, students, and administrators (Lima et al., 2018). These findings allow for the refinement of teaching practices and course curricula, ultimately leading to improvements in the overall quality of education. By providing a systematic and data-driven evaluation, Enade plays a pivotal role in enhancing the educational landscape in Brazil.

In this paper, we present a scoping review (SR) (Ralph; Baltes, 2022), which aims at identifying and discussing the methods used to analyze Enade data. Our focus was on the microdata provided by Inep, and on works that implement diagnostic actions to track the competencies needed by the undergraduate student who takes the exam. This SR aims to benefit HEI and researchers who intend to conduct studies using Enade data or learn about tools proposed by the scientific community for monitoring students who take the exam. After searching the databases, we found a total of 129 studies. Applying the selection criteria, we selected 32 studies described in this paper in detail.

The evaluation of undergraduate education in Brazil, as in many other countries, plays a fundamental role in the pursuit of improving educational quality. This study is particularly relevant in educational research as it focuses on the critical analysis of the methods used to assess students' competencies and skills across various fields of knowledge. This evaluation encompasses not only students' performance but also the effectiveness of courses and higher education institutions (HEIs) as a whole.

Through an analysis of these methods, it becomes possible to identify both the strengths and areas in need of improvement in courses and HEIs. These findings provide valuable insights that can be used to enhance curricula, teaching practices, and assessment strategies. This, in turn, contributes to elevating the quality of education, ensuring that students are acquiring the competencies and skills necessary to face the challenges of an ever-evolving job market and society.

Other research studies, such as Lima et al. (2019), have conducted a systematic literature review with the aim of identifying how the data collected by the Enade and Enem assessments are used, highlighting the need for more in-depth analyses beyond the descriptive statistics provided by the National Institute for Educational Studies and Research Anísio Teixeira (Inep). However, Lima et al. (2019) found articles published between 2013 and 2016, and the research points to the necessity of exploring other types of analysis, such as data mining, to make the most of the vast volume of available data and gain a better understanding of student profiles and the Brazilian education system, enabling preventive actions. This work aims to delve further into this subject.

The remainder of this paper is organized as follows: Section 2 presents the SR methodology, Section 3 provides the SR results regarding the included studies and addressing the research questions. In Section 4, the findings are discussed in detail, and finally, Section 5 offers the concluding remarks and future work.

2 Method

2.1 Protocol

We based the protocol for this scoping review (SR) on Keele at al. (2007). The guidelines suggest three main phases: planning, conducting, and reporting the review. In the first planning phase, we intended to identify the study's needs to define its research questions (RQ). Inep provides datasets containing detailed information about the Enade, such as students' responses to the questions, student's answers to the student questionnaire, institution data, and other relevant information. Our research seeks to comprehend how this Enade data is analyzed and to identify the computational and statistical tools researchers employ in these studies. Additionally, we have focused on identifying educational resources that aim to prepare students for the Enade assessment.

After identifying these aspects, we defined the research protocol, encompassing the mapping objectives, research databases, search string, and selection criteria. The protocol underwent an incremental construction process and was reviewed by all authors involved in this research.

We executed the defined protocol in the second phase. Firstly, we conducted searches in the databases using the search string. Then, we read only the abstracts of the papers and evaluated them based on the selection criteria. With a reduced number, we performed a comprehensive reading of the documents, reviewing them against the selection criteria. Following this step, we selected the final articles and completed a full reading of them. Subsequently, we created a shared spreadsheet that included the title, document type, year, and detailed research questions. This allowed all authors to monitor the completion of all items effectively.

Finally, in the third phase, we analyzed the results by generating graphs and tables based on the data extraction form. Subsequently, we wrote the report, presented in this paper, and incrementally validated it.

2.2 Research questions

The research questions (RQ) to guide this work were defined using the PICO strategy (Population Intervention Control Outcomes) (Stone, 2002), which helped structure this mapping and select relevant RQ. Figure 1 presents the process of constructing the RQ. The population of our study is composed of undergraduate students and higher education institutions (HEI). Software and data analysis methods related to the Enade belong to the intervention. Enade microdata and studies conducted with students are the control. Finally, the algorithms, statistical solutions, and developed software focus on Enade assessment are the outcomes.

Figure 1
RQ build process and search string.

The main research question of our study is: What are the software, statistical techniques, and machine learning methods commonly used to analyze Enade data from HEIs?

2.3 Search string

Based on the definition of the PICO elements for this research, we determined the keywords composed of three parts: (I) Enade; (II) Assessments; and (III) Computer and Statistical Techniques. We used synonyms and subtopics of them to enlarge the search string. The final search string used in the databases was: (“High education” OR “Large-scale assessment” OR “Higher education assessment” OR “Item Response Theory” OR “Classical Test Theory” OR “Knowledge assessment”) AND “Enade” AND (“Data mining” OR “Machine Learning” OR “Data analysis” OR “Algorithm” OR “Analysis tool” OR “Learning analytics”)¹.

After defining the search string, we conducted tests on Google Scholar² to assess the relevance of the results, allowing the authors to refine the string if necessary. The databases used for the automated search in this research were Google Scholar and Periodicals Capes³. The choice of these databases is related to this research topic, which is focused on a specific assessment by the Brazilian MEC. We conducted the search between March 15th and 18th, 2022.Table 1 presents the six derived research questions.

Table 1
Research Questions

Description
RQ1 - Which areas of Enade were the primary focus of the research?
RQ2 - What were the studies' approaches and goals?
RQ3 - Which statistical techniques were employed in the data analysis?
RQ4 - What were the data mining or algorithmic techniques utilized in the studies?
RQ5 - What were the key characteristics of the systems focused on Enade preparation?
RQ6 - What are the benefits and limitations of those systems highlighted by the authors of the papers?

Source: Own elaboration

2.4 Study Selection and Data Extraction

The inclusion criteria (IC) used in this research were: IC1- articles presenting algorithms or machine learning techniques for conducting investigations on Enade; IC2- papers using statistical techniques for conducting investigations on Enade; and IC3- documents proposing Web or mobile systems using Enade questions to prepare students or simulate the assessments.

The exclusion criteria (EC) were: EC1- studies published more than ten years ago (before 2012); EC2- articles that do not have Enade as the central object of study; EC3- studies not written in Portuguese or English; EC4- studies that are short papers (less than two pages); EC5- preliminary studies; EC6- unavailable studies; EC7- duplicate studies; and EC8- secondary studies.

To perform the data extraction from the articles, we chose to use a shared Google Sheet⁴ with all authors. The spreadsheet contains a list of articles in its rows and the answers to essential questions for this research in its columns. These questions include, among others, the method used in the study, the source of the data used, the areas in which the study was applied, and the techniques employed to analyze the Enade data. The results of this analysis are presented in the next section.

3 Results

After defining the SR protocol, we conducted the research. The database searches yielded 129 studies, of which 32 met the selection criteria outlined in the protocol and were included in this research. The remaining 97 studies were excluded.

Among the 32 studies, 14 met EC1 criterion, 14 met EC2, and 4 met EC3. It is worth noting that the number of studies included in this research is lower than the total number of studies found by Lima et al. (2018) related to this topic. This difference could be attributed to the research timeframe and the search string's structure. Lima et al. (2018) examined studies from 2005 to 2016, whereas our study focused on 2012 to 2022.

Regarding the distribution over the years, we found 1 study from 2022, 8 studies from 2021, 7 studies from 2020, 3 studies from 2019, 5 studies from 2018, 3 studies from 2017, 3 studies from 2016, 1 study from 2014, and 1 study from 2013. It is important to note that the number of studies from 2022 is small due to the timeframe of the search conducted.

3.1 RQ1 - Which areas of Enade were the primary focus of the research?

Figure 2 presents the knowledge areas on which the studies were focused and the number of studies in each knowledge area. In the field of exact and earth sciences, twelve studies were found: Silva et al. (2017), Piton-Goncalves (2020), Fonseca et al. (2013); and the papers Silva et al. (2019), Vista et al. (2017), Lima et al. (2021), Landes e Magalhães (2018), Figueiredo et al. (2018), Rosa et al. (2021), Lima et al. (2018), Charão et al. (2020), e Cunha et al. (2021, which are specific to the field of computer science.

Figure 2
Total articles by area of knowledge.

In our review, we identified two studies within the field of Humanities, those by Medeiros Filho (2019) and Silva et al. (2017). Similarly, in the field of Engineering, we came across two studies: one by Melguizo and Wainer (2016) and another by Nunes (2018). However, in the fields of Biological Sciences, Health Sciences, and Applied Social Sciences, we found only one study in each. The study in Biological Sciences was conducted by Alves (2020), while Silva et al. (2020) contributed to Health Sciences, and Camargo et al. (2016) to Applied Social Science.

Nine studies did not focus on a specific field. Some studies merged multiple areas, such as Melguizo and Wainer (2016), which combined mathematics, social sciences, and biological sciences courses, and Coelho (2014), which combined chemical engineering, administration, and statistics courses. Other studies used data from all undergraduate programs in a single state (Martins, 2021) or from a specific region such as Matos et al. (2020), which used data from all courses in the Northern Region of Brazil. The remaining studies presented data from all undergraduate programs in a specific year (Casitaghi, 2021; Araújo, 2019), while others covered a particular period (Rodrigues, 2021; ARAUJO, 2020; Leão, 2018).

The studies by Barbosa and Carvalho (2020), França et al. (2021), Ribeiro and C. Junior (2022), and Santos et al. (2016) did not utilize Enade data directly. Those authors propose instead information systems specifically focused on Enade preparation (i.e., Web simulators, exercises similar to Enade questions).

3.2 RQ2 - What were the studies' approaches and goals?

The studies included in this research used mainly three approaches. Two of them focused on analyzing Enade data: the first group uses machine learning techniques, as depicted in Figure 4; and the second approach applies statistical methods, as presented in Figure 3. The third approach in this SR involved the development of systems focused on Enade preparation. The main characteristics of these systems are outlined in Table 2. Some studies combined the first and second approaches, often utilizing descriptive statistics with some machine learning methods (Silva et al., 2020; Vista et al., 2017).

Table 2
Computational systems features.

Features	Barbosa and Carvalho (2020)	Ribeiro and Costa Junior (2020)	Lima et al. (2018)	França et al. (2021)	Santos et al. (2016)	Araújo et al. (2019)
Questions	X	X	X	X	X
Dashboard	X	X		X	X
Gamification				X
Machine learning						X
Microdata			X			X
Desktop			X		X
Mobile		X
Web	X	X		X		X
Test Online	X	X		X	X

Source: Own elaboration.

Figure 3
Statistical techniques by goal

Figure 4
Machine learning techniques by objective

Regarding statistical techniques, we noted the included studies have more heterogeneous objectives (Figure 3), even those using the same statistical method. Thirteen objectives were identified. The most recurring goal among these studies was to describe the Enade dataset using descriptive statistics.

Regarding the studies that utilized machine learning techniques, we identified five objectives (Figure 4). The most recurring goal was to identify factors associated with scores, and for this purpose, most of the algorithms used were classification algorithms. Some studies focused on clustering the data to assess the performance of HEI according to the Enade Concept (EC)⁵ (INEP, 2020). In contrast, others aimed to investigate patterns among the Student Questionnaire (SQ) questions.

3.3 RQ3 - Which statistical techniques were employed in the data analysis?

Figure 3 presents the statistical techniques identified in the studies included in this research. A large portion of the studies made use of descriptive statistics (DS) (Casitaghi & Aragão, 2021; Charão et al., 2020; Landes & Manhães, 2018; Lima et al., 2021; Lima et al., 2018; Medeiros Filho, 2019; Silva et al., 2020). Some of them used DS to describe the Enade microdata (Landes & Manhães, 2018; Lima et al., 2021; Vista et al., 2017), others to describe questions (Lima et al., 2018), others to discover factors that impact student scores (Casitaghi & Aragão, 2021; Medeiros Filho, 2019), others to improve the course pedagogical project (CPP) based on the data description (Charão et al., 2020), as well as to make comparisons using the Enade concept (EC) (Silva et al., 2020).

Item Response Theory (IRT)⁶ (Camargo et al., 2016) was also present in some analyses (Alves et al., 2020; Coelho, 2014; Fonseca et al., 2013; Piton-Goncalves, 2020). The authors applied it for measuring student performance, with all studies utilizing the Three-Parameter Logistic Model (3PL). Other techniques used include entropy balancing (Araujo et al., 2020), confirmatory factor analysis (Santos et al., 2017), Cohen's d (Melguizo & Wainer, 2016), Cronbach's alpha, Mann-Whitney U, and Kruskal-Wallis tests (Medeiros Filho, 2019).

3.4 RQ4 - What were the data mining or algorithmic techniques utilized in the studies?

Figure 4 presents the studies using data mining or algorithmic techniques. It displays the objective, the technique, and the number of studies using a specific method for a particular goal. 14 studies of our SR used machine learning algorithms (Araújo et al., 2019; Cunha et al., 2021; Figueiro et al., 2017; Leão et al., 2018; Martins et al., 2021; Matos et al., 2020; Nunes, 2018; Rodrigues & Gouveia, 2021; Rosa et al., 2021; Silva et al., 2020; Silva et al., 2021; L. Silva et al., 2017; Vista et al., 2017). The studies of Silva et al. (2020) and Vista et al. (2017) employed statistical techniques too.

A significant portion of the algorithms (68.18%) aimed at classification, and the most commonly employed technique was Decision Tree, which was implemented in 5 studies (Araújo et al., 2019; Martins et al., 2021; Rodrigues and Gouveia, 2021; Rosa et al., 2021; Silva et al., 2021).

Martins et al. (2021) used C4.5 algorithm to generate decision rules and predict an increase in EC in subsequent years of Enade based on these rules. Hierarchical clustering was employed by both Vista et al. (2017) and Figueiro et al. (2017) to group institutions with excellent and poor ECs. Nunes (2018) utilized Enade microdata for training an artificial neural network (ANN) capable of predicting the EC of HEIs. Additionally, Silva et al. (2020) employed K-means clustering to group the data based on EC and identify which administrative categories and regions of HEIs have lower and higher EC.

Six studies examined the factors associated with student performance in Enade. Rosa et al. (2021) conducted a study to determine which algorithm-Decision Tree, Random Forest, or Support Vector Machine (SVM)- provided a better classification of the factors influencing student performance. They determined these factors using socioeconomic data. In the authors' studied scenario, the C4.5 algorithm achieved the highest accuracy (71.54%). On the other hand, Silva et al. (2021) employed Decision Tree, Random Forest, Gradient Boosting Tree, and Naive Bayes algorithms, and Naive Bayes performed the best in classifying factors associated with grades, achieving an accuracy of 98.21%. Silva et al. (2021) utilized selected questions from the Student Questionnaire (SQ) as the factors.

With the same purpose of investigating the characteristics related to student performance in Enade, Silva et al (2019) used the Apriori algorithm and presented ten association rules with a confidence level greater than 80% with a lift greater than three. For this purpose, the authors used ten questions from the SQ. Similarly, Leão et al. (2018) employed association rules and identified 30 rules based on the SQ. The authors found that attributes Q46, Q52, Q60, Q61, and Q62⁷ had the highest occurrence in the rules. Based on this, they concluded that these attributes should be focused on for further development in the Higher Education Institution (HEI) to improve the quality of education and enhance student satisfaction. On the other hand, Silva et al. (2017) used linear regression to identify variables associated with student grades. The authors identified 13 out of 31 selected variables that impacted student grades.

Regarding student grades, Araújo et al. (2019) used decision trees to predict student performance. The classification model that achieved the highest accuracy in their study was CART, which reached an accuracy of 79.78% using 70% of the instances for training and 30% for testing. To investigate the relationships between the questions in the SQ, Matos et al. (2020) employed the Apriori algorithm and successfully identified 20 associations for the year 2014, 72 associations for 2015, and 84 for 2016. It was possible to verify, for example, that 79% of the students have the same profile, consisting of Brazilians who studied their entire high school years in public schools and did not participate in academic programs abroad Matos et al. (2020). These discovered rules exhibited a confidence level above 75%.

Rodrigues and Gouveia (2021) examined models considering factors related to students' length of stay. The study found that XGBoost models consistently performed well across all scenarios. The analysis used 30 attributes from the Enade microdata in a classification model. The difference between the year of high school completion and the year of the student's graduation start was calculated to determine the length of stay.

3.5 RQ5 - What were the key characteristics of the systems focused on Enade preparation?

Table 2 presents the main characteristics of these systems. Two systems are web platforms Barbosa and Carvalho (2020) and França et al. (2021), one study focused on a desktop system (LIMA et al., 2018), and the work by Ribeiro and Costa Junior (2020) was designed for both web and mobile platforms.

The works Barbosa and Carvalho (2020), Ribeiro and Costa Junior (2020), Lima et al. (2018), França et al. (2021) and Santos et al. (2016) proposed datasets of Enade questions. The intention is to use this database of questions to propose mock exams and preparation exercises for students to practice before the Enade tests.

In Barbosa and Carvalho (2020) and Ribeiro and Costa Junior (2020), for example, teachers or system administrators carry out the registration of questions. In Lima et al. (2018), those managing the platform register the questions; in França et al. (2021), teachers and students can register questions. However, if a student registers a question, it requires validation by a teacher. In the works of Ribeiro and Costa Junior (2020), Barbosa and Carvalho (2020), the questions are associated with Enade competencies and contents. In the work of Lima et al. (2018), the questions are classified by themes associated with word dictionaries. Santos et al. (2016) proposed a desktop system that stored a database of questions entered by teachers and generated the test in the format of Enade.

In the works of Barbosa and Carvalho (2020), Ribeiro and Costa Junior (2020) and França et al. (2021), the organization of questions into practice exams is possible, and students can provide their responses within the system. Performance reports are generated for both teachers and students. Barbosa and Carvalho (2020), Ribeiro and Costa Junior (2020) and França et al. (2021) offer dashboards for students to present their performance in a specific assessment, while teachers can assess the class performance through indicators. We did not find in the work of Lima et al. (2018) the use of dashboards.

We identified gamification elements in França et al. (2021). In this platform, each quiz is presented as a game board with a path to be followed, and the board contains different difficulty levels in its spaces.

The study by Lima et al. (2018) Inep's microdata to create a system. This system enables users to select and process specific microdata, automatically categorizes exam questions into predefined themes using dictionaries, calculates individual student scores based on these themes, and performs statistical analyses. The system generates reports as the final output. Araújo et al. (2019) introduced an automated exploratory analysis tool for Enade microdata, which, along with a classification model, could predict student performance and compare courses and HEI.

3.6 RQ6 - What are the benefits and limitations of those systems highlighted by the authors of the papers?

Barbosa and Carvalho (2020) and Lima et al. (2018) identified the system's benefits as the ease of diagnosing student competencies and content. Ribeiro and Costa Junior (2022) indicated that the system could support the teaching-learning process and evaluation in both virtual and face-to-face environments. Additionally, França et al. (2021) highlighted the benefits of using gamification, which enhanced the tool's interactivity.

The limitations identified are as follows: Barbosa and Carvalho (2020) emphasized the lack of regular simulated assessments for diagnosing students and the need for student training in answering Enade-format questions. Ribeiro and Costa Junior (2022) mentioned the absence of a help interface to assist users in understanding the system. Lima et al. (2018) highlighted the challenge of generalizing their proposal to courses other than the one they studied (Computer Science). Finally, França et al. (2021) noted that their tool lacked machine learning strategies, which could have improved its performance.

4 Discussion

This section presents a discussion related to the 32 studies included in this research.

4.1 The use of statistical techniques with Enade data

Figure 4 shows that the most common statistical techniques used are descriptive statistics (CASITAGHI and ARAGÃO, 2021; CHARÃO et al., 2020; LANDES & MANHÃES, 2018; LIMA et al., 2021; SILVA et al., 2020); and logistic regression (IRT) (ALVES et al., 2020; CAMARGO et al., 2016a; COELHO, 2014; FONSECA et al., 2013; PITONGONCALVES, 2020).

This scoping review (SR) highlights the key role of statistical analyses in recent studies on Enade data. These analyses help describe the data and provide clearer insights from the available microdata. They also address important teaching concerns, like improving the evaluation process and enhancing program curricula. The results from these analyses contribute to valuable discussions on improving the quality of courses offered by educational institutions, as noted by Charão et al. (2020) and Lima et al. (2018).

Based on the analyses of the studies, statistical analyses have revealed insights into the characteristics of Enade test-takers that are not included in Inep's official report. Microdata was the key resource that enabled researchers to uncover these valuable insights. Studies such as Lima et al. (2018) classified questions into knowledge areas using microdata Landes and Manhães (2018) identified score trends among students across courses to assess educational quality, and Charão et al. (2020) conducted a detailed exploration of results by question in Enade exams with the aim of gathering insights for the Pedagogical Project review. These studies showcase how leveraging microdata enhances researchers' understanding of various aspects related to Enade test-takers.

Furthermore, one study employed Item Response Theory (IRT) to examine the psychometric properties of the test; two studies utilized IRT to evaluate the feasibility of adaptive testing; and another study examined item characteristics and scores. Two studies advanced by combining statistical techniques with machine learning methods, as highlighted by Silva et al. (2020) and Vista et al. (2017).

4.2 The use of machine learning with Enade data

In contrast to Lima et al. (2019), which identified only 2 studies that utilized machine learning techniques, this study presented data from 14 studies that employed these methods. The decision tree was the most commonly used technique, employed in 35% of the studies. Furthermore, 71% of the studies aimed to investigate factors associated with the test scores.

The results of this systematic review indicate that using machine learning techniques allows for the identification of information in Inep's microdata that can influence student performance, both positively and negatively. Moreover, these techniques enable the analysis of students' social groups based on indicators such as the administrative dependency of higher education institutions (HEIs). Institutions can use this data to enhance their teaching and learning processes.

Additionally, the use of clustering techniques in some studies made it possible to identify similarities among the analyzed instances and investigate the factors influencing performance, whether positive or negative. This analysis provides support for the development of more targeted educational interventions.

4.3 Proposed Systems with a Focus on Enade in Computing

The research has uncovered a limited number of system proposals dedicated to Enade, with a total of six papers focused on this area. These proposals vary in their approaches, primarily centering on the use of questionnaires featuring Enade-related questions and the analysis of Inep microdata. Notably, only one of these systems integrated gamification into its design.

These systems generate performance reports for both individual students and entire classes. This data could offer valuable feedback for educators and institutions, indicating content where students may need additional support. Only one study, Barbosa and Carvalho (2020), demonstrated some type of impact of usage and evaluation on students' performance throughout their undergraduate courses. More research is required to identify the advantages and disadvantages of adopting this type of system during the undergraduate program and its impact on course performance in Enade.

However, there is room for the evolution of these systems to provide more comprehensive assessments of competencies. Incorporating models that can track students' knowledge and demonstrate their progress would be a substantial step forward. By doing so, these systems could not only provide feedback but also allow for timely and targeted interventions in the teaching and learning process, in line with the proposals by França et al. (2021).

One of the system proposals stands out by introducing gamification elements, which has shown promise in enhancing student motivation to improve their competencies (França, 2021). While the initial results are encouraging, the effectiveness of gamification in this context warrants further investigation. Several studies have suggested potential benefits of gamification in educational settings (Van Elderen and Van Der Stappen, 2019), and exploring its impact within the Enade context can provide valuable insights into the role of gamification in improving student engagement and competencies.

Recent studies on gamification, such as those by Salman et al. (2024), reveal a clear correlation between students' preferences for gamification components, such as points and badges, and the need for adaptation and feedback to enhance the effectiveness of the educational process. Dusmukhamedova et al. (2024) investigate the integration of gamification with artificial intelligence, concluding that this combination may provide a viable solution to the challenges students face regarding motivation and retention. Rey (2024) emphasizes the transformative potential of gamification, exemplified by the CodeQuest application, for online education in objectoriented programming, addressing challenges related to students' comprehension and engagement in technical disciplines.

Thus, exploring the impact of gamification in the context of Enade can offer valuable insights into its role in improving student engagement and competencies, complementing discussions on the effectiveness of gamification in educational environments.

5 Final considerations

This study systematically reviewed the literature to identify techniques used in analyzing Enade data and implementing diagnostic actions for the required competencies of students taking the exam. Initially, 129 papers were found, and after applying selection criteria, 32 studies were included. We observed that the analysis of Enade data is essential for ensuring educational quality, monitoring student performance, identifying knowledge gaps, guiding the labor market, and promoting academic research.

The implications of this study are significant, particularly in the context of understanding the statistical methodologies and machine learning approaches employed in the evaluation of Enade data and in preparing students for the exam. These findings have the potential to provide valuable insights and benefits to both higher education institutions (HEIs) and researchers. Specifically, they can aid HEIs in enhancing their strategies for student preparation and exam performance analysis.

For researchers interested in conducting studies related to Enade or those exploring tools recommended by the scientific community for monitoring students' progress in the exam, the findings of this study offer a foundation of knowledge. These insights can guide the selection of appropriate statistical and machine learning techniques for analyzing Enade data.

5.1 Limitations and threats to validity

This study has certain limitations, as it focuses on statistical techniques, machine learning methods, and developed systems, including works published over the past 10 years, up to 2022.

To address validity threats, systematic methods were employed to minimize biases and enhance result analysis objectivity. A research protocol was established and discussed with domain experts at each stage of the process. However, the execution of the protocol by two researchers may introduce a potential partial bias. It is possible that important studies were not included in this analysis, possibly due to sources not being indexed in the databases used, the search string not covering certain studies, or studies being in languages other than English or Portuguese. To mitigate this threat, we used relevant databases commonly used for systematic reviews and multiple attempts were made to define the search string.

5.2 Perspectives and future directions

Our review covers only the period from 2012 to 2022, as mentioned in Section 5.1. However, a non-systematic search in 2023 and 2024 demonstrates the continued relevance of Enade analysis and the application of computational and statistical techniques. Research such as that of Castilho (2024) focuses on data mining in Information Systems courses, while Sanchez Trujillo (2024) and Sakashita (2023) explore similar subjects in Computer Science courses. Furthermore, Guilhen (2024) investigates predictive analysis using machine learning algorithms based on data from engineering students who took the Enade exam in 2019. This continuity of research highlights the growing importance of these analytical approaches in education.

In this context, a new perspective emerges with the adoption of Generative Artificial Intelligences and Large-Scale Language Models, such as ChatGPT. These tools have the potential to develop simulators, act as explanatory problem-solving machines, and provide study references for students preparing for the exam. For example, Mendonça (2024) demonstrated that version 4.0 of ChatGPT achieved scores comparable to those of the best human participants in the 2021 Enade for the Bachelor's degree in Computer Science, excelling in open-ended and multiple-choice questions that included visual elements. This performance not only illustrates the effectiveness of these technologies, but also highlights the potential of AI in Enade, offering support for tasks such as advanced tutoring systems, educational assessments aligned with student needs, and personalized learning experiences aimed at exam preparation.

5.3 Future work

Future work involves the development of computational models aimed at diagnosing students' performance based on the competencies assessed by Enade. Leveraging machine learning techniques and data analytics, these models can provide a deeper understanding of students' strengths and weaknesses, allowing for more targeted interventions and support. Such models have the potential to enhance the effectiveness of educational programs and interventions.

Another promising area for exploration is the application of natural language processing (NLP) to automate the classification of exam questions. Automated question classification can streamline the process of organizing and categorizing questions based on their content and difficulty, making it easier for educators and students to access relevant materials. This could significantly improve the efficiency of exam preparation and enhance the overall learning experience.

Furthermore, this research suggests the evaluation of implementing adaptive testing concepts in exam preparation systems. Adaptive testing involves tailoring the difficulty of questions to the individual student's performance, ensuring a more personalized and effective learning experience. Investigating the feasibility and effectiveness of such adaptive systems can lead to the development of innovative and student-centric exam preparation tools.

• 1
  In Portuguese: (“educação superior” or “avaliação de larga escala” or “avaliação educacional superior” or “teoria de resposta ao item” or “teoria clássica dos itens” or “avaliação do conhecimento”) and “enade” and (“mineração de dados” or “aprendizagem de máquina” or “análise de dados” or “algoritmo” or “ferramenta de análise” or “analítica da aprendizagem”)
• 2
  https://scholar.google.com/
• 3
  https://www-periodicos-capes-gov-br.ezl.periodicos.capes.gov.br
• 4
  https://docs.google.com/spreadsheets/u/0/
• 5
  Enade Concept is a quality indicator that evaluates courses based on student success on the Enade exam.
• 6
  IRT allows for the evaluation of students' performance based on the complexity of the subject covered by each exam question
• 7
  https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exameseducacionais/enade/questionario-do-estudante

References

• ALVES, Kauê Tortato. Análise das propriedades psicométricas da prova de conhecimentos específicos de licenciatura em ciências biológicas no Enade 2020 Dissertação (Mestrado em Métodos e Gestão em Avaliação) - Universidade Federal de Santa Catarina, Florianópolis, 2020. Disponível em: https://mobile.repositorio.ufsc.br/bitstream/handle/123456789/216664/PMGA0048D.pdf?sequence=-1&isAllowed=y Acesso em: 7 nov. 2024.
  » https://mobile.repositorio.ufsc.br/bitstream/handle/123456789/216664/PMGA0048D.pdf?sequence=-1&isAllowed=y
• ARAUJO, Antonia Amanda et al. Diferencial de desempenho dos estudantes cotistas no Exame Nacional de Desempenho de Estudantes: evidências sobre as instituições de ensino superior federais. Revista Brasileira de Educação, Rio de Janeiro, v. 25, e250064, 2020. Disponível em: https://www.scielo.br/j/rbedu/a/sDcnBvg4kNMDsLnZZWMMX7R/abstract/?lang=pt Acesso em: 7 nov. 2024.
  » https://www.scielo.br/j/rbedu/a/sDcnBvg4kNMDsLnZZWMMX7R/abstract/?lang=pt
• ARAÚJO, Rodrigo Alexandrino et al. Análise dos microdados do Enade: proposta de uma ferramenta de exploração utilizando mineração de dados. 2019. Dissertação (Mestrado em Ciência da Computação) - Universidade Federal de Goiás, Goiânia, 2019. Disponível em: https://repositorio.bc.ufg.br/tedeserver/api/core/bitstreams/cd220565-f157-4617-a71dfaeb36defa70/content Acesso em: 7 nov. 2024.
  » https://repositorio.bc.ufg.br/tedeserver/api/core/bitstreams/cd220565-f157-4617-a71dfaeb36defa70/content
• BARBOSA, Pedro Luis Saraiva; CARVALHO, Windson Viana. Uso continuado de ferramenta computacional baseada em questões do ENADE para a avaliação diagnóstica de alunos do curso de Sistemas de Informação: um estudo de caso. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO, 31, 2020. [Anais]. Porto Alegre, Sociedade Brasileira de Computação (SBC), 2020. p. 1743-1752. Disponível em: https://sol.sbc.org.br/index.php/sbie/article/view/12930/12784 Acesso em: 7 nov. 2024.
  » https://sol.sbc.org.br/index.php/sbie/article/view/12930/12784
• BRASIL. Lei nº 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior - SINAES e dá outras providências. Diário Oficial da União, Brasília, Presidência da República, 14 de abril de 2004. Disponível em: https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm Acesso em: 7 nov. 2024.
  » https://www.planalto.gov.br/ccivil_03/_ato2004-2006/2004/lei/l10.861.htm
• CAMARGO, Raphael Vinicius Weigert et al. Desempenho dos alunos de ciências contábeis na prova ENADE/2012: uma aplicação da Teoria da Resposta ao Item. Revista de Educação e Pesquisa em Contabilidade (REPeC), Brasília, v. 10, n. 3, art. 6, p. 332-355, jul./set. 2016. Disponível em: https://www.repec.org.br/repec/article/view/1401/1183 Acesso em: 7 jun. 2024.
  » https://www.repec.org.br/repec/article/view/1401/1183
• CASITAGHI, Bruna; ARAGÃO, Júlio César Soares. Avaliação do ensino superior brasileiro: desempenho dos estudantes em formação geral. Revista Portuguesa de Educação, Braga, v. 34, n. 1, 2021. Disponível em: https://revistas.rcaap.pt/rpe/article/view/20821/18429 Acesso em: 7 jun. 2024.
  » https://revistas.rcaap.pt/rpe/article/view/20821/18429
• CASTILHO, Mércia de Souza Maguerroski. Análise de perfil, desempenho e provas do ENADE do curso de Sistemas de Informação da UFSC 2024. TCC (Graduação em Sistemas de Informação) - Universidade Federal de Santa Catarina, Florianópolis, 2024.
• CHARÃO, Andrea S. et al. Explorando resultados por questão no Enade em ciência da computaçao para subsidiar revisão de projeto pedagógico de curso. In: WORKSHOP SOBRE EDUCAÇAO EM COMPUTAÇAO (WEI), 28, 2020. [Anais]. Porto Alegre, Sociedade Brasileira de Computação (SBC), 2020. p. 16-20. Disponível em: https://sol.sbc.org.br/index.php/wei/article/view/11121/10992 Acesso em: 7 nov. 2024.
  » https://sol.sbc.org.br/index.php/wei/article/view/11121/10992
• COELHO, Edy Célia. Teoria da resposta ao item: desafios e perspectivas em exames multidisciplinares. 2014. Tese (Doutorado em Ciências) - Universidade Federal do Paraná, Curitiba, 2014. Disponível em: https://acervodigital.ufpr.br/xmlui/bitstream/handle/1884/36872/R%20-%20T%20%20EDY%20CELIA%20COELHO.pdf?sequence=3&isAllowed=y Acesso em: 7 nov. 2024.
  » https://acervodigital.ufpr.br/xmlui/bitstream/handle/1884/36872/R%20-%20T%20%20EDY%20CELIA%20COELHO.pdf?sequence=3&isAllowed=y
• CUNHA, Renan; SALES, Claudomiro; SANTOS, Reginaldo. Análise automática com os microdados do ENADE para melhoria do ensino dos cursos de ciência da computação. In: WORKSHOP SOBRE EDUCAÇÃO EM COMPUTAÇÃO, 29, Porto Alegre. [Anais]. Porto Alegre, Sociedade Brasileira de Computação (SBC), 2021. p. 208-217. Disponível em: https://sol.sbc.org.br/index.php/wei/article/view/15912/15753 Acesso: 7 nov. 2024.
  » https://sol.sbc.org.br/index.php/wei/article/view/15912/15753
• DIAS SOBRINHO, José. Avaliação e transformações da educação superior brasileira (1995-2009): do provão ao SINAES. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 15, n. 1, p. 195-224, 2010. Disponível em: https://www.scielo.br/j/aval/a/SkVnKQhDyk6fkNngwvZq44c/abstract/?lang=pt Acesso em: 7 nov. 2024.
  » https://www.scielo.br/j/aval/a/SkVnKQhDyk6fkNngwvZq44c/abstract/?lang=pt
• DUSMUKHAMEDOVA, Shoira et al. A novel based design of ai kind of models for gamification in enhanced learning type of environment. In: INTERNATIONAL CONFERENCE ON ADVANCE COMPUTING AND INNOVATIVE TECHNOLOGIES IN ENGINEERING (ICACITE), 4., 2024. [Anais]. [s. l] IEEE, 2024. p. 584-588. Disponível em: https://www.researchgate.net/publication/383010887_A_Novel_Based_Design_of_AI_kind_of_Models_for_Gami-Fication_in_Enhanced_Learning_type_of_Environment Acesso em: 7 nov. 2024.
  » https://www.researchgate.net/publication/383010887_A_Novel_Based_Design_of_AI_kind_of_Models_for_Gami-Fication_in_Enhanced_Learning_type_of_Environment
• EUROPEAN ASSOCIATION FOR QUALITY ASSURANCE IN HIGHER EDUCATION (ENQA). Standards and guidelines for quality assurance in the European Higher Education Area (ESG). Brussells, 2015. Disponível em: https://www.enqa.eu/wpcontent/uploads/2015/11/ESG_2015.pdf Acesso em: 7 nov. 2024.
  » https://www.enqa.eu/wpcontent/uploads/2015/11/ESG_2015.pdf
• FIGUEIRO, Michele Ferraz et al. Análise de agrupamento hierárquico aplicada aos microdados do ENADE do curso de graduação em Ciência da Computação. Revista Eletrônica Argentina-Brasil de Tecnologias da Informação e da Comunicação, Rio Grande do Sul, v. 1, n. 8, p. 1-13, 2017. Disponível em: https://revistas.setrem.com.br/index.php/reabtic/article/view/267 Acesso em: 7 nov. 2024.
  » https://revistas.setrem.com.br/index.php/reabtic/article/view/267
• FONSECA, Luís Carlos Costa; ALMADA COUTINHO, Lidinalva; SANTOS, Lanylldo Araujo. Ferramenta para estimar a proficiência de indivíduos baseada na teoria de resposta ao item. A RENOTE - Revista Novas Tecnologias na Educação, Porto Alegre, v. 11, n. 1, p. 1-11, jul. 2013. Disponível em: https://seer.ufrgs.br/renote/article/view/41680 Acesso em: 7 nov. 2024.
  » https://seer.ufrgs.br/renote/article/view/41680
• FRANÇA, Tarsys Fonseca et al. TôSabendo: a platform to create engaging teaching and learning experiences. In: LATIN AMERICAN CONFERENCE ON LEARNING TECHNOLOGIES (LACLO), 16., 2021, Peru. Anais […]. Peru: IEEE, 2021. p. 275-281.
• GUILHEN, Bruno Anselmo. Avaliação de algoritmos para a análise preditiva das notas de Engenharia no ENADE utilizando dados socioeconômicos. 2024. Dissertação (Mestrado em Ciências - Engenharia Elétrica e Sistemas Digitais) - Escola Politécnica da Universidade de São Paulo, Universidade de São Paulo - USP, 2024.
• INEP. Conceito Enade. Brasília: Inep, 2020.Disponível em: https://www.gov.br/inep/ptbr/areas-de-atuacao/pesquisas-estatisticas-e-indicadores/indicadores-de-qualidade-daeducacao-superior/conceito-enade Acesso em: 11 out. 2023.
  » https://www.gov.br/inep/ptbr/areas-de-atuacao/pesquisas-estatisticas-e-indicadores/indicadores-de-qualidade-daeducacao-superior/conceito-enade
• INEP. Exame Nacional de Desempenho dos Estudantes (Enade). Brasília, MEC, 2023. Disponível em: https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exameseducacionais/enade Acesso em: 11 out. 2023.
  » https://www.gov.br/inep/pt-br/areas-de-atuacao/avaliacao-e-exameseducacionais/enade
• INEP. Microdados do Enade 2017 (Atualizado em 30/6/2022). Brasília: Inep, 2017. Disponível em: https://www.gov.br/inep/pt-br/acesso-a-informacao/dadosabertos/microdados/enade Acesso em: 7 nov. 2024.
  » https://www.gov.br/inep/pt-br/acesso-a-informacao/dadosabertos/microdados/enade
• KEELE, Staffs et al. Guidelines for performing systematic literature reviews in software engineering. Keele: ST5, 2007. Disponível em: https://legacyfileshare.elsevier.com/promis_misc/525444systematicreviewsguide.pdf Acesso em: 7 nov. 2024.
  » https://legacyfileshare.elsevier.com/promis_misc/525444systematicreviewsguide.pdf
• LANDES, Flávio Barreiro; MANHÃES, Laci Mary Barbosa. Análise dos exames do ENADE para os cursos de computação utilizando o software R. Revista do Seminário Internacional de Estatística com R, Niterói, v. 3, n. 1, p. 1-14, 2018. Disponível em: https://periodicos.uff.br/anaisdoser/article/view/29225 Acesso em: 7 nov. 2024.
  » https://periodicos.uff.br/anaisdoser/article/view/29225
• LIMA, Priscila da Silva Neves et al. Análise de conteúdo das provas do Enade para os alunos do curso de Bacharelado em Ciência da Computação. Revista Brasileira de Informática na Educação, Porto Alegre, v. 29, p. 385-413, 2021. Disponível em: https://journalssol.sbc.org.br/index.php/rbie/article/view/2981 Acesso em: 7 nov. 2024.
  » https://journalssol.sbc.org.br/index.php/rbie/article/view/2981
• LIMA, Priscila da Silva Neves et al. Análise de dados do Enade e Enem: uma revisão sistemática da literatura. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 24, n. 1, p. 89-107, mar./maio 2019. Disponível em: https://www.scielo.br/j/aval/a/L4J43gBxhXmjYhT5cX6BTTM/ Acesso em: 7 nov. 2024.
  » https://www.scielo.br/j/aval/a/L4J43gBxhXmjYhT5cX6BTTM/
• LIMA, Priscila da Silva Neves et al. Content analysis of student assessment exams. Frontiers in Education Conference (FIE), San Jose, p. 1-9, oct. 2018. Disponível em: https://www.researchgate.net/publication/329131186_Content_Analysis_of_Student_Assessment_Exams Acesso em: 7 nov. 2024.
  » https://www.researchgate.net/publication/329131186_Content_Analysis_of_Student_Assessment_Exams
• LOPES, Fernanda Luzia; VENDRAMINI, Claudette Maria Medeiros. Propriedades psicométricas das provas de pedagogia do ENADE via TRI. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 20, n. 1, p. 27-47, mar. 2015. Disponível em: https://www.scielo.br/j/aval/a/64h6cvwZrKPR9v4P4nChZ6t/abstract/?lang=pt Acesso em: 7 nov. 2024.
  » https://www.scielo.br/j/aval/a/64h6cvwZrKPR9v4P4nChZ6t/abstract/?lang=pt
• MARTINS, Ernane Rosa; ALARCÃO, Davi Taveira Alencar; VIEIRA, Sibelius Lellis. Descoberta de conhecimento aplicado ao Enade. In: MARTINS, Ernane Rosa (org.). Engenharia de produção: gestão de qualidade, produção e operações. Guarujá: Científica Digital, 2021. v. 2. Disponível em: https://downloads.editoracientifica.com.br/articles/211006382.pdf Acesso em: 7 nov. 2024.
  » https://downloads.editoracientifica.com.br/articles/211006382.pdf
• MATOS, A. M et al. Mineração dos dados do Enade: avaliação do questionário do estudante das instituições do norte do Brasil. In: ECOINFO - CONGRESSO DE COMPUTAÇÃO E TECNOLOGIA DA INFORMAÇÃO, 20., 2020. [Anais]. Universidades Luterana do Brasil, Palmas, 2020. Disponível em: https://ulbra-to.br/encoinfo/wp-content/uploads/2020/03/Minera%C3%A7%C3%A3o-dos-Dados-do-Enade-Avalia%C3%A7%C3%A3o-do-question%C3%A1rio-do-estudante-das-Institui%C3%A7%C3%B5es-do-Norte-do-Brasil.pdf Acesso em: 7 nov. 2024.
  » https://ulbra-to.br/encoinfo/wp-content/uploads/2020/03/Minera%C3%A7%C3%A3o-dos-Dados-do-Enade-Avalia%C3%A7%C3%A3o-do-question%C3%A1rio-do-estudante-das-Institui%C3%A7%C3%B5es-do-Norte-do-Brasil.pdf
• MEDEIROS FILHO, Antonio Evanildo Cardoso. Percepção discente e desempenho dos estudantes dos cursos de licenciatura em Educação Física no Enade 2017. 2019. Dissertação (Mestrado Acadêmico ou Profissional) - Universidade Estadual do Ceará, 2019. Disponível em: http://siduece.uece.br/siduece/trabalhoAcademicoPublico.jsf?id=84781 Acesso em: 7 de novembro de 2024.
  » http://siduece.uece.br/siduece/trabalhoAcademicoPublico.jsf?id=84781
• MELGUIZO, Tatiana; WAINER, Jacques. Toward a set of measures of student learning outcomes in higher education: evidence from Brazil. Higher Education, v. 72, p. 381-401, 2016. DOI 10.1007/s10734-015-9963-x. Disponível em: https://link.springer.com/article/10.1007/s10734-015-9963-x Acesso em: 7 nov. 2024.
  » https://doi.org/10.1007/s10734-015-9963-x.» https://link.springer.com/article/10.1007/s10734-015-9963-x
• MENDONÇA, Nabor C. Evaluating ChatGPT-4 Vision on Brazil’s National Undergraduate Computer Science Exam. ACM Transactions on Computing Education, Nova York, v. 24, n. 3, p. 1-56, 2024. Disponível em: https://dl.acm.org/doi/10.1145/3674149 Acesso em: 7 nov. 2024.
  » https://dl.acm.org/doi/10.1145/3674149
• NUNES, Ricardo Tocantins Modugno. Cálculo preditivo de classificação das notas do enade utilizando redes neurais artificiais. RTA - Revista de Tecnologia Aplicada, Campo Limpo Paulista, SP, v. 7, n. 2, p.71-81, 2018. Disponível em: https://www.cc.faccamp.br/ojs-2.4.82/index.php/RTA/article/view/1171 Acesso em: 7 nov. 2024.
  » https://www.cc.faccamp.br/ojs-2.4.82/index.php/RTA/article/view/1171
• PITON-GONCALVES, Jean. Testes adaptativos para o Enade: uma aplicação metodológica. Revista Meta: Avaliação, Rio de Janeiro, v. 12, n. 36, p. 665-688, 2020. Disponível em: https://revistas.cesgranrio.org.br/index.php/metaavaliacao/article/view/2735 Acesso em: 7 nov. 2024.
  » https://revistas.cesgranrio.org.br/index.php/metaavaliacao/article/view/2735
• RALPH, Paul; BALTES, Sebastian. Paving the way for mature secondary research: the seven types of literature review. In: PROCEEDINGS OF THE 30TH ACM JOINT EUROPEAN SOFTWARE ENGINEERING CONFERENCE AND SYMPOSIUM ON THE FOUNDATIONS OF SOFTWARE ENGINEERING, 22., 2022. [Anais]. Singapore, 2022. p. 1632-1636. Disponível em: http://assets.empirical-software.engineering/pdf/fse22-literature-reviews.pdf Acesso em: 7 nov. 2024.
  » http://assets.empirical-software.engineering/pdf/fse22-literature-reviews.pdf
• REY, William P. CodeQuest mobile app: a comprehensive analysis of system performance and user evaluation for enhanced learning experiences. In: INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND TECHNOLOGIES IN EDUCATION (ICCCBDA), 6., 2024. [Anais]. [s. l], 2024. p. 6-10. Disponível em: https://www.researchgate.net/publication/381808092_CodeQuest_Mobile_App_A_Comprehensive_Analysis_of_System_Performance_and_User_Evaluation_for_Enhanced_Learning_Experiences Acesso em: 7 nov. 2024.
  » https://www.researchgate.net/publication/381808092_CodeQuest_Mobile_App_A_Comprehensive_Analysis_of_System_Performance_and_User_Evaluation_for_Enhanced_Learning_Experiences
• RIBEIRO, Tiago C.; COSTA JUNIOR, Wilson F. Plataforma para auxílio na preparação de estudantes para as avaliações do ENADE e POSCOMP 2020. TCC (Graduação em Ciência da Computação) - Instituto Federal de Santa Catarina, 2020. Disponível em: https://repositorio.ifsc.edu.br/bitstream/handle/123456789/2458/Tiago_C_Ribeiro%3bWilson_F_C_Junior_TCCGRAD_2020.pdf?sequence=1&isAllowed=y Acesso em: 7 nov. 2024.
  » https://repositorio.ifsc.edu.br/bitstream/handle/123456789/2458/Tiago_C_Ribeiro%3bWilson_F_C_Junior_TCCGRAD_2020.pdf?sequence=1&isAllowed=y
• RODRIGUES, Ebony M.; GOUVEIA, Roberta M. M. Técnicas de machine learning para predição do tempo de permanência na graduação no âmbito do ensino superior público brasileiro. In: CONGRESSO SOBRE TECNOLOGIAS NA EDUCAÇÃO, 6., 2021. [Anais]. Sociedade Brasileira de Computação, Pau dos Ferros, RN, 2021. p. 128-137. Disponível em: https://sol.sbc.org.br/index.php/ctrle/article/view/17557 Acesso em: 7 nov. 2024.
  » https://sol.sbc.org.br/index.php/ctrle/article/view/17557
• ROSA, Eliene Ribeiro et al. Estudo exploratório através de análises longitudinais aplicado à ciência da computação a partir da base de dados do ENADE. Revista Brasileira de Informática na Educação, Porto Alegre, v. 29, p. 1463-1486, 2021. Disponível em: https://journals-sol.sbc.org.br/index.php/rbie/article/view/2073 Acesso em: 7 nov. 2024.
  » https://journals-sol.sbc.org.br/index.php/rbie/article/view/2073
• RYBINSKI, Krzysztof. Are rankings and accreditation related? Examining the dynamics of higher education in Poland. Quality assurance in education, Reino Unido, v. 28, n. 3, p. 193204, 2020. Disponível em: https://www.emerald.com/insight/content/doi/10.1108/qae-032020-0032/full/html Acesso em: 7 nov. 2024.
  » https://www.emerald.com/insight/content/doi/10.1108/qae-032020-0032/full/html
• RYBINSKI, Krzysztof. Assessing how QAA accreditation reflects student experience. Higher Education Research & Development, Londres, v. 41, n. 3, p. 898-918, 2022. Disponível em: https://www.tandfonline.com/doi/abs/10.1080/07294360.2021.1872058 Acesso em: 7 nov. 2024.
  » https://www.tandfonline.com/doi/abs/10.1080/07294360.2021.1872058
• SAKASHITA, Renan GA; BERTOLINI, Diego; SCHWERZ, André Luis. Predição do conceito ENADE dos cursos de computação no Brasil. In: SIMPÓSIO BRASILEIRO DE INFORMÁTICA NA EDUCAÇÃO, 34., 2023. [Anais]. Sociedade Brasileira de Computação, Porto Alegre, 2023. p. 1305-1316. Disponível em: https://sol.sbc.org.br/index.php/sbie/article/view/26756 Acesso em: 7 nov. 2024.
  » https://sol.sbc.org.br/index.php/sbie/article/view/26756
• SALMAN, Osama K. et al. Tailoring gamification to individual learners: A study on personalization variables for skill enhancement. International Journal of Data and Network Science, Canada, v. 8, n. 2, p. 789-796, 2024. Disponível em: https://www.growingscience.com/ijds/Vol8/ijdns_2023_250.pdf Acesso em: 7 nov. 2024.
  » https://www.growingscience.com/ijds/Vol8/ijdns_2023_250.pdf
• SANCHEZ TRUJILLO, Daniel Erick. Mineração de dados aplicada ao Exame Nacional de Desempenho dos Estudantes (ENADE). 2024. TCC (Graduação em Engenharia da Computação) - Universidade de Brasília, Brasília, 2024. Disponível em: https://164.41.168.37/bitstream/10483/38462/1/2024_DanielErickSanchezTrujillo_tcc.pdf Acesso em: 7 nov. 2024.
  » https://164.41.168.37/bitstream/10483/38462/1/2024_DanielErickSanchezTrujillo_tcc.pdf
• SANTOS, Agenor Sousa; DANTAS, Maria José Pereira; MACHADO, Ricardo Luiz. Structural equation modeling applied to assess industrial engineering students’ satisfaction according to ENADE 2011. Production, São José dos Campos, SP, v. 27, 2017. Disponível em: https://www.scielo.br/j/prod/a/vn8pYS3cW5qmB7h7x3MG9mp/?lang=en Acesso em: 7 nov. 2024.
  » https://www.scielo.br/j/prod/a/vn8pYS3cW5qmB7h7x3MG9mp/?lang=en
• SANTOS, Perycles Jannser Lopes et al. E-prova: sistema para elaboração de avaliações no padrão Enade. Brasília: MEC, 2016.
• SILVA, Alexsander Figueiredo; HOED, Raphael Magalhães; SARAIVA, Pedro Fábio. Análise do desempenho dos alunos de cursos superiores em computação no enade - uma abordagem usando mineração de dados. CONFERÊNCIAS IADIS IBERO-AMERICANAS WWW/INTERNET E COMPUTAÇÃO APLICADA, Lisboa, 2019. Anais […]. Lisboa, Portugal: IADIS, 2019. p. 207-214. Disponível em: https://www.iadisportal.org/components/com_booklibrary/ebooks/201914L026.pdf Acesso em: 7 nov. 2024.
  » https://www.iadisportal.org/components/com_booklibrary/ebooks/201914L026.pdf
• SILVA, I; SILVA, M; COSTA NETO, P. Descoberta dos fatores preditivos de sucesso dos alunos do Curso de Tecnologia em Logística, nos exames do Enade, por meio da Mineração de Dados Educacionais. In: INTERNATIONAL CONFERENCE ON NETWORK ENTERPRISES AND LOGISTICS MANAGEMENT, 2., 2021. [Anais]. São Paulo, 2021. Disponível em: https://www.researchgate.net/publication/351457514_Descoberta_dos_fatores_preditivos_de_sucesso_dos_alunos_do_Curso_de_Tecnologia_em_Logistica_nos_exames_do_ENADE_por_meio_da_Mineracao_de_Dados_Educacionais Acesso em: 7 nov. 2024.
  » https://www.researchgate.net/publication/351457514_Descoberta_dos_fatores_preditivos_de_sucesso_dos_alunos_do_Curso_de_Tecnologia_em_Logistica_nos_exames_do_ENADE_por_meio_da_Mineracao_de_Dados_Educacionais
• SILVA, Jesaías Carvalho Pereira et al. Data analysis of the performance of Brazilian higher education medicine courses. The ACM Digital Library, New York, n. 13, p. 1-8, 2020. Disponível em: https://dl.acm.org/doi/10.1145/3401895.3401922 Acesso em: 7 nov. 2024.
  » https://dl.acm.org/doi/10.1145/3401895.3401922
• SILVA, L. G. F.; ROCHA, M. E. P. S.; FAGUNDES, RA de A. ENADE: Math and Science Students' Performance Analysis. IEEE Latin America Transactions, Estados Unidos, v. 15, n. 9, p. 17421746, 2017. DOI 10.1109/TLA.2017.8015081. Disponível em: https://ieeexplore.ieee.org/document/8015081 Acesso em: 7 nov. 2024.
  » https://doi.org/10.1109/TLA.2017.8015081.» https://ieeexplore.ieee.org/document/8015081
• STONE, Patricia W. Popping the (PICO) question in research and evidence-based practice. Applied nursing research, Amsterdam, 15, n. 3, p. 197-198, ago. 2002. Disponível em: https://www.sciencedirect.com/science/article/abs/pii/S0897189702000101?via%3Dihub Acesso em: 7 nov. 2024.
  » https://www.sciencedirect.com/science/article/abs/pii/S0897189702000101?via%3Dihub
• TIVES, Heloise Acco et al. Mining Enade data from the Ulbra network institution. In: INFORMATION technology: new generations. [s. l]:Springer International Publishing, 2018. p. 287-294. Disponível em: https://www.researchgate.net/publication/324503235_Mining_ENADE_Data_from_the_Ulbra_Network_Institution Acesso em: 7 nov. 2024.
  » https://www.researchgate.net/publication/324503235_Mining_ENADE_Data_from_the_Ulbra_Network_Institution
• VAN ELDEREN, Jan; VAN DER STAPPEN, Esther. The potential impact of gamification elements on the acceptance of technology in the context of education: a literature review. CONFERÊNCIA BLED, HUMANIZANDO A TECNOLOGIA PARA UMA SOCIEDADE SUSTENTÁVEL, 32., 2019, ESLOVÊNIA. p. 177-201. Anais [...]. ESLOVÊNIA: BLED, 2019. Disponível em: https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1009&context=bled2019 Acesso em: 7 nov. 2024.
  » https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1009&context=bled2019
• VISTA, Nicolas Pastorio Boa; FIGUEIRÓ, Michele Ferraz; CHICON, Patricia Mariotto Mozzaquatro. Técnicas de mineração de dados aplicadas aos microdados do ENADE para avaliar o desempenho dos acadêmicos do curso de Ciência da Computação no Rio Grande do Sul utilizando o software R. In: SEMINÁRIO DE PESQUISA CIENTÍFICA E TECNOLÓGICA, 1., 2017. [Anais]. Disponível em: https://www.semanticscholar.org/paper/T%C3%A9cnicas-deminera%C3%A7%C3%A3o-de-dados-aplicadas-aos-do-o-o-Vista-Figueir%C3%B3/f4d64fd50a1484134afe250503d22ab5699b1ac2 Acesso em: 7 nov. 2024.
  » https://www.semanticscholar.org/paper/T%C3%A9cnicas-deminera%C3%A7%C3%A3o-de-dados-aplicadas-aos-do-o-o-Vista-Figueir%C3%B3/f4d64fd50a1484134afe250503d22ab5699b1ac2

Publication Dates

• Publication in this collection
  16 Dec 2024
• Date of issue
  2024

History

• Received
  13 Oct 2023
• Accepted
  11 Oct 2024
• Reviewed
  01 Nov 2024