Student self-assessment of the graduate course: a multidimensional model proposal

Autoavaliação do aluno do curso de pós-graduação: uma proposta de modelo multidimensional

Autoevaluación del estudiante del curso de posgrado: una propuesta de modelo multidimensional

Abstract

The objective of this work is to develop a self-assessment model by students and alumni of Graduate Courses. A survey was carried out with students and alumni at a Higher Education institution in southern Brazil, and a valid sample of 2037 respondents was obtained. A model was proposed considering the perceptions of students and alumni regarding the institution’s infrastructure, the course subjects, the course professors, the advisor’s and the administrative supports as antecedents and satisfaction as a result of the course assessment. The results showed that the student’s perception of the subjects and the professors are the dimensions that most impact the assessment of the course. Furthermore, student satisfaction with the course is positively influenced by their self-assessment.

Self-Assessment; Graduate Courses; Students

Resumo

O objetivo deste trabalho é desenvolver um modelo de autoavaliação para alunos e egressos de cursos de pós-graduação. Foi realizada uma pesquisa com estudantes e ex-alunos de uma instituição de ensino superior do sul do Brasil e obteve-se uma amostra válida de 2.037 respondentes. Foi proposto um modelo considerando as percepções dos alunos e ex-alunos sobre a infraestrutura da instituição, as disciplinas do curso, os professores do curso, os orientadores e os apoios administrativos como antecedentes e a satisfação como resultado da avaliação do curso. Os resultados mostraram que a percepção do aluno sobre as disciplinas e dos professores são as dimensões que mais impactam na avaliação do curso. Além disso, a satisfação do aluno com o curso é influenciada positivamente pela sua autoavaliação.

Autoavaliação; Pós-Graduação; Estudantes

Resumen

El objetivo de este trabajo es desarrollar un modelo de autoevaluación por parte de estudiantes y egresados de Cursos de Posgrado. Se realizó una encuesta con estudiantes y ex alumnos de una institución de educación superior en el sur de Brasil, y se obtuvo una muestra válida de 2.037 encuestados. Se propuso un modelo considerando las percepciones de los estudiantes y egresados respecto a la infraestructura de la institución, las asignaturas, los profesores del curso, el asesor y los apoyos administrativos como antecedentes y satisfacción como resultado de la evaluación del curso. Los resultados mostraron que la percepción del alumno de las asignaturas y de los profesores son las dimensiones que más impactan en la evaluación del curso. Además, la satisfacción de los estudiantes con el curso está influenciada positivamente por su autoevaluación.

Autoevaluación; Cursos de Posgrado; Estudiantes

1 Introduction

The assessment process of Higher Education Institutions (HEI) and Post Graduate Courses (PGC) is a fundamental mechanism that generates indicators for both control bodies and course coordinators to optimise their practices to offer high-performance professionals to the academia or the job market, as well as diagnose the aspects that must be improved. In this perspective, the qualification of PGCs becomes a decisive factor to enable the improvement of educational systems (Astakhova et al., 2016).

It is the universities’ responsibility to develop and encourage the employability of their students. In this perspective, universities must show society that they can train students with qualities useful to the job market through quality Education, in addition to citizen training (Caballero; Alvarez-González; López-Miguens, 2021; Leal, 2023). Among their purposes, Bispo and Costa (2016), add that stricto sensu graduate courses train high-level researchers in a given area, professionals capable of working in Higher-Education teaching, and high-performance professionals through professional master’s courses.

Aiming at a greater efficiency in the teaching and learning process of the graduate courses, and the qualification of the courses offered, their assessment is necessary. In Brazil, this process is done through the Brazilian graduate assessment system. Among other goals and based on established criteria, the Coordination of Improvement of Higher Education Personnel (Capes) assesses existing courses and approves the proposal of new courses (Marques; Veiga; Borges, 2020). Capes has been undergoing constant updates encompassing technological, scientific and educational advances and included self-assessment as one of the evaluation criteria for graduate courses in 2018. Self-assessment is a way of self-knowledge of what happens in a course or institution, and allows an evaluation of the present situation of the course and the expectations. Such assessment also permits the evaluation of what has already been accomplished, how the administration takes place, what information is available for analysis and interpretation, and what the strengths and weaknesses of the institution are (Zainko; Pinto, 2008). Self-assessment favours the construction of identity, heterogeneity and involvement of the courses assessed, in addition to the minimum standards guaranteed by external assessment (CAPES, 2019). Also, Seifert and Feliks (2018) claim that self-assessment establishes what is being built, considering the teaching-learning process.

Meeting the CAPES assessment criteria, which include issues related to the dissertation and theses development process; course infrastructure; professors’ performance; curriculum structure, among others, this work aims to develop a Self-Assessment model of Graduate Courses from the student’s perspective.

The study presents an innovative bias for providing new evidence for the emerging literature on self-assessment applied to the graduate courses scenario, since the inclusion of this dimension in the Capes assessment system is recent and highlights a new aspect in the courses assessment field. It also innovates for being the pioneer on building a model that considers the aspects of self-assessment and for dealing with an empirical approach that can be replicated to different realities of the various graduate courses.

In addition, understanding the relationship between the different aspects of self-assessment and identifying the influencers in the assessment process and in the satisfaction of students and alumni is essential for graduate courses to achieve better results, as self-assessment provides relevant and updated information (Angst; Alves, 2018). Furthermore, the study can be useful to identify the central points to be addressed in institutional strategies aimed at improving graduate courses at HEI.

2 Model development

The development of a nation is directly related to the Education quality (Astakhova et al., 2016). Permanent assessment and continuous development of the educational context are necessary to meet social demands, measure quality and provide visibility to educational institutions. The assessment process with organised information favours the understanding of situations and relationships, the construction of meanings and knowledge about subjects, structures and activities that occur in an educational institution at a given time (Leite, 2008).

In Brazil, the assessment is carried out every four years, seeking to analyse and assess the permanence of the PGCs offer (CAPES, 2014). That evaluation contemplates three central dimensions. First, regarding the Program, the assessment seeks to check the operation, structure and planning of the Postgraduate Program (PPG) considering its characteristics and objectives. Second, the Educational Background considers the quality of the students, the performance of professors and the production of knowledge associated with the research and training activities of the program. Third the Impact on Society verifies the innovativeness of the intellectual production, internationalization, impact and social relevance of the program (BRASIL, 2019).

In the last reformulation, implemented for the 2017–2020 quadrennium, the self-assessment of the graduate course started to be recommended and assessed by Capes. Self-assessment is present in the “Course” query, specifically in the “The processes, procedures and results of the course self-assessment, focusing on student training and intellectual production” item (CAPES, 2018).

Self-assessment is characterised by a process of self-reflection and self-analysis that highlights strengths and challenges that remain, aiming to improve the quality of academic work, training concepts, development and learning. Self-assessment is planned, executed and explored by the individuals who are responsible for the actions being assessed. It also allows for reflections on the context and policies applied, as well as the organisation of data that result in decision-making (CAPES, 2019).

The self-assessment process must be broad both in its assessment dimensions and in the assumption of the participation of all those involved (Angst, 2017; Myalkina, 2019). When creating a model, one must consider the need to cover all the activities of academic institutions, teaching, research, extension and management in a coordinated way, as well as crediting and supporting the various interested parties for the information (Monticelli et al., 2021).

Therefore, considering the need to implement effective self-assessment systems to meet Capes’s requirements, the internal assessment systems of educational institutions, and the multidimensionality necessary for a self-assessment model, this study proposes a Student Self-Assessment model with seven main dimensions: “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”, “Course Assessment” and “Satisfaction with the Course”, which are related as shown in Figure 1 .

Figure 1
Student Self-Assessment Model

The “course assessment” represents the graduate student’s self-assessment. This dimension is formed by three first-order constructs: the “Graduate Course Impacts”, “Professional Development” and “Selection Processes”. The “Graduate Course Impacts” dimension assesses the course performance in terms of teaching capacity, self-management and regional and national impacts, in addition to international performance. The “Professional Development” dimension analyses both the ability of the course to provide specific knowledge that effectively contributes to career development and to build researchers capable of producing nationally and internationally. On the other hand, the “Selection Process” dimension refers to the form, clarity and adequacy of the processes for selecting students and distributing scholarships (if any).

As determinants of the student’s self-assessment, five fundamental factors are considered: HEI Infrastructure, Course Subjects, Course Professors, Advisor Support, and Administrative Support. The first three factors are assessment parameters provided for in the assessment model of graduate courses used by Capes. The last two, which refer to the presence of support for students, are important support dimensions for the conclusion of a stricto sensu graduate course.

The “HEI Infrastructure” factor identifies the student’s perception of the adequacy of physical facilities, equipment, Internet access and security provided by the educational institution. The “Course Subjects” dimension comprehensively assesses the student’s perception of the adequacy of contents, bibliographies, activities developed, evaluation forms, and the quality of classes. In “Course Professors”, the student’s perception is sought both in terms of mastery of knowledge, teaching learning strategies and the form of service and access to the subject’s professors. In this sense, it is considered that in Higher Education, teaching-learning strategies generally require students to act more independently and proactively, with feedback being essential (Leal, 2023).

In the support-related dimensions, both the course secretariat and the advisors are assessed. For the secretariat, the focus is on the adequate provision of information, accessibility, courtesy, and willingness to help students. The insertion of the “Advisor Support” dimension is justified by the evidence that the advisor-student relationship has a relevant role in retaining students in courses, in addition to being important in the knowledge construction (Moore, 2017; Vieira et al. , 2022), since it helps students to fulfil their academic, personal and career goals (Martinez; Elue, 2020; Meurer et al ., 2020). In the student’s perception, the dimension identifies the various facets of the guidance process, ranging from issues related to knowledge sharing to issues of access and feedback from the advisor.

The last dimension of the model is “Satisfaction with the Graduate Course”, which is directly impacted by the course assessment. This dimension represents the student’s level of satisfaction with the course in a comprehensive way, considering the main assessment parameters of a course according to the Capes assessment system.

Therefore, the Student Self-Assessment model is multidimensional, with seven central dimensions and three constructs that form the course assessment dimension. Of the seven dimensions, five are considered assessment antecedents (“HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”), and “Satisfaction” is the consequent dimension, that is, the one that is directly impacted by the course assessment. Table 1 presents the set of items that measure each of the model dimensions and their respective scales.

Table 1
Items and scales of the student self-assessment model dimensions

Dimension		Items	Scale
HEI Infrastructure	1	Resources made available by UFSM libraries.	1 = Inadequate 2 = Not suitable 3 = Adequate 4 = Very suitable 5 = Totally adequate
	2	Infrastructure available at the Teaching Centres (or campuses outside the headquarters) for holding conferences and events.
	3	Safety measures practised at the university.
	4	The campus infrastructure.
	5	The infrastructure used by the program.
	6	The adequacy of the infrastructure used by the program for PNE (Persons with Special Needs).
	7	The equipment available.
	8	The labs available.
	9	Internet access in the classrooms used by the program.
	10	Internet access in common areas at the university.
	11	Internet access in my workroom.
Course Professors	12	The professors are responsive.	1 = I totally disagree 2 = I partially disagree 3 = I neither disagree nor agree 4 = I partially agree 5 = I totally agree
	13	The professors are accessible.
	14	The professors adopt good teaching practices.
	15	The professors have specific knowledge that they add to my training.
	16	The professors have general knowledge that adds to my training.
	17	The professors have international academic contacts.
	18	The professors publish in international journals.
Course Subjects	19	I can understand the concepts discussed in the subjects.
	20	I can apply the concepts discussed in the subjects.
	21	I believe the classes are of good quality.
	22	The subject’s activities are adequate.
	23	The bibliography is predominantly based on international literature.
	24	The bibliography is updated.
	25	The use of writing articles as an assessing criterion is appropriate.
	26	The assessment criteria are adequate.
	27	I receive feedback from the activities I perform in the course.
	28	Grading is consistent with my performance.
	29	The recommended readings are appropriate to what is discussed in the classroom.
	30	The amount of readings is adequate to what is discussed in the classroom.
	31	The course workload is fulfilled.
	32	The program content is fulfilled.
Advisor Support	33	My advisor has a good relationship with me.
	34	My advisor treats me respectfully.
	35	My advisor is a good researcher.
	36	My advisor is patient with me.
	37	My advisor always assists me when I need it.
	38	My advisor quickly returns the requests I submit.
	39	My advisor is easily accessible.
	40	My advisor encourages me to attend events.
	41	My advisor helps me publish articles in national journals.
	42	My advisor helps me publish articles in international journals.
	43	My advisor helps me get international academic contacts.
	44	My advisor places me in activities (meetings, presentations, projects, articles etc.).
	45	My advisor masters the knowledge related to the subject of my dissertation/thesis.
	46	My advisor masters the methodological procedures necessary for the development of my dissertation/thesis.
	47	My advisor discusses my research with me regularly.
	48	My advisor teaches me to find work that is relevant to the development of my work.
	49	My advisor provides constructive feedback and guidance for my work.
	50	My advisor is essential for the elaboration of my dissertation/thesis.
Administrative Support	51	The secretary is courteous.
	52	The secretary is always willing to help.
	53	The secretary provides correct information.
	54	The secretary is accessible during working hours.
	55	The secretary answers my questions.
	56	The secretary treats me respectfully.
	57	The secretary updates me with academic proceedings.
Graduate Course Impacts	58	The teaching quality of the course is high.
	59	The international performance of the course is relevant.
	60	The regional impact of the course is high.
	61	The national impact of the course is high.
	62	The course is properly coordinated.
	63	The course has an efficient information system (website, emails, media).
Professional Development	64	My national academic production evolved from the learning acquired in the course.
	65	My international academic production evolved from the learning acquired in the course.
	66	The course improved my specific knowledge in my field of expertise.
	67	The completion of the course has been important for my professional career.
Selection Processes	68	The requirements for joining the program are clear.
	69	The selection process is fair.
	70	The criteria for awarding scholarships are clear.
	71	The curriculum structure of the course (research lines, subjects, credits) is adequate to the course goals.
Satisfaction with the Course	72	Course curriculum structure.	0 (completely unsatisfied) to 10 (completely satisfied)
	73	Course faculty competencies.
	74	Course internationalisation.
	75	Regional impacts of the course.
	76	Incentive to Academic Production.
	77	Course Selection Criteria.
	78	Career/labour market preparation.
	79	Overall satisfaction with the advisor.
	80	Overall satisfaction with the course.

Source: Prepared by the authors (2022)

It is also noteworthy that internationalization, a challenge institutionalized both by national public policies and by institutional policies (Heinzle; Pereira, 2023), depends on different actors to be executed (Vázquez; Jiménez García, Canan, 2022). The internationalization creates new opportunities, hastens the implementation of innovative work methods, improves mutual understanding between peoples and cultures, and contributes to the Education of a new generation for work in the global labor Market (Akkari et al ., 2023) was incorporated into the model based on specific items in different dimensions. Questions related to social impact were also incorporated into the model, which are represented by activities with the potential to highlight the link and impact of programs with society, in a virtuous circle between practice, research and professional qualification (Ferraço; Farias, 2021).

3 Method

The present study is characterised as descriptive, carried out through a quantitative approach, from a survey-type research. According to Malhotra (2019), the survey method is based on an interrogation of the participants carried out through a structured questionnaire, with the purpose of eliciting specific information from the interviewees.

The research was carried out with regularly enrolled students and alumni (since 2017) of graduate courses at the Federal University of Santa Maria (UFSM), constituting the population of this study (4,544 students and 7,970 alumni). The entire population received an invitation email from the institution, which resulted in a sample of 2,037 respondents.

As a collection tool, a questionnaire with 85 questions was used and organised into seven blocks. In the first block (“Graduate Courses Infrastructure”), a five-point adequacy scale was used (1– Totally inadequate; 5 – Totally adequate). The other blocks addressed issues related to the professors’ academic and professional quality, the advisor’s relationship and accessibility, the support from the secretariat, the general assessment of the graduate course, and the satisfaction with the graduate course. To measure the responses of these blocks, a five-point Likert scale was used (1 – Totally disagree; 5 – Totally agree). Finally, the last block was intended to characterise the respondents’ profile and collect some general information, with questions about gender, age, marital status, relationship, performance and length of service in the graduate courses.

In the first phase of the quantitative stage, an Exploratory Factor Analysis (EFA) was carried out to validate the proposed constructs using version 23 of the SPSS software (Statistical Package for the Social Sciences). Internal consistency was assessed by calculating Cronbach’s Alpha. As the analysis is exploratory, values equal to or greater than 0,7 were considered acceptable (Hair Junior et al ., 2014). In the second phase of the quantitative stage, Structural Equation Modelling (SEM) was used in order to validate the first-order constructs: “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”, “Satisfaction with the Graduate Course”, and the second-order construct “Course Assessment”. The models were estimated using the Amos Graphics software.

To validate the constructs, the statistical significance of the indices was analysed: Chi-square (χ ² ), Chi-square/degrees of freedom ratio, Comparative Fit Index (CFI), Normed Fit Index (NFI), Tucker-Lewis Index (TLI), Root Mean Square Residual (RMR), Root Mean Square Error of Approximation (RMSEA). For the chi-square/degrees of freedom ratio, the recommendations are for values less than five. For CFI and TLI, values greater than 0,950 are suggested, and the RMR and RMSEA should be below 0,080 and 0,060, respectively (Byrne; 2010; Hair Junior et al. , 2014; Hooper; Coughlan; Mullen; 2008; Kline; 2015).

As for the reliability measures, the coefficients of the Average Variance Extracted (AVE), Composite Reliability and Cronbach’s Alpha were considered. For the AVE, values equal to or greater than 0,5 are desirable (Fornell; Larcker, 1981), with values from 0,7 admitted for the Composite Reliability and Cronbach’s Alpha (Hair Junior et al. , 2014). Discriminant validity was assessed in the nine constructs, comparing the AVE and the correlations. For this, it was necessary to consider the premises (Fornell; Larcker, 1981), in which the correlation of all pairs of the model constructs was verified, comparing it with the square root of the AVE and the orientations of Bagozzi, Yi and Phillips (1991) through the Chi-square difference test. Finally, the estimation of the integrated model occurred through the SEM. In this step, the chi-square indices (χ ² ), the Chi-square/degrees of freedom ratio, CFI, NFI, TLI, RMR and RMSEA were considered in the validation of the integrated model.

4 Results

In order to identify the self-assessment dimensions of the PGCs, and the satisfaction and general assessment of the courses, the EFA was applied, with the varimax rotation method and as an extraction criterion, eigenvalues greater than one were defined. Also, all initial variables showed a commonality greater than 0,5 and were maintained in the analysis. The exploratory factor results of the analyses are presented in Table 2 .

Table 2
Factor analysis results

Variable	Factor loading	Variance	Cronbach’s Alpha
Factor 1 - HEI Infrastructure
Question 9	0,863	60,76	0,918
Question 11	0,828
Question 10	0,825
Question 7	0,790
Question 5	0,787
Question 8	0,743
Question 4	0,740
Question 2	0,722
Question 3	0,691
Question 1	0,656
Question 6	0,646
Factor 2 – Course Subjects
Question 32	0,719	35,83	0,930
Question 31	0,711
Question 28	0,698	35,83	0,930
Question 29	0,685
Question 30	0,671
Question 22	0,671
Question 26	0,669
Question 21	0,639
Question 27	0,628
Question 19	0,612
Question 20	0,596
Question 25	0,584
Question 23	0,558
Question 24	0,550
Factor 3 – Course Professors
Question 16	0,762	31,35	0,901
Question 15	0,735
Question 12	0,720
Question 13	0,702
Question 14	0,696
Question 18	0,676
Question 17	0,617
Factor 4 – Advisor Support
Question 42	0,818	26,83	0,955
Question 43	0,795
Question 41	0,784
Question 47	0,780
Question 48	0,778
Question 50	0,760
Question 49	0,749
Question 44	0,732
Question 40	0,698
Question 45	0,694
Question 46	0,668	26,83	0,955
Question 38	0,609
Question 35	0,567
Factor 5 – Administrative Support
Question 55	0,924	21,47	0,958
Question 52	0,923
Question 54	0,890
Question 51	0,887
Question 53	0,870
Question 56	0,854
Question 57	0,820
Factor 6 – Graduate Course Impacts
Question 69	0,846	23,93	0,879
Question 68	0,787
Question 67	0,729
Question 70	0,606
Question 61	0,599
Question 71	0,531
Factor 7 – Professional Development
Question 62	0,762	20,59	0,775
Question 64	0,759
Question 65	0,700
Question 63	0,683
Factor 8 – Selection Processes
Question 59	0,835	20,1	0,777
Question 58	0,811
Question 60	0,686
Question 66	0,405
Factor 9 – Satisfaction with the Course
Question 80	0,861	62,87	0,918
Question 73	0,854
Question 78	0,827
Question 76	0,811
Question 72	0,808
Question 75	0,802
Question 77	0,767
Question 74	0,765
Question 79	0,591

Source: Prepared by the authors (2022)

Bartlett’s test rejected the null hypothesis that the data matrix is an identity matrix for all dimensions, indicating the data factorability. The factor loadings of the variables range from 0,405 to 0,924, demonstrating that some variables have a greater contribution to the item’s formation. Cronbach’s alphas are adequate, as they all resulted above 0,7. Therefore, the EFA confirms the dimensions predicted in the theoretical model and their respective items.

4.1 Validation of constructs

In order to validate the first-order constructs, the Confirmatory Factor Analysis (CFA) was applied for individual validation of each of the nine constructs. Then, in order to confirm whether the models are adequate, the fit indices were analysed, which are presented in Table 3 .

Table 3
Adjustment indices of first-order constructs

Indices	Limits	HEI Infrastructure		Course Subjects		Course Professors		Advisor Support		Administrative Support		Graduate Course Impacts		Professional Development		Selection Processes		Satisfaction with the Course
		IM	FM	IM	FM	IM	FM	IM	FM	IM	FM	IM	FM	IM	FM	IM	FM	IM	FM
X 2 (value)	-	2297,456	79,003	3162,662	254,244	629,026	37,603	757,859	194,880	1088,604	17,106	702,589	702,589	230,900	0,007	5,261		544,394	72,602
DF/X 2	-	52,215	4,158	41,074	4,889	116,359	4,178	42,429	4,753	77,757	2,138	78,065	78,065	115,450	0,007	2,630		20,163	4,536
CFI	> 0,950	0,837	0,994	0,831	0,988	0,831	0,997	0,887	0,994	0,934	0,999	0,896	0,896	0,915	1,00	0,999		0,956	0,995
TLI	> 0,950	0,796	0,99	0,800	0,983	0,747	0,993	0,864	0,988	0,901	0,999	0,826	0,826	0,745	1,002	0,996		0,941	0,989
RMSR	< 0,080	0,080	0,016	0,049	0,015	0,079	0,009	0,067	0,016	0,026	0,003	0,058	0,058	0,057	0,000	0,008		0,140	0,047
RMSEA	< 0,060	0,159	0,039	0,140	0,044	0,238	0,040	0,143	0,043	0,194	0,024	0,195	0,195	0,237	0,000	0,028		0,097	0,042
CR	> 0,700	....	0,902	....	0,932	...	0,889	...	0,951	...	0,954	......	......	...	0,781	0,794		...	0,918
AVE	> 0,500	....	0,510	....	0,513	...	0,538	...	0,604	...	0,750	......	......	...	0,482	0,500		...	0,560
Alpha	> 0,700	0,918	0,906	0,930	0,934	...	0,901	...	0,955	...	0,958	0,879	0,879	...	0,775	0,777		...	0,918

• IM: Initial Model; FM: Final Model; DF: Degrees of Freedom; CR: Composite Reliability; AVE: Average Variance Extracted.
Source: Prepared by the authors (2022)

Upon verifying the initial models, it can be seen that the values are outside the pre-established standards. In this sense, the necessary procedures were carried out to adjust the indices, adopting two main measures: the removal of variables with a coefficient lower than 0,5 (Hair Junior et al. , 2014) and the insertion of correlations between the variables errors, which were suggested by the software and that made theoretical sense.

For the nine constructs, it is observed that the models initially proposed refer to the model with all the variables of the original scale, and only the “Selection Processes” construct presented an adequate initial model, with no adjustments required. The adjustment indices of the “HEI Infrastructure”, “Course Subjects”, “Graduate Course Impacts”, “Professional Development” constructs corroborate that the model is not adjusted because the division between the Chi-Square and the degrees of freedom is greater than 5, the CFI and TLI are less than 0,95, and the RMSEA is greater than 0,06.

Thus, in order to reach the validity and reliability adjustment indices, the correlations indicated by the software were inserted, in addition to the variables 1, 10, 23, 71 and 79 were removed. After these changes, all adjustments, reliability and unidimensionality indices of the constructs showed satisfactory values.

Once the model adjustment procedures were performed, the Composite Reliability and Cronbach’s Alpha of the constructs were calculated, and all resulted above 0,700, as indicated in the literature (Hair Junior et al. , 2014). It is observed that the convergent validity of the constructs also reached adequate values through the Average Variance Extracted (AVE), which demonstrates the results obtained with values close, equal and higher than 0,5, as suggested by Hair Junior et al. (2014). It is noteworthy that the AVE from the “Professional Development” construct (0,482) does not compromise its convergent validity since the value is close to that suggested in the literature, and the other indices are adequate, indicating the adequacy of the construct.

In order to verify the discriminant validity of the constructs set, Fornell and Lacker’s criteria (1981) were used, in which the correlation of all construct pairs was verified and compared with the square root of the AVE. Table 4 shows the results.

Table 4
Discriminating Validity

	Inf-IES	Dis-C	Do-C	Su-O	Su-A	Im-PPG	DV	Pr-S	Sa-PPG
HEI Infrastructure (Inf-IES)	0,714
Course Subjects (Dis-C)	0,517	0,716
Course Professors (Do-C)	0,500	0,892	0,733
Advisor Support (Su-O)	0,291	0,520	0,625	0,777
Administrative Support (Su-A)	0,273	0,395	0,473	0,337	0,866
Graduate Course Impacts (Im-PPG)	0,478	0,730	0,747	0,515	0,401	0,749
Professional Development (DV)	0,417	0,707	0,693	0,605	0,324	0,754	0,694
Selection Processes (Pr-S)	0,408	0,701	0,668	0,472	0,408	0,684	0,641	0,707
Satisfaction with the Course (Sa-PPG)	0,335	0,582	0,590	0,428	0,284	0,615	0,587	0,530	0,748

Source: Prepared by the authors (2022)

According to the data in Table 4 , it can be seen that all constructs pairs obtained the values of the correlations between the constructs smaller than the square root of the AVE, indicating the discriminant validity. The following relationships were the exceptions: “Course Subjects” – “Course Professors” (0,892), “Course Professors” – “Graduate Course Impacts” (0,747) and “Graduate Course Impacts” – “Professional Development” (0,754), which presented correlation values greater than the AVE. As an alternative procedure to assess the discriminant validity of these constructs, the Chi-square difference test was performed (Bagozzi, Yi and Phillips, 1991). The results of this test indicated differences between the restricted model and the free model greater than 3,84, which suggests the existence of discriminant validity. In addition, it is observed that the correlations between the constructs are less than 0,85, also meeting the criterion proposed by Kline (2015).

After carrying out all the procedures related to the validation and reliability of the nine first-order constructs presented, it is confirmed that the results of the indices reached the proposed parameters, demonstrating that the constructs have convergent validity, reliability and discriminant validity.

4.2 Integrated model analysis

To analyse the integrated model, the first procedure performed was the validation of the second-order construct, called “Course Assessment”. The results of the validation procedure are shown in Table 5 .

Table 5
Adjustment indices of the second-order construct and the integrated model

Indices	Criteria	Course Assessment (second-order construct)		Integrated Model
		IM	FM	IM	FM
x2 (value)	---	162,246	367,400	17465,421	6104,581
x2 (probability)	>0,05	0,000	0,000	0,000	0,000
x2 /degrees of freedom	< 5	28,443	7,654	6,687	2,769
CFI – Comparative Fit Index	> 0,95	0,889	0,977	0,885	0,966
TLI – Tucker-Lewis Index	> 0,95	0,847	0,963	0,878	0,963
RMSR – Root Mean Square Residual	< 0,08	0,069	0,032	0,250	0,044
RMSEA – R.M.S. Error of Approximation	< 0,06	0,116	0,057	0,053	0,029

Source: Prepared by the authors (2022)

Once the second-order construct was validated, the analysis of the integrated model was performed. To verify the validity of the integrated model, analyses of the Chi-square, Chi-square/degrees of freedom, CFI, TLI, RMSR and RMSEA indices were used. The results are also presented in Table 5 .

The proposed model initially presented inadequate adjustment indices. Therefore, the strategy of including correlations between the variables errors that made theoretical sense was adopted. Thus, Figure 2 presents the final integrated model with standardised coefficients and significance of the relationships between the constructs.

Figure 2
Final Model

The relationship evidenced in the theoretical model shown in Figure 2 demonstrates that the assessment of graduate courses is directly influenced by the “HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support” and “Administrative Support” factors. The highest impact is exerted by the course subjects (0,525) and the course professors (0,200). This is in line with studies carried out by De Rezende et al. (2017), who found that the faculty also directly impacts the academic performance of graduate students. They are also consistent with the literature that highlights the role of the teacher by promoting a harmonious and democratic dialogue with students with respect, availability, care and receptivity, (Alves; Espindola; Bianchetti, 2012) and so, directly contributes to a positive self-evaluation of the course.

It can also be seen that the course assessment is formed by the first-order constructs: “Graduate Course Impacts”, “Professional Development” and “Selection Processes”, which have a regression load higher than 0,8, indicating a strong influence on the second-order construct.

Another point to be highlighted from the model results is that the course assessment has a significant influence on the course satisfaction. In other words, there is a causal relationship between two constructs so that better course evaluation results imply a better perception of satisfaction by students and alumni.

It is also necessary to highlight that the variables defended in the projection of the initial model were, for the most part, accepted after the estimations and validity tests, corroborating the adequacy of the proposed model. The results of the factor loadings of each item (observed variable) in relation to its first-order construct are shown in Appendix A.

Finally, the averages were estimated to identify the perception of students and alumni in each of the model dimensions of the model.

For the five antecedents of the course assessment, there were four averages: “HEI Infrastructure” (3,63), “Course Subjects” (4,21), “Course Professors” (4,23), “Advisor Support” (4,35) and “Administrative Support” (4,52). Considering that the scale varies from 1 to 5, it appears that the respondents consider the “HEI Infrastructure” very adequate and that they agree with most of the issues related to Subjects, Professors and Support. Also, for these three constructs: “Graduate Course Impacts” (4,26), “Professional Development” (4,35) and “Selection Processes” (4,32), and for the “Course Assessment” factor (4,31), the averages are slightly higher than 4, stating that students and alumni positively assess the graduate course. Furthermore, it is observed that the “Satisfaction with the Graduate Course” is high, as it reached an average of 8,106 on a scale from zero (totally unsatisfied) to ten (totally satisfied). Therefore, the average results indicate that the institution’s graduate courses are well-assessed, and that students and alumni are satisfied.

Thus, the estimation and validation results presented in this section demonstrate the possibility of using a very comprehensive and complete model for student self-assessment. In line with the self-assessment literature that suggests the need to create models capable of highlighting the complexity, plurality and challenges of self-assessment in university institutions, since it is influenced by structural and situational elements (Lehfeld et al., 2010). Such results are essential because they are associated with the development of students’ skills and their employability (ST Jorre; Oliver, 2018).

5 Final Considerations

Self-assessment as an assessment process has several dimensions and encompasses the perception of different stakeholders, that is, it includes professors, students, the community, and alumni. In this study, the main actor is the student, and the main goal is their self-assessment of the graduate course.

For this, a multidimensional model of self-assessment was developed with five evaluation antecedents (“HEI Infrastructure”, “Course Subjects”, “Course Professors”, “Advisor Support”, “Administrative Support”) and “Satisfaction” is the dimension directly impacted by the course assessment.

In the process of building the model, several validation tests were performed, which analysed and indicated the unidimensionality and the convergent and discriminant validity of all the proposed dimensions. The estimation of the integrated model confirmed the impact of antecedents on the graduate course assessment and its direct influence on satisfaction. Altogether, these results indicated that the proposed self-assessment model is robust and parsimonious.

Since Capes has included self-assessment as one of the assessments of graduate courses, the use of student self-assessment models should be adopted by all programs, so the self-assessment model presented in this article can become useful to several courses. The use of a multidimensional measure allows a broad approach to the different aspects that impact directly on the course assessment and indirectly on the satisfaction of students and alumni. It is also worth noting that the measure as a whole or some of its dimensions can be used to assess other stakeholders in the self-assessment process, such as alumni.

The practice of student self-assessment, by itself, is already a fundamental process for self-regulated and lifelong learning (Yan, 2020), as it leads students to reflect on the gap between current and desirable performance levels (Yan; Carless, 2022). However, Brazil still does not have a culture favorable to the self-assessment process, which is often criticized for inducing behaviors and valuations, either positive and negative, based on indicators that are not always accepted or seen as the most appropriate (Leite et al., 2020).

Thus, the construction of adequate measurement models is just one of the first conditions for the implementation of a self-assessment policy capable of effectively indicating the strengths and possible improvements to graduate courses managers. Just with the development of a continuous self-assessment policy that promotes effective feedback a country’s graduate system evolves. Thus, in addition to choosing an appropriate model, it is necessary for graduate courses to establish their own self-assessment policy, taking into account both the rules established by Capes and their own criteria.

As for the research limitations, we can mention the application of the questionnaire via the Internet and the fact that the research was carried out only at one university. It is suggested that future works use other samples and analyse the possible differences arising from the course level (master’s and doctorate). New scale validation tests can also be performed.

References

• AKKARI, A. et al. Internationalization of higher education in Kazakhstan: from political will to implementation. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 31, n. 119, e0223730, apr./jun. 2023. https://doi.org/10.1590/S0104-40362023003103730
  » https://doi.org/10.1590/S0104-40362023003103730
• ALVES, V. M.; ESPINDOLA, I. C. P.; BIANCHETTI, L. A relação orientador-orientando na Pós-graduação stricto sensu no Brasil: a autonomia dos discentes em discussão. Revista Educação em Questão, Natal, v. 43, n. 29, p. 135-156, maio/ago. 2012.
• ANGST, F. A. Dispositivos de regulação do processo de (auto) avaliação de qualidade do ensino superior em Moçambique. Revista Electrónica de Investigação e Desenvolvimento, Ponta-Gêa, v. 1, n. 7, p. 117-151, 2017.
• ANGST, F..; ALVES, J. M. (Auto) avaliação de qualidade do ensino superior em Moçambique: um estudo de caso. Revista Portuguesa de Investigação Educacional, Braga, n. esp., p. 117-151, 2018. https://doi.org/10.34632/investigacaoeducacional.2018.3448
  » https://doi.org/10.34632/investigacaoeducacional.2018.3448
• ASTAKHOVA, K. V. et al. The role of education in economic and social development of the country. International Review of Management and Marketing, v. 6, n.1,p.53-58, 2016.
• BAGOZZI, R. P.; YI, Y.; PHILLIPS, L. W. Assessing construct validity in organizational research. Administrative Science Quarterly, Ithaca, v. 36, p. 421-458, Sep. 1991. https://doi.org/10.2307/2393203
  » https://doi.org/10.2307/2393203
• BISPO, M. S.; COSTA, F. J. Artigos como avaliação discente em disciplinas de pós-graduação: instrumento educativo ou subsistema de linha de montagem?. Cadernos EBAPE. BR, Rio de Janeiro, v. 14, n. 14, p. 1001-1010, 2016. https://doi.org/10.1590/1679-395141927
  » https://doi.org/10.1590/1679-395141927
• BRASIL. Ministério da Educação. Mudanças na ficha de avaliação valorizam a pós-graduação. Brasília, DF, 2019. Available from: http://portal.mec.gov.br/component/content/index.php?option=com_content&view=article&id=73911:mudancas-na-ficha-de-avaliacao-valorizam-a-pos-graduacao&catid=225&Itemid=86 Access in: 2021 June 19.
  » http://portal.mec.gov.br/component/content/index.php?option=com_content&view=article&id=73911:mudancas-na-ficha-de-avaliacao-valorizam-a-pos-graduacao&catid=225&Itemid=86
• BYRNE, B. M. Structural equation modeling with AMOS: basic concepts, applications, and programming. 2. ed. [S. l.]: Routledge, 2010.
• CABALLERO, G.; ÁLVAREZ-GONZÁLEZ, P.; LÓPEZ-MIGUENS, M. J. Which are the predictors of perceived employability? An approach based on three studies. Assessment & Evaluation in Higher Education, [s. l.], v. 47, n. 6, p. 878-895, 2022. https://doi.org/10.1080/02602938.2021.1983769
  » https://doi.org/10.1080/02602938.2021.1983769
• CAPES – Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Grupo de Trabalho sobre a Autoavaliação de Programas de Pós-Graduação Brasília, DF, 2018. Disponível em: https://www.gov.br/capes/pt-br/acesso-a-informacao/institucional Acesso em: 8 jun. 2021.
  » https://www.gov.br/capes/pt-br/acesso-a-informacao/institucional
• CAPES – Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Autoavaliação de programas de pós-graduação: grupo de trabalho. Brasília, DF, 2019. Disponível em: https://www.gov.br/capes/pt-br/centrais-de- conteudo/10062019-autoavaliacao-de-programas-de-pos-graduacao-pdf Acesso em: 9 nov.2021
  » https://www.gov.br/capes/pt-br/centrais-de- conteudo/10062019-autoavaliacao-de-programas-de-pos-graduacao-pdf
• CAPES – Coordenação de Aperfeiçoamento de Pessoal de Nível Superior. Sobre a avaliação. Brasília, DF, 2014. Disponível em: https://www.gov.br/capes/pt-br/acesso-a-informacao/acoes-e-programas/avaliacao/sobre-a-avaliacao/avaliacao-o-que-e/sobre-a-avaliacao-conceitos-processos-e-normas/conceito-avaliacao Acesso em: 19 jun. 2021
  » https://www.gov.br/capes/pt-br/acesso-a-informacao/acoes-e-programas/avaliacao/sobre-a-avaliacao/avaliacao-o-que-e/sobre-a-avaliacao-conceitos-processos-e-normas/conceito-avaliacao
• FERRAÇO, C. E.; FARIAS, I. M. S. Inserção social: em busca de sentidos e de indicadores para a avaliação da pós-graduação na área da Educação. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 29, p. 420-440, abr./jun. 2021. https://doi.org/10.1590/S0104-40362020002802413
  » https://doi.org/10.1590/S0104-40362020002802413
• FORNELL, C.; LARCKER, D. F. Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, Chicago, v. 18, n. 1, p. 39-50, Feb. 1981. https://doi.org/10.1177/002224378101800104
  » https://doi.org/10.1177/002224378101800104
• HAIR JUNIOR, J. F. et al. Multivariate data analysis (Pearson Education Limited), Upper Saddle River: Prentice-Hall, 2014
• HEINZLE, M. R. S.; PEREIRA, P. Políticas de internacionalização em universidades fundacionais: produção intelectual, intercâmbio, currículo e internacionalização integral. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 31, e0233354, abr./jun. 2023. https://doi.org/10.1590/S0104-40362023003103354
  » https://doi.org/10.1590/S0104-40362023003103354
• HOOPER, D.; COUGHLAN, J.; MULLEN, M.R. Structural equation modelling: Guidelines for determining model fit. Electronic Journal of Business Research Methods, [s. l.], v. 6, n. 1, p.53-60, 2008.
• KLINE, R. B. Principles and practice of structural equation modelling New York: Guilford, 2015. (Methodology in the Social Sciences).
• LEAL, F. Estratégias de estudo e de aprendizagem de estudantes de Ensino Superior. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 31, e0233349, apr./jun. 2023. https://doi.org/10.1590/S0104-40362023003103349
  » https://doi.org/10.1590/S0104-40362023003103349
• LEHFELD, N. A. S. et al. Reflexões sobre o processo de autoavaliação institucional: o olhar de uma comissão própria de avaliação. Avaliação: Revista da Avaliação da Educação Superior (Campinas), v. 15, n. 1, p. 177-194, 2010. https://doi.org/10.1590/S1414-40772010000100010
  » https://doi.org/10.1590/S1414-40772010000100010
• LEITE, D. Ameaças pós-rankings sobrevivência das CPAS e da auto-avaliação. Avaliação: Revista da Avaliação da Educação Superior (Campinas), v. 13, n. 3, p. 833-840, set./dez. 2008.
• LEITE, D. et al. A autoavaliação na Pós-Graduação (PG) como componente do processo avaliativo CAPES. Avaliação: Revista da Avaliação da Educação Superior (Campinas), v. 25, p. 339-353, 2020. https://doi.org/10.1590/S1414-4077/S1414-40772020000200006
  » https://doi.org/10.1590/S1414-4077/S1414-40772020000200006
• MALHOTRA, N. K. Pesquisa de marketing: uma orientação aplicada. Porto Alegre: Bookman, 2019.
• MARQUES, C. A.; VEIGA, A. C.; BORGES, L. M. C. A avaliação da pós-graduação no Brasil: resultados e determinantes da avaliação da Capes (2013-2016). Meta: Avaliação, Rio de Janeiro, v. 12, n. 37, p. 979-1003, 2020. https://doi.org/10.22347/2175-2753v12i37.2825
  » https://doi.org/10.22347/2175-2753v12i37.2825
• MARTINEZ, E.; ELUE, C. From community college to graduate school: Exploring the role of academic advisors in promoting graduate education at baccalaureate degree-granting community colleges. The Journal of Higher Education, [s. l.], v. 91, n. 7, p. 1003-1027, Feb. 2020. https://doi.org/10.1080/00221546.2020.1732176
  » https://doi.org/10.1080/00221546.2020.1732176
• Meurer et al. 2020. Sentimentos assumidos pelos orientandos nas fases de orientação das dissecções em contabilidade. Revista Contabilidade & Finanças, São Paulo, v. 32, n. 85, p. 158-173, Jan./Apr. 2020. https://doi.org/10.1590/1808-057x202010550
  » https://doi.org/10.1590/1808-057x202010550
• MONTICELLI, N. A. M. et al. Avaliação institucional e gestão estratégica-vínculos necessários para o desenvolvimento institucional. Avaliação: Revista da Avaliação da Educação Superior (Campinas), v. 26, p. 315-342, jan./abr. 2021. https://doi.org/10.1590/S1414-40772021000100017
  » https://doi.org/10.1590/S1414-40772021000100017
• MOORE, N. J. Mentoring for advising: developing relationships between academic advisors and advisees. The Chronicle of Mentoring & Coaching, v. 2, n. esp. 10, Nov., 2017.
• MYALKINA, E. V. Diagnostics of the education quality in the higher educational institution. Vestnik of Minin University, Chelyuskintsev, v. 7, n. 3, p. 4, 2019. https://doi.org/10.26795/2307-1281-2019-7-3-4
  » https://doi.org/10.26795/2307-1281-2019-7-3-4
• REZENDE, .M. S. et al. Stress e desempenho acadêmico na pós-graduação stricto sensu em ciências contábeis no Brasil. Education Policy Analysis Archives, [s. l.], v. 25, p. 96-96, 2017. https://doi.org/10.14507/epaa.25.2931
  » https://doi.org/10.14507/epaa.25.2931
• SEIFERT, T.; FELIKS, O. Online self-assessment and peer-assessment as a tool to enhance student-teachers’ assessment skills. Assessment & Evaluation in Higher Education, Abingdon, v. 44, n. 2, p. 169-185, 2018. https://doi.org/10.1080/02602938.2018.1487023
  » https://doi.org/10.1080/02602938.2018.1487023
• ST JORRE, T. J.; OLIVER, B. Want students to engage? Contextualise graduate learning outcomes and assess for employability. Higher Education Research & Development, Sydney, v. 37, n. 1, p. 44-57, 2018. https://doi.org/10.1080/07294360.2017.1339183
  » https://doi.org/10.1080/07294360.2017.1339183
• VÁZQUEZ, J. M.; JIMÉNEZ GARCÍA, S. A. J.; CANAN, S. R. La política de internacionalización de la Educación Superior. Efectos, brechas y asimetrías persistentes. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 30, n. 117, p. 1047-1068, out./dez. 2022. https://doi.org/10.1590/S0104-403620220003002939
  » https://doi.org/10.1590/S0104-403620220003002939
• VIEIRA, K. M. et al. Autoavaliação discente: avaliando a relação orientador-orientando e a satisfação com o curso. Revista Pretexto, Belo Horizonte, v. 23, n.3, p. 63-81, jul./dez. 2022.
• YAN, Z. Self-assessment in the process of self-regulated learning and its relationship with academic achievement. Assessment & Evaluation in Higher Education, Abingdon, v. 45, n. 2, p. 224-238, 2020. https://doi.org/10.1080/02602938.2019.1629390
  » https://doi.org/10.1080/02602938.2019.1629390
• YAN, Zi; CARLESS, David. Self-assessment is about more than self: the enabling role of feedback literacy. Assessment & Evaluation in Higher Education, v. 47, n. 7, p. 1116-1128, 2022. https://doi.org/10.1080/02602938.2021.2001431
  » https://doi.org/10.1080/02602938.2021.2001431
• ZAINKO, M. A. S.; PINTO, M. L. A. T. Gestão da instituição de ensino e ação docente Curitiba: Ibpex, 2008.

• Data, Instrument and Appendix A: The entire data set, the description of the instrument items and Appendix A are published on Scielo Data at the link https://doi.org/10.48331/scielodata.RHGLG9

Data availability

Data, Instrument and Appendix A: The entire data set, the description of the instrument items and Appendix A are published on Scielo Data at the link https://doi.org/10.48331/scielodata.RHGLG9

Publication Dates

• Publication in this collection
  23 Feb 2024
• Date of issue
  Jan 2024

History

• Received
  22 Oct 2022
• Accepted
  22 Nov 2023