Reflections on the Insertion of the Self-Evaluation Policy in Brazilian Postgraduate Studies: The Experts' View

Abstract

Recent changes in the assessment model used by the Coordination for the Improvement of Higher Education Personnel (CAPES) assume, among other things, the consideration of the need for new assessment methods that include a more active involvement of the academic community. As of 2018, Self-Assessment has become the fundamental structure within this model. This article aims to understand the integration of the Self-Assessment Policy in the CAPES Assessment System, reflecting on the possible interactions and implications that arise in this context. It is a theoretical-empirical article with a qualitative approach, using documental research and interviews with experts who were part of the CAPES Self-Assessment Working Group (WG). The results shed light on the antecedents, conceptual and motivational aspects of implementing this practice, as well as suggest other potential connections, such as learning, innovation, and elements shaping an emerging context. The research provides preliminary insights for discussions on this practice within the Brazilian graduate studies landscape, offering a fertile ground to inspire further research endeavors.

Keywords:
educational assessment; self-assessment policy; brazilian postgraduate assessment system

Resumo

As recentes transformações no modelo de avaliação empregado pela Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES) presumem, entre outros sentidos, a ponderação sobre a necessidade de novas formas avaliativas que contemplem um envolvimento mais ativo da comunidade acadêmica. Desse modo, a partir de 2018, a autoavaliação passou a compor a estrutura fundamental dentro desse modelo. Nessa seara, este artigo objetiva compreender a inserção da Política de Autoavaliação no Sistema de Avaliação da CAPES, refletindo acerca das possíveis interlocuções e significados que emergem nesse cenário. Trata-se de um artigo teórico-empírico, de abordagem qualitativa, que recorre à pesquisa documental e entrevistas com especialistas que atuaram no Grupo de Trabalho (GT) de Autoavaliação da CAPES. Os resultados possibilitam apreender sobre antecedentes, aspectos conceituais e motivacionais da implantação dessa prática, bem como depreender outras possíveis articulações, como aprendizagem e inovação e elementos que configuram um contexto emergente. A pesquisa promove contribuições, ainda que introdutórias, para as discussões sobre essa prática no contexto da pós-graduação brasileira, constatando um campo fértil para estimular outras pesquisas.

Palavras-chave:
avaliação educacional; política de autoavaliação; sistema de avaliação da pósgraduação brasileira

Resumen

Las recientes transformaciones en el modelo de evaluación utilizado por la Coordinación de Perfeccionamiento del Personal de Educación Superior (CAPES) suponen, entre otras cosas, considerar la necesidad de nuevas formas de evaluación que incluyan una participación más activa de la comunidad académica. Por ello, a partir de 2018 la Autoevaluación comenzó a constituir la estructura fundamental dentro de este modelo. En este ámbito, este artículo tiene como objetivo comprender la inserción de la Política de Autoevaluación en el Sistema de Evaluación CAPES, reflexionando sobre los posibles diálogos y significados que emergen en ese escenario. Se trata de un artículo teórico-empírico, con enfoque cualitativo, que utiliza investigación documental y entrevistas con expertos que trabajaron en el Grupo de Trabajo de Autoevaluación (GT) de la CAPES. Los resultados permiten conocer los antecedentes, aspectos conceptuales y motivacionales de la implementación de esta práctica, así como comprender otras posibles articulaciones, como el aprendizaje y la innovación y elementos que configuran un contexto emergente. La investigación promueve contribuciones, aunque sean introductorias, a las discusiones sobre esta práctica en el contexto de los estudios de posgrado brasilehos, estableciendo un campo fértil para estimular otras investigaciones.

Palavras clave:
evaluación educativa; política de autoevaluación; sistema brasileho de evaluación de posgrado

1 Introduction

Over the past forty years, various incremental changes have been made to the Brazilian Postgraduate Assessment System (PG), systematically conducted by the Coordination of Higher Education Personnel (CAPES), with the aim of continuous improvement. These transformations aim to ensure the qualified development and strengthening of the National Postgraduate System (SNPG) through the creation of new courses and Postgraduate Programs (PPG) (Vogel, 2015; Ferreira; Ferenc; Wassem, 2018). Currently, there are 4,559 Programs¹, and progress has been made in the production of scientific and technological knowledge (Magalhães; Real, 2018).

In the historical development of the field of Educational Evaluation, there have been significant efforts to democratize public decisions embedded in "structural changes in the conception and positioning of evaluation, reflecting its multifaceted nature and the diverse interests surrounding its practices" (Silva; Gomes, 2018, p. 352). Concerning the CAPES Evaluation System, various discussions and concerns about distortions and inconsistencies in external evaluation, driven in part by the necessity for programs to be acknowledged, appreciated, and assessed based on their unique characteristics (Magalhães; Real, 2018; Patrus, 2018; Barata, 2019), have prompted considerations for alternative approaches and evaluation models that would ensure increased engagement of the academic community, not only as evaluators but also as active participants in the external evaluation process (Leite et al., 2020).

The recent changes involve, in particular, the inclusion, since 2018, of self-evaluation as a component of external evaluation (CAPES, 2019a; Leite et al., 2020). This means that self-evaluation will gradually become a significant element in the evaluation of Brazilian graduate studies, with programs being required to implement a self-evaluation policy and carry out a systematic process of internal analysis involving different organizational actors (Barata, 2019; Leite et al., 2020).

Although it is only one of the items evaluated in the "Program" section of the new Evaluation Form, accounting for 10% of the score (CAPES, 2020), it is a complex practice. Its purpose is directed towards a self-managed process of analysis, evaluation, and internal reflection aimed at improving the program (CAPES, 2019b). In this process, the individuals involved assume roles both evaluated and evaluating agents within their sphere of action, with implications for the overall political and managerial framework of a PPG.

Studies on self-evaluation in graduate studies (Saul, 2002; Hortale; Moreira, 2008) have already addressed the systematic nature of this evaluation mechanism within graduate programs. They highlight it as a tool for reflecting on teaching and research activities for program development, as well as for understanding the characteristics and limitations of this practice to facilitate critical analysis and decisionmaking. However, these investigations did not consider CAPES' regulatory aspects concerning the necessity for programs to conduct internal evaluations, indicating a significant theoretical gap that needs to be addressed.

More recently, Leite et al. (2020) conducted a study on "Self-evaluation in Postgraduate Studies (PG) as a component of the CAPES evaluation process", presenting various perspectives on the significance of self-evaluation as a tool for improving the Evaluation System, the appreciation of democratic practices, and the agenda for improvements in the context of PG. According to the authors, "self-evaluation becomes an entrepreneurial act when, following discussion and analysis, it is accompanied by actions for enhancement" (p. 348). In an effort to enrich the initial dialogues on self-evaluation within Brazilian graduate studies, this article seeks to explore the integration of the Self-Evaluation Policy into the CAPES Evaluation System, contemplating the potential interactions and implications arising from this framework.

The discussion is structured, in addition to this introduction, in four sections: the second part presents the theoretical aspects of Educational Evaluation, Self-Evaluation Policy, and their relationship with the PG scenario; the third part discusses the methodological aspects of the research; the reflections based on the evidence then constructed; and in the final section, the final considerations are presented.

2 Educational Evaluation and Self-Evaluation Policy: Movements and Regulation in the CAPES Evaluation System

In the Brazilian educational context, evaluation policies are based on two main trends, according to Belloni (1999): a) Control and hierarchization, where the emphasis is on regulation and control to ensure compliance with requirements and norms. This approach aims to create rankings by selecting the "best" institutions through performance analysis, establishing standards of excellence, and emphasizing meritocratic aspects; and b) Improvement and change, guided by a transformative perspective that seeks to enhance the scientific and social effectiveness of institutions. This involves identifying successes and challenges to improve institutional quality and excellence.

The evaluation of higher education at the undergraduate level has a brief history of success, although it is marked by various criticisms. At the postgraduate level, however, despite its long and consolidated history, there is a recurring issue concerning the evaluation model (Leite et al., 2020). According to the literature, CAPES' external evaluation has been carried out satisfactorily in terms of regulation, receiving different criticisms from the scientific community (Alves; Oliveira, 2014; Vogel, 2015; Magalhães; Real, 2018; Patrus, 2018; Barata, 2019).

However, considering the maturity of the evaluation model throughout the history of the SNPG and the tensions caused by this community, CAPES corroborated the need, among other changes, to include self-evaluation in the Evaluation System (CAPES, 2019b; Leite et al., 2020). It was recognized that external evaluation is not formative (Patrus; Shigaki; Dantas, 2018; Maldonado; Bitencourt, 2019), even if it ensures answers to the basic parameters.

This change, in particular, contributes to overcoming the concept of summative evaluation, which results in the generation of rankings based on essentially quantitative indicators. This approach goes against what Leite et al. (2020) argue for when they advocate for the implementation of other autonomous and formative practices that actively engage the academic community at various levels. While self-evaluation experiences already exist in higher education, particularly within the framework of the National Higher Education Evaluation System (SINAES) (Griboski; Peixoto; Hora, 2018), this aspect was not observed in the PG evaluation process (Leite et al., 2020). There is, therefore, progress in this direction, which may even be a component of autonomy (Leite; Pinho, 2017) for stricto sensu PPGs.

Thus, programs are required to define their self-evaluation policies, procedures, and uses, understanding it as an evaluation process of internal reflection, planned and self-managed by the subjects that make it up, involving actors at different levels (CAPES, 2019b). It is assumed that self-evaluation is associated with academic customs that uphold the principles of autonomy, collegiality, and institutional democracy. This implies that, besides encouraging and acknowledging the unique qualities and excellence based on the distinctions of each program (Leite, 2006), self-evaluation as a policy implemented in the PPGs can promote a "[...] reorientation towards expanding the culture of diversity, inclusion, and emerging contexts in postgraduate education" (Leite et al., 2020, p. 347).

This is because, in the way it was set up, the CAPES Evaluation System takes little account of the particularities of the programs and the emerging contexts in the external evaluation process (Verhine; Dantas, 2009; Patrus, 2018), imposing requirements and parameters that shape an environment capable of subjecting organizations to conditions of uniformity, limiting their performance in the face of their own identities (Andrade et al., 2018). On the other hand, the implementation of self-evaluation strengthens conceptions of exercising democracy through evaluation, presupposing an effort to defend the quality represented in the distinctions between PPGs (Belloni, 1999; Leite, 2006). In this sense, it is considered that:

A well-executed self-evaluation leads to an understanding of a specific reality as perceived by those who are connected to it and shape it, within a specific place, context, and historical period. The generation of this understanding, the act of acquiring knowledge, is inherently a social, professional, and public obligation of the program or institution (Leite et al., 2020, p. 343).

They also emphasize that reflection on the results obtained is a central element in conducting the process (Leite et al., 2020). Thus, the operationalization of a self-evaluation process enables the construction of in-depth knowledge about the programs that can enable more assertive actions. In other words, "self-evaluation should result in decision-making that ultimately implies changes" (CAPES, 2019b, p. 08). Evaluation "should be used by both evaluators and those being evaluated to make decisions, rather than being merely a formal and technical act to be shelved" (Andriola; Souza, 2010, p. 47).

Along these lines, we must not lose sight of the fact that the purpose of evaluation goes beyond the institutional sphere. It is understood as a strategic device to promote an effective relationship between the educational institution and the social reality, reaffirming the commitment to the reconstruction of the social space and the democratization of education (Belloni, 1999). Therefore, self-evaluation can be a fundamental process for the PPGs to provide the evaluation committee with a diagnosis of their contexts and trajectories (Barata, 2019). Furthermore, it can be "[...] a moment of pause to reflect, analyze, and move towards innovative future projects" (Leite et al., 2020, p. 351).

3 Methodological Choices

In light of the proposal to integrate the Self-Evaluation Policy into the CAPES Evaluation System and to reflect on the potential interactions and implications arising in this context, the interpretivist paradigm (Burrell; Morgan, 1979) was adopted to explore the phenomenon under study by valuing the meanings ascribed by the research participants and the analysis of documents, taking into account the subjective elements of this process.

The research adopts a qualitative approach and aims to describe and interpret a phenomenon. The reflections were produced and stimulated by "[...] exploring people's experiences and their views and perspectives on these experiences" (Gray, 2012, p. 38), in order to highlight the way in which things are related (Marconi; Lakatos, 2007). Reflections were constructed based on the perceptions of experts in educational assessment, who are representatives of the CAPES Self-Assessment Working Group, and through the interpretation of documents.

In order to select the research subjects and conduct the interviews, the criterion of availability was considered. Ten members of the CAPES Self-Assessment Working Group were contacted via email based on the information available on the Sucupira Platform, and two of them responded positively. Consequently, two interviews were carried out using the Google Meet tool, following the principles of online social research (Flick, 2013). The interviews were recorded with the subjects' consent, later transcribed, and analyzed. The protection of the interviewees and ethical procedures of the research were ensured through the signing of the Informed Consent Form (ICF) by the subjects and the approval of the research project, to which this study is related, by the Research Ethics Committee (CEP) - Certificate of Submission for Ethical Appraisal (CAAE) No. 40448520.3.0000.5546. In this study, the subjects were identified as Interviewee A and Interviewee B.

With regard to documents, we accessed the Report of the Working Group on Self-Assessment of Graduate Programs and the Final Document of the National Commission for Monitoring the PNPG 2011-2020, which contributed to constructing the narrative about the inclusion of self-assessment in Brazilian graduate programs.

Finally, the Content Analysis technique was used to analyze the evidence, following the three stages established by Bardin (2016): pre-analysis, exploration of the material, and treatment of the results. The goal was to enable a systematic analysis. Initially, the documentary records and transcribed interviews were selected and processed to create the research analysis corpus, followed by a floating reading. Subsequently, the materials were explored to define thematic axes or analytical categories. Lastly, the evidence was described and analyzed (inference and interpretation), leading to the discussion presented in the following section of this article. The analysis used the categories of Background, Self-evaluation Concept and Policy, Motivations, Learning and Innovation, and Emerging Context. It is important to note that these categories emerged from an iterative process of returning to the field, as recommended by the analysis process.

4 Insertion of the Self-Evaluation Policy in Brazilian Postgraduate Studies from the perspective of experts

Subdivided into two parts, this section presents the construction of the narrative based on the evidence, covering the five analytical categories mentioned above.

4.1 Understanding the Self-Evaluation Policy in the Brazilian Postgraduate Scenario: Background, Concept, and Motivations

a) Background

The rationale behind incorporating self-evaluation in the postgraduate stricto sensu is linked to the evaluation practices of Brazilian higher education, primarily derived from SINAES, which has significant experience in institutional evaluation and self-assessment as explained by Interviewee A. SINAES developed due to advancements in educational assessment, with self-assessment being one of the key aspects. This self-assessment is carried out by the Assessment Committees (CPA) of each Higher Education Institution (HEI) in alignment with the Institutional Assessment guidelines and standards set by the National Commission for Higher Education Assessment (CONAES) (Griboski; Peixoto; Hora, 2018).

Interviewee B recalled that these experiences evolved from the practices that originated in the Program for Institutional Assessment of Brazilian Universities (PAIUB) in the 1990s. Interviewee B stated, "[...] it was a Program created within the universities and the universities, the deans, entities, and us, the professors involved in assessment, we created the Program considering the engagement of the actors, the participation of the actors". In other words, it aimed to involve all actors and promote their participation in building a collective, reliable, adjustable, and transparent evaluation. The focus was on qualitative evaluation and self-assessment within the Higher Education Institutions (HEIs) (Griboski; Peixoto; Hora, 2018).

Throughout 2012 and 2013, CONAES organized a series of seminars on self-evaluation to highlight the importance of this process in higher education, particularly in undergraduate programs, due to certain identified weaknesses. According to Interviewee A, "ENADE [National Student Performance Exam] worked, these visits worked, but institutional assessment, institutional self-assessment, was the weak point and should be the central point".

This is because institutional evaluation, implemented in HEIs, is not limited to itself but should be seen as an important part of a set of public policies that contribute to a broader process aimed at revalorizing higher education and Brazilian social development (Falleiros; Pimenta; Valadão Júnior, 2016).

In this context, a way of transferring this expertise was through professionals specialized in Educational Evaluation, such as Interviewees A and B and their colleagues. They transitioned between evaluating higher education at undergraduate and postgraduate levels, focusing particularly on working within the Evaluation System and/or the CAPES Self-Evaluation Working Group.

In 2015, CAPES established a commission to reconsider its external evaluation process. Recognizing the deficiencies of this process in light of numerous criticisms from the academic community and inspired by the successful practices observed in SINAES, the Commission started advocating for self-evaluation. As stated by Interviewee A,

[...] in 2015, I was invited to participate in a commission created by CAPES to reassess its evaluation process. In that Commission [...] we thought 'Why don't we encourage this self-assessment concept?' [...]. And we... [...] implemented this concept [which] later was incorporated into the commission's final report completed in 2016.

The Commission's final report was presented and considered by the Ministry of Education (MEC). However, due to the political scenario at the time², these results were lost amidst the political reorganization in the realm of government action in the public sphere.

Subsequently, in 2018, with the change in the Evaluation Directorate and the presidency of CAPES' Technical-Scientific Council for Higher Education (CTC), space was opened up for professionals and specialists to resume discussions on self-evaluation, considering the vision already established in previous instances. It was during this period that CAPES established, among other working groups, the Self-Evaluation Working Group, comprising a team of ten professionals with expertise in Educational Evaluation, including Interviewees A and B.

Along these lines, the working group (WG) focused on developing a study aimed at enhancing self-evaluation concepts and guidelines. To achieve this goal, various researchers and experts proposed improvements during seminars, which were subsequently reviewed during the group's meetings. Additionally, with the support of CAPES, some coordinators visited Germany and the Netherlands to study international practices in postgraduate assessment, mentioned by Interviewee B.

The culmination of these efforts resulted in the drafting of a document emphasizing the significance of incorporating self-assessment as a policy within the PPGs. Given the context of changes in the CAPES Evaluation System and the ongoing efforts of other WGs in various areas, such as restructuring the Evaluation Form, there was a consensus to integrate this aspect into the new form. In essence, as stated by Interviewee B, "[...] there were numerous challenges to overcome before we reached the point where the self-assessment was included in the form".

A review of the final report of the Self-Evaluation Working Group revealed arguments supporting the inclusion of self-evaluation as a component of external evaluation, as outlined in the following excerpts:

[...] with the maturing of its evaluation process, CAPES believes it is necessary to broaden its focus. Recognizing that self-assessment, now widely used in international experiences, can provide more support for the development of the System with quality (CAPES, 2019b, p. 4-5).

External evaluation undoubtedly ensures basic standards, which is important in a continental country, but it has limitations. One of them is the fact that it is not formative, meaning that those involved in the process are not engaged in solving identified problems (CAPES, 2019b, p. 5).

Thus, self-evaluation, developed systematically and continuously, is the approach to be emphasized, as it ensures proximity between the evaluator and the evaluated, allowing for qualitative and contextualized in-depth studies (CAPES, 2019b, p. 05).

In parallel to the Working Group's activities, as the PNPG 2011-2020 (CAPES, 2010) was being finalized, the committee responsible for monitoring it also had the task of proposing improvements to the Evaluation Form and the evaluation process. Various meetings were held, and the committee received suggestions from organizations such as the National Association of Directors of Federal Higher Education Institutions (ANDIFES), the National Union of Teachers of Higher Education Institutions (ANDES), among others. As Interviewee B pointed out, "[...] oddly enough, all [the suggestions] mentioned self-evaluation". In view of this, the commission responsible for monitoring the PNPG also produced a study that resulted in a document approved by CAPES that same year, which also promoted the idea of self-evaluation.

Accessing the Final Document of the National Commission for Monitoring the PNPG 2011-2020, which contains the proposal to improve the PG evaluation model, we identified self-assessment as a key theme suggested by at least 75% of the consulted entities. This is evident in the following excerpt:

The postgraduate evaluation system should use program self-assessment as a fundamental aspect of the evaluation process. Although CAPES currently encourages program self-evaluations, they are not officially required, not integrated into the formal evaluation process, and lack support from guidelines provided externally by CAPES or any other specialized entity. It is therefore recommended that self-evaluation be conducted regularly and that programs be provided with guidance and tools to facilitate successful implementation (CAPES, 2018, p. 19).

It should be noted that even before these discussions, there was already a section in the Sucupira Report designated for including information about self-evaluation. However, Interviewee A reflected that the programs, in general, provided results that were not well-organized, lacking in reflection, and merely fulfilling a requirement to complete this section in the report. In contrast, as noted in the literature, the primary perspective advocated by the agency is the establishment of a self-assessment policy as an internal tool improving each program individually (CAPES, 2019b; Leite et al., 2020). According to Interviewee A,

we are not interested in the results; rather, we want to know if the program has a policy, a self-evaluation project, and the ability to conceptualize self-evaluation. They should be able to define their vision of quality, identify the instruments and procedures they will use, determine the frequency of evaluations, and understand the results and their implications [...].

In this way, the evidence points to an understanding that the idea is, on the one hand, to assess whether there is an institutionalized policy of self-evaluation within the PPGs dealing with these elements, and, on the other hand, to promote advances in line with best practices in Educational Evaluation through the more active participation of the academic community (Leite et al., 2020). In other words, as an item to be judged in the external evaluation of the PG, the self-assessment process seeks to enhance the qualitative character, where the transparency and consistency of aspects about the policies, procedures, and uses of self-assessment implemented by each program will be analyzed (CAPES, 2020).

This pertains to conceptions regarding the fundamentals of democratic evaluation practices (Leite, 2006) and contemplation on the establishment of an evaluation process where the evaluator and the evaluated collaborate in pursuit of qualitative changes (Andriola; Souza, 2010).

b) Self-evaluation Concept and Policy

The concept of self-evaluation observed in the evidence aligns with the meanings found in the literature. The perception of self-evaluation is rooted in the understanding of internal evaluation as a process established by the organization itself be it an entity, institution, program, or course which may involve various levels of evaluation, "[...] with the aim of identifying problems and identifying solutions and monitoring these solutions to see if they are having the desired effect" (Interviewee A), or, in other words, to enable the "[...] improvement of the quality of its institutional work" (Leite, 2006, p. 466). In essence, self-evaluation is embraced as a self-managed process of internal reflection that should lead to the generation of knowledge about the program and its various aspects, enabling the recognition of weaknesses, areas for improvement, and proposals for enhancement from the viewpoint of organizational actors (CAPES, 2019b; Leite et al., 2020; Interviewees A and B).

The evidence highlights several key aspects to consider during the implementation of the self-assessment policy. One crucial aspect is that programs need to develop their self-assessment policies, establish committees to oversee the self-assessment process, and define procedures, tools, and discussions regarding the outcomes (CAPES, 2019b). It is essential to establish close communication with the Postgraduate (PG) deaneries to align institutional policies and the Higher Education Institutions' (HEIs) Academic Assessment Committees (CPAs), leveraging their extensive experience in conducting evaluation processes at an institutional level, as emphasized in Interviewee A's statement.

This CPA, who has been at your institution since 2004 or 2005, already has a lot of experience in self-evaluation at the institutional level. However, we need to extend this practice to the programs and courses as well. Typically, CPAs focus on self-evaluation at the institutional level and may not engage in self-evaluation at the course level. Therefore, fostering interaction between the dean's office and the CPA is also very important [...].

Similarly, it is important to share experiences with other PPGs, especially those that are closer to the subject, such as the field of Administration, as exemplified by Interviewee B, and thus have expertise to share.

Furthermore, it is acknowledged the significance of establishing a participatory process that involves both internal and external actors in order to incorporate diverse perspectives. Besides using these organizations as a reference point, we emphasize the crucial role of students and graduates in evaluating their progress. The insights provided by these two groups are essential for assessing the program's quality and its ability to incorporate various viewpoints that support institutional growth.

Another essential aspect concerns the link between self-evaluation and strategic planning, as these two mechanisms work together to pursue improvements and quality in programs, as Interviewee A reveals:

[...] with planning, you are including self-assessment as an element of the plan, using self-assessment to accompany the plan, and even to build the plan. Self-evaluation serves to make a diagnosis. [...] instead of being asked to improve through an external evaluation, you internally seek improvement based on an understanding of your own reality. You do this in conjunction with the plan, and these two things together, we think it's fundamental for improving programs.

Once their mission, objectives, goals, and institutional commitment have been clearly established, the programs are in a position to carry out an analysis of their reality, which is related to strategic planning. In line with this, the idea is for the PPGs to have the autonomy to define their mission and work systematically to achieve it. Through self-assessment, they will contextualize their realities in terms of their mission.

Like education, evaluation is a political act that influences the lives of those being evaluated, from individuals to programs or public policies (Leite et al., 2020). This process can be viewed as an authoritarian exercise of judgment or, alternatively, as a pursuit of qualitative progress through collaboration between the evaluator and the evaluated (Andriola; Souza, 2010).

In light of these considerations and upon reviewing the final report of the PNPG 2011-2020 commission, the significance of enhancing the connection between programs and universities is underscored. This is evident in the excerpt from the document:

There was also a consensus on the idea that universities should take a more active role in planning and evaluating their graduate programs within their specific context and operational area. Self-evaluation is crucial, but it should be aligned with and supported by the institutional strategic plan of the university. This involves fostering and appreciating strategic partnerships and institutional collaborations to enhance the quality, efficiency, and effectiveness of the Graduate Program System (SNPG). This approach aims to encourage the establishment of more robust Graduate Programs (PPGs) that are in line with both institutional policies and the development strategies of the regions in which they are located (CAPES, 2018, p. 9).

In this regard, Interviewee A mentioned that concrete experiences are expected to emerge from 2022 onwards, following the Quadrennial Assessment (2017-2020) scheduled for 2021. This explains why self-evaluation, as observed in CAPES (2019b), carries a relatively low weight in the Evaluation Form, ranging from 10% to 20% depending on the area of knowledge. The tendency is for this weight to gradually rise over time, highlighting the growing emphasis on self-assessment (Barata, 2019).

c) Motivations

As observed throughout the analysis of the interviews, the motivations for including self-evaluation in the CAPES Evaluation System are associated, at first, with factors that aim to enhance the qualitative development of the PPGs. In addition to enabling an agenda for improvement in the programs, self-evaluation emerges as an instrument for enhancing the Evaluation System, as well as valuing democratic practice, by expanding the role of participation by the academic community (Leite et al., 2020).

Internal evaluation addresses aspects that external evaluation cannot reach, such as the training process itself, classroom relationships, interactions between students and supervisors, and the advancement of scientific research, which are primarily qualitative in nature. CAPES' external evaluation focuses on assessing outcomes on a broad scale rather than processes. This perspective directly addresses criticisms regarding the overemphasis of external evaluation on quantitative indicators, highlighting their indiscriminate use in evaluating the quality of training processes, scientific output, and societal impact (Verhine; Dantas, 2009; Alves; Oliveira, 2014). Nevertheless, both evaluation approaches can complement each other to provide the most effective evaluation method, as argued by Interviewee A:

Internal evaluation can address processes, whereas external evaluation cannot. The most crucial process in a Graduate Program is training, which is a process. However, this external evaluation by CAPES focuses on outcomes rather than processes. It assesses products. Therefore, it does not provide insight into what occurs in the classroom or in the student-advisor relationship. Such information is not evident from a report submitted to CAPES but can be discerned internally. If quality is a concern, efforts should be directed towards enhancing teaching, guidance, research, research opportunities, and research incentives, all internally. Now, we understand that these two aspects complement each other.

Therefore, it is clear that there is an emergence of valuing processes, especially training, in the context of external evaluation. As evidenced, self-evaluation has the potential to contribute in this direction, as highlighted in the final report of the Self-Evaluation Working Group: "[...] evaluation must capture this dimension, which is essentially a process and not just a product" (CAPES, 2019b, p. 5).

Considering the future prospect of self-evaluation becoming much more important in the Brazilian graduate studies scenario, it is worth noting that in many countries, such as the United States, external evaluation usually takes place every ten years, while internal evaluation is a continuous process, as institutions are primarily concerned with maintaining their quality standards (Barata, 2019). In this way, there is a criticism that "self-assessment as the central element in the assessment of postgraduate courses [...] exists worldwide, it is Brazil that is lagging behind in this sense, of starting to emphasize self-assessment now" (Interviewee A).

In agreement, Interviewee B said that self-assessment is used in different countries for accreditation, which is perhaps different from the perspective CAPES is adopting. Abroad, there are accrediting agencies, there is no government control of the PG, and all programs are autonomous. However, if they want to be accredited, it is essential to draw up a self-evaluation report. She concludes that "[...] since this Brazilian characteristic accredits, funds, and evaluates all within a single institution, it's a bit complicated". However, this reflection on self-evaluation already demonstrates a certain progress within the Brazilian SNPG (Interviewee B).

In SINAES, [...] self-evaluation is highly emphasized. PAIUB [...] also highlights self-evaluation, whereas CAPES never prioritized self-evaluation as it used evaluation for resource allocation and aimed for a standardized distribution of resources. Self-evaluation is not intended to rank programs but solely to identify issues and propose solutions (Interviewee A).

Although the SNPG has been in existence for about forty years, this issue has been observed since the establishment of the Evaluation System in the 1970s. The primary goal was to set criteria to direct the allocation of scholarships (Castro; Soares, 1983; Balbachevsky, 2005). In essence, the evaluation process aimed to distribute public funds to promote scientific production by ranking the PPGs, leading to a summative approach (Patrus; Shigaki; Dantas, 2018; Maldonado; Bitencourt, 2019). Consequently, regional disparities have widened, resulting in the institutionalization of "centers of excellence" and "peripheral courses" (Verhine; Dantas, 2009; Alves; Oliveira, 2014; Nobre; Freitas, 2017; Tavares, 2019).

4.2 Other potential interactions in the field: Learning, Innovation, and the Emerging Context

a) Learning and Innovation

Upon reviewing the final report of the CAPES Self-Assessment Working Group, it was noticed a link between self-assessment results and knowledge generation. It also observed nuances suggesting a connection with aspects of Organizational Learning and Innovation.

When asked about the contribution of the self-evaluation process to learning and innovation within the programs, Interviewees A and B revealed a feasible association. According to Interviewee A, both learning and innovation are relevant aspects to discuss in this context, since the focus of self-evaluation is precisely on understanding the programs and providing insights for making decisions about enhancements, which are typically accumulated through experiences. In his words,

[...] our hope [...] is to create a culture that values self-evaluation as means of capturing learning [...] and providing input for decision-making on improvements. So, its two aspects [...] are very appropriate (Interviewee A).

Interviewee B perceives self-evaluation as a "qualified organizer" that aids in acquiring knowledge about the program and for the program through the collection of data and information. This process leads to the development of shared knowledge within the PPGs. In essence, it enables the collection and organization of data based on the preferences of those involved in the evaluation process, and as Interviewee B stated, "[...] with this data you produce information and from this information you make knowledge. That's what I call the learning sequence".

Thus, "[...] we think self-assessment is important to capture learning because learning is a process that we don't capture well through an external assessment, especially the CAPES type of assessment" (Interviewee A). Therefore, the objective of self-assessment is formative, according to CAPES (2019b), and should help reflect on the context and policies adopted, in addition to systematizing data to support decision-making processes, which perhaps underpins the summative nature of external assessment, as discussed by Patrus, Shigaki and Dantas (2018), and Maldonado and Bitencourt (2019).

Still on the subject of learning, Interviewee B added that:

I think the main thing we learn is democracy. We have to listen to each other in order to reach a conclusion on how we are going to make an assessment. We have to listen to each other, including the students, employees, technicians working with us, and ideally our employers. We also need to listen to our graduates.

In other words, both self-assessment and Organizational Learning have the basic premise of involving different agents in the development of knowledge. Organizational Learning can be understood as a practice that articulates the creation, apprehension, and institutionalization of knowledge at the organizational level (Basten; Haamann, 2018; Patky, 2020), based on interactions between subjects (Abel, 2015). The final report of the Self-Assessment Working Group also addresses the technical operationalization of self-assessment, considering it as a learning process within the scope of the Meta-Assessment stage. It directs the understanding towards the use of the results aimed at identifying a "[...] list of changes and innovations to be implemented" (CAPES, 2019b, p. 15).

With regard to innovations, Damanpour (2014) suggests that organizations, as adaptive systems, tend to implement organizational changes in response to external pressures and internal expectations to enhance performance. In essence, the adoption of innovations is aimed at ensuring adaptive behavior for improved effectiveness. Therefore, there exists a crucial relationship between the evaluator, the evaluated entity, and the decision-maker in steering organizational change. In other words,

[...] we understand that the best way to bring about change is through self-evaluation. The program itself recognizes its problems, and collectively builds not only a vision of the problem but also a vision of the solution. If the vision of the solution is collectively constructed, the likelihood of actually realizing this solution is much greater (Interviewee A).

[...] with this information, we'll be able to understand something that's happening. So, this new knowledge will help us to comprehend a situation. When we perceive a situation, [...] we begin to identify the weaknesses (Interviewee B).

These weaknesses are typically associated with the program's performance through its internal processes, which can only be identified by observing concrete experiences within the PPGs. Therefore, the understanding is that

[...] the programs themselves have a clear goal in mind. [...] If they are successful in reaching their goal, how can you identify any obstacles that are hindering their progress and how can you overcome these obstacles to improve the program's performance (Interviewee A).

From these nuances, innovations can emerge in the PPGs, as "[...] learning injects new ideas into the organization [...], favoring the presence of innovation" (Padilha et al., 2016, p. 329).

First of all, Interviewee B highlighted the issue of the paradigm shift in the CAPES Evaluation System concerning the implementation of the self-assessment policy itself. In other words, there is a clash of paradigms: the top-down paradigm, involving the explicit control of a country's executive power over its institutions, is gradually giving way to a paradigm of democratic participation. This new paradigm emerges from the bottom up, reflecting the self-analysis of individuals and actors in their quest to comprehend their organizational status. Therefore, a paradigm shift paves the way for the exploration of new possibilities, as expressed by Interviewee B:

So, the self-analysis that needs to be done, the diagnoses that need to be made, introduce another way of thinking. It's one thing to answer a questionnaire, but structuring and organizing this situation represent a different paradigm. This paradigm is where the democratic process comes in (Interviewee B, emphasis added).

In addition, considering the objectives of training teachers and researchers in Brazil, as outlined in the Sucupira Report (Brazil, 1965), have already been achieved, Interviewee B noted that it is necessary to expand beyond social areas and focus on postgraduate training that encompasses other sectors such as business and industry. In other words, we have reached a stage where social, economic, and cultural needs have been addressed, as gradually indicated by the academic community. CAPES needs to implement changes to broaden the activities and goals of the National Postgraduate System (SNPG). One way to achieve this is through self-evaluation to facilitate this evolution, by guiding strategies that consider processes beyond staff training for Higher Education Institutions (HEIs).

In light of these facts, concerning the potential for innovation, there is a plausible connection with self-assessment. Programs that engage in this process of self-analysis can surpass the expectations outlined in the CAPES Assessment Form. Essentially, the key question is how each PPG will formulate its policy for self-improvement to recognize areas for enhancement and implement innovative changes within the program, as highlighted by Interviewee B.

b) Emerging Context

The different arguments presented by Interviewees A and B converged on an understanding of the importance of the specific context of each PPG, as evidenced by the self-evaluation. This evidence has been systematically organized in a category named Emerging Context.

Interviewee A began by stating that it is essential to understand that there is only one Postgraduate (PG) system in Brazil, unlike higher education, which has two sectors (federal and state), and basic education, which comprises several. Therefore, there are minimum quality standards for evaluating all Postgraduate Programs (PPGs), "[...] as [...] the diploma that each student receives holds the same value nationwide. Therefore, we cannot give a higher rating to a subpar program just because it is in a more challenging location, right? That would not be logical". However, the external evaluation was developed based on a standardized approach that pays little attention to regional specificities and the emerging contexts of different programs (Verhine; Dantas, 2009; Patrus, 2018), which, as noted by Patrus (2018), exacerbates the inequalities observed in the National Postgraduate System (SNPG) to date.

There are expectations that, through self-assessment, the PPGs will be able to highlight their realities and their contributions within their context of operation. Interviewee A stated, "[...] because with self-assessment you will contextualize your reality [...] in terms of your mission", that this process will help CAPES capture the difficulties faced in different regions and direct more specific actions to contribute to the development of these programs and locations. On the other hand, Interviewee B expressed uncertainty about whether self-assessment could help reduce inequalities in higher education. She illustrated her point by referring to her experience at a PPG in Education in the interior of the state of Mato Grosso. In line with its mission, the program carried out multiple actions

[...] working with riverside dwellers, addressing the needs of the community, collaborating with schools in settlements, addressing educational challenges, working with teachers from indigenous schools, despite all the challenges. The institution has multiple campuses in the state's interior [...]. To attend a postgraduate meeting, for instance, which used to be conducted in person, a teacher often had to commute by boat, travel by road, sometimes by plane, and occasionally arrive late... In other words, all the imaginable challenges of teaching in this country were encountered in that Postgraduate Program. I can also mention that even the indigenous language, or rather languages, as there are several, could pose difficulties for those interested in pursuing a formal education program (Interviewee B).

In this context, the products resulting from these actions, in addition to the training process itself for the development of the communities, included booklets in various indigenous languages. "So you can imagine what this work entails, creating a booklet in the indigenous language, translating it, and adapting it" (Interviewee B).

Interviewee B said that although the PPG developed work that was fully integrated with the community, this was not recognized by the CAPES Evaluation System, given the criteria established in an Evaluation Form, which PPGs from inland institutions often find difficult to achieve. And because of this lack of recognition, the scores of the professors, the intellectual production and the students' output were always below what was expected to be "deserved", given that that product did not consist of "[...] a publication of as many pages as CAPES could demand, it was not a publication in English, as CAPES could demand, it was not an international publication, as CAPES could demand", criticized Interviewee B. In other words, "[...] evaluation is something very shocking because it can often fail to recognize a reality that is very rich and pass over it and ignore it" (Interviewee B).

When examining the literature on the foundations of educational evaluation policies, Belloni (1999) categorizes them into two trends: evaluation as improvement and change; and evaluation as control and hierarchization. CAPES' evaluation aligns with the trend of evaluation policy as control and hierarchization, emphasizing regulation and control to ensure compliance with requirements and standards. This also echoes the concern raised by Bertero (2009) about the coercive role of CAPES, as it holds the power to disqualify a program, leading to significant consequences such as the loss of support from the agency for its progress.

In view of this, the evidence indicates the necessity for CAPES to make direct investments in enhancing programs, particularly in more challenging regions. Self-evaluation can help to illuminate these contexts. Interviewee A stated,

[...] I think CAPES should untie this funding from evaluation. I mean...They create a situation where strong programs get stronger and weak programs get weaker because more money is allocated to the strong ones and less to the weak ones.

And he suggested that this funding for programs could be linked to improvement projects. He proposed, "[...] a consolidation project that the program presents to CAPES, in which the program conceives what it needs to do to consolidate itself. CAPES funds this project based on an evaluation to ensure that the project is well-conceptualized, etc." (Interviewee A).

On the contrary, it can be inferred that, even if the self-evaluation process brings these nuances to light, an evaluation model has not yet been developed to measure the inequalities observed in the Brazilian postgraduate scenario, despite the actions undertaken in this direction, such as the inter-institutional master's and doctoral programs (MINTER and DINTER), as stated by Verhine and Dantas (2009), Vogel (2015), and Patrus (2018). In this way, Interviewee B was not very hopeful and considered that the evaluation committees follow the logic of the "hard" sciences, but she hoped that there would be a change in the way differences are viewed, in the sense that particularities could be valued in different ways, as a central issue. In the words of Interviewee B,

[...] I expect the evaluation groups, including our fellow evaluators, to be sensitive to differences. We are living in a time when various media outlets, such as television, emphasize differences, and many organizations highlight diversity. Is it only in postgraduate studies that we fail to acknowledge these differences? It's impossible!

These aspects can indicate a significant opportunity in terms of valuing the organizational identities of each program. This is due to the potential for PPGs to articulate, through their self-evaluation processes, their unique characteristics, individualities, and contexts in relation to their mission and institutional objectives. Moreover, self-evaluation can play a crucial role in the process of reshaping program identities.

Therefore, it is necessary to re-evaluate the CAPES Evaluation System based on the principles of Educational Evaluation policies. As Belloni (1999, p. 38) states, the evaluation of educational institutions is linked to analyzing their performance in alignment with their objectives and mission. This analysis should consider the social, economic, political, and cultural context in which the organization operates. The primary goal is to identify the factors that contribute to success and those that lead to challenges, with the intention of addressing "[...] the factors favorable to good progress and those responsible for difficulties in order to overcome them".

Finally, Table 1 summarizes the main findings of the research.

Table 1
Summary of the main findings.

Category	Main findings
Background	■ Experiences of Brazilian specialists and researchers in Educational Evaluation; ■ Consolidated higher education evaluation practices in Brazil - SINAES; ■ Transferring knowledge between undergraduate and postgraduate assessment levels; ■ Contributing to the design of the self-evaluation based on suggestions and criticisms from the academic community; ■ International experiences on evaluation in the postgraduate studies (PG) - Germany and the Netherlands; ■ Results of the analysis by the National Monitoring Committee for the PNPG 2011-2020.
Self-evaluation Concept and Policy	■ Self-evaluation as a process created by the program itself to identify problems and propose solutions; ■ Systematic and articulated self-evaluation process, including the establishment of procedures, instruments, and methods of self-evaluation; ■ The creation of a commission and regulations as a way of institutionalizing self-evaluation; ■ Articulation with the program's strategic planning; ■ The importance of support from the Postgraduate Dean's Office and CPA, and alignment with the HEI's Institutional Development Plan; ■ Participation of multiple internal and external actors; ■ Contextualization of the program's reality in terms of its mission.
Motivations	■ Emphasis on improving the quality of programs; ■ Valuing process evaluation over product evaluation; ■ Delay in evaluation policies in the Brazilian postgraduate scenario, compared to the international context; ■ Valuing the more active participation of the academic community in the external evaluation process.
Learning and Innovation	■ A culture of self-evaluation serves as means of capturing learning and providing input for decision-making on improvements; ■ Self-evaluation as a "qualified organizer" that helps to build knowledge about and for the program; ■ Learning about democracy; ■ Production of shared knowledge within the program; ■ The process of collecting and systematizing data according to the interests of the PPGs; ■ Implementation of changes or solutions to address identified problems; ■ The perspective on changing the paradigm of external evaluation, emphasizing democratic participation; ■ Insertion of the self-evaluation policy as an innovation; ■ New way of thinking and reflecting on internal processes, mission, and evaluation; ■ The focus of the SNPG needs to be broadened to include other areas of civil society (such as business and industry); ■ The autonomy to use the self-evaluation policy beyond what is required by the CAPES Evaluation Form.
Emerging context	■ Multiple realities and contexts emerging from the different programs; ■ Relevance of actions to mitigate inequalities in the SNPG; ■ Investment in projects to improve PPGs; ■ Valuing actions with a local and regional impact based on the social, economic, and cultural development of each program; ■ Self-evaluation as an important component in the process of rebuilding the identities of the PPGs.

• Source: Own elaboration.

5 Final Considerations

Different changes have marked the contemporary context of the CAPES Evaluation System, aiming to mitigate historical obstacles that include the need to institutionalize new evaluation models in response to contextual challenges. The inclusion of self-evaluation can represent a significant advancement in terms of qualitative aspects, expanding the participation of PPGs in the external evaluation process and acknowledging the formative nature of this practice. This approach assumes the quality factor is rooted in valuing and respecting the diversity of training processes, scientific production, and the impact on society.

The reflections undertaken have enabled an understanding, based on the perceptions of representatives of the CAPES Self-Assessment Working Group and document analysis, of the context, motivations, and other potential interactions that have sparked discussions about the self-evaluation policy in the Postgraduate (PG) field. It was observed that, despite acknowledging the experiences already present in Brazilian higher education through the SINAES evaluation policies and various international experiences with self-evaluation, CAPES did not prioritize it in the PG evaluation process. As a response to the ongoing changes and criticisms from the academic community, the concept of promoting self-evaluation in this educational context emerged. It is evident that, given the nature of this process, there are diverse approaches that can address the long-standing inconsistencies in the evaluation model, allowing for various institutional identities and contexts to be represented in the external evaluation process.

Silva and Gomes (2018) emphasize the importance of discussing, problematizing, and understanding the control exerted by evaluation in assessing, regulating, justifying, or discrediting public programs and policies, as well as its impacts, particularly in the field of education. With the emergence of self-evaluation at the postgraduate level, there is a necessity for studies that offer the essential knowledge to comprehend the implications of this practice within the National Postgraduate System (SNPG). Therefore, this research contributes, albeit in an introductory manner, to the discussions and reflections on self-evaluation in the context of Brazilian Graduate Studies. It also reveals potential connections with other themes, indicating a promising area for further research to advance knowledge.

• 1
  Information as reported by GeoCAPES, considering indicators updated on November 5, 2021. Available at: https://geocapes.capes.gov.br/geocapes/.
• 2
  Dilma Rousseff's (PT) government was ousted in August 2016 after being convicted of what was considered a "crime of fiscal responsibility. Three months of impeachment proceedings in Brazil's Federal Senate culminated in the vice president at the time, Michel Temer (PMDB), assuming the office.

Referências

• ABEL, M-H. Knowledge map-based web platform to facilitate organizational learning return of experiences. Computers in Human Behavior, EUA, v. 51, p. 960-966, oct. 2015. Disponível em: https://www.sciencedirect.com/science/article/abs/pii/S0747563214005354 Acesso em: 22 abr. 2022.
  » https://www.sciencedirect.com/science/article/abs/pii/S0747563214005354
• ALVES, M. F.; OLIVEIRA, J. F. Pós-graduação no Brasil: do Regime Militar aos dias atuais. Revista B. Política e Administração da Educação, Goiânia, v. 30, n. 2, p. 351-376, maio/ago. 2014. Disponível em: https://seer.ufrgs.br/index.php/rbpae/article/view/53680/33095 Acesso em: 22 abr. 2022.
  » https://seer.ufrgs.br/index.php/rbpae/article/view/53680/33095
• ANDRADE, E. F. S. et al. Inovação em um contexto isomórfico nos Programas De Pós-Graduação Stricto Sensu em administração. Revista Gestão & Tecnologia, Pedro Leopoldo, v. 18, n. 2, p. 100-127, maio/ago. 2018. Disponível em: https://doi.org/10.20397/2177-6652/2018.v18i2.1415 Acesso em: 22 abr. 2022.
  » https://doi.org/10.20397/2177-6652/2018.v18i2.1415
• ANDRIOLA, W. B.; SOUZA, L. A. Representações sociais dos gestores e dos técnicos das unidades acadêmicas da Universidade Federal do Ceará (UFC) acerca da Autoavaliação Institucional. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 15, n. 2, p. 45-72, jul. 2010. Disponível em: https://www.scielo.br/j/aval/a/yBQXMMzNYbMBrLTKBqmtQjK/?lang=pt Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/aval/a/yBQXMMzNYbMBrLTKBqmtQjK/?lang=pt
• BALBACHEVSKY, E. A pós-graduação no Brasil: novos desafios para uma política bem sucedida. In: BROCK, C.; SCHWARTZMAN, S. (org.). Os desafios da educação no Brasil. Rio de Janeiro: Nova Fronteira, 2005. p. 275-304.
• BARATA, R. C. B. Mudanças necessárias na avaliação da pós-graduação brasileira. Interface – Comunicação, Saúde, Educação, Botucatu, v. 23, 2019. Disponível em: https://www.scielo.br/j/icse/a/gBkWRwqC5svbVNL3R8QN4sx/ Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/icse/a/gBkWRwqC5svbVNL3R8QN4sx/
• BARDIN, L. Análise de conteúdo. São Paulo: Edições 70, 2016.
• BASTEN, D.; HAAMANN, T. Approaches for organizational learning: a literature review. SAGE Open – Literature Review, EUA, p. 1-20, jul./sep. 2018. Disponível em: https://journals.sagepub.com/doi/10.1177/2158244018794224 Acesso em: 22 abr. 2022.
  » https://journals.sagepub.com/doi/10.1177/2158244018794224
• BELLONI, I. Avaliação institucional: um instrumento de democratização da educação. Linhas Críticas, Brasília, v. 5, n. 9, p. 31-58, jul./dez. 1999. Disponível em: https://periodicos.unb.br/index.php/linhascriticas/article/view/2752 Acesso em: 22 abr. 2022.
  » https://periodicos.unb.br/index.php/linhascriticas/article/view/2752
• BERTERO, C. O. Ensino e pesquisa em administração. São Paulo: GVPesquisa, 2009.
• BRASIL. Ministério da Educação. Conselho Federal de Educação. Parecer n. 977, 03 de dezembro de 1965. Definição dos cursos de pós-graduação. Brasília, 1965. Disponível em: https://www.scielo.br/j/rbedu/a/NsLTtFBTJtpH3QBFhxFgm7L/?format=pdf&msclkid=0f232296c26a11ecab0a0c078fce7c66 Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/rbedu/a/NsLTtFBTJtpH3QBFhxFgm7L/?format=pdf&msclkid=0f232296c26a11ecab0a0c078fce7c66
• BURRELL, G.; MORGAN, G. Sociological paradigms and organizational analisys: elements of the sociology of corporate life. London: Heinemann, 1979.
• CAPES - COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR. Plano Nacional de Pós-Graduação (PNPG) 2011-2020. Brasília: Ministério da Educação, 2010. v. 1. Disponível em: https://www.gov.br/capes/pt-br/acesso-a-informacao/institucional/plano-nacional-de-pos-graduacao-pnpg/plano-nacional-de-pos-graduacao-pnpg-2011 -2020 Acesso em: 22 abr. 2022.
  » https://www.gov.br/capes/pt-br/acesso-a-informacao/institucional/plano-nacional-de-pos-graduacao-pnpg/plano-nacional-de-pos-graduacao-pnpg-2011 -2020
• CAPES - COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR. Proposta de Aprimoramento do Modelo de Avaliação da PG. Documento Final da Comissão Nacional de Acompanhamento do PNPG 2011 -2020. Brasília: Ministério da Educação, 2018. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/2018-pnpg-cs-avaliacao-final-10-10-18-cs-final-17-55-pdf?msclkid=f5f827d7c26b11ec851ce8f372d3c165 Acesso em: 22 abr. 2022.
  » https://www.gov.br/capes/pt-br/centrais-de-conteudo/2018-pnpg-cs-avaliacao-final-10-10-18-cs-final-17-55-pdf?msclkid=f5f827d7c26b11ec851ce8f372d3c165
• CAPES - COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR. Processo de avaliação da pós-graduação é aprimorado. Brasília: Ministério da Educação, 2019a. Disponível em: https://www.gov.br/capes/pt-br/assuntos/noticias/processo-de-avaliacao-da-pos-graduacao-e-aprimorado Acesso em: 22 abr. 2022.
  » https://www.gov.br/capes/pt-br/assuntos/noticias/processo-de-avaliacao-da-pos-graduacao-e-aprimorado
• CAPES - COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR. Autoavaliação de Programas de Pós-Graduação. Relatório Grupo de Trabalho. Ministério da Educação: Brasília, 2019b. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-autoavaliacao-de-programas-de-pos-graduacao-pdf?msclkid = 1d8912ccc26d11 ec81 ed8f633b63351f Acesso em: 22 abr. 2022.
  » https://www.gov.br/capes/pt-br/centrais-de-conteudo/10062019-autoavaliacao-de-programas-de-pos-graduacao-pdf?msclkid = 1d8912ccc26d11 ec81 ed8f633b63351f
• CAPES - COORDENAÇÃO DE APERFEIÇOAMENTO DE PESSOAL DE NÍVEL SUPERIOR. Ficha de Avaliação. Área 27: Administração Pública e de Empresas, Ciências Contábeis e Turismo. Brasília: Ministério da Educação, 2020. Disponível em: https://www.gov.br/capes/pt-br/centrais-de-conteudo/documentos/avaliacao/FICHA ADMINISTRACAO P ATUALIZADA.pdf?msclkid=3f6405c7c26d 11 eca4f12057ebacae3a Acesso em: 22 abr. 2022.
  » https://www.gov.br/capes/pt-br/centrais-de-conteudo/documentos/avaliacao/FICHA ADMINISTRACAO P ATUALIZADA.pdf?msclkid=3f6405c7c26d 11 eca4f12057ebacae3a
• CASTRO, C. M.; SOARES, G. A. D. Avaliando as avaliações da Capes. Revista de Administração de Empresas, Rio de Janeiro, v. 23, n. 3, p. 113-73, jul./set. 1983. Disponível em: https://periodicos.fgv.br/rae/article/view/39515/38257 Acesso em: 22 abr. 2022.
  » https://periodicos.fgv.br/rae/article/view/39515/38257
• DAMANPOUR, F. Footnotes to Research on Management Innovation. Organization Studies, v. 35, n. 9, p. 1265-1285, 2014. Disponível em: https://doi.org/10.1177/0170840614539312 Acesso em: 22 abr. 2022.
  » https://doi.org/10.1177/0170840614539312
• FALLEIROS, A. E. S.; PIMENTA, M. L.; VALADÃO JÚNIOR, V. M. O significado da autoavaliação institucional na perspectiva de técnicos-administrativos de uma universidade pública. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 21, n. 2, p. 593-618, 2016. Disponível em: https://www.scielo.br/j/aval/a/YZW4pnFB9MgY8xr9FTkcgtx/?lang=pt Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/aval/a/YZW4pnFB9MgY8xr9FTkcgtx/?lang=pt
• FERREIRA, A. C. S. P.; FERENC, A. V. F.; WASSEM, J. Teaching Work and Capes Assessment: estrangement and naturalization. Educação & Realidade, Porto Alegre, v. 43, n. 4, p. 1321-1341, oct./dec. 2018. Disponível em: https://doi.org/10.1590/2175-623684892 Acesso em: 22 abr. 2022.
  » https://doi.org/10.1590/2175-623684892
• FLICK, U. Introdução à metodologia de pesquisa: um guia para iniciantes. Porto Alegre: Penso, 2013.
• GRAY, D. E. Pesquisa no mundo real. Porto Alegre: Penso, 2012.
• GRIBOSKI, C. M.; PEIXOTO, M. C. L.; HORA, P. M. Avaliação externa, autoavaliação e o PDI. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 23, n. 1, p. 178-197, mar. 2018. Disponível em: https://www.scielo.br/j/aval/a/NctWXzzfBGWJ5YCTnk846xS/abstract/?lang=pt Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/aval/a/NctWXzzfBGWJ5YCTnk846xS/abstract/?lang=pt
• HORTALE, V. A.; MOREIRA, C. O. F. Auto-avaliação nos programas de pós-graduação na área da saúde coletiva: características e limitações. Ciência & Saúde Coletiva, Rio de Janeiro, v. 13, n. 1, p. 223-233, 2008. Disponível em: https://doi.org/10.1590/S141381232008000100026 Acesso em: 22 abr. 2022.
  » https://doi.org/10.1590/S141381232008000100026
• LEITE, D. Avaliação da Educação Superior. In: MOROSINI, Marilia Costa. Enciclopédia de Pedagogia Universitária. Glossário. Brasília: INEP/RIES, 2006. v. 2.
• LEITE, D.; PINHO, I. Concluding remarks: evaluation and collaboration. In: LEITE, D.; PINHO, I. Evaluating collaboration networks in higher education research. Drivers of excellence. Cham: Palgrave MacMillan, 2017. p. 103-107.
• LEITE, D. et al. A autoavaliação na Pós-Graduação (PG) como componente do processo avaliativo CAPES. Avaliação: Revista da Avaliação da Educação Superior, Campinas; Sorocaba, v. 25, n. 02, p. 339-353, jul. 2020. Disponível em: https://www.scielo.br/j/aval/a/whfJzmNx7Vgpcr7c6Zj5kXz/?format=html Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/aval/a/whfJzmNx7Vgpcr7c6Zj5kXz/?format=html
• MAGALHÃES, A. M. S.; REAL, G. C. M. Situando os debates sobre a avaliação da pós-graduação: os estudos do campo pelo campo. EccoS: Revista Científica, São Paulo, n. 46, p. 131-148, maio/ago. 2018. Disponível em: https://doi.org/10.5585/eccos.n46.7947 Acesso em: 22 abr. 2022.
  » https://doi.org/10.5585/eccos.n46.7947
• MALDONADO, M. M. C.; BITENCOURT, L. P. Impasses e desafios para uma avaliação formativa em Programas de Pós-graduação. Revista de Educação Pública, Cuiabá, v. 28, n. 68, p. 429-439, maio/ago. 2019. Disponível em: https://periodicoscientificos.ufmt.br/ojs/index.php/educacaopublica/article/view/8400 Acesso em: 22 abr. 2022.
  » https://periodicoscientificos.ufmt.br/ojs/index.php/educacaopublica/article/view/8400
• MARCONI, M. A.; LAKATOS, E. M. Metodologia do trabalho científico: procedimentos básicos, pesquisa bibliográfica, projeto e relatório, publicações e trabalhos científicos. São Paulo: Atlas, 2007.
• NOBRE, L. N.; FREITAS, R. R. A evolução da pós-graduação no Brasil: histórico, políticas e avaliação. B. Journal of Production Engineering, São Mateus, v. 3, n. 2, p. 18-30, 2017. Disponível em: https://periodicos.ufes.br/bjpe/article/view/v3n2 3 Acesso em: 22 abr. 2022.
  » https://periodicos.ufes.br/bjpe/article/view/v3n2 3
• PADILHA, C. K. et al. Capacidade de aprendizagem organizacional e desempenho inovador: percepção dos atores de uma empresa têxtil. Race, Joaçaba, v. 15, n. 1, p. 327-350, 2016. Disponível em: https://doi.org/10.18593/race.v15i1.7911 Acesso em: 22 abr. 2022.
  » https://doi.org/10.18593/race.v15i1.7911
• PATKY, J. The influence of organizational learning on performance and innovation: a literature review. Journal of Workplace Learning, EUA, v. 32, n. 3, p. 229-242, 2020. Disponível em: https://doi.org/10.1108/JWL-04-2019-0054 Acesso em: 22 abr. 2022.
  » https://doi.org/10.1108/JWL-04-2019-0054
• PATRUS, R. Desigualdade social e pós-graduação em Administração: o papel da avaliação. Revista de Administração de Empresas, São Paulo, v. 58, n. 5, p. 506-510, set./out. 2018. Disponível em: https://doi.org/10.1590/S0034-759020180506 Acesso em: 22 abr. 2022.
  » https://doi.org/10.1590/S0034-759020180506
• PATRUS, R.; SHIGAKI, H. B.; DANTAS, D. C. Quem não conhece seu passado está condenado a repeti-lo: distorções da avaliação da pós-graduação no Brasil à luz da história da Capes. Cadernos EBAPE.BR, Rio de Janeiro, v. 16, n. 4, out./dez. 2018. Disponível em: https://www.scielo.br/j/cebape/a/TGP3X57NqRVfnt4xnydbH5g/ Acesso em: 22 abr. 2022.
  » https://www.scielo.br/j/cebape/a/TGP3X57NqRVfnt4xnydbH5g/
• SAUL, A. M. A sistemática de auto-avaliação do Programa de Pós-Graduação em Educação (currículo) da PUC/SP. Estudos em Avaliação Educacional, São Paulo, v. 26, p. 97-110, 2002. Disponível em: https://publicacoes.fcc.org.br/eae/article/view/2187 Acesso em: 22 abr. 2022.
  » https://publicacoes.fcc.org.br/eae/article/view/2187
• SILVA, A. L.; GOMES, A. M. Avaliação Educacional: concepções e embates teóricos. Estudos em Avaliação Educacional, São Paulo, v. 29, n. 71, p. 350-384, maio/ago. 2018. Disponível em: https://doi.org/10.18222/eae.v29i71.5048 Acesso em: 22 abr. 2022.
  » https://doi.org/10.18222/eae.v29i71.5048
• TAVARES, F. F. Análise da instrumentação da ação pública na pós-graduação stricto sensu por meio dos Planos Nacionais de Pós-Graduação. 2019. Dissertação (Mestrado em Desenvolvimento, Sociedade e Cooperação Internacional) - Centro de Estudos Avançados Multidisciplinares, Universidade de Brasília, Distrito Federal, 2019.
• VERHINE, R. E.; DANTAS, L. M. V. Reflexões sobre o sistema de avaliação da capes a partir do V Plano Nacional de Pós-graduação. Revista de Educação Pública, Cuiabá, v. 18, n. 37 p. 295-310, maio/ago. 2009. Disponível em: https://periodicoscientificos.ufmt.br/ojs/index.php/educacaopublica/article/view/481 Acesso em: 22 abr. 2022.
  » https://periodicoscientificos.ufmt.br/ojs/index.php/educacaopublica/article/view/481
• VOGEL, M. J. M. Avaliação da Pós-Graduação Brasileira: análise dos quesitos utilizados pela CAPES e das críticas da comunidade acadêmica. 2015. Tese (Doutorado em Ciência da Informação) - Escola de Comunicações e Artes, Universidade de São Paulo, São Paulo, 2015.

Editores de Seção: André Pires; Milena Pavan Serafim | Editora de Layout: Silmara Pereira da Silva Martins

Publication Dates

• Publication in this collection
  19 Aug 2024
• Date of issue
  2024

History

• Received
  08 Aug 2022
• Accepted
  22 Mar 2024
• Reviewed
  26 June 2024