Formulation and Management of a Policy: democratic impasses in the SEAP-RS

Abstract:

This article presents an analysis of the State Participative Evaluation System (SEAP-RS) formulation at the Rio Grande do Sul State Secretariat of Education, during the 2011-2014 government period. Institutional movements, concepts and assumptions of this institutional evaluation policy are examined. Dilemmas and challenges of/in managing such process were drawn from narratives and documents that illustrate the formation of the agenda and the regulatory norms. Two impasses are highlighted: the discourse on democratic management in face of a centralized policy formulation; and the demand for immediate responses from public officers in face of the new participatory evaluation requirements.

Keywords:
Institutional Evaluation; Democratic management; Policy Formulation

Resumo:

O artigo apresenta uma análise do processo de formulação do Sistema Estadual de Avaliação Participativa (SEAP-RS) na gestão 2011-2014 da Secretaria de Estado da Educação do Rio Grande do Sul. Considera os movimentos institucionais, conceitos e pressupostos desta política de avaliação institucional. Revela dilemas e desafios da/na gestão, a partir de narrativas e documentos que ilustram a formação da agenda e as normas regulamentares. Dois impasses são salientados: o discurso sobre a gestão democrática em face do centralismo na formulação da política; e a exigência de respostas imediatas dos dirigentes públicos em face dos novos modos participativos de avaliação.

Palavras-chave:
Avaliação Institucional; Gestão Democrática; Formulação de Política

Introduction

Large-scale evaluations were introduced in Brazil thirty years ago, linking ourselves in a synchronized and tuned way the educational reform movements propagated in Latin America, in which transnational bodies stimulated the institutionalization of national evaluation systems for primary education. From the expansion of school systems targeting universal access, we started to address education quality and equity, with attention to distribution - issues relevant to the current processes of democratization, with the recognition of rights in the new constitutional and legal framing of education and with progressive schooling rates. However, as highlighted by Casassus (2013), policies that were created to improve the quality of learning have been diverted from their pedagogical foundations to a notion according to which the promotion of good education should be evidence-based. Hence, efforts in producing standardized tests and information systems have been accumulated, and such have become a central element of the educational policy.

Standardized tests became the norm that guides decisions in most fields of education, including several aspects such as curricular policy, allocation of resources, teacher education and career. The reform of education, once seen as a complex and multidimensional process, ended for being guided mainly by a single and linear indicator: the result of standardized tests (Casassus, 2013, p. 22)⁴.

Large-scale evaluations have grown in importance and scope. The insertion of nations in globalization and the international evaluations weighting down on the national education policy intensified the requirement of external and comparable school evaluations.

Then, in the era of evaluation as policy, we draw our attention to that because we deem it powerful in the formulation of educational policies. We are committed to unveil and discuss which evidence is relevant to decisions about the school we long for and the organization of the educational system. What are the information and analyses, and who, how, and when were they produced? To whom and why?

We present an analysis of the formulation process of the State Participative Evaluation System (SEAP-RS), in the 2011-2014 administration of the State Secretariat of Education of Rio Grande do Sul (SEDUC-RS), considering institutional movements, and concepts and assumptions of the evaluation policy that was conceived as an inflection on the hegemonic orientation. From narratives and documents that illustrate the formation of the agenda and the regulatory norms of this proposal, which officially had a short life but may have seeded important values and learnings, we highlight two impasses: the discourse on democratic management in face of a centralized formulation of the policy; and the demand for results in face of the new participatory evaluation requirements.

As we intend to demonstrate, these impasses were not new nor unlikely. Thus, the importance of the investigative record and the theoretical reflection that place the educational policy in the field of public policies and political practice, and which positions the educational policy before evaluation policies on student achievement and structural and contingent conditions of schools.

We highlight Carlos Matus as one of the authors contributing to this theoretical reflection, a necessary link between science and practical action in the public sphere, for understanding and working within a complex reality that does not exist regardless of the subject. In his Theory of Social Game (Matus, 2005), the researcher on planning discusses the divorce between politics and science, theory and practice, government capacity and complexity of social processes. Our analysis of the SEAP-RS is an attempt to create a path towards such sense.

According to the definition of public policy by Souza (2006) and the social game theory by Matus (2005), we may say that SEAP-RS put Government into action and brought possibilities, even if uncertain, for the field of institutional evaluation. Political actors, the governor, central advisors, secretary of education, and the ruling body of the State Schools Network (Rede Estadual de Ensino - Rede), external consultants, and civil society could control economic, political, and cognitive resources, had strength, or relative power, and position in face of the problem.

Democratic management and participation were founding concepts for the new state leaders; however, there was no participation of many actors in the planning of this policy. With the following elements, we situate the movements of the State Government and the concepts and assumptions of the proposed institutional evaluation, within the scenario in which dominates a national evaluation policy of primary education that, aiming at promoting the development of students, refers to the comparison of their proficiency levels and to the classification and awards of students and schools, as expressed by a global agenda (Sousa, 2013).

The Situational Context and the Game

Carlos Matus, influenced by Dror and Lasswel, experienced the crises of the second generation of authors who rethought the analysis of policies from the 1950s and 1960s (Friedmann, 1991). The Chilean author argued that planning, management, and politics could not be unbound, thus drawing a horizontal line between science and government technique. He approached universities and the practice of social process, believing that the space for science in politics was much larger, as long as practice is theorized. He also indicated that the decision-making process consists of coalitions created by political actors and that a single diagnosis of reality would not be possible - hence, he sought a situational explanation (Matus, 1997; 2005).

Thus, he showed predicting the future is not possible, since linear actions whose cause leads to a consequence do not exit, but making a calculation and a bet is possible. He stated that all players have limitations of resources and information, but they intend to win the game; so, the uncertainties that prevail come from the combination of two variables difficult to enumerate: possibilities and probabilities. In the combination of the latter, social practice occurs, between well and poorly defined possibilities and well or poorly known probabilities, when action can be chosen, but never the circumstance in which it will occur.

Matus (1996b) has worked with situational context, flows, social game, and social rules, in which genostructures are the social rules that give identity to the social game, and fenostructures are the ability of production and accumulation of political, economic, and organizational facts and flows that are in the game. Social production is, therefore, held by a social actor linked with several others; it depends not only on a center of power to control it, but also on the ability of collective action. The author explained that each social actor involved with planning develops tecnopolitics and, therefore, they are social scientists that,

[...] worried about the action, without complexes to explore the diffuse and uncertain future, capable of understanding that the action cannot wait for developing theories; attentive to avoid deviations of the academic researcher who delights with the past and the distant future; alert to the understanding that professional departments tend to reason in an one-dimensional way, with focus and attention limited by an artificial cutoff from reality; prepared to understand a complex universe, with multiple scarce resources and varied criteria of efficiency and effectiveness; aware of the political and technocratic barbarism, while worried about the political planning, understood as situational calculus related to creative, hazy, uncertain processes, contaminated with subjectivity affecting the present and the future (Matus, 2000, p. 97).

To explain social production, Matus (1996b) made an analogy with game. As in a game, guessing its completeness is impossible, because each player has an strategy, and the rules may be modified in the course of the game and changed every moment; with them, the resources of each player increase, decrease, or change. Every social production is, therefore, multidimensional.

A process for formulation of public policies involves actions deriving from political actors, public and private institutions, individual or collective actors alternating their power and influencing their generation, creating different correlations of powers, and affecting the government and political agenda within a field of uncertainties. Processes are different in each public policy, actors, and coalitions.

According to Matus (2005), in the situational context political speeches and disputes between interest groups, games between social actors, legitimate the assumptions of the policy. Contradictions result from social games in dispute or agreements; and this can be noted in textual interventions, with possibilities and limits, what Matus called field of uncertainties.

Situational reality, in this line of thought (Matus, 1996c), becomes a field of negotiations in which some rules are revised by the manager and others are not, making (im)possible to listen to some voices, making those who claim and contest seek for gaps to manifestations through texts, boycotts, and others.

Winning the game is not the end goal for the player, but the possibility of new games, with the maintenance or creation of new rules, situated in the field of relationships between social actors with some power resource or aimed at understanding how these relationships are influenced by the social sphere. Keeping the dynamic is paramount (Matus, 1996a; 1996c).

Hence, we do not aim, in this article, to portray the past, but we seek to understand that there is an update that creates possibilities; historicity requires analytical detachment, since implementation always modifies public policies, because the “[...] practice of any public policy is actually promoted by the agents responsible for implementation” (Arretche, 2001, p. 2).

We explain the reality by analyzing a policy formulation process by people who lived the experience (concept of social actor). The reason, explained and discussed (which would be in the social game), and the way we explained and discussed demonstrate the context (situation/situational). We recognize the existence of the other in several explanations of reality that interfere in the social game, in which several actors coexist.

In this study, besides Matus, who defends the methodical movement, we also consider other authors renowned in the field of policy analysis, institutional evaluation, democratic management, and participation, supporting the theoretical-conceptual basis: Arretche (2001); Freitas (2013); Frey (2000); Lima (2001); Souza (2006); Sousa (2013), among others.

A New Evaluation

Schools in the state of Rio Grande do Sul (RS), Brazil, until 2011, had two systems of evaluation: the national Evaluation System of Primary Education (Sistema de Avaliação da Educação Básica - SAEB); and the Evaluation System of School Performance of Rio Grande do Sul (Sistema de Avaliação do Rendimento Escolar do Rio Grande do Sul - SAERS). In the state, an own policy of institutional evaluation did not exist, even though the National Curricular Guidelines of Primary Education (CNE/CEB, 2010)⁵ already advocated such practice.

In the 2011-2014 management, the state executive power established the State Participative Evaluation System (Sistema Estadual de Avaliação Participativa - SEAP-RS), by Decree no. 48,744, in December 2011 (Rio Grande do Sul, 2011a), and no longer implemented the SAERS. With such system, implementation measures were adopted for the Network (State Schools Network), comprising the Secretariat of Education (SEDUC-RS), 38 regional coordinating offices (CRE), and all the state schools, totaling 2,570 schools at the time. Declared as a State public policy, focusing on institutions and school workers, SEAP-RS was distinguished for having a systemic character and enabling comprehensive analyses in its dimensions, structures, goals, relationships, activities, commitments, and responsibilities within the three instances of management. Aiming at having annual cycles of institutional evaluation in all units, the initial goal was to provide information for the definition of public actions in education.

SEAP-RS expanded institutional evaluation with a diagnosis unrestricted to students’ proficiency tests. It sought to investigate and discuss, in a participatory way and with self-assessment, other dimensions of the situational reality and, thus, promoting the development of action plans for improving the provision of school education.

Organized in a digital platform, and with printed support material, it standardized instruments for each instance of the Network: Institutional Evaluation Notebook no. 2 to schools; no. 3, to CRE; and no. 4, to SEDUC. Notebook no. 1 comprised guidelines for creating the Institutional Evaluation and was widely distributed and discussed at meetings with leaders of the Network.

This institutional evaluation was structured with 50 indicators and their respective descriptors, distributed into six dimensions: Institutional Management; Physical Space of the Institution; Work Organization and Environment; Access Conditions, Permanence and Success in School; Training of Teachers; Teaching and Evaluation practices. It comprised quantitative and qualitative evaluation, five-point scale, according to their descriptors and justifications.

SEAP-RS also included an external evaluation carried out by Universidade Federal do Rio Grande do Sul (UFRGS), responsible for analyses and reviews of results and processes. In Chart 1, we show some elements for comparison between SAERS and SEAP-RS.

Chart 1:
Some differences between SAERS and SEAP-RS concerning Concept of Management, External Evaluation, and System Structure

SAERS	SEAP-RS
Management
Government policy, with intermittent and periodic changes; creation focused on productivity, based on managerialism and performativity (WERLE; KOETZ, 2012). Focus on student achievement of elementary school, considering tests of Portuguese Language and Mathematics, the principal’s and teachers’ management, per school, consequently. Two-year cycles, but discontinuous, mainly census in state schools; on occasion, also comprising samples of municipal and private schools. Created and administered by contracted evaluation agencies.	Proposed as a State policy with systemic creation, focusing on the institution and on individuals and providing analyses of dimensions, structures, goals, relationships, activities of different instances and public facilities of the state education. Census and annual cycles in all network of state schools and their management body. Developed and managed by the SEDUC team, it required collective and participatory evaluation throughout the Network unit (schools, CRE, and SEDUC).
External Evaluation
Contracted to produce instruments and reports of results, in addition to the administration of tests. Reports focused on quantitative results without context analysis and data interpretation.	Contracted as a longitudinal, quantitative, and qualitative research project for monitoring and advising the institutionalization process, aiming at legitimation, mobilization, and interpretative-demonstrative analysis of results.
Structure
Syllabus focusing on skills and abilities according to the SAEB syllabi, published as Lições do Rio Grande [Lessons from Rio Grande] (2009).	In situ analysis and verification of Dimensions, Institutional Indicators and Descriptors, instrumentalized by SEAP-RS Notebooks.

• Source: Prepared by the authors based on the document Guia Interno [Internal Guideline] created by SEDUC to orient the debate on SEAP-RS [n.d].

Indeed, SAERS consisted of student achievement tests created and verified by an external consultancy, whose quantitative results were disclosed without analyzing contextual factors. SEAP-RS aimed at the institutional self-assessment performed by actors of several instances of the state education network, comprising various dimensions, structures, objectives, and the practice of administrative and pedagogical relationships.

Actors and the Situational Context for Formulating the Agenda

The agenda of the SEAP-RS showed peculiarities due to themes that occupied a prominent position for the rulers and a viable alternative to be implemented at that time.

In 2011, Rio Grande do Sul had a new State administration, with the traditional political alternation: in this case, from PSDB to an alliance led by PT, committed to continue the decentralization of public administration, on one hand; and, on the other hand, to articulate actions and enhance services and public policies under the coordination of the Federal Government. Therefore, how to sustain and expand democracy, decision-making arenas, and social games that consisted of the public government and various civil society sectors, with the managerialist demand that was also imposed? The discourse of education modernization, as in other areas, required increasing its efficiency and effectiveness - in a production-oriented approach, less analytical, as the zero waste concept - linking and mistaking decentralization for modernization, whereas the project for education democratization would imposed the social relevance criterion to its management- inclusion, equity, and social quality (Sander, 2007).

The Federal Government, through new technologies, started demanding more feedback on programs and projects to implement the release of resources, i.e., even decentralizing, it ended centralizing to greater control of the government spending. The decentralization process was aimed to ensure social quality in education, through greater control.

The state election was aligned with the federal path. According to Barbosa and Souza (2010), the national policy had a developmental cutoff, with three lines of action of the Federal Government: temporary measures of fiscal and monetary stimulus to accelerate growth and increase the productive potential of the economy; acceleration of social development by increasing income transfers and raising the minimum wage; increased public investment and recovery of the role of the State in long-term planning. Consequently, there was greater control of the proposed programs and projects.

After the neoliberal avalanche of the 1980s and 1990s, in which prevailed the concept of minimal public service, the conjuncture questioned the nature of the State, the scope and limits of planning and management of public policies within the contemporary capitalism. The state of Rio Grande do Sul was a field of possibilities for accomplishments, but also for uncertainties about its effects.

The neoliberal thought questioned education as a public policy of full responsibility of the State, imposing its own categories of analysis according to hegemonic interests - i.e. under the so-called laws of the market and private initiative. In this context, regarding educational policy, Azevedo (1997, p. 15) stated that the “[...] neoliberal virus does not infect it [educational policy] in the same proportion in which it affects other social policies,” since the responsibility of the Government to ensure access to primary educational is not questioned. However, neoliberals propose its division with the business sector to stimulate competition and to expand public funding to private stakeholders, by providing families with means for a free choice concerning school for their children. Hence, the author claims, the State monopoly is put into question, and the educational policy is successful if capital oriented.

Santos (2016) interpreted this move as a broad process of disorganization of the Democratic State, in five transitions: a) from collective responsibility to individual responsibility; b) from State action based on taxation to action based on credit; c) from recognition of the existence of public goods (health and education) and strategic interests (water, telecommunications, and postal services); d) from primacy of the State to that of civil society and market; and e) from social rights to assistance services, in situations of extreme poverty. These neoliberal transitions build the Minimal State.

SEAP-RS was created against this dynamic. Associated with other social policies, it gathered a self-assessment and collective methodology to, in a reflective way, think about the school territory and its community and, who knows, to stimulate criticism about the confrontation of neoliberal transitions presented by Santos (2016), at least concerning the public good of education. The studies by Silva and Machado (2015), Lima and Golbspan (2016), Bairros and Marchand (2016), and Saraiva and Luce (2015), among others, demonstrated several testimonies in favor of this participatory and systemic proposal towards a more egalitarian education.

Following them, we also mention Castro and Oliveira (2014), who addressed the legitimacy of public policy:

In contemporary societies, the State must provide public policies that meet society needs. To carry out state functions with legitimacy, planning and permanent interaction between governments and society is needed, in such a way that they agree upon objectives and goals that guide the formulation and implementation of public policies (Castro; Oliveira, 2014, p. 23).

According to Souza (2006, p. 26), we understand that the formulation of public policies “[...] consists of the stage in which democratic governments translate their purposes and election platforms into programs and actions that will produce results or changes in the real world.” The development of public policies such as SEAP-RS resulted, then, in a political process in which several perceptions of the problem are in dispute, and even of the model to be implemented. These, which emerged from a complex demand, were supported or faced the internal resistance as that of society itself. Hence, the difference between a political decision and a public policy (Rua, 2009).

In the case of SEAP-RS, the core team of SEDUC-RS put faith on the differential proposed for the institutional self-evaluation, and had a strong support from the Executive Power. The theoretical proposal and the implementation possibility converged.

The only indicators valued until 2011 were standardized ones, which unsatisfactorily classified the schools and the state of RS; and the lack of a own institutional evaluation prevented any counterpoint to large-scale values, nationally and regionally.

The problem definition and its solution were established by the delegative democracy, which reinforced the importance of process analysis, to do justice to the complex and dynamic empirical reality that composes this research field. SEAP-RS, according to the analyzed documents, was unlike anything that had been done before.

The official text of the policy explained the influence of the national agenda, when proposing the articulation of SEAP-RS (art. 9, § 1, and 2) with SAEB and the use of the Primary Education Development Index (Índice de Desenvolvimento da Educação Básica - IDEB) as a criterion to select the CRE and the schools at which additional tests would be applied, by sampling.

The center of government planning was at the Sala de Gestão [Management Room]⁶, having been important to be aware of how the relationships between State, civil society, and the public sphere were established, within the dilemmas between politics and the policy analysis as well as in flows arising from these relations.

With Matus (1996b; 2000), we understand that the high-level government has high degree of influence on the creation of the government agenda, but has low influence on the formulation and control of actions. Also, legislative actors influence the government agenda and may contribute to the generation of actions, since they have legal authority to formulate laws, also paramount to changes, and access to regulatory information. The coalition created during election campaigns is defining. Economic interest groups, overall businessmen, and groups interested in social policies, the organized civil society, can affect the government agenda in both directions, in a positive or negative way.

In the political game, media also interferes in the spread of information between social actors, sieving and highlighting information that eventually weigh on dissemination of the agenda and the policy formulation. However, it always supports the orientation of the attention of several actors; the formation of public opinion evidences its effect.

Less visible social actors can be inside or outside government. They are: civil servants, analysts of interest groups, parliamentary advisers, researchers and consultants, i.e., those who influence the development of policies, but do not possess decision power. In Chart 2, we indicate the main actors and their respective degree of influence on the formulation of the SEAP-RS agenda.

Chart 2:
Actors, Belonging Structure, and Power of Influence in the Formulation of the Agenda, Formulation and Reformulation of SEAP-RD Policy

Actors	Belonging structure	Influence in the agenda formulation	Influence in the policy formulation	Influence in the policy reformulation stage
Governor	State Public Authority and Leaders of the Network	High	Low	Low
Secretary		High	High	Average
SEDUC Directors		Average/High	Average/High	Average
Advisors (SEAP-RS)		Average	High	High
CRE		Null	Low	High
State Council of Education		Null	Null	Null
Legislature		Average	Null	Null
School Councils	Schools and Local Community	Null	Null	High
Students and Parents		Null	Null	High
Directors, Faculty, and Staff of Schools		Null	Null	High
Hired Researchers	External Evaluation and Funding	Null	Null	Average
World Bank		Average/High	Null	Low
CPERS Syndicate	Social Interest Groups	Null	Low	Average
Media		Null	Average	Average

• Source: Prepared by the authors.

Indeed, actors who influence the formation of the agenda are distinct from those having influence in the agenda formulation and reformulation. Whereas the reformulation occurred during implementation, it is noteworthy that, in the case of the SEAP-RS, there was limited participation between the stage of formulation and reformulation, meanwhile participation was crucial process to the success of such policy.

What decisively contributes for an agenda to become a policy is the involvement of actors since the formulation process. Observing the degree of influence in the formulation of the agenda, eight groups were key actors for the reformulation/implementation of the policy with a null degree of participation, because at this stage of SEAP-RS, actors who (inter)acted were the Governor, the Secretary, and the Director of the Department of Pedagogy, influencing the context for future actions towards separation, transformation, and/or mutual support (Matus, 1996b). The World Bank was in the field of the agenda formulation as a spectator, because it was waiting for the moment to create the financing rule. That is why we placed an average/high degree of influence to it, moreover considering that one of the five elements of power mentioned by Matus (1996b) is the control of resources.

In the formulation of SEAP-RS, the degree of influence remained focused only on one level below the institutional peak, when considering the role of advisors of the Secretary and the Educational Board, and introduced the participation of some other actors, through consultation, for preparing the dimensions, indicators, and descriptors of SEAP-RS.

In the stage of policy reformulation/implementation, social actors with high influence belonged to the group in which they worked, subjects of this process. However, they do not perceive their power in defining the game rules, which somehow silenced the disputes. Some ignorance regarding the influence power remind us of the assertion by Lima (2001): the greater the distance between the creation and the implementation of a public policy, the greater the intervention capacity of social actors, since these may interpret and reinterpret formal rules to the point of being able to replace them with possible alternative rules. In the case of this policy, the CPERS-Union and the media influenced in some step backs of the policy for its implementation; for other actors, the distancing did not cause an emancipating movement under a regulation.

Researchers had large influence at this stage, because they occupied the place of analysis through the external evaluation; their criticisms worked for reorienting the policy. On the other hand, the Brazilian Ministry of Education (MEC) featured a low degree of influence, because it only supported with the technological basis, and the state legislature, in this context, exercised power over the legal text.

The political will of who establishes the evaluation spaces “[...] constitutes a necessary condition, but not enough to enable the authentic dialogue between people, the learning of listening to contradictory opinions respecting them” (Sordi; Bertagna; Silva, 2016, p. 189). It is in the intensive and methodical exercise of participation that such spaces are consolidated and leveraged. According to the authors, “[...] we learn to participate by participating” (Sordi; Bertagna; Silva, 2016, p. 189). SEAP-RS was created as a space for legitimate participation and overcoming the limits of representative democracy, contributing to the confrontation of managerial reforms, currently proposed.

The power of influence exerts pressure and defines strategies for the changes in the government agenda, interfering in processes and in the context of formulation of policies. The game of forces can maintain social actors united in coalitions that, in a dispute, will produce changes in public policies. As Matus (1996b) proposed, the accomplishment of the agenda and the implementation of the new evaluation system would require a balance of five elements: strategic quality, scientific and technological bases, control of resources, ability, and motivation. This occurs in cases of mutual reinforcement of fenostructures and genostructures. What keeps the actors together is a set of common beliefs, in addition to their commitment to coordinated actions in favor of a policy.

The cooperation of MEC to SEAP-RS was providing access to the Integrated System for Monitoring, Implementation and Control (Sistema Integrado de Monitoramento Execução e Controle - SIMEC) for customization by Companhia de Processamento de Dados do Estado do Rio Grande do Sul (PROCERGS). This action would hardly have occurred without harmony of interests between the State and Federal Governments and direct action by the State Governor.

In the stage of policy formulation, the government used different strategies for solving problems: those of imposition force, such as legal devices, or of indicative force and dependent on the support of other social actors, such as cooperation, opposition, or confrontation (Matus, 1996a).

Formulators of the SEAP-RS policy, believing in the absence of accumulation of experiences in institutional evaluation in the involved groups - SEDUC, CRE and schools -, structured a theoretical-methodological support aiming at the (self-)awareness of their strengths and weaknesses, through training actions to qualify and deepen what should be translated into practice: an innovative and complement system concerning the large-scale evaluation models.

In its turn, the Secretary of Education was aware of the differential proposed for the institutional self-evaluation and won full support from the Executive Power. Both the theoretical proposal and the possibility of implementation converged, but the CPERS-Union had a different understanding and, in an interview to the newspaper Correio do Povo (NUNES, 2011), its President stated that SEAP-RS would be an attempt to pass on the State responsibility to the education professionals. The State Government answered through a printed communication:

The evaluation we advocate results from a participatory and institutional process focused on schools, occurring on all levels and modalities, including the regional departments and coordinating offices, validated by experts of RS universities and MEC itself, in all stages of the process. A process that will comprise several aspects [...], structural, pedagogical, management, and training aspects concerning the socioeconomic relationship and reality of the community where the school is located. The result will consist of a diagnostic for finding solutions, [...], as well as it will enable the interpretation of the IDEB of schools, and will not be linked to any punishment or award according to performance (Rio Grande do Sul, 2011b, p. 1).

The coalition of forces was important in facing the tensions. Knowing who were the actors present in the political game was part of the strategy as they influenced the discussions and debates during the (re)formulation of alternatives for public policies.

Interest of Actors, the Social Game, and Institutional Limits

The SEAP-RS was presented in a vertical and descending way, despite advocating the rupture with the traditional top-down evaluation and planning at the Network. With determination, the Secretariat opened space for the Participatory Institutional Evaluation, taking advantage of the critiques on large-scale evaluations, believing in the educational and democratic quality of its proposal. As Freitas (2013) stated, the culture of institutional evaluation was introduced in Brazil by a practice of large-scale student achievement tests. Supported by the theories of measurement and audit, it becomes a metaregulation, which implies accountability of school actors and drives meritocracy. This was the situational context we found, with an education perspective based on measurement of controlled achievement, associated with awards and punishments.

In a tactical act, the Governor made a statement on SEDUC-RS⁷ website, in which he sought to distinguish merit from meritocracy. He highlighted merit as recognition and professional qualification of the public service; and meritocracy as a business cause, forged in private structures.

SEDUC-RS, when removing SAERS and proposing the SEAP-RS, intended to stress an inversion of logic: abandoning the monitoring to which social actors were subjected in the educational system towards something that would fit the evaluation of policies; highlighting the evaluation process instead of their results; breaking with a culture of accountability and meritocracy. This was an attempt to change the focus of media on rankings, as the recent 9th place obtained by RS in the IDEB and high school failure rates.

The competition was fierce. In the game, on the one hand, actors who were educators and, on the other hand, those identified by Ravitch (2011) as managerialist reformers of education - politicians, media, entrepreneurs, consulting institutes and foundations, and educational technologies business, all aligned with the competitive and private logic. Two concepts were particularly put into question: social quality of education and management democratization. For managers of SEDUC-RS, a higher grade in large-scale evaluations would not necessarily mean quality; as for the school management, participation in decisions is a requirement for autonomy. However, a problematic gap was that the concept of social quality of education was unclear in the Notebooks of SEAP-RS.

We understand that quality is negotiated, dynamic, and transient, and contains traits that stimulate the comparisons. However, as stated by Silva (2009), in Brazil, social policies absorb the mercantilist logic in a transposed formulation, almost without screening, in the area of social rights, including public education. In this process, the international financial bodies act with a managerialist bias. On the pretext of ensuring economic and social development, they prioritize financial criteria and the quality of education depends on them. However, inside the school, the quality of education is considered as

[...] the organization of pedagogical work and school management; school projects; interlocutory contexts between school and families; a healthy environment; the effective inclusion policy; respect for differences and dialogue as a basic premise; the collaborative work and the effective operational practices of school committees and/or boards (Silva, 2009, p. 224).

According to Farenzena, Cardoso and Schuch (2015), SEDUC-RS complemented the concept of social quality of education with other strategic actions, such as permanent training policies, physical recovery of school spaces, technological modernization, and the dialogue with educators - without, however, being able to significantly improve the professional conditions of teachers, a permanent demand of the Network, also recognized by the society.

In this perspective, a low IDEB would not mean lack of social quality of education and vice versa; the context and other aspects that interfere in the conditions of the school should be considered. If IDEB is precarious as an indicator of quality of education, we should reflect on what indicator(s) would be capable of addressing it, and could allow the monitoring of public policies in a more pedagogical and democratic perspective.

Another challenge arose: How to emancipate and regulate? How to change the power into shared authority, into democratic rights and responsibilities? Would it be possible to transform knowledge-regulation into knowledge-emancipation?

The democratization of management, access, and knowledge belong to an emergency paradigm, to the reinvention of space-times that should promote democratic deliberation and emancipatory knowledge. Public education is supported by universal assurance, since it is a social duty sustained by the egalitarian, secular, democratic, free, and universal State.

SEDUC-RS would continue with the deployment of the multidimensional participatory evaluation, aiming to organize, run, maintain, manage, coordinate, and control the State Education System, safeguarding state and national legislation and the implementation of the guidelines provided by the State Council of Education (Conselho Estadual de Educação - CEEd) and the National Council of Education (Conselho Nacional de Educação - CNE). However, the formulation process of SEAP-RS introduced participation only as a methodological proposition of a representative cutoff. It is well known that participation does not occur by mandate, if the actors involved in it do not want it. Nevertheless, a formal mandate has a symbolic value, as a standard, in the social game context of practice; in this case, the rule that prevailed was the transparency of records in the electronic system, made by the School Council, the CRE or SEDUC-RS, as the external evaluation reports. How and why some records were not in fact representative is an issue noted but not interpreted at this point.

We reinforce that the political will of managers manifested in legal and political texts is not enough. All social actors involved (Chart 2) are protagonists. In the established game, the Governor stated that SEDUC-RS would be one of the pillars of the agreement between State Government and the World Bank. Despite the concept of evaluation proposed by SEDUC-RS being quite distinct from that advocated by the funding body, the project was accepted, and WB also played a role of external evaluator, monitoring and demanding accountability every six months.

SEAP-RS went in force within the agreed deadlines, and almost 100% of those involved participated in the process. We observed, in the external reports (UFRGS, 2016), that schools have progressively made more severe criticisms of its conditions regarding the physical space, rates of approval, failure, and dropout, external indicators, and school legal documents (PPP, Study Plans etc.). Would this be a move towards greater confidence in the policy and/or the other actors?

Practice has changed the policy and demanded the permanent (re)formulation of it, always considering the participation of new social actors and their new games within the interactions between the original formulation and the actual conditions of tis practice. SEAP-RS may be an example in this sense, changing according to every practical application in light of the need for adjustments and changes to meet the challenges of reality.

The time of creativity and uncertainty of the social game, in the case of SEAP-RS, resulted from an interaction between the self and the other, i.e., the government action depended on the action of coordinating offices, the action of the coordinating offices depended on the action of schools, and vice versa - by action or omission of different actors.

The continuing education procedures and the spaces created for disseminating the System brought support and opposition manifestations regarding the proposed evaluation. Sometimes its methodology was contested, sometimes its objectives and purposes. Some social actors questioned the importance of their participation in the institutional self-evaluation activities, because of experiences with conventional assessments that gather information more related to the legitimacy of superior decisions. Reports from some schools and questions by the teachers union evidenced they were uninterested in legitimizing decisions made by others, which were not derived from the participation process itself.

Instruments of SEAP-RS highlighted pedagogical changes, requiring an understanding of fundamental concepts of the educational project and the evaluation, inducing a culture of self-evaluation and the consequent use of their results with maximum participation of the institutional community. Participants needed discernment on the proposal because SEAP-RS involved all levels of the educational administration, from the State Secretariat to regional administration to school and the classroom. The proposed self-evaluation was not an individual and isolated action; collective, it aimed to integrate the school community and, in its discussions, society. In 2014, the last year of the management, SEAP-RS Notebooks were revised and Notebook 5 was created, with emphasis on the planning based on institutional evaluation, as another effort to legitimate the policy.

Nevertheless, the dissemination of results - external evaluation reports - did not entirely reach school communities; it depended on actions of SEDUC and CRE, the central power. Publicizing occurred in the website of SEDUC, with little knowledge of the social actors involved, “[...] thus creating a gap in communications, maybe the one responsible for the absence of an effective changing impact” (Vianna, 2003, p. 27). The lack of discussion on the reports can be one of the factors that would explain the discontinuity of evaluation in a structured way, so far. Participants did not pressure the following state administration for SEAP permanence. We understand that there was insufficient appropriation of the process to demand its continuity.

Surely, a strength of SEAP-RS was to bring innovation into state educational policies. But, if we presume that a state public policy is that which, in addition to the mentioned aspects, has continuity, nowadays it is not possible to recognize SEAP-RS as a state policy, like it was conceived and announced. With the electoral defeat of the political field that established it, the System was reduced to a government policy. We note this recurrence when the mechanisms for continuity and improvement of the policy are not provided.

In fact, the political history of RS has many examples of this phenomenon, since there is no record of reelection of governors, and the fierce disputes have buried many important and effective economic and social policies, in spite of its democratic production and/or intent.

We assume that SEAP-RS could have achieved more legitimacy if greater listening between the three instances and effective participation of the entire community had occurred, in a systematical and empowered way, in the pedagogical and political-administrative use of the external evaluation, and in providing the reformulations suggested for its improvement. Many potential participants did not actually act, the new practices were not in fact legitimized and institutionalized.

Matus (1998; 2000) stated that being a good politician is not enough; we must know the field to be ruled, otherwise, the manager will not be able to mediate the present-future. Political and intellectual actions have to be complementary and simultaneous, since the management time is finite and the social time is irreversible. In this case, the manager’s profile and of his team weighted heavily on the policy formulation. The System brought in its epistemological and scientific-technological basis the pedagogical work for creating the social quality of education, by proposing planning and practice in permanent movements towards the situational reality. This methodological ability, even though presented in a descendent way, would enable an inverse process - participatory, even though by representation, in the border between regulation and emancipation.

Social games have generated institutional movements that (re)signified the policy, which proved to be powerful to indicate differences and able to give voice to subjects in local institutional contexts through self-evaluation. Obviously, professionals of SEDUC, CRE, and the schools lived a distinct experience and, although absent in the policy formulation, they could have more protagonism. However, time was short, and this movement was discontinued, followed by a resumption of the large-scale student achievement policy.

Dilemmas and Challenges of the SEAP-RS Management

The process between the decision and the constitution of the policy takes place by social games, because those who turn the opponent into an ally or take advantage of the opposition to win more support win the debate; it is not a question of eliminating the contenders. In scenario of policy text production of the SEAP-RS were the actors of this game shortly after winning the elections to State Government from the conservative political field; in preparing the government program, they had sought persuasion in various sectors; in the post-election game, they defined a cooperative political alignment with the Federal Government, with a broader scope; and changes in the articulations supporting the state management occurred. We understand these intersections of fields of power and opportunities in the formulation of social actions, because “[...] in the democratic contest we can establish a distinct principle: the ideal of a strategy is cooperation to achieve a situation-objective, turning a zero-sum game into a positive-sum game [...]” (Matus, 1996a, p. 38).

Indeed, the most important dilemma was taking the vertical formulation of the agenda to an implementation with high degree of participation, involving many actors to whom they intended to give voice and space for major roles in participatory modes of evaluation. In the policy formulation, the government used several strategies to accomplish its project: presentations and persuasive dialogues, collaboration agreements, enforcement by legal instruments of the policy institutionalization, as well as explicit opposition and confrontation. To the latest, Matus (1996a, p. 38) referred to as “[...] the Chimpanzee or the Maquiavel political strategies, [in which] you fight for and seek to establish inequality from unequal forces. This is about avoiding the duel to ensure success”.

This policy entailed in ingenuity and operational capabilities on a large scale, permeating the big Network. Centralization was the strategic method for induction and control of the policy, even though other directions were possible, and aimed to ensure the debate and the practice of self-evaluation. We recognize that the mobilization triggered towards different institutional levels (schools, CRE, and SEDUC) ensured the possibility of multiple evaluative perceptions through participation, even if shy in several schools and in some CRE. Furthermore, it stipulated an action with greater potential of political commitment, the democratic management of/in education if the involvement of different actors (Chart 2) in its formulation and implementation was achieved.

Then, how to practice the founding concepts of the SEAP-RS, democratic management and participation, from the structural design and the established flow? How to instrumentalize the proclaimed autonomy of schools, for the creation of an education system, although from a centralized definition? The technological tool used was not enough to incite the wide participation; stimuli and demands were needed, and yet, in many cases participation was shy.

We also consider the dilemma of innovation, in addition to the problematic access and use of digital technologies in the Network as a whole. The practice of the new system should be innovative to provoke discussions about their indicators to complement, or better, (re)situate the large-scale evaluation and its forms of dissemination, in a participative and democratic design articulated in the three institutional levels. Improving the quality of education requires measures to reverse the precarious conditions on the basis in which such takes place. SEAP-RS was designed to allow various actors to participate in the analysis of the school management and, then, mobilize to change strategies, based on the local reality. An innovation of SEAP-RS was the articulation of the three institutional levels that comprised the management (School, CRE, and SEDUC) in a single process. Another was comprising factors within and outside schools, requiring consideration of participating subjects concerning the social, cultural, and economic context, immediate or not and, explicitly, the confrontation with the concept of quality not reduced to academic achievement and measured by self-evaluation.

SEAP-RS indicated in the assumptions and concepts of the policy a rupture with tradition. Concepts of social quality of education and democratization of the management required complying with the principles of participation, because the rules of the democratic game demanded performance in the decisions to establish effective autonomy. However, the concept of social quality of education was not clear in the Notebooks of SEAP-RS.

Such facts were also problematic: meeting the emancipation regulation and turning the power into shared authority; and the centrally planned participation to strengthen the democratic management was paradoxical. An uncertainty of the social game, in the case of SEAP-RS, was the governance interaction, between Departments and Coordinating Offices, since the success of this policy greatly depended on the (re)action in the schools. As we observed, communication with the schools was not as effective in the continuing education procedures nor in the feedbacks of the external evaluation.

However, practice modifies the policy, requiring permanent reformulation. Implementation was modified in three editions of SEAP-RS, from 2012 to 2014. Aware of improvements in operational resources, effort in continuous formation of CRE and schools was increased so that the culture of participatory evaluation could be in force. Despite the prevalence of the control and certain dissimulations, the self-evaluation proved how autonomy can be exercised in schools. Settling and silencing must be overcome so that possibilities of/in democratic management can be potentiated.

The centralized formulation could have been a gap for greater involvement of social actors. After all, the distance between the creation and the implementation of a policy makes room for criticism and reaction. The social actors could have subverted the rules, but it was unclear. The managers’ strategy of creating the Notebook no. 5 on the Participatory Planning in School, in the second edition of SEAP-RS, was designed as an incentive to the creative organization of emancipatory actions.

However, one administration period is finite. The training [formation] of the permanent professional body was not sufficient for self-evaluation to continue, neither for improving the systemic planning of multiple factors of education democratization. Even so, schools of the 30 CRE, over three years, show increasing choice of the excellent situation for the indicator 1, concerning the planning and monitoring of goals/actions of democratic management (UFRGS, 2015). All in all, the experience helps to think critically about participatory policies and about more inclusive and comprehensive evaluation policies. As researchers, we want to contribute to reflection and debate, moreover, to the impact on new and better ways of evaluation in education, in which we perceive ethical and philosophical values supporting the notion of quality in an eminently democratic education, the scientific basis about learning and the possibilities of their assessments, as well as the political interests inherent in measuring and comparing large-scale educational evaluation.

We have understood the emerging moment of reinvention of policies in a participatory institutional evaluation policy, with possibility of ressignification of the democratic processes in public education of the State of Rio Grande do Sul, which can no longer endure so many restrictions on access, permanence, and success of its schools.

Although conceived in a restricted environment, SEAP-RS corresponds to principles legitimized in the constitutional and legal order of the Brazilian education and to the political bases of the state government of its period. With SEAP institutional evaluation policy achieved a new and prominent role; instituted spaces of potential and broad participation, overcoming the limits of representative democracy; contributed to the confrontation with the logic of standardized evaluations restricted to the performance of students and professionals of basic education. The dilemmas and challenges here exposed only suggest more complex problems, still waiting for solutions coming from science and from political debate. Thus, in the intensive and methodical exercise of publicized research and of participation in evaluation, the democratic spaces of education are consolidated and leveraged.

Referências

• ARRETCHE, Marta Tereza da Silva. Uma Contribuição para Fazermos Avaliações Menos Ingênuas. In: BARREIRA, Maria Cecília; CARVALHO, Maria do Carmo (Org.). Tendências e Perspectivas na Avaliação de Políticas e Programas Sociais. São Paulo: IEE/PUC, 2001.
• AZEVEDO, Janete. A Educação como Política Pública. Campinas: Autores Associados, 1997.
• BAIRROS, Mariângela; MARCHAND, Patrícia Souza. Sistema Estadual de Avaliação Participativa no Rio Grande do Sul: uma análise a partir das instituições. Políticas Educativas, Santa Maria, v. 9, n. 2, p. 54-74, 2016.
• BARBOSA, Nelson; SOUZA, José Antônio Pereira. A Inflexão do Governo Lula: política econômica, crescimento e distribuição de renda. In: SADER, Emir; GARCIA, Aurélio (Org.). Brasil: entre o passado e o futuro. São Paulo: Boitempo, 2010.
• CASASSUS, Juan. Política y Metáforas: un anlysis de la evaluación estandardizada en el contexto de la política educativa. In: BAUER, Adriana; GATTI, Bernadete Angelina (Org.). Vinte e Cinco Anos de Avaliação de Sistemas Educacionais no Brasil: origens e pressupostos. Florianópolis: Insular, 2013. P. 21-46.
• CASTRO, Jorge Abrahão de; OLIVEIRA, Márcio Gimene de. Políticas Públicas e Desenvolvimento. In: MADEIRA, Lígia Mori (Org.). Avaliação de Políticas Públicas. Porto Alegre: UFRGS/CEGOV, 2014.
• FARENZENA, Nalú, CARDOSO, Caroline Cristiano; SCHUCH, Cleusa Conceição Terres. Uma Análise da Formulação do Sistema Estadual de Avaliação Participativa do Rio Grande do Sul. Dialogia, São Paulo, n. 22, dez. 2015.
• FREITAS, Luiz Carlos. Caminhos da Avaliação de Sistemas Educacionais no Brasil: o embate entre a cultura da Auditoria e a cultura da Avaliação. In. BAUER, Adriana; GATTI, Bernadete (Org.). Vinte e Cinco Anos de Avaliação de Sistemas Educacionais no Brasil: implicações nas redes de ensino, no currículo e na formação de professores. Florianópolis: Insular , v. 2, 2013.
• FREY, Klaus. Políticas Públicas: um debate conceitual e reflexões referentes à prática da análise de políticas públicas no Brasil. Planejamento e Políticas Públicas, Brasilia, n. 21, jun. 2000.
• FRIEDMAN, John. Planificación en el Ámbito Publico. Madrid: INAP, 1991.
• LIMA, Iana Gomes; GOLBSPAN, Ricardo Boklis. O Sistema Estadual de Avaliação Participativa na Educação Estadual do Rio Grande do Sul: contribuições para o desenvolvimento da gestão escolar democrática. Políticas Educativas , Santa Maria, v. 9, n. 2, p. 15-31, 2016.
• LIMA, Licínio. A Escola como Organização Educativa. São Paulo: Cortez, 2001.
• MATUS, Carlos. Chimpanzé, Maquiavel e Ghandi: estratégias políticas. São Paulo: FUNDAP, 1996a.
• MATUS, Carlos. Política, Planejamento e Governo. Brasília: IPEA, 1996b.
• MATUS, Carlos. Adeus Senhor Presidente. São Paulo: FUNDAB, 1996c.
• MATUS, Carlos. Los 3 Cinturones de Gobierno. Caracas: Fundación Altadir, 1997.
• MATUS, Carlos. Teoria de la Produción y el Juego Social. Caracas: Altadir, 1998.
• MATUS, Carlos. O Líder Sem Estado-Maior. Tradução de Gisela Barroso. São Paulo: Fundar, 2000.
• MATUS, Carlos. Teoria do Jogo Social. São Paulo: Fundap, 2005.
• NUNES, Paulo. Proposta Prevê Mudança na Avaliação. Correio do Povo, Porto Alegre, v. 116, n. 365, 30 set. 2011.
• RAVITCH, Diane. A Vida e a Morte do Grande Sistema Escolar Americano, como os Testes Padronizados e o Modelo de Mercado Ameaçam a Educação. Porto Alegre: Sulina, 2011.
• RIO GRANDE DO SUL. Decreto nº 48.744, de 28 de dezembro de 2011. Institui o Sistema Estadual de Avaliação Participativa do Estado do Rio Grande do Sul - SEAP/RS e dá outras providências. Diário Oficial do Estado do Rio Grande do Sul, Porto Alegre, ano LXIX, n. 250, p. 15, 29 dez. 2011a.
• RIO GRANDE DO SUL. Palácio Piratini. Proposta ao CPERS Sindicato, Porto Alegre, 16 mar. 2011b.
• RUA, Maria das Graças. Políticas Públicas. Florianópolis: Departamento de Ciências da Administração/UFSC. [Brasília] CAPES-UAB, 2009.
• SANDER, Benno. Administração da Educação no Brasil: genealogia do conhecimento. Brasília: Liber Livro, 2007.
• SANTOS, Boaventura de Sousa. A Difícil Democracia: reinventar as esquerdas. São Paulo: Boitempo , 2016.
• SARAIVA, Mateus; LUCE, Maria Beatriz. A Apropriação da Avaliação em Larga Escala Pelas Comunidades Escolares: uma análise do ENEM e do SEAP. Políticas Educativas , Santa Maria, v. 9, n. 1, p. 136-153, 2015.
• SILVA, Maria Beatriz Gomes; MACHADO, Maria Goreti Farias. Sistema Estadual de Avaliação Participativa no Rio Grande do Sul: diagnóstico das coordenadorias regionais de educação em 2013. Políticas Educativas , Santa Maria, v. 9, n. 1, p. 103-120, 2015.
• SILVA, Maria Abádia da. Qualidade Social da Educação Pública: algumas aproximações. Caderno Cedes, Campinas, v. 29, n. 78, p. 216-226, maio/ago. 2009.
• SORDI, Mara Regina Lemes; BERTAGNA, Regiane Helena; SILVA, Margarida Montejano. A Avaliação Institucional Participativa e os Espaços Políticos de Participação Construídos, Reinventados, Conquistados na Escola. Caderno Cedes , Campinas, v. 36, n. 99, p. 175-192, maio-ago., 2016.
• SOUSA, Sandra Maria Zakia. Avaliação Externa e em Larga Escala no Âmbito do Estado Brasileiro: interface de experiências estaduais e municipais de avaliação da educação básica com iniciativas do governo federal. In: BAUER, Adriana; GATTI, Bernadete Angelina (Org.). Vinte e Cinco Anos de Avaliação de Sistemas Educacionais no Brasil: implicações nas redes de ensino, no currículo e na formação de professores . Florianópolis: Insular , 2013. P. 61-85.
• SOUZA, Celina. Políticas Públicas: uma revisão da literatura. Sociologias, Porto Alegre, ano 8, n. 16, p. 20-45, jul./dez. 2006.
• UFRGS. Universidade Federal do Rio Grande do Sul. 12º Relatório Parcial acerca da análise e avaliação externa de processos e de resultados atinentes à avaliação institucional participativa do Sistema Estadual de Avaliação Participativa - SEAP (Doc.195/2016). Maria Beatriz Luce (coordenadora geral). Porto Alegre: UFRGS, 2015.
• VIANNA, Heraldo Marelim. Fundamentos de um Programa de Avaliação Educacional. Estudos em Avaliação Educacional, São Paulo, n. 28, jul.-dez. 2003.
• WERLE, Flávia Obino Corrêa; KOETZ, Carmen Maria. Trajetória do Sistema de Avaliação do Rendimento Escolar do Estado do Rio Grande do Sul. Ensaio, Rio de Janeiro, v. 20, n. 77, p. 677-700, out./dez. 2012.

• Translated from portuguese by Tikinet.
• 1
  All citations were free translated.
• 2
  Available from: <http://portal.mec.gov.br/dmdocuments/rceb007_10.pdf>.
• 3
  The decisive political and administrative structure located in Palácio Piratini, shared by the Governor with the Departments for monitoring and implementating government projects, of which participated representatives of civil society, members of the Brazilian Council for Economic and Social Development, the so-called “Conselhão” (something as “Big Council”) (FGV, 2017).
• 4
  Matéria Diálogo e investimentos marcam nova política educacional gaúcha, diz Tarso Genro. Publication: 04/18/2011, 11h30min at: <https://bit.ly/2lg40gY>. Access on: June 16, 2018.

Publication Dates

• Publication in this collection
  Oct-Dec 2018

History

• Received
  17 July 2018
• Accepted
  26 Sept 2018