DEBATE DEBATE
André-Pierre Contandriopoulos
Debate sobre o artigo de Zulmira M. A. Hartz
Debate on the paper by Zulmira M. A. Hartz
Is the institutionalization of evaluation sufficient to guarantee its practice?
Zulmira Hartz, through her impassioned analysis of the institutionalization of evaluation in France and the situation in the Anglo-Saxon countries as a backdrop, aims essentially to understand what Brazil can learn from other countries' experiences. Her purpose is to provide precise elements for the Brazilians in charge of creating the "Council on Higher Studies and Technological Evaluation" and the "Department of Policy Evaluation" under the Ministry of Health in order to help them create the bodies needed for the emergence of a "culture of evaluation" in the country. She makes her basic premise clear in the introduction to her article: "I believe [...] that one can take advantage of the lessons learned from the more advanced countries in evaluation programs for the evaluation of programs, as in the case of the US model for agencies in charge of public health interventions [...] or the difficulties experienced by those who have more recently begun to build an evaluation policy for the evaluation of policies", and in the conclusion, where she states, "I believe I have learned from the example of what France and other countries have attempted to do. [...] I feel that such initiatives should not be seen as a déjà vu of others (that were not always successful); on the contrary, they are a stimulus for leapfrogging stages, insights for a regulatory and organizational framework drawing us closer to a situation characterized by the notion of what "seem the right points". [...] Nevertheless, without an effort at institutionalization by political and government structures so as to introduce technical and financial incentives and encourage a culture of evaluation for decision-making and program budget allocation, all this knowledge will be nothing but an academic exercise, powerless to help solve the problems identified."
This last observation raises two issues: First, is the institutionalization of evaluation sufficient to generalize the practice of evaluation, i.e., for a "culture of evaluation" to emerge in society? Second, does the institutionalization of evaluation not create the risk of limiting debates and innovations, thus consecrating the domination of the techno-scientific approach in collective decisions, to the detriment of the other three types of regulatory logic used in the health system (professional logic, economic logic, and democratic logic)?
1) Institutionalization of evaluation and practice of evaluation
The history of the institutionalization of evaluation in France shows quite clearly that it is one of the state's responses to the need to rationalize expenditures. Health systems are currently deteriorating in countries all over the world. Their internal dynamics, based on technological development, demographic and epidemiological transitions, new knowledge, and globalization of the economy pushes them in a direction incoherent with society's values. The chasm is growing between what the population wants and the status quo offers. The crisis is profound (Contandriopoulos, 1995).
Most countries are left in a paradoxical situation that can be summed up as follows: on the one hand, economic imperatives linked to globalization of the economy and expansion of the public debt, forcing governments to reduce expenditures to balance their insufficient budgets, making them dependent, while both domestic and international financial markets rob them of enough autonomy to govern democratically and maintain competitive positions in the world. On the other hand, cuts imposed on social programs, especially in the health system, challenge the foundations of the state's legitimacy and thus its ability to deal with the rationalization imposed by economic logic.
To the extent that one cannot consider merely one of the components in the paradox (that of maintaining the status quo in the health system) to the detriment of the other (balancing the budget), since they are interconnected, whatever choice is made challenges the very existence of society; thus the need to innovate. It is necessary to change the system in order to allow it to continue to respond to the people's demands by offering quality services to all those whose state of health requires them, increasing their efficiency. The institutionalization of evaluation, which is conceived of as a formal mechanism for the production of information on public programs and policies, is a prime component of rationalization policies in countries worldwide. It aims to enhance the performance of interventions by the public sector.
Yet one might ask, is such institutionalization of evaluation sufficient to achieve the planned gains in efficiency? That is, in order for evaluation to become a routine practice by all social actors, what are its spheres of responsibility?
It is not enough that the demand for efficiency be perceived by public opinion as sufficiently legitimate to have evaluation institutionalized; rather, the various actors must incorporate this new social standard into their ways of conceiving their own responsibilities and those of others. Evaluation in the field of health must become a normal, routine practice, a natural need for the various actors. Only then will evaluation become truly operational.
The conditions for this new culture to spread in the health field (as well as in other areas of the public sector) are difficult to define, but are certainly highly demanding. Generalization of a new culture in a society is a major change. It will no doubt face resistance from all those for whom the prevailing order is advantageous. In order to grasp the nature of the difficulties involved in such change, it is crucial to have a clear understanding that the institutionalization of evaluation is a process that runs the risk of considerably increasing the weight and influence of actors whose legitimacy rests on the value of technocratic logic, to the detriment of others whose credibility is based on other types of regulatory logic.
2) Institutionalization of evaluation and democratic culture
We can say rather schematically that four major groups of actors interact in the health system (Evans, 1981). Professionals, managers (those that manage, plan, and control the health system: payers, administrators, and public sector employees), the market, and the political sphere, consisting of all participants in state political activities.
The population as such does not constitute a group of actors. However, the legitimacy of each of the other four groups depends to a major extent on the possibility of speaking in the people's name. In order to prevail, each group attempts to convince society that it is the true representative of collective interests. Health professionals speak of both their patients and the various diseases, while health care managers prefer such terms as "users", "beneficiaries", or "policy-holders". The world of the market speaks of consumers, customers, clients. The state, in turn, speaks of citizens, voters, taxpayers, or the general public. By definition, individuals can only speak for themselves and call themselves by the group to which they belong (employees, managers, functionaries, workers, men, women...), but none of them individually can speak for the population in the broader sense.
Each group of actors can be characterized by its view of the health care system, its concept of health, disease, and their determinants, the types and relevance of the resources it controls, its will to improve its position in society by increasing its control over the health care system's resources (financial, human, material, and symbolic), and the type of logic it feels should serve to regulate changes and interactions in the system. We see this as the circumstances providing the basis for actors' perceptions as to the role and pertinence of the types of regulatory models coexisting in the health system.
Professional logic is that which prevailed in the developed countries during the first half of the 20th century. Its legitimacy derives in part from the huge progress made by scientific medicine in both its understanding of disease and ability to intervene, on the one hand, and from the fact that despite such progress, medicine remains an art. Application of medical knowledge to individual problems is never automatic or direct. It depends in a determinant way on an expert professional opinion and patient trust. Patients lack both the necessary knowledge to decide alone on their disease and the ability to exercise "rational" judgment when they are ill, suffering, and/or stressed. In this context, patients are represented by health professionals in general and physicians in particular. The health care system is conceived of as a means whereby patients and physicians meet and the latter can exercise their expertise on behalf of patients as freely and completely as possible. Physicians are thus at the center of the health care system, and allocation of resources rests on their decisions. The medical profession controls the amount and quality of the various services available to the population.
All decisions constraining the resources to which physicians have access pose a potential threat to patients. Under this model, the role of public powers is to reduce insofar as possible the barriers against utilization of health services, especially by employing health insurance systems, adequately funding the health care system, and taking charge of public health programs.
According to this logic, the medical profession's responsibility consists of applying self-regulatory mechanisms so as to provide a public guarantee of adequate physician training, quality services, and professional respect for a code of ethics, aimed at assuring that services are neither under- nor over-utilized. In a word, the profession aims to guarantee the community's interest.
Technocratic logic is the counterweight to professional logic. Its basic premise is that it is possible to rationalize by mobilizing scientific approaches in order to define how to best employ resources so as to meet the population's needs. Decisions concerning the health system should all be subject to a rational planning process based on an analysis and prioritization of needs, detailed programming of activities, optimum definition of the resources required to optimize treatment of priorities, and evaluation of the results.
According to this perspective, decisions revolve around the experts, through their command of analytical approaches to rational decision-making. Medical activity should be evaluated, just as all other activities. It is possible, from the outside, to judge the efficacy and pertinence of the use of different services and to define which services should be provided, based on convincing data, i.e., "evidence-based medicine". To avoid arbitrary conduct by physicians, clinical decisions should be strictly framed within guidelines for practice, which is increasingly facilitated by new information and communications technologies.
The state, and especially the government apparatus, should take charge of the health system's planning so as to guarantee the population that public resources are applied rationally and that no group of actors monopolizes their use. Technocratic logic purports to be rational, apolitical, and completely devoted to maximizing the collective interest.
Economic logic refers back to professional and technocratic logic. Based on classical neoliberal economic theory, it demonstrates that resource allocation is optimized when supply and demand operate freely in competing markets. Proponents of this view contend that health services are not truly different from other goods, that their characteristics (information imbalance, random disease occurrence, presence of externalities) are insufficient to keep the free market from functioning. The state has no more reason to intervene in the health domain than in any other, much less to predefine a budget package. The state should be content to curtail, insofar as possible, market imperfections (by eliminating monopoly situations created by professional corporatism, restricting its participation to financing only a minimum packet of basic-need services and public health services that are truly public goods, facilitating dissemination of information on health services), leaving regulation of the system to the care of the market's invisible hand.
According to a democratic logic, citizens have the right and responsibility to influence socio-political actions and decisions within society. This democratic right can be exercised directly or indirectly. It is usually exercised indirectly through representatives that have been elected or coopted. Within the health system, democratic logic allows for the association of each member of the population regardless of income, schooling, age, or place of residence, with regard to both the formulation of needs, problems, priorities, and solutions and the very management and administration of the system as a whole and each of its parts. In order for democratic logic to be expressed, it is not enough to organize elections; it is also (or perhaps especially) necessary for there to be true room for debate in society. Democracy aims not only to allow citizens to express their preferences by voting, but also to use discussion and controversy to improve their democratic culture. This is indispensable for everyone to be able to reflect critically on what they conceive of as fair for themselves, for the groups to which they belong, and for society as a whole (Touraine, 1994)
Evaluation, which we can conceive of as a formal mechanism for producing information on a given intervention (such as a policy or program) in order to help decision-makers allocate resources optimally, is inscribed quite naturally within technocratic logic.
Institutionalization of evaluation provides a strong legitimacy for actors who value this form of regulation and reduces that of others who defend other types of regulatory logic. The latter, in a context where scientific rationality prevails, resume, in their own way, the discourse concerning the need to base decisions on reliable data (evidence-based decisions). The stakes shift in the struggle for legitimacy. What matters is no longer to officially defend a logic of regulation, but to control the methods (or frame of reference) used to produce information on interventions. Control of the evaluation mechanism thus becomes the central wager in the struggle among the four major groups of actors.
Professionals attempt to impose the idea that a given intervention's quality lies in the use of methods from biomedical research and clinical epidemiology. Randomized trials have become the "gold standard" for proper evaluation. The results of research using these approaches feed the groups of experts expected to define best practices standards. They are thus the origin of the benchmarks used to fit professionals' clinical activity in what has come to known as "evidence-based medicine". Such methods are also recommended in the burgeoning field of pharmaco-economics, in which the contribution from economics per se is limited to calculating the cost of interventions.
The experimental method is a research mechanism that allows one to isolate an intervention from its surroundings in order to precisely assess its observed effects. This approach is most applicable when the intervention under study is limited in time and not contingent on a given environment (such as the evaluation of a drug, technology, or treatment modality). It focuses on measuring the effects and is not concerned with explanatory factors for the observed result. One can see why professionals value this approach, which applies perfectly well to the technical component of their activity. Meanwhile, this approach allows them to publicly subscribe to the idea that medical practice can be subject to evaluation with an on-going concern for improving quality, while excluding from the field of evaluation the part of their practice which cannot be evaluated by clinical trials, i.e., referring to the portion of their activity where interpersonal relations are superimposed on the application of techniques. In fact, medical professionals' exclusive focus on the experimental method produces little agreement as to evaluation's credibility in relation to new forms of organizing and financing the health care system, whose evaluation requires other methods.
For planners and managers, evaluation is a normal activity in their practice. It aims primarily to place a critical focus on decisions in order to improve them. It also serves to predict the consequences of alternative models for organizing the health care system (strategic analysis). The most widely adopted approach aims to compare resources or activities with a benchmark (normative evaluation) with a view towards improving the health care system as a whole and its component organizations. Planners also approach evaluation through the implementation of pilot projects. Managers and planners generally acknowledge the validity of social science research methods. The fact that the results of their work are by nature contingent on a specific context allows them to express their adherence to the principle of evaluation, while maintaining an important margin of autonomy in decisions pertaining to their organizations.
According to economic logic, formal evaluation of interventions is conceived of as a method allowing various decision-makers to make rational decisions concerning resource allocation in areas where not all conditions are present for the proper functioning of the free market. This is particularly valid for the health field, ripe with uncertainties and externalities. For liberal economists, evaluation is an inevitably imperfect substitute for the market's invisible hand. For others, evaluation is a formal mechanism providing decision-makers with the information they need to make the best possible decisions on behalf of the community. Economists have the same expectations as planners vis-à-vis economic calculations. Both postulate that it is possible to measure the consequences of different interventions and compare them. This epistemological position is refuted by proponents of the democratic logic of regulation.
According to Touraine (1994) democracy is "a regime which recognizes individuals and groups as subjects, that is, which protects them and encourages them in their desire 'to live life', to provide a unity and meaning to the life experience. Thus, the limit to power is not only a set of procedural rules, but the positive will to each and every individual's freedom. Democracy is the subordination of social organization, especially political power, to an objective, which is not social but moral: individual freedom. Such a task would become contradictory if it were fully accomplished, since it would dissolve society, yet it is put to work in democratic societies, as opposed to the forces of domination and social control, in order to increase each individual's portion of initiative and quest for happiness, making known to each social actor the rights of others to formulate projects and conserve their memory." (Touraine, 1994:262).
To the extent that evaluation participates in the development of a truly democratic culture, thereby institutionalizing, at all levels of society, whatever favors individual and collective learning processes (Crozier et Freidberg, 1977), it becomes a tool for change and innovation (Palmade, 1996; Denis et al., 1997). In this case, evaluation is no longer a tool for the power of a group of actors; rather, it allows for a critical view of the established order and becomes a true instrument of liberation. It allows all actors to step back from the exclusive undertaking of mere regulatory logic.
The institutionalization of evaluation in favor of the emergence of an authentically democratic culture fosters the subordination of vested interest groups' power to that of individuals who collectively constitute society.
CONTANDRIOPOULOS, A.-P., 1995. Reformar o sistema de saúde: uma utopia para sair de um status quo inaceitável. Saúde em Debate, 49-50:53-64.
CROZIER et FREIDBERG, E., 1977. L'Acteur et le Système. Paris: Seuil.
DENIS, J.-L. et al., 1997. Organiser l'innovation, imaginer le contrôle dans le système de santé. Ruptures, 4.
EVANS, R., 1981. Incomplete vertical integration: the distinctive structure of the health care industry, In: Health, Economies and Health Economics (J. Van der Gaag & M. Pelman, ed.), pp. 320-354, North Holland.
PALMADE, J., 1996. Rationalité économique/finalité en valeur, une tension indépassable? 17ème Colloque Jean-Yves-Rivard "Organiser l'Innovation, Imaginer le Contrôle dans le Système de Santé", Montréal: Département d'Administration de la Santé, Université de Montréal.
TOURAINE, A., 1994. Qu'est-ce que c'est la Démocratie? Paris: Fayard.
Publication Dates
-
Publication in this collection
18 Mar 2003 -
Date of issue
Apr 1999
