Evaluation: the French chefs are still searching for "la nouvelle cuisine"

DEBATE DEBATE

Gérard de Pouvourville

Debate sobre o artigo de Zulmira M. A. Hartz

Debate on the paper by Zulmira M. A. Hartz

Groupe Image, École Nationale de la Santé Publique Saint-Maurice, France.

Evaluation: the French chefs are still searching for "la nouvelle cuisine"

I wish to begin by complimenting Zulmira Hartz for her article on the institutionalization of evaluation in the French context, since in a few pages she provides a brilliant analysis of the history of program evaluation in France. Second, she raises a number of important issues on which I would like to comment briefly here.

The first point concerns the role of evaluation in our parliamentary democracies. Indeed, institutionalization of evaluation should be seen as an attempt to restore to elected officials the power to control the functioning of public administration, the so-called technocracy. Because public action has become so diverse and complex, it is obvious that members of Parliament are increasingly less able to judge by themselves whether the policies they have approved are actually achieving their objectives. Although there are independent inspection bodies in France, quite often their investigations are more audits than true evaluations (i.e., the study of the relationship between means and ends). From that perspective, I must say that I am quite attracted by the US philosophy quoted by Hartz, of ex ante social experimentation. If one recalls the famous distinction made by Hirschman (1970), improving democracy (the power of voice) is thus a legitimate goal. But implementation of such evaluations runs up against numerous obstacles: members of Parliament still depend heavily on the public administration in terms of expertise and access to data, and their own time is scarce. The public administration is also reluctant to release information and data, and there is still a strong feeling that members of Parliament, not being experts, will fail to grasp the complexity of all the issues. Is there a solution to the problem, other than to merely provide more resources and regulatory power to Parliament to conduct autonomous evaluations?

Indeed, if evaluation were considered an intrinsic part of public services management, as Hartz suggests, then at least part of the work would be done on a decentralized basis. It seems to me that what is at stake is much more than just rationalizing production of public services, i.e., just one more managerial fashion. The turn of this century is marked by emphatic rejection of the Welfare State, on the basis that public administration, not under the pressure of competition, cannot be efficiently managed. Governments appear to be under pressure from international financial markets and are forced to demonstrate their capacity to master the public sector's growth. This generates heavy ideological pressure to transfer provision of services to the private sector, under the assumption that if such services come under competitive pressure, there will be a strong incentive for more efficient use of resources. However, because public services must be delivered under the constraints of accessibility and equity, privatization of delivery generally goes with some form of contract with the state, limiting competition and demarcating service providers' autonomy. So the question is, if the game is not real competition, why is it impossible to achieve the same results with public services, to which more autonomy is given, but from which more accountability should be expected? Thus, evaluation would be the condition for accountability and one of the necessary conditions for modernizing the public sector.

Another way to decentralize evaluation is to encourage its development as a legitimate scientific activity, which is not always the case, at least in France. Evaluative research, as it has been labeled by Hartz, is still considered "applied research", as compared to conceptual or basic research. Since it is often funded by public agencies (or private firms, for the economic evaluation of drugs), versus public research money, academic evaluation committees also consider it mere consultancy. It is acknowledged that such work is a true service to the community, but there are major doubts as to whether it is really science. I shall not answer this question completely. I shall merely suggest that mobilization of scientific methods to empirically explore public action and explain its impact on society seems to me to be quite close to science, if one considers that the main aim of science is to explore our reality with controlled methods. It is true that public agencies often ask research teams the impossible task of evaluation programs "due yesterday", and if possible at no expense to the budget (after all, research is already funded by public money elsewhere), thus fostering "quick-and-dirty" studies. When studies are funded by the private sector, they are seen as defending private interests, and as such suspected of bias. For example, drug research in France is funded mainly by the drug industry. Thus, it allows public agencies to disqualify the studies on the basis of their partisanship and to bemoan the lack of independent studies. Funded by whom? Therefore, allow me to plead for special research funding for evaluative research and for the development of a structured research community to define good scientific work.

The three preceding situations ­ evaluation by Parliament, evaluation as a management practice of public services, evaluation as research ­ refer to different facets, different interpretations of the same word. Yet they are complementary. Public services seldom have complete responsibility for a given program. For example, health programs often require intervention and coordination by many actors with different statutes. All may be accountable for their respective parts of the job and use evaluation as an action-oriented activity, articulated with management. Nevertheless, evaluation of the program to which they contribute requires an integrative perspective, which can be fostered by special investigation or research. Hartz mentions another important aspect of evaluation, that it is a never-ending process. Evaluations always come either too soon (the program does not appear to have been fully implemented and stabilized) or too late (irreversibility has been created). But this judgment is based on an erroneous perception of evaluation, conceived of as a one-shot judgment at one given moment in time, allowing for a stop-or-go decision. Rather, evaluation should be seen as a learning process, each step identifying what is already known and what remains to be learned.

Finally, if one sees evaluation as a process to improve conditions for democratic debate in our parliamentary systems, then one must raise the issue of equal access to the expertise required for evaluation. Equal access has two main dimensions. The first relates to public disclosure of evaluations conducted by public services or parliamentary offices, i.e., results that should be publicized as widely as possible. I am aware that many share a pessimistic view of human nature as to whether access to information and quality knowledge improves our societies (Revel, 1988). A Machiavellian view of governance also tends to argue over la Raison d'État. But an organization seldom has the capacity to adapt itself from the inside, and it often needs "exogenous shocks" to improve. Moreover, in the case of public services, citizens are often captive customers. Evaluation makes public services more accountable to the people they are supposed to serve. My second point is more utopian. In democratic nations, access to free legal counsel is guaranteed for those who cannot afford to hire a lawyer to defend them in court. There is no guarantee that this lawyer will do the best work in the world, but at least free legal aid is provided. Access to evaluation is certainly not distributed equally among socioeconomic categories. Some actors have the resources to build their own evaluation of public services and use this to lobby, promote, or protect their interests. Is it possible to imagine that politically weaker constituencies could hope to counterbalance economic power and be supported by public money to develop their own evaluation?

Have we really met these requirements in France, as the record of achievements listed by Hartz might suggest? Actually, we are still far short of many objectives, in both the health sector and others. The role of parliamentary evaluation is modest because of the modest level of resources invested, evaluation is far from being accepted as a normal management practice in public services, and evaluative research lacks legitimacy. But it may well be that health will be a model for other sectors of public intervention, because of severe exogenous shocks, not only financial, but also scientific, and through the emergence of major public health issues such as "new poverty", AIDS, prions, population aging, and others.

HIRSCHMAN, A. O., 1970. Exit, Voice and Loyalty. Boston: Harvard University Press.

REVEL, J. F., 1988. La Connaissance Inutile. Paris: Grasset.

Publication Dates

• Publication in this collection
  18 Mar 2003
• Date of issue
  Apr 1999