Evaluation of health programs, strategies, and actions: a dialogue with critical realism

ABSTRACT

Critical realism can be defined as an approach within the philosophy of science and social theory, which proposes a scientific model of explanation about the reality that denies the traditional epistemological poles of positivism and relativism (or idealism). It gathers different authors and has implications on the debate about the future of contemporary sociological theory. Regarding the field of evaluation of social policies and programs in health, the critical realistic approach has brought news perspectives regarding evidence, impact, and results. This theoretical essay aimed to present the main contributions of critical realism to the evaluation of policies, programs, and interventions in health, as well as to dialogue with authors that have addressed and reflected directly or indirectly on the ontology of the programs in the practice of evaluation and its implications for the design of such interventions.

KEYWORDS
Health evaluation; Program evaluation; Health inequalities

RESUMO

O realismo crítico se constitui em uma abordagem, no âmbito da filosofia da ciência e da teoria social, que propõe um modelo científico de explicação acerca da realidade que evita os polos epistemológicos tradicionais do positivismo e do relativismo (ou idealismo). Reúne diferentes autores e traz implicações para o debate sobre os caminhos da teoria sociológica contemporânea. No que se refere ao campo da avaliação de políticas e programas sociais em saúde, a abordagem realista crítica tem trazido novas perspectivas e contornos para a problematização sobre evidências, impactos e resultados. Assim, o ensaio teórico teve por objetivo apresentar as principais contribuições do realismo crítico para a avaliação de políticas, programas e intervenções em saúde e dialogar com autores que se apropriaram, direta ou indiretamente, do debate e da reflexão sobre a ontologia dos programas na prática da avaliação e suas implicações para o desenho das intervenções.

PALAVRAS-CHAVE
Avaliação em saúde; Avaliação de políticas e programas; Iniquidades em saúde

Introduction

In the last decades, public policies have been characterized by being complex initiatives that involve multiple causal mechanisms, non-linear social processes, open social systems, and uncertain, contingent, and unforeseen contexts. The intricate nature of these initiatives implies the recognition that their implementation is linked to the various institutional and community dynamics existing in each context. Moreover, public policy interventions tend to mobilize different resources and reach different targets. Barnes, Matka, and Sullivan¹ believe that complex initiatives require multilevel changes, that is, changes in the pattern of individual choices, the nature of community ties, the epidemiological profile, and the institutional organization.

The dynamics of health intervention contexts challenge evaluative research, both concerning the initiatives per se and also due to the need to use comprehensive theoretical and methodological references. Regarding the assessment of complex health initiatives, it makes less sense to evaluate policy in isolation since its effects on the population cannot be artificially separated². In the Brazilian case, the Family Health Strategy is an example of intersectoral action that requires joint efforts in the territory and multi-professional teams.

It is based, therefore, on the premise that it is impossible to isolate the effects of public policies since they mix and acquire singular characteristics, given the diversity of social situations and conditions and the infinite possibilities of appropriation by the respective users. Therefore, it is pertinent to use evaluation theories that allow understanding the social mechanisms and contexts underlying the changes to be brought about by policies, programs, and initiatives, as proposed by Chen³.

Despite the contributions of the different approaches used in policy evaluation, some limits can be found. Generally speaking, gaps regarding the dialogue with social theory – understood as a dimension of sociology or social sciences that addresses major theoretical problems⁴ – are found, while giving less value to the understanding of social conditions and contexts in which the results were achieved^5-10. Theoretical and methodological inputs were produced within social sciences, in the sense of providing supporting elements for the field of evaluation, and incorporated by authors who, in the 1980s, criticized approaches (more widespread) that did not consider the context and were centered on results of programs and interventions¹¹.

Such criticisms resulted in the analysis of the components, resources, objectives, and intended effects of the programs in an interdependent manner and guided by the challenges and opportunities specific to each social context. Thus, a reasonable consensus that different policies and programs are a materialization of ideas, expectations, conceptions, and theories about social processes of change emerged. The restricted focus on so-called outputs or the financial resources employed is unlikely to favor learning about which components are most affected by local dynamics, and improve decision-making based on relevant evidence¹². Undoubtedly, the emphasis on program theory represents a significant shift in evaluative practice. Chen¹³ affirms that the failure or success of the actions must be linked to the theory about the problem and the process of implementing the programs that aim to face it.

To dialogue with evaluation perspectives that discuss the production of evidence in health and consider the intricate contexts of interventions, this essay aims to bring supporting elements for the evaluation of health policies and programs from authors who, directly or indirectly, debated critical realism issues to reflect on the ontology of programs and the practice of evaluation. In this perspective, the approach brought by critical realism on the nature of public policies and implementation contexts proves to be a comprehensive and challenging theoretical and methodological alternative. The essay intends to establish connections between critical realism and evaluation, highlighting some of the multiple aspects that are influenced by it, such as ‘theory-driven evaluation’. Particular emphasis is placed on the contributions of critical realism and realistic evaluation to the debate on health policies and programs effectiveness.

Health evaluation: possible dialogues with critical realism

This essay presents how critical realism is a scientific model of explanation about reality, and opposed to others within social and human sciences. These models support perspectives on science and theories (including evaluation), and, depending on what is adopted, the explanatory capacity may be associated with a greater or lesser emphasis on the position of the observer or the disciplinary field to which he/she is affiliated.

The realism model is based on some assumptions, and critical realism is located in this debate, which is why it is necessary to bring some concepts and terms from Roy Bhaskar (1944-2014), one of the primary authors who proposes a realistic ontology. Once the debate has been outlined, the essay seeks to establish theoretical-methodological relationships between realism and the evaluation of health programs, and how the theory-driven evaluation appropriated this debate, thus allowing the formulation of other perspectives of analysis. Given the explanatory scope of critical realism, we can conclude that theory-driven evaluation has produced essential inputs for the analysis of intersectoral, multi-strategic, and equality-focused programs.

Realism is an approach within the philosophy of science and social theory, which proposes a scientific model of explanation about the social reality that denies the traditional epistemological poles of positivism and relativism (or idealism). We can affirm that, even today, realism is one of the key aspects of the construction of knowledge. In the case of contemporary sociological theory, Sell¹⁴ calls attention for the debate between realists and anti-realists. Realism is called upon to answer the question, “to what extent do scientific theories access and describe the world as it is?”¹⁴⁽²⁰⁾, and supports the thesis of the correspondence between theory and social reality. It differs, therefore, from anti-realism (or idealism), which argues that all knowledge is determined by the observer’s context, thus denying the possibility of an independent or external world. Schematically, we can, thus, illustrate:

Figure 1
Summary of the dimensions of reality

However, despite the previous illustration, and as emphasized by Sell and Hamlin¹⁵, it is not a matter of merely transposing the debate from philosophy to sociology, nor is it possible to establish rigid boundaries between realism and anti-realism. Furthermore, other authors, such as Bhaskar, will make these limits porous and tighten the relationship between knowledge and social reality.

Sell emphasizes that a controversy (and dispute) is currently found between the realistic and anti-realistic approaches to scientific-social knowledge, in which the ability to access the constitutive properties of external reality is in check. The realistic stance, among other aspects, presents the conception that there is an external world independent of the observer, and that, even so, it is possible to know it in its structures and properties even considering the social conditions subsumed in the perception¹⁴. We should underscore that this is not a contrast between the social construction of reality and the reality that is independent of this construction. Authors who dialogue with the tensions between realism and anti-realism argue that other possibilities of analysis arise, including those that operate with the principle of symmetry when considering the processes of co-construction.

A fundamental aspect of critical realism¹⁵ is that it is an alternative to positivist and postmodern approaches. The first, which tends to identify the real with what we can know empirically, and the second, by conceiving it as a social construct. In Hamlin’s words:

Bhaskar’s criticism of dominant philosophies gave rise to a general ontology that defines reality as something different, structured, and constantly changing¹⁵⁽⁴⁷⁾.

An approximation of critical realism requires knowing some key concepts and ideas developed by Bhaskar^16,17. In the case of health, the questions asked by the author are central: first, by claiming a stratified ontology of reality; second, for presenting a critical view concerning the social practices studied, including social sciences and other theories.

Critical realism and Roy Bhaskar

It is essential to bring some concepts developed in ‘Philosophy and scientific realism’ and ‘A realist theory of Science’. As highlighted by Bhaskar, any philosophy of science must find a way of handling the central paradox of science, namely: human beings in their social activities produce knowledge that is a social product as any other and, therefore, dependent on its production and who produces it; this is one side of knowledge as per this author¹⁶. The other side refers to the knowledge of things that are not produced by human beings.

Here we find an essential division between what is ‘independent’ of human activity (for example, the virus and other examples from the natural sciences) and is established as ‘intransitive objects of knowledge’ and what ‘depends’ on human activity and, therefore, in contrast, represents the ‘transitive objects’. Bhaskar calls the following ‘raw science materials’: facts and theories, paradigms and models, research methods, and techniques available in a given scientific school¹⁶. As summarized by the author, intransitive objects are generally invariable concerning our knowledge about them and, thus, represent the real things and structures, mechanisms and processes, events, and possibilities in the world and, mostly, are independent of us¹⁶. In the diagram below, a synthesis illustrates the epistemological (which concerns knowledge) and ontological (which concerns being) realms:

Figure 2
Summary diagram of the stratified reality

An ‘adequate philosophy of science’ should support both the social character of science regarding the independence of science from the objects of scientific thought. ‘Health science’ (biomedicine and public health, for example), like any object of knowledge, also consists of transitive and intransitive objects. What we call viruses and bacteria and their effects on the human body, for example, are independent of knowledge about them.

Another vital aspect of Bhaskar’s approach refers to the distinction he makes between empirical realism, transcendental idealism, and transcendental realism or, as summarized, the three traditions in the (Western) philosophy of science. As synthesized by Sayer, the objects of knowledge are neither phenomena (empiricism) nor human constructs imposed on the phenomenon (idealism), but, rather, real structures that persevere and operate independently of our knowledge, our experience and the conditions that allow accessing them¹⁸. From the perspective of transcendental realism, science is not an epiphenomenon of nature, nor is nature a human product.

Only transcendental realism can support the idea of a world governed by laws that are independent of human beings, which is why it is necessary to understand science¹⁶. Besides other issues developed by Bhaskar (and which cannot be arrested due to the scope of the essay), we should underline that, while transcendental realism explicitly asks “what the world must be like for science to be possible?”, classical philosophy asks “what science would have to be like for the knowledge yielded to be justified?”¹⁶⁽³³⁾. In the second question, the author believes it is assumed that knowledge is justified, and not seen as a ‘process on the move’.

Bhaskar characterized one of the problematic aspects of this view as an ‘epistemic fallacy’ that is, the possibility that the ontological questions (‘what is this?’) can be rephrased as if they were epistemological questions (‘how can we know this?’)^16,19,20. In the words of Vandenberghe^21(179):

Those who confuse our knowledge and models of reality with reality perpetrate the epistemic fallacy. Like Kuhn, they think that the world also changes with every paradigm shift. The world is what it is. It is not, as Wittgenstein thought, the totality of facts, but the totality of intricate things, interactive processes, and structural relationships with causal powers.

Bhaskar believes that the anthropocentric and epistemic biases of classical philosophy led to the dissolution of the concept of an ontological realm, and which we need to make the transitive process of science intelligible. A fundamental aspect is, therefore, elaborated by Bhaskar, to reconstruct the dissolution of the ontology of science, a task that depends on the philosophy of science. In the reconstitution of this ontology, in the author’s words, the causal laws have been mysterious entities: “What can it mean to say that they have a real basis independent of events?”¹⁶⁽³⁵⁾. The answer to this question requires the development of a non-anthropocentric ontology of structures, generative mechanisms, and active things.

In developing this ontology, it is essential to understand that laws are not empirical statements, but statements about the forms of activities typical of things in the world. Thus, he distinguishes between the real structures and the mechanisms of the world, and the current patterns of events that are produced¹⁶⁽³⁷⁾. He says:

The world consists of mechanisms not events. Such mechanisms combine to generate the flux of phenomena that constitute the actual states and happenings of the world. They may be said to be real, though it is rarely that they are actually manifest and rarer still that they are empirically identified by men. They are the intransitive objects of scientific theory.

He also adds that “the world consists of things, not events”¹⁶⁽⁴¹⁾ and that most things are complex objects since they involve a set of trends, obligations, and powers. The complexity of such objects is directly related to the experimental component of science (in closed and open systems) whose plasticity, as reiterated, is not given: it is an ‘experimental design’ that contains substantial theoretical work. As formulated by the author¹⁶⁽⁴⁴⁾:

In stressing the practical component of experimental activity, it is important not to forget the theoretical side. In an experiment men put a question to nature. But they must put it in a language that nature understands, as well as in a form that makes possible an unambiguous reply.

Two types of problems arise from his analysis of experimental activity. First, much of what we know about science has proceeded as a way of ‘thinking’ rather than a real experiment. Second, in many other fields of science (history, social sciences, and even biology), experimental activity is impossible in the sense of ‘mental experiments’.

Causal structures and mechanisms that generate nature can exist and act independently of the conditions that allow human beings to access them. For this reason, they must be assumed to be structured and intransitive. Mechanisms, events, and experiences then constitute three overlapping realms of reality, that is, the realms of the real, the actual, and the empirical. As summarized by Hamlin, it is essential to remember that the realm of the empirical can be accessed from direct observation (experience), that of the actual refers to events that occur in reality, whether they are perceived or not, and that of the real includes the mechanisms; that is: “underlying processes or structures that generate the events”²²⁽⁵⁵⁾.

It is crucial to understand how these three realms overlap, as it is the activity of science that makes them possible. The experiences and the facts they generate are social products and the conjunctions of events that – when apprehended in the experience – provide the empirical bases for the causal laws that are also social products¹⁶. Because they are social products, what was transformed through scientific training was not the knowledge itself, but rather the human beings themselves. Therefore, causal powers concern human beings in their ability to transform the natural world of which they are a part. However, as Bhaskar noted, the concept of the empirical world is anthropocentric, given that the world is what humans experience. In this sense, the concepts of ‘empirical’ and ‘sensation-experience’ belong unequivocally to the social world of science.

Belonging to the social world of science means saying that experiences are a part of it, and, when placed in the context of social activity, they acquire an epistemologically critical part. However, as he warns, precisely because they are a part, experiences cannot be used to define the world. They are the result of social production processes and constitute the end of the journey, not the beginning. Under artificially established and controlled production conditions, it is then possible to access the active and permanent structures that generate phenomena in the world. For this reason, he states that empirical realism depends on a reduction from the real to the realized and from the realized to the empirical and, thus, presupposes a closed world and complete science.

As informed by Sayer²³, critical realism is a variant of transcendental realism (philosophy of and for natural science). In transcendental realism (or critical realism), laws are categorically independent of human beings just as events are of experiences. Knowledge, cognitive skills and causal powers are part of the social context of science whose course is continually being extended and for which there are no a priori limits. According to this perspective, for this extension to occur, science is incomplete: some laws, at any given moment, are unknown, and there is no need for any particular law to exist.

Critical realism is related to critical social science. In this case, social phenomena are found in the intransitive realm of science, which poses other challenges. Sayer emphasizes that, in critical realism, the world is independent of one’s thoughts about it. Therefore: “When theories change (transitive realm), this does not mean that what they deal with (intransitive realm) necessarily changes”²³⁽⁸⁾. For this reason, social scientists play the role of ‘interpreting’, not ‘building’ the social world²³. Therefore, the “stratified ontology” or the “realistic ontology”, in the words of Sayer²³⁽¹¹⁾:

[...] is the recognition of the possibility that the powers can exist even when not exercised, and, thus, what happened or what is known to have happened, does not exhaust what could have happened or everything that happened. Like real objects, it allows and presents, at a given moment, restrictions to what can happen, but does not determine what will happen, [...] therefore, it facilitates the understanding about how we could be or become things that we currently are not.

For Sayer, in critical or transcendental realism:

the world has an ontological depth: events derive from the operation of mechanisms, which derive from the structures of objects, and these are located in geo-historical contexts²³⁽¹⁵⁾.

However, as social phenomena are “intrinsically significant”, there is an interpretive or hermeneutic realm in social science, or “double hermeneutics”²³⁽¹⁸⁾. This is a distinctive factor of the social sciences compared to the natural sciences (a single hermeneutics) since an interpretative work (researcher/researched) is required.

However, how does critical realism propose to understand meanings? According to Sayer: “the meanings are related to material circumstances and practical contexts in which communication occurs and to which it refers”²³⁽¹⁹⁾. That is why the author puts an emphasis, among other aspects, on the material, non-discursive realm of social life. Although he affirms the hermeneutic or interpretative realm of the social sciences, he seeks causal explanations (in the sense of causation) for material changes, and takes as a principle that reasons can also be causes that move towards actions: “actions presuppose an already existing set of structures that include shared meanings”²³⁽²⁰⁾. Furthermore, it is essential to remember that it is not enough to interpret subjective meanings:

[...] much of what happens does not correspond to the actors’ understanding; there are unintended consequences and unrecognized conditions, and things can happen to people regardless of their understanding²³⁽²²⁾.

In this sense, Sayer draws attention to the risk of naive subjectivism and objectivism, since knowledge and practice are interconnected. Thus, answering the empirical questions presupposes also answering the questions set by the concepts used to identify the objects.

Finally, as highlighted by Crotty²⁴, it is worth remembering that ontological and epistemological issues tend to emerge together, which is why addressing the construction of meaning implies dealing with the construction of a meaning of reality. The author argues that the existence of a world without a mind is considerable, but it is not the case for a meaning without a mind, which is why one can reconcile realism in ontology and constructionism in epistemology. Giddens and Sutton²⁵ indicate that critical realism is now seen as an alternative to social constructionism since it conceives the production of knowledge on other bases. Albert et al.²⁶ question the understanding of the ontological and epistemological realms of critical realism by observing that they tend to be ‘insufficiently realistic about epistemology’ and propose alternatives.

In summary, from the perspective of critical realism, some considerations become fundamental for the understanding of natural and social sciences and, especially for health evaluation. Among these, the following stand out: the world is independent of the knowledge we have of it (‘the world is what it is’); it is stratified and differentiated (domain of the real, the actual and the empirical); it consists not only of events but of objects, including structures that have powers and possibilities to generate events; social phenomena (actions, texts, and institutions) are concept-dependent; science and any kind of knowledge is a social practice; mechanisms and structures are in interaction; the social sciences operate in a double hermeneutics; the material (non-discursive) realm of social life is beyond interpretationism.

These propositions dialogue with the approaches of ‘theory-driven evaluation’ (and its strand of realistic assessment) and bring challenges to the field of public health. When incorporating critical realism, it is necessary to develop methodological designs (of research and evaluations) that consider the transitive and intransitive domains of knowledge, as well as the ontological and epistemological. In other words, the emphasis on the transitive domain often leads the collective health debate to focus on modes of knowledge without necessarily asking itself about its objects and what is this field, and what are its limits and possibilities. On the other hand, different ontological perspectives inform about the reality and the ways of acting in health from different socio-historical contexts.

Realism, critical realism and evaluation of health programs

We can affirm that critical realism has influenced the debate about the evaluation of programs through authors such as Pawson and Tilley, who, more than 20 years ago, proposed the ‘realistic evaluation’²⁷. The dissemination of critical realism in health studies occurred through research in evaluation, and can, therefore, be expanded. The critical realism, disseminated by Bhaskar, started to be incorporated (directly and indirectly) by a varied set of authors who carry out research in the evaluation and favor dialogue with the social sciences.

Evaluation research has followed a path marked by different and multiple approaches, and, for some authors, theory occupies a central place, which allows relating the practice of evaluation to studies in social theory and philosophy of science. The theory, among these authors, has different meanings, but, in common, they recognize the importance of knowing the theoretical models on which the research under evaluation is based. The place held by theory is crucial and differentiates these authors, either because they favor the construction of theoretical models, or because they attribute to theories a key element in the evaluation process. Authors who propose an approach to evaluation that prioritizes theory have developed research that takes public programs and policies as a reference. Having the programs as the object of evaluation demands an investigative question about the nature (or framework) of the programs, which requires concepts and knowledge about the conception and execution of actions or interventions to be implemented in specific social contexts.

Theory-driven evaluation, according to several authors^13,28-30, is part of the study of program ontology. This evaluative perspective gathers different authors and constitutes an extensive debate from criticisms and contributions accumulated in the last decades. It covers multiple nomenclatures. However, realist evaluation and theory-driven evaluation are not synonymous but are intertwined. Therefore, different denominations were adopted, thus being characterized by different approaches. According to the systematic review carried out by Coryn et al.^31(201), the term is adopted to:

to denote any evaluation strategy or approach that explicitly integrates and uses stakeholder, social science, some combination of, or other types of theories in conceptualizing, designing, conducting, interpreting, and applying an evaluation.

A more in-depth study on the (still scarce) dissemination of theory-driven evaluation is required in Latin America and Brazil to understand better the dialogue between the authors and the evaluative experiences put into practice. Studies on health promotion question the biomedical model of evidence production, gather authors who dialogue with the theory-driven evaluation. As highlighted by Potvin and Chabot³²⁽³³⁾:

Epidemiology has traditionally formed the methodological basis for evaluating public health interventions. [...] Social processes, however, are not part of the same nature as risky behaviors. They only acquire and produce meaning concerning their spatial and temporal contexts. This network of social relationships must be captured by the evaluation of health promotion to leave the realm of classical epidemiology.

In an attempt to overcome this realm, studies carried out by researchers in Latin America^33-37, in the scope of public/collective health and health promotion, have drawn attention to investigations and the theory that underlies the evaluation and, especially, to its conversion into a reflexive apparatus.

This perspective of analysis seeks to value both broader institutional standards and perceptions, interests, and resistance of professionals, technicians, and other stakeholders. It starts from the premise that there are discrepancies between what was planned in the original program design and the actions developed in the implementation process. In this direction, it intends to explore the relationships between theoretical models that structure the programs, the causal associations between activities, contexts, and predicted and unforeseen results.

The evaluation guided by the theory of the program is justified by several aspects, among which two are highlighted by Pawson³⁸: i) the evaluation intends to find out whether the programs work; ii) the programs are theories (embodied theories). Consequently, evaluation is a test of theory (cf. Rein quoted in Weiss³⁹) or as reported by Pawson and Tilley^28(217): “Outcome patterns are a test of theory”. In other words, every program carries with it a theory of how change can be produced, theories underlying the ideas of public policy planners: “If we provide the resources for these people, can this change their behavior?”^20(472). These resources can be material, social, cognitive, or even emotional, as the authors point out.

Based on the understanding that social programs are part of a general theory of changes in social systems²⁸ and that social programs are a specific case of change, policymakers also try to ‘trigger’ a series of changes. When designing a program, the success of the initiatives depends on the extent to which the program’s theory was able to predict and control the interpretive spiral of ideas and social conditions. In this sense, an important aspect is to consider why social programs have the potential to bring about change.

Pawson and Tilley²⁸⁽⁵⁶⁾ laid the foundations for the so-called realistic evaluation and were, among other things, mainly concerned with explaining ‘how things change’, which requires a significant shift in thinking about critical priorities and reviewing how to organize empirical work. They take realism as a theoretical foundation and thus conceptualize it:

[…] a model of scientific explanation which avoids the traditional epistemological poles of positivism and relativism. Realism’s key feature is its stress on the mechanics of explanation, and its attempt to show that the usage of such explanatory strategies can lead to a progressive body of scientific knowledge.

They assume their admittedly controversial tone to the book ‘Realistic Evaluation’, which they also consider to be a manifesto that gives evaluation a scientific status. They draw the attention to how recent the practice of evaluation is (the 1960s) and, even so, with a very diversified arc concerning the designs they acquire and their taxonomies. In the model constructed by the authors (realistic evaluation), a fundamental reference is the work of Chen and Rossi, who coined the term ‘theory-driven evaluation’. This is not a sociological theory, but the theory per se or the ‘worldly distinctions’ that are made and that conform the prior knowledge of the diverse circumstances that involve a public policy²⁸.

A central aspect, emphasized by Chen and Rossi (1981), was the question about ‘what is a program’ and, seeking to answer it, they propose to exit the ‘black box of evaluation’ restricted then to the relationship between inputs and results programs. The emphasis on the question of an ontological nature (what is a program) instead of an epistemological one (getting to know a program) leads the authors to essential shifts in the scope of the evaluation. To answer it, they assume that there is a theory underlying each program (the programs operate according to some theories about social reality), and programs generate or intend to generate changes. In this sense, social sciences are called upon to contribute to a theoretical foundation in order to support the understanding of the processes by which changes are achieved or not. For this reason, an understanding of each context in which changes occur and on which programs they seek to intervene is essential. They also emphasize that some generative mechanisms are activated in complex interventions that intend to promote changes.

In the realistic assessment, the epistemological perspective (how to know) prioritizes causal explanation based on generative (change) principles. It assumes that regularities, in the patterns of social activities, are brought about by specific mechanisms made up of people and the resources they can mobilize in particular contexts, and which are often independent of them. The task of the research is to ‘test the theories’ of how the program results are generated by specific mechanisms and contexts, which requires inter- and intra-program comparisons to identify which Context-Mechanism-Output configurations (CMOc) are most effective.

The programs are seen as an attempt at ‘embodied knowledge’ once ‘what works, for whom and under what circumstances’²⁸ is identified. Pawson and Tilley believe that knowledge is cumulative, and we can generate knowledge through successive attempts to intervene in programs. Added to this knowledge is the one produced by research, providing policymakers with different ‘families’ of theories set by different CMOc. They believe that a teaching-learning process produces knowledge.

From this perspective, there is a growing need to carry out evaluations that start from the discussions about the evidence to be generated, and which produces knowledge³⁵. The theory-driven evaluation questions, for example, the idea that health policies can be decontextualized, verticalized, alien to the perspectives of multiple agents. Added to this view is that health policies tend to be intersectoral, multifaceted, and encompass different purposes (individual, collective, territorial) by local context. From this perspective, it is crucial to understand the nature of the challenges present in each reality. In the Latin American and Brazilian context, marked by political discontinuity, institutional fragility, low investment in social programs, poverty and violence, the pursuit of improving evaluation strategies to support decision-making, and ensure effective changes is inescapable.

Conclusions

Bhaskar’s critical realism influenced authors who devoted themselves to the studies of evaluation or, at least, intended the possibilities of correspondence between reality and scientific theory, in this case, between social reality and health theories. Scientific theories on health (and the field in which it is inserted) inform about programs and public policies. The explanatory capacity of health evaluation is compromised if they are not understood.

The dialogue with critical realism not only contributes to health analyses⁴⁰ but also other areas such as social theory and evaluation. In the case of health, it is essential to recognize the importance of evaluative research for improving the process of implementing programs and achieving substantive changes in the equity profile⁴¹. It is also essential to understand that programs in the area tend to be complex and require investment in the analysis of the different viewpoints of social agents⁸. In approaches to theory-driven and realist evaluation, other interpretative perspectives on health are raised by breaking with the positivist and idealistic stance in science⁴². When reconstructing the theories and conceptions that underlie them, confronting them with their implementation from the (visible and invisible) generative mechanisms, one faces the challenge of producing contextualized results and relevant information for decision-making.

Regardless of whether or not it is an evaluation paradigm, the theory-evaluation model embraces different perspectives, from the one initially grounded by Chen and Rossi to others, such as the realistic (or realist) evaluation, as later named by Pawson and Tilley. Through another methodological and argumentative path, Brousselle and Buregeya⁴³ advocate a new generation of evaluation consisting of theory-based driven evaluations, and in which critical realism is central. The authors believe that the new generation (which includes the approaches of logical analysis, realist evaluation, and contribution analysis) is based on the paradigm of critical realism that articulates a realistic ontology, a relativistic epistemology, and rationality of judgment.

Concepts of sociological theory were adopted indirectly by authors in the field of public health, whose audience is formed by public policy professionals and, notably, by evaluation experts, which helps to understand why social theory was understood in a particular way. When addressing theory (or the theory of the program, a metatheory), the evaluation is seen as an investigative, scientific, and reflexive enterprise. Thus, the program cannot be understood in itself and out of context, which is why the study of its implementation is fundamental. Finally, according to Pawson^44(472): “evaluation is theory-testing”.

How public health programs are conceived relates to different ontological perspectives about reality³⁶. Potvin, Gendron, and Bilodeau argue that an empirical realistic, an idealist and a critical realistic conception result in different ways of analyzing health programs and the problematic situations to which they respond. In this regard, is it possible to ask how ‘reality’ should appear to be for a particular problem to be possible? Starting from the way it should be, health programs are designed and implemented, and the ways of analyzing and evaluating them are superimposed on them, thus generating specific results according to such theoretical and methodological choices.

The evaluation of programs, in the area of public health and collective health, is inserted in intricate contexts and marked by extreme inequalities that directly impact the living and health conditions of the populations. Such conditions result from a world (or reality) that is stratified, differentiated, and structured. The area of health interventions combines multiple tensions from different theoretical perspectives, which sometimes lean towards biomedicine studies, and sometimes move away from it. The conception of ‘health’ varies over time and from place to place, expressing disputes and controversies about the production of knowledge, including science. The debate on ‘collective health’ highlights the challenges and the multiple meanings of social and human sciences and, consequently, the conditions of possibilities for knowledge achievements about the real from the biological and social dimensions.

Acknowledgments

We are grateful to the Stricto Sensu Postgraduate Program in Public Health (Sergio Arouca National School of Public Health/Oswaldo Cruz Foundation (Ensp/Fiocruz), for the selection for the Papers’ Workshop (2018); F. Vandenberghe (Institute of Philosophy and Social Sciences/Federal University of Rio de Janeiro) and Sociofilo, Co-Laboratory of Social Theory for comments on the original version.

• Financial support: non-existent
• *
  Orcid (Open Researcher and Contributor ID).

Referências

• ¹
  Barnes M, Matka E, Sullivan H. Evidence, understanding and complexity: evaluation in non-linear systems. Eval. rev. 2003; 9(3):265-284.
• ²
  Magalhães R. Implementação de programas multiestratégicos: uma proposta de matriz avaliativa. Ciênc. Saúde Colet. 2014; 19(7):2115-2123.
• ³
  Chen H. Applications of theory-driven evaluations. In: Chen H. Theory-driven evaluations. Londres: Sage; 1990. p. 77-86.
• ⁴
  Barnard A. History and theory in Anthropology. United Kingdom: Cambridge University Press; 2000.
• ⁵
  Pawson R, Tilley N. Realistic evaluation. London; Thousand Oaks: Sage; 1997.
• ⁶
  Patton MQ. Evaluation focus options: developmental evaluation and other alternatives. In: MQ Patton. Utilization Focused Evaluation. 4. ed. USA: Londres: Sage; 2008. p. 271-305.
• ⁷
  Salazar L. Reflexiones y posiciones alrededor de la evaluación de intervenciones complejas. Santiago (CL): Universidad del Valle; 2011.
• ⁸
  Magalhães R, Bodstein R. Avaliação de iniciativas e programas intersetoriais em saúde: desafios e aprendizados. Ciênc. Saúde Colet. 2009; 14(3):861-868.
• ⁹
  Eslava-Schmalbach J, Garzón-Orjuela N, Elias V, et al. Incorporación de equidad en salud en la investigación sobre implementación: revisión de los modelos conceptuales. Rev panam salud pública. 2017; 41(126):1-8.
• ¹⁰
  Hartz ZMA. Avaliação em saúde: dos modelos conceituais a prática na análise da implantacao de programas. Rio de Janeiro: Fiocruz; 1997.
• ¹¹
  Marchal B, Van Belle S, Van Olmen J, et al. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Eval rev. 2012; 18(2):192-212.
• ¹²
  Magalhães R. Avaliação da Política Nacional de Promoção da Saúde: perspectivas e desafios. Ciênc. Saúde Colet. 2016; 21(6):1767-1776.
• ¹³
  Chen H. Theory-driven evaluations. London (UK): SAGE Publications; 1990.
• ¹⁴
  Sell CE. Introdução. O que é a teoria social contemporânea? In: Sell CE, Martins CB, organizadores. Teoria sociológica contemporânea: autores e perspectivas. São Paulo: Annablume; 2017. p. 13-43.
• ¹⁵
  Hamlin CL. Realismo crítico. In: Sell CE, Martins CB, organizadores. Teoria sociológica contemporânea: autores e perspectivas. São Paulo, SP: Annablume; 2017. p. 48-74.
• ¹⁶
  Bhaskar R. Philosophy and scientific realism. In: Archer M, Bhaskar R, Collier A, et al. editores. Critical realism: essential readings. London; New York: Routledge; 1998. p. 16-47. [acesso em 2020 jan 23]. Disponível em: https://internationalcentreforcriticalrealism.files.wordpress.com/2013/04/archer-et-al_cr-essential-readings-1998-bhaskar-genl-intro.pdf
  » https://internationalcentreforcriticalrealism.files.wordpress.com/2013/04/archer-et-al_cr-essential-readings-1998-bhaskar-genl-intro.pdf
• ¹⁷
  Bhaskar R. Postscript to the second edition. In: Roy Bhaskar. A realist theory of science. London; New York: Routledge; 2008. p. 243-53.
• ¹⁸
  Sayer A. Características chave do realismo crítico na prática: um breve resumo. Estudos de Sociologia. Rev. do Programa de Pós-Graduação em Sociologia da Universidade Federal de Pernambuco. 2014; 6(2):7-32.
• ¹⁹
  Vandenberghe F. O maremoto do Realismo Crítico. Teor cult. 2014; 9(1):8-30.
• ²⁰
  Vandenberghe F. What's critical about critical realism? essays in reconstructive social theory. London; New York: Routledge; 2014.
• ²¹
  Vandenberghe F. Obituário Roy Bhaskar. Polif. 2016; 23(3):179-182.
• ²²
  Hamlin CL. Realismo crítico. In: Sell CE, Martins CB, organizadores. Teoria sociológica contemporânea: autores e perspectivas. São Paulo: Annablume; 2017. p. 48-74.
• ²³
  Sayer A. Características chave do realismo crítico na prática: um breve resumo. Estudos de Sociologia. Rev. do Programa de Pós-Graduação em Sociologia da Universidade Federal de Pernambuco. 2014; 6(2):7-32.
• ²⁴
  Crotty M. The foundations of social research: meaning and perspective in the research process. Thousand Oaks (CA): Sage Publications; 1998.
• ²⁵
  Giddens A, Sutton PW. Conceitos essenciais de sociologia. São Paulo: Unesp; 2016.
• ²⁶
  Albert K, Brundage JS, Sweet P, et al. Radicalizing epistemology in critical realism. XIX ISA World Congress of Sociology. Presentation, Toronto, Canada. 2018.
• ²⁷
  Jagosh J, Tilley N, Stern E. Realist evaluation at 25: Cumulating knowledge, advancing debates and innovating methods. Eval rev. 2016; 22(3):267-269.
• ²⁸
  Pawson R, Tilley N. Realistic evaluation. Thousand Oaks (CA): Sage Publications; 1997.
• ²⁹
  Potvin L, McQueen DV, Hall M. Health promotion evaluation practices in the Americas: values and research. New York: Springer; 2008.
• ³⁰
  Weiss CH, Weiss CH. Evaluation: methods for studying programs and policies. 2. ed. Upper Saddle River (NJ): Prentice Hall; 1998.
• ³¹
  Coryn CLS, Noakes LA, Westine CD, et al. A systematic review of theory-driven evaluation: practice from 1990 to 2009. American Journal of Evaluation. 2011; 32(2):199-226.
• ³²
  Potvin L, Chabot P. O esplendor e a miséria da epidemiologia na avaliação da promoção da saúde. In: Hartz Z, Potvin L, Bodstein R, orgnizadores. Avaliação em promoção da saúde: uma antologia comentada da parceria entre o Brasil e a cátedra de abordagens comunitárias e iniquidades em saúde (CACIS), da Universidade de Montreal de 2002 a 2012. Brasília, DF: CONASS; 2014. p. 23-37.
• ³³
  Magalhães R. Avaliação da Política Nacional de Promoção da Saúde: perspectivas e desafios. Ciênc. Saúde Colet. 2016; 21(6):1767-1776.
• ³⁴
  Magalhães R. Implementação de programas multiestratégicos: uma proposta de matriz avaliativa. Ciênc. Saúde Colet. 2014; 19(7):2115-2123.
• ³⁵
  Salazar L. Reflexiones y posiciones alrededor de la evaluación de intervenciones complejas. Santiago (CL): Universidad del Valle; 2011.
• ³⁶
  Potvin L, Gendron S, Bilodeau A. Três posturas ontológicas concernentes à natureza dos programas de saúde: implicações para a avalição. In: Bosi MLM, Mercado FJ, organizadores. Avaliação qualitativa de programas de saúde: enfoques emergentes. Petrópolis, RJ: Vozes; 2006. p. 65-86.
• ³⁷
  Souza LV, McNamee S, Santos MA. Avaliação como construção social: investigação apreciativa. Psicol Soc. 2010; 22(3):598-607.
• ³⁸
  Pawson R. Nothing as Practical as a Good Theory. Evaluation. 2003; 9:471-490.
• ³⁹
  Weiss CH. Understanding the program. In: Weiss Carol H. Evaluation: Methods for Studying Programs and Policies. 2. ed. United States: Prentice Hall; 1998, p. 46-71.
• ⁴⁰
  Roy Bhaskar, Danermark B. Metatheory, interdisciplinarity and disability research: a critical realist perspective. Scand J Disabilty Res. 2006; 8(4):278-297.
• ⁴¹
  Sridharan S, Zhao K, Nakaima A. Editors' Notes: New Dir Eval. 2017; (154):9-16.
• ⁴²
  Bodstein R. The complexity of the discussion on effectiveness and evidence in health promotion practices. IUHPE. 2007; 14(1):16-20.
• ⁴³
  Brousselle A, Buregeya JM. Theory-based evaluations: Framing the existence of a new theory in evaluation and the rise of the 5th generation. Eval. rev. 2018; 24(2):153-68.
• ⁴⁴
  Pawson R. Nothing as Practical as a Good Theory. Eval. rev. 2003; 9(4):471-490.

Publication Dates

• Publication in this collection
  13 July 2020
• Date of issue
  2019

History

• Received
  14 Mar 2019
• Accepted
  31 Oct 2019