Evaluability study of the health research induction program in Brazil to mitigate problems related to extreme poverty

ABSTRACT

The multidimensionality of poverty was the central premise of the Brazilian Without Misery Plan (2011-2014), which aimed at eradicating misery in the country. The Oswaldo Cruz Foundation coordinated a health research induction program that promoted research aimed at producing knowledge to mitigate problems related to misery. The evaluability study was guided by the theoretical and methodological instrument of the Theory of Change, which allowed us to trace the path from the initial scenario to an expected scenario, based on the induction results, represented in the logical model of the program, in which the structure, processes, and results were also described. Through document analysis and interviews with stakeholders, it was possible to identify the objectives, expected results of the program, as well as the public benefited by it and by the evaluation of the program’s results. This study guided the planning of the assessment that followed, as well as the definition of the Canadian model, to measure the return on health research investment as a methodological strategy. The limitations were recognized, as well as the issues that favored the implementation and development of the induction program. It is further hoped that this study may contribute to evaluations and/or pre-evaluations of other similarly targeted programs.

KEYWORDS: Poverty areas; Poverty; Brazil; Program evaluation

RESUMO

A multidimensionalidade da pobreza foi premissa central do Plano Brasil Sem Miséria (2011-2014), que visava a erradicar a extrema pobreza no País. A Fundação Oswaldo Cruz coordenou um programa de indução à pesquisa em saúde que promoveu investigações destinadas a produzir conhecimento para mitigar problemas relacionados à miséria. O estudo de avaliabilidade foi orientado pelo instrumento teórico-metodológico da Teoria da Mudança, que permitiu traçar o caminho do cenário inicial para um cenário esperado, com base nos resultados da indução, representados no modelo lógico do programa, no qual também foram descritos a estrutura, os processos e os resultados. A partir da análise documental e entrevistas com stakeholders das instituições envolvidas foi possível identificar os objetivos do programa, os resultados esperados, o público beneficiado pelo programa e o público beneficiado pela avaliação dos resultados do programa. Este estudo orientou o planejamento da avaliação que se seguiu, bem como a definição do modelo canadense de mensuração do retorno de investimento em pesquisa em saúde como estratégia metodológica. As limitações foram reconhecidas, bem como os pontos que favoreceram a implantação e o desenvolvimento do programa de indução. Espera-se, ainda, que este estudo possa contribuir para avaliações e/ou pré-avaliações de outros programas com escopo semelhante.

PALAVRAS-CHAVE: Áreas de pobreza; Pobreza; Brasil; Avaliação de programas e projetos de saúde

Introduction

Brazil is known as a challenging and complex country in terms of mitigating poverty. Together with the Middle East and Sub-Saharan Africa, Brazil remains among the three most unequal regions in the world, according to the 2018 World Inequality Report, which states that “income inequality has remained relatively stable, at extremely high levels”¹⁽⁶⁾.

Despite the increase in economic development in the last five decades, when Brazil entered the group of the 10 largest economies in the world in absolute Gross Domestic Product (GDP), the country never distributed this wealth in order to face historical inequality. It is notable that the Gini index, a globally accepted indicator of inequality, decreased continuously between 2001 and 2014, and, after changes in economic policies, as the political crisis in Brazil worsened, it rose again².

The best period of inequality reduction in Brazil is explained by a set of macroeconomic measures based on policies to open jobs, continually increasing the net value of the minimum wage and income transfer, through the Bolsa Família Program (PBF). During this period, the issue of tackling extreme poverty was highlighted on the Brazilian national agenda. In fact, the PBF was the largest conditional cash transfer program in the world and allowed millions of Brazilians to get out of extreme poverty. However, the diagnosis made in 2010³ found that poverty remained significant, which meant about 16 million people living below the poverty threshold, in places where the state did not even reach.

To face this scenario, the Brazilian Without Misery Plan⁴ (PBSM) was launched in 2011. Through this plan, the federal government has set the goal of eradicating extreme poverty in 4 years, from 2011 to 2014. Under the coordination of the Ministério do Desenvolvimento Social e Combate à Fome, MDS (Ministry of Social Development and Fight against Hunger), as it was named and structured at that time, the promotion of rights was at the center of the narrative of the plan’s official policy⁵. The PBSM supported its guidelines on the articulation of actions to guarantee income in addition to actions aimed at improving the living conditions of the extremely poor population, considering the multidimensionality of the situation of poverty^6,7.

Based on the integration and articulation of social policies, programs, and actions, the purpose of PBSM was to strengthen existing actions and link them to new programs, with the objective of including the poorest population in the opportunities generated by the Brazilian economic growth of that period. According to the MDS report⁸ , from the perspective of income and according to the standards of the period, the end of extreme poverty was reached still in 2013, with the inclusion of about 20 million Brazilians who lived below the poverty threshold and with the achievement of all goals foreseen in the PBSM.

In the field of health, the theme ‘neglected diseases, poverty eradication, and the Brazil Without Misery Plan’ stood out, linked to the axis of access to public services. This inclusion was the result of the articulation of the MDS and the Oswaldo Cruz Foundation (Fiocruz), with the final decision of the Presidency of the Republic. As Fiocruz is the largest non-university academic institution in the country and responsible for more than 25 doctoral courses in different areas of health, one of the main developments of this articulation was the implementation of a program to induce health research, from 2013 to 2018, through the cooperation among Fiocruz, MDS and the Coordination for the Improvement of Higher Education Personnel (Capes), an official body that coordinates and finances higher education at the doctoral level.

This induction program, called the Fiocruz-Capes-Brazil Without Misery Program, was one of Capes’ first induction programs, an unprecedented experience at Fiocruz and a desirable approximation between the academy and social policies. The main purpose of the program was to generate health knowledge to mitigate problems related to extreme poverty, through the implementation of scholarships for doctoral students and postdoctoral researchers. With the participation of 16 doctoral courses, after the development period, the products were presented in the form of theses, scientific articles, technical and educational productions.

This article presents the first results of the evaluation research of this induction program, also called an intervention, through the study of evaluability⁹ , which brought together a set of actions to plan the evaluation, considering the parties interested in its results, in order to maximize its usefulness and answer if the intervention was eligible for evaluation.

The results showed that it was possible to identify the purposes, structure, activities, and expected products that involved the intervention and whether these elements responded to the problem situation that was intended to be solved, as well as to represent them graphically in the logical model of the intervention. The political context in which the intervention was designed and implemented and the evaluation questions, which should be prioritized in the subsequent stages of the evaluation, are also the outcomes of this study.

Material and methods

This evaluability study adopted a descriptive-qualitative approach with the following steps: i) analysis of official documents to build the first version of the logical model and identification of stakeholders; ii) validation of the program’s logical model with stakeholders, through semi-structured interviews; and iii) formulation and selection of evaluation questions that supported the evaluation^10,11. There is also the theoretical support of the ‘Theory of Change’^12,13 as a process by which ‘change pathways’ can be traced, that is, what was expected from the results and effects of the program when it was conceived.

The document analysis^14,15 gathered six official documents from the induction program. Namely: i) Technical note 1/2011/IOC-Fiocruz; ii) Decree No. 7,492, of 6/2/2011, which instituted the Brazilian Without Misery Plan; iii) Document approved by the Technical Teaching Chamber of Fiocruz - September/2011; iv) Cooperation Agreement No. 123/2011 - Fiocruz/Capes; v) Cooperation Agreement No. 128/2011 - Fiocruz/MDS; and vi) Strategic planning by the Ministry of Health (2011-2015). Approved by the Human Research Ethics Committee, CAEE: 62571016.5.0000.5248, opinion No. 1.950.248.

This analysis guided the first version of the logical model of the intervention, as it was a source of data collection considered adequate for this stage since it was aimed at subsidizing the description of the program to represent it through graphics^16,17.

This initial version was presented at the end of the six interviews with stakeholders from the three institutions involved, to obtain contributions and validation. The interviews took place from October 2017 to February 2018, followed a semi-structured script, and allowed for the characterization of the political context in which the program was designed, as well as the purposes, the expected results, the public benefited by the program and by the evaluation, and the questions that should be answered in the assessment. All participants signed the Free and Informed Consent Form and agreed to the audio recording, which was transcribed and analyzed through an interpretative analysis¹⁸.

Stakeholders (SH) were selected due to the key positions they occupied during the program’s creation period, in a representative situation, with autonomy and decision-making power in the three institutions involved.

From MDS, the strategic formulator of the PBSM, three major stakeholders were invited and agreed to participate in the study (SH1,2,3); the director of scholarship programs at that time at Capes, the agency that made the intervention possible through grants (SH4); the representative of the postgraduate general coordination and the then institutional development coordinator were interviewed from Fiocruz (research institution) (SH5,6).

Results and discussion

The first result of the evaluability study was the elaboration and validation of the logical model of the intervention by the stakeholders, shown in figure 1, in which it was possible to identify and describe the following issues: i) problematic situation that generated the need to create the program; ii) the objective of the program; iii) the target audience of the program; iv) the program structure and activities, and v) the expected results.

Figure 1
Logical model of the ‘Fiocruz-Capes-Brazil Without Misery’ health research induction program

It is possible to observe in the logical model that, although the intervention involved three institutions, the responsibility for the structures and processes was centralized at Fiocruz, which corroborates the centrality of the institution identified in the number of official documents on the intervention, objects of document analysis.

The second result of this study was the characterization of the political context external and internal to Fiocruz and the description, from the perspective of the stakeholders, of the objectives of the program and the expected results, which made it possible to trace the path of the change intended with this intervention.

Characterization of the political context

The political context, in general, is not described in official program documents, and this was the case with the intervention under analysis. However, to plan the evaluation, it is desirable to know it since the success or not of an intervention can be (co-)determined by the context in which it is inserted.

In this sense, the interview participants were asked to describe the political context in which the Fiocruz-Capes-Brazil Without Misery induction program was designed.

It was possible to identify convergence and complementarity in the reports of the key actors, in the sense of a very favorable context in the country for the implementation of PBSM and its actions. The reports also revealed the managers’ attention/concern regarding the limitation of the social programs in force until 2010 and the need for action focused on the extremely poor. According to the main stakeholder of the MDS:

Without entering into a more macro-political situation, I think there is an issue that is especially important to focus on. It was a very high level of adherence [to PBSM], and I think that there is an important element [...] And it [PBSM] was a challenge, in this case, a management challenge, because it was not just saying ‘I will end misery’, it was like saying ‘I will end misery in its multiplicity, I will involve all the agendas and face the misery of not having water, the misery of having no income, the misery of having schistosomiasis’. It was a cross-cutting challenge. And it is not just a generous view of the State’s performance, but it is a very complex view of how to organize the State around a given agenda. (SH1).

In addition to the contribution of Fiocruz, the renowned academic institution, partners were sought to give legitimacy to PBSM. Thus, two aspects were considered by the MDS managers: a managerial one, the ability to present results and produce health knowledge, with a focus on neglected diseases; and another political one, the symbolic potential of the institution.

Although the external context was favorable and the proposal was converged with Fiocruz’s agenda and with its own mission in the face of social inequalities, there was resistance on behalf of the academy, as it was guided by the political agenda, as highlighted by the then coordinator of management and institutional development:

At Fiocruz, we also had an incredibly positive moment of integration, of unity, which made it possible to incorporate this agenda internally. So, we had a connection with the MDS, which unfolded in various activities [...].

The technical note was a precursor to this process, but political involvement made it much easier to carry on with the project, which nonetheless faced the cultural issue of scientific autonomy. (SH6, Fiocruz).

This finding revealed that there was no consensus at Fiocruz on cooperation and guided an aspect in the next stages of the evaluation, which was to identify the reason for the non-adherence and/or low adherence of graduate students to the induction program, as well as the concentration of research in a single graduate course: Tropical Medicine, which had 40% of the induction scholarships.

It is worth mentioning that the favorable internal context in the design of the program, even if not consensual, was not maintained during its implementation, since the foundation went through a moment of internal division, which had direct effects on the evaluated intervention. Among the effects, the restructuring of the management committee of the induction program stands out, with the departure of key actors since its conception. In the light of the theory of change and the path that was sought to be represented in the logical model, this finding was essential for the definition of management indicators in the evaluation of this intervention.

From the perspective of Capes, which was experiencing a moment of expansion, with an increase in the budget forecast to finance research, through graduate scholarships, the director of the scholarship program at the time started promoting thematic inductions, which came to meet Fiocruz’s proposal to induce health research to tackle the diseases of poverty:

Capes was going through a moment of great euphoria. Growth euphoria. We did not want that these scholarships only called the attention. There was a lot of interest in creating scholarships for induction programs, inductions that would allow us to create settings to expand graduate education at certain times, so I started doing some induction programs, involving ministries, that is, Ministry A, Ministry B would enter the program with some resources to fund and we would cover the scholarships for that program [...]. (SH4).

Although Capes was key to this intervention, be it for the resources allocated to scholarships or for the enthusiasm to promote health research focused on the public of the Brazil Without Misery Plan, the institution did not follow the process of implementation and results of the program. One of the reasons to be highlighted about this non-involvement is the historical problem of discontinuity of Brazilian public actions that reflect the characteristics and dynamics of the decision-makers of the moment, with a sequence of interruptions and ruptures that directly interfere in the achievement of program results¹⁹.

Objectives of the induction program

When asked about what was expected in 2011 to induce health research and what results were projected, the interviewees highlighted the applicability of knowledge, that is, that in some way the knowledge generated would return directly or indirectly to the poorest and to society in general.

According to representatives of Fiocruz, the greatest contribution of the induction program to PBSM was systematized knowledge, and one of the main means was the Technical Notes, result of research, to guide managers and decision makers in the formulation of public policies.

Contributing to the eradication of poverty, producing theses and technical notes for the Ministry of Health was our requirement. (SH5).

Stakeholders 1 and 4, from MDS and Capes, respectively, shared the idea that the academy had to be concerned with the country’s problems:

The academy is concerned with solving the problems of the country, of our country. (SH4).

It [PBSM] was not aimed at a goal for research results, it was aimed at bringing it to the surface, also involving the research area, in that sense, and I think it was successful. It shed light for researchers in Brazil. I think that, in addition to looking at the disease itself, even the researcher on the bench had to leave the bench and think, after all... what am I doing here behind the bench? (SH1).

The articulation of public policies to improve of public policies themselves, with a consequent improvement in life for the population, was also an expected result of the intervention.

Regarding the audacious objectives of the stakeholders, it is important to highlight that the induction program was experimental, and, in the scope of training researchers, the objectives were, to some extent, much more a ‘will’ than something possibly accomplished. This finding was particularly important when tracing the path of change since the translation of knowledge and the improvement of public policies with direct effects on the population are historical challenges to the country.

The public benefited by the induction program

The public benefited by the induction program, according to the interviewees, is remarkably diverse, including researchers, political actors, managers, and decision makers, based on the improvement and/or implementation of public policies for the general population, through the application of the knowledge generated.

The initial idea that these products ... they need to be delivered to those who manage politics. Academic production (thesis, article) is fantastic, but it is aimed at a restricted audience, so let’s try to give this production a greater utility [...]. (SH6).

The population that lived in extreme poverty, in a country that has a lack of resources in education, science, and technology [...] we need to know if the induction met the priority. (SH4).

Maybe we will not be able to even directly link to what happened with the specific results, but I honestly think that it matters a lot to the population, so I think there will be specific results that will help the population, that is, technologies, methods, processes that will help us to overcome illnesses that were reaching this population.

Second: I think that research has certainly profited from that in Brazil. The target audience is also researchers, there is one more line [...]. (SH1).

The breadth and diversity of the public benefited by the induction program, identified during the construction of the path of change, reflect the immense gaps and needs of one of the most unequal countries in the world. They also reflect the false dichotomy between academic knowledge and the idea of ‘utility’, revealed during the interviews.

The public benefited by the evaluation of the induction program

The public benefited by the program evaluation, according to the interviewees, are especially the institutions involved, as well as the public policymakers and decision-makers since the results can reorganize the formulation of programs with a similar scope.

In addition to Fiocruz, the citizen who was interviewed, who participated in the research was also benefited. According to the government, there are still people in government concerned with the country, with the issue of inequality. There is a gap, and we also need to create those loopholes. (SH5).

I would say that whoever financed ... mainly Capes ... Fiocruz was the plaintiff and the whole society ... in order to show that the investment that was made in this priority line was (or was not) successful. (SH4).

Those who develop public policies, to improve, to think about new proposals. (SH3).

The benefits of evaluating the program, from the perspective of the stakeholders, converge to the learning perspective of the authors when they were committed to evaluate and learn from this experience, to verify whether this path was promising or not. The expectation is that, based on evaluation research, which begins with this evaluability study, knowledge will be produced for new actions with similar characteristics.

Evaluation questions

This study started and was conducted under the premise that the ongoing evaluation would strongly consider the perspective of those interested in the program. Thus, the evaluation questions raised by the representatives of the three institutions were strategic for this study of evaluability and were decisive for the decision of the methodological strategy of the evaluation.

The questions are organized by institution, on purpose, to allow that different institutional perspectives are viewed.

• Oswaldo Cruz Foundation - Stakeholders 5 and 6:

  1. Was there, in the training of researchers, a differential look at poverty and/or social policies?

  2. Have the Technical Notes contributed to the formulation of public policies?

  3. What are the reasons for the difference in adherence between Fiocruz graduate programs?

  4. Was there knowledge application? Knowledge translation?

• Coordination for the Improvement of Higher Education Personnel - Stakeholder 4:

  1. Has this investment really brought possible future developments or solutions for society?

• Ministry of Social Development and Fight against Hunger - Stakeholders 1, 2 and 3:

  1. Can research contribute to optimize/improve public policies?

  2. Has the research contributed to health protocols, qualification of professionals, daily routine of health professionals?

  3. Has the program contributed to placing Fiocruz in broad social policies?

  4. Has the program had results beyond what was expected, such as the engagement of students without scholarships?

Identifying the evaluation questions of the stakeholders was an important step in planning the evaluation. This survey allowed us to understand what was important to be answered for those who designed the program and also allowed us to identify that the moment of the interview was, for some, the first moment when they thought about this evaluation, a fact that corroborates the challenge of institutionalizing the evaluation in public institutions in the country²⁰.

The last step was to select which of these questions could be answered, considering issues such as time to carry out the research, feasibility of collecting and analyzing the data, and methodological possibilities. The selected questions were those involving academic and technical products, including qualitative analysis. Questions that had a broader proposal on institutional changes and the effects or consequences of the knowledge produced did not fall within the scope of the assessment, although both merit and legitimacy are recognized.

Final considerations

The purposes of the evaluability study were achieved based on the following results: i) the logical model of the program developed and validated by the interested parties; ii) the political context described as favorable to the implementation and development of the program; and iii) the evaluation questions, which were mostly identified and recognized as possible to be answered.

This study was also crucial to choose the methodological strategy of the evaluation, which was the model developed by the Canadian Academy of Health Sciences, available in the document ‘Making an Impact: A preferred Framework and Indicators to Measure Returns on Investment in Health Research’²¹ since the great guidance for the evaluation was to answer whether the public and/or the society benefited from the public investment in the production of knowledge.

It is worth recognizing that the discontinuity of public policies in Brazil is the main limitation of this study and a posteriori evaluation, especially from the perspective of learning adopted for the evaluation that followed this study, that is, that based on the results and recommendations of this evaluation, new programs to induce health research could be promoted and improved.

Also, considering that an important part of the results of the induction program was suitable, based on the knowledge produced, to subsidize the (re)organization of current public policies and/or to guide the formulation of new policies and that this depended on political articulations with the government, which was interrupted in 2016, the linearity expected and presented in the logical model did not happen and it will hardly happen in the current government, which has a political project diametrically opposed to the one that conceived the induction program.

However, there were more than 15 million Brazilians living, in 2018, below the extreme poverty threshold, according to the Brazilian Institute of Geography and Statistics (IBGE)²². If new studies conducted by the World Bank are considered, which take into account the multiple dimensions of well-being, we reach the number of 50 million Brazilians living below the poverty threshold. These data demonstrate the relevance of the object of the study and the recognition that there is much to be done to eradicate poverty in the country, and both academia and the health field play a leading role in the care of millions of Brazilians who remain in the poverty-disease-poverty cycle.

• Financial support: non-existent

References

• ¹ Alvaredo F, Chancel L, Piketty T, et al., coordenadores. World inequality report: english version. [Paris]: World Inequality Lab; 2018.
• ² Neri MC. What was the impact of the crisis on poverty and income distribution? Rio de Janeiro: FGV; 2018.
• ³ Instituto Brasileiro de Geografia e Estatística. Universo preliminar do censo demográfico 2010. Rio de Janeiro: IBGE; [2010].
• ⁴ Brasil. Decreto nº 7.492, de 02 de junho de 2011. Institui o Plano Brasil Sem Miséria. Diário Oficial da União. 3 Jun 2011.
• ⁵ Paes-Sousa R. Plano Brasil Sem Miséria: incremento e mudança na política de proteção e promoção social no Brasil. Work. pap. 2013; Ago 113.
• ⁶ Paes-Sousa R, Vaitsman J. The zero hunger and Brazil without extreme poverty programs: a step forward in brazilian social protection policy. Ciênc. Saúde Colet. 2014; 19(11)4351-4360.
• ⁷ Campello T, Mello J. O processo de formulação e os desafios do plano Brasil sem miséria: por um país rico e com oportunidades para todos. In: Ministério do Desenvolvimento Social e Combate à Fome (Brasil). O Brasil sem miséria. Brasília, DF: Ministério do Desenvolvimento Social; 2014.
• ⁸ Brasil. Ministério do Desenvolvimento Social. Plano Brasil sem miséria: caderno de resultados (junho/2011 a dezembro/2014). Brasília, DF: Ministério do Desenvolvimento Social; 2015.
• ⁹ Leviton LC, Collins CB, Laird BL, et al. Teaching evaluation using evaluability assessment. London: Evaluation; 1998.
• ¹⁰ Thurston WE, Ramaliu A. Evaluability assessment of survivors of a torture program: lessons learned. Can. J. Program Eval. 2005; (20):1-25.
• ¹¹ Mendes MF. Avaliabilidade ou pré-avaliação de um programa. In: Samico I, Felisberto E, Figueiró AC, et al., organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010.
• ¹² Anderson AA. The community builder's approach to theory of change: a practical guide to theory development. New York: Aspen Institute Roundtable on Community Change; 2006.
• ¹³ Arnold M. Theories of change in internacional development: communications, learning or ccountability? London: Justice and Security Research Programme; 2014.
• ¹⁴ Cutolo LR, Madeira KH. O trabalho em equipe na estratégia saúde da família: uma análise documental. Arq. Catarin. Med. 2010; 39(3):79-84.
• ¹⁵ Prates JC, Prates FC. Problematizando o uso da técnica de análise documental no serviço social e no direito. Soc. Debate. 2009; 15(2):111-125.
• ¹⁶ Bezerra LCA, Cazarin G, Alves CKA. Modelagem de programas: da teoria à operacionalização. In: Samico I, Felisberto E, Figueiró AC, et al., organizadores. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010.
• ¹⁷ Cassiolato MM, Gueresi S. Como elaborar modelo lógico: roteiro para formular programas e organizar avaliação. Brasília, DF: Ipea; 2010.
• ¹⁸ Vianna IO. Metodologia do trabalho científico: um enfoque didático da produção científica. São Paulo: EPU; 2001.
• ¹⁹ Brousselle A, Champagne F, Contandriopoulos A, et al., organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Fiocruz; 2011.
• ²⁰ Carvalho AL, Shimizu HE. A institucionalização das práticas de monitoramento e avaliação: desafios e perspectivas na visão dos gestores estaduais do Sistema Único de Saúde (SUS). Interface. 2017; 21(60):23-33.
• ²¹ Canadian Academy of Health Sciences. Making an impact: a preferred framework and indicators to measure returns on investment in health research. Canadian Academy of Health Sciences: Canada; 2009.
• ²² Instituto Brasileiro de Geografia e Estatística. Síntese de indicadores sociais: uma análise das condições de vida da população brasileira. Rio de Janeiro: IBGE; 2018.

Publication Dates

• Publication in this collection
  24 May 2021
• Date of issue
  Jan-Mar 2021

History

• Received
  14 Oct 2019
• Accepted
  20 July 2020