Evaluability Assessment application on health field: an integrative review

ABSTRACT

It was aimed to analyze the scientific production on the application of the Evaluability Assessment in the health field. This is an integrative review of the literature carried out from September to October 2017, with the search of evaluative research with the application of an Evaluability Assessment, in the Virtual Health Library, Scopus and Web of Science. 21 articles were analyzed, 57.1% published in the last five years. The results show that there are important aspects of Evaluability Assessment theory in the studies, such as theoretical model, use of the stages of an Evaluability Assessment, modeling of the program, and association of sources of evidence. It is concluded that the use of Evaluability Assessment has been increasing in health, however, there is a need to improve its quality, especially regarding the objective and research subject, description of involvement of the authors and authorization of the research site, stakeholders involvement, description of data analysis procedures, strategies to ensure data validity and reliability, and contribution to academic knowledge, explaining the potential contributions of Evaluability Assessment and health evaluation.

KEYWORDS
Health evaluation; Program evaluation; Health care evaluation mechanisms; Evaluation studies

RESUMO

Objetivou-se analisar a produção científica sobre a aplicação dos Estudos de Avaliabilidade na área da saúde. Trata-se de uma revisão integrativa da literatura realizada de setembro a outubro de 2017, com busca de pesquisas avaliativas com aplicação de um Estudo de Avaliabilidade, na Biblioteca Virtual em Saúde, Scopus e Web of Science. Analisaram-se 21 artigos, 57,1% publicados nos últimos cinco anos. Os resultados demonstram que há incorporação de aspectos importantes da teoria dos Estudos de Avaliabilidade nos estudos, tais como modelo teórico norteador, uso das etapas próprias de um Estudo de Avaliabilidade, realização da modelagem do programa e associação de fontes de evidências. Conclui-se que o uso do Estudo de Avaliabilidade vem aumentando na área da saúde, entretanto, há necessidade de melhorar sua qualidade, especialmente quanto ao objetivo e questão de pesquisa, descrição do envolvimento dos autores e autorização do local de pesquisa, envolvimento dos stakeholders, descrição dos procedimentos de análise dos dados, estratégias para garantir validade e confiabilidade dos dados, e contribuição para aumento do conhecimento acadêmico, explicitando as contribuições potenciais do Estudo de Avaliabilidade e da avaliação em saúde.

PALAVRAS-CHAVE
Avaliação em saúde; Avaliação de programas e projetos de saúde; Mecanismos de avaliação da assistência à saúde; Estudos de avaliação

Introduction

The Evaluability Assessment (EA) is a type of study of the evaluative process that can be used as a pre-assessment at some stage of the development and implementation of a program, as well as throughout its life cycle¹. The EA enables stakeholder participation (interested in the assessment), enabling them to broaden their empowerment and improve understanding of the program and its progress, aspects which facilitates the occurrence of change^2,3.

The conception of EA has changed over the course of history. Initially, in the 1970s, Wholey understood it as a pre-assessment activity, that is, to be carried out prior to the assessment itself, serving as the basis for deciding whether the program is ready to be evaluated. In this context, its purpose was to serve as a strategy to improve the cost-effectiveness of the assessment so that managers could use the results of the EA to improve the program and ensure that it was ready to have its results evaluated¹.

In the 1980s, Rutman expanded the initial model of Wholey and described that EA was composed of two proposals: analyzing the characteristics of a program and evaluating the feasibility of reaching the assessment proposal. Rutman specified EA as a first step in identifying issues that impede assessment of effectiveness and the identification of strategies to improve the evaluation of the program, such as formative, implementation and outcome assessment. He also advocated the option of evaluating program components rather than the program as a whole¹.

In the year 1989, Smith proposed a model similar to the previous ones, but that could advance in its eagerness to involve the stakeholders in the EA¹. During the 1990s, there was an expansion in the use of the EA, and in the early 2000s, there was a theoretical increase in the field, so that Thurston and Potvin² developed a model for using EA as a continuous participatory assessment through a system of seven elements^1,2.

The models presented so far are of paramount importance for the elaboration of an EA. However, EA should not be considered a rigid, linear, but, rather, cyclic process, in which the ‘stages’ overlap during the process of construction of the study. In this direction, in 2015, Trevisan and Walser¹ proposed a new model of EA through four essential components: EA focus; development of the initial theory of the program; compile feedback from the theory of the program; use of EA, which should not be understood as steps or stages.

Although EA studies have been conducted since the 1970s, only in recent years its popularity has grown in the health field, and has helped this area to improve its programs, considering that they can serve a number of important purposes, such as: (a) provide the program team with quick and constructive feedback on program operations; (b) assist in the main planning functions and contribute to the feasibility of proposed actions in the field of public health, helping to develop realistic objectives and provide rapid and low-cost feedback on implementation; (c) translate research into practice, examining the feasibility, acceptability and adaptation of evidence-based practices in new contexts and populations; and (d) translate practice into research, identifying promising new approaches to achieving public health objectives⁴.

The EA is an approach that can be used at some point in the development and implementation of a program as well as throughout the program life cycle¹, that is, it can be applied at different stages of the intervention cycle – before, during or after – and aims to adjust the design of the intervention to what it is trying to achieve; raise awareness about the institutional context to support an appropriate assessment; and provide information to be used in the assessment⁵.

The desired product of an EA is a full description of the program, the key issues to be addressed by the assessment, an evaluation plan and an agreement among stakeholders about the process. Conducting an EA, therefore, also means evaluating and criticizing until the program design description is coherent and logical³. The need to modify program descriptions and activities is often identified in an EA³.

Based on the above, it is important to understand the application of the EAs in health programs, in order to highlight the importance of this stage of assessment and identify gaps to be observed in future studies. Thus, the present study aims to analyze the scientific production on the application of the EAs in the health area.

Material and methods

An integrative review study of the literature, which is characterized by a method that allows searching, critical assessment and synthesis of available evidence about the theme investigated. Its product consists of the current state of knowledge, implementation of interventions and identification of gaps that guide the development of other studies⁶.

The operational stages of the integrative review are: identification of the theme and selection of the hypothesis or research subject; establishment of criteria for inclusion and exclusion of studies/sampling or search in the literature; definition of the information to be extracted from the selected studies/categorization of the studies; evaluation of studies included in the integrative review; interpretation of results; and presentation of the review/synthesis of the knowledge⁶.

The guiding question of the present study was: How has the Evaluability Assessment been applied in the health area?

The search for the references occurred in the Virtual Health Library (VHL), which consolidates the main databases of national and international scope, selecting all the databases contained therein. In addition, the search on the Scopus and Web of Science databases was carried out. The data collection period occurred between September and October 2017. The keywords ‘evaluability assessment’ OR ‘estudo de avaliabilidade’ for the VHL, and ‘evaluability assessment’ for the Scopus and Web of bases Science were used. The use of a keyword is justified since the term ‘evaluability study’ is not a descriptor in health and was the object of this study.

Articles published in the Portuguese and English languages dealing with evaluative research in the health area were included, indicating in the method the application of an EA. Repeated studies were excluded from one or more databases; articles that addressed only theoretical revision on EA; evaluative researches that were not in the format of an article, such as theses and dissertations; and articles that did not provide a complete description of the EA, since the lack of information made the analysis of the study impossible. The authors chose not to establish year of publication of the studies, conducting an open research because it is a recent topic in health and still little performed in this field, both in Brazil and in other countries where the assessment field is more developed.

The selection of the material analyzed occurred in two stages and by two independent reviewers. In the first stage, the reading of the title and summary of the articles were carried out, as well as a consensus among the reviewers to define which articles would go to the second stage. At that moment, the reading of the complete texts was carried out, the verification if they met the inclusion criteria and the realization of a new consensus to decide which articles would be the object of the study.

To perform the data collection and analysis, a structured questionnaire was used. The first part of the questionnaire, elaborated by the authors, dealt with the characterization of the articles with the items: title, authors, journal, quality of the journal according to Journal Impact Factor (JCR), year, language and place of publication, objective and if it was about an academic study or service. The second stage used an instrument adapted by the authors of Walser and Trevisan⁷, which analyzed the application of the EA in theses and dissertations. The adaptation came about through the individual analysis by the authors, and later discussion and consensus among them. In the face of the divergences, two specialists were consulted in the area of health assessment to assist in the decision on the final version of the questionnaire.

After adaptation for analysis of the methodological quality of the EA and characterization of the application in articles, the questionnaire was composed of 26 questions grouped into five categories: (a) model of EA; (b) proposition of the EA; (c) procedures of the EA; (d) results of the EA and (e) use of the EA.

For the question ‘Are there strategies to guarantee the validity and reliability of the results?’, have been considered: a) Development of the EA in a systematic way (used a theoretical model of EA or followed adequate steps for an EA; b) use of more than one data source and if this was reliable; c) there was some kind of validation (agreement/consensus) with specialists in the area and/or stakeholders; d) presented the limitations of the study. The judgment of the quality was classified as: yes – meeting all criteria; partial – meeting one or three criteria; no – meeting no criteria.

The data collection stage and analysis of the articles was also performed by two reviewers, and it was done through an ongoing process in blocks of five articles, until totaling the number of references analyzed. At the end of each set of reviews, the authors compared their responses regarding each of the 26 questions. The divergences were debated until a consensus was reached from the items defined in the research instrument. The results were presented through a table containing absolute and relative frequency, in addition to a description of the data found. After presenting the results and interpretations, the authors elaborated recommendations on the use of the EA in health programs.

Results

After performing the systematic search in the databases, 180 references were found, which were submitted to the selection process as shown in figure 1. At the end of the search, 21 articles were analyzed^8-28.

Figure 1
Flowchart of the selection process of the articles. Florianópolis (SC). 2017

Regarding the characterization of the articles, presented in table 1, the majority (85.7%) was published in health journals. As for quality, this was low, being the majority with JCR <1 (52.4%). Most of the articles were published in the last five years (57.1%), the country with the highest number of publications was Brazil (61.9%), and the predominant language was Portuguese (61.9%). Of the 21 researches conducted, 15 (71.4%) were under academic demand. Concerning the themes, most of the EAs has concentrated on programs and projects (85.7%) focused on several areas of health care.

Table 1
Characterization of the Evaluability Assessments. Florianópolis (SC). 2017 (N=21)

Variables	n	%
Journal
Health	18	85,7
Assessment of programs	2	9,5
Education	1	4,8
Qualis/JCR of the journal
JCR >1	7	33,3
JCR < 1	11	52,4
Without JCR	3	14,3
Year
< 5 years (2013-2017)	12	57,1
> 5 years (<2013)	9	42,9
Country
Brazil	13	61,9
Canada	3	14,3
United States of America	2	9,5
Western Africa	2	9,5
Scotland	1	4,8
Language
Portuguese	13	61,9
English	8	38,1
Demand
Academic	15	71,4
Service	6	28,6
Theme
Health policies	3	14,3
Programs and projects	18	85,7

The analysis of the methodological quality of the EAs and their application in the selected articles are presented in table 2, according to the established categories.

Table 2
Result of the review of articles according to the analysis category and review subject. Florianópolis (SC). 2017 (N=21)

Category	Review subject	n	%
Model of EA	1. Which theoretical reference of EA was used?
	Thurston e Ramaliu	7	33.3
	Leviton	3	14.3
	Wholey	2	9.5
	Grouped authors (Wholey, Rutman, Smith, Thurston and Potvin)	1	4.8
	Did not define	8	38.1
	2. Has followed model steps?
	Yes	11	52,4
	Partly	2	9,5
	Does not apply	8	38,1
	3. Has followed proper steps for EA?
	Yes	19	90.5
	No	2	9.5
	4. Has the EA been addressed in the introduction or literature review?
	Yes	16	76.2
	No	5	23.8
	5. Has the program modeling been carried out and presented?
	Yes	17	81.0
	Carried out, but not presented	4	19.0
	6. What type of model was performed?
	Logical model by activities	13	61.9
	Logical model by objectives	1	4.8
	Logical model by activities and theoretical model	2	9.5
	Theoretical/Logical Model	2	9.5
	Does not inform	3	14.3
Proposition of the EA	7. What is the proposal of the EA? *
	Verify if the program is evaluable	12	57.1
	Guide the evaluation	5	23.8
	Understand the program	4	19.0
	Enable use of assessment	2	9.5
	Improve the knowledge of the stakeholders on the program	2	9.5
	Improve the program	2	9.5
	Plan a program	1	4.8
	Define objectives of the Program	1	4.8
	Update the program	1	4.8
	Verify if the program is useful	1	4.8
Proposition of the EA	8. Were the objectives and research question appropriate for an EA?
	Yes	13	61.9
	No	4	19.0
	Partly	4	19.0
	9. Was there alignment between purpose/questions, procedures, results, and recommendations?
	Yes	13	61.9
	No	3	14.3
	Partly	5	23.8
Procedures of the EA	10. What data collection procedures were used?
	Documentary analysis	19	90.5
	Interview	19	90.5
	Meetings and discussion groups with stakeholders	9	42.9
	Literature review	5	23.8
	Local visit/observation	4	19.0
	Questionnaire	1	4.8
	11. Did the author describe his/her relation with the research local? If there was a dual role (that is, a student and an employee or affiliate of the organization for which an EA is being conducted), did the author describe how it was addressed?
	Yes, he/she described the relation and how he/she was treated.	1	4.8
	Yes, he/she described the relation, but not how he/she was treated.	3	14.3
	No	17	81.0
	12. Has the research been approved by the service/institution staff investigated?
	Yes	5	23,8
	No	8	38,0
	Did not inform	7	33.3
	Does not apply	1	4.8
	13. Has EA involved stakeholders?
	Yes	6	28.6
	No, they were only a source of evidence	14	66.7
	No, there was no participation	1	4.8
	14. Was the involvement sufficiently representative throughout the EA process?
	Yes	3	50.0
	No	3	50.0
	15. Was the data analysis procedures sufficiently described and appropriate?
	Yes	7	33.3
	No	14	66.7
Results of the EA	16. How have the results been presented?
	Narrative	21	100.0
	Figure	19	90.5
	Table	7	33.3
	Board	6	28.6
Results of the EA	17. Was the focus of the results on the process or on the results of the EA?
	Results	15	71.4
	Process	6	28.6
	18. Are there strategies to ensure the validity and reliability of the results?
	Yes	3	14.3
	No	6	28.6
	Partly	12	57.1
	19. Have limitations been described?
	Yes	3	14.3
	No	18	85.7
	20. Has it presented MAJ or matrix of indicators?
	Yes	6	28.6
	No	15	71.4
	21. Has it brought an evaluative question?
	Yes	4	19.0
	No	17	81.0
Use of the EA	22. Has the article presented strategies that could facilitate the use of EE?
	Yes	18	81,0
	No	3	19,0
	23. Has the use of EA contributed to professional knowledge? (for example, solving a real problem).
	Yes	16	94.1
	No	1	5.9
	24. Has the use been consistent with the purpose of EA?
	Yes	17	81.0
	No	1	4.8
	Does not apply	3	14.3
	25. Has the work contributed to academic knowledge (academic use) about EA and/or assessment?
	Yes	4	19.0
	No	14	66.7
	Does not apply	3	14.3
	26. Has EA presented itself as an important strategy for the assessment process of the program in question?
	Yes	21	100.0
	No	-	-

• ^*
  The total n of this question is greater than 21; some articles presented more than one EA proposal.

In the ‘model of EA’ category, 13 (61.9%) studies^{10-17,21,24-27} presented a guiding theoretical frame, and 11 (52.4%)^{10,11,13,15-17,21,24-27} followed all steps of the referential. In the review, it was also analyzed whether articles, regardless of adopting a theoretical reference, followed the own stages of an EA, and most (90.5%)^{8-19,21,22,24-28} did so.

Of the 21 articles analyzed, 16 (76.2%)^{8,10-16,19-23,26-28} addressed EA in the introduction or in the literature review, all performed the modeling of the program^8-28 and 17 (81.0%)^{8,9,19,20,22,25-28,10-17} showed it. Regarding the type of model performed, 13 (61.9%)^{8-12,14,15,17,20,24-26,28} did a logical model by activities.

In relation to the second category, ‘proposition of the ‘EA’, it was verified that 12 studies^{9,11,12,14-18,20,23,25,28}, which corresponds to slightly more than half (57.1%), had as purpose to verify if the program was able to be submitted to an assessment. Of these, 4 studies^15,18,20,25 were exclusively for this purpose, and the others associated others.

In general (61.9%), the studies presented appropriate objectives and research questions for EA^{8,11,12,15,17-19,21-25,28}, with alignment between purpose, questions, procedures, results and recommendations^{8,11,12,15,17,19,21-26,28}.

The third category refers to ‘procedures of the EA’, in which documentary analysis (90.5%)^{8-10,12-18,20-28} and interview (90.5%)^{8-13,15-19,21-28} were the most commonly used for data collection, in addition to meetings and discussion groups with stakeholders, which also had significant use, that is, 9 (42.9%)^{8,13-15,17,19,21,23,24} of the studies. It is noteworthy that 1 (4.8%)¹¹ study used only one source, and the others associated two or more data sources.

Another aspect investigated in relation to procedures of the EA was the description of the involvement of the authors with the research site, in which 4 (19.0%)^8,11,24,27 described; and of these, 1 (4.8%)⁸ reported how this relationship was treated. Regarding the approval of the service to carry out the study, 5 (23.8%)^{14,19,21,24,27} had express authorization from the managing institution of the program studied. Of these, 2 studies^19,21 were carried out for the initial interest of the programs themselves.

In addition, the involvement of stakeholders was analyzed, in which 6 (28.6%)^{14,15,17,19,21,24} studies did so, and half of them^15,17,19 did so in a sufficient way, characterized by involvement of stakeholders throughout the EA. For most studies (66.7%)^{8-13,16,18,22,23,25-28}, the stakeholders were only a source of evidence. Regarding the analysis of the data, the majority (66.7%)^{9,12-14,17-21,23,24,26-28} did not make the description sufficiently and adequately, that is, with methodological details that allowed the reader to understand in detail the form of analysis; and in general there was a succinct description.

Regarding the ‘results of the EA’, all studies presented in a narrative form, and most (90.5%)^{8-20,22,23,26-28}, also by means of figures, especially for presentation of the modeling, being the focus of the studies predominantly (71.4%)^{8,9,11-14,16-20,22,24-26} on the results of the EA.

Regarding the strategies to guarantee validity and reliability of the results, most of the studies (57.1%)^{9-13,19,22-27} presented partial results, that is, one or three of the established criteria were met. Regarding the limitations of the study, only 3 (14.3%)^8,11,27 described them.

In the planning of the evaluation process, 6 (28.6%)^{12,14,15,17,18,28} of the studies presented Matrix of Analysis and Judgment (MAJ) or indicators; and 4 (19.0%)^12,13,17,28, an evaluative question.

Regarding ‘use of EA’, most of the studies presented strategies to facilitate its use (81.0%)^{8-19,21-24,26,28}, especially in the form of recommendations; and all studies reported capacity to contribute to the knowledge of professionals about the programs investigated, and the majority (81.0%)^{8-19,21-24,28} brought the consistent use with the purpose of the EA, that is, when the study presented use in line with the proposition and results of the EA. However, regarding the academic contribution of new knowledge on evaluation and/or EA, only 4 (19.0%)^8,11,15,22 contributed to a greater basis in the literature.

Based on the findings of the present study, the authors elaborated recommendations according to the main weaknesses found in the studies analyzed, in order to improve the quality of future researches that have the EA as method, which are presented in chart 1.

Chart 1
Recommendations of the review study. Florianópolis (SC). 2017

Category	Recommendations
Model of EA	1. Explain the EA model and concept that guides the study;
	2.Carry out and present thoroughly the modeling of the program
Proposition of the EA	3. Establish appropriate objectives and research subject for EA;
	4. Align purpose/questions, procedures, results and recommendations;
Procedures of the EA	5. Use two or more procedures of data collection;
	6. Describe the relationship of the researcher with the research location;
	7. Describe agreement of the team of the program with the performance of the EA;
	8. Involve stakeholders in all stages of EA;
	9. Describe the procedures of analysis sufficiently;
Results of the EA	10. Establish and describe data validity and reliability criteria;
	11. Present the limitations of the study;
	12. When the program is assessable, present an evaluative question;
Use of the EA	13. Contribute to academic knowledge in the area of the study object through recommendations and contributions on EA and/or health assessment;
	14. Contribute to the knowledge increase about the object by the professionals, which allows collaborating to solve problems in the implementation of programs.

Discussion

A literature review of published documents on EA from 1979 to 2011 has identified an expressive increase in materials over the years²⁹, a trend that has been applied to the health area, expressed by the increase in EA in the last years, especially at the national level, revealing that EA has been gaining ground in the field of health assessment.

The EA became popular in the United States in the 1980s, after being promoted by Joseph Wholey³⁰, a US government official. Subsequently, this method fell into disuse for a decade, given that, over the past 15 years, international development agencies have begun to use EA²⁹.

The first reference found in the present study was published in 2003, concentrating a greater number of publications in the last five years, which corroborates Leviton et al.⁴.

In the present study, with regard to the quality of journals, it was verified that only 7 articles were published in a JCR journal greater than one, which may indicate little acceptance of this type of study because it is a recent and still little used method in the area, or the need to improve the studies for their dissemination in the scientific environment.

It was observed, as well, that the studies are concentrated in Brazil and through academic demand, assuming that the services have not yet consolidated the systematic culture of the assessment or, if evaluated, they do not divulge their work in the scientific environment³¹.

Regarding the analysis of the review questions, it was possible to identify that the studies have strengths and weaknesses, which need to be discussed in order to improve the quality of the EA, thus increasing its use and dissemination, which also contributes to qualify the knowledge about the theme.

In the first category of analysis, ‘model of EA’, most used a theoretical frame and followed the appropriate steps for an EA, which is an essential element of any EA. Following a rigorous model in the execution of the steps shows an understanding of the role of the theory of the program in EA and supports the accuracy of the evaluation findings⁷.

Despite being a minority, five studies did not bring the EA into the literature introduction or review, not making clear which concept of EA is being used by the authors, which makes it difficult to analyze the coherence between study purpose, results and recommendations.

All the studies performed the modeling of the program, which constitutes an essential step for an EA. However, four studies did not present the model, which weakens the analysis and discussion of the findings, thus conferring less validity and reliability to these studies. The logical model is useful for defining inputs, clarifying program objectives, defining planned activities, ensuring that objectives are realistic, exploring possible evaluation approaches and detailing implicit or explicit logic of production for the desired result³². In addition, the logical model can also direct improvements in theory and implementation of programs¹⁵.

In the ‘purpose of the EA’ category, it was observed as potentiality the fact that the studies, in general, are surpassing the traditional concept of pre-assessment. Although more than half of the studies have the purpose of verifying that the program is evaluable, only four have been considered as an exclusive purpose, and other issues are incorporated, such as guiding/making an assessment, understanding, planning, improving and updating a program, instead of verifying if it is useful.

The EA is used in public health as a pre-assessment to determine if a program is ready for a full assessment, to assist planners of the programs in identifying the necessary improvements of the program and to ensure that an evaluation produces useful information^4,32. It is part of the evaluation process¹⁵ and can be used at any stage of a program, either in the planning, implementation, or over the life cycle of the program¹, in order to obtain in-depth knowledge about the research object and prior appreciation of the assessment possibilities^20,33.

The purpose of the EA needs to be aligned with procedures, results and recommendations, as well as having adequate objectives and research questions for an EA. On this aspect, the studies presented fragility, with satisfactory answer for only 61.9% of these. In general, they weakened when presenting the objectives, which did not focus on the purpose of the EA, with a lack of justification for the need to perform an EA in a given program. In addition, the research subject was not presented clearly.

The findings of the present study are similar to the review study of theses and dissertations by Walser and Trevisan⁷, which recommended that the objectives of the study and issues should be consistent with the intended purposes and uses of the EA. The alignment between purpose, procedures, results and recommendations demonstrates good planning of the EA, its methodological quality and favors its use by both stakeholders (for the program in question) and by experts in assessment (broadening the field of knowledge on the subject).

After setting objectives and research subject, it is necessary to choose the appropriate procedures to achieve the proposed. It was evidenced that all the studies used two or more procedures of data collection, a positive aspect to increase the validity and reliability of the data.

Review studies have found similar data regarding the types of procedures, with interview and review of the most commonly used documents^7,34, which indicates a consensus of the researchers regarding the execution of the EA, strengthening the establishment of this method in the evaluation process.

Another fragile aspect of most studies was the lack of description of the involvement of the authors with the research site and the approval/agreement process of the team of the program. Because it is an evaluation process in the universe of a program that involves several actors, it is fundamental to take the methodological care to clarify the role and involvement of stakeholders and researchers in order to identify possible conflicts of interest and biases in the execution of the study, making feasible clarify the limitations and ways of minimizing them, which gives higher quality to the study.

The present study has evidenced that the EAs have little involvement of the stakeholders, using them only as a source of evidence. One possible explanation for this phenomenon is that the studies, mostly, were by academic demand, in which there is often no involvement and interest on the part of those involved with the program.

Stakeholders are people with an interest in a program, whose decisions can affect the future of the program in very important ways, and that must be involved in all components of the EA^7,35. Their involvement is fundamental to the success of the assessment process, as they contribute to the execution of the EA in order to improve the use of the resources of the program, to comply with the evaluation criteria, to guarantee greater data reliability, increased use of assessment and program improvement potential^1,7,24,36.

As for the description of the procedures of data analysis, the studies also presented weaknesses, with absence or description carried out succinctly, not allowing the understanding of this item of the method. It should be emphasized that an EA is, by definition, exploratory, and its methodology is similar to that of any other research, and must follow methodological rigor²².

In the ‘results of the EA’ category, these were presented through narrative and figures, focusing on the results, which corroborates studies already done on the subject^7,34.

In this category, a fact that deserves attention is that the articles do not present satisfactory strategies to guarantee the validity and reliability of the data. A quality study requires sufficiently valid and reliable information, collection, systematic review and verification of information, explicit evaluation reasoning, and transparency and disclosure in relation to study limitations⁷. In addition, the accomplishment of agreement/consensus among stakeholders is a common strategy used in EA to provide a feedback of the program, making feasible to compare reality with the model of the elaborated program, and to perform necessary adjustments^1,2,4, which leads to the formulation of a more reliable model for the reality investigated.

Among the review questions of the present study, the authors have opted to investigate whether the articles presented analysis matrix and/or evaluative question, which was not very common. Considering that all the studies verified that the programs were apt to be evaluated after the completion of the EA, it is considered that, minimally, the presentation of the evaluative question would be an important aspect to guide the continuity of the evaluative process, as well as to base the use of the assessment.

The use of assessment is essential for a quality assessment and reflects the quality of the EA, considering its purpose^1,4. Thus, the ‘use of EA’ category showed that the studies, for the most part, presented coherent strategies to make use feasible. Among the articles analyzed, it is worth mentioning the good quality of the study by Sanou et al.¹⁵, who presented a specific topic about the use of the results, which were discussed with stakeholders. It is also valid to emphasize the study by Honeycutt et al.²³, which presented several strategies for disseminating the results, which is an important resource to promote the use of EA.

All the studies have contributed to the knowledge of professionals and revealed that the EA is useful for the health program, which is extremely important, since the realization of an EA, primarily, should contribute so that the professionals can make use of it and positively impact on the assessed program. However, especially since most studies are by academic demand, they could have a greater contribution to increasing academic knowledge about assessment and/or EA in order to strengthen the subject and improve the method.

A quality EA assists in the identification of program areas that need improvement and ensures that a complete assessment produces useful information²⁴, thus, every EA should be applicable to the practice of health professionals. In many cases, EA reveals results and ideas that would not have been thought otherwise^8,37.

It is considered that a study of the EA demonstrates contributing to academic knowledge in the subject area of study when it makes recommendations and potential contributions regarding the academic use of EA and/or health assessment⁷.

Like any type of study, the integrative review presents advantages and limitations of the method itself, and care should be taken with regard to conclusions related to the findings. However, the authors highlight the potentiality of the method used, since it has validity among experts in the review area. In addition, the study had as limitation the fact that some items of the questionnaire were difficult to analyze because of their subjectivity, but such limitation was minimized through discussion and consensus among the reviewers, and consultation with experts in the area in the face of divergences.

Conclusions

The study evidenced that the applicability of EA in health programs is recent, with growth in the last five years, presenting itself as an important strategy to improve the assessment process.

Many fragilities have been identified, with shortcomings with respect to: the objective and research question appropriate for EA; consistency between purpose, procedures, results and recommendations; description of involvements of the authors with local of the research; information about authorization of the local of the research; involvement of stakeholders in all phases of the EA; description of data analysis procedures; strategies to ensure data validity and reliability; an evaluative question, and little contribution to academic knowledge, considering the lack of approach to the potential contributions of the EA and health assessment.

As potentialities, the use of a theoretical reference of EA, adoption of an EA concept, diversified EA purposes were verified, overcoming single concept of pre-evaluation, source of evidence coherent with EA and use of more than one source, coherent strategies to enable the use of EA, and contribution to professional knowledge.

Increasing the quality of EA is of paramount importance to improve the theoretical basis on the subject and, consequently, on its use. In addition, it was verified that few studies were published in high-impact journals, thus, the improvement of studies also facilitates the expansion of the field of scientific publication, which contributes to increase access to knowledge on EA by researchers and professionals, potentializing the use of EA in health programs.

• Financial support: non-existent
• *
  Orcid (Open Researcher and Contributor ID).

Referências

• ¹
  Trevisan MS, Walser TM. Evaluability Assessment: improving evaluation quality and use. Publications, SAGE; 2015.
• ²
  Thurston WE, Potvin L. Evaluability Assessment: A Tool for Incorporating Evaluation in Social Change Programmes. Evaluation. 2003; 9(4):453-69.
• ³
  Thurston WE, Ramaliu A. Evaluability Assessment of a Survivors of Torture Program: Lessons Learned. Can J Progr Eval. 2005; 20(2):1-25.
• ⁴
  Leviton LC, Khan LK, Rog D, et al. Evaluability Assessment to Improve Public Health Policies, Programs, and Practices. Annu Rev Public Health. 2010; 31(1):213-33.
• ⁵
  Peersman G, Guijt I, T Pasanen. Evaluability Assessment impact evaluation: guidance, checklists and decision support. Australian Government: Departament of Foreign Affairs and Trade; 2015.
• ⁶
  Mendes KDS, Silveira RCCP, Galvão CM. Revisão integrativa: método de pesquisa para incorporação de evidências na saúde e na enfermagem. Texto Context enferm. 2008; 17(4):758-64.
• ⁷
  Walser TM, Trevisan MS. Evaluability Assessment Thesis and Dissertation Studies in Graduate Professional Degree Programs: Review and1. Walser TM, Trevisan MS. Evaluability Assessment Thesis and Dissertation Studies in Graduate Professional Degree Programs: Review and Recommendat. Am J Eval. 2016; 37(1):118-38.
• ⁸
  Belford M, Robertson T, Jepson R. Using evaluability assessment to assess local community development health programmes: a Scottish case-study. BMC Med Res Methodol. 2017; 17(1):1-12.
• ⁹
  Chaves SCL, Silva GAP, Rossi TRA. Avaliabilidade do Projeto de Mobilização Social para Prevenção e Controle da Dengue no Estado da Bahia. Saúde debate. 2017; 41(esp):138-55.
• ¹⁰
  Nascimento LV, Machado WD, Gomes DF. Estudo de avaliabilidade da Política Nacional de Atenção Integral à Saúde do Homem no município de Sobral, Ceará. Rev Baiana Saúde Pública. 2014; 38(1):95-114.
• ¹¹
  D'Ostie-racine L, Dagenais C, Ridde V. An evaluability assessment of a West Africa based Non-Governmental Organization's (NGO) progressive evaluation strategy. Eval Program Plann. 2013; 36(1):71-9.
• ¹²
  Rocha BNGA, Uchoa SAC. Avaliação da atenção humanizada ao abortamento: um estudo de avaliabilidade. Physis. 2013; 23(1):109-27.
• ¹³
  Oliveira LGD, Natal S, Camacho LAB. O programa de controle da tuberculose em unidades prisionais de dois estados brasileiros Tuberculosis control program in two Brazilian states ' prisons. Cad Saude Colet. 2012; 20(2):250-7.
• ¹⁴
  Bezerra LCDA, Alves CK A, Reis YAC, et al. Identificação e caracterização dos elementos constituintes de uma intervenção: pré-avaliação da política ParticipaSUS. Ciênc. Saúde Colet. 2012; 17(4):883-900.
• ¹⁵
  Sanou A, Kouyaté B, Bibeau G, Nguyen VK. Evaluability Assessment of an immunization improvement strategy in rural Burkina Faso: Intervention theory versus reality, information need and evaluations. Eval Program Plann. 2011; 34(3):303-15.
• ¹⁶
  Natal S, Samico I, Gonçalves L, et al. Estudo de avaliabilidade da rede de formação de Recursos Humanos da Secretaria de Vigilância em Saúde do Ministério da Saúde. Cad. Saúde Colet. 2010; 18(4):560-71.
• ¹⁷
  Medeiros PFP, Bezerra LCA, Santos NTV, et al. Um estudo sobre a avaliabilidade do Programa + Vida: política de redução de danos em álcool, fumo e outras drogas do município de Recife, Brasil. Rev bras saúde mater infant. 2010; 10(supl1):s209-17.
• ¹⁸
  Lima LRF, Silva LMV. Ampliação do acesso à atenção oftalmológica?: um estudo sobre a avaliabilidade da campanha "De Olho na Visão", Goiás, 2004. Ciênc. Saúde Colet. 2008; 13(supl2):2059-64.
• ¹⁹
  Dwyer JJM, Hansen B, Barrera M, et al. Maximizing children's physical activity: An evaluability assessment to plan a community-based, multi-strategy approach in an ethno-racially and socio-economically diverse city. Health Promot Int. 2003; 18(3):199-208.
• ²⁰
  Correia PCI, Goulart PM, Furtado JP. A avaliabilidade dos Núcleos de Apoio à Saúde da Família (Nasf). Saúde debate. 2017; 41(esp):345-59.
• ²¹
  Thurston WE, Hatfield J. Evaluability assessment: A Catalyst for Program Change and Improvement. Eval Health Prof. 2003; 206-21.
• ²²
  Dagenais C, Plouffe L, Gagné C, et al. Dupont D. Improving the health and safety of 911 emergency call centre agents: An evaluability assessment of a knowledge transfer strategy. Int J Occup Saf Ergnomics. 2016; 1-20.
• ²³
  Honeycutt S, Hermstad A, Carvalho ML, et al. Practice to Evidence: Using Evaluability Assessment to Generate Practice-Based Evidence in Rural South Georgia. Heal Educ Behav. 2017; 44(3):454-62.
• ²⁴
  Abildso CG, Shawley S, Owens S, et al. An Evaluability Assessment of the West Virginia Physical Activity Plan, 2015: Lessons Learned for Other State Physical Activity Plans. Prev Chronic Dis. 2016; 13:160307.
• ²⁵
  Oliveira CM, Cruz MM, Kanso S, et al. Avaliabilidade do Programa de Valorização do Profissional da Atenção Básica (PROVAB): desafios para gestão do trabalho. Ciênc. Saúde Colet. 2015; 20(10):2999-3010.
• ²⁶
  Padilha MA, Oliveira CM, Figueiró AC. Estudo de avaliabilidade do Programa Academia Carioca da Saúde: desafios para a promoção da saúde. Saúde debate. 2015; 39(105):375-86.
• ²⁷
  Pereira NC, Luiza VL, Cruz MM. Serviços farmacêuticos na atenção primária no município do Rio de Janeiro: um estudo de avaliabilidade. Saúde debate. 2015; 39(105):451-68.
• ²⁸
  Coelho AA, Martiniano CS, Brito EWG, et al. Tuberculosis care: an evaluability study. Rev Lat Am Enfermagem. 2014; 22(5):792-800.
• ²⁹
  Davies R, Payne L. Evaluability Assessments: Reflections on a review of the literature. Evaluation. 2015; 21(2):216-31.
• ³⁰
  Wholey JS. Evaluation: promise and performance. Washington (DC): The Urban Institute; 1979.
• ³¹
  Hartz ZMA, Silva LMV. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Rio de Janeiro: Fiocruz; 2005.
• ³²
  Wholey JS, Hatry HP, Newcomer KE. Handbook of practical program evaluation. 3. ed. San Francisco: Jossey-Bass; 2010.
• ³³
  Guimarães EAA, Hartz ZMA, Luz ZMP. Desenvolvimento de modelos para avaliação das redes de conhecimento: um estudo de avaliabilidade no Centro de Pesquisa René Rachou (Fiocruz, Minas), Brasil. An Inst Hig Med Trop (Lisb) [internet]. 2016; 15(supl2):17-26.
• ³⁴
  Trevisan MS. Evaluability Assessment From 1986 to 2006. Am J Eval. 2007; 28(3):290-303.
• ³⁵
  Smith MF. Evaluability assessment: a practical approach. Boston, MA: Kluwer Academics; 1989.
• ³⁶
  Salvatierra da Silva D, Jacobson SK, Monroe MC, et al. Using evaluability assessment to improve program evaluation for the Blue-throated Macaw Environmental Education Project in Bolivia. Appl Environ Educ Commun [internet]. 2016; 15(4):312-24.
• ³⁷
  Hester LL, Pollution A, Branch RH, et al. Roles of the State Asthma Program in Implementing Multicomponent, School-Based Asthma Interventions. J Sch Heal. 2013; 83(12):833-41.

Publication Dates

• Publication in this collection
  06 May 2019
• Date of issue
  Jan-Mar 2019

History

• Received
  28 May 2018
• Accepted
  21 Jan 2019