Cross-cultural validation of ImpRes-Tool-BR: a tool for project development and implementation research

ABSTRACT

This study aims to present the results of the translation, cross-cultural validation and preliminary evaluation of a tool, originally developed in the United Kingdom, to guide Brazilian researchers in the elaboration of projects and rigorous and high-quality implementation research: ImpRes-BR. Following currently established good practices for cross-cultural validation of instruments and scales, the tool, together with its user guide, was translated and back-translated, subjected to a pilot test with 20 health professionals and evaluated by a panel of 10 specialists who assigned the values used for the calculations of the Content Validity Index at the item level (CVI-I) and scale (CVI-E). In this process, in addition to conceptual validity indices greater than 90%, an IVC-I of at least 0.90 was observed in all domains of the tool and its guide, and an IVC-E of 0.98. Having established the validity of the tool and its guide, it was applied in 14 research projects in the planning or execution phase and was recognized as a powerful instrument for self-analysis of the teams in the qualification of their projects and strengthening them in relation to the principles of the Implementation Science.

KEYWORDS
Implementation Science; Validation study; Methods

RESUMO

Este estudo tem como objetivo apresentar os resultados da tradução, validação transcultural e avaliação preliminar de uma ferramenta, originalmente desenvolvida no Reino Unido, para orientar pesquisadores brasileiros na elaboração de projetos e pesquisas de implementação rigorosos e de alta qualidade: ImpRes-BR. Seguindo boas práticas atualmente estabelecidas para validação transcultural de instrumentos e escalas, a ferramenta, juntamente com seu guia de utilização, foi traduzida e retrotraduzida, submetida a um teste piloto com 20 profissionais de saúde e avaliada por um painel de 10 especialistas que atribuíram os valores utilizados para os cálculos do Índice de Validade de Conteúdo ao nível do item (IVC-I) e escala (IVC-E). Nesse processo, além de índices de validade conceitual superiores à 90%, foi observado um IVC-I de pelo menos 0,90 em todos os domínios da ferramenta e seu guia e um IVC-E de 0,98. Estabelecida a validade da ferramenta e seu guia, a mesma foi aplicada em 14 projetos de pesquisa em fase de planejamento ou execução e foi reconhecida enquanto um instrumento potente para autoanálise das equipes na qualificação de seus projetos e fortalecimento destes em relação aos princípios da Ciência de Implementação.

PALAVRAS-CHAVES
Ciência da Implementação; Estudo de validação; Métodos

Introduction

With great expression in Australia, the United Kingdom and the United States, and occupying a prominent position with important international research funders such as the National Institutes of Health (NIH) and the Medical Research Council (MRC), implementation research corresponds to the scientific study of the various processes that guide the translation of public policies and health interventions into effective changes in the practice of individuals and/or health services¹. Greenhalgh et al.² further define implementation research as active and planned efforts to integrate an innovation within an organization. Its scope considers aspects that include the factors that affect implementation, the strategies used to interfere in the context of the intervention or implementation of a policy, as well as the results achieved at the individual and collective level, and the establishment of strategies to promote the sustainability of the proposal and its use on a large scale³, ⁴.

The growing interest of funding agencies in this type of research does not happen by chance. Fixsen et al.⁵ demonstrated that while the average success rate of conventional health interventions was 14% and took around 17 years to be incorporated into practice, interventions conducted through implementation research had a success rate of 80% over a period of 3 years. The success of this type of research can be attributed especially to the radical approach it takes in relation to the study of the context and the participation of all interested parties in the various stages of the research.

The applicability of implementation research has occurred in several areas, especially in order to address persistent care gaps. In this sense, the tools of implementation research seek to provide elements that allow identifying whether when a health policy or intervention fails - as often happens - this is because the policy or intervention was ineffective in the environment in which it was proposed (intervention failure), or if a good policy or intervention was implemented incorrectly (implementation failure)⁶.

Despite its recognized contributions to the study of health policies and practices, implementation research is still an incipient field in Brazil. In this sense, among the challenges posed for the development of this type of research in the country is the identification, compilation and adaptation of international bibliographies compatible with the Brazilian reality. It is worth mentioning that, internationally, several resources for implementation research are spread across the web or in academic products, such as compilations of measurement instruments⁷, compilations of implementation strategies⁸, guidance on the use of theories, structures and models, etc.⁹.

As a result, healthcare researchers without access to expertise in Implementation Science are tasked with identifying and assimilating design guidance and recommendations from different sources when designing their research. This is a challenging task and not always carried out successfully. The challenge of designing implementation research is further exacerbated by the fact that this type of research crosses diverse scientific fields, which makes it difficult to evaluate and synthesize relevant literature to inform design decisions¹⁰.

When faced with a similar reality, a few years ago, a group of researchers from the Centre for Implementation Science, linked to the Health Services & Population Research department at King’s College London, coordinated an initiative that resulted in the creation of the ImpRes Tool, a tool for improve the quality of research projects in Implementation Science. According to its authors, until that moment, there was a lack of guidelines and recommendations that described how to design implementation research, therefore, ImpRes was developed to fill this gap. Combined with a guide, this tool aims to enable research teams to design high-quality implementation research and, as a result, implement evidence-based interventions into service practice, thereby reducing research waste and improving health outcomes¹⁰. It can also be used in the retrospective evaluation of projects, in order to identify gaps that need to be taken into account in the process of evaluating the results of studies originating from these projects.

The ImpRes tool contains ten domains that, based on current evidence, cover the core Implementation Science principles and methods that researchers should consider when designing such research. These include (1) characteristics of implementation research, (2) implementation theories, frameworks and models, (3) determinants of implementation: contextual factors, (4) implementation strategies, (5) service and user outcomes, (6) implementation outcomes, (7) unintended consequences, (8) economic evaluation, (9) stakeholder involvement and engagement, (10) user and community involvement and engagement. Its construction took place through an extensive literature review process, followed by an international panel of multidisciplinary experts engaged in an interactive process of brainstorming and consensus building¹⁰.

Through application in 15 National Institute of Health Research (NIHR) implementation science projects led by the South London Collaboration for Leadership in Applied Health Research and Care (CLAHRC), ImpRes has proven to be a useful tool for (1) self-reflection of project teams about the implementation process, (2) identifying project areas where core elements of implementation research were missing, (3) strengthening Implementation Science in projects, and (4) auditing the rigor and quality of implementation research being conducted¹⁰. It is noteworthy that its application has occurred in a wide range of clinical areas, including diabetes, mental health, dementia, maternity and women’s health.

In addition to the tool itself, the work led by Hull et al.¹⁰ culminated in a usage guide, thus enabling researchers less familiar with the area of Implementation Science to use the tool autonomously. The guide provides the justification for the inclusion of each of the domains of the ImpRes tool, guidance on its application, and directs researchers to other specialized literature and resources. It is worth noting that to date, its development, application and preliminary evaluation study, published in 2019¹⁰, has been accessed more than 20,000 times and the tool and its guide have been downloaded from the King’s Improvement Science website more than 6,500 times.

Considering the potential contribution of the ImpRes tool to the qualification of projects and implementation research and, consequently, the promotion of this field in the country, this study aims to present the results of the translation into Portuguese, cross-cultural validation and preliminary evaluation of the ImpRes tool and its user guide.

Material and methods

This is a mixed study conducted between May 2021 and November 2022. Its implementation involved the translation and cross-cultural validation of the ImpRes tool and its usage guide, as well as its application in project design and implementation research with subsequent preliminary assessment of its structure, content and usefulness.

The transcultural translation and validation process was based on instructions from the compilation of good practices for cross-cultural validation of instruments and scales proposed by Sousa et al.¹¹. In this sense, five steps were carried out that included (1) the translation of the original instruments (English) into the target language (Portuguese); (2) comparison of the two translated versions of the instruments; (3) blind back-translation of preliminary translated versions of the instruments; (4) comparison of the two backtranslated versions of the instruments; (5) pilot testing of pre-final versions of the instruments in the target language (Portuguese) with a monolingual sample.

For a preliminary assessment of the structure, content and usefulness of the tool, it was applied by research teams in designing projects and implementation research, to subsequently share their impressions about the use of the tool through a structured questionnaire based on previously used questions in the evaluation in the original version¹⁰.

Translation of original instruments into the target language

The tool and its guide in its source language (English) were translated into the target language (Portuguese) by two independent translators whose mother tongue was Portuguese. Both translators were bilingual and bicultural, that is, fluent in the language of origin and destination of the instruments and with cultural experience in countries with both languages. Furthermore, the two translators had different backgrounds, the first being a doctor, with knowledge of healthcare terminology and the content area of instrument construction in the target language. The second translator, in turn, was an economist, familiar with colloquial expressions, slang and jargon of the target language, but with no knowledge of terminology in the health area and/or related to the construction of the instruments.

Comparison of the two translated versions of the instruments

The instructions and items of the two translated versions of the instruments were compared with their original version by a third independent, bilingual and bicultural translator, regarding ambiguities and discrepancies in words, phrases and meanings. Ambiguities and discrepancies were discussed and resolved by consensus using a committee approach that included the participation of the third translator, the two first-stage translators, and the study’s principal investigator (CAST).

Blind back-translation of preliminary translated versions of instruments

The preliminary versions of the instruments translated into the target language were back-translated into the source language by two other independent translators whose mother tongue was English. This process resulted in two back-translated versions of the instruments in their original language. As in the first stage, both translators were bilingual and bicultural with different backgrounds, one being a nurse and the other an English teacher. Both translators were completely unaware of the original version of the instruments.

Comparison of the two back-translated versions of the instruments

The instructions and items from the two back-translations were compared by a multidisci-plinary committee with the instructions and items from the instruments in their original language regarding format, wording and grammatical structure of the sentences, similarity of meaning and relevance. The committee was composed of the main researcher of the study (CAST), all four bilingual and bicultural translators involved in translating the instruments into the target language and back-translating the instruments into the source language, the translator who participated in the comparison of the two versions of the translated instruments and the researcher responsible for preparing the original instruments (LH).

Ambiguities and discrepancies regarding the cultural meaning and colloquialisms or idiomatic expressions in the words and phrases of the instructions and items between the two back-translations and between each of the two back-translations and the original instrument were discussed and resolved by consensus among the committee members in order of deriving a pre-final version of the instruments in the target language.

Pilot testing the pre-final version of the instrument in the target language with a monolingual sample

The pre-final versions of the instruments were subjected to a pilot test with 20 participants whose native language was Portuguese, in order to evaluate the clarity of their items and instructions. Participants were selected among students and professionals interested in implementation research linked to research groups that had been carrying out some type of study in this field. The groups were identified through searches in research directories and databases of projects funded by research agencies using the keywords “Implementation Research” and “Implementation Science”. The inclusion criteria were being over 18 years old, having higher education and previous health research experience.

Each participant was asked to evaluate the instructions and items presented in the instruments using a dichotomous scale (clear or unclear). Participants who classified the instructions or any item on the instruments as unclear were asked for suggestions on how to rewrite the statements in order to make the language clearer. Instructions and instrument items that were considered unclear by at least 20% of the sample were reevaluated and subjected to a second round of pilot testing with another 20 participants¹².

To determine the conceptual and content equivalence of the instruments translated into the target language, a panel of experts was also used to evaluate the items and instrument instructions for clarity. The panel of experts was composed of ten members with recognized knowledge about the content areas of the instruments’ constructs and of the target population in which the instrument will be used and whose mother tongue was Portuguese. The experts were identified through their literary production through searches in databases using the keywords “Implementation Research” and “Implementation Science”, or through their link to a research group dedicated to the study implementation research. The inclusion criteria were: having a doctorate, minimum experience of 10 years in health research and having coordinated or participated in implementation research. After identification, the experts were contacted and invited to participate in the study.

In addition to evaluating conceptual equivalence in a similar way to the other 20 participants, the panel of experts was invited to evaluate each item of the instrument in relation to its content equivalence (relevance) using the following scale: 1= not relevant; 2=not very relevant; 3= relevant, but needs small changes; 4= very relevant and succinct. Items classified as 1 (not relevant) or 2 (not very relevant) should be reviewed. Based on the experts’ responses, the Content Validity Index at the item level (CVI-I) and at the scale level (CVI-E) was calculated using the average calculation method (CVI-E/Med). A CVI-I of 0.78 and a CVI-E of 0.90 were defined as minimum acceptable indices¹³, ¹⁴.

Application of the tool in designing projects and implementation research and preliminary assessment of its structure, content and usefulness

The validated tool (called ImpRes-BR) together with its usage guide was made available for application in project design or implementation research. The projects in question constituted the final work of an introductory course on implementation research offered through the Campinas Reproductive Health Research Center in partnership with the Postgraduate Program in Public Health at the State University of Campinas, between October and November 2022. Its preparation should take into account the items and instructions provided by the tool.

Participation in the course was subject to the following inclusion criteria: being over 18 years old, having higher education and having previous experience in health research. All 18 course graduates were invited to participate in the study, of which 14 accepted. The latter were asked to answer a previously structured questionnaire with 10 statements about the structure, content and usefulness of the tool based on the questions used to evaluate its original version¹⁰.

Statistical analysis

Quantitative analyzes were performed using the Stata 11 statistical package (Stata Corp., College Station, United States). Descriptive statistics were used to calculate the median and interquartile range of numerical variables and proportion of categorical variables. Calculations were performed based on valid data, missing data were excluded from the analysis.

Ethical aspects

The study was submitted and approved by an Ethics Committee accredited at the Faculty of Medical Sciences of the State University of Campinas under registration CAAE: 00827918.8.0000.5404 opinion number 5,299,417, following Brazilian regulatory standards and guidelines for research involving human beings - Resolution CNS 466/201215, in addition to the provisions of the Declaration of Helsinki. Ethical principles were ensured by guaranteeing the right not to participate in the research from the first contact; anonymity and adoption of the Free and Informed Consent Form.

Results

Translation and back-translation of instruments

As a first step, the translation of the instruments culminated in two preliminary versions of the tool and its guide, which upon comparison showed divergence regarding some relevant aspects from a conceptual point of view. These divergences were especially related to the domains ‘characteristics of implementation research’, ‘implementation theories, structures and models’ and ‘implementation outcomes’.

In the domain ‘characteristics of implementation research’, the divergences referred to specific words or concepts related to the characterization of the type of implementation study and phases of the study. The options for translating or not words such as ‘design’, use of the concept of ‘Evidence-Based Intervention’ or ‘factual intervention’, ‘change readiness’ or ‘readiness for change’ and confusion regarding the use of the concepts of efficacy or effectiveness. In the committee approach, it was decided to translate all words that could be translated, in addition to searching for concepts already used in the field in Portuguese, such as ‘readiness for change’ and ‘Evidence-Based Practice’.

In the domain ‘theories, structures and implementation models’, the main divergence occurred regarding the translation of the word ‘framework’, for which in one of the versions it was understood that it should be maintained, while another version indicated the word ‘bases’ as a translation option. Understanding that the word ‘bases’ changed the intended meaning and that there was a choice to translate as many words as possible, a third option was considered, thus using the word ‘structure’.

In the ‘implementation outcomes’ domain, the main conceptual divergence referred to the definition of acceptability and adequacy outcomes. Expressions such as ‘agreed’ and ‘degree of agreement’ were assigned to the definition of acceptability, both of which were discarded in the committee approach. In their place, expressions such as ‘pleasant’ and ‘palatable’ were included. The adequacy outcome had initially been translated as suitability and appropriation, less common words in the Portuguese language. Furthermore, in one of the translations the expression ‘perception of sustainability’ had been attributed as one of its definitions, which was suppressed given the possibility of overlap with the outcome of sustainability implementation.

In the back-translation stage, no divergences were identified that signaled changes to the original content of the tool and its guide. In the committee approach, which included the participation of the main author of the original instruments, the divergences found were related to the use of more common words in the Portuguese language than those corresponding to the literal translation of the expressions used in the source language. The option to use the expression ‘users’ instead of ‘patients’, ‘leaders’ instead of ‘champions’, ‘workers’ instead of ‘providers’ or ‘clinicians’, ‘facilitators’ instead of ‘ drivers’, ‘intervention’ rather than ‘recommended treatment’, ‘perform’ rather than ‘deliver’ and ‘consultancy’ rather than ‘counseling’. Other divergences concerned verbal tenses or the use of the gerund/infinitive.

First round of pilot testing of the pre-final version of the instruments in the target language with a monolingual sample

The first round of pilot testing of the pre-final version of the instruments involved 20 participants, the majority of whom were female (65% n=13), with an average age of 36.7 (SD= 9.40) years. Other characteristics of the participants, including their training and familiarity with implementation research can be seen in table 1.

Table 1
Characteristics of participants included in the first and second round of pilot tests of the pre-final version of the instruments

	Round 1		Round 2
	n	%	n	%
Gender
Male	7	35%	5	25%
Female	13	65%	15	75%
Age
Up to 30 years	7	35%	6	30%
Up to 30 years	8	40%	10	50%
41 years or more	5	25%	4	20%
Education
Master - on going	7	35%	10	50%
Master - complete	1	5%	1	5%
Doctorate - on going	9	45%	7	35%
Doctorate - complete	3	15%	2	10%
Profession
Physician	4	20%	3	15%
Nurse	1	5%	1	5%
Psychologist	9	45%	11	55%
Occupational Therapist	6	30%	4	20%
Nutritionist	0	0%	1	5%
Previous completion of courses on implementation research
No	16	80%	17	85%
Yes	4	20%	3	15%
Prior participation in implementation research
No	5	25%	6	30%
Yes	15	75%	14	70%

• Source: Own elaboration.

In the first round of pilot testing of the pre-final version of the instruments, two domains were considered ‘clear’ by less than 80% of participants: the domain ‘theories, structures and implementation models’ and the domain ‘implementation outcomes’. Therefore, they were revised based on participants’ suggestions and subjected to a second round of evaluation. The performance of each domain, in addition to the cover page and introductory elements one and two of the instruments, can be seen in graph 1.

Graph 1
Performance of each component in the first round of pilot testing of the pre-final version of the instruments

Second round of pilot testing of the pre-final version of the instruments in the target language with a monolingual sample

The second round of pilot testing of the pre-final version of the instruments involved another 20 participants and sought to evaluate the new wording of the tool and its usage guide with regard to domains two and six. The characteristics of the participants who made up this stage of the evaluation can be seen in table 1.

In the second round of pilot testing of the pre-final version of the instruments, the domains ‘implementation theories, structures and models’ and ‘implementation outcomes’ were considered ‘clear’ by more than 80% of participants: 95% and 85%, respectively. Even with the minimum validity criteria met, suggestions for improvement in both domains were evaluated and incorporated into the version submitted to the experts.

Panel of experts: conceptual and content equivalence of the pre-final version of the instruments in the target language with a monolingual sample

The expert panel was made up of 10 participants, the majority of whom were female (60% n=6) and the average age was 45.5 (SD= 8.9) years. According to the inclusion criteria, all had completed a doctorate, had at least 10 years of experience in health research and had coordinated or participated in implementation research. Additionally, 40% (n = 4) of the participants had completed some training course in this field. Regarding the area of training, 30% (n= 3) of the participants were doctors, 20% (n= 2) nurses, 20% (n=2) economists, in addition to a dental surgeon (10%), a psychologist (10%) and a nutritionist (10%).

Regarding conceptual equivalence (clarity), all domains of the tool together with its usage guide were evaluated as ‘clear’ by at least 90% of participants. Concomitantly, in the assessment of content equivalence (relevance), only one evaluator assigned the concepts ‘Not relevant’ or ‘Slightly relevant’ to one of the tool’s domains, namely, the domain ‘characteristics of implementation research’, which therefore, obtained a Content Validity Index at item level (CVI-I) of 0.90. All other domains of the tool together with its usage guide presented a CVI-I of 1.00, thus resulting in an Average Content Validity Index at Scale level (CVI-E/Med) of 0.98. The results obtained from the panel of experts for each of the tool’s domains can be seen in table 2.

Table 2
Results referring to the evaluation of the conceptual and content equivalence of each domain with the panel of experts in the pilot test of the pre-final version of the instruments

	Conceptual Equivalence			Content Equivalence			CVI-I
	Clear	Unclear	Not relevant	Not very relevant	Relevant, but needs small changes	Very relevant and succinct
	n (%)	n (%)	n (%)	n (%)	n (%)	n (%)
Domain 1	9 (90%)	1 (10%)	0 (0%)	1 (10%)	4 (40%)	5 (50%)	0.90
Domain 2	9 (90%)	1 (10%)	0 (0%)	0 (0%)	5 (50%)	5 (50%)	1.00
Domain 3	10 (100%)	0 (0%)	0 (0%)	0 (0%)	3 (30%)	7 (70%)	1.00
Domain 4	10 (100%)	0 (0%)	0 (0%)	0 (0%)	3 (30%)	7 (70%)	1.00
Domain 5	9 (90%)	1 (10%)	0 (0%)	0 (0%)	3 (30%)	7 (70%)	1.00
Domain 6	10 (100%)	0 (0%)	0 (0%)	0 (0%)	4 (40%)	6 (60%)	1.00
Domain 7	9 (90%)	1 (10%)	0 (0%)	0 (0%)	2 (20%)	8 (80%)	1.00
Domain 8	10 (100%)	0 (0%)	0 (0%)	0 (0%)	1 (10%)	9 (90%)	1.00
Domain 9	9 (90%)	1 (10%)	0 (0%)	0 (0%)	4 (40%)	6 (60%)	1.00
Domain 10	9 (90%)	1 (10%)	0 (0%)	0 (0%)	4 (40%)	6 (60%)	1.00
CVI-E/Med							0.98

• Source: Own elaboration.
• CVI-I: Content Validity Index at item level; CVI-E/Med: Average Content Validity Index at Scale level.

Based on the recommendations received during the expert panel, a final review of the tool and its usage guide was carried out in order to consider the suggestions and thus give rise to the final version of the instruments. Suggestions included replacing words that had been kept in the original language, such as insights (translated to percepções) and experts (translated to especialistas), in addition to simplifying expressions, completing sentences, changing table titles and inserting links between content and graphic materials (e.g. ‘see figure 1’). After completion, both instruments were made available in an open access institutional repository¹⁶.

Application of the tool in designing projects and implementation research and preliminary assessment of its structure, content and usefulness

The application of the tool in implementation projects was carried out by 14 participants whose characteristics can be seen in table 3. The projects were mostly (n=9) in the planning phase and were distributed in the care areas of Maternal-Infant, Medical-Surgical, Community Mental Health, Oncology, Pediatrics and Women’s Health. According to the classification proposed by Curran et al.¹⁷, the projects consisted mostly of pure implementation studies (n=7) and type 2 hybrid studies (n=5).

Table 3
Characteristics of participants included in the application of the tool

	n	%
Gender
Male	2	14,3
Female	12	85,7
Age
up to 30 years	2	14,3
up to 40 years	6	42,9
41 years or more	6	42,9
Education
Specialization – complete	3	21,4
Master – on going	3	21,4
Master – complete	2	14,3
Doctorate – on going	2	14,3
Doctorate – complete	4	28,6
Profession
Physician	4	28,6
Nurse	7	50
Psychologist	2	14,3
Physiotherapist	1	7,1
Previous completion of courses on implementation research
No	3	24,4
Yes	11	78,6
Prior participation in implementation research
No	7	50
Yes	7	50
Project concentration area
Maternal-Infant	5	35,7
Medical-Surgical	1	7,1
Community Mental Health	3	21,4
Oncology	1	7,1
Pediatrics	1	7,1
Women’s Health	3	21,4
Project stage
Execution	5	35,7
Planning	9	64,3
Type of implementation research
Pure	7	50
Hybrid type 1	1	7,1
Hybrid type 2	5	35,7
Hybrid type 3	1	7,1

• Source: Own elaboration.

The results relating to the evaluation of the tool in terms of its structure, content and usefulness can be seen in table 4, where the medians found are presented, as well as their respective interquartile ranges, for each of the statements used.

Table 4
Assessment of the structure, content and usefulness of the tool

Median (IRQ)
Structure and content
The ImpRes-BR tool is easy to understand	4 (4-4)
Completing the ImpRes-BR tool is time-consuming	4 (3-4)
The ImpRes-BR tool is too long	4 (3-4)
The order of ImpRes-BR questions is logical	4 (4-5)
The ImpRes-BR tool covers the main components that must be considered when designing or conducting an Implementation Project	4 (4-5)
Utility
The ImpRes-BR tool is useful for self-analysis of the team involved in the project regarding the elements of implementation research	4,5 (4-5)
The ImpRes-BR tool is useful for identifying project areas where Implementation Science elements are missing	4 (4-5)
ImpRes-BR is a useful tool to strengthen projects in relation to Implementation Science	5(4-5)
Giving feedback to research teams on their projects based on the ImpRes-BR tool would be useful to improve the quality of implementation research	5(4-5)
ImpRes-BR is a useful tool for evaluating the quality of implementation research	4 (4-4)

• Source: Own elaboration.

Discussion

In addition to improving the understanding of its content to a greater number of potential beneficiaries, carrying out translation and cultural adaptation procedures, supported by a clear and rigorous theoretical framework, ensure equivalence in the conceptual, item, semantic, operational, measurement and functionality levels of a given tool, making it responsive to the culture and context of a given country¹⁸. In this sense, the results obtained through this study indicate that ImpRes-BR reached satisfactory levels of validity, thus constituting an appropriate tool to guide the design and qualification of projects and implementation research in the Brazilian context.

It is worth noting that, although in the translation phases there were no major conflicts between the versions prepared by independent translators, in the back-translation phases it was possible to observe that some of the expressions agreed upon in the committee approach, which generated the first translated version of the tool and its guide, did not correspond to those present in the original version of the instruments. This was because the literal translation of some of the original expressions is not equivalent to everyday language in the Brazilian context. Among the expressions related to health services, the treatment of health workers as ‘providers/provedores’ or ‘clinicians/clímcos’ and also the provision of care as ‘service delivery/entrega de serviços’ stand out. In a broader sense, the concept of ‘campeões’ also stands out, which although it can have the same meaning as ‘champions’, the original expression, is rarely assimilated in this way in the Portuguese language.

These aspects corroborate the relevance of subjecting ImpRes to translation and validation procedures for its use in Brazil. In addition to enabling reading in Portuguese, this process resulted in a greater approximation of the contents of the tool and its guide to the reality experienced in services and the health system in the country. An important point to be highlighted in this sense was the participation of the main author of these instruments in the evaluation stage of back-translated versions. With this, it was possible to clarify the meanings attributed to the original expressions, so that the best equivalent expression was selected.

Despite the efforts made to develop versions of the tool and its guide that could be easily assimilated by its potential users, in a first round of pilot testing, two domains were considered ‘clear’ by less than 80% of participants. It is noteworthy that the domains in question referred to ‘Implementation Theories, Structures and Models’ and ‘Implementation Outcomes’, domains that, together with ‘Implementation Strategies’, are properly original to the field of Implementation Science in the study of implementation processes. Therefore, it is suggested that part of the need for greater investment in clarifying these contents may be associated with the incipience of this type of study in the country.

In the expert assessment, conceptual equivalence was attested by at least 9 of the 10 experts for all domains evaluated. Similarly, content equivalence was evaluated negatively in only one of the domains by one of the experts. In other cases, the domains were evaluated positively, requiring only minor changes. In this sense, domain 2 stands out again, related to ‘Theories, Structures and Implementation Models’, which received suggestions for changes from 5 experts. However, the changes suggested at this stage mostly included changes in the nominal form, changes to the table titles and insertion of links between the contents and graphic materials. These changes contributed to reading fluency and made the material more user-friendly.

It is worth mentioning that this constitutes the first translation and cross-cultural adaptation effort of ImpRes, and it is not possible to compare the results observed with previous experiences, as is common in this type of study¹⁹. However, it is hoped that this study can become a reference for future efforts, since there is an important demand for materials for designing projects and implementation research in other Latin American countries, where Spanish is the predominant language²⁰.

On the other hand, in a similar way to what was carried out with the original version of ImpRes¹⁰, it is worth highlighting the carrying out of a preliminary assessment of the use of ImpRes-BR based on its application in 14 implementation projects, in the planning or execution phase, carried out in six different areas of health care. Through application in projects, it was possible to observe a high degree of agreement among participants regarding the ease of understanding ImpRes-BR, its logicality and coverage of the main components of an implementation project. However, just like the original version, ImpRes-BR was also considered long and time-consuming to complete.

It is noteworthy that in the original version it was observed that filling out the ImpRes was a demotivating task for some applicants whose projects had already been carried out or were in progress¹⁰. In this sense, it was anticipated that the users most likely to evaluate ImpRes-BR as long and time-consuming would be those with ongoing projects. However, a stratification of the data demonstrated that the interquartile range fluctuated upwards in these two items when evaluated only among those whose project was yet to be carried out (IQR=4-4). Among the factors that may explain this result, it is worth highlighting that while the original version of ImpRes was applied by participants recruited from a research organization (NIHR CLAHRC South London), in this study, the participants were mostly junior researchers, linked to postgraduate programs, in the phase of defining the dissertation or thesis project. This may have made filling out the ImpRes an arduous task, since many aspects of the project that had not yet been considered needed to be evaluated.

Despite being considered long and time-consuming to complete, ImpRes-BR, similarly to the original version10, had a high degree of agreement regarding its usefulness. Among the sentences with the best evaluation were those related to usefulness for team self-analysis (median=4.5), ability to strengthen projects (median=5) and use as a form of feedback to teams in order to improve the quality of research (median =5). In this way, the potential of ImpRes-BR for qualifying projects and implementation research in the Brazilian context is suggested, thus making it opportune to conduct potentially more successful implementation efforts and, consequently, with greater impact for its beneficiaries.

Some limitations should be observed when interpreting the results of this study. As the development of implementation research guided by the theoretical framework of Implementation Science is still incipient in the country, recruiting researchers to apply ImpRes-BR was a challenge. Alternatively, its offer was used as a final work in an introductory course on implementation research aimed at researchers who were planning or developing some type of study in this area. If, on the one hand, this made the preliminary assessment of ImpRes-BR viable, on the other hand, it restricted its application among a predominantly academic audience, not including, for example, policymakers or decision-makers.

In this sense, it would be important for ImpRes-BR to undergo a more comprehensive evaluation process, involving agents from different institutions as well as people who have used it in different stages of the project and across the entire implementation process. Furthermore, as pointed out by Hull et al.¹⁰ regarding the original version, it would be important to conduct an objective evaluation process that allows identifying how much the use of the tool can qualify projects and implementation research and how much this qualification translates into more practical results in real-world contexts.

Conclusions

ImpRes-BR, together with its usage guide, demonstrated satisfactory validity rates for use in the design and qualification of projects and implementation research in the Brazilian context. Although it has a long and complex structure, ImpRes-BR was recognized as a powerful tool for self-analysis by research teams in qualifying their projects and strengthening them in relation to the principles of Implementation Science.

• Financial support: Fundação de Amparo à Pesquisa do Estado de São Paulo (Fapesp), process No 2020/14309-7

Referências

• ¹
  Proctor EK, Landsverk J, Aarons G, et al. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 2009 [acesso em 2020 jul 12]; 36(1):24-34. Disponível em: https://doi.org/10.1007/s10488-008-0197-4
  » https://doi.org/10.1007/s10488-008-0197-4
• ²
  Greenhalgh T, Robert G, Macfarlane F, et al. Diffusion of innovations in service organizations: systematic review and recommendations. Milbank Q. 2004 [acesso em 2020 jul 12]; 82(4):581-629. Disponível em: https://doi.org/10.1111/j.0887-378X.2004.00325.x
  » https://doi.org/10.1111/j.0887-378X.2004.00325.x
• ³
  Peters DH, Adam T, Alonge O, et al. Republished research: Implementation research: what it is and how to do it: implementation research is a growing but not well understood field of health research that can contribute to more effective public health and clinical policies and programmes. This article provides a broad definition of implementation research and outlines key principles for how to do it. Br J Sports Med. 2014 [acesso em 2020 jul 13]; 48(8):731-6. Disponível em: http://dx.doi.org/10.1136/bmj.f6753
  » http://dx.doi.org/10.1136/bmj.f6753
• ⁴
  Powell BJ, Proctor EK, Glass JE. A Systematic Review of Strategies for Implementing Empirically Supported Mental Health Interventions. Res Soc Work Pract. 2014 [acesso em 2020 jul 13]; 24(2):192-212. Disponível em: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4002057/
  » https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4002057/
• ⁵
  Fixsen DL, Blase KA, Timbers GD, et al. In search of program implementation: 792 replications of the Teaching Family Model. In Bernfeld GA, Farrington DP, Leschied AW, organizadores. Offender rehabilitation in practice: Implementing and evaluating effective programs. Inglaterra: John Wiley & Sons; 2001. p. 149-66.
• ⁶
  Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011 [acesso em 2020 jun 12]; 38(2):65-76. Disponível em: https://doi.org/10.1007/s10488-010-0319-7
  » https://doi.org/10.1007/s10488-010-0319-7
• ⁷
  Lewis CC, Fischer S, Weiner BJ, et al. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015 [acesso em 2020 jul 12]; 10(155):1-17. Disponível em: https://doi.org/10.1186/s13012-015-0342-x
  » https://doi.org/10.1186/s13012-015-0342-x
• ⁸
  Powell BJ, Waltz TJ, Chinman MJ, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci. 2015 [acesso em 2020 maio 12]; 10(21):1-14. Disponível em: https://doi.org/10.1186/s13012-015-0209-1
  » https://doi.org/10.1186/s13012-015-0209-1
• ⁹
  Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci. 2015 [acesso em 2020 jul 12]; 10(53):1-13. Disponível em: https://doi.org/10.1186/s13012-015-0242-0
  » https://doi.org/10.1186/s13012-015-0242-0
• ¹⁰
  Hull L, Goulding L, Khadjesari Z, et al. Designing high-quality implementation research: development, application, feasibility and preliminary evaluation of the implementation science research development (ImpRes) tool and guide. Implement Sci. 2019 [acesso em 2020 jan 12]; 14(80):1-20. Disponível em: https://doi.org/10.1186/s13012-019-0897-z
  » https://doi.org/10.1186/s13012-019-0897-z
• ¹¹
  Sousa VD, Rojjanasrirat W. Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: a clear and user-friendly guideline. J Eval Clin Pract. 2011 [acesso em 2020 jan 12]; 17(2):268-74. Disponível em: https://doi.org/10.1111/j.1365-2753.2010.01434.x
  » https://doi.org/10.1111/j.1365-2753.2010.01434.x
• ¹²
  Topf M. Three estimates of interrater reliability for nominal data. Nurs Res. 1986 [acesso em 2021 jul 12]; 35(4):253-5. Disponível em: https://doi.org/10.1097/00006199-198607000-00020
  » https://doi.org/10.1097/00006199-198607000-00020
• ¹³
  Lynn MR. Determination and quantification of content validity. Nurs Res [periódico na Internet]. 1986 [acesso em 2021 jul 12]; 35(6):382-6. Disponível em: https://journals.lww.com/nursingresearchonline/citation/1986/11000/determination_and_quantification_of_content.17.aspx
  » https://journals.lww.com/nursingresearchonline/citation/1986/11000/determination_and_quantification_of_content.17.aspx
• ¹⁴
  Waltz CF, Strickland OL, Lenz E. Measurement in Nursing and Health Research, 3. ed. New York: Springer Publishing Company; 2005.
• ¹⁵
  Conselho Nacional de Saúde. Resolução nº 466, de 12 de dezembro de 2012. Aprova as diretrizes e normas regulamentadoras de pesquisas envolvendo seres humanos. Diário Oficial da União, Brasília, DF. 13 Jun 2013; Seção I:59.
• ¹⁶
  King’s College London. ImpRes-BR (Brazilian Portuguese version) Implementation Science Research Development Tool. 2022. [acesso em 2021 jul 12]. Disponível em: https://kingsimprovementscience.org/resources/
  » https://kingsimprovementscience.org/resources/
• ¹⁷
  Curran GM, Bauer M, Mittman B, et al. Effectiveness-implementation hybrid designs: combining elements of clinical effectiveness and implementation research to enhance public health impact. Med Care. 2012 [acesso em 2020 jul 12]; 50(3):217-26. Disponível em: https://doi.org/10.1097/mlr.0b013e3182408812
  » https://doi.org/10.1097/mlr.0b013e3182408812
• ¹⁸
  Alexandre NMC, Coluci MZO. Validade de conteúdo nos processos de construção e adaptação de instrumentos de medidas. Ciênc. saúde coletiva. 2011 [acesso em 2020 jan 12]; 16(7):3061-8. Disponível em: https://doi.org/10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ¹⁹
  Erazo-Chavez LJ, La-Rotta EIG, Onocko-Campos RT. Adaptação transcultural do Recovery Self-Assessment RSA-R família/Brasil: evidências de validade baseada no conteúdo. Ciênc. saúde coletiva. 2021 [acesso em 2022 jul 12]; 26(supl2):3693-3704. Disponível em: https://doi.org/10.1590/1413-81232021269.2.32692019
  » https://doi.org/10.1590/1413-81232021269.2.32692019
• ²⁰
  Marten R, Reveiz L, Aslanyan G, et al. The value and impact of embedded implementation research: insights from Latin America and the Caribbean. Rev Panam Salud Publica. 2021 [acesso em 2022 jul 12]; 45:e110. Disponível em: https://iris.paho.org/handle/10665.2/54568
  » https://iris.paho.org/handle/10665.2/54568

Edited by

Responsible editor: Maria Lucia Frizon Rizzotto

Publication Dates

• Publication in this collection
  15 July 2024
• Date of issue
  Apr-Jun 2024

History

• Received
  10 May 2023
• Accepted
  22 Mar 2024