Systematic review studies: a guide for careful synthesis of the scientific evidence

Abstracts

INTRODUÇÃO: Agregar evidências de pesquisa para guiar a prática clínica é uma das principais razões para se desenvolverem estudos que sintetizam a literatura, mas não é a única. As revisões sistemáticas são desenhadas para ser metódicas, explícitas e passíveis de reprodução. Esse tipo de estudo serve para nortear o desenvolvimento de projetos, indicando novos rumos para futuras investigações e identificando quais métodos de pesquisa foram utilizados em uma área. MÉTODOS: Uma revisão sistemática requer uma pergunta clara, a definição de uma estratégia de busca, o estabelecimento de critérios de inclusão e exclusão dos artigos e, acima de tudo, uma análise criteriosa da qualidade da literatura selecionada. O processo de desenvolvimento desse tipo de estudo de revisão inclui caracterizar cada estudo selecionado, avaliar a qualidade deles, identificar conceitos importantes, comparar as análises estatísticas apresentadas e concluir sobre o que a literatura informa em relação a determinada intervenção, apontando ainda problemas/questões que necessitam de novos estudos. Um trabalho de revisão sistemática segue a estrutura de um artigo original. CONCLUSÃO: Boas revisões sistemáticas são recursos importantes ante o crescimento acelerado da informação científica. Esses estudos ajudam a sintetizar a evidência disponível na literatura sobre uma intervenção, podendo auxiliar profissionais clínicos e pesquisadores no seu cotidiano de trabalho.

revisão sistemática; ECA; síntese da literatura

METHODOLOGICAL ARTICLES

Systematic review studies: a guide for careful synthesis of scientific evidence

Sampaio RF; Mancini MC

Physical Therapy and Occupational Therapy Departments, Physical Education, Physical Therapy and Occupational Therapy School, Federal University of Minas Gerais, Belo Horizonte, MG - Brazil

^{Correspondence to}

ABSTRACT

Introduction: Aggregation of research evidence to guide clinical practice is one of the main reasons for conducting studies that provide a synthesis of the literature, but it is not the only reason. Systematic reviews are designed to be methodical, explicit and replicable. Such studies may help to guide project development, by indicating new directions for further investigations and identifying which research methods have been used within a given area. Methods: Systematic reviews require straightforward questions, search strategy definition, establishment of study inclusion and exclusion criteria and careful analysis of the selected literature. The development process for reviews of this type includes characterization of each selected study, evaluation of their quality, identification of important concepts, comparison of statistical analyses used and conclusions regarding what the literature tells us about a specific intervention. Such reviews also suggest problems/questions that need further investigation. Systematic review studies follow the structure of original articles. Conclusion: Good systematic reviews are important resources, in the light of the accelerated growth of scientific information. These studies help in producing syntheses of the evidence available in the literature on specific interventions, and may help clinicians and researchers in their work process.

Key words: systematic review, RCT, synthesis of the literature.

INTRODUCTION

The demand for maximal quality in health care, allied to the need of rational use of both public and private resources, have contributed to increase the pressure over the professionals of the area, in the sense of assuring the implementation of a scientific evidence-based practice. The expression "evidence-based medicine" came up in the 1980s to describe the learning based in problems, used by the MacMaster University Medicine School¹. Evidence-based practice (EBP) and EB health care comprehend the same concepts and principles of the evidence-based medicine, being employed by different professionals and in many health contexts¹.

EBP has been defined as the conscious, explicit and careful use of the best and most actual research evidence in the process of clinical decisions making about the patient care². The EBP has been sustained in a tripod that takes in account the synthesis of the best external or research evidence, the experience of the professional, and the values and preferences of the patient, that is, centered in the patient and his family. Research developed in a careful way provides assurances to aid clinical decision making, but they never substitute reasoning and the professional's experience to decide which intervention is effective or not for a specific patient care³.

The EBP process is similar in all health professions, but there are peculiarities related to the different practice domains, and with the theoretical models adapted by a certain group of professionals as in the profession that compose de rehabilitation area. Clinical questions are identified based in the need for information in the decision-making process, respecting patients or groups of patients under treatment, as well as the context that this is being offered. The search for answers in the scientific literature is made for the search of the best evidence available, not all studies are all developed; in this way, it becomes necessary a careful evaluation of its validity and of the clinical applicability of the results.

The EBP involves the overcome of certain challenges: how to keep oneself updated in front of the growing availability of information in the health area? Which are the best sources of information? How to join the selected evidences with the clinical experience facing the needs presented by patients? The research evidence analysis requires from the professionals new knowledge and abilities to capacitate them to have autonomy in the critical evaluation of the scientific information that are to be used to diminish uncertainties of the clinical decision-making.

It is consensual that randomized clinical trials (RCT) are the most adequate studies to provide evidences about the effects of an intervention. However, the results of only one of these studies are not enough to enlighten certain research question, or a clinical inquiry. The conclusions are stronger when different studies investigate the effects of an intervention and provide data that support the same conclusions. In this sense, systematic reviews and methanalysis are the most adequate and up-to-date methods to summarize and synthesize evidences about the effectiveness and intervention effects⁵. Systematic methods are used to avoid bias and to make possible a more objective analysis of the results, facilitating to make a conclusive synthesis about certain intervention. The current study has as objective to define and to describe the stages involved in the development of a systematic literature review.

What is a systematic review?

Systematic review, as well as other kinds of review studies, is a research that uses as data source the literature about a determined issue. This kind of investigation provides a sum of the evidences related to a strategy of specific intervention, by an application of explicit and systematized method of research, critical appreciation and synthesis of the selected information. The systematic reviews are particularly useful to integrate the information of a group of studies carried out separately about certain therapeutics/intervention, which may present conflicting or coincident results, as well as to identify themes that need evidences, helping the orientation of future investigations⁵.

By making available a resume of all studies about certain intervention, systematic reviews allow us to incorporate a greater spectrum of relevant results, instead of limiting our conclusions to the reading of only a few articles. Other advantages include the possibility of evaluating the consistence and generalization of the results between populations or clinical groups, as well as treatment protocols' specificities and variations⁶. It is important to highlight that this is a kind of retrospective study, and secondary, that is, the review is usually drawn and conducted after the publication of many experimental studies about a theme. In this sense, a systematic review depends on the quality of the primary source.

There is inconsistence in the terminology used to describe systematic reviews, considering that some include a statistic synthesis of the studies' result, and some don't. Some authors point out that systematic reviews with methanalyses are different from the other reviews by its methanalitical component⁶. Methanalyses is the analyses of the analyses, that is, it is a study of literature review in which the results of several independent studies are combined and synthesized by way of statistical procedures, in a way to produce a single estimative or index that characterizes the effect of a certain intervention⁷. In methanalyses studies, by combining samples from several studies, the total sample is increased, empowering the statistical analyses, as well as the precision of the estimative of the treatment effect⁶.

Presently, there are still few systematic reviews that apply methanalyses in physical therapy and occupational therapy, as well as in other health related areas. According to Magee, the reasons that difficult the implementation of this type of study include the utilization of different research protocols (i.e. effect measurement, selection and intervention) and variations in the methodological quality (i.e. which may be classified as strong, moderate or weak)⁸. The present article will have as focus studies of systematic review without methanalyses.

The position occupied by the systematic review in the hierarchy of the evidence illustrates its importance to clinic and research. In this hierarchy, when we search for evidence about the effectiveness of the intervention or treatment, systematic review studies with or without methanalyses, that include RCT and experimental studies, tend generally to present a stronger evidence, that is, they are more adequate studies to answer questions about the efficacy of an intervention^9,10. This hierarchy guides the classification criteria of levels of evidence for different kinds of study (prognostic, diagnostic, therapeutic, prevalence and economical analyses studies)^*.

Before beginning a systematic review, three stages must be considered, which are: to define the object of the review, to identify the literature and to select the studies to be included. These preliminary stages are important, once they aid the researcher to adequate the problem of the review based on the information available about the theme of interest¹¹. It is worth to point out that a systematic review follows the structure of an original article, including the sections of introduction, methods, results and discussion.

DESCRIPTION AND ELABORATION OF A SYSTEMATIC REVIEW

A description of the stages that constitute the process of elaboration of a systematic review is presented below. In each stage, an illustration of the specific content will be presented, taking as example information published in a systematic review about the effectiveness of muscle force training programs for individuals with cerebral palsy¹².

The implementation of a systematic review involves a work of at least two investigators that will evaluate, independently, the methodological quality of each selected article. It is important that the investigators elaborate a research protocol that include the following items: how the studies will be found, article inclusion and exclusion criteria, definition of interest outcomes, verification of the results accuracy, determination in the studies quality and analyses of the used statistics.

Stage 1: Defining the question

As any other scientific investigation, a good systematic review requires an inquiry or a clear, well-formulated question. It must contain a description of the disease, or interest condition, the population, the context, the intervention, and the outcome.

"The objective of this review was to determine whether strengthening of the muscles produces benefic outcomes for individuals with cerebral palsy (CP)" 12.

Stage 2: Searching for evidence

Investigators must certificate that all articles considered important, or that may have some impact on the conclusion of the review, are included. The search for evidence starts with the definition of terms or keywords, followed by search strategies, databases definition, and other sources of information to be researched. For more details, see www.birreme.br and Oxman.¹³

The search in electronic databases and in other sources is an important ability on the process of implementation of a systematic review, considering that searching efficiently maximizes the possibility of finding relevant articles in a reduced time^6,13. An efficient search involves not only a strategy that includes proper terms, but also the choice of databases that include more specifically the theme (i. g., Cochrane Lybrary, MEDLINE, EMBASE, SciELO, among others). Sackett et al.² described a strategy orientation that increases the sensitivity and the specificity of the search.

"Electronic databases (MEDLINE, PubMed, EMBASE, CINAHL, Sports Discus, DARE, PsychInfo, ERIC, AusportMed, AMI, Cochrane, PEDro) were consulted retrospectively until the year of 1966, using the following keywords: 'cerebral palsy' combined with 'exercise', 'strength', and 'physical training'. The search was limited to articles written in English" 12.

Stage 3: Reviewing and selecting the studies

During the study selection, the evaluation of the titles and abstracts identified in the initial search must be done by at least two researchers, independently and blindly, strictly obeying to the inclusion and exclusion criteria defined in the research protocol. When the title and the abstract are not clear, the full article must be searched so that there is no risk of leaving important studies out of the systematic review.

The inclusion and exclusion criteria are defined based on the question that guides the review: appropriate search time (for example: 5 years), aimed population, (adults, children, athletes), interventions, measuring of the interest outcomes, methodological criteria, language, type of study, among others. The disagreements that may eventually occur must be resolved consensually¹⁴.

"[ ] articles identified by the initial search strategy were evaluated independently by two authors, by the following inclusion criteria: (1) population (adult or child), (2) intervention (strength training or a program of progressive resistance exercises, (3) outcome (measurement) of changes in strength, activity, or participation" 12.

In the attachment 2 there are some rules that may help in the initial section of articles.

Stage 4: Analyzing the methodological quality of the studies

The quality of a systematic review depends on the validity of the studies included on it. In this phase, it is important that the investigators consider all the possible error sources (bias) that may undermine the relevance of the analyzed studies. A deep knowledge of investigation of methods and statistical analyses, as well as of the measurements and measuring tools employed, is an indispensable requisite for investigators to perform their task.

There are different scales that aid in the evaluation of the studies, such as lists of Delphi, PEDro, OTSeeker, Maastricht criteria, Jadad scale, among others. Presently, the most used on the rehabilitation area is the PEDro scale (http://www.pedro.fhs.usyd.edu.au). This scale was developed by the Physiotherapy Evidence Database to be employed in experimental studies and has a total score of 10 points, including internal value evaluation criteria and employed statistical analyses presentation. For each criterion defined on the scale, one point (1) is attributed to the presence of the presented evidence quality indicators, and no point (0) is attributed to the absence of these indicators. The PEDro scale is composed by the following criteria:

1) inclusion criteria specification (non-scored item); 2) randomized allocation; 3) allocation secrecy; 4) group similarity in the initial phase; 5) subject masking; 6) therapist masking; 7) evaluation masking; 8) measuring of at least one primary outcome in 85% of the allocated subjects; 9) analysis of the intention to treat; 10) comparisons between groups of at least one primary outcome; 11) account of the variability measures and parameters estimative of at least one primary variable^15,16.

After the investigators have used the scale independently, it is appropriate to analyze the agreements between them. On this analysis, it may be employed, for example the Kappa index (k). Considering that the PEDro scale presents moderate levels of reliability between evaluators (ICC= 0,68; IC 95% = 0,57-0,76), disagreements among researchers may be resolved with discussion and consensus, when possible¹⁶.

"The studies that fulfilled the inclusion criteria were evaluated on the methodological quality with the PEDro scale, based in the Delphi list, described by Verhagen et al. [ ] studies with low methodological quality (PEDro score lesser than 3) were excluded. Were also excluded articles that presented repeated information or were available in other articles" 12.

Stage 5: Presenting the results

The articles in the systematic review may be presented in a chart that details its main characteristics, such as: authors, year of publication, methodological design, number of individuals (N), comparison groups, intervention protocol characterization (time, intensity, session frequency, etc.), dependent variables and main results. Attachment 1 illustrates an example of presentation of part of the results of a systematic review¹².

The method section is especially important and needs to be well detailed (for example, search strategies, how the studies were selected for inclusion in the systematic review among others) and susceptible to reproduction. Information about the reliability between examiners in the evaluation of the quality of the evidence needs to be presented as well as criteria used to solve the disagreements between them.

According to Law et al.⁷, many systematic reviews authors tend to communicate only the positive results from clinical trials, that is, the intervention that did come up with an effect. It is important to present also the negative results from studies, since the professionals that are in the clinic need this information to change their practice. To publish in the systematic reviews the positive and negative aspects of the interventions/treatments will only increase the knowledge about their effectiveness and limitations.

FINAL COMMENTARIES

To know the process of development of systematic reviews may aid the reader to comprehend this type of study. However, the reader must still prepare himself to evaluate the quality of the systematic review^17,18 and to select what is interesting between different reviews about the same theme¹⁹. It is important to consider how the conclusion from this kind of study may be applied on the clinical practice, taking into account the patient and the context in which it will be applied.

The publication of systematic review studies, as well as others that synthesize research results, is a step for an evidence-based practice. However, for that to occur, it becomes necessary a behavior change from the health professionals. This change implicates not only in consuming the available literature, but also in taking this information to the daily clinical practice. The final object of this process is to improve the quality of the care by health professionals.

REFERENCES

Received: 20/09/2006

Revised: 20/12/2006

Accepted: 27/12/2006

ANNEX 1

Presentation example of part of the results of a systematic review.

^{Clique para ampliar}

ANNEX 2

Orientation for article selection that may provide valid results§.

• 1. Bennett S, Bennett JW. The process of evidence-based practice in occupational therapy: informing clinical decisions. Austral Occup Ther J. 2000;47:171-80.
• 2. Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB. Evidence-Based Medicine: how to practice and teach EBM. 2Ş ed. London: Churchill Livingstone; 2000.
• 3. Law M, Baum C. Evidence-based practice. Can J Occup Ther. 1998;65:131-5.
• 4. Sampaio RF, Mancini MC, Fonseca ST. Prática baseada em evidência: buscando informação para fundamentar a prática clínica do fisioterapeuta e do terapeuta ocupacional. Rev. Bras. Fisioter. 2002;6(3):113-8.
• 5. Linde K, Willich SN. How objective are systematic reviews? Differences between reviews on complementary medicine. J R Soc Med. 2003;96:17-22.
• 6. Akobeng AK. Understanding systematic reviews and meta-analysis. Arch Dis Child. 2005;90:845-8.
• 7. Law M, Philp I. Systematically reviewing the evidence. In: Law M. Evidence-based rehabilitation: a guide to practice. Thorofare (NJ): SLACK Inc; 2002.
• 8. Magee DJ. Systematic reviews (meta-analysis) and functional outcome measures (apostila). Developmental Editor: B. Aindow, 1998.
• 9. Akobeng AK. Understanding randomized controlled trials. Arch Dis Child. 2005;90:840-4.
• 10. Evans D. Hierarchy of evidence: a framework for ranking evidence evaluating healthcare interventions. J Clin Nurs. 2003; 12:77-84.
• 11. Domholdt E. Rehabilitation research: principles and applications. Missouri: Elsevier Saunders; 2005.
• 12. Dodd KJ, Taylor NF, Damiano DL. A systematic review of the effectiveness of strength-training programs for people with cerebral palsy. Arch Phys Med Rehabil. 2002;83:1157-64.
• 13. Oxman AD, Sackett DL, Guyatt GH. Users' guide to the medical literature I: how to get started. JAMA. 1993;270:2093-5.
• 14. Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, et al. BMJ. 2000;320:537-40.
• 15. Chagas PSC, Mancini MC, Barbosa AP, Silva PTG. Análise das intervenções utilizadas para a promoção da marcha em crianças portadoras de paralisia cerebral: uma revisão sistemática da literatura. Rev. Bras. Fisioter. 2004;8(2):155-63.
• 16. Maher CG, Sherrington C, Herbert RD, Moseley AM, Elkins M. Reliability of the PEDro scale for rating quality of randomized controlled trials. Phys. Ther. 2003;83:713-21.
• 17. Oxman AD, Cook DJ, Guyatt GH. Evidence-based medicine working group. User´s guide to the medical literature: IV. How to use an overview. JAMA [periódico na Internet]. 1994; 272:1367-71. Disponível em: URL: <http://www.cche.net/principles/content_all.asp>.
• 18. Greenhalgh T. How to read a paper: papers that summarise other papers (systematic reviews and meta-analyses). BMJ, 1997;315:672-675. Disponível em: URL: <http://www.bmj.com/collections/red.shtml>.
• 19. Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. Can Med Assoc J. 1997;156:1411-6.

Correspondence to:

Rosana F. Sampaio

Rua Juvenal dos Santos, 222, apto. 602, Luxemburgo

CEP 30380-530, Belo Horizonte, MG - Brazil

*

For definition of evidence levels, see <

http://www.cebm.net/levels_of_evidence.asp#/levels>.

†

Figure adapted from Evans (2003); Akonbeng (2005b).

‡

Adapted from Domholdt (2005), Law & Philp (2002) and Magee (1998).

Publication Dates

• Publication in this collection
  23 Mar 2007
• Date of issue
  Feb 2007

History

• Accepted
  27 Dec 2006
• Reviewed
  20 Dec 2006
• Received
  20 Sept 2006