Quality assessment of randomized clinical trial in intensive care

Abstracts

OBJETIVO: O ensaio clínico aleatório é um estudo prospectivo que compara o efeito e o valor das intervenções em seres humanos, utilizando um ou mais grupos contra o grupo controle. O objetivo deste estudo foi avaliar a qualidade dos ensaios clínicos aleatórios publicados em terapia intensiva no Brasil. MÉTODOS: Todos os ensaios clínicos aleatórios encontrados através da busca manual na Revista Brasileira de Terapia Intensiva de janeiro de 2001 a março de 2008 foram analisados para avaliar sua descrição através da escala de qualidade. Foi utilizada uma estatística descritiva e intervalo de confiança de 95% para a variável primária. A variável primária foi a qualidade dos ensaios clínicos aleatórios. RESULTADOS: Foram encontrados 185 artigos originais, sendo 14 de ensaios clínicos aleatórios. Apenas um artigo original analisado (7,1%) mostrou-se de boa qualidade. Não houve significância estatística entre os dados coletados e os dados mostrados na hipótese desta pesquisa. CONCLUSÃO: Pode-se concluir que na amostra de artigos avaliada 7% dos ensaios clínicos aleatórios em terapia intensiva publicados em uma revista no Brasil são de boa qualidade metodológica.

Avaliação em saúde; Ensaios clínicos controlados aleatórios como assunto; Publicações; Unidade de terapia intensiva

ORIGINAL ARTICLE

Quality assessment of randomized clinical trial in intensive care

Giulliano Peixoto Gonçalves^I; Fabiano Timbó Barbosa^II; Luciano Timbó Barbosa^III; José Lira Duarte^IV

^IMedical Student at the Universidade Estadual de Ciências da Saúde de Alagoas; Member of the Anesthesiology, Pain and Intensive Care League of Alagoas - Maceió (AL), Brazil

^IIPhysician of the Clínica Santa Juliana - Maceió (AL), Brazil; Tutor of the Anesthesiology, Pain and Intensive Care League of Alagoas - Maceió (AL), Brazil

^IIIPhysician of the Hospital da Agroindústria do Açúcar e do Álcool do Estado de Alagoas - Maceió (AL), Brazil

^IVPhysician of the Emergency Unit, Doutor Armando Lages - Maceió (AL), Brazil

Author for correspondence

ABSTRACT

OBJECTIVE: A randomized clinical trial is a prospective study that compares the effect and value of interventions in human beings, of one or more groups vs. a control group. The objective of this study was to evaluate the quality of published randomized clinical trials in Intensive care in Brazil.

METHODS: All randomized clinical trials in intensive care found by manual search in Revista Brasileira de Terapia Intensiva from January 2001 to March 2008 were assessed to evaluate their description by the quality scale. Descriptive statistics and a 95 % confidence interval were used for the primary outcome. Our primary outcome was the randomized clinical trial quality.

RESULTS: Our search found 185 original articles, of which 14 were randomized clinical trials. Only one original article (7.1%) showed good quality. There was no statistical significance between the collected data and the data shown in the hypothesis of this search.

CONCLUSION: It can be concluded that in the sample of assessed articles 7% of the randomized clinical trials in intensive care published in a single intensive care journal in Brazil, present good methodological quality.

Keywords: Health evaluation; Randomized controlled trials as topic; Publications; Intensive care units

INTRODUCTION

The randomized clinical trial is a type of prospective study that compares the effect and value of interventions in human beings, involving one or more groups of intervention and at least one control group, with randomized allocation of participants and using control measures.^(1,2) In the majority of cases Intervention under consideration is compared to the most common and more adequate procedure to date. Should there be no such procedure; the experimental group is compared to a placebo group.⁽¹⁾ Randomized clinical trials are viewed as studies that form the basis for the advance of science, because they have less possibility of biases during the investigation of the phenomenon of interest.⁽³⁾

Methodological quality assessment enables the reader to analyze a survey's performance and verify the applicability of these findings in his daily clinical practice. Some authors already published their results related to methodological quality assessment in other areas in or outside of medicine, with variable results, as there are few articles with good methodological quality.^(4-7)

The objective of this survey was to assess the quality of original articles of the randomized clinical trials in the field of intensive care in Brazil.

METHODS

This survey was submitted to the Committee on Ethics in Research of the Universidade de Ciências da Saúde of Alagoas (protocol 861). A term of informed consent is not applicable in this type of survey. The authors were in charge of the expenses inherent to this survey. The hypothesis tested in this survey was that 5% of the original articles of randomized clinical trials published on intensive care in Brazil were of good methodological quality.

This was a descriptive study for assessment of the quality of randomized clinical trials. Inclusion criterion was: original article of a randomized clinical trial published on intensive care in Brazil. Exclusion criteria were: articles regarding research with animals, articles with incomplete descriptions and original articles involving the pediatric age bracket.

The primary variable analyzed was quality of randomized clinical trials, defined as the probability that a clinical trial will generate unbiased outcomes.⁽⁸⁾ Secondary variables were: forwarding of the survey to a Committee of Ethics in Research, use of the informed consent, description of the funding source, sample size calculation, site of origin, statistical test utilized and classification of the type of study.

Original articles from the "Revista Brasileira de Terapia Intensiva" published from January 2001 to March 2008, were analyzed by a manual search for the words random, randomized, double blind, placebo or any other words suggesting that the article was about a randomized clinical trial. Titles, summaries and key words were initially reviewed to separate randomized clinical trials.

Randomized clinical trials were separated, read in their entirety and submitted to quality assessment using the quality scale that can be seen in chart 1.

The quality scale assessment criteria used in the survey were:⁽⁸⁾ randomizing, double blind masking and the set of losses and exclusions. These items may be described as follows:

a) For randomization: the method of generating the randomized sequence was considered appropriated when it permitted that each participant had the same chance of receiving each intervention and when the researcher was unable to foresee which would be the next treatment.

b) For double blind masking: a study was considered double blind when the term double blind was used. The method was considered appropriate when neither the patient nor the person in charge of data collection were able to identify the type of treatment given to each one, or in absence of this statement, if the use of an identical placebo or imitations were mentioned.

c) For the losses and exclusions: participants who entered the study but did not fulfill the observation period or were not included in the analysis and who were described by the authors of original articles. Number and reasons for losses in each group must be stated. When there are no losses, this must also be stated in the article. When there was no description of loss, the item was ascribed a zero.

A maximum of five points could be achieved by this scale: one point for each yes, an additional point for an adequate randomization method and another additional point for an adequate masking method.

When the term double blind was not mentioned, however there was a description of the patient's and researcher's masking of the variables, this item was scored in the quality scale. A study was considered to be of poor quality when it had only two points or less on the quality scale.⁽⁸⁾

Original articles of randomized clinical trials were analyzed and classified for concealment of allocation as follows:

A - Concealment of allocation properly performed and described;⁽²⁾

B - Study was described as randomized, but there is no description of randomization nor of concealment of allocation;

C - The study was described as randomized, but the survey method was inadequate, such as: use of the number on medical charts; birth date; alternation of the week days or any other form that had not been considered transparent for the authors of this survey;

D - The study was not a randomized clinical trial.

Articles that were not about a randomized clinical trial were analyzed to confirm their exclusion from the quality assessment and in the search for secondary variables. Secondary variables were also analyzed in the original articles of randomized clinical trials.

Descriptive statistics were used in addition to calculation of the 95% confidence intervals (CI 95%) for results of the methodological quality analysis. Survey data were compared with those of the hypothesis using the Chi-square, considering a 5% significance level.

RESULTS

One hundred and eighty five original articles on intensive care, published in Brazil were found and analyzed. Of these 185, 15 were initially classified as randomized clinical trials.^(9-23) These 15 original articles were analyzed regarding concealment of allocation, and six articles^(17,19-23) (40%) ranked as A, other six articles^{(10,13-16,18)} (40%) ranked as B, two articles^(11,12) (13.3%) as C and one article⁽⁹⁾ (6.7%) that was not a randomized clinical trial was ranked as D.

Quality analysis could only be performed in 14 original articles of randomized clinical trials because one article⁽⁹⁾ was excluded from this analysis. Results of the quality assessment disclosed that 7% (1/13) of the articles were of good quality⁽²²⁾ (Table 1). Score distribution on the quality scale was: zero for 8 (57.2%) original articles^(10-16,18) 2 points for 5 (35.7%)^{(17,19-21,23)} and 4 points for one (7.1%) original article that presented good methodological quality.⁽²²⁾

The assessment of the secondary variables of all original articles showed that: 110 (59.5%) original articles described submitting the project to a Committee of Ethics in Research, 53 (28.6%) described use of informed consent, 19 (10.3%) described sample size calculation, 146 (78.9%) described the variables analyzed, 118 (63.8%) described inclusion criteria, 90 (48.6%) described exclusion criteria and in only 3 (1.6%) was there a description of the funding source.

Regarding origin of the original articles, it was noted that the largest contribution came from the state of São Paulo with 88 (47.3%) original articles (Graph 1).

Classification of the original articles by type of study is presented in table 2, and the cohort study was the most common, described in 106 (57.3%) of original articles.

Results of the statistical tests used in the original articles are shown in table 3. The test most often used was the Chi-square in 72 (38.9%) articles, followed by the Student's t test in 59 (31.9%) articles.

Results of the quality analysis of the original articles on randomized clinical trials of this survey were compared with data of the hypothesis and there was no statistical significance (p=0.71).

DISCUSSION

Currently, randomized clinical trials are being described as the gold standard for the assessment of therapeutic issues in health.⁽²⁴⁾ This type of study reduced the probability of obtaining biased data in a survey.⁽²⁵⁾ Occurrence of systematic error may be avoided by using randomization masking principles and analysis of all the recruited subjects.⁽²⁶⁾

In this survey 14 original articles of randomized clinical trials were found, however only one presented good methodological quality according to the quality scale.⁽⁸⁾ The fact that the authors state that the study was randomized does not warrant to the reader or those assessing the article, that it truly was a clinical trial. The randomization principle in this type of study means that the allocation of the surveyed subjects to these study groups must be randomized,⁽¹⁾ participants must have the same probability of receiving interventions as well as being tested regarding their control.⁽²⁷⁾ If this principle is respected and adequately preformed, randomization reduces risk of systematic errors balancing the risk factors that could influence the clinical outcome to be measured.⁽²⁸⁾ Some of the authors analyzed did not describe the randomization method and therefore lost the point for this item on the scale.

When assessing masking it was perceived that most authors, directly or indirectly mentioned that there was masking, but did not describe it adequately. The masking principle states that, whenever possible it must be guaranteed that individuals involved as subjects of survey as well as those obtaining the data, do not know which group is receiving intervention or belongs to the control, thus being characterized as double-blind.⁽²⁹⁾ Masking helps to avoid occurrence of biases due to subjectivity of the researcher as well as of the patients.

A joint description of losses and exclusions that took place after randomization of patients was not found in any of the 14 original articles, therefore no article scored in this item. It is important that the reader has access to the rates and reasons of these losses and exclusions to enable him to assess the feasibility of performing the intervention in his daily practice.

In this survey it was observed that 100 (59.5%) original articles mentioned contact with the Research Ethics Committee, 53 (28.6%) mentioned use of the informed consent and, 3 (1.6%) articles described a funding source. These items were also analyzed by other scales and lists as integral part of the methodological quality assessment and the description gives quality and credibility to the publication.^(30,31) Such items should be described in all publications regarding research with human beings.

In this survey it was observed that 19 (10.3%) original articles described sample size calculation. Sample size has an inverse influence on the p value, that is why, very large samples tend to present low p values and lead to errors in decision making concerning the differences found in the survey.⁽³²⁾ Failure to present sample size calculation jeopardizes the validity of outcomes in any survey.

In this survey 146 (78.9%) original articles described the analyzed variables, 118 (63.8%) described inclusion criteria and 90 (48.6%) the exclusion criteria. Description of these items leads to a better understanding of eligibility criteria used. This permits the reader of original articles to judge if it is possible to extrapolate results found in a survey to patients attended in daily practice.⁽³³⁾

The state of São Paulo had the largest contribution of original articles 88 (47.3%) published. These results can be explained by the larger number of universities, researchers, training centers and scientific events in that city.

Classification of the original articles disclosed that the most frequent survey was the cohort study with 106 (57.3%) articles. This portrays the authors' intention to carry out a survey involving an intervention, however the best way to confirm the effect of a treatment is the randomized clinical trial.⁽²⁾

The statistical test most often used is the Chi-square (38.9%). Tests of the hypothesis are expressed in p values that represent the probability of an event taking place in a sample, even if this event is null in the populations that originated the sample.⁽³²⁾ The Chi-square serves to analyze the proportions of the categorical variables and together with the Student's t test may be considered as one of the most often used in health surveys.⁽³³⁾

Comparison between data of this survey and the formulated hypothesis disclosed that there was no statistical significance (p=0.7). When results are negative and there is no mention of sample size calculation, some authors state that the publication regarding the survey does not deserve any credit by the readers.⁽³³⁾ For this reason we calculated sample size after concluding the survey considering a hypothesis of 5%, an absolute precision of 4% and a significance level of 5% and obtained as an outcome 114 original articles. This number of original articles needed to confirm our hypothesis was lower than the number of articles assessed in this survey, confirming our results that the number of good quality original articles is as low as 5%.

Limitations of this survey were: use of only one scale for quality assessment, assessment of only one of the Brazilian journal that publishes intensive care in Brazil and quality assessment by only one reviewer of the articles. The quality scale used in this survey⁽⁸⁾ assesses only the items of internal validity, neglecting those of external validity and the statistical method utilized. Other forms of the methodological quality assessment could also have been used such as individual items and lists.⁽³¹⁾ Assessment of only one journal prevented the authors from observing the clinical trials carried out in Brazil but published in other journals, besides Brazilian works published in international journals. It is recommended that more than one reviewer assesses the articles and that discrepancies be resolved by consensus meetings, as such conduct would help to reduce biases.⁽⁸⁾

CONCLUSION

It can be concluded that in the sample of articles assessed, 7% of the randomized clinical trials on intensive care published in one of the Brazilian journals, are of good methodological quality.

REFERENCES

• 1. Castelo Filho A, Sesso RCC, Atallah NA. Epidemiologia clínica: uma ciência básica para o clínico. J Pneumol. 1989;15(2):89-98.
• 2. Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Version 4.2.5 [updated May 2005]. Chichester, England: John Wiley & Sons Ltd; Appendix 8a:215-229. [cietd 2006 Apr 24]. Available from: http://www.cochrane.org/resources/handbook/hbook.htm
• 3. Schulz KF. Unbiased research and the human spirit: the challenges of randomized controlled trials. CMAJ. 1995;153(6):783-6. Comment in: CMAJ. 1995;153(12):1708-10. Comment on: CMAJ. 1995;153(6):769-79.
• 4. Greenfield ML, Rosenberg AL, O'Reilly M, Shanks AM, Sliwinski MJ, Nauss MD. The quality of randomized controlled trials in major anesthesiology journals. Anesth Analg. 2005;100(6):1759-64.
• 5. Bauer DC. Randomized trial reporting in general endocrine journals: the good, the bad, and the ugly. J Clin Endocrinol Metab. 2008;93(10):3733-4. Comment on: J Clin Endocrinol Metab. 2008 Oct;93(10):3810-6.
• 6. Harrison JE. Clinical trials in orthodontics II: assessment of the quality of reporting of clinical trials published in three orthodontics journal between 1989 and 1998. J Orthod. 2003;30(4):309-15; discussion 297-8.
• 7. Jüni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. BMJ. 2001;323(7303):42-6.
• 8. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1-12.
• 9. Rodrigues Júnior GR, Amaral JLG. Impacto psicológico da internação na Unidade de Terapia Intensiva. Rev Bras Ter Intensiva. 2001;13(4):92-8.
• 10. Oliveira LRC, José A, Dias EC, Santos VLA, Chiavone PA. Protocolo de desmame da ventilação mecânica: efeitos da sua utilização em uma Unidade de Terapia Intensiva. Um estudo controlado, prospectivo e randomizado. Rev Bras Ter Intensiva. 2002;14(1):22-32.
• 11. Piras C. A gasometria arterial na relação tempo entre a coleta e realização do exame. Rev Bras Ter Intensiva. 2002;14(3):95-8.
• 12. Figueirêdo LC de, Araújo S, Kosour C, Veloso CA, Petrucci Júnior O. Comparação entre dois métodos de retirada do suporte ventilatório mecânico no pós-operatório de cirurgia cardíaca. Rev Bras Ter Intensiva. 2003;15(3):114-20.
• 13. Vieira SRR, Lu Q, Rouby JJ. A scanographic assessment of alveolar recruitment and overinflation during high frequency ventilation - preliminary results in three patients. Rev Bras Ter Intensiva. 2004;16(3):138-41.
• 14. Yamaguti WPS, Alves LA, Kauss IAM, Galvan CCR, Brunetto AF. Comparação entre a pressão inspiratória máxima medida pelo método da válvula unidirecional e pelo convencional em pacientes submetidos ao processo de desmame da ventilação mecânica invasiva. Rev Bras Ter Intensiva. 2004;16(3):142-5.
• 15. Fiore Júnior JF, Oliveira ACS, Pinho E, Benassule E, Francischini J, Chiavagato LD. O uso de baixos níveis de pressão suporte influencia a avaliação de parâmetros de desmame? Rev Bras Ter Intensiva. 2004;16(3):146-9.
• 16. Ferreira JC, Valiatti J, Schettino GPP, Bonassa J, Iwata L, Carvalho CCR. Comparação do modo VAPS com os modos volume controlado e pressão controlada em pacientes com insuficiência respiratória aguda. Rev Bras Ter Intensiva. 2005;17(2):89-93
• 17. Viana JM, Martins MA, Campos AD, Basile-Filho A. Análise comparativa entre os pesos corpóreos estimados e os medidos obtidos de pacientes em estado crítico. Rev Bras Ter Intensiva. 2005;17(4):238-45.
• 18. Moritz RD, Souza RL, Machado FO. Avaliação de um algoritmo para a adequação da sedoanalgesia de pacientes internados em UTI e submetidos à ventilação mecânica. Rev Bras Ter Intensiva. 2005;17(4):265-9.
• 19. Arantes AS, Christiano Júnior AC, Abreu SP, Moraes JMNF, Gandolfi JV, Leite LG, Lobo SM. Doses baixas de dobutamina e fluidos no pós-operatório de pacientes de alto risco: efeitos sobre a oxigenação tecidual, resposta inflamatória e morbidade. Rev Bras Ter Intensiva. 2007;19(1):5-13.
• 20. Lage SG, Carvalho RT, Kopel L, Bastos JF, Ribeiro MA, Fagundes Júnior AAP, et al. Estudo de segurança e eficácia da enoxaparina sódica na profilaxia e terapêutica antitrombótica. Rev Bras Ter Intensiva. 2007;19(1):67-73.
• 21. Rosa FK da, Roese CA, Savi A, Dias AS, Monteiro MB. Comportamento da mecânica pulmonar após a aplicação de protocolo de fisioterapia respiratória e aspiração traqueal em pacientes com ventilação mecânica invasiva. Rev Bras Ter Intensiva. 2007;19(2):170-5.
• 22. Molon M, Piva J, Karcher P, Baldissera T. Clonidina associada à morfina e midazolam em crianças submetidas à ventilação mecânica: estudo aleatório, duplamente encoberto e placebo controlado. Rev Bras Ter Intensiva. 2007;19(3):284-91.
• 23. Azevedo JRA, Azevedo RP, Cossetti RJD, Lima ERM, Sousa GB. Insuficiência adrenal relativa no choque séptico: comparação da resposta ao tratamento com hidrocortisona em pacientes diagnosticados através de dosagem única do cortisol plasmático versus teste da corticotropina. Rev Bras Ter Intensiva. 2008;20(1):18-23.
• ²⁴
  Council for International Organizations of Medical Sciences(CIOMS). World Health Organization (WHO). International ethical guidelines for biomedical research involving human subjects. Washington: OPAS; 1996.
• 25. Atallah AN, Castro AA. Medicina baseada em evidências: fundamentos da pesquisa clínica. São Paulo: Lemos Editorial; 1998.
• 26. Schulz KF, Chalmers I, Grimes DA, Altman DG. Assessing the quality of randomization from reports of controlled trials published in obstetrics and gynecology journals. JAMA. 1994;272(2):125-8. Comment in: JAMA. 1995;274(23):1835- 6. JAMA. 2001;286(20):2546-7. JAMA. 2001;286(20):2546; author reply 2547.
• 27. Yusuf S, Collins R, Peto R. Why do we need some large, simple randomized trials? Stat Med. 1984;3(4): 409-22.
• 28. Collins R, Peto R, Gray R, Parish S. Large-scale randomized evidence: trials and overviews. In: Weatherall DJ, Ledingham JGG, Warrell DA. Oxford Textbook of Medicine, 3a. ed. Oxford: Oxford University Press; 1996. P. 21-32.
• 29. Newell DJ. Intention-to-treat analysis: implications for quantitative and qualitative research. Int J Epidemiol. 1992;21(5):837- 41. Comment in: Int J Epidemiol. 1993;22(3):566.
• 30. Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA. 1995;273(5):408-12. Comment in: JAMA. 2001;286(20):2546-7.
• 31. Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG. The Delphi list: a criteria list for quality assessment of randomized clinical trials conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998;51(12):1235-41.
• 32. Cavalcanti AB, Akamine N, Sousa JMA. Avaliação crítica da literatura. In: Knobel E. Condutas no paciente grave. 3a. ed. São Paulo: Atheneu; 2006. P. 2635-47.
• 33. Dawson B, Trapp RG. Bioestatística básica e clínica. 3a. ed. Rio de Janeiro: McGraw Hill; 2003.

Publication Dates

• Publication in this collection
  04 May 2009
• Date of issue
  Mar 2009

History

• Accepted
  10 Feb 2000
• Received
  28 July 2008