Should systematic reviews and meta-analyses include data from preprints?

As attentive observers of the rapidly developing literature on the effect of the Covid-19 pandemic in mental health, we have been witnessing an interesting conundrum: should systematic reviews and meta-analyses include data from preprints in their results?¹

Publication of manuscripts in a traditional peer-reviewed journal can take a long time, while preprints are readily available online. Preprint articles were already becoming more common, mainly due to the need to protect intellectual property, but preprint platforms gained a remarkable impulse since the arrival of the Covid-19 pandemic.² Today, there is a pressing need to make data on Covid-19 available as quickly as possible and the scientific community has been willing to be more flexible in terms of what would be the ideal process to consider results as both scientifically valid and clinically useful.

Searching for unpublished data is common when conducting systematic reviews and meta-analyses. The so called ‘grey literature’ has been a source of data for these types of studies and it is common to read in the methods of systematic reviews and meta-analyses that authors have also searched for conference abstracts or even personal communications between researchers. It is thought that this type of search could help to reduce publication bias, providing readers a more balanced view of the knowledge on the topic.³ In addition, the fact that a study has not been peer-reviewed does not necessarily mean it lacks quality or clinical applicability.

In a recent study, Carneiro et al. contrasted the quality of data reporting in independent samples of preprints with published studies.⁴ They also made pairwise comparisons of preprint versions with their published versions. They found that, on average, the quality of reporting is slightly better for peer-reviewed journal articles compared to preprint articles. Compared to peer-reviewed articles, preprint articles were less likely to report conflicts of interest, funding sources, and unit-level (most commonly individual people) data. These data are not in accordance with other studies which did not find significant differences between the two versions.⁵-⁷ Conflicts of interest are a very relevant variable which is commonly used in sensitivity analyses in meta-analyses of interventional studies.⁸ It should be noted that preprint repositories have recently changed their policy for reporting conflicts of interest, with medRxiv requiring disclosure of conflicts of interest, while bioRxiv was not requiring it at the time the study by Carneiro et al. was conducted.⁴

Preprint articles have the unquestionable advantage of quick availability online. While an article may take several months or even more than a year to be published in a journal, publication on a preprint server takes less than a week. Currently, many journals are applying fast-tracked processes to articles related to COVID-19, but the peer-review process still requires time for qualified referees.⁹

To add an additional layer of complexity, Schmucker et al. performed a methodological research study of systematic reviews and meta-analyses in medicine.¹⁰ Their aim was to investigate if including unpublished data would have an impact on results. They found that, although there were some cases that not including unpublished data led to an overestimation of the results, this only affected a minority of reviews. However, when researchers propose systematic reviews and meta-analyses in which a significant volume of information will come from unpublished material, this generates new sources of biases. For example, most preprint datasets do not provide a search strategy that is equivalent to major databases, such as PubMed. So, it becomes impossible to guarantee the reproducibility of the retrieval, because how many of the studies would or would not be found is subject to significant variation.

This is only one of the arguments against inclusion of preprints in systematic reviews and meta-analyses. Other criticisms are:

1. The possibility that the results change further and the findings of the review and meta-analyses must also be modified;

2. Some preprints will never pass the minimum quality criteria for publication and in theory should be excluded from a meta-analysis;

3. Possibility of conflicting information between the preprints and the final form of the published article (e.g. authorship, additional analysis, discussion of the results). Notwithstanding, these challenges should not lead us to ignore that results on a research question were generated, even when they are made public only in a preprint.

Future approaches to this topic could include the obligation to disclose conflicts of interests, funding, and unit-level data in preprints, a better description of the search strategy in the preprint datasets, improvement of the advanced search tools for the preprint datasets, standardized analysis of the quality of the gray literature and preprints, inclusion of additional exploratory analysis of peer-reviewed and not peer-reviewed material, and, finally, a balanced discussion of the impact of inclusion of these two types of literature on future systematic reviews and meta-analyses. It also should be highlighted that, irrespective of the inclusion criteria adopted in systematic reviews and meta-analyses, they should be considered temporary and be updated periodically. The improvement of meta-science methods can be a good collateral effect of the Covid-19 pandemic. It may not only benefit the literature in the field but also other subjects in mental health, accelerating interpretation of evidence and knowledge translation.

References

• ¹
  Lukito AA, Pranata R, Henrina J, Lim MA, Lawrensia S, Suastika K. The effect of metformin consumption on mortality in hospitalized COVID-19 patients: a systematic review and meta-analysis. Diabetes Metab Syndr. 2020;14:2177-83.
• ²
  Lai X, Liu J, Zhang T, Feng L, Jiang P, Kang L, et al. Clinical, laboratory and imaging predictors for critical illness and mortality in patients with COVID19: protocol for a systematic review and meta-analysis. BMJ Open. 2020;10:e039813.
• ³
  Joober R, Schmitz N, Lawrence Annable D, Boksa P. Publication bias: what are the challenges and can they be overcome? J Psychiatry Neurosci. 2012;37:149-52.
• ⁴
  Carneiro CFD, Queiroz VGS, Moulin TC, Carvalho CAM, Haas CB, Rayee D, et al. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. Res Integr Peer Rev. 2020;5:16.
• ⁵
  Klein M, Broadwell P, Farb SE, Grappone T. Comparing published scientific journal articles to their pre-print versions. Int J Digit Libr. 2019;20:335-50.
• ⁶
  Polka K, Dey G, Palfy M, Nanni F, Brierley L, Fraser N, et al. Preprints in motion: tracking changes between posting and journal publication. bioRxiv. 2021.02.20.432090. Epub 2021 Feb 20. doi.org/10-.1101/2021.02.20.432090
  » doi.org/10-.1101/2021.02.20.432090
• ⁷
  Bero L, Lawrence R, Leslie L, Chiu K, McDonald S, Page MJ, et al. Cross-sectional study of preprints and final journal publications from COVID-19 studies: discrepancies in results reporting and spin in interpretation. BMJ Open. 2021;11:e051821.
• ⁸
  Elmore SA. Preprints: what role do these have in communicating scientific results? Toxicol Pathol. 2018;46:364-5.
• ⁹
  Martins RS, Cheema DA, Sohail MR. The pandemic of publications: are we sacrificing quality for quantity? Mayo Clin Proc. 2020;95:2288-90.
• ¹⁰
  Schmucker CM, Blumle A, Schell LK, Schwarzer G, Oeller P, Cabrera L, et al. Systematic review finds that study data not published in full text articles have unclear impact on meta-analyses results in medical research. PLoS One. 2017;12:e0176210.

Publication Dates

• Publication in this collection
  14 Apr 2023
• Date of issue
  2023

History

• Received
  01 June 2021
• Accepted
  14 July 2021