Reference |
Area |
Recruitment |
# of studies |
# of authors |
Study sample |
Experiment selection |
Replications/experiment |
Replication criteria |
Replication rate (%) |
Errington et al.3131. Errington TM, Iorns E, Gunn W, Tan FE, Lomax J, Nosek BA. Science forum: an open investigation of the reproducibility of cancer biology research. Elife. 2014;3:e04333. |
Cancer biology |
Science exchange |
18 (16 completed) |
84 |
High-profile studies (2010-2012) |
Main results |
1 |
p < 0.05, meta-analysis (experiments), subjective assessment (article) |
31-68 |
Klein et al.3232. Klein RA, Ratliff K, Vianello M, Adams RB Jr, Bahník Š, Bernstein MJ, et al. Investigating variation in replicability. Soc psychol. 2014;45(3):142-52. |
Psychology |
Open call |
12 |
51 |
Ad-hoc selection |
Ad-hoc selection |
36 |
p < 0.05 (aggregate), effect size comparison |
76-85 |
Nosek et al.1010. Nosek BA, Aarts AA, Anderson CJ, Anderson JE, Kappes HB; Collaboration OS. Estimating the reproducibility of psychological science. Science. 2015;349(6251):aac4716. |
Psychology |
Open call |
100 |
270 |
Three psychology journals (2008) |
Last study |
1 |
p < 0.05, effect size comparison, 95%CI, meta-analysis, subjective assessment |
36-47 |
Klein et al.3333. Klein RA, Vianello M, Hasselman F, Adams BG, Adams RB Jr, Alper S, et al. Many Labs 2: investigating variation in replicability across samples and settings. Adv Methods Pract Psychol Sci. 2018;1(4):443-90. |
Psychology |
Open call |
28 |
177 |
Ad-hoc selection (community) |
Ad-hoc selection |
57-58 |
p < 0.05 or < 0.0001 (aggregate) |
50-54 |
Camerer et al.3434. Camerer CF, Dreber A, Forsell E, Ho TH, Huber J, Johannesson M, et al. Evaluating replicability of laboratory experiments in economics. Science. 2016;351(6280):1433-6. |
Economics |
Unspecified |
18 |
18 |
Two economics journals (2011-2014) |
Main result |
1 |
p < 0.05, 95%CI, meta-analysis, 95%PI |
61-78 |
Ebersole et al.3535. Ebersole CR, Atherton OE, Belanger AL, Skulborstad HM, Allen JM, Banks JB, et al. Many Labs 3: evaluating participant pool quality across the academic semester via replication. J Exp Soc l Psychol. 2016;67(1)68-82. |
Psychology |
Open call |
10 |
64 |
Ad-hoc selection (community) |
Ad-hoc selection |
20 |
p < 0.05 (aggregate) |
30-60 |
Cova et al.3636. Cova F, Strickland B, Abatista A, Allard A, Andow J, AttieM, et al. Estimating the reproducibility of experimental philosophy. Rev Phil Psych. 2018;1-36. |
Experimental philosophy |
Open call |
40 |
42 |
Most cited + random studies (2003-2015) |
First study |
1 |
p < 0.05, subjective assessment, 95%CI |
70-78 |
Camerer et al.2525. Camerer CF, Dreber A, Holzmeister F, Ho T-H, Huber J, Johannesson M, et al. Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nat Hum Behav. 2018 Sep;2(9):637-44. |
Social Sciences |
Open call |
21 |
24 |
Nature/Science (2010 and 2015) |
First study |
1 |
p < 0.05, meta-analysis, 95%PI, “small telescopes”, default Bayes factor, bayesian mixture model |
57-67 |