National Institutes of Health. Advancing NIH’s Mission through a Unified Strategy https://www.nih.gov/about-nih/nih-director/statements/advancing-nihs-mission-through-unified-strategy (2026).
National Institutes of Health. Leading in Gold Standard Science—an NIH Implementation Plan https://www.nih.gov/sites/default/files/2025-08/2025-gss.pdf (2025).
National Institutes of Health. 2024 NIH Public Access Policy https://grants.nih.gov/grants/guide/notice-files/NOT-OD-25-047.html (2024).
National Institutes of Health. Autism Data Science Initiative https://dpcpsi.nih.gov/autism-data-science-initiative (2026).
Kaiser, J. NIH sets out to double check biomedical studies. Science 387, 126 https://doi.org/10.1126/science.adv7964 (2025).
Cobey, K. D. et al. Biomedical researchers’ perspectives on the reproducibility of research. PLoS Biol. 22, e3002870 https://doi.org/10.1371/journal.pbio.3002870 (2024).
Amaral, O. B., Neves, K., Wasilewska-Sampaio, A. P. & Carneiro, C. F. The Brazilian Reproducibility Initiative. eLife 8, e41602 https://doi.org/10.7554/elife.41602 (2019)
Digital Science, Springer Nature & Figshare. The State of Open Data 2025: a Decade of Progress and Challenges 2026 https://doi.org/10.6084/m9.figshare.30823079 (2026).
The National Academies of Sciences, Engineering, and Medicine. Reproducibility and Replicability in Science (National Academies Press, 2019).
Goodman, S. N., Fanelli, D. & Ioannidis, J. P. What does research reproducibility mean?. Sci. Transl. Med. 8, 341ps12–341ps12 (2016).
Barba, L. A. Terminologies for reproducible research. Preprint at https://doi.org/10.48550/arXiv.1802.03311 (2018).
Plesser, H. E. Reproducibility vs. replicability: a brief history of a confused terminology. Front. Neuroinform. 11, 76 (2018).
Pellizzari, E. D., Lohr, K. N., Blatecky, A. R. & Creel, D. R. Reproducibility: a Primer on Semantics and Implications for Research (RTI Press, 2017).
House of Commons Science, Innovation and Technology Committee. Reproducibility and Research Integrity. Sixth Report of Session 2022–2023 (HC 101) https://publications.parliament.uk/pa/cm5803/cmselect/cmsctech/101/report.html (2023).
UNESCO. UNESCO Recommendation on Open Science https://unesdoc.unesco.org/ark:/48223/pf0000379949/ (2021).
Steyerberg, E. W. Clinical Prediction Models: a Practical Approach to Development, Validation, and Updating (Springer, 2009).
MIT Critical Data. Secondary Analysis of Electronic Health Records (Springer, 2016).
Harrell, F. E. Jr, Lee, K. L. & Mark, D. B. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 15, 361–387 https://doi.org/10.1002/(SICI)1097-0258(19960229)15 (1996).
Ramspek, C. L., Jager, K. J., Dekker, F. W., Zoccali, C. & van Diepen, M. External validation of prognostic models: what, why, how, when and where? Clin. Kidney J. 14, 49–58 https://doi.org/10.1093/ckj/sfaa188 (2021).
Degtiar, I. & Rose, S. A review of generalizability and transportability. Annu. Rev. Stat. Appl. 10, 501–524 https://doi.org/10.1146/annurev-statistics-042522-103837 (2023).
Desai, A., Abdelhamid, M. & Padalkar, N. R. What is reproducibility in artificial intelligence and machine learning research? AI Mag. 46, e70004 https://doi.org/10.1002/aaai.70004 (2025).
Open Science Collaboration Estimating the reproducibility of psychological science. Science 349, aac4716 (2015).
Patil, P., Peng, R. D. & Leek, J. T. A statistical definition for reproducibility and replicability. Preprint at bioRxiv https://doi.org/10.1101/066803 (2016).
Heyard, R. et al. A scoping review on metrics to quantify reproducibility: a multitude of questions leads to a multitude of metrics. R. Soc. Open Sci. 12, 242076 https://doi.org/10.1098/rsos.242076 (2025).
Baker, M. Reproducibility crisis: blame it on the antibodies. Nature 521, 274 https://doi.org/10.1038/521274a (2015).
Baker, M. 1,500 scientists lift the lid on reproducibility. Nature 533, 452–454 https://doi.org/10.1038/533452a (2016).
Haven, T. L. & Ioannidis, J. P. A. Reproducibility failure in biomedical research: problems and solutions. Ann. Rev. Med. 77, 537–549 https://doi.org/10.1146/annurev-med-050124-050859 (2026).
Begley, C. G. & Ellis, L. M. Raise standards for preclinical cancer research. Nature 483, 531–533 (2012).
Errington, T. M. et al. Investigating the replicability of preclinical cancer biology. eLife 10, e71601 (2021).
Errington, T. M., Denis, A., Perfito, N., Iorns, E. & Nosek, B. A. Challenges for assessing replicability in preclinical cancer biology. eLife 10, e67995 (2021).
Horbach, S. P. & Halffman, W. The ghosts of HeLa: how cell line misidentification contaminates the scientific literature. PLoS ONE 12, e0186281 (2017).
Ayoubi, R. et al. Scaling of an antibody validation procedure enables quantification of antibody performance in major research applications. eLife 12, RP91645 https://doi.org/10.7554/eLife.91645 (2023).
Debray, T. P. et al. A new framework to enhance the interpretation of external validation studies of clinical prediction models. J. Clin. Epidemiol. 68, 279–289 (2015).
Collins, G. S. et al. TRIPOD+ AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ 385, q902 https://doi.org/10.1136/bmj-2023-078378 (2024).
Van Calster, B., Steyerberg, E. W., Wynants, L. & van Smeden, M. There is no such thing as a validated prediction model. BMC Med. 21, 70 https://doi.org/10.1186/s12916-023-02779-w (2023).
Van Calster, B. et al. A calibration hierarchy for risk models was defined: from utopia to empirical data. J. Clin. Epidemiol. 74, 167–176 (2016).
Vickers, A. J. & Elkin, E. B. Decision curve analysis: a novel method for evaluating prediction models. Med. Decis. Making 26, 565–574 https://doi.org/10.1177/0272989X06295361 (2006).
Wong, A. et al. External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients. JAMA Int. Med. 181, 1065–1070 (2021).
Wynants, L. et al. Prediction models for diagnosis and prognosis of COVID-19: systematic review and critical appraisal. BMJ 369, m1328 https://doi.org/10.1136/bmj.m1328 (2020).
Marchetti, M. A. et al. Prospective validation of dermoscopy-based open-source artificial intelligence for melanoma diagnosis (PROVE-AI study). NPJ Digit. Med. 6, 127 https://doi.org/10.1038/s41746-023-00872-1 (2023).
Chambers, C. D. & Tzavella, L. The past, present and future of registered reports. Nat. Hum. Behav. 6, 29–42 https://doi.org/10.1038/s41562-021-01193-7 (2022).
Cadwallader L., Morton L. & Hrynaszkiewicz I. Trends in preprint, data, and code sharing, 2019–2022. PLOS Blog https://theplosblog.plos.org/2023/04/open-science-indicators/ (2023).
Hardwicke, T. E. & Wagenmakers, E. -J. Reducing bias, increasing transparency and calibrating confidence with preregistration. Nat. Hum. Behav. 7, 15–26 https://doi.org/10.1038/s41562-022-01497-2 (2023).
Nature is expanding Registered Reports to all the fields in which we publish. Nature 653, 981 https://doi.org/10.1038/d41586-026-01629-y (2026).
Stodden, V., Seiler, J. & Ma, Z. An empirical analysis of journal policy effectiveness for computational reproducibility. Proc. Natl Acad. Sci. USA 115, 2584–2589 https://doi.org/10.1073/pnas.1708290115 (2018).
Gabelica, M., Bojčić, R. & Puljak, L. Many researchers were not compliant with their published data sharing statement: a mixed-methods study. J. Clin. Epidemiol. 150, 33–41 https://doi.org/10.1016/j.jclinepi.2022.05.019 (2022).
Xiong, X. & Cribben, I. The state of play of reproducibility in statistics: an empirical analysis. Am. Stat. 77, 115–126 (2023).
Han, H. Challenges of reproducible AI in biomedical data science. BMC Med. Genet. 18, 8 https://doi.org/10.1186/s12920-024-02072-6 (2025).
Siddiq, M. L., Islam-Gomes, A., Sekerak, N. & Santos J. C. S. Large language models for software engineering: a reproducibility crisis. Preprint at https://doi.org/10.48550/arXiv.2512.00651 (2025).
McDermott, M. B. et al. Reproducibility in machine learning for health research: still a ways to go. Sci. Transl. Med. 13, eabb1655 (2021).
Gundersen, O. E. & Kjensmo, S. State of the art: reproducibility in artificial intelligence. In Proc. AAAI Conference on Artificial Intelligence Vol. 32 https://doi.org/10.1609/aaai.v32i1.11503 (AAAI, 2018).
Kapoor, S. & Narayanan, A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns 4, 100804 https://doi.org/10.1016/j.patter.2023.100804 (2023).
Singh, P. P. & Benayoun, B. A. Considerations for reproducible omics in aging research. Nat. Aging 3, 921–930 https://doi.org/10.1038/s43587-023-00448-4 (2023).
Freedman, L. P., Cockburn, I. M. & Simcoe, T. S. The economics of reproducibility in preclinical research. PLoS Biol. 13, e1002165 (2015).
Prinz, F., Schlage, T. & Asadullah, K. Believe it or not: how much can we rely on published data on potential drug targets? Nat. Rev. Drug Discov. 10, 712 https://doi.org/10.1038/nrd3439-c1 (2011).
Academy of Medical Sciences. Reproducibility and Reliability of Biomedical Research: Improving Research Practice https://acmedsci.ac.uk/policy/policy-projects/reproducibility-and-reliability-of-biomedical-research (2015).
Redman, B. K. & Caplan, A. L. Limited reproducibility of research findings: implications for the welfare of research participants and considerations for institutional review board. IRB Ethics Hum. Res. 38, 4 (2016).
Wang, S. V., Sreedhara, S. K., Schneeweiss, S. & REPEAT Initiative. Reproducibility of real-world evidence studies using clinical practice data to inform regulatory and coverage decisions. Nat. Commun. https://doi.org/10.1038/s41467-022-32310-3 (2022).
Wang, S. V., Sreedhara, S. K., Bessette, L. G. & Schneeweiss, S. Understanding variation in the results of real-world evidence studies that seem to address the same question. J. Clin. Epidemiol. 151, 161–170 https://doi.org/10.1016/j.jclinepi.2022.08.012 (2022).
European Medicines Agency. HMA-EMA Catalogues of Real-World Data Sources and Study Protocols https://catalogues.ema.europa.eu/legal/ (2026).
National Institute for Health and Care Research. PROSPERO Registry for Systematic Reviews https://www.crd.york.ac.uk/prospero/ (2026).
Glasziou, P. et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 383, 267–276 (2014).
Begg, C. et al. Improving the quality of reporting of randomized controlled trials: the CONSORT statement. JAMA 276, 637–639 https://doi.org/10.1001/jama.1996.03540080059030 (1996).
Moher, D., Schulz, K. F. & Altman, D. G. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials. Lancet 357, 1191–1194 https://doi.org/10.1016/S0140-6736(00)04337-3 (2001).
Moher, D. et al. CONSORT 2010 explanation and elaboration: updated guidelines for reporting parallel group randomised trials. BMJ 340, c869 (2010).
Hopewell S., et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ 389 https://doi.org/10.1136/bmj-2024-081123 2025
Turner, L. et al. Consolidated standards of reporting trials (CONSORT) and the completeness of reporting of randomised controlled trials (RCTs) published in medical journals. Cochrane Database Syst. Rev. 11, MR000030 (2012).
Hopewell, S. et al. CONSORT for reporting randomised trials in journal and conference abstracts. Lancet 371, 281–283 (2008).
Campbell, M. K., Piaggio, G., Elbourne, D. R. & Altman, D. G. Consort 2010 statement: extension to cluster randomised trials. BMJ 345, e5661 (2012).
Zwarenstein, M. et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ 337, a2390 (2008).
Calvert, M. et al. Reporting of patient-reported outcomes in randomized trials: the CONSORT PRO extension. JAMA 309, 814–822 (2013).
Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C. & Altman, D. G. Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles. JAMA 291, 2457–2465 (2004).
Chan, A.-W. et al. SPIRIT 2013 statement: defining standard protocol items for clinical trials. Ann. Intern. Med. 158, 200–207 (2013).
Von Elm, E. et al. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. Lancet 370, 1453–1457 (2007).
Benchimol, E. I. et al. The REporting of studies Conducted using Observational Routinely-collected health Data (RECORD) statement. PLoS Med. 12, e1001885 (2015).
Langan, S. M. et al. The reporting of studies conducted using observational routinely collected health data statement for pharmacoepidemiology (RECORD-PE). BMJ 363, k3532 https://doi.org/10.1136/bmj.k3532 (2018).
Wang, S. V. et al. Reporting to improve reproducibility and facilitate validity assessment for healthcare database studies V1.0. Value Health 20, 1009–1022 https://doi.org/10.1016/j.jval.2017.08.3018 (2017).
Wang, S. V. et al. HARmonized Protocol Template to Enhance Reproducibility of hypothesis evaluating real-world evidence studies on treatment effects: A good practices report of a joint ISPE/ISPOR task force. Pharmacoepidemiol. Drug Saf. 32, 44–55 https://doi.org/10.1002/pds.5507 (2023).
Moher, D., Liberati, A., Tetzlaff, J. & Altman, D. G. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ 339, b2535 (2009).
Collins, G. S., Reitsma, J. B., Altman, D. G. & Moons, K. G. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 102, 148–158 (2015).
Liu, X. et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit. Health 2, e537–e548 (2020).
Rivera, S. C. et al. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit. Health 2, e549–e560 (2020).
Mongan, J., Moy, L. Jr. & Kahn, C. E. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol.Artif. Intell. 2, e200029 (2020).
Simera, I. et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR Network. BMC Med. 8, 24 (2010).
Moher, D., Schulz, K. F., Simera, I. & Altman, D. G. Guidance for developers of health research reporting guidelines. PLoS Med. 7, e1000217 (2010).
Heus, P. et al. Poor reporting of multivariable prediction model studies: towards a targeted implementation strategy of the TRIPOD statement. BMC Med. 16, 120 (2018).
Collins, G. S. et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open 11, e048008 (2021).
Dhiman, P. et al. Risk of bias of prognostic models developed using machine learning: a systematic review in oncology. Diagn. Progn. Res. 6, 13 (2022).
Kilkenny, C., Browne, W. J., Cuthill, I. C., Emerson, M. & Altman, D. G. Improving bioscience research reporting: the ARRIVE guidelines for reporting animal research. Osteoarthritis Cartilage 20, 256–260 (2012).
Landis, S. C. et al. A call for transparent reporting to optimize the predictive value of preclinical research. Nature 490, 187–191 (2012).
Ram, K. Git can facilitate greater reproducibility and increased transparency in science. Source Code Biol. Med. 8, 7 (2013).
Fenner, M. et al. A data citation roadmap for scholarly data repositories. Sci. Data 6, 28 (2019).
Hrynaszkiewicz, I. & Cockerill, M. J. Open by default: a proposed copyright license and waiver agreement for open access research and data in peer-reviewed journals. BMC Res. Notes 5, 494 (2012).
Piwowar, H. A., Vision, T. J. & Whitlock, M. C. Data archiving is a good investment. Nature 473, 285 (2011).
Boettiger, C. An introduction to Docker for reproducible research. ACM SIGOPS Oper. Syst. Rev. 49, 71–79 (2015).
Project, J. et al. Binder 2.0—Reproducible, interactive, sharable environments for science at scale. In Proc. 17th Python in Science Conference (eds Akici, F. et al.) 113–120 (SciPy, 2018).
Perkel, J. M. Why Jupyter is data scientists’ computational notebook of choice. Nature 563, 145–147 (2018).
Köster, J. & Rahmann, S. Snakemake—a scalable bioinformatics workflow engine. Bioinformatics 28, 2520–2522 (2012).
Di Tommaso, P. et al. Nextflow enables reproducible computational workflows. Nat. Biotechnol. 35, 316–319 (2017).
Wilson, G. et al. Best practices for scientific computing. PLoS Biol. 12, e1001745 (2014).
Wilson, G. et al. Good enough practices in scientific computing. PLoS Comput. Biol. 13, e1005510 (2017).
Merkel, D. Docker: lightweight linux containers for consistent development and deployment. Linux J. 239, 2 (2014).
Wilkinson, M. D. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci. Data 3, 160018 (2016).
Price, W. N. & Cohen, I. G. Privacy in the age of medical big data. Nat. Med. 25, 37–43 (2019).
Evans, B. J. Much ado about data ownership. Harv. JL Tech. 25, 69 (2011).
Howe, D. et al. The future of biocuration. Nature 455, 47–50 (2008).
Hodge Jr, J. G., Gostin, L. O. & Jacobson, P. D. Legal issues concerning electronic health information: privacy, quality, and liability. JAMA 282, 1466–1471 (1999).
Benitez, K. & Malin, B. Evaluating re-identification risks with respect to the HIPAA privacy rule. J. Am. Med. Inf. Assoc. 17, 169–177 (2010).
Weissgerber, T. L. et al. Understanding the provenance and quality of methods is essential for responsible reuse of FAIR data. Nat. Med. 30, 1220–1221 https://doi.org/10.1038/s41591-024-02879-x (2024).
Nosek, B. A. et al. Promoting an open research culture. Science 348, 1422–1425 (2015).
Munafò, M. R. et al. A manifesto for reproducible science. Nat. Hum. Behav. 1, 0021 (2017).
Nosek, B. A., Spies, J. R. & Motyl, M. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspect. Psychol. Sci. 7, 615–631 (2012).
Moher, D. et al. Increasing value and reducing waste in biomedical research: who’s listening? Lancet 387, 1573–1586 (2016).
Ioannidis, J. P. et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 383, 166–175 (2014).
Bossuyt et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. Radiology 277, 826–832 (2015).
Cohen, J. F. et al. STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration. BMJ Open 6, e012799 (2016).
Schulz, K. F., Altman, D. G. & Moher, D. CONSORT 2010 statement: updated guidelines for reporting parallel group randomised trials. BMC Med. 8, 18 (2010).
Stodden, V. et al. Enhancing reproducibility for computational methods. Science 354, 1240–1241 (2016).
Peng, R. D. Reproducible research in computational science. Science 334, 1226–1227 (2011).
Laakso, M. et al. The development of open access journal publishing from 1993 to 2009. PLoS ONE 6, e20961 (2011).
Piwowar, H. et al. The state of OA: a large-scale analysis of the prevalence and impact of Open Access articles. PeerJ 6, e4375 (2018).
Björk, B. -C. & Solomon, D. Open access versus subscription journals: a comparison of scientific impact. BMC Med. 10, 73 (2012).
Gargouri, Y. et al. Self-selected or mandated, open access increases citation impact for higher quality research. PLoS ONE 5, e13636 (2010).
McCabe, M. & Snyder, C. M. The economics of open-access journals. SSRN https://doi.org/10.2139/ssrn.914525 (2004).
Percie du Sert, N. et al. The ARRIVE guidelines 2.0: updated guidelines for reporting animal research. J. Cereb. Blood Flow Metab. 40, 1769–1777 (2020).
Page, M. J. et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71 (2021).
Vandenbroucke, J. P. et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Int. J. Surg. 12, 1500–1524 (2014).
Vilhuber, L., Son, H. H., Welch, M., Wasser, D. N. & Darisse, M. Teaching for large-scale reproducibility verification. J. Stat. Data Sci. Educ. 30, 274–281 (2022).
Konkol, M., Kray, C. & Pfeiffer, M. Computational reproducibility in geoscientific papers: insights from a series of studies with geoscientists and a reproduction study. Int. J. Geogr. Inf. Sci. 33, 408–429 (2019).
National Institutes of Health. Final NIH Policy for Data Management and Sharing. NOT-OD-21-013 https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html (2026).
National Science Foundation. Proposal and Award Policies and Procedures Guide https://www.nsf.gov/policies/pappg/ (2026).
Wellcome Trust. Data, Software and Materials Management and Sharing Policy https://wellcome.org/research-funding/guidance/policies-grant-conditions/data-software-materials-management-and-sharing-policy/ (2026).
European Commission. Guidelines on FAIR Data Management in Horizon 2020. H2020 Programme https://www.kowi.de/Portaldata/2/Resources/horizon2020/h2020-guide-data-mgt.pdf (2016).
Vines, T. H. et al. Mandated data archiving greatly improves access to research data. FASEB J 27, 1304–1308 (2013).
Borgman, C. L. Big Data, Little Data, no Data: Scholarship in the Networked World (MIT Press, 2017).
Lynch, C. How do your data grow? Nature 455, 28–29 (2008).
Bouwmeester, W. et al. Reporting and methods in clinical prediction research: a systematic review. PLoS Med. 9, e1001221 (2012).
Michener, W. K. Ten simple rules for creating a good data management plan. PLoS Comput. Biol. 11, e1004525 (2015).
Federer, L. M., Lu, Y. -L., Joubert, D. J., Welsh, J. & Brandys, B. Biomedical data sharing and reuse: attitudes and practices of clinical and scientific research staff. PLoS ONE 10, e0129506 (2015).
Michener, W. K. & Jones, M. B. Ecoinformatics: supporting ecology as a data-intensive science. Trends Ecol. Evol. 27, 85–93 (2012).
Roche, D. G., Kruuk, L. E., Lanfear, R. & Binning, S. A. Public data archiving in ecology and evolution: how well are we doing?. PLoS Biol. 13, e1002295 (2015).
Tenopir, C. et al. Data sharing by scientists: practices and perceptions. PLoS ONE 6, e21101 (2011).
Resnik, D. B., Rasmussen, L. M. & Kissling, G. E. An international study of research misconduct policies. Account. Res. 22, 249–266 (2015).
Emanuel, E. J., Wendler, D. & Grady, C. An ethical framework for biomedical research. In The Oxford Textbook of Clinical Research Ethics (eds Emanuel, E. J. et al.) 123–135 (Oxford Univ. Press, 2008).
Sarafoglou, A., Hoogeveen, S., Matzke, D. & Wagenmakers, E. -J. Teaching good research practices: protocol of a research master course. Psychol. Learn. Teach. 19, 46–59 (2020).
Pownall, M. et al. Teaching open and reproducible scholarship: a critical review of the evidence base for current pedagogical methods and their outcomes. R. Soc. Open Sci. 10, 221255 (2023).
Sánchez-Caballé, A., Gisbert Cervera M & Esteve-Mon, F. M. Integrating digital competence in higher education curricula: an institutional analysis. Educar 57, 0241–0258 (2021).
Ziemann, M., Poulain, P. & Bora, A. The five pillars of computational reproducibility: bioinformatics and beyond. Brief. Bioinform. 24, bbad375 https://doi.org/10.1093/bib/bbad375 (2023).
Whyte, A. & Tedds, J. Making the Case for Research Data Management (Digital Curation Centre, 2011).
Katz, D. S. et al. Recognizing the value of software: a software citation guide. F1000Res. 9, 1257 (2021).
Borgman, C. L., Wallis, J. C. & Enyedy, N. Little science confronts the data deluge: habitat ecology, embedded sensor networks, and digital libraries. Int. J. Digit. Libr. 7, 17–30 (2007).
Park, H. & Wolfram, D. Research software citation in the Data Citation Index: current practices and implications for research software sharing and reuse. J.Informetr. 13, 574–582 (2019).
Tenopir, C., Sandusky, R. J., Allard, S. & Birch, B. Research data management services in academic research libraries and perceptions of librarians. Libr. Inf. Sci. Res. 36, 84–90 (2014).
Cox, A. M., Kennan, M. A., Lyon, L. & Pinfield, S. Developments in research data management in academic libraries: Towards an understanding of research data service maturity. J. Assoc. Inform. Sci. Technol. 68, 2182–2200 (2017).
Pineau, J. et al. Improving reproducibility in machine learning research (a report from the neurips 2019 reproducibility program). J. Mach. Learn. Res. 22, 1–20 (2021).
Button, K. S., Chambers, C. D., Lawrence, N. & Munafò, M. R. Grassroots training for reproducible science: a consortium-based approach to the empirical dissertation. Psychol. Learn. Teach. 19, 77–90 (2020).
National Information Standards Organization. NISO RP-31-2021, Reproducibility Badging and Definitions https://www.niso.org/publications/rp-31-2021-badging/ (2021).
Association for Computing Machinery. Artifact Review and Badging. Version 1.1 https://www.acm.org/publications/policies/artifact-review-and-badging-current/ (2020).
McKiernan, E. C. et al. Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations. eLife 8, e47338 (2019).
Moher, D. et al. Assessing scientists for hiring, promotion, and tenure. PLoS Biol. 16, e2004089 (2018).
Allen, C. & Mehler, D. M. Open science challenges, benefits and tips in early career and beyond. PLoS Biol. 17, e3000246 (2019).
McKiernan, E. C. et al. How open science helps researchers succeed. eLife 5, e16800 (2016).
Munafò, M. R. & Davey Smith, G. Robust research needs many lines of evidence. Nature 553, 399–401 (2018).
Fanelli, D. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4, e5738 (2009).
Ioannidis, J. P., Munafo, M. R., Fusar-Poli, P., Nosek, B. A. & David, S. P. Publication and other reporting biases in cognitive sciences: detection, prevalence, and prevention. Trends Cogn. Sci. 18, 235–241 (2014).
Nosek, B. A., Ebersole, C. R., DeHaven, A. C. & Mellor, D. T. The preregistration revolution. Proc. Natl Acad. Sci. USA 115, 2600–2606 (2018).
Franco, A., Malhotra, N. & Simonovits, G. Publication bias in the social sciences: Unlocking the file drawer. Science 345, 1502–1505 (2014).
Schmidt, S. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Rev. Gen. Psychol. 13, 90–100 (2009).
Peikert, A. & Brandmaier, A. A reproducible data analysis workflow with R Markdown, Git, Make, and Docker. Quant. Comput. Methods Behav. Sci. 1, e3763 (2021).
Marwick, B., Boettiger, C. & Mullen, L. Packaging data analytical work reproducibly using R (and friends). Am. Stat. 72, 80–88 (2018).
Gentleman, R. & Temple Lang, D. Statistical analyses and reproducible research. J. Comput. Graph. Stat. 16, 1–23 (2007).
Gorgolewski, K. J. & Poldrack, R. A. A practical guide for improving transparency and reproducibility in neuroimaging research. PLoS Biol. 14, e1002506 (2016).
Barnes, N. Publish your computer code: it is good enough. Nature 467, 753–753 (2010).
Lee, B. D. Ten simple rules for documenting scientific software. PLoS Comput. Biol. 14, e1006561 (2018).
Kohrs, F. E. et al. Eleven strategies for making reproducible research and open science training the norm at research institutions. eLife 12, e89736 (2023).
Lynch, C. Stewardship in the ‘Age of Algorithms’. First Monday 22, 12 (2017).
Berman, F. & Cerf, V. Who will pay for public access to research data?. Science 341, 616–617 (2013).
The National Academies of Sciences, Engineering, and Medicine Global Affairs Committee on Responsible Science. Fostering Integrity in Research (National Academies Press, 2017).
Federer, L. M. et al. Data sharing in PLoS ONE: an analysis of data availability statements. PLoS ONE 13, e0194768 (2018).
Houtkoop, B. L. et al. Data sharing in psychology: a survey on barriers and preconditions. Adv. Methods Pract. Psychol. Sci. 1, 70–85 (2018).
Asendorpf, J. B. et al. Recommendations for increasing replicability in psychology. Eur. J. Pers. 27, 108–119 (2013).
Wagenmakers, E. -J., Wetzels, R., Borsboom, D., van der Maas, H. L. & Kievit, R. A. An agenda for purely confirmatory research. Perspect. Psychol. Sci. 7, 632–638 (2012).
Hardwicke, T. E. et al. Data availability, reusability, and analytic reproducibility: evaluating the impact of a mandatory open data policy at the journal Cognition. R. Soc. Open Sci. 5, 180448 (2018).
Hardwicke, T. E. et al. An empirical assessment of transparency and reproducibility-related research practices in the social sciences (2014–2017). R. Soc. Open Sci. 7, 190806 (2020).
Sherry, C. E. et al. Assessment of transparent and reproducible research practices in the psychiatry literature. Gen. Psychiatr. 33, e100149 (2020).
Scheel, A. M., Tiokhin, L., Isager, P. M. & Lakens, D. Why hypothesis testers should spend less time testing hypotheses. Perspect. Psychol. Sci. 16, 744–755 (2021).
Nosek, B. A. et al. Replicability, robustness, and reproducibility in psychological science. Annu. Rev. Psychol. 73, 719–748 https://doi.org/10.1146/annurev-psych-020821-114157 (2022).
Mulugeta, L. et al. Credibility, replicability, and reproducibility in simulation for biomedicine and clinical applications in neuroscience. Front. Neuroinform. 12, 18 (2018).
Kahn, M. G. et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. Egems 4, 1244 (2016).
Wedderkopp, N. & Rutz, E. Scientific integrity and transparency in academic writing: the foundation of credible science. Children 11, 1191 https://doi.org/10.3390/children11101191 (2024).
Ivie, P. & Thain, D. Reproducibility in scientific computing. ACM Comput. Surv. 51, 1–36 https://doi.org/10.1145/3186266 (2018).
Van Calster, B., McLernon, D. J., Van Smeden, M., Wynants, L. & Steyerberg, E. W. Calibration: the Achilles heel of predictive analytics. BMC Med. 17, 230 https://doi.org/10.1186/s12916-019-1466-7 (2019).
O’Sullivan, J. W. & Ioannidis, J. P. A. Reproducibility in the UK biobank of genome-wide significant signals discovered in earlier genome-wide association studies. Sci. Rep. 11, 18625 https://doi.org/10.1038/s41598-021-97896-y (2021).
Collins, G. S. & Altman, D. G. Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK2. BMJ 344, e4181 https://doi.org/10.1136/bmj.e4181 (2012).
Hippisley-Cox, J. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ 357, j2099 https://doi.org/10.1136/bmj.j2099 (2017).
Button, K. S. et al. Power failure: why small sample size undermines the reliability of neuroscience. Nat. Rev. Neurosci. 14, 365–376 (2013).
Button, K. S. et al. Confidence and precision increase with high statistical power. Nat. Rev. Neurosci. 14, 585 (2013).
Clayton, J. A. & Collins, F. S. Policy: NIH to balance sex in cell and animal studies. Nature 509, 282–283 (2014).
Voelkl, B., Vogt, L., Sena, E. S. & Würbel, H. Reproducibility of preclinical animal research improves with heterogeneity of study samples. PLoS Biol. 16, e2003693 (2018).
Ioannidis, J. P. Why most published research findings are false. PLoS Med. 2, e124 (2005).
Simmons, J. P., Nelson, L. D. & Simonsohn, U. False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychol. Sci. 22, 1359–1366 (2011).
Kerr, N. L. HARKing: hypothesizing after the results are known. Pers. Soc. Psychol. Rev. 2, 196–217 (1998).
Price, W. & Nicholson, I. Black-box medicine. Harv. JL Tech. 28, 419 (2014).
Bradbury, A. & Plückthun, A. Reproducibility: standardize antibodies used in research. Nature 518, 27–29 (2015).
American Type Culture Collection Standards Development Organization Workgroup. Cell line misidentification: the beginning of the end. Nat. Rev. Cancer 10, 441–448 https://doi.org/10.1038/nrc2852 (2010).
Errington, T. M. et al. An open investigation of the reproducibility of cancer biology research. eLife 3, e04333 (2014).
Tenopir, C. et al. Changes in data sharing and data reuse practices and perceptions among scientists worldwide. PLoS ONE 10, e0134826 (2015).
Grant, S. et al. TOP 2025: an update to the transparency and openness promotion guidelines. Res. Integr. Peer Rev. 11, 40 https://doi.org/10.1186/s41073-026-00223-0 (2026).
Sandve, G. K., Nekrutenko, A., Taylor, J. & Hovig, E. Ten simple rules for reproducible computational research. PLoS Comput. Biol. 9, e1003285 (2013).
Taschuk, M. & Wilson, G. Ten simple rules for making research software more robust. PLoS Comput. Biol. 13, e1005412 (2017).
Stodden, V. Reproducing statistical results. Annu. Rev. Stat. Appl. 2, 1–19 (2015).
National Institutes of Health. NIH Releases Strategic Plan for Data Science https://www.nih.gov/news-events/news-releases/nih-releases-strategic-plan-data-science/ (2018).
Open Science Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science. Perspect. Psychol. Sci. 7, 657–660 (2012).
