ML Reproducibility Challenge 2023

Welcome to the ML Reproducibility Challenge 2023 (MLRC 2023). This is the seventh edition of the event (v1, v2, v3, v4, v5, v6). The primary goal of this event is to encourage the publishing and sharing of scientific results that are reliable and reproducible. In support of this, the objective of this challenge is to investigate reproducibility of papers accepted for publication at top conferences by inviting members of the community at large to select a paper, and verify the empirical results and claims in the paper by reproducing the computational experiments, either via a new implementation or using code/data or other information provided by the authors.

Call For Papers

We invite contributions from academics, practitioners and industry researchers of the ML community to submit novel and insightful reproducibility studies. Please read our blog post regarding our retrospectives of running the challenge and the future roadmap. We are happy to announce the formal partnership with Transactions of Machine Learning Research (TMLR) journal. The challenge goes live on October 23, 2023.

We recommend you choose any paper(s) published in the 2023 calendar year from the top conferences and journals (NeurIPS, ICML, ICLR, ACL, EMNLP, ICCV, CVPR, TMLR, JMLR, TACL) to run your reproducibility study on.

In order for your paper to be submitted and presented at MLRC 2023, it first needs to be accepted and published at TMLR. While TMLR aims to follow a 2-months timeline to complete the review process of its regular submissions, this timeline is not guaranteed. If you haven’t already, we therefore recommend submitting your original paper to TMLR by February 16th, 2024, that is a little over 3 months in advance of the MLRC publication announcement date.

Key Dates

  • Challenge goes live: October 23, 2023
  • Deadline to share your intent to submit a TMLR paper to MLRC: February 16th, 2024 at the following form: https://forms.gle/JJ28rLwBSxMriyE89. This form requires that you provide a link to your TMLR submission. Once it gets accepted (if it isn’t already), you should then update the same form with your paper camera ready details. Your accepted TMLR paper will finally undergo a light AC review to verify MLRC compatibility.
  • We aim to announce the accepted papers by May 31st, 2024, pending decisions of all papers.

Contact Information