FAQ
This page consists of some frequently asked questions about the challenge.
Can I reproduce workshop papers from the listed conferences? Unfortunately no, as of now we are not accepting reports on workshop papers from the listed conferences. Please work on any accepted papers from the conference proceedings.
I want to reproduce paper(s) from conferences not listed in the challenge. Can I? We recommend you choose any paper(s) published in the 2023 calendar year from the top conferences and journals (NeurIPS, ICML, ICLR, ACL, EMNLP, ICCV, CVPR, TMLR, JMLR, TACL) to run your reproducibility study on.
I want to reproduce a papers from a previous year conference paper. Can I? While we recommended you to choose paper(s) to work on from the current calendar year, you can also choose to reproduce an older paper published in the same conferences.
I am a course instructor, how do I participate officially with my course? Many thanks for your participation! You can just drop us a mail (reproducibility.challenge@gmail.com) with details of your course, and we will list it on our website!
I am from industry, can I participate? Yes you are more than welcome to! Please consider sharing the word about the challenge to your peers in your company too!
Where can I get GPUs to run experiments? Check the Resources tab for more information.
Can I contact the authors? Yes! It is highly recommended to contact the authors of the paper you are reproducing, to clarify doubts and implementation details.
How do I contact the authors? You can send the authors mail directly to initiate a discussion. The contact details can be found on the paper, which is linked in the pdf of the paper which is available for each paper.
How do I get the code of the paper? You can either search the pdf of the paper for the code, or find the link to PapersWithCode page of the paper, which is usually updated with the publicly released code of the authors.
How much of the code am I allowed to use? There is no restriction on the extent of the original code you can use for the reproducibility effort.
Is the submission double blind? Yes, the report to be submitted should be double blind, according to TMLR’s submission policies. When submitting code for review, include your codebase in the Supplementary Materials, or link to an Anonymous Github URL.
What is the format of the paper? Please consult TMLR’s author guidelines.
This is super exciting, how can I help? Thanks for your interest in our challenge! You can help out by spreading the news. If you are a course-instructor you can help by enrolling your course in the challenge. You can also sign up to be a reviewer when we share the call for reviewing for the challenge! If you are a company you can help sponsor by providing compute resources. Please contact us at reproducibility.challenge@gmail.com to list your generous offer in the Resources section.