Resources

This page lists some useful resources which you can use for the challenge.

Reproducible Code

Code submission are mandatory for all submitted papers. We recommend that you:

• Publish your code in a repository (e.g. on GitHub, GitLab, BitBucket) and anonymize it according to our double blind guidelines.
• Document your code appropriately
• Have a README.md file which describes the exact steps to run your code. You can refer to the ML Code Completeness Checklist to write the README file and make sure your code submission is complete.
• See this blog post on best practices for reproducibility.

Compute Resources

• Google Colaboratory provides free GPU backed Jupyter Notebooks
• Instructors can apply for Google Cloud credits for their students.
• If you are a company that can offer cloud computing credits, please contact reproducibility.challenge@gmail.com

Suggested Readings

• Online Proceedings of MLRC
• Arvind Narayanan et al, 2023; Talk: Evaluating LLMs is a Minefield
• ACL 2022 Tutorial on “Towards Reproducible Machine Learning Research in Natural Language Processing”
• NAACL 2022 Reproducibility Track
• ML Reproducibility Checklist
• ML Code Completeness Checklist
• ML reproducibility tools and best practices
• Joelle Pineau’s Keynote talk on Reproducibility at NeurIPS 2018