A practical checklist for reviewing research code

Peer reviewers are increasingly handed code along with a manuscript. Few were trained for it, and nobody has a free week to audit a pipeline. The good news is that a structured, limited review catches most of the problems that matter.

This checklist is designed for a reviewer with a few hours and general programming literacy. It is not a security audit or a software engineering review. It asks one question: does the code support the claims in the paper?

Before you start: can you review this at all?

Check the languages and tools. If the analysis is in a language you do not read, tell the editor early. That is not a failure. It helps the editor find a second reviewer for the code or split the work.

Agree a scope with yourself. A reasonable target is: confirm the code is available and documented, run it once, and inspect the steps behind the main results. Write that scope into your report.

1. Is the code actually available?

• Is there a code availability statement, and does it point to a specific, archived version, not just a repository homepage?
• Does the link work? Is the repository public, or was access provided for review?
• Is there a license file? Without one, others cannot legally reuse the code.
• Does the archived version match the version described in the paper?

If code is withheld, is the reason specific and credible? Is there a route for qualified readers to get access?

2. Can a stranger follow the instructions?

• Is there a README explaining what the code does and how to run it?
• Are dependencies listed with versions, for example in a lockfile, an environment file or a container definition?
• Is it clear which data are needed and where they come from?
• Is there a mapping from scripts to figures and tables? Something like "run make_fig2.py to produce Figure 2" saves a reviewer an hour.

Missing instructions are worth reporting even if you manage to work around them. The next reader will not have your patience.

3. Does it run?

Try once, in a clean environment. A container, a fresh virtual environment or a hosted service like Binder all work. Time box this step. If it does not run within a reasonable effort, stop and report what failed.

git clone https://example.org/lab/analysis.git
cd analysis
conda env create -f environment.yml
conda activate analysis
python run_all.py   # note runtime, errors, warnings

If the full analysis takes days or needs a cluster, ask whether the authors provide a reduced example, test data or cached intermediate results. A small run that exercises the same code paths is often enough.

4. Does the code match the methods section?

This is where most substantive problems hide. Pick the two or three claims the paper depends on most and trace them through the code.

• Are the statistical tests, models and parameters the ones described in the paper?
• Are exclusion criteria and filters applied as stated, and in the stated order?
• Are random seeds set where results depend on randomness?
• Are there hard coded values, such as thresholds, paths or constants, that are not mentioned in the methods?
• Do numbers produced by the code match the numbers in the tables and figures?

You do not need to read every line. Follow the data from input to the key result.

5. Is there evidence the code was checked?

• Are there tests? Even a few tests on the core calculations are a good sign.
• Do the tests pass?
• Is there any validation against known results, simulated data with a known answer, or an independent implementation?

Missing tests are not grounds for rejection on their own in most fields. But they are worth noting, especially for custom statistical code.

Red flags worth noting

• Manual steps between scripts, such as "then edit the spreadsheet by hand."
• Results that only reproduce with one specific, undocumented package version.
• Commented out blocks that look like alternative analyses, with no explanation of why they were dropped.
• Silent error handling that swallows failures.
• Credentials, private paths or personal data in the repository. Flag these to the editor privately rather than in a public report.

How should you write it up?

Keep the code review as a clear, separate section of your report. Structure it so the editor can act on it:

1. Scope. What you checked, in what environment, and how long you spent.
2. Blocking issues. Problems that undermine the main claims or prevent anyone from running the code.
3. Requests. Missing documentation, license, version pinning, tests.
4. Suggestions. Nice to have improvements that should not hold up publication.
5. What you could not check. Parts outside your expertise, steps that needed resources you did not have.

Be kind about style. Research code is often written under pressure by people who were never taught software practices. Comment on what affects correctness and reproducibility, not on variable naming taste.

What if the authors push back?

Sometimes authors respond that code review is out of scope, or that their field does not share code. Keep the conversation on the paper's claims. If a key result cannot be traced to the code, that is a question about the evidence, not about software style. Separate what is essential for the claims from what is good practice, and let the editor decide how strictly to apply journal policy. A short, specific request, such as "please add the script that produces Table 3 and pin the package versions," is much easier to act on than a general call for better code.

Where to learn more

Dedicated venues have built detailed review criteria that are worth borrowing. The Journal of Open Source Software publishes its reviewer checklist, and rOpenSci maintains a guide for reviewing R packages. Both are written for reusable software rather than one-off analyses, but many of their questions apply directly. For checking whether published results can be reproduced, ReScience C publishes replications of computational studies and shows what a full reproduction effort looks like.