How journals check code: the main models

"Code available on request" used to be the norm. Today many journals ask for more, and a growing number check what they receive. But "the journal checks code" can mean very different things. At one end it means an editor confirms a link exists. At the other, a specialist downloads your materials and tries to reproduce every table before the paper is accepted.

This post describes the main models in general terms. Policies differ between journals and change over time, so always read the author guidelines of the journal you are targeting. The point here is to help you recognise which model you are dealing with and what it will ask of you.

What are the main models?

There are four broad approaches, and many journals combine more than one:

Model	What is checked	Who checks	When
Availability policy	That code is shared and described	Editorial office	Submission or acceptance
Reviewer code review	Code quality and fit with the paper	Peer reviewers	During review
Verification or reproducibility check	That code reproduces reported results	Data or reproducibility editor, or a contracted service	Usually after conditional acceptance
Software or replication journal	The software or replication is the publication	Reviewers, openly	The whole review

The models are not ranked from bad to good. They make different trade-offs between cost, speed and confidence.

What does an availability-only policy require?

It requires that code be shared and that the paper say where. It does not require anyone to run it. Typical requirements are a code availability statement, a link to a public repository or archive and, increasingly, a persistent identifier such as a DOI.

Policies vary in strength. Some encourage sharing. Some require it but accept "available on request". Stronger ones require a public archive at publication and ask authors to justify any exception. Our post on why journals should require code makes the case for the stronger end.

The value of this model is that it is cheap and scales to every paper. Its limit is obvious: a link can point to code that is incomplete, undocumented or simply does not run. Availability is necessary for checking, but it is not checking.

For authors, this is the easiest bar to meet. Archive a release, cite the DOI, and write a clear code availability statement that names the archive and version.

What happens when reviewers review the code?

Reviewers are asked, or allowed, to look at the code as part of normal peer review. Some journals add a question about code to the review form. Others invite a reviewer specifically for the software or statistics.

In practice the depth varies a lot. A reviewer might:

• check that the repository contains what the paper says it does;
• read the code that implements a key method and compare it with the description;
• try to install it and run an example;
• attempt to reproduce one figure or table.

The strength of this model is that reviewers know the field and can spot when code does something the text does not describe. The weakness is that reviewers are unpaid, often short of time, and not always comfortable with the language or tooling. Without guidance, code review can be skipped quietly. A shared checklist helps, such as our checklist for reviewing research code.

For authors, the practical lesson is to make the reviewer's first ten minutes easy. A README with installation steps, a small example that runs quickly and a map from scripts to figures make a real difference.

How do verification and reproducibility editors work?

A dedicated person or service tries to reproduce the paper's results from the submitted materials, separately from scientific peer review. This model is well established in parts of economics and political science, where some journals appoint a data editor or work with an external verification team. It is spreading to other fields.

The usual sequence looks like this:

1. The paper goes through normal peer review on its scientific merits.
2. After conditional acceptance, authors submit a replication package: code, data or data access instructions, and a README.
3. The verifier runs the package in a clean environment and compares the output with the manuscript.
4. Discrepancies go back to the authors, who fix them and resubmit.
5. Final acceptance waits until the results reproduce, or until remaining differences are explained.

This is the most thorough of the common models and the most expensive. It needs trained staff, computing resources and time. It also has limits. Verifiers usually check that the code produces the reported numbers, not that the analysis is the right one. Restricted data can make full verification impossible, in which case the check may be partial.

For authors, preparation matters more here than anywhere else. Expect to provide a single entry point that runs everything, pinned software versions, and expected run times. Containers or environment files with pinned versions make the verifier's job, and yours, much easier.

What are overlay journals like JOSS and ReScience C?

They are journals where the code is not supporting material but the thing being reviewed. Both run open review on GitHub and publish short articles that point to archived software.

The Journal of Open Source Software publishes short papers about research software. Reviewers work through a public checklist covering installation, functionality, documentation, tests and community guidelines. The software must be open source and archived at acceptance. We cover the process in detail in publishing in JOSS.

ReScience C publishes replications of computational studies. Authors reimplement a published model or analysis from the description and report whether they could reproduce the original results. Failed replications are publishable, which is part of the point.

These journals are sometimes described as overlay or companion venues because they sit alongside the traditional literature. A paper in a domain journal can cite a JOSS paper for the software it relies on. The software gets reviewed by people who care about software, and the research paper gets reviewed by people who care about the science.

What makes a good replication package?

One that a stranger can run without writing to you. Whatever model the journal uses, the same package serves every level of checking, so it is worth building once and well. A good package usually contains:

• a README that lists the steps in order, the software required and how long each step takes;
• one script or command that runs the whole analysis from raw inputs to final outputs;
• a clear mapping from each figure and table in the paper to the code that produces it;
• an environment file, lockfile or container definition with exact versions;
• the data, or precise instructions for obtaining restricted data, with a small sample if possible;
• a licence for the code and a persistent identifier for the archived version.

Test it yourself before you submit. Clone the archive into a fresh folder or a new machine and follow your own README. Most problems verifiers find, such as hard-coded paths, missing files or undocumented steps, show up in that first run.

Which model gives the most confidence?

For a single paper, independent verification gives the strongest assurance that the reported numbers come from the shared code. For software that many papers will use, a dedicated software review, as at JOSS, checks things verification does not, such as documentation, tests and usability. Reviewer code review sits in between and depends heavily on the reviewer.

Availability alone gives the least direct assurance, but it is the foundation for everything else. Without shared code, no later check is possible, by the journal or by readers.

Many journals combine models: a firm availability policy for all papers, reviewer guidance for code, and verification for a subset. Our journals overview is a starting point for comparing policies.

What should editors consider when choosing a model?

• Volume. Verification for every paper requires staff in proportion to submissions.
• Field norms. Where data are often restricted, a strict reproduction requirement may need exceptions.
• Reviewer pool. Asking for code review only works if reviewers have the skills and time.
• Timing. Checks after conditional acceptance avoid spending effort on papers that will be rejected.
• Clarity. Whatever the model, authors need to know what is required before they submit.

Starting small is reasonable. A clear availability policy with a required archive and identifier is a meaningful step on its own.

What can you do this week?

If you are an author, look up the code policy of the journal you plan to submit to and decide which model it uses. Then prepare for one level above it. If the journal only asks for availability, make the code runnable anyway. If you are an editor, write down in one paragraph what your journal currently checks and what it does not. That paragraph is often the most useful thing an author guideline can contain.