Reproducibility, replicability and code: what the debate means for your scripts

Most researchers have heard that science has a reproducibility problem. Fewer agree on what the word means. In one paper it means rerunning the authors' code on their data. In another it means running a new experiment and finding the same effect. These are very different tests, and code plays a very different role in each.

This post untangles the terms and then looks at the practical question: what does the debate mean for the scripts on your laptop? Where does sharing code genuinely help, and where are the limits?

What is the difference between reproducibility and replicability?

In the most widely used definitions, reproducibility means getting consistent results using the same data, code and methods. Replicability means getting consistent results in a new study that collects its own data to answer the same question.

This is the usage adopted in the 2019 report from the US National Academies of Sciences, Engineering, and Medicine, Reproducibility and Replicability in Science. It is also close to how many methodologists in the life and social sciences use the words. Under these definitions, reproducibility is a check on the computation. Replicability is a check on the finding.

A third phrase you will see is computational reproducibility, which removes any doubt: the question is whether the same inputs and the same code produce the same outputs.

The distinction matters for how you read a failed check. If a reanalysis of the original data cannot reproduce the published numbers, the problem is in the computation or its description. If a new study fails to replicate, the original analysis may have been flawless, and the reasons lie in the design, the sample, chance or the effect itself. Mixing the two leads to unfair conclusions in both directions.

Why do the definitions vary between fields?

Because different communities started using the words independently, and some ended up with them the other way round. Parts of computer science historically used replicability for rerunning the same setup and reproducibility for independent confirmation. The ACM, for example, has used these terms for its artifact badges and later revised its wording to line up with other fields.

You do not need to win this argument. You do need to say what you mean. A sentence in a paper or README such as "the results in Tables 2 and 3 can be regenerated from the archived data and code" is clearer than any single word.

A helpful way to keep the ideas apart is a two by two grid, popularised by The Turing Way community handbook:

	Same data	Different data
Same analysis	Reproducible	Replicable
Different analysis	Robust	Generalisable

Each cell is a different question about a result. Code is central to the first, useful for the second and third, and only indirectly related to the fourth.

Where does shared code help most?

Shared code helps most with reproducibility in the narrow sense. If the data and code are available, anyone can check whether the reported numbers actually come out of the stated analysis. That catches a real class of problems:

• Transcription errors between output and manuscript.
• Analysis steps that differ from the methods section.
• Bugs that change the result, such as a wrong join, a dropped filter or a sign error.
• Results that depend on a specific software version or an unrecorded setting.

Without the code, a reader can only trust the description. With it, the description becomes testable. That is the core argument behind journal code policies, which we cover in why journals should require code.

Code also makes robustness checks much cheaper. If someone suspects a result depends on one exclusion rule or one model specification, they can change that line and rerun. Without code, they would have to rebuild the whole pipeline from a methods section, which rarely captures every choice.

Where does code not help?

Code cannot tell you whether a finding is true. A perfectly reproducible analysis can still rest on a biased sample, a confounded design, a measurement problem or a statistical choice that does not fit the question. Rerunning it will faithfully produce the same flawed answer every time.

Code also does not make a study replicate. Replication depends on whether the effect holds in new samples, settings and conditions. That is a question about the world, not about the script.

And code cannot recover what was never recorded. If a key step happened by hand in a spreadsheet, or a data cleaning decision lived only in someone's head, the shared scripts will have a gap exactly where the interesting judgement was made.

Reproducibility is a necessary floor, not a ceiling. It shows the numbers follow from the analysis. It says nothing about whether the analysis was the right one.

Is bit for bit identical output the goal?

Not usually. Identical output is a strong and useful property, but it is not always achievable, and chasing it can waste time.

Floating point arithmetic can give slightly different results on different hardware, compilers or library versions. Parallel code may sum numbers in a different order. Some algorithms are randomised by design. In these cases, the honest goal is results that agree within a stated tolerance, and conclusions that do not change.

Where exact reproduction is possible, aim for it. Fix random seeds, pin dependency versions and record the environment. The Reproducible Builds project shows how far software engineers have pushed bit for bit reproducibility for compiled software, and some of its thinking carries over. Where it is not possible, say so and give the tolerance. Our post on reproducible environments covers the tooling.

What should your scripts actually do?

A script that supports reproducibility makes it easy for someone else to go from raw inputs to the reported results with no hidden steps. In practice:

1. One entry point. A single command or script that runs the full analysis in order.
2. No manual steps. If something has to happen by hand, document it precisely, or better, script it.
3. Recorded environment. Dependencies with versions in an environment file, lockfile or container definition. Tools like Docker or Apptainer help when system libraries matter.
4. Seeds and settings in one place. Random seeds, thresholds and parameters at the top of a config file, not scattered through the code.
5. A map to the paper. Which script produces which figure and table.

A minimal top level script can be as simple as this:

#!/usr/bin/env bash
set -euo pipefail

python src/01_clean.py   data/raw/      data/clean/
python src/02_model.py   data/clean/    results/
python src/03_figures.py results/       figures/   # Figures 1-4

It is short, it runs in order and it fails loudly if a step breaks. That is most of what a reproducibility check needs.

How can code help with robustness and replication?

By making your analysis choices visible and easy to change. Robustness checks need someone to vary the decisions you made. Replication needs someone to apply the same analysis to new data.

• Separate data loading from analysis, so a replicator can plug in their own dataset with the same structure.
• Document the expected input format: columns, units, coding of categories.
• Expose key decisions as parameters, such as outlier rules or model terms, rather than burying them.
• Record decisions you considered and rejected, with a short reason.

None of this guarantees that a finding will hold up. It does mean that when someone tests it, they are testing your idea and not your ability to describe code in prose. ReScience C, a journal for replications of computational work, shows what careful reimplementation looks like in practice.

What to do this week

Take one current analysis and check it against the grid above:

• Write one sentence in the README stating which kind of check the repository supports, using words a reader from another field will understand.
• Find any manual step in the pipeline and either script it or document it exactly.
• Move seeds and key parameters into one config file.
• Add a table mapping scripts to figures and tables.
• Note one robustness check you would expect a sceptical reviewer to ask for, and make it a one line change.

Then ask a colleague to run it from a fresh clone. That is the quickest honest test of reproducibility you can get.