Common errors in research code and how to catch them
Most bugs in research code are not exotic. They are the same handful of mistakes, repeated across fields. Knowing what they look like makes them much easier to catch before a reviewer, or a reader, does.
- A small set of error types causes a large share of trouble: data auto-conversion, indexing, units, randomness, missing values, paths and float comparisons.
- Most of them fail silently. The code runs, the numbers look plausible, and only a deliberate check reveals the problem.
- Cheap checks catch most of them: assertions on data shape and range, fixed seeds, explicit missing value handling and a few tests.
The bugs that end up in published papers are rarely clever. They are ordinary mistakes that produced output which looked reasonable. Nobody noticed, because the code did not crash and the numbers were in the expected range.
This post collects the errors we see most often when reviewing research code, explains why each one slips through, and gives a concrete check for each. None of these checks needs a software engineering background. Most take a few lines.
Why do research code errors go unnoticed?
Because most of them fail silently. A crash gets fixed. A wrong number that looks plausible gets published. Research code also tends to be run by its author only, on one dataset, with no independent expectation of what the output should be.
The context makes it worse. Analyses evolve over months, scripts get copied between projects, and a step that was correct for one dataset is quietly wrong for the next. A column changes meaning, a file format gains a header row, a collaborator sends data in different units. Each change is small, and none of them announces itself.
The fix is not to write perfect code. It is to add checks that turn silent failures into loud ones. An assertion that fails is a gift: it tells you exactly where your assumption broke.
Can a spreadsheet really change your data?
Yes, and this is one of the best documented examples. Spreadsheet programs try to be helpful by converting text that looks like a date or a number. Gene symbols such as SEPT2 or MARCH1 were turned into dates when files were opened and saved. Identifiers that look like scientific notation, such as some clone IDs containing an E followed by digits, were turned into floating point numbers.
The problem was described in the bioinformatics literature in 2004, and a later study in 2016 found it was still widespread in supplementary files of published papers. In 2020 the HUGO Gene Nomenclature Committee renamed several affected genes, for example SEPT1 became SEPTIN1 and MARCH1 became MARCHF1, partly to avoid the issue. Spreadsheet vendors have since added options to turn some conversions off, but old files and default settings are still around.
The same pattern applies outside genomics. Leading zeros vanish from postcodes and IDs. Long numeric identifiers lose precision. Text such as "1-2" becomes a date.
Check: treat spreadsheets as a viewing tool, not a step in the pipeline. Read raw files with explicit column types, and compare a few identifiers against the source after every import.
What are off-by-one errors and where do they appear?
An off-by-one error is a loop, slice or range that includes one element too many or too few. They are common in research code because languages disagree. R and Julia index from 1. Python and C index from 0. Python slices exclude the end point. Genomic coordinate systems differ too, with some formats using 0-based half open intervals and others 1-based closed intervals.
The result is a first or last item silently dropped, a window shifted by one position, or a time series aligned with the wrong lag.
Check: test edges. Run the code on a tiny input where you can count the answer by hand, and assert lengths explicitly.
window = series[start:end]
assert len(window) == end - start, "unexpected window length"
assert window[0] == series[start]
How do unit mix-ups get into analysis code?
Usually at boundaries: when data from two sources are combined, when a function written by someone else is reused, or when a constant is copied from a paper. The best known example outside research is NASA's Mars Climate Orbiter, lost in 1999 after one software component produced results in imperial units and another expected metric.
In analysis code the effects are quieter. Milligrams treated as grams, minutes as seconds, degrees as radians, percentages as proportions. If the factor is large, a plot looks absurd and someone notices. If the factor is small, nobody does.
Check: put units in variable and column names (dose_mg, time_s), convert once at input, and assert plausible ranges. Unit aware libraries exist in most languages for code where this matters a lot.
Why does unseeded randomness matter?
Because without a fixed seed, every run gives a slightly different answer, and nobody can reproduce yours exactly. Bootstraps, permutation tests, train and test splits, simulations, and many fitting algorithms all draw random numbers.
A missing seed also hides fragility. If a result only appears on some runs, that is important information. You will not see it if you ran the analysis once and kept the output.
Check: set seeds explicitly at the top of each script, and record them. Then deliberately rerun with a few different seeds to see whether conclusions hold.
import numpy as np
rng = np.random.default_rng(20260723) # use rng, not global state
sample = rng.choice(data, size=100, replace=True)
What goes wrong with missing values?
Different tools treat missing values differently, and many do so without telling you. In R, mean(x) returns NA if any value is missing, unless you pass na.rm = TRUE. In pandas, mean() skips missing values by default. Code ported between the two can change meaning without changing shape.
Readers can also invent missing values. By default, pandas treats the string "NA" as missing when reading a CSV. That is a problem if "NA" is a real value, such as a country code or a chemical symbol. Some models drop incomplete rows silently, so the sample size in the output differs from the one in the methods section.
Check: count missing values per column after every load and every join. Log the number of rows going into each model. Set missing value markers explicitly when reading files.
Why are hard-coded paths and constants a problem?
A path like C:/Users/anna/Desktop/final_v3/data.csv breaks the moment anyone else runs the script. That is annoying but visible. The quieter problem is a script that runs because it finds a different, older file at a similar path.
Hard-coded constants are similar. A threshold typed directly into a function, and changed in one place but not another, produces an analysis that no longer matches the methods.
Check: use paths relative to the project root, keep parameters in one config file, and print the full path and modification time of each input when the script starts. Our guide to organising a research project shows a layout that makes this easy.
Why should you not compare floats with ==?
Because floating point numbers are approximations. In most languages 0.1 + 0.2 == 0.3 is false, because the stored values differ in the last few bits. Code that tests equality on computed values may branch the wrong way on some inputs or some machines.
Check: compare with a tolerance. Use math.isclose or numpy.isclose in Python, all.equal in R, and pytest.approx in tests written with pytest.
A quick reference
| Error | Typical symptom | Check |
|---|---|---|
| Spreadsheet auto-conversion | Dates or numbers where identifiers should be | Read raw files with explicit types, spot check IDs |
| Off-by-one | First or last item missing, shifted windows | Tiny hand-checked input, assert lengths |
| Unit mix-up | Values off by a constant factor | Units in names, convert once, range asserts |
| Unseeded randomness | Results change between runs | Fixed, recorded seeds, rerun with others |
| Silent missing values | Sample size smaller than reported | Count missing values, log rows per model |
| Hard-coded paths | Runs only on one machine, or reads a stale file | Relative paths, config file, log inputs |
| Float equality | Branches or tests fail unpredictably | Compare with a tolerance |
What to do this week
- Pick one analysis script and add three assertions: row count after loading, missing values per key column, and the range of one measured variable.
- Search the codebase for absolute paths and the
==operator on computed numbers. - Set and record a random seed in every script that uses randomness.
- Write one small test on a hand-checked input. Our post on testing research code shows how to start.
If you review other people's code, the table above doubles as a shortlist of what to look for. It pairs well with our checklist for reviewing research code. The Carpentries lessons also cover several of these habits for beginners.
Questions
Which of these errors is most common?
We do not have reliable numbers across fields, and it varies by domain. Silent missing value handling and hard-coded paths come up very often in reviews, and spreadsheet conversion is well documented in genomics.
Do I need a full test suite to catch these?
No. A few assertions on data shape, range and missing values, plus one or two tests on small hand-checked inputs, catch a large share of these problems.
Is it safe to open data files in a spreadsheet at all?
Viewing is fine, but avoid saving files back from a spreadsheet as part of the pipeline. If you must, import columns as text and check identifiers afterwards.
Keep reading. More from the desk.
A practical checklist for reviewing research code
You agreed to review a paper and the code came with it. You do not need to audit every line. You need a plan that fits into a few hours and tells the editor something useful.
Running code review inside a research group
Code review is routine in software teams and rare in research labs. It does not need heavy process. A small, regular habit catches errors early and spreads skills across the group.
How to organise a research project folder so others can follow it
A research project that only its author can navigate is hard to review, hard to hand over and hard to reproduce. A simple, consistent folder layout fixes most of that, and it takes an afternoon to set up.