Testing research code: where to start

Ask a room of researchers whether their code is tested and most will say no. Ask whether they check their results, and almost all will say yes. They plot the output, compare it with a previous run, look at whether the numbers are plausible. That is testing. It is just done by hand, once, and then forgotten.

Automated tests turn those checks into code that runs every time something changes. You do not need a testing philosophy or full coverage to benefit. This guide explains where to start, which kinds of tests pay off first in research, how to handle floating point numbers, and how to get tests running automatically.

Why test research code?

Tests tell you quickly when a change breaks a result. Without them, a bug introduced while tidying code can travel all the way into a paper, because nothing flags it.

Research code has particular risks. Analyses are revised for months. Several people edit the same scripts. Packages update underneath you. A small change in one function, such as a different default in a library call, can shift a number that ends up in a table. Tests are the cheapest way to notice that the same day rather than during review, or after publication.

Tests also document intent. A test that says "this function returns 0 for an empty input" tells the next reader something the code alone may not.

Where should I start if I have no tests at all?

Start with one regression test on the whole pipeline: run it on a small, fixed input and check that the output matches a saved, trusted result. This protects everything at once, even if it cannot tell you exactly what broke.

The steps are straightforward:

1. Make a small input dataset that runs in seconds. A few hundred rows is often enough.
2. Run your analysis on it and inspect the output carefully. When you are happy it is correct, save it as the reference result.
3. Write a test that reruns the analysis on the small input and compares it with the reference.
4. Commit the input, the reference output and the test.

import pandas as pd
from pandas.testing import assert_frame_equal
from myproject.pipeline import run_analysis

def test_pipeline_matches_reference():
    result = run_analysis("tests/data/small_input.csv")
    expected = pd.read_csv("tests/data/expected_summary.csv")
    assert_frame_equal(result, expected, check_exact=False, rtol=1e-6)

When the test fails, you have two possibilities. Either you introduced a bug, or you changed the method on purpose. In the second case, inspect the new output, update the reference, and say why in the commit message. That makes every change to a result deliberate and recorded.

Which functions deserve unit tests?

Write unit tests for the functions that do the science. These are the places where a small error changes a conclusion.

• Transformations and conversions: units, coordinate systems, time zones, log scales.
• Statistical steps: normalisation, weighting, outlier rules, any formula copied from a paper.
• Data cleaning rules: what gets dropped, merged or imputed, and when.
• Edge cases: empty inputs, a single row, missing values, zeros where division happens.

A good unit test uses an input where you know the answer without running the code. That might come from a hand calculation, a textbook example, a published worked example, or a mathematical property. Properties are especially useful. A probability distribution should sum to one. Rotating a vector by 360 degrees should return the original. Converting a value and converting it back should give the starting value.

You can skip tests for code that only plots, formats or moves files around. Check those by eye.

How do I compare floating point numbers?

Compare floats with a tolerance, never with exact equality. Floating point arithmetic rounds at almost every step, so two mathematically equal calculations can differ in the last digits.

>>> 0.1 + 0.2 == 0.3
False

Every common testing tool offers tolerant comparison:

Language	Tolerant comparison
Python with pytest	assert x == pytest.approx(0.3, rel=1e-9)
Python with NumPy	numpy.testing.assert_allclose(a, b, rtol=1e-7)
R with testthat	expect_equal(x, 0.3, tolerance = 1e-8)
Julia	@test x ≈ 0.3 atol=1e-10

Choosing the tolerance is a scientific decision, not a technical one. Ask how much difference would matter for your result. A relative tolerance suits values across many orders of magnitude. An absolute tolerance suits values near zero, where relative differences become meaningless. If your code uses random numbers, fix the seed in tests so the output is repeatable, or test statistical properties over many draws with a tolerance that reflects the expected spread.

Results can also differ slightly between machines, operating systems and library versions. If a regression test fails only on a colleague's laptop by a tiny margin, the tolerance may be too strict. If it fails by a lot, look harder.

Which tools should I use?

Use the standard testing framework for your language. They are mature, well documented and familiar to anyone who reads your code.

• Python: pytest. Put files called test_*.py in a tests/ folder, write functions starting with test_ that use plain assert, and run pytest from the project root.
• R: testthat, the standard choice for R packages. Tests live in tests/testthat/ and use functions such as expect_equal() and expect_error(). It also works for analysis projects that are not packages.
• Julia: the built-in Test standard library with @test and @testset.
• MATLAB: the built-in unit testing framework, with script-based tests as the simplest entry point.

A minimal testthat example looks like this:

test_that("weights sum to one", {
  w <- compute_weights(c(2, 3, 5))
  expect_equal(sum(w), 1, tolerance = 1e-12)
})

Structure matters more than tool choice. Tests need to import your functions, so code that lives only inside one long script or notebook is hard to test. Moving logic into functions in a module is often the first real step. We cover that in making Jupyter notebooks reproducible.

What is continuous integration and do I need it?

Continuous integration, or CI, runs your tests automatically on a clean machine every time you push to the repository. It is worth setting up once you have a few tests, because it catches the problems you would not catch on your own laptop.

A clean machine is the point. It installs your dependencies from your environment file, so it fails if you forgot to list a package. It runs every test, not just the ones you remembered. It also shows collaborators and reviewers a visible pass or fail for each change.

GitHub Actions and GitLab CI/CD are the common choices, built into the platforms. A basic configuration is a short text file in the repository that says which language version to install, how to install dependencies, and which command runs the tests. Both platforms provide starter templates for Python and R. Keep CI fast by running tests on small data. Full analyses that take hours belong elsewhere.

If you do not use Git yet, start there first. Our guide to version control for researchers covers the basics.

How many tests are enough?

Enough to make you confident that a change has not quietly broken a result. There is no fixed number, and coverage percentages are a weak guide for research code.

A practical target for a typical paper:

• one end-to-end regression test on a small dataset;
• unit tests for every function that implements a formula or a cleaning rule;
• a test for every bug you fix, so the same bug cannot return unnoticed.

The last habit is the most valuable over time. Each bug you find tells you where the code is fragile, and a test pins that spot down.

Reviewers notice tests. A repository with a tests/ folder and a passing CI badge signals that someone has checked the code, which is exactly what our research code review checklist asks reviewers to look for.

What should I do this week?

Pick one project that is still active and add the first test. Keep it small.

1. Create a tests/ folder and install pytest or testthat.
2. Build a tiny input dataset and save a trusted output from it.
3. Write the regression test and make sure it passes.
4. Add one unit test for the function you trust least.
5. Commit everything, and next time you fix a bug, write its test first.

If your institution has a research software engineering group, they are often happy to help set up CI. The Society of Research Software Engineering is a good place to find that community if you do not know where to look locally.