MozFest 2017 session notes: why research code stays closed

These notes summarize the Code is Science session at MozFest 2017 in London. They have been rewritten by the current editors from the original volunteer write up. Individual participants are not named.

The session opened with a simple observation from the facilitators: a lot of work in research depends on code, and very little of that code is shared. That means errors in it are rarely caught. The ideal they put forward was open source as the gold standard for research code, or at the very least code that has been reviewed before a paper is published.

Who was in the room?

Attendance grew during the hour to around 20 people, including the facilitators. Roughly half described themselves as researchers or academics and roughly half as developers, most of whom already worked on open source. A few sat in both camps. That mix shaped the conversation: researchers spoke about pressure and incentives, developers spoke about practice and tooling, and the overlap was where the useful ideas came from.

Why do people keep research code closed?

The first activity asked everyone to write down, one per post-it, why researchers choose not to share their code. The wall filled up fast. The reasons, roughly in the words people used:

• Embarrassment about the quality of the code. This came up more than once.
• Fear that others will find mistakes, or will judge the features they did not build.
• Not knowing that sharing is an option, or not knowing how to do it.
• No time, and the effort needed to clean code up before release.
• No reason or incentive to share. Nobody asks, nobody rewards it.
• Sharing is not the default, so not sharing is easier.
• The researcher does not own the copyright. The employer does.
• Patents, commercial interest or perceived monetary value.
• Giving away a competitive advantage for future papers.
• Credentials, configuration files or private keys left in the code that would need to be scrubbed first.
• Code that touches sensitive or restricted data.
• Fear of a maintenance burden: once code is public, people expect support even if none is offered.
• A belief that nobody else would want it anyway.
• No venue to have the code peer reviewed, so no obvious path to recognition.

What themes came out of it?

When the notes were clustered, they fell into a handful of groups:

1. Skills and awareness. Many researchers have never been taught how to publish code, choose a licence or remove secrets from a repository.
2. Fear and shame. Showing your working means showing your mistakes. In a competitive field that feels risky.
3. Time and effort. Making code truly reusable takes work, and that work is rarely counted.
4. Ownership, IP and commercial conflict. Especially where the employer holds the rights or there is a patent in view.
5. Lack of incentive. Apathy, unclear benefit, and the worry of losing an edge for future work.
6. Maintenance and documentation burden. The fear that publishing creates an obligation.

Several of these are not really technical. They are about culture and reward. That was one of the clearest messages of the hour: better tools help, but they do not change what researchers are rewarded for.

How do you review research code?

The second half of the session turned to review. Several people pointed out that review venues for research software already existed. rOpenSci runs open peer review of R packages against a published set of standards. The Journal of Open Source Software reviews research software packages and publishes short papers about them.

But the room noted a gap. Those venues review reusable packages. The analysis scripts that sit behind a specific paper, the ones that load data, call those packages and produce the figures, are mostly not reviewed by anyone. Even when a well reviewed package is used, how it was applied in a given study is often a black box.

The discussion then circled a few hard questions:

• What is worse: no tests, or failing tests? There was no consensus. Failing tests at least show someone tried to check the code. No tests show nothing at all.
• Should there be a minimum quality bar? Most people thought yes. Not perfection, but enough to run the code and understand what it does.
• Do we need a dedicated facility for reviewing code? Some thought journals should handle it. Others thought a separate service would work better.
• Is reviewing code just hard? Yes, and the room was honest about it.

Is there such a thing as the perfect reviewer?

No, and that was a useful conclusion. A paper might combine statistics, domain science and code in three languages. Expecting one reviewer to be expert in all of it is unrealistic. Several points followed from that:

• Reviewers should accept that the code is part of what they are reviewing, not an optional extra.
• Reviewers need to know in advance which languages and tools are involved. Review invitations rarely say, so people agree to review work they cannot fully check.
• A reviewer who is not comfortable in the language should say so, and the editor should find someone who is, or split the review.
• Clear communication of what was and was not checked is itself valuable.

What happened next?

The session ended with interest in continuing the work: collecting journal policies on code, writing practical guidance for researchers and making the case for code review in publishing. The list of themes above became a kind of map for the project. Each reason people gave for keeping code closed points to something that can be addressed, whether with training, clearer policies, better tooling or changes to what gets credit.

Reading these notes years later, what stands out is how human the reasons are. Very few people in that room argued that code should stay closed. They described why it does, and most of those reasons are fixable.