Reviewing software and code: an update
On June 14th 2013 Mozilla Science Lab announced their collaboration with PLOS Computational Biology. The collaboration was a trial series of software reviews conducted by Mozilla engineers for small pieces of code from articles already published with PLOS Computational Biology. The project began amidst discussion about the presentation of scientific code, such as this piece by Carl Boettiger, who stated: “We are not really taught to review software, any more than we are taught to write it in the first place.”
The project was spearheaded by Kaitlin Thaney from Mozilla Science Labs and Greg Wilson from Software Carpentry and announced by posts on Kaitlin’s blog and on PLOS Biologue. The full results have now been released on arXiv and summarised on the Mozilla Science blog, which explains that many of the scientists had never had their code reviewed before. Kaitlin states that while “the scientists aimed to produce readable, re-usable code, the reviewers felt their software was less reusable by others”. However, she also concludes that “the authors still found the comments useful, particularly feedback on usability, ease of re-use, organization of README files, code structure, performance questions and optimization”.
Since the first results were revealed, Mozilla Science Labs have announced that they are embarking on a second round of code reviews. In this version of the project, mentors from Mozilla will work with small groups of scientists, at first performing code reviews but also training the scientists to do the reviews themselves. Anyone who is interested in taking part as either a mentor or part of a team should use this email to get in touch.
It should be noted that this was a small-scale experiment and so much of the evidence provided is anecdotal. However, we believe that scientific research could benefit from an increase of these kinds of interactions. Overall we at PLOS Computational Biology are thrilled to have been a part of this project and hope that the results contribute to the growing world of scientific software.