Does your code stand up to scrutiny?Posted on March 13, 2018
This article originally appeared as an editorial in Nature. Read the original here.
Computer code written by scientists forms the basis of an increasing number of studies across many fields — and an increasing number of papers that report the results. So, more papers should include these executable algorithms in the peer-review process. From this week, Nature journal editors handling papers in which code is central to the main claims or is the main novelty of the work will, on a case-by-case basis, ask reviewers to check how well the code works, and report back.
The move builds on growing demand in recent years for authors to publish the details of bespoke software used to process and analyse data. And it aims to make studies that use such code more reliable. Computational science — like other disciplines — is grappling with reproducibility problems, partly because researchers find it difficult to reproduce results based on custom-built algorithms or software.
This policy is the latest stage in the evolution of our editorial processes, which aims to keep up with technological change across the research community. All Nature journals, for example, already require that authors make materials, data, code and associated protocols promptly available to readers on request, without undue qualifications. In 2014, the Nature journals adopted a “code availability” policy to ensure that all studies using custom code deemed central to the conclusions include a statement indicating whether and how the code can be accessed, and explain any restrictions to access.
Some journals have for years gone a step further and ensured that the new code or software is checked by peer reviewers and published along with the paper. When relevant, Nature Methods, Nature Biotechnology and, most recently, journals including Nature and Nature Neuroscience encourage authors to provide the source code, installation guide and a sample data set, and to make this code available to reviewers for checking.
To assist authors, reviewers and editors, we have updated our guidelines to authors and have developed a code and software submission checklistto help authors compile and present code for peer review. We also strongly encourage researchers to take advantage of repositories such as GitHub, which allow code to be shared for submission and publication.
According to the guidelines, authors must disclose any restrictions on a program’s accessibility when they submit a paper. Nature understands that in some cases — such as commercial applications — authors may not be able to make all details fully available. Together, editors and reviewers will decide how the code or mathematical algorithm must be presented and released to allow the paper to be published.
Occasionally, other exceptions will be made — for example, when custom code or software needs supercomputers, specialized hardware or very lengthy running times that make it unfeasible for reviewers to run the necessary checks. We also recognize that preparing code in a form that is useful to others, or sharing it, is still not common in some areas of science.
Nevertheless, we expect that most authors and reviewers will see value in the practice. Last year, Nature Methods and Nature Biotechnologybetween them published 47 articles that hinged on new code or software. Of these, approximately 85% included the source code for review.
As with other scientific fields, the impact of computational tools is determined by their uptake. Open implementation increases the likelihood that other researchers can use and build on techniques. So, although many researchers already embrace the idea of releasing their code on publication, we hope this initiative will encourage more to do so.
- Featured Researcher: Nick Tusay
- Multi-institutional team to use AI to evaluate social, behavioral science claims
- NSF invests in cyberinfrastructure institute to harness cosmic data
- Center for Immersive Experiences set to debut, serving researchers and students
- Distant Suns, Distant Worlds
- CyberScience Seminar: Researcher to discuss how AI can help people avoid adverse drug interactions
- AI could offer warnings about serious side effects of drug-drug interactions
- Taking RTKI drugs during radiotherapy may not aid survival, worsens side effects
- Cost-effective cloud research computing options now available for researchers
- Costs of natural disasters are increasing at the high end
- Model helps choose wind farm locations, predicts output
- Virus may jump species through ‘rock-and-roll’ motion with receptors
- Researchers seek to revolutionize catalyst design with machine learning
- Resilient Resumes team places third in Nittany AI Challenge
- ‘AI in Action’: Machine learning may help scientists explore deep sleep
- Clickbait Secrets Exposed! Humans and AI team up to improve clickbait detection
- Focusing computational power for more accurate, efficient weather forecasts
- How many Earth-like planets are around sun-like stars?
- Professor receives NSF grant to model cell disorder in heart
- SMH! Brains trained on e-devices may struggle to understand scientific info
- Whole genome sequencing may help officials get a handle on disease outbreaks
- New tool could reduce security analysts’ workloads by automating data triage
- Careful analysis of volcano’s plumbing system may give tips on pending eruptions
- Reducing farm greenhouse gas emissions may plant the seed for a cooler planet
- Using artificial intelligence to detect discrimination
- Four ways scholars say we can cut the chances of nasty satellite data surprises
- Game theory shows why stigmatization may not make sense in modern society
- Older adults can serve communities as engines of everyday innovation
- Pig-Pen effect: Mixing skin oil and ozone can produce a personal pollution cloud
- Researchers find genes that could help create more resilient chickens
- Despite dire predictions, levels of social support remain steady in the U.S.
- For many, friends and family, not doctors, serve as a gateway to opioid misuse
- New algorithm may help people store more pictures, share videos faster
- Head named for Ken and Mary Alice Lindquist Department of Nuclear Engineering
- Scientific evidence boosts action for activists, decreases action for scientists
- People explore options, then selectively represent good options to make difficult decisions
- Map reveals that lynching extended far beyond the deep South
- Gravitational forces in protoplanetary disks push super-Earths close to stars
- Supercomputer cluster donation helps turn high school class into climate science research lab
- Believing machines can out-do people may fuel acceptance of self-driving cars
- People more likely to trust machines than humans with their private info
- IBM donates system to Penn State to advance AI research
- ICS Seed Grants to power projects that use AI, machine learning for common good
- Penn State Berks team advances to MVP Phase of Nittany AI Challenge
- Creepy computers or people partners? Working to make AI that enhances humanity
- Sky is clearing for using AI to probe weather variability
- ‘AI will see you now’: Panel to discuss the AI revolution in health and medicine
- Privacy law scholars must address potential for nasty satellite data surprises
- Researchers take aim at hackers trying to attack high-value AI models
- Girls, economically disadvantaged less likely to get parental urging to study computers
- Seed grants awarded to projects using Twitter data
- Researchers find features that shape mechanical force during protein synthesis
- A peek at living room decor suggests how decorations vary around the world
- Interactive websites may cause antismoking messages to backfire
- Changing how government assesses risk may ease fallout from extreme financial events
- Algorithm aims to alert consumers before they use illicit online pharmacies
- Scientists tap AI betting agents to help solve research reproducibility concerns
- Search engine could help researchers scour internet for privacy documents