By Genna Reed, Union of Concerned Scientists
Ever think about reproducibility in science? Turns out you’re not alone! The National Academies of Science (NAS) just spent a year and a half studying the status quo and have released some important findings. An NAS committee released a report this week that EPA Administrator Andrew Wheeler, Department of Interior Secretary David Bernhardt and OMB Acting Director Russell Vought should really read, titled Reproducibility and Replicability in Science. The group of experts was charged with answering questions about reproducibility and replicability, mandated by the 2017 American Innovation and Competitiveness Act. There are two key takeaways that are incredibly important for federal agency heads to understand as they are issuing sweeping policies that include language about these scientific concepts under the guise of transparency.
Reproducibility and replicability are important but not the be-all end-all of good science
The NAS committee was charged with defining reproducibility and replicability across scientific fields. Reproducibility is obtaining consistent results using the same input data, computational steps, methods, and code, and conditions of analysis. Replicability is obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data. While the report acknowledges that reproducible and replicable studies help to generate reliable knowledge, it also is very clear throughout that these standards can be features of a scientifically rigorous study, but are not necessarily essential. The committee writes, “A predominant focus on the replicability of individual studies is an inefficient way to assure the reliability of scientific knowledge. Rather, reviews of cumulative evidence on a subject, to assess both the overall effect size and generalizability, is often a more useful way to gain confidence in the state of scientific knowledge.” There are many reasons why a study might not be able to be reproduced or replicated, not the least of which to protect the privacy, trade secrets, intellectual property and other confidentiality concerns associated with the underlying data. Challenges also arise when studying environmental hazards. We must use observational data for studies of air and water pollution and it is often not possible or ethical to recreate the conditions under which people were exposed to a contaminant.
As my colleague, Andrew Rosenberg, explained in a recent blog:
“Maybe we all learned that doing an experiment in a lab many times over can give you confidence in the results and that is the “scientific method.” Made sense in grade school. But lots and lots of critical scientific information and even analyses are not “reproducible” in this sense. Take, for example, the impact of a toxic pollutant on a local community. Should we release it again to see if it is really harmful? Or the study of a natural disaster? Should we wait for it to happen again to reproduce the results? The Environmental Data and Governance Initiative illustrated the many real-world examples of scientific studies that are neither feasible nor ethical to reproduce.”
In the EPA’s proposed restricted science rule issued last April, EPA argues that part of the reason for the policy is to allow regulators to better determine that key findings are “valid and credible.” It claims that the benchmark upon which validation and credibility are measured is reproducibility and replication of studies. But as EPA fails to understand and the NAS committee rightfully points out, “reproducibility and replicability are not, in and of themselves, the end goals of science, nor are they the only way in which scientists gain confidence in new discoveries.” The report explains that policy decisions should be based on the body of evidence, rather than any one study (replicable or not), and likewise, that one study should not be used to refute evidence backed by a large body of research. Further, systematic reviews and meta-analyses, whereby large bodies of evidence are evaluated, are an important method of increasing confidence of scientific results. The EPA and other agencies should have the flexibility to use their own criteria to judge the rigor and validity of the science informing rules as applicable, and should not rely on reproducibility and replicability as the principal criteria of scientific credibility.
Challenges of transparency and reproducibility in science are best handled within the research community, not the White House or EPA
Improvements in transparency can be and are being made by researchers, journals, funders, and academic institutions and the report gives many neat examples of ongoing efforts. It certainly is not one agency’s job to solve issues around science transparency. Indeed, they couldn’t do this even if they tried. The recommendations of the report are aimed at scientific institutions to work on educating researchers and ensure best practices in recordkeeping and transparency that may lead more reproducible and replicable studies. Nowhere does it suggest that federal agencies that are users of such science should be involved in deciding how transparent authors must be. The scientific community needs to drive the bus. End users of scientific information are not in a position to address challenges in the scientific community at large, especially considering the lack of infrastructure and resources needed to ensure privacy protections for sensitive data within agency rulemaking. Instead of making sweeping transparency requirements that would limit the government’s ability to use the best science, the report recommends that funding agencies invest in the research and development of open-source tools and related trainings for researchers so that transparency is fostered at the beginning of the scientific process instead of being used as an opportunity to exclude crucial public health studies that have already been conducted.
No crisis of reproducibility, no time for complacency
During the report release webinar, the study authors summarized their findings by saying that there wasn’t a crisis of reproducibility, nor was it a time to be complacent about issues related to transparency in science. This is a fair assessment of the situation and one that should be reexamined by the EPA as it reviews the 590,000 public comments it received on its restricted science proposal. There are absolutely ways we can use technology to improve recordkeeping and transparency throughout the scientific process so that researchers can better build on one another’s findings and advance knowledge. Smart minds at NAS and elsewhere are already working on this. The committee report highlights some of the ways this is happening thanks to the leadership of academic institutions, funders, and journals. Government has a role to play in helping to fund the infrastructure that will foster more open and accessible science and arm researchers with the tools to abide by best practices. The EPA, DOI, and OMB should listen to the scientific community and learn how best to accomplish that task. There is absolutely no role for the White House or federal agencies like the EPA to issue sweeping, prescriptive rules that limit the way that science is used to inform regulations.