EPA administrator Scott Pruitt promised in late April to raise standards for the science behind environmental regulations — restricting usable studies to those that can be replicated or independently verified. That’s the rationale for a rule he proposed called “Strengthening Transparency in Regulatory Science.”
It sounds harmless on a superficial level, but there’s always a trade-off: The more certainty required of the science, the greater the risk people may be exposed to dangerous pollutants or toxic products.
Pruitt has framed the discussion to focus on the possibility of false positives — studies that exaggerate risks. But any serious effort to improve the environmental science used to make regulations would need to start by determining if indeed there is an excess of false positive results, or if instead there’s a problem with false negatives that underreport or miss real risks. If history is any guide, false negatives could prove to be more costly.
The focus on false positives is premised on the assumption that problems recently uncovered in psychology and some areas of medical research also extend to environmental science. In a 27-page document describing the rule, Pruitt made reference to the so-called “replication crisis” — concerns over systematic reviews revealing that more than half of published studies could not be replicated, but only in those limited fields. There’s no evidence so far that the same kind of crisis affects physics, astrophysics, chemistry, climatology and other fields.
Scientists quickly expressed opposition to Pruitt’s transparency rule, arguing that he would use it as an excuse to roll back regulation critical for protecting public health. It wasn’t clear from Pruitt’s proposal whether the transparency was supposed to apply only to future regulations, or existing ones.
UCLA statistics and epidemiology professor Sander Greenland helped me consider the importance of false negatives. He said that the tendency to produce false positives or negatives varies from one field to another, and may depend on the incentives faced by researchers. This echoed a discussion on incentives I had last year with Dana Farber Cancer Center biologist Bill Kaelin. He told me that there’s major pressure to get positive results in his field. The publish-or-perish culture has intensified, while rigor has declined.
Before about 1980, he said, scientists strove to publish papers that proved one thing in multiple ways. Now, he said, top journals want multiple things proven one way. The claims have to be bigger — loftier. But the evidence can be weaker.
In psychology, too, researchers face a strong incentive to get positive results — indications of some real effect. A study by a Harvard psychologist, for example, appeared to prove that assuming certain “power poses” gave people a hormonal boost. That soon led a to record-breaking TED talk and bestselling book — though subsequent analysis cast doubt on her conclusions.
But in other fields, there are incentives to find negative results. In pharmaceutical research, for example, it’s much better for a drug’s inventors if no serious side effects are found.
It’s hard to imagine a more egregious case of bad U.S. environmental science than the studies that purported to show leaded gasoline was likely to be harmless. Now, there are multiple lines of evidence showing that during the 1960s and 1970s, millions of children suffered irreversible brain damage.
The harm from this false-negative science isn’t just a matter of weighing a small number of lives against the economy — the economic impact of all that brain damage may have outweighed any economic benefits accrued by continuing to use leaded gasoline and lead paint.
The scientists who’ve objected to Pruitt’s proposed rule are right to worry. The rule is vaguely worded and premised on assumptions and innuendo.
Pruitt’s rule could go into effect after a 30-day comment period, though The Washington Post reports that it could face opposition in court. The rule itself is based on an alleged problem for which there’s no solid evidence at all.
Faye Flam is a Bloomberg View columnist.