There’s a lot of hyperventilation in the science world lately about reproducibility. Oh, sorry, is my bias showing? If you read this in Scientific American, or this in The New York Times, or this in The Economist, you might think that the entire scientific enterprise is an exercise in futility, or perhaps, even, a world of utterly depraved carelessness, incompetence, and blind ambition.
Of course, anyone who has ever worked in a lab, or knows a real scientist, is aware that most of the people engaged in scientific research are dedicated and careful. Most of them, far from being cowboys looking for the next big Science paper, are, if anything, overly cautious. The whole discussion sort of reminds me of those polls asking people what they think of Congress (news flash: they hate it) vs. what they think of their own congressional representative (Awesome! Totally NOT part of the problem!)
OK, so aside from the “who, exactly, are these unprincipled monster scientists?” problem, plus the “if half of published scientific results are wrong, and scientific results build on each other, why hasn’t scientific progress ground to a dead halt?” problem, I have another partial explanation for the (so-called) reproducibility crisis.
Here it is. Prepare to be gobsmacked: Biology is complicated.
I’ve written about this before, in The Curious Incident of the Fly in the Night and Bugs and Features in Fruit Flies, but a recent article in Science News, entitled “’Dirty’ Mice Better than Lab-raised Mice for Studying Human Disease” provides too nice an example to pass up.
In the story, journalist Tina Hesman Saey describes two recently published articles, one in Nature, one in Cell Host & Microbe, (both links are to abstracts; the full articles are behind pay walls. However, if you write to the corresponding authors, they’ll happily send you the full text, as Josh Rosenau explained here) that point toward the same unbelievably-obvious-as-soon-as-you-think-about-it-for-one-second conclusion: mice raised in a sterile environment are not always good models for the study of infectious disease in humans. Why? Because their immune systems have not undergone the exposure to myriad bacteria, viruses, and parasites, both beneficial and harmful, that all animals in the real world experience. (This is why immune systems exist in the first place.) Once you think about it, the idea that a mouse with an entirely naïve immune system would be a good model for any human other than a newborn infant becomes problematic. Sort of like throwing a new piano concerto at someone who’s got ten fingers, but never learned to play.
Don’t get me wrong. I’m not saying that anyone who ever studied infectious disease in mice raised in a hyper-clean environment was a stupid idiot who should have known better. It turns out that appreciating the complex interaction among the immune system, pathogens, and the complex assemblage of microbes known as the microbiome that we now know to be essential to health, is a fairly recent development in biology. And now that we know it, it isn’t very hard to look back and find a whole bunch of studies that gave inconsistent results, or where results in mice didn’t pan out in humans. The variable of “what is the history of microbial exposure in this model laboratory animal and how does that affect the process being studied” simply wasn’t taken into account.
I know, I know. Failing to take the microbiome into account does not explain every case of apparent irreproducibility. It’s just an example of the multitude of variables that characterize biological systems, many of which are not very well known, if they’re known at all. They’re what one former U.S. Secretary of Defense memorably called the unknown unknowns. I’d wager that you could ask just about any biologist for an example from his or her particular specialty; previously unknown unknowns turn up all the time.
Uncovering one of those unknown unknowns is one of the most satisfying experiences a scientist can have. Suddenly a whole wealth of seemingly inconsistent or inexplicable (or irreproducible) results start to make sense. Suddenly, entirely new paths of inquiry open up, and new and better ways to design experiments become clear. It doesn’t make the earlier results wrong. Or maybe it does, but in an OK way. Because now we know why they were wrong, and we know a little more about how the natural world works. In other words, science happened.
