In a recent post, I wrote about the establishment of a new Statistical Board of Reviewing Editors at Science magazine – a response, in part, to emerging concerns about poorly applied statistical methods in published research results. As I wrote, I believe that the establishment of a new research board is characteristic of how a healthy scientific community should react to signs of problems: no one benefits when insufficiently justified results are published.
However, I also mentioned that I believe the problem is not as dire as some headlines suggest, and I’d like to tell a story to illustrate my point.
It is a given in science that results should be reproducible. That’s why authors are required to provide detailed experimental methods, and must agree to share unique reagants, cell lines, or software tools with researchers wishing to repeat or build on their experiments or models.
So it came as a shock when the pharmaceutical company, Amgen, reported that it was unable to reproduce the results of nearly 60% of what it described as “groundbreaking” results in cancer research. Amgen’s conclusion was that pharmaceutical companies are potentially at risk of sinking millions of dollars into pursuing research results that are somehow flawed. That’s a serious contention, and it is being taken seriously, with many in the biomedical community calling for the establishment of private third-party laboratories to reproduce critical results – an extremely expensive proposition.
And one that is likely to be highly frustrating. Because I suspect that a lot of results will, indeed, not be reproducible. And I also suspect that in many, many cases, the lack of reproducibility will not mean that those results were wrong. Sound unscientific? Let me explain.
Bear in mind that Amgen is concerned about research in basic and clinical biomedicine. In other words, in the kinds of research that study living organisms. Living things - even the ones we blithely call “simple”, like bacteria - are in fact extraordinarily complicated. Successful research, even on the benchtop in a laboratory, requires controlling for a multitude of variables – temperature, pH levels, infinitesimal variations in the concentrations of salts, co-factors, and trace metals. A new batch of a critical enzyme or culture broth, a different passage number of the same cell line, even adding reagants in a different order can and do change results.
All researchers struggle with reproducing their own results, much less results from another laboratory. Of course, when research leaves the more controlled environment of the laboratory and reaches out into patient populations or ecosystems, even more variables are introduced.
Successful research, therefore, in part involves systematically tracking down and controlling for more and more potential variables. Often, the process of figuring out why one’s own results aren’t reproducible can point the way toward a source of variability that was previously unknown. Indeed, to an alert scientist, problems with reproducibility can open up an entirely new line of research.
Mimi Shirasu
-Hiza, assistant professor of genetics and development at Columbia University, is just such a scientist, whose response to puzzling results has led her not once, but twice, to important insights about how life works. She told the story of those insights during a talk at the American Society for Microbiology Conference on Beneficial Microbes in 2012.
Shirasu-Hiza studies fruit flies (Drosophila melanogaster) at Columbia University. Her postdoctoral studies were focused on how a certain bacterium that infects the flies wreaks its havoc. These studies required injecting the bacteria into the flies’ abdomens. (You know those little flies that buzz around your compost bucket? Yeah, those little teeny ones? Those are the ones she works on. And, yes, she injected bacteria into their teeny-tiny abdomens. I’d rather bail out Lake Erie with a teaspoon, but I guess she REALLY wanted those experiments to work!)
Anyway, a self-described workaholic, Shirasu-Hiza frequently found herself infecting flies in the middle of the night. And she noticed something curious. The flies she injected at night were more likely to become sick and die than flies she injected during the day. In other words, she couldn’t reproduce her own results if she carried out the experiments at a different time of day.
She took this perplexing result to her postdoctoral advisor, who said, "you must be doing something different when you infect them in the middle of the night." Though skeptical, he helped her rig up an experimental system (think light-proof boxes) that would allow her to fool a group of flies into thinking it was the middle of the night when it was not. This way, she could infect flies at the same time of day for her, but half of the flies “thought” it was daytime, and half “thought” it was nighttime. When she infected the real daytime and the false daytime flies at the same time: their immune responses were different! Unbeknownst to her, or her advisor, the immune response of the flies was affected by their circadian rhythm – a result that she went on to explore, and which she continues to study.
By the way, this result is important because if immune response differs by time of day, it might be something worth knowing not only when studying pathogenicity in fruit flies, but even when scheduling your next surgical procedure. (I hope I don’t have to add that results shown in fruit flies are likely to be relevant to humans because … evolution.)
In part II - tiny flies, tinier partners.....
