Reproduce, Reproduce, Reproduce!

Is one “successful” result enough to claim fame in academia?

In the latest years, there has been a cry out within the science community about a so called reproducibility crisis. What does that really mean and does it imply we cannot trust scientific results? Also, what does lack of reproducibility in science mean in a development perspective?

70% of researchers who tried to reproduce other scientists’ findings failed to do so according to a survey by Nature. Add to that: more than half of respondents, who reported they had tried, even failed to reproduce their own results. As one respondent to the survey commented:

‘The challenge is not to eliminate problems with reproducibility in published work. Being at the cutting edge of science means that sometimes results will not be robust. We want to be discovering new things but at the same time not generate too many false leads’

What does lack of reproducibility mean in a development perspective?
Lack of reproducibility in science can be a problem, since in the best of worlds, we expect policy makers to base their decisions on robust scientific findings that represent reality. Decisions that affect citizens’ lives. Policymakers seldom take decisions based on cutting edge research, but rather synthesized reports where a number of studies points in the same direction. Still,  naturally there is a risk that trust in science goes down if studies can´t be reproduced.

Another threat to reproducibility is when research is based on big data that comes from one single corporate source and is released to researchers under restricted conditions, as in the Orange D4D challenge. This characteristic of limited replicability and thus verifiability is important when considering the politics of data as representation. Another problem is that of unknown bias, when using big data, since data obtained my not be representative of population at large (Taylor, 2015) Related to non-representative data, read more about lack of sex-disaggregated data here.

Why is reproducing other researchers work not sexy (enough)?
When studies, do not reproduce, researchers often assume there is a perfectly valid (and probably boring) reason. What’s more, incentives to publish positive replications are low in academia. Researchers are more rewarded for novelty and breaking new ground in research than for contributing to robustness. Respondents who had published a failed replication said that editors and reviewers even demanded that they play down comparisons with the original study.

Is there even too much replication going on?
Another voice in the reproducibility crisis debate came from psychologist Courtenay Norbury, who found a paper that came to the same conclusion as she found 12 years ago. She turned to Twitter and asked: How many times does a research finding need to be replicated before the field says “ok, how do we move this forward?” and a new angel of the discussion started off.

 Particularly psychology has been the discipline with more results questioned to hold for reproducibility.  Some well-knows psychologists, including Harvard’s Daniel Gilbert, have pushed back against the very idea of a replication crisis and criticized the critics for sniping at their colleagues rather than pursuing their own research. (Bartlet, 2016)

 

TED talk on ‘power poses’ goes viral, while key results don’t hold for replication
Social Psychologist Amy Cuddy’s TED talk Your body language may shape who you are is one of the most popular TED-talks ever. It is easy to understand what makes it so clickable.  The take-home message is that standing in a confident pose for two minutes is enough to transform a timid person into a fierce go-getter.

What went unmentioned during Cuddy’s more and more frequent public appearances talking about power poses, was the fact that the underlying study had started to crumble and was impossible to replicate for several other research teams.
Eva Ranehill, one of the researchers who tried to replicate Cuddy´s study was intrigued by power poses. She set up an almost identical experiment as Cuddy- but with the difference that a computer gave instructions instead of a human, for a less personal approach.  Also Ranehill’s sample was more than four times as big (200 people) as the original study, but she didn’t get close to the same results as Cuddy. (Bartlet, 2016)

So how could such questionable research migrate from a journal to a viral video to a bestseller, circulating for years, retweeted, forwarded and praised by millions, with almost no pushback? Most certainly because the take home message is fun, easy to understand and goes down very well beyond and academic audience. TED has recieved criticism for oversimplifying research; being based on the Model of American Idol; and straight up ‘Lying to us’ (Denskus & Esser, 2015)

Action taken by TED- and what´s the harm?
So, recently TED added a science curator to their staff and published a specific section on their web called TED Science Standards. A disclaimer has also been added to the description of Cuddy’s talk, stating that there is an ongoing debate about robustness related to her findings. Cuddy has also been invited to comment on this.

One might wonder what the harm is in doing some power posing every now and then? After all, this talk is not about the cure for cancer.  Uri Simonsohn reasons on this in The Data Vigilante: ‘The main reason to be worried about power posing is not power posing -that is  trivial entertainment’ he says. But it would be great if it didn’t come at and expense of the science brand.

Sources:
Baker, M. (2016). 1500 scientists lift the lid on reproducibility (Nature)

Baker, M. (2015). First results from psychology´s largest reproducibility test (Nature)

Bartlet, T. (2016). When Big Ideas Go Bad: When big ideas go bad. The Chronicle of Higher Education

Chawla, D. (2016). How many replication studies are enough? (Nature)

Cuddy, A. (2012). Your body language may shape who you are. TED Talk https://www.youtube.com/watch?v=Ks-_Mh1QhMc

Denskus, T., & Esser, D (2015). TED talks on International Development: Trans-Hegemonic Promise and Ritualistic Constraints. Communication Theory. 25 (2), 166–187.  https://doi.org/10.1111/comt.12066

Ranehill, E. et. al (2015) Assessing the Robustness of Power Posing. No Effect on Hormones and Risk Tolerance in a Large Sample of Men and Women. Psycological science, 26 (5) 653-656. https://doi-org.ezproxy.ub.gu.se/10.1177/0956797614553946

Taylor, L., & Schroeder, R. (2015). Is bigger better? The emergence of big data as a tool for international development policy. GeoJournal80(4), 503–518. https://doi.org/10.1007/s10708-014-9603-5

Image credit: Paul Blow

Back to Top