Big Data: Silver Bullet, 21st Century oil, or just a plain old thermometer?

Part 1

Development has seen a number of “silver bullets” since the middle of the last century and it’s tempting to place Big Data as the latest and greatest solution to the development question. But as competing silver bullets ricochet through workshops and high-level panel discussions in the develo-sphere, in this post I want to dodge the Big Data for Development bullet for a moment, and consider instead what it means for a “post-truth” world.

In How Statistics lost their power – and why we should fear what comes next, (Jan 2017) William Davies positions the emergence of Big Data as a direct challenge to the dominance of statistical truth which emerged with the Enlightenment. As the questioning of institutions and received wisdom became a cornerstone of emerging democracies, statistics were the stable reference points to build an argument – essential in a Habermasian context of public sphere. Rational informed debate and consensus coalesced around statistical tools to “see” the nation collectively. This could then be used to challenge and ultimately replace the divine rule and intervention of kings.

So where and why did this collapse? The current situation can perhaps be seen as a frustration that one form of authority (Church and King) has simply been replaced with another (State and Experts). The data itself was neutral, but the power that it enabled was still held by an elite.

“From one perspective, grounding politics in statistics is elitist, undemocratic and oblivious to people’s emotional investments in their community and nation….From the opposite perspective, statistics are quite the opposite of elitist. They enable journalists, citizens and politicians to discuss society as a whole, not on the basis of anecdote, sentiment or prejudice, but in ways that can be validated.”

The digital and data revolution has taken that dichotomy to a dizzying conclusion. Information on citizen’s lives, activities, sentiments are now accumulated by default. In the previous paradigm, statisticians started with a question or problem, then sought the numbers to clarify the scale of the situation, or the progress of planned solutions. In the Big Data world, data is captured first. It’s up to “us” what questions to pose to these vast data sets.

In theory then, there should be an unprecedented potential to use data to serve the public good, and Big Data evangelists prioritise this potential, often at the expense of four other important factors:

  • Facts are less important in big data sets than sentiment and opinion.
  • Big Data prioritises correlation over causation and thus removes individual agency from analysis. We no longer need to know why people do what they do – just to accept they do, track and measure it. With enough data, the numbers speak for themselves (The End of Theory).
  • Who has the access, capacity and expertise to interpret and question the data;
  • Big Data is as prone to sample bias and error (if not more so) than traditional statistical analysis. For example, collecting sentiment data through searching for key words on Twitter doesn’t tell you what the nation feels, but what Twitter users report. Who gets left out?

The second two points extend the technical and ethical limitations of traditional statistical analysis, and I will return to them in my next post. It is the first two which relate most to the “Post-Truth” world.

Post-Truth, the word of the year in 2016, is defined as: “relating to or denoting circumstances in which objective facts are less influential in shaping public opinion than appeals to emotion and personal belief.”

In Regimes of Posttruth, Postpolitics, and Attention Economies, Jayson Harsin discusses the collapse of the “truth regimes” propped up by statistical evidence and summises that this has not led to their demise, “but rather to a more complex reorganization of functions, among which are efforts to mobilize new digital “participatory culture” to proliferate truth games—that is, to generate an overall regime of posttruth (ROPT).”

What does the proliferation of “truth games” actually mean? Unless you’ve been living in a cave for the past year, you will have come across the mainstream media’s frenzied obsession with the rejection of truth in politics. Truth isn’t falsified, or even contested; it’s just of secondary importance. The fact-checking of Trump’s rallies and debates (and now his press conferences) didn’t impact his supporters who wanted to believe his emotional rhetoric.

An Economist article claims that, “Once, the purpose of political lying was to create a false view of the world. The lies of men like Mr Trump do not work like that. They are not intended to convince the elites, whom their target voters neither trust nor like, but to reinforce prejudices.”

Big Data analytics rejects the traditional demographic segmentation of pollsters and statisticians. The vast new data sets allow analysts to determine patterns, trends, correlations and emergent moods by tracking key words in social media. Most importantly, it becomes of way of people defining their own identities (such as “#ImwithCorbyn” or “entrepreneur”) rather than imposing classifications upon them which conform to statistical and canvassing models.

“With the authority of statistics waning, and nothing stepping into the public sphere to replace it, people can live in whatever imagined community they feel most aligned to and willing to believe in. Where statistics can be used to correct faulty claims about the economy or society or population, in an age of data analytics there are few mechanisms to prevent people from giving way to their instinctive reactions or emotional prejudices.”

When members of interest groups coalesce online in the echo-chambers created by algorithms of preference, opinions can rapidly and intractably be accepted as fact.

“A lie gets halfway around the world before the truth has a chance to get its pants on.”

Winston Churchill’s observation has never been more true than in the current inter-connected world as the fake-news crisis dominates political discourse. And it knows no nation-state borders as the expose of Macedonia teens creating fake-news for advertising revenue, revealed.

But it’s not just this. Opinions tend to simplify while traditional statistical analysis is often bewilderingly complex or contradictory (part of why it has been rejected). When Trump was ridiculed for exclaiming that healthcare was complicated, the ridicule perhaps misses the point. “Policy is complicated, yet post-truth politics damns complexity as the sleight of hand experts use to bamboozle everyone else.”

But therein lies a greater danger. As the political system becomes dysfunctional in the “truth-games” rhetoric that we saw so clearly in the Trump election and Brexit referendum, it creates it’s own feedback loop of frustration, amplified in echo-chambers online. The poor results of policies created by appeals to opinion, can extend the same alienation and lack of trust that created the conditions for post-truth discourse in the first place. The mainstream media’s role to hold leaders to account, in a post-truth world is condemned as complicity in a fact-obsessed and broken system. And this is contagious. Just last week, Hun Sen in Cambodia cited Trump’s banning of the mainstream media from the Whitehouse to legitimize his own clamping down on mainstream press freedom in Cambodia.

But in any public debate, we know it’s not just facts that persuade an electorate. It’s always been about feelings as well. Now with Big Data we have the capacity to map feelings, not facts, in the kind of detail that focus group discussions, the bedrock of much statistical qualitative research, could never come close to.


  1. derek

    Do you think that to consider post truth, is to believe there was truth before that. If yes, isn’t that our fundamental issue, that we are still believing in a truth that is unique, even though throughout history, a general agreed truth has been very rare. So what truth are you talking about? Or which truth is the dictionary referring too? I found this page very suprisingly, but I feel these are very actual topics and looking forward to see where you go with it regarding South issues…if there is such a thing…. all the best, derek, uganda

    1. Ian

      Hey Derek – thanks for taking the time to comment. And it’s an interesting point to which I don’t really have an answer. A historical perspective would show that any notion of truth is constantly in a process of disruption/replacement – whether that’s the truth that the earth is flat, or that the universe revolves around the earth. The point of a notion of truth is as a function to hold society together. That can and does change and one truth is replaced by another. The difference with post-truth isn’t a replacing of one truth with another, around which society can hold together, but a rejection of any notion of truth whatsoever. That creates a problem for the “social glue” – and so you get consensus and social cohesion through opinion and (sadly) prejudice. “Imagined communities” of shared preferences online replace geographical communities. The question is perhaps more about how those preferences are engineered by social networks and the power that those networks have. It would be an interesting experiment to try and screw with the social network algorithms, by posting, sharing and commenting on things normally on the opposite end of the spectrum to your usual online presence. At the moment, many of my posts are on films and filmmaking – and the advertising that pops up reflects that – mainly ads for Aaron Sorkin’s scriptwriting masterclass (which I’m sure I need!). How many times would I have to post things from Infowars or Breitbart before the algorithm changes and I start to see more ads for survival and protection products? A worthy experiment??? That’s the idea behind the title of the blog – Al Gore Rhythm.


    1. Ian

      Awesome article – I had wanted to try this out as an experiment – glad someone has done it for me:
      “Facebook uses algorithms to decide what shows up in your feed. It isn’t just a parade of sequential updates from your friends and the things you’ve expressed an interest in. In 2014 the News Feed is a highly-curated presentation, delivered to you by a complicated formula based on the actions you take on the site, and across the web. I wanted to see how my Facebook experience would change if I constantly rewarded the robots making these decisions for me, if I continually said, “good job, robot, I like this.”


Leave a Reply

Your email address will not be published. Required fields are marked *