#2: Big Data

By Krystle van Hoof

Screen Shot 2015-10-26 at 19.41.59

Big Data is one of those buzzwords that everyone seems to be using but no one can clearly define. While I believe it’s always preferable to be clear, the lack of clarity around what is or is not big data is not necessarily a bad thing.

According Kenneth Cukier, data editor at The Economist, “to define [big data] is to constrain it.”

And why would we want to constrain big data?
The optimists will tell you that big data represents a relatively un-tapped spring of knowledge that can help solve some of the worlds most pressing and complex problems—from poverty to climate change.

The pessimists, on the other hand, may tell you about a scenario where big-data algorithms could enable authorities to predict and preemptively act on things that have yet to (and may never) happen in a dystopian, Minority Report pre-crime sort of way.

As with most extreme predictions about new advances in technology, I tend to think the truth will likely take a far more boring route along the middle road.

So what is it?
Lack of one clear, agreed-upon definition aside, there are a few characteristics that most people are willing to agree upon when it comes to big data.

The 3 (or 4 or 5 or 6…) Vs of Big Data

  1. Volume: There’s a lot of it. so much that any meaningful analysis requires a level of automation (i.e. with computers).
  2. Variety: It comes from a variety of sources and is in a variety of forms (documents, cell phone GPS data, environmental sensors)
  3. Velocity: More being added all the time. More and more very recent data. Fast to store and retrieve.

Some other Vs you might encounter in a search for big data definitions could include: Value, Veracity, Variability, Viscosity, Virality, Visibility…

Another aspect of big data that comes up in a lot of definitions is that it [the data] is made up of information that was not collected for the purpose of mining it. The way it is being used new, in the context of big data, is a secondary purpose. For instance, a store may have been collecting sales information for years. But now, they are analyzing that information across data points that allow them to anticipate and prepare for particular purchasing trends. 

If we must force the idea into a nutshell-sized explanation, I personally like this definition from the guardian:

“Big data is a moniker for the astonishing amount of information that is created as a byproduct of the growing digitisation of our lives – our use of mobile phones, social networks, mobile money, search engines, online shopping, dating apps and so on. What excites policymakers and development practitioners is that if we can mine these datasets we could suddenly have a whole range of information about people that previously would only have been available with months of painstaking planning, travelling and surveying, or, as is often the case in the poorest countries, not at all.”

Big Data for Development
The big data revolution isn’t just about corporations’ bottom lines—the potential benefits for development cooperation are just starting to be uncovered.

The untapped potential of massive amounts of digital information is a promise that has made the UN sit up and take notice. In 2014, UN Secretary-General Ban Ki-moon set up a group of experts to make recommendations on how to bring about a data revolution for sustainable development.

The Data Revolution Group put out its report in November 2014, which outlines several high-level recommendations; lays out some ideas about what the data revolution means for sustainable development; identifies gaps in current data that need to be filled; and provides a few case studies, which illustrate how the data revolution is playing out around the world.

Some Challenges
Apart from the threat of turning into a Big-Brother-like dystopian world, the success of Big Data in the service of development has some challenges:

  1. Existence of data/reliable collection systems
  2. Barriers to open data (government and corporate control)
  3. Privacy Issues
  4. Access & Representation (who is able to provide data? Who has access/can use it?)
  5. Standardisation (for better/more accurate comparison)
  6. Timeliness (can the right people get access and react in time?)
  7. High-Quality Analysis (do the people with access have the experience and expertise to accurately interpret the data?)

Some Opportunities  

  1. Early warning: When I was working for WFP in Mali, we collaborated with a number of other organizations on the SAP, which was an early warning system designed to anticipate and allow us to react to food security emergencies. This system was mainly fed by traditional survey data, which takes weeks to collect and analyse. Big data, collected and analyzed in a timely way could have a significant impact in cases like this.
  2. Real-time: If programmes can respond in a nimble way, real-time information can mean better programs and policies.
  3. Immediate feedback: If you can continually monitor a population across several data sources, you can respond to adjust and improve policies and programs where needed.

Making it Work (for real)
According to the UN Global Pulse report, to get Big Data working for development, we need two key ingredients:

  1. Contextualization (if you don’t know what’s normal in a particular country or region, you won’t be able to accurately analyse the data)
  2. Becoming sophisticated users of information: This comes back to some of the challenges listed above. If you’re planning to spend months or years analyzing your big data, writing white papers on your big findings and setting up committees to discuss it before you do anything useful with it, you might as well flush it down the toilet. (Not mentioning any bureaucracy in particular…)

Tags: , , , , ,


  1. Yes, Kenneth Cukier’s definition of Big Data is accurate. Others come up with different definitions.
    Boyd and Crawford believe that Big Data is a poor term: they claim that “BD is less about Data that is BIG than it is about a capacity to search,aggregate, and cross-reference large data sets”
    Boyd and Crawford (2012:63)

  2. […] I don’t know about you, but it makes my head spinning if I start to think that this project “consists of over a quarter-billion event records in over 300 categories covering the entire world from 1979 to present.” That makes A LOT of DATA. […]

  3. this is the new phenomenon of the coming generation and it is alarm for f doing further….. which is gereat innovation for every planet of the world.

  4. office.com/setup – MS Office setup is very easy to install, download and redeem. Use of it is also simple and the user can learn the use of it easily. Online help option is also available in all application which provides an instant guideline.

  5. HP Printer Support Number – When it comes to the Printer, HP is one of the first choices for many users. They design and develop devices that are reliable, easy to use and efficient. The company also offers the services for its hardware as well as software components to the customers. HP provides products for home, small businesses, government sector and large enterprises.

  6. TurboTax Support – Get 24×7 complete TurboTax Support from the best TurboTax Technical Support experts. Call our support number 1-844-456-8733 (US/CA) for an immediate solution.

  7. Quicken Support – Get 24×7 complete Quicken Support from the best Quicken Technical Support experts. Call our support number 1-844-456-8733 (US/CA) for an immediate solution.

  8. QuickBooks is an accounting software developed by Intuit which has completely changed how the business accounting is done in the industry nowadays. Commonly used by small and medium businesses, QuickBooks has both clouds as well as on places. The QuickBooks has several variations including Enterprise, Accountant, Pro, and Premier etc.

  9. McAfee is not just an antivirus it is a complete security tool which gives protection to Computers, Mobile Devices and Mac from the threats and suspicious website in a more intelligent way than other anti-viruses. It also provides protection to the cloud files and passwords as well as identity theft and data privacy on every device. By visiting mcafee.com/activate, you can download and install the latest version of McAfee product which you want or you can buy it from a retail store.

  10. Thank you for sharing wonderful post

  11. Thank you for the information

  12. Thank you for the information

  13. I was confused in buying an antivirus. I searched on Google and find out this website mcafee activation code free and contact this number 1-855-567-5335. they guided and helped me to choose which one is the best. For more information visit here :- http://www.mcafee-help-setup.co.uk/

  14. Positive site, where did u come up with the information on this posting? I have read a few of the articles on your website now, and I really like your style. Thanks a million and please keep up the effective work.

  15. Thanks for sharing it. Love your content!
    Keep up the good work! Will definitely follow it up…
    Our company is an end-to-end, customer oriented company that is focused on providing tailor-made travel solutions.

  16. Allassignmenthelp has a team with expertise and experience in programming assignments. Our team has professionals with relevant industry experience. Our experts provide best Java assignment help for students.