Big Data, Bigger Mistakes?

Almost everyone these days is talking about "Big Data" and how it's shaking the business landscape. Financial Times (FT) Magazine quips that big data is a "vague term for a massive phenomenon that has rapidly become an obsession with entrepreneurs, scientists, governments and media".

What exactly is big data anyway? Simply put, it is identifying, collecting and analysing large volumes of information. This information must first be captured digitally and can then be used for a variety of purposes including making better business decisions. Here's one of several articles to come looking at big data with a critical eye.

The Google experience

Several years ago Google announced that "Google Flu Trends" was able to easily and quickly track the spread of influenza across the US. According to FT Magazine, Google did this by taking a sample of some top 50 million search terms and applied various algorithms to it. 

However, four years after Google published their paper in Nature, a prestigious science journal, the "theory-free, data-rich" model began to fall apart. Google's estimates of the spread of flu-like illnesses were overstated compared to the Centers for Disease Control and Prevention.

The problem was that Google was simply looking for statistical patterns in the data. Moreover, it appears they didn't think about whether search terms really reflected the spread of flu. For example, searching for "flu vaccine" in Brisbane may not be caused by flu outbreaks in Brisbane itself. Perhaps a celebrity tweeted about it, or perhaps health or pharmaceutical organisations were engaging in a major online marketing campaign.

A bump in the (big data) street

FT Magazine also highlights an app released by the City of Boston in the USA. Using the Street Bump smartphone app, the city claimed it could detect potholes automatically. 

The method sounded ingenious at first - using the app, anyone could just drive around on their normal routes and the phone accelerometer would detect bumps and automatically notify the city council. 

However, what Street Bump produces is most likely a map of potholes or other extraneous data that "systematically favours" the people that are most likely to use it. Possibly young people who download the app... possibly people who actually care that much about using an app to fix potholes!

In any case the City of Boston claims that the app provides the city with "real-time information to help improve roads" and Street Bump won a "Digital Government Achievement Award" in 2013.

No doubt apps, mobile and desktop websites and other tools can collect useful crowdsourced data. And systems such as Google Maps, Apple Maps and Waze do genuinely attempt to "digest" and present such data in relevant, interesting and effective ways. Perhaps as Street Bump gets wider use around the world and its methods improve it will turn out to be truly effective for city and state governments.


FT Magazine sums it up as "the challenge now is to solve new problems and gain new answers - without making the same old statistical mistakes on a grander scale than ever."

Many businesses and organisations are rushing to the first big data water fountain they find and dipping their heads in it. However, merely collecting large amounts of data and making purely mathematical assumptions about it is ill-advised.

In any case, businesses will have to shift from paper-based systems and processes to a purely digital workflow before they can implement big data solutions. Team Wired specialises in data entry services. Whether it's gathering customer feedback, key business intelligence, or if you're looking to convert a room full of paper forms to electronic format, contact us today. We'll make sure your very first step into big data isn't a big mistake.

Photo credit: Flickr/Jer Thorp. Research credit: Big data: are we making a big mistake? FT.Com