In the beginning, there was data. And it was good. So good, in fact, companies demanded more of it. Data could be used to identify past trends, shape current policies and predict future events. Eventually, the volumes of information being generated became so large and complex that a grander term was needed. It had to reflect the need for powerful mass-scale number-crunching algorithms, instead of traditional data processing software.
Rather underwhelmingly, the phrase coined for this phenomenon was big data. This term is now used to identify any database or collection of information that requires sophisticated analysis to identify patterns or extract value. And in truth, that applies to much of today’s internet. With an estimated 20 billion devices now uploading information as part of the connected Internet of Things, data volumes are increasing at an astonishing rate. Cisco has calculated that the IoT alone will be generating 600 zettabytes of data a year by 2020. That’s 1,000,000,000,000,000 gigabytes of information. And that’s a big number.
Vast amounts of data can’t be processed by an Excel spreadsheet or managed from an iPad. Big data requires specialist storage, processing, and analysis to be of value. For instance, modern antivirus algorithms can scan colossal volumes of web traffic, looking for tell-tale signs of viruses. When your bank phones to query recent transactions, computers rather than cashiers have identified deviations from your normal banking activities. The word ‘patterns’ crops up a lot in big data discussions, particularly across industries like physics and financial technology. The latter is also known as fintech – a rapidly-growing sector supporting cutting-edge banking and monetary activities.
One of big data’s defining characteristics is its constant expansion. The IoT is no different. Every time your fridge takes a photo of its shelves (a shelfie?), or your toothbrush uploads data about your brushing techniques, new information is piled on top of existing archives to build a more detailed picture. Your dentist might only need to know about your typical brushing techniques once a year when your check-up is due. Yet fintech specialists like stock market investment companies need to be able to recognize pattern changes over the last twelve seconds, never mind the last twelve months. Incredibly sophisticated analytics tools have been developed to detect fluctuations in data patterns and predict future events, in ways even the world’s finest statisticians couldn’t dream of.
Looking into the Future
Big data will be used in many applications in the years to come. Autonomous vehicles and neurobiology are prime examples, with the human brain’s sheer complexity currently foiling attempts to map it at a cellular level. Big data will never be sexy, and you won’t see it promoted on an advertising billboard. But it’ll probably determine which ads appear on that billboard. Next time you’re poring over sales flowcharts trying to spot patterns, you’re dipping a toe into an ocean of big data. And it’s an ocean that’s constantly expanding, as today’s digital world generates more and more data that requires analyzing and interpreting.