Shining A Light On ‘Dark (Big) Data’

18th August, 2016 by

The big data gold rush is prompting many companies to gather as much data as possible in the hope that it might be useful in future, but often those organizations already hold a treasure trove of useful data that they’re failing to make use of – so-called ‘dark data’.

You might have historical sales records and customer information locked up in sales or CRM systems, product design information sitting in stores of old CAD/CAM files, operational data from ERP systems or other stores of structured data secreted away on standalone systems.

Typically these individual big data silos are only accessible by the team that created or manages them. They may be indexed in an idiosyncratic fashion, with no clear way to perform wider or comparative analyses on the information they contain, analyses which could well give you valuable new insights into how you might design better products, build more fruitful relationships with customers, improve your productivity, and otherwise boost your business.

A New Layer Of Intelligence

So how do you start shining a light on dark data in order to uncover the jewels that potentially lie within? For most companies, it’s not practical to reformat all of their big data, so it conforms to a common structure. Rather, the answer is to add a new intelligent layer that draws in data from all these old silos, freeing it up and allowing it to be usefully searched by anyone in the organization.

One of the best ways to do this – and to help break down organizational silos in the process – is to enable your employees to become curators of the data, adding metadata to this new layer that gives the old information new context. Typically, this would conform to a far looser taxonomy than the highly structured formats demanded by the individual systems where the data resides. For example, every time someone pulls up an old record or creates a new one, they should add their own descriptive tags and keywords that gives the information more context and helps it to be found by others in the organization.

Let Your Big Data Manage Itself

It may seem a daunting task to index multiple stores of dark data in this way, even in an unstructured way, but as you adopt this method for new data you create new opportunities. Modern big data management tools can increasingly do much of the work for you. For example, this may entail learning what keywords and tags people tend to use, and then intelligently applying similar tags to historical data automatically, according to the textual content of that data. They might not get it 100% right all the time, but the more data that’s indexed in this way, the more accurate they become.

Lifting The Veil

Once the data is universally accessible and contextualized with tags and keywords, you can put increasingly more sophisticated data analysis tools to work on that data. By doing so you are able to look at it holistically, uncovering hidden patterns and insights to lift the veil of darkness and ensure your business’s future is ever brighter.