Get ready to hear a lot about “Dark Data” in 2018. Gartner defines this as, “the information assets organizations collect, process and store during regular business activities, but generally fail to use for other purposes (for example, analytics, business relationships and direct monetizing).”
In other words, dark data is the data being collected but not used, and every organisation has it. Within it is information that has potential for creating new revenue opportunities. An example of this is unused customer information, such as internet interactions where feedback and comments about are left. This goldmine of customer information exists, but left unanalysed returns no value.
Research firm, International Data Corporation forecast that worldwide revenues for big data and business analytics would grow from $130.1 billion in 2016 to more than $203 billion in 2020. With this strong focus on data and its analysis, exploration of ways to tap into dark data assets looks set to be become a big deal this coming year (cue #DarkData).
Dark data is not trivial. The unanalysed log file from one department may be highly valuable to another. When an organisation has this unanalysed raw data it undermines and restricts the usefulness of the existing analytics/Business Intelligence systems. So the enterprise that solves its own dark data puzzle will have an advantage from more complete information, leading to more accurate business planning.
Further, with the arrival of GDPR in May 2018 it will be vital for organisations to find and remove currently invisible data when requested. Consider a former employee whose data should have been removed, yet his/her National Insurance number exists in a long since forgotten spreadsheet in HR, and now there has been a data breach. With penalties reaching €20 million or 4% or annual global turnover – whichever is higher – dark data discovery should be high on every organisation’s agenda.
So where do you start? It may seem to be a daunting task but there are solutions with APIs such as Cognitive Services. Also, customising SQL Server can deliver an enterprise application to analyse specified dark data which can then be integrated with the existing big data. For more information on solving your dark data problem give us a call on 01342 410223 or email firstname.lastname@example.org.