# Dark Data

## Summary


Organizations typically collect, process and store lots of information in the normal course of their operations. A lot of this is done simply for compliance purposes. However, the same data could be useful to plan product roadmaps, aid business decisions or optimize operations. This is possible today due to modern techniques of machine learning and data analytics. 

**Dark data** is simply data that's available to organizations but not being used. The term "dark" does not refer to something evil or illegal. Nor is it specifically about security or privacy. Rather, it's about data that's hidden from view, easy to ignore, and hard to access or analyse.

In physics, we know that dark matter comprises most of the universe. Similarly, dark data is lot bigger than data that organizations typically analyse to aid decision making.

## Discussion

### Could you explain dark data with some examples?

Consider a banking application for credit card approval. The focus is on customer information and eligibility. However, how the user arrived at the online form could be useful information. This data is available but is not used. 

In medical domain, paper records are digitized. These are used by doctors. Since they're are stored as scanned images, information retrieval or analysis systems ignore them. 

Another example is when a business acquires users or gathers feedback via different channels. Call centre logs are captured as audio files whereas website visits are captured as web server logs. This is another source of dark data when new insights are inaccessible just because data sources exist independently and can't be easily combined. 

Consider an Accounts Payable department. Dark data could include overviews in the ERP system, previous transactions, account history, CRM information, and so on. This data is not just unused but also presents a risk due to its sensitive and confidential nature. 


### What are the different types or sources of dark data?

Dark data is different in each industry. Some categories of dark data include customer information, ex-employee information, log files, survey data, financial statements, notes, presentations, emails, email attachments, inactive databases, old versions of documents, call-centre transcripts, customer reviews, etc. 

Dark data has come about because organizations realize that data is valuable and storage is cheap. So they end up storing lots of it without using them properly. Data lakes and their associated technologies help store large volumes of multiformat data. 

An important reason for dark data is big data. Big data's traits (volume, variety and velocity) make it difficult for organizations to process it. So they store it and defer the processing until later. From this perspective, the term "dusty data" has been used. 

Where credibility of data is essential, data that can't be traced to its source won't be used for analysis. 


### How does dark data differ from unstructured data?

Often dark data is unstructured because unstructured data is harder to analyse. Unstructured data "becomes dark" when organizations don't have the know-how to analyse them and obtain insights. 

However, structured data can also be part of dark data. For example, two structured datasets might exist independently in their own silos. If they were to be combined, new insights could be obtained. Combining two datasets might not be trivial, particularly when they exist in different formats, stored in different systems or controlled by different teams. 

In general, dark data can be structured or unstructured but they share some common characteristics. Data might be redundant, obsolete or trivial. This view expands the definition of what's dark data. It's not just valuable data that's not used but includes useless data that's simply taking up storage space. 


### How can I make use of or analyse dark data?

Those who manage dark data should regularly do audits and attempt to structure the data. You may decide not to dump dark data but it should be encrypted and stored in a secure manner. For unstructured data, at least attach metadata labels that so they can be easier to find for future analysis. Classify or organize data in a single repository so that users can find them faster. 

Before dark data can throw up insights, you have to ask the right questions, questions that are relevant to your business. The question could be about 'What' or 'Why'. Each type of question might need a different approach. 

Tools to work with dark data include DeepDive, Snorkel, and Dark Vision. Docugami is a company that's looking at ways to enable computers to understand documents written by and for humans. They call the problem "document dysfunction". 

Any effort to use dark data manually will be almost impossible. Automation is the key. Robotic Process Automation (RPA) can be enhanced with cognitive abilities to understand dark data. 

## Milestones

Jul  
2012

A researcher at Gartner mentions the term *Dark Data* in a blog post. This may well be the first mention of the term. He notes that dark data is stored "just in case it might come in handy" at a later point. To get any value out of dark data, it's important that business users ask the right questions. In subsequent literature, Gartner is credited with coining the term. 

Mar  
2013

Gartner Research publishes a report titled *Innovation Insight: File Analysis Innovation Delivers an Understanding of Unstructured Dark Data*. Two months later, Gartner publishes a separate article on *Dark Data* in its online glossary of information technology terms. 

2015

IBM reports that 90% of IoT sensor data is not used. Moreover, 60% of this data loses its value within milliseconds. Technologies that allow us to process this data in almost real time is therefore critical. If not, we end up with dark data. 

2016

A study by Veritas involving 22 countries shows that 52% of all data collected by organizations is dark. They say it's data "beneath the line of sight of senior management". 

Jan  
2017

Chan Zuckerberg Initiative (CZI) acquires **Meta**, which is a search engine focused on scientific literature. This is an effort to unlock the value in dark data. 

May  
2017

Apple acquires **Lattice Data** that specializes in dark data. Lattice Data commercialized DeepDive, a tool developed at Stanford University. DeepDive uses machine learning to convert dark data into structured data that can be combined within existing structured data sources. DeepDive extracts relationships and makes inferences. It's development can be traced back to 2014. 

2018

Some folks extend the definition of dark data to include data that's hidden behind firewalls or not accessible by search engines. Thus, dark data includes data from the deep web as well.