ICAEW.com works better with JavaScript enabled.

Dark data: from overload to insight

Businesses today have the opportunity to capture and analyse unprecedented amounts of data – from internal systems to the internet, sensors, satellites, social media and beyond. By analysing data about the needs, preferences and actions of customers, suppliers and competitors, businesses can replace guesswork and assumptions with hard evidence, transforming their decision-making. And yet, in most cases, businesses are not making the most of this data. According to research company IDC, 90% of unstructured data is never analysed by businesses.

Practical data challenges

There are many reasons for data remaining ‘dark’ and unexplored by businesses. In many cases, disparate and unintegrated legacy systems make it difficult to use data across the business. This is often compounded by a lack of data standards. As a result, most data projects start with a lot of work to extract, clean and standardise data before any analytics can get started. This takes time and money that can be difficult to justify in the short-term. 

But most businesses are also hampered by real difficulties in connecting data with their strategy and operations – I have all this data, but what do I with it? This is still a common experience of many businesses. As data volumes look set to increase rapidly over the coming years, businesses need to get a handle on its value if they are to avoid being entirely overwhelmed. And, where competitors are using data to improve their services and operations, this will be a growing competitive disadvantage. 

While we may think of the digital ‘FANG’ companies (Facebook, Amazon, Netflix and Google) as the natural leaders in the field, there are many examples of more traditional businesses exploiting big data. Airlines, for example, can purchase data about customer end-to-end journeys to pinpoint profitable new routes. Manufacturers can predict when assets need maintenance based on granular data about their condition. Small businesses can also make better use of data – one example shows cheap sensors being used to record footfall past the shop, and the business changing their opening hours to take advantage of more passing trade.  

Start with a question

The irony is that using data effectively doesn’t start with data. It starts with business questions. What do you want to know about your customers, their preferences or their use of your products? What about the risks or performance of suppliers? Or how about the development of new products and what the market really wants?

By being forensic about what you want to know, you can then move onto what data is available and what analysis you can do. It’s important to think broadly – looking at external data as well as internal data, and identifying where you could capture new data.  In many cases, good visualisation can then tell a story from the data, showing outliers and trends. Sometimes, more sophisticated statistical models can patterns or build predictions. But it all comes back to knowing how the data will help your business decisions.

Building the right skills

Making the most of data also needs a mix of different skills. There is certainly a need for IT, data and statistics skills to work with the data and analytical models. But working out the right questions and interpreting the results also needs good business understanding, as well as strong critical thinking and scepticism. This skills set is a natural fit for many Chartered Accountants, who typically combine a strong overview of how the business works and is controlled, with attention to detail and a focus on data quality. Given this background, they can work effectively with both data specialists and other business functions to deliver greater value from data.

There are also importance cultural aspects around data, in particular the need to move it from a ‘technology’ box to mainstream decision-making. Given the mix of skills, a collaborative culture and cross-departmental working is important. The change to decision culture can also be challenging to management, as it can replace many years of experience with hard data and evidence. As a result, leadership and commitment to the value of data is a critical success factor for this wider organisational transformation.