The Price You Pay for Poor Data Quality

When vacation-goers booked flights with Hawaiian Airlines last spring, they were surprised to find that their tickets — which were intended to be free award flights — actually cost tens of thousands of dollars. The culprit of this was a faulty airline booking application that accidentally charged customer accounts in dollars instead of airline miles. A ticket that was supposed to be redeemed for 674,000 miles turned into a sky-high price of $674,000 USD!

This is yet another example of the impact that poor data quality can have, sometimes with these types of embarrassing results. The value of a company can be measured by the performance of its data; however, data quality often carries heavy costs in terms of financial, productivity, missed opportunities and reputational damage.

The Financial Cost of Data Quality

Erroneous decisions made from bad data are not only inconvenient but also extremely costly. According to Gartner research, “the average financial impact of poor data quality on organizations is $9.7 million per year.” IBM also discovered that in the US alone, businesses lose $3.1 trillion annually due to poor data quality.

Data Quality’s Cost to Productivity

This all goes beyond dollars and cents. Bad data slows employees down so much so that they feel their performance suffers. For example, every time a salesperson picks up the phone, they rely on the belief that they have the correct data – such as a phone number – of the person on the other end. If they don’t, they’ve called a person that no longer exists at that number, something that wastes more than 27 percent of their time.  

Accommodating bad data is both time-consuming and expensive. The data needed may have plenty of errors, and in the face of a critical deadline, many individuals simply make corrections themselves to complete the task at hand.

Data quality is such a pervasive problem, in fact, that Forrester reports that nearly one-third of analysts spend more than 40 percent of their time vetting and validating their analytics data before it can be used for strategic decision-making.

The crux of the problem is that as businesses grow, their business-critical data becomes fragmented. There is no big picture because it’s scattered across applications, including on-premise applications. As all this change occurs, business-critical data becomes inconsistent, and no one knows which application has the most up-to-date information. 

These issues sap productivity and force people to do too much manual work. The New York Times noted that this can lead to what data scientists call ‘data wrangling’, ‘data munging’ and ‘data janitor’ work. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.

Data Quality’s Reputational Impact

Poor data quality is not just a monetary problem; it can also damage a company’s reputation. According to the Gartner report Measuring the Business Value of Data Quality, organizations make (often erroneous) assumptions about the state of their data and continue to experience inefficiencies, excessive costs, compliance risks and customer satisfaction issues as a result. In effect, data quality in their business goes unmanaged.

The impact on customer satisfaction undermines a company’s reputation, as customers can take to social media (as in the example at the opening of this article) to share their negative experiences. Employees too can start to question the validity of the underlying data when data inconsistencies are left unchecked. They may even ask a customer to validate product, service, and customer data during an interaction — increasing handle times and eroding trust.

Case Study: Poor Data Quality at a Credit Card Company

Every time a customer swipes their credit card at any location around the world, the information reaches a central data repository. Before being stored, however, the data is analyzed according to multiple rules, and translated into the company’s unified data format.

With so many transactions, changes can often fly under the radar:

• A specific field is changed by a merchant (e.g., field: “brand name”).
• Field translation prior to reporting fails, and is reported as “null”.
• An erroneous drop in transactions appears for that merchant’s brand name.
• A drop goes unnoticed for weeks, getting lost in the averages of hundreds of other brands they support.

Setting back the data analytics effort, the data quality team had to fix the initial data and start analyzing again. In the meantime, the company was pursuing misguided business strategies – costing lost time for all teams, damaging credibility for the data analytics team, adding uncertainty as to the reliability of their data and creating lost or incorrect decisions based on the incorrect data. 

Anodot’s AI-Powered Analytics solution automatically learns normal behavior for each data stream, flagging any abnormal behavior. Using Anodot, changes leading to issues such as null fields would be immediately alerted on, so that it could be fixed. This prevents wasted time and energy and ensures that decisions are made based on the complete and correct data.

Applying Autonomous Business Monitoring to Ensure Good Data Quality

Reducing the causes of poor data is crucial to stopping the negative impact of bad data. An organization’s data quality is ultimately everyone’s business, regardless of whether or not they have direct supervision over the data.

Artificial Intelligence can be used to rapidly transform vast volumes of big data into trusted business information. Machine learning can automatically learn your data metrics’ normal behavior, then discover any anomaly and alert on it. Anodot uses machine learning to rapidly transform vast volumes of critical data into trusted business information.

Data scientists, business managers and knowledge workers alike all have a responsibility to implement the best tools to ensure that false data doesn’t impact critical decisions.