Can Your Big Data Company Forego Anomaly Detection?

While enterprise leaders are constantly looking to innovate, there’s one area where “business as usual” should be a focus — spotting anomalies in your data.

When it comes to time-series data, “business as usual” is the baseline or expected behavior of the KPIs you track. Any unexpected deviations in those patterns can be classified as anomalies. However it’s important to keep in mind that anomalies can be either negative or positive.

Negative vs. Positive Anomalies

Negative anomalies could be extreme spikes in web traffic that point to DDoS attacks. When Blizzard Entertainment was hit by a DDoS attack in 2017, the downtime lasted an entire weekend — all because the company couldn’t track the root cause of the anomaly. Attacks like these can cost an enterprise more than $2 million, which makes anomaly detection essential to protecting revenue.

Positive anomalies can help you spot opportunities for growth. When MeUndies started experimenting with lookalike audiences in Facebook and Instagram ads, they saw an upticks in sales. Detecting an outlier in sales performance alongside those marketing experiments compelled MeUndies to increase ad spend and drive a 97 percent increase in incremental purchases.

The Challenge of Detecting Anomalies

In either case, some anomalies may be easier to spot than others. At times, you’ll see massive deviations in your data. And then there are those subtle, unnoticed glitches that can cause small revenue leaks, amounting to millions of dollars over time.

Using root cause analysis to identify anomalies like these and take action is nothing new. However, the sheer speed and volume of big data have created unique challenges that make automated anomaly detection more important than ever.

Detecting anomalies in big data is humanly impossible

You’ve been manually tracking anomalies in key business KPIs for years. But your time series data is becoming so complex that your traditional processes can’t keep pace. Tracking the thousands (or even millions) of unique metrics across your business make automated solutions a must.

For 80 percent of executives, the overload has led to significant investment in business intelligence tools like visualization, dashboards, and reporting.

The problem with traditional BI tools lies in anomaly detection, especially in:

• Unactionable Insights: Traditional BI tools provide general visibility into business data, but don’t supply the insights you can use to cut time to remediation.
• Over-Simplification: Dashboards give you big-picture overviews, totals, and averages for your business metrics, but fail to uncover isolated, significant anomalies.
• Static Thresholds: Basic BI tools don’t actually detect anomalies; they simply alert you when a metric hits your manual thresholds. Given how a metric’s behavior can fluctuate due to seasonality or complex factors, this set-up is a recipe for alert storms.
• Reactive Monitoring: Traditional BI tools suffer from latency and can’t provide real-time insights, which means incidents are surfaced later than needed and you may miss critical opportunities.

These are just a few reasons BI tools don’t act as an adequate substitute for dedicated anomaly detection. Already, 59 percent of executives recognize that their big data initiatives would improve if they included AI.

To get the most out of their big data stack, many companies are integrating AI-based anomaly detection that utilizes machine learning and works in real time  — both to handle large volumes of data and to address opportunities proactively.

3 key attributes of advanced anomaly detection

Regardless of the KPIs contained in your time series data, advanced anomaly detection is all about finding and fixing incidents as they’re happening. Identifying the unknown unknowns within big data will help you plug revenue leaks, capitalize on opportunities to increase sales and proactively address incidents that impact customer experience.

With artificial intelligence enabling powerful new capabilities in the analytics space, there are now anomaly detection solutions that provide:

• Dynamic Monitoring: AI-powered anomaly detection is more accurate. Rather than setting manual thresholds, you can use machine learning algorithms to create a dynamic baseline. They continuously analyze your business data to understand its normal behavior under different conditions and seasons, or what constitutes “business as usual”.
• Real-Time Analysis: With dynamic baselines for business data, advanced anomaly detection can correlate incidents to root causes faster than traditional monitoring tools. And once root causes are identified, real-time analysis gives you a prioritized set of opportunities to cut time to remediation.
• Holistic Visibility: Unlike BI tools, advanced anomaly detection sees beyond traditional data silos. By integrating data from applications, databases, storage, CRMs, monitoring and analytics tools, IT infrastructure and more, you benefit from cross-metric correlation that uncovers more granular insights.

Without these advanced anomaly detection features, you could invest millions of dollars in big data solutions just to have business incidents slip through the cracks and hurt your ROI. In fact, many enterprises are already supplementing basic BI tools with anomaly detection that uses AI to operate autonomously.

The real-world impact of anomaly detection

There are many different ways that large enterprises can use autonomous anomaly detection to improve their businesses.

For AppNexus, AI-powered anomaly detection enabled agents to proactively address performance issues for their adtech clients. Rather than having data scientists manually sifting through big data, autonomous anomaly detection pointed out revenue leaks and allowed agents to optimize transactions in real time.

In other cases, real-time anomaly detection can make all the difference in preventing business disasters. One global telco operator recognized that traditional monitoring wouldn’t maintain network performance as voice, video, and data usage skyrocketed in recent years. With millions of time series datasets to monitor, a single anomaly could result in total outages and lost business. Using advanced anomaly detection streamlined processes and delivered granular insights to help the telco mitigate damages before they became disasters.

The benefits of AI anomaly detection also extend to customer experience. LivePerson was struggling to analyze millions of unique metrics in real time to maintain quality experiences for 18,000+ customers. But with advanced anomaly detection, the engineering team discovered isolated issues on platforms and features that could have driven customers to their competitors if left unresolved.

But not every use case is about resolution. For eCommerce companies looking to increase revenue, anomaly detection can be used to analyze sales spikes. Before implementing advanced anomaly detection, one eCommerce brand relied on intuition to analyze sales spikes presented by dashboards. But with anomaly detection, sales and marketing were able to present practical opportunities for growth to decision makers.

—

In each of these examples, big data alone wasn’t enough to optimize business performance. If you want to maximize your big data ROI, implement advanced anomaly detection and get the granular insights that will drive success.