Real-Time Anomaly Detection: Solving Problems and Finding Opportunities

Success in today’s high-velocity business environments means having the correct information to make the right decisions at the right time. As marketplaces grow more competitive and customer expectations continually rise, the “right time” is often real-time. Every transaction generates a plethora of data.

Anomalies within your company’s data set can represent opportunities and threats to the business. Real-time detection of anomalies empowers enterprises to make the right decisions to seize revenue opportunities and avoid potential losses.

What are the main types of anomalies?

There are three commonly accepted types of anomalies in statistics and data science: Global outliers, contextual outliers, and collective outliers.

1. Global outliers represent rare events that likely have never happened before. An example might be if a customer spends no more than $200 per week on e-commerce purchases, suddenly spending $10,000 in a single day.
2. Contextual outliers represent events that fall within the normal range from a global sense but are abnormal in the context of seasonal patterns. If a customer only ever spends $2500 every December on gifts but racks up $2000 in charges in July, it would be considered a contextual anomaly. While that month’s spending isn’t outside their normal global range, it occurs at an unusual time.
3. Collective outliers represent events that on their own do not fall outside of the standard expected behavior, but when combined, represent an anomaly. A group of customers with a history of order cancellations all canceling their orders at the same time would be a collective outlier.

The limitations of manual anomaly detection

In the past, when businesses only had a handful of metrics to track across their business, manual monitoring methods were feasible. Now, there are potentially millions of metrics to manage and multiple types of anomalies to consider and evaluate. On top of the inherent complexities, many real-life business anomalies require immediate action.

A bad software update could cause a business to lose money every second. And since discovering the problem is the first step in resolving it, eliminating the delay between when the problem occurs and when the problem is detected immediately brings you one crucial step closer to rolling back that update and restoring revenue flow.

Manual detection is also insufficient when the anomaly represents an opportunity rather than a problem. For example, an unusual uptick in mobile app installations from a specific geographical area may be due to a successful social media marketing campaign that has gone viral in that region. Given the short lifespan of such surges, your business has a limited time window to capitalize on this popularity and turn all those shares, likes, and tweets into sales.

Even when anomalies don’t require an immediate response, manual anomaly detection and dashboards can fall tragically short. You can always postpone action on an instant alert, but you can never react to a delayed alert in real-time.

The Secrets of Fast and Scalable Autonomous Anomaly Detection

If manual anomaly detection is inadequate, then automated anomaly detection must be used to achieve real-time anomaly detection at scale. Incremental machine learning algorithms for anomaly detection have the added benefit of scalability.

In anomaly detection, there are two paradigms. In the first, a system detects an anomaly that has already occurred and displays it in a traditional dashboard using batch machine learning algorithms. That’s only useful as a retrospective on what’s happened in the past to inform future decisions. This has some value with long-term planning, but most online businesses need real-time decision-making to seize opportunities and prevent negative impacts.

For example, sudden spikes or dips in purchases could present opportunities for action to generate more sales. The only way to take advantage of real-time trends is to know what’s going on at the moment that it’s happening. That requires automated, fast, and scalable anomaly detection in real-time.

In this example, an ecommerce company was alerted as soon as there was an unusual drop in approval rates for PayPal payments. Fixing the issue quickly prevented a significant loss in revenue.

Scaling for Growth

Incremental machine learning algorithms are easily scalable, thus making them ideal for the incomprehensibly large data sets of today’s businesses. If your company is continuously growing, then scalability is a valid concern. Incremental machine learning algorithms are the best option for companies with more metrics and large data sets.

Advanced Anomaly Detection

Systems that use advanced anomaly detection are more effective than those that use more straightforward techniques. A Gartner report on advanced anomaly detection explained that “They can detect subtle anomalies that might otherwise escape notice, provide earlier warning of impending problems or more time to capitalize on emerging opportunities, reduce the human effort required to develop a monitoring or measuring application, and reduce the time to solution for implementing complicated anomaly detection systems.”

Many systems try to find events and data outside the normal range to identify opportunities and threats. Yet, often these systems fail, identifying too many anomalies (false positives) or not enough (false negatives). Applying machine learning to rapidly changing environments means constantly updating and training models incrementally to prevent false positives.

Our AI analytics solution offers autonomous analytics with contextualized alerts that minimize false positives. Instead of your data scientists asking many questions and carrying out complex data analysis, Anodot does the work and provides the Autonomous Business Analytics answers to understand why an incident happened.

This real customer example illustrates how Anodot immediately detected a drop in completed purchases for an online store and correlated the anomalous events to a specific product and device in order to expedite remediation.

AI/ML-based anomaly detection in the real world

Scientists already harness the power of AI and machine learning to spot anomalies and the opportunities they present far faster than humans ever could on their own. An AI system developed by NASA’s Jet Propulsion Laboratory was able to detect and command an orbital satellite to image a rare volcanic event in Ethiopia – before volcanologists even asked NASA for that satellite to take images of the eruption.

When working with thousands or millions of metrics, real-time decision-making requires incremental machine learning algorithms. Whether it’s saving your business money or gleaning scientific insights from a brief volcanic eruption, real-time anomaly detection has enormous potential for catching critical deviations in data sets that can have tremendous real-world impacts.