Two Secrets of Swift and Scalable Anomaly Detection

Streamlining the way your digital business works – ensuring customers get what they need, conversions occur without a glitch, etc – is the goal of any company with an online presence, but it can be challenging with so many moving parts. As we discussed in a previous post, detecting anomalies is one of the best ways to make sure that everything is running smoothly and keep up with current trends.

In digital businesses business, many processes happen simultaneously, and each activity may be monitored by a different person or team. Changes in different departments or even external partners can show up as an unexpected change in a totally different area, but the association might never be made if the metrics are not analyzed on a holistic level.

The solution? An anomaly detection system that can understand all these different types of metrics, identify the normal behavior and alert when something has changed.

When designing an anomaly detection system, there are certain principles within the design that are essential to its success. This post will give an overview of two of those secrets to success: timeliness and scale. In future posts, we’ll take a look at the other three key principles: Rate of Change, Conciseness and Definition of Incidents.

How quickly do you need your anomalies detected?

In anomaly detection, there are two types of decision making. First, detection can be done in non-real-time, meaning that the results are retroactively seen by the user. In this case, the anomalies are used for a retrospective analysis of what happened, which helps in making decisions about the future. The other option is real-time detection, where you see the results of metrics as they happen.

When would you want non-real-time decision making? This model is useful for long term planning. Basically, the data received and understood is not relevant to the immediate situation of the company, and is not necessary for immediate action. An example of when you might use non-real-time decision making is when reviewing data from marketing campaigns to plan future strategy, scheduled maintenance, budget planning, etc.

In this situation, data is collected over a period of time, and when that period finishes, a batch machine learning algorithm can be used to find out what anomalies occurred during the set amount of time. While viewing these results in non-real-time, your business can see the results of a longer course of action and thus make non-urgent decisions for future action.

However, most online businesses are in dire need of real-time decision making. For example, sudden spikes or dips in purchases could present opportunities for action that would generate more sales. Knowing exactly what is going on with your digital business at the moment that it is happening enables you to take advantage of real-time trends for the furtherance of your business goals.

Online machine learning algorithms are the best way to process data in real-time. Using these algorithms also helps in our next point.

Scaling for Growth

Online machine learning algorithms are easily scalable, thus making them ideal for large data sets. However, online machine learning algorithms are not without their faults. They tend to be more prone to false positives.

If your company is continuously growing, then scalability is a valid concern. Thus, online machine learning algorithms are still the best option for businesses that have more metrics and large data sets. There are ways to reduce false positives, which we discuss in our White Paper “Building a Large Scale, Machine Learning-Based Anomaly Detection System, Part 1: Design Principles.”

Conclusion

Online machine learning algorithms are a viable solution to the needs of businesses in the digital age. As we’ve explained, real-time decision making and ability to scale are two of the secrets of building a successful online machine learning anomaly detection system.

For more information about the design principles of an anomaly detection system, read the full white paper: Building a Large Scale, Machine Learning-Based Anomaly Detection System, Part 1: Design Principles.