Have You Got Black Swans Swimming in Your Big Data?

9/11, Hurricane Katrina, the 2008 credit crisis, the BP Gulf oil spill in 2010, and the Japanese earthquake and resulting tsunami that put the Fukashima nuclear facility offline. These are all recent examples of what are known as Black Swan events, incidents that occur randomly and unexpectedly and have wide-spread ramifications.

Black Swans are a hot topic for business too. The term puts a face on the type of event that companies fear most – the ‘Unknown Unknown’ that, despite all of the preparations that might have been made, organizations are still taken by surprise and often shaken to their core.

Plenty of information

Businesses are collecting all kinds of data, trying to detect potentially costly problems and discover hidden opportunities. Those problems and opportunities may appear as anomalies – data points that are unusual or unexpected – surfacing a change. They are the proverbial needles in a haystack. The problem is that the needle is sitting in an increasingly larger haystack.  Moreover, the subtle anomalies can just slip under the radar and go undetected for much longer.

The longer a subtle anomaly persists the greater the expense. Subtle anomalies, or micro glitches, can appear in the form of incorrect pricing, online service outages due to sudden spikes in website traffic and products being incorrectly priced or incorrectly coded for checkout.

Where can data analysts  look for insights in this needle in the haystack of business data?

The Arrival of the Black Swan

Before the discovery of Australia, people were convinced that all swans were white, a notion completely confirmed by empirical evidence. Then in 1697, the Dutch navigator, Willem de Vlamingh, found black swans in Western Australia, showing how one can’t so quickly declare something impossible.

The phrase, black swan event, now illustrates the frailty of inductive reasoning and the danger of making sweeping generalizations from limited data.

Black Swan Outliers

Outliers are normally defined as any data point that lies far outside the expected range for a value. These often expensive outliers are considered “black swans,” after Nassim Nicholas Taleb characterized the expression in his 2007 book “The Black Swan.” Taleb defines black swans as rare, high-impact events that seem improbable and unforeseeable but, in hindsight, are explainable and he is a huge proponent of keeping an eye on data sets that could end up with such black swan type of events.

In many situations, data outliers are errant data which can skew averages, and thus are just filtered out and excluded by statisticians and data analysts before attempting to extract insights from the data. They may consider those outliers to be reporting errors or some other issue that they needn’t worry about. Also, since genuine outliers are considered to be relatively rare like a black swan, they aren’t seen as indicating a deeper, urgent problem in the system being monitored.

However, outliers do hold great significance. An outlier in a single metric could reflect a one-off event or a new opportunity, like an unexpected increase in sales for a key demographic.

Outlier Detection and Revenue Opportunities

Hidden in the huge mass of enterprise data are latent patterns. If only the data could be interpreted properly, then precious business secrets could be revealed.

For example, applying outlier detection to social media metrics can both impact the brand management front, and also discover new revenue opportunities. Positive buzz can probably be reflected in a sharp increase in unique visitors to a company’s online store, turning into a sharp surge in orders.

Yet one doesn’t always know why this is happening or just how to leverage this into an opportunity. That large spike in sales volume could mean also mean that a stampede of online shoppers are taking advantage of a costly pricing glitch. Then, the average revenue per transaction would actually be slipping, depending on the ratio of glitch sales to normal sales. At other times of the year, this subtle dip might actually be considered completely normal (like the normal retail slow periods which occur outside of the holiday season).  But not during a promotion! For that case, the low values of average revenue per transaction would be a contextual outlier.

For e-commerce companies, micro glitches are occurring in greater frequency, especially during the holidays. Like the black swan waiting to be discovered, these glitches can go unseen, while sucking thousands of dollars from the business. These losses manifest themselves in customers that grew frustrated by an inability to check out or faced a page error that left them unable to even read about an item they may be considering to purchase—then taking their business elsewhere.

Identifying the Black Swans in Real Time

Success in business hinges on making the right decisions at the right time. You can only make smart decisions, however, if you also have the insights you need at the right time.

While not the earth-shaking black swan events that have hit the world in recent years, under the burden of massive amounts of data, often updated in seconds, subtle anomalies can signal deeper problems. Quickly detecting and analyzing the outliers can help adjust course in time to seize opportunities or avoid losses.

Regardless of the industry, no matter the data source, the anomaly detection capabilities in Anodot’s AI-powered analytics can find all types of outliers in time series data, in real time, and at the scale of millions of metrics. This not only simplifies the detection and correlation of outliers in the data, but provides the real-time insights needed for real-world savings.