#GlitchHappens. That’s an unavoidable consequence of the scale and speed of ecommerce today, especially when lines of code set and change prices in seconds. Unavoidable, however, doesn’t have to mean catastrophic, especially if a real-time anomaly detection system is deployed. In two real-world glitch incidents, we’ll see the cost of not employing real-time automated anomaly detection in ecommerce.
Our connected world is made possible not only by glass threads pulsing with data, but also by the connections between vendors, clients, consumers and government agencies which enable goods, services, and even financial assistance to be delivered to those who need them. As we’ll find out, Walmart learned a break in that chain anywhere can lead to pain everywhere.
EBT spending limits hit the roof…and Walmart picks up the tab
Due to a series of failures in the electronic benefit transfer (EBT) system in certain areas, the system allowed card holders to make food purchases at retailers, but without spending limits. Even though Walmart’s management realized something was wrong, they still decided to allow all these EBT purchases, rather than deny food to low-income families.
In Walmart stores in two Louisiana cities in particular, entire shelves were emptied as EBT shoppers brought full shopping carts to the checkout. Some Walmart stores in the are were forced to close due the number of customers inside exceeding fire safety limits. All of this occurred in the narrow two-hour window the EBT spending limit glitch lasted. When the system was fixed and Walmart announced the spending limits were restored, some shoppers were forced to abandon their carts. In one case, a woman with a forty-nine cent balance on her card was stopped just as she approached the checkout with $700 of food in her cart.
The cumulative cost inflicted by the glitch hasn’t been publicly disclosed, but if the $700 example is representative, those Louisiana Purchases probably reached six figures.
An anomaly detection system would have been tremendously helpful
Had Walmart been using Anodot to monitor all of the metrics across the company, a number of anomalies would have been detected in minutes in the individual store-level time series data:
- On-shelf inventory of food items (sudden decrease)
- Sales volume (sharp increase)
- Volume of EBT transactions (large increase)
- Average dollar amount of EBT transactions (skyrocketing increase)
Not only would Anodot’s anomaly detection system detect each of these individual metrics, but it would also combine these separate signals into an actionable alert which told the complete story of what was going on. It would become clear that the anomalies originated from the same stores and within a single state; therefore, Walmart would have known within minutes that there was a multi-store problem with EBT in Louisiana.
Walmart’s EBT glitch shows the potential corporate damage of a massive volume of glitch purchases. A glitch incident at Bloomingdale’s, however, shows that the other extreme occurs too: a much smaller volume of high-dollar glitch purchases.
Bloomingdale’s bonus points glitch
A simple coding error in the software powering Bloomingdale’s “Loyalist” points system caused store credit balances to equal the point balances, not the equivalent cash value of those point balances. Since the two are separated by a factor of 200, this left a few Bloomingdale’s shoppers pleasantly surprised.
Word quickly spread on social media informing more customers of the opportunity. Some made online purchases, many of which were canceled by Bloomingdale’s after the glitch was discovered and fixed a day later.
Yet, like the Walmart glitch, it’s the in-store purchases which really inflict the most damage during a glitch. One man spent $17,000 on in-store purchases, but could have walked away with even more merchandise since the glitch gave him $25,000 of credit.
How Anodot could have prevented this bug from blooming
Anodot would have detected a sudden, large jump in the gift card value to points ratio as soon as that data was reported and fed into the real-time anomaly detection system. Bloomingdale’s could have then temporarily disabled the Loyalist points of the affected accounts. Anodot would have also detected the large, sudden uptick in the average dollar value of online purchases made with Loyalist points, both online and at physical stores.
The increase of both Bloomingdale’s mentions on social media and gift card usage would have been correlated with the other anomalies to not only show the specific problem, but also draw attention to the fact that many, many people were actively taking advantage of it. Quickly and definitively proving that this was a business incident which needed to be fixed now.
Shoppers today are savvy users of online deal sharing websites and social media, and wield these tools to help themselves and others instantly pounce on drastic discounts or free buying power, regardless if it’s from a legitimate promotion or system glitch. In this commerce environment, companies must react even faster to identify, contain and fix glitches when they happen.
Not everyone will stop at $17,000.
Written by Anodot
You'll believe it when you see it
