When vacation-goers booked flights with Hawaiian Airlines last spring, they were surprised to find that their tickets -- which were intended to be free award flights -- actually cost tens of thousands of dollars. The culprit of this was a faulty airline booking application that accidentally charged customer accounts in dollars instead of airline miles. A ticket that was supposed to be redeemed for 674,000 miles turned into a sky-high price of $674,000 USD! This is yet another example of the impact that poor data quality can have, sometimes with these types of embarrassing results. The value of a company can be measured by the performance of its data; however, data quality often carries heavy costs in terms of financial, productivity, missed opportunities and reputational damage. The Financial Cost of Data Quality Erroneous decisions made from bad data are not only inconvenient but also extremely costly. According to Gartner research, “the average financial impact of poor data quality on organizations is $9.7 million per year.” IBM also discovered that in the US alone, businesses lose $3.1 trillion annually due to poor data quality. Data Quality’s Cost to Productivity This all goes beyond dollars and cents. Bad data slows employees down so much so that they feel their performance suffers. For example, every time a salesperson picks up the phone, they rely on the belief that they have the correct data - such as a phone number - of the person on the other end. If they don’t, they’ve called a person that no longer exists at that number, something that wastes more than 27 percent of their time.   Accommodating bad data is both time-consuming and expensive. The data needed may have plenty of errors, and in the face of a critical deadline, many individuals simply make corrections themselves to complete the task at hand. Data quality is such a pervasive problem, in fact, that Forrester reports that nearly one-third of analysts spend more than 40 percent of their time vetting and validating their analytics data before it can be used for strategic decision-making. The crux of the problem is that as businesses grow, their business-critical data becomes fragmented. There is no big picture because it’s scattered across applications, including on-premise applications. As all this change occurs, business-critical data becomes inconsistent, and no one knows which application has the most up-to-date information.  These issues sap productivity and force people to do too much manual work. The New York Times noted that this can lead to what data scientists call ‘data wrangling’, ‘data munging’ and ‘data janitor’ work. Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets. Data Quality’s Reputational Impact Poor data quality is not just a monetary problem; it can also damage a company’s reputation. According to the Gartner report Measuring the Business Value of Data Quality, organizations make (often erroneous) assumptions about the state of their data and continue to experience inefficiencies, excessive costs, compliance risks and customer satisfaction issues as a result. In effect, data quality in their business goes unmanaged. The impact on customer satisfaction undermines a company’s reputation, as customers can take to social media (as in the example at the opening of this article) to share their negative experiences. Employees too can start to question the validity of the underlying data when data inconsistencies are left unchecked. They may even ask a customer to validate product, service, and customer data during an interaction — increasing handle times and eroding trust. Case Study: Poor Data Quality at a Credit Card Company Every time a customer swipes their credit card at any location around the world, the information reaches a central data repository. Before being stored, however, the data is analyzed according to multiple rules, and translated into the company’s unified data format. With so many transactions, changes can often fly under the radar: A specific field is changed by a merchant (e.g., field: “brand name”). Field translation prior to reporting fails, and is reported as “null”. An erroneous drop in transactions appears for that merchant’s brand name. A drop goes unnoticed for weeks, getting lost in the averages of hundreds of other brands they support. Setting back the data analytics effort, the data quality team had to fix the initial data and start analyzing again. In the meantime, the company was pursuing misguided business strategies – costing lost time for all teams, damaging credibility for the data analytics team, adding uncertainty as to the reliability of their data and creating lost or incorrect decisions based on the incorrect data.  Anodot’s AI-Powered Analytics solution automatically learns normal behavior for each data stream, flagging any abnormal behavior. Using Anodot, changes leading to issues such as null fields would be immediately alerted on, so that it could be fixed. This prevents wasted time and energy and ensures that decisions are made based on the complete and correct data. [CTA id="076c8680-fa50-4cd7-b342-37f878bd14fc"][/CTA] Applying Autonomous Business Monitoring to Ensure Good Data Quality Reducing the causes of poor data is crucial to stopping the negative impact of bad data. An organization’s data quality is ultimately everyone’s business, regardless of whether or not they have direct supervision over the data. Artificial Intelligence can be used to rapidly transform vast volumes of big data into trusted business information. Machine learning can automatically learn your data metrics’ normal behavior, then discover any anomaly and alert on it. Anodot uses machine learning to rapidly transform vast volumes of critical data into trusted business information. Data scientists, business managers and knowledge workers alike all have a responsibility to implement the best tools to ensure that false data doesn’t impact critical decisions. Related Guides: Top 13 Cloud Cost Optimization: Best Practices for 2025 Understanding FinOps: Principles, Tools, and Measuring Success Related Products: Anodot: Cost Management Tools  

The global predictive maintenance market is expected to grow to $6.3 billion by 2022, according to a report by Market Research Future. However, a new paradigm is required for analyzing real-time IoT data. The Impact of Predictive Maintenance in Manufacturing Predictive maintenance, which is the ability to use data-driven analytics to optimize capital equipment upkeep, is already used or will be used by 83 percent of manufacturing companies in the next two years. And it’s considered one of the most valuable applications of the Internet of Things (IoT) on the factory floor. Benefits of Predictive Maintenance The CXP Group report, Digital Industrial Revolution with Predictive Maintenance, revealed that 91 percent of predictive maintenance manufacturers’ see a reduction of repair time and unplanned downtime, and 93 percent see improvement of aging industrial infrastructure. According to a PWC report, predictive maintenance in factories could: Reduce cost by 12 percent Improve uptime by 9 percent Reduce safety, health, environment, and quality risks by 14 percent Extend the lifetime of an aging asset by 20 percent Challenges in Leveraging IoT Data for Predictive Maintenance The CXP Group report provides examples in which, for companies like EasyJet, Transport for London (TfL), and Nestle, predictive maintenance and understanding can boost the efficiency of its technicians, benefit the customer experience, and improve unplanned downtime. Realizing these advantages is not without challenge. Most IoT data, according to Harvard Business Review, is not currently leveraged through machine learning, squandering immense economic benefit. For example, less than one percent of unstructured data is analyzed or used at all and less than half of structure data is actively used in decision-making. Because traditional Business Intelligence (BI) platforms were not designed to handle the plethora of IoT data streams and don’t take advantage of machine learning, use of BI reports and dashboards only periodically leads to late detection of issues. In addition, currently, alerts are set with static thresholds, leading to false alerts in case of low thresholds, and failed detection in the instance of high thresholds. What’s more, data may change by time of day, week or season with those irregularities being outside the limited scope of static thresholds. Last, BI platforms were not designed to correlate between the myriad of sensor data—a correlation that exponentially boosts the likelihood of detecting issues. Let’s say an engine that is about to fail may rotate faster than usual, have an unusual temperature reading and low oil level. Connecting the dots between these sensor readings through machine learning can multiply the likelihood of detection. Automated Anomaly Detection By addressing the deficiencies of existing BI platforms, Anodot’s automated anomaly detection paves the way for factories to realize the full value of predictive maintenance. Analyzing big data from production floors and machinery to deliver timely alerts, Anodot’s visualizations and insights facilitate optimization, empower predictive maintenance and deliver bottom-line results. Anodot leverages the IoT-generated stream data such as meter readings, sensor data, error events, voltage readings and more. Monitoring data over time—and in real-time—it uses machine learning to learn the metric stream’s normal behavior. It then automatically detects out-of-the-ordinary data and events, serving up diagnoses and making preemptive recommendations that represent significant cost savings on upkeep and downtime. Rejecting old-school static thresholds, Anodot identifies anomalies in data that changes over time, for example, with built-in seasonality. This dynamic way of understanding what is happening in real time detects and alerts for real issues, often much earlier than a static threshold would have alerted. Without manual configuration, data selection or threshold settings, this platform uses machine learning to automatically calibrate to achieve the best results. Anodot algorithms can control data of any size or complexity, such as seasonality, trends and changing behavior because they are sufficiently robust to handle an army of data variables, intelligently correlating anomalies whose connections may escape the limitations of a human observer. Conclusion: The Future of Predictive Maintenance with Anodot Predictive maintenance offers a hefty opportunity for factories to save money on maintenance, downtime and while extending the life of their capital equipment. Automatic anomaly detection, such as Anodot, offers the best way to expose and preempt issues that require real maintenance in real time.   Related Guides: Top 13 Cloud Cost Optimization: Best Practices for 2025 Understanding FinOps: Principles, Tools, and Measuring Success Related Products: Anodot: Cost Management Tools

Start measuring, monitoring, analyzing, experimenting and improving at the speed of light. Read more here about growth hacks for data-driven marketing.

Although user experience is very important and issues around UX and application performance sometimes relay to revenue loss, not all revenue loss can be seen when exploring user performance and not all user performance issues affect revenue.

In today's payment ecosystem, the ability to monitor and use payment data effectively represents a real competitive advantage. Intelligent payment operations enables organizations to build a future-proof operations infrastructure. In a recent webinar hosted by Anodot, we talked to a panel of experts in payments operations to discuss how to leverage data to optimize payment processes. Experts from Thunes, Payoneer, 888 Holdings and Anodot joined in the roundtable. Liron Diamant, Anodot's Global Payment expert set the stage discussing today's environment in which payment data is becoming a commodity - a digital product. She said payment companies and financial institutions are realizing that smart operations aren't necessarily related to performance but also to the company's ability to learn and adapt using automation and complex data analysis. The panel started the webinar discussing the process of collecting data, specifically which data they find most useful in analyzing. Collecting useful data for payment operations Elie Bertha, Product Director at Thunes, said it's most useful to collect and monitor payment data that enables users to detect issues as fast as possible and communicate it properly. He also said it's important to link all data sources together for a 360 degree view of the business and the customer. Ari Kohn, the Risk Team Leader at Payoneer, said data that is managed and measured properly is the foundational layer of a successful payments business. He said Payoneer's approach to using data for analysis is constantly evolving. He says the company has multiple sources of data stored in multiple formats. His teams have to wrangle all of that to get a 360 degree view of what's going on in order to identify risk. . Anodot's Chief Data Scientist, Ira Cohen,  discussed what happens on the other side of data collection - machine learning. Ira agreed it's important to be notified as soon as possible when something is happening. He said the speed of incident detection has a lot to do with the volume and velocity of data. Cohen says the challenge in data collection that feeds into AI and machine learning is to understand what level of granularity to go by. Cohen says the two options of granularity are by time and space. For example, you can break down transactions by location - down to a particular user. You can also aggregate transactions in time as well - in windows of one minute, five minutes, one hour, etc. Cohen says a good monitoring system allows you to play with both of these attributes, but the dimensionality of the data and the timescale resolution of the data.   Payment use cases  Elie Bertha from Thunes says one of the company's interesting use cases is to segment customers and compare them which helps detect anomalies from a business perspective. Amit Levy at 888 holdings says they strive for end-to-end monitoring that correlates technical issues with business KPIs such as revenue, and how they are related. Ari Kohn from Payoneer discussed use cases in risk management. He says different products carry different risks. For example, when Payoneer is issuing a debit card, the primary concern is fraud. In order to protect customers from card theft, they have to look for signals that indicate that kind of behavior. However, when issuing capital for a seller that needs an advance, they are worried more about delinquency. Kohn says both of those use cases rely heavily on the availability of data - data that is specific to the types of risk they monitoring. The panel also discussed how they prioritize payment incident alerts and how they democratize data across the company for self service analytics. You can watch the roundtable discussion in its entirety here.

Running machine learning anomaly detection on streaming data can play a significant role in your overall revenue. Here’s why.

A jumping-off point for improving your business monitoring capabilities and the way you measure its effectiveness.

Traditional Analytics Tools for eCommerce can’t include Each and Every Metric Number of sessions, total sales, number of transactions, competitor pricing, clicks by search query, cart abandonment rate, total cart value…the analytics tools commonly used by eCommerce companies for performance monitoring can’t include every metric, and even if they did the analysts using them wouldn’t be able to keep up with the amounts of changing data. This of course, inevitably leads to overlooked business incidents and lost revenue whenever these tools are used in the fast-paced world of eCommerce. In eCommerce, minutes matter. Your infrastructure and your competitors’ ad bidding strategies can change in an instant. Any metric can signal an important business incident. When these tools are the foundation of your performance monitoring and business, incident detection doesn’t occur anywhere near the speed of business, so your analysts can spend less time analyzing and more time head-scratching. The need to go granular with performance monitoring Traditional analytics tools like KPI dashboards and lists fall flat on their face when it comes to performance monitoring in the fast-paced, multi-faceted world of eCommerce. These tools take a high-level approach that tries to simplify the complex through generalization, causing BI teams to overlook plenty of metrics for eCommerce analytics. This is a design flaw since even though those tools may automate reporting and visualization, they still require humans to manually monitor the visualized data and spot the anomalies which point to business incidents. Many interesting things can happen in the metrics you’re not monitoring, leading you to miss things completely or discover them too late after the financial and reputation damage is already done. Also, missing just one of a metric’s many dimensions can cause you to miss significant business incidents. Think of metrics as the general kind of quantity and dimensions as the specific slices of that data (e.g. daily sales per brand, daily sales per browser). In effect, monitoring each dimension multiplies the number of metrics that could be monitored, easily resulting in far too many ecommerce analytics metrics for a single person, or even a team, to constantly monitor. A performance monitoring horror story To illustrate why etailers need to take this granular approach to performance monitoring, consider an eCommerce company that sells physical goods in the US. Like many online retailers, this one accepts a wide variety of payment options, from PayPal and credit cards to e-wallets like Google Wallet and Apple Pay. The etailer’s BI team notices on their dashboard that the total daily revenue dropped very slightly. The almost imperceptible dip in this high-level KPI gets passed over by the analysts because they have about five other dashboards to monitor anyway, so they attribute it to statistical noise. Meanwhile, a crucial payment processor has changed their API, breaking the etailer’s ability to process orders made with American Express cards, resulting in those customers abandoning their carts. Since orders with AMEX cards make up such a small portion of the total order volume for this merchant, the total daily revenue barely budges, glossing over the frustration of those AMEX cardholders. Had this company been monitoring daily revenue, not as a single KPI, but broken out across each payment option (daily revenue from AMEX orders, daily revenue from Apple Pay orders, etc.), the sudden drastic drop in successful AMEX orders would have been obvious. Even if this team was using a reasonable static threshold on this metric (an approach which doesn’t scale, as we’ve discussed before), they would have been alerted and the team could contact the payment provider to fix their broken API or implement a workaround in their own code. Problems like these, which impact a small subset of your target market or existing customer base occur quite often in eCommerce, and can paralyze a company’s growth. And what if the company in our hypothetical scenario had just launched a line of premium smartphone accessories for international business travelers – the exact demographic most likely to shop with an American Express card? Good luck recovering from that misstep. The value of real-time monitoring of every eCommerce metric With every passing day that the problem goes undetected, lost revenue piles up and this merchant’s success in breaking into that wealthier clientele is less and less likely. Missed problems lurking in overlooked eCommerce analytics metrics can stop growth in its tracks. The only performance monitoring solution which is adequate for eCommerce is one that can monitor all the dimensions of a given metric in real-time. By missing the crucial business incidents that can make or break eCommerce success, analytics tools that overlook many vertical-specific metrics imperil the merchants who use them. As we’ll see in the next article of this series, this is just as true in fintech as it is in eCommerce.

Social Media Platforms Promote Fake News and Spread Unreliable Content Picture this: You’re at work and you’ve been given an assignment by your boss to research a possible new product. So you go out and do some googling, you find several blog posts, including a very intriguing one with several quotes from industry leaders. You go fetch yourself a cup of coffee and settle in to read. There’s one very big problem with this post, however: It’s completely fake. According to a recent post in the Wall Street Journal, “[r]eal-sounding but made-up news articles have become much easier to produce thanks to a handful of new tools powered by artificial intelligence.” This could be one more instance where ‘fake news’ has penetrated mainstream venues, underscoring how fake news can flourish online. In fact, since the 2016 presidential election, awareness of fake news has soared. Detecting and preventing the spread of unreliable media content is a difficult problem, especially given the rate at which news can spread online. Google and Facebook blamed algorithm errors for these events. Overwhelming amounts of data challenge Social Media to Take Action on Fake News The reach and speed of social media networks (Facebook alone has nearly two billion users) make it easy for such stories to spread before they can be debunked. Part of the challenge lies in how Facebook and Google rely on algorithms, especially when it comes to making complex news decisions. Already in the 2020 presidential campaign, we’ve seen disinformation spread, including manufactured sex scandals against former Mayor Pete Buttigieg of South Bend, Ind., and Sen. Elizabeth Warren (D-Mass.), and a smear campaign claiming Sen. Kamala Harris is “not an American black” because of her mixed-race heritage. Further examples illustrate the impact of fake news on both mainstream media and the public’s mind share: 10 most-viewed ‘fake news’ stories on Facebook Fake bushfire images and maps spreading in Australia Fake news leading to violence in Hong Kong protests Local ‘fake news’ factory spreads disinformation Fake news used to sell diet supplements Climate disaster denialism in Australia  While the algorithms are geared to support the social media giants’ business model for generating traffic and engagement, they’re largely run by engineers who rely on data to choose which content will trend. Are Machine Learning Algorithms Reliable or Are More Human Editors the answer? While computer programs may be cheaper than real-life human editors, Fortune asserts, “The reality is that Facebook needs to hire humans to edit and review the content it promotes as news—and it needs to hire a lot of them.” Facebook was using human editors, but then in 2016 the company fired them after it was reported that they routinely suppressed conservative news stories from trending topics. Now, however, Facebook has brought back human editors to curate certain news content. Appeasing all audiences won’t be easy, though. As New York magazine explains, “the algorithms are biased, and if Facebook hires editors and moderators to double-check decisions made by algorithms, those editors will be denounced as biased too.”  With the sheer volume of data and speed of appearance, MIT has suggested that the use of artificial intelligence tools could help. But artificial intelligence alone isn’t the answer, writes Samual Wooley, who argues that the future will involve “some combination of human labor and AI that eventually succeeds in combating computational propaganda, but how this will happen is simply not clear. AI-enhanced fact-checking is only one route forward.” AI-powered Analytics Using Anomaly Detection Can Hold Back the Spread of Fake News The problem is with the trending algorithms that the social media platforms use – these are machine learning algorithms. They have no context and therefore make these errors. In light of the recent South Park motivated Alexa mishap, we suggested that there should be systems in place to detect when something out of place happens, in order to let the right people know. AI-powered analytics tools would include stance classification to determine whether a headline agreed with the article body, text processing to analyze the author’s writing style, and image forensics to detect Photoshop use. To determine the reliability of an article, Algorithms could extract even relatively simple data features, like image size, readability level, and the ratio of reactions versus shares on Facebook. The fake news issue can also be detected by focusing on anomalies. When a social media algorithm starts pushing a trending post or article to the top, if AI-powered analytics tracked the sudden surge of a new topic, correlating this data with the source site or Facebook page, it would emerge as an obvious anomaly and be paused from gaining any further momentum until a human at Facebook or Google can validate the specific item, rather than needing human review of all topics. You can’t prevent anyone from writing fake news, but by applying AI-powered analytics that employs anomaly detection, we can prevent the “simple-AI” algorithms from spreading and promoting fake news stories. The power of this application of AI-powered analytics to spot anomalies, far faster than humans could, can be used when working with thousands or millions of metrics. Real-time anomaly detection can catch even the most subtle, yet important, deviations in data.