How to showcase the business impact of machine learning for fraud

In Ravelin's latest Fraud and Payments Survey, machine learning emerged as one of the most popular tools to stop fraud in ecommerce, with 48% of merchants describing it as the most effective tool available, as high as 58% in some locales.

This is great to see, as machine learning is fast, scalable, efficient and can be customized according to individual business priorities.

Machine learning for fraud prevention involves training an algorithm to become a bespoke model. Fraud teams input a merchant’s historical data and create features to teach the model what fraud signals to look out for. Trained models can score customers based on how risky they appear, and make real time decisions on who to block.

However, it can be a challenge to explain how machine learning actually works against fraud. Often thought of as a black box, technical metrics can be confusing. In this article, we’ll explain why it’s important to focus on business objectives and impact when talking about machine learning and fraud.

Traditional ML approach focuses on precision and recall

Machine learning is often explained using precision, recall and false negatives/false positives. Simply put, these are the meanings of these terms:

• Precision: the proportion of blocked (prevented) customers that were actually fraudsters
• Recall: the proportion of fraudsters that were actually blocked
  
• False positive rate: the proportion of genuine customers that were blocked
• False negative rate: the proportion of fraudsters that were allowed

While these are accurate, they can be tricky to relate to leadership and other business teams. How do these lofty concepts actually impact the bottom line? We’ll explain further why it's hard to quantify these exactly.

Instead of precision and recall, block rate and chargeback rate can better communicate the benefits of machine learning for fraud in a way that makes sense across the business. Let’s look at how these metrics compare...

Block and chargeback rates track business objectives

The definitions of block rate and chargeback rate are succinct and self-explanatory:

Block rate: the percentage of transactions that get blocked.

Chargeback rate: the percentage of transactions that resulted in a chargeback.

In fact, precision and recall are almost synonymous with block rate and chargeback rate. Both sets of metrics describe the impact of a machine learning model’s risk threshold – what transactions it should prevent, review or allow based on customer fraud scores.

• If a merchant has really good precision, they will have a low false positive rate - which translates to a lower block rate.

• If a merchant has very good recall, they will have a low false negative rate - which translates to a lower chargeback rate.

Merchants need to balance their block rate against the chargeback rate to optimize their fraud prevention model based on their current priorities.
So, how does this work in practice?

Two example merchants tracking fraud: GottaGrow and Shoeify

Let's look at the fraud approaches of two example merchants with different risk appetites and priorities to show how block and chargeback rates are more fit-for-purpose than precision and recall.

GottaGrow: a startup focused on customer acquisition

GottaGrow is a startup that sells and delivers plants. It’s in the early stages of growth and wants to focus on customer acquisition. As GottaGrow is still fairly unknown, it’s not a big target for fraudsters yet. It wants to block as few customers as possible, and will accept the risk of more chargebacks.

In terms of precision and recall, GottaGrow wants a model with high precision and low recall, but what does this actually mean?

GottaGrow’s fraud team has to unpack these metrics to be able to justify the higher cost of chargebacks to C-level. Block rate and chargeback rate make the practical business impact immediately clear: GottaGrow wants to block only 0.01% of transactions and will accept a higher chargeback rate of 1%.

Shoeify: an established enterprise focused on profitability

Shoeify is an established luxury shoes merchant that wants to focus on profitability and prevent as much fraud as possible. Shoeify’s fraud team wants high recall and will accept more false positives and low precision. To translate what this means for the business, they will have a higher block rate of 1% but a lower chargeback rate of only 0.01%

Position fraud protection as a business-enabler, not a revenue blocker

Using these simple metrics, fraud teams can adopt a more customer centric approach. Shoeify may realise their model blocks too many genuine customers and when it starts to damage their brand reputation, they can adjust the risk threshold lever to lower their block rate slightly.

Likewise, these metrics put the focus on the merchant priorities. Chargeback rates directly impact profitability, affecting revenue and costs - with every chargeback, a business loses the value of the transaction and incurs chargeback fees. GottaGrow can track the cost of their chargebacks month to month, and when costs start to outweigh the value of increased customer acquisition, they can tweak their approach.

Less dependence on hard-to-measure false positives

False positives are a big problem for merchants. When a merchant blocks a genuine customer, they lose the potential revenue from that sale, but can also lose the customer for life.

In the US, 33% of consumers said they would never order again from a merchant after being wrongly declined. When you factor in damage to brand reputation and hidden costs such as wasted marketing budgets, false positives get very expensive.

But after you’ve blocked a transaction, how can you find out if it was actually a fraudster or just a genuine customer that looked risky? Fraud teams can attempt to measure false positives through manual reviews (which are resource intensive and vulnerable to bias) or allow a control set of transactions to go through with no fraud prevention (which is extremely high risk).

Ultimately, there’s not a simple way to understand the true false positive rate.

Precision and recall are all tied up with these hard-to-measure false positive rates. But looking at block and chargeback rate doesn’t depend on false positives. Instead, the block rate is focused on how much revenue was lost through the blocked transactions, and how the cost of fraud compares.

Summary

Machine learning for fraud can have a huge impact on your business profits and priorities.

It’s important to communicate the impact to business leaders in a way that positions fraud protection as a business-enabler, not a revenue blocker. To learn more about how to explain machine learning in business terms, watch our straight-talking webinar here.

Further reading

• On-demand webinar: How to tackle fraud with a bespoke ML fraud model
  
• On-demand webinar: Demystifying machine learning
• Why fraud prevention without natural language processing is missing a trick
• Podcast: Stephen Whitworth on data engineering and fraud detection