3DS rule experimentation and optimization

How we’ve balanced fraud risk and friction: Deliveroo’s Journey with 3DS rule experimentation

Table of Contents

  1. Introduction
  2. What is 3DS and why is this important?
  3. How does Deliveroo use 3DS?
  4. Deliveroo’s 3DS best practices
  5. Running Experiments on 3DS Rules
  6. Thinking in terms of cost benefit
  7. Our results


Hello there! I’m Jack, a Data Scientist working in Deliveroo’s Trust team. The Trust team works on safeguarding Deliveroo and its customers against fraudulent activities and abuse. We tackle various challenges, including payment fraud, compensation abuse, and promotions abuse, among others. Lately, my focus has been on optimising our payment fraud rules. In this blog post, I will share my insights and offer some actionable advice for teams trying to do the same.

What is 3DS and why is this important?

Any merchant receiving online transactions is at risk of payment fraud - this is where fraudsters use stolen credit cards to transact on their platform. The main way merchants protect themselves and their customers from payment fraud is through the use of 3D Secure (3DS). 3DS is an authentication protocol designed to provide an additional layer of security for online transactions, for example by requiring the cardholder to provide a one-time passcode to help verify their identity.

Payment fraud can impose significant financial burdens to a company’s profitability through various means, such as chargeback expenses and severe financial penalties imposed by card networks like VISA and Mastercard in cases of excessively high fraud rates. While preventing fraud is of paramount concern for companies, employing excessive and inefficient fraud prevention measures can have equally adverse consequences. These measures can detrimentally impact the experience of legitimate customers, leading to a decrease in order volume, profitability, and customer retention.

How does Deliveroo use 3DS?

Deliveroo decides which transactions to send to 3DS through two main methods: Machine learning Models (such as Ravelin’s Machine learning model) Custom 3DS rules. These are rules manually created by our operations team, using Ravelin, to help quash emerging fraud trends they’ve identified, or that we think we need extra protection against.

Ravelin is a fraud detection and prevention company that offers machine learning-based solutions to help businesses combat online fraud. In the payments fraud space, we use their models to prevent fraud, and also use their platform to create 3DS rules.

It is worth noting that other parties involved in the payment processing flow (PSPs and Card Issuers) can also trigger 3DS based on their own fraud engines or fraud rules, so the overall 3DS customer experience is not only dependent on Deliveroo.

So, why not just 3DS everyone?

Scooby Doo meme, showing Fred unmasking the baddie. Before unmasking the baddie is captioned with "We've eliminated all fraud", and after unmasking says "by eliminating all orders"

Whilst 3DS is highly effective in preventing fraud, it comes with its trade offs. Whenever you authenticate a customer, you introduce friction into their purchase journey, and inevitably cause a percentage of transactions to be lost. So, whilst we could send all orders to 3DS and eliminate almost all fraud, we’d see sizable drops in both order volume, profit, customer retention/experience, and negatively impact what is mostly genuine customers. Indeed, extremely high 3DS rates sometimes result in lower authorisation rates as Issuers rate the merchants as riskier.

Deliveroo’s 3DS best practices

The below outlines how we at Deliveroo are thinking about 3DS:

Running Experiments on 3DS Rules

The question we first and foremost needed to answer was: how good are our 3DS rules? To do this, we conducted A/B tests (or experiments). We conducted large scale A/B tests over Q4 2022. For each rule that we experimented on, we divided our users into two groups – the control group and the experimental group. The control group was subject to the existing 3DS rule as usual, while the experimental group was exempted from 3DS for that particular rule. We then monitored the performance of both groups. We chose to experiment on the highest volume rules first for speed and practicality. There are a lot of interesting quirks around 3DS experimentation such as the randomization unit, and challenges with overlapping rules which are out of scope for this blog post.

In practice, we ran all of the experiments through Ravelin. Ravelin’s rule platform and UI makes it extremely easy to run A/B tests, without the need for engineering work traditionally required for experiments such as adding feature flags and relevant data logging. We used the Ravelin tags feature to split our population into groups, and made use of Ravelin’s rule hierarchy to exempt the correct orders from 3DS.

Thinking in terms of cost benefit

To evaluate the rules, we devised an equation that accounted for various financial factors beyond fraud prevention alone that the rule could have. Instead of just thinking about the fraud it prevented, we added in information about the operational profit, 3DS fees, and any additional compensation costs. Below shows the equation we used.

Cost benefit = Additional Operational profit + 3DS fees saved - Additional Chargeback costs (inc. fees) - Additional Compensation costs

Another thing we consider when removing rules is fraud rates - Card schemes (such as VISA and Mastercard) impose strict penalties if you breach certain fraud levels. We always make sure to keep well within card scheme rules by keeping our fraud rates very low.

Our results

Based on our analysis of the data, we identified a large number of rules that were overly restrictive and were on balance net negative to the business. By optimising those rules we have been able to and reduce our 3DS rate by nearly 40% without exposing Deliveroo, our customers or partners to additional fraud risks. As a result, we’ve been able to reduce friction for genuine customers, improving conversion for Deliveroo and increasing orders for our partners and riders. A win-win-win-win.

About Jack Dai

A picture of Jack Dai

I’m a Data scientist in the Trust team at Deliveroo - working on stopping fraud and abuse.