Fraud Detection

RecSysSearchAdvanced16 min

By: Anacodic Team

TL;DR — Fraud detection is heavily imbalanced binary classification under a real-time latency budget: maybe 0.1% of transactions are fraud, so accuracy is a trap (predict "never fraud" → 99.9% accurate, catches nothing). You optimize precision/recall with a cost-sensitive threshold, evaluate with PR-AUC / F-beta (not ROC-AUC or accuracy), engineer velocity / graph / device features, score in <100 ms, and pair a rules engine with an ML model (hybrid). Two problems make fraud uniquely hard: label lag (you learn a charge was fraud weeks later, after chargebacks) and adversarial drift (fraudsters actively change tactics, so yesterday's model rots). Domains: payments (Stripe/PayPal/Visa), banking, healthcare claims (PHI), account takeover.

1. Simple explanation

Fraud detection asks, for each event (a payment, login, claim): "Is this legitimate or fraudulent?" — and it has to decide now, while ~999 of every 1000 events are legit, and the crooks are trying to look legit.

Analogy — airport security for a needle in a haystack. Millions of travelers (transactions), a tiny few are threats (fraud). If security waved everyone through they'd be "99.99% correct" and useless (that's accuracy — worthless here). So they set a threshold: how suspicious before you pull someone aside? Set it loose and you catch every threat but strip-search grandmothers (high recall, low precision — furious customers, huge review cost). Set it tight and lines move fast but threats slip through (high precision, low recall — big losses). The whole job is tuning that threshold to the cost of each mistake, using smart signals (a passenger who bought a one-way ticket in cash 5 minutes ago = a velocity/behavior feature) — while the smugglers keep inventing new tricks (adversarial drift).

2. Diagram

                     REAL-TIME FRAUD DETECTION  (score every event in <100ms)

  transaction / login / claim
        │
        ▼
  ┌──────────────────────┐    FEATURE ENRICHMENT (from a low-latency store)
  │ feature computation  │    velocity: #tx last 1m/1h/24h · $ sum · #cards/device
  │ (streaming + store)  │    graph: shared device/IP/email across accounts
  │                      │    device/behavior: fingerprint, geo mismatch, IP risk
  └──────────┬───────────┘
             ▼
  ┌──────────────────────┐    RULES ENGINE (fast, deterministic, explainable)
  │  RULES  ──hard block──┼──► block obvious fraud / allowlist trusted  (>$X new country…)
  └──────────┬───────────┘
             ▼  (ambiguous cases)
  ┌──────────────────────┐    ML MODEL → fraud probability p ∈ [0,1]
  │  ML SCORER (GBDT/NN)  │    trained on IMBALANCED data (class weights / resampling)
  └──────────┬───────────┘
             ▼
  ┌──────────────────────┐    THRESHOLD (cost-sensitive):
  │  DECISION            │      p ≥ τ_high → block   ·   τ_low ≤ p < τ_high → review (humans)
  │                      │      p < τ_low  → approve
  └──────────┬───────────┘
             ▼
     approve / review / block
             │
             ▼   (days–weeks later: chargebacks, confirmations)
        DELAYED LABELS ──► retrain (fight drift) ──► monitor PR-AUC / recall / $loss

3. How it works (the system-design flow)

3.1 Requirements

Functional: score each event in real time; output approve / review / block; explain why (regulatory).

Non-functional (the numbers):

Constraint	Typical target
Latency	p99 ≤ ~50–100 ms (inline with checkout/auth)
Class balance	fraud ~0.1–2% → severe imbalance
Availability	~99.99% (can't drop payments)
Explainability	required (chargeback disputes, fair-lending, regulated finance)
Objective	minimize $ loss + review cost + customer friction, not accuracy

Framing: this is a cost-sensitive decision, not a symmetric accuracy problem. A missed fraud (false negative) costs the transaction amount + chargeback fees; a false alarm (false positive) costs a blocked good customer + manual-review labor. Those costs are wildly asymmetric and known — bake them into the threshold.

3.2 Data & features

Fraud signal lives in relationships and velocity, not single-row attributes:

  VELOCITY (aggregations over time windows) — the workhorse:
     # transactions / $ amount per card|user|device|IP over 1m / 1h / 24h / 7d
     # distinct cards per device, # distinct devices per account, failed-auth count
  GRAPH / NETWORK:
     shared device/IP/email/shipping-address across accounts (fraud rings cluster);
     entity-graph features, connected-component size, PageRank-style risk propagation
  DEVICE / BEHAVIORAL:
     device fingerprint, browser/OS, IP geolocation vs. billing geo mismatch,
     typing/session behavior, time-since-account-creation, is-emulator
  TRANSACTION / IDENTITY:
     amount vs. user's normal, merchant category risk, hour-of-day, new-payee flag,
     mismatched name/BIN/country

Served from a feature store with strong online/offline consistency — velocity counters are computed by a streaming layer (Kafka + Flink/Redis) and must match what training saw, or you get training–serving skew.

3.3 Class-imbalance handling ★

Because positives are rare, you must fight the model's tendency to ignore them:

Class weights / cost-sensitive loss: weight the minority (fraud) class higher in the loss — cheap, no data distortion, usually the first thing to try.
Resampling: random under-sampling the majority (fast, loses data) or oversampling the minority — SMOTE synthesizes new minority points between neighbors; SMOTEENN cleans noisy overlap. Resample only the training fold, never validation/test.
Anomaly / one-class methods (Isolation Forest, autoencoders) for novel/unseen fraud where you have almost no labels.
Ensembles: GBDTs (XGBoost/LightGBM) dominate tabular fraud; add graph models (GNNs) for ring detection.

3.4 Model & real-time scoring

Gradient-boosted trees are the industry default (strong on tabular, fast, interpretable via SHAP). Serve behind a low-latency service; precompute/enrich features from the store; keep the model small enough to score inline. Output a probability, not a hard label — the threshold decision is separate and tunable without retraining.

3.5 Rules + ML hybrid ★

Production fraud systems are not pure ML:

Rules catch known patterns instantly, are explainable, and let analysts respond in minutes to a new attack ("block > $5k from a new device in a high-risk country"). But they're brittle and gameable.
ML generalizes to subtle, novel combinations rules miss.
Together: rules hard-block/allowlist the obvious; ML scores the ambiguous middle; ambiguous scores route to human review. Rule outputs can also be ML features.

3.6 Threshold & the precision–recall tradeoff ★

The model gives p; the business picks thresholds:

Lower threshold → higher recall (catch more fraud), lower precision (more false alarms). Fraud usually leans toward recall (missing fraud is expensive) — but not blindly, because false positives have real cost.
Often two thresholds: p ≥ τ_high auto-block, τ_low ≤ p < τ_high → human review queue, p < τ_low approve. This bounds both losses and review workload.
Set thresholds by the cost matrix (§4.3), then re-tune as fraud rates and costs shift.

3.7 Feedback delay & drift ★ (what makes fraud special)

Label lag: the ground truth ("this was fraud") arrives days–weeks later via chargebacks/confirmations. So you train on stale labels and must use proxy/early signals and delayed-label-aware evaluation. Recent transactions are effectively unlabeled for a while.
Adversarial / concept drift: fraudsters adapt to your model — feature distributions and the fraud definition shift continuously. Counter with frequent retraining, drift monitoring (PSI/feature distribution alarms), champion/challenger models, and human-in-the-loop rule updates for fast response.

3.8 Evaluation ★

Never accuracy or ROC-AUC alone (both look great under imbalance):

PR-AUC (area under precision–recall) — the right summary metric under imbalance; focuses on the positive (fraud) class.
Recall at fixed precision (or vice-versa), F-beta (β>1 favors recall), precision@k for the review queue.
Business metrics: $ fraud caught vs. $ lost, review rate, false-positive rate on good customers, customer-friction rate. Monitor these live; alert on drift.

4. The math

4.1 Precision, recall, F-beta

   Confusion:   TP (fraud caught)  FP (good flagged)  FN (fraud missed)  TN (good approved)

   Precision = TP / (TP + FP)     of everything we FLAGGED, how much was really fraud?
   Recall    = TP / (TP + FN)     of all real fraud, how much did we CATCH?  (aka TPR)

   Fβ = (1 + β²) · (Precision · Recall) / (β²·Precision + Recall)
        β = 1 → F1 (balance);  β = 2 → weights RECALL 2× (common in fraud)

Why accuracy lies: at 0.1% fraud, "always legit" gives Accuracy = TN/(all) = 99.9% while Recall = 0. Accuracy is dominated by the majority class → meaningless.

Why PR-AUC over ROC-AUC: ROC uses FPR = FP/(FP+TN); with a huge TN, FPR barely moves even for many false positives, so ROC-AUC looks flattering. PR curves use precision, which does react to false positives relative to the rare positives — a truer picture under imbalance.

4.2 Class weighting (cost-sensitive loss)

   weighted log-loss:  L = − Σ_j  w_{y_j} · [ y_j·log(p_j) + (1−y_j)·log(1−p_j) ]
     w_fraud  ≫  w_legit   (e.g. w_fraud = N_legit / N_fraud  → balance the classes)
   → each rare fraud example "counts" as many, so the model can't ignore them.

4.3 Cost-sensitive threshold ★

Pick the threshold τ that minimizes expected cost, not error count:

   ExpectedCost(τ) =  C_FN · FN(τ)  +  C_FP · FP(τ)  [ + C_review · REVIEW(τ) ]
     C_FN = cost of a missed fraud  ≈ transaction $ + chargeback fee   (usually the biggest)
     C_FP = cost of a false alarm   ≈ lost sale + review labor + friction
   Decision rule: block when  p·C_FN  >  (1−p)·C_FP   ⇔   p > C_FP / (C_FP + C_FN) = τ*
   → the OPTIMAL threshold is set by the cost RATIO, not 0.5.

So if a miss costs 20× a false alarm, τ* = 1/21 ≈ 0.048 — you block at just ~5% predicted fraud probability. This single formula answers most "how do you pick the threshold" questions.

5. Real code

Imbalanced training (class weights) + cost-based threshold + PR-AUC — the real workflow.

import numpy as np
from sklearn.metrics import precision_recall_curve, average_precision_score
from xgboost import XGBClassifier

# ---- 1) train with class weighting for imbalance (no data distortion) ----
neg, pos = (y_train == 0).sum(), (y_train == 1).sum()
model = XGBClassifier(
    n_estimators=400, max_depth=6, learning_rate=0.05,
    scale_pos_weight=neg / pos,          # up-weight the rare fraud class ~ N_legit/N_fraud
    eval_metric="aucpr",                 # optimize PR-AUC, NOT accuracy/auc
)
model.fit(X_train, y_train)
p_val = model.predict_proba(X_val)[:, 1]

# ---- 2) evaluate with PR-AUC (correct metric under imbalance) ----
print("PR-AUC:", average_precision_score(y_val, p_val))   # accuracy would be ~useless here

# ---- 3) pick threshold by COST, not 0.5 ----
C_FN, C_FP = 500.0, 25.0                  # miss costs 20x a false alarm
tau_star = C_FP / (C_FP + C_FN)           # optimal analytic threshold ≈ 0.048
print("cost-optimal tau:", round(tau_star, 4))

# ---- (optional) two-threshold policy with a human-review band ----
def decide(p, tau_low=0.05, tau_high=0.90):
    if p >= tau_high: return "BLOCK"
    if p >= tau_low:  return "REVIEW"     # send to analyst queue
    return "APPROVE"

# ---- 4) report recall at a business-mandated precision (e.g. keep precision >= 0.80) ----
prec, rec, thr = precision_recall_curve(y_val, p_val)
ok = prec[:-1] >= 0.80
best = np.argmax(rec[:-1] * ok)           # max recall while precision >= 0.80
print(f"recall @ precision>=0.80: {rec[best]:.3f} at threshold {thr[best]:.3f}")

6. Real-world example

Stripe / PayPal / Adyen (payments): real-time inline scoring at checkout, velocity + device + network features, gradient-boosted models + rules, human review queues, and continuous retraining against chargeback labels. Stripe's Radar is the canonical productized version.
Card networks & banks (Visa, Capital One): sub-100 ms authorization scoring at massive QPS, explainability mandatory for adverse-action/regulatory reasons, cost-sensitive thresholds tied to real $ loss, graph features for fraud rings. Regulated-finance relevance (Capital One): models must be documented, monitored for bias/drift, and every decline must be explainable — which is exactly why GBDT + SHAP + rules (not opaque deep nets) dominate.
Healthcare claims / account integrity (UnitedHealth Group, PHI): anomaly detection over claims with the same imbalance + label-lag problems, but under strict PHI governance — feature stores and logs must be access-controlled, auditable, and privacy-preserving. The author's UHG healthcare/PHI experience maps directly: the ML is standard imbalanced classification, but the data-handling, auditability, and explainability constraints dominate the design — the same governance discipline as regulated finance.
Author tie-in (natural): the rules + ML hybrid and human-in-the-loop review mirror the mediator/feedback loop in Recommendation Systems; the imbalanced eval discipline (PR-AUC, threshold tuning) is the anomaly-detection counterpart to the ranking metrics used in Feed Ranking and Search Systems.

7. Interview questions companies actually ask

 Q [easy]  "Why is accuracy a bad metric for fraud detection?"
   A With ~0.1% fraud, predicting 'never fraud' scores 99.9% accuracy while catching zero fraud.
     Accuracy is dominated by the majority class. Use precision/recall, F-beta, and PR-AUC, which
     focus on the rare positive class.

 Q [easy]  "Precision vs recall — which matters more for fraud?"
   A Depends on cost. Fraud usually leans toward RECALL (a missed fraud costs the transaction +
     chargeback), but false positives block good customers and cost review labor, so you tune the
     tradeoff by the cost ratio, often with a human-review middle band.

 Q [medium] "How do you handle the class imbalance?"
   A Class weights / cost-sensitive loss (first choice — no distortion), resampling (undersample
     majority, or SMOTE oversample minority — TRAIN fold only), anomaly methods for unlabeled/
     novel fraud, and evaluate with PR-AUC/recall-at-precision, never accuracy.

 Q [medium] "Why PR-AUC instead of ROC-AUC here?"
   A ROC's FPR = FP/(FP+TN) barely moves when TN is huge, so ROC-AUC looks flattering under
     imbalance. PR uses precision, which reacts to false positives relative to the rare positives,
     giving a truer picture of minority-class performance.

 Q [medium] "What features would you engineer for card fraud?"
   A Velocity aggregations (# tx / $ per card|device|IP over 1m/1h/24h), graph features (shared
     device/email/IP across accounts → rings), device/behavior (fingerprint, geo mismatch, IP
     risk), and transaction anomaly vs. the user's own baseline. Served from a low-latency store.

 Q [hard]  "How do you set the decision threshold?"
   A Minimize expected cost, not error count: block when p·C_FN > (1−p)·C_FP, i.e. τ* = C_FP/
     (C_FP+C_FN) — set by the cost RATIO, not 0.5. Add a review band (τ_low..τ_high) to cap both
     $ loss and analyst workload; re-tune as fraud rates/costs shift.

 Q [hard]  "Labels arrive weeks late via chargebacks. How does that change things?"
   A Label lag means recent data is effectively unlabeled; you train on stale labels. Use
     delayed-label-aware evaluation, early proxy signals, and don't 'confirm' a window as clean too
     soon. Retrain on a lag-adjusted schedule and monitor live business metrics as leading signals.

 Q [hard]  "Fraudsters adapt to your model (adversarial drift). How do you stay ahead?"
   A Monitor feature/score drift (PSI), retrain frequently, run champion/challenger models, keep a
     fast RULES layer + human-in-the-loop for same-day response to new attacks, and add adversarial-
     robustness features. Treat it as a moving target, not a one-time train.

 Q [hard]  "Rules vs ML — why not just one?"
   A Rules are instant, explainable, and let analysts react in minutes but are brittle/gameable.
     ML generalizes to novel combinations but is slower to update and needs labels. Hybrid: rules
     hard-block/allowlist the obvious, ML scores the ambiguous, humans review the middle; rule
     outputs feed the model as features.

 Q [hard]  "Regulated / PHI context (finance, healthcare) — what constraints dominate?"
   A Explainability (every decline must be justified — favors GBDT+SHAP over black-box nets),
     auditability and access-controlled feature stores/logs, bias/fairness monitoring, and
     documented model governance + drift monitoring. Compliance often shapes the architecture more
     than raw model accuracy.

Sources: Precision vs Recall in Credit Card Fraud · Fraud Detection & Imbalanced Classification — Comet · Precision & Recall: When Conventional Fraud Metrics Fall Short — Equifax · Fraud Detection Under Imbalanced Classes (arXiv) · Handling Unbalanced Data — Towards Data Science

8. When to use / tradeoffs

  Class weights        → first move for imbalance; no data distortion, works with GBDTs natively.
  Under/over-sampling  → undersample when majority is huge (faster train); SMOTE when positives are
                         too few to weight; NEVER resample the validation/test set.
  Anomaly / one-class  → little/no labeled fraud, or novel attack types; noisier, harder to tune.
  GBDT (XGBoost/LGBM)  → the tabular default: strong, fast, explainable (SHAP). Deep nets only for
                         huge/sequence/graph data and where explainability is negotiable.
  Rules + ML hybrid    → essentially always in production: rules for speed/explainability/fast
                         response, ML for generalization, humans for the ambiguous middle.
  Threshold policy     → cost-based, often two thresholds (block / review / approve); re-tune with drift.

Core tensions: precision ↔ recall (cost-driven), catch-rate ↔ customer friction, model power ↔ explainability (regulated domains), and fresh model ↔ label lag.

Fraud detection = imbalanced binary classification in real time; accuracy and ROC-AUC mislead — use PR-AUC, recall-at-precision, F-beta.
Handle imbalance with class weights / resampling (SMOTE); engineer velocity, graph, device features from a consistent feature store.
Set the threshold by cost (τ* = C_FP/(C_FP+C_FN)), often with a human-review band; combine a rules engine + ML hybrid.
The hard, fraud-specific parts: label lag (chargebacks arrive late) and adversarial drift (retrain, monitor, keep humans in the loop).
In regulated/PHI settings (finance, healthcare) explainability, auditability, and governance shape the design as much as accuracy.

Resources

Credit Card Fraud dataset (imbalanced classic) — https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud
imbalanced-learn (SMOTE etc.) — https://imbalanced-learn.org/stable/
Precision-Recall vs ROC under imbalance — https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-imbalanced-classification/
Cost-sensitive learning — https://machinelearningmastery.com/cost-sensitive-learning-for-imbalanced-classification/

← Previous: Search Systems Next: ML Inference Systems →