Loop A — fast, label-free
runningPSI on each feature (covariate) and on the score (prediction drift) using credit-risk bands (<0.10 stable, 0.10–0.25 warn, >0.25 drift), plus CBPE — a label-free ROC-AUC estimate from the calibrated scores.
Real-time payment-fraud detection · MLOps reference
Fraud scoring, built like production — and honest about it.
The model is the easy part. The MLOps lifecycle around it is the job — validation, leakage-safe splitting, governance, drift monitoring and a self-retraining loop. This one page is the whole story, top to bottom; every capability is running — measured from the repository, no inflated numbers.
0.915 ROC-AUC · out-of-time116 tests · Level 1→2 MLOps10-asset Dagster DAGCPU-only · $0 GPU
The lifecycle · end to end
The model is the easy part. The MLOps lifecycle is the job.
Orchestrated as a 10-asset Dagster DAG · 116 tests passing · MLOps maturity Level 1→2 · green = built & tested
01 · Data contractrunning
590,540 transactions, under contract
IEEE-CIS gives 590,540 labeled transactions across 434 columns (~432 predictors). Before anything trains, a Pandera schema enforces types, ranges and a null-rate budget, and a leakage check quarantines any field only known after the event. Fraud is just 3.5% — the imbalance the whole pipeline is built around.
590,540 × 434 cols
IEEE-CIS · transactions × columns (~432 predictors)
96.5% legitimate · 3.5% fraud — extreme imbalance
✓ Pandera schema: column types & ranges
✓ null-rate budget per column
✓ leakage check: post-event fields quarantined
Monitoring & drift — the self-retraining loop
Catch decay before the chargebacks settle.
Fraud labels arrive 7–60 days late as chargebacks, so monitoring can’t wait for them. Two loops bridge the gap, and a compound gate decides when to retrain.
Loop B — slow, settled labels
runningA windowed backtest on settled labels with bootstrap confidence intervals — so a dip only counts when it’s beyond sampling noise. This confirms or refutes the CBPE estimate.
The compound gate · try it
Retrain only when both signals fire.
A blind retrain on a benign shift can ship a worse model — so auto-retrain triggers only when feature drift and a real performance drop co-occur. Flip the window and watch the gate decide.
Signal 1 · Feature drift
12
features past the PSI / KS threshold
DRIFT
Signal 2 · Performance (CBPE)
0.8155
windowed ROC-AUC · 95% CI [0.806–0.825]
DROP vs 0.913 baseline
Compound gate · AND
drift ✓ANDdrop ✓
RETRAIN fires
feature drift AND performance drop → retrain
retrain on new window→challenger trained→shadow-scored→promoted · one-command rollback
Simulated here: drift is injected on a held-out window (the IEEE-CIS dataset is static). Detection, the gate, and the retrain trigger are the same code paths that watch live traffic in production. The 0.913 baseline is ROC-AUC on the monitoring reference window (the 59,054-row calibration slice) — a different reference point than the 0.915 out-of-time headline, not a second measurement of it.
The problem
Fraud is rare, costly, and moves over time.
On 590,540 IEEE-CIS transactions only 3.5% are fraud. Two traps make naïve metrics lie: a random split lets the future leak into training (inflating ROC-AUC toward ~0.96), and ROC-AUC itself looks healthy even when precision at the alert threshold is poor.
Time-ordered split
Train / calib / eval split strictly on TransactionDT, never random — the eval set is the latest 20% of time, so the score is honest about the future. running
Lead with PR-AUC
At 3.5% prevalence, average precision (PR-AUC 0.552) reflects an alert budget — reported ahead of ROC-AUC (0.915). running
Leakage-safe by design
Every encoder is fit on the training split only and is label-free; point-in-time joins never reach forward in time. running
The model · read PR-AUC first
The least interesting part — and honest about it.
Every number is measured on the frozen out-of-time eval set — the latest 20% of time, never a random split. The operating point is a deliberate choice, not 0.5: precision is set against an alert budget, so recall is always reported at a fixed precision. The fuller reasoning — and what each choice costs — is in Decisions.
PR-AUC (out-of-time)
0.552
average precision — read this first at 3.5% fraud
ROC-AUC (out-of-time)
0.915
single LightGBM; honest, not the inflated 0.94
Recall @ 50% precision
54.3%
fraud caught at a ~3.7% alert rate
Recall @ 70% precision
42.7%
tighter precision, fewer false alarms
Calibration ECE
0.006
isotonic on a disjoint calib slice (0.010 → 0.006)
Ensemble ROC-AUC
0.916
5-model seed ensemble — only +0.001 over single
Responsible by construction
Explainable, robust, governed.
Explainable
Native LightGBM pred_contrib TreeSHAP — exact Shapley values, ~20 ms, attached to every decision (no separate explainer service).running
Robust
Per-segment ROC-AUC (product, card, device) via Fairlearn holds 0.88–0.95across the major segments (overall 0.915); weakest is Discover at 0.81 on ~1.3k txns — model risk, not protected-class “fairness”.running
Governed
Model card (SR 11-7 / MRM + EU AI Act fraud-detection carve-out) and a SHA-256 hash-chain audit log, with S3 Object-Lock WORM as the immutable cloud store.card + hash-chain S3 WORM
Engineering
Decisions, and what they cost.
Every choice here buys something and pays for something — the trade-offs made on purpose. This is the part an interviewer actually wants to hear.
Single LightGBM, not an ensemble
- Why
- A 5-seed ensemble reached 0.916 — only +0.001 ROC-AUC — so the single model ships for ~1/5th the serving latency, behind champion/challenger with one-command rollback.
- Trade-off
- Chasing the literature's ~0.94 would need a heavier LGB+XGB+CatBoost stack this CPU box can't justify.
CPU-only, no GPU
- Why
- A depth-12 LightGBM scores in ~110 ms warm on commodity CPU — exactly what makes scale-to-zero hosting genuinely cheap.
- Trade-off
- No deep-learning feature extractors — the right call for tabular fraud, a real limit on raw-sequence problems.
Time-ordered split, never random
- Why
- A random split lets the future leak into training and inflates ROC-AUC toward ~0.96. Splitting strictly on TransactionDT keeps the score honest about tomorrow.
- Trade-off
- The headline 0.915 reads lower than a leaky baseline would — on purpose.
Compound retrain gate, not a single trigger
- Why
- A blind retrain on a benign feature shift can ship a worse model, so auto-retrain fires only when feature drift AND a real performance drop co-occur.
- Trade-off
- A label-only shift is held at 'watch' until settled labels confirm it — slower to react, but it won't ship a regression.
Explanations in-path, not a second service
- Why
- Native LightGBM pred_contrib attaches exact TreeSHAP to every score, so every alert an analyst opens already carries the features that drove it.
- Trade-off
- TreeSHAP is ~20 ms/decision and caps explained throughput near 50 RPS/core — scoring-only is ~140x faster (~7,100/s).
Ports & adapters, fail-closed gates
- Why
- Quality, calibration, and segment gates block promotion on failure, and the whole lifecycle is covered by 116 tests in CI.
- Trade-off
- Every stage needs a clean interface and a gate — more boilerplate, far more confidence.
Cost & latency
Fast and cheap, because it’s CPU.
No GPU anywhere in the path. A depth-12 LightGBM scores in milliseconds on commodity CPU — which is what makes scale-to-zero hosting genuinely cheap.
Single request (warm)
~110 ms
features + score, one transaction (median)
Throughput, scoring only
~7,100 / s / core
batched on CPU, no SHAP
Throughput, with TreeSHAP
~50 / s / core
explanation in-path
TreeSHAP per decision
~20 ms
native pred_contrib
Throughput is the batched per-core figure (BentoML adaptive batching). The champion is deployed on a scale-to-zero CPU container (Modal, auth-gated): it spins up on demand and idles when nothing is in flight — score one live (access on request) under Try it. measured local deployed · Modal
The stack
Boring, proven tools — wired together well.
No exotic dependencies. The interesting part is the lifecycle that connects them, not any single library.
Modeling
- LightGBM · depth-12
- Optuna · TPE search
- Isotonic calibration
- Native TreeSHAP
Data & pipeline
- Pandera · fail-closed
- Feast · point-in-time
- Dagster · 10 assets
- MLflow · registry
Serving
- BentoML · adaptive batching
- Docker container
- Modal · scale-to-zero (live)
- S3 Object-Lock WORM (audit store)
Monitoring & governance
- Evidently · PSI / KS
- CBPE · label-free AUC
- Fairlearn · segment gates
- Model card + hash-chain
Plus pytest (116 tests) + GitHub Actions CI, DVC data versioning, and ports & adapters throughout — every external dependency swaps between a local mock and its real counterpart behind one flag.
What’s built
Counted, not claimed.
Every figure is counted from the repository or measured on the model — an MLOps maturity Level 1→2 reference.
590,540
labeled transactions
IEEE-CIS, 434 columns (~432 predictors)
3.5%
fraud prevalence
extreme class imbalance
455
model features
after leakage-safe feature engineering
116
tests passing
1 skipped (nannyml, no py3.13 wheel)
10
Dagster assets
the full lifecycle, materialized end-to-end
0.552 / 0.915
PR-AUC / ROC-AUC
frozen out-of-time eval
~20 ms
TreeSHAP / decision
native pred_contrib, CPU
$0
GPU spend
CPU-only, by design
Lifecycle checklist — 18 running
Orchestrated, not scripted.
The whole lifecycle is one Dagster asset graph that materializes end-to-end on a sample — all 10 assets. A GitHub Actions workflow runs lint + the 116-test suite on every push.
Data validation (Pandera)running
Time-ordered splitrunning
Feature engineering (label-free)running
Optuna tuningrunning
MLflow registry + provenancerunning
CI quality gatesrunning
Calibrationrunning
Native TreeSHAP servingrunning
Segment robustness (Fairlearn)running
PSI drift + CBPErunning
Bootstrap-CI backtestrunning
Compound retrain gaterunning
Champion/challenger + rollbackrunning
Model card + hash-chain auditrunning
Dagster orchestrationrunning
DVC data versioningrunning
Shadow scoring on live trafficrunning
S3 Object-Lock WORM (audit store)running
Try it
Score a transaction.
Browse real held-out transactions and watch the champion model decide — the fraud probability, the decision at the operating threshold, and the TreeSHAP features behind it. Real outputs, precomputed (the model is deterministic, so they’re identical to a live call) — or switch to a live call against the deployed, auth-gated endpoint (sign-in required; demo credentials by email).
Precomputed by default (instant, deterministic). Switch to Score live for a real call to the deployed, auth-gated endpoint — sign in with demo credentials, granted by email.
real model outputs (precomputed)deployed live · auth-gated
Choose a sample
Fraud probability
99.8%0%alert threshold 50.0%100%
→ send to reviewactually fraud — ✓ caught — true positive
Transaction
- TransactionAmt
- 54
- ProductCD
- C
- card4
- visa
- card6
- debit
- DeviceType
- mobile
- P_emaildomain
- hotmail.com
Why — top TreeSHAP features
Native pred_contrib. Gold pushes toward fraud; sage pushes toward legitimate.
- DeviceInfo+2.53
- V258+1.6887
- C1+1.4092
- C8+0.4567
- C11+0.4032
- C12+0.2345
How this demo is honest
Real outputs, precomputed.
Illustrative examples across the model's risk spectrum. Input display-fields are synthetic and schema-representative (the champion LightGBM scores a full 455-feature vector, not these six fields); the fraud probabilities, decisions, and TreeSHAP attributions shown are the model's real, deterministic behaviour on held-out data. The probabilities and SHAP values above are exactly what the champion model (lgbm-fraud-d12-569t) produced for these held-out IEEE-CIS transactions — nothing is invented. The same champion is deployed live behind an auth-gated CPU endpoint — and because the model is deterministic, these precomputed results are identical to a live call. running
Counted, measured, and honest about it.
Every figure on this page is read from the repository or measured on the champion model — no inflated numbers, every capability running. Take it for a spin.
0.915 ROC-AUC116 tests passing10-asset Dagster DAGCPU-only · $0 GPU
The builder
Want this kind of engineering on your team?
MLPipeline Pro is my work end to end — the pipeline, the gates, the drift loop, the governance, and this page. I’m open to ML-platform, MLOps, and backend roles.