Model bake-off

When fraud isn't an anomaly

August 2026 · 7 min · Code on GitHub

Fraud is rare and fraud is abnormal, so anomaly detection should be the right tool. That is the intuition, it is extremely common, and on this dataset it is wrong. Here is the bake-off that shows it: four unsupervised anomaly detectors against two supervised classifiers, same stratified 70/30 split, 492 frauds hidden among 284,807 transactions.

The contenders

On the unsupervised side, Isolation Forest, One-Class SVM, Local Outlier Factor, and Elliptic Envelope, each fit on legitimate transactions only so they learn the shape of normal and flag departures from it. On the supervised side, a Random Forest and an MLP trained on the full labeled training set. Features standardized, and accuracy deliberately excluded from the scorecard: predicting "not fraud" every single time scores 99.83 percent accurate and catches nothing, which makes accuracy an actively misleading metric here rather than merely a weak one.

The scoreboard

Model	Precision	Recall	F1
Local Outlier Factor	~0.01	~0.01	~0.01
One-Class SVM	~0.11	~0.79	~0.19
Elliptic Envelope	~0.14	~0.17	~0.15
Isolation Forest	~0.21	~0.23	~0.22
MLP Classifier	~0.88	~0.71	~0.79
Random Forest	~0.96	~0.76	~0.85

It is not close. Every unsupervised model lands below 0.25 F1; both supervised models clear 0.79.

ROC curves for the supervised models, and a live demonstration of why ROC misleads here. The MLP posts the *higher* AUC (0.945 against 0.931) while losing on precision, 0.88 to 0.96, because the true negative pool is so large that false positive rate stays tiny even when a meaningful share of flags are wrong. Read it alongside the precision column, never instead of it.

Confusion matrix for the Random Forest classifier — Random Forest confusion matrix at the default 0.5 threshold. The two error cells are the entire business conversation: 36 frauds missed on one side, 5 legitimate customers declined on the other. That ratio is a choice, not a fact, and the slider in the dashboard is where it gets made.

Read the One-Class SVM row again

Recall 0.79 looks like a win. It catches four of every five frauds. But precision 0.11 means roughly nine of every ten transactions it flags are perfectly legitimate, so in production that is either an investigations team buried in false alarms or thousands of good customers getting declined at checkout. High recall at low precision is not a fraud detector. It is an alarm that never stops ringing, which in practice means an alarm everyone learns to ignore.

Why the intuition breaks

Anomaly detection assumes fraud sits in a statistically unusual region of feature space. Mostly it doesn't. Fraudsters actively work to look like ordinary customers, so fraudulent transactions land inside or beside dense regions of legitimate behavior. Local Outlier Factor collapsing to near zero across the board is this exact failure in its purest form: local fraud density looks like local normal density, so there is nothing for it to find. Elliptic Envelope fails differently, assuming Gaussian structure and fitting an ellipse around data that is not remotely Gaussian.

Underneath both is base rate arithmetic. At 0.17 percent prevalence, 99.83 percent of everything scored is legitimate, so even a small false positive rate produces a flood of false alarms in absolute terms. Precision degrades savagely as prevalence falls, and no amount of unsupervised cleverness escapes that arithmetic.

The labels are the signal. When ground truth about fraud exists, discarding it to go hunting for "anomalies" throws away the most valuable column in the dataset. Unsupervised methods earn their place when labels are absent, not when they are inconvenient.

The threshold belongs to the business

A probability model has no single operating point, only a curve of them, and choosing where to sit on that curve is a business decision wearing a technical costume. A missed fraud costs a chargeback. A false positive costs a customer relationship. Those prices are not equal, they are not stable across products, and no data scientist should be setting them alone.

So the project ships a Streamlit dashboard where the decision threshold is a slider. Drag it and watch precision and recall trade off live, with the confusion matrix updating underneath. It is a small feature that turns the most important conversation in fraud modeling, how much customer friction we will accept per fraud caught, into something a non-technical stakeholder can hold in their hands instead of taking on faith.

What I would actually ship

Random Forest, tuned to whatever threshold the fraud team's cost ratio implies rather than the default 0.5, with feature importances exposed so analysts can sanity check what drives a flag. And I would keep one anomaly detector running in the background, not as a competitor, but as a canary: it is the only member of this lineup that can flag a pattern nobody has labeled yet, which is precisely the case the supervised model is blind to.

Code, pipeline, and dashboard: github.com/ChrisJ1751/CC_Fraud_Detection. Dataset: Kaggle credit card fraud, European cardholders, 2013.