training a random forest on 100,000 synthetic customers to spot the quiet signals of intent — and finding that behaviour beats demographics, every single time.
the marketing team had a budget to contact 10,000 customers about a mortgage. the customer base was 1.2 million. who do we call, and why them?
The Hypothesis
demographics — salary, region, tenure — will explain some of the signal. but the strongest predictor of intent isn't who a customer is; it's what they've been quietly doing in the last 90 days. a random forest, given enough behavioural features, will outperform a logistic regression trained on the obvious ones.
The Data
100,000 synthetic customers, generated to mirror a mid-sized retail bank's book. ten raw fields — salary, tenure, product count, app logins, mortgage-page visits, savings balance, email engagement, previous campaign responses, salary-credited flag, and region.
from those ten, i engineered eighteen behavioural features: mortgage-page visits in the last 30 days, ratio of mortgage visits to total sessions, app-login velocity (delta between the last two 30-day windows), savings-to-salary ratio, months-since-last-large-deposit, email open rate x click rate, response rate on the last three campaigns, tenure buckets, product-density (products per year of tenure), a "just-got-a-payrise" flag (salary jump > 15% quarter-on-quarter), region x salary deciles, weekend-vs-weekday login skew, and a composite "engagement momentum" score.
the target —
applied_for_mortgage
in the next 60 days — was set at a 4.2% base rate, deliberately imbalanced to mirror reality.
The Analysis
three models, same train/test split (80/20), same stratified sampling, same hyperparameter budget.
model comparison — hold-out set
accuracy is a liar on an imbalanced problem — a model that predicts "no" for everyone scores 95.8%. what matters is precision (of the customers we call, how many actually apply?) and recall (of the customers who would apply, how many did we catch?). the random forest wins on both, and its ROC-AUC of 0.91 means it separates the two classes cleanly.
the decision tree overfits — high variance, brittle splits. logistic regression is honest but under-powered; it can't model the interactions (payrise × mortgage-page-visits × low savings) that the forest picks up for free.
The Decision
deploy the random forest. score the full 1.2m base weekly. rank descending. call the top 10,000.
but the more interesting decision was what to stop doing. feature importance told a story that contradicted the team's intuition:
top 10 features — random forest importance
salary — the feature the business had built its previous targeting on — landed ninth. region was almost noise. the top three signals were all behavioural, recent, and free to collect: page visits, visit ratios, and app-login acceleration. the model was telling us: stop guessing from demographics. watch what people do.
The Outcome
the pilot campaign ran for 8 weeks against a control group targeted the old way (top-decile salary + homeowner age band).
conversion on the model-targeted list: 11.3%
conversion on the demographic control: 3.1%
3.6x lift, £4.2m in incremental mortgage originations against a £180k campaign cost.
more importantly, the second-order finding: the "payrise_flag" alone — a single derived feature — identified a micro-segment converting at 18%. that's now a real-time trigger, not a quarterly batch.
the model went into production behind a nightly scoring job. it's been retrained monthly for a year. drift has been minimal because the behavioural features are, by their nature, always current.
Further reading
why not just use logistic regression?
honestly, we tried. and for a while we shipped it — it's interpretable, it's fast, the coefficients read like a report. but logistic regression assumes each feature contributes independently and linearly to the log-odds of applying. reality doesn't work that way.
the customers who apply for mortgages aren't the ones with the highest salary. they're the ones whose salary just jumped, who've been opening the app more often than last month, who visited the mortgage calculator twice on a sunday, and whose savings are sitting at roughly 20% of a typical deposit. that's an interaction — four features conspiring. logistic regression needs you to hand-craft that interaction as a new column. a random forest finds it for you, and finds forty more you'd never have thought of.
why not xgboost?
we tested it. it edged the random forest by ~0.008 on AUC and took four times as long to explain to the risk committee. not worth it. the random forest was the sweet spot: strong enough to win, simple enough to defend.
what i'd change
calibrate the probabilities. random forests are notoriously overconfident. platt scaling on the hold-out fixed it.
monitor drift on the top features. if
mortgage_visits_30d
importance drops, something changed upstream (a site redesign, a broken tracker) — treat it as an alarm.
stop reporting accuracy internally. it's the wrong number and it invites the wrong conversations.
