the tree that could explain itself

a retention team had budget to call 1,500 customers a month out of a book of 400,000. the ask was not "predict churn" — a model already did that. the ask was: can you show me why, on one page, to a management committee that has forty minutes and no appetite for a black box?

so the question became two questions. which customers leave within 90 days — and which model do you put in front of senior management when the most accurate one is the hardest to explain?

that a shallow decision tree would lose a little accuracy and win a lot of trust. specifically:

• the strongest churn signal would be relationship depth, not demographics — salary no longer landing in the account, balances drifting down, products falling away.
• a tree deep enough to memorise 75,000 customers would score worse on hold-out data than one pruned to four levels.
• the logistic regression from the previous investigation would rank customers slightly better, but say nothing a committee could repeat back.

75,000 synthetic customers, generated with a fixed seed so every number on this page can be rebuilt from the script at the bottom. nine features, one binary target: churned within 90 days.

churn base rate: 3.61% — 2,706 leavers. that imbalance is the whole story of the metrics below.

feature	mean	stayers	leavers
tenure (months)	35.4	35.6	30.1
salary credited	0.86	0.87	0.60
product count	1.75	1.76	1.48
app logins (30d)	12.3	12.3	10.9
balance trend	+0.009	+0.011	−0.049
complaints (12m)	0.25	0.25	0.43
email engagement	0.31	0.31	0.27
branch visits (90d)	1.45	1.44	1.65
mortgage page visits	0.40	0.40	0.39

read the last two columns side by side and the shape of a leaver appears before any model is trained: salary has stopped arriving, the balance is sliding, there is a complaint on file, and they are turning up in a branch rather than the app.

the leaver, feature by feature (indexed: stayer = 100)

mortgage page visits sit almost exactly on 100 — a feature that mattered enormously for the propensity model and matters not at all here. intent to borrow is not intent to leave.

preprocessing. no imputation was required — the panel is complete by construction. no outlier trimming: extreme balance swings are exactly the cases the retention team cares about.

salary_credited

is already binary. every other feature is numeric and left on its native scale for the tree, which is scale-invariant.

encoding and scaling. the decision tree takes raw values (splits are thresholds, so monotone transforms change nothing). the logistic regression is fitted on

StandardScaler

-transformed inputs so the coefficients are directly comparable in size.

split. a single 80/20 hold-out, stratified on the churn label to keep 3.61% in both halves — 60,000 training rows, 15,000 test rows.

random_state=42

. no test row is touched until the final scoring pass.

class imbalance.

class_weight="balanced"

on both models rather than resampling: it keeps the dataset honest and makes the two models comparable on identical rows.

regularisation of the tree.

min_samples_leaf=50

, so no rule fires on fewer than fifty customers. depth is the tuned parameter — swept from 2 to unlimited.

evaluation. roc-auc as the ranking metric (the one that matters when you can only call 1,500 people), precision and recall at the default 0.5 threshold, plus top-decile lift — the churn rate among the 1,500 highest-scored customers divided by the base rate. accuracy is reported only to show why it should be ignored.

1. how deep should the tree be?

depth was swept from 2 to unlimited, everything else fixed.

hold-out roc-auc by tree depth

depth	leaves	roc-auc	recall	precision	accuracy
2	4	0.672	0.682	0.060	0.600
3	8	0.693	0.654	0.065	0.650
4	16	0.707	0.688	0.063	0.617
5	31	0.710	0.689	0.067	0.644
6	55	0.707	0.688	0.064	0.626
8	139	0.703	0.580	0.071	0.710
12	394	0.662	0.566	0.063	0.682
unlimited	712	0.654	0.547	0.062	0.683

the fully grown tree ends with 712 leaves and a worse auc than a tree with four. that is overfitting rendered as a picture: 712 rules, each one a memory of a handful of customers who happened to leave.

depth 5 edges depth 4 by 0.003 auc and doubles the rule count. depth 4 was chosen — 0.003 of auc is not worth fifteen extra leaves you have to explain.

2. the tree, visualised

sixteen leaves, and the top three levels read like a retention playbook:

salary_credited <= 0.5                          [salary no longer arriving]
├── complaints_12m <= 0.5
│   ├── email_engagement <= 0.45  ────────────► LEAVING
│   └── email_engagement >  0.45
│       ├── balance_trend <= 0.04  ───────────► LEAVING
│       └── balance_trend >  0.04  ───────────► staying
└── complaints_12m > 0.5  ─────────────────────► LEAVING  (all branches)

salary_credited > 0.5                           [salary still arriving]
├── balance_trend <= -0.04
│   ├── product_count <= 2.5
│   │   ├── app_logins_30d <= 16.5  ──────────► LEAVING
│   │   └── app_logins_30d >  16.5  ──────────► staying
│   └── product_count > 2.5
│       ├── complaints_12m <= 1.5  ───────────► staying
│       └── complaints_12m >  1.5  ───────────► LEAVING
└── balance_trend > -0.04
    ├── complaints_12m <= 0.5  ───────────────► staying
    └── complaints_12m > 0.5
        ├── email_engagement <= 0.30  ────────► LEAVING
        └── email_engagement >  0.30  ────────► staying

the first split is the whole model in one line: has the salary stopped landing? everything after it is a modifier. a customer whose salary still arrives, whose balance is not sliding and who has not complained is, essentially, safe — that single leaf holds most of the book.

3. feature importance

gini importance — depth-4 tree

tenure, branch visits and mortgage page visits score exactly zero — not because they carry no signal (tenure differs by five months between leavers and stayers) but because a depth-4 tree only has sixteen decisions to spend, and stronger correlated features spend them first. the logistic regression, which is not forced to choose, still gives branch visits a coefficient of +0.19.

that is the first honest caveat to put in front of a committee: zero importance in a shallow tree means "not needed", not "not related".

4. tree vs. logistic regression

same rows, same split, same class weighting.

hold-out performance — depth-4 tree vs logistic regression

metric	tree (depth 4)	logistic regression
roc-auc	0.707	0.748
recall	0.688	0.621
precision	0.063	0.077
accuracy	0.617	0.717
top-decile lift	3.01×	3.70×
explainable on one slide	yes	partly

the regression wins where it counts operationally. at the top decile — the only part of the ranking anyone will ever action — it finds churners at 3.70× the base rate against the tree's 3.01×. on a 1,500-call budget that is roughly 200 extra at-risk customers reached per month, for free.

and note the precision numbers: 6–8%. on a 3.6% base rate that is a doubling, but it still means twelve of every thirteen calls go to someone who was never going to leave. any framing that hides this from management is a framing that will be found out.

present the tree. deploy the regression.

they answer different questions and the mistake is pretending one artefact must do both.

the depth-4 tree goes on the slide. it is the only version of this work that a committee can read, argue with, and remember on the way out. "salary stopped arriving, balance sliding, complaint on file" is a sentence a regional director can act on without a data scientist in the room. it also makes the model falsifiable by people who know the business — the fastest way to find a broken feature is to show a rule to someone who has worked the front line for twenty years.

the logistic regression goes into the call list. it ranks better (0.748 vs 0.707 auc, 3.70× vs 3.01× lift), it produces a smooth score rather than sixteen buckets, so the cut-off can be moved when the budget moves, and its standardised coefficients still line up with the tree's story rather than contradicting it.

what was rejected: deploying the tree because it is explainable (it leaves 200 findable churners on the table every month), and presenting the regression because it scores better (a committee cannot interrogate nine standardised coefficients, so they will either rubber-stamp it or kill it — both bad).

the one-page recommendation, as it would be sent:

recommendation. adopt the logistic regression as the scoring engine for the monthly retention list. adopt the depth-4 decision tree as the explanation layer published alongside it, and re-fit both quarterly on the same split.

why. the regression identifies churners at 3.70× the base rate in the contacted decile against 3.01× for the tree — about 200 additional at-risk customers reached per month at no extra cost. the tree gives the committee, the regional teams and the regulator a version of the logic they can read: salary credit stopping is the dominant signal, amplified by a falling balance and any complaint on file.

what this model cannot do. precision at the operating threshold is 7.7%. twelve in thirteen calls reach a customer who was not leaving. the correct measure of success is therefore not "were we right" but "did the retained cohort beat a randomised control" — so 10% of the monthly list should be held back untreated from day one.

known blind spots. tenure, branch visits and mortgage page visits contribute nothing to the tree. that is a property of a shallow tree, not evidence they are irrelevant — branch visits remain positively associated with churn in the regression and should stay in the feature set.

the honest caveat. this is a propensity model, not a causal one. it says who is likely to leave. it does not say who can be persuaded to stay. the next piece of work is an uplift model on a randomised retention offer.

the depth sweep is the part worth keeping even if everything else is forgotten. an unconstrained tree grew 712 leaves and scored 0.654 — worse than one with four. more model, less knowledge. the constraint was not the compromise. the constraint was the finding.