When an open-source rule set beats a churn prediction model
A churn prediction model turns past cancellations into a probability for each current account. Kortix is the open-source CompanyOS: your agents, their skills, your company memory, and every connector in one git repo you own, with the agents working on real cloud computers.
Four signals, one combination rule
Usage is declining when core activity drops 20% or more against the prior 30 days, and dormant after 14 days with almost no activity. Support friction is rising when thread volume climbs above the account's history or threads stay unresolved. Cancellation language counts as escalated. Billing is at risk after a missed or failed payment. A renewal is imminent within 30 days. No single signal is high risk on its own. Rank accounts with declining usage, rising support friction and an imminent renewal or failed payment first. Give every listed account a reason and one next step. Never write to Postgres, Plain or Stripe, and never contact a customer.
What a churn prediction model is
A churn prediction model estimates a probability for each current account: the chance it cancels or stops using the product inside a set window. The model learns that estimate from accounts that already left. It compares their behavior in the months before cancellation with the behavior of accounts that stayed. Amplitude's guide to churn prediction describes the same three parts, a label, a set of behavioral features and a score that ranks accounts from low to high risk.
The label is a business decision before it is a data one. A subscription business may call an account churned when it does not renew; a product-led team may call 30 days without activity churn. That definition decides which accounts become training examples, and a model fitted to one definition will not score the other. The training window fixes when the features are read. For each account that churned in the past year, you take its usage and support history from the weeks before the cancellation. The model then sees the run-up rather than the aftermath.
Only then does the algorithm matter. Logistic regression, decision trees and random forests, neural networks and survival analysis all fit this shape of problem. The same guide sets out what each is good for. Regression gives readable coefficients, trees give rules a person can read, forests take mixed event data and neural networks want larger datasets. Survival analysis covers the case where the timing of churn matters as much as the fact.
The accuracy figure is the least useful number in a churn tutorial
A Python tutorial using the public Telco Customer Churn dataset trains a random forest on that file. It reports 0.78 accuracy on its test split, and its own confusion matrix counts 181 churners caught and 187 missed (checked November 2026). The figure measures the model against one snapshot of one consumer business, with its contract lengths, prices and churn definition baked in. A score fitted to that snapshot says nothing about a B2B book with annual contracts or a handful of cancellations a quarter.
The number is also the wrong output for the person who has to act on it. A probability tells the customer-success lead that an account is likely to leave. It does not say whether the account went quiet because the champion left, the integration broke or the invoice failed. Working that out is the Monday job, and it starts after the model has finished. The useful comparison, then, is what the system hands back at the end: a score, or a score with a reason and a next step attached.
What a rule set does instead
The churn-risk scan that Kortix runs scores accounts from four signals. Each signal is written as a rule a person can read. Usage is declining when core activity drops 20% or more against the prior 30 days, and an account is dormant after 14 days with almost no activity. Support friction rises when thread volume climbs above the account's own history or threads stay unresolved, and cancellation language counts as escalated. Billing is at risk after a missed or failed payment, and a renewal is imminent inside 30 days.
Each threshold uses the account as its own baseline, which is the quiet advantage of a rule set over a fixed cutoff. A 20% drop in core activity reads the same for a large account and a small one. It compares this month with that account's last month. Support volume is judged against the account's history for the same reason: an absolute ticket count would punish every large customer. Two weeks of near silence is longer than a normal gap for a product people use weekly. Dormancy at 14 days catches the account that has already stopped. A renewal inside 30 days matters because that window is where a conversation can still change the outcome. A failed payment is involuntary churn, which looks identical in the data and is usually fixable before the renewal.
Then one combination rule does the ranking. No single signal is high risk on its own, and accounts with declining usage, rising support friction and an imminent renewal or failed payment come first. That caution carries most of the value. A quiet month can be a holiday, a support spike can follow a launch, and a failed card is often an expired card. Three signals together narrow the explanation enough for a person to act.
Every entry in the list carries the signals that fired, a reason in words and one next step. The lead reads a sentence, not a dashboard. A threshold is also cheap to argue with. When the list is wrong, someone can point at the rule that produced it and change the sentence.
The trade, for a team with a few thousand accounts
| Dimension | Trained model | Rule set |
|---|---|---|
| Explainability | A probability, plus feature attributions | A stated reason per account |
| Cold start | Needs labeled churn history | Needs a baseline and agreed definitions |
| Maintenance | Retraining, monitoring, an owner | Editing thresholds in a file |
| Drift | Decays without saying so | Fails where someone can see it |
Cold start is where most model projects stall. A supervised model needs labeled examples, and the label is only honest if your churn definition held across the whole history. A team that loses a handful of accounts a quarter has very few labeled examples after a year. That is not enough to fit a model that will generalize to next quarter. A rule set needs a 30-day baseline per account and definitions people agree on, and both exist on day one.
Drift is the other cost, and it arrives quietly. Product changes move the behavior that a feature measures, and a change to contract terms moves the label; the model keeps returning numbers while its ranking decays. Rules fail in a way you can see. An account that should have led the list did not, and the rule that should have caught it is a sentence in a file.
Maintenance follows from both. A rule change is a short diff a teammate can review before the next run, and the scan picks it up without a rebuild. A model needs a training pipeline, a monitoring job and a threshold decision that someone owns, which is a standing cost for a team without a data scientist.
Explainability separates them for the customer-success team. A model can be explained with feature attributions, which say how far each input moved the score. A rule set states the reason in the same words the team would use on a call.
When a trained model is the right call
A trained model earns its cost when four things hold at once. The base is large, with many events per account. The history is clean across product, billing and support. Someone owns retraining and monitoring. And the decision needs a ranking at a horizon rather than an explanation of one account. Under a few thousand accounts, that combination is rare.
At scale a rule set starts to multiply. Segments, plan types and contract lengths each want their own thresholds, and the combination table grows past what anyone will maintain. Nonlinear interactions between dozens of event types are where a fitted model does something a rule cannot.
There is a useful order to the two. Build the rule set first and let it run for a quarter. Its list records the accounts the team acted on, the ones that churned anyway and the ones that stayed. That is training data with a known label and a known intervention. A model trained on that history starts from your book. A model trained on a public dataset starts from someone else's.
Running the scan every morning
The churn-risk agent runs on a cron at 06:00 UTC. It opens a session with read-only access to Postgres for usage, Plain for support threads and Stripe for payments and renewals. It scores every account and posts one ranked list to the customer-success channel in Slack. Only the Slack post leaves the run. Connector credentials are brokered server-side and never enter the machine. Hand it an outcome; agents plan the work, run it on their own computers at any hour, get it checked by a separate agent and a change request, and learn from it.
Changing the rules as the team learns
A monthly review closes the loop. List the accounts that churned, show which signals fired in the weeks before, and edit the threshold that produced the noise. In Kortix the rules live in a skill file in the project's git repo, so a change is a diff a teammate can read, and it reaches the default branch through a change request. Approval gates you set. Off until you set them.
None of this makes the model the wrong tool forever. A model becomes the better one once the base is large enough to support it and the history is clean enough to learn from. Until then, a rule set that names the reason and the next step is the one that changes what happens on Monday.
Churn prediction model questions
The definitions and rates people ask about most often.
- What is a predictive churn model?
- A predictive churn model estimates, for each current account, the probability that it will cancel or stop using the product within a set period. It learns the pattern from past accounts that churned and stayed, then scores today's accounts. The score ranks accounts; it does not explain them.
- What model to use for churn prediction?
- Logistic regression is the usual first choice, because each coefficient has a direction you can explain to a customer-success lead. Decision trees give readable rules, random forests average many of them to handle mixed event data, and neural networks need far more data and are harder to interpret, so they come later, if at all.
- Is a 5% churn rate good?
- Only when compared with your own baseline and the period it covers. Five percent a year and five percent a month describe very different businesses. Compare accounts with the same contract length and segment, and watch the trend rather than the absolute number.
- What does a 20% churn rate mean?
- It means one account in five, or a fifth of recurring revenue if you measure revenue churn, left during the period. Twenty percent a year is a slow leak; the same rate every month empties the base far faster. Always name the period when you quote the number.