Decisions · Technology

Should loyalty use machine learning, an LLM, or rules

Use deterministic rules for every member-visible accrual, tier, and redemption calculation, because those are liabilities you must be able to audit. Deploy machine learning only in back-office scoring for fraud or churn where outcomes are already labelled and a human can review. Do not put an LLM in the live balance path.

The decision

The decision is not whether to adopt machine learning or an LLM, but which part of the loyalty system may be non-deterministic. A loyalty programme has two distinct clocks. The first is the balance ledger: earn, burn, expire, tier. The second is the inference layer: who will churn, who is defrauding, which offer might lift spend. A rule is an explicit conditional that anyone can read. Machine learning is a fitted model that optimises an objective on past behaviour. An LLM is a token predictor that generates plausible language, not arithmetic. Loyalty balances are a liability, so the ledger should run on rules. The inference layer can tolerate probabilistic output because a human reviews it before any member-facing action.

What it costs you

The recommended split has real costs. Rules do not write themselves; every promotion, partner earn rate, or expiry exception is a change request. That consumes engineering time and creates a backlog. Machine learning in the back office needs labelled outcomes, model retraining, and monitoring; if those stop, performance decays silently. Keeping the LLM out of the balance path costs you some automation convenience, but that is the point. The cost of an incorrectly approved rule is fixed and calculable; the cost of an LLM's plausible but wrong credit is unbounded until a member complains.

Illustrative calculation. Assume a programme with 100,000 members and a base earn rate of one point per dollar. A rule change that accidentally grants 2 points per dollar for one day on a partner with 50,000 dollars of daily sales creates 50,000 extra points. At a conservative liability of 0.8 cents per point, that is 400 dollars in extra redemption cost. If the same error comes from an LLM that invents a "double points for everyone" bonus and 100 members redeem an extra 10,000 points each before correction, that is 1,000,000 points, or 8,000 dollars at 0.8 cents per point. The rule error is visible in a code diff; the LLM error is visible only after redemptions.

How to decide

Apply the balance test. Trace any proposed feature from input to output. If the output debits, credits, expires, or tiers a member, it must be a rule. If the output is a probability or ranking that a human then approves, it may be machine learning. If the output is natural language sent to a member, it may come from an LLM only when a human confirms the text and the programme has already encoded the stated terms as a rule. Ask what happens when the system is wrong and a member screenshots it. If the correction requires apologising for points already spent, the system is in the wrong layer.

What would change our mind

The recommendation flips when points stop being a balance sheet liability. If the issuer can and does void every erroneous LLM credit in real time before a member redeems it, and members have accepted those terms in advance, then an LLM may operate in the balance path because the failure cost is zero. Until that is true, no amount of prompt engineering makes a token predictor an auditor.

Related