Chasing ML for fraud detection — and ending up with rules instead

Earlier this year I took on a part-time project for a telecom operator: see if machine learning could replace, or at least augment, their existing fraud and abuse detection. The brief was about as open as it gets — try to make this AI-first, figure out how, here's the data, ask if you have questions. The data was Call Detail Records (CDRs): one row per call, with fields like duration, cost, source and destination numbers, status, and a couple of timestamps. In practice, the people who understood the data best were busy running the systems that produced it, so a lot of the "asking questions" turned into reverse-engineering the data myself and checking my guesses when I could get ten minutes with someone.

This is a writeup of what I tried, in roughly the order I tried it, what each attempt taught me, and why I ended up walking away from the idea of a magic-bullet model entirely. Names, exact configuration values and anything that would make this traceable to a specific company are left out — what's left is, I think, a pretty honest account of what "do ML fraud detection" actually looks like once you're past the tutorial dataset.

1. The real problem wasn't the model#

I spent weeks trying different modeling approaches before it clicked that every single one of them was getting blocked by the same handful of issues in the raw data. None of these are exotic — if anything, they're the kind of thing you'd expect from any sufficiently old, sufficiently large operational system — but together they make "just throw an anomaly detector at it" a much harder proposition than it sounds.

• The chicken-and-egg problem of ground truth. To train or even properly evaluate a fraud model, you need verified examples of fraud. To get more verified examples, ideally you'd have a better detection system. The incident data I was handed was real and useful, but by construction incomplete — it's whatever got caught, reported and confirmed by other means, not a representative sample of all fraud that happened. Any model trained on it inherits that selection bias, and any metric computed against it is a lower bound at best.

• Timestamps are not as simple as they look. A CDR typically carries at least two timestamps: one for when the call was set up, and one for the last time the record was touched. The first tells you when a call started, not whether it's still running. The second is more current but gets updated by all sorts of unrelated housekeeping, not just "the call changed state." Picking the wrong one — or not realizing there's a choice to make at all — quietly breaks any time-windowed analysis downstream.

• Cumulative metrics fool anomaly detectors. Cost and duration for an ongoing call only ever go up. A perfectly normal, long-running call looks to a generic anomaly detector exactly like a steadily escalating one, because mathematically it is one. This single property generated more false positives in my experiments than anything else, and it's not something you can parameter-tune your way out of — the feature itself needs to be a delta, not a running total.

• "Anomalous" is relative to the user. What counts as an unusually expensive call depends heavily on which plan a customer is on. A global threshold either misses the users for whom that cost is genuinely unusual, or constantly flags the users for whom it's routine. Anything useful has to be relative to an individual's own history, not a population-wide cutoff.

• One call, several records. A single logical call can generate more than one database record, particularly around setup, as more information becomes available. Naively counting or aggregating records instead of calls silently double-counts activity — a trap that's easy to fall into and genuinely annoying to catch.

• You're working with a subset of a subset. The data available for analysis was itself a projection of a larger, primary dataset, and a short retention window at that — a matter of months, not years. That's enough to catch short-term behavioural shifts, but it rules out anything that needs to learn what "normal" looks like over a full yearly cycle, and there was no guarantee a field I might have wanted was even exposed in this copy of the data.

None of this is really a "problem" to be fixed at the source — it's the normal shape of an operational dataset that happens to have been generated by a billing and routing system, not designed up front for data science. The implication, though, is important: unless a model (or a human-designed system) is explicitly built to handle these quirks, the signal from genuine fraud gets lost in structural noise long before it gets anywhere near an algorithm's decision boundary.

2. Five approaches, four dead ends#

2.1 Off-the-shelf models#

The first thing worth checking, before building anything, is whether somebody has already solved this. The search engine I had available for the data ships with a machine-learning plugin offering a library of pre-built models, and the wider ecosystem around it points to the usual hub of community models for anything off-the-shelf. Almost all of them turned out to be NLP models — sentence transformers and the like, built to turn text into vectors of meaning. They're genuinely powerful tools, just for a different problem: they have no built-in notion of call duration, cost sequences, or per-user behaviour over time, because they were never meant to. Structured, temporal, numeric data isn't text, and no amount of creative feature-to-text encoding closes that gap convincingly. I dropped this line fast — it's not that the models were bad, it's that I was pointing a very good hammer at a problem that wasn't a nail.

2.2 Graph neural networks, in theory#

Next I went looking at the academic literature, partly to see how the state of the art tackles telecom fraud and partly to refresh my own footing on current neural network architectures. A lot of the recent work is genuinely compelling: Graph Neural Networks, in particular, are a natural fit for the problem, since who calls whom is literally a graph, and GNNs are built to exploit exactly that kind of relational structure.

Two things got in the way of turning any of this into something usable. First, the data-mapping problem: these papers are trained and evaluated on specific, carefully structured research datasets, and translating a real-world CDR schema into the feature format a given paper expects is its own non-trivial research project, not a weekend's work. Second, and more mundane, is what I ended up calling "code rot": most papers simply don't publish code, and of the ones that do, the repositories are frequently built against long-superseded library versions, with dependency chains that no longer resolve on a modern machine. One paper¹ stood out precisely because it did publish code, so I used it as a case study — and it confirmed both problems at once: the feature mapping didn't line up with our schema, and the repository itself needed an afternoon of archaeology before it would even install.

The literature review wasn't wasted effort, though. Along the way I came across the GSMA's International Revenue Share Fraud (IRSF) database² — an industry-maintained, continuously updated list of high-risk destination numbers used in this specific type of fraud. That's immediately actionable in a way a research paper usually isn't: a lookup table you can join against today, no model required.

2.3 A custom model, from scratch#

The obvious next question was what it would actually take to build something bespoke. Conversations here were more useful than any amount of solo research: on the infrastructure side, the data pipelines needed to gather and prepare training data at scale didn't exist yet and would take real engineering time to build, the retention window limited how much history was available, and meaningful compute and storage for training would need a formal resourcing request, not something I could spin up ad hoc. On the ground-truth side, the incident data I had was small and, as above, not a reliable or complete positive-label set.

One idea that came out of talking this through was to treat that small incident set as seed data and lean on synthetic data generation to grow it into something a model like XGBoost could learn from. It's a reasonable long-term strategy, but it was clearly going to take longer to bear fruit than the time I had, and at that point the next approach — using the search engine's own built-in anomaly detection — looked like a faster path to a concrete result. I shelved the custom-model idea rather than abandoning it; it's the right answer eventually, just not the right answer for a part-time, time-boxed engagement.

2.4 Anomaly detection, plus a lot of post-processing#

This is where most of the time on the project actually went. The plan was to use the search engine's native anomaly detection plugin — built on Random Cut Forest, an algorithm that's reasonably good at handling seasonality, which matters a lot for telecom traffic — and wrap it in a custom, time- and context-aware post-processing layer written in Python. Detectors were configured against key metrics (call counts, cost, duration) with a mix of short and long detection windows, trying to catch both sudden spikes and slower, day-over-day drifts.

It went through three real iterations, and each one taught me something the previous one hadn't surfaced:

• First pass. Simple spike detection on volume, cost and error rate. This is where the timestamp choice first bit: using the call-setup timestamp caused the analysis to silently miss calls that had been set up earlier but were still active. The detectors also had no concept of per-entity baselines — they looked at aggregate behaviour across everyone, so one user going haywire was invisible against the noise of everyone else. Output was far too noisy to act on.

• Second pass. Switched to the other timestamp, added a per-user discriminator, and ran a proper evaluation against known incidents for the first time. The result was blunt: thousands of false positives, and zero of the known incidents detected — a 100% false-negative rate on the only ground truth I had. Digging into why surfaced two more structural issues: the discriminator I'd chosen (source IP) didn't actually match how the incident data itself was labelled (by user), making evaluation apples-to-oranges from the start; and the "new" timestamp had its own pollution problem, since cumulative fields kept climbing across record updates that had nothing to do with the call itself.

• Third pass. Narrowed the feature set down to the two metrics that mattered most, and pre-aggregated the data into deltas — change in cost and duration per user per interval — specifically to kill the cumulative-metric problem at the source rather than fighting it downstream. This also meant wrestling with the search engine's indexing internals to make a user identifier usable as a grouping key, which turned out to need a schema change, not just a query change. Even with all of that, the outcome didn't move: still a high volume of false positives, still nothing from the known-incident set detected.

After three iterations with a consistent result, I called it. The detectors from this effort are still configured and technically running, but nobody should trust their output as-is. The honest lesson here wasn't about tuning harder — it was that I'd been too optimistic, too early, that the core anomaly signal would be there at all once enough post-processing was layered on top. In hindsight, the right move would have been to spend a day confirming the raw detector could flag even one known incident before investing weeks in the filtering logic around it.

2.5 A transparent risk-scoring system#

The last approach was a deliberate change of philosophy: instead of a black-box anomaly detector trying to decide "anomaly, yes or no," build something transparent that scores "how risky is this behaviour, and why." For each user and each metric (cost, duration, call count), the system compares the current value over a short window against that same user's own historical baseline for an equivalent window, turns that comparison into a ratio, multiplies it by a hand-set weight reflecting how indicative that metric is of fraud, and sums the weighted ratios into a single risk score per user. Every number in that score is traceable back to a concrete cause, which matters enormously when the output eventually needs to be explained to someone who isn't going to take "the model said so" as an answer.

Architecturally it came together in three stages: scheduled queries aggregate raw CDRs into per-user time-series data; a small Python service picks up the latest aggregates, computes the deltas and weighted score, and pushes the result on; and that score lands in a metrics and alerting stack, so a sustained high score for a user can trigger an actual alert rather than sitting in a dashboard nobody checks.

The core pipeline — search index in, Python scoring service, metrics backend out — got built and worked end to end, which on a project with this much prior churn felt like real progress. It wasn't a clean finish, though. The export format I'd built around for the real-time path turned out not to support custom timestamps, which only became a problem once a second requirement showed up: replaying the system against past, backdated data to validate it against history. That forced a second export path using a different, more flexible format, and by the end there were three slightly different variants of the same exporter doing overlapping jobs — workable, but not something I'd call finished. The original estimate for this whole project was a week and a half of full-time-equivalent work; by the time this approach was fully scoped it had already eaten several times that, and there wasn't runway left to fully tune and validate it against history the way it deserved.

3. What actually shipped#

The data-collection and scoring pipeline for the risk-scoring system is in place and running, with a first, deliberately simple set of weights. It's being actively tested against the known incidents to get a real baseline read on precision and recall before anyone tunes it further. If a single global set of weights turns out not to be good enough — which, given everything above, is the likely outcome — the planned next step is to segment users by type or usage pattern and give each segment its own tuned weights, rather than trying to find one configuration that fits everyone. A known limitation going in is that it only looks at calls after they've ended; extending the same logic to score calls while they're still in progress is on the list but wasn't in scope for this phase.

Nothing here is a finished fraud-detection product. What exists is a transparent, inspectable foundation that's honest about what it does and doesn't know, built on top of a much clearer understanding of why the obvious approaches didn't work.

4. The lesson#

If I had to compress a part-time project's worth of dead ends into one sentence: the hard part was never finding a sufficiently clever algorithm, it was designing something that could work with the specific, structurally messy nature of the data in front of me. Timestamp ambiguity, cumulative metrics, per-user relativity, incomplete ground truth — none of these are solved by a better model architecture, because they're upstream of modeling entirely. A sophisticated algorithm fed confused inputs just produces confused — and confidently false — outputs faster.

That's not a knock on machine learning so much as a reminder of where it actually adds value. Every "failed" attempt here — the NLP mismatch, the GNN papers, the custom-model infrastructure gap, three rounds of anomaly-detector tuning — narrowed down the real shape of the problem, and that narrowing is what eventually made a simple, transparent, rules-based system the right call instead of a consolation prize. Sometimes the most useful output of an open-ended "go figure it out" mandate isn't a model at all. It's finally understanding your own data well enough to know what would actually need to be true for a model to help.