AI-DRIVEN FINANCIAL TRANSACTION MONITORING: A HYBRID FRAMEWORK FOR FRAUD DETECTION AND AML ALERT PRIORITIZATION

Abstract

Banks and payment providers process billions of transactions a day, and hidden within that torrent is a small but costly stream of fraud and money laundering. The systems most institutions rely on to catch it were built decades ago around fixed rules—flag any transfer over a threshold, freeze any account touching a sanctioned country—and while those rules are transparent and auditable, they are also blunt. They generate enormous numbers of alerts, the overwhelming majority of which turn out to be false, and they miss the sophisticated schemes that deliberately stay under the radar. This paper presents a hybrid framework that combines rule-based screening with machine learning to both detect fraud more accurately and prioritize anti-money-laundering (AML) alerts so that investigators spend their time on the cases that matter. The framework layers a supervised fraud classifier, an unsupervised anomaly detector for novel patterns, and a learning-to-rank model that orders the resulting alerts by estimated risk, all sitting on top of a preserved rule layer that guarantees regulatory coverage. We evaluated it on a large, highly imbalanced transaction dataset and found that the hybrid approach substantially improved fraud detection while cutting the false-positive burden that consumes investigator time. The alert-prioritization component pushed genuine suspicious activity toward the top of the queue, so that a team reviewing only a fraction of alerts still caught most of the true cases. Compared against a rules-only baseline and standalone machine-learning models, the hybrid consistently delivered a better balance of catch rate and workload. Crucially, the design keeps the rule layer intact for auditability, addressing the explainability demands that regulators impose on financial institutions. We argue that the right way to modernize transaction monitoring is not to replace rules with black-box models but to let learning triage and prioritize what the rules surface.