Technical Showcase

Our initial GraphSAGE model scored a perfect 100.0% accuracy on the test set. In machine learning, a perfect score is usually a warning sign that the model is cheating rather than learning. We paused the project to conduct a thorough code audit, treating this perfect number with scientific skepticism rather than celebrating.

Our audit revealed two structural leaks: first, standard random splitting placed some transactions from the same mule ring in the training set and others in the test set, allowing the model to peek at connected answers. Second, our synthetic data generator accidentally linked mules to shared devices, giving the model an unrealistic shortcut to flag fraud without learning transaction graph structures.

We refactored the split logic to use component-disjoint splits, holding entire mule rings out together in the test set. We also injected realistic noise, such as legitimate families sharing devices and natural multi-hop transaction patterns. This forced the GNN to learn real transaction topologies instead of relying on metadata shortcuts.

With the leaks fixed, our tabular baseline (Isolation Forest) scored a weak 64.7% AUC, proving tabular data is insufficient. GraphSAGE scored 99.9% AUC on the clean full graph, and 96.7% AUC under a 35%-edge-masked robustness test. This edge-masked score simulates real-world conditions where transaction visibility is incomplete.

We didn't stop at our own synthetic graphs. We validated the GNN on the public IBM AMLSim (HI-Small) benchmark—a NeurIPS 2023 dataset with 56,000 accounts and 6,357 labeled laundering profiles across 8 distinct typologies. The model was evaluated on structure it had never seen before.

GraphSAGE achieved a strong, believable 85.49% AUC on the IBM AMLSim benchmark. This lower performance on real-world simulated topologies is expected—real-world data has higher complexity and less obvious tells, validating the model's capacity to generalize without overfitting to synthetic patterns.

During a recalibration run, the evaluation scores shifted slightly. Rather than ignoring the variation, we tracked it down to unseeded randomness in PyTorch's execution and Python's default hash ordering. We fixed the random seeds and dictionary sorting across the entire training pipeline to guarantee exact results.

The model was frozen and verified across three independent training runs. The resulting split hashes and accuracy metrics matched exactly to six decimal places, locking in 85.4943% AUC and 30.7471% F1-score (F1-optimal threshold). These locked parameters represent a fully reproducible, robust, and empirically validated GNN detection engine.