I built this, so disclosure first: this is my stablecoin riskgate model, and I want the benchmark torn apart, not praised.

The narrow job: before a locally-run agent signs a stablecoin payment, decide approve / hold / reject. The payment itself is irreversible once signed, so the decision has to happen before signing, and the checks run offline — no API call at decision time.

Base Qwen3-4B got 59.8% of risk-gate decisions right on my 1,175-case test split, and approved 34.6% of unsafe payments. After fine-tuning on synthetic labels from a deterministic rule engine, the same 4B model got 99.5% right and approved only 0.47% of unsafe payments (4 of 847, all prompt injections).

The part I am less sure about, and the reason I am posting here rather than just announcing numbers: the dispute side is weaker, at 77.8%, and it over-denies valid refunds. And the honest question behind the fine-tune numbers: did the model learn the task, or did it learn the generator’s habits? The labels are synthetic, the generator is not public, and the benchmark checks 0 exact-row / 0 ID leakage against training, but that does not settle the generator-habits question.

One deterministic check I trust more than the model alone: a new payee that shares 7 or more characters with a known payee, at least 3 at each end, is rejected outright as a lookalike. A random address matches that pattern about once in 140 million tries. A merely new payee is just held for a human to confirm. That check was verified clean in an audit of core 0.15.0 (471 tests pass, 3 skip).

A newer delegated-account check shipped in SDK 0.16.0 (eth_getCode, hold when code is a 0xef0100 delegation to a delegate outside an allowlist) — that one is NOT yet audited, so I am not claiming it as verified.

Runs locally: ollama pull sriram1983007/sra-riskgate ollama run sriram1983007/sra-riskgate

Links: model https://huggingface.co/sriram1983007/SRA-RiskGate-4B — benchmark https://huggingface.co/datasets/sriram1983007/sra-stablecoin-risk-bench — code https://github.com/sriram1983007-dev/sra-riskgate — Apache 2.0, 8,908 last-month downloads on the main model.

What would you test first to separate “learned the task” from “learned the generator” — a held-out human-labelled set, adversarial payees, or something else?