Qwen3-Coder-30B-A3B-Audit

A Solidity security-audit model fine-tuned from Qwen/Qwen3-Coder-30B-A3B-Instruct on 10,821 real vulnerability findings adjudicated by human auditors in public audit contests.

Given a Solidity snippet, it returns a JSON array of security findings — severity, category, function, line range, and a one-line reason — or an empty array when the code is clean.

Fine-tuning cuts false positives by 75% and raises the clean-code pass rate from 5% to 73%. The base model flags a vulnerability in 95% of clean contracts; this model does not.

Serving

vllm serve qtum/Qwen3-Coder-30B-A3B-Audit --max-model-len 16384

A Mixture-of-Experts model with 30B total and ~3B active parameters — throughput close to a 3B dense model at the capability of a much larger one. bf16 weights, ~57 GB.

Prompt format

The model is trained for one specific system prompt. Deviating from it degrades output quality.

<|im_start|>system
You are a smart contract security auditor. You are given a Solidity code snippet taken from a
real audited codebase. The comment `// ... (第 A-B 行)` tells you the original file line numbers
of the snippet body.

Report only real, exploitable security vulnerabilities. Do NOT report style issues, gas
optimizations, or speculative concerns. Most snippets contain at most one or two real issues;
many contain none.

Respond with ONLY a JSON array. Each element must have exactly these fields:
  "severity": "High" | "Medium" | "Low"
  "category": one of the following slugs, nothing else:
- reentrancy
- access-control
- input-validation
- arithmetic
- accounting
- oracle
- mev-slippage
- dos
- liquidation
- rewards-fees
- token-integration
- signature-replay
- init-upgrade
- crosschain
- flashloan-donation
- callback-hook
- other
  "function": the name of the function containing the bug
  "lines": the original file line range of the bug, e.g. "412-418" (keep it tight, under 40 lines)
  "reason": one sentence explaining the vulnerability

If the snippet has no real vulnerability, respond with exactly: []
No prose, no markdown fences, no explanation outside the JSON array.<|im_end|>
<|im_start|>user
Audit this snippet:

```solidity
{contract_code}
```<|im_end|>
<|im_start|>assistant

Recommended sampling: temperature=0, max_tokens=4096, thinking disabled, one sample.

Greedy decoding is not just the default here — it is the best configuration we measured. Drawing five samples at temperature=0.7 and taking the union raises recall from 0.080 to 0.147 but pushes false positives from 0.23 to 2.78 per snippet; filtering that union by agreement across samples (keep findings seen in ≥2 of 5) lands at recall 0.077 / FP 0.38 — strictly worse than a single greedy sample on both axes. Vote count does carry a weak signal (hit rate rises from 3.4% at ≥1 vote to 10.2% at ≥4), but not enough to separate the extra findings from the noise.

Results vs. base model

Evaluated on two independent held-out sets built from audit contests the model never saw: EVAL-A (Code4rena, 383 questions / 313 gold findings) and EVAL-B (Sherlock, 804 / 765). Both sets include 30–35% clean contracts to measure false positives.

Metric	Base	This model	Change
Clean-code pass rate (EVAL-A / B)	0.05 / 0.07	0.73 / 0.64	14×
False positives per question	0.56 / 0.52	0.23 / 0.26	−58%
Severity accuracy	0.38 / 0.36	0.61 / 0.53	+61%
Location + category F1	0.089 / 0.094	0.281 / 0.236	+215%
End-to-end F1 (semantically verified)	0.010 / 0.043	0.081 / 0.057	+8.1× / +1.3×

Confirmed findings rose from 4/313 to 25/313 on EVAL-A and 36/765 to 41/765 on EVAL-B. Pooled across both sets the improvement is statistically significant.

The largest gains are on the false-positive side, and recall rises together with precision — the model is not simply reporting less.

Putting the numbers in context

The gold set is the union of every finding submitted by every participant in a contest — typically 37 auditors per contest. No individual reaches it. Measured across 155 Sherlock contests using the published "Found by" rosters:

	Share of a contest's findings
All participants combined (= the gold set)	100%
Best single human auditor in a contest	median 59%
Median human participant	median 12%
This model	8.6%
Base model	1.3%

On that scale the model performs roughly like a median contest participant rather than "9% of a perfect score". Three caveats, in both directions: the tasks are not equivalent and the difference favours the model (auditors work a whole codebase and must write PoCs; the model is handed a function-level snippet that contains the bug); the 12% median includes many low-effort submissions; and the semantic verifier used here is conservative — it rejects 23% of genuine paraphrases, so the true figure is likely 10–11%.

The meaningful target is not 100% but the ~59% a strong human auditor reaches.

Examples

Each example is verbatim output on a held-out contract the model never saw during training.

Oracle staleness — EthOracle.sol (bakerfi)

// gold:  [Medium / oracle]  Min and maxAnswer never checked for oracle price feed
// base:  wrong reason, over-rated severity
[{"severity":"High","category":"oracle","function":"getLatestPrice",
  "reason":"Missing check for stale price data ..."}]
// this model:
[{"severity":"Medium","category":"oracle","function":"getLatestPrice","lines":"20-44",
  "reason":"Chainlink's `latestRoundData` can return stale or incorrect results"}]

Signature replay — TimelockTokenPool.sol (Taiko)

// gold:  [High / signature-replay]  Signatures can be replayed in `withdraw()`
// base:  misclassified as an access-control issue
[{"severity":"High","category":"access-control","function":"withdraw",
  "reason":"The withdraw function allows anyone to specify any recipient address ..."}]
// this model:
[{"severity":"High","category":"signature-replay","function":"withdraw","lines":"158-183",
  "reason":"TimelockTokenPool#withdraw can be called multiple times"}]

Wrong exchange rate — xRenzoDeposit.sol (Renzo)

// gold:  [Medium / oracle]  Incorrect exchange rate provided to Balancer pools
// base:  flagged an unrelated function entirely
[{"severity":"High","category":"access-control","function":"setAllowedBridgeSweeper",
  "reason":"The function lacks proper access control validation ..."}]
// this model:
[{"severity":"Medium","category":"oracle","function":"getRate","lines":"446-468",
  "reason":"`xRenzoDeposit`'s `getRate()` function returns wrong value"}]

Knowing when to stay silent — PeripheryPayments.sol, a clean contract

// gold: []
// base: hallucinates a High-severity reentrancy
[{"severity":"High","category":"reentrancy","function":"unwrapWETH9",
  "reason":"... transfers ETH without checking if the recipient can reenter ..."}]
// this model:
[]

Training

Method — LoRA (rank 32, alpha 64, dropout 0.05) on attention projections (q_proj, k_proj, v_proj, o_proj); 26.7M trainable parameters, 0.0875% of the base. Loss is computed on the assistant response only. 2 epochs, lr 1e-4 cosine, sequence length 6144, bf16 on 2×H100. Adapter merged into the base weights for this release.

Data — 3,492 examples derived from public audit-contest findings:

Source	Code4rena (363 contests) + Sherlock (226 contests)
Raw findings parsed	10,821 — of which 5,060 carry file path and line anchors
Positive examples	2,407 function-level snippets paired with their adjudicated findings
Negative examples	1,268 clean contracts from the same contests (35%)
Severity balance	High:Medium oversampled to 1742:2145

Two design choices mattered more than hyperparameters:

• Negatives at 35%. Without clean-code examples the model reports a vulnerability in almost every snippet — the base model's clean-code pass rate is 0.05. This ratio was chosen to match the evaluation prior, and it is the single largest contributor to the false-positive reduction.
• Multi-finding targets. Snippets containing several adjudicated findings are trained as a multi-element array. An earlier version trained one finding per example and learned to never emit more than one, capping recall by construction.

Evaluation methodology

Numbers above come from a purpose-built benchmark rather than a generic code-security set.

Leak prevention, three gates. Evaluation uses only contests from 2024 onward; any protocol appearing on both sides of that boundary is removed entirely (16 protocols, 121 findings — a time-only split would have leaked); and every evaluation snippet is fingerprinted against the training corpus to catch forked code.

Scoring, three layers. A prediction counts only if (1) its function or line range overlaps the gold anchor, within a 40-line span cap, (2) its category matches the gold category from a controlled 17-slug vocabulary, and (3) a separate verifier model confirms the two descriptions refer to the same bug. The third layer removes roughly 60% of rule-level matches — the same function often contains more than one issue.

The scorer is itself tested, in both directions. An all-empty baseline must score 0; a hardcoded five-template baseline that never reads the code must score ≤0.10 (measured: 0.020); and feeding the gold answers back in must score ≥0.85 (measured: 0.891). The upper-bound test is not optional — it is what catches a scorer that silently under-counts correct answers.

What does not help

These were measured on the same held-out benchmark and are reported so you do not spend time re-discovering them. All comparisons use hit counts against 313 gold findings with Poisson standard errors; "no effect" means within two standard errors.

Idea	Result
Feeding whole files instead of function-level snippets	Worse. On the base model — a fair control, since it was trained on neither format — recall is unchanged (0.013 → 0.013) while false positives rise 2.7×. The findings this model misses are not missed for lack of surrounding code.
Multi-sample decoding, with or without agreement filtering	No threshold beats one greedy sample. See the sampling note above.
A stronger base model	Fine-tuning Qwen3-32B on identical data lands at 0.058 / 0.084 (EVAL-A / B) versus 0.081 / 0.057 here — +1.2 standard errors pooled, indistinguishable. Notably the bare-base ranking does not survive fine-tuning: bare Qwen3-32B scores 3× the bare Coder-30B, yet the two fine-tunes converge.
Supplying the protocol's specification	No effect. Adding a compact protocol brief to the prompt: 25 → 22 hits. Adding the contest's full audit brief — which literally contains an "attack ideas / where to look for bugs" section — gives 25 → 26.
Retrieval-augmented prompting or retrieval-aware training (RAFT)	Retrieval lifts the base model (0.010 → 0.040) but adds nothing after fine-tuning, and RAFT is indistinguishable from plain SFT on two independent evaluation sets. Whatever the retrieved patterns supply, fine-tuning has already absorbed.

The consistent pattern is that extra context, extra samples, extra parameters and extra domain knowledge all fail to move the number. Seven configurations land between 0.054 and 0.084. That band appears to be the ceiling of the task as posed — one snippet in, one JSON verdict out, no iteration and no verification — rather than a limit of any particular model or dataset.

Limitations

This model assists human auditors; it does not replace them.

• Absolute recall is low. 25 of 313 gold findings on EVAL-A — about 9% of the ceiling a perfect answer would reach on the same metric. Treat its output as a triage signal.
• Roughly 0.25 false positives per snippet. Far below the base model, but every finding still needs human confirmation.
• Function-level snippets only. Trained and evaluated on single functions with surrounding context, not whole repositories or cross-contract call graphs. Vulnerabilities that only appear across contract boundaries are out of scope.
• Solidity / EVM only.
• Fixed output schema. The 17-category vocabulary and JSON shape are baked in by training.

Details

Base model	Qwen/Qwen3-Coder-30B-A3B-Instruct
Method	LoRA r=32 → merged
Parameters	30.5B total / ~3B active (MoE)
Precision	bf16
Context	16K recommended (32K supported by base)
Task	Solidity vulnerability detection, JSON output
License	apache-2.0 (inherited from base model)

---

Fine-tuned by qtum. Training data is derived from publicly published smart-contract audit contest reports. Findings produced by this model are automated suggestions and are not a security guarantee; independent review is required before relying on them.