henil08/verilog-coder-7b-lora

verilog-coder (7B, LoRA)

Qwen/Qwen2.5-Coder-7B-Instruct fine-tuned for Verilog/SystemVerilog generation from a natural-language spec (optionally with the module interface). Code, data pipeline, and evaluation harness: https://github.com/Henil-08/verilog-coder-ft.

Training data

OpenRTLSet: 131k Verilog modules from GitHub, each paired with an LLM-written description. After dropping 54.6k fuzzer-generated modules, deduplicating, and capping examples at 2048 tokens, 53.7k remain; this run used 40,000 of them (~93% Verilog, ~7% SystemVerilog). About 30% of training prompts describe the ports in prose instead of giving the header as code, which keeps the model usable when no header is provided.

The raw corpus contains UVM testbench code in only 3 rows, and none of them survive filtering. This model was not trained or evaluated on UVM and makes no claim of UVM capability.

Training setup

• Method: LoRA, 1 epoch, 29,906,052 supervised tokens
• LoRA rank 64, alpha 128, dropout 0.05, targets: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• LR 0.0001 (cosine, 3% warmup), effective batch 64, max length 2048, bf16
• HF Transformers + PEFT + DeepSpeed ZeRO-2 on 1× H100 (Modal); 79 min of training, ~$5.37
• Loss on the code (assistant) tokens only; final eval loss 0.2653
• Selected among LoRA/QLoRA/full-fine-tune configs by pass@1 on a validation split, so the test numbers below were not used for model selection.

Evaluation (pass@1, greedy decoding)

Benchmark	Base	Fine-tuned	Items gained / lost
OpenRTLSet held-out (diff-sim)	33.2% (131/394, 95% CI 28.8–38.0)	46.5% (183/394, 95% CI 41.6–51.4)	+59 / -7 (McNemar p<0.001)
VerilogEval v2 spec-to-RTL (published prompts)	31.4% (49/156, 95% CI 24.6–39.1)	23.7% (37/156, 95% CI 17.7–31.0)	+10 / -22 (McNemar p=0.050)
VerilogEval v2 + interface code block	38.5% (60/156, 95% CI 31.2–46.3)	28.8% (45/156, 95% CI 22.3–36.4)	+9 / -24 (McNemar p=0.013)

• OpenRTLSet held-out: 394 standalone modules not seen in training. A generated module passes if it matches the reference's ports and its outputs equal the reference's on every half-cycle of 500 cycles of random stimulus (Verilator). Random stimulus is evidence of equivalence, not proof.
• VerilogEval v2: the official testbenches and Icarus Verilog v12 pass rule, with this project's chat prompt (scores are not directly comparable to leaderboard numbers that use other prompts).

Limitations

• On VerilogEval this fine-tune scores below its base model (23.7% vs 31.4% with the published prompts). Fine-tuning on OpenRTLSet's real-world modules and LLM-written descriptions improved held-out OpenRTLSet modules but cost accuracy on VerilogEval's textbook-style problems; for that style of task the base model is the better choice.
• Greedy decoding sometimes falls into repetition loops on long modules; about 13% of eval outputs hit the 2048-token cap.
• Trained on single-module completions; multi-module designs and testbenches are out of scope. Not evaluated for synthesis quality, timing, or area.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM
base = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-7B-Instruct", dtype="auto")
model = PeftModel.from_pretrained(base, "henil08/verilog-coder-7b-lora")

Prompt format (system + user), as in training:

Specification:
<description>

Module interface:
```verilog
module ...(...);