drafter-understanding

Domain-specific draft model for speculative decoding, trained on Understanding tasks.

Model Details

Parameter	Value
Base model	Lite-Mistral-150M-v2-Instruct (156M params)
Architecture	MistralForCausalLM
Target model	TurboSparse-Mistral-Instruct (7B)
Domain	Understanding (21 Flan clusters)
Training samples	395K
Epochs	4.5 (early stop from 25)
Training time	4.3 hours (1x RTX 3090)
Loss	Mixed: 0.5 x CE + 0.5 x KL (T=1.0)
Final eval_loss	1.193
Final top1_accuracy	65.05%
Overlap Area (AR proxy)	0.7558 on own domain

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-understanding")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-understanding")

Training Data

MikhailRudenko/domain-aware-sd-synthetic

Citation

Part of the Domain-Aware Speculative Decoding research project: GitHub