drafter-mixed-ut

Domain-specific draft model for speculative decoding, trained on Mixed (Understanding + Text Reformulation) tasks.

Model Details

Parameter	Value
Base model	Lite-Mistral-150M-v2-Instruct (156M params)
Architecture	MistralForCausalLM
Target model	TurboSparse-Mistral-Instruct (7B)
Domain	Mixed (Understanding + Text Reformulation) (32 Flan clusters)
Training samples	722K
Epochs	5
Training time	12.6 hours (1x RTX 3090)
Loss	Mixed: 0.5 x CE + 0.5 x KL (T=1.0)
Final eval_loss	1.788
Final top1_accuracy	56.64%
Overlap Area (AR proxy)	0.7191 on own domain

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-mixed-ut")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-mixed-ut")

Training Data

MikhailRudenko/domain-aware-sd-synthetic

Citation

Part of the Domain-Aware Speculative Decoding research project: GitHub