drafter-text-reformulation

Domain-specific draft model for speculative decoding, trained on Text Reformulation tasks.

Model Details

Parameter	Value
Base model	Lite-Mistral-150M-v2-Instruct (156M params)
Architecture	MistralForCausalLM
Target model	TurboSparse-Mistral-Instruct (7B)
Domain	Text Reformulation (11 Flan clusters)
Training samples	327K
Epochs	5
Training time	4.2 hours (1x RTX 3090)
Loss	Mixed: 0.5 x CE + 0.5 x KL (T=1.0)
Final eval_loss	2.151
Final top1_accuracy	54.34%
Overlap Area (AR proxy)	0.7026 on own domain

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("MikhailRudenko/drafter-text-reformulation")
tokenizer = AutoTokenizer.from_pretrained("MikhailRudenko/drafter-text-reformulation")

Training Data

MikhailRudenko/domain-aware-sd-synthetic

Citation

Part of the Domain-Aware Speculative Decoding research project: GitHub