mistral-7b-qlora-dpo

Mistral-7B-v0.1 fine-tuned with QLoRA + DPO on multiple-choice science/commonsense QA, improving ARC-Challenge (25-shot) from 61.35 → 65.27 acc_norm.

This repository contains a merged, bitsandbytes 4-bit (NF4) checkpoint: the LoRA adapter was merged into the 4-bit quantized base model with merge_and_unload() and pushed as-is, so the safetensors weights are stored quantized (~4.5 GB) and loading requires bitsandbytes on a CUDA device.

Approach

ARC-Challenge is evaluated (via lm-evaluation-harness) by computing the log-likelihood of each answer choice and picking the highest — no instruction following is involved. Standard supervised instruction tuning (e.g., on Alpaca-style data) actually degraded the base model's ARC score in our ablations, because the training objective was not aligned with the likelihood-ranking evaluation.

Instead, this model uses Direct Preference Optimization (DPO) to directly increase the likelihood of the correct answer relative to distractors: every multiple-choice question is converted into preference pairs (chosen = correct answer, rejected = each wrong answer), which matches the evaluation protocol exactly.

Results

ARC-Challenge, 25-shot, acc_norm, evaluated with EleutherAI lm-evaluation-harness (--batch_size 8, dtype=float16). The base-model row was measured with the same harness command on the unquantized fp16 model, which is why it can differ slightly from other published numbers for Mistral-7B-v0.1:

Setup	acc_norm
Mistral-7B-v0.1 (base)	61.35
SFT (QLoRA) on Alpaca 0.5k	58.96
SFT (QLoRA) on SciQ	55.38
SFT (QLoRA) on QASC	43.17
SFT (QLoRA) on CommonsenseQA	56.83
DPO on CommonsenseQA only	64.16
DPO on multi-dataset curriculum (this model)	65.27
SimPO on the same multi-dataset mix	62.12

Notes from the experiment series:

• All SFT variants scored below the base model — evidence that the failure mode was the training/evaluation mismatch (and likely catastrophic forgetting), not data quality.
• SimPO trained in about half the time of DPO (44 min 03 s vs 1 h 28 min 13 s) but scored ~3.2 points lower under this setup.

Training data

Three multiple-choice QA datasets (train + validation splits) were normalized to a common 4-choice format:

Dataset	Source	Questions used	Preprocessing
CommonsenseQA	tau/commonsense_qa	1,500	5 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled
QASC	allenai/qasc	500	8 choices → 4: gold answer + 3 randomly sampled distractors, order shuffled
OpenBookQA	allenai/openbookqa (config additional)	1,500	native 4 choices kept as-is

Each dataset was shuffled with seed=42 before subsampling. The three subsets were concatenated in the order above, following a curriculum-inspired design: datasets on which the base model scored higher in a no-training difficulty probe come first, the hardest last. (Since the trainer's default sampler shuffles examples each epoch, this should be read as the composition of the data mixture rather than a strict ordering guarantee.) SciQ and MMLU science subsets were also explored but excluded from the final mix.

Every question then yields 3 preference pairs (one per distractor), for a total of 10,500 DPO pairs from 3,500 questions:

prompt   = "Question: {question}\nChoices: (A) {a} (B) {b} (C) {c} (D) {d}\nAnswer:"
chosen   = " {correct answer text}"
rejected = " {distractor text}"

Training procedure

Quantization (QLoRA)

Setting	Value
Quantization	bitsandbytes 4-bit, NF4
Double quantization	off
Compute dtype	float16 (as recorded in the released config.json)
Attention	FlashAttention-2

LoRA

Setting	Value
r	8
lora_alpha	16
target_modules	q_proj, v_proj
lora_dropout	0.05
bias	none
task_type	CAUSAL_LM

The model was prepared with peft.prepare_model_for_kbit_training before wrapping.

DPO (TRL DPOTrainer)

Hyperparameter	Value
beta	0.1
epochs	2
per_device_train_batch_size	32
gradient_accumulation_steps	1
learning_rate	2e-4
lr_scheduler	cosine
warmup_ratio	0.03
weight_decay	0.001
max_prompt_length	256
max_completion_length	128

Tokenizer setup: add_eos_token=True, padding_side="left" during training, and pad_token = eos_token (</s>, id 2) since the Mistral tokenizer has no dedicated pad token.

Training ran on a single CUDA GPU (CUDA_VISIBLE_DEVICES=0) and took ~1 h 28 min. After training, the LoRA adapter was merged into the 4-bit base (merge_and_unload()) and the merged quantized model was pushed. Note that merging LoRA weights into a 4-bit quantized model introduces small rounding differences relative to merging in full precision.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nevertmr/mistral-7b-qlora-dpo"

# Weights are stored 4-bit (bitsandbytes); requires a CUDA GPU and `pip install bitsandbytes`
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

prompt = (
    "Question: Which property of a mineral can be determined just by looking at it?\n"
    "Choices: (A) luster (B) mass (C) weight (D) hardness\n"
    "Answer:"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=8, use_cache=True)
print(tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

Tips:

• The released config.json has use_cache: false (a training-time leftover) — pass use_cache=True to generate() for fast decoding.
• The model is optimized for likelihood-based multiple-choice scoring in the prompt format shown above. It is not instruction-tuned or chat-aligned; treat it as a base-style LM specialized for MCQA.
• For evaluation-style usage, score each candidate answer's log-likelihood after "Answer:" and pick the argmax, rather than free-form generation.

Evaluation setup

lm_eval --model hf \
    --model_args pretrained=nevertmr/mistral-7b-qlora-dpo,dtype=float16 \
    --tasks arc_challenge \
    --device cuda:0 \
    --batch_size 8 \
    --num_fewshot 25

Framework versions

Pinned environment used for training:

• torch (cu126 build), transformers==4.50.0, accelerate==1.5.2, bitsandbytes==0.45.3, datasets==3.4.1, flash_attn==2.7.4.post1, safetensors==0.5.3, tokenizers==0.21.1
• peft @ git+https://github.com/huggingface/peft.git@e79fdd78f63ca274174fa973d79779107448e660
• trl @ git+https://github.com/huggingface/trl.git@a0a53171cce596642c5fbb9e7b88b8f26a44ecad
• lm-evaluation-harness @ git+https://github.com/EleutherAI/lm-evaluation-harness@fd93c6c4fcb008b348fe21049a7b8b46d2bdc88e

Limitations

• Specialized for 4-choice science/commonsense QA likelihood ranking; general instruction following, chat, and open-ended generation are out of scope.
• Inherits all limitations and biases of Mistral-7B-v0.1 and of the training datasets; no safety alignment was performed.
• The checkpoint is only usable with bitsandbytes on CUDA hardware (weights are stored 4-bit); there is no full-precision variant of the merged model.