nadeem172/fedquant-mistral-7b-lora

FedQuant Mistral-7B LoRA Adapter

A LoRA adapter for Mistral-7B-Instruct-v0.3, trained with FedQuant, a federated fine-tuning method for devices with different memory sizes. Domain: Indian legal NLP.

What is FedQuant?

• Each client gets a LoRA rank based on its RAM tier: r = 8, 16 or 32.
• The server projects all adapters to the lowest rank (r* = 8), then takes a weighted average.
• The averaged adapter is zero-padded back to each client's own rank.

Training setup

Item	Value
Base model	Mistral-7B-Instruct-v0.3 (7.25B)
Federated rounds	12
Training clients	4 (client ranks 8, 8, 16, 32)
LoRA target layers	q, k, v, o projections
LoRA alpha / dropout	2r / 0.05
Optimizer	AdamW, lr 2e-4
Batch size	1 (grad. accumulation 4)
Max sequence length	256 tokens
Precision	BF16
Hardware	1x NVIDIA H100
Training time	about 12.3 hours
Training data	64,999 examples (10 legal tasks)

Results (Round 12)

• Final training loss, highest-rank client (r = 32): 0.091
• Average perplexity over 10 tasks: 3.323
• Task metrics use 100 validation samples per task.

How to use

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "mistralai/Mistral-7B-Instruct-v0.3"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "nadeem172/fedquant-mistral-7b-lora")

Limitations

• Training data was generated using rule-based and template methods from Indian statutory text.
• Covers Indian statutory law only.
• Clients were simulated on one GPU, not on physical phones.
• Outputs are for information only. They are not legal advice and need human review.