tiwarinitin94/therapy-mistral-7b-qlora

🧠 Mistral Mental Health Therapy v0.1

Model Overview

Mistral Mental Health Therapy v0.1 is a fine-tuned version of Mistral-7B-Instruct-v0.2, trained using QLoRA (4-bit quantization) on a mental health counseling dataset.
It is designed to produce empathetic, supportive, and structured conversational responses similar to a therapist’s tone.

• Base model: Mistral-7B-Instruct-v0.2
• Fine-tuning method: QLoRA (LoRA + 4-bit quantization)
• Framework: Hugging Face Transformers, PEFT, BitsAndBytes
• Dataset: fadodr/mental_health_therapy
• Language: English
• Model version: v0.1
• Release date: October 2025

---

Intended Use

This model is intended for research, education, and experimentation in the domain of mental health dialogue systems and empathy modeling.

✅ Suitable for:

• Research on conversational empathy.
• Fine-tuning or continued training.
• Generating supportive dialogue samples.

⚠️ Not suitable for clinical or real-world therapeutic use.
This model is not a substitute for mental health professionals.

---

Training Details

Dataset

Used dataset: fadodr/mental_health_therapy
Each sample follows:

Instruction: [Task or situation]
User: [User query or context]
Therapist: [Expected response]

Preprocessing

• Combined all fields into a conversational format:

  Instruction: <instruction>
  User: <input>
  Therapist: <output>
  

• Tokenized with Mistral tokenizer (max_length = 512).
• Truncated longer sequences.

Fine-Tuning Configuration

Parameter	Value
Method	QLoRA
Rank (r)	8
LoRA Alpha	16
Dropout	0.05
Batch Size	1
Gradient Accumulation	8
Learning Rate	2e-4
Max Steps	200
Precision	FP16
Save Steps	25

Hardware

• GPU: NVIDIA RTX 4070 (8–16 GB VRAM)
• Frameworks: PyTorch, PEFT, Transformers, BitsAndBytes

---

Example Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "your-username/mistral-mental-health-v0.1"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype=torch.float16)

prompt = "I've been feeling very anxious about my studies. What can I do to calm myself down?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=200, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

---

Evaluation (v0.1)

Observations

• Generates calm, empathetic, and structured responses.
• Occasionally produces repetitive or generic answers.
• Handles therapy-style tone effectively within trained context.

Areas for Improvement

• Broader dataset for diverse mental health topics.
• Validation with real-world dialogue structures.
• Add reinforcement or reward-based tuning for sensitivity and safety.

---

Limitations & Ethics

• Not a licensed therapy model.
• May produce inaccurate or emotionally insensitive outputs.
• Avoid using in real-world therapeutic contexts.

---

Citation

If you use this model, please cite it as:

Nitin Tiwari. (2025). Mistral Mental Health Therapy v0.1. Hugging Face.  
https://huggingface.co/your-username/mistral-mental-health-v0.1

---

Author: Nitin Tiwari
Contact: tiwarinitin94@gmail.com
Version: v0.1
Status: Experimental Research Model