💊 Pharma-DrugInteraction-Qwen-0.5B
Pharmaceutical Drug-Interaction Learning Prototype
A small domain-specific language model prototype created by fine-tuning Qwen/Qwen2.5-0.5B-Instruct with LoRA (Low-Rank Adaptation) on a curated pharmaceutical drug-interaction dataset.
Version: v1.0 Base Model:
Qwen/Qwen2.5-0.5B-InstructFine-Tuning: LoRA Model Size: 0.5B parameters Purpose: Educational / Research Status: Learning Prototype
Usage
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch
BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct"
ADAPTER = "mkdiscovery/Pharma-DrugInteraction-Qwen-0.5B"
device = "cuda" if torch.cuda.is_available() else "cpu"
print("Loading model...")
tokenizer = AutoTokenizer.from_pretrained(BASE_MODEL)
model = AutoModelForCausalLM.from_pretrained(
BASE_MODEL,
torch_dtype=torch.float16 if device == "cuda" else torch.float32,
device_map="auto" if device == "cuda" else None,
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()
print("Model loaded!")
print("Ask questions below. Type 'exit' to quit.\n")
while True:
question = input("You: ")
if question.lower() in ["exit", "quit", "q"]:
print("Bye!")
break
messages = [
{
"role": "user",
"content": question
}
]
text = tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True,
)
inputs = tokenizer(
text,
return_tensors="pt"
).to(model.device)
with torch.no_grad():
outputs = model.generate(
**inputs,
max_new_tokens=256,
temperature=0.2,
do_sample=True,
)
response = tokenizer.decode(
outputs[0][inputs["input_ids"].shape[1]:],
skip_special_tokens=True
)
print(f"Model: {response}\n")
🚀 What is this?
This project explores how a small Large Language Model (LLM) can be adapted toward the pharmaceutical drug-interaction domain using parameter-efficient fine-tuning.
Instead of training an LLM from scratch, this project takes an existing instruction-tuned model:
Qwen/Qwen2.5-0.5B-Instruct
and trains a small set of additional LoRA parameters using pharmaceutical drug-interaction examples.
Architecture
Qwen 0.5B
│
│
▼
┌──────────────────┐
│ LoRA Adapter │
│ Fine-Tuning │
└────────┬─────────┘
│
▼
Pharma-DrugInteraction
Qwen-0.5B
│
▼
Drug Interaction
Questions
│
▼
Response
🎯 Project Goal
The goal of V1 was not to build a clinical-grade medical system.
The goal was to practically understand:
- How LLMs work
- How datasets are prepared for fine-tuning
- How pharmaceutical data can be normalized
- How different datasets can be combined
- How conversational training data is generated
- How LoRA fine-tuning works
- How a fine-tuned model behaves compared with the original model
- What limitations appear when using a very small model
This is primarily a hands-on AI/ML learning project.
🧠 Model
Base Model
Qwen/Qwen2.5-0.5B-Instruct
The base model provides the general language understanding and generation capabilities.
Fine-Tuning Method
LoRA — Low-Rank Adaptation
Instead of updating all parameters of the Qwen model, LoRA adds trainable low-rank matrices to selected layers.
Conceptually:
Original Qwen Model
│
├── Frozen Parameters
│
└── LoRA Parameters
│
▼
Pharmaceutical Domain
This makes the training process significantly lighter than full-model fine-tuning.
📊 Dataset
The training data was created from pharmaceutical/drug-interaction datasets.
The preparation pipeline included:
Raw Pharmaceutical Data
│
▼
Drug Name Normalization
│
▼
Drug Matching
│
▼
Interaction Records
│
▼
Conversational Training Examples
│
▼
train.jsonl
For this V1 prototype, the pharmaceutical drug selection was intentionally limited to a smaller subset because of available compute resources.
V1 scope
- Top 100 drugs
- 13,534 matched interaction records
- Multiple conversational question variations were generated for the interaction records
- Final training file:
train.jsonl
The smaller scope was intentional so the project could be trained and tested on available hardware.
🧪 Training Format
The training examples use a conversational format compatible with instruction-tuned models.
Example:
{
"messages": [
{
"role": "user",
"content": "Does Zolpidem Tartrate interact with Itraconazole?"
},
{
"role": "assistant",
"content": "Interaction: increase drug exposure.\n\nItraconazole increased the exposure of zolpidem..."
}
]
}
Multiple question formulations were generated for interaction records to expose the model to different ways a user might ask about the same drug interaction.
⚙️ Training Configuration
| Parameter | Value |
|---|---|
| Base model | Qwen/Qwen2.5-0.5B-Instruct |
| Fine-tuning | LoRA |
LoRA rank (r) |
16 |
| LoRA alpha | 32 |
| LoRA dropout | 0.05 |
| Bias | None |
| Task | Causal Language Modeling |
| Target modules | q_proj, k_proj, v_proj, o_proj |
| Epochs | 1 |
| Batch size | 1 |
| Gradient accumulation | 4 |
| Learning rate | 2e-4 |
| Precision | FP32 |
🔬 Before vs After Fine-Tuning
One of the main objectives of this project was to compare the base model with the fine-tuned model.
Before Fine-Tuning
The original Qwen model could generate general pharmaceutical-looking responses, but it could also:
- misunderstand drug names
- confuse medications
- generate unsupported explanations
- produce generic responses
After Fine-Tuning
The model became more aligned with the structure and terminology of the drug-interaction training data.
It learned patterns such as:
User:
Does Drug A interact with Drug B?
Model:
Interaction: <interaction category>
<supporting pharmaceutical text>
However, the V1 model can still produce incorrect or mismatched evidence.
This is an important limitation of using fine-tuning alone for precise pharmaceutical knowledge retrieval.
⚠️ Limitations
This model is a learning/research prototype.
It should not be considered a reliable medical information system.
Known limitations include:
- Small model size: 0.5B parameters
- Limited V1 drug coverage
- Limited training compute
- Possible hallucinations
- Possible incorrect drug-pair associations
- Fine-tuning does not guarantee exact factual retrieval
- Training data may contain inconsistencies
- No retrieval/database verification layer
- No clinical validation
- No medical professional verification of generated responses
The model should therefore not be used for diagnosis, prescribing, dosage decisions, or clinical decision-making.
Always verify drug-interaction information using authoritative pharmaceutical references and qualified healthcare professionals.
💻 Quick Start
1. Install dependencies
pip install torch transformers peft
Depending on the environment, compatible versions of the Hugging Face ecosystem may also be required.
2. Download the base model
The model is based on:
Qwen/Qwen2.5-0.5B-Instruct
The LoRA repository contains the adapter rather than a complete copy of the base model.
3. Load the LoRA adapter
The included inference.py loads:
Qwen/Qwen2.5-0.5B-Instruct
+
LoRA Adapter
↓
Pharma-DrugInteraction-Qwen-0.5B
Run:
python3 inference.py
🧑💻 Example
💊 You: Does Zolpidem Tartrate interact with Itraconazole?
🤖 Assistant:
Interaction: increase drug exposure.
Itraconazole increased the exposure of zolpidem...
Type:
exit
to close the application.
📁 Repository Structure
Pharma-DrugInteraction-Qwen-0.5B/
│
├── adapter_config.json
├── adapter_model.safetensors
│
├── tokenizer_config.json
├── tokenizer.json
├── special_tokens_map.json
├── added_tokens.json
├── merges.txt
├── vocab.json
│
├── inference.py
├── README.md
└── LICENSE
Important
adapter_model.safetensors contains the trained LoRA adapter weights.
The complete Qwen base model is not duplicated in this repository.
🔧 How the Model Works
At inference time:
User Question
│
▼
Qwen Tokenizer
│
▼
Qwen 0.5B Base Model
│
+
│
LoRA Adapter
│
▼
Generated Response
The LoRA adapter modifies the behavior of the base model toward patterns learned from the pharmaceutical training examples.
🧪 Why LoRA?
Full fine-tuning would require updating the entire model.
LoRA instead trains a relatively small number of additional parameters while keeping the original model largely frozen.
This makes it particularly useful for:
- learning experiments
- smaller compute environments
- domain adaptation
- rapid prototyping
- parameter-efficient fine-tuning
🛣️ Future Work
Possible future versions could explore:
V2 — Larger Dataset
Increase drug coverage beyond the V1 top-100 selection.
V3 — Retrieval-Augmented Generation
Introduce a retrieval layer so that the model can retrieve the exact drug-interaction record instead of relying entirely on information encoded during fine-tuning.
Question
↓
Drug Pair Retrieval
↓
Relevant Evidence
↓
LLM
↓
Answer
V4 — Evaluation
Build a dedicated evaluation dataset and measure:
- Exact-match accuracy
- Interaction-category accuracy
- Drug-pair coverage
- Hallucination rate
- Seen vs unseen pair performance
📌 Model Card Summary
| Property | Details |
|---|---|
| Project | Pharma Drug Interaction |
| Version | v1.0 |
| Base LLM | Qwen/Qwen2.5-0.5B-Instruct |
| Parameters | 0.5B base model |
| Fine-tuning | LoRA |
| Domain | Pharmaceutical drug interactions |
| V1 Drug Scope | Top 100 |
| Interaction Records | 13,534 |
| Training Format | Conversational JSONL |
| Primary Purpose | Educational / Research |
| Clinical Use | ❌ Not recommended |
🙏 Acknowledgements
This project was created as a hands-on exploration of LLMs, pharmaceutical datasets, data preprocessing, and parameter-efficient fine-tuning.
The project builds upon the capabilities of:
- Qwen
- Hugging Face Transformers
- Hugging Face Datasets
- PEFT / LoRA
⚖️ Disclaimer
Educational and research purposes only.
This model is not a medical device and has not been clinically validated.
Generated responses may be inaccurate, incomplete, or misleading. Do not use this model as a substitute for professional medical advice, prescribing information, official drug labels, or validated drug-interaction databases.
For real-world medical decisions, consult qualified healthcare professionals and authoritative pharmaceutical references.
👥 Contributors
This project was a collaborative effort combining pharmaceutical research and AI engineering.
Ayushi Nair
Concept, Research, Data Collection & Validation
- Led the project concept and research direction
- Collected and curated pharmaceutical datasets
- Validated data quality and interaction records
LinkedIn: https://www.linkedin.com/in/ayushi--nair/
Midhun Krishna
Engineering & Infrastructure
- Dataset preprocessing and preparation
- LoRA fine-tuning pipeline
- Model training and inference
- Hugging Face model packaging and deployment
- Repository development and documentation