ehab215/DR-AI-V1

Dr. AI v1 — Stage-1 CPT LoRA adapter (Arabic / English medical)

LoRA adapter produced by Stage 1 (Continued Pre-Training / CPT) of the Dr. AI project. It adapts google/medgemma-4b-it (Gemma-3 4B) to Egyptian Arabic + Arabic + English medical text, giving the base model much stronger in-domain and in-dialect fluency before instruction tuning.

This is stage 1 only — a domain/language-adapted base. For the ready-to-chat assistant use the merged standalone model ehab215/DR-AI-V2.

What it is

• Type: PEFT / LoRA adapter (not a full model).
• Objective: causal-LM continued pre-training (next-token) on mixed Arabic/English medical corpora.
• Role in the pipeline: Stage 1. Stage 2 (SFT) is trained on top of this and both adapters are merged into the base to produce DR-AI-V2.
• Note on base: the adapter was trained targeting the MedGemma-4B family; adapter_config.json records google/medgemma-1.5-4b-it, and it is loaded/merged on google/medgemma-4b-it in Stage 2.

Training data (~1.1M documents, 80/20 mix)

Source	Role	Share
UBC-NLP/nilechat-lhv-egy	Egyptian-Arabic language backbone	~80%
GBaker/MedQA-USMLE-4-options	Medical replay (EN)	part of 20%
qiaojin/PubMedQA (pqa_artificial)	Medical replay (EN)	part of 20%

Cleaned (drop empty, truncate long docs to 4096 chars), shuffled so each batch mixes language/domain.

Training configuration

Item	Value
Method	LoRA, r=64, α=128, dropout=0.05
Target modules	q,k,v,o,gate,up,down proj
Max seq length	1024
Effective batch	32 (4 × 8 grad-accum)
LR / schedule	2e-4, cosine, 3% warmup
Epochs	1
Precision / HW	bf16, A100 80GB

Results (held-out validation, 2000 samples)

Metric	Base MedGemma	+ Stage-1 adapter
Perplexity	13.01	3.29

→ 3.95× perplexity reduction (Δ NLL ≈ 1.37) — a strong, healthy CPT signal.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "google/medgemma-4b-it"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "ehab215/DR-AI-V1")   # attach Stage-1 adapter

Limitations

Not a medical device. This is a base/pre-trained adapter, not instruction-tuned — it will not behave like a chat assistant on its own. Use DR-AI-V2 for that. Outputs may be wrong or unsafe; always have a qualified clinician review. Arabic skews Egyptian; trained at a 1024-token context.

License

Gemma Terms of Use. Inherits base-model and dataset licenses.