ssuvetha/pharma-tinyllama-domain-lora

Pharma TinyLlama Domain LoRA

Model description

This repository contains a LoRA adapter trained on pharmaceutical domain text using QLoRA.

Base model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T

This is non-instruction fine-tuning (continued pretraining / domain adaptation), not chat fine-tuning.

Training objective

The model was trained as a causal language model on cleaned pharma-domain text extracted from a PDF corpus. Its objective is next-token prediction in the pharmaceutical domain.

Training pipeline

• PDF text extraction
• Text cleaning and normalization
• Paragraph dataset creation
• Hugging Face dataset conversion
• Tokenization
• Token packing into 512-token blocks
• QLoRA fine-tuning
• Adapter saving

Intended use

This adapter is intended for:

• pharmaceutical domain adaptation
• text continuation
• downstream instruction tuning
• educational and research experiments

Limitations

• Not instruction fine-tuned
• Not a chatbot
• Not medically validated
• Not for clinical decision-making

How to load

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

base_model_id = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
adapter_id = "ssuvetha/pharma-tinyllama-domain-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()