nvikou/llama3-phd-abstract-qa-ru

LLaMA 3 LoRA — PhD Abstract QA (Russian)

Parameter-efficient QLoRA / LoRA adapters for document-grounded question answering over a PhD dissertation abstract (avtoreferat) treated as a scientific text.

This model is the Russian specialization branch used in a bilingual comparative study of:

• parametric adaptation (QLoRA / LoRA)
• non-parametric adaptation (RAG)

evaluated with BLEU, ROUGE, BERTScore, LLM-as-a-Judge, and latency.

🔗 Model page: nvikou/llama3-phd-abstract-qa-ru

Sister model (English): nvikou/llama3-phd-abstract-qa-en

---

Model Details

Field	Value
Developed by	nvikou (Nel Nelson)
Model type	LoRA adapters (PEFT)
Base model	unsloth/llama-3-8b-Instruct-bnb-4bit (LLaMA 3 8B Instruct, 4-bit)
Language	Russian
License	LLaMA 3 community license (inherits from base model)
Framework	PEFT / Transformers / Unsloth
Intended task	Document-grounded QA on a PhD abstract

Adapter configuration

Hyperparameter	Value
Method	QLoRA + SFT
Rank r	16
lora_alpha	16
lora_dropout	0.05
Target modules	q_proj, v_proj
Trainable params	≈ 0.08% of base model
Epochs	10
Effective batch size	16 (2 × 8 grad accumulation)
Learning rate	2e-4
Max sequence length	8192
Max new tokens (eval)	256

---

Intended Use

Direct use

• Answering questions in Russian about the content of a PhD dissertation abstract.
• Experimental comparison with RAG systems for scientific document QA.
• Research / educational demos of PEFT specialization.

---

How to Use

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_id = "unsloth/llama-3-8b-Instruct-bnb-4bit"
adapter_id = "nvikou/llama3-phd-abstract-qa-ru"

tokenizer = AutoTokenizer.from_pretrained(base_id)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    device_map="auto",
    torch_dtype=torch.float16,
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

system_prompt = (
    "Вы профессиональный консультант по тексту диссертации / реферата. "
    "Отвечайте строго на основе содержания документа, ясно и по делу. "
    "Отвечайте на русском языке!"
)

question = "Какова практическая значимость результатов исследования?"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": question},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        do_sample=False,
    )

answer = tokenizer.decode(
    outputs[0][inputs["input_ids"].shape[-1]:],
    skip_special_tokens=True,
)
print(answer)

---

Training Data

• Supervised instruction-tuning pairs derived from a PhD abstract (Russian branch).
• Training set size: 219 question–answer pairs.
• Held-out evaluation set: 10 questions.

---

Citation

If you use this model, please cite the associated experimental study and this repository:

@misc{nvikou2026llama3phdabstractqaru,
  author       = {Nelson, Nel},
  title        = {LLaMA 3 LoRA for PhD Abstract QA (Russian)},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/nvikou/llama3-phd-abstract-qa-ru}}
}

---

Model Card Contact

• Hugging Face: @nvikou