ksumanthreddy96/studymate-ctp-dapt

---

StudyMate CTP — Domain-Adaptive Pretraining (DAPT)

About

StudyMate CTP DAPT is a domain-adapted language model developed for the Construction Technology and Practices (CTP) course. It uses lecture materials from scanned course PDFs to adapt the Qwen2.5-0.5B language model to construction-related educational content.

This repository contains LoRA adapter weights, not a standalone complete language model.

Project Workflow

1. Document processing: Processed 11 scanned CTP lecture PDFs.
2. OCR extraction: Used PyMuPDF, Tesseract OCR, EasyOCR, and Python to extract text from lecture materials.
3. Domain-adaptive pretraining: Continued training the Qwen2.5-0.5B model on the extracted course text.
4. Adapter training: Used parameter-efficient fine-tuning with LoRA and 4-bit model loading.
5. Next stage: The resulting DAPT adapter was used as the starting point for supervised fine-tuning (SFT).

Training Details

Parameter	Value
Base model	Qwen/Qwen2.5-0.5B
Training approach	Domain-Adaptive Pretraining
Adapter method	LoRA
Model loading	4-bit quantization
Training framework	Unsloth, PEFT, Transformers, PyTorch
Epochs	3
Optimizer steps	24
Final training loss	Approximately 3.1623
Training data	OCR-extracted CTP lecture content

The reported training loss is a training metric; independent DAPT evaluation results are not available.

Intended Use

This adapter is intended for research and educational experimentation with construction-related course content. It serves as an intermediate model for further fine-tuning and is not intended to guarantee accurate answers to every construction question.

Loading the Adapter

The adapter can be loaded using Hugging Face Transformers and PEFT. Use a compatible base model and the adapter configuration provided in this repository.

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "Qwen/Qwen2.5-0.5B"
adapter_name = "ksumanthreddy96/studymate-ctp-dapt"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    device_map="auto"
)

model = PeftModel.from_pretrained(
    base_model,
    adapter_name
)

model.eval()

This is a basic loading example. For memory-constrained environments, use a compatible quantized loading configuration.

Limitations

• The training corpus was extracted from scanned PDFs and may contain OCR errors or incomplete text.
• Training loss alone does not establish factual accuracy or answer quality.
• The adapter requires a compatible base model.
• Further evaluation and fine-tuning may be needed for reliable question answering.

Related Model

The subsequent SFT adapter is available at:

StudyMate CTP SFT

Disclaimer

This is an experimental educational project developed for learning and research. Please verify generated information against the original course materials.