shvaq/hybrid-peft-la-research

Hybrid-PEFT-LA: Layer-Aware Hybrid Parameter-Efficient Fine-Tuning

Research Project by shvaq | ML Intern

Quick Links

• 🤗 Hub Repo: https://huggingface.co/shvaq/hybrid-peft-la-research
• 📓 Phase 1 Notebook (Baselines): https://huggingface.co/shvaq/hybrid-peft-la-research/blob/main/phase1_baselines.ipynb
• 📓 Phase 2 Notebook (Proposed): https://huggingface.co/shvaq/hybrid-peft-la-research/blob/main/phase2_proposed.ipynb
• 📄 IEEE Paper Draft: https://huggingface.co/shvaq/hybrid-peft-la-research/blob/main/ieee_paper.tex

What This Is

A complete research project exploring layer-aware parameter-efficient fine-tuning. Instead of applying LoRA uniformly across all 12 transformer layers, we learn which layers actually need adaptation by tracking gradient signals.

How to Run (Kaggle / Colab)

Step 1: Phase 1 Baselines

Download phase1_baselines.ipynb → import to Kaggle → run with GPU accelerator.

Step 2: Phase 2 Proposed Method

Download phase2_proposed.ipynb → import to Kaggle → run with GPU accelerator.

Step 3: Compare Results

Use phase3_report.py (in the repo) to generate comparison tables and statistical tests.

Topic

Hybrid-PEFT-LA: Layer-Aware Hybrid Parameter-Efficient Fine-Tuning

Research Gap

Existing hybrid PEFT methods (UniPELT, S4, MAM Adapter) apply adapter combinations uniformly across all layers. None learn which layers need which adapter type in a data-driven way. This matters because different transformer layers encode fundamentally different linguistic features.

Algorithm

1. Warmup (2 epochs): LoRA on all layers, track per-layer gradient norms
2. Selection: Keep top 75% layers, prune bottom 25% (where LoRA isn't helping)
3. Finetune (18 epochs): Train with sparse/pruned configuration

Datasets

GLUE benchmark (MNLI, SST-2, QQP, QNLI, CoLA) via nyu-mll/glue

Model

RoBERTa-base (125M params) — FacebookAI/roberta-base

Expected Compute

~25-30 GPU hours on A100 for full sweep (5 methods × 5 tasks × 3 seeds)

References

1. He et al. "Towards a Unified View of PEFT." ICLR 2022.
2. Hu et al. "LoRA: Low-Rank Adaptation." ICLR 2022.
3. Han et al. "PEFT Survey." arXiv:2403.14608, 2024.
4. Mao et al. "UniPELT." ACL 2022.
5. Li & Liang. "Prefix-Tuning." ACL 2021.

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = 'shvaq/hybrid-peft-la-research'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

For non-causal architectures, replace AutoModelForCausalLM with the appropriate AutoModel class.