likhith231/roberta-base-lora-text-classification

Model Card for Model ID

This model classifies whether the text/email is spam or ham.

Model Details

This model is a fine-tuned version of roberta-base using LoRA specifically for a binary classification task containing emails (subject + message) and a label whether it is spam or ham.

Model Description

The base model for this fine-tuning is roberta-base, which is a transformer-based model pre-trained on a large corpus of English data in a self-supervised fashion. RoBERTa is an optimized version of BERT, designed to perform better on natural language understanding tasks. We have applied LoRA to adapt the original RoBERTa model to the specific nuances of our binary classification problem. LoRA introduces low-rank matrices that are trained during the fine-tuning process, enabling the model to learn task-specific adaptations without altering the pre-trained weights directly.

• Developed by: Likhith231
• Model type: Text Classification
• Finetuned from model: Roberta Base

Training Data

The model was finetuned on truncated version of SetFit/enron_spam dataset. The SetFit/enron_spam consists of 33716 rows. The truncated version consists of 1000 train samples and 1000 test samples with columns, text and label.

Model Sources

• Repository: https://huggingface.co/FacebookAI/roberta-base

Training Details

• pretrained model= Roberta-base
• all params = 125,313,028
• trainable params= 665,858
• trainable% = 0.531355766137899

Parameters

• weight_decay = 0.01
• lr = 1e-3
• batch_size = 4
• num_epochs = 10

Results

Epoch	Training Loss	Validation Loss	Accuracy
1	No log	0.172788	0.957
2	0.194500	0.202991	0.956
3	0.194500	0.229950	0.958
4	0.038400	0.267390	0.954
5	0.038400	0.283116	0.963
6	0.007000	0.254960	0.961
7	0.007000	0.299375	0.961
8	0.007900	0.276321	0.966
9	0.007900	0.275304	0.967
10	0.002000	0.271234	0.967

Framework versions

• PEFT 0.8.2