gemma-2-toxic-classification

This model is a fine-tuned version of unsloth/gemma-2-2b-it-bnb-4bit on the None dataset. It achieves the following results on the evaluation set:

• F1 Threshold 0.5: 0.8325
• Loss: 0.0854

Model description

This model is optimized for binary classification of toxic comments.

The fine-tuning was performed using QLoRA (4-bit quantization) via the Unsloth framework, which ensured high computational stability and training speed on a single Tesla T4 GPU. To capture long-form text nuances and prevent data truncation, the context length was extended to 512 tokens.

• Developed by: nnl26
• Language(s) (NLP): English
• Model type: Causal Language Model with a Sequence Classification head

Intended uses & limitations

Intended Uses

• Automated moderation of English text, comments, and forum posts.
• Identification of toxic behavior, cyberbullying, harassment, and aggressive language.

Limitations

• Language: This specific model is tailored and evaluated only for English text.
• Sarcasm & Nuance: While the model captures many subtle toxic patterns due to the extended context 512 tokens and enhanced LoRA capacity r=32, highly implicit sarcasm or niche cultural metaphors may still cause false classifications.

Training and evaluation data

The original dataset possessed a strong imbalance ~10% toxic / 90% clean comments. To build a reliable and robust classifier, a controlled oversampling strategy was applied to the training split.

Dataset Splits

• Training Set: 30,000 rows with a controlled 30% Toxic / 70% Clean ratio.
• Validation Set: 10,000 rows with the natural 10% / 90% imbalance to ensure realistic evaluation.
• Test Set: 10,000 rows with the natural 10% / 90% imbalance for the final unbiased benchmark.

Evaluation Results on a 10,000-row Independent Test Set

The model achieved a balanced performance at a stable, near-standard classification threshold:

• Optimal Threshold: 0.46
• Validation Loss: 0.0854 achieved at step 250 via Early Stopping
• Test Set F1-Score (Toxic Class): 0.8300
• Test Set Precision (Toxic Class): 0.84
• Test Set Recall (Toxic Class): 0.82
• Overall Accuracy: 0.97

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 8e-05
• train_batch_size: 16
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 32
• optimizer: Use OptimizerNames.PAGED_ADAMW_8BIT with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 0.1
• training_steps: 1000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	F1 Threshold 0.5	Validation Loss
0.3702	0.0533	50	0.6390	0.2893
0.2158	0.1067	100	0.6709	0.3042
0.2464	0.16	150	0.8068	0.1019
0.1254	0.2133	200	0.8273	0.0978
0.2041	0.2667	250	0.8325	0.0854
0.1734	0.32	300	0.6104	0.3610
0.1432	0.3733	350	0.8353	0.0971
0.1990	0.4267	400	0.7781	0.1316

Framework versions

• PEFT 0.21.2
• Transformers 5.17.0
• Pytorch 2.11.0+cu130
• Datasets 4.8.5
• Tokenizers 0.23.3