Hugging Face's logo

migaraa
/
Gemma2B-LORAfied

PEFT
Safetensors
trl
sft
Generated from Trainer
ipex
GPU Max 1100
Model card Files Community
Gemma2B-LORAfied / README.md
Migara Amarasinghe
End of training
f7e512d verified
preview code
|
raw
history blame
1.65 kB
metadata
license: gemma
library_name: peft
tags:
  - trl
  - sft
  - generated_from_trainer
datasets:
  - generator
base_model: google/gemma-2b
model-index:
  - name: Gemma2B-LORAfied
    results: []

Gemma2B-LORAfied

This model is a fine-tuned version of google/gemma-2b on the generator dataset. It achieves the following results on the evaluation set:

  • Loss: 2.1460

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 1e-05
  • train_batch_size: 2
  • eval_batch_size: 8
  • seed: 42
  • gradient_accumulation_steps: 4
  • total_train_batch_size: 8
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_ratio: 0.05
  • training_steps: 593

Training results

Training Loss Epoch Step Validation Loss
2.8443 0.82 100 2.5332
2.4577 1.64 200 2.3103
2.275 2.46 300 2.2143
2.2331 3.28 400 2.1686
2.1737 4.1 500 2.1460

Framework versions

  • PEFT 0.10.0
  • Transformers 4.39.3
  • Pytorch 2.0.1a0+cxx11.abi
  • Datasets 2.18.0
  • Tokenizers 0.15.2