metadata
license: apache-2.0
library_name: peft
tags:
- generated_from_trainer
base_model: mistralai/Mistral-7B-v0.1
model-index:
- name: mistrAND-7B-v1
results: []
MistrAND-7B-v1
MistrAND 7B v1 is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the OASST2 dataset converted to a custom Andalusian Spanish orthography.
This project is part of the Master's Degree Final Thesis titled Conservational AI for endangered languages: a preservation strategy for Tsakonian Greek upon the Andalusian Spanish case, aiming to preserve endangered languages by storing them in QLoRA adapters for unlimited use in the future.
Links
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0002
- train_batch_size: 2
- eval_batch_size: 8
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 1
- num_epochs: 1
Training results
| Training Loss | Epoch | Step | Validation Loss |
|---|---|---|---|
| 0.9639 | 0.1796 | 1000 | 0.8628 |
| 0.753 | 0.3593 | 2000 | 0.7898 |
| 0.713 | 0.5389 | 3000 | 0.7348 |
| 0.6756 | 0.7185 | 4000 | 0.6888 |
| 0.6665 | 0.8981 | 5000 | 0.6530 |
Framework versions
- PEFT 0.10.1.dev0
- Transformers 4.41.0.dev0
- Pytorch 2.3.0+cu121
- Datasets 2.19.1
- Tokenizers 0.19.1