Edit model card

Visualize in Weights & Biases

MistrAND-7B-v1

MistrAND 7B v1 is a fine-tuned version of mistralai/Mistral-7B-v0.1 on the OASST2 dataset converted to a custom Andalusian Spanish orthography.

This project is part of the Master's Degree Final Thesis titled Conservational AI for endangered languages: a preservation strategy for Tsakonian Greek upon the Andalusian Spanish case, aiming to preserve endangered languages by storing them in QLoRA adapters for unlimited use in the future.

Links

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 0.0002
  • train_batch_size: 2
  • eval_batch_size: 8
  • seed: 42
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: linear
  • lr_scheduler_warmup_steps: 1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss
0.9639 0.1796 1000 0.8628
0.753 0.3593 2000 0.7898
0.713 0.5389 3000 0.7348
0.6756 0.7185 4000 0.6888
0.6665 0.8981 5000 0.6530

Framework versions

  • PEFT 0.10.1.dev0
  • Transformers 4.41.0.dev0
  • Pytorch 2.3.0+cu121
  • Datasets 2.19.1
  • Tokenizers 0.19.1
Downloads last month
2
Unable to determine this model’s pipeline type. Check the docs .

Adapter for