lora_evo_ta_all_layers_16

This model is a fine-tuned version of togethercomputer/evo-1-8k-base on the None dataset. It achieves the following results on the evaluation set:

• Loss: 2.5463

Model description

Trained on single ID token 5K dataset filtered to 10k sequences (20% for test data = 2000)

lora_alpha = 128

lora_dropout = 0.1

lora_r = 128

epochs = 3

learning rate = 3e-4

warmup_steps=200

gradient_accumulation_steps = 1

train_batch = 2

eval_batch = 2

ONLY on attention layers and MLPs of last 31 layers <--------------------

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0003
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: constant
• lr_scheduler_warmup_steps: 500
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss
2.8598	0.4998	1999	2.6289
2.5927	0.9995	3998	2.5852
2.5467	1.4992	5997	2.5717
2.5487	1.999	7996	2.5554
2.4987	2.4988	9995	2.5546
2.4934	2.9985	11994	2.5463

Framework versions

• PEFT 0.11.1
• Transformers 4.41.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1