Mistral-7B-Instruct-v0.1-LC-PI-.5-noSW

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.1 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.8995

Model description

This model is a fine-tuning of Mistral-7B-Instruct-v0.1. This FT was done with full attention (removing the 4k SWA). This FT was using a Position Interpolation factor of 0.5 (Linear RoPE scaling). Please note that the RoPE scaling factor should be determined by L/L' where L is the pre-training max context length and L' is the new max context length. In our case, we are just making experiments (and for us we would have had L/L' = 8096/7200 > 1 which did not require any PI scaling).

Intended uses & limitations

More information needed

Training and evaluation data

Data is a 9k sample from the RedPajama datset. The context is <=7200 with a decreasing exponential distribution of scale 1500.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 32
• total_train_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 20
• training_steps: 300

Training results

Training Loss	Epoch	Step	Validation Loss
2.1056	0.18	50	1.9680
2.1266	0.36	100	1.9213
1.978	0.55	150	1.9084
1.8576	0.73	200	1.9022
2.0311	0.91	250	1.8999
1.9197	1.09	300	1.8995

Framework versions

• Transformers 4.34.1
• Pytorch 2.0.0+cu117
• Datasets 2.14.6
• Tokenizers 0.14.1