Mistral-7B-Instruct-v0.2-advisegpt-v0.3

This model is a fine-tuned version of mistralai/Mistral-7B-Instruct-v0.2 on the generator dataset. It achieves the following results on the evaluation set:

• Loss: 0.0862
• Bleu: {'bleu': 0.9549627224896852, 'precisions': [0.9768137794223292, 0.9601226611596732, 0.9485784293167555, 0.9390826620297074], 'brevity_penalty': 0.9988666836798081, 'length_ratio': 0.998867325397811, 'translation_length': 1126143, 'reference_length': 1127420}
• Rouge: {'rouge1': 0.9750644838957752, 'rouge2': 0.9567876902653232, 'rougeL': 0.9732849754530062, 'rougeLsum': 0.9746665365645586}
• Exact Match: {'exact_match': 0.0}

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 3
• eval_batch_size: 1
• seed: 42
• gradient_accumulation_steps: 10
• total_train_batch_size: 30
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Bleu	Exact Match	Validation Loss	Rouge
0.0708	0.9991	907	{'bleu': 0.9446206529600942, 'brevity_penalty': 0.9986864062670457, 'length_ratio': 0.9986872682762413, 'precisions': [0.9714789420395403, 0.9503978305171663, 0.9365333504686326, 0.9256591913728304], 'reference_length': 1127420, 'translation_length': 1125940}	{'exact_match': 0.0}	0.1052	{'rouge1': 0.9694819646914797, 'rouge2': 0.9464199252414252, 'rougeL': 0.9665470510722093, 'rougeLsum': 0.9687792447488508}
0.0611	1.9991	1814	0.0878	{'bleu': 0.9535151066703249, 'precisions': [0.9762399786139381, 0.9589412451791418, 0.9470130412549163, 0.9372328452904729], 'brevity_penalty': 0.9987103859226171, 'length_ratio': 0.998711216760391, 'translation_length': 1125967, 'reference_length': 1127420}	{'rouge1': 0.9743797099363829, 'rouge2': 0.9554568193403455, 'rougeL': 0.9724812167922234, 'rougeLsum': 0.9739500654981077}	{'exact_match': 0.0}
0.051	2.9982	2721	0.0862	{'bleu': 0.9549627224896852, 'precisions': [0.9768137794223292, 0.9601226611596732, 0.9485784293167555, 0.9390826620297074], 'brevity_penalty': 0.9988666836798081, 'length_ratio': 0.998867325397811, 'translation_length': 1126143, 'reference_length': 1127420}	{'rouge1': 0.9750644838957752, 'rouge2': 0.9567876902653232, 'rougeL': 0.9732849754530062, 'rougeLsum': 0.9746665365645586}	{'exact_match': 0.0}

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.3.0+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1