metadata
language:
- en
pipeline_tag: text-generation
tags:
- facebook
- meta
- pytorch
- llama
- llama-3
license: other
license_name: llama3
license_link: LICENSE
datasets:
- JeanKaddour/minipile
- raincandy-u/SlimOrca-Llama-3-Preference-DPO-Pairs
Llama-3-5B-Sheard
Pruned version of Llama-3-8b.
Tool used: PrunMe, Mergekit.
Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.
Training
After sliced by mergekit, the model is continue-pretrained on minipile for 1 epoch and ~100k samples. Then we trained it using ORPO on Llama-3-70b generated DPO pairs.
Disclaimer
This model is for testing purposes only, and when the system prompt is not empty, the output may repeat and not stop!