---
language:
- en
pipeline_tag: text-generation
tags:
- facebook
- meta
- pytorch
- llama
- llama-3
license: other
license_name: llama3
license_link: LICENSE
datasets:
- JeanKaddour/minipile
- raincandy-u/SlimOrca-Llama-3-Preference-DPO-Pairs
---



![image/png](https://cdn-uploads.huggingface.co/production/uploads/657eb5b256c9c67605a6e8b5/6X2l7zKi9ttuxfRGLucUY.png)

# Llama-3-5B-Sheard

Pruned version of Llama-3-8b.

Tool used: [PrunMe](https://github.com/arcee-ai/PruneMe), Mergekit.

**Meta Llama 3 is licensed under the Meta Llama 3 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.**

## Training

After sliced by mergekit, the model is continue-pretrained on minipile for 1 epoch and ~100k samples. Then we trained it using ORPO on Llama-3-70b generated DPO pairs.

## Disclaimer

This model is for testing purposes only, and when the system prompt is not empty, the output may repeat and not stop!

<h1><a href="https://discord.gg/Dw8SkfBm">Join our discord</a></h1>