|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
Paper: https://arxiv.org/pdf/2310.06694.pdf |
|
Code: https://github.com/princeton-nlp/LLM-Shearing |
|
|
|
License: Must comply with license of Pythia since it's a model derived from Pythia. |
|
|
|
Sheared-Pythia-160m is a model pruned and further pre-trained from [EleutherAI/pythia-410m](https://huggingface.co/EleutherAI/pythia-410m). We dynamically load data from different domains in the Pile dataset to prune and contune pre-train the model. We use 0.4B tokens for pruning and 50B tokens for continued pre-training the pruned model. This model can be loaded with HuggingFace via |
|
|
|
``` |
|
model = GPTNeoXForCausalLM.from_pretrained("princeton-nlp/Sheared-Pythia-140m") |
|
``` |
|
|
|
The model's overall performance is better than EleutherAI/pythia-160m. |
|
|
|
## Bibtex |
|
``` |
|
@article{xia2023sheared, |
|
title={Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning}, |
|
author={Xia, Mengzhou and Gao, Tianyu, and Zeng, Zhiyuan and Chen, Danqi}, |
|
year={2023} |
|
} |
|
``` |
|
|
