File size: 1,059 Bytes
dfb9989 24d91d3 dfb9989 24d91d3 7e7aa2a 8b74e44 7e7aa2a 0e518c0 7e7aa2a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 |
---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
tags:
- 8bit
- sharded
- open_llama
inference: False
---
# open_llama_13b-sharded-8bit
This is [open_llama_13b](https://huggingface.co/openlm-research/open_llama_13b) sharded into 2 GB shards, and in 8-bit precision using `bitsandbytes==0.38.0`. Please refer to the original model card for details.
<a href="https://colab.research.google.com/gist/pszemraj/166ad661c6af1e024d4e2897621fc886/open_llama_13b-sharded-8bit-example.ipynb">
<img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>
## loading
```sh
pip install -U -q sentencepiece transformers accelerate bitsandbytes
```
load the model and tokenizer:
```python
import torch
from transformers import LlamaTokenizer, LlamaForCausalLM
model_name = "ethzanalytics/open_llama_13b-sharded-8bit"
tokenizer = LlamaTokenizer.from_pretrained(model_name, use_fast=False)
model = LlamaForCausalLM.from_pretrained(
model_name,
load_in_8bit=True,
device_map="auto",
)
``` |