|
--- |
|
license: mit |
|
datasets: |
|
- databricks/databricks-dolly-15k |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
tags: |
|
- dolly |
|
- dolly-v2 |
|
- instruct |
|
- sharded |
|
--- |
|
|
|
# dolly-v2-7b: sharded checkpoint |
|
|
|
This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-7b` model. Refer to the [original model](https://huggingface.co/databricks/dolly-v2-7b) for all details. |
|
|
|
- this enables low-RAM loading, i.e. Colab :) |
|
|
|
## Basic Usage |
|
|
|
|
|
install `transformers`, `accelerate`, and `bitsandbytes`. |
|
|
|
```bash |
|
pip install -U -q transformers bitsandbytes accelerate |
|
``` |
|
|
|
Load the model in 8bit, then [run inference](https://huggingface.co/docs/transformers/generation_strategies#contrastive-search): |
|
|
|
```python |
|
from transformers import AutoTokenizer, AutoModelForCausalLM |
|
|
|
model_name = "ethzanalytics/dolly-v2-7b-sharded" |
|
tokenizer = AutoTokenizer.from_pretrained(model_name) |
|
|
|
model = AutoModelForCausalLM.from_pretrained(model_name, load_in_8bit=True) |
|
``` |
