---
license: mit
datasets:
- databricks/databricks-dolly-15k
language:
- en
pipeline_tag: text-generation
tags:
- dolly
- dolly-v2
- instruct
- sharded
widget:
  - text: Imagine Einstein was part of a comedy duo. What would be their stage name?
    example_title: Einstein's comedy duo
  - text: What do you think Einstein's favorite Swiss chocolate brand would be?
    example_title: Einstein's chocolate
  - text: If Einstein were to enter a yodeling competition in Switzerland, what
      would his yodel sound like?
    example_title: Einstein's yodel
  - text: If Einstein had to create a Swiss-themed superhero, what would their name
      and superpower be?
    example_title: Swiss superhero
  - text: What kind of wild party would Einstein throw at ETH Zurich?
    example_title: Einstein's party
  - text: If Einstein had a pet Swiss cow, what would he name it and why?
    example_title: Einstein's cow
  - text: You've discovered a secret Swiss cheese that grants the power of genius.
      How would you use it to become the next Einstein?
    example_title: Genius cheese
inference:
  parameters:
    max_length: 64
    min_length: 32
---

# dolly-v2-7b: sharded checkpoint 

<a href="https://colab.research.google.com/gist/pszemraj/6eb7ccce28ea6aa07b8ec86388ac010e/sharded-instruction-model-playground.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-7b` model. Refer to the [original model](https://huggingface.co/databricks/dolly-v2-7b) for all details.

- this enables low-RAM loading, i.e. Colab :)

## Basic Usage


install `transformers`, `accelerate`, and `bitsandbytes`.

```bash
pip install -U -q transformers bitsandbytes accelerate
```

Load the model in 8bit, then [run inference](https://huggingface.co/docs/transformers/generation_strategies#contrastive-search):

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "ethzanalytics/dolly-v2-7b-sharded"
tokenizer = AutoTokenizer.from_pretrained(model_name)

model = AutoModelForCausalLM.from_pretrained(
          model_name, load_in_8bit=True, device_map="auto",
        )
```