--- license: mit datasets: - databricks/databricks-dolly-15k language: - en pipeline_tag: text-generation tags: - dolly - dolly-v2 - instruct - sharded inference: False --- # dolly-v2-12b: sharded checkpoint Open In Colab This is a sharded checkpoint (with ~4GB shards) of the `databricks/dolly-v2-12b` model. Refer to the [original model](https://huggingface.co/databricks/dolly-v2-12b) for all details. - this enables low-RAM loading, i.e. Colab :) ## Basic Usage install `transformers`, `accelerate`, and `bitsandbytes`. ```bash pip install -U -q transformers bitsandbytes accelerate ``` Load the model in 8bit, then [run inference](https://huggingface.co/docs/transformers/generation_strategies#contrastive-search): ```python from transformers import AutoTokenizer, AutoModelForCausalLM model_name = "ethzanalytics/dolly-v2-12b-sharded" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForCausalLM.from_pretrained( model_name, load_in_8bit=True, device_map="auto", ) ```