| license: mit | |
| language: | |
| - en | |
| tags: | |
| - llm-rs | |
| - ggml | |
| pipeline_tag: text-generation | |
| datasets: | |
| - databricks/databricks-dolly-15k | |
| # GGML converted version of [Databricks](https://huggingface.co/databricks) Dolly-V2 models | |
| ## Description | |
| Dolly is trained on ~15k instruction/response fine tuning records databricks-dolly-15k generated by Databricks employees in capability domains from the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA and summarization. | |
| ## Converted Models | |
| $MODELS$ | |
| ## Usage | |
| ### Python via [llm-rs](https://github.com/LLukas22/llm-rs-python): | |
| #### Installation | |
| Via pip: `pip install llm-rs` | |
| #### Run inference | |
| ```python | |
| from llm_rs import AutoModel | |
| #Load the model, define any model you like from the list above as the `model_file` | |
| model = AutoModel.from_pretrained("rustformers/dolly-v2-ggml",model_file="dolly-v2-12b-q4_0-ggjt.bin") | |
| #Generate | |
| print(model.generate("The meaning of life is")) | |
| ``` | |
| ### Rust via [Rustformers/llm](https://github.com/rustformers/llm): | |
| #### Installation | |
| ``` | |
| git clone --recurse-submodules https://github.com/rustformers/llm.git | |
| cd llm | |
| cargo build --release | |
| ``` | |
| #### Run inference | |
| ``` | |
| cargo run --release -- gptneox infer -m path/to/model.bin -p "Tell me how cool the Rust programming language is:" | |
| ``` |