metadata

license: apache-2.0
language:
  - en
pipeline_tag: text-generation

OLMo 7B-Instruct-GGUF

For more details on OLMO-7B-Instruct, refer to Allen AI's OLMo-7B-Instruct model card.

OLMo is a series of Open Language Models designed to enable the science of language models. The OLMo base models are trained on the Dolma dataset. The Instruct version is trained on the cleaned version of the UltraFeedback dataset.

OLMo 7B Instruct is trained for better question answering. They show the performance gain that OLMo base models can achieve with existing fine-tuning techniques.

This version of the model is derived from ssec-uw/OLMo-7B-Instruct-hf as GGUF format, a binary format that is optimized for quick loading and saving of models, making it highly efficient for inference purposes.

In addition to the model being in GGUF format, the model has been quantized, to reduce the computational and memory costs of running inference. We are currently working on adding all of the Quantization Types.

These files are designed for use with GGML and executors based on GGML such as llama.cpp.

Get Started

To get started using one of the GGUF file, you can simply use llama-cpp-python, a Python binding for llama.cpp.

Install llama-cpp-python with pip.
```
pip install llama-cpp-python
```
Download one of the GGUF file. In this example, we will download the OLMo-7B-Instruct-Q4_K_M.gguf, when the link is clicked.

Open up a python interpreter and run the following commands. For example, we can ask it: What is a solar system?

You will need to modify the model_path argument to where the GGUF model has been saved in your system

from llama_cpp import Llama
llm = Llama(
      model_path="path/to/OLMo-7B-Instruct-Q4_K_M.gguf"
)
result_dict = llm(prompt="What is solar system?", echo=True, max_tokens=500)
print(result_dict['choices'][0]['text'])

That's it, you should see the result fairly quickly! Have fun! 🤖

Contact

For errors in this model card, contact Don or Anant, {landungs, anmittal} at uw dot edu.