First steps with llama
Tags: 100daystooffload, llm, ai, python
Published: 2023-07-31 06:39:09

Time to try out some coding with generative AI and LLM (Large Language Model).

I had one requirement for this and that was to run it locally on my machine and not use a web API, like the one provided by [OpenAI](https://platform.openai.com/docs/api-reference).

To do this is I managed to stumble over this project [llama.cpp](https://github.com/ggerganov/llama.cpp) by [Georgi Gerganov](https://twitter.com/ggerganov), which also has a couple of bindings to other programming languages.

I went with Python, [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), since my goal is just to get a small project up and running locally.

Set up the project.

```sh
# Create a folder for the project
mkdir test-llama-cpp-python && cd $_

# Create a virtual environment
pyenv virtualenv 3.9.0 llm
pyenv activate llm

# Install the llama-cpp-python package
python -m pip install llama-cpp-python
python -m pip freeze > requirements.txt

# Create an empty main.py
touch main.py

# Open up the project in VS Code
code .
```

In the main file, add a simple skeleton prompt loop.

```python
import os

def get_reply(prompt):
    """Local inference with llama-cpp-python"""
    return ""

def clear():
    """Clears the terminal screen."""
    os.system('cls' if os.name == 'nt' else 'clear')

def main():
    """The prompt loop."""
    clear()

    while True:
        cli_prompt = input("You: ")

        if cli_prompt == "exit":
            break
        else:
            answer = get_reply(cli_prompt)

            print(f"""Llama: {answer}""")


if __name__ == '__main__':
    main()
```

From the [examples](https://github.com/abetlen/llama-cpp-python/tree/main/examples/high_level_api) on GitHub we can see that we need to import the class **Llama** into our main file and we will also need a model.

There is a popular AI community, [Hugging Face](https://huggingface.co/), where we can find models to use. There is one requirement about the model file, and that is that it has to be in the GGML file format. There is a [converter](https://github.com/ggerganov/llama.cpp#prepare-data--run) available in the llama.cpp GitHub project to do that.

However I searched for a model that already was in that format and took the first one I found, [TheBloke/Llama-2-7B-Chat-GGML](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML). I downloaded the following model at the end, [llama-2-7b-chat.ggmlv3.q4_1.bin](https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/blob/main/llama-2-7b-chat.ggmlv3.q4_1.bin).

Picking correct/best model is for another topic and is out of scope for this post.

When the model is downloaded to the project folder we can update our main.py file to start using the Llama class and the model.

```python
from llama_cpp import Llama

llama = Llama(model_path='llama-2-7b-chat.ggmlv3.q4_1.bin', verbose=False)

def get_reply(prompt):
    """Local inference with llama-cpp-python"""
    response = llama(
        f"""Q: {prompt} A:""", max_tokens=64, stop=["Q:", "\n"], echo=False
    )
            
    return response["choices"].pop()["text"].strip()
```

We first import Llama class and initialize an Llama object. The constructor needs the path to our model file which is given with _model_path_. I’m also setting the _verbose_ flag to false to suppress noise messages from the llama-cpp package.

The _get_reply_ method is doing all local inference with the llama-cpp-python package. The prompt to generate text from needs to be formatted in a specific way and therefore the **Q** and **A** is added.

Here is the final result of the code.

```python
import os

from llama_cpp import Llama

llama = Llama(model_path="llama-2-7b-chat.ggmlv3.q4_1.bin", verbose=False)


def get_reply(prompt):
    """Local inference with llama-cpp-python"""
    response = llama(
        f"""Q: {prompt} A:""", max_tokens=64, stop=["Q:", "\n"], echo=False
    )

    return response["choices"].pop()["text"].strip()


def clear():
    """Clears the terminal screen."""
    os.system("cls" if os.name == "nt" else "clear")


def main():
    """The prompt loop."""
    clear()

    while True:
        cli_prompt = input("You: ")

        if cli_prompt == "exit":
            break
        else:
            answer = get_reply(cli_prompt)

            print(f"""Llama: {answer}""")


if __name__ == "__main__":
    main()

```

Test run it by executing following in your cli.

```sh
python main.py
```

And ask a question, rember that **exit** will close the prompt.

```prompt
You: What are the names of the planets in the solar system?
Llama: The planets in our solar system, in order from closest to farthest from the Sun, are: Mercury Venus Earth Mars Jupiter Saturn Uranus Neptune
You: exit
```

Until next time!