Quick tip: How to Build Local LLM Apps with Ollama and SingleStore
Tags: singlestoredb, ollama, langchain, vectordatabase
Published: 2024-05-07 15:54:06

## Abstract
In an era of heightened data privacy concerns, the development of local Large Language Model (LLM) applications provides an alternative to cloud-based solutions. Ollama offers one solution, enabling LLMs to be downloaded and used locally. In this short article, we'll explore how to use Ollama with LangChain and SingleStore using a Jupyter Notebook.

The notebook file used in this article is available on [GitHub](https://github.com/VeryFatBoy/ollama-examples/blob/main/ollama_langchain_demo.ipynb).

## Introduction
We'll use a Virtual Machine running Ubuntu 22.04.2 as our test environment. An alternative would be to use `venv`.

## Create a SingleStoreDB Cloud account
A [previous article](https://dev.to/singlestore/quick-tip-using-dbt-with-singlestoredb-161g) showed the steps required to create a free SingleStore Cloud account. We'll use **Ollama Demo Group** as our Workspace Group Name and **ollama-demo** as our Workspace Name. We'll make a note of our **password** and **host** name. For this article, we'll temporarily allow access from anywhere by configuring the firewall under **Ollama Demo Group > Firewall.** For production environments, firewall rules should be added to provide increased security.

## Create a Database
In our SingleStore Cloud account, let's use the **SQL Editor** to create a new database. Call this `ollama_demo`, as follows:

```sql
CREATE DATABASE IF NOT EXISTS ollama_demo;
```

## Install Jupyter
From the command line, we'll install the classic Jupyter Notebook, as follows:

```shell
pip install notebook
```

## Install Ollama
We'll install Ollama, as follows:

```shell
curl -fsSL https://ollama.com/install.sh | sh
```

## Environment Variable
Using the password and host information we saved earlier, we'll create an environment variable to point to our SingleStore instance, as follows:

```shell
export SINGLESTOREDB_URL="admin:<password>@<host>:3306/ollama_demo"
```

Replace `<password>` and `<host>` with the values for your environment.

## Launch Jupyter
We are now ready to work with Ollama and we'll launch Jupyter:

```shell
jupyter notebook
```

## Fill out the notebook
First, some packages:

```shell
!pip install langchain ollama --quiet --no-warn-script-location
```

Next, we'll import some libraries:

```python
import ollama
from langchain_community.vectorstores import SingleStoreDB
from langchain_community.vectorstores.utils import DistanceStrategy
from langchain_core.documents import Document
from langchain_community.embeddings import OllamaEmbeddings
```

We'll create embeddings using `all-minilm` (45 MB at the time of writing):

```python
ollama.pull("all-minilm")
```

Example output:

```
{'status': 'success'}
```

For our LLM we'll use `llama2` (3.8 GB at the time of writing):

```python
ollama.pull("llama2")
```

Example output:

```
{'status': 'success'}
```

Next, we'll use the example text from the [Ollama website](https://ollama.com/blog/embedding-models):

```python
documents = [
    "Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels",
    "Llamas were first domesticated and used as pack animals 4,000 to 5,000 years ago in the Peruvian highlands",
    "Llamas can grow as much as 6 feet tall though the average llama between 5 feet 6 inches and 5 feet 9 inches tall",
    "Llamas weigh between 280 and 450 pounds and can carry 25 to 30 percent of their body weight",
    "Llamas are vegetarians and have very efficient digestive systems",
    "Llamas live to be about 20 years old, though some only live for 15 years and others live to be 30 years old"
]

embeddings = OllamaEmbeddings(
    model = "all-minilm",
)

dimensions = len(embeddings.embed_query(documents[0]))

docs = [Document(text) for text in documents]
```

We'll specify `all-minilm` for the embeddings, determine the number of dimensions returned for the first document, and convert the documents to the format required by SingleStore.

Next, we'll use LangChain:

```python
docsearch = SingleStoreDB.from_documents(
    docs,
    embeddings,
    table_name = "langchain_docs",
    distance_strategy = DistanceStrategy.EUCLIDEAN_DISTANCE,
    use_vector_index = True,
    vector_size = dimensions
)
```

In addition to the documents and embeddings, we'll provide the name of the table we want to use for storage, the distance strategy, that we want to use a vector index and the vector size using the dimensions we previously determined. These and other options are explained in further detail in the [LangChain documentation](https://python.langchain.com/docs/integrations/vectorstores/singlestoredb/).

Using the **SQL Editor** in SingleStore Cloud, let's check the structure of the table created by LangChain:

```sql
USE ollama_demo;

DESCRIBE langchain_docs;
```

Example output:

```
+----------+------------------+------+------+---------+----------------+
| Field    | Type             | Null | Key  | Default | Extra          |
+----------+------------------+------+------+---------+----------------+
| id       | bigint(20)       | NO   | PRI  | NULL    | auto_increment |
| content  | longtext         | YES  |      | NULL    |                |
| vector   | vector(384, F32) | NO   | MUL  | NULL    |                |
| metadata | JSON             | YES  |      | NULL    |                |
+----------+------------------+------+------+---------+----------------+
```

We can see that a `vector` column with 384 dimensions was created for storing the embeddings.

Let's also quickly check the stored data:

```sql
USE ollama_demo;

SELECT SUBSTRING(content, 1, 30) AS content, SUBSTRING(vector, 1, 30) AS vector FROM langchain_docs;
```

Example output:

```
+--------------------------------+--------------------------------+
| content                        | vector                         |
+--------------------------------+--------------------------------+
| Llamas weigh between 280 and 4 | [0.235754818,0.242168128,-0.26 |
| Llamas were first domesticated | [0.153105229,0.219774529,-0.20 |
| Llamas are vegetarians and hav | [0.285528302,0.10461951,-0.313 |
| Llamas are members of the came | [-0.0174482632,0.173883006,-0. |
| Llamas can grow as much as 6 f | [-0.0232818555,0.122274697,-0. |
| Llamas live to be about 20 yea | [0.0260244086,0.212311044,0.03 |
+--------------------------------+--------------------------------+
```

Finally, let's check the `vector` index:

```sql
USE ollama_demo;

SHOW INDEX FROM langchain_docs;
```

Example output:

```
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| Table          | Non_unique | Key_name   | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type       | Comment | Index_comment | Index_options                         |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
| langchain_docs |          0 | PRIMARY    |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | COLUMNSTORE HASH |         |               |                                       |
| langchain_docs |          1 | vector     |            1 | vector      | NULL      |        NULL |     NULL |   NULL |      | VECTOR           |         |               | {"metric_type": "EUCLIDEAN_DISTANCE"} |
| langchain_docs |          1 | __SHARDKEY |            1 | id          | NULL      |        NULL |     NULL |   NULL |      | METADATA_ONLY    |         |               |                                       |
+----------------+------------+------------+--------------+-------------+-----------+-------------+----------+--------+------+------------------+---------+---------------+---------------------------------------+
```

We'll now ask a question, as follows:

```python
prompt = "What animals are llamas related to?"
docs = docsearch.similarity_search(prompt)
data = docs[0].page_content
print(data)
```

Example output:

```
Llamas are members of the camelid family meaning they're pretty closely related to vicuñas and camels
```

Next, we'll use the LLM, as follows:

```python
output = ollama.generate(
    model = "llama2",
    prompt = f"Using this data: {data}. Respond to this prompt: {prompt}"
)

print(output["response"])
```

Example output:

```
Llamas are members of the camelid family, which means they are closely related to other animals such as:

1. Vicuñas: Vicuñas are small, wild relatives of llamas and alpacas. They are native to South America and are known for their soft, woolly coats.
2. Camels: Camels are also members of the camelid family and are known for their distinctive humps on their backs. There are two species of camel: the dromedary and the Bactrian.
3. Alpacas: Alpacas are domesticated animals that are closely related to llamas and vicuñas. They are native to South America and are known for their soft, luxurious fur.

So, in summary, llamas are related to vicuñas, camels, and alpacas.
```

## Summary
In this short article, we've seen that we can connect to SingleStore, store the documents and embeddings, ask questions about the data in the database, and use the power of LLMs locally through Ollama.