#SemanticKernel: Local LLMs Unleashed on #RaspberryPi 5
Tags: englishpost, codesample, llama3
Published: 2024-05-09 15:26:55

---
title: #SemanticKernel: Local LLMs Unleashed on #RaspberryPi 5
published: true
date: 2024-05-01 12:00:00 UTC
tags: EnglishPost,CodeSample,EnglishPost,Llama3
canonical_url: 
---

Hi!

Welcome to the exciting world of local Large Language Models (LLMs) where we’re pushing the boundaries of what’s possible with AI.

Today let’s talk about a cool topic: run models locally, especially on devices like the Raspberry Pi 5. Let’s dive into the future of AI, right in our own backyards.

## Ollama and using Open Source LLMs

[OLLAMA stands out as a platform that simplifies the process of running open-source LLMs locally on your machine](https://cheatsheet.md/llm-leaderboard/ollama.en). It bundles model weights, configuration, and data into a single package, making it accessible for developers and AI enthusiasts alike. The key benefits of using Ollama include:

- **Simplicity** : Easy setup process without the need for deep machine learning knowledge.
- **Cost-Effectiveness** : Eliminates cloud costs, making it wallet-friendly.
- **Privacy** : Ensures data processing happens on your local machine, enhancing user privacy.
- [**Versatility** : Suitable for various applications beyond Python, including web development](https://cheatsheet.md/llm-leaderboard/ollama.en).

## Using Local LLMs like Llama3 or Phi-3

Local LLMs like Llama3 and Phi-3 represent a significant shift towards more efficient and compact AI models. [Llama3, with its Mixture-of-Experts (MoE) architecture, offers specialized neural networks for different tasks, providing high-quality outputs with a smaller parameter count](https://cheatsheet.md/llm-leaderboard/ollama.en).

[Phi-3, developed by Microsoft, uses advanced training techniques like quantization to maximize efficiency, making it ideal for deployment on a wide range of devices](https://cheatsheet.md/llm-leaderboard/ollama.en).

The use of local LLMs offers several advantages:

- **Reduced Latency** : Local models eliminate network latency associated with cloud-based solutions.
- **Enhanced Privacy** : Data remains on your local device, offering a secure environment for sensitive information.
- [**Customization** : Local models allow for greater flexibility to tweak and optimize the models as per your needs](https://cheatsheet.md/llm-leaderboard/ollama.en).

## How to Set Up a Local Ollama Inference Server on a Raspberry Pi 5

I already wrote a couple of times, my own version of the 1st time setup for a Raspberry Pi ([link](https://dev.to/elbruno/raspberrypi-1st-setup-no-monitor-wifi-auto-connect-ssh-rename-update-docker-rust-and-more-update-2023-jan-04-1gp7-temp-slug-9533435)). Once the device is ready, setting up Ollama on a Raspberry Pi 5 (or older) is a straightforward process. Here’s a quick guide to get you started:

[**Installation** : Use the official Ollama installation script to install it on your Raspberry Pi OS](https://cheatsheet.md/llm-leaderboard/ollama.en).

The main command is:

```

curl -fsSL https://ollama.com/install.sh | sh

```

[**Running Models** : After installation, you can run various LLMs like tinyllama, phi, and llava, depending on your RAM capacity](https://cheatsheet.md/llm-leaderboard/ollama.en).

In example to install and run llama 3, we can use the following command:

```

ollama run llama3

```

Once ollama is installed and a model is downloaded, the console should look similar to this one:

![log view of the installation of ollama and the run of llama 3 model](https://brunocapuano.files.wordpress.com/2024/04/image-2.png?w=1024)

[For a detailed step-by-step guide, including setting up Docker and accessing the Ollama WebUI, check out the resources available on GitHub](https://cheatsheet.md/llm-leaderboard/ollama.en).

Tip: to check the realtime journal of the ollama service, we can run this command:

```

journalctl -u ollama -f

```

**Important:** by defaul ollama server is available only for local calls. In order to enable access from other machines, you need to follow these steps:

– Edit the systemd service by calling this command.

```

sudo systemctl edit ollama.service

```

– This will open an editor.

– Add a line Environment under section [Service]:

```

[Service]
Environment="OLLAMA_HOST=0.0.0.0"

```

– Save and exit.

– Reload systemd and restart Ollama:

More information in the [ollama FAQ](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server).

## How to Use Semantic Kernel to Call a Chat Generation from a Remote Server

Let’s switch and write some code. This is a “Hello World” sample using Semantic Kernel and Azure OpenAI Services.

{% gist https://gist.github.com/elbruno/ff2d9d2c67d1b84db2c84f81a84f89af %}

You can learn more about this ai samples in: [https://aka.ms/dotnet-ai](https://aka.ms/dotnet-ai).

Now , to use a remote LLM, like Llama 3 in a Raspberry Pi, we can add a service to the Builder, that uses OpenAI API specification. In the next sample this change in the line 35:

{% gist 
https://gist.github.com/elbruno/13603c34827c382e0b1343cba380760e %}

This makes the trick! And with just a single change in a line.

And we also have the question about the performance, adding a StopWatch we can get a sense of the time elapsed for the call. For this simple call, the response is around 30-50 seconds.

![](https://brunocapuano.files.wordpress.com/2024/04/image-3.png?w=990)

Not bad at all for a small device!

## Conclusion

The advent of local LLMs like Ollama is revolutionizing the way we approach AI, offering unprecedented opportunities for innovation and privacy. Whether you’re a seasoned developer or just starting out, the potential of local AI is immense and waiting for you to explore.

* * *

_This blog post was generated using information from various online resources, including cheatsheet.md, anakin.ai, and techcommunity.microsoft.com, to provide a comprehensive guide on local LLMs and Ollama._

Happy coding!

Greetings

El Bruno

* * *