Get Started with Local Language Models
Tags: llm, llama, tutorial, ai
Published: 2023-09-17 14:12:17

---
title: 'Get Started with Local Language Models'
description: "My personal take on Local LLMs, why they are exciting, and how to use them with Langroid"
published: true
canonical_url: 
series: Building LLM Applications with Langroid
tags: llm, llama, tutorial, ai
comments: true
---

I recently dove into the world of Local LLMs, to see how I can hook them up with [Langroid](https://github.com/langroid/langroid), the agent-oriented open-source LLM framework. These are some notes from what I learned, plus a pointer to a tutorial on how to use them with Langroid.

## Why local models?
There are commercial, remotely served models that currently appear to beat all open/local
models. So why care about local models? Local models are exciting for a number of reasons:

- **cost**: other than compute/electricity, there is no cost to use them.
- **privacy**: no concerns about sending your data to a remote server.
- **latency**: no network latency due to remote API calls, so faster response times, provided you can get fast enough inference.
- **uncensored**: some local models are not censored to avoid sensitive topics.
- **fine-tunable**: you can fine-tune them on private/recent data, which current commercial models don't have access to.
- **sheer thrill**: having a model running on your machine with no internet connection,
  and being able to have an intelligent conversation with it -- there is something almost magical about it.

The main appeal with local models is that with sufficiently careful prompting,
they may behave sufficiently well to be useful for specific tasks/domains,
and bring all of the above benefits. Some ideas on how you might use local LLMs:

- In a mult-agent system, you could have some agents use local models for narrow 
  tasks with a lower bar for accuracy (and fix responses with multiple tries).
- You could run many instances of the same or different models and combine their responses.
- Local LLMs can act as a privacy layer, to identify and handle sensitive data before passing to remote LLMs.
- Some local LLMs have intriguing features, for example llama.cpp lets you 
  constrain its output using grammars.

## Running LLMs locally

There are several ways to use LLMs locally. See the [`r/LocalLLaMA`](https://www.reddit.com/r/LocalLLaMA/comments/11o6o3f/how_to_install_llama_8bit_and_4bit/) subreddit for
a wealth of information. There are open source libraries that offer front-ends
to run local models, for example [`oobabooga/text-generation-webui`](https://github.com/oobabooga/text-generation-webui)
(or "ooba-TGW" for short) but the focus in this tutorial is on spinning up a
server that mimics an OpenAI-like API, so that any Langroid code that works with
the OpenAI API (for say GPT3.5 or GPT4) will work with a local model,
with just a simple change: set `openai.api_base` to the URL where the local API
server is listening, typically `http://localhost:8000/v1`. It really is as simple as that!

There are two libraries I'd recommend for setting up local models with OpenAI-like APIs:

- [ooba-TGW](https://github.com/oobabooga/text-generation-webui) mentioned above, for a variety of models, including llama2 models.
- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) (LCP for short), specifically for llama2 models.

If you have other recommends, feel free to add them in the comments.

## Building Applications with Local LLMs

We open sourced Langroid to simplify building LLM-powered applications, whether with local or commercial LLMs. If you’re itching to play with local LLMs in simple python scripts, head over to our [tutorial](https://langroid.github.io/langroid/blog/2023/09/14/using-langroid-with-local-llms/) on using Langroid with local LLMs.