Create Phi-3 Chatbot with 20 Lines of Code (Runs Without Wifi) 🚀 🤖
Tags: python, ai, beginners, programming
Published: 2024-06-06 16:21:10

Made in collaboration with Andrew Dang
## The Power of Local Chatbots
Chatbots are fun and powerful tools. They excel at organization, summarization, and conversation. The versatility of chatbots makes platforms like ChatGPT popular. However, these platforms train their chatbots on user data, which comes at the cost of user privacy. Hence the appeal of local large language models (LLMs). Users can deploy local LLMs without the need for an internet connection and on consumer hardware. Stick around to learn how to deploy a local chatbot in just 20 lines of code!

***
## Let's get started 🔥 
This video will introduce you to the capabilities of the Chatbot and provide an overview of how it is built.
{% youtube gzzEVK8p3VM %}
***

## Framework 🖼️ 

#### LLMWare
For our new readers, LLMWARE is a comprehensive, open-source framework that provides a unified platform for application patterns based on LLMs, including Retrieval Augmented Generation (RAG).

#### Streamlit
Streamlit is an open-source Python library that allows developers to create interactive web applications quickly and easily. It's designed for machine learning and data science projects, enabling users to visualize data, run models, and share results in a user-friendly interface.

Please run `pip install llmware` and `pip install streamlit` in the command line to download these packages.

***

## Importing Libraries and Configurations 📚 

```python
import streamlit as st
from llmware.models import ModelCatalog
from llmware.gguf_configs import GGUFConfigs

GGUFConfigs().set_config("max_output_tokens", 500)
```
**Streamlit(`st`)**: Used for creating web applications. It provides functions to display content and handle user interactions.

**ModelCatalog**: A component of `llmware` that manages selecting the desired model, loading the model, and configuring the model using the parameters in the Model Loading section below.

**GGUFConfigs**: A component of `llmware` that handles global (applies to all currently loaded models) configurations for models, such as output token limits. You may find the full list of global configurations [here](https://github.com/llmware-ai/llmware/blob/main/llmware/gguf_configs.py) in the variable `_conf_libs` under the `GGUFConfigs` class.

***

## Model Loading 🪫🔋

```python
model = ModelCatalog().load_model(model_name, temperature=0.3, sample=True, max_output=450)
```
**Model Name**: The name of the model you want to load from ModelCatalog.

**Temperature**: This controls the randomness of the output. Valid values range between 0 and 1, where lower values make the model more deterministic, and higher values make the model more random and creative.

**Sample**: Determines if the output is generated deterministically or probabilistically. False generates deterministic output. True generates probabilistic output.

**Max Output**: Specifies the max length of the generated output (ex. if this is set to 100, there will be at most 100 words in the output).

***

## Session State Management 🧮
Manages chat history within the session state of Streamlit, ensuring that previous messages are preserved across reruns of the app.
#### Ensure there is a list called "messages" in session state

```python
if "messages" not in st.session_state:
   st.session_state.messages = []
```
#### Display Chat History
`role` is either "user" or "assistant" and will retrieve the corresponding message.

```python
for message in st.session_state.messages:
   with st.chat_message(message["role"]):
       st.markdown(message["content"])
```
#### Prompt the User and Risplay Responses
If the user enters text, display their text
```python
prompt = st.chat_input("Say something")
if prompt:

   with st.chat_message("user"):
      st.markdown(prompt)
```
#### Generate and Display Responses

```python
   with st.chat_message("assistant"):
      bot_response = st.write_stream(model.stream(prompt))
```
#### Update Session State
Updates the session state by appending the new user input and the assistant's response to the chat history.

```python
   st.session_state.messages.append({"role": "user", "content": prompt})
   st.session_state.messages.append({"role": "assistant", "content": bot_response})
```

***

## Main Block ⚙️
The typical Python idiom to ensure that the script runs only when it is executed as the main program, not when imported as a module.

Initializes the application with a predefined list of model names and starts the chat application with the first model in the list.

This script is a straightforward example of integrating AI models into a web application for interactive purposes, showcasing the use of Streamlit for UI and `llmware` for backend model handling.

```python
if __name__ == "__main__":
 
   chat_models = ["phi-3-gguf",
               "llama-2-7b-chat-gguf",
               "llama-3-instruct-bartowski-gguf",
               "openhermes-mistral-7b-gguf",
               "zephyr-7b-gguf",
               "tiny-llama-chat-gguf"]
   
   model_name = chat_models[0]
   simple_chat_ui_app(model_name)
```
**Model List**: This list contains the identifiers of various chat models available through the `llmware` library. These models are pre-trained and configured for generating conversational responses.

**Purpose**: At runtime, one of these models is selected to power the chat interface. This allows for flexibility in choosing different models based on their capabilities or performance characteristics.

***

## Fully Integrated Code 📄
**To run, go to command line -> 
streamlit run "path/to/gguf_streaming_chatbot.py"**
```python
import streamlit as st
from llmware.models import ModelCatalog
from llmware.gguf_configs import GGUFConfigs

GGUFConfigs().set_config("max_output_tokens", 500)


def simple_chat_ui_app (model_name):

    st.title(f"Simple Chat with {model_name}")

    model = ModelCatalog().load_model(model_name, temperature=0.3, sample=True, max_output=450)

    if "messages" not in st.session_state:
        st.session_state.messages = []

    for message in st.session_state.messages:
        with st.chat_message(message["role"]):
            st.markdown(message["content"])

    prompt = st.chat_input("Say something")
    if prompt:

        with st.chat_message("user"):
            st.markdown(prompt)

        with st.chat_message("assistant"):

            bot_response = st.write_stream(model.stream(prompt))

        st.session_state.messages.append({"role": "user", "content": prompt})
        st.session_state.messages.append({"role": "assistant", "content": bot_response})

    return 0


if __name__ == "__main__":

    chat_models = ["phi-3-gguf",
                   "llama-2-7b-chat-gguf",
                   "llama-3-instruct-bartowski-gguf",
                   "openhermes-mistral-7b-gguf",
                   "zephyr-7b-gguf",
                   "tiny-llama-chat-gguf"]

    model_name = chat_models[0]

    simple_chat_ui_app(model_name)
```
You may also find the fully integrated code on our Github repo [here](https://github.com/llmware-ai/llmware/blob/main/examples/UI/gguf_streaming_chatbot.py)
***

## Conclusion 🔒

As we have demonstrated, deploying a local chatbot can be achieved with just 20 lines of code. This simplicity, combined with the enhanced privacy and control, makes local LLMs an attractive option for users seeking powerful and secure chatbot solutions.

Thank you for exploring this topic with us. We hope you are now equipped with the knowledge to deploy your own local chatbot and harness its full potential.

Please check out our Github and leave a star! https://github.com/llmware-ai/llmware

Follow us on Discord here: https://discord.gg/MgRaZz2VAB