Practical AI - Local LLM and machine learning for plant care
Tags: llm, tutorial, python, raspberrypi
Published: 2024-03-29 15:23:31

### Create a custom AI assistant with local data security and no usage charges in less than 50 lines of code.

## Story

I've never bought a house plant. And yet, somehow my house and porch have slowly started to fill with all varieties of flora and fauna. Even with my aversion to seeking them out (thank you, all our plant-gifting friends!), I do need to keep the leafy beings alive and healthy. To help myself and anyone else who may be plant-sitting, I built an AI-powered greenery consultant that could identify the various species and provide helpful tips for taking care of the growing population within my home.


![various potted plants gathered in a room, sitting on the floor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/df6bao3belutue0v7uv7.png)

This smart machine uses speech-to-text to listen for a question, recognizes the plant type using a pre-trained machine learning model and computer vision, and passes that context into a local large language model (LLM) before sharing the response out loud using text-to-speech. This provides a natural and intuitive way of interacting with your new assistant while keeping the data and processing in your control.

### The plant identification model

This project might sound like a lot to put together from scratch, but [Viam's](https://viam.com) built-in and community modules make it possible to compose each of these capabilities together; adding or removing features as needed while being independent of specific hardware components, software packages or ML model types. For example, when I approached this project I knew I needed to use a machine learning model to recognize plant types, but was not sure which model.

- I knew that Viam makes it easy to [train and create your own custom ML models](https://docs.viam.com/ml/train-model/), but I did not feel comfortable enough training a model on plants I know nothing about.
- Next, I looked in the [Viam Registry](https://www.viam.com/product/registry) - as a new feature was just released that allows people to [share and use ML models](https://www.viam.com/post/machine-learning-models-now-available-through-the-modular-registry) directly on the Viam Registry. I [found a model](https://app.viam.com/ml-model/arielle/plantdetections) that my colleague Arielle had created at a plant store when working with Nvidia on a webinar to highlight [Viam’s integration with Nvidia’s Triton server](https://www.youtube.com/watch?v=jqniRZmHR5o), but since it was for demo purposes it was only trained on 5 plant types.
- I searched around the internet, and found a [model for houseplants](https://universe.roboflow.com/houseplants-dataset-for-bibitunggulid-bangkit-2023/houseplants-image-detection) that was trained on over 6 thousand images - could it work with Viam? Yes, because the Viam Registry has modules that allow you to point to and use [Roboflow](https://app.viam.com/module/viam-labs/roboflow-vision) and [HuggingFace](https://app.viam.com/module/viam-labs/YOLOv8) models.

To use any of these models, first choose the kind of model and add a couple of details.

![screenshot of Viam vision service configuration in Viam app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/68y5ssf3se7fnzej9kdb.png)

Then, whether it be a model you trained yourself, a model you find on the Viam Registry, or a HuggingFace, Roboflow or [DirectAI](https://directai.io/) model, you can “see” the world with the same single line of code:

```python
detections = await roboflow_plants.get_detections(image)
```

### Large Language Models (LLMs)

LLMs (Large Language Models), like ChatGPT, have gained a lot of well-deserved attention in the past year, and have been integrated into many services in most industries. However, only now are we seeing LLMs that are both small and useful enough to run locally on portable devices. This gets interesting for a few reasons:

- LLMs in the cloud can get costly, as they charge for API access.
- For some use cases, you want to maintain privacy and not have to worry about your interactions with an LLM going to the cloud and potentially even be used to add to the LLMs knowledge base (imagine a custom LLM used for corporate document analysis, for example).
- A local LLM can maintain its own knowledge base and does not rely on internet connectivity.

A Viam module exists that allows you to [deploy a local LLM](https://app.viam.com/module/viam-labs/local-llm) to your machine in just a few clicks. This module currently defaults to [TinyLlama 1.1B](https://github.com/jzhang38/TinyLlama), which provides reasonably accurate and performant results on small, portable single board computers like the [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) and [OrangePi 5B](http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5B.html).

For my use-case, the AI LLM can cover a wide range of topics, such as watering frequency, lighting conditions, temperature preferences, and fertilization schedules. The LLM's ability to understand context and generate personalized responses makes it an ideal tool for guiding users in their plant care journey. The model and prompt can be tuned to be more specific or broad, depending on the use case.

## Supplies

Viam allows you to be mostly hardware-agnostic, so you can try this all with a Macbook, but because I wanted to create a small portable device I used the following:

- A single-board computer ([Orange Pi 5B](http://www.orangepi.org/html/hardWare/computerAndMicrocontrollers/details/Orange-Pi-5B.html) or other octa-core CPU for best performance, a [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/) will also work)
- MicroSD card for the computer OS
- [USB webcam with a microphone](https://www.amazon.com/EMEET-1080P-Webcam-Microphone-Noise-Canceling/dp/B0CP6BTQKR)
- [USB speaker](https://www.amazon.com/HONKYOB-Speaker-Computer-Multimedia-Notebook/dp/B075M7FHM1/)
- Power supply for the board

## Build instructions

### 1. Install viam-server

The first step in this project build is to install viam-server on the single-board computer (SBC) with the [Viam installation instructions](https://docs.viam.com/get-started/installation/). Then in the [Viam App](https://app.viam.com/), add a new machine instance. I named mine “smart-assistant”, but you can choose another name if you want.


![screenshot of new machine creation text input with "smart assistant" as the value](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hqfryxqr3jdktaxiqd49.png)

### 2. Configure the camera

We will now set up the components and services, beginning with a camera. Be sure to have the USB devices (webcam and speaker) connected to board when turning it on.

Viam has support for a variety of [cameras](https://docs.viam.com/components/camera/); we chose to use a webcam, which generally can be [set up](https://docs.viam.com/components/camera/webcam/) with just a few clicks as Viam will auto-detect any attached cameras.  


![screenshot of camera component configuration in Viam app, selecting the video path of the webcam on the machine](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/pyp0xfebu5x1oa0edz1i.png)

### 3. Configure local LLM chat module

Now, we can start adding capabilities to our machine. First, we’ll add the local LLM, which we can find in the Service configuration menu, under ‘chat’.


![screenshot of adding the chat LLM service to machine in Viam app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rhywyfvlxfso4tygs71g.png)

Configuration attributes are documented in the [local LLM module README](https://github.com/viam-labs/local-llm), but we can start with the defaults.

After configuring the LLM module, we can test it by running a small Python script using the [Viam Python SDK](https://python.viam.dev/). If you’ve not installed the Viam Python SDK, install it and the chat API used by the local LLM module by running this command on your terminal:

```console
pip install viam-sdk chat_service_api@git+https://github.com/viam-labs/chat-service-api.git@main
```

Then, create this script called `assistant.py` on your machine, but substitute api_key, api_key_id, and your robot address with the correct values from the Viam app’s [Code sample tab](https://docs.viam.com/fleet/machines/#code-sample).

```python
import asyncio
from viam.robot.client import RobotClient
from chat_service_api import Chat

async def connect():
    opts = RobotClient.Options.with_api_key(
      api_key='',
      api_key_id=''
    )
    return await RobotClient.at_address('', opts)

async def main():
    robot = await connect()
    
    llm = Chat.from_robot(robot, name="llm")
    response = await llm.chat("what is a robot?")
    print(response)

    # Don't forget to close the machine when you're done!
    await robot.close()

if __name__ == '__main__':
    asyncio.run(main())
```

Run it with `python assistant.py` to produce a result like:

> A robot is an autonomous machine designed to perform specific tasks with limited human control. They are programmed to follow predetermined instructions or rules, making them highly efficient and reliable. Robots are used in various industries, from manufacturing and assembly to healthcare and transportation.


![colorful illustrated baby llama](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/whylhdtwe4l7562ye0x0.png)

You now have a LLM running and generating responses all on your local machine!

### 4. Add a speech interface

We want to interact with our local LLM in a natural way, so the next thing to add is a module that gives our machine capabilities to:

- Listen to us speak and convert that speech to text that we can be passed to the LLM
- Convert the LLM response to audio so we can hear the response spoken aloud

We’ll use a [speech module](https://app.viam.com/module/viam-labs/speech) from the Viam registry with these capabilities. To configure it, we’ll go back to the **Services** configuration tab in the Viam app and choose the "speechio" module.


![screenshot of adding speechio module as a service to the machine in the Viam app](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fea5i2nw1cyurqgr0k35.png)

There are [options](https://github.com/viam-labs/speech?tab=readme-ov-file#configure-your-speech-service) to customize the speech module’s behavior. Since we’ll want to be able to talk to our plant assistant, we need to allow the module to listen to the microphone, and also to have a trigger prefix (the speech that will trigger a response, you can set this to whatever you prefer). Add this configuration to the attributes field of the speech service, and save.

```json
{
  "listen": true,
  "listen_trigger_command": "hey there"
}
```

Once you’ve configured the module, add the speech API in the location you are running your Python scripts by running this command in the terminal:

```console
pip install speech_service_api@git+https://github.com/viam-labs/speech-service-api.git@main
```

Now update the `assistant.py` program to import the speech API, initialize the speech service, listen, and have the speech service speak the LLM response out loud.

We will have the script running in a loop. When the speech module hears a command, it will add it to a buffer. We’ll pull that command out of the buffer and send it to the LLM.

```python
import asyncio
from viam.robot.client import RobotClient
from chat_service_api import Chat
from speech_service_api import SpeechService

async def connect():
    opts = RobotClient.Options.with_api_key(
      api_key='',
      api_key_id=''
    )
    return await RobotClient.at_address('', opts)

async def main():
  robot = await connect()
    
  llm = Chat.from_robot(robot, name="llm")
  speech = SpeechService.from_robot(robot, name="speech")

  while True:
    commands = await speech.get_commands(1)
    if len(commands) > 0:
      command = commands[0]
      print(command)
             response = await llm.chat(command)
             print(response)
      await speech.say(response, True)

    # Don't forget to close the machine when you're done!
    await robot.close()

if __name__ == '__main__':
    asyncio.run(main())
```

You now have an AI assistant that you can converse with when you run this script and say something that is prefixed with the trigger you chose, like:

> _Hey there_, how often should I water my ficus plant?

> _Hey there_, does my spider plant need a lot of sun?

> _Hey there_, my cat peed on my verbena plant, what should I do?

### 5. Enhance your LLM assistant with computer vision

Now that we have our speech interface to our local LLM working, the last step is to connect it to the machine learning model we set up previously so that we can leverage computer vision to see and recognize our plants.

We’ll then have it replace mentions of “this plant”, “that plant” and “the plant” with the name of the plant our computer vision service most recently saw. We can then point our assistant at one of our plants and say:

> _Hey there_, how do I take care of this plant?

What will get updated to:

> How do I take care of this snake plant?

We can make this possible with a few changes in our code. We’ll first import the [‘re’ module](https://docs.python.org/3/library/re.html#regular-expression-examples), which allows us to use regular expressions to replace the name of the plant in our text. Then, we’ll import and instantiate our plant detector and use it to detect any plants it sees.

```python
import asyncio
import re
from viam.robot.client import RobotClient
from viam.services.vision import VisionClient
from chat_service_api import Chat
from speech_service_api import SpeechService

async def connect():
    opts = RobotClient.Options.with_api_key(
      api_key='',
      api_key_id=''
    )
    return await RobotClient.at_address('', opts)

async def main():
  robot = await connect()
    
  llm = Chat.from_robot(robot, name="llm")
  speech = SpeechService.from_robot(robot, name="speech")
  roboflow_plants = VisionClient.from_robot(robot, "roboflow-plants")
  plant = ""

  while True:
    detections = await roboflow_plants.get_detections_from_camera('cam')
    if len(detections) > 0:
      print(detections[0].class_name)
      plant = detections[0].class_name

    commands = await speech.get_commands(1)
      if len(commands) > 0:
        command = commands[0]
        if plant != "":
          command = re.sub(r'(this|that|the) plant',  "a " + plant, command)
        print(command)

        response = await llm.chat(command)
        print(response)
        await speech.say(response, True)

    # Don't forget to close the machine when you're done!
    await robot.close()

if __name__ == '__main__':
    asyncio.run(main())
```

You now have an AI and computer vision enabled plant care assistant, and all of this is possible with less than 50 lines of code!

{% embed https://youtu.be/i-NAXwLE87Y %}

_Thank you to Matt Vella for recording this working demo!_

## Next steps

Making this project was a great way to test the possibilities of running an AI LLM locally, and combining that with other capabilities like machine learning and computer vision. Other things I am hoping to do next:

- I want to try fine-tuning a local LLM like TinyLlama with a custom dataset so that it can have specialized knowledge. For example, I could train it with the [Viam documentation](https://docs.viam.com) so that it can help me build smart machines.
- Despite the LLM being local, the speech service I used defaults to using Google service for text-to-speech (TTS) and speech-to-text (STT). I want to integrate the new [`stt-vosk`](https://app.viam.com/module/viam-labs/stt-vosk) and [`tts-piper`](https://app.viam.com/module/viam-labs/tts-piper) modules to perform these tasks locally.
- I want to create a portable case for the components so I truly can use this as a portable device.

I am interested in how you might use local LLMs. Leave a comment below or [join us in Discord](https://discord.gg/viam) to let the community know!

_This post was originally shared on the [Viam blog](https://www.viam.com/post/practical-ai-local-llm-and-machine-learning-for-plant-care)._