How to Build Flet Chat App with Barcode and Gemini APIs
Published: 2024-01-08 07:05:45

Gemini is Google's latest AI model, which can be used for free with a limit of [60 queries per minute](https://ai.google.dev/pricing), and is capable of recognizing text from images. Generally, 1D barcodes are accompanied by human-readable text, which can be used to verify the accuracy of barcode recognition results. In this article, we will use the Flet Python API to build a desktop chat app integrated with both barcode and Gemini APIs. The app will read barcodes from images using Dynamsoft Barcode Reader and perform OCR on text within images using Gemini's text recognition capabilities.


<video src="https://github.com/yushulx/flet-chat-app-gemini-barcode/assets/2202306/ef159de0-cdf9-48e2-91d9-3da3ce9f85a0" controls="controls" muted="muted" style="max-height:640px; min-height: 200px; max-width: 100%;"></video>

## Installation

```bash
pip install -U google-generativeai dbr flet 
```

## Prerequisites
- [Gemini API Key](https://makersuite.google.com/app/apikey)
    
     ![Gemini API key](https://github.com/yushulx/flet-chat-app-gemini-barcode/assets/2202306/a556a3dc-622f-4de0-b3d5-9067de44a5e5)

- [Dynamsoft Barcode Reader License Key](https://www.dynamsoft.com/customer/license/trialLicense?product=dbr)
    
    ![Dynamsoft Barcode Reader license key](https://github.com/yushulx/flet-chat-app-gemini-barcode/assets/2202306/59535e1a-3f8e-4711-adb0-2610476848fd)


## Flet Python API for Desktop Applications
Flet empowers developers to create desktop applications using Python. It offers a [crash course](https://flet.dev/docs/tutorials/python-realtime-chat) for constructing a real-time chat application, which serves as an excellent starting point.

Our application features a list view for displaying chat messages, a text input field, a button for uploading images, a button for sending messages, and a button to clear the chat history.

![Flet chat app UI](https://www.dynamsoft.com/codepool/img/2024/01/flet-chat-app-ui-design.png)

- Chat messages:
    
    ```python
    
    chat = ft.ListView(
            expand=True,
            spacing=10,
            auto_scroll=True,
        )
    ```

- Text input field:
    
    ```python
    new_message = ft.TextField(
        hint_text="Write a message...",
        autofocus=True,
        shift_enter=True,
        min_lines=1,
        max_lines=5,
        filled=True,
        expand=True,
        on_submit=send_message_click,
    )
    ```

- Button to load an image:
    
    ```python
    
    def pick_files_result(e: ft.FilePickerResultEvent):
        global image_path
        image_path = None
        if e.files != None:
            image_path = e.files[0].path
            # TODO

    def pick_file(e):
        pick_files_dialog.pick_files()

    pick_files_dialog = ft.FilePicker(on_result=pick_files_result)
    page.overlay.append(pick_files_dialog)

    ft.IconButton(
        icon=ft.icons.UPLOAD_FILE,
        tooltip="Pick an image",
        on_click=pick_file,
    )
    ```

- Button to send a message:
    
    ```python
    def on_message(message: Message):
        if message.message_type == "chat_message":
            m = ChatMessage(message)

            chat.controls.append(m)
            page.update()

    page.pubsub.subscribe(on_message)

    def send_message_click(e):
        global image_path
        if new_message.value != "":
            page.pubsub.send_all(
                Message("Me", new_message.value, message_type="chat_message"))

            question = new_message.value

            new_message.value = ""
            new_message.focus()
            page.update()

            page.pubsub.send_all(
                Message("Gemini", "Thinking...", message_type="chat_message"))

            # TODO

    ft.IconButton(
        icon=ft.icons.SEND_ROUNDED,
        tooltip="Send message",
        on_click=send_message_click,
    ),
    ```

    `PubSub` facilitates asynchronous communication across page sessions. The `subscribe` method enables the receipt of broadcast messages from other sessions, while the `send_all` method allows for sending messages to all active sessions. Whenever a new message is received, the list view is automatically updated to display this new message.

- Button to clear the chat history:
    
    ```python
    def clear_message(e):
        global image_path
        image_path = None
        chat.controls.clear()
        page.update()

    ft.IconButton(
        icon=ft.icons.CLEAR_ALL,
        tooltip="Clear all messages",
        on_click=clear_message,
    )
    ```

## Integrating the Dynamsoft Barcode Reader
The Dynamsoft Barcode Reader is an efficient library designed for barcode scanning. To enable barcode scanning in your app, you must integrate this library. Here's how you can do it:

1. Import the Dynamsoft Barcode Reader library and initialize a barcode reader instance using your license key.
    
    ```python
    from dbr import *
    license_key = "LICENSE-KEY"
    BarcodeReader.init_license(license_key)
    reader = BarcodeReader()
    ```

2. Decode the barcode from the uploaded image and send the result to the chat.
    
    ```python
    def pick_files_result(e: ft.FilePickerResultEvent):
        global image_path, barcode_text
        barcode_text = None
        image_path = None
        if e.files != None:
            image_path = e.files[0].path
            page.pubsub.send_all(
                Message("Me", image_path, message_type="chat_message", is_image=True))

            text_results = None
            try:
                text_results = reader.decode_file(image_path)
            except BarcodeReaderError as bre:
                print(bre)

            if text_results != None:
                barcode_text = text_results[0].barcode_text
                page.pubsub.send_all(
                    Message("DBR", barcode_text, message_type="chat_message"))
    ```


## Utilizing Google's Gemini AI for Text Recognition
Gemini can extract text from images. Once you've decoded a barcode, you can employ Gemini to verify the accuracy of the text decoded from the barcode. Here are the steps to use Gemini:
1. Set up the API key for Gemini.
    
    ```python
    import google.generativeai as genai
    import google.ai.generativelanguage as glm

    genai.configure(api_key='API-KEY')
    ```

2. Initialize the text and vision models. The vision model takes both text and images as input.

    ```python
    model_text = genai.GenerativeModel('gemini-pro')
    chat_text = model_text.start_chat(history=[])
    model_vision = genai.GenerativeModel('gemini-pro-vision')
    chat_vision = model_vision.start_chat(history=[])
    ```

3. Customize the command to effectively recognize text from the barcode image.
    
    ```python
    def send_message_click(e):
        global image_path
        if new_message.value != "":
            ...

            if question == ":verify":
                question = "recognize text around the barcode"
                response = model_vision.generate_content(
                    glm.Content(
                        parts=[
                            glm.Part(
                                text=question),
                            glm.Part(
                                inline_data=glm.Blob(
                                    mime_type='image/jpeg',
                                    data=pathlib.Path(
                                        image_path).read_bytes()
                                )
                            ),
                        ],
                    ))

                text = response.text
                page.pubsub.send_all(
                    Message("Gemini", text, message_type="chat_message"))

    ```    

## Verifying the Barcode Decoding Results with the Accompanying Text
Now, we can check whether the text read from the barcode exists in the text recognized from the image. Since the text extracted by Gemini might include spaces, it's essential to eliminate these spaces prior to comparison.
    
```python
if barcode_text == None:
    return

text = text.replace(" ", "")
if text.find(barcode_text) != -1:
    page.pubsub.send_all(
        Message("Gemini", barcode_text + " is correct ✓", message_type="chat_message"))
else:
    page.pubsub.send_all(
        Message("Gemini", barcode_text + " may not be correct", message_type="chat_message"))
```

Launch the desktop application and test it with some images that contain 1D barcodes:

```
flet run chatbot.py
```

![Flet chat app with barcode and gemini APIs](https://www.dynamsoft.com/codepool/img/2024/01/flet-chat-app-gemini-barcode-api.png)

## Source Code
[https://github.com/yushulx/flet-chat-app-gemini-barcode](https://github.com/yushulx/flet-chat-app-gemini-barcode)