Build a smart product data generator from image with GPT-4o and Langchain
Tags: openai, langchain, python, tutorials
Published: 2024-06-25 05:34:10

---
title: "Build a smart product data generator from image with GPT-4o and Langchain"
description: "How to create an AI tool to generate essential product's info based on an image with Langchain and OpenAI GPT-4o in Python."
published: true
cover_image: "https://res.cloudinary.com/mayashavin/image/upload/v1719260748/articles/llm/aaii_tool.png"
tags: ['OpenAI', 'Langchain', 'Python', 'tutorials']
series: Experimenting with Generative AI
canonical_url: "https://mayashavin.com/articles/product-generator-langchain-openai-gpt4opart1"
---

_When listing new products to an online store, owners or marketers often find it too time-consuming to fill in the essential information such as title, description, and tags for each product from scratch. Most of the information can be retrieved from the product image itself. With the right combination of LLM and AI tools, such as Langchain and OpenAI, we can automate the process of writing product's information using an input of image, which is our focus in today's post._

## Table of contents

- [Table of contents](#table-of-contents)
- [Brief introduction about Langchain and OpenAI](#brief-introduction-about-langchain-and-openai)
  - [Setting up Langchain and OpenAI](#setting-up-langchain-and-openai)
- [The flow of generating product data](#the-flow-of-generating-product-data)
- [Step 1: Load an product image into base64 format](#step-1-load-an-product-image-into-base64-format)
- [Step 2: Ask GPT to generate a product's metadata](#step-2-ask-gpt-to-generate-a-products-metadata)
  - [Setting up OpenAI API key](#setting-up-openai-api-key)
  - [Creating a model to process the image and prompt](#creating-a-model-to-process-the-image-and-prompt)
- [Step 3: Extract the result from GPT in a structured Product format](#step-3-extract-the-result-from-gpt-in-a-structured-product-format)
  - [Define the Product structure](#define-the-product-structure)
  - [Create a function to extract the product information](#create-a-function-to-extract-the-product-information)
- [Chaining all the steps together using Langchain](#chaining-all-the-steps-together-using-langchain)
- [Resources](#resources)
- [Summary](#summary)

## Brief introduction about Langchain and OpenAI

Langchain is a powerful tool that allows you to architect and run AI-powered functions with ease. It provides a simple interface to integrate with different LLMs (Large-Language-Models) APIs and services such as OpenAI, Hugging Face, etc. It also offers an extensible architecture that allows you to create and manage custom chains (pipelines), agents, and workflows tailored to your specific needs.

OpenAI is a leading AI research lab that has developed several powerful LLMs, including GPT-3, GPT-4 and Dall-E. These models can generate human-like text and media based on the input prompt, making them ideal for a wide range of applications, from chatbots to content/image generation.

### Setting up Langchain and OpenAI

In this post, we will use GPT-4o model from OpenAI for better image anayzing and text completion, along with the following Langchain Python packages:

- `langchain-openai` - A package that provides a simple interface to interact with OpenAI API.
- `langchain_core` - The core package of Langchain that provides the necessary tools to build your AI functions.

To install these packages, you use the following command:

```bash
python -m pip install langchain-openai langchain-core
```

Next, let's define the flow of how we generate product information based on a given image.

## The flow of generating product data

Our tool will perform the following steps upon receiving an image URL from the user:

1. Load the given product image into base64 data URI text format.
2. Ask GPT to analyze and generate the required product's metadata based on such data.
3. Extract the result from GPT in a structured Product format.

The below diagram demonstrates how the our work flow looks like:

<img src="https://res.cloudinary.com/mayashavin/image/upload/f_auto,q_auto/v1719259328/articles/llm/flow_generator" loading="lazy" alt="Diagram flow of generating product data" class="mx-auto">

With this flow in mind, let's walk through each step's implementation in detail.

## Step 1: Load an product image into base64 format

Before we can ask GPT to generate a product's metadata from a given image URL, we need to convert it into a format that GPT can understand, which is `base64` data URI. To do so, we will create an `image.py` with the following code:

```python
import base64

def encode_image(image_path):
  with open(image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode('utf-8')
```

The above `encode_function` function takes an `image_path`, opens and reads the image into bytes format, and then returns the encoded `based64` text version.

We then write a `load_image` function, which performs the following:

- Receives `inputs` as a dictionary, which contains an `image_path` key with the path to the image file, 
- Reads `inputs[image_path]` into base64 format using `base64.b64encode()` method.
- Assigns the result to `image` property of the returned object for the function.

The code is as follows:

```python
def load_image(inputs: dict) -> dict:
    """Load image from file and encode it as base64."""
    image_file = inputs["image_path"]
    image_base64 = encode_image(image_file)
    return {
        "image": image_base64
    }
```

Now we have the image processing step implemented. Next, we will create a function to communicate with GPT for the information desired based on this image data.

## Step 2: Ask GPT to generate a product's metadata

In this step, since we are going to send request to GPT API, we need to set up its API's key for related Langchain OpenAI package to pick up and initialize the service.

### Setting up OpenAI API key

The most straighforward way is to create an `.env` file with an `OPENAI_API_KEY` variable, whose value can be found under Settings panel, as shown below:

<img src="https://res.cloudinary.com/mayashavin/image/upload/f_auto,q_auto/v1719259328/articles/llm/api" loading="lazy" alt="Screenshot of how to retrieve API key in OpenAI Panel" class="mx-auto">

```bash
OPENAI_API_KEY=your-open-ai-api-key
```

Then, we install `python-dotenv` package using the below command:

```bash
python -m pip install python-dotenv
```

And in our `generate.py` file, we add the following code to load the key from the `.env` file into our project for usage:

```python
import os
from dotenv import load_dotenv

load_dotenv()
```

And with that, we can implement the function that will invoke the GPT model for answers.

### Creating a model to process the image and prompt

In `generate.py`, we create a function `image_model` that takes `inputs` as a dictionary containing the fields: `image` and `prompt`, where `image` is the base64 data URI from step 1.

```python
def image_model(inputs: dict):
 """Invoke model with image and prompt."""

    image = inputs["image"]
    prompt = inputs["prompt"]
```

From the given inputs, we compute a user's message to pass to the model. To do so, we use `HumanMessage` class from `langchain_core.messages` package:

```python
message = HumanMessage(
    content=[
        {"type": "text", "text": prompt},
        {
            "type": "image_url", 
            "image_url": {
                "url": f"data:image/jpeg;base64,{ image }"
            }
        },
    ]
)
```

In the above code, we pass to `HumanMessage` an array of `content` containing:

* A `text` object with the `prompt` text
* An `image_url` object with the base64-encoded `image` data as the URL

Once we have the `message` ready, we then initialize a model instance of `ChatOpenAI` using `gpt-4o`, an `0.5` temperature and a maximum number of `1024` tokens:

```python
from langchain_openai import ChatOpenAI

def image_model(inputs: dict):
 """Invoke model with image and prompt."""

    #... previous code
    model = ChatOpenAI(temperature=0.5, model="gpt-4o", max_tokens=1024)
```

And invoke the model with the `message` and return the `content` of the response, as follows:

```python
def image_model(inputs: dict):
 #... previous code
 result = model.invoke(message)
 return result.content
```

At this stage, we have the content of the response from GPT. In the next step, we will extract that content in a structured Product format.

## Step 3: Extract the result from GPT in a structured Product format

The response from GPT is always in a text format, which requires us to parse and extract the relevant information in a structured Product format. This is not a straightforward step. Fortunately, Langchain provides us several tools to help us with this task, starting with defining the output structure format.

### Define the Product structure

We will define a `Product` class as a Pydantic model using `BaseModel` and `Field` from the `langchain.pydantic_v1` package, as shown below:

```python
# Product.py
from langchain_core.pydantic_v1 import BaseModel, Field

class Product(BaseModel):
    '''Product description'''
    title: str = Field(..., title="Product Title", description="Title of the product")
    description: str = Field(..., title="Product Description", description="Description of the product")
    tags: list = Field([], title="Product Tags", description="Tags for SEO")
```

The above class defines a `Product` model with the following fields:

* `title` - The title of the product
* `description` - The description of the product
* `tags` - The tags for SEO

Next, we declare a parser function that will extract the GPT response into the `Product` structure.

### Create a function to extract the product information

We can use `JsonOutputParser` class to create a custom parser by passing our `Product` structure as its `pydantic_object`, as follows:

```python
from langchain_core.output_parsers import JsonOutputParser

#... previous code
parser = JsonOutputParser(pydantic_object=Product)
```

Great. All left is to modify our `content` array in Step 2 to include the parser's format instructions, by adding the following element to the array:

```python
content = [
    #... previous code
    {"type": "text", "text": parser.get_format_instructions()},
    {
        "type": "image_url", 
       # ... code
    },
]
```

And with that, all the components for the flow is ready. It's time to chain them together.

## Chaining all the steps together using Langchain

Chaining is similar to a train of action carriage, where each carriage can be a step of LLM call, data transformation, or any tool connected together, supporting streaming, async and batch processing out of the box. In our case, we will use `TransformChain` for transforming our `image_path` input into a proper base64 data input as a pre-processing step of the main flow.

```python
from langchain.chains import TransformChain

load_image_chain = TransformChain(
    input_variables=['image_path'],
    output_variables=["image"],
    transform=load_image
)
```

From there, we create another `generate_product_chain` that chains all the flow components together using `|` operator, starting with loading and transforming the image path into a base64 data URI text, then passing its output as the input to our image model for generating the desired data, and finally parsing the result into our target Product format:

```python
generate_product_chain = load_image_chain | image_model | parser
```

Finally, we define `get_product_info` function to invoke the chain with the initial input `image_path` and `prompt` as follows:

```python
def get_product_info(image_path: str) -> dict:
generate_product_chain = load_image_chain | image_model | parser

prompt = f"""
   Given the image of a product, provide the following information:
   - Product Title
   - Product Description
   - At least 13 Product Tags for SEO purposes
"""

return generate_product_chain.invoke({
    'image_path': image_path, 
    'prompt': prompt
})
```

And that's it! We have successfully built a smart product information generator. You can now use the `get_product_info` function to generate product information by giving it a valid image path:

```python
product_info = get_product_info("path/to/image.jpg")
print(product_info)
```

<img src="https://res.cloudinary.com/mayashavin/image/upload/f_auto,q_auto/v1719259328/articles/llm/example_flow_generate" loading="lazy" alt="Diagram flow of generating product data" class="mx-auto">

## Resources

- [Langchain documentation](https://python.langchain.com/v0.2/docs/introduction/)
- [OpenAI API documentation](https://platform.openai.com/docs/overview)
- [Product Information Generator Repo](https://github.com/mayashavin/product-info-ai-generator)

## Summary

In this post, we have explored how to generate essential product data such as title, description and tags based on a given image using Langchain, Open AI GPT-4o. We have walked through the flow, including loading an image into base64 text format, asking GPT to generate a product's metadata, and extracting the result from GPT in a structured Product format. We have also seen how to chain all the steps together using Langchain to create a working product information generator.

In the next post, we will explore how to deploy this tool as a web service API using Flask. Until then, happy coding!

👉 _Learn about Vue 3 and TypeScript with my new book [Learning Vue](https://www.oreilly.com/library/view/learning-vue/9781492098812/)!_

👉 _If you'd like to catch up with me sometimes, follow me on [X](https://x.com/MayaShavin) | [LinkedIn](https://www.linkedin.com/in/mayashavin)._

Like this post or find it helpful? Share it 👇🏼 😉