*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to use multimodal prompts

How to use multimodal prompts
=============================

Here we demonstrate how to use prompt templates to format multimodal inputs to models.

In this example we will ask a model to describe an image.

    import base64import httpximage_url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"image_data = base64.b64encode(httpx.get(image_url).content).decode("utf-8")

    from langchain_core.prompts import ChatPromptTemplatefrom langchain_openai import ChatOpenAImodel = ChatOpenAI(model="gpt-4o")

**API Reference:**[ChatPromptTemplate](https://api.python.langchain.com/en/latest/prompts/langchain_core.prompts.chat.ChatPromptTemplate.html) | [ChatOpenAI](https://api.python.langchain.com/en/latest/chat_models/langchain_openai.chat_models.base.ChatOpenAI.html)

    prompt = ChatPromptTemplate.from_messages(    [        ("system", "Describe the image provided"),        (            "user",            [                {                    "type": "image_url",                    "image_url": {"url": "data:image/jpeg;base64,{image_data}"},                }            ],        ),    ])

    chain = prompt | model

    response = chain.invoke({"image_data": image_data})print(response.content)

    The image depicts a sunny day with a beautiful blue sky filled with scattered white clouds. The sky has varying shades of blue, ranging from a deeper hue near the horizon to a lighter, almost pale blue higher up. The white clouds are fluffy and scattered across the expanse of the sky, creating a peaceful and serene atmosphere. The lighting and cloud patterns suggest pleasant weather conditions, likely during the daytime hours on a mild, sunny day in an outdoor natural setting.

We can also pass in multiple images.

    prompt = ChatPromptTemplate.from_messages(    [        ("system", "compare the two pictures provided"),        (            "user",            [                {                    "type": "image_url",                    "image_url": {"url": "data:image/jpeg;base64,{image_data1}"},                },                {                    "type": "image_url",                    "image_url": {"url": "data:image/jpeg;base64,{image_data2}"},                },            ],        ),    ])

    chain = prompt | model

    response = chain.invoke({"image_data1": image_data, "image_data2": image_data})print(response.content)

    The two images provided are identical. Both images feature a wooden boardwalk path extending through a lush green field under a bright blue sky with some clouds. The perspective, colors, and elements in both images are exactly the same.

[Edit this page](https://github.com/langchain-ai/langchain/edit/master/docs/docs/how_to/multimodal_prompts.ipynb)

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchain/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to pass multimodal data directly to models

](/v0.2/docs/how_to/multimodal_inputs/)[

Next

How to create a custom Output Parser

](/v0.2/docs/how_to/output_parser_custom/)