*   [](/v0.2/)
*   [How-to guides](/v0.2/docs/how_to/)
*   How to pass multimodal data directly to models

On this page

How to pass multimodal data directly to models
==============================================

Prerequisites

This guide assumes familiarity with the following concepts:

*   [Chat models](/v0.2/docs/concepts/#chat-models)

Here we demonstrate how to pass multimodal input directly to models. We currently expect all input to be passed in the same format as [OpenAI expects](https://platform.openai.com/docs/guides/vision). For other model providers that support multimodal input, we have added logic inside the class to convert to the expected format.

In this example we will ask a model to describe an image.

    import * as fs from "node:fs/promises";import { ChatAnthropic } from "@langchain/anthropic";const model = new ChatAnthropic({  model: "claude-3-sonnet-20240229",});const imageData = await fs.readFile("../../../../examples/hotdog.jpg");

The most commonly supported way to pass in images is to pass it in as a byte string within a message with a complex content type for models that support multimodal input. Here’s an example:

    import { HumanMessage } from "@langchain/core/messages";const message = new HumanMessage({  content: [    {      type: "text",      text: "what does this image contain?",    },    {      type: "image_url",      image_url: {        url: `data:image/jpeg;base64,${imageData.toString("base64")}`,      },    },  ],});const response = await model.invoke([message]);console.log(response.content);

    This image contains a hot dog. It shows a frankfurter or sausage encased in a soft, elongated bread bun. The sausage itself appears to be reddish in color, likely a smoked or cured variety. The bun is a golden-brown color, suggesting it has been lightly toasted or grilled. The hot dog is presented against a plain white background, allowing the details of the iconic American fast food item to be clearly visible.

Some model providers support taking an HTTP URL to the image directly in a content block of type `"image_url"`:

    import { ChatOpenAI } from "@langchain/openai";const openAIModel = new ChatOpenAI({  model: "gpt-4o",});const imageUrl =  "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg";const message = new HumanMessage({  content: [    {      type: "text",      text: "describe the weather in this image",    },    {      type: "image_url",      image_url: { url: imageUrl },    },  ],});const response = await openAIModel.invoke([message]);console.log(response.content);

    The weather in the image appears to be pleasant and clear. The sky is mostly blue with a few scattered clouds, indicating good visibility and no immediate signs of rain. The lighting suggests it’s either morning or late afternoon, with sunlight creating a warm and bright atmosphere. There is no indication of strong winds, as the grass and foliage appear calm and undisturbed. Overall, it looks like a beautiful day, possibly spring or summer, ideal for outdoor activities.

We can also pass in multiple images.

    const message = new HumanMessage({  content: [    {      type: "text",      text: "are these two images the same?",    },    {      type: "image_url",      image_url: {        url: imageUrl,      },    },    {      type: "image_url",      image_url: {        url: imageUrl,      },    },  ],});const response = await openAIModel.invoke([message]);console.log(response.content);

    Yes, the two images are the same.

Next steps[​](#next-steps "Direct link to Next steps")
------------------------------------------------------

You’ve now learned how to pass multimodal data to a modal.

Next, you can check out our guide on [multimodal tool calls](/v0.2/docs/how_to/tool_calls_multimodal).

* * *

#### Was this page helpful?

  

#### You can also leave detailed feedback [on GitHub](https://github.com/langchain-ai/langchainjs/issues/new?assignees=&labels=03+-+Documentation&projects=&template=documentation.yml&title=DOC%3A+%3CPlease+write+a+comprehensive+title+after+the+%27DOC%3A+%27+prefix%3E).

[

Previous

How to generate multiple embeddings per document

](/v0.2/docs/how_to/multi_vector)[

Next

How to use multimodal prompts

](/v0.2/docs/how_to/multimodal_prompts)

*   [Next steps](#next-steps)