{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4facdf7f-680e-4d28-908b-2b8408e2a741",
   "metadata": {},
   "source": [
    "# How to pass multimodal data directly to models\n",
    "\n",
    "Here we demonstrate how to pass multimodal input directly to models. \n",
    "We currently expect all input to be passed in the same format as [OpenAI expects](https://platform.openai.com/docs/guides/vision).\n",
    "For other model providers that support multimodal input, we have added logic inside the class to convert to the expected format.\n",
    "\n",
    "In this example we will ask a model to describe an image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0d9fd81a-b7f0-445a-8e3d-cfc2d31fdd59",
   "metadata": {},
   "outputs": [],
   "source": [
    "image_url = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "fb896ce9",
   "metadata": {},
   "outputs": [],
   "source": [
    "from langchain_core.messages import HumanMessage\n",
    "from langchain_openai import ChatOpenAI\n",
    "\n",
    "model = ChatOpenAI(model=\"gpt-4o\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4fca4da7",
   "metadata": {},
   "source": [
    "The most commonly supported way to pass in images is to pass it in as a byte string.\n",
    "This should work for most model integrations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9ca1040c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import base64\n",
    "\n",
    "import httpx\n",
    "\n",
    "image_data = base64.b64encode(httpx.get(image_url).content).decode(\"utf-8\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "ec680b6b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weather in the image appears to be clear and pleasant. The sky is mostly blue with scattered, light clouds, suggesting a sunny day with minimal cloud cover. There is no indication of rain or strong winds, and the overall scene looks bright and calm. The lush green grass and clear visibility further indicate good weather conditions.\n"
     ]
    }
   ],
   "source": [
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": \"describe the weather in this image\"},\n",
    "        {\n",
    "            \"type\": \"image_url\",\n",
    "            \"image_url\": {\"url\": f\"data:image/jpeg;base64,{image_data}\"},\n",
    "        },\n",
    "    ],\n",
    ")\n",
    "response = model.invoke([message])\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8656018e-c56d-47d2-b2be-71e87827f90a",
   "metadata": {},
   "source": [
    "We can feed the image URL directly in a content block of type \"image_url\". Note that only some model providers support this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a8819cf3-5ddc-44f0-889a-19ca7b7fe77e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The weather in the image appears to be clear and sunny. The sky is mostly blue with a few scattered clouds, suggesting good visibility and a likely pleasant temperature. The bright sunlight is casting distinct shadows on the grass and vegetation, indicating it is likely daytime, possibly late morning or early afternoon. The overall ambiance suggests a warm and inviting day, suitable for outdoor activities.\n"
     ]
    }
   ],
   "source": [
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": \"describe the weather in this image\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n",
    "    ],\n",
    ")\n",
    "response = model.invoke([message])\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c470309",
   "metadata": {},
   "source": [
    "We can also pass in multiple images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "325fb4ca",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Yes, the two images are the same. They both depict a wooden boardwalk extending through a grassy field under a blue sky with light clouds. The scenery, lighting, and composition are identical.\n"
     ]
    }
   ],
   "source": [
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": \"are these two images the same?\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n",
    "    ],\n",
    ")\n",
    "response = model.invoke([message])\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71bd28cf-d76c-44e2-a55e-c5f265db986e",
   "metadata": {},
   "source": [
    "## Tool calls\n",
    "\n",
    "Some multimodal models support [tool calling](/docs/concepts/#functiontool-calling) features as well. To call tools using such models, simply bind tools to them in the [usual way](/docs/how_to/tool_calling), and invoke the model using content blocks of the desired type (e.g., containing image data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cd22ea82-2f93-46f9-9f7a-6aaf479fcaa9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'name': 'weather_tool', 'args': {'weather': 'sunny'}, 'id': 'call_BSX4oq4SKnLlp2WlzDhToHBr'}]\n"
     ]
    }
   ],
   "source": [
    "from typing import Literal\n",
    "\n",
    "from langchain_core.tools import tool\n",
    "\n",
    "\n",
    "@tool\n",
    "def weather_tool(weather: Literal[\"sunny\", \"cloudy\", \"rainy\"]) -> None:\n",
    "    \"\"\"Describe the weather\"\"\"\n",
    "    pass\n",
    "\n",
    "\n",
    "model_with_tools = model.bind_tools([weather_tool])\n",
    "\n",
    "message = HumanMessage(\n",
    "    content=[\n",
    "        {\"type\": \"text\", \"text\": \"describe the weather in this image\"},\n",
    "        {\"type\": \"image_url\", \"image_url\": {\"url\": image_url}},\n",
    "    ],\n",
    ")\n",
    "response = model_with_tools.invoke([message])\n",
    "print(response.tool_calls)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}