Repository: langchain
Issue: Issue with LangChain LCEL-based Chain, gemini models and google storage bucket inputs  #gemini 
Labels: Ɑ:  core

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

```from langchain_google_vertexai import ChatVertexAI
from langchain_core.output_parsers import PydanticOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_core.runnables import RunnableSequence,RunnablePassthrough 
from langchain_google_vertexai import HarmBlockThreshold, HarmCategory
from typing import List


class Highlight(BaseModel):
    startTime: str = Field(description="Start time of the highlight.")
    endTime: str = Field(description="End time of the highlight.")
    caption: str = Field(description="A short caption.")
    scene_type: str = Field(description="Type of scene.")


class Highlights(BaseModel):
    highlights: List[Highlight]

parser = PydanticOutputParser(pydantic_object=Highlights)

prompt = ChatPromptTemplate.from_messages([
    ("system", "{system_instruction}\n'{format_instructions}'\n"),
    ("human", '{human_message}')
])

chain = prompt | ChatVertexAI(model="gemini-1.5-flash-002", temperature=0.4, max_tokens=8192, top_p=0.95, safety_settings={
                    HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
                    HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
                    HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
                    HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
                }) | parser


system_instruction_template = """
#### You are an AI-agent specializing in creating video hightlights."""


text_message1 = """Please analyze the video clip and extract highlights scences."""




media_message1 ={
    "type": "media_input",
    "video_url": {
        "url": "https://storage.googleapis.com/sample-video/video.mp4",
    },
}


result = await chain.ainvoke({
    "language": "English",
    "system_instruction": system_instruction_template,
    "format_instructions": parser.get_format_instructions(),
    "human_message": [media_message1, text_message1]
})

print(result.json(indent=2))```

### Error Message and Stack Trace (if applicable)

_No response_

### Description

I’m working on a GenAI-based  application where video files are uploaded to Google Cloud Storage and processed through the Gemini Flash 1.5 model via LangChain. The application has been running successfully since early September, but recently it started returning responses that are irrelevant to the actual video content.

The issue seems to lie in how LangChain is orchestrating prompts and handling the video input. Despite the video files being correctly retrieved via gsutil, it appears that the files are not being passed correctly to the LLM, resulting in hallucinated or off-topic responses. When I directly use the Gemini Flash 1.5 API without LangChain, the responses are accurate.

Expected Behavior:
The model should return video content-specific insights (e.g., extracting key scenes based on dialogues and interactions).

Current Behavior:
The model is generating irrelevant responses that do not reflect the video content, suggesting a breakdown in the input-output handling when using LangChain.



### System Info

Python 3.8.10
langchain-google-vertexai==1.0.6
langchain-core==0.2.41
Model version "gemini-1.5-flash-002"