Repository: langchain
Issue: [Google Generative AI] Structured Output doesn't work with advanced schema
Labels: 🤖:bug,🔌: google,Ɑ:  Tool Calling,todo

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

Collab link : https://colab.research.google.com/drive/1BCat5tBZRcxUhjQ3vGJD3Zu1eiqYIAWz?usp=sharing
Code :
```
!pip install -qU langchain langchain-community langchain-core
!pip install -qU langchain-google-genai
!pip install -qU langchain-text-splitters tiktoken
!pip install -qU faiss-gpu
```
```python
import os
import getpass

os.environ["GOOGLE_API_KEY"] = getpass.getpass("Google API Key:")

import re

import requests
from langchain_community.document_loaders import BSHTMLLoader

# Download the content
response = requests.get("https://en.wikipedia.org/wiki/Car")
# Write it to a file
with open("car.html", "w", encoding="utf-8") as f:
    f.write(response.text)
# Load it with an HTML parser
loader = BSHTMLLoader("car.html")
document = loader.load()[0]
# Clean up code
# Replace consecutive new lines with a single new line
document.page_content = re.sub("\n\n+", "\n", document.page_content)

from typing import List, Optional

from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_core.pydantic_v1 import BaseModel, Field


class KeyDevelopment(BaseModel):
    """Information about a development in the history of cars."""

    year: int = Field(
        ..., description="The year when there was an important historic development."
    )
    description: str = Field(
        ..., description="What happened in this year? What was the development?"
    )
    evidence: str = Field(
        ...,
        description="Repeat in verbatim the sentence(s) from which the year and description information were extracted",
    )


class ExtractionData(BaseModel):
    """Extracted information about key developments in the history of cars."""

    key_developments: List[KeyDevelopment]


# Define a custom prompt to provide instructions and any additional context.
# 1) You can add examples into the prompt template to improve extraction quality
# 2) Introduce additional parameters to take context into account (e.g., include metadata
#    about the document from which the text was extracted.)
prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "You are an expert at identifying key historic development in text. "
            "Only extract important historic developments. Extract nothing if no important information can be found in the text.",
        ),
        ("human", "{text}"),
    ]
)

from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model="gemini-pro")

extractor = prompt | llm.with_structured_output(
    schema=ExtractionData,
    include_raw=False,
)

from langchain_text_splitters import TokenTextSplitter

text_splitter = TokenTextSplitter(
    # Controls the size of each chunk
    chunk_size=2000,
    # Controls overlap between chunks
    chunk_overlap=20,
)

texts = text_splitter.split_text(document.page_content)

from langchain_community.vectorstores import FAISS
from langchain_core.documents import Document
from langchain_core.runnables import RunnableLambda
from langchain_google_genai import GoogleGenerativeAIEmbeddings
from langchain_text_splitters import CharacterTextSplitter

texts = text_splitter.split_text(document.page_content)
vectorstore = FAISS.from_texts(texts, embedding=GoogleGenerativeAIEmbeddings(model="models/embedding-001"))

retriever = vectorstore.as_retriever(
    search_kwargs={"k": 1}
)  # Only extract from first document

rag_extractor = {
    "text": retriever | (lambda docs: docs[0].page_content)  # fetch content of top doc
} | extractor

results = rag_extractor.invoke("Key developments associated with cars")
```

### Error Message and Stack Trace (if applicable)

InvalidArgument                           Traceback (most recent call last)
[/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py](https://localhost:8080/#) in _chat_with_retry(**kwargs)
    177         try:
--> 178             return generation_method(**kwargs)
    179         # Do not retry for these errors.

25 frames
[/usr/local/lib/python3.10/dist-packages/google/ai/generativelanguage_v1beta/services/generative_service/client.py](https://localhost:8080/#) in generate_content(self, request, model, contents, retry, timeout, metadata)
    826         # Send the request.
--> 827         response = rpc(
    828             request,

[/usr/local/lib/python3.10/dist-packages/google/api_core/gapic_v1/method.py](https://localhost:8080/#) in __call__(self, timeout, retry, compression, *args, **kwargs)
    130 
--> 131         return wrapped_func(*args, **kwargs)
    132 

[/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py](https://localhost:8080/#) in retry_wrapped_func(*args, **kwargs)
    292             )
--> 293             return retry_target(
    294                 target,

[/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py](https://localhost:8080/#) in retry_target(target, predicate, sleep_generator, timeout, on_error, exception_factory, **kwargs)
    152             # defer to shared logic for handling errors
--> 153             _retry_error_helper(
    154                 exc,

[/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_base.py](https://localhost:8080/#) in _retry_error_helper(exc, deadline, next_sleep, error_list, predicate_fn, on_error_fn, exc_factory_fn, original_timeout)
    211         )
--> 212         raise final_exc from source_exc
    213     if on_error_fn is not None:

[/usr/local/lib/python3.10/dist-packages/google/api_core/retry/retry_unary.py](https://localhost:8080/#) in retry_target(target, predicate, sleep_generator, timeout, on_error, exception_factory, **kwargs)
    143         try:
--> 144             result = target()
    145             if inspect.isawaitable(result):

[/usr/local/lib/python3.10/dist-packages/google/api_core/timeout.py](https://localhost:8080/#) in func_with_timeout(*args, **kwargs)
    119 
--> 120             return func(*args, **kwargs)
    121 

[/usr/local/lib/python3.10/dist-packages/google/api_core/grpc_helpers.py](https://localhost:8080/#) in error_remapped_callable(*args, **kwargs)
     80         except grpc.RpcError as exc:
---> 81             raise exceptions.from_grpc_error(exc) from exc
     82 

InvalidArgument: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[key_developments].items: missing field.


The above exception was the direct cause of the following exception:

ChatGoogleGenerativeAIError               Traceback (most recent call last)
[<ipython-input-18-49ad0989f74d>](https://localhost:8080/#) in <cell line: 1>()
----> 1 results = rag_extractor.invoke("Key developments associated with cars")

[/usr/local/lib/python3.10/dist-packages/langchain_core/runnables/base.py](https://localhost:8080/#) in invoke(self, input, config, **kwargs)
   2794                     input = step.invoke(input, config, **kwargs)
   2795                 else:
-> 2796                     input = step.invoke(input, config)
   2797         # finish the root run
   2798         except BaseException as e:

[/usr/local/lib/python3.10/dist-packages/langchain_core/runnables/base.py](https://localhost:8080/#) in invoke(self, input, config, **kwargs)
   4976         **kwargs: Optional[Any],
   4977     ) -> Output:
-> 4978         return self.bound.invoke(
   4979             input,
   4980             self._merge_configs(config),

[/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py](https://localhost:8080/#) in invoke(self, input, config, stop, **kwargs)
    263         return cast(
    264             ChatGeneration,
--> 265             self.generate_prompt(
    266                 [self._convert_input(input)],
    267                 stop=stop,

[/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py](https://localhost:8080/#) in generate_prompt(self, prompts, stop, callbacks, **kwargs)
    696     ) -> LLMResult:
    697         prompt_messages = [p.to_messages() for p in prompts]
--> 698         return self.generate(prompt_messages, stop=stop, callbacks=callbacks, **kwargs)
    699 
    700     async def agenerate_prompt(

[/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py](https://localhost:8080/#) in generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs)
    553                 if run_managers:
    554                     run_managers[i].on_llm_error(e, response=LLMResult(generations=[]))
--> 555                 raise e
    556         flattened_outputs = [
    557             LLMResult(generations=[res.generations], llm_output=res.llm_output)  # type: ignore[list-item]

[/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py](https://localhost:8080/#) in generate(self, messages, stop, callbacks, tags, metadata, run_name, run_id, **kwargs)
    543             try:
    544                 results.append(
--> 545                     self._generate_with_cache(
    546                         m,
    547                         stop=stop,

[/usr/local/lib/python3.10/dist-packages/langchain_core/language_models/chat_models.py](https://localhost:8080/#) in _generate_with_cache(self, messages, stop, run_manager, **kwargs)
    768         else:
    769             if inspect.signature(self._generate).parameters.get("run_manager"):
--> 770                 result = self._generate(
    771                     messages, stop=stop, run_manager=run_manager, **kwargs
    772                 )

[/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py](https://localhost:8080/#) in _generate(self, messages, stop, run_manager, tools, functions, safety_settings, tool_config, generation_config, **kwargs)
    765             generation_config=generation_config,
    766         )
--> 767         response: GenerateContentResponse = _chat_with_retry(
    768             request=request,
    769             **kwargs,

[/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py](https://localhost:8080/#) in _chat_with_retry(generation_method, **kwargs)
    194             raise e
    195 
--> 196     return _chat_with_retry(**kwargs)
    197 
    198 

[/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py](https://localhost:8080/#) in wrapped_f(*args, **kw)
    334             copy = self.copy()
    335             wrapped_f.statistics = copy.statistics  # type: ignore[attr-defined]
--> 336             return copy(f, *args, **kw)
    337 
    338         def retry_with(*args: t.Any, **kwargs: t.Any) -> WrappedFn:

[/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py](https://localhost:8080/#) in __call__(self, fn, *args, **kwargs)
    473         retry_state = RetryCallState(retry_object=self, fn=fn, args=args, kwargs=kwargs)
    474         while True:
--> 475             do = self.iter(retry_state=retry_state)
    476             if isinstance(do, DoAttempt):
    477                 try:

[/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py](https://localhost:8080/#) in iter(self, retry_state)
    374         result = None
    375         for action in self.iter_state.actions:
--> 376             result = action(retry_state)
    377         return result
    378 

[/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py](https://localhost:8080/#) in <lambda>(rs)
    396     def _post_retry_check_actions(self, retry_state: "RetryCallState") -> None:
    397         if not (self.iter_state.is_explicit_retry or self.iter_state.retry_run_result):
--> 398             self._add_action_func(lambda rs: rs.outcome.result())
    399             return
    400 

[/usr/lib/python3.10/concurrent/futures/_base.py](https://localhost:8080/#) in result(self, timeout)
    449                     raise CancelledError()
    450                 elif self._state == FINISHED:
--> 451                     return self.__get_result()
    452 
    453                 self._condition.wait(timeout)

[/usr/lib/python3.10/concurrent/futures/_base.py](https://localhost:8080/#) in __get_result(self)
    401         if self._exception:
    402             try:
--> 403                 raise self._exception
    404             finally:
    405                 # Break a reference cycle with the exception in self._exception

[/usr/local/lib/python3.10/dist-packages/tenacity/__init__.py](https://localhost:8080/#) in __call__(self, fn, *args, **kwargs)
    476             if isinstance(do, DoAttempt):
    477                 try:
--> 478                     result = fn(*args, **kwargs)
    479                 except BaseException:  # noqa: B902
    480                     retry_state.set_exception(sys.exc_info())  # type: ignore[arg-type]

[/usr/local/lib/python3.10/dist-packages/langchain_google_genai/chat_models.py](https://localhost:8080/#) in _chat_with_retry(**kwargs)
    188 
    189         except google.api_core.exceptions.InvalidArgument as e:
--> 190             raise ChatGoogleGenerativeAIError(
    191                 f"Invalid argument provided to Gemini: {e}"
    192             ) from e

ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 * GenerateContentRequest.tools[0].function_declarations[0].parameters.properties[key_developments].items: missing field.

### Description

Hi !

Since yesterday, I try to follow this official guide in the v0.2 documentation : https://python.langchain.com/v0.2/docs/how_to/extraction_long_text/

However, it doesn't work well with Chat Google Generative AI
The collab link is here, if you want to try : https://colab.research.google.com/drive/1BCat5tBZRcxUhjQ3vGJD3Zu1eiqYIAWz?usp=sharing

I have followed the guide step by step, but it keep having an error about missing field on the request.
For information, Chat Google Generative AI have Structured Output : https://python.langchain.com/v0.2/docs/integrations/chat/google_generative_ai/
And also, it's not about my location either (I have already success for others use of Chat Google Generative AI)

I have try differents things with schema, and I go to the conclusion that I can't use scheme that define other scheme in it like (or List):
```python
class ExtractionData(BaseModel):
    """Extracted information about key developments in the history of cars."""

    key_developments: List[KeyDevelopment]
```

However I can use without problem this scheme :
```python
class KeyDevelopment(BaseModel):
    """Information about a development in the history of cars."""

    year: int = Field(
        ..., description="The year when there was an important historic development."
    )
    description: str = Field(
        ..., description="What happened in this year? What was the development?"
    )
    evidence: str = Field(
        ...,
        description="Repeat in verbatim the sentence(s) from which the year and description information were extracted",
    )
```

(but responses with scheme tend to have very bad result with Chat Google, like it's 90% time non-sense)

Sorry for my english which is not really perfect and thank you for reading me !

- ToyHugs

### System Info

https://colab.research.google.com/drive/1BCat5tBZRcxUhjQ3vGJD3Zu1eiqYIAWz?usp=sharing