Repository: UKGovernmentBEIS/inspect_ai
Issue: Gemini API call structured incorrectly
Root cause: Interface Contract Mismatch

The API calls to gemini models incorrectly nest `systemInstruction`, `safetySettings`, `tools`, and `toolConfig` inside the `generation_config` object. According to the [[official Gemini API documentation](https://ai.google.dev/api/generate-content)](https://ai.google.dev/api/generate-content), these should be root-level fields.

## Current Behavior

Currently the API structure is like this:

```json
{
  "contents": [...],
  "generation_config": {
    "systemInstruction": [...],    // ❌ Wrong location
    "safetySettings": [...],        // ❌ Wrong location
    "tools": [...],                 // ❌ Wrong location
    "toolConfig": {...},            // ❌ Wrong location
    "httpOptions": {...}            // ❌ Not part of API spec
  },
  "safety_settings": [...],         // Duplicate
  "tools": [...],                   // Duplicate
  "tool_config": {...}              // Duplicate
}
```

## Expected Behavior

According to the [[official API specification](https://ai.google.dev/api/generate-content#request-body)](https://ai.google.dev/api/generate-content#request-body), the request should be:

```json
{
  "contents": [...],
  "systemInstruction": {...},      // ✅ Root level
  "safetySettings": [...],         // ✅ Root level
  "tools": [...],                  // ✅ Root level
  "toolConfig": {...},             // ✅ Root level
  "generationConfig": {            // ✅ Only generation parameters
    "temperature": 1.0,
    "maxOutputTokens": 2048
  }
}
```

## Impact

- Unpredictable behavior when fields are duplicated at multiple locations. The models do actually see the system prompt despite the wrong location presumably because of very lenient parsing.
- Non-compliance with official API specification


## Minimal Reproduction
```
#!/usr/bin/env python3
"""
Minimal example testing Gemini with a simple tool.
This helps diagnose whether tools/safetySettings are being incorrectly nested in generation_config.
"""

from inspect_ai import Task, eval as inspect_eval, task
from inspect_ai.dataset import Sample
from inspect_ai.solver import generate, system_message, use_tools
from inspect_ai.tool import tool


# Simple toy tool
@tool
def get_weather():
    async def execute(city: str):
        """
        Get the weather for a city.

        Args:
            city: The name of the city

        Returns:
            A string describing the weather
        """
        # Mock weather data
        weather_data = {
            "San Francisco": "Foggy, 60°F",
            "New York": "Sunny, 75°F",
            "London": "Rainy, 55°F",
        }
        return weather_data.get(city, f"Weather data not available for {city}")

    return execute


@tool
def add_numbers():
    async def execute(a: int, b: int):
        """
        Add two numbers together.

        Args:
            a: First number
            b: Second number

        Returns:
            The sum of a and b
        """
        return f"The sum of {a} and {b} is {a + b}"

    return execute


# Create a simple task that should use the tool
@task
def gemini_tool_test():
    """Test Gemini with a simple tool calling task."""
    return Task(
        dataset=[
            Sample(
                input="What's the weather like in San Francisco?",
                target="foggy",
            ),
            Sample(
                input="What is 42 plus 58?",
                target="100",
            ),
        ],
        solver=[
            system_message("You are a helpful assistant"),
            use_tools([get_weather(), add_numbers()]),
            generate(),
        ],
    )


def main():
    print("=" * 60)
    print("Testing Gemini with tool calling")
    print("=" * 60)

    # Test with Gemini model
    model = "google/gemini-2.5-flash"

    print(f"\nRunning eval with model: {model}")
    print(
        "This will help diagnose if tools are being incorrectly nested in generation_config"
    )
    print()

    results = inspect_eval(
        tasks=[gemini_tool_test()],
        model=model,
        log_dir="logs/gemini_tool_test",
    )

    print("\n" + "=" * 60)
    print("Eval completed successfully!")
    print("=" * 60)
    print(f"Results: {results}")


if __name__ == "__main__":
    main()
```

Thanks for reporting this! The good news is that the bug is just in our reporting/diagnostics of the API call. Under the hood we are calling the Gemini Python API which forms the actual HTTP request (our diagnostics are just a simulation of the request body). 

We've updated our diagnostics to correctly report the actual request payload here: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2625