Repository: UKGovernmentBEIS/inspect_ai
Issue: How to control the agent thinking depth
Root cause: Unimplemented Feature Gap

We are building an eval to test multi-agent communication and in multi turn format. Such as a client-agent talk to the customer-service agent.

In this scenario, we would love the order to be

Turn 1:
   - client: "Hello, I have a baggage check in question about my flight.?"
   - customer_service: "can I verify your loyalty member ID so I can try to look up your details that way?"

Turn 2:
  - client: "yes my loyalty member ID is ABCD"
  - customer_service: "Thanks for confirming, I am able to identify your information, how can I help you?"

I expect to use a for loop to ask these two agents building the context incrementally, and each turn only fixed one small question. 

However, with my implemenation

```
customer_service_state = AgentState(
    messages=[
        ChatMessageUser(
            content="Hello, I have a baggage check in question about my flight."
        )
    ]
)

updated_customer_service_state = await customer_service_agent(
            customer_service_state
        )
```

When I print out the thinking trace, I found

```
"_messages": [
    {
      "id": "PoMXwrW3KWDPou5isew6qM",
      "content": "Some system prompt.\n",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "system"
    },
    {
      "id": "aZionQtXWoti46oALkN34v",
      "content": "Hello, I have a baggage check in question about my flight.",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "user",
      "tool_call_id": null
    },
    {
      "id": "MJ5H6ibDT2axb8Lt665UVB",
      "content": [
        {
          "internal": null,
          "type": "text",
          "text": "To assist you with your baggage check-in question, could you please provide your client confirmation number or loyalty member ID so I can look up your booking information? \n[get_client_booking_info(), get_client_membership_info()]",
          "refusal": true,
          "citations": null
        }
      ],
      "source": "generate",
      "metadata": null,
      "internal": null,
      "role": "assistant",
      "tool_calls": null,
    },
    {
      "id": "Qi6ri3ztg8SKTKmjYFdS2j",
      "content": "False",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "user",
      "tool_call_id": null
    },
    {
      "id": "GYhWcukviN5ZW8SjJ6vKbZ",
      "content": [
        {
          "internal": null,
          "type": "text",
          "text": "It seems that the tool call was not successful. Can you please provide your client confirmation number or loyalty member ID so I can assist you with your baggage check-in question?",
          "refusal": true,
          "citations": null
        }
      ],
      "source": "generate",
      "metadata": null,
      "internal": null,
      "role": "assistant",
      "tool_calls": null,
    },
    {
      "id": "JV7bSY8wQVHFQEKS7QynWi",
      "content": "False",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "user",
      "tool_call_id": null
    },
    {
      "id": "aDCJTcaZ7T4FWJmmd566HZ",
      "content": [
        {
          "internal": null,
          "type": "text",
          "text": "To assist you, I need some information about your booking. Can you please provide your client confirmation number? \n[get_client_booking_info(client_confirmation_number=\"provided_by_client\")]",
          "refusal": true,
          "citations": null
        }
      ],
      "source": "generate",
      "metadata": null,
      "internal": null,
      "role": "assistant",
      "tool_calls": null,
    },
    {
      "id": "G7pZsr8Szuubjk7rahToVj",
      "content": "False",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "user",
      "tool_call_id": null
    },
    {
      "id": "HSCKQ8TfAwgjfY9niaJCDt",
      "content": [
        {
          "internal": null,
          "type": "text",
          "text": "It seems I'm having trouble retrieving your information. Can you please provide your loyalty member ID so I can try to look up your details that way? \n[get_client_membership_info(client_loyalty_member_id=\"provided_by_client\")]",
          "refusal": true,
          "citations": null
        }
      ],
      "source": "generate",
      "metadata": null,
      "internal": null,
      "role": "assistant",
      "tool_calls": null,
    },
    {
      "id": "AnwndBGyV93MAeavtRCWXb",
      "content": "False",
      "source": null,
      "metadata": null,
      "internal": null,
      "role": "user",
      "tool_call_id": null
    },
    {
      "id": "Ev4XAyTuqHrF4oWoARX54X",
      "content": [
        {
          "internal": null,
          "type": "text",
          "text": "To proceed, I need either your client confirmation number or loyalty member ID. Can you please provide one of these so I can assist you with your baggage check-in question?",
          "refusal": true,
          "citations": null
        }
      ],
      "source": "generate",
      "metadata": null,
      "internal": null,
      "role": "assistant",
      "tool_calls": null,
    },
```

As one can see it keep on trying to fetch all the information at the current turn, and in my case, it asks for >10 times and eventually make things up. 

In the system prompt I already ask it to do

```
Please stop thinking if cannot identify the user information and immediately return the latest assistant message to the client agent
```

Since the agent work too hard to get the answer 
```

Is there a way such that I can make this agent stop thinking after few tries? Seems the submit() function is feasible but I am not sure how to use that.