Repository: UKGovernmentBEIS/inspect_ai
Issue: Cannot use BedrockAPI for Claude 4 Series Model
Root cause: Validation Gap

Hi, I am trying to run Inspect SWE bench for Claude 4 Series Model through BedrockAPI
```
inspect eval inspect_evals/swe_bench_verified_mini \
  --model bedrock//anthropic.claude-sonnet-4-20250514-v1:0  \
  --message-limit 500 \
  --limit=10
```

But get the following error 
```
RuntimeError: ValidationException('An error occurred (ValidationException) when calling the Converse operation: Invocation of model ID anthropic.claude-sonnet-4-20250514-v1:0 with 
on-demand throughput isn't supported. Retry your request with the ID or ARN of an inference profile that contains this model.')
```

I am not sure about the details. But looks like it's related to Inspect `BedrockAPI` impl use `ConverseClientConverseRequest` ? https://github.com/UKGovernmentBEIS/inspect_ai/blob/0270f1888a8e1c02024feaf640b0a275b3d0b35c/src/inspect_ai/model/_providers/bedrock.py#L350-L369 . 

I don't see this issue with Sonnet 3.7. 

Is there anyway to run Inspect to eval Claude 4 Series Model via BedrockAPI -- is there a way not to use Converse operation (what's this for?) which seems not supported by Claude 4 on BedrockAPI ? Thanks! 🙏 

The preferred way to run Anthropic models on Bedrock is to use the `anthropic` client directly, which generally has far fewer of these sorts of issues: https://inspect.aisi.org.uk/providers.html#anthropic-on-aws-bedrock

Thanks @jjallaire . I tested `AnthropicAPI` with Bedrock model and it works, looking at the implementation, it seems indeed more recommended over `BedrockAPI`. At a separate note,  I was once get into the following `HTTPClientError` with `BedrockAPI` sometimes when connection is high (e.g. >50); and I have to do some temporary workaround to cache the client (not just boto Session). 
```
HTTPClientError: An HTTP Client raised an unhandled exception: [Errno 24] Too many open files
```

But looking at the implementation, `AnthropicAPI` looks promising since it already holds the single client. 

-------------------

For my specific issue, it turns out i miss `us.` prefix. Changing to use `us.anthropic.claude-sonnet-4-20250514-v1:0 ` also works for `BedrockAPI`. But agree `AnthropicAPI` seems preferred with potential other improvements. 



^^ stacktrace for `[Errno 24] Too many open files`:

```
┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/httpsession.py:221 in send                    │
│                                                                                                                      │
│   218 │   │   │   │   data = _IOBaseWrapper(data)                                                                    │
│   219 │   │   │                                                                                                      │
│   220 │   │   │   url = URL(url, encoded=True)                                                                       │
│ > 221 │   │   │   session = await self._get_session(proxy_url)                                                       │
│   222 │   │   │   response = await session.request(                                                                  │
│   223 │   │   │   │   request.method,                                                                                │
│   224 │   │   │   │   url=url,                                                                                       │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/httpsession.py:175 in _get_session            │
│                                                                                                                      │
│   172 │                                                                                                              │
│   173 │   async def _get_session(self, proxy_url):                                                                   │
│   174 │   │   if not (session := self._sessions.get(proxy_url)):                                                     │
│ > 175 │   │   │   connector = self._create_connector(proxy_url)                                                      │
│   176 │   │   │   self._sessions[proxy_url] = (                                                                      │
│   177 │   │   │   │   session                                                                                        │
│   178 │   │   │   ) = await self._exit_stack.enter_async_context(                                                    │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/httpsession.py:165 in _create_connector       │
│                                                                                                                      │
│   162 │   │   │   │   # inline self._setup_ssl_cert                                                                  │
│   163 │   │   │   │   ca_certs = get_cert_path(self._verify)                                                         │
│   164 │   │   │   │   if ca_certs:                                                                                   │
│ > 165 │   │   │   │   │   ssl_context.load_verify_locations(ca_certs, None, None)                                    │
│   166 │   │                                                                                                          │
│   167 │   │   return aiohttp.TCPConnector(                                                                           │
│   168 │   │   │   limit=self._max_pool_connections,                                                                  │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
OSError: [Errno 24] Too many open files

During handling of the above exception, another exception occurred:

┌───────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────┐
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/_eval/task/run.py:678 in task_run_sample       │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/solver/_plan.py:106 in __call__                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/agent/_as_solver.py:66 in solve                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/agent/_react.py:176 in execute                 │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/agent/_react.py:431 in _agent_generate         │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/agent/_react.py:436 in generate                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/model/_model.py:407 in generate                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/model/_model.py:699 in _generate               │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:189 in async_wrapped         │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:111 in __call__              │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:153 in iter                  │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/_utils.py:99 in inner                            │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/__init__.py:400 in <lambda>                      │
│                                                                                                                      │
│ /Users/wx/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:449 in │
│ result                                                                                                               │
│                                                                                                                      │
│   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                         │
│   447 │   │   │   │   │   raise CancelledError()                                                                     │
│   448 │   │   │   │   elif self._state == FINISHED:                                                                  │
│ > 449 │   │   │   │   │   return self.__get_result()                                                                 │
│   450 │   │   │   │                                                                                                  │
│   451 │   │   │   │   self._condition.wait(timeout)                                                                  │
│   452                                                                                                                │
│                                                                                                                      │
│ /Users/wx/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:401 in │
│ __get_result                                                                                                         │
│                                                                                                                      │
│   398 │   def __get_result(self):                                                                                    │
│   399 │   │   if self._exception:                                                                                    │
│   400 │   │   │   try:                                                                                               │
│ > 401 │   │   │   │   raise self._exception                                                                          │
│   402 │   │   │   finally:                                                                                           │
│   403 │   │   │   │   # Break a reference cycle with the exception in self._exception                                │
│   404 │   │   │   │   self = None                                                                                    │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:114 in __call__              │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/model/_model.py:639 in generate                │
│                                                                                                                      │
|
(...redacted...)
|
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:189 in async_wrapped         │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:111 in __call__              │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:153 in iter                  │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/_utils.py:99 in inner                            │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/__init__.py:420 in exc_check                     │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/__init__.py:187 in reraise                       │
│                                                                                                                      │
│ /Users/wx/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:449 in │
│ result                                                                                                               │
│                                                                                                                      │
│   446 │   │   │   │   if self._state in [CANCELLED, CANCELLED_AND_NOTIFIED]:                                         │
│   447 │   │   │   │   │   raise CancelledError()                                                                     │
│   448 │   │   │   │   elif self._state == FINISHED:                                                                  │
│ > 449 │   │   │   │   │   return self.__get_result()                                                                 │
│   450 │   │   │   │                                                                                                  │
│   451 │   │   │   │   self._condition.wait(timeout)                                                                  │
│   452                                                                                                                │
│                                                                                                                      │
│ /Users/wx/.local/share/uv/python/cpython-3.12.9-macos-aarch64-none/lib/python3.12/concurrent/futures/_base.py:401 in │
│ __get_result                                                                                                         │
│                                                                                                                      │
│   398 │   def __get_result(self):                                                                                    │
│   399 │   │   if self._exception:                                                                                    │
│   400 │   │   │   try:                                                                                               │
│ > 401 │   │   │   │   raise self._exception                                                                          │
│   402 │   │   │   finally:                                                                                           │
│   403 │   │   │   │   # Break a reference cycle with the exception in self._exception                                │
│   404 │   │   │   │   self = None                                                                                    │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/tenacity/asyncio/__init__.py:114 in __call__              │
|
(...redacted...)
|
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/inspect_ai/model/_providers/bedrock.py:382 in generate    │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/client.py:394 in _make_api_call               │
│                                                                                                                      │
│   391 │   │   │   │   self.meta.config, request_dict, operation_model                                                │
│   392 │   │   │   )                                                                                                  │
│   393 │   │   │   apply_request_checksum(request_dict)                                                               │
│ > 394 │   │   │   http, parsed_response = await self._make_request(                                                  │
│   395 │   │   │   │   operation_model, request_dict, request_context                                                 │
│   396 │   │   │   )                                                                                                  │
│   397                                                                                                                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/client.py:420 in _make_request                │
│                                                                                                                      │
│   417 │   │   self, operation_model, request_dict, request_context                                                   │
│   418 │   ):                                                                                                         │
│   419 │   │   try:                                                                                                   │
│ > 420 │   │   │   return await self._endpoint.make_request(                                                          │
│   421 │   │   │   │   operation_model, request_dict                                                                  │
│   422 │   │   │   )                                                                                                  │
│   423 │   │   except Exception as e:                                                                                 │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/endpoint.py:142 in _send_request              │
│                                                                                                                      │
│   139 │   │   │   │   total_retries                                                                                  │
│   140 │   │   │   )                                                                                                  │
│   141 │   │   if exception is not None:                                                                              │
│ > 142 │   │   │   raise exception                                                                                    │
│   143 │   │   else:                                                                                                  │
│   144 │   │   │   return success_response                                                                            │
│   145                                                                                                                │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/endpoint.py:194 in _do_get_response           │
│                                                                                                                      │
│   191 │   │   │   )                                                                                                  │
│   192 │   │   │   http_response = first_non_none_response(responses)                                                 │
│   193 │   │   │   if http_response is None:                                                                          │
│ > 194 │   │   │   │   http_response = await self._send(request)                                                      │
│   195 │   │   except HTTPClientError as e:                                                                           │
│   196 │   │   │   return (None, e)                                                                                   │
│   197 │   │   except Exception as e:                                                                                 │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/endpoint.py:296 in _send                      │
│                                                                                                                      │
│   293 │   │   │   return True                                                                                        │
│   294 │                                                                                                              │
│   295 │   async def _send(self, request):                                                                            │
│ > 296 │   │   return await self.http_session.send(request)                                                           │
│   297                                                                                                                │
│   298                                                                                                                │
│   299 class AioEndpointCreator(EndpointCreator):                                                                     │
│                                                                                                                      │
│ /Users/wx/.virtualenvs/openai/lib/python3.12/site-packages/aiobotocore/httpsession.py:282 in send                    │
│                                                                                                                      │
│   279 │   │   except Exception as e:                                                                                 │
│   280 │   │   │   message = 'Exception received when sending urllib3 HTTP request'                                   │
│   281 │   │   │   logger.debug(message, exc_info=True)                                                               │
│ > 282 │   │   │   raise HTTPClientError(error=e)                                                                     │
│   283                                                                                                                │
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
HTTPClientError: An HTTP Client raised an unhandled exception: [Errno 24] Too many open files
```

Thanks for the additional details on the HTTPClientError -- that one I think needs to continue to be a hard error as with too many open files I think bedrock client will just keep failing if we retry (it needs the hard reset of the process exiting to free up the file handles). What it is doing this I am not at all sure!