Repository: UKGovernmentBEIS/inspect_ai
Issue: [Feature request] Flag to disable all API calls
Root cause: Unimplemented Feature Gap

I want to make sure my test suite behaves the same locally and in CI. 

Locally I have environment variables set for my API keys (e.g. OPENAI_API_KEY) and so if any tests were to call an OpenAI model they would be able to succeed.

In CI environments we try not to call models and there are no API keys set.

This means that I might implement a test which passes locally and only find out that it's calling a model API in CI, which hurts the dev feedback loop.

The way we solved this in our internal framework is to have a `--disable-api-calls` flag which always prevents model APIs from being called. This flag could be set in individual tests or globally by pytest via an environment variable. It worked quite nicely.

Just to clarify my understanding, this is different from the `--runapi` flag you introduced in that its a failsafe guard that we'd implement in the model interface to make sure that no API code was actually every run? Maybe for consistency then we call it `--disableapi`?

Implementation would then be:

- Introduce `--disableapi` as a Pytest CLI flag that sets an envrionment variable
- Main model dispatching code checks this environment variable and throws if its set (other than if mockllm is in use)

Let me know if this captures it....


> Just to clarify my understanding, this is different from the --runapi flag you introduced in that its a failsafe guard that we'd implement in the model interface to make sure that no API code was actually every run? Maybe for consistency then we call it --disableapi?

Yes! I'd forgotten about runapi. But the behaviour sounds correct.

One clarification point: I want to be able to use this as an end user, not inspect developer. So I'm not sure if a pytest flag works here. I had imagined it would be a parameter to `eval` (and possibly `task`, `eval-set`, `eval-retry`, etc)

But yes, the core idea is that I run my tasks as usual, but I set the disable API flag, and then it just blows up saying "API Calls disabled" if it ever gets to a point that it'd call any LLM except for mockllm

You are correct that my main use case is in tests, so if you wanted to find a way to expose this to users for testing only then I think that'd work

Okay, got it. I'd prefer not to clutter up eval/eval-set with this since its so particular. So I'd say we should go with one of two approaches:

(1) Environment variable that could apply to both user and test scenarios
(2) CLI flag for test scenarios (or a combination of the above and this)

LMK which of these you think is the most fitting.


Thank you! I think an environment variable would be great. CLI flag would be good as a bonus if it's easy, but probably not necessary.

resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/794/

Thanks for implementing this JJ! 

I think unfortunately it seems to be a bit too high up. When I set the variable it causes some of our tests to fail despite the fact they do not in fact call models. I think it's because the check has been added at the level of instantiating the Model API rather than the point where the model call is actually made.  Not 100% sure.

When I have some time I will attempt a more specific repro for you

I can absolutely move this down to generate Let me take a crack at that.

Here is a PR for this: https://github.com/UKGovernmentBEIS/inspect_ai/pull/811

Note that because we are checking in `generate()` rather than `get_model()` the result is a failed task (i.e status == "error") rather than an exception blowing out the top of the stack. This is because `generate()` is called within task execution, which doesn't treat errors as fatal but rather a failed task with a log file including completed samples, error message, etc. If you want errors to blow out the top of stack you can set `debug_errors=True` (typically used so that debuggers can break on exceptions, but would work to cause a hard error in this case as well).

Thanks for the heads up! Can I also use `fail_on_error=True`? We set that in tests by default anyway so I think it should be just fine

No, `fail_on_error=True` is already the default (stop the task if a sample has an error). `fail_on_error=False` would actually push the error down to the sample level (allowing the eval to have "success" even in the case of a sample failing).

I think though that this should still work well with any test that is looking at the eval log to verify success (i.e. checking for status == "success", checking for samples, asserting there is no error, etc.). If all the test does is just succeed if there is no exception then I think it should be enhanced to do a minimal check for success.


Thanks for the clarification. Very helpful! 

Resolved with https://github.com/UKGovernmentBEIS/inspect_ai/pull/811

Sorry, another request for this one! Currently we hit the exception even if the response is cached, but what I had in mind was that cached model calls would go through and we would only raise if the model API actually gets called. That way we can still run our tests which use the cache. 

So i suppose we would move it [right after we know it's a cache miss / right before the actual API call](https://github.com/UKGovernmentBEIS/inspect_ai/blob/c70f53945bf8aec0a028a977ee6f2a53e7c148e0/src/inspect_ai/model/_model.py#L348)?

Could we update it again? Apologies for not being clear enough. 

https://github.com/UKGovernmentBEIS/inspect_ai/pull/843