Repository: UKGovernmentBEIS/inspect_ai
Issue: Negative working time can be reported if there is concurrency contention within a sample
Root cause: Algorithmic Error

Inspect's `concurrency()` context manager reports sample waiting time when it [acquires the semaphore](https://github.com/UKGovernmentBEIS/inspect_ai/blob/main/src/inspect_ai/util/_concurrency.py#L107). However, the waiting time is reported for the running sample by every task in that sample that is blocked on acquiring the semaphore, which means that if multiple tasks within a sample are simultaneously blocked waiting for the same shared resource, they may report overlapping/duplicated waiting time when acquiring the semaphore, meaning that the working time figure is no longer correct and can end up going negative in extreme cases.

A real-world example of where this might occur is when we make [multiple simultaneous calls to `generate()`](https://github.com/METR/triframe_inspect/blob/0eacfc42f1ca600c66b42631ed6f9b68fd2774d7/triframe_inspect/generation.py#L45-L57) within our triframe agent to emulate the `n` parameter of the OpenAI chat completion API in the Anthropic and OpenAI responses APIs.

Minimal example to reproduce:

```python
import asyncio
import time

from inspect_ai import Task, eval
from inspect_ai.dataset import Sample
from inspect_ai.log import transcript
from inspect_ai.model import (
    ChatMessage,
    GenerateConfig,
    Model,
    ModelAPI,
    ModelOutput,
    get_model,
    modelapi,
)
from inspect_ai.solver import Generate, Solver, TaskState, solver
from inspect_ai.tool import ToolChoice, ToolInfo
from inspect_ai.util import sample_limits

GLOBAL_START_TIME = time.monotonic()


@modelapi(name="slow-api")
class SlowAPI(ModelAPI):
    def __init__(self, model_name: str = "slow-api", **kwargs):
        super().__init__(model_name)
        self.start_time = time.monotonic()

    async def generate(
        self,
        input: list[ChatMessage],
        tools: list[ToolInfo],
        tool_choice: ToolChoice,
        config: GenerateConfig,
    ) -> ModelOutput:
        sample_input = input[0].text if input else "unknown"

        elapsed = time.monotonic() - GLOBAL_START_TIME
        transcript().info(f"[{elapsed:.2f}s] Starting generate for: {sample_input}")
        print(f"[{elapsed:.2f}s] Starting generate for: {sample_input}")

        # Simulate a slow API call
        await asyncio.sleep(5)

        elapsed = time.monotonic() - GLOBAL_START_TIME
        transcript().info(f"[{elapsed:.2f}s] Completed generate for: {sample_input}")
        print(f"[{elapsed:.2f}s] Completed generate for: {sample_input}")
        print(
            f"  --> and working usage is now {sample_limits().working.usage:.3f} seconds"
        )

        return ModelOutput.from_content(
            model=self.model_name,
            content=f"response to '{sample_input}'",
        )


@solver
def parallel_generate(n: int = 5) -> Solver:
    async def solve(state: TaskState, generate: Generate) -> TaskState:
        model = get_model()

        async def do_generate(i: int) -> ModelOutput:
            elapsed = time.monotonic() - GLOBAL_START_TIME
            print(f"[{elapsed:.2f}s] Solver launching generate call {i + 1}/{n}")
            return await model.generate(state.messages)

        results = await asyncio.gather(*[do_generate(i) for i in range(n)])

        state.output = results[-1]
        return state

    return solve


samples = [
    Sample(input=f"request {i + 1}", target=f"response to 'request {i + 1}'")
    for i in range(2)
]

result = eval(
    tasks=Task(dataset=samples, solver=parallel_generate(n=5)),
    model=Model(
        api=SlowAPI(),
        config=GenerateConfig(max_connections=2),
    ),
    max_samples=2,
)
```

Here's a real-world example of that - you can see in this eval file that the total working time, and the reported working start of most of the events, are negative:

[2025-10-27T08-33-52+00-00_debug-small-libs_hwZdyn8wdBu47zQWEt5nTj.eval.zip](https://github.com/user-attachments/files/24101943/2025-10-27T08-33-52%2B00-00_debug-small-libs_hwZdyn8wdBu47zQWEt5nTj.eval.zip)

I think I can take a look at improving the working time counts tomorrow or over the weekend. Also note we updated the grok provider to not use internal retries so once we resolve this you should be able to get accurate numbers for grok either with our without the native provider.

@pipmc here's a fix that should resolve this: https://github.com/UKGovernmentBEIS/inspect_ai/pull/2930

Would also appreciate a review on this one.