Repository: UKGovernmentBEIS/inspect_ai
Issue: ensure that timed out docker exec() calls terminate processes in the container
Root cause: Resource Mishandling

Related discussion below

_Ted_
I think there's an issue with timeouts for the docker sandbox exec — setting a timeout doesn't always seem to lead to the process inside the container dying at the requested time. If you have a rambunctious agent that e.g. likes to write code with infinite loops, this can lead to resource problems! It exhibits as the inspect sample moving forwards with further messages, as if the exec were finished, while actually the process inside the container is still going merrily along. You can end up stacking up multiple running processes in there at the same time.
I'm working around it by only ever exec'ing commands which make use of the linux timeout command inside of the container, which seems reliable, but in case helpful I think the issue is as follows (partly speculation, haven't confirmed):
For reference, here's the inspect exec timeout handling code:

```python
            except asyncio.exceptions.TimeoutError:
                # terminate timed out process -- try for graceful termination
                # then be more forceful if requied
                try:
                    proc.terminate()
                    await asyncio.sleep(2)
                    if proc.returncode is None:
                        proc.kill()
                except Exception as ex:
                    logger.warning(
                        f"Unexpected error terminating timed out process '{args}': {ex}"
                    )

                # raise standard Python TimeoutError
                raise TimeoutError
````

The proc is a handle to the local (outside-container) docker exec client process. From chatting with LLMs, my understanding of is that when you call proc.terminate() it (a) kills the outside-container exec process and (b) maybe gives the process inside the container a SIGHUP signal because its controlling terminal went away (this signal might be ignored, depending on the behavior of the in-container process)
If that's right, then you can end up with a dead docker exec client process, but a living in-container process. At that point you're kind of in trouble — there's now a rogue container process you don't have a handle to. Subsequently proc.returncode makes clear that the docker exec client process is indeed dead, and so proc.kill() is likely never run.
If this dx is correct (again, haven't verified), a couple of possible solutions:
- Set a timeout on the process inside the container using e.g. the linux timeout command.
- Track the in-container process from inspect — possibly by finding a way to assign it a unique id at its inception. Then can later look up the pid and hard-kill it if necessary using a new docker exec kill -SIGKILL.

Related docker issue: https://github.com/moby/moby/issues/9098 (edited) 

_Craig Walton_
These are good observations! I don't have anything to add on the Docker implementation and haven't verified this behaviour myself, just noting that for the [K8s sandbox](https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox/blob/38de8871296684532bdcc4c2e7007cc749430e59/src/k8s_sandbox/_pod/execute.py#L90), we use your suggestion 1 of a timeout command running in the container which wraps the user command.
It is also worth bearing mind that even with a timeout command or some process tracking followed by a kill -SIGKILL, processes spawned by the agent command can continue to run. For example, an agent issuing sleep infinity & to the bash tool with a timeout of 10s will result in the sleep running forever, because the bash process exits quickly so the timeout command has nothing to do. (The sleep is orphaned.)
It might also depend on how one interprets the (currently loosely defined) timeout parameter to SandboxEnvironment.exec() - it doesn't specify what will happen to the process (or any subprocesses spawned) upon timeout.

_JJ Allaire_
We could change the implementation of exec to run commands under  timeout  -- one issue here is that we actually need to know for certain when a timeout occurs (as opposed to another error) so we can retry it (needed to work around some docker daemon bugs we discovered). I guess we could do both and just set the command timeout a bit longer merely as an insurance policy.

_Craig Walton_
That could work! For another reference point, in k8s, we interpreted an exit code of 124 to imply a TimeoutError should be raised ([source](https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox/blob/38de8871296684532bdcc4c2e7007cc749430e59/src/k8s_sandbox/_pod/execute.py#L135C9-L137C80)). Though this does have limitations when the timeout command is stripped down to a non-standard one in some busybox images https://github.com/UKGovernmentBEIS/inspect_k8s_sandbox/issues/15

_Ted_
Agree with JJ that if you're seeing docker unreliability, might need both timeouts so you can retry appropriately.
Also: come on, Docker!
Agree with you Craig about detached processes not being caught by this
timeout by default sends SIGTERM, which if succeeds results in exit code 124. However timeout also has a -k parameter which can be used to send SIGKILL after an additional grace period (a signal which the OS guarantees cannot be ignored by the recipient). If this happens, timeout will have exit code 137.  Technically this 137 is ambiguous between (a) SIGKILL from timeout, and (b) SIGKILL for other reasons, e.g. the OS killing the command being run due to OOM.
This is a limitation of the timeout command (it could've been implemented differently). Not sure how much it matters.

Another approach here in general would be using a shell wrapper script rather than using timeout . Said script would (1) run the user command in a detached process, and capture its pid, then (2) sleep for a time according to the timeout arg, (3) attempt to SIGTERM the user process if it's still around, (4) sleep for a short grace period, (5) SIGKILL the user process if it's still around.
This would save you from platform differences w.r.t timeout, and would mean you have full control of the exit code, i.e. can always raise 124 for timeouts, even when you had to SIGKILL the user process. A disadvantage is that it'd be less accurate w.r.t timing, maybe especially under load (not sure). In a brief test I saw it taking a few extra seconds to kill user processes. Aside from timing accuracy issues, I don't know whether it'd be significantly less reliable than timeout. (It's certainly not more reliable... but in practice may work well enough. sh and sleep and kill are institutions tested by time) (edited)

Possibly fixed by https://github.com/UKGovernmentBEIS/inspect_ai/pull/1356

My belief is that https://github.com/UKGovernmentBEIS/inspect_ai/pull/1356 likely fixed this (as the symptoms described here exactly mirror what I'd expect if we didn't kill the process on timeout). Please feel free to open a new issue if you find that this problem still occurs.