Repository: python-trio/trio

Issue #1104: Should we have a conventional way to "spawn a process into a nursery"?
Created: 2019-06-12T01:18:41Z
We have run_process, which is a convenient way to run a process like it's a regular subroutine, with error checking (check=), cancellation support, automatic waiting, etc. And we have trio.Process, which is a fully general way to spawn a process and then do whatever with it.
In the discussion in #872, @Badg brought up some dissatisfaction with both of these options. I think I mostly get the issue now, so I'll try to rephrase in my own words:
In some ways, when a task spawns a subprocess it's very similar to spawning a new task: the subprocess is a chunk of code that runs simultaneously with the Trio task, and they can potentially interact.
Trio has some strong and convenient conventions for spawning tasks:

trio always makes sure that both tasks are finished before exiting the nursery block
if the new task crashes, the exception propagates until caught, and automatically cancels sibling tasks
if the original task crashes, the new task is automatically cancelled too
if the surrounding context is cancelled, then this automatically cancels both tasks

It would be nice to be able to get this combination of features for subprocesses too. But we don't currently have any simple way to do that.
The closest thing we have is to use async with on a Process object. But that's stuck between two conflicting sets of conventions. The Process API mostly treats processes as independent objects, and our async with process mostly follows the general conventions for async with <resource>: it never cancels the body of the with (even if the process crashes). It doesn't care whether the body of the with block completed with an exception or not. And it doesn't do any particular checks on the subprocess's return code. This is also reasonable and what you'd expect for like, with file_obj. OTOH, the conventions above are specific to how Trio handles code, and really only show up with nurseries and nursery-like objects. Which a Process... kind of is, and kind of isn't.
I don't think it makes sense to try to wedge all those nursery semantics into async with process_obj. In particular, it would be very weird for async with process_obj to suddenly enable the equivalent of run_process's check=True. And then we'd need some way to allow it to be overridden with check=False on a case-by-case basis, and there's no obvious way to spell that. And if we use check=True and the subprocess fails and the body fails, do we raise a MultiError? In multierror v2, does that mean we need to wrap all outgoing exceptions in MultiError? It would involve a lot of weirdness.
Another idea that's been proposed is to extend run_process so that it can be used like:
process_obj = await nursery.start(run_process, ...)
This has several attractive aspects: run_process's job is to "domesticate" a subprocess so it acts like a subroutine – for example, the check= argument translates error-reporting from subprocess-style to subroutine-style. So here we're reusing that, plus a nursery's general ability to take any subroutine and push it off into a parallel task. You get nursery semantics because you used a nursery. Elegant!
It's also kinda weird. It's effectively a totally different 'mode' of using run_process: normally run_process treats the process_obj as hidden, and the return value as important; this flips that around completely, so you can mess with the process_obj, and never see the return value. Normally, the capture_* arguments are a major part of why people use run_process; here, they're completely useless, and would probably need to be forbidden. The use of start is sorta gratuitous: the actual process startup is synchronous, so you could just as well have a synchronous version. The start is just a trick to start a task and get a return value at the same time. (never mind, see #1109)
Are there other options to consider? Brainstorming:
async with trio.open_process(...) as process_obj:  # creates a nursery
    ...

process_obj = nursery.process_start_soon(...)
# or start_process_soon or start_soon_process?

async with trio.open_nursery() as nursery:
    await nursery.start(trio.process_as_task, ...)

async with trio.open_nursery() as nursery:
    process_obj = trio.start_process(nursery, ...)

Duplicate issue #1844: open_process() sometimes "swallows" exceptions
Created: 2020-12-29T15:11:44Z
I had an error in my code:
import trio

async def run():
    async with await trio.open_process('python3 -c "while True: pass"', shell=True):
        print('before')
        1 + ''
        print('after')

trio.run(run)
Running the above felt as if the program just hangs:
$ .venv/bin/python3 swallowed-error.py
before
If I press Ctrl-C, I finally get my TypeError printed out. It looks as if it was buffered before:
$ .venv/bin/python3 swallowed-error.py
before
^CTraceback (most recent call last):
  File "<string>", line 1, in <module>
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/user/src/swallowed-error.py", line 6, in run
    1 + ''
TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:
...
(Of course, I haven't noticed TypeError at first, because the interesting part comes before deep KeyboardInterrupt traceback.)
I would expect Trio not to hide exceptions from me :-)
Trio 0.17.0, MacOS 10.14.6, Python 3.9.0.

Full output, seems to block on `await wait_child_exiting()`
$ .venv/bin/python3 swallowed-error.py
before
^CTraceback (most recent call last):
  File "<string>", line 1, in <module>
KeyboardInterrupt
Traceback (most recent call last):
  File "/home/user/src/swallowed-error.py", line 6, in run
    1 + ''
TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/src/swallowed-error.py", line 9, in <module>
    trio.run(run)
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_core/_run.py", line 1928, in run
    raise runner.main_task_outcome.error
  File "/home/user/src/swallowed-error.py", line 7, in run
    print('after')
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_abc.py", line 261, in __aexit__
    await self.aclose()
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_subprocess.py", line 198, in aclose
    await self.wait()
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_subprocess.py", line 221, in wait
    await wait_child_exiting(self)
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_subprocess_platform/kqueue.py", line 41, in wait_child_exiting
    await _core.wait_kevent(process.pid, select.KQ_FILTER_PROC, abort)
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_core/_generated_io_kqueue.py", line 30, in wait_kevent
    return await GLOBAL_RUN_CONTEXT.runner.io_manager.wait_kevent(ident, filter, abort_func)
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_core/_io_kqueue.py", line 132, in wait_kevent
    return await _core.wait_task_rescheduled(abort)
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_core/_traps.py", line 166, in wait_task_rescheduled
    return (await _async_yield(WaitTaskRescheduled(abort_func))).unwrap()
  File "/home/user/src/.venv/lib/python3.9/site-packages/outcome/_impl.py", line 138, in unwrap
    raise captured_error
  File "/home/user/src/.venv/lib/python3.9/site-packages/trio/_core/_run.py", line 1178, in raise_cancel
    raise KeyboardInterrupt
KeyboardInterrupt

Closing comment (2020-12-29T16:14:45Z): Ok, this seems like a duplicate of #1104 and PR #1568 is supposed to fix it.