Repository: fabric/fabric

Issue #237: Remote Python execution hooks
Created: 2011-08-19T01:59:23Z
Description
Just a placeholder that eventually, if people need the feature and it's easy enough to integrate / makes sense to integrate instead of just saying "do it yourself, it's just Python" -- we should consider making it easy to leverage tools for remote Python execution/RPC. Modern-ly useful candidates include:

execnet

actively developed but may only support Python up to 3.4?
transport is popen (local), ssh (via binary 😒) or direct socket (with bootstrappy remote script/server in-repo)
serialization is completely handrolled, using struct
used for distributed pytest so is at least somewhat battle tested
API relatively sane (gateway and channel objects, send/receive, light async available) if lacking explicit modern-style API docs (seems to be ONLY example/prose driven?)


Pyro(4)

actively developed, seems to support at least up to 3.6 if not also 3.7
serializes with an AST-based standalone library called serpent (which explicitly states in its README why it exists and why Pyro is not using XML, JSON, Pickle etc)
transports with TCP/IP, Unix sockets, and can optionally use SSL/TLS
extremely featureful, including things like raising remote exceptions on the caller's end, handling reconnects automatically, etc
nice clean API (understandably, at a high level, similar to execnet's re: objects representing the remote interpreter, if slightly more modern feeling) including option to use decorators to declare RPC calls


Mitogen

actively developed (maintainer is in comments below), supports up to Python 3.6 at least
serializes w/ pickle, unfortunately, though there is rationale
transport: actually not 100% sure but feels like direct socket connections combined with each Mitogen child being capable of routing requests onwards in a topology?
Relatively featureful and the features it does have are all very powerful-sounding

can run even if remote end has no free disk space, etc - neat!
Fancy caller-side module forwarding; not sure if/how the other options even handle non-stdlib imports.
remote-end logging module calls automatically forwarded back to caller
less immediately relevant to the average Fab single-node/list-of-sibling-nodes use case: some very neat emphases on trees of child processes, headless trees, and other topology strategies


Bootstraps automatically as long as remote end has SSH and Python, and does not require transferring a file first (compared to e.g. execnet requiring use of a remote script)
API somewhat more complex than eg Pyro/execnet (about 1 more layer of objects) but not necessarily in a bad way given its additional features/layout


IPython's kernel side

actively maintained (is it ever!)
uses ZMQ for transport between some client and a jupyter/ipython kernel process
serializes with JSON, alternately msgpack/pickle



Options that may not be feasible but are worth tracking:

Pushy (link goes to blog entry specifically discussing integrating Fabric with it) - unsupported/archived since 2016, though per below, the author earlier expressed collaboration interest
Salt - remote (pun not intended) chance one could try reusing salt's ZMQ-driven remote execution bits, but at a very quick glance they do not appear to be factored out in a reusable fashion or anything
Celery - similar to salt re: high level pub/sub/message-passing arch and need to try extracting a lower level flow, except requires a broker/queue system (vs ZMQ which can do direct socket to socket style connections) so even less feasible?
etc (Python wiki list of projects)  - last scanned early 2019, all obvious useful candidates already pulled out


Offhand I think the main draw would be to hook up your Fabric host/user/etc settings into such a tool. So probably something more along the lines of our Django contrib, instead of a core integration (especially since many users may not have Python remotely, so we absolutely would not want to require use of such a tool).

The author of Pushy also mentions the possibility of using his tool in Fabric as a potential layer on top of Paramiko; I guess the same approach could be applied to the other tools as well.
Not entirely sure if that would be a good way to go; plus is an eventual decoupling from SSH specifically; minus is that, well, we're pretty SSH specific in a lot of ways (and use a lot of core Paramiko stuff) and I'm not positive it would be worth the loss of focus.

Originally submitted by Jeff Forcier (bitprophet) on 2010-09-20 at 11:43am EDT
Relations

Duplicated by #246: Ability to easily remotely execute small Python snippets (rather than shell commands)

Duplicate issue #246: Ability to easily remotely execute small Python snippets (rather than shell commands)
Created: 2011-08-19T01:59:28Z
Description
Example usage:
install_dir = run_python("os.path.join('var', 'www')")
(although maybe with a better name than run_python).
Motivation: Sometimes it is much easier to write a bit of Python code to be executed remotely than a shell command that does the same thing. Also, let's you stay in the Python mindset, worry less about the details of the remote shell, etc.
Why it should be in Fabric:
(1) The search for the remote Python executable is not that hard, but having to look potentially in a couple of places makes it nice to have in a library.
(2) Trying to write it as "python -c ..." is a bummer, because there's shell escaping, quotation marks to look out for, etc. Better to use stdin, but the current run() doesn't make that easy.

Originally submitted by Josh Snyder (josharian) on 2010-10-28 at 08:59am EDT
Relations

Duplicates #237: Remote Python execution hooks


Closed as Duplicate on 2010-10-28 at 11:10am EDT

Closing comment (2011-08-19T02:08:36Z): Jeff Forcier (bitprophet) posted:

This is basically a dupe of #237, insofar as it's requesting functionality to make using a remote Python interpreter easier. Your request is slightly different insofar as I think you're asking for Fabric to do the heavy lifting re: quoting/escaping, determining location of remote interpreter, etc -- but it's the same overall effect as using execnet et al.
Minor clarification -- how does run make leveraging stdin difficult? Outside of the quoting problem, it's certainly possible to do run('echo "import sys; print sys.path" | python'). If you meant interactive stdin, see my response to your other ticket :)

on 2010-10-28 at 09:55am EDT