devcluster 1.3.2
Developer tool for running the Determined cluster
License: Apache License 2.0
Requires Python: >=3.6

# `devcluster`

A developer tool for running the Determined cluster.

## Configuring and Running

You can run the code from the git root by running `python -m devcluster`, or
install it into your python environment with `pip install -e .` and then just
call `devcluster` by itself.

The first time you run it, it will give you the option to automatically install
the default configuration file (see `devcluster/example.yaml`).

By default, devcluster needs to run from the root of a `determined` source
tree.  You can do this in one of a few ways:
- always run `devcluster` from the root of your `determined` source tree
- set `cwd: /path/to/determined` to run `devcluster` from anywhere
- run `devcluster` with the `-C /path/to/determined` option

You can either specify a config file via the `--config` or `-c` option, or
`devcluster` will try to read the default config file at
`~/.devcluster.yaml`.

## How `devcluster` Works

devcluster runs a cluster with a linear dependency graph of stages.  With the
default config (`devcluster/example.yaml`), those stages are:

- `DEAD`
- `DB`
- `MASTER`
- `AGENT`

Because of the linear dependency graph, the `DB` stage can run by itself, the
`MASTER` stage is only started after `DB` is already up, and the `AGENT` is only
started after the `MASTER` is up.  The `DEAD` stage indicates that nothing
is running.  The non-`DEAD` stages correspond to the `stages` setting in your
config file.

The status bar has two rows.  The first row is the state row.  It shows all of
the stages in your dependency graph.  All active stages are highlighted orange.
There is also a 'target state', which is indicated by a less-than character
(`<`).  You set the target state by pressing the key corresponding to a
particular stage.

When the target state is modified, devcluster will walk up or down the stage
list, starting or killing processes as needed to reach the target stage.  In
the default configuration, the MASTER and AGENT stages are rebuilt each time
they are started.

A common pattern during development might be to:
- make a change to the master code
- press `1` to set the target state to `DB` (killing the `MASTER` and the
  `AGENT`)
- press `3` to set the target state back to `AGENT` (rebuilding/restarting the
  `MASTER` and `AGENT` on the way).

The second row in the status bar is the logging row.  Every stage collects a
separate stream of logs, and each stream can be toggled by pressing the
corresponding keys.

## Keybindings

- `1`-`3` set target states for the cluster (as shown in the status bar)
- `!`-`#` toggle logs for the corresponding stage (as shown in the status bar)
- `q` or `ctrl`+`c` once to quit, or twice to force-quit (which may leave
  dangling processes or docker containers laying around)
- `u`/`d` will scroll up/down, and `U`/`D` will scroll up/down faster
  Scrolling support is only partially line-aware; it will scroll by
  literal log chunks, which may each be more or less than a line
- `x` will reset scrolling to continue following logs
- Spacebar will emit a visual separator with a timestamp to the console stream.
- With the default configuration, the keys `p`, `w`, and `c` will trigger
  building the python harness, webui, and docs, respectively.

## Server/Client Mode

You can run `devcluster` as a headless server with various configurations for
listening for clients:

    # listen on 0.0.0.0:1234
    devcluster --quiet --listen 1234

    # listen on localhost:1234
    devcluster -q -l 127.0.0.1:1234

    # listen on a unix socket
    devcluster -q -l /path/to/socket

    # listen on the default unix socket (/tmp/devcluster/sock)
    devcluster -q

    # configure multiple listeners
    devcluster -q -l 1234 /path/to/socket

Sending a `SIGINT` (via `ctrl`+`c`) or a `SIGTERM` (via `kill`) to the headless
server will close it.

You can then connect the UI to a headless server:

    devcluster 1234
    devcluster host:1234
    devcluster /path/to/socket

In client mode, pressing `q` or `ctrl`+`c` will only close the UI; it will not
affect the server.

## Oneshot Mode

You can run `devcluster` in a special oneshot mode (`-1`/`--oneshot`), suitable
for automated usage.  In automated mode, the `stdout` from `devcluster` will be
suitable for redirecting to a file, and the `stderr` will emit the line:

    devcluster is up

to indicate when the cluster is ready for use.

## Philosophy

`devcluster` tries as much as possible to be a standalone tool for running
arbitrary versions of the Determined cluster.  It is tested as far back as
v0.12.3.

Some defaults are kept up-to-date with tip-of-main to simplify the config
file for the most-common use case, but in general preference was given to being
configurable and explicit.

## TODO

- config file reloading