bmdf 0.6.2
Execute commands in Markdown files, embed output, generate TOCs
License: MIT
Requires Python: >=3.10

# `bmdf`
**B**ash **m**ark**d**own **f**ences: execute commands in Markdown files, embed output, generate TOCs

e.g.:
<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```
<!-- `python test/print-ci-yml-ref.py mdcmd` -->
<p>

☝️ This block is updated programmatically (by [`mdcmd`]), and verified [in CI](https://github.com/runsascoded/bash-markdown-fence/blob/v0.6.2/.github/workflows/ci.yml#L32-L35).
</p>

[![](https://img.shields.io/pypi/v/bmdf?label=bmdf&color=blue)][bmdf]

<a id="toc"></a>
<!-- toc -->
- [Overview](#overview)
- [Install](#install)
- [`mdcmd`: execute commands in Markdown files, embed output](#mdcmd)
    - [`bmdf` example](#mdcmd-bmdf-example)
    - [HTML example](#mdcmd-html-example)
- [`bmd`: format `bash` command and output as Markdown](#bmd)
    - [`bmdf` (`bmd -f`): command+output mode](#bmdf)
    - [`bmdff` (`bmd -ff`): two-fence mode](#bmdff)
    - [`bmdfff` (`bmd -fff`): &lt;details&gt; mode](#bmdfff)
    - [Piping](#piping)
    - [Env vars](#env-vars)
    - [`-w/--workdir` / `$BMDF_WORKDIR`](#workdir)
- [`mktoc`: Markdown Table of Contents](#mktoc)
- [Examples](#examples)
<!-- /toc -->
<!-- `python test/print-ci-yml-ref.py mktoc` -->
<p>

☝️ This TOC is generated programmatically (by [`mktoc`]), and verified [in CI](https://github.com/runsascoded/bash-markdown-fence/blob/v0.6.2/.github/workflows/ci.yml#L22-L25).
</p>

## Overview <a id="overview"></a>
This package provides 3 CLIs:
- [`mdcmd`]: execute shell commands in Markdown files, embed output
- [`bmd`]: run Bash commands, wrap output for Markdown embedding
  - Useful in conjunction with `mdcmd`
  - [`bmdf`], [`bmdff`], [`bmdfff`] provide different types of "fencing" for command output
- [`mktoc`]: update Markdown table of contents (with custom "id"s for sections)

## Install <a id="install"></a>

Global install via [pipx] or [uv] (recommended):
```bash
pipx install bmdf
# or: uv tool install bmdf
```

You can also install in the current (v)env:
```bash
pip install bmdf
```

## `mdcmd`: execute commands in Markdown files, embed output <a id="mdcmd"></a>

<!-- `bmdfff -- mdcmd --help` -->
<details><summary><code>mdcmd --help</code></summary>

```
Usage: mdcmd [OPTIONS] [PATH] [OUT_PATH]

  Parse a Markdown file, updating blocks preceded by <!-- `[cmd...]` -->
  delimiters.

  If no paths are provided, will look for a README.md, and operate "in-place"
  (same as ``mdcmd -i README.md``).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -C, --no-concurrent             Run commands in sequence (by default, they
                                  are run concurrently)
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --dry-run                   Print the commands that would be run, but
                                  don't execute them
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  -x, --execute TEXT              Only execute commands that match these
                                  regular expressions
  -X, --exclude TEXT              Only execute commands that don't match these
                                  regular expressions
  --help                          Show this message and exit.
```
</details>

```bash
# Modify README.md in-place
mdcmd -i README.md
# Same as above; no args defaults to `-i README.md`
mdcmd
```

That's how the various command examples in this file are generated / updated!

### [`bmdf`] example <a id="mdcmd-bmdf-example"></a>

The example at the top of this file is generated by a line like:

  ```
  <!-- `bmdf seq 3` -->
  ```

`mdcmd` transforms that into:
  ````
  <!-- `bmdf seq 3` -->
  ```bash
  seq 3
  # 1
  # 2
  # 3
  ```
  ````

Notes:
- HTML comments (`<!-- ... -->`) are hidden in rendered markdown, so all the user sees is the output of `bmdf seq 3`
  - [`bmdf`] formats output as a "Bash fence" block
  - [`bmd`] (and variants) are useful for displaying commands (and their output) in Markdown (especially in conjunction with [`mdcmd`]).
- **`mdcmd` is idempotent:**
  - It looks for the block immediately following the `` <!-- `[cmd...]` --> `` line, and replaces that with the output of running `[cmd...]`.
  - If there's already output there, it will be replaced with new/current output.

### HTML example <a id="mdcmd-html-example"></a>
Scripts that output raw HTML also work, e.g. [print-table.py](test/print-table.py) generates this table:

<!-- `python test/print-table.py` -->
<table>
  <tr>
    <th>header 1</th>
    <th>header 2</th>
  </tr>
  <tr>
    <td>cell 1</td>
    <td>cell 2</td>
  </tr>
</table>

That table is generated by a line like:

  ```
  <!-- `python test/print-table.py` -->
  ```

[`mdcmd`] maintains an output block immediately after it:

  ````
  <!-- `python test/print-table.py` -->
  <table>
    <tr>
      <th>header 1</th>
      <th>header 2</th>
    </tr>
    <tr>
      <td>cell 1</td>
      <td>cell 2</td>
    </tr>
  </table>
  ````

## `bmd`: format `bash` command and output as Markdown <a id="bmd"></a>

<!-- `bmdfff -- bmd --help` -->
<details><summary><code>bmd --help</code></summary>

```
Usage: bmd [OPTIONS] COMMAND...

  Format a command and its output to markdown, either in a `bash`-fence or
  <details> block, and copy it to the clipboard.

Options:
  -A, --strip-ansi                Strip ANSI escape sequences from output
  -C, --no-copy                   Disable copying output to clipboard
                                  (normally uses first available executable
                                  from ['pbcopy', 'xclip', 'clip']
  -e, --error-fmt TEXT            If the wrapped command exits non-zero,
                                  append a line of output formatted with this
                                  string. One "%d" placeholder may be used,
                                  for the returncode. Defaults to
                                  $BMDF_ERR_FMT
  -E, --env TEXT                  k=v env vars to set, for the wrapped command
  -f, --fence                     Pass 0-3x to configure output style: 0x:
                                  print output lines, prepended by "# "; 1x:
                                  print a "```bash" fence block including the
                                  <command> and commented output lines; 2x:
                                  print a bash-fenced command followed by
                                  plain-fenced output lines; 3x: print a
                                  <details/> block, with command <summary/>
                                  and collapsed output lines in a plain fence.
  -i, --include-stderr / -I, --no-include-stderr
                                  Capture and interleave both stdout and
                                  stderr streams; falls back to
                                  $BMDF_INCLUDE_STDERR
  -s, --shell / -S, --no-shell    Disable "shell" mode for the command; falls
                                  back to $BMDF_SHELL, but defaults to True if
                                  neither is set
  -t, --fence-type TEXT           When -f/--fence is 2 or 3, this customizes
                                  the fence syntax type that the output is
                                  wrapped in
  -u, --expanduser / -U, --no-expanduser
                                  Pass commands through `os.path.expanduser`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDUSER
  -v, --expandvars / -V, --no-expandvars
                                  Pass commands through `os.path.expandvars`
                                  before `subprocess`; falls back to
                                  $BMDF_EXPANDVARS
  -w, --workdir TEXT              `cd` to this directory before executing
                                  (falls back to $BMDF_WORKDIR
  -x, --executable TEXT           `shell_executable` to pass to Popen
                                  pipelines (default: $SHELL)
  --help                          Show this message and exit.
```
</details>

`bmd` (and aliases [`bmdf`], [`bmdff`], [`bmdfff`]) takes a `bash` command as input, and renders the command and/or its output in various Markdown-friendly formats:

### `bmdf` (`bmd -f`): command+output mode <a id="bmdf"></a>

Suppose you want to embed a command and its output in a README.md, like this:

<!-- `bmdf seq 3` -->
```bash
seq 3
# 1
# 2
# 3
```

(Note how the command is `bash`-highlighted, and output lines are rendered as comments)

Put a placeholder like this in your README.md:
  ````
  <!-- `bmdf seq 3` -->
  ````

then [run `mdcmd`][`mdcmd`] to update your README containing this embedded command block.

### `bmdff` (`bmd -ff`): two-fence mode <a id="bmdff"></a>
`bmdff` (alias for `bmd -ff`) renders two code fences, one with the Bash command (syntax-highlighted appropriately), and a second (non-highlighted) block with the output, e.g.:

  ````
  <!-- `bmdff seq 5` -->
  ````

becomes:

<!-- `bmdff seq 5` -->
```bash
seq 5
```
```
1
2
3
4
5
```

### `bmdfff` (`bmd -fff`): &lt;details&gt; mode <a id="bmdfff"></a>

When a command's output is large, rendering it as a `<details><summary>` (with the output collapsed, by default) may be preferable.

`bmdfff` (3 `f`s, alias for `bmd -fff`) transforms placeholders like this:

  ````
  <!-- `bmdfff seq 10` -->
  ````

to:

<!-- `bmdfff seq 10` -->
<details><summary><code>seq 10</code></summary>

```
1
2
3
4
5
6
7
8
9
10
```
</details>

### Piping <a id="piping"></a>
Piping works too, e.g.:

  ````
  <!-- `bmdf -- seq 10 | wc -l` -->
  ````

will become:

<!-- `bmdf -- seq 10 | wc -l` -->
```bash
seq 10 | wc -l
# 10
```

(the `--` is needed so that that `-l` isn't parsed as an opt to `bmdf`)

### Env vars <a id="env-vars"></a>
By default, `shell=True` is passed to `subprocess` calls (but can be disabled via `-S`).

This means env vars are expanded; they can also be set via `-E`, e.g.:

  ````
  <!-- `bmdf -E FOO=bar echo $FOO` -->
  ````

yields:
<!-- `bmdf -E FOO=bar echo $FOO` -->
```bash
FOO=bar echo '$FOO'
# bar
```

<details>
<summary>
More examples of quoting/splitting behavior
</summary>

Quoting `"$FOO"`:
  ````
  <!-- `bmdf -E FOO=bar echo "$FOO"` -->
  ````

yields:
<!-- `bmdf -E FOO=bar echo "$FOO"` -->
```bash
FOO=bar echo '$FOO'
# bar
```

Arg with spaces:
  ````
  <!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->
  ````
yields:
<!-- `bmdf -E FOO=bar echo "FOO: $FOO"` -->
```bash
FOO=bar echo 'FOO: $FOO'
# FOO: bar
```

Escaping `$`:
  ````
  <!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->
  ````
yields:
<!-- `bmdf -E FOO=bar echo "\$FOO=$FOO"` -->
```bash
FOO=bar echo '\$FOO=$FOO'
# $FOO=bar
```

</details>

### `-w/--workdir` / `$BMDF_WORKDIR` <a id="workdir"></a>

By default, `bmdf` runs in the current working directory. This can be overridden with `-w`:

  ````
  <!-- `bmdf -w .github ls` -->
  ````

<!-- `bmdf -w .github ls` -->
```bash
ls
# workflows
```

## `mktoc`: Markdown Table of Contents <a id="mktoc"></a>

<!-- `bmdfff -- mktoc --help` -->
<details><summary><code>mktoc --help</code></summary>

```
Usage: mktoc [OPTIONS] [PATH] [OUT_PATH]

  Insert a table of contents (TOC) in a markdown file.

  Looks for a pair of sentinel lines to insert or update the TOC between:
  ``<!-- toc -->``, ``<!-- /toc -->``.

  If an empty line follows the opening ``<!-- toc -->`` line, the TOC will be
  inserted there (along with the closing sentinel); this is useful when
  initially generating a TOC.

  If no ``out_path`` is provided, will operate "in-place" on ``README.md`` (as
  if ``mktoc -i README.md`` was passed).

Options:
  -a, --amend                     Squash changes onto the previous Git commit;
                                  suitable for use with `git rebase -x`
  -i, --inplace / -I, --no-inplace
                                  Edit the file in-place
  -n, --indent-size INTEGER       Indent size (spaces)
  -T, --no-cwd-tmpdir             In in-place mode, use a system temporary-
                                  directory (instead of the current workdir,
                                  which is the default)
  --help                          Show this message and exit.
```
</details>

Put a block like this in your README.md:
````
<!-- toc -->
<!-- /toc -->
````

Then put empty `<a>` tags next to the headings you want to include, e.g.:

 ```markdown
 ## My section heading <a id="my-section"></a>
 ```

(This allows for customizing and shortening the `id`s, as well as skipping sections)

Then run:
```bash
# Modify README.md in-place
mktoc -i README.md
# Same as above; no args defaults to `-i README.md`
mktoc
```

And the `<!-- toc  -->` section will have a table of contents injected (like the one at the top of this file).

## Examples <a id="examples"></a>
The examples in this file are rendered using [`bmdf`] and [`mdcmd`], and [the TOC](#toc) is rendered using [`mktoc`]. Both are verified by the [`ci.yml`] GitHub Action.

These repos' READMEs also use [`bmdf`] / [`mdcmd`] / [`mktoc`] to execute/verify examples commands (and in some cases are also verified by GitHub Actions):

- [runsascoded/juq]
- [runsascoded/utz]
- [runsascoded/dvc-utils]
- [ryan-williams/dvc-helpers] ([GHA][dvc-helpers GHA])
- [ryan-williams/git-helpers]
- [ryan-williams/parquet-helpers] ([GHA][pqt-helpers GHA])
- [ryan-williams/tdbs-dask]
- [TileDB-Inc/scverse-ml-workshop-2024]

[`ci.yml`]: .github/workflows/ci.yml
[`bmd`]: #bmd
[`bmdf`]: #bmdf
[`bmdff`]: #bmdff
[`bmdfff`]: #bmdfff
[`mdcmd`]: #mdcmd
[`mktoc`]: #mktoc

[runsascoded/utz]: https://github.com/runsascoded/utz?tab=readme-ov-file#utz
[TileDB-Inc/scverse-ml-workshop-2024]: https://github.com/TileDB-Inc/scverse-ml-workshop-2024?tab=readme-ov-file#training-models-on-atlas-scale-single-cell-datasets
[ryan-williams/tdbs-dask]: https://github.com/ryan-williams/tdbs-dask?tab=readme-ov-file#tdbs-dask
[ryan-williams/dvc-helpers]: https://github.com/ryan-williams/dvc-helpers?tab=readme-ov-file#dvc-helpers
[dvc-helpers GHA]: https://github.com/ryan-williams/dvc-helpers/actions
[ryan-williams/git-helpers]: https://github.com/ryan-williams/git-helpers?tab=readme-ov-file#git-helpers
[ryan-williams/parquet-helpers]: https://github.com/ryan-williams/parquet-helpers
[pqt-helpers GHA]: https://github.com/ryan-williams/parquet-helpers/actions/
[runsascoded/dvc-utils]: https://github.com/runsascoded/dvc-utils
[runsascoded/juq]: https://github.com/runsascoded/juq

[bmdf]: https://pypi.org/project/bmdf/
[pipx]: https://pipx.pypa.io/stable/
[uv]: https://docs.astral.sh/uv/