Experimentations on Bazel: genrule
Tags: bazel
Published: 2021-03-21 19:37:24

Bazel allows several ways to define [rules] in a workspace. In this article, we are focusing on [`genrule`], the [easiest way](https://docs.bazel.build/versions/master/skylark/tutorial-creating-a-macro.html) to create a custom rule (with some limitations).

```python
# full signature of genrule
genrule(name, srcs, outs, cmd, cmd_bash, cmd_bat, cmd_ps, compatible_with, deprecation, distribs, exec_compatible_with, exec_properties, exec_tools, executable, features, licenses, local, message, output_licenses, output_to_bindir, restricted_to, tags, target_compatible_with, testonly, toolchains, tools, visibility)
```

The goal of a rule is to produce `outs` as form of one or several files. So the minimal arguments to set are :

- [`name`] used to identified the `outs` then you can refer them as a group into input for other `rule` or what to build.
- [`outs`] a non empty list of files generated by the rule
- the command to generate the [`outs`], via at least of [`cmd`], [`cmd_bash`](https://docs.bazel.build/versions/master/be/general.html#genrule.cmd_bash), [`cmd_bat`](https://docs.bazel.build/versions/master/be/general.html#genrule.cmd_bat), [`cmd_ps`](https://docs.bazel.build/versions/master/be/general.html#genrule.cmd_ps)

The following example create a file `hello20s.txt` after 20s. When [`outs`] is a list of one file, then `$@` can be used as a shortcut to identify the output file.

## Go back to hello20s

Let's start with the rule `hello20s` defined in previous article.

```python
# BUILD.bazel
genrule(
    name = "hello20s",
    outs = ["hello20s.txt"],
    cmd_bash = "sleep 20 && echo 'hello20s' >$@",
)
```

```shell
❯ bazel build //:hello20s
INFO: Analyzed target //:hello20s (5 packages loaded, 7 targets configured).
INFO: Found 1 target...
Target //:hello20s up-to-date:
  bazel-bin/hello20s.txt
INFO: Elapsed time: 20.334s, Critical Path: 20.02s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
```

The target was run but the output was not as expected in the same as folder `BUILD.bazel`, it is at `bazel-bin\hello20s.txt`, because bazel keeps separated what is generated and what is not.

Also note, that instead of requesting to build every rules, we request to build `hello20s`. There are several [syntaxes for the target](https://docs.bazel.build/versions/master/guide.html#target-patterns), in this case `//<package_name>:<name>` is a full qualified target name (start by `//`)   with `package_name` empty because `BUILD.bazel` is at the root of the workspace.

What happens if we moved this to a sub-level package named `exp_genrule` ?

```sh
# create a package is made by creating a folder with a BUILD file
mkdir exp_genrule
mv BUILD.bazel exp_genrule
```

```sh
❯ bazel build //:hello20s
ERROR: Skipping '//:hello20s': no such package '': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/david/src/github.com/davidB/sandbox_bazel
WARNING: Target pattern parsing failed.
ERROR: no such package '': BUILD file not found in any of the following directories. Add a BUILD file to a directory to mark it as a package.
 - /home/david/src/github.com/davidB/sandbox_bazel
INFO: Elapsed time: 0.030s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)
```

It failed as expected, because the full qualified name need to include the new package'name.

```sh
❯ bazel build //exp_genrule:hello20s
INFO: Analyzed target //exp_genrule:hello20s (1 packages loaded, 1 target configured).
INFO: Found 1 target...
Target //exp_genrule:hello20s up-to-date:
  bazel-bin/exp_genrule/hello20s.txt
INFO: Elapsed time: 20.058s, Critical Path: 20.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
```

Note that the new package name is also applied into the path of the output `bazel-bin/exp_genrule/hello20s.txt`

## Build environment

When a rule is built the [`cmd`] part is executed into a sandboxed environment instead of "in-place" like lot of build tool. It could be interesting to explore this sandbox a little beat to better understand it for our future custom build.

Let's start by grabbing information about the environment. Add into `exp_genrule/BUILD.bazel` the rule `envinfo`:

```python
genrule(
    name = "envinfo",
    outs = ["envinfo.txt"],
    cmd_bash = """( env; echo "-----"; pwd; ls; echo "-----"; whoami; echo "outs = $@" ) | tee $@""",
)
```

In this rule, we pipe the output into `tee` instead of redirect to also have the output displayed on terminal. It is easier for debug (in CI or in interactive).

```sh
❯ bazel build //exp_genrule:envinfo
INFO: Analyzed target //exp_genrule:envinfo (1 packages loaded, 1 target configured).
INFO: Found 1 target...
INFO: From Executing genrule //exp_genrule:envinfo:
PWD=/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/9/execroot/__main__
TMPDIR=/tmp
SHLVL=1
PATH=/home/david/.cache/bazelisk/downloads/bazelbuild/bazel-4.0.0-linux-x86_64/bin:/home/david/.nix-profile/bin:/home/david/.nix-profile/bin:/home/david/.poetry/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/david/.cargo/bin:/home/david/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin
_=/usr/bin/env
-----
/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/9/execroot/__main__
bazel-out
external
-----
david
outs = bazel-out/k8-fastbuild/bin/exp_genrule/envinfo.txt
Target //exp_genrule:envinfo up-to-date:
  bazel-bin/exp_genrule/envinfo.txt
INFO: Elapsed time: 0.043s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
```

We didn't trunk the output, so as you can see :

- the rule is executed with our account
- the environment variables and shell is not the one from where the command was called, notice that
  - `$PATH` is the same as in the caller shell
  - `$HOME` is not defined (like other environment)
- the current working directory is under `$HOME/.cache/...`
- the value of `$@` is a relative path that generate into `bazel-out` not into `bazel-bin`

But what change if we need input files ?

```sh
# create 2 input files
echo "A" > exp_genrule/a.txt
echo "B" > exp_genrule/b.txt
```

Modify `envinfo` to depend of the input files:

```python
genrule(
    name = "envinfo",
    srcs = [
        "a.txt",
        "b.txt",
    ],
    outs = ["envinfo.txt"],
    cmd_bash = """( env; echo "-----"; pwd; ls -l; echo "-----"; whoami; echo "outs = $@" ) | tee $@""",
)
```

```sh
❯ bazel build //exp_genrule:envinfo
INFO: Analyzed target //exp_genrule:envinfo (1 packages loaded, 3 targets configured).
INFO: Found 1 target...
INFO: From Executing genrule //exp_genrule:envinfo:
PWD=/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/13/execroot/__main__
TMPDIR=/tmp
SHLVL=1
PATH=/home/david/.cache/bazelisk/downloads/bazelbuild/bazel-4.0.0-linux-x86_64/bin:/home/david/.nix-profile/bin:/home/david/.nix-profile/bin:/home/david/.poetry/bin:/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin:/home/david/.cargo/bin:/home/david/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/var/lib/snapd/snap/bin
_=/usr/bin/env
-----
/home/david/.cache/bazel/_bazel_david/76e87152cc51687aee6e05b5bdcf89aa/sandbox/linux-sandbox/13/execroot/__main__
total 12
drwxr-xr-x 3 david david 4096 Mar 21 17:46 bazel-out
drwxr-xr-x 2 david david 4096 Mar 21 17:46 exp_genrule
drwxr-xr-x 3 david david 4096 Mar 21 17:46 external
-----
david
outs = bazel-out/k8-fastbuild/bin/exp_genrule/envinfo.txt
Target //exp_genrule:envinfo up-to-date:
  bazel-bin/exp_genrule/envinfo.txt
INFO: Elapsed time: 0.044s, Critical Path: 0.01s
INFO: 2 processes: 1 internal, 1 linux-sandbox.
INFO: Build completed successfully, 2 total actions
```

Bazel add a folder `exp_genrule` and if you modify the rule to use `ls -lR`, you will see that bazel also added symlink to `a.txt` and `b.txt`. It means that if we want to generate a file that take both file as input, we'll need to change directory to go inside `exp_genrule` or to accept path instead of filenames. To help us bazel provide function and [Make Variables](https://docs.bazel.build/versions/master/be/make-variables.html) (like `$@`). Try to display some of them:

```python

genrule(
    name = "envinfo",
    srcs = [
        "a.txt",
        "b.txt",
    ],
    outs = ["envinfo.txt"],
    cmd_bash = """(
        env
        echo "-----"
        pwd
        ls -l
        echo "-----"
        echo "whoami     = $$(whoami)"
        echo "TARGET_CPU = $(TARGET_CPU)"
        echo "BINDIR     = $(BINDIR)"
        echo "GENDIR     = $(GENDIR)"
        echo "OUTS       = $(OUTS)"
        echo "SRCS       = $(SRCS)"
        echo "RULEDIR    = $(RULEDIR)"
        echo "a.txt      = $(location :a.txt)"
        echo "b.txt      = $(location :b.txt)"
        ) | tee $@""",
)
```

```sh
...
TARGET_CPU = k8
BINDIR     = bazel-out/k8-fastbuild/bin
GENDIR     = bazel-out/k8-fastbuild/bin
OUTS       = bazel-out/k8-fastbuild/bin/exp_genrule/envinfo.txt
SRCS       = exp_genrule/a.txt exp_genrule/b.txt
RULEDIR    = bazel-out/k8-fastbuild/bin/exp_genrule
a.txt      = exp_genrule/a.txt
b.txt      = exp_genrule/b.txt
...
```
> NOTE the `:` at the begining of filename, it's because `location` takes a target as parameter, and every files can be identified as a target

So we can use those variables and the `location` function to compute the path that will be needed to run our tool inside bazel. But it will not be as simpler than with other builder. This complexity/sandbox is also what enforce that :
- every inputs (`srcs`, `tools`, `cmd`, environment variables, ...) are declared
- every outputs are generated

Try remove "a.txt" from `srcs` or to not generate content into `$@`, bazel will failed.

Enough experimentation about `genrule` for today,...

## Tips

We can also retrieve some of information with a built-in bazel command, give a try to:

```sh
bazel info --show_make_env
```

## Conclusion

We only overlook part of [`genrule`], we'll continue to discover when we'll used it in the following articles. If you want to know more, then follow the links from this article to the official doc.

You can also take a look at {% youtube nf9mOUvLkjE %}

The sandbox_bazel is hosted on github (not with the same history, due to errors), use tag to have the expected view at end of article: [`article/2_genrule`](https://github.com/davidB/sandbox_bazel/tree/article/2_genrule).

[rules]: https://docs.bazel.build/versions/master/skylark/rules.html
[`genrule`]: https://docs.bazel.build/versions/master/be/general.html#genrule
[`name`]: https://docs.bazel.build/versions/master/be/general.html#genrule.name
[`outs`]: https://docs.bazel.build/versions/master/be/general.html#genrule.outs
[`cmd`]: https://docs.bazel.build/versions/master/be/general.html#genrule.cmd