I agree. We moved to Bazel at my last job and it probably took about 6 person months. I was most of that person and it includes other tooling not related to Bazel. Some extremely competent engineers also moved over our frontend code (including stuff like Cypress) and Python code (which needs to run against 3 different versions of Python). They had no Bazel experience beforehand and asked me maybe a couple hours worth of questions and just got it done. So I don't think you need to be a Bazel genius to get this done, but it helps to have someone with a Vision, which was me in this case. All in all, I'd do it again. I'm in the process of moving all my open source code to a monorepo (jrockway/monorepo, which really should have been called jrockway/jrockway) because the development experience is so much better.

The biggest goal for me in doing the project at work was because new employees couldn't run the code they were working on. We hired people. They tried. They weren't very productive. That's my fault, and I wanted to fix that while supporting our policy of "you can use any Linux distribution you want, or you can use an arm64 Mac". Many people suggested things like "force everyone to use NixOS" which I would be in favor of, but it wasn't the solution that won. (I honestly prefer Debian myself and didn't think that my preference should dictate how the team works. The fact that I disagreed with the proposed solution is a good indicator that people would be unhappy with anything I declared by fiat.) Rather, using Bazel to provide a framework for retrieving third-party tooling and also building our code was a comfortable compromise.

A secondary goal was test caching. If you edit README.md, CI doesn't need to rerun the Cypress tests. (As a corollary, if you edit "metadata_server.go", the "pfs_server.go" tests don't need to run, as the PFS server does not depend on the Metadata server.)

The biggest piece of slowness and complexity in the workflow would be building our code into a container to run in k8s. We used goreleaser and that involved building the code twice, one for each architecture, to build into an OCI image index which was our main release artifact. The usual shell scripts for local development just reused this and it was terribly slow. Throw in docker to do the builds, which deletes the Go build cache after every build, and you have a recipe for not getting anything done. Bazel is a much better way to build containers. Containers are just some JSON files and tar files. Bazel (rules_oci) just assembles your build artifacts into the necessary JSON files and tar files. To build a multi-architecture image index, you build twice and add a JSON file. Bazel handles this with platform transitions; you make a rule to build for the host architecture (technically transitioned to linux on a macos host), and then the image index rule just builds that target with two configurations (cross-compiled, not emulated with binfmt_misc like "docker buildx") and assembles the two artifacts into the desired multi-arch image. When running locally, you skip 2 of the 3 steps, just build for the host machine. Combined with proper build caching (thanks BuildBuddy!), this means that making an image to run in k8s takes about 10 seconds instead of 6 minutes. With the previous system you could try your code 80 times a day. With the new system, you could try it 2880 times ;) This increased productivity.

I also wrote a bunch of tools to make setting up k8s easier, which would have been perfectly possible without Bazel, but it helped. (Before, everyone pushed their built image to DockerHub and then reconfigured k8s to pull that. Now we have a local registry, and if two people do a build at the same time, you always get yours and not theirs. I did not design this previous system, I merely set out to fix it because it's Wrong.) Bazel makes vendoring tools pretty easy. For our product we needed things like kubectl, kind, skopeo, postgres, etc. These are all in //tools/whatever and can be run for your host machine with `bazel run //tools/whatever`. So once you ran my program to create and update your environment, you automatically had the right version of the tool to interact with it. We upgraded k8s regularly, nobody noticed. They would just get the new tool the next time they tried to run it. (A centrally managed linux distribution would do the same thing, but it couldn't revert you to an old tool when you checked out an old version to debug. A README with versions would work, but I learned that nobody really reads the READMEs until you ask them to. "How do I do X" "See this section of the README" "Oh damn I wish I thought of that" "Me too." ;)

The biggest problem I had with Bazel in the past was dealing with generated files. Editor support, "go get <our thing>", etc. I got by when I used Blaze at Google, but realistically, there was no editor support for Go at that time, so I didn't notice how badly it worked. There is now GOPACKAGESDRIVER, which technically helps, but it didn't work well for me and I wasn't going to inflict it upon my team. I punted this time and continued to check in generated files. We have a target //:make_proto that rebuilds the proto files, and a test that checks that you did it. You check in the generated protos when you change the protos. It works and I have a general rule for all generations like this. (We also generate a bunch of JSON from Jsonnet files; this mechanism helps with that.)

All in all, you can get a fresh machine, install git and Bazelisk, check out our repo, and get "bazel test ..." to succeed. That, to me, is the minimum dev experience that your employer owes you, and if you joined my team, you'd get it. That made me happy and it wouldn't be as good without Bazel. I'd do it again!

Just as an aside, after the Bazel conversion, I did a really complicated change, and Bazel didn't make it any harder. We made our main product depend on pg_dump, and adjusting the container building rules from "distroless plus a go binary" to "debian plus postgres plus a go binary" was pretty easy. rules_debian is very nifty, and it gives me a sense of reproducibility that I never got from "FROM debian:stable@sha256...; RUN apt-get update && apt-get install ....". Indeed, the reproducibility is there. You can take any release tag, "bazel build //oci:whatever" and see that the resulting multi-arch image has the same sha256 of what's on DockerHub. I couldn't have done that without Bazel, or at least not without writing a lot of code.

I don't work there anymore but I'm really happy about the project. I don't even do Build & Release stuff. I just add features to the product. But this needed to be done and I was happy to wear the hat.