Gitpod notes and thoughts
Tags: gitpod
Published: 2022-07-30 04:48:12

A few months ago, a colleague reposted a Gitpod [blog post](https://www.gitpod.io/blog/gitpod-for-opensource) in my employer's slack, mentioning that if you're a member of some reputable GitHub organization, you get a plan with unlimited hours. Thank you, the past me who spent a weekend packaging an app for nixpkgs!

I knew these kinds of cloud IDEs have been around for some time, but that message inspired me to give it a bit of try, who can resist free lunch after all? Well, I'm now using it on a daily basis, so this post will record some notes and thoughts along the way. The content is not quite well structured, but I hope they can provide a bit of insight for other Gitpod users as well.

## Hacking GHC

The first Gitpod use case I tried is hacking [GHC](https://gitlab.haskell.org/ghc/ghc). And that's quite a roadbump because GHC is hosted on their own GitLab instance!

Yes, you probably can fiddle with tokens and such, to make Gitpod work with self-hosted GitLab. It's mentioned in Gitpod docs, I haven't tried, since there's another problem ahead:

Gitpod requires you to check in `.gitpod.yml` into the repo you're working with. That's reasonable for most single-repo use cases, but I don't have any mental energy to upstream a Gitpod config, nor do I want to pollute my GHC working branches with this config file.

So I made a dedicated GitHub [repo](https://github.com/TerrorJack/gitpod-ghc) for hacking GHC. Gitpod allows you to specify scripts to run during workspace initialization, so I can just clone GHC to `/workspace` and open it later, profit.

## Git authentication with SSH

All looks good except when I need to push. For GitHub repos I own, pushes to `https` remotes should work out of the box, but well, this doesn't work for GHC's self-hosted GitLab.

This hasn't been a problem when I used my employer's remote build machine, since I can always enable SSH agent forwarding, so I can do the pushes on that machine. Gitpod allows you to SSH into an active workspace, but honestly I don't want to open a separate terminal window just for git operations, I prefer to stay inside that browser tab all the time.

So. Given Gitpod allows specifying secret environment variables, I generated a passwordless SSH private key and encoded it as a secret. The `gitpod-ghc` init script will start an `ssh-agent` and automatically add that key, so future `git` operations will work out of the box.

Yes, doesn't sound like a good security story, other than permuting the key frequently and limiting the key's scope, I still need to trust Gitpod not to leak my secret, no less than they trust me not to abuse their machines to run crypto miners.

## What's persisted and what's not

There are multiple ingredients in making a Gitpod workspace:

- The base `Dockerfile` or a Docker Hub image. Use whatever you like, but given the Gitpod runtime will yolo a bunch of Linux binaries at runtime, it better be a typical glibc-based FHS distro.
- The `.gitpod.yml` config, which allows you to specify init scripts.

Having two places to write scripts may seem redundant, but the `.gitpod.yml` scripts have access to the `/workplace` directory. Across restarts of the same Gitpod workspace, only changes in `/workplace` is persisted!

It's quite an annoyance since for a long-lived workspace, you often accumulate implicit state (e.g. language-specific caches) not only in your project directory, but also in the home directory.

Luckily, there's a limited workaround for that: bind mounts. In `gitpod-ghc` config, I bind mount a lot of home directory contents from `/workplace` instead, things like `.config`, `.cache`, `.cabal`. Even the entire Nix store!

It's a pity I can't bind mount the entire home directory though, likely because of busy resources during workplace initialization. And of course, all system package manager invocations are forgotten after workplace restart.

To think of it in a more positive way, Gitpod is enforcing the [erase your darlings](https://grahamc.com/blog/erase-your-darlings) philosophy. It doesn't strictly prohibit state, it forces you to make your state explicit.

## Prebuilds

Gitpod prebuilds is a CI service that runs on the same infra to power the workspaces. When a prebuild is triggered, the Docker image gets built, a fresh `/workplace` with a checkout of that commit is mounted, and the expensive (`init`) step of `.gitpod.yml` scripts get run. The updated `/workplace` is saved and reused upon workplace start up, so the `init` step can be skipped at that time.

For a large project like GHC, enabling Gitpod prebuilds is essential to ensure a sane startup time for new workspaces. Here's a non-comprehensive list of things done during `gitpod-ghc` prebuild time:

- Install latest release versions of `ghcup`, `ghc` and `cabal`
- Compile and install `haskell-language-server`
- Check out GHC tree into `/workspace`
- Go through the `configure` step, build `hadrian`, build up the `.hie-bios` cache, so the language server shall work nearly instantly for a fresh workspace

## In-browser VSCode caveats

Oops. I've written a lot in this post and haven't even mentioned VSCode even once. Yes, Gitpod supports multiple IDEs, but the one I'm using on a daily basis is the in-browser VSCode. Some caveats to keep in mind:

- Most default key bindings work. ^W closes the browser tab instead of the editor tab though, luckily there'll be a prompt about unsaved work. Don't silence that prompt.
- Flaky connection is fine, but don't drop offline for more than 3 minutes.
- Settings do get synced properly. A minor annoyance is you can't declare editor settings in `.gitpod.yml`, apart from the extension list.
- The default extension store is OpenVSX instead of the Microsoft-hosted one. There do exist extensions that are severely outdated in OpenVSX (like `rust-analyzer`), but as a fallback measure, you can specify URLs to `.vsix` artifacts in `.gitpod.yml`.

## Other things known to work (or not)

There are things that work, and things that don't work yet. I didn't really spend time to dig into Gitpod's issue tracker, so I'll be more than happy to be corrected in the comments shall the list requires some update!

- `sudo` works. Of course you're still in a container jail, so you can't really mess with `sysctl` config and such.
- `systemd` doesn't work. You need to bring your own userspace supervisor if you need such a thing.
- `nix` works with single-user installation and `sandbox = false`. Multi-user mode with `nix-daemon` should still be possible, but not worth the effort if sandboxing won't work anyway. So don't ever push to binary caches from a Gitpod workspace.
- `docker` works out of the box as rootless mode, and the state directory is in `/workplace`. `podman` doesn't, despite unprivileged user namespace is on.
- `gdb` works. `strace` works. `lldb` doesn't. `rr` doesn't.
- `proot` latest version works.
- `tailscale` userspace mode works, but running the daemon as `root` seems quite fragile. Haven't tried other VPNs yet.
- `htop` kinda works. The CPU/RAM stats don't reflect the workspace, but reflects the entire metal below it.

## Community support

Gitpod development happens on GitHub, they allow self-hosting and the sausages are made in the open.

Apart from that, there's an official discord server. Not a super active one, but the staff are there, and questions do get answered within a day or two, regardless of whether you're a paid user or not.

## Conclusion

I believe I've accumulated enough experience to say: Gitpod is nice, and it's worth a try.