Building a syntax highlighting extension for VS Code
Tags: vscode, slim, rails, regex
Published: 2024-03-01 04:28:55

I spent a few days of my spare time building a [VS Code extension](https://marketplace.visualstudio.com/items?itemName=borama.ruby-slim) that would bring better syntax highlighting for the [Slim template language](https://github.com/slim-template/slim) to the editor. I quite enjoyed most of the process so I’d like to share what I learned.

### Why?

First of all, **I like Slim**. I like the beauty and cleanness of Slim templates, to me they are way more readable than regular ERB templates and I think they fit in the ruby/Rails ecosystem very well. Slim is a close cousin to [Haml](https://haml.info/), without the ugly percent characters, haha. I've used Slim exclusively in my projects since about 2016.

But why bother building a new syntax highlighting extension for VS Code especially when there [already is](https://marketplace.visualstudio.com/items?itemName=sianglim.slim) a quite popular one? To answer that I need to add a bit more of a personal context (short, I promise)… As a long-time and mostly happy user of the [RubyMine IDE](https://www.jetbrains.com/ruby/) from Jetbrains I recently noticed I somehow got used to this subdued, passive role: when I encountered a bug or a missing feature of the editor, I could of course file an [issue in the tracker](https://youtrack.jetbrains.com/issues/RUBY) but – that was about it. As RubyMine is closed source code and the [plugins ecosystem](https://plugins.jetbrains.com/docs/intellij/developing-plugins.html) was hard to grasp for me, I never felt the courage to deep dive into building a custom plugin (actually, I did try hard once – and failed – many years ago, and I can see that things might be a bit easier now with Kotlin, the Gradle toolkit and finally the recent [LSP support](https://blog.jetbrains.com/ruby/2023/07/the-rubymine-2023-2-beta-updated-ai-assistant-lsp-api-for-plugin-developers-and-more/#lsp-support-for-plugin-developers)). So, I waited patiently for the developers to notice the issue and fix it. And sometimes they did! But more often not which is quite understandable for a niche bug or language like Slim.

Now, fast forward to last year's [Rails World conference](https://rubyonrails.org/world) that I was a lucky attendee of. What a breeze of fresh air! Among the many many inspiring people, talks and presentations, I noticed one thing: most people use VS Code, some use Vim but – more importantly – **a lot of [people](https://youtu.be/iRjei4nj41o?feature=shared&t=1261) [tweak](https://youtu.be/GnqRMQ0iQTg?feature=shared&t=2019) their editor / IDE** almost as routinely as they tweak the code they work on professionally! And I thought: I want that too, how come I've lost this mindset here? I’ve taken for granted that I can tweak every imaginable aspect of my Linux OS as well as the Gnome environment so why not my IDE – the program that I literary spend most hours a day in? That was the final nudge for me to try to switch to something – anything really – that would be feasible for me to tweak and that’s how I ended up in VS Code. I’m not saying this will be my final IDE destination (looking at you [Zed](https://zed.dev/), [Fleet](https://www.jetbrains.com/fleet/) or perhaps even Vim) but I know I want to stay closer to where a more active developer community around the editor is.

### OK, but why another Slim syntax extension?

After the switch, an opportunity for a small initial tweak came almost immediately: I didn't like the syntax highlighting of the Slim templates in our project. It almost resembled the old days when RubyMine highlighted [slashes](https://tailwindcss.com/docs/width#percentage-widths) in some of the Tailwind classes as ugly errors (to be fair, they have fixed it a long time ago):

![An old version of RubyMine marking a Tailwind class with a slash as an error](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k97r1e03zudvy4x7lbkv.png)

[The only Slim extension](https://marketplace.visualstudio.com/items?itemName=sianglim.slim) available for VS Code at that time had somewhat similar problems: it couldn’t recognize slashes in class names, it did not understand multi-line comments, attribute values and expressions, support for some embedded language blocks such as Markdown or SASS was missing.

![Slim extension having problems with a multi-line attribute](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/blvrmru9s846wb2yx82c.png)

I said to myself: _it couldn’t be that hard to fix, could it?_ Well… When I looked at [the repository](https://github.com/sianglim/vscode-slim), I soon found out that it wouldn’t be that easy: the grammar syntax was in some hardly comprehensible XML format ([PList](https://en.wikipedia.org/wiki/Property_list)) and looked like being a bare copy from somewhere else, there were no comments or helpful tips anywhere and, above all, the whole thing was basically unmaintained: last changes from several years ago, no issues resolved, no PRs merged. I’m not judging here (of course I made several ”zombie projects“ myself, too), I just hope it’s clear now that I _had to_ go down the rabbit hole and rebuild the extension from the bottom up.

### Down the rabbit hole

The official [Syntax Highlighting Guide](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide) was very nice and helpful. It was not hard to create a blank grammar extension. What turned out much harder was the grammar syntax itself, partly because there was no formal specification available. The official guide showed a [basic grammar example](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide#contributing-a-basic-grammar) but mostly linked to a few articles such as the [Textmate grammars](https://macromates.com/manual/en/language_grammars) from which the VS Code ones originate. 

The most helpful resource for me, in the end, was [this post by Matt Neuburg](https://www.apeth.com/nonblog/stories/textmatebundle.html). It was a long one and I loved how sincere it was when describing the time involvement needed to deep dive into writing language grammars: 

![Excerpt from Matt Neuburg’s article about grammars](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/1453lbs7k0k30syiqvwe.png)

Strangely enough, it got me hooked, it looked like a fun thing to try! 🙂

#### Regular expressions

I quickly learned that I needed to invest myself in one more thing before trying to actually edit the files: **regular expressions**. All grammar matching is based on them. I refreshed the basics and re-read thoroughly the [lookahead](https://www.regextutorial.org/positive-and-negative-lookahead-assertions.php) and [lookbehind](https://www.regextutorial.org/positive-and-negative-lookbehind-assertions.php) chapters which turned out to be tremendously useful. 

As a ruby developer, I was happy to find that VS Code / TextMate grammar files use the same regular expression engine called [Oniguruma](https://github.com/kkos/oniguruma) as ruby itself. Thus, I could be sure that when trying my regular expressions in my favorite online regex tool, [rubular.com](https://rubular.com/), there would be no inconsistencies due to the engine inner workings.

#### Unit-testing the beast

Oh, one more thing – and actually the most important one – **unit tests**! I would never be able to finish a working extension without having them at hand, I don’t even understand how did people make language grammars without unit-testing them, it seems just like a too complex task to me. I am no [TDD](https://en.wikipedia.org/wiki/Test-driven_development)-ist but it was unit tests that gave me the necessary confidence and guided me throughout the process.

I spent some time looking for ways to unit-test the grammar. The official VS Code docs didn’t say a word about that and their [Testing Extensions chapter](https://code.visualstudio.com/api/working-with-extensions/testing-extension) dealt rather with integration tests for the extension itself. I needed something at a lower level.

Luckily, there is a project that has fitted my needs perfectly: **[`vscode-grammar-test`](https://github.com/PanAeon/vscode-tmgrammar-test)**. It’s a command line tool that builds on the VS Code [regex engine](https://github.com/microsoft/vscode-oniguruma) and [grammar file parser](https://github.com/microsoft/vscode-textmate) and allows to run unit tests directly against a given grammar file. 

The format of the test files themselves is inspired by the relatively new initiative by the [Sublime Text](https://www.sublimetext.com/) team when they [introduced](https://www.sublimetext.com/blog/articles/sublime-text-3-build-3103) a new grammar file format called [Sublime Syntax](https://www.sublimetext.com/docs/syntax.html) and – more importantly – **a way to [unit test](https://www.sublimetext.com/docs/syntax.html#testing) grammars**. It’s using some lovely human-friendly magic comments that allow to specify what [scopes](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide#textmate-tokens-and-scopes) should the grammar file produce for a given position on a given line. 

For example, the following Slim line is tested using asserts in the comment lines below it: the `<-` ”operator“ targets the first character on the tested line and the `^`s target the ”underlined“ part of the tested line:

```slim
/ unit_tests.slim (i.e. this is a Slim file with comments)

' Verbatim <b>#{text}</b> without processing.
/ <- punctuation.section.verbatim.slim
/ ^^^^^^^^^^^^ text.html.embedded.slim
/          ^^^ meta.tag.inline.b.start.html
/              ...
```

This is cool. Even cooler is the fact that the Sublime Text team also provide their own official and full-featured [grammar for Slim templates](https://github.com/SublimeText/Slim/tree/master) as well as unit tests! So, in the end, I just grabbed the [test file](https://github.com/SublimeText/Slim/blob/master/tests/syntax_test_slim.slim) and used it as a basis for all my development of the VS Code extension grammar file. ❤️

#### TL;DR recap

Chances are that your head is already exploding so let’s recap the dependencies and inspiration sources for building a VS Code syntax grammar file: 

- VS Code syntax highlighting works with TextMate editor grammar files,
- they are written in an old but well-thought-out specification based on some clever regexp matching
- for VS Code they can be written in PList, JSON or YAML formats,
- the Sublime Text team introduced a newer format for the same task (which is not directly usable in VS Code) and another one for testing the grammars,
- they also created grammar files for various languages, they are usually well-maintained and include the test files,
- the `vscode-grammar-test` tool combines the Sublime Text test files with VS Code / TextMate grammar files and thus allows unit testing the VS Code syntax grammars.

### Building the grammar

With all the tools and some basic theoretical knowledge of the problem at hand, I began building the grammar file. Of course I didn’t start from scratch though, I used the official TextMate grammar file (the [one in the YAML format](https://github.com/slim-template/ruby-slim.tmbundle/blob/master/Syntaxes/Ruby%20Slim.YAML-tmLanguage)) from the [Slim language repository](https://github.com/slim-template/ruby-slim.tmbundle). 

I opened up the grammar file and the unit test file in my editor and started working **iteratively**:
1. I converted my YAML grammar file to the JSON format using the `js-yaml` tool (see my [wiki](https://github.com/borama/vscode-ruby-slim/wiki/Development-hints) for more instructions).

2. I [ran the tests](https://github.com/borama/vscode-ruby-slim/wiki/Development-hints#tests) with the output redirected to a `results.txt` log file. This flooded the results log with hundreds of assertion errors. I opened up the log file and scrolled to the first error, for example similar to this one:

        ✖ tests/unit_tests.slim failed
          at [tests/unit_tests.slim:5:1:2]:
          5: /Comment
             ^
          missing required scopes: comment.block.code.slim punctuation.definition.comment.slim
          actual: source.slim comment.block.slim comment.line.slash.slim punctuation.definition.comment.slim

3. Next, I tried to understand what the error was about: was this a real error or a missing feature in the grammar file? Or, was this just a mismatch between what scopes the original grammar file produced vs. what scopes the – much younger – Sublime Text unit test file expected? What scopes does VS Code need anyway? While it was hard to asses these things in the beginning, I slowly began to follow… (I will share some hints below.)

    > BTW, I found the [GitHub Copilot: Explain This](https://dev.to/github/understand-your-code-using-github-copilot-5375) feature in VS Code quite useful. It provided me with explanations that helped me initially, even though I still had to do the hard part – deciding what the error really was about – myself.
    > ![Copilot explaining a grammar test for me](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lfo7guh2b0ynakg0k2ol.png)

4. Having an idea about the cause of the error I edited the grammar file and/or the test file and jumped back to #1. And that was all there was to it! 😄

It may sound scary but **it was actually quite fun**! Although trying to untangle some of the problems definitely _was_ frustrating, the **very fast feedback loop** that this TDD-ish workflow provided was very satisfying and kept me reassured and oriented along the path. The most pleasant were the situations when I fixed a single thing in the grammar file and then watched tens or even hundreds of errors magically disappear!

#### Selected grammar file quirks

I will not try to explain how the grammar file works, it would be useless to repeat stuff from other great materials such as the [Matt’s post](https://www.apeth.com/nonblog/stories/textmatebundle.html). Instead, I want to briefly mention a few of the peculiarities that one likely hits when playing with the VS Code grammar files:

- The order of patterns in the grammar file is important - **first patterns are tried first**.

- The regexps can and often **should use groups** (either numbered or named groups). They can then be targeted by the `capture` keys mapping the groups to the actual [VS Code scopes](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide#textmate-tokens-and-scopes) that the given language pattern should produce.

- Although the regexps in the grammar can match new lines `\n`, they will **never match multiple lines of text** even though they are written like they should. In other words, regexps in grammars are forced to always match a single line of text only.

- The `end` key of a pattern, which specifies the regexp that the matched language pattern ends with, often acts in a surprising way that one has to keep in mind: if you match some _nested_ patterns inside this one, its **`end` regexp will become a ”floating“ one**. It will match the (final) part of the language pattern _after all nested patterns have finished their matching_. 

    So, for example, if the `end` regexp matches a new line but patterns nested inside the main one also match some new lines, the main pattern `end` will match a new line only after all the nested patterns are satisfied. In other words, what the `end` regexp effectively means is that it’s the _first_ match the pattern _can have_ if no other nested patterns prolong the whole pattern.

- The `end` regexp can also **[backreference](https://www.regular-expressions.info/backref.html) groups from the `begin` regexp**. This is very helpful in Slim templates because it allows the `end` regexp to match lines that are nested out of the `begin` line.

- The grammar syntax also allows defining a **repository of named patterns**. The repository very conveniently cleans up the grammar file and allows reusing of patterns.

- When working with **embedded languages**, the appropriate language grammar file for the embedded language is needed to better understand how embedding works and to run tests that target embedded language scopes (see the `-g` option of the `vscode-tmgrammar-test` tool). It should not be packed into the extension, though, this is a dev-only dependency.

- VS Code has a [convention for scopes naming](https://macromates.com/manual/en/language_grammars#naming_conventions). **Sometimes this convention differs** from conventions in other editors, notable those used in the new grammar files in Sublime Text which I used as a basis for the Slim extension. Thus, I had to rename some of the scopes to match the VS Code ones.

### Debugging and testing the extension

I used a few ways to try and test the syntax highlighting extension I was working on. First of all, the most important one were the **unit tests**. Once I got used to the error messages the testing tool reported, I was able to quickly pinpoint the problematic location in the grammar file and could start playing with the matching rules there. To verify that they work, I again ran the tests and watched the changes in the output log.

In more complicated cases, **I used the [Scope inspector](https://code.visualstudio.com/api/language-extensions/syntax-highlight-guide#scope-inspector)**. I created a [launch configuration](https://github.com/borama/vscode-ruby-slim/blob/main/.vscode/launch.json) to run the extension in the [VS Code Extension host](https://code.visualstudio.com/api/advanced-topics/extension-host). Then I could run the extension any time, open up a Slim template and start the Scope inspector via the ”Editor: Inspect Editor Tokens and Scopes“ quick action (for which I soon learned the shortcut key). This way I was able to inspect the behavior of the extension in a live document. 

By the way, I extracted the official [Literate tests](https://github.com/slim-template/slim/blob/main/test/literate/TESTS.md) from the Slim repository into a [Slim template](https://github.com/borama/vscode-ruby-slim/blob/main/tests/test_suite.slim) that would be always ready for some manual testing.

From time to time I also **[packed](https://github.com/borama/vscode-ruby-slim/wiki/Development-hints#pack-extension) the extension** into a `.vsix` file and installed it locally into my VS Code (using the ”Extension: Install from VSIX…“ quick action) so that I could watch it highlighting some real code during my regular working days.

### Final notes

So here we are, the [Slim highlighting extension](https://marketplace.visualstudio.com/items?itemName=borama.ruby-slim) is ready for you to try. I mostly enjoyed the process of building it, although at times frustrating and having an arcane feel. Thanks to the fast feedback loop, very clear problem boundaries and a good notion of progress it was fun most of the time!

I now realize I quickly made most of the extension four months ago and it was only after I finished covering the grammar file by the unit tests that I somehow lost interest. I had the extension installed locally and it was working well, so I didn’t feel the push to finish the last 5% and actually publish the thing. Until now, luckily. 😅

So, enjoy the extension if you work in Slim, and I encourage you to try to build your own language grammar if you like working with a vintage, sometimes weird but apparently well-thought-out piece of technology.

_Want more stuff like this? Follow me here or [on Twitter](https://twitter.com/boramacz). Cheers!_