Create a VSCode Extension for my esoteric programming language in C++ with Web Assembly
Tags: cpp, vscode, emscripten, webassembly
Published: 2023-03-06 20:50:15

---
title: Create a VSCode Extension for my esoteric programming language in C++ with Web Assembly
published: true
description: Create a VSCode extention, mainly in C++, to have semantic highlighting and auto-completion
tags: [cpp, vscode, emscripten, webassembly]
series: PainPerdu
cover_image: https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dfgkhwono0nnvyglpkbs.png
---

## Article::Article
I cleaned up the project, I compiled it to Wasm using emscripten, that's nice; but something is missing : tools to develop in my langage. Who would like to use something without a highlighter and auto-completion? Even the brainfuck langage has some VSCode extensions to highlight the code!   

But this issue is already something of the past now, because I created [a VScode extension](https://marketplace.visualstudio.com/items?itemName=Baduit.painperdu-bakery) providing auto-completion, semantic highlighting.

## Why semantic highlighting instead of syntax highlighting?
VScode offers two APIs to provide highlighting, syntactic and semantic. With the syntax one, I need to provide a [TextMate grammars](https://macromates.com/manual/en/language_grammars) and it looks like the opposite of fun. With the semantic one, I need to provide a list of tokens with their meaning. It may seem more complicated, but I already have a complete parser, so it will be a piece of cake.

## The informations VSCode needs
### For the highlighting
As I said, for the semantic highlighting VSCode needs a list of tokens. Each token must have the following information:
* the line where it starts
* the column where it ends
* the length
* the type

VSCode already provides some types by default:
* _variable_: for PainPerdu references
* _function_: for PainPerdu labels
* _comment_: (do it really need to say it?)
* _string_: for PainPerdu file inclusion (because it is between two double quotes)
* _number_: for integers (there is no decimal in PainPerdu)
* _operator_: for symbols

### For the auto-completion
VSCode will only need the list of labels and references. That's it.

## The C++ implementation
### Define the token
The token is defined with the following structure:
```C++
struct Token
{
    enum class Type
    {
        REFERENCE,
        LABEL,
        COMMENT,
        STRING,
        NUMBER,
        OPERATOR
    };

    bool operator==(const Token&) const = default;

    Type type;
    std::size_t line;
    std::size_t start_column;
    std::size_t length;
};
```
I won't linger on this code; it is a very simple structure containing the information stated in the previous section.

### Get the tokens
I just need to get the tokens, therefore I won't need to build a whole parse tree, I will only use the callback system that I already used to handle the errors in [this previous article](https://baduit.github.io/2023/01/08/remove-clunky-parser-and-use-pegtl.html) to just fill a vector of token. 

For the `Move right` operator `>` it would look like this:
```C++
template <>
struct ToTokenAction<operators::MoveRight>
{
    template <typename ParseInput>
    static void apply(const ParseInput& in, std::vector<Token>& tokens)
    {
        tokens.push_back(
            Token
            {
                .type = Token::Type::OPERATOR,
                .line = in.position().line,
                .start_column = in.position().column,
                .length = in.size()
            });
    }
};
```

But because I'm lazy and I don't want to write almost the same code twenty times, I will use the following macro:

```C++
#define DefineAction(Match, TokenType)                                                \
    template <>                                                                       \
    struct ToTokenAction<Match>                                                   \
    {                                                                                 \
        template <typename ParseInput>                                                \
        static void apply(const ParseInput& in, std::vector<Token>& tokens)           \
        {                                                                             \
            tokens.push_back(                                                         \
                Token                                                                 \
                {                                                                     \
                    .type = TokenType,                                                \
                    .line = in.position().line,                                       \
                    .start_column = in.position().column,                             \
                    .length = in.size()                                               \
                });                                                                   \
        }                                                                             \
    };
// You can then use it like this:
DefineAction(operators::MoveRight, Token::Type::OPERATOR)
```

There is still one issue, I need to differentiate the `labels` and the `references`, because for now they are only considered as `identifiers`.

I know that a `label` must be preceded by some specific operators, same for the references. Each time the parser see one of these operators, it knows that the next identifier will be either a `reference` or a `label`.

I can update the macro to reflect this and then handle correctly the `labels` and `references`:
```C++
// Little helper struct to centralize some information
struct ToTokenState
{
    enum NextIdentifierType
    {
        NONE, // If there is an identifier with NONE, this is a bug
        LABEL,
        REFERENCE
    };

    std::vector<Token> tokens;
    NextIdentifierType next_identifier_type;
};

// A little helper function
inline Token::Type get_token_type(ToTokenState::NextIdentifierType next_identifier_type)
{
    switch (next_identifier_type)
    {
        case ToTokenState::NextIdentifierType::LABEL :
        {
            return Token::Type::LABEL;
        }
        case ToTokenState::NextIdentifierType::REFERENCE :
        {
            return Token::Type::REFERENCE;
        }
        default:
        {
            throw std::runtime_error("Someone did a shitty code trololol");
        }
    }
}

// Empty default action
template<typename Rule>
struct ToTokenAction {};

// The macro has one more argument
#define DefineAction(Match, TokenType, IdentifierType)                                \
    template <>                                                                       \
    struct ToTokenAction<Match>                                                       \
    {                                                                                 \
        template <typename ParseInput>                                                \
        static void apply(const ParseInput& in, ToTokenState& state)                  \
        {                                                                             \
            state.tokens.push_back(                                                   \
                Token                                                                 \
                {                                                                     \
                    .type = TokenType,                                                \
                    .line = in.position().line,                                       \
                    .start_column = in.position().column,                             \
                    .length = in.size()                                               \
                });                                                                   \
            state.next_identifier_type = IdentifierType; // Update accoringly       \
        }                                                                             \
    };

// Now use the macro
DefineAction(operators::DefineLabel, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::MoveRight, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::MoveLeft, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::Increment, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::Decrement, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(ResetCase, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)
DefineAction(operators::DefineReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::UndefineReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::MoveToReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::GoToLabel, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::Rewind, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::LABEL)
DefineAction(operators::IfCurrentValueDifferent, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::IfCursorIsAtReference, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(operators::IfReferenceExists, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::REFERENCE)
DefineAction(GetChar, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)
DefineAction(PutChar, Token::Type::OPERATOR, ToTokenState::NextIdentifierType::NONE)

DefineAction(Comment, Token::Type::COMMENT, ToTokenState::NextIdentifierType::NONE)
DefineAction(ReadFile, Token::Type::STRING, ToTokenState::NextIdentifierType::NONE)
DefineAction(Integer, Token::Type::NUMBER, ToTokenState::NextIdentifierType::NONE)

// Handle the identifier
template <>                                                                       
struct ToTokenAction<Identifier>
{
    template <typename ParseInput>
    static void apply(const ParseInput& in, ToTokenState& state)
    {
        if (state.next_identifier_type != ToTokenState::NextIdentifierType::NONE)
        {
            state.tokens.push_back(
                Token
                {
                    .type = get_token_type(state.next_identifier_type),
                    .line = in.position().line,
                    .start_column = in.position().column,
                    .length = in.size()
                });
            state.next_identifier_type = ToTokenState::NextIdentifierType::NONE;
        }
    }
};

// Because I'm not that dirty, I under my macro since I don't need it anymore
#undef DefineAction
```

And finally, I the code to use this machinery and get all the tokens:
```C++
std::vector<Token> Parser::get_tokens(std::string_view input)
{
    ToTokenState token_state;

    pegtl::memory_input mem_input(input.data(), input.size(), "");
    pegtl::parse<Grammar, ToTokenAction>(mem_input, token_state);

    return token_state.tokens;
}
```

### Get the labels and references
Now that I am able to get the tokens, this part will be very easy.

The only way to define a label is like this: `:my_label` and the only way to define a reference is like this : `#my_reference`. I will detect these patterns to get the desired list.

There is two solutions to do it:
* Make a parse tree and with a custom Selector to get only label/reference with identifiers
* Use the same technique as the tokens

Both are really simple but I chose the second one because the code is really compact. Here's the code for the reference, but for the labels, the code is almost the same:
```C++
template<typename Rule>
struct GetDefinedReferencesAction {};

template<>
struct GetDefinedReferencesAction<operators::DefineReference>
{
	// This is called when a # is encountered
	template <typename ParseInput>
	static void apply(const ParseInput&, std::vector<std::string>&, bool& is_next_reference)
	{
		is_next_reference = true;
	}
};

template<>
struct GetDefinedReferencesAction<Identifier>
{
	// This is called for all identifiers
	template <typename ParseInput>
	static void apply(const ParseInput& in, std::vector<std::string>& references, bool& is_next_reference)
	{
		// If the previous caracter was a #
		if (is_next_reference)
		{
			// Add it in the list
			references.push_back(in.string());
			is_next_reference = false;
		}
	}
};
```
That's it!

### Create the bindings
Now to create the bindings I need a new CMake target, it is the same way as the web interpreter with the difference that the nodejs shipped with VSCode does not use the flag `--experimental-wasm-eh`. This mean I won't use the flag `-fwasm-exceptions` in the CMakeLists.txt:
```CMake
# Create the target
add_executable(ExtentionBindings ExtentionBindings.cpp)
# Basic options
target_compile_options(ExtentionBindings PRIVATE -Wextra -Wall -Wsign-conversion -Wfloat-equal -pedantic -Wredundant-decls -Wshadow -Wpointer-arith -O3)
# Don't forget to link with --bind
target_link_options(ExtentionBindings PRIVATE --bind)
# Obviously I need my PainPerdu library
target_link_libraries(ExtentionBindings PRIVATE PainPerdu)
# Put the output files directly in the good directory
set_target_properties(ExtentionBindings PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_CURRENT_LIST_DIR}/../../vscode_extention/painperdu-bakery/generated")
```

Same for the cpp files, it will be very similar as what I did in my [previous article](https://baduit.github.io/2023/01/29/PainPerdu-emscripten-web.html) about emscripten.

Add some getter functions to the `Token` structure:
```C++
struct Token
{
	enum class Type
	{
		REFERENCE,
		LABEL,
		COMMENT,
		STRING,
		NUMBER,
		OPERATOR
	};
	using TypeIntType = std::underlying_type_t<Type>;

    bool operator==(const Token&) const = default;

	Type type;
	std::size_t line;
	std::size_t start_column;
	std::size_t length;

	// Getter for the bindings
	Type get_type() const { return type; }
	// Little spoiler, I added this function because VSCode will only need an index :)
	TypeIntType get_type_index() const { return static_cast<TypeIntType>(type); }
	std::size_t get_line() const { return line; }
	std::size_t get_start_column() const { return start_column; }
	std::size_t get_length() const { return length; }

};
```

Then I can really create the bindings

```C++
#include <vector>
#include <string>

// Do not forget this inscmude
#include <emscripten/bind.h>

#include <PainPerdu/PainPerdu.hpp>

using namespace emscripten;

// Little helper functions
std::vector<PainPerdu::parser::Token> get_tokens(const std::string& input)
{
	return PainPerdu::Parser().get_tokens(input);
}

std::vector<std::string> get_defined_labels(const std::string& input)
{
	return PainPerdu::Parser().get_defined_labels(input);
}

std::vector<std::string> get_defined_references(const std::string& input)
{
	return PainPerdu::Parser().get_defined_references(input);
}

// The module
EMSCRIPTEN_BINDINGS(PainPerduParserModule) {
	// I need to declare the vectors
	register_vector<std::string>("VectorString");

	// And the Token class with all its getters
	class_<PainPerdu::parser::Token>("PainPerduToken")
		.function("get_type_index", &PainPerdu::parser::Token::get_type_index)
		.function("get_line", &PainPerdu::parser::Token::get_line)
		.function("get_start_column", &PainPerdu::parser::Token::get_start_column)
		.function("get_length", &PainPerdu::parser::Token::get_length);

	// Then declare what a vector of Token is
	register_vector<PainPerdu::parser::Token>("VectorToken");

	// Last step, declare functions I will use in javascript
	function("get_tokens", &get_tokens);
	function("get_defined_labels", &get_defined_labels);
	function("get_defined_references", &get_defined_references);

}
```

## The javascript implementation
### The setup
I just followed the [vscode tutorial](https://code.visualstudio.com/api/get-started/your-first-extension). To be honest if I go any deeper I would either be paraphrasing or be less clear than the tutorial itself. It is very clear and well written.

Basically, I just did:
```
npm install -g yo generator-code
yo code
```
And it generated the structure of the extension, then I created a folder to place my wasm code with its js glue code (generated from C++). In the rest of this article, my whole code will be in a file named `extension.js` and at the beginning it looks like this:
```js
// Include vscode stuff
const vscode = require('vscode');

// Use the bindings to my PainPerdu library
const bindings = require('./generated/ExtentionBindings')


/**
 * @param {vscode.ExtensionContext} context
 */
function activate(context) {

}

function deactivate() { }

module.exports = {
	activate,
	deactivate
}
```

### The semantic highlighting
As I said earlier, we need to provide a list of tokens to VSCode and it provides utility classes to build it: `SemanticTokensBuilder` and `SemanticTokensLegend`. 

The `SemanticTokensLegend` is just a little class describing the types of token that will be provided. There is a lot of predefined type of token like `variable` or `function`. You can see the list of predefined type of token or how to create your own in [the documentation](https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide). The `SemanticTokensLegend` is created from a list of string and I will use this legend for now on:
```js
const tokenTypeStrings = [
	'variable',
	'function',
	'comment',
	'string',
	'number',
	'operator'
];
const legend = new vscode.SemanticTokensLegend(tokenTypeStrings);
```

The `SemanticTokensBuilder` can then be used with the legend previously created:
```js
const builder = new vscode.SemanticTokensBuilder(legend);
builder.push(start_line, start_column, length, index_of_the_type);
return builder.build();
```
Note that VSCode considers than the line and columns of a document start at 0. Also for the type it uses an index, the index of `variable` is 0, `function` is 1, etc.

Now I need a `DocumentSemanticTokensProvider`, when using TypeScript it is possible to implement the interface `vscode.DocumentSemanticTokensProvider`. It must have a method named the provideDocumentSemanticTokens that returns the list of tokens. I used it like this:
```js
class DocumentSemanticTokensProvider {
    async provideDocumentSemanticTokens(document) {
        // Use the C++ code to create the list of tokens
        const allTokens = bindings.get_tokens(document.getText());
    
        // Create the builder using the lgend
        const builder = new vscode.SemanticTokensBuilder(legend);

        // Convert my list of tokens into one that VSCode understand
        for (var i = 0; i < allTokens.size(); ++i) {
            let token = allTokens.get(i);

            // In my C++ code, column and lines start at 1, but with VSCode it starts at 0
            builder.push(token.get_line() - 1, token.get_start_column() - 1, token.get_length(), token.get_type_index());
        }

        return builder.build();
    }
}
```

And the last step is to make VSCode use the `DocumentSemanticTokensProvider`:
```js
function activate(context) {
    // I want only to use it for my language
    const selector = { language: 'painperdu' };
    // Some boiler plate code 
    
context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider(selector, new DocumentSemanticTokensProvider(), legend));
}
```




Here's what it can look like with some code from my [Brainfuck interpreter written in PainPerdu](https://github.com/Baduit/LostBrain):
![PainPerdu code with color highlighting](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dfgkhwono0nnvyglpkbs.png)

### The auto-completion
Auto-completion almost work the same, except that instead of a list of tokens I need to return a list of element with their type: in this case either a variable (for a PainPerdu reference) or a function (for a PainPerdu label).

I will create 2 `GoCompletionItemProvider`, one giving the list of reference, the other one the list of labels:
```js
class GoCompletionItemProviderLabels {
    provideCompletionItems(document) {
        const labels = bindings.get_defined_labels(document.getText());

        let result = [];
        for (var i = 0; i < labels.size(); ++i) {
            let label = labels.get(i);
            let completionItem = new vscode.CompletionItem();
            completionItem.label = label;
            completionItem.kind = vscode.CompletionItemKind.Function;
            result.push(completionItem);
        }
        return result;
    }
}

class GoCompletionItemProviderReference {
    provideCompletionItems(document) {
        const refs = bindings.get_defined_references(document.getText());

        let result = [];
        for (var i = 0; i < refs.size(); ++i) {
            let ref = refs.get(i);
            let completionItem = new vscode.CompletionItem();
            completionItem.label = ref;
            completionItem.kind = vscode.CompletionItemKind.Variable;
            result.push(completionItem);
        }
        return result;
    }
}
```
And now I just need VSCode to use them:
```js
    const selector = { language: 'painperdu' };
    context.subscriptions.push(vscode.languages.registerDocumentSemanticTokensProvider(selector, new DocumentSemanticTokensProvider(), legend));
    // These characters can only be followed by a label

    context.subscriptions.push(vscode.languages.registerCompletionItemProvider(selector, new GoCompletionItemProviderLabels(), '.', '*', '&'));
    // These characters can only be followed by a reference
    context.subscriptions.push(vscode.languages.registerCompletionItemProvider(selector, new GoCompletionItemProviderReference(), '#', '@', '$', '>', '<', '+', '-', '?', '!'));
}
```

And voilà! It was that simple!

## Article::~Article
It is my very first VSCode extension and even if it may seem dumb, I'm proud of it. Don't hesitate to point some errors or some things I could have done better, I'm not very comfortable using Javascript and I don't know well the VSCode API.

The extension could be improved, but I doubt that anybody will really use this extension. The whole project around this language is just an excuse to try some tools and have some fun.

The extension is named _PainPerdu Bakery_ and you can find it on the [vscode marketplace](https://marketplace.visualstudio.com/items?itemName=Baduit.painperdu-bakery).

## Sources
* [https://code.visualstudio.com/api/references/vscode-api](https://code.visualstudio.com/api/references/vscode-api)
* [https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide](https://code.visualstudio.com/api/language-extensions/semantic-highlight-guide)
* [https://code.visualstudio.com/api/references/vscode-api#Terminal](https://code.visualstudio.com/api/references/vscode-api#Terminal)