How to write your own language plugin for IDEA (part 1)
Tags: java, kotlin, idea, jetbrains
Published: 2022-04-26 18:22:12

Disclaimer: I don't work for JetBrains, so there will be and most likely there may be inaccuracies and errors in the article and code.

#Introduction
I am working as a backend developer, and sometimes I am tired of the work routine. Going deep into some opensource helps me, besides I am keen on Jetbrains' product and IDEA. IDEA can be customized with plugins, which you can write yourself.
The biggest part of IDEA's code is open-source. Developers from Jetbrains always help you to make pull requests and improve your main tool.
I have already written an article about how to write [a small plugin](https://dev.to/pyltsinm/how-to-create-a-new-inspecion-for-idea-10i1) with one inspection. But if you want to understand how IDEA works, you should try to write a language plugin.
Let's do this.
As a "guinea pig" we will take a simple  language - [Monkey](https://monkeylang.org/), the interpreter and compiler for which are described in the [books on Golang](https://interpreterbook.com/). Since my goal was not to cover everything, the plugin covers a limited subset of this language. The interpreter can be found [here](https://github.com/pyltsin/monkey-source).

Example of calculating the Fibonacci number on Monkey:

```java
let fibonacci = fn(x){
                    if (x == 0){
                        0;
                    }
                    else{
                        if (x == 1){
                            return 1;
                        }
                        else{
                            fibonacci(x - 1) + fibonacci(x - 2);
                        };
                    };
};
```

We will write this plugin with Java and Kotlin (JVM)

#Prerequisites
The easiest way to create a new plugin is to use [the template](https://github.com/JetBrains/intellij-platform-plugin-template). It is being actively developed and had already included the most needed features.

Also, we need DevKit plugin, which you can enable in IDEA
![Where you can find DevKit](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rkhd07pkgcjxijj7rzc6.png)
 
[Grammar-Kit](https://plugins.jetbrains.com/plugin/6606-grammar-kit) plugin will be used to generate a lexer and parser

Examples of other plugins:
- [Java](https://github.com/JetBrains/intellij-community) (in the code of IDEA)
- [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin.git) (now this plugin is known as Goland)
- [Haskell](https://github.com/carymrobbins/intellij-haskforce)
- [Erlang](https://github.com/ignatov/intellij-erlang/blob/master/grammars/erlang.bnf)
- [Frege](https://github.com/IntelliJ-Frege)
- [Monkey plugin](https://github.com/pyltsin/monkey-plugin) (in this article)

#Creating the basis of a language plugin

The first stages of creating a language plugin are well described in [the documentation](https://plugins.jetbrains.com/docs/intellij/language-and-filetype.html).

- It is required to declare a new language (examples in [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/GoLanguage.java), [frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/FregeLanguage.java), [monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/MonkeyLanguage.kt))

Example for Monkey:
```kotlin
import com.intellij.lang.Language

class MonkeyLanguage : Language("Monkey") {
    companion object {
        @JvmStatic
        val INSTANCE = MonkeyLanguage()
    }
}

```

- Declare an icon (examples in [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/GoIcons.java), [frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/FregeIcons.java), [monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/MonkeyIcons.kt))

- Declare a new file type and combine everything together (examples in [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/GoFileType.java), [frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/FregeFileType.java), [monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/MonkeyFileType.kt))

Example for Monkey
```kotlin
import com.intellij.openapi.fileTypes.LanguageFileType
import javax.swing.Icon

class MonkeyFileType : LanguageFileType(MonkeyLanguage.INSTANCE) {
    override fun getName(): String {
        return "Monkey File"
    }

    override fun getDescription(): String {
        return "Monkey language file"
    }

    override fun getDefaultExtension(): String {
        return "monkey"
    }

    override fun getIcon(): Icon {
        return MonkeyIcons.FILE
    }

    companion object {
        @JvmStatic
        val INSTANCE = MonkeyFileType()
    }
}
```
After that, you need to enable a new file type via the extension point. All the features that plugins provide are enabled through one or more extension points. They must be declared in the file plugin.xml (example for [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/resources/META-INF/gogland.xml), [frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/resources/META-INF/plugin.xml)). Other examples of using extension points will be given below or can be checked in the [documentation](https://plugins.jetbrains.com/docs/intellij/welcome.html).

Example for Monkey (resources/META-INF/plugin.xml)
```xml
<extensions defaultExtensionNs="com.intellij">
  <fileType name="Monkey File"
            implementationClass="com.github.pyltsin.monkeyplugin.MonkeyFileType"
            fieldName="INSTANCE"
            language="Monkey"
            extensions="monkey"/>
</extensions>
```
#Creation PSI-tree

It is impossible to describe the whole theory that is required to parse the code in one article. Fundamental knowledge for this topic can be found in the "[Dragon Book](https://en.wikipedia.org/wiki/Compilers:_Principles,_Techniques,_and_Tools)"
![Cover of "Dragon Book"](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lpwld9cljps0mxiiyaje.png)
 
According to this book, the process of the compiler working with the code consists of the following steps:
- Lexical Analyzer
- Syntax Analyzer
- Semantic Analyzer
- Intermediate Code Generator
- Machine-Independent Code Optimizer
- Code Generator
- Machine-Dependent Code Optimizer

It is required to implement the first 3 analyzers for any IDE to work successfully:
- Lexical analyzer (to read a stream of characters and to group them into tokens)
- Parser (to receive a stream of tokens and to build a syntax tree from them - [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree))
- Semantic analyzer (to use a tree to verify the source code according to some rules).

In IDEA, an analog called `PSI tree` (Program structure Interface) is used instead of an AST tree.

The process of creating a PSI tree is well shown in the illustration from the [documentation](https://plugins.jetbrains.com/docs/intellij/implementing-parser-and-psi.html):
![PSI tree](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/rxsh8l8bige992uw2dv6.png)

In order to see PSI-tree, you can use the [PSI Viewer](https://www.jetbrains.com/help/idea/psi-viewer.html) (Tools->View PSI structure)
 ![Example of usage of PSI Viewer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5v6k2btlp1ioooqgana3.png)

In IDEA, the abstract TreeElement class is mainly used to implement the PSI tree
```java
public abstract class TreeElement extends ElementBase implements ASTNode, Cloneable {
  private TreeElement myNextSibling;
  private TreeElement myPrevSibling;
  private CompositeElement myParent;
...
}
```

In IDEA, you can use the [GrammarKit](https://plugins.jetbrains.com/plugin/6606-grammar-kit) plugin to create a lexer and a parser.

#Lexer

Perhaps, the easiest way to create a lexer for IDEA is to use [JFlex](https://jflex.de/). The GrammarKit plugin already contains implementation and allows you to generate a lexer from `.bnf` file or from `.flex` file (with more customization options). An example for the Monkey language can be viewed [here](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/java/com/github/pyltsin/monkeyplugin/Monkey.flex), a more complex one for Frege - [here](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/lexer/FregeLexer.flex).

To generate Lexer, you need to configure the Gradle plugin or use the context menu in .flex file - "Run JFlex Generator".

After that, you need to declare a class that implements `com.intellij.lexer.Lexer`. There is already an adapter for the generated JFlex lexer - `com.intellij.lexer.FlexAdapter`

#Parser
In IDEA, code generation by the GrammarKit plugin is mainly used to create a parser. Unfortunately, there is not much documentation about it and it is only presented in [Tutorial](https://github.com/JetBrains/Grammar-Kit/blob/master/TUTORIAL.md) and [HOWTO](https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md).

The grammar of the language is described in [BNF](https://en.wikipedia.org/wiki/Backus%E2%80%93Naur_form). The only difference that is used is `::=` as "is".

Example grammar for expressions can be found [here](https://github.com/JetBrains/Grammar-Kit/blob/master/testData/generator/ExprParser.bnf)

As you can see, the bnf file consists of 2 parts: 
- the first part describes the meta-information (and the description of tokens, if flex files are not used)
- the second part describes the grammar itself.

Let's consider some parts of the meta-information:

`parserClass` - name and location of the generated parser class

`parserUtilClass` - a reference to a class containing a set of additional methods for the parser (usually the `com.intellij.lang.parser.GeneratedParserUtilBase` class or its inheritors)

`extends = <some class>` - a reference to the base class from which all PSI elements (tree nodes) will be inherited. Usually `com.intellij.extapi.psi.ASTWrapperPsiElement` or its inheritors.

`extends(<regexp for tree nodes>) = <psi-element>` (for example: `extends(".*expr")=expr`) - all psi elements will be inherited from the specified psi element.

`psiClassPrefix`, `psiImplClassSuffix` - respectively the prefix for classes and interfaces (usually by the name of the language) and the suffix for implementing interfaces (usually - `Impl`)

`psiPackage` and `psiImplPackage` are respectively a package for interfaces and their implementations.

`implements` - similar to `extends`, but for interfaces

`elementTypeHolderClass` - a generated storage for all types of elements

`elementTypeClass` - a class for types of elements (not generated, inheritors to `com.intellij.psi.tree.IElementType`)

`elementTypeFactory` - creating a factory for generating element types (used for Stub - about them below)

`psiImplUtilClass` - a class with a set of static methods that are used as an implementation of the required methods for psi elements. 
Let's suppose we have such lines (from go-plugin)
```java
ImportSpec ::= [ '.' | identifier ] ImportString {
  stubClass="com.goide.stubs.GoImportSpecStub"
  methods=[getAlias getLocalPackageName shouldGoDeeper isForSideEffects isDot getPath getName isCImport]
}
```
For that, we need to declare and implement a static method with the same name in `psiImplUtilClass`:
```java
public static String getAlias(@NotNull GoImportSpec importSpec)
```
After that `getAlias` will be generated as:
```java
  public String getAlias() {
    return GoPsiImplUtil.getAlias(this);
  }

```

Now let's move on to the `bnf` rules themselves. Modifiers can be used for each rule (for example, `private`, `fake`, and so on). Their description is given [here](https://github.com/JetBrains/Grammar-Kit#rule-modifiers). So, for example `private` in

```java
private boolean_group ::= xor_expr | between_expr | is_not_expr
```
indicates that the PSI element for `boolean_group` will not be generated.

If it is not possible to describe the grammar in a `bnf` file correctly, then it will be useful to describe it in code using external rules.

One of the important parts of grammar is the rules of dealing with errors. Two keywords are used for this: `pin` and `recoverWhile`.

`pin` - indicates the token number. As soon as we get to it, the parser starts waiting only for the current declaration. For example, declaring a structure in Golang
```java
Struct Type ::= struct '{' Fields? '}' {pin=1}
```

`recoverWhile` - specifies which tokens can be consumed only after matching with all rules is completed. Recommendations for the use of this attribute are described [here](https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md#22-using-recoverwhile-attribute).

You should also pay attention to the [recommendations](https://github.com/JetBrains/Grammar-Kit/blob/master/HOWTO.md#24-compact-expression-parsing-with-priorities) for parsing expressions based on priority.

It seems to me that creating correct and convenient grammar rules for future work is one of the most difficult parts of implementing a plugin for a language. To get started, you can use examples: [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/grammars/go.bnf), [Frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/Frege.bnf), [Monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/java/com/github/pyltsin/monkeyplugin/Monkey.bnf) (for Monkey, only a subset of this language is implemented for simplification).

After creating a `bnf` file and generating a parser from it, you need to define the class for `File` (inheritor from `com.intellij.extapi.psi.PsiFileBase`) (examples from [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/psi/GoFile.java), [Frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/kotlin/com/plugin/frege/psi/FregeFile.kt), [Monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/psi/MonkeyFile.kt)) and the definition class for your parser (heir from `com.intellij.lang.ParserDefinition`) (example from [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/GoParserDefinition.java), [Frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/java/com/plugin/frege/parser/FregeParserDefinition.java), [Monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/parser/MonkeyParserDefinition.kt)), and then enable it via the extension point.
```xml
<lang.parserDefinition language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.parser.MonkeyParserDefinition"/>
```


#Annotators

In the previous parts, we looked at how a lexer and a parser are created. Now let's move on to the third part - semantic analysis. Studying the code of IDEA and its plugins, I found two ways to implement it (excluding inspections).

The first way is used in a plugin for Java language. Consider the following invalid code:
![Example of invalid code](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/k354nf0ikebcp9zdzqnl.png)

IDEA highlighted it and said `Operator'-'cannot be applied to 'java.lang.String', 'java.lang.String'`. It works thanks to the following extension point: 
```xml
<highlightVisitor implementation=
"com.intellij.codeInsight.daemon.impl.analysis.HighlightVisitorImpl"/>
```
The class must implement `com.intellij.codeInsight.daemon.impl.HighlightVisitor`

```java
public interface HighlightVisitor {

  boolean suitableForFile(@NotNull PsiFile file);

  void visit(@NotNull PsiElement element);

  boolean analyze(@NotNull PsiFile file, 
                  boolean updateWholeFile, 
                  @NotNull HighlightInfoHolder holder, 
                  @NotNull Runnable action);
}
```

We use the method `analyze` to configure, launch (`action.run()`) and close resources

```java
//From HighlightVisitorImpl
	@Override
  public void visit(@NotNull PsiElement element) {
  // some code
      element.accept(this);
  // some code
  }

//Example for ClsJavaModuleImpl
  @Override
  public void accept(@NotNull PsiElementVisitor visitor) {
    if (visitor instanceof JavaElementVisitor) {
      ((JavaElementVisitor)visitor).visitModule(this);
    }
    else {
      visitor.visitElement(this);
    }
  }
```

The [visitor](https://en.wikipedia.org/wiki/Visitor_pattern) pattern is used there.
`HighlightVisitorImpl` extends `JavaElementVisitor`

```java
public abstract class JavaElementVisitor extends PsiElementVisitor {
  public void visitAnonymousClass(PsiAnonymousClass aClass) {
    visitClass(aClass);
  }

  public void visitArrayAccessExpression(PsiArrayAccessExpression expression) {
    visitExpression(expression);
  }

  public void visitArrayInitializerExpression(PsiArrayInitializerExpression expression) {
    visitExpression(expression);
  }
...
```

The second way is applied in go-plugin and Frege.
I used it too in the Monkey plugin. It uses an annotator extension point.
```xml
<annotator language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.annotator.MonkeyWarningAnnotator"/>
```
Your class must implement the next interface:

```java
public interface Annotator {
  void annotate(@NotNull PsiElement element, 
                @NotNull AnnotationHolder holder);
}
```

An error message can be registered with next way:
```java
holder.newAnnotation(HighlightSeverity.ERROR, errorMsg)
.range(element)
.create()
```

Examples for [Frege](https://github.com/IntelliJ-Frege/intellij-frege/blob/master/src/main/kotlin/com/plugin/frege/annotator/FregeWarningAnnotator.kt), [go-plugin](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/highlighting/GoAnnotator.java), [Monkey](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/annotator/MonkeyWarningAnnotator.kt).

For Monkey language, I have currently implemented 2 rules - the inability to resolve links (resolve references - about them below) and the simple type check (via [DSL](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/psi/impl/TypeResolver.kt)).

#Highlighting brackets
In this part, we'll look at a couple more extension points.

The first point of extension: `lang.braceMatcher`. 
```xml
<lang.braceMatcher language="Monkey"
implementationClass="com.github.pyltsin.monkeyplugin.editor.MonkeyBraceMatcher"/>
```
This extension point enables highlighting a pair of brackets and adding a closing bracket
![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lt2y18zszc5gel1s2zej.png)

The class must implement `com.intellij.lang interface.PairedBraceMatcher`

The implementation that I made for the Monkey language can be viewed [here](https://github.com/pyltsin/monkey-plugin/blob/main/src/main/kotlin/com/github/pyltsin/monkeyplugin/editor/MonkeyBraceMatcher.kt), for the go-plugin [here](https://github.com/go-lang-plugin-org/go-lang-idea-plugin/blob/master/src/com/goide/editor/GoBraceMatcher.java), for Java - [here](https://github.com/JetBrains/intellij-community/blob/master/java/java-impl/src/com/intellij/codeInsight/highlighting/JavaBraceMatcher.java) and [here](https://github.com/JetBrains/intellij-community/blob/master/java/java-impl/src/com/intellij/codeInsight/highlighting/JavaPairedBraceMatcher.java).

The second point of extension: `highlightVisitor`. I have already mentioned it when we were creating a semantic analyzer. I didn't use it in my plugin, but it is used in the popular [Rainbow Brackets](https://plugins.jetbrains.com/plugin/10080-rainbow-brackets) plugin, which colors pairs of brackets in unique colors.

![Rainbow Brackets](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e4ue69d4j0p19q960b22.png)

If you look in his plugin.xml, then you can find this line
```xml
<highlightVisitor implementation="com.github.izhangzhihao.rainbow.brackets.visitor.DefaultRainbowVisitor"/>
``` 

The class implements the interface - `com.intellij.CodeInsight.daemon.impl.HighlightVisitor`. The implementation can be viewed [here](https://github.com/izhangzhihao/intellij-rainbow-brackets/blob/2020.3/src/main/kotlin/com/github/izhangzhihao/rainbow/brackets/visitor/DefaultRainbowVisitor.kt). The coloring takes place in the `com.github.izhangzhihao.rainbow.brackets method.visitor.RainbowHighlightVisitor#setHighlightInfo`

```java
holder.add(HighlightInfo
    .newHighlightInfo(rainbowElement)
    .textAttributes(attr)
    .range(element)
    .create())
```
Continuation [here](https://dev.to/pyltsinm/how-to-write-your-own-language-plugin-for-idea-part-2-9g2-temp-slug-7793113).