Season finale: Dhall documentation generator
Tags: haskell, dhall, gsoc
Published: 2020-08-26 12:45:59

This is my last post and final report as part of GSoC 2020, working on the [Dhall documentation generator](https://summerofcode.withgoogle.com/projects/#5366051979657216)

# Initial problem

Dhall is a relatively new language. The only way to see how to use a Dhall package was to manually inspect the code and reading the comments. The [Dhall Prelude](https://github.com/dhall-lang/dhall-lang/tree/v17.1.0/Prelude) is a good example of this approach by adding a comment to its headers as shown in [this](https://github.com/dhall-lang/dhall-lang/blob/2cb2e1804aa5ba8379137df1fc1fd1e9afb17ec7/Prelude/List/concatMap.dhall#L1-L4) example. Some package doesn't even have comments, so you'd need to actually read the code to see how to use the package.

# Main contribution

As part of my GSoC project, I built `dhall-docs` that takes a Dhall package and generates HTML documentation by running static analysis over each file on the package. The tool has the following features:

* Takes a folder with Dhall files, extracts its headers and outputs HTML that will render that documentation ([#1845](https://github.com/dhall-lang/dhall-haskell/pull/1845))
* Renders the HTML with some simple CSS and JS that improves the UI/UX ([#1848](https://github.com/dhall-lang/dhall-haskell/pull/1848), [#1895](https://github.com/dhall-lang/dhall-haskell/pull/1895))
* The index's dhall files section includes the type of each file (extracted from source code) ([#1898](https://github.com/dhall-lang/dhall-haskell/pull/1898))
* Assert expressions are extracted and rendered in a dedicated section in the generated HTML ([#1899](https://github.com/dhall-lang/dhall-haskell/pull/1899))
* A comment's syntax specification was created to correctly handle indentation before using a Markdown parser ([#1929](https://github.com/dhall-lang/dhall-haskell/pull/1929))
* Renders the source code with jump-to-definition (JTD) capabilities on the following places:
  * Import expressions ([#1959](https://github.com/dhall-lang/dhall-haskell/pull/1959))
  * Let binding names ([#1966](https://github.com/dhall-lang/dhall-haskell/pull/1966))
  * Lambda argument names ([#1982](https://github.com/dhall-lang/dhall-haskell/pull/1982))
  * Record (literals and types) field names ([#1991](https://github.com/dhall-lang/dhall-haskell/pull/1991))

Besides the work outlined in the PRs of the above list, I made the following technical improvements to the `dhall-docs` package at the [`dhall-haskell`] repository:

* Add a [golden tests](https://softwareengineering.stackexchange.com/questions/358786/what-is-golden-files) setup to ensure that the generated documentation is preserved through code changes ([#1899](https://github.com/dhall-lang/dhall-haskell/pull/1899))
* Generate documentation for the [Prelude](https://hydra.dhall-lang.org/job/dhall-haskell/master/prelude-dhall-docs/latest/download/1/docs), [`dhall-kubernetes`](https://hydra.dhall-lang.org/job/dhall-haskell/master/kubernetes-dhall-docs/latest/download/1/docs), and the [test demo package](https://hydra.dhall-lang.org/build/70746/download/1/docs/) that the golden-test setup of `dhall-docs` uses.

The list of all PRs I authored on the [`dhall-haskell`] repository can be found [here](https://github.com/dhall-lang/dhall-haskell/pulls?q=is%3Apr+author%3Agerman1608+)

...and I made some upstream changes to the [Dhall Language Standard](https://github.com/dhall-lang/dhall-lang):

* Add more tests for record field labels ([#1013](https://github.com/dhall-lang/dhall-lang/pull/1013))
* And changes related to how does `dhall-docs` manages packages ([#1026](https://github.com/dhall-lang/dhall-lang/pull/1026), [#1053](https://github.com/dhall-lang/dhall-lang/pull/1053))

I openly and actively discussed things related to `dhall-docs` behavior not only with my mentors but with the wide community via Discourse posts. The most relevant posts were:

* [The first discussion about the `dhall-docs` goals and my introduction to the community](https://discourse.dhall-lang.org/t/dhall-docs-cli-utility/267)
* [The announcement of the first release of `dhall-docs`](https://discourse.dhall-lang.org/t/announcing-the-first-experimental-release-of-dhall-docs/300)

If you want to see my messages on discourse.dhall-lang.org, go [here](https://discourse.dhall-lang.org/u/german1608/activity)

...and to end this section, although it's not directly related to my project, I familiarized myself with the existing codebase by taking small issues on the community bonding period. I already talked about that in [this](https://dev.to/german1608/prelude-48a1) post.

All the source code can be found [here](https://github.com/dhall-lang/dhall-haskell/tree/master/dhall-docs). We already have a release of `dhall-docs` (on a [GitHub release](https://github.com/dhall-lang/dhall-haskell/releases/tag/1.34.0) and on [Hackage](https://hackage.haskell.org/package/dhall-docs)) __but__ it doesn't contain all features. We're waiting for [this release PR on the standard](https://github.com/dhall-lang/dhall-lang/pull/1059) to be merged to make another release of the [`dhall-haskell`] packages, including `dhall-docs`.

# Pending work

If we compare the work done vs. my proposal, the things that are missing are:

* Syntax-highlighting on rendered source code. We had it at the beginning, but adding jump-to-definition required changing how we render source code. It is still possible to add this with the current `fragments` function used for JTD
* Type-on-hover. We have a [WIP PR](https://github.com/dhall-lang/dhall-haskell/pull/2005) that I couldn't finish, although it's in good shape.
* Read comments from other places in the source-code. In this moment, `dhall-docs` only supports the Header comment of the file, and it would be great to add support for other places such as record fields, lambdas and function-type arguments. The most difficult part for this work is to properly preserve whitespace, but [we're making progress](https://github.com/dhall-lang/dhall-haskell/issues/145#issuecomment-660479611).
* Jump-to-definition on expressions that involve imports. We currently jump to another URL when the clicked expression is a relative and remote import. But things like `(./Import.dhall).field.deep` are not handled yet and it would be really useful. A common pattern seen on dhall packages is that they use a `package.dhall` file that contains every file in the package, and using the above example with it can improve navigability a lot

I'm not sad that these features weren't done. I made a couple of improvements on the Dhall parser to preserve more whitespace that `dhall-docs` needed for JTD and that took more weeks as expected. The best part is that the work done there can also be used to fix a [long-standing](https://github.com/dhall-lang/dhall-haskell/issues/145) issue on the `dhall format` command. It was a good trade-off, after all :smile:

# Screenshots & Demo package

The following are some screenshots of how the generated documentation looks. These were taken from the [`dhall-haskell`] CI/CD results for the `dhall-docs` execution on sample packages, and since the job is not updated with the latest [`dhall-lang`] commit, almost all links on relative imports are broken (`dhall-docs` doesn't report broken links)

The Dhall Prelude package index:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/yqcxff0uo297c65e60vu.png)

`Prelude/Bool/package.dhall` is essentially a sub-package inside `Prelude` and it re-exports all files inside `Prelude/Bool`

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/qsrutb9s5h0iolwa6viv.png)

The `Prelude/Bool/fold` is really complete as it shows the extracted header as markdown, assertions, and jump-to-definition capabilities:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/kwjk9zrh2czfekvahgx1.gif)

On the [test demo package](https://hydra.dhall-lang.org/build/70746/download/1/docs/) a good example of jump-to-definition on record fields is the "jump-to-definition with nested labels":

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/h4tmb5ek42hn91ztfeir.gif)

We can use relative imports links to navigate to another expression in the same package:

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/edq7tpiniacxo91gpmlz.gif)

# Final thoughts

I still remember that I literally yelled at my room after receiving the email from the GSoC team that my proposal was accepted, and a lot of things have changed since then. The German from that moment wouldn't believe all the things that He learned during these last 4 months, not only about Haskell but about Open Source projects.

The most important part of an Open Source project is its community: without it, a project is just a bunch of code. And it's difficult to be proactive (quoting from Gabriel, my mentor) if you're not really passionate or if you don't get any funds from it. Google's initiative on this program helps a lot of projects finish their ongoing issues, and to offer new issues for arising tools (such as Dhall, for instance).

Contributing to Open Source for the first time may be complicated, but once you get used to it it's really simple and you get a lot of benefits:

* Be a member of a community
* Help your favorite ecosystem grows
* Improve your career in technical and non-technical aspects

Before contributing to [`dhall-haskell`], my Haskell experiences came from university projects. I could say that those projects used ok-ish Haskell: I'm really proud of those. I didn't have somebody reviewing my code, and I felt that a lot of things could be improved and I didn't have any point of comparison.

When I cloned [`dhall-haskell`] and opened any file I knew that this was completely new to me. Things such as LANGUAGE pragmas were completely unknown. Even things like:

```haskell
{-# LANGUAGE OverloadedStrings #-}

l :: Data.Text.Text
l = "string literal"
```

confused me a lot at the beginning (how a `String` is compatible with `Data.Text.Text`???????), though right now it's really simple!

Working properly with Functors, Applicatives, and Monads (even implementing my own!) were things that I wanted to do so many time ago, and after a couple of PRs on the repository started to feel really natural (not a pro, yet).

I started creating more data-types to make illegal states unrepresentable, a thing that I felt afraid (I over-abused `Maybe` and `[]` on my old projects) and lenses, although I didn't use them a lot for `dhall-docs`, stopped to scaring me.

There are even more things that I could write, but I don't want this post longer than it currently is.

Although this is the end of my work on the GSoC program, I'm pretty sure that I'll keep contributing in the short-coming future

# Acknowledgments

To my mentors [Gabriel](https://github.com/Gabriel439), [Simon](https://github.com/sjakobi) and [Philip](https://github.com/Profpatsch): Thanks for the opportunity, for always being there and for your patience over all the project. Thanks for the guidance on the development of `dhall-docs` and for telling me about your experiences on Open Source.

To the Dhall community, to be proactive on their responses to my open-questions.

To [Haskell.org](https://haskell.org) for hosting the project and help me decide what project to choose :wink:

And last but not least: To Google for providing the GSoC program. It's a really good way to improve your career and help the Open Source community grow bigger and stronger.

Thanks :heart:

[`dhall-haskell`]: https://github.com/dhall-lang/dhall-haskell
[`dhall-lang`]: https://github.com/dhall-lang/dhall-lang