Data and System Visualization Tools That Will Boost Your Productivity
Tags: tooling, programming, devops, productivity
Published: 2022-06-13 18:50:45

As files, datasets and configurations grow, it gets increasingly difficult to navigate them. There are however many tools out there, that can help you to be more productive when dealing with large JSON and YAML files, complicated regular expressions, confusing SQL database relationships, complex development environments and many others.

## JSON

JSON is one of those formats which is nice for computers, but not for humans. Even relatively small JSON object can be pretty difficult to read and traverse, but there's a tool that can help!

![JSON Editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fq3rg192y3tnbx7usc9v.png)

[JSON Visio](https://github.com/AykutSarac/jsonvisio.com) is a tool that generates graph diagrams from JSON objects. These diagrams are much easier to navigate than the textual format and to make it even more convenient, the tool also allows you to search the nodes. Additionally, the generated diagrams can also be downloaded as image.

You can use the web version at <https://jsonvisio.com/editor> or also run it locally as [Docker container](https://github.com/AykutSarac/jsonvisio.com#-docker).

## Regular Expressions

Regular expressions (RegEx) are notorious for being extremely unreadable. There are 2 tools I recommend to help make sense of complex RegExes - first one them being <https://regex101.com/>:

![RegEx 101](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/2fq8y7ug4zuixbsgkiep.png)

Which can help you build and test RegExes, as well as break them down and identify its individual parts.

The second one is <https://regex-vis.com> which generates a graph from a RegEx which is very helpful for understanding what the expression actually does:

![RegEx-Vis](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/elxzf3yhzux38m551a9d.png)

## YAML

YAML, the language that was meant to be readable, expect it oftentimes isn't. AS we all know, long YAML documents with many levels of indentations can be very hard to navigate and troubleshoot.

To avoid spending unreasonable amount of time trying to find that one wrong indent, I recommend you use schema validation and let your IDE do all the work. You can use validation schemas from <https://schemastore.org/json> or custom schemas such as [these](https://github.com/instrumenta/kubernetes-json-schema) for Kubernetes to validate your files. These will work both with _JetBrains_ products (e.g. [Pycharm](https://www.jetbrains.com/help/pycharm/json.html#ws_json_using_schemas), [IntelliJ](https://www.jetbrains.com/help/idea/json.html#ws_json_using_schemas)) as well as _VSCode_ (see [this guide](https://developers.redhat.com/blog/2020/11/25/how-to-configure-yaml-schema-to-make-editing-files-easier#))

If you prefer to use `vim` as your editor, I recommend using custom formatting that can help you spot mistakes. My preferred configuration looks like so:

```bash
# Add this line to "~/.vimrc"
autocmd FileType yaml setlocal ai et cuc sw=2 ts=2
```

And the resulting formatting will look like:

![vim YAML](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cjazkovcwo4396wnx6bi.png)

On top of the above-mentioned tools, it's also a good idea to use YAML linter such [this one](http://www.yamllint.com/) or its [CLI equivalent](https://github.com/adrienverge/yamllint), which will validate and cleanup your documents.

Finally, if you're working with OpenAPI/Swagger YAML specs, then you can use <https://editor.swagger.io/> to view/validate/edit you schemas.

## SQL

There's a lot of software for working with relational databases, most of them however focus on connecting to database instances and running SQL queries. These capabilities are very handy, but don't help with the fact that navigating database with possibly hundreds of tables can be very difficult. One tool that can solve that is _Jailer_:

![Jailer Data Browser](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/ku334ol43f2yzppgwlis.png)

[Jailer](https://wisser.github.io/Jailer/data-browsing.html) is a tool which can - among other things - navigate through the database by following foreign keys. Couple videos showcasing all the features of Jailer can be found [here](https://wisser.github.io/Jailer/videos.html).

## Git

Git - a software with absolutely terrible user experience which we all use every day - could also use some tooling for navigating history (log), staging/committing files, viewing diffs or for example rebasing branches.

My tool of choice for all the above is _IntelliJ/PyCharm_ `git` integration - in my opinion there's really no better tool for dealing with git-related stuff.

If you're not an IntelliJ/PyCharm user or if you prefer to stay in terminal, then following commands can make your life a little bit easier:

```bash
git log --graph --abbrev-commit --decorate --all \
    --format=format:"%C(bold blue)%h%C(reset) - %C(bold cyan)%aD%C(dim white) \
    - %an%C(reset) %C(bold green)(%ar)%C(reset)%C(bold yellow)%d%C(reset)%n %C(white)%s%C(reset)"
```

The above command will print a somewhat readable and pretty graph of commits history, which would look something like:

![git log](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/33idsgfk2qiidr8lzktu.png)

If you also desire a bit improved diffing functionality, then you can use `git diff ... --word-diff --color-words`:

![git word diff](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dgx6ikdyllxvqt44tvf8.png)

As you can see from above 2 examples, with enough effort you might be able to make commandline `git` experience somewhat bearable, for more tips like these see my previous article at <https://martinheinz.dev/blog/43>.

If you want to avoid bare-bones `git` CLI altogether, you can try using `forgit` - a TUI for using `git` interactively:

![Forgit git log](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/i69tl2afdme7cxekictv.png)

The above screenshot shows interactive and pretty `git log`. The tool supports pretty much all of the `git` subcommands, most importantly `rebase`, `cherry-pick`, `diff` and `commit`. For full list of features and more screenshots see project's [repository](https://github.com/wfxr/forgit#-features).

Beyond using `git` or your IDEs features you can also grab other tools that can help with complex diffing of files. One very powerful tool for this is _[Difftastic](https://github.com/Wilfred/difftastic)_. Difftastic language-aware diffing tools which has support for wide range of [languages](https://difftastic.wilfred.me.uk/languages_supported.html). For demo of the tool see [this video](https://github.com/Wilfred/difftastic#one-minute-demo).

If you want a tool specifically for diffing structured languages like JSON, XML, HTML, YAML, then _[Graphtage](https://github.com/trailofbits/graphtage)_ is a great choice. Graphtage compares documents semantically and will find differences correctly even when you change ordering of elements. 

Finally, if you're more into visual tools, then you might find [Meld](https://meldmerge.org/) useful as it provides similar experience to JetBrains' products.

## Docker

On the DevOps side of things, when working with Docker, it's not uncommon to spin up bunch of containers with `docker-compose` and end-up with a mess that's very hard to troubleshoot.

[Lazydocker](https://github.com/jesseduffield/lazydocker) is a great tool for dealing with multiple Docker containers at the same time. If you don't believe me, then check out the [_"elevator pitch"_](https://github.com/jesseduffield/lazydocker#elevator-pitch) in the project's repository.

![lazydocker](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/5ktkhfajvotwnthei9fk.png)

If you're more into browser-based tools you might want to try out [Portainer](https://docs.portainer.io/v/ce-2.11/user/docker/containers/view) which provides dashboard for navigating/inspecting Docker containers, volumes, images, etc.

![Portainer](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/e2djeav78vcfla1bf9zh.png)

## Kubernetes

A lot of things to cover when it comes to Kubernetes, considering that each API resource could use a visualization tool. There are however, a couple notable areas (pain points really), that oftentimes require visual tool to setup/manage effectively.

First one being _NetworkPolicy_ which can be visualized and also configured using <https://editor.cilium.io/>. Even if you prefer to craft your policies by hand, I would still recommend visually verifying them with this tool.

![NetworkPolicy Editor](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/lomc4rhjk6172syfy84e.png)

Another similar tool is [Network Policy Viewer](https://artturik.github.io/network-policy-viewer/), which focuses just on the visualization of policies and has simpler and more readable diagrams.

I'd recommend using [this collection](https://github.com/ahmetb/kubernetes-network-policy-recipes) of network policy recipes to test out these 2 tools and see how they can be helpful to your workflow.

Another pain point in configuring Kubernetes is RBAC, more specifically _Roles_, _RoleBindings_ and their _Cluster_-scoped alternatives. There are a couple of tools that can help with those:

[Krane](https://github.com/appvia/krane) is a tool that can generate graph showing relationships between all roles and subjects. Krane also has many more features, including RBAC risk assessment, reporting and alerting, as well as querying/interrogating RBAC rules with CypherQL.

A simpler alternative to Krane is [`rbac-tool`](https://github.com/alcideio/rbac-tool), which can be installed as `kubectl` plugin. It can also analyze, audit, interrogate RBAC rules, but most importantly, it can visualize them:

![rbac-tool](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/nlr0s92roqa4ik84turh.png)

Finally, if you're more concerned with easily configuring RBAC rather than viewing pretty diagrams, then [Permission manager](https://github.com/sighupio/permission-manager) is a tool for you. See [GIFs in GitHub repository](https://github.com/sighupio/permission-manager/blob/master/docs/assets) for demonstration of what the tool can do.

Instead of specialized tools for things like network policies or RBAC you can also make use of general purpose dashboards, such as:

- [Lens](https://github.com/lensapp/lens) - the Kubernetes IDE which brings some sanity into managing clusters, especially when paired with [Lens Resource Map](https://github.com/nevalla/lens-resource-map-extension/) which displays Kubernetes resources and their relations as a real-time force-directed graph.

![Lens](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/cl1pg4leejkkh6mz4yh0.png)
![Lens Resource Map](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/byf9ancwgakrr2cjy996.png)

- As usual, there's also CLI based tool which provides more capabilities than the bare `kubectl`. It's called [`k9s`](https://k9scli.io/) and definitely does make it easier to navigate, observe and manage your deployed applications in Kubernetes:

![k9s](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/p2dt5x02h4cog6qkeg8b.png)

## Closing Thoughts

This article focused on developer/DevOps tooling, but there are 2 more things - honorable mentions - that deserve a shout-out. First being [Mermaid.js](https://github.com/mermaid-js/mermaid) for creating beautiful diagrams (as a code), which is [now integrated with GitHub markdown](https://github.blog/2022-02-14-include-diagrams-markdown-files-mermaid/). The other one - _MathJax_ - for visualizing mathematical expressions, as of recent [also supported by GitHub markdown](https://github.blog/2022-05-19-math-support-in-markdown/).

These are the tools I like or the tools I made a use of in the past, so it's by no means an exhaustive list of things out there. If you have found different (better?) tools, please do share, so others can make use of them too.