Data Exploration Quick Wins with VisiData
Tags: python, cli, visidata
Published: 2021-11-21 15:26:31

---
title: Data Exploration Quick Wins with VisiData
published: true
description: VisiData recently celebrated its fifth anniversary. It gets more capable and polished with every release, and hasn't lost its focus on quick exploration.
tags: python, cli, visidata
---

- [VisiData turns five!](#visidata-turns-five)
- [Tour of a quick win - browsing Python package information](#tour-of-a-quick-win---browsing-python-package-information)
	- [Step 1 - Browser Dev Tools](#step-1---browser-dev-tools)
	- [Step 2 - Throw it to VisiData](#step-2---throw-it-to-visidata)
	- [Step 3 - Make it easier for next time](#step-3---make-it-easier-for-next-time)
- [The compounding value of quick wins](#the-compounding-value-of-quick-wins)
- [References](#references)

### VisiData turns five!

[VisiData](https://www.visidata.org/) recently [celebrated its five-year anniversary](https://www.visidata.org/blog/2021/5-years-old/). The anniversary post modestly describes VisiData as an interactive tabulator, but for me the real prize comes later when talking about its support for a wide variety of data sources:

> [...] even some things you might not think of at first: like filesystem metadata and API results and packet captures. Because **everything is data**.

VisiData is useful to me in so many situations that "just throw it to VisiData" has become an instinct.

### Tour of a quick win - browsing Python package information

I was looking at the [Python Package Index](https://pypi.org/) recently to browse [release history](https://pypi.org/project/c7n/#history) for the [Cloud Custodian](https://cloudcustodian.io/) package. Some contributors were discussing the total number of releases, and I was curious about the count from the PyPI side. I noticed that the release history page _listed_ all releases but didn't show a total count. What next?

#### Step 1 - Browser Dev Tools

I was already in the browser, and the data I needed was right there. So I opened Firefox dev tools and found a usable CSS selector for a release label:

![CSS selector for a PyPI release](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/a571ibwvujpnghj4tw6i.png)

And checked how many times that class showed up on the page:

![Release count in the browser console](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3tsaresjmor6gc1163t0.png)

That answered my immediate question, and would have been a fine place to stop.

But I figured this would be a handy thing to check from the terminal. And PyPI has a [JSON API](https://warehouse.pypa.io/api-reference/json.html). That makes my VisiData sense start to tingle.

#### Step 2 - Throw it to VisiData

I can get a JSON blob of Python package details by hitting PyPI's `/pypi/<project>/json` endpoint. If I already know what I'm looking for, I might use a combination of [HTTPie](https://httpie.io/cli) and [jq](https://stedolan.github.io/jq/manual/) to get it. But in recon mode, I blindly throw it to VisiData:

```bash
vd https://pypi.org/pypi/c7n/json
```

And as is often the case, it just works! It gives me a single row of data for the c7n package, including a count of releases.

![Hitting the PyPI API with VisiData](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6dax4n8p7l153w8wf9b7.png)

The bits of nested data are all collapsed by default, which effectively provides a summary view. If I want to dig deeper I can open one of those cells in its own sheet, or expand its children into their own rows/columns.

Again, this would be a fine place to stop.

#### Step 3 - Make it easier for next time

Up until this point I wasn't really sure how useful VisiData would be for this use case. But it feels like a pretty good fit, and I want to make it a bit smoother for next time. VisiData is [extensible](https://www.visidata.org/docs/api/extensible.html), and one of the simplest ways to extend its functionality is to add little bits of helper code to a `~/.visidatarc` file. For example, I can use this snippet to tell VisiData how to handle a URL with a `pypi` scheme:

```python
from visidata import VisiData, vd

@VisiData.api
def openurl_pypi(vd, path, filetype='json'):
    sheet = vd.openPath(vd.Path(f'https://pypi.org/pypi/{path.name}/json'), filetype=filetype)
    sheet.name = f'pypi_{path.name}'
    return sheet
```

And then, I can run:

```bash
vd pypi://c7n
```

And VisiData knows what to do - I get the same sheet I saw before, only with a more descriptive name and less typing.

### The compounding value of quick wins

One great thing about VisiData is that it's easy to accrue quick wins over time, and their value can stack. As one example of this, consider that the JSON structure we get from the PyPI API has some embedded Markdown in it. That's viewable in VisiData as-is:

{% asciinema 450512 %}

But it certainly looks prettier rendered using something like [glow](https://github.com/charmbracelet/glow)!

{% asciinema 450513 %}

And that's quick to slot into my workflow because a while back I wanted to look at cell's with an external program. As in this post, I started with a little helper in `~/.visidatarc`. Eventually it seemed generally useful enough to justify a life outside my home directory, so I split it out into [vpager](https://github.com/ajkerrigan/visidata-plugins#vpager-open-long-cell-values-in-the-system-pager).

And that's how it can often go with VisiData. There's a beautiful exploration and evolution path, from local fiddling to personal helper code to full-fledged plugins. You can stick to the out-of-the-box experience, or take a ride on the extensibility train for as many stops as you want. VisiData doesn't judge, it just handles whatever you throw at it and leaves you with a lollipop and a smile. Enjoy :).

### References

- [Intro to VisiData Tutorial](https://jsvine.github.io/intro-to-visidata/index.html)
- [Ten Ways to Use VisiData](https://www.visidata.org/blog/2020/ten/)
- [VisiData Turns 5](https://www.visidata.org/blog/2021/5-years-old/)
- [VisiData's New Menu System](https://www.youtube.com/watch?v=QixtGeSbSLU)
- [My VisiData Plugins](https://github.com/ajkerrigan/visidata-plugins)