Exploring AWS from the terminal with VisiData
Tags: aws, python, cli, visidata
Published: 2020-10-15 03:25:46

---
title: Exploring AWS from the terminal with VisiData
published: true
description: VisiData is an interactive data analysis tool for your terminal. Let's dig into one specific use case - exploring data files in S3.
tags: aws, python, cli, visidata
---

### VisiData for Exploratory DevOps

The folks behind [VisiData](https://www.visidata.org) recently [released version 2.0](https://www.visidata.org/blog/2020/v2.0/).

VisiData is a tool I wish I found sooner - a tool for quickly exploring many types of files from your terminal. To celebrate its 2.0 release I want to help show some ways it's helped me. Many of VisiData's uses are well-covered by [tutorials](https://jsvine.github.io/intro-to-visidata/), [video case studies](https://www.youtube.com/playlist?list=PLxu7QdBkC7drrAGfYzatPGVHIpv4Et46W) and [documentation](https://www.visidata.org/docs/). I'm not a data scientist though, so I'll start somewhere totally different. Let's explore S3!

### Exploring S3 from the Terminal

I often want to quickly peek at data files in S3. With VisiData I can browse a bucket in a way that recalls terminal file explorers like Midnight Commander. I can open files (CSV, deeply nested JSON, Parquet...) directly as new sheets in VisiData's tabular interface.

Here's a demo that uses VisiData to explore weather data in a versioned S3 bucket:

{% asciinema 365458 %}

### Extensible with Python

Most of the functionality in the demo above comes out of the box with VisiData. The only work I needed to do was create a [plugin](https://github.com/ajkerrigan/visidata-plugins#vds3-open-amazon-s3-paths-and-objects) that tells VisiData how to speak S3. And to be fair, that's mostly a wrapper around the excellent [s3fs](https://github.com/dask/s3fs/) library! VisiData 2.0 includes a [Plugin Author's Guide](https://www.visidata.org/docs/api/) that can help you extend VisiData as you see fit.

### VisiData as a Piece of the Puzzle

Sometimes if I'm on a screenshare, people notice VisiData and ask about it. The strange part is that as much as I love VisiData, it doesn't actually _replace_ other tools I use. It's replaced specific uses for sure...

* [jq](https://stedolan.github.io/jq/): I still use and love it, but VisiData feels smoother for interactive exploration.
* [glom](https://glom.readthedocs.io/): Rather than inspecting/exploring data with glom _first_, I explore with VisiData and potentially use glom to define repeatable transforms.

VisiData also works very well as a side "quick view" tool to consume output from other tools I use regularly:

* [Terraform](https://www.terraform.io/): Add a few lines of Python to your `~/.visidatarc` file to [automatically open .tfstate files as JSON](https://gist.github.com/ajkerrigan/3f18a9603caf5a9f190b4de8b2c03a33).
* [Cloud Custodian](https://cloudcustodian.io/docs/index.html): Open local policy output files or use [S3 glob matching](https://asciinema.org/a/340202) to check output for a range of dates.
* [AWS CLI](https://aws.amazon.com/cli/): Pipe its JSON output to `vd -f json`.

Some of those use cases deserve their own post! The gist is that if you feel at home in the terminal and need a tool for ad-hoc data exploration and analysis... VisiData is probably worth a peek. It'll do as much or as little as you need, and let you get on with your life.