Getting started with FX: Powerful and handy JSON manipulation from the command line
Tags: fx, json, scripting, node
Published: 2020-12-03 09:46:33

## Why this post?

If you spend a lot of your time-consuming APIs and having to **build data pipelines**, you will enjoy this post (if you don't know [**fx**](https://github.com/antonmedv/fx) yet).

According to several sources, **data scientists spend between 70-80% of their time normalizing data before they begin to play with it**. That's a lot of time, so it's a good time investment to have powerful tools at your disposal without a steep learning curve.

When we talk about data processing, one of my favorite anecdote is the following one: [Command-line Tools can be 235x Faster than your Hadoop Cluster](https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-scluster.html). Sometimes people spend thousands of dollars on new software without realizing they can do the same task faster and cheaper with some alternatives. You've got me, I'm an old-school command-line guy ;-)

The point I'm trying to make here is that there is a ton of tools out there, but my advice to you is: "**be conservative with your toolset**". Spend more time with the tools you use daily. If you detect a recurrent issue in several scenarios, then it's time to look for alternatives.

I used **jq** for a couple of years but every time I struggled with some data transformation, the pain came from the same place: *learning the concrete syntax for that tool*. In that sense, **fx** freed me from that inconvenience.


> Don't get me wrong. **jq** it's a very useful tool, but it takes time to control it. In my case, I don't want to spend more time learning a concrete syntax that is only useful with **jq**.

In this post, I'm not going to cover how to use [**jq**](https://stedolan.github.io/jq/), but if you're interested in it, here are some useful references:

- [jq: JSON like a Boss (talk)](https://www.youtube.com/watch?v=_ZTibHotSew)
- [JSON on the command line with jq](https://shapeshed.com/jq-json/)


## fx

### How to install

> **Prerequisite**: You have to install nodejs on your computer.

```bash
npm i -g fx
```

**fx** can do a lot of things (see *What can I do with fx* below), but let me first explain the two modes on which **fx** operates:

### fx modes

- **Interactive**: When you are not familiar with the data (JSON) you're playing with, this mode is quite useful because it lets you explore the data structure, find values, filter them, apply some transformation... Think **interactive** mode as a playground. Here you can see a sneak peak of fx in action:

![fx sneak peak in action](https://medv.io/assets/fx.gif)

> As you can see, it's pretty intuitive! You can explore any JSON data in the same way you do when you are accessing an object in Javascript. Bonus point: it supports auto-completion.

 - **Commandline (CLI)**: Once you know the data, it's time to apply some transformations. This mode can be used in scripts, it's pipe-friendly, so you can concatenate several fx commands in a one-liner. Think **cli mode** as a **grep**,**sed** or **awk** command, but **fx** reads JSONs instead of lines. Let's see another visual example:

![Applying some transformation](https://dev-to-uploads.s3.amazonaws.com/i/n5o8m09ffgjfeu1sttoh.gif)


> TIP: When you want to select text in interactive mode you need to press **Alt / Fn key** depending on your terminal

Once we saw the **available modes** of **fx**, let's practice with some examples.

## What can I do with fx

> If you want to follow the examples, just type in your terminal:

```bash
curl -sS "https://jsonplaceholder.typicode.com/users" -o users.json
```

### Quick JSON exploration

When you are in **interactive mode** you can search for strings or use regular expressions. If you are familiar with **vim editor** you'll feel at home. If you're not familiar with regular expressions, you can start just typing "**/**" followed by the string you are looking for.

> By default, the search is **case-insensitive**, so you don't have to worry about that.

To navigate across search results, just press Enter to go to the next match.

![search fx](https://dev-to-uploads.s3.amazonaws.com/i/4jtaucjnt28esa16hpam.gif)

> If you don't feel comfortable with regular expressions but you want to practice, I recommend you to take a look at [RegEx101](https://regex101.com/). It's a playground where you can start to master regular expressions.

### Transform

> From each user, I want to keep only the website and the "geo" keys.

Easy peasy! Because in **fx** we can use plain Javascript, let's translate this scenario to a Javascript code, and later apply this directly in **fx**:

For each user (object) we can make use of destructuring technique, to get the keys we want (website,geo) from the object and discard the rest:

```javascript
var user = {
  "id": 1,
  "name": "Leanne Graham",
  "username": "Bret",
  "email": "Sincere@april.biz",
  "address": {
    "street": "Kulas Light",
    "suite": "Apt. 556",
    "city": "Gwenborough",
    "zipcode": "92998-3874",
    "geo": {
      "lat": "-37.3159",
      "lng": "81.1496"
    }
  },
  "phone": "1-770-736-8031 x56442",
  "website": "hildegard.org",
  "company": {
    "name": "Romaguera-Crona",
    "catchPhrase": "Multi-layered client-server neural-net",
    "bs": "harness real-time e-markets"
};
```

```javascript
var {website,geo,...rest} = user; // user is our object
// website = "hildegard.org"
// geo = {"lat": "-37.3159", "lng": "81.1496"}
```

Now we want to apply this operation on each user, so let's do it with ***.map***

```javascript
users.map(({website,geo,...rest}) => ({website,geo}))
```

In **fx**, we will do it like this:

```bash
cat users.json | fx '.map(({website,geo,...rest}) => ({website,geo}))'
```

Isn't it beautiful?

You may think: "this is very basic stuff Jorge." Yep, indeed. But put yourself in the shoes of someone who has to do lots of different data transformations every day, or someone who is just getting insights from different  data sources each time... Do you think that person is going to write a script every time?

That's the beauty of **fx** for me, it lets you do things very quickly without the need to learn anything more!

### Filter

> From our JSON file, I want to filter the company names of those which email has the domain **.biz**

```bash
cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => user.company.name)'
```

Got it!

> Extra ball: Could I convert the results above in a **CSV format** (company and mail by line)?

```bash
cat users.json | fx '.filter(({email,...rest}) => /\.biz$/.test(email))' '.map(user => `${user.company.name};${user.email}`)' '.join("\n")'
```


### Use your favorite npm module along fx

**fx** offers a way to include npm modules in the execution context.

When you are dealing with data structures in Javascript, [lodash](https://lodash.com/) is a very handy option. Also, [dayjs](https://day.js.org/) let us play with dates and time data easily.

Let's see how to use it along with **fx**:

1. Create *.fxrc* file in `$HOME` directory, and require any packages or define global functions.

2. Install **lodash** and **dayjs** globally in your computer:

```bash
npm i -g lodash dayjs
```

3. Set `NODE_PATH` env variable. This step is **IMPORTANT** to allow **fx** to make use of globally installed packages.

```bash
export NODE_PATH=`npm root -g`
```

4. Put in your `.fxrc` file:

```js
Object.assign(global, require('lodash/fp'))
global.dayjs = require("dayjs");
```


> Now, let's play this scenario: I want to have a list of the 5 most recent github repositories I have that includes the day of the week I created each of them.

```bash
curl -sS "https://api.github.com/users/ntkog/repos" |  \
fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \
'sortBy("created_at")' \
'reverse' \
'take(5)' \
'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'
```

Let's look at each step (we can chain several transformations in **fx** as you can see)

> "\" at the end of the line it's just for separating one command into several lines

1. Get all info of my github repos and pipe it to **fx**
```bash
curl -sS "https://api.github.com/users/ntkog/repos" |
```

2. Keep only **name,created_at,clone_url** from each object
```bash
fx '.map(({name,created_at,clone_url,...rest}) => ({name,created_at,clone_url}))' \
```
3. Sort array by **created_at** key
```bash
'sortBy("created_at")' \
```

4. Invert the order of the results
```bash
'reverse' \
```

5. Take 5 objects
```bash
'take(5)' \
```

6. Add **weekDay** key to each object
```bash
'map(repo => ({...repo, weekDay : dayjs(repo.created_at).format("dddd")}))'
```

It's a very **expressive way to transform the data step by step**, don't you think?


## Explore more possibilities

I didn't find a lot of articles talking about **fx**, but [**this talk**](https://www.youtube.com/watch?v=ktfeRxKog98) from Антон Медведев gave me a lot of ideas on how to get the most out of **fx**.

Give **fx** a try, you won't regret it! Let me know if you find other tricks!  

Happy hacking :-)


<span>Photo by <a href="https://unsplash.com/@samthewam24?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Samuel  Sianipar</a> on <a href="https://unsplash.com/s/photos/pipes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></span>