JQ Recipes for HashiCorp Vault
Tags: vault, jq, api, bash
Published: 2020-10-07 22:48:50

---
title: JQ Recipes for HashiCorp Vault
published: true
description: Series of JQ Examples for working with the Vault API
tags: vault, jq, api, bash
cover_image: https://www.vaultproject.io/img/og-image.png
---

This post will show some recipes for working with the `jq` utility in Bash to get data from HashiCorp Vault Enterprise via its ReST API. 

You can install `jq` from here: 

https://stedolan.github.io/jq

And if you don't have a "Bash" environment, you can get one from here: https://git-scm.com ([make sure to select the "Git Bash" option when given the choice in the installation process](https://www.stanleyulili.com/git/how-to-install-git-bash-on-windows/))

If you're on Windows you might have some troubles or need to use some different name for `jq-64.exe`.

You also need to download and unzip Vault from https://vaultproject.io, and run it in a Git Bash like this: 

```
$ cd ~/Downloads
$ echo 'alias vault="~/Downloads/vault.exe"' >> ~/.bash_profile ##(Only do this if you're on Windows)
$ source ~/.bash_profile ## Again, needed only if you're on Windows
$ ./vault server -dev -dev-root-token-id=s.JzcMzvFCAVHvxE4xusglvoHF

==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls
: "disabled")
               Log Level: info
                   Mlock: supported: false, enabled: false
           Recovery Mode: false
                 Storage: inmem
                 Version: Vault v1.5.3
             Version Sha: 9fcd81405feb320390b9d71e15a691c3bc1daeef

WARNING! dev mode is enabled! In this mode, Vault runs entirely in-memory
and starts unsealed with a single unseal key. The root token is already
authenticated to the CLI, so you can immediately begin using Vault.

You may need to set the following environment variable:

PowerShell:
    $env:VAULT_ADDR="http://127.0.0.1:8200"
cmd.exe:
    set VAULT_ADDR=http://127.0.0.1:8200

The unseal key and root token are displayed below in case you want to
seal/unseal the Vault or re-authenticate.

Unseal Key: t/5BTQzZ5LbD0JIYkpQ9dY3fLEqZEMyena3aCh4+sWw=
Root Token: s.JzcMzvFCAVHvxE4xusglvoHF

Development mode should NOT be used in production installations!

==> Vault server started! Log data will stream in below:

2020-10-07T15:14:47.286-0700 [INFO]  proxy environment: http_proxy= https_proxy= no_proxy=
2020-10-07T15:14:47.289-0700 [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value s
hould be manually set
2020-10-07T15:14:47.327-0700 [INFO]  core: security barrier not initialized
2020-10-07T15:14:47.327-0700 [INFO]  core: security barrier initialized: stored=1 shares=1 threshold=1
2020-10-07T15:14:47.332-0700 [INFO]  core: post-unseal setup starting
2020-10-07T15:14:47.349-0700 [INFO]  core: loaded wrapping token key
2020-10-07T15:14:47.349-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.356-0700 [INFO]  core: no mounts; adding default mount table
2020-10-07T15:14:47.360-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.363-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.365-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.370-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.370-0700 [INFO]  rollback: starting rollback manager
2020-10-07T15:14:47.370-0700 [INFO]  core: restoring leases
2020-10-07T15:14:47.376-0700 [INFO]  expiration: lease restore complete
2020-10-07T15:14:47.376-0700 [INFO]  identity: entities restored
2020-10-07T15:14:47.376-0700 [INFO]  identity: groups restored
2020-10-07T15:14:47.377-0700 [INFO]  core: post-unseal setup complete
2020-10-07T15:14:47.380-0700 [INFO]  core: root token generated
2020-10-07T15:14:47.380-0700 [INFO]  core: pre-seal teardown starting
2020-10-07T15:14:47.380-0700 [INFO]  rollback: stopping rollback manager
2020-10-07T15:14:47.380-0700 [INFO]  core: pre-seal teardown complete
2020-10-07T15:14:47.381-0700 [INFO]  core.cluster-listener.tcp: starting listener: listener_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO]  core.cluster-listener: serving cluster requests: cluster_listen_address=127.0.0.1:8201
2020-10-07T15:14:47.381-0700 [INFO]  core: post-unseal setup starting
2020-10-07T15:14:47.381-0700 [INFO]  core: loaded wrapping token key
2020-10-07T15:14:47.381-0700 [INFO]  core: successfully setup plugin catalog: plugin-directory=
2020-10-07T15:14:47.382-0700 [INFO]  core: successfully mounted backend: type=system path=sys/
2020-10-07T15:14:47.383-0700 [INFO]  core: successfully mounted backend: type=identity path=identity/
2020-10-07T15:14:47.383-0700 [INFO]  core: successfully mounted backend: type=cubbyhole path=cubbyhole/
2020-10-07T15:14:47.384-0700 [INFO]  core: successfully enabled credential backend: type=token path=token/
2020-10-07T15:14:47.384-0700 [INFO]  rollback: starting rollback manager
2020-10-07T15:14:47.384-0700 [INFO]  core: restoring leases
2020-10-07T15:14:47.384-0700 [INFO]  identity: entities restored
2020-10-07T15:14:47.384-0700 [INFO]  expiration: lease restore complete
2020-10-07T15:14:47.384-0700 [INFO]  identity: groups restored
2020-10-07T15:14:47.384-0700 [INFO]  core: post-unseal setup complete
2020-10-07T15:14:47.384-0700 [INFO]  core: vault is unsealed
2020-10-07T15:14:47.401-0700 [INFO]  core: successful mount: namespace= path=secret/ type=kv
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: collecting keys to upgrade
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: done collecting keys: num_keys=1
2020-10-07T15:14:47.411-0700 [INFO]  secrets.kv.kv_8bdaeac5: upgrading keys finished
```

Leave Vault running in that terminal window, and open another one to run the following stuff in: 

`export VAULT_ADDR=http://127.0.0.1:8200`
`export VAULT_TOKEN=s.JzcMzvFCAVHvxE4xusglvoHF`

Now, to prepare further, create some auth methods: 

`./vault auth enable kubernetes`
`./vault auth enable userpass`
`./vault auth enable okta`
`./vault auth enable aws`

##Example 1: Get a list of all of the Auth Methods that are of type `okta`

Shameless theft from this SO dude: https://stackoverflow.com/questions/18592173/select-objects-based-on-value-of-variable-in-object-using-jq/31911811#comment112920282_18608100

First, look at the response without `jq`: 

`curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth`

Here's the output from that command: 

```
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth
{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config":{"default_lease_t
tl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"
description":"","external_entropy_access":false,"local":false,"options":null,"se
al_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"fo
rce_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descripti
on":"token based credentials","external_entropy_access":false,"local":false,"opt
ions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-0580
59545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default_l
ease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-servi
ce"},"description":"","external_entropy_access":false,"local":false,"options":nu
ll,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dcd5
84a"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"","external_entropy_access":false,"local":false,"options":null,"seal_wrap
":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894a"},"request_id
":"dc3dbab5-e001-498b-514c-05665baf09d2","lease_id":"","renewable":false,"lease_
duration":0,"data":{"kubernetes/":{"accessor":"auth_kubernetes_9bc9df5a","config
":{"default_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"
default-service"},"description":"","external_entropy_access":false,"local":false
,"options":null,"seal_wrap":false,"type":"kubernetes","uuid":"72df2827-2ac1-1fb1
-7d5a-c6262a4f5300"},"okta/":{"accessor":"auth_okta_fb869c57","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"okta","uuid":"1a445f3a-d302-ac6b-4508-def7a289894
a"},"token/":{"accessor":"auth_token_332e9370","config":{"default_lease_ttl":0,"
force_no_cache":false,"max_lease_ttl":0,"token_type":"default-service"},"descrip
tion":"token based credentials","external_entropy_access":false,"local":false,"o
ptions":null,"seal_wrap":false,"type":"token","uuid":"ba81c552-e8aa-9ba9-5243-05
8059545ccc"},"userpass/":{"accessor":"auth_userpass_1dd44d0b","config":{"default
_lease_ttl":0,"force_no_cache":false,"max_lease_ttl":0,"token_type":"default-ser
vice"},"description":"","external_entropy_access":false,"local":false,"options":
null,"seal_wrap":false,"type":"userpass","uuid":"e02a55d0-bd07-efeb-f5de-37fb5dc
d584a"}},"wrap_info":null,"warnings":null,"auth":null}
```

Pretty messy, right? Let's clean it up a bit: 

`curl --request GET --header "X-Vault-Token $VAULT_TOKEN" --header "X-Vault-Namespace $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq`

The above command will show the output in more of a structured format with indentation to indicate deeper levels of the object.

Here's the output from the above command: 

```
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   105k      0 --:--:-- --:--:-- --:--:--  105k
{
  "token/": {
    "accessor": "auth_token_332e9370",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "token based credentials",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "token",
    "uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
  },
  "userpass/": {
    "accessor": "auth_userpass_1dd44d0b",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "userpass",
    "uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
  },
  "okta/": {
    "accessor": "auth_okta_fb869c57",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "okta",
    "uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
  },
  "kubernetes/": {
    "accessor": "auth_kubernetes_9bc9df5a",
    "config": {
      "default_lease_ttl": 0,
      "force_no_cache": false,
      "max_lease_ttl": 0,
      "token_type": "default-service"
    },
    "description": "",
    "external_entropy_access": false,
    "local": false,
    "options": null,
    "seal_wrap": false,
    "type": "kubernetes",
    "uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
  },
  "request_id": "633adc4e-bd04-e579-58d4-34bde3dc91ee",
  "lease_id": "",
  "renewable": false,
  "lease_duration": 0,
  "data": {
    "kubernetes/": {
      "accessor": "auth_kubernetes_9bc9df5a",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "kubernetes",
      "uuid": "72df2827-2ac1-1fb1-7d5a-c6262a4f5300"
    },
    "okta/": {
      "accessor": "auth_okta_fb869c57",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "okta",
      "uuid": "1a445f3a-d302-ac6b-4508-def7a289894a"
    },
    "token/": {
      "accessor": "auth_token_332e9370",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "token based credentials",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "token",
      "uuid": "ba81c552-e8aa-9ba9-5243-058059545ccc"
    },
    "userpass/": {
      "accessor": "auth_userpass_1dd44d0b",
      "config": {
        "default_lease_ttl": 0,
        "force_no_cache": false,
        "max_lease_ttl": 0,
        "token_type": "default-service"
      },
      "description": "",
      "external_entropy_access": false,
      "local": false,
      "options": null,
      "seal_wrap": false,
      "type": "userpass",
      "uuid": "e02a55d0-bd07-efeb-f5de-37fb5dcd584a"
    }
  },
  "wrap_info": null,
  "warnings": null,
  "auth": null
}
```

Now, let's get all of the Auth Methods that are of type `okta`: 

`curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'`

The first part gets you the contents of the "data" part of the response. The second part, with the `'to_entries[]`, gets you the data in a format where you can read out the key names separately. The `| select(.value.type=="okta")` part gets you the auth methods that have the value `"okta"` under the key `type`. The `| .key'` part gets you the key of the auth method, usually ending in a `/`.

Here's the output for my example (should just be `"okta/"`): 

```
$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq .data | jq 'to_entries[] | select(.value.type=="okta") | .key'
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   125k      0 --:--:-- --:--:-- --:--:--  125k
"okta/"
```

##Example 2: How to get an accessor based on a Bash variable: 

`export OKTAAUTH='okta/'`

```
cat << EOF > jqparams.txt
."data"."${OKTAAUTH}/"."accessor"
EOF
```
`curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt`

Here's an example: 

```
$ export OKTAAUTH='okta/'

$ cat << EOF > jqparams.txt
> ."data"."${OKTAAUTH}"."accessor"
> EOF

$ curl --request GET --header "X-Vault-Token: $VAULT_TOKEN" --header "X-Vault-Namespace: $VAULT_NAMESPACE" $VAULT_ADDR/v1/sys/auth | jq -f jqparams.txt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2695    0  2695    0     0   125k      0 --:--:-- --:--:-- --:--:--  125k
"auth_okta_fb869c57"

$
```

##Example 3: How to get an Accessor Without a Bash Variable

In this case, it's the same as Example 2, except that you needn't bother with a `jqparams.txt` file. 

."data"."okta/"."accessor"

Further reading for the nerds: 

https://remysharp.com/drafts/jq-recipes

https://stedolan.github.io/jq/tutorial/

https://stedolan.github.io/jq/manual/