Giving --format to your gcloud output
Tags: googlecloud, gcloud
Published: 2019-08-13 23:05:37


When using the Google Cloud command-line tool ([gcloud cli](https://cloud.google.com/sdk/gcloud/)), you often get various forms of output, depending on the command called.

For example, a general command, like listing your current gcloud configuration:

```
$ gcloud config list
[core]
account = katie@glasnt.com
disable_usage_reporting = True
project = glasnt-unicodex
[run]
region = us-central1
```


Or something more specific, like which GCP services are currently enabled for your project: 

```
$ gcloud services list --enabled
NAME                              TITLE
bigquery-json.googleapis.com      BigQuery API
cloudapis.googleapis.com          Google Cloud APIs
cloudbuild.googleapis.com         Cloud Build API
...
```

But there are ways to alter this output in order to get consistent, more context useful results. In just these two examples, we had two very different formats being presented by default. 

If you're after a consistent, specific format, or are only interested in a subsection of the output, using a `--format` option could be useful for you!

---
> Each API is different, but using these steps will help you introspect each on the fly, and help you build a command that will help you quickly. 
---

For all output here, we'll be using output from the most up-to-date version of `gcloud`. Our terminal input will be prefaced with `$`, and output will be as is, except where specifically truncated for clarity, which will be marked with `...`. For example: 

```
$ gcloud --version
Google Cloud SDK 257.0.0
alpha 2019.05.17
beta 2019.05.17
...
```

Let's start with that services listing from earlier.

```
$ gcloud services list
NAME                              TITLE
bigquery-json.googleapis.com      BigQuery API
cloudapis.googleapis.com          Google Cloud APIs
cloudbuild.googleapis.com         Cloud Build API
...
```

What is this output, anyway? `--help` can give us more insight.

```
$ gcloud services list --help

NAME
    gcloud services list - list services for a project

SYNOPSIS
    gcloud services list [--available | --enabled] [--filter=EXPRESSION]
        [--limit=LIMIT] [--page-size=PAGE_SIZE] [--sort-by=[FIELD,...]]
        [GCLOUD_WIDE_FLAG ...]

DESCRIPTION
    This command lists the services that are enabled or available to be enabled
    by a project. You can choose the mode in which the command will list
    services by using exactly one of the --enabled or --available flags.
    --enabled is the default.

FLAGS
     At most one of these may be specified:

       --available
          Return the services available to the project to enable. This list
          will include any services that the project has already enabled.

       --enabled
          (DEFAULT) Return the services which the project has enabled.

LIST COMMAND FLAGS
     --filter=EXPRESSION
        Apply a Boolean filter EXPRESSION to each resource item to be listed.
        If the expression evaluates True, then that item is listed. For more
        details and examples of filter expressions, run $ gcloud topic filters.
        This flag interacts with other flags that are applied in this order:
        --flatten, --sort-by, --filter, --limit.

     --limit=LIMIT
        Maximum number of resources to list. The default is unlimited. This
        flag interacts with other flags that are applied in this order:
        --flatten, --sort-by, --filter, --limit.

     --page-size=PAGE_SIZE
        Some services group resource list output into pages. This flag
        specifies the maximum number of resources per page. The default is
        determined by the service if it supports paging, otherwise it is
        unlimited (no paging). Paging may be applied before or after --filter
        and --limit depending on the service.

     --sort-by=[FIELD,...]
        Comma-separated list of resource field key names to sort by. The
        default order is ascending. Prefix a field with ``~'' for descending
        order on that field. This flag interacts with other flags that are
        applied in this order: --flatten, --sort-by, --filter, --limit.

GCLOUD WIDE FLAGS
    These flags are available to all commands: --account, --billing-project,
    --configuration, --flags-file, --flatten, --format, --help,
    --impersonate-service-account, --log-http, --project, --quiet,
    --trace-token, --user-output-enabled, --verbosity. Run $ gcloud help for
    details.

EXAMPLES
    To list the services the current project has enabled for consumption, run:

        $ gcloud services list --enabled

    To list the services the current project can enable for consumption, run:

        $ gcloud services list --available

NOTES
    These variants are also available:

        $ gcloud alpha services list
        $ gcloud beta services list
```



Yes, this is a long output. But let's step through it. 

The FLAGS section here is super useful as it shows us that the DEFAULT for the `gcloud services list` command is to show us only enabled output. 

If we ask it for all available services, we get a very different list.

```
$ gcloud services list --available
NAME                                                  TITLE
abusiveexperiencereport.googleapis.com                Abusive Experience Report API
acceleratedmobilepageurl.googleapis.com               Accelerated Mobile Pages (AMP) URL API
...
```

This list is really long. How long? We can check by piping the command through utility that will [count the number of lines](https://linux.die.net/man/1/wc).

```
$ gcloud services list --available | wc -l
244
```

What would be useful here is a list of services showing what is enabled and what is not enabled. At the moment we can only get either: all enabled services, or all available services. It would be useful to see all available services, and if they are enabled or disabled. 

---

We can see from earlier that we have a series of GCLOUD WIDE FLAGS, and the one we're going to focus on today is `--format`.

```
$ gcloud help
...
     --format=FORMAT
        Set the format for printing command output resources. The default is a
        command-specific human-friendly output format. The supported formats
        are: config, csv, default, diff, disable, flattened, get, json, list,
        multi, none, object, table, text, value, yaml. For more details run $
        gcloud topic formats.
...
```

Depending on which data format is most readable to you, you can use a format that gives you a deep look at the data structure for the command in question, which we can then introspect to get the values we want to focus on (in our case, service name, and it's state (status? state? Let's find out!))

```
$ gcloud services list --available --format=json

[
  {
    "config": {
      "authentication": {},
      "documentation": {
        "summary": "A data platform for customers to create, manage, share and query data."
      },
      "name": "bigquery-json.googleapis.com",
      "quota": {},
      "title": "BigQuery API",
      "usage": {
        "requirements": [
          "serviceusage.googleapis.com/tos/cloud"
        ]
      }
    },
    "name": "projects/1009201337834/services/bigquery-json.googleapis.com",
    "state": "ENABLED"
  },
   {
    "config": {
      "authentication": {},
      "documentation": {
        "summary": "This is a meta service for Google Cloud APIs for convenience. Enabling this service enables all commonly used Google Cloud APIs for the project. By default, it is enabled for all projects created through Google Cloud Console and Google Cloud SDK, and should be manually enabled for all other projects that intend to use Google Cloud APIs. Note: disabling this service has no effect on other services.\n"
      },
      "name": "cloudapis.googleapis.com",
      "quota": {},
      "title": "Google Cloud APIs",
      "usage": {
        "requirements": [
          "serviceusage.googleapis.com/tos/cloud"
        ]
      }
    },
    "name": "projects/1009201337834/services/cloudapis.googleapis.com",
    "state": "ENABLED"
  },
...
```

We can see that we have an array of objects (a list of dictionaries, a number of rich objects; your descriptive terms may vary). We can see, though, the fields that will be useful for us: title, and state. 

One of the formatting options we have is "value", a tab-separated list of values. This is good as a short-form for debugging our query, as it's easily read in the terminal. 

It's also useful to read up the Formats section of Formats (as suggested earlier in the `--format` tag description). 


```
$ gcloud topic formats

...
    A format expression has 3 parts:

     NAME
        name
     ATTRIBUTES
        [ [no-]attribute-name[=value] [, ... ] ]
     PROJECTION
        ( resource-key [, ...] )

    NAME is required, ATTRIBUTES are optional, and PROJECTIONS may be required
    for some formats. Unknown attribute names are silently ignored.
...
```

The `value` format type has an attribute "separator", which we need to specify in the following format: 

```
--format='value[separator=","](thing,anotherthing...)'
```

**Important note here**: the parameter we pass to `--format` is a complex string, so it must be wrapped in quotation marks. If we don't do this, our terminal will get _very_ confused with the parentheses and brackets going on. And since we have to declare the comma as the separator, we need to use _another_ set of quotation marks. We could also escape our quotation marks, but this version looks cleaner. 

So if we want title and state, we need to specify the fields by their full names: 
 - config.title, and
 - state

```
$ gcloud services list --available --format="value[separator=' → '](config.title,state)"

Abusive Experience Report API → DISABLED
Accelerated Mobile Pages (AMP) URL API → DISABLED
...
BigQuery API → ENABLED
...
```

In this last example, I used an unusual separator just to emphasise the breadth of what you could use. 

The output format you want is greatly going to depend on what is going to be consuming  the output. 

A shell script? Maybe it should be `--format=csv` with limited values.

An internal README? `--format=table` can be quite cool. 

```
$ gcloud services list --available \
      --format="table[box,margin=3,title='✨ Enabled Services ✨'](config.title,state)"

┌───────────────────────────────────────────────────────────────┐
│                     ✨ Enabled Services ✨                    │
├────────────────────────────────────────────────────┬──────────┤
│                       TITLE                        │  STATE   │
├────────────────────────────────────────────────────┼──────────┤
│ Abusive Experience Report API                      │ DISABLED │
│ Accelerated Mobile Pages (AMP) URL API             │ DISABLED │
...

│ BigQuery API                                       │ ENABLED  │
...

│ Zync Render API                                    │ DISABLED │
└────────────────────────────────────────────────────┴──────────┘

```


The `gcloud topic formats` help also gives a number of EXAMPLES of useful calls. For example:

```
    List the project authenticated user email address:

        $ gcloud info --format="value(config.account)"

    List the URIs for all compute instances:

        $ gcloud compute instances list --format="value(uri())"
```


Hopefully using these discovery tips can help you introspect your system with relative and just-in-time-discovery ease!

---

You can read more about using `gcloud --format` on the [GCP Blog.](https://cloud.google.com/blog/products/gcp/filtering-and-formatting-fun-with)