Getting started using Google APIs: Workspace & OAuth client IDs (3/3)
Tags: python, googlecloud, node, googleapi
Published: 2023-06-12 15:30:00

<!-- Getting started using Google APIs: Workspace (Part 3) -->

## Introduction

Are you a developer but a complete beginner using Google APIs? This series is for you because I'm showing you how get started from scratch, beginning with the Google Workspace ("GWS") APIs like Google Drive and Sheets. The [first post](https://dev.to/wescpy/getting-started-using-google-apis-workspace-9a8) covered the base requirements for using _any_ Google APIs: _authentication_ ("authn") and _authorization_ ("authz"). The authn requirements are met by having a Google account while the authz requirements are met by having a [developer project](https://cloud.google.com/resource-manager/docs/creating-managing-projects) with appropriate application [credentials](https://developers.google.com/workspace/guides/create-credentials#credential_types).

The [second post](https://dev.to/wescpy/getting-started-using-google-apis-workspace-23-3ch8) outlined the next steps to using Google APIs: choosing a development language and APIs to use, installing the appropriate client libraries, and enabling the selected API(s) in the DevConsole. For programming language, I picked **Python** and **Node.js** and chose the [Google Drive API](http://developers.google.com/drive) to craft a simple script that dumps out the first 100 files/folders in your Google Drive. Once the Drive API is enabled and client libraries installed, you'll be ready to code.

I'm excited about this third and longest post, not just because it completes the 3-part series, but because I can finally show some code! While I'm most comfortable with Python, I'm also learning Node, so I'm happy to be able to demo the "same" script in both languages. But, let's start with Python first.


## Python

This next section describes the major sections of the Python script. The code is described in more detail in the codelab at <http://g.co/codelabs/gsuite-apis-intro>, however, it's based on the older Python auth libraries whereas all the [Drive API documentation](http://developers.google.com/drive) has switched to the newer libraries. Regardless, this post give you enough to understand how the script [`python/drive_list-new.py`](http://github.com/wescpy/gsuite-apis-intro/python/drive_list-new.py) works.


### Imports

```python
from __future__ import print_function
import os.path

from google.auth.transport.requests import Request
from google.oauth2 import credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient import discovery
```

The standard library `import`s come first, with `__future__.print_function` enabling Python 2-3 compatibility by bringing in the Python 3 `print()` function to Python 2 apps. (This import is ignored in Python 3.) The other brings in file path utilities for managing OAuth tokens.

The next block imports all the packages necessary to talk to GWS APIs, including the Google APIs client library (`googleapiclient`) and the required security libraries.


### Security

#### Code snippet

```python
creds = None
SCOPES = 'https://www.googleapis.com/auth/drive.metadata.readonly'
TOKENS = 'storage.json'
if os.path.exists(TOKENS):
    creds = credentials.Credentials.from_authorized_user_file(TOKENS)

if not (creds and creds.valid):
    if creds and creds.expired and creds.refresh_token:
        creds.refresh(Request())
    else:
        flow = InstalledAppFlow.from_client_secrets_file(
                'client_secret.json', SCOPES)
        creds = flow.run_local_server()

with open(TOKENS, 'w') as token:
    token.write(creds.to_json())
```


#### Permission scopes

`SCOPES` represents the permission(s) an app will request from the end-user; it's either a single string or an array of such strings (Python `list`). In this case, it's one string representing the Google Drive metadata read-only _permission scope_. While it looks like a URL, it is translated to a sentence in the language specified by your browser's or computer's locale. In English, it would be: **"See information about your Google Drive files."** For your own apps, the scope(s) will differ depending on what GWS APIs it uses.

For example, I've got another script that accesses the Drive and Sheets APIs as well as the Google Cloud (GCP) Storage and Vision APIs (see [this blog post](http://goo.gle/3nPxmlc)) where the [sample app](https://github.com/googlecodelabs/analyze_gsimg/blob/master/final/analyze_gsimg.py#LL41C1-L46C2)'s scopes look like this:

```python
SCOPES = (
    'https://www.googleapis.com/auth/drive.readonly',
    'https://www.googleapis.com/auth/devstorage.full_control',
    'https://www.googleapis.com/auth/cloud-vision',
    'https://www.googleapis.com/auth/spreadsheets',
)
```

In this example, the Drive scope requested differs slightly: yes, it's also read-only, but rather than file metadata, this scope requests permission to access the file _content_. For GCS and Sheets, both scopes are read-write, and the Cloud Vision API only processes data (doesn't read or write personal data).


#### OAuth flow

The previous post instructed you to download the JSON credentials file after creating an OAuth client ID (and secret), saving it to your local filesystem as `client_secret.json`. When connecting to Google servers to request API access, they are presented along with the scopes the script is requesting (`InstalledAppFlow.from_client_secrets_file()`). Once the user permits access, OAuth tokens (`creds`) are sent back and stored in a local token-storage file (`TOKENS`). These OAuth tokens are what are required to access Google APIs, and they're cached locally so the user won't be prompted for authz each time the script is executed.

There are a pair of OAuth tokens (_access token_ and _refresh token_): the access token provides API access but expires while the refresh token doesn't expire (but can be revoked). If the access token has expired, the refresh token is used to request a new (non-expired) access token. More on this entire flow can be found in [Google's documentation](https://developers.google.com/identity/protocols/oauth2#basicsteps).

Described just now is how it all works, but this is what you see implemented in the script:

1. Check for an existing token storage file, and if so, load its contents
1. Check whether credentials (both OAuth2 tokens) exist and are valid
1. If credentials exist but expired, use refresh token to request another valid (access) token
1. If credentials don't exist, create "OAuth flow" and render to end-user for authz
1. Regardless of how credentials were obtained, (re)save latest tokens (last few lines)


### Application

```python
DRIVE = discovery.build('drive', 'v3', credentials=creds)
files = DRIVE.files().list().execute().get('files', [])
for f in files:  # 4 fields returned: mimeType, kind, id, name
    print(f['name'], f['mimeType'])
```

Funny story: the actual application takes up fewer lines of code than implementing the necessary security described above. The first line creates an endpoint to the Google Drive API (v3 is the current version). Think of this as an "API client." The second line calls `list()` from the `files()` collection of methods available via the API and returns the file list. The files are then looped over with filenames and MIMEtypes displayed. If no files/folders are found in the user's Google Drive, an empty array (`list`) is assigned and nothing is displayed.

That's it, there are no additional lines of code you can't see here. This script along with its older auth library equivalent ([`drive_list.py`](http://github.com/wescpy/gsuite-apis-intro/python/drive_list.py)) featured in the codelab can be found in the `python` folder in the repo at <https://github.com/wescpy/gsuite-apis-intro>. Now let's take a look at the Node versions.


## Node.js/JavaScript

The Node.js version of the script is the file, [`drive_list.js`](http://github.com/wescpy/gsuite-apis-intro/nodejs/drive_list.js):


### Imports

```javascript
const fs = require('fs').promises;
const path = require('path');
const process = require('process');
const {authenticate} = require('@google-cloud/local-auth');
const {google} = require('googleapis');
```

The standard Node.js filesystem (`fs`), filepath (`path`), and `process` packages are imported along with the Google auth and API client libraries. If you prefer the ES module style, here are the equivalent `import` statements you'll find in [`drive_list.mjs`](http://github.com/wescpy/gsuite-apis-intro/nodejs/drive_list.mjs):

```javascript
import fs from 'node:fs/promises';
import path from 'node:path';
import process from 'node:process';
import {authenticate} from '@google-cloud/local-auth';
import {google} from 'googleapis';
```

The rest of the code below is identical in both JavaScript versions.


### Security

The equivalent JavaScript security code operates exactly like the Python version. The only difference is that the security steps are broken out into their own `async` functions, so there's a bit more code to look at:

```javascript
async function loadSavedCredentialsIfExist() {
  try {
    const content = await fs.readFile(TOKEN_PATH);
    const credentials = JSON.parse(content);
    return google.auth.fromJSON(credentials);
  } catch (err) {
    return null;
  }
}

async function saveCredentials(client) {
  const content = await fs.readFile(CREDENTIALS_PATH);
  const keys = JSON.parse(content);
  const key = keys.installed || keys.web;
  const payload = JSON.stringify({
    type: 'authorized_user',
    client_id: key.client_id,
    client_secret: key.client_secret,
    refresh_token: client.credentials.refresh_token,
    access_token: client.credentials.access_token,
  });
  await fs.writeFile(TOKEN_PATH, payload);
}

async function authorize() {
  var client = await loadSavedCredentialsIfExist();
  if (client) return client;
  client = await authenticate({
    scopes: SCOPES,
    keyfilePath: CREDENTIALS_PATH,
  });
  if (client.credentials) await saveCredentials(client);
  return client;
}
```

The typical usage pattern is a call to `authorize()` followed immediately (upon resolution of the Promise) by the code you want to run, in our case, the actual app via the `listFiles()` function shown below.


### Application

```javascript
async function listFiles(authClient) {
  const drive = google.drive({version: 'v3', auth: authClient});
  const res = await drive.files.list();
  const files = res.data.files || [];
  for (let file of files) {
    console.log(`${file.name} (${file.mimeType})`);
  }
}

authorize().then(listFiles).catch(console.error);
```

Like the Python version, `listFiles()` creates a Drive API client and calls its `files.list()` method. Filenames and MIMEtypes are displayed as long as at least one file/folder is returned. The last line in the script comprises the "main" app, calling `authorize()` to get the credentials which are passed to `listFiles()`. That's it, there are no additional lines of code you can't see here. Both scripts can be found in the repo's `nodejs` folder at <https://github.com/wescpy/gsuite-apis-intro>.

> **TypeScript NOTE**
>
> Some of you porting this to TypeScript may run into some challenges getting the security code working due to a conflict between `JSONClient` and `OAuth2Client` types. Specifically, this error may show up: `error TS2322: Type 'OAuth2Client' is not assignable to type 'JSONClient'. Type 'OAuth2Client' is missing the following properties from type 'Impersonated': sourceClient, targetPrincipal, targetScopes, delegates, and 3 more.` The only clue I found touching this topic is in this [closed bug on GH](https://github.com/googleapis/google-auth-library-nodejs/issues/1402). Drop me a comment below if you're able to get a working version, and I'll update this post.


## Running the script

Now that coding is complete, it's time to reap the rewards by executing the script. There are two phases of execution. The first is to go through the security process. When complete the code will then run. If no permission changes occur between executions, the security flow should only happen once upon initial execution. After that, the OAuth tokens are "self-managing:"

1. If access token still valid, execution proceeds as normal
1. If access token expired, refresh token used to request new access token; execution proceeds

Regardless, these steps are not exposed to the end-user who has only to give their initial permissions once. Let's see what these steps look like in practice.


### Authn and authz

When executing the script for the first time, users muse _authenticate_ (sign-in to or select a signed-in Google account) then _authorize_ the app, granting it the requested permission(s). For then authn part, you'll get a Google account-chooser from which to select the one you want to use, something that looks like this:

![Google account-chooser](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tobxg2kqsuavmhwfaszk.png)
<figcaption>[IMAGE] Google account chooser</figcaption>

After selecting the desired account, you'll likely get a **"Google hasn’t verified this app"** screen:

![Google unverified app dialog](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/u2zyo38kymik2yevzg83.png)
<figcaption>[IMAGE] Google unverified app dialog</figcaption>


While it may look a bit apprehensive, its purpose is to let users know that Google has not verified this app which will be asking for permissions to access your data. This is all to protect end-users. There's nothing to worry about for now as it's your code and you're only going to access your own data with it. Learn more about this notification in [Google's unverified apps support page](https://support.google.com/cloud/answer/7454865#unverified-app-screen).

Click on the **Advanced** link at the bottom which opens up the ability to move on if you understand the risks:

![Google unverified app dialog, advanced section](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/qqj2v5m3e5hmxrlhf5ie.png)
<figcaption>[IMAGE] Unverified app dialog, advanced section</figcaption>

Since _you're_ the developer, hopefully you trust yourself enough to proceed. If so, click on the **"Go to _YOUR-APP-NAME_ (unsafe)"** link (where _YOUR-APP-NAME_ is what you configured in the OAuth consent screen earlier). That takes you to the long-awaited OAuth2 authz flow dialog:

![OAuth2 authorization dialog](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/dtz0gvm7mi71uwt3etd4.png)
<figcaption>[IMAGE] OAuth2 authorization dialog</figcaption>

Here, the end-user is prompted to give the developer the requested permission(s) to access their private data. For the purposes of this demo, you're the end-user, the data owner, and also the developer. Basically, you'll be giving your script access to your Drive files.

When you click **Continue**, the security flow completes, and your code will execute. If users click **Cancel** instead, no OAuth tokens are returned, and the script fails; this is all by design. The bottom-line is that because this is private user data, owner permission _must_ be granted before an API access can occur.

To review the difference between authn & authz or to learn more about this security flow, review the [Part 1 post](https://dev.to/wescpy/getting-started-using-google-apis-workspace-9a8) as well as [this security overview page](https://developers.google.com/workspace/guides/configure-oauth-consent) in the GWS documentation.


### Script execution

Once the end-user has opted-in, execution of the script begins. The `listFiles()` method returns the first 100 files/folders in the end-user's Google Drive, so when running the app, expect to see their names and MIMEtypes. Your output will definitely vary, but below is some sample output from my Google Drive when I crafted this script originally for a [post on my old blog](http://wescpy.blogspot.com/2014/11/authorized-google-api-access-from-python.html):

```bash
$ python3 drive_list-new.py  # or $ node drive_list.js
Google Maps demo application/vnd.google-apps.spreadsheet
Overview of Google APIs - Sep 2014 application/vnd.google-apps.presentation
tiresResearch.xls application/vnd.google-apps.spreadsheet
6451_Core_Python_Schedule.doc application/vnd.google-apps.document
out1.txt application/vnd.google-apps.document
tiresResearch.xls application/vnd.ms-excel
6451_Core_Python_Schedule.doc application/msword
out1.txt text/plain
Maps and Sheets demo application/vnd.google-apps.spreadsheet
ProtoRPC Getting Started Guide application/vnd.google-apps.document
gtaskqueue-1.0.2_public.tar.gz application/x-gzip
Pull Queues application/vnd.google-apps.folder
gtaskqueue-1.0.1_public.tar.gz application/x-gzip
appengine-java-sdk.zip application/zip
taskqueue.py text/x-python-script
Google Apps Security Whitepaper 06/10/2010.pdf application/pdf
```

It's not pretty, and my sample Drive folder doesn't even have 100 files/folders in it, but it's just a proof-of-concept and not very flashy. The exciting thing is that you see filenames you recognize, and they're showing up on the command-line, far away from your web browser and the Drive user interface.

For those who prefer more visual content, below is a video I made a few years ago covering much of what's in this post and demonstrating the same (Python) app. Some of the code or video screenshots may have been updated, but the core functionality is mostly identical.

{% youtube Z5G0luBohCg %}
<figcaption>[VIDEO] Listing first 100 files/folders in Google Drive</figcaption>

> **Command-line scripts vs. web or mobile apps**
>
> Command-line scripts aren't going to be the most widely implemented type of application. It's more likely you want to add GWS API usage to a web or mobile app. The calls will differ, but the OAuth flow will be similar. Google has a specific page in their docs for [managing the OAuth flow process for web apps](https://developers.google.com/identity/protocols/oauth2/web-server) as well as [one for mobile apps](https://developers.google.com/identity/protocols/oauth2/native-app).


## Summary

The goal of this 3-part blog post series is to start with nothing and bring developers to a fully working command-line script using GWS APIs (e.g., Google Drive). APIs for the GWS products use OAuth client ID as the authz mechanism. Now that you've completed this "Hello World" application, it's up to your imagination of what you can do now... expand Drive API usage, connect with other GWS APIs like for Docs, Sheets, Slides, Forms, Calendar, or Gmail.

In upcoming posts, I'll look at other Google API families which use different kind of authz mechanism, namely _API keys_ (Maps) and _service accounts_ (GCP). Stay tuned for those as well as additional posts on authn and API client libraries. If neither Python or Node.js "are your thing," you'll also find the Google Drive API QuickStart in [Java](https://developers.google.com/drive/api/quickstart/java), Go, and client-side JavaScript (see left-nav). Finally, if you have ideas on what you'd like to hear about or how I can improve my Node.js (or Python) code, please drop a comment below!


## References

- "Listing your files on Google Drive" [codelab](http://g.co/codelabs/gsuite-apis-intro)
- "Listing your files on Google Drive" [video](http://goo.gl/ZIgf8k)
- Codelab and this post's [GitHub repo](https://github.com/wescpy/gsuite-apis-intro)
- Using Google Workspace APIs: [Part 1/3 blog post](https://dev.to/wescpy/getting-started-using-google-apis-workspace-9a8)
- Using Google Workspace APIs: [Part 2/3 blog post](https://dev.to/wescpy/getting-started-using-google-apis-workspace-23-3ch8)

---
<small>
**WESLEY CHUN**, MSCS, is a [Google Developer Expert](https://developers.google.com/experts) (GDE) in Google Cloud (GCP) &amp; Google Workspace (GWS), author of Prentice Hall's bestselling ["Core Python"](https://corepython.com) series, co-author of ["Python Web Development with Django"](https://withdjango.com), and has written for Linux Journal & CNET. He runs [CyberWeb](https://cyberwebconsulting.com) specializing in GCP & GWS APIs and serverless platforms, [Python & App Engine migrations](https://appenginemigration.com), and Python training & engineering. Wesley was one of the original Yahoo!Mail engineers and spent 13+ years on various Google product teams, speaking on behalf of their APIs, producing sample apps, codelabs, and videos for [serverless migration](http://bit.ly/3xk2Swi) and [GWS developers](http://goo.gl/JpBQ40). He holds degrees in Computer Science, Mathematics, and Music from the University of California, is a Fellow of the Python Software Foundation, and loves to travel to meet developers worldwide at conferences, user group events, and universities. Follow he/him [@wescpy](https://twitter.com/wescpy) & his [technical blog](https://dev.to/wescpy). Find this content useful? [Contact CyberWeb](https://forms.gle/bQiDMiGyGrrwv5sy5) if you may need help or [buy him a coffee (or tea)](http://buymeacoffee.com/wescpy)!
</small>