Granular access control to Google Cloud Storage objects using IAM Conditions
Tags: cloud, gcp, cloudstorage, iam
Published: 2019-12-20 22:36:15

At Meteopress we leverage Google Cloud Storage very intensively. Due to its serverless characteristics and unbeatable performance and reliability, it often makes a core part of our services. More traditionally we also use it as an unlimited data storage and for serving static website resources.

But one feature we missed deeply - the ability to limit access to "directories" (subpaths) in Cloud Storage Bucket. Although Cloud IAM permissions had already been a very powerful mechanism, there was no possible way to bound each application (service account) to specific parts of a bucket.

Practicing [least privilege principle](https://en.wikipedia.org/wiki/Principle_of_least_privilege) we see a lot of sense in controlling shared bucket access - mainly in cases when production and experimental services meet on the same bucket resource.

This superpower comes with brand new [Cloud IAM Conditions](https://cloud.google.com/storage/docs/access-control/using-iam-permissions#conditions-iam) feature.

## Satellite processing service

To demonstrate _Cloud IAM Conditions_ capabilities the following lines are going to describe and simulate a simplified satellite image processing service. The service continuously reads data downloaded from our satellite dish and processes each file to render enhanced images. We need the service to be able to read all incoming data and write processed data to a specified Cloud Storage "directory". After each new file is processed, the manifest file is updated to contain the lastest image metadata.

The service is using `iamconditionsdemo` bucket which is created with _uniform bucket-level permissions_.

## Reading data

In the beginning, the service cannot read any data.

```
> gsutil cp gs://iamconditionsdemo/satellite/raw/12_49.jpg .

ServiceException: 401 Anonymous caller does not have storage.objects.get access to iamconditionsdemo/satellite/12_49.jpg.
```

Cloud Storage bucket is not publicly accessible so the service is going to use a service account to identify itself.

```
> gcloud auth activate-service-account --key-file gcp_credentials.json

Activated service account credentials for: [iamconditionstest@meteopress-radars.iam.gserviceaccount.com]
```

We assign _Storage Object Viewer_ role in bucket's permissions panel to the service account.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/sbs2h81llfynu22k7sha.png)

Now the service is able to read objects from the bucket. It has access to all of the objects no matter their name.

```
> gsutil cp gs://iamconditionsdemo/satellite/raw/12_49.jpg .

Copying gs://iamconditionsdemo/satellite/raw/12_49.jpg...
- [1 files][835.5 KiB/835.5 KiB]
Operation completed over 1 objects/835.5 KiB.
```

## Writing data

After our service processes any given file it needs to store a new object with `satellite/processed/` path prefix. This operation is not allowed at the moment.

```
gsutil cp 12_49_processed.jpg gs://iamconditionsdemo/satellite/processed/12_49.jpg

Copying file://12_49_processed.jpg [Content-Type=image/jpeg]...
AccessDeniedException: 403 iamconditionstest@meteopress-radars.iam.gserviceaccount.com does not have storage.objects.create access to iamconditionsdemo/satellite/processed/12_49.jpg.
```

To allow object writes a service account can obtain _Storage Object Creator_ role. But the role actually allows any object to be written to the bucket. Any potentially dangerous write can appear and for example, contaminate our input dataset by creating an object starting with `satellite/raw/` pathname.

Cloud IAM Conditions are the right tool to prevent those unwanted writes. We can limit the permission to write an object just to specified object paths. This can be achieved by clicking on _Add condition_ besides the IAM role.

Using _Edit condition_ dialog we set _Type_ to be a cloud storage object and resource _Name_ to filter object names by a prefix. According to the [documentation](https://cloud.google.com/storage/docs/key-terms#resource-name) object's resource name has the following form `projects/_/buckets/[BUCKET_NAME]/objects/[OBJECT_NAME]`.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/ctoijfwrjbiis2ndly23.png)

Or we can switch to write the rule in the built-in text editor using Common Expression Language.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/pnwwsqt0gljqz709kxlh.png)

The service is now able to write objects as required.

```
> gsutil cp 12_49_processed.jpg gs://iamconditionsdemo/satellite/processed/12_49.jpg

Copying file://12_49_processed.jpg [Content-Type=image/jpeg]...
\ [1 files][835.5 KiB/835.5 KiB]
Operation completed over 1 objects/835.5 KiB.
```

At the same time, it can't write any objects out of `satellite/processed` "directory".

```
> gsutil cp 12_49_processed.jpg gs://iamconditionsdemo/satellite/raw/12_49_processed.jpg

Copying file://12_49_processed.jpg [Content-Type=image/jpeg]...
AccessDeniedException: 403 iamconditionstest@meteopress-radars.iam.gserviceaccount.com does not have storage.objects.create access to iamconditionsdemo/satellite/raw/12_49_processed.jpg.
```

## Updating manifest

After each new image is processed its metadata needs to be saved to the predefined manifest file `satellite/manifest_processed.json`. This object is not writable at the moment.

Since _Storage Object Creator_ role doesn't give "storage.objects.delete" access (which is necessary to rewrite an object) we are going to add _Storage Object Admin_ permission just to `satellite/manifest_processed.json` object.

"Unfortunately" the `iamconditionstest` bucket was created with _uniform bucket-level permissions_ so we can't simply update object's permissions (not even mentioning it doesn't exist yet). IAM Conditions can be used again.

![Alt Text](https://thepracticaldev.s3.amazonaws.com/i/3ojy4cm13xyzpp9iqk1o.png)

Now the manifest file can be created or updated after each image is processed.

```
> gsutil cp manifest.json gs://iamconditionsdemo/satellite/manifest_processed.json

Copying file://manifest.json [Content-Type=application/json]...
/ [1 files][   22.0 B/   22.0 B]
Operation completed over 1 objects/22.0 B.

...manifest updated...

> gsutil cp manifest.json gs://iamconditionsdemo/satellite/manifest_processed.json

Copying file://manifest.json [Content-Type=application/json]...
/ [1 files][   44.0 B/   44.0 B]
Operation completed over 1 objects/44.0 B.
```

## Conclusion

This illustrational use case demonstrates how _Cloud IAM Conditions_ can be used to limit a service account's access on a granular Cloud Storage Bucket level. In reality, we use multiple buckets to better segregate raw and processed data but principals stay the same.

Cloud IAM Conditions are going to help us better control access to particular Cloud Storage resources and we can assure each service is not (re)writing data it shouldn't.

We can also use Cloud IAM Conditions for our customers' service accounts which they use to read a bucket with data they are subscribed to. Now we don't have to copy datasets to each domain bucket but we can set correct access with IAM Conditions based on a customer's subscription.

Cloud IAM Conditions are not limited just to the Cloud Storage resource type. The [documentation](https://cloud.google.com/iam/docs/conditions-overview#supported-services) describes other resources that can be controlled.

Unfortunately, there is no Terraform support yet.

What is your use case for _Cloud IAM Conditions_?