Investigating Service Account Key Origins and Usage with Best Practices
Tags: googlecloud, security, cloudcomputing
Published: 2023-10-11 18:51:06

Service accounts can pose a security risk for your Google Cloud project if not managed properly. Because they are often highly privileged, anyone who is able to authenticate as a service account can likely take sensitive actions in your environment. It’s important to control access and have visibility into which identities — whether human or machine — have excessive and possibly dangerous permissions.

In this blog post, we’ll cover analyzing service account authentication patterns. You’ll be able to discover unused service accounts and keys and understand what API actions your service accounts are being used to take. If data access audit logs are enabled on your project, you’ll also be able to understand which users are authenticating via impersonation, the IPs that service account keys are being used to authenticate from, and determine which users originally created service account keys.

## Required permissions:

You’ll need the following IAM predefined roles to follow the steps discussed in this blog post:

- `iam.serviceAccountViewer`
- `monitoring.viewer`
- `logging.privateLogViewer`

## Cloud monitoring authentication events

Google’s Cloud Monitoring API allows you to view and query service account and service account  key authentication events that occurred within the past six weeks. An “authentication event” is defined as any time a service account or service account key was used to call any Google API. 

### Google Cloud Console

If you know which service account you are interested in, you can navigate to the “service accounts” page for your desired project [here](https://console.cloud.google.com/iam-admin/serviceaccounts). You can then click on a specific service account and view its metrics. 

![Service account metrics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8tr3bqrilkis19uapb2t.png)

You’ll be shown authentication traffic for the account, separated by impersonation events and key authentication events, as well as data on the API calls made from the service account.

![Service account metrics](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/8dxpbe9hmpwefp5wfnaa.png)


You can also view aggregated metrics for all service accounts and keys via the Cloud Monitoring Metrics explorer, located [here](https://console.cloud.google.com/monitoring/metrics-explorer). 

Use the drop-down and search to select either the “Service account authentication events” metric or the “Service account key authentication” metric.

![service account authentication events](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sfy0cvh3i2nep45usra7.png)

“Service account authentication events” will include requests directly from an attached service account or requests made via service account impersonation, while “Service account key authentication events” will include requests authenticated with a specific key. 

![service account key authentication events](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/002bi4yz2a5q1ao5y4fp.png)

Google also provides various options for filtering, grouping, and aggregating the data.

![service account key authentication events](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/7u7f14e8ip7cmr656j62.png)

You can group by `unique_id` to view the data grouped by Service Account ID, or group by `key_id` to view the data grouped by Service Account Key ID. 

![group by key ID](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/zdszx7s4qf32jpyvsu20.png)


### Google REST APIs

To query service account authentication events directly from the Cloud Monitoring API you can use the `projects.timeSeries.list` API route ([docs](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.timeSeries/list)).

You’ll need to specify `metric.type = "iam.googleapis.com/service_account/authn_events_count"` or `metric.type = "iam.googleapis.com/service_account/key/authn_events_count"` depending on whether you are looking for key events or direct authentication events. By default, this call will return raw time series data grouped by service account ID and service account key ID, but you can also request aggregated data via the `aggregation` parameter.

## Identity access monitoring

The Cloud Monitoring data provides us with a lot of insights into service account usage, but some details are missing. For example, who accessed these service accounts? Which of these accesses are human users and which are machines? To answer these questions, we can look to Google’s Data Access audit logs.

To view the data in this section, you’ll need to have had data access audit logs enabled in your project for the time period you wish to analyze. These logs are not enabled by default, but can be very helpful for analyzing access patterns in your cloud environment. See Google’s [documentation](https://cloud.google.com/logging/docs/audit/configure-data-access) for steps on how to enable data access logs.  

### Uncover service account key origins

One difficulty of understanding service account key usage is that there’s no way to know who has access to a key. Users may have downloaded it to their computer and shared it with others, which is something that you cannot track. However, you can make progress towards understanding which users have access to a key by determining who created the key, since Google only lets you download the key at creation time. 

To do this, we can query the [Logs Explorer](https://console.cloud.google.com/logs) to search for the `CreateServiceAccountKey` event with the desired key ID. 

```
protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"
protoPayload.response.name="projects/[PROJECT]/serviceAccounts/[SA_EMAIL]/keys/[KEY_ID]"
```

You can narrow your search to the day the service account key was created by using the “Creation Time” field in the Cloud Console or returned by the [IAM Service Account Keys API](https://cloud.google.com/iam/docs/keys-list-get). 

The resulting log will have an `authenticationInfo` field that will tell you who created the key. 

### Investigate service account impersonation

In the previous section, we covered obtaining timestamps for service account impersonation events. With logs, you can get more details on each event, including the principal that performed the impersonation and the specific request that was made.

Each audit log contains an `authenticationInfo` field with information on how the request was authenticated. If the method was service account impersonation, there will be a `serviceAccountDelegationInfo` field inside `authenticationInfo`, which contains an array of all the principals in the impersonation chain. You can use this to query for all impersonation events. 

If you want to query for all impersonation events involving human users, you can use the following: 

```
protoPayload.authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail =~ "@[YOUR DOMAIN]"
```

You can also query for all impersonation events involving service accounts via: 

```
protoPayload.authenticationInfo.serviceAccountDelegationInfo.firstPartyPrincipal.principalEmail =~ "iam.gserviceaccount.com"
```

Filter for a specific service account by adding the following to your query: 

```
protoPayload.authenticationInfo.principalEmail="[SA_EMAIL]"
```

### Examine service account key usage

Similarly, you can use the audit logs to get a more detailed log of service account key usage. Because there is no identity tied to the key usage, this unfortunately will not provide information on who used the key.

However, the audit log includes the IP address from which the request was made, which can allow you to further investigate the source of the key usage. 

To search for all events on a specific key, you can use: 

```
protoPayload.authenticationInfo.serviceAccountKeyName="//iam.googleapis.com/projects/[PROJECT_ID]/serviceAccounts/[SA_EMAIL]/keys/[KEY_ID]"
```

To search for events on any key, you can use the following: 

```
protoPayload.authenticationInfo.serviceAccountKeyName =~ "//iam.googleapis.com/"
```

The resulting log will have a `requestMetadata` field that contains a `callerIp` field and a `callerSuppliedUserAgent` field. You can use these to try to determine whether the key is being accessed programmatically or by a human user: for example, by cross-referencing the caller IP with your workload IP addresses. 

## Take action and reduce risk with IAM best practices

It’s important to secure your service account permissions, because they may be allowing your users to have unintended access in your system and a common privilege escalation attack vector in Google Cloud is managing to authenticate as a privileged service account. Having excessively privileged service accounts in your infrastructure can leave you more vulnerable to both internal and external threats. 

Here are some best practices to manage privileged access:  

- Delete unused service accounts.
- Right-size service account roles. You should not have service accounts with “Editor” on your project. Instead, give them access to only the APIs that they’re being used to call. If you are using the same service account for a large range of functionality, instead consider replacing it with several different service accounts that each have more limited access.
- Delete unused service account keys.
- Avoid using service account keys unless absolutely necessary. Oftentimes, you can instead use service account impersonation or direct user credentials, both of which are more secure. See [this flowchart](https://cloud.google.com/docs/authentication#auth-decision-tree) from Google to understand alternative authentication methods you may be able to use.
- Apply the [Disable service account key creation](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-service-accounts#disable_service_account_key_creation) and [Disable service account key upload](https://cloud.google.com/resource-manager/docs/organization-policy/restricting-service-accounts#disable_service_account_key_upload) organization policy constraints.

## Secure service accounts and keys for free with P0

Tired of having to check multiple Google APIs and querying logs in search for the information you need in order to secure your cloud configuration? [Sign up for P0 for free](https://p0.app/create-account?utm_source=blog_iam_detective&utm_medium=blog_cta&utm_campaign=iam_best_practices) and conduct your first IAM assessment in fifteen minutes or less. P0 will show you all your cloud principals (human and service accounts) along with all resources they have access to and information about their authentication history. P0 allows you to view service accounts and service account keys by usage easily, and will even alert you to changes in your security posture, such as new unused service account keys.