Securing Grafana for Web Access
Tags: tutorial, devops, security, grafana
Published: 2024-02-27 16:39:36

This guide will demonstrate how to secure an instance of Grafana behind Pomerium proxy to provide users with a seamless login to Grafana using your identity provider.

# What is Grafana?

[Grafana](https://grafana.com/) is an open-source analytics visualization and monitoring tool. It provides many user-contributed Dashboards that make it popular for enthusiasts as well as professionals.

# Getting Started

_Note: This guide uses GitHub as the pre-configured Identity Provider (IdP), but you can use any supported IdP._

To complete this guide, you need:

- [Docker](https://www.docker.com/) and [Docker Compose](https://docs.docker.com/compose/install/)
- A pre-configured [identity provider](https://www.pomerium.com/docs/identity-providers) (IdP)

See the Pomerium [Quick-start guide](https://www.pomerium.com/docs/quickstart) to run Pomerium in a containerized Docker environment.

## Build a Grafana Route

In your `config.yaml` file, add the following Grafana route:

```yaml
- from: https://grafana.localhost.pomerium.io
  to: http://grafana:3000
  host_rewrite_header: true
  pass_identity_headers: true
  policy:
    - allow:
        or:
          - email:
              is: user@example.com
```

In the code above:

- [`host_rewrite_header`](https://www.pomerium.com/docs/reference/routes/headers#host-rewrite) rewrites the `Host:` header to match an incoming header value. Without this option enabled, Grafana will throw a `403: Origin not allowed` error after login when you attempt to add users.

This behavior is due to a CSRF check added in Grafana v8.3.5 (this guide uses v9.2.4) that requires the [Origin request header](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Origin) to match the server origin.

See this [GitHub issue](https://github.com/grafana/grafana/issues/45117#issuecomment-1033842787) for more information.

## Configure Grafana to use JWT Verification

Configuring Grafana to use JWT authentication makes it possible for Grafana to accept identity claims sent upstream by Pomerium.

Since you're not using a `grafana.ini` file to configure your Grafana instance, you can override Grafana's configuration using environment variables. See the [Grafana docs](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#override-configuration-with-environment-variables) for more information.

In your `docker-compose.yaml` file, add the Grafana Docker image as a service:

```yaml
grafana:
  image: grafana/grafana:latest
  ports:
    - 3000:3000
  environment:
    - GF_AUTH_SIGNOUT_REDIRECT_URL=https://grafana.localhost.pomerium.io/.pomerium/sign_out
    - GF_AUTH_JWT_ENABLED=true
    - GF_AUTH_JWT_HEADER_NAME=X-Pomerium-Jwt-Assertion
    - GF_AUTH_JWT_EMAIL_CLAIM=email
    - GF_AUTH_JWT_JWK_SET_URL=https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json
    - GF_AUTH_JWT_CACHE_TTL=60m
  volumes:
    - ./grafana-storage:/var/lib/grafana
```

Here's what the environment variables do:

- `GF_AUTH_SIGNOUT_REDIRECT_URL`: signs users out of Pomerium when they sign out of Grafana
- `GF_AUTH_JWT_ENABLED`: enables JWT authentication (Grafana disables it by default)
- `GF_AUTH_JWT_HEADER_NAME`: tells Grafana which HTTP header to look at for the JWT (see [Identity Verification](https://www.pomerium.com/docs/capabilities/getting-users-identity) for more information)
- `GF_AUTH_JWT_EMAIL_CLAIM`: associates the `email_claim` in the JWT with the email of the Grafana user
- `GF_AUTH_JWT_JWK_SET_URL`: defines the URL with the signing key to validate the JWT
- `GF_AUTH_JWT_CACHE_TTL`: sets a 60-minute cache time for the token

Congratulations, your Grafana now has a secured route. Let's add users.

## Add users to Grafana

At this stage, Grafana is configured to use the `email` claim in the JWT to associate an incoming connection with a user (you will configure Pomerium to include identity information via the JWT in the next section).

But, the user first must exist in Grafana to be associated. Otherwise, you will see the following error in the browser after authenticating:

```json
{"message": "User not found"}
```

To avoid this error, create a user with administrator privileges. (If you plan on administering Grafana through a direct connection to the service, skip this section.)

To add users without requiring them to accept an invitation:

1. Log in to Grafana directly as an admin user at `https://grafana.localhost.pomerium.io` (if this is your first time signing up for Grafana, `admin` is the [default username and password](https://grafana.com/docs/grafana/v9.0/setup-grafana/configure-grafana/#security))

1. Select **Server Admin**, **Users**
![The users option under the server admin menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/sneinwmd7aafkcwau9ga.png)
_Caution: This is distinct from the **Users** option under the **Configuration** cog wheel, which will only finalize a new user when they accept an invite via email or link._

1. Select **New user** to create a new user. Make sure that the email address matches the one provided by Pomerium via your IdP. If you want to check the data provided by Pomerium in the JWT, you can access the special endpoint `/.pomerium` from any Pomerium route to view it.

After creating a new user in Grafana, that user should be logged in automatically when they access Grafana from the Pomerium route (after first authenticating with your IdP, of course).

# Manage access at scale

First, ensure Grafana is up to date to take advantage of `auto_sign_up`, as it is only available for JWT as of version 8.4.0.

The steps outlined above work to confirm the configuration for small teams, but adding users individually and manually does not scale for larger organizations. To add users to Grafana at scale, you can use the Grafana's `auto_sign_up` configuration to auto sign up users as they log in:

```ini
[auth]
signout_redirect_url = https://grafana.localhost.pomerium.io/.pomerium/sign_out
[auth.jwt]
enabled = true
header_name = X-Pomerium-Jwt-Assertion
email_claim = email
jwk_set_url = https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json
cache_ttl = 60m
username_claim = sub ;sets the username to the value of the "sub" claim
auto_sign_up = true ;sets the login to automatically create a new user if one doesn't exist
[users]
auto_assign_org = true ;auto assigns the user to the existing default organization
auto_assign_org_role = Editor ;auto assigns the user the "Editor" role
```

Note that the value of `auto_assign_org_role` could also be "Admin" or "Viewer".

This will automatically create a user with their email and username populated. 

Pomerium autopopulates the JWT `name` claim by default with the value provided by the identity provider to save you time. This configuration will allow seamless authentication and authorization without any additional toil for your team.

For teams using Google as an IdP, the user field is populated by a numeric value that you cannot configure. Note that Grafana can accept the `name` field as a username, including spaces:

```abnf
username_claim = name
```

# Troubleshooting

### Local Signing Key

In instances where Grafana cannot get the signing key for the JWTs from the Pomerium authenticate service, you can place a copy of the key locally.

For example, wildcard certificates signed by LetsEncrypt may still be cross-signed by the [expired DST R3 root]. While many browsers still trust these certificates (as long as they are also signed by a valid root), some applications reject them, including Grafana:

```log
logger=context error=Get "https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json": x509: certificate signed by unknown authority
```

To circumvent this issue, you can use `curl` or `wget` to download the signing key locally:

## curl

From the Grafana host:

```bash
curl https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json > /etc/grafana/jwks.json
```

## wget

From the Grafana host:

```bash
wget -O /etc/grafana/jwks.json https://grafana.localhost.pomerium.io/.well-known/pomerium/jwks.json
```

Edit `grafana.ini` and add the `jwk_set_file` key to provide it to Grafana:

```ini
[auth.jwt]
enabled = true
header_name = X-Pomerium-Jwt-Assertion
email_claim = email
jwk_set_file = /etc/grafana/jwks.json
cache_ttl = 60m
```

If you have other issues with this guide, please [reach out to Pomerium on our forums](https://discuss.pomerium.com/)!