Format and Parse Amazon S3 URL
Tags: webdev, javascript, typescript, programming
Published: 2024-03-10 06:17:04

Amazon S3 URLs come in different flavors. There are those starting with `s3:`, `http:`, or `https:`. Then, there are the ones with `s3.amazonaws.com`, `s3.us-east-1.amazonaws.com`, or even `s3-us-west-2.amazonaws.com` (note the dash instead of the dot between s3 and the region code). And where do you put the bucket: is it `<bucket>.s3.us-east-1.amazonaws.com/<key>` or `s3.us-east-1.amazonaws.com/<bucket>/<key>`? And when it comes to static website hosting, of course, there is also `<bucket>.s3-website-us-east-1.amazonaws.com` and `<bucket>.s3-website-us-east-1.amazonaws.com` (again, note the dash and the dot).

There are even more when you include the dual-stack, FIPS, access point, and S3 control endpoints. Here's the full list of [Amazon S3 endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html). But for this post, I will focus on the more common URLs that I mentioned before.

## Global
The global URL has the simplest format with the following structure: `s3://<bucket>/<key>`. This URL is also displayed by the AWS management console.

![Image description](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/q6srngrpblpx3a3m7z9g.png)

## Path-style vs. Virtual-hosted-style
The difference between [path-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access) and [virtual-hosted-style](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access) URLs is how the bucket name is included in the URL. Path-style URLs have the bucket name in the pathname of the URL:

```
https://s3.<region>.amazonaws.com/<bucket>/<key>
```

On the other hand, virtual-hosted-style URLs have the bucket name in the hostname of the URL:

```sh
https://<bucket>.<region>.s3.amazonaws.com/<key>
```

Having the bucket name in the host has the advantage of using DNS to route different buckets to different IP addresses. If the bucket name is in the path, all requests have to go to one IP address even for different buckets. That is the reason path-style URLs are deprecated, and support for this style was supposed to end in 2020, but AWS changed their plan and continues to support this style for buckets created on or before September 30, 2020. There's an interesting blog post about the background: [Amazon S3 Path Deprecation Plan – The Rest of the Story](https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/)

## Legacy vs. Regional
Some regions like US East (N. Virginia) `us-east-1` have a legacy global endpoint that doesn't need a region code in the hostname:

```
# Legacy hostname with path-style
https://s3.amazonaws.com/<bucket>/<key>
# Legacy hostname with virtual-hosted-style
https://<bucket>.s3.amazonaws.com/<key>
```

If you use this type of URL for other regions that don't support it, you might either get an _HTTP 307 Temporary Redirect_ or, in the worst case, an _HTTP 400 Bad Request_ error, depending on when the bucket was created.

AWS recommends always using the regional endpoints with the region code in the hostname:

```
# Regional hostname with path-style
https://s3.<region>.amazonaws.com/<bucket>/<key>
# Regional hostname with virtual-hosted-style
https://<bucket>.<region>.s3.amazonaws.com/<key>
```

### Dot-style vs. Dash-style
But also here is a caveat: some regions used to have a dash `-` instead of a dot `.` between `s3` and `<region>`:

```
# Dot-style
https://s3.<region>.amazonaws.com/<bucket>/<key>
# Dash-style
https://s3-<region>.s3.amazonaws.com/<bucket>/<key>
```

For example, the US West (Oregon) `us-west-2` region would support the legacy dash-style URL like `https://s3-us-west-2.amazonaws.com/<bucket>/<key>`. Nevertheless, the standard format `https://s3.us-west-2.amazonaws.com/<bucket>/<key>` is also available for these outliers.

## REST vs. Website
All the URL formats we have seen so far, except the global S3 URL, are called REST endpoints. They are hosted on either the `s3.amazonaws.com` or `s3.<region>.amazonaws.com` hostname, but more importantly, they support secure HTTPS connections. That means all these URLs work with `https://` as the protocol.

Amazon S3 also has a website endpoint for static website hosting. The website endpoint does not support HTTPS, only HTTP. These URLs have the following formats:

```
# Website hostname with dot-style
http://<bucket>.s3-website.<region>.amazonaws.com/<key>
# Website hostname with dash-style
http://<bucket>.s3-website-<region>.amazonaws.com/<key>
```

Again, depending on the region, there is a dash `-` or a dot `.` separating `s3-website` and `<region>`. To see which one is right for your region, you have to check the list of [Amazon S3 website endpoints](https://docs.aws.amazon.com/general/latest/gr/s3.html#s3_website_region_endpoints).

## Format and Parse S3 URLs
Depending on how you interact with Amazon S3, you might use one of the previous URLs. For example, the AWS CLI for S3 expects the S3 URL in the global format `s3://<bucket>/<key>`. Other clients and SDKs probably use the regional REST endpoint with the bucket name either in the hostname or pathname.

If you're using the wrong format or endpoint, you might get an error like this:

```
com.amazonaws.services.s3.model.AmazonS3Exception:
The bucket is in this region: eu-west-1.
Please use this region to retry the request (Service: Amazon S3; Status Code: 301; Error Code: PermanentRedirect;)
```

The right URL really depends on the individual client and how it is requesting from S3. To lift some of this burden, I created a tiny JavaScript library to check, format, and parse S3 URLs in the various formats I described earlier.

{% embed https://github.com/zirkelc/amazon-s3-url %}

At the moment, the library exports only three functions: `formatS3Url`, `parseS3Url`, and `isS3Url`.

```ts
import { formatS3Url, parseS3Url, isS3Url, S3Object } from 'amazon-s3-url';

/* Types */
type S3UrlFormat =
  | "s3-global-path"
  | "s3-legacy-path"
  | "s3-legacy-virtual-host"
  | "https-legacy-path"
  | "https-legacy-virtual-host"
  | "s3-region-path"
  | "s3-region-virtual-host"
  | "https-region-path"
  | "https-region-virtual-host";

type S3Object = {
  bucket: string;
  key: string;
  region?: string;
};

/* Signatures */
function formatS3Url(s3Object: S3Object, format?: S3UrlFormat): string;
function parseS3Url(s3Url: string, format?: S3UrlFormat): S3Object;
function isS3Url(s3Url: string, format?: S3UrlFormat): boolean;

/* Examples */
// Global path
// Without format param (defaults to s3-global-path)
formatS3Url({ bucket: 'bucket', key: 'key' });
parseS3Url('s3://bucket/key');
isS3Url('s3://bucket/key');

// Legacy path-style
// With format param for explicit formatting and parsing
formatS3Url({ bucket: 'bucket', key: 'key' }, 'https-legacy-path');
parseS3Url('https://s3.amazonaws.com/bucket/key', 'https-legacy-path');
isS3Url('https://s3.amazonaws.com/bucket/key', 'https-legacy-path');

// Regional virtual-hosted-style
// With region property for regional endpoints
formatS3Url({ region: 'us-west-1', bucket: 'bucket', key: 'key' }, 'https-region-virtual-host');
parseS3Url('https://bucket.s3.us-west-1.amazonaws.com/key', 'https-region-virtual-host');
isS3Url('https://bucket.s3.us-west-1.amazonaws.com/key', 'https-region-virtual-host');
```

### Limitations
The library does only rudimentary URL validation on the structure of the URL, but it doesn't validate the bucket name, object keys, and regions. Also, it doesn't support the dual-stack, FIPS, access point, control, and website endpoints yet. But I'm happy to welcome any external contribution.