A Beginner's Guide to Curl: Part 5 - Setting Headers
Tags: bash, http, zsh, curl
Published: 2023-04-16 22:23:45

HTTP requests are sent with metadata contained in headers. This includes data such as the request method, authentication credentials, and cookies.

Headers consist of a "field name" and "field value" separated by a colon.  Multi-word field names are separated by dashes and are case-insensitive.

`Field-Name: Some Value` or `field-name: Some Value`

You can see exactly which headers are being passed by adding the `-v` or `--verbose` flag to your curl requests.  There is also a free service that has an API endpoint that echos your request headers back to you as a JSON response.

Type the following command to see the default headers sent by curl.

```bash
$ curl https://httpbin.org/headers
```

I got this JSON response in return.

```json
{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.85.0"
  }
}
```

By default curl accepts any content type in return, "\*/\*".  Normally the "Accept" header would specify the MIME type the request is expecting in return, such as "text/html" or "application/json".

The "Host" header is set because of the URL supplied to curl.

The "User-Agent" in this case is the curl application, but let's switch that now.  

Send this instead.

```bash
$ curl -H 'User-Agent: Mozilla/5.0' https://httpbin.org/headers
```

Now, as you can see, the response shows your agent header set to 'Mozilla/5.0'.  All headers can be set in Curl with the `-H` or `--header`.  Headers can be placed before or after the URL. 
 If you want to set multiple headers at once, you'll need us use multiple flags.  Here's a way you can construct a complex curl request on multiple lines using the backlash to escape newlines.

(By the way, you can make up your own headers, and you don't have to prefix them with an 'X'.)

```bash
$ curl https://httpbin.org/headers \
    -H 'Accept: application/json' \
    --header 'Referer: https://dev.to' \
    -H 'My-Cool-Custom-Header: foo'
```

**Authorization Header**

One of the main usages of HTTP headers is authentication.  Interestingly the header for this is called 'Authorization'.  I guess these two concepts are similar enough. 

Authorization headers work by specifying a `scheme` followed by a space and the value.  Here's an example.

`Authorization: basic username:password`

Here's an API to test out basic auth.  You construct the endpoint using `basic-auth` as the first segment followed by the username of your choice, and finally the password.  In our case, we will attempt to authenticate via the endpoint `basic-auth/username/password`.

The API will either return a `200` or `401` response.  Supplying the `-i` or `--include` flags will reveal the response headers and status code.

```bash
$ curl -i https://httpbin.org/basic-auth/username/password \
    -H 'Authorization: basic username:password`
```

Huh?  Why is it a 401?

```
HTTP/2 401
```
This is because the internet specifies that basic HTTP auth credentials be encoded using base64.

Base 64 is not encryption.  Nothing about it is secure, it's just a different encoding.  This [video](https://www.youtube.com/watch?v=aUdKd0IFl34) offers a great explanation.

You can easily encode anything you want in Base64 without leaving the command line.  Just echo an expression and pipe it to the `base64` cli.

```bash
$ echo -n username:password | base64
```

You should see the result as `dXNlcm5hbWU6cGFzc3dvcmQ=`.

Ok, now that we have the string 'username:password' encoded in Base64, let's try the API again, and see if we can successfully authenticate.

```bash
$ curl -i https://httpbin.org/basic-auth/username/password \
   -H 'Authorization: basic dXNlcm5hbWU6cGFzc3dvcmQ='
```
And here's the response!

```bash
HTTP/2 200
{
  "authenticated": true,
  "user": "username"
}
```
Hooray!

But that seems pretty tedious.  Thankfully, the developers of Curl already thought of that.  That's why Curl provides a special option for the authorization header.  Just provide the `-u` or `--user` flag followed by the `username:password` in plain text.  You can specify the the scheme with the `--basic`; however, it's already basic by default.  Curl will then encode your username and password for you.

```bash
$ curl -i https://httpbin.org/basic-auth/apple/banana
    --user 'apple:banana'
```

To verify that Curl really did encode the credentials, just supply the `-v` or `--verbose` flag to see all request headers.  In my case with apple and banana, my header looked like this.

`authorization: Basic YXBwbGU6YmFuYW5h`

Another type of HTTP authorization is called `bearer`.  Instead of supplying a username and password, you would use the word 'bearer' followed by a space and some sort of token.  Oftentimes these tokens are randomly generated keys or actual data that's gone through a hashing algorithm, such as a JSON Web Token (JWT) which contains JSON data.

Once you're confident supplying header data in your Curl requests, you'll find it easy and fast to test out APIs straight from the command line.