Http Requests Cheatsheet
Tags: cheatsheet, httprequests
Published: 2021-12-27 09:21:09

*Note: this article was originally published on my personal blog. Check it out [here](https://bastien-antoine.fr).*

When working around APIs, whether it is building or playing with, you often end up making API requests over and over that all look alike, and yet there are some stuff that you can't remember. Like how would you provide form data with cURL? Or basic authentication with `requests`? This article serves as a cheatsheet on all those common stuff.



- [Selecting the HTTP method](selecting-the-http-method)
- [Headers](headers)
- [Authentication](authentication)
	- [HTTP Basic](http-basic)
	- [HTTP Bearer](http-bearer)
- [Query parameters](query-parameters)
- [Body parameters](body-parameters)
- [Sending files](sending-files)
- [Disabling SSL verification](disabling-ssl-verification)
- [3XX redirects](3xx-redirects)
- [Logging](logging)



# Selecting the HTTP method

## `requests` (Python) 

3 possible ways:

1. Using the `requests.request` method:

```python
resp = requests.request('GET', 'https://httpbin.org/get')
resp = requests.request('POST', 'https://httpbin.org/post')
resp = requests.request('PUT', 'https://httpbin.org/put')
resp = requests.request('PATCH', 'https://httpbin.org/patch')
resp = requests.request('DELETE', 'https://httpbin.org/delete')
resp = requests.request('HEAD', 'https://httpbin.org/get')
```

2. Using the `requests.<method>` methods:

```python
resp = requests.get('https://httpbin.org/get')
resp = requests.post('https://httpbin.org/post')
resp = requests.put('https://httpbin.org/put')
resp = requests.patch('https://httpbin.org/patch')
resp = requests.delete('https://httpbin.org/delete')
resp = requests.head('https://httpbin.org/get')
```

3. Using a prepared request:

```python
req = requests.Request('GET', 'https://httpbin.org/get')
r = req.prepare()
s = requests.Session()
s.send(r)
```

From [`requests.PreparedRequest` documentation](https://docs.python-requests.org/en/latest/api/#requests.PreparedRequest).

## cURL 

```shell
curl -X GET https://httpbin.org/get
curl -X POST https://httpbin.org/post
curl -X PUT https://httpbin.org/put
curl -X PATCH https://httpbin.org/patch
curl -X DELETE https://httpbin.org/delete
```

If the `-X/--request` is not provided, cURL will default to `GET`.

[[doc of `-X/--request` option]](https://curl.se/docs/manpage.html#-X)

*Note on the `HEAD` requests*: to make a proper `HEAD` request (ie. requesting only the headers), the `-X HEAD` option will not work, you will need to use [`-I/--head`](https://curl.se/docs/manpage.html#-I) instead:

```shell
> curl -X HEAD https://httpbin.org/get
Warning: Setting custom HTTP method to HEAD with -X/--request may not work the
Warning: way you want. Consider using -I/--head instead.
curl: (18) transfer closed with 255 bytes remaining to read
> curl -I https://httpbin.org/get
HTTP/2 200
date: Sun, 19 Dec 2021 11:03:31 GMT
content-type: application/json
content-length: 255
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true
```

(Tested on cURL 7.64.0 and 7.77.0)

# Headers

## `requests` (Python) 

```python
requests.get(
  'https://httpbin.org/headers',
  headers={
      'X-My-Header': 'Value',
  }
)
```

[[doc of `headers` parameter]](https://2.python-requests.org/en/master/user/quickstart/#custom-headers)

## cURL 

```shell
curl -H 'X-My-Header: Value' https://httpbin.org/headers
```

[[doc of `-H/--headers` option]](https://curl.se/docs/manpage.html#-H)

# Authentication

## HTTP Basic

### `requests` (Python) 

2 possible ways:

1. Using the [`auth`](https://2.python-requests.org/en/master/user/authentication/#basic-authentication) option

```python
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  auth=('user', 'passwd')
)
```

2. Using the [headers](https://docs.python-requests.org/en/latest/user/quickstart/#custom-headers) option

```python
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': 'Basic dXNlcjpwYXNzd2QK',
  }
)

from base64 import b64encode
requests.get(
  'https://httpbin.org/basic-auth/user/passwd',
  headers={
      'Authorization': f'Basic {b64encode('user:passwd')}',
  }
)
```

*Note on using this along with a `.netrc` file*: Authorization headers that are provided this way will be overriden by credentials set in the `.netrc` file [[doc]](https://2.python-requests.org/en/master/user/authentication/#netrc-authentication).

As a workaround you can either temporarily comment the authentication part in your `.netrc`, set the `NETRC` env var to an empty file, or temporarily set `requests.utils.NETRC_FILES = []` [[source code]](https://github.com/psf/requests/blob/2d2447e210cf0b9e8c7484bfc6f158de9b24c171/requests/utils.py#L178) .

### cURL 

2 possible ways:

1. Using the [`-u/--user`](https://curl.se/docs/manpage.html#-u) option

```shell
curl -u user:passwd https://httpbin.org/basic-auth/user/passwd
```

Few things to note when using this option:

- cURL will split the username and password on the first colon. This means you can't have one in the username, but you can in the password. If needed, use the solution below with the `-H/--header` option.
- If only the username is provided, cURL will prompt for the password before sending the request

2. Using the [`-H/--header`](https://curl.se/docs/manpage.html#-H) option

```shell
curl -H 'Authorization: Basic dXNlcjpwYXNzd2QK' https://httpbin.org/basic-auth/user/passwd
curl -H "Authorization: Basic $(openssl base64 <<< user:passwd)" https://httpbin.org/basic-auth/user/passwd
```

## HTTP Bearer

### `requests` (Python) 

```python
requests.get(
  'https://httpbin.org/bearer',
  headers={
      'Authorization': 'Basic dGhpc0lzQUJlYXJlclRva2VuCg==',
  }
)
```

### cURL 

```shell
curl -H 'Authorization: Bearer dGhpc0lzQUJlYXJlclRva2VuCg==' https://httpbin.org/bearer
```

# Query parameters

## `requests` (Python) 

```python
requests.get(
  'https://httpbin.org/anything',
  params={
      'arg1': 'val1',
      'arg2': 'val2',
  }
)
```

[[doc of `params` parameter]](https://2.python-requests.org/en/master/user/quickstart/#passing-parameters-in-urls)

## cURL 

No useful options here unfortunately

```shell
curl https://httpbin.org/anything?arg1=val1&arg2=val2
```

# Body parameters

## `requests` (Python) 

There are multiple ways to provide a body, each with their side consequences:

1. Providing a dict in the `data` parameter: this will set the `Content-Type` header to `application/x-www-form-urlencoded`, unless told otherwise by the `headers` parameter.

```python
requests.post(
  "https://httpbin.org/post",
  data={'key1': 'value1', 'key2': 'value2'},
)
```

2. Providing a raw string in the `data` parameter: this will not set any value to the `Content-Type` header.

```python
requests.post(
  "https://httpbin.org/post",
  data='theBodyOfMyRequest',
)
```

3. Providing a dict in the `json` parameter: this will dumps the provided payload as JSON and set the `Content-Type` header to `application/json`, unless told otherwise by the `headers` parameter. The following two request are equivalents:

```python
import json
requests.post(
  "https://httpbin.org/post",
  data=json.dumps({'key1': 'value1 and value2', 'key2': 'value2'}),
  headers={'Content-Type': 'application/json'},
)
requests.post(
  "https://httpbin.org/post",
  json={'key1': 'value1 and value2', 'key2': 'value2'},
)
```

[[doc]](https://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests)

## cURL 

cURL offers multiple options to send a request with a body, each with their side consequences as well.

1. Using the `-F/--form` option

```shell
curl -F key1=value1 -F key2=value2 https://httpbin.org/post
```

Few things to note when using this option:
- it forces the method to `POST`, unless told otherwise with `-X/--request` option
- it sets the `Content-Type` header to `multipart/form-data`, unless told otherwise aswell with `-H/--header` option

Using this option, it is possible to send files or read input from a file by prefixing the filename with `@` or `<`, respectively. This means that if you need to have either a `@` or `<` in the key and/or the value, you need to use the `--form-string` option.

[[doc of `-F/--form` option]](https://curl.se/docs/manpage.html#-F)

[[doc of `--form-string` option]](https://curl.se/docs/manpage.html#--form-string)

2. Using `-d/--data` option

```shell
curl -d key1=value1 -d key2=value2 https://httpbin.org/post
```

Few things to note when using this option:
- it forces the method to `POST`, unless told otherwise with `-X/--request` option
- it sets the `Content-Type` header to `application/x-www-form-urlencoded`, unless told otherwise aswell with `-H/--header` option

Using this option, it is possible to read input from a file by prefixing the filename with `@`. This means that if you need to have a `@` in the key and/or the value you need to use the `--data-raw` option.

It is also possible to URL-encode the parameter using the `--data-urlencode` option.

[[doc of `-d/--data` option]](https://curl.se/docs/manpage.html#-d)

[[doc of `--data-raw` option]](https://curl.se/docs/manpage.html#--data-raw)

[[doc of `--data-urlencode` option]](https://curl.se/docs/manpage.html#--data-urlencode)

# Sending files

## `requests` (Python) 

```python
requests.post(
  'https://httpbin.org/post',
  files={'file': ('index.html', open('index.html', 'rb'), 'text/html', {'Expires': '0'})}
)
```

[[doc]](https://docs.python-requests.org/en/latest/user/quickstart/#post-a-multipart-encoded-file)

## cURL 

```shell
curl -F "file=@index.html" https://httpbin.org/post
curl -F "file=@index.html;type=text/html" https://httpbin.org/post
```

[[doc of `-F/--form` option]](https://curl.se/docs/manpage.html#-F)

# Disabling SSL verification

## `requests` (Python) 

```python
resp = requests.get(
    'https://httpbin.org/get',
    verify=False,
)
```

[[doc of `verify` parameter]](https://2.python-requests.org/en/master/api/#requests.Session.verify)

## cURL 

```shell
curl -k https://httpbin.org/get
```

[[doc of `-k/--insecure` option]](https://curl.se/docs/manpage.html#-k)

# 3XX redirects

## `requests` (Python) 

`requests` default behavior is to follow redirect, unless making a `HEAD` request (see below for more informations). Use the parameter `allow_redirects` to disable it:

```python
resp = requests.get(
    'https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302',
    allow_redirects=False,
)
```

*Note on redirects with HEAD requests*: when making `HEAD` requests, `requests` will not follow 3XX redirects in most case, depending on how the request is made:

| How the request is made                                               |Follow 3XX on default|
|-----------------------------------------------------------------------|---------------------|
|[`requests.head(...)`](requests-head-source-code)                      | No                  |
|[`requests.Session(...).head(...)`][requests-session-head-source-code] | No                  |
|[`requests.request('HEAD', ...)`][requests-request-source-code]        | Yes                 |

[requests-head-source-code]: https://github.com/psf/requests/blob/b0e025ade7ed30ed53ab61f542779af7e024932e/requests/api.py#L90
[requests-session-head-source-code]: https://github.com/psf/requests/blob/b0e025ade7ed30ed53ab61f542779af7e024932e/requests/sessions.py#L568
[requests-request-source-code]: https://github.com/psf/requests/blob/b0e025ade7ed30ed53ab61f542779af7e024932e/requests/sessions.py#L470

## cURL 

cURL's default behavior is to not follow 3XX redirects. To enable this use the `-L/--location` option:

```shell
curl -L https://httpbin.org/redirect-to?url=bastien-antoine.fr&status_code=302
```

[[doc of `-L/--location` option]](https://curl.se/docs/manpage.html#-L)

# Logging

## `requests` (Python) 

```python
import requests
import logging
import http.client

http.client.HTTPConnection.debuglevel = 1

logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True
```

[[source]](https://stackoverflow.com/a/26460395)

## cURL 

```shell
curl -v https://httpbin.org/get
```

Note that this option only shows requests headers and full response. If you want to see the whole process, including request body, checkout the [`--trace`](https://curl.se/docs/manpage.html#--trace) and [`--trace-ascii`](https://curl.se/docs/manpage.html#--trace-ascii) options.

[[doc of `-v/--verbose` option]](https://curl.se/docs/manpage.html#-v)