Requests Library in Python
Tags: beginners, python, api, tutorial
Published: 2022-07-18 22:31:36

As programmers, we have to deal with APIs on an everyday basis. Regardless of your position, there will be a time when you have to use API. If you are working with Python language and want to learn how to use APIs with Python then this blog is for you.

This blog will show you how to use the [Requests](https://docs.python-requests.org/en/latest/) library for different API operations. We will go through the GET, POST, PATCH and DELETE methods of API. To demonstrate all operations, I will use [JSON Placeholder](https://jsonplaceholder.typicode.com/) API. JSON Placeholder is a free fake API for testing and prototyping.

# Purpose of Requests library in Python

To get any kind of data over the internet, we need to use the HTTP protocol. Whether you are fetching a webpage or calling an API, any data coming over the internet has to follow the HTTP protocol. **Requests** library allows us to use this HTTP protocol outside the browser.

> Why use Requests when there are plenty of other options, you may ask?🤔

If you look at the stats of this project, you will believe me. Not only the stats, but I know many people who prefer the Requests library over other HTTP modules. According to the [official page](https://docs.python-requests.org/en/latest/) of Requests python library,  "_Requests allows you to send HTTP/1.1 requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your POST data. Keep-alive and HTTP connection pooling are 100% automatic, thanks to [urllib3](https://github.com/urllib3/urllib3)._" 

![Requests Library stats on GitHub](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hpfyp9ri0zmtvdt3jjqa.png)
Requests on [GitHub](https://github.com/psf/requests)

# Install Requests in Python
Before we can use any third-party library, we have to install it. To install Requests run the following command in your terminal.

```shell
pip install requests
```

On Mac and Linux-based OS, this command might show an error mentioning pip is not available. If it does then try

```shell
pip3 install requests
```

If you are still getting an error, then you might need to install pip first. To install pip on your system, check out this [documentation](https://pip.pypa.io/en/stable/installation/).

To verify installation you can try **import requests** in python shell.
# Different HTTP Methods in Requests
## GET Method in Requests Library
As the name suggests get request is used for fetching data. **So, how do we do that?**
To use the GET method, first, we need to import it. You can use any of the following methods to import GET from the requests module.
```python
from requests import get
response = get('https://jsonplaceholder.typicode.com/posts/1')
```
OR
```python
import requests
response = requests.get('https://jsonplaceholder.typicode.com/posts/1')
```

Here, we have used JSONPLaceholder API. If this call is successful, API will return the following data.
```json
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
```
> So, the question is how do we know that the API call was successful and we got the data.    

To check that we can use either `response.ok`, `response.status_code` or `response.raise_for_status()`. Let me show you what all of these three properties and objects return based on the success of the API call.

```python
from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/1')
print(f"OK: {response.ok}")
print(f"Status Code: {response.status_code}")
print(f"Error: {response.raise_for_status()}")

"""
OUTPUT:
-------
OK: True
Status Code: 200
Error: None
"""
```

For the code above we can see that if the API call is successful then `response.ok` will return True and  `response.status_code` will return a status code of 200 series.  But one thing to notice here is `response.raise_for_status()` did not return anything. ***Why?*** Because we did not receive any error from our backend or API call.

To understand all this better now let's take a look at what happens when an API call is not successful.
```python
from requests import get

response = get('https://jsonplaceholder.typicode.com/garbage')
print(f"OK: {response.ok}")
print(f"Status Code: {response.status_code}")
print(f"Error: {response.raise_for_status()}")

"""
OUTPUT:
-------
OK: False
Status Code: 404
Traceback (most recent call last):
  File "C:\Users\Sahil\Documents\pCloud\Blog\Python\Code\API.py", line 6, in <module>
    print(f"Error: {response.raise_for_status()}")
  File "C:\Users\Sahil\AppData\Local\Programs\Python\Python310\lib\site-packages\requests\models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://jsonplaceholder.typicode.com/garbag
"""
```

You might have noticed already but let me throw some light on it. Unsuccessful  API call returned False for `response.ok` property and 404 for `response.status_code` property. But, unlike successful call, we received an error from `response.raise_for_status()` method. As the name suggests and from the Requests module's official documentation, the **raise_for_status** method will raise HTTPError, if it occurred. In-general, anything except the 200 series is considered an error.

So like many people, I prefer to use the **status_code** property so that I can handle errors manually and it is in my hand how my code behaves when I get an error. No doubt you can use generic error handling if you prefer to use **raise_for_status()**.

> Our request is successful, now what?

To extract data from **response** we need to convert it into JSON format. In this blog, I will focus on textual data. If you want me to write about other kinds of data reach out to me on [Twitter](https://twitter.com/Sahil_Fruitwala3KjwgZV) or [LinkedIn](https://linedin.com/in/sahilfruitwala).

To convert response in JSON, we can use **json()** method.
```python
from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/1')
data = response.json()
print(data)

"""
OUTPUT:
------
{
  "userId": 1,
  "id": 1,
  "title": "sunt aut facere repellat provident occaecati excepturi optio reprehenderit",
  "body": "quia et suscipit\nsuscipit recusandae consequuntur expedita et cum\nreprehenderit molestiae ut ut quas totam\nnostrum rerum est autem sunt rem eveniet architecto"
}
"""
```

By default, Python converts any JSON data to its native dictionary format. If you print **type(data)**, it will return `<class 'dict'>`. So, now we can operate on data as we operate on dictionaries in python. Sometimes, we need to pass an authentication token with the URL to the server. I will show you how to send those later in this blog. First, I will show you how to use the POST method in the Requests library.

## POST Method in Requests Library
The POST method is generally used for the creation of the record. For example, you have to build an e-commerce website and now you want to add the product to your website from the admin panel. To do this, you will create make a POST request with all the necessary data and credentials. Once you make a successful POST API call, you can get new data using the GET method. To make a post request you mainly need two things, a proper endpoint and data. You will need to provide authentication data as well in most of the requests. But for now, let's focus on endpoint and data.

So, if we want to add a new post in JSON Placeholder API, we have to use `/posts` as an endpoint. We will receive the added record as the response from JSON Placeholder API. The response of any method depends on how it is configured. It is not required to send data again but generally, it is a standard to send newly added or updated records back.
```python
from requests import post

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = post('https://jsonplaceholder.typicode.com/posts', data=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 101,
  "title": "foo",
  "body": "bar"
}
"""
```

Here to pass data we have used a Python dictionary. But, if you wish you can use JSON data as well. To use the JSON data instead of `data` as an argument you need to use `json`.
```python
from requests import post

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = post('https://jsonplaceholder.typicode.com/posts', json=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 101,
  "title": "foo",
  "body": "bar"
}
"""
```
You might be thinking it seems the same then why there are two different arguments. For our `new_post_data` variable python automatically converts it into JSON and vice versa. But sometimes we might receive JSON data from other API requests and need to pass it into the body of the POST request. In this kind of scenario, we need to use the `json` argument. Python will convert data to its internal data types but it is not the case every time so most of the time, instead of using `json` people tend to modify or convert data into an internal format and use the `data` argument instead.
## PATCH & PUT Methods in Requests Library
PATCH and PUT, both methods are used for updating data. Most of the time you can use them alternatively. If you want to know more, try this [**Stack Overflow**](https://stackoverflow.com/questions/21660791/what-is-the-main-difference-between-patch-and-put-request) explanation. Here, I will show you how you can use the PATCH method with the Requests package. The PATCH/PUT method needs the same kind of arguments as the POST method. But the main difference is, that instead of calling general endpoint we use an endpoint to the specific record.
For example, to update data for the 2nd record in JSON Placeholder API, we will use the `/posts/2` endpoint.
```python
from requests import get, patch

response = get('https://jsonplaceholder.typicode.com/posts/2')
data = response.json()
print(data)
print('-'*15)

new_post_data = { 'title': 'foo', 'body': 'bar', 'userId': 1}
response = patch('https://jsonplaceholder.typicode.com/posts/2', json=new_post_data)
data = response.json()
print(data)

"""
OUTPUT:
-------
{'userId': 1, 'id': 2, 'title': 'qui est esse', 'body': 'est rerum tempore vitae\nsequi sint nihil reprehenderit dolor beatae ea dolores neque\nfugiat blanditiis voluptate porro vel nihil molestiae ut reiciendis\nqui aperiam non debitis possimus qui neque nisi nulla'}
---------------
{'userId': 1, 'id': 2, 'title': 'foo', 'body': 'bar'}
"""
```
As you might have noticed, we used the same arguments and process as we used in the POST method.

## DELETE Method in Requests Library
As the name suggests, the DELETE method is used to make a record deleted or inactive. We use the specific endpoint as we did in the PATCH method. There are some cases where people create an endpoint to remove all records but the process is the same for that as well. You will have to change the endpoint only.
In our case, to delete 3rd record from JSON Placeholder API. We will use the following code:
```python
from requests import delete

response = delete('https://jsonplaceholder.typicode.com/posts/3')
data = response.json()
print(data)
print(response.status_code)

"""
OUTPUT:
-------
{}
200
"""
```
In the case of the DELETE method, we have not received any data which is the reason we are seeing `{}` in the output. 

Before we move to the next section, I would like to mention that we can use `response.text` as well to retrieve the data. But it will return in form of a string. We have to convert it to dict/json ourselves.
```python
from ast import literal_eval
from json import loads
from requests import get

response = get('https://jsonplaceholder.typicode.com/posts/3')

text_data = response.text
print(text_data)
print(type(text_data))
print('-'*15)

json_data = loads(response.text)
print(json_data)
print(type(json_data))
print('-'*15)

json_data = loads(response.text)
dict_data = literal_eval(text_data)
print(dict_data)
print(type(dict_data))

"""
OUTPUT:
-------
{
  "userId": 1,
  "id": 3,
  "title": "ea molestias quasi exercitationem repellat qui ipsa sit aut",
  "body": "et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut"
}
<class 'str'>
---------------
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
<class 'dict'>
---------------
{'userId': 1, 'id': 3, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
<class 'dict'>
"""
```

If you want to send data in the body, you can use `json.dumps()` to convert **dict** into the **string**.
```python
import json
from requests import post

payload_data = {'userId': 1, 'title': 'ea molestias quasi exercitationem repellat qui ipsa sit aut', 'body': 'et iusto sed quo iure\nvoluptatem occaecati omnis eligendi aut ad\nvoluptatem doloribus vel accusantium quis pariatur\nmolestiae porro eius odio et labore et velit aut'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data))

print(response.json())
"""
OUTPUT:
-------
{'id': 101}
"""
```
# Authentication and Custom Headers in Requests Module
There are many cases when you need to authenticate/identify yourself. This is the scenario in most third-party libraries.
To pass the custom headers in any request. We have to use the **headers** argument.
```python
import json
from requests import post

payload_data = {'userId': 1, 'title': 'foo', 'body': 'bar'}

headers = {'content-type': 'application/json'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data), headers=headers)
print(response.json())
print('-'*15)

headers = {'content-type': 'application/text'}
response = post('https://jsonplaceholder.typicode.com/posts', data=json.dumps(payload_data), headers=headers)
print(response.json())

"""
OUTPUT:
-------
{'userId': 1, 'title': 'foo', 'body': 'bar', 'id': 101}
---------------
{'id': 101}
"""
```
Here, you can notice differences when we use `application/text` and `application/json`. Sometimes, we need to mention explicitly what kind of data we are sending. This is just one case there are many kinds of headers you can use. You can find the whole list on the official [documentation](https://docs.python-requests.org/en/latest/user/quickstart/#custom-headers).

Now comes what you are waiting for. How to pass authentication data. In the example below, I have used Twitter API to fetch tweet data. Twitter has made required to pass either **Bearer Token** or other **OAuth Tokens** when making any requests. 
```python
import json
from requests import get

URL = "https://api.twitter.com/2/tweets?ids=1505694856240377860&tweet.fields=lang,public_metrics"
headers = {'content-type': 'application/json', 'Authorization': f'Bearer {ACCESS_TOKEN}'}

response = get(URL, headers=headers)
print(response.json()['data'][0])

"""
OUTPUT:
-------
{
  "text": "Inspiration is perishable—act on it immediately.\n\n— @naval",
  "id": "1505694856240377860",
  "lang": "en",
  "public_metrics": {
    "retweet_count": 29,
    "reply_count": 7,
    "like_count": 351,
    "quote_count": 0
  }
}
"""
```
As you can see, you just need to add the `Authentication` key-value pair to your headers dictionary. Now I am using a bearer token but if you are using other kind of token then the value will be different from this **Authentication** key.

What if you want to use OAuth authentication instead of Bearer token. To use OAuth authentication, you can use the `requests-oauthlib` library.  You can install it using the following terminal command:
```shell
pip install requests-oauthlib
```

Once you install the **requests-oauthlib** library and verify it, use the following code snippet to use OAuth1 authentication with the Requests module.
```python
import json
from requests import get
from requests_oauthlib import OAuth1

URL = "https://api.twitter.com/2/tweets?ids=1505694856240377860&tweet.fields=lang,public_metrics"
auth = OAuth1(CONSUMER_KEY, CONSUMER_SECRET,ACCESS_KEY, ACCESS_SECRET)

response = get(URL,  auth=auth)
print(response.json()["data"][0])

"""
OUTPUT:
-------
{
  "text": "Inspiration is perishable—act on it immediately.\n\n— @naval",
  "id": "1505694856240377860",
  "lang": "en",
  "public_metrics": {
    "retweet_count": 29,
    "reply_count": 7,
    "like_count": 351,
    "quote_count": 0
  }
}
"""
```

---
# Conclusion
ufffff! 13 Min read ha😆!  

That was it friends. I hope you got a better idea of how to use the Requests library in Python. Apart from calling APIs, there are so many things you can do with the Requests library with a combination of other libraries.

Make sure to share any thoughts, questions, or concerns. I would love to see them. Let me know if you need any help or want to discuss something. Reach out to me on [Twitter](https://twitter.com/Sahil_Fruitwala) or [LinkedIn](https://linkedin.com/in/SahilFruitwala).