Recording and Replaying HTTP Interactions with Ease: A Guide to VCR.py
Tags: http, python, testing, tutorial
Published: 2023-07-05 03:19:49

## Introduction

Testing code that interacts with external services can be tricky. You need to make sure that your code is handling all the possible responses correctly, and that it's not making too many unnecessary requests. But how do you test this kind of code reliably and efficiently?

That's where VCR.py comes in. The idea behind VCR.py is simple: it records HTTP interactions made by your code, and then replays them when your tests run. That way, your tests can simulate HTTP requests without actually hitting the external service. Pretty neat, right?

In this article, we'll take a closer look at VCR.py and some of its features. We'll cover how to use VCR.py to record and replay HTTP interactions, how to customize its behavior with matchers and filters, and how to work with the cassette file that VCR.py uses to store recorded interactions.

## Getting Started

To get started with VCR.py, you'll need to install it first. You can do this using pip:

```shell
pip install vcrpy
```

Once you have VCR.py installed, you can start using it in your tests. Here's a simple example:

```python
import requests
import vcr

with vcr.use_cassette('test_api.yaml', record_mode='once'):
    def test_api():
        response = requests.get('https://api.example.com')
        assert response.status_code == 200
```

In this example, we're using VCR.py to intercept a GET request to `https://api.example.com`. The first time this test is run, VCR.py will record the HTTP interaction in a cassette file named `test_api.yaml`. On subsequent runs, VCR.py will replay the recorded interaction from the cassette file instead of making a new request.

Record_mode that passed in with the VCR cassette determines how the interactions with an external API are recorded. There are four record modes available:

1. `once`: This mode will record the interactions with the external API the first time the test is run. Subsequent runs of the test will use the recorded interactions instead of making new requests to the API.

2. `new_episodes`: This mode will record any interactions with the external API that haven't been previously recorded. If an interaction has already been recorded, it will be replayed instead of making a new request.

3. `all`: This mode will record all interactions with the external API, regardless of whether they have been previously recorded or not. This is useful for building up a complete set of recordings for an API.

4. `none`: This mode will turn off recording altogether and will only replay previously recorded interactions. This mode is useful when you want to ensure that your tests are only using recorded interactions and not making any new requests to the external API.

In the previous example, the `record_mode` argument is set to `'once'`, which means that the interactions with the external API will be recorded the first time the test is run and subsequently replayed on all subsequent runs.




## Working with the Cassette File


![VCR Casstte](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/fapg8kh8cq5m3up6p3hz.gif)

The cassette file is where VCR.py stores recorded HTTP interactions. By default, the cassette file is a YAML file, but you can also use JSON or SQLite. The cassette file can be easily read and edited, which makes it easy to inspect the contents of the file and make changes if necessary.

Here's an example of a cassette file:

```yaml
- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "123", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK
```

In this example, we have a single interaction that was recorded. The interaction consists of a POST request to `https://api.example.com/foo` with a JSON request body, and a response with a JSON body containing an ID and a foo value.

One important thing to keep in mind when working with the cassette file is that it can become quite large if you have a lot of interactions recorded. To help with this, VCR.py provides several options for controlling how interactions are recorded and stored in the cassette file.



## Matchers

By default, VCR.py matches HTTP interactions based on the HTTP method, URL, and request body. But what if you want to match on other criteria, like headers or query parameters?

That's where matchers come in. Matchers allow you to specify custom criteria for matching HTTP interactions. Here's an example:

```python
import vcr

def custom_matcher(request1, request2):
    # Your custom matching logic here
    return True

with vcr.use_cassette('my_test.yaml', 
                      match_on=['method', 'uri', custom_matcher]):
    def test_my_function():
        # Code that makes HTTP requests
```

In this example, we define a custom matcher function `custom_matcher` that takes two requests and returns `True` if they match based on our custom criteria. We then use this custom matcher in our `use_cassette` call, along with the default matchers for HTTP method and URI.

## Filters

Sometimes you might want to filter or modify requests or responses before they are recorded or played back. For example, you might want to filter out sensitive data from the response before it gets stored in the cassette file. Or you might want to modify the request headers before the request is sent.

VCR.py provides two options for filtering: `before_record` and `before_playback`. These options allow you to specify functions that are called before requests are recorded or played back, respectively. Here's an example:

```python
import vcr

def filter_request(request):
    request.headers['Authorization'] = 'Bearer xxxxxxx'
    return request

def filter_response(response):
    response.headers.pop('Set-Cookie', None)
    return response

with vcr.use_cassette('my_test.yaml',
                      before_record=filter_request,
                      before_playback=filter_response):
    # Your test code here
```

In this example, we define two filter functions `filter_request` and `filter_response` that modify the request and response respectively. We then pass these filter functions to the `before_record` and `before_playback` options of `use_cassette`, respectively.

## Placeholders

Sometimes you might have dynamic data in your responses that you don't want to hard-code in your tests. For example, if you have a test that creates a new resource and returns a unique ID, you might want to use a placeholder to represent that ID in the cassette file.

VCR.py supports placeholders in the cassette file. Placeholders are strings that are replaced with actual values during playback. Here's an example:

```yaml
- request:
    body: !!binary |
      eyJ0ZXN0IjoidGVzdCJ9
    headers:
      Content-Type: application/json
    method: POST
    uri: https://api.example.com/foo
  response:
    body: !!binary |
      {"id": "{{ ID }}", "foo": "bar"}
    headers:
      Content-Type: application/json
    status:
      code: 200
      message: OK
```

In this example, we're using the `{{ ID }}` placeholder in the response body to represent the unique ID that is generated by the API. When VCR.py replays the response, it will substitute the placeholder with the actual ID value that was returned by the API during the recording.

## Basic Illustration

Let's assume that we have a simple Flask web application that acts as an external web service API. It returns a JSON response  when we make a GET request to the `/hello` endpoint:

```python
from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/hello')
def hello():
    return jsonify({'message': 'Hello, World!'})

if __name__ == '__main__':
    app.run()
```

Now, let's write a vcrpy test to test this API. We'll use the `requests` library to make the HTTP request to the API:

```python
import vcr
import requests

def test_hello_api():
    with vcr.use_cassette('hello_api.yaml', record_mode='once'):
        response = requests.get('http://localhost:5000/hello')
        assert response.status_code == 200
        assert response.json() == {'message': 'Hello, World!'}
```

In this test, we're using the `use_cassette` context manager to wrap the HTTP request to our Flask app. We're specifying that we want to use the cassette file `hello_api.yaml` to record the interactions with the API, and we're using the `record_mode` argument to specify that we only want to record the interactions once (i.e., on the first run of the test).

When we run this test for the first time, vcrpy will record the interaction with the Flask app and save it to the `hello_api.yaml` file. On subsequent runs of the test, vcrpy will replay the interaction from the cassette file instead of making a new HTTP request to the Flask app.

To run this test, we can simply execute the test file using a test runner such as pytest:

```shell
$ pytest test_hello_api.py
```

This will run the test and output the results of the assertions. If the test passes, we should see output similar to the following:

```shell
============================ test session starts ============================
collected 1 item

test_hello_api.py .                                                   [100%]

============================= 1 passed in 0.13s =============================
```

If the test fails, we'll see an error message indicating which assertion failed.

here's what the `hello_api.yaml` cassette file might look like after vcrpy records the HTTP interaction with the Flask app (ie. our external API):

```yaml
interactions:
- request:
    body: null
    headers:
      Accept-Encoding: identity
      Connection: keep-alive
      Host: localhost:5000
      User-Agent: python-requests/2.25.1
    method: GET
    uri: http://localhost:5000/hello
  response:
    body:
      encoding: utf-8
      string: '{"message": "Hello, World!"}'
    headers:
      Connection: Keep-Alive
      Content-Length: '25'
      Content-Type: application/json
      Date: Mon, 05 Jul 2023 12:34:56 GMT
      Keep-Alive: timeout=5, max=100
      Server: Werkzeug/2.0.2 Python/3.9.5
    status:
      code: 200
      message: OK
  recorded_at: 2023-07-05T12:34:56.789012Z
  recorded_with: vcrpy/4.1.1
```

This file contains a YAML representation of the HTTP interaction between the test and the Flask app. The `interactions` section contains a list of all the HTTP interactions that were recorded, in this case just one. The `request` and `response` sections contain the request and response headers and bodies, respectively. The `recorded_at` and `recorded_with` fields indicate when and with what version of vcrpy the interaction was recorded.


## Conclusion

VCR.py is a powerful tool for testing code that interacts with external services. Its ability to record and replay HTTP interactions makes it easy to write tests that are reliable and efficient. With VCR.py, you can customize how interactions are matched and filtered, and work with the cassette file to inspect and modify recorded interactions.

I hope this article has given you a good introduction to VCR.py and its features. If you're interested in learning more, I encourage you to check out the [official documentation](https://vcrpy.readthedocs.io/en/latest/) and start experimenting with VCR.py in your own tests.