Validate Your PHP API Tests Against OpenAPI Definitions – a Laravel Example [2023 update]
Tags: php, openapi, testing, laravel
Published: 2022-03-22 12:03:37

[OpenAPI](https://swagger.io/specification/) is a specification for describing RESTful APIs in JSON and YAML format, aiming at being understandable by both humans and machines.

OpenAPI definitions are language-agnostic and can be used in several ways:

> _An OpenAPI definition can be used by documentation generation tools to display the API, code generation tools to generate servers and clients in various programming languages, testing tools, and many other use cases._

– [The OpenAPI Specification](https://swagger.io/specification/)

This article demonstrates how to write integration tests that compare API responses to OpenAPI 3+ definitions to validate that the former conform to the latter.

We will use the [OpenAPI HttpFoundation Testing](https://github.com/osteel/openapi-httpfoundation-testing) package in a fresh [Laravel](https://laravel.com/) installation for which we'll also generate a [Swagger UI](https://swagger.io/tools/swagger-ui/) documentation using the [L5 Swagger](https://github.com/DarkaOnLine/L5-Swagger) package.

The first part of the post will describe the issue and provide further context – if you're just here for the code, you're welcome to skip ahead and go to the _Laravel example_ section straight away.

## The issue

APIs are commonplace nowadays and the decent ones come with documentation describing the endpoints and how to use them. These documents come in various shapes and forms, but a fact they can't escape is that they need to be updated every time the API changes.

To many developers, API documentation is an afterthought – it's boring, tedious, and often unrewarding. Using annotations to keep the code and the documentation in one place helps to some extent, but these are often painful to write and even the most willing developer is likely to introduce errors that won't necessarily be caught by reviewers.

The result is documentation and API getting out of sync, leading to frustrated consumers.

Another goal of API maintenance is to ensure that existing endpoints keep functioning properly – regressions can and will appear over time, and may go unnoticed if there isn't a proper testing strategy in place.

One approach is to write integration tests controlling the API's behaviour and automatically detecting breaking changes as soon as they are introduced. While this is a good strategy, it isn't foolproof – there's no guarantee that the expectations set in the tests will always 100% match the documentation.

How can we make sure they remain in sync?

## A solution

Let's assume that we have an API documented with OpenAPI, some integration tests, and that we want to align the expectations of the latter with the operations described in the former.

OpenAPI comes with a growing number of [tools](https://openapi.tools) built on top of it, pushing its utility far beyond the documenting aspect.

One tool destined for the PHP community and maintained by [The PHP League](https://thephpleague.com) is [OpenAPI PSR-7 Message Validator](https://github.com/thephpleague/openapi-psr7-validator), a package validating [PSR-7](https://www.php-fig.org/psr/psr-7) HTTP messages against OpenAPI definitions.

The idea is to compare HTTP requests and responses with OpenAPI definitions and return errors when they don't match.

In other words, we can use this package to add an extra layer on top of our integration tests that will compare API requests and responses with the OpenAPI definitions describing our API and fail the test if they don't match.

Here is a fancy diagram to illustrate the relationship between the API, the integration tests and the OpenAPI definition:

![Relationship between OpenAPI, API and tests](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/6h1moahl5vf6olvl7vsy.png)

The definition describes the API, and the tests use the definition to make sure the API behaves the way the definition says it does.

All of a sudden, our OpenAPI definition becomes a reference for both our code and our tests – it acts as the API's single source of truth.

## PSR-7

One caveat of the OpenAPI PSR-7 Message Validator package is that, as its name indicates, it only works with [PSR-7](https://www.php-fig.org/psr/psr-7) messages.

The issue is that not all frameworks support this standard by default – in fact, a lot of them use Symfony's [HttpFoundation component](https://symfony.com/components/HttpFoundation) under the hood, whose requests and responses do not implement that standard out of the box.

The Symfony folks thought of this, however, and provided [a bridge](https://symfony.com/doc/current/components/psr7.html) that converts HttpFoundation objects to PSR-7 ones. The bridge simply needs a PSR-7 and [PSR-17](https://www.php-fig.org/psr/psr-17) factory, for which they suggest using [Tobias Nyholm](https://github.com/Nyholm)'s [PSR-7 implementation](https://github.com/Nyholm/psr7).

All these pieces are brought together by [OpenAPI HttpFoundation Testing](https://github.com/osteel/openapi-httpfoundation-testing), a package allowing developers to back their integration tests with OpenAPI definitions in applications relying on the HttpFoundation component.

Let's see how to use it in a Laravel project, which falls into this category.

## A Laravel example

The code featured in this section is also available as a [GitHub repository](https://github.com/osteel/openapi-httpfoundation-testing-laravel-example/) for reference.

First, let's create a new Laravel 10 project, using Composer:

```shell
$ composer create-project --prefer-dist laravel/laravel openapi-example "10.*"
```

Enter the project's root folder and instal a couple of dependencies:

```shell
$ cd openapi-example
$ composer require --dev osteel/openapi-httpfoundation-testing
$ composer require darkaonline/l5-swagger -W
```

The first one is the [OpenAPI HttpFoundation Testing](https://github.com/osteel/openapi-httpfoundation-testing) package mentioned earlier, which we instal as a development dependency as it's intended to be used as part of our test suite.

The second one is [L5 Swagger](https://github.com/DarkaOnLine/L5-Swagger), a popular package bringing [Swagger PHP](https://github.com/zircote/swagger-php) and [Swagger UI](https://swagger.io/tools/swagger-ui/) to Laravel. We actually don't need Swagger PHP here, as it uses Doctrine annotations to generate OpenAPI definitions and we're going to manually write our own instead. We do need Swagger UI, however, and L5 Swagger conveniently adapts it to work with Laravel.

If you're wondering, the `-W` option is simply here to also update related dependencies and thus avoid conflicts.

To make sure Swagger PHP doesn't overwrite the OpenAPI definition, let's set the following environment variable in the `.env` file, at the root of the project:

```ini
L5_SWAGGER_GENERATE_ALWAYS=false
```

Let's also make YAML the default format:

```ini
L5_FORMAT_TO_USE_FOR_DOCS=yaml
```

Create a file named `api-docs.yaml` in the `storage/api-docs` folder (which you need to create), and add the following content to it:

```yaml
openapi: 3.1.0

info:
    title: OpenAPI HttpFoundation Testing Laravel Example
    version: 1.0.0

servers:
    - url: http://localhost:8000/api

paths:
    '/test':
    get:
        responses:
        '200':
            description: Ok
            content:
            application/json:
                schema:
                type: object
                required:
                    - foo
                properties:
                    foo:
                    type: string
                    example: bar
```

This is a simple OpenAPI definition describing a single operation – a `GET` request to the `/api/test` endpoint, that should return a JSON object containing a required `foo` key.

Let's check whether Swagger UI displays our OpenAPI definition correctly. Start the PHP development server (from the project's root):

```shell
$ php artisan serve
```

Open [localhost:8000/api/documentation](http://localhost:8000/api/documentation) – you should see the OpenAPI definition rendered as a Swagger UI documentation:

![Swagger UI](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tp9tw7gznj4tlqk8i1pz.png)

Expand the `/test` endpoint and try it out – it should fail with a `404 Not Found` error because we haven't implemented it yet.

Let's fix that now. Open the `routes/api.php` file and replace the example route with this one:

```php
Route::get('/test', function (Request $request) {
    return response()->json(['foo' => 'bar']);
});
```

Go back to the Swagger UI tab and try the endpoint again – it should now return a successful response.

Time to write a test! Open `tests/Feature/ExampleTest.php` and replace its content with the following:

```php
<?php

namespace Tests\Feature;

use Osteel\OpenApi\Testing\ValidatorBuilder;
use Tests\TestCase;

class ExampleTest extends TestCase
{
    /**
     * A basic test example.
     *
     * @return void
     */
    public function testBasicTest()
    {
        $response = $this->get('/api/test');

        $validator = ValidatorBuilder::fromYamlFile(storage_path('api-docs/api-docs.yaml'))->getValidator();

        $result = $validator->validate($response->baseResponse, '/test', 'get');

        $this->assertTrue($result);
    }
}
```

Let's unpack the above. If you're unfamiliar with Laravel, `$this->get()` is a test method provided by the `MakesHttpRequests` trait to perform a `GET` request to the provided endpoint, executing the request's lifecycle without leaving the application. It returns a response that is identical to one you would obtain if you'd perform the same request from the outside.

We then create a validator using the `Osteel\OpenApi\Testing\ValidatorBuilder` class, to which we feed the YAML definition we wrote earlier via the `fromYamlFile` static method (the `storage_path` function is a helper returning the path to the `storage` folder, where we placed the definition).

Had we had a JSON definition instead, we could have used the `fromJsonFile` method (note that other methods are available and listed in the [documentation](https://github.com/osteel/openapi-httpfoundation-testing#usage "Usage")).

The builder returns an instance of `Osteel\OpenApi\Testing\Validator`, on which we call the `get` method, passing the path and the response as parameters (`$response` is an instance of `Illuminate\Testing\TestResponse` here – a wrapper for the underlying HttpFoundation object, which can be retrieved through the `baseResponse` public property).

The above is basically the equivalent of saying:

> _I want to validate that this response conforms to the OpenAPI definition of a `GET` request to the `/test` path._

It could also be written this way:

```php
$result = $validator->get($response->baseResponse, '/test');
```

That's because the validator exposes a shortcut method for each of the HTTP methods supported by OpenAPI (`GET`, `POST`, `PUT`, `PATCH`, `DELETE`, `HEAD`, `OPTIONS` and `TRACE`), to make it simpler to test responses for the corresponding operations.

Note that the specified path must exactly match one of the OpenAPI definition's [paths](https://swagger.io/specification/#paths-object).

You can now run the test, which should be successful:

```shell
$ php artisan test tests/Feature
```

Open `routes/api.php` again, and change the route for this one:

```php
Route::get('/test', function (Request $request) {
    return response()->json(['baz' => 'bar']);
});
```

Run the test again – it should now fail because the response contains `baz` instead of `foo`, and the OpenAPI definition says the latter is expected.

Our test is officially backed by our OpenAPI definition!

The above is an oversimplified example for demonstration purposes, but in real-life situations, a good practice would be to overwrite the `MakesHttpRequests` trait's `call` method, so it performs both the request and the OpenAPI check.

This way, our test would boil down to this line:

```php
$this->get('/api/test');
```

This could be implemented as a new `MakesOpenApiRequests` trait that would "extend" the `MakesHttpRequests` one, and that would first call the parent `call` method to get the response. It would then work out the path from the URI, and validate the response against the OpenAPI definition before returning it, for the calling test to perform any further assertions, as needed.

The OpenAPI HttpFoundation Testing package has more to offer, like [extension points](https://github.com/osteel/openapi-httpfoundation-testing#extending-the-package) and [caching](https://github.com/osteel/openapi-httpfoundation-testing#caching) to speed up the execution of your tests (Laravel's caching API is supported out of the box) – please refer to the [documentation](https://github.com/osteel/openapi-httpfoundation-testing) for details.

## Conclusion

While this setup is a great step towards improving an API's robustness, it is no silver bullet – it requires that every single endpoint is covered with integration tests, which is not easily enforceable in an automated way and ultimately still requires some discipline and vigilance on the part of developers.

It may even feel a bit coercive at first since, as a result, maintainers are basically _forced_ to keep the documentation up to date to write successful tests.

The added value, however, is that said documentation is now guaranteed to be much more accurate, leading to happy consumers which, in turn, should lead to less frustrated developers, who will spend less time hunting down pesky discrepancies.

All in all, making OpenAPI definitions the single source of truth for both the API documentation and integration tests is in itself a strong incentive to keep them up to date. They naturally become a priority, where they used to be an afterthought.

As for maintaining the OpenAPI definition itself, doing so manually can feel a bit daunting. Annotations are a solution, but I personally don't like them and prefer to maintain a YAML file directly. IDE extensions like [this one for VSCode](https://marketplace.visualstudio.com/items?itemName=42Crunch.vscode-openapi) make it much easier, but if you can't bear the sight of a YAML or JSON file, you can also use [tools](https://openapi.tools/#gui-editors) like [Stoplight Studio](https://stoplight.io/studio) to do it through a more user-friendly interface.

And since we're talking about Stoplight, [this article about API Design-First vs Code First](https://stoplight.io/blog/api-design-first-vs-code-first/) by [Phil Sturgeon](https://twitter.com/philsturgeon) is a good starting point for API documentation in general and might help you choose an approach to documenting that works for you.

## Resources

* [The OpenAPI Specification](https://swagger.io/specification)
* [The OpenAPI HttpFoundation Testing package](https://github.com/osteel/openapi-httpfoundation-testing)
* [Laravel example repository](https://github.com/osteel/openapi-httpfoundation-testing-laravel-example)
* [Swagger UI](https://swagger.io/tools/swagger-ui)
* [The L5 Swagger Laravel package](https://github.com/DarkaOnLine/L5-Swagger)
* [OpenAPI.Tools](https://openapi.tools)
* [The OpenAPI PSR-7 Message Validator package](https://github.com/thephpleague/openapi-psr7-validator)
* [The HttpFoundation component](https://symfony.com/components/HttpFoundation)
* [PSR-7: HTTP message interfaces](https://www.php-fig.org/psr/psr-7)
* [The PSR-7 Bridge](https://symfony.com/doc/current/components/psr7.html)
* [PSR-7 implementation](https://github.com/Nyholm/psr7)
* [API Design-First vs Code First](https://stoplight.io/blog/api-design-first-vs-code-first)