Spring Boot – Black Box Testing
Tags: java, spring, testing, tutorial
Published: 2022-11-13 13:16:50

In this article, I’m showing you

1. What’s the difference between white box and black testing
1. What are the benefits of the latter
1. How you can implement it in your Spring Boot application
1. How to configure the OpenAPI generator to simplify code and reduce duplications

You can find the code examples in [this repository](https://github.com/SimonHarmonicMinor/spring-boot-black-box-testing-example).

## Domain

We’re developing a restaurant automatization system. There are two domain classes. `Fridge` and `Product`. A fridge can have many products, whilst a product belongs to a single fridge. Look at the classes declaration below.

```java
@Entity
@Table(name = "fridge")
public class Fridge {
    @Id
    @GeneratedValue(strategy = IDENTITY)
    private Long id;

    private String name;

    @OneToMany(fetch = LAZY, mappedBy = "fridge")
    private List<Product> products = new ArrayList<>();
}

@Entity
@Table(name = "product")
public class Product {
    @Id
    @GeneratedValue(strategy = IDENTITY)
    private Long id;

    @Enumerated(STRING)
    private Type type;

    private int quantity;

    @ManyToOne(fetch = LAZY)
    @JoinColumn(name = "fridge_id")
    private Fridge fridge;

    public enum Type {
        POTATO, ONION, CARROT
    }
}
```

> I'm using [Spring Data JPA](https://spring.io/projects/spring-data-jpa) as a persistence framework. Therefore, those classes are [Hibernate](https://hibernate.org/) entities.

## White box testing

This type of testing makes an assumption that we know some implementation details and may interact them. We have 4 REST API endpoints in the system:

1. Create a new `Fridge`.
2. Add a new `Product`.
3. Change the `Product` quantity.
4. Remove the `Product` from the `Fridge`.

Suppose we want to test the one that changes the `Product` quantity. Take a look at the example of the test below.

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
class ProductControllerWhiteBoxTest extends IntegrationSuite {
    @Autowired
    private FridgeRepository fridgeRepository;
    @Autowired
    private ProductRepository productRepository;
    @Autowired
    private TestRestTemplate rest;

    @BeforeEach
    void beforeEach() {
        productRepository.deleteAllInBatch();
        fridgeRepository.deleteAllInBatch();
    }

    @Test
    void shouldUpdateProductQuantity() {
        final var fridge = fridgeRepository.save(Fridge.newFridge("someFridge"));
        final var productId = productRepository.save(Product.newProduct(POTATO, 10, fridge)).getId();

        assertDoesNotThrow(() -> rest.put("/api/product/{productId}?newQuantity={newQuantity}", null, Map.of(
            "productId", productId,
            "newQuantity", 20
        )));

        final var product = productRepository.findById(productId).orElseThrow();
        assertEquals(20, product.getQuantity());
    }
}
```

Let’s examine this piece of code step by step.
We inject repositories to manipulate rows in the database. Then the `TestRestTemplate` comes into play. This bean is used to send HTTP requests. Then you can see that the `@BeforeEach` callback deletes all rows from the database. So, each test runs deterministically. And finally, here is the test itself:

1.  We create a new `Fridge`.
1. Then we create a new `Product` with a quantity of `10` that belongs to the newly created `Fridge`.
1. Afterwards, we invoke the REST endpoint to increase the `Product` quantity from `10` to `20`.
1. Eventually we select the same `Product` from the database and check that the quantity has been increased.

The test works fine. Anyway, there are nuances that should be taken into account:

1. Though the test verifies the entire system behavior (aka functional test) there is a coupling on implementation details (i.e. the database).
2. What the test validates is not the actual use case. If somebody wants to interact with our service, they won’t be able to insert and update rows in the database directly.

As a matter of fact, if we want to test the system from the user perspective, we can only use the public API that the service exposes.

## Black box testing

This type of testing means loose coupling on the system's implementation details. Therefore, we can only depend on the public API (i.e. REST API).

Check out the previous white box test example. How can we refactor it into the black box kind? Look at the `@BeforeEach` implementation below.

```java
@BeforeEach
void beforeEach() {
    productRepository.deleteAllInBatch();
    fridgeRepository.deleteAllInBatch();
}
```

A black box test should not interact with the persistence provider directly. Meaning that there should be a separate REST endpoint clearing all data. Look at the code snippet below.

```java
@RestController
@RequiredArgsConstructor
@Profile("qa")
public class QAController {
    private final FridgeRepository fridgeRepository;
    private final ProductRepository productRepository;

    @DeleteMapping("/api/clearData")
    @Transactional
    public void clearData() {
        productRepository.deleteAllInBatch();
        fridgeRepository.deleteAllInBatch();
    }
}
```

Now we have a particular controller that encapsulates all the clearing data logic. If the test depends only on this endpoint, then we can safely put changes and refactor the method as the application grows. And our black box tests won’t break. The `@Profile(“qa”)` annotation is crucial. We don’t want to expose an endpoint that can delete all user data in production or even development environment. So, we register this endpoint, if `qa` [profile](https://www.baeldung.com/spring-profiles) is active. We’ll use it only in tests.

> The `qa` abbreviation stands for the [quality assurance](https://www.techtarget.com/searchsoftwarequality/definition/quality-assurance). 

And now we should refactor the test method itself. Have a look at its implementation below again.

```java
@Test
void shouldUpdateProductQuantity() {
    final var fridge = fridgeRepository.save(Fridge.newFridge("someFridge"));
    final var productId = productRepository.save(Product.newProduct(POTATO, 10, fridge)).getId();

    assertDoesNotThrow(() -> rest.put("/api/product/{productId}?newQuantity={newQuantity}", null, Map.of(
        "productId", productId,
        "newQuantity", 20
    )));

    final var product = productRepository.findById(productId).orElseThrow();
    assertEquals(20, product.getQuantity());
}
```

There are 3 operations that should be replaced with direct REST API invocations. These are:

1. Creating new `Fridge`.
1. Creating new `Product`.
1. Checking the the `Product` quantity has been increased.

Look at the whole black box test example below.

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@ActiveProfiles("qa")
class ProductControllerBlackBoxTest extends IntegrationSuite {
    @Autowired
    private TestRestTemplate rest;

    @BeforeEach
    void beforeEach() {
        rest.delete("/api/qa/clearData");
    }

    @Test
    void shouldUpdateProductQuantity() {
        // create new Fridge
        final var fridgeResponse =
            rest.postForEntity("/api/fridge?name={fridgeName}", null, FridgeResponse.class, Map.of(
                "fridgeName", "someFridge"
            ));
        assertTrue(fridgeResponse.getStatusCode().is2xxSuccessful(), "Error during creating new Fridge: " + fridgeResponse.getStatusCode());
        // create new Product
        final var productResponse = rest.postForEntity(
            "/api/product/fridge/{fridgeId}",
            new ProductCreateRequest(
                POTATO,
                10
            ),
            ProductResponse.class,
            Map.of(
                "fridgeId", fridgeResponse.getBody().id()
            )
        );
        assertTrue(productResponse.getStatusCode().is2xxSuccessful(), "Error during creating new Product: " + productResponse.getStatusCode());

        // call the API that should be tested
        assertDoesNotThrow(
            () -> rest.put("/api/product/{productId}?newQuantity={newQuantity}",
                null,
                Map.of(
                    "productId", productResponse.getBody().id(),
                    "newQuantity", 20
                ))
        );

        // get the updated Product by id
        final var updatedProductResponse = rest.getForEntity(
            "/api/product/{productId}",
            ProductResponse.class,
            Map.of(
                "productId", productResponse.getBody().id()
            )
        );
        assertTrue(
            updatedProductResponse.getStatusCode().is2xxSuccessful(),
            "Error during retrieving Product by id: " + updatedProductResponse.getStatusCode()
        );
        // check that the quantity has been changed
        assertEquals(20, updatedProductResponse.getBody().quantity());
    }
}
```

The benefits of black box testing in comparison to white box testing are:

1. The test checks the path that a user shall do to retrieve the expected result. Therefore, the verification behavior becomes more robust.
1. Black box tests are highly stable against refactoring. As long as the API contract remains the same, the test should not break.
1. If you accidentally break the backward compatibility (e.g. adding a new mandatory parameter to an existing REST endpoint), the black box test will fail and you'll determine the issue way before the artefact is being deployed to any environment.

However, there is a slight problem with the code that you have probably noticed. The test is rather cumbersome. It’s hard to read and maintain. If I didn’t put in the explanatory comments, you would probably spend too much time figuring out what’s going on. Besides, the same endpoints might be called for different scenarios, which can lead to code duplication.

Luckily there is solution.

## OpenAPI and code generation

Spring Boot comes with a brilliant [OpenAPI](https://swagger.io/specification/) support. All you have to do is to add two dependencies. Look at the [Gradle](https://gradle.org/) configuration below.

```groovy
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'org.springdoc:springdoc-openapi-ui:1.6.12'
```

After adding these dependencies, the OpenAPI specification is available by `GET /v3/api-docs` endpoint.

> The [SpringDoc](https://springdoc.org/) library comes with lots of annotations to tune your REST API specification precisely. Anyway, that's out of context of this article.

If we have the OpenAPI specification, it means that we can generate Java classes to call the endpoints in a type-safe manner. What’s even more exciting is that we can apply those generated classes in our black box tests!

Firstly, let's define the requirements for the upcoming OpenAPI Java client:

1. The generated classes should be put into `.gitignore`. Otherwise, if you have [Checkstyle](https://checkstyle.sourceforge.io/), [PMD](https://pmd.github.io/), or [SonarQube](https://www.sonarqube.org/) in your project, then generated classes can violate some rules. Besides, if you don't put them into `.gitignore`, then each pull request might become huge due to the fact that even a slightest fix can lead to lots of changes in the generated classes.
1. Each pull request build should guarantee that generated classes are always up to date with the actual OpenAPI specification.

How can we get the OpenAPI specification itself during the build phase? The easiest way is to write a separate test that creates the web part of the Spring context, invokes the `/v3/api-docs` endpoint, and put the retrieved specification into `build` folder (if you are Maven user, then it will be `target` folder). Take a look at the code example below.

```java
@SpringBootTest(
    webEnvironment = RANDOM_PORT
)
@AutoConfigureTestDatabase
@ActiveProfiles("qa")
public class OpenAPITest {
    @Autowired
    private TestRestTemplate rest;

    @Test
    @SneakyThrows
    void generateOpenApiSpec() {
        final var response = rest.getForEntity("/v3/api-docs", String.class);
        assertTrue(response.getStatusCode().is2xxSuccessful(), "Unexpected status code: " + response.getStatusCode());
        // the specification will be written to 'build/classes/test/open-api.json'
        Files.writeString(
            Path.of(getClass().getResource("/").getPath(), "open-api.json"),
            response.getBody()
        );
    }
}
```

> The `@AutoConfigureTestDatabase` configures the in-memory database (e.g. [H2](https://www.h2database.com/)), if you have one in the classpath. Since the database provider does not affect the result OpenAPI specification, we can make the test run a bit faster by not using [Testcontainers](https://www.testcontainers.org/).

Now we have the result specification. How can we generate the Java classes based on it? We have another Gradle plugin for that. Take a look at the `build.gradle` configuration below.

```groovy
plugins {
    ...
    id "org.openapi.generator" version "6.2.0"
}

...

openApiGenerate {
	inputSpec = "$buildDir/classes/java/test/open-api.json".toString()
	outputDir = "$rootDir/open-api-java-client".toString()
	apiPackage = "com.example.demo.generated"
	invokerPackage = "com.example.demo.generated"
	modelPackage = "com.example.demo.generated"
	configOptions = [
			dateLibrary    : "java8",
			openApiNullable: "false",
	]
	generatorName = 'java'
	groupId = "com.example.demo"
	globalProperties = [
			modelDocs: "false"
	]
	additionalProperties = [
			hideGenerationTimestamp: true
	]
}
```

> In this article, I'm showing you how to configure the corresponding [Gradle plugin](https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator-gradle-plugin). Anyway, there is [Maven plugin](https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator-maven-plugin) as well and the approach won't be different much.

There is an important detail about OpenAPI generator plugin. It creates the whole Gradle/Maven/SBT project (containing `build.gradle`, `pom.xml`, and `build.sbt` files) but not just Java classes. So, we set the the`outputDir` property as `$rootDir/open-api-java-client`. Therefore, the generated Java classes go into the Gradle subproject.

We should also mark the `open-api-java-client` directory as a subproject in the `settings.gradle`. Look at the code snippet below.

```groovy
rootProject.name = 'demo'
include 'open-api-java-client'
```

All you have to do to generate OpenAPI Java client is to run these Gradle commands:

```shell
gradle test --tests "com.example.demo.controller.OpenAPITest.generateOpenApiSpec"
gradle openApiGenerate
```

## Applying the Java client

Now let’s try our brand new Java client in action. We’ll create a separate `@TestComponent` for convenience. Look at the code snippet below.

```java
@TestComponent
public class TestRestController {
    @Autowired
    private Environment environment;

    public FridgeControllerApi fridgeController() {
        return new FridgeControllerApi(newApiClient());
    }

    public ProductControllerApi productController() {
        return new ProductControllerApi(newApiClient());
    }

    public QaControllerApi qaController() {
        return new QaControllerApi(newApiClient());
    }

    private ApiClient newApiClient() {
        final var apiClient = new ApiClient();
        apiClient.setBasePath("http://localhost:" + environment.getProperty("local.server.port", Integer.class));
        return apiClient;
    }
}
```

Finally, we can refactor our black box test. Look at the final version below.

```java
@SpringBootTest(webEnvironment = RANDOM_PORT)
@ActiveProfiles("qa")
@Import(TestRestControllers.class)
class ProductControllerBlackBoxGeneratedClientTest extends IntegrationSuite {
    @Autowired
    private TestRestControllers rest;

    @BeforeEach
    @SneakyThrows
    void beforeEach() {
        rest.qaController().clearData();
    }

    @Test
    void shouldUpdateProductQuantity() {
        final var fridgeResponse = assertDoesNotThrow(
            () -> rest.fridgeController()
                .createNewFridge("someFridge")
        );
        final var productResponse = assertDoesNotThrow(
            () -> rest.productController()
                .createNewProduct(
                    fridgeResponse.getId(),
                    new ProductCreateRequest()
                        .quantity(10)
                        .type(POTATO)
                )
        );

        assertDoesNotThrow(
            () -> rest.productController()
                .updateProductQuantity(productResponse.getId(), 20)
        );

        final var updatedProduct = assertDoesNotThrow(
            () -> rest.productController()
                .getProductById(productResponse.getId())
        );
        assertEquals(20, updatedProduct.getQuantity());
    }
}
```

As you can see, the test is much more declarative. Moreover, the API contract become statically typed and the parameters validation proceeds during compile time!

## The `.gitignore` caveat and the separate test source

I told that we should put the generated classes to the `.gitignore`. However, if you mark the `open-api-java-client/src` directory as the unindexed by Git, then you suddenly realize that your tests do not compile in CI environment. The reason is that the process of generation the OpenAPI specification (i.e. `open-api.json` file) is an individual test as well. And even if you tell the Gradle to run a single test directly, it will compile everything in the `src/test` directory. In the end, tests don’t compile successfully.

Thankfully, the issue can be solved easily. Gradle provides [source sets](https://docs.gradle.org/current/dsl/org.gradle.api.tasks.SourceSet.html). It's a logical group that splits the code into separate modules that you can compile independently.

Firstly, let's add the [gradle-testsets](https://github.com/unbroken-dome/gradle-testsets-plugin) plugin and define a separate test source that'll contain the `OpenAPITest` file. It's the one that generates the `open-api.json` specification. Take a look at the code example below.

```groovy
plugins {
    ...
    id "org.unbroken-dome.test-sets" version "4.0.0"
}

...

testSets {
    openApiGenerator
}

tasks.withType(Test) {
    group = 'verification'
    useJUnitPlatform()
    testLogging {
        showExceptions true
        showStandardStreams = false
        showCauses true
        showStackTraces true
        exceptionFormat "full"
        events("skipped", "failed", "passed")
    }
}

openApiGenerator.outputs.upToDateWhen { false }

tasks.named('openApiGenerate') {
    dependsOn 'openApiGenerator'
}
```

The `testSets` block declares a new source set called `openApiGenerator`. Meaning that Gradle treats the `src/openApiGenerator` directory like another test source.

The `tasks.withType(Test)` declaration is also important. We need to tell Gradle that every task of `Test` type (i.e. the `test` task itself and the `openApiGenerator` as well) should run with JUnit.

I put the `upToDateWhen` option for convenience. It means that the test that generates `open-api.json` file will be always run on demand and never cached.

And the last block defines that before generating the OpenAPI Java client we should update the specification in advance.

Now we just need to move the `OpenAPITest` to the `src/openApiGenerator` directory and also make a slight change to the `openApiGenerate` task in `build.gradle`. Look at the code snippet below.

```groovy
openApiGenerate {
    // 'test' directory should be replaced with 'openApiGenerator'
    inputSpec = "$buildDir/classes/java/openApiGenerator/open-api.json".toString()
    ....
}
```

Finally, you can build the entire project with these two commands.

```shell
gradle openApiGenerate
gradle build
```

## Conclusion

The black box testing is a crucial part of the application development process. Try it and you’ll notice that the test scenarios become much more representative. Besides, black box test are also great documentation for the API. You can even apply [Spring REST Docs](https://spring.io/guides/gs/testing-restdocs/) and generate a nice manual that’ll be useful both for the API users and QA engineers.

If you have any questions or suggestion, leave your comments down below. Thanks for reading!

## Resources

1. [The repository with code examples](https://github.com/SimonHarmonicMinor/spring-boot-black-box-testing-example)
1. [Spring Data JPA](https://spring.io/projects/spring-data-jpa)
1. [Hibernate](https://hibernate.org/)
1. [Spring Profile](https://www.baeldung.com/spring-profiles)
1. [Quality Assurance](https://www.techtarget.com/searchsoftwarequality/definition/quality-assurance)
1. [OpenAPI](https://swagger.io/specification/)
1. [Gradle](https://gradle.org/)
1. [SpringDoc](https://springdoc.org/)
1. [Checkstyle](https://checkstyle.sourceforge.io/)
1. [PMD](https://pmd.github.io/)
1. [SonarQube](https://www.sonarqube.org/)
1. [H2 database](https://www.h2database.com/)
1. [Testcontainers](https://www.testcontainers.org/)
1. [OpenAPI generator Gradle plugin](https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator-gradle-plugin)
1. [OpenAPI generator Maven plugin](https://github.com/OpenAPITools/openapi-generator/tree/master/modules/openapi-generator-maven-plugin)
1. [Gradle source sets](https://docs.gradle.org/current/dsl/org.gradle.api.tasks.SourceSet.html)
1. [Gradle testsets plugin](https://github.com/unbroken-dome/gradle-testsets-plugin)
1. [Spring REST Docs](https://spring.io/guides/gs/testing-restdocs/)