GraphQL Schema Test with GraphQL Inspector
Tags: graphql, testing, apitesting, tutorial
Published: 2020-07-30 23:52:23

### Background 
We were nearing the alpha release of **one of our new modules** which includes the release of a **new mobile application.** Our web app, new mobile app and our desktop app, all consume our graphQL API. 
To increase the reliability of our platform, we needed to formalize and implement a backwards compatibility and deprecation policy for our GraphQL API.
The rules to add or delete from schema were
* Only add new fields. 
* If you want to remove a field, deprecate it and remove it when it’s no longer in use. 
* If you want to change an existing field, create a new field, deprecate the old field and remove it once it’s no longer in use.

However, since the mobile app has a much larger footprint as users might be using the app with the older version with old graphQL schema, having a test for it seemed inevitable. Our existing e2e automation on mobile did not cover all graphQL endpoints, and it would have been too expensive and time consuming to cover all endpoints from a user interface perspective. Moreover, we wanted backward compatibility test to be a test that runs before we run e2e tests to detect breaking changes earlier in the cycle.

Since we needed something light weight and graphQL compatible, we decided to use [GraphQL Inspector Tool](https://graphql-inspector.com/) to **ensure backward compatibility** and prevent schema breaking changes.

### Tech Stack
* Bitbucket (Git)
* Codeship Pro for CI/CD workflow
* Docker
* GraphQL

### Code Structure
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/40iho56itks2rj4ygbor.png)

### Why GraphQL Inspector Tool?
We decide to use GraphQL Inspector Tool for graphQL backward capability test because of following reasons:
* GraphQL compatibility 
* Ease of implementation with custom rules 
* Easy CLI command for CI integration 

### Steps to Implement
##### Step1: Create a dockerfile for [GraphQL Inspector](https://graphql-inspector.com/) 
* Dockerfile template can be found here at [graphQL inspector docker hub](https://hub.docker.com/r/kamilkisiela/graphql-inspector)
* Dockerfile can be named as `Dockerfile.graphqlInspector` in `./codeship` folder
```dockerfile
FROM node:10-slim

ENV LOG_LEVEL "debug"
ARG PRIVATE_SSH_KEY

RUN apt update 
RUN apt install -y ssh
RUN yarn global add @graphql-inspector/cli@2.1.0 graphql
RUN apt install -y git-all

# Copy the decrypt ssh private key for Bitbucket Readonly User to allow codeship access to 
# Our repository from codeship container  
RUN mkdir -p /root/.ssh
RUN echo $PRIVATE_SSH_KEY >> /root/.ssh/id_rsa
RUN chmod 600 /root/.ssh/id_rsa
RUN ssh-keyscan -H bitbucket.org >> /root/.ssh/known_hosts

RUN mkdir /app
WORKDIR /app
```
##### Step2: Create an entrypoint file
* This file will be triggered after docker image is built 
* Name file `entrypoint-graphql-inspector.sh`  in `./codeship` folder
   * To add git remote for our repository
      * Note: it uses remote name as `api` because `origin` already taken and it doesn't work in codeship
   * To fetch the master branch which will use for `graphql inspector command` in codeship
```bash
#!/bin/bash
git remote add api git@bitbucket.org:<owner_name>/<repo_name>.git
git fetch api master
exec "$@"
```
##### Step3: Create graphQL inspector test service
* `graphql-inspector-test` service will be placed in `./codeship-services.yml`
* Build from `Dockerfile.graphqlInspector`
* Include _entrypoint_ link with `entrypoint-graphql-inspector.sh` to trigger the command line after finishing pulling and building the docker image from _dockerfile_
* For `volumes`, it can be different depending on your code structure or same as created above in _dockerfile_.
   * How to find which volume to mount
   * Install jet if you have not  (for locally running commands for _codeship_)
   * `cd <repo>`
   * `jet run SERVICE_NAME pwd`
   * Example: `jet run graphql-inspector-test pwd`
   * With `pwd`, it will show the path of the volume where it should be mounted 
```yml
services:
  graphql-inspector-test:
    build:
      context: ./
      dockerfile: ./codeship/Dockerfile.graphqlInspector
      encrypted_args_file: <path_to_encrypted_git_ssh_key>
    encrypted_env_file: <path_to_encrypted_git_ssh_key>
    entrypoint: ["sh", "./codeship/entrypoint-graphql-inspector.sh"]
    cached: true
    volumes:
      - .:/app
```
##### Step4: Create graphql-inspector-test step
* *GraphQL Schema Inspector Test* test step under _Jest Tests_ in `./codeship-steps.yml`
* Use `graphql-inspector-test` service
* To trigger command `graphql-inspector diff git:api/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js`

```yml
- type: serial
  name: Continuous Integration
  encrypted_dockercfg_path: dockercfg.encrypted
  steps:
  - name: Tests
    type: parallel
    exclude: ^(production)
    steps:
    - name: Jest Tests
      type: serial
      steps:
      - name: Jest Tests for E2E Tests
        service: e2e-runner
        command: yarn test
      - name: ESLint for E2E Tests
        service: e2e-runner
        command: yarn run lint
      - name: Graph QL Test
        service: graphql-test
        command: yarn run graphql-test
      - name: Graph QL Schema Inspector Test
        service: graphql-inspector-test
        command: graphql-inspector diff
                 git:api/master:./src/generated/graphql/schema.graphql             
                 ./src/generated/graphql/schema.graphql --rule 
                 ./src/generated/graphql/custom-rule.js
  - name: Deploy to Dev and Production
```
##### Step5: How to create custom rules for graphQL inspector test 
* For details, you can see [graphql inspector custom rules](https://graphql-inspector.com/docs/essentials/diff#custom-rules)
* We use _custom-rules_ to skip certain attributes that we don’t want the test to run 
   * Example: the field may not be available to mobile apps yet and still under development, so there may be many changes to the field.
* Create file name `custom-rules.js` in the folder where you have your `schema.graphql` 
```javascript
// custom-rules.js
/**
 * @summary exceptionSet is an array of schema attributes that can be skipped during graphql schema diff test
 * for breaking changes. 
 * !important: by adding to exceptionSet, there is a risk of graphql schema test will skip these attributes
 * example: new Set(['Form.user]);
 */
const exceptionSet = new Set([]);

/**
 * @summary changes is an object array of all breaking changes [{error1},{error2}]
 * Each error object contains these key value pairs as follow 
 * critical: { level: 'BREAKING', reason: "some reason"}
 * type: 'FIELD_TYPE_CHANGED' or 'FIELD_REMOVED' or 'ENUM_VALUE_REMOVED', etc... 
 * message: "Field 'XYZ' was removed from interface 'ABC'"
 * path: graphql schema attribute, eg: 'Comment.description' 
 */
module.exports = ({ changes }) => {
	const included = [];
	const excluded = [];
	for (const change of changes) {
		if (exceptionSet.has(change.path)) {
			excluded.push(change);
			continue;
		}
		included.push(change);
	}
	if (excluded.length > 0) {
		const warnColor = '\x1b[33m';
		console.log('The following changes have been excluded from breaking changes test:');
		for (const change of excluded) {
			// print something sensible - should show that the field was excluded
			// and what potential problem we're ignoring ;-)
			const message = `!!! ${change.path}: ${change.message}`;
			console.log(warnColor, message);
		}
	}
	return included;
};

```
### How to execute test locally 
##### Prerequisite
```bash
npm install --global @graphql-inspector/cli graphql
```
##### Steps
```bash
cd <repo_folder> 
graphql-inspector diff git:origin/master:./src/generated/graphql/schema.graphql ./src/generated/graphql/schema.graphql --rule ./src/generated/graphql/custom-rule.js

```
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/mt5t0d9giz9eldcygkfn.png)

### Conclusion
We were successfully able to add *graphQL backward capabilities test* that is easy to maintain, and is platform agnostic by using [GraphQL Inspector Tool](https://graphql-inspector.com/). We are able to gain a better understanding of how API should be handled for projects, where the same API is used across multiple platforms (i.e. web, desktop and mobile).