NestJS GraphQL image upload into a S3 bucket
Tags: node, typescript, graphql, nestjs
Published: 2022-12-26 13:05:51

## Intro

In this tutorial I will explain how to upload images to a S3-compatible object storage ([AWS S3](https://aws.amazon.com/s3), [DigitalOcean Spaces](https://www.digitalocean.com/products/spaces), [Linode Object Storage](https://www.linode.com/products/object-storage/), etc.) bucket using [NestJS](https://nestjs.com/) and [GraphQL](https://graphql.org/) for both [Apollo](https://www.apollographql.com/docs/apollo-server/) and [Mercurius](https://mercurius.dev/#/) drivers.

## Common

Most of the set-up as well as the entirety of the upload process is the same for both drivers.

### Set Up

Start by installing the [class-transformer](https://github.com/typestack/class-transformer) and [class-validator](https://github.com/typestack/class-validator) packages for dto validation (see more in the [docs](https://docs.nestjs.com/techniques/validation)), [sharp](https://sharp.pixelplumbing.com/) for image optimization, and the S3 client packages.

<small>Note: I assume you have UUID installed, however I still added it to the following command</small>

```bash
$ yarn add class-transformer class-validator sharp @aws-sdk/client-s3 @aws-sdk/signature-v4-crt uuid
$ yarn add -D @types/sharp @types/uuid
```

### Configuration

Assuming you are using nestjs [ConfigModule](https://docs.nestjs.com/techniques/configuration) start by creating an `interfaces` folder.

_Bucket Data_:

```ts
// bucked-data.interface.ts

export interface IBucketData {
  // name of the bucket
  name: string;
  // folder of the service
  folder: string;
  // uuid for the app
  appUuid: string;
  // the bucket url to create the key
  url: string;
}
```

_Upload Middleware Options_:

```ts
// uploader-middleware-options.interface.ts

export interface IUploaderMiddlewareOptions {
  maxFieldSize?: number;
  maxFileSize?: number;
  maxFiles?: number;
}
```

_Uploader Options_:

```ts
// uploader-options.interface.ts
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';
import type { IUploaderMiddlewareOptions } from './uploader-middleware-options.interface';

export interface IUploaderOptions {
  clientConfig: S3ClientConfig;
  bucketData: IBucketData;
  middleware: IUploaderMiddlewareOptions;
}
```

On the `config.ts` function add the following:

```ts
import { IConfig } from './interfaces/config.interface';

export function config(): IConfig {
  const bucketBase = `${process.env.BUCKET_REGION}.${process.env.BUCKET_HOST}.com`;

  return {
    // ...
    uploader: {
      clientConfig: {
        forcePathStyle: false,
        region: process.env.BUCKET_REGION,
        endpoint: `https://${bucketBase}`,
        credentials: {
          accessKeyId: process.env.BUCKET_ACCESS_KEY,
          secretAccessKey: process.env.BUCKET_SECRET_KEY,
        },
      },
      bucketData: {
        name: process.env.BUCKET_NAME,
        folder: process.env.FILE_FOLDER,
        appUuid: process.env.SERVICE_ID,
        url: `https://${process.env.BUCKET_NAME}.${bucketBase}/`,
      },
      middleware: {
        maxFileSize: parseInt(process.env.MAX_FILE_SIZE, 10),
        maxFiles: parseInt(process.env.MAX_FILES, 10),
      },
    },
    // ...
  };
}
```

### Module Creation

The uploader module set up changes depending if it is a library on a nestjs (or nx) mono-repo, or a monolith NestJS GraphQL API.

_Monolith API_

Create the `uploader` module and service:

```bash
$ nest g mo uploader
$ nest g s uploader
```

On the `uploader.service.ts` file add the S3 Client, the bucket data and the logger:

```ts
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';

@Injectable()
export class UploaderService {
  private readonly client: S3Client;
  private readonly bucketData: IBucketData;
  private readonly loggerService: LoggerService;

  constructor(
    private readonly configService: ConfigService,
  ) {
    this.client = new S3Client(
      this.configService.get<S3ClientConfig>('uploader.clientConfig'),
    );
    this.bucketData = this.configService.get<IBucketData>('uploader.bucketData');
    this.loggerService = new Logger(UploaderService.name);
  }
  
  // ...
}
```

On the `uploader.module.ts` file add the global decorator:

```ts
import { Global, Module } from '@nestjs/common';
import { UploaderService } from './uploader.service';

@Global()
@Module({
  providers: [UploaderService],
  exports: [UploaderService],
})
export class UploaderModule {}
```

_Library_

Create the `uploader` library:

- NestJS mono-repo:
    ```bash
    $ nest g lib uploader
    ```
- NX mono-repo:
    ```bash
    $ npx nx @nrwl/nest:library uploader --global --service --strict --no-interactive
    ```

To be able to create a `DynamicModule`, we need to create a library options, start by creating an `interfaces` folder and move the `bucket-data.interface.ts`, and add an `options.interface.ts`:

```ts
import type { S3ClientConfig } from '@aws-sdk/client-s3';
import type { IBucketData } from './bucket-data.interface';

export interface IOptions {
  clientConfig: S3ClientConfig;
  bucketData: IBucketData;
}
```

Create a new `constants` folder and add the options constant:

```ts
// options.constant.ts

export const UPLOADER_OPTIONS = 'UPLOADER_OPTIONS';
```

Unlike a normal module the class parameters come from the options passed to the module and not from the `ConfigService`:

```ts
import { S3Client, S3ClientConfig } from '@aws-sdk/client-s3';
import { Injectable, Logger, LoggerService } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IBucketData } from './interfaces/bucket-data.interface';
import { IOptions } from './interfaces/options.interface';

@Injectable()
export class UploaderService {
  private readonly client: S3Client;
  private readonly bucketData: IBucketData;
  private readonly loggerService: LoggerService;

  constructor(@Inject(UPLOADER_OPTIONS) options: IOptions) {
    this.client = new S3Client(options.clientConfig);
    this.bucketData = options.bucketData;
    this.loggerService = new Logger(UploaderService.name);
  }
  
  // ...
}
```

Turn the module into a dynamic global module:

```ts
import { DynamicModule, Global, Module } from '@nestjs/common';
import { UPLOADER_OPTIONS } from './constants/options.constant';
import { IOptions } from './interfaces/options.interface';
import { UploaderService } from './uploader.service';

@Global()
@Module({})
export class UploaderModule {
  public static forRoot(options: IOptions): DynamicModule {
    return {
      global: true,
      module: UploaderModule,
      providers: [
        {
          provide: UPLOADER_OPTIONS,
          useValue: options,
        },
        UploaderService,
      ],
      exports: [UploaderService],
    };
  }
}
```

### Image Uploading

_Upload Scalar DTO_

The graphql-upload package `Upload` scalar will be process into the following DTO, so add it into a `dtos` folder:

```ts
import { IsMimeType, IsString } from 'class-validator';
import { ReadStream } from 'fs';

export abstract class FileUploadDto {
  @IsString()
  public filename!: string;

  @IsString()
  @IsMimeType()
  public mimetype!: string;

  @IsString()
  public encoding!: string;

  public createReadStream: () => ReadStream;
}
```

_File Validation_

Since we want to upload only images the first thing we need to check is the `mimetype`:

```ts
//...

@Injectable()
export class UploaderService {
  // ...

  private static validateImage(mimetype: string): string | false {
    const val = mimetype.split('/');
    if (val[0] !== 'image') return false;

    return val[1] ?? false;
  }

  // ...
}
```

_Stream Processing_

As you saw the `Upload` scalar returns a read stream that we need to transform into a buffer:

```ts
//...
import { Readable } from 'stream';

@Injectable()
export class UploaderService {
  // ...

  private static async streamToBuffer(stream: Readable): Promise<Buffer> {
    const buffer: Uint8Array[] = [];

    return new Promise((resolve, reject) =>
      stream
        .on('error', (error) => reject(error))
        .on('data', (data) => buffer.push(data))
        .on('end', () => resolve(Buffer.concat(buffer))),
    );
  }

  // ...
}
```

_Image Compression and Conversion_

Before uploading the image we need to optimize it by compressing the image. Start by creating the constants necessary for compression, therefore on a `constants` folder add a file named `uploader.constant.ts`:

```ts
// The max width an image can have
// reducing the pixels will reduce the size of the image
export const MAX_WIDTH = 2160;

// an array with the percentage of quality ranging from 90 to 10%
export const QUALITY_ARRAY = [
  90, 80, 75, 70, 65, 60, 55, 50, 45, 40, 35, 30, 25, 20, 15, 10,
];

// Max image size, I recommend 250kb
export const IMAGE_SIZE = 256000;
```

I also recommend converting all images into a lighter format such as jpeg or webp, in this tutorial I will use jpeg:

```ts
// ...
import sharp from 'sharp';
import {
  IMAGE_SIZE,
  MAX_WIDTH,
  QUALITY_ARRAY,
} from './constants/uploader.constant';

@Injectable()
export class UploaderService {
  // ...
  
  private static async compressImage(
    buffer: Buffer,
    ratio?: number,
  ): Promise<Buffer> {
    let compressBuffer: sharp.Sharp | Buffer = sharp(buffer).jpeg({
      mozjpeg: true,
      chromaSubsampling: '4:4:4',
    });

    if (ratio) {
      compressBuffer.resize({
        width: MAX_WIDTH,
        height: Math.round(MAX_WIDTH * ratio),
        fit: 'cover',
      });
    }

    compressBuffer = await compressBuffer.toBuffer();

    if (compressBuffer.length > IMAGE_SIZE) {
      for (let i = 0; i < QUALITY_ARRAY.length; i++) {
        const quality = QUALITY_ARRAY[i];
        const smallerBuffer = await sharp(compressBuffer)
          .jpeg({
            quality,
            chromaSubsampling: '4:4:4',
          })
          .toBuffer();

        if (smallerBuffer.length <= IMAGE_SIZE || quality === 10) {
          compressBuffer = smallerBuffer;
          break;
        }
      }
    }

    return compressBuffer;
  }

  // ...
}
```

The ratio although not mandatory is a nice to have, so optionally you can create an enum with all common image ratios:

```ts
// ratio.enum.ts

export enum RatioEnum {
  SQUARE = 1, // 192 x 192
  MODERN = 9 / 16, // 1920 x 1080
  MODERN_PORTRAIT = 16 / 9 // 1080 x 1920
  OLD = 3 / 4, // 1400 x 1050
  OLD_PORTRAIT = 4 / 3 // 1050 x 1400
  BANNER = 8 / 47, // 1128 x 192
  ULTRA_WIDE = 9 / 21, // 2560 x 1080
  SUPER_WIDE = 9 / 32, // 3840 x 1080
}
```

This will add consistent with all image sizes on your application.

_File upload_

With the file buffer created, we just need to add a key to our file, and do a `PutObjectCommand` to our bucket:

```ts
import {
  // ...
  PutObjectCommand,
  // ...
} from '@aws-sdk/client-s3';
import {
  // ...
  InternalServerErrorException,
  // ...
} from '@nestjs/common';
import { v4 as uuidV4, v5 as uuidV5 } from 'uuid';
// ...

@Injectable()
export class UploaderService {
  // ...
  
  private async uploadFile(
    userId: number,
    fileBuffer: Buffer,
    fileExt: string,
  ): Promise<string> {
    const key =
      this.bucketData.folder +
      '/' +
      uuidV5(userId.toString(), this.bucketData.appUuid) +
      '/' +
      uuidV4() +
      fileExt;

    try {
      await this.client.send(
        new PutObjectCommand({
          Bucket: this.bucketData.name,
          Body: fileBuffer,
          Key: key,
          ACL: 'public-read',
        }),
      );
    } catch (error) {
      this.loggerService.error(error);
      throw new InternalServerErrorException('Error uploading file');
    }

    return this.bucketData.url + key;
  }

  // ...
}
```

I highly recommend using UUIDs both for the user ID and filename therefore if there is ever a breach on your bucket you are the only one that know which person owns which images.

I assumed that users are saved in a SQL database, if the user ID is not an int but a `UUID` or a `MongoID`, just change it to a string and it should work as expected.

_Image upload_

Putting all the previous private method together, we are able to create a public method that only accepts and uploads optimized images to our bucket:

```ts
// ...
import {
  BadRequestException,
  // ...
  InternalServerErrorException,
  // ...
} from '@nestjs/common';
import { RatioEnum } from './enums/ratio.enum';
// ...

@Injectable()
export class UploaderService {
  // ...
  
  /**
   * Upload Image
   *
   * Converts an image to jpeg and uploads it to the bucket
   */
  public async uploadImage(
    userId: number,
    file: Promise<FileUploadDto>,
    ratio?: RatioEnum,
  ): Promise<string> {
    const { mimetype, createReadStream } = await file;
    const imageType = UploaderService.validateImage(mimetype);

    if (!imageType) {
      throw new BadRequestException('Please upload a valid image');
    }

    try {
      return await this.uploadFile(
        userId,
        await UploaderService.compressImage(
          await UploaderService.streamToBuffer(createReadStream()),
          ratio,
        ),
        '.jpg',
      );
    } catch (error) {
      this.loggerService.error(error);
      throw new InternalServerErrorException('Error uploading image');
    }
  }

  // ...
}
```

_File deletion_

Files should be deleted asynchronously as it is often not a main event and can be done in the background:

```ts
import {
  DeleteObjectCommand,
  // ...
} from '@aws-sdk/client-s3';
// ...

@Injectable()
export class UploaderService {
  // ...
  
  /**
   * Delete File
   *
   * Takes a file url and deletes the file from the bucket
   */
  public deleteFile(url: string): void {
    const keyArr = url.split('.com/');

    if (keyArr.length !== 2 || !this.bucketData.url.includes(keyArr[0])) {
      this.loggerService.error('Invalid url to delete file');
    }

    this.client
      .send(
        new DeleteObjectCommand({
          Bucket: this.bucketData.name,
          Key: keyArr[1],
        }),
      )
      .then(() => this.loggerService.log('File deleted successfully'))
      .catch((error) => this.loggerService.error(error));
  }

  // ...
}
```

## Driver specific

### Apollo Driver

Using the latest version of [graphql-upload](https://www.npmjs.com/package/graphql-upload) with NestJS can be quite tricky since it uses mjs, this means we need to use dynamic imports, alternative you can use [graphql-upload-minimal](https://github.com/flash-oss/graphql-upload-minimal), or graphql-upload version 13.

In order to cover both approaches I will divide this section in 2 parts, one for version 16 and one for version 13.

#### Version 16

Using version 16 is not the best approach for production projects as it forces us to use a lot of experimental features through our app.

Start by installing `graphql-upload`:

```bash
$ yarn add graphql-upload
```

_Middleware Set up_

On your main file dynamically import the `graphql-upload/graphqlUploadExpress.mjs`:

```ts
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  // ...

  const { default: graphqlUploadExpress } = await import(
    'graphql-upload/graphqlUploadExpress.mjs'
  );
  app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(configService.get<number>('port'));
}

bootstrap();
```

_Asynchronous Scalar_

The scalar now will be a function that you add to your `Field` and `Args` decorators:

```ts
// upload-scalar.util.ts
import { GraphQLScalarType } from 'graphql/type';

let GraphQLUpload: GraphQLScalarType;

import('graphql-upload/GraphQLUpload.mjs').then(({ default: Upload }) => {
  GraphQLUpload = Upload;
});

export const uploadScalar = () => GraphQLUpload;
```

In the end a DTO (`ArgsType`) would look something like this:

```ts
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { FileUploadDto } from './file-upload.dto';
import { uploadScalar } from '../utils/upload-scalar.util';

@ArgsType()
export abstract class PictureDto {
  @Field(uploadScalar)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}
```

_Testing_

Using mjs will break your tests, so to be able to use mjs on your app, add the `--experimental-vm-modules` flag to jest:

```json
{
  "...": "...",
  "scripts": {
    "...": "...",
    "test": "yarn node --experimental-vm-modules $(yarn bin jest)",
    "test:watch": "yarn node --experimental-vm-modules $(yarn bin jest)  --watch",
    "test:cov": "yarn node --experimental-vm-modules $(yarn bin jest)  --coverage",
    "test:e2e": "yarn node --experimental-vm-modules $(yarn bin jest)  --config ./test/jest-e2e.json",
    "...": "..."
  },
  "...": "..."
}
```

#### Version 13

Using version 13 is what I recommend, start by installing the package:

```bash
$ yarn add graphql-upload@13
$ yarn add -D @types/graphql-upload
```

_Middleware Set up_

With version 13 we can just do a normal import:

```ts
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import { graphqlUploadExpress } from 'graphql-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';


async function bootstrap() {
  const app = await NestFactory.create(AppModule);
  const configService = app.get(ConfigService);
  // ...
  app.use(graphqlUploadExpress(configService.get<IUploaderMiddlewareOptions>('uploader.middleware')));

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(configService.get<number>('port'));
}

bootstrap();
```

_DTOs_

The `GraphQLUpload` scalar will just be an export from `graphql-upload`:

```ts
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import { GraphQLUpload } from 'graphql-upload';
import { FileUploadDto } from './file-upload.dto';

@ArgsType()
export abstract class PictureDto {
  @Field(() => GraphQLUpload)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}
```

### Mercurius

Mercurius has its own adaptation of graphql-upload, start by installing the [mercurius-upload](https://www.npmjs.com/package/mercurius-upload) package:

```bash
$ yarn add mercurius-upload
```

_Middleware Set up_

As any fastify middleware you just need to import the default value of the library:

<small>Note: do not forget to set `"esModuleInterop": true` on your `tsconfig.json` file</small>

```ts
import { ValidationPipe } from '@nestjs/common';
import { ConfigService } from '@nestjs/config';
import { NestFactory } from '@nestjs/core';
import {
  FastifyAdapter,
  NestFastifyApplication,
} from '@nestjs/platform-fastify';
import mercuriusUpload from 'mercurius-upload';
import { AppModule } from './app.module';
import { IUploaderMiddlewareOptions } from './config/interfaces/uploader-middleware-options.interface';

async function bootstrap() {
  const app = await NestFactory.create<NestFastifyApplication>(
    AppModule,
    new FastifyAdapter(),
  );
  const configService = app.get(ConfigService);
  // ...

  app.register(
    mercuriusUpload,
    configService.get<IUploaderMiddlewareOptions>('uploader.middleware'),
  );

  // ...
  app.useGlobalPipes(new ValidationPipe());
  await app.listen(
    configService.get<number>('port'),
    '0.0.0.0',
  );
}

bootstrap();
```

_DTOs_

`mercurius-upload` is dependent on `graphql-upload` version 15 so you need to import default the scalar from `graphql-upload/GraphQLUpload.js`:

```ts
import { ArgsType, Field } from '@nestjs/graphql';
import { Type } from 'class-transformer';
import { ValidatePromise } from 'class-validator';
import GraphQLUpload from 'graphql-upload/GraphQLUpload.js';
import { FileUploadDto } from './file-upload.dto';

@ArgsType()
export abstract class PictureDto {
  @Field(() => GraphQLUpload)
  @ValidatePromise()
  @Type(() => FileUploadDto)
  public picture: Promise<FileUploadDto>;
}
```

## Conclusion

A complete version of this code can be found in this [repository](https://github.com/tugascript/nestjs-graphql-image-upload). 

## About the Author

Hey there my name is Afonso Barracha, I am a Econometrician made back-end developer that has a passion for GraphQL.

I try to do post once a week here on Dev about Back-End APIs and related topics.

If you do not want to lose any of my posts follow me here on [dev](https://dev.to/tugascript), or on [LinkedIn](https://www.linkedin.com/in/afonso-barracha/).