How I Built a Video Transcoder using AWS MediaConvert
Tags: aws, serverless, node
Published: 2022-08-19 05:13:58

---
title: How I Built a Video Transcoder using AWS MediaConvert
published: true
date: 2022-08-15 16:50:27 UTC
tags: AWS, serverless, nodejs
canonical_url: https://devswisdom.com/how-i-built-a-video-transcoder-using-aws-mediaconvert/
---

In this post, you are going to see how I created a Serverless Video Transcoding Pipeline using AWS MediaConvert using Nodejs.

There will be two parts to this project this is the first part where I will be showing you how I built the backend for this using AWS Serverless.

Let’s get started by creating a blank folder with `serverless.yml` file which will be the core file to deploy our Serverless stack to AWS.

## Creating Serverless.yml file

```yaml
service: video-transcoding-pipeline

provider:
  name: aws
  region: ${file(./env.yml):${opt:stage}.REGION}
  runtime: nodejs14.x
  versionFunctions: false
  tracing:
    lambda: true

functions:
  - ${file(./lambdaFunctions.yml)}

resources:
  - ${file(./permissions.yml)}
  - ${file(./db.yml)}
  - ${file(./s3.yml)}
```

As you can see here that we are importing a bunch of _yml_ files which we will be creating next, we are also setting the region which is imported from the env file of the project.

To know more about _serverless.yml_ file check out “What is a serverless.yml file?” section [here](https://devswisdom.com/use-websockets-with-aws-serverless/).

## Creating S3 buckets

```yaml
Resources:
  MediaInputBucket:
    Type: AWS::S3::Bucket
    Properties:
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders: ["*"]
            AllowedMethods: [GET, PUT, POST]
            AllowedOrigins: ["*"]

  MediaOutputBucket:
    Type: AWS::S3::Bucket
    Properties:
      CorsConfiguration:
        CorsRules:
          - AllowedHeaders: ["*"]
            AllowedMethods: [GET, PUT, POST]
            AllowedOrigins: ["*"]
      AccessControl: PublicRead
```

Now we will be creating the `s3.yml` file which will be responsible for creating the S3 buckets, we are creating two buckets here.

The `MediaInputBucket` is the input bucket where the video file will be uploaded to get transcoded.

The `MediaOutputBucket` is the output bucket where the transcoded video will be saved by AWS MediaConvert.

- **CorsRules** : This config is used to set the Cors for the buckets so we can interact with the buckets through the client side (these can be changed according to the need).
- **AccessContro** l: This is giving the public access to the bucket so transcoded videos can be played publicly.

To check more configurations provided for S3 bucket creation, check out the official [documentation](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket.html).

## Creating a DynamoDB Table

```yaml
Resources:
  VideoStatus:
    Type: AWS::DynamoDB::Table
    Properties:
      TableName: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
      PointInTimeRecoverySpecification:
        PointInTimeRecoveryEnabled: true
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - AttributeName: id
          KeyType: HASH
      BillingMode: PAY_PER_REQUEST
```

Here a DynamoDB table is being created, this table will be used to store the AWS MediaConvert Job status (more on this later).

Also, you can see that the table name is also being imported from the env file, so let’s create this file now.

## Creating the env.yml file

```yaml
prod:
  MEDIA_INPUT_BUCKET: !Ref MediaInputBucket
  MEDIA_OUTPUT_BUCKET: !Ref MediaOutputBucket
  REGION: us-east-2
  VIDEO_STATUS_TABLE: VideoStatusTable
  ACCOUNT_ID: [REPLACE_THIS_WITH_YOUR_ACCOUNT_ID]
  MEDIA_ENDPOINT: [REPLACE_THIS_WITH_YOUR_ENDPOINT]
  MEDIA_CONVERT_ROLE: !GetAtt MediaConvertRole.Arn
```

Here we are creating a bunch of env variables under the _prod_ stage name.

- **MEDIA\_ENDPOINT** : This is the endpoint for MediaConvert which you can get from your AWS Console by going under the _Account_ section in the MediaConvert dashboard.
- **MEDIA\_CONVERT\_ROLE** : This is the IAM role for AWS MediaConvert.

## Creating permissions.yml file

Now it’s time to create the `permissions.yml` file, there will be two roles created in this file, one will be used by all the Lambda functions and another one will be used by AWS MediaConvert.

Let’s break down this file as it is a bit long.

### Creating policy for interacting with DynamoDB

```yaml
Resources:
  LambdaRole:
    Type: "AWS::IAM::Role"
    Properties:
      RoleName: "LambdaRole-${opt:stage}"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: "sts:AssumeRole"
      Policies:
        - PolicyName: "LambdaRolePolicy-${opt:stage}"
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - "dynamodb:PutItem"
                  - "dynamodb:UpdateItem"
                  - "mediaconvert:*"
                Effect: Allow
                Resource: "*"
```

This policy will allow the lambda functions to interact with the DynamoDB table.

### Creating policy for interacting with AWS MediaConvert

```yaml
Policies:
  - PolicyName: 'MediaConvertLambdaPolicy-${opt:stage}'
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Sid: PassRole
          Effect: Allow
          Action:
            - 'iam:PassRole'
          Resource: !GetAtt MediaConvertRole.Arn
        - Sid: MediaConvertService
          Effect: Allow
          Action:
            - 'mediaconvert:*'
          Resource:
            - '*'
        - Sid: MediaInputBucket
          Effect: Allow
          Action:
            - 's3:*'
          Resource:
            - '*'
```

This policy will allow the Lambda functions to interact with AWS MediaConvert, to read more about how these permissions work check out [this](https://docs.aws.amazon.com/mediaconvert/latest/ug/auth_access_required-permissions.html) official documentation by AWS.

### Creating policy to write CloudWatch Log streams

```yaml
Policies:
  - PolicyName: 'CloudWatchLogsPolicy-${opt:stage}'
    PolicyDocument:
      Version: '2012-10-17'
      Statement:
        - Action:
            - 'logs:CreateLogGroup'
            - 'logs:CreateLogStream'
            - 'logs:PutLogEvents'
          Effect: Allow
          Resource:
            - >-
              arn:aws:logs:${file(./env.yml):${opt:stage}.REGION}:${file(./env.yml):${opt:stage}.ACCOUNT_ID}:*

```

This is straightforward as we are allowing the lambda log to be created in the same region and AWS account where we are deploying the stacks.

Now we will create the second role which will be attached to the MediaConvert.

### Creating IAM role for AWS MediaConvert

```yaml
MediaConvertRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: "MediaConvertRole-${opt:stage}"
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - "mediaconvert.amazonaws.com"
                - "mediaconvert.us-east-2.amazonaws.com"
            Action:
              - sts:AssumeRole
      Policies:
        - PolicyName: "MediaConvertPolicy"
          PolicyDocument:
            Statement:
              - Effect: "Allow"
                Action:
                  - "s3:*"
                Resource:
                  - "*"
              - Effect: "Allow"
                Action:
                  - "cloudwatch:*"
                  - "logs:*"
                Resource:
                  - "*"
```

This role will allow AWS MediaConvert to interact with S3 and also be able to write AWS CloudWatch logs to the AWS account.

That was a lot to take in but now you are done with creating core _yml_ files, now there is only one _yml_ file left that will create all the lambda functions which are needed, so let’s start with that.

## Creating lambdaFunctions.yml file

```yaml
startJob:
  handler: resolvers/job/startJob.handler
  name: ${opt:stage}-startJob
  timeout: 600
  role: LambdaRole
  description: Lambda function to start the media convert job
  environment:
    VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
    MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
    MEDIA_OUTPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_OUTPUT_BUCKET}
    MEDIA_ENDPOINT: ${file(./env.yml):${opt:stage}.MEDIA_ENDPOINT}
    REGION: ${file(./env.yml):${opt:stage}.REGION}
    MEDIA_CONVERT_ROLE: ${file(./env.yml):${opt:stage}.MEDIA_CONVERT_ROLE}
  events:
      - s3:
          bucket: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
          event: s3:ObjectCreated:*
          existing: true

getSignedUrl:
  handler: resolvers/getSignedUrl.handler
  name: ${opt:stage}-getSignedUrl
  timeout: 600
  role: LambdaRole
  description: Lambda function to get the signed url to upload the video
  environment:
    MEDIA_INPUT_BUCKET: ${file(./env.yml):${opt:stage}.MEDIA_INPUT_BUCKET}
  events:
      - http:
          path: getSignedUrl
          method: post
          cors: true

updateJobStatus:
  handler: resolvers/job/updateJobStatus.handler
  name: ${opt:stage}-updateJobStatus
  timeout: 600
  role: LambdaRole
  description: Lambda function to update the media convert job status in the DB
  environment:
    VIDEO_STATUS_TABLE: ${file(./env.yml):${opt:stage}.VIDEO_STATUS_TABLE}
    REGION: ${file(./env.yml):${opt:stage}.REGION}
  events:
      - cloudwatchEvent:
          event:
            source:
              - 'aws.mediaconvert'
            detail-type:
              - 'MediaConvert Job State Change'
```

There are three lambda functions that are being created here.

- **startJob** : This lambda function will be responsible to start the AWS MediaConvert job and it will be called whenever any file will be uploaded to the input S3 bucket which you created earlier.
- **getSignedUrl** : This lambda function will return the signed URL to upload the video file to the input bucket from the client side.
- **updateJobStatus** : This lambda function will be updating the MediaConvert job status to the DynanmoDB table and it will be called whenever the job status gets changed in MediaConvert.

Now you are done with creating all the required _yml_ files, let’s move on to creating the resolvers for the lambda functions.

## getSignedUrl Lambda Resolver

This lambda function will be called first to get the signed URL back and then that signed URL will be used to upload the video file to S3 from the client side so we are uploading the video from the backend.

### Adding validations

```javascript
const {
	fileName,
	metaData
} = JSON.parse(event.body)

if (!fileName || !fileName.trim()) {
	return sendResponse(400, {
		message: 'Bad Request'
	})
}
```

Here you are getting the file name and the metadata from the client side and you are also checking that the file name must exist otherwise 400 Status code is being returned.

The `sendResponse` is a utility function that is just sending the response to the API request, you can find it in the source code.

### Creating the signed URL

```javascript
const params = {
	Bucket: process.env.MEDIA_INPUT_BUCKET,
	Key: fileName,
	Expires: 3600,
	ContentType: 'video/*',
	Metadata: {
		...metaData
	}
}

const response = s3.getSignedUrl('putObject', params)
```

Here parameters are being created and `getSignedUrl` API call is made to get the signed URL, `ContentType` is set to _video/\*_ because only videos will be uploaded to the S3 bucket from the client side.

Now when the file will get uploaded to the S3 bucket by the client application using this signed URL, `startJob` lambda function will be triggered which will start the AWS MediaConvert job, let’s see what this lambda function looks like.

## startJob Lambda Resolver

The first thing that I wanna show you is what are the imports that are added in this lambda resolver.

### Imports

```javascript
const {
	sendResponse
} = require('../../utilities/index')
const AWS = require('aws-sdk')
AWS.config.mediaconvert = {
	endpoint: `https://${process.env.MEDIA_ENDPOINT}.mediaconvert.${process.env.REGION}.amazonaws.com`
}
const MediaConvert = new AWS.MediaConvert({
	apiVersion: '2017-08-29'
})
const s3 = new AWS.S3()
const params = require('./mediaParams.js')
const dbClient = new AWS.DynamoDB.DocumentClient()
```

Notice here that I am updating the endpoint for the MediaConvert config, also there is a file named `mediaParams.js` which is being imported here.

This file will hold the configuration for starting the MediaConvert job, so we will now create this file first.

### Creating mediaParams.js config file

```javascript
module.exports = {
  Settings: {
    TimecodeConfig: {
      Source: 'ZEROBASED'
    },
    OutputGroups: [
      {
        Name: 'Apple HLS',
        Outputs: [
          {
            ContainerSettings: {
              Container: 'M3U8',
              M3u8Settings: {}
            },
            VideoDescription: {
              Width: '',
              Height: '',
              CodecSettings: {
                Codec: 'H_264',
                H264Settings: {
                  MaxBitrate: '',
                  RateControlMode: 'QVBR',
                  SceneChangeDetect: 'TRANSITION_DETECTION'
                }
              }
            },
            AudioDescriptions: [
              {
                CodecSettings: {
                  Codec: 'AAC',
                  AacSettings: {
                    Bitrate: 96000,
                    CodingMode: 'CODING_MODE_2_0',
                    SampleRate: 48000
                  }
                }
              }
            ],
            OutputSettings: {
              HlsSettings: {}
            },
            NameModifier: 'hgh'
          }
        ],
        OutputGroupSettings: {
          Type: 'HLS_GROUP_SETTINGS',
          HlsGroupSettings: {
            SegmentLength: 10,
            MinSegmentLength: 0,
            DestinationSettings: {
              S3Settings: {
                AccessControl: {
                  CannedAcl: 'PUBLIC_READ'
                }
              }
            }
          }
        }
      },
      {
        CustomName: 'Thumbnail Creation Group',
        Name: 'File Group',
        Outputs: [
          {
            ContainerSettings: {
              Container: 'RAW'
            },
            VideoDescription: {
              Width: 1280,
              Height: 720,
              CodecSettings: {
                Codec: 'FRAME_CAPTURE',
                FrameCaptureSettings: {
                  FramerateNumerator: 1,
                  FramerateDenominator: 5,
                  MaxCaptures: 5,
                  Quality: 80
                }
              }
            }
          }
        ],
        OutputGroupSettings: {
          Type: 'FILE_GROUP_SETTINGS',
          FileGroupSettings: {
            DestinationSettings: {
              S3Settings: {
                AccessControl: {
                  CannedAcl: 'PUBLIC_READ'
                }
              }
            }
          }
        }
      }
    ],
    Inputs: [
      {
        AudioSelectors: {
          'Audio Selector 1': {
            DefaultSelection: 'DEFAULT'
          }
        },
        VideoSelector: {},
        TimecodeSource: 'ZEROBASED'
      }
    ]
  },
  AccelerationSettings: {
    Mode: 'DISABLED'
  },
  StatusUpdateInterval: 'SECONDS_60',
  Priority: 0
}

```

As you can see that there are a lot of parameters added here but most of these values are static in this project, you will only be modifying transcoded video width/height and bitrate (many many more configurations can be made dynamic according to the requirement).

### Fetching the metadata from the uploaded file

```javascript
const fileKey = event.Records[0].s3.object.key
const {
	metaData
} = await fetchMetaData(fileKey)
```

Here you are getting the uploaded file key (which will be received in the lambda trigger attached to the S3 bucket) and calling `fetchFromS3` function.

#### Creating fetchFromS3 function

```javascript
async function fetchMetaData (key) {
  try {
    const params = {
      Bucket: MEDIA_INPUT_BUCKET,
      Key: key
    }

    const response = await s3.headObject(params).promise()
    return { metaData: response.Metadata }
  } catch (err) {
    throw new Error(err)
  }
}
```

### Creating the params for starting the MediaConvert job

```javascript
const input = `s3://${MEDIA_INPUT_BUCKET}/${fileKey}`
const output = `s3://${MEDIA_OUTPUT_BUCKET}/`

params.Role = MEDIA_CONVERT_ROLE
params.Settings.OutputGroups[0].OutputGroupSettings.HlsGroupSettings.Destination = output
params.Settings.OutputGroups[1].OutputGroupSettings.FileGroupSettings.Destination = output
params.Settings.Inputs[0].FileInput = input
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Width = metaData.videowidth || 1920
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.Height = metaData.videoheight || 1080
params.Settings.OutputGroups[0].Outputs[0].VideoDescription.CodecSettings.H264Settings.MaxBitrate = metaData.videobitrate || 6000000

const response= await MediaConvert.createJob(params).promise()
```

We are setting the IAM role for MediaConvert and other settings with metadata as we discussed previously.

### Creating initial entry for the created job in the DB

```javascript
const vodObj = {
	TableName: VIDEO_STATUS_TABLE,
	Item: {
		id: response.Job.Id,
		createdAt: new Date().toISOString(),
		vodStatus: 'SUBMITTED'
	},
	ConditionExpression: 'attribute_not_exists(id)'
}

await dbClient.put(vodObj).promise()
```

We are taking the created job id and making it a Sort key in the DynamoDB table and we are also setting the initial job status to SUBMITTED.

Now it is time to work on the last lambda function resolver.

## updateJobStatus Lambda Resolver

```javascript
try {
    const { VIDEO_STATUS_TABLE, REGION } = process.env
    const { jobId, status, outputGroupDetails } = event.detail

    const params = {
      TableName: VIDEO_STATUS_TABLE,
      Key: {
        id: jobId
      },
      ExpressionAttributeValues: {
        ':vodStatus': status
      },
      UpdateExpression: 'SET vodStatus = :vodStatus',
      ReturnValues: 'ALL_NEW'
    }

    if (status !== 'INPUT_INFORMATION') {
      if (status === 'COMPLETE') {
        const splitOutput = outputGroupDetails[0].outputDetails[0].outputFilePaths[0].split('/')
        params.ExpressionAttributeValues[':outputPath'] = `https://${splitOutput[2]}.s3.${REGION}.amazonaws.com/${splitOutput[3]}`
        params.UpdateExpression += ', outputPath = :outputPath'
      }

      await dbClient.update(params).promise()
    }
  } catch (err) {
    return sendResponse(500, { message: 'Internal Server Error' })
  }
```

This will be the final lambda function resolver you will need, this lambda will be called whenever the status of the MediaConvert job changes and it will update the new status to the DynamoDB table using the job id we stored earlier.

There are three main stages of a job progression –

- **SUBMITTED** : This is the initial job status when it gets started and this is being stored by `startJob` lambda function.
- **PROGRESSING** : This is the status when the job is going on and it will be set through this lambda function.
- **COMPLETE** : This is the final status when the job gets successfully completed.

If you want to read more about the different stages of a job, you can check [here](https://docs.aws.amazon.com/mediaconvert/latest/ug/how-mediaconvert-jobs-progress.html).

AND we are done, pat yourself on the back if you have reached this far, there are a lot of improvements that can be done in this project.

## Improvements

- MediaConvert endpoint can be fetched using `describeEndpoints` API, read more [here](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/emc-examples-getendpoint.html).
- More configuration can be added to the AWS MediaConvert startJob parameters.
- Multi-part upload can be implemented to upload larger video files.
- Job status can be pushed to SNS topic to use in other places.
- AWS CloudFront distribution can be used to distribute the transcoded video.

## Conclusion

Today you saw how we can create a video transcoder using AWS MediaConvert with Serverless and Nodejs, you can play around with it and have fun adding new things, there will be a part 2 to this series where i will be showing how to make the Frontend for this

Find the full source code [here](https://github.com/shivangchauhan7/serverless-video-transcoder).

_Check more posts:_

What is [AWS Athena](https://devswisdom.com/aws-athena-all-you-need-to-know/)?

[AWS Cognito Pricing](https://devswisdom.com/aws-cognito-authentication-with-serverless-and-nodejs/)

[DynamoDB VS MongoDB: Detailed Comparison](https://devswisdom.com/dynamodb-vs-mongodb-detailed-comparison/)

[AWS DynamoDB Pricing](https://devswisdom.com/aws-dynamodb-pricing-and-features/) Model and Features Explained

The post [How I Built a Video Transcoder using AWS MediaConvert](https://devswisdom.com/how-i-built-a-video-transcoder-using-aws-mediaconvert/) appeared first on [DevsWisdom](https://devswisdom.com).