Customize the deployment of AWS Lambda Layer anyhow you like
Tags: aws, serverless, devops
Published: 2022-12-29 23:56:23

## Separate NPM Dependencies into layers

One way to improve deployment time when developing AWS Lambda functions is to separate your function's dependencies into layers. You can save time by deploying the layers once so that the subsequent deployments of your main functions code are as lightweight as possible.

Serverless Framework has built-in support for deploying layers by creating your desired dependencies in a separated `package.json` file and putting the file in the `nodejs` directory under the layer path.

```cmd
# directory structure
layers/
  stablelibs/
    nodejs/
      package.json
  otherLibs/
    nodejs/
      package.json  
serverless.yml
package.json
```

```yml
# serverless.yml

layers:
  commonlibs:
    path: layers/commonlibs

functions:
  ...
```

## The problems

However, this method has some problems.

1. You have to maintain multiple `package.json` files. It will be very difficult to keep the dependencies version in sync in every file, especially if you are using Github's `dependabot` to keep updating the dependencies version.

2. It will involve multiple manual steps for each deployment. For example, you have to go into each layer and run `yarn install` to populate `node_modules` before running the actual deploy command.

3. The deployment time can be even worst because it now has to take time to install dependencies and package each layer for every deployment.

For the second and third points, the existing plugin such as [serverless-layer](https://www.npmjs.com/package/serverless-layers) is a good solution, but in exchange, you will lose some built-in features such as the ability to exclude unnecessary files inside `node_modules` for some packages.

Different projects have different requirements for the workflow. It would be unfortunate to let the framework or plugins limit what you want to do.

I have been researching this problem, and It turns out that Serverless Framework is more flexible and powerful than I thought. It is very easy to customise the deployment workflow anyhow I like. And what follows is an example of what I have done with my current project.

## 1. Separate layers deployment from functions

In my project, I already have multiple `serverless.yml` for each of my API gateways. For example, I have `serverless.internal-api.yml` and `serverless.external-api.yml` for my internal and external API gateways.

The functions in both files share the same dependencies. So I create another file dedicated to the layers only.

```cmd
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml <-- this
package.json
```

Let's start with the simple version of `serverless.layers.yml`

```yml
service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs
```

And then use these layers with the functions in `serverless.internal-api.yml` and `serverless.external-api.yml`

```yml

package:
  individually: true

  # include only the functions code in the dist folder
  patterns:
    - "!**"
    - "dist/**"

providers:
  layers:
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-awslibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-platformlibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-stablelibs:latest
    - arn:aws:lambda:${aws:region}:${aws:accountId}:layer:dependency-layers-${sls:stage}-otherlibs:latest

plugins:
  # This plugin will replace the 'latest' after the ARNs to the actual version number
  - serverless-latest-layer-version
```

Now I can deploy the layers independently from the functions. I have this npm script in `package.json`

```json
scripts: {
  "deploy:layer": "serverless deploy -c serverless.layers.yml",
  "deploy:internal-api": "serverless deploy -c serverless.internal-api.yml",
  "deploy:external-api": "serverless deploy -c serverless.external-api.yml"
}
```

## 2. Use hooks to populate layer's files at built-time

Notice that we will need the following folder structure for the layer files.

```cmd
# directory structure
layers/
  awslibs/
    nodejs/
      node_modules
      package.json
  platformlibs/
    nodejs/
      node_modules
      package.json  
  stablelibs/
    nodejs/
      node_modules
      package.json  
  otherlibs/
    nodejs/
      node_modules
      package.json  
serverless.internal-api.yml
serverless.external-api.yml
serverless.layers.yml
package.json
```

We will populate these files dynamically when running the `deploy:layers` script. To do that, I insert `dependencyIncludeRegexPatterns` for each layer configuration to tell which dependencies to be copied from the main `package.json` to the layer's `package.json`.

```yml
service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"
  platformlibs:
    path: layers/platformlibs
    name: ${self:service}-${sls:stage}-platformlibs
    installCommand: yarn install
    dependencyIncludeRegexPatterns:
      - "^@opentelemetry"
      - "^@nestjs"
  stablelibs:
    path: layers/stablelibs
    name: ${self:service}-${sls:stage}-stablelibs

    dependencyIncludeRegexPatterns:
      - "^date-fns$"
      - "^rxjs$"
  otherlibs:
    path: layers/otherlibs
    name: ${self:service}-${sls:stage}-otherlibs
    dependencyIncludeRegexPatterns:
      - "."
```

Then I use [serverless-plugin-scripts](https://www.serverless.com/plugins/serverless-plugin-scripts) to define a hook script that will run right after we kick off the script `deploy:layer`.

```yml
plugins:
  - serverless-plugin-scripts

custom:
  scripts:
    commands:
      hooks:
        "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
```

Basically, we can do anything in `tools/initializeLayerDependencies.ts`, but here I populate the `package.json` for each layer and run the `installCommand`

```typescript
import { mkdirSync, readFileSync, rmSync, writeFileSync } from "fs";
import * as yaml from "js-yaml";
import packageJson from "../package.json";
import { run } from "./libs/process-utils";

const configFile = process.argv[2];

initializeLayerDependencies(configFile);

// ################################################################

interface LayerConfig {
    path: string;
    installCommand: string;
    yarnCacheFolder: string;
    dependencyIncludeRegexPatterns: string[];
    dependencyExcludeRegexPatterns?: string[];
}

async function initializeLayerDependencies(configFile: string) {
    const config = yaml.load(readFileSync(configFile, "utf8")) as {
        layers: { [layerName: string]: LayerConfig };
    };

    const remainingDependencies = packageJson.dependencies;
    for (const [_layerName, layerConfig] of Object.entries(config.layers)) {
        console.log({ _layerName, layerConfig });

        if (!validConfig(layerConfig)) {
            console.warn("Invalid layerConfig!");
            continue;
        }
        const dependencyPath = `${layerConfig.path}/nodejs`;
        rmSync(dependencyPath, { recursive: true, force: true });
        console.log(`Initializing dependencies for ${layerConfig.path} ...`);
        const layerDependencies = Object.keys(remainingDependencies).reduce(
            (filteredDeps, key) => {
                if (
                    shouldInclude(
                        key,
                        layerConfig.dependencyIncludeRegexPatterns,
                        layerConfig.dependencyExcludeRegexPatterns,
                    )
                ) {
                    filteredDeps[key] = remainingDependencies[key];
                    delete remainingDependencies[key];
                }

                return filteredDeps;
            },
            {},
        );

        mkdirSync(dependencyPath, { recursive: true });
        writeFileSync(
            `${dependencyPath}/package.json`,
            JSON.stringify({ dependencies: layerDependencies }, null, 4),
        );

        writeFileSync(`${dependencyPath}/yarn.lock`, "");
        await run(layerConfig.installCommand, dependencyPath, {
            YARN_CACHE_FOLDER: layerConfig.yarnCacheFolder,
        });
    }
}

function shouldInclude(
    packageName: string,
    includePatterns?: string[],
    excludePatterns?: string[],
) {
    if (
        excludePatterns &&
        excludePatterns.some((regex) => packageName.match(regex))
    ) {
        return false;
    }

    if (
        includePatterns &&
        includePatterns.some((regex) => packageName.match(regex))
    ) {
        return true;
    }

    return false;
}

function validConfig({
    path,
    yarnCacheFolder,
    installCommand,
    dependencyIncludeRegexPatterns,
}: LayerConfig) {
    return (
        path &&
        yarnCacheFolder &&
        installCommand &&
        dependencyIncludeRegexPatterns &&
        true
    );
}

export async function run(
    cmd: string,
    cwd: string,
    env: Record<string, string> = {},
) {
    try {
        execSync(cmd, {
            cwd,
            env: {
                ...process.env,
                ...env,
            },
            maxBuffer: 1024 * 1024 * 500,
            stdio: "inherit",
        });
    } catch (error) {
        console.log(error.stdout?.toString());
        throw error;
    }
}

```

## 3. Improve installment time with yarn cache

We can further speed up the `yarn install` by utilizing `yarnCacheFolder`

```yml
service: dependency-layers
layers:
  awslibs:
    path: layers/awslibs
    name: ${self:service}-${sls:stage}-awslibs
    installCommand: yarn install
    yarnCacheFolder: ../../../.yarn/cache <-- here
    dependencyIncludeRegexPatterns:
      - "^@aws-sdk"
      - "^aws"
```

## 4. Save the dependencies and skip deployment

Imagine the sequence to run in our CI/CD would be like

```
  - yarn install
  - yarn deploy:layer # serverless deploy -c serverless.layers.yml
  - yarn deploy:internal-api # serverless deploy -c serverless.internal-api.yml
  - yarn deploy:external-api # serverless deploy -c serverless.external-api.yml
```

So far so good, but it still takes non-trivial time to build the layers. Serverless Framework can check the deployment hash and skip the deployment, but it does not know when to skip the build (packaging) step.

This is easy. First, we will cache the `dependencies` from the main `package.json` in the SSM parameter store together with the deployment package.

```yml
# serverless.layers.yml

resources:
    Resources:
        LayerDependencies:
            Type: AWS::SSM::Parameter
            Properties:
                Type: String
                Name: /${self:service}/${sls:stage}/dependencies
                Value: ${file(./tools/getDependenciesAsString.js):value}
```

```typescript
# tools/getDependenciesAsString.js

module.exports = async () => {
    const fs = require("fs");
    const packageJson = JSON.parse(fs.readFileSync("package.json", "utf-8"));
    return {
        value: JSON.stringify(packageJson.dependencies),
    };
};

```

Next, we can create a command to compare the cached dependencies with the current dependencies in the main `package.json` and run `yarn deploy:layer` when we detect the diff.

```yml
# serverless.layers.yml

custom:
    scripts:
        commands:
            update: ts-node tools/updateDependencyLayers.ts "/${self:service}/${sls:stage}/dependencies" ap-southeast-1 "yarn deploy:lambda:layer"
        hooks:
            "before:package:initialize": ts-node tools/initializeLayerDependencies.ts serverless.layers.yml
```

```typescript
# tools/updateDependencyLayers.ts

import packageJson from "../package.json";
import _ from "lodash";
import { GetParameterCommand, SSMClient } from "@aws-sdk/client-ssm";
import { run } from "./libs/process-utils";

main(process.argv[2], process.argv[3], process.argv[4]);

async function main(ssmPath: string, region: string, deployCommand: string) {
    const ssm = new SSMClient({ region });
    const response = await ssm.send(
        new GetParameterCommand({
            Name: ssmPath,
        }),
    );
    const cachedDependenciesString = response.Parameter.Value;

    if (!cachedDependenciesString) {
        console.log("Not found dependencies in cache, should deploy layers");
        await run(deployCommand, ".");
        return;
    }
    const cachedDependencies = JSON.parse(cachedDependenciesString);
    if (!_.isEqual(cachedDependencies, packageJson.dependencies)) {
        console.log(
            "Found dependencies in cache and is not equal to dependencies in package.json, should deploy layers",
        );
        await run(deployCommand, ".");
        return;
    }

    console.log("The dependencies are up-to-date, should skip deploy layers");
}

```

Now we can run `npx serverless update -c serverless.layers.yml` to invoke this script.

## Conclusion

It will be much faster for developers to edit the function code and deploy it because the function code is lightweight.

Only when adding new libraries or updating their versions should we then deploy the layers.

The CI/CD pipeline can always run the update, which will skip the deployment if the dependencies are up-to-date.