How we keep our Serverless deploy times short and avoid headaches
Tags: serverless, javascript, cloud, aws
Published: 2022-02-01 09:30:37

At [BubblyDoo](https://bubblydoo.be) we're building the world's most powerful product personalization platform, and we've gotten this far by using open-source software through all of our projects.

We're using Serverless Framework to deploy most of our backend. AWS Lambda, Cloudflare Workers and Deno Deploy are the Serverless platforms we've been using. Unfortunately, not all projects can be deployed to isolate-based platforms like Cloudflare Workers and Deno Deploy, as many still have binary dependencies or need filesystem access. That's why most of our infrastructure is deployed on AWS Lambda.

But how do you deploy a large Node.js project with hundreds of dependencies, and avoid long deploy times?

We've encountered this problem as well, and we've come up with a solution: the [Serverless Externals Plugin](https://github.com/bubblydoo/serverless-externals-plugin).

## Without any plugins

You create a Javascript file (`lambda.js`) which requires some Node modules. You include the whole `node_modules` folder in the Serverless deployment.
Serverless has some built-in optimizations: it can exclude your dev dependencies, which already helps reduce the size.

```yaml
# serverless.yml
package:
  excludeDevDependencies: true
```

However, there's no tree-shaking, and a lot of unnecessary files are uploaded (e.g. documentation). For some of our deployments this would create zips of 100MB+.

Next to that `excludeDevDependencies` is inefficient and takes a very long time.

## With a bundler

You use a bundler like Webpack, Rollup or esbuild to turn your code and all `node_modules` into a single bundled file (`bundle.js`).

You then exclude all node_modules from the deployment.

```yaml
# serverless.yml
package:
  excludeDevDependencies: false
  patterns:
    - '!node_modules/**'
```

But there's a problem! Not all Node modules can be bundled. There are issues in bundlers, issues in packages, but there are also inherent problems: what if a Node module includes a binary file? In that case, it can't be bundled.

To solve this, we need a way to exclude some modules from the bundle, and keep them external. We can then upload only these modules in the deployment package.

## With Serverless Externals Plugin

We don't like plugins that add magic, so you'll have to configure a few things.

Let's say we made a function that uses `readable-stream`, a module that can't be bundled.

```js
const { Readable } = require('readable-stream');
const _ = require('lodash');

module.exports.handler = () => {
  ... // code using _ and Readable
};
```

The desired result is a bundle that has bundled `lodash`, but keeps the call to `require('readable-stream')`.

You use Rollup, a bundler, to create a single bundled file.

In `rollup.config.js`:

```ts
import { rollupPlugin as externals } from "serverless-externals-plugin";

export default {
  input: { file: "src/lambda.js" },
  output: { file: "dist/bundle.js" },
  ...,
  plugins: [
    externals(__dirname, {
      modules: ["readable-stream"] // <- list external modules
    }),
    commonjs(),
    nodeResolve({ preferBuiltins: true, exportConditions: ["node"] }),
    ...
  ],
}
```

After running `rollup -c`, you'll have your bundle inside `dist/bundle.js`, and a report inside `dist/node-externals-report.json`:

```json
{
  "isReport": true,
  "importedModuleRoots": [
    "node_modules/readable-stream"
  ],
  ...
}
```

Using this report, Serverless knows which node_modules it needs to upload.

In `serverless.yml`:

```yml
plugins:
  - serverless-externals-plugin

functions:
  handler:
    handler: dist/bundle.handler
    package:
      patterns:
        # include only dist
        - "!./**"
        - ./dist/**
    externals:
      report: dist/node-externals-report.json
```


## Advantages of using this plugin

- Node spends a lot of time resolving the correct Node module because it is I/O-bound. This is not great for cold starts. By inlining all code, a bundler basically removes this problem.
- The bundled code is much smaller than the raw files. It's also tree-shaken, meaning unused code is removed.
- The plugin can be added incrementally. If you're already bundling your code but you have one node_module you can't bundle, this plugin is for you.

## How does it do that?

1. The Rollup plugin looks at your `package-lock.json` or your `yarn.lock` and builds a dependency tree for your application.

2. It uses your configuration to mark the right modules and all of their production dependencies as external.

3. It looks at the bundled file and checks which modules are actually imported. If a module isn't imported, it's not packaged.

This is why it doesn't matter if you add too many dependencies to the modules array, the unused ones will be filtered out.

The dependency tree is quite complicated when you take different versions into account, see our [README](https://github.com/bubblydoo/serverless-externals-plugin) for an example. This plugin handles different versions correctly.

### Example

Let's say you have two modules in your `package.json`, `pkg2` and `pkg3`. `pkg3` is a module with native binaries, so it can't be bundled.

```
root
+-- pkg3@2.0.0
+-- pkg2@0.0.1
    +-- pkg3@1.0.0
```

Because `pkg3` can't be bundled, both `./node_modules/pkg3` and `./node_modules/pkg2/node_modules/pkg3` should be included in the bundle. `pkg2` can just be bundled, but should import `pkg3` as follows: `require('pkg2/node_modules/pkg3')`. It cannot just do `require('pkg3')` because `pkg3` has a different version than `pkg2/node_modules/pkg3`.

In the Serverless package, only `./node_modules/pkg3/**` and `./node_modules/pkg2/node_modules/pkg3/**` will be included, all the other contents of `node_modules` are already bundled.

When uploading the whole `node_modules` folder, all requires from `./node_modules/pkg2` to `pkg3` would already require `pkg2/node_modules/pkg3` because of the Node resolution algorithm. Because Rollup isn't made to make only subdependencies external, this plugin rewrites those calls to `require('pkg2/node_modules/pkg3')`.

## How does this compare to other plugins?

### [Serverless Jetpack](https://github.com/FormidableLabs/serverless-jetpack)

Jetpack is great but it doesn't go the bundling way. It does something like a bundler and analyzes the files on which the Lambda code depends, and generates include patterns from there. (in trace mode)
Because of this it doesn't have the benefits of bundling, namely fast module resolution and tree-shaking.

### [Serverless Webpack](https://github.com/serverless-heaven/serverless-webpack)

By default, Serverless Webpack doesn't support externals, but Webpack can use [Webpack Node Externals](https://github.com/liady/webpack-node-externals) to exclude all modules from the bundle. All included modules have to be allowlisted, but this plugin doesn't look at subdependencies.
When used with `custom.webpack.includeModules`, the non-allowlisted modules are added to the deployment zip.

### [Serverless Plugin Tree Shake](https://github.com/sergioramos/serverless-plugin-tree-shake)

There's not much documentation about this plugin, but it also doesn't use bundling. However, it uses `@vercel/nft` to analyze the files on which the Lambda code depends. It seems to support Yarn PnP, which this plugin doesn't.
It overrides the zip function of Serverless to achieve this.

## Used in production

This plugin is used for all our AWS Lambda deployments, using a wide range of Node modules, some with more quirks than others. We use it together with [Lambda Layer Sharp](https://github.com/bubblydoo/lambda-layer-sharp) and [Chrome AWS Lambda](https://github.com/alixaxel/chrome-aws-lambda).

## Webpack and esbuild Plugin

Although Rollup is great, Webpack and esbuild are more feature-rich and faster, respectively. I'd like to create plugins for these bundlers as well if the community is interested. Feel free to open an issue or comment here!