Overcoming Next.js Deployment Challenges on Google App Engine: Image Cache, Application Size, and Warm-Up Requests
Tags: nextjs, googlecloud
Published: 2023-07-18 14:08:43

## Introduction

In this article, we're going to address the challenges we encountered when deploying a Next.js application to Google App Engine and share the solutions we developed for each. The challenges arose after deploying the app created by running the `npx create-next-app` command. They were as follows:

1. Write errors occurring on App Engine when requests were made to images using next/image
2. The App Engine image size was somewhat large at around 400MB
3. Uncertainty about how to configure warmup requests to improve App Engine's performance

## 1. Write errors occurring on App Engine when requests were made to images using next/image

By using `next/image`, you can easily implement features like WebP compatibility, image quality adjustment, and responsive image delivery. For more details, refer to the [official Next.js documentation](https://nextjs.org/docs/api-reference/next/image). 
However, when we deployed to App Engine and made requests to images (PNG, JPG, GIF files, etc., that require image optimization processing other than SVG) using `next/image`, we encountered the following error in the App Engine logs:

`Failed to write image to cache uWMUG7XySLA2rI+bcHWmDuWP7Bt4LUSGBKIB2Vd-X08= [Error: ENOENT: no such file or directory, mkdir '/workspace/.next/cache/images'] {`

Code for verification:
- https://github.com/shoito/nextjs-appengine-experiments/tree/main/nextjs-appengine-next-image-cache-error
- https://github.com/shoito/nextjs-appengine-experiments/tree/main/nextjs-appengine-next-image-unoptimized

This is an error that occurs when `next/image` is unable to create a cache directory when caching images. The official documentation states that, in the case of App Engine, `/workspace/.next/cache/images` is used as the cache directory. However, in the App Engine Standard environment, writing is only permitted in the `/tmp` directory, so writing to the `/workspace/.next/cache/images` cache directory used by `next/image` is not possible.

https://nextjs.org/docs/pages/api-reference/components/image#caching-behavior
> Caching Behavior - Next.js
> Images are optimized dynamically upon request and stored in the <distDir>/cache/images directory. 

https://cloud.google.com/appengine/docs/standard/nodejs/using-temp-files
> Reading and writing temporary files - Google App Engine
> While Cloud Storage is the recommended solution for reading and writing files in App Engine, if your app only needs to write temporary files, you can use language-specific methods to write files to a directory named /tmp.

You can avoid the error by disabling image optimization by setting `images.unoptimized` to `true` in next.config.js, but this will increase the size of your images, which could adversely affect your Web Core Vitals score. However, it does prevent the cost of optimizing every request due to the absence of caching for optimized images.

```ts
const nextConfig = {
    images: {
        // Disable Next.js' default behavior of optimizing images.
        // https://nextjs.org/docs/app/api-reference/components/image#unoptimized
        unoptimized: true,
    }
}

module.exports = nextConfig
```

You can also delegate the image optimization process to a cloud provider using a custom loader, which may be a good option.
https://nextjs.org/docs/app/api-reference/next-config-js/images

Incidentally, we investigated whether it is possible to specify the cache directory under `/tmp`, but as of now (07-18-2023), it is hard-coded as follows and cannot be changed in next.config.js.
https://github.com/vercel/next.js/blob/0084166caaba402c1f8218da48e32855bc7ee48b/packages/next/src/server/image-optimizer.ts#L290

Discussion about the cache directory on the `vercel/next.js` repository.
> Allow for external cache/images dir for next/image processing #38066
https://github.com/vercel/next.js/discussions/38066

## 2. The image size of App Engine is quite large, around 400MB 

We found that the image size of App Engine was around 400MB after deploying using the `npx create-next-app` command.

![400MB](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/tytkryajrpys57qh8vmu.png)

Code for verification
- https://github.com/shoito/nextjs-appengine-experiments/tree/main/nextjs-appengine-initial-setup
- https://github.com/shoito/nextjs-appengine-experiments/tree/main/nextjs-appengine-standalone

We executed and confirmed from `npx create-next-app` to `gcloud app deploy`, `gcloud app browse` in the following flow.

```sh
$ npx create-next-app
Need to install the following packages:
  create-next-app@13.4.10
Ok to proceed? (y) 
✔ What is your project named? … nextjs-appengine-initial-setup
✔ Would you like to use TypeScript? … No / Yes
✔ Would you like to use ESLint? … No / Yes
✔ Would you like to use Tailwind CSS? … No / Yes
✔ Would you like to use `src/` directory? … No / Yes
✔ Would you like to use App Router? (recommended) … No / Yes
✔ Would you like to customize the default import alias? … No / Yes
Creating a new Next.js app in ~/Workspaces/nextjs-appengine-experiments/nextjs-appengine-initial-setup.

Using npm.

Initializing project with template: app-tw 


Installing dependencies:
- react
- react-dom
- next
- typescript
- @types/react
- @types/node
- @types/react-dom
- tailwindcss
- postcss
- autoprefixer
- eslint
- eslint-config-next


added 350 packages, and audited 351 packages in 36s

131 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
Initialized a git repository.

Success! Created nextjs-appengine-initial-setup at ~/Workspaces/nextjs-appengine-experiments/nextjs-appengine-initial-setup
```

We created an `app.yaml` for App Engine settings.

```sh
$ cat <<EOF > app.yaml
runtime: nodejs20
instance_class: F1
service: default
EOF
```

Deployed to App Engine and checked in the browser.

```sh
$ gcloud app deploy
$ gcloud app browse
```

By default, Next.js mainly has a large `node_modules` directory that impacts the image size of App Engine. To address this, we set it up as follows in next.config.js, and tried using standalone mode.

```ts
const nextConfig = {
    output: 'standalone'
}

module.exports = nextConfig
```

https://nextjs.org/docs/pages/api-reference/next-config-js/output
> Automatically Copying Traced Files
Next.js can automatically create a standalone folder that copies only the necessary files for a production deployment including select files in node_modules.

Next, we build in standalone mode and try to deploy to App Engine.
After `npm run build`, which only copies necessary files, we collect directories such as `.next/standalone/{.next, node_modules, package.json, server.js}`, `.next/static`, `public` for deployment, and then perform `gcloud app deploy`.

```sh
mkdir dist
# Build Next.js app for standalone deployment
npm ci && npm run build
cp -rf .next/standalone/. dist/
cp -rf .next/static/. dist/.next/static/
cp -rf public/. dist/public/
# Copy app.yaml for App Engine
cp app.yaml .gcloudignore package*.json dist/
# Deploy to App Engine
gcloud app deploy dist/app.yaml
```

However, this does not result in the expected deployment outcome.
By default, the gcloud CLI does not upload the node_modules folder, and npm install is performed during deployment, so even if built in standalone mode, the image size of App Engine remained around 400MB.

https://cloud.google.com/appengine/docs/standard/nodejs/runtime?hl=en
> Node.js Runtime Environment - App Engine
> Dependencies
> During deployment, the runtime installs your dependencies using the npm install command. 
> Note: By default, the gcloud CLI does not upload your node_modules folder. 

Even though it's easy with Cloud Run...
https://zenn.dev/team_zenn/articles/nextjs-standalone-mode-cloudrun

When we checked the logs of Cloud Build executed after `gcloud app deploy`, we found that `npm install` was indeed being executed. It seems that `npm` is executed if `package-lock.json` is present in the directory, and `yarn` is executed if `yarn.lock` is present.
https://cloud.google.com/docs/buildpacks/nodejs?hl=en
https://github.com/GoogleCloudPlatform/buildpacks/blob/main/cmd/nodejs/npm/main.go

Even if you want to use standalone mode, due to the above behavior of App Engine, `npm install` is executed when deploying with `gcloud app deploy`, resulting in a large image size.

Therefore, when deploying with `gcloud app deploy`, we thought of deleting `package.json` and `package-lock.json` to prevent `npm install` from being executed, and including the `node_modules` built in standalone mode in the `.gcloudignore` file for upload, in order to reduce the image size.

Furthermore, by default, App Engine runs the application by `node server.js`, or if package.json is present, by `npm run start`. This time, we assumed that `package.json` is unnecessary because we run `server.js` built in standalone mode.

https://cloud.google.com/appengine/docs/standard/nodejs/runtime?hl=en#application_startup
> Application Startup
> By default, the runtime starts your application by running node server.js. If you specify a start script in your package.json file, the runtime runs the specified start script instead. 

However, there's a risk of exceeding the allocation limit of App Engine, which is `10,000 files per app`, a default limit on the number of files. In an experimental scenario where I installed several npm modules, the number of files exceeded 10,000, resulting in a failed deployment. Even if this limit can be circumvented, having a large number of files can lead to time-consuming uploads/deployments. 
https://cloud.google.com/appengine/quotas#Files

To tackle this, we archive `node_modules` and upload it as a `node_modules.tar.gz` file to reduce the number of files and avoid reaching the allocation limit on App Engine. In addition, we remove `dependencies` and `devDependencies` from `package.json` to prevent `npm install` from running during deployment (this might feel a bit forced).

https://github.com/shoito/nextjs-appengine-experiments/blob/main/nextjs-appengine-standalone/build-standalone.sh

```sh
#!/bin/bash

rm -rf dist .next
mkdir dist
# Build Next.js app for standalone deployment
npm ci && npm run build
cp -rf .next/standalone/. dist/
cp -rf .next/static/. dist/.next/static/
cp -rf public/. dist/public/
# Copy app.yaml for App Engine
cp app.yaml .gcloudignore package*.json dist/
# Change directory to dist for gcloud app deploy command
cd dist
# Archive node_modules for skipping npm install on App Engine
tar -zcvf node_modules.tar.gz node_modules
# Remove dependencies from package.json for skipping npm install on App Engine with jq command
jq 'del(.dependencies)' package.json > package.json.tmp && mv package.json.tmp package.json
jq 'del(.devDependencies)' package.json > package.json.tmp && mv package.json.tmp package.json
jq 'del(.optionalDependencies)' package.json > package.json.tmp && mv package.json.tmp package.json
# Remove scripts from package.json for skipping npm run start on App Engine with jq command
jq 'del(.scripts.start)' package.json > package.json.tmp && mv package.json.tmp package.json
```

Finally, we use a custom build step to unpack the uploaded `node_modules.tar.gz` file within the deployment flow of `gcloud app deploy`.

> Running custom build steps - App Engine
> By default, if a build script is detected in the package.json file, the node.js runtime runs npm run build. If you need further control over the build step before starting your application, you can specify a custom build step. To run a custom build step, add gcp-build to your package.json file.
https://cloud.google.com/appengine/docs/standard/nodejs/running-custom-build-step?hl=ja

We add `gcp-build` to `scripts` in `package.json` as shown below.

```json
{
  "name": "nextjs-appengine-standalone",
  "version": "0.1.0",
  "private": true,
  "scripts": {
    ...
    "gcp-build": "tar zxvf node_modules.tar.gz && rm node_modules.tar.gz"
  },
  ...
}
```

Now, when you deploy with `gcloud app deploy`, the `node_modules` built in standalone mode will be unpacked, and `npm install` will be skipped.

![standalone](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/jeea8j5g8rpgkdgptffz.png)

Ultimately, **the image size was reduced from 397.3MB to 31.5MB🚀**

To confirm everything is working correctly, access the deployed application with `gcloud app browse`.

![Screenshot](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gyu9ham1wpff96k0s4v6.png)

## 3. Uncertain how to configure warm-up requests for performance improvement

In Google App Engine, you can improve the performance of your application by sending warm-up requests.
If you want to set `min_instances` or `min_idle_instances` in the `autoscaling` settings in `app.yaml`, you need to configure it to accept warm-up requests.
https://cloud.google.com/appengine/docs/standard/nodejs/configuring-warmup-requests?hl=ja

Therefore, it is necessary to add the `/_ah/warmup` request endpoint to the application.
https://nextjs.org/docs/app/api-reference/next-config-js/basePath

Test code:
- https://github.com/shoito/nextjs-appengine-experiments/tree/main/nextjs-appengine-standalone-warmup

This time, we'll try adding a `/_ah/warmup` request endpoint to the custom server used in standalone mode during build.

```sh
# Inject warmup endpoint to server.js
awk '{
  if ($0=="      await nextHandler(req, res)") {
    print "      if (req.url === \"/_ah/warmup\" && req.method === \"GET\") {"
    print "        res.statusCode = 200"
    print "        res.end(\"\")"
    print "        return"
    print "      }"
  }
  print
}' server.js > tmp && mv tmp server.js
```

The script for building looks like this.
If the implementation of `server.js` changes due to an update of Next.js, this script will also need to be modified, so it seems better to implement a custom server and replace `server.js` at build time.

https://github.com/shoito/nextjs-appengine-experiments/blob/main/nextjs-appengine-standalone-warmup/build-standalone.sh

## Conclusion

We introduced the issues encountered during the deployment of a Next.js application to App Engine and each corresponding solution.
Note that we have hacked the default behavior of App Engine, so the situation may change due to future updates of Next.js or App Engine.