How to create a sitemap with Next.js app directory
Tags: nextjs, sitemap, contentlayer
Published: 2022-12-07 05:34:05

---
title: How to create a sitemap with Next.js app directory
published: true
tags: nextjs,sitemap,contentlayer
canonical_url: https://sdorra.dev/posts/2022-11-15-sitemaps-with-appdir
cover_image: https://images.unsplash.com/photo-1452421822248-d4c2b47f0c81?fit=crop&w=1000&h=420
---

To gain the best of SEO we should use a sitemap for our sites. Search engines like Google can use those sitemaps to crawl our sites more efficiently.

## Create a sitemap

We could simply create a sitemap, like the following and put it into the public directory of our app:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://sdorra.dev/</loc>
  </url>
  <url>
    <loc>https://sdorra.dev/posts/2022-11-15-sitemaps-with-appdir</loc>
  </url>
</urlset>
```

But this is very error prone. Every time we create a new blog post or add another page to the site we have to manually modify the sitemap.

I think we can do better.

## Generate sitemap

We could generate our sitemap at build time by traversing the app directory and write it afterwards to the public directory:

```javascript
import { writeFile } from "fs/promises";
import { globby } from "globby";

const PAGE = "https://sdorra.dev";

const createPath = (p) => {
  const path = "/" + p.replace("page.tsx", "");
  if (path.endsWith("/") && path.length > 1) {
    return path.substring(0, path.length - 1);
  }
  return path;
};

const collectPaths = async () => {
  const paths = await globby("./**/page.tsx", {
    cwd: "app",
  });
  return paths.map(createPath);
};

const createSitemap = async (routes) => {
  return `
  <?xml version="1.0" encoding="UTF-8"?>
  <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
      ${routes
        .map((route) => {
          return `
            <url>
                <loc>${`${PAGE}${route}`}</loc>
            </url>
          `;
        })
        .join("")}
  </urlset>
  `;
};

(async () => {
  const paths = await createPaths();
  const sitemap = await createSitemap(paths);
  await writeFile("./public/sitemap.xml", sitemap, { encoding: "utf-8" });
})();
```

The snippet above uses [globby](https://github.com/sindresorhus/globby) to find each `page.tsx` in the app directory (`collectPaths`). After the file paths are collected, each path is transformed to the url path (`createPath`). The last step is to create the sitemap (`createSitemap`) and write to the public directory.

This works well, until we introduce [dynamic routes](https://beta.nextjs.org/docs/routing/defining-routes#dynamic-segments) to our app.

### Dynamic routes

A dynamic route can be identified by the square brackets around a variable e.g.: `app/posts/[slug]/page.tsx`. To handle those routes in the sitemap generator could become very tricky.

I will use a simple example how it could be implemented if the site uses [contentlayer](https://www.contentlayer.dev/):

```javascript
import { writeFile } from "fs/promises";
import { globby } from "globby";
import { allPosts } from "../.contentlayer/generated/index.mjs";

const PAGE = "https://sdorra.dev";

const expandPath = (p) => {
  if (p === "/posts/[slug]") {
    return allPosts.map((p) => `/posts/${p._raw.flattenedPath}`);
  }
  return [p];
};

const createPath = (p) => {
  const path = "/" + p.replace("page.tsx", "");
  if (path.endsWith("/") && path.length > 1) {
    return path.substring(0, path.length - 1);
  }
  return path;
};

const collectPaths = async () => {
  const paths = await globby("./**/page.tsx", {
    cwd: "app",
  });
  return paths.map(createPath).flatMap(expandPath);
};

// rest of scripts/sitemap.mjs
```

The first interesting line in this snippet is the import of the posts generated by contentlayer (`allPosts`). I had to specify the whole path including extension, otherwise typescript threw errors at me. In the `expandPath` function we will check the path and if it is our dynamic route, we will replace it with the slugs of all posts. This function is finally used by the `collectPath` function to create our flat array of paths.

**Warning:** This method covers only the basics. If you are use something like [grouped routes](https://beta.nextjs.org/docs/routing/defining-routes#route-groups) or the old `pages` directory is used in addition to the app directory, you have to go a few extra miles.

### Build

Now it is time to add our generator to the build:

```json
{
  "scripts": {
    "build": "next build && node scripts/sitemap.mjs"
  }
}
```

Whenever our page is build using the `build` script, our sitemap is build afterwards. The order is important, because contentlayer have to run first in order to generate our content types.

### robots.txt

We could specify the url to our sitemap in the `robots.txt`, this makes it easier for search engines to find our sitemap. If we use the default path `/sitemap.xml` most of the search engines should find it without the entry in the `robots.txt`. However the entry could look like the following:

```
User-Agent: *
Allow: /

Sitemap: https://sdorra.dev/sitemap.xml
```

### Git

Since our sitemap is generated, we should not include it in our repository. So we should ignore it:

```
/node_modules
/.next/
.contentlayer
public/sitemap.xml
```

If we would not do that, we would see changes to the `sitemap.xml` every time we change the structure of the page.