Generate an XML Sitemap for a Static Website in GitHub Actions
Tags: webdev, showdev, github, programming
Published: 2022-11-23 14:00:00

I use GitHub Pages for my personal website, as well as for several project sites. Although some static site generators include support for sitemap generation (e.g., [Jekyll has a plugin for sitemaps](https://github.com/jekyll/jekyll-sitemap)), my personal website is generated by a custom static site generator that I built for a few specialized reasons, and most of my project sites for Java libraries consist of a single hand-written HTML page combined with javadoc-generated documentation. So a while back I implemented a GitHub Action, [generate-sitemap](https://github.com/cicirello/generate-sitemap), that can generate an XML sitemap by crawling a GitHub repository containing the HTML of the site. It uses the last commit date of each file to produce the `<lastmod>` tags. By default, it includes URLs for HTML and PDF files in the sitemap, and skips other file extensions in the repository. But it can be configured to include URLs corresponding to whatever file extensions you want included. It checks the head of HTML pages for `noindex` meta tags, and excludes such files from the sitemap, and it likewise excludes files from the sitemap if they match a `Disallow` rule in your `robots.txt`. The [generate-sitemap](https://github.com/cicirello/generate-sitemap) can be configured in a few other ways as well (see the documentation in the GitHub repository for all details). The [generate-sitemap](https://github.com/cicirello/generate-sitemap) action is implemented in Python as a [container action](https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action).

__Table of Contents:__ This post is organized as follows:
* [Prerequisite Workflow Step](#prerequisite-workflow-step)
* [Example Workflow](#example-workflow)
* [Learn More](#learn-more)
* [Where You Can Find Me](#where-you-can-find-me)

## Prerequisite Workflow Step

In order for the `<lastmod>` dates to be correctly determined, the step that checks out your repository must use `actions/checkout`'s optional input `fetch-depth: 0` in order to get the full git history, such as with a step like the following:

```YAML
    steps:
    - name: Checkout the repo
      uses: actions/checkout@v3
      with:
        fetch-depth: 0 
```

## Example Workflow

Here is an example workflow. It runs on pushes to the branch `main`. It then starts with the checkout as described above. The generate-sitemap action assumes that the entire repository is the website by default (you can change that behavior with the input `path-to-root`). The most important input is probably `base-url-path`, which specifies the URL to the root of your site. This example workflow includes html and pdf files in the sitemap by default. There are optional inputs that can be used to exclude either of these, and an optional input `additional-extensions` that can be used to additionally include files of any specific type you desire in the sitemap.

```YAML
name: Generate xml sitemap

on:
  push:
    branches: [ main ]

jobs:
  sitemap_job:
    runs-on: ubuntu-latest
    name: Generate a sitemap

    steps:
    - name: Checkout the repo
      uses: actions/checkout@v3
      with:
        fetch-depth: 0 

    - name: Generate the sitemap
      uses: cicirello/generate-sitemap@v1
      with:
        base-url-path: https://www.example.com/

    - name: Commit and push
      run: |
        if [[ `git status --porcelain sitemap.xml` ]]; then
          git config --global user.name 'github-actions'
          git config --global user.email '41898282+github-actions[bot]@users.noreply.github.com'
          git add sitemap.xml
          git commit -m "Automated sitemap update" sitemap.xml
          git push
        fi
```

The [generate-sitemap](https://github.com/cicirello/generate-sitemap) action doesn't commit and push, so you need a step in your workflow to do that. In the above example workflow, the last step uses a simple shell script to commit and push. This example does the commit as the `github-actions` bot. If you'd rather be the committer, then adjust that step as necessary. There are also actions in the GitHub Marketplace that can be used for the commit and push step if you prefer.


## Learn More

You can find more information about this GitHub Action in its GitHub repository:

{% github cicirello/generate-sitemap %}

You can also find information about this GitHub Action, as well as others I've implemented and maintain at the following site (which by the way is served via GitHub Pages, and uses this action to generate its sitemap):

{% embed https://actions.cicirello.org/ %}



## Where You Can Find Me

Follow me here on DEV:

{% user cicirello %}

Follow me on GitHub:

{% github cicirello/cicirello %}

Or visit my website:

{% embed https://www.cicirello.org/ %}