Incremental Builds with Jamify
Tags: webdev, gatsby, ghost, cms
Published: 2020-09-18 08:15:20

---
title: Incremental Builds with Jamify
published: true
date: 2020-06-13 13:38:30 UTC
tags: #webdev #gatsby #ghost #cms 
canonical_url: https://www.jamify.org/2020/06/13/incremental-builds-with-jamify/
---

With Jamify, you can now fully leverage the new Gatsby Cloud incremental builds feature to build your sites even faster! We are proud to be the first to bring this exciting feature to your publication pipeline when sourced from a headless Ghost CMS.

 ![Incremental Builds with Jamify](https://images.unsplash.com/photo-1543364221-e76ae14091f3?ixlib=rb-1.2.1&q=80&fm=jpg&crop=entropy&cs=tinysrgb&w=2000&fit=max&ixid=eyJhcHBfaWQiOjExNzczfQ)

Gatsby Cloud recently introduced a new _Incremental Builds_ feature that makes publishing your site _even_ faster! **Today, I'm excited to announce best in class feature support for incremental builds in Jamify.** We are proud to be the _very first_ to bring Ghost CMS incremental build support to your publication pipeline!

This will allow you to make content changes and rebuild only the parts of your site that are affected by the change. With this feature, you can easily achieve **sub-ten seconds build times** following a content change. This makes a big difference especially for large sites, where the entire site had to be re-built for even the smallest data change.

One of the biggest obstacles towards a wide adoption of static site generators has always been build processing times. Incremental builds is a huge step forward towards removing a strong argument against static site builders. I am sure this technology leap will invigorate new momentum to Gastby and the Jamstack in general.

## Content vs. Configuration Changes

Incremental Builds are currently only supported for _content changes_. If you publish a new post, update an existing post or add a new tag or author, and incremental build is triggered. However, if you make a configuration change such as installing a new plugin or changing a setting in `gatsby-node.js`, a full rebuild of your site is necessary.

> On production sites, content changes are the frequent and most important ones, so publication authors will directly profit from reduced publication times!

## Explore Incremental Builds

With Jamify, incremental builds are software enabled by default: **zero configuration** needed! All the prerequisites have been included in the Jamify starter, such as supporting the latest `gatsby` version as well as ensuring best in class _[CMS integration](https://www.gatsbyjs.com/docs/integration-guide/)_. Let's dive in and see incremental builds in action.

### Jamify starter

To begin, you need to make a copy of the Jamify starter and save it into your personal Github account. You can either use a fork or make use of the template approach.

Open the [Jamify starter project](https://github.com/styxlab/gatsby-starter-try-ghost) and press the _Use this template_ button to make a copy.

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-starter-template.png)

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-starter-incremental-builds-1.png)

You can choose any repository name, so I took _incremental-builds_. After pressing _Create repository from template_, you'll see a new repository with that name under your account.

It contains a self-contained software package for sourcing content from a headless Ghost CMS that transforms your content into a _flaring fast React front-end_ that can be directly served from the edges of a Content Delivery Network (CDN).

> The Jamify starter can be enhanced with plugins. Check out the list of available [plugins for Jamify](https://www.gatsbyjs.org/plugins/?=jamify) or simply write your own!

### Gatsby Cloud

Now, you are ready to connect this repository to [Gatsby Cloud](https://www.gatsbyjs.com/). Go to the Gatsby Cloud dashboard and click on _Create new site_ which walks you through the set up process:

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatby-cloud-new-site-repo-3.png)

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-cloud-ghost-content-api-keys-1.png)

This is a three step process. First choose _I already have a Gatsby site, next, connect to your repository (e.g., github-user/incremental-builds) and_ leave the branch options as they are.

The final step is where you need to provide your Ghost Content API keys for both the _Preview_ and the _Build_ environments.

> If you skip the last step, your project is still building and sourcing in content from a demo server. However, you need to be able to make changes to your content in order to test out the incremental build feature. Thus, you'll have a lot more fun, if you provide the keys of your own Ghost CMS here.

### Initial Builds

After completing the last step, the build process is directly triggered, producing the standard demo site:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-build-demo-site-1.png)

As you can see from the below screenshot, the total build process was less than a minute and took **55 seconds** _:_

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-initial-2.png)

As this was the _initial build_ that started from an empty cache, it's interesting to see how long it takes to rebuild the site. Use the _Trigger build -> Build master branch_ option to trigger a rebuild:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-rebuild-1.png)

In this case, the rebuild is _ **20% faster** _ because previously processed images can be directly fetched from the cache.

### Triggering incremental builds

It's important to understand that a _rebuild_ is **not** an _incremental build_. An incremental build in Gatsby Cloud can only be triggered by _webhooks_!

> Incremental Builds in Gatsby Cloud are available only on the professional plans. However, Gatsby offers a free 14-day trial period for new users on the professional plan. This should be sufficient for testing this stunning feature.

Gatsby Cloud automatically generates webhooks for every site. You just have to find them! Go to the _Settings -> General -> Webhooks_ and copy the webhook under _Preview Webhook_ to your clipboard:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-cloud-builds-webhook.png)

You'll want to configure this webhook in your headless Ghost CMS, but for a first test you can directly trigger it from the command line by sending a `POST` request with `curl`:

```
$ curl -X POST https://webhook.gatsbyjs.com/hooks/data_source/publish/86992927-dffe-4481-ad6a-ebb0985d84d1
```

Use your own webhook and trigger an _incremental build_ which is then labeled with an `incremental` badge:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-cloud-incremental-builds.png)

> **Wow** 🎉! A build in **just three seconds**. Take a smoothie and celebrate!

A sceptic would grumble: Nice build times, but _nothing_ has changed on your site, no wonder it's so fast! Okay, sceptic. I hear you, let's do some real content edits...

## Ghost CMS Content Edits

You need to define a trigger in your Ghost CMS, so content or site edits initiate a `POST` request to Gatsby Cloud webhook endpoint.

> If you do not yet run a headless Ghost CMS, check out the tutorial that describes how to set up [Ghost CMS on Hetzner Cloud](https://cms.jamify.org/2020/04/07/ghost-cms-on-hetzner-cloud/).

In the Ghost Admin panel, go to _Integrations -> Add custom integration_ and give it a name, e.g. Gatsby Cloud. Next add a new webhook:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/ghost-cms-webhook-for-incremental.png)

give it a name, choose _Site changed (rebuild)_ as an event and paste the previously copied Gatsby Cloud webhook URL into the target field.

If you inspect the list of events that Ghost provides, you may be tempted to choose more fine grained triggers, such as _Post published_, _Post updated_ and others. Unfortunately, most of these triggers didn't work (tested with Ghost version 3.18.1), so the _Site changed (rebuild)_ is currently the only useful option.

### In-Page Content Edit

It's time to make your first content edit. Let's start simple and change the headline of the welcome post (left before, right after the edit):

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/ghost-cms-edit-title-before.png)

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/ghost-cms-edit-title-after.png)

To trigger the incremental build press the _Update_ button on the top right corner:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-incremental-preview-demo.png)

This incremental build was ready in **just six seconds** :

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-build-title-edit-2.png)

On successive runs, I observed some variance in the 4 to 8 seconds range.

> I experienced stability issues with incremental builds on the production branch. Sporadically, a subsequent incremental build would not change the site, despite showing a successful build status. I therefore switched to the preview branch, which always showed consistent results.

### Unpublishing an article

A common use case is to take an article out of your site. Again, this change takes about **five seconds** to build:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-preview-unpublished.png)

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-unpublish-1.png)

### Publishing a new article

Reversing the last step is equivalent to publishing a new article. As this is also a change with just on page, you can expect comparable build times, in this case **4 seconds** :

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-publish.png)

### Changing Settings

Finally, we are going to make a change to the site settings by changing the blog title, which takes just **three seconds** to rebuild!

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-incremental-settings.png)

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/gatsby-cloud-incremental-settings.png)

I encourage you to play a little with different type of edits and especially test out common content changes, such as adding or removing tags and authors or changing a a feature or inline image.

I performed a wide range of different edits and was getting consistent incremental build times in the **three to fifteen seconds range:** this _is_ truly amazing.

## Conditional page builds

While I want to give full credit to Gatsby for pulling out this game changing incremental build feature, a lot of people I spoke with were rather disappointed when they heard that incremental builds are exclusively available on Gatsby Cloud. Here is quote from a Gatsby team member, in the context of describing incremental builds:

> It's important to clarify that in order to achieve the results that we have, we need to own the CI/CD pipeline. This isn't something that can be packaged in our open-source repo.  
>   
> [— Josh Comeau](https://joshwcomeau.com/gatsby/incredimental-builds/)

While I fully understand that Gatsby needs to protect their technological investments in Gatsby Cloud, I really don't buy into this argument. When you build locally or on your own server, you also _own the pipeline_ including the _cache_ and the _full build history_, so what is it exactly that cannot be packaged as open-source?

I really hope Gatsby does not loose credibility in the open source community because of these apparent marketing communications and that they will eventually provide more technical details into the incremental build process so that _all parts that are portable_ can be integrated into the open source tool-chain.

Fortunately, small portions of the content-diffing methods that are part of incremental builds have already been provided as an experimental feature called [conditional page builds](https://www.gatsbyjs.org/docs/conditional-page-builds/).

## Locally built conditional page builds

In the following I am going to share with you my latest test results based on this experimental feature. As before, I am going to build the Jamify starter demo site. You can easily repeat these steps on your own machine. First clone the starter:

```
$ git clone https://github.com/styxlab/gatsby-starter-try-ghost.git jamify-conditional-page-builds
```

and change into the newly created directory:

```
$ cd jamify-conditional-page-builds
```

Resolve all package dependencies by issuing the `yarn`. This may take a couple of seconds to complete but only needs to be called once.

```
[jamify-conditional-page-builds]$ yarn
```

> Note that the conditional page build feature is only available for the build command, so it does not affect the development mode.

You enable this feature by setting the lengthy environment variable `GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES` to `true`. The build command is then called as follows:

```
GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
```

The option `--log-pages` shows a summary at the end of the build log with details about the pages that were deleted, updated or created.

### First experimental feature test

Now, it's time to build the demo site for the first time:

```
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 38.91s.
```

This is the initial build time starting from an empty cache. You can see that all feature pictures are pulled in and subsequently processed with the `transformer-sharp` tools. Also notice the routes to all pages are listed at the end of the log. The build time of roughly **40 seconds** is comparable with the measured times on Gatsby Cloud for exactly the same project site. You can preview this site by calling `gatsby serve` and visiting `http://localhost:9000`:

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-conditional-page-builds-demo.png)

Let's _conditionally rebuild_ this site without making any content changes:

```
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 20.83s.
```

This is a **factor 2 improvement** compared to the initial build, but can that be attributed to the experimental feature?

In order to check that, let's repeat the previous two build commands, this time without conditional page builds. First, clear the cache:

```
[jamify-conditional-page-builds]$ yarn clean
```

Next, use the `gatsby build` command directly, without any additional parameters:

```
[jamify-conditional-page-builds]$ gatsby build
...
Done in 36.04s.
```

and a second time, with the cache populated:

```
[jamify-conditional-page-builds]$ gatsby build
...
Done in 21.91s.
```

> As the benchmarks suggest, the factor two improvement between first and second is **solely due to caching** and cannot be attributed to the experimental build feature!

### CMS page edits

While the conditional page build feature did not make any difference in the previous test, let's check if that feature makes a difference in a more practical use case.

Therefore, we are going to make the same page edit test that we previously made on Gatsby Cloud. To refresh initial test conditions with conditional page builds enabled, please call the following three commands:

```
[jamify-conditional-page-builds]$ yarn clean
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
```

In the headless Ghost CMS, you can now change the title of the welcome post:

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/ghost-cms-edit-title-before-1.png)

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-conditional-page-build-edit.png)

Make sure to press the `Ctrl+S` or the _Update_ button in the top right corner in order to publish the changes:

```
[jamify-conditional-page-builds]$ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
...
Done in 21.83s.
```

Well... _not so impressive_. As you can see from the logs, all pages are rebuild instead of the expected one page.

I've made a couple of more tests and was able to see that _under special conditions_, only one page is recreated. For example, when adding a Ghost page with _no relations_ to other pages (using different tags and authors), unpublishing it would only delete this one page.

### Conclusions on conditional page builds

For the time being, the experimental page build feature **does not improve the build process** more significantly than what the standard Gatsby cache mechanism provides. Gatsby Cloud incremental builds are boosting performance by a factor of 5 on top of what the standard caching gives you, a level that is currently out of reach with the conditional page build feature.

## Incremental builds on Netlify?

Shortly after incremental builds were announced on Gatsby Cloud, Netlify was quick to jump on the band wagon and publish a blog article on [enabling Gatsby incremental build support on Netlify](https://www.netlify.com/blog/2020/04/23/enable-gatsby-incremental-builds-on-netlify).

I was really thrilled to see a little competition on accelerating Gatsby build processing times and was eager to test it out. I'm going to repeat my tests here, so you can follow along and don't just need to take my word for it.

### Set-up

To enable this feature on Netlify, you need to install a build plugin. If you inspect the `netlify.toml` file that is shipped with the Jamify Starter, this is already configured for you, so this plugin will be automatically installed during the build process.

```
# ./netlify.toml
[build]
  command = "gatsby build"
  publish = "demo/public"

[[plugins]]
  package = "netlify-plugin-gatsby-cache"
```

As always, I'm starting from scratch. First you need a copy of the Jamify starter template, and give it a unique name such as `netlify-incremental`:

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-starter-template-1.png)

 ![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/jamify-github-create-template.png)

Once the template is copied to your repo, clone it so you can make some changes locally:

```
$ git clone https://github.com/<your-github-user>/netlify-incremental.git
$ cd netlify-incremental
```

Due to a current bug described in an article entitled _[Incremental build time increases](https://community.netlify.com/t/incremental-gatsby-build-time-increases/14722/5),_ we must switch from `yarn` to `npm` package management. You must therefore delete the `yarn.lock` in your starter project and use `npm install` to generate a `package-lock.json` instead:

```
[netlify-incremental]$ rm -rf yarn.lock
[netlify-incremental]$ npm install
```

Next, change the build command in the `netlify.toml` file:

```
# ./netlify.toml
[build]
  command = "GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages"
  publish = "demo/public"

[[plugins]]
  package = "netlify-plugin-gatsby-cache"
```

Stage and check-in the changes with git:

```
[netlify-incremental]$ git commit -a -m "switch to npm"
[netlify-incremental]$ git push
```

### Deploy to Netlify

First make sure to have their cli tool installed:

```
$ sudo npm -g install netlify-cli
```

You can now deploy the site from the command line:

```
[netlify-incremental]$ netlify init
```

Choose _Create & configure a new site_, your team, optionally a site name, _Authorize with GitHub through app.netlify.com_ and use the defaults for build command and deploy directory. The latter two settings are overwritten by your `netlify.toml` file.

### Netlify build times

Switch to your new site on Netlify with

```
[netlify-incremental]$ netlify open
```

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/netlify-initial-build-time.png)

As you can see, the initial build took approximately **three minutes**. This time includes bootstrapping the build environment, so a second build should be faster. Let's test that with issuing a webhook trigger:

```
curl -X POST https://api.netlify.com/build_hooks/5ee4c313b2863fd798b7411a
```

![Incremental Builds with Jamify](https://cms.jamify.org/content/images/2020/06/netlify-incremental-webhook.png)

A re-build is utilizing the cache and is therefore much faster, it only takes **less than a minute** to complete. I made a couple of more test with `netlify-plugin-gatsby-cache` and with the experimental page build setting `GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true` enabled. Here are the results:

<!--kg-card-begin: markdown-->

| Description | init | build | post | total (min) |
| --- | --- | --- | --- | --- |
| Initial build (new cache) | 1:24 | 1:27 | 0:06 | 2:57 |
| Trigger panel (keep cache) | 0:26 | 0:26 | 0:03 | 0:55 |
| Tigger Webhook | 0:27 | 0:27 | 0:03 | 0:57 |
| CMS content edit (1 Page) | 0:30 | 0:32 | 0:03 | 1:05 |

<!--kg-card-end: markdown-->

When compared to builds with the standard  `gatsby develop` command (`netlify-plugin-gatsby-cache` enabled!) the picture is almost the same:

<!--kg-card-begin: markdown-->

| Description | init | build | post | total (min) |
| --- | --- | --- | --- | --- |
| Initial build (new cache) | 1:35 | 1:25 | 0:13 | 3:13 |
| Trigger panel (keep cache) | 0:33 | 0:35 | 0:04 | 1:12 |
| Tigger Webhook | 0:27 | 0:30 | 0:03 | 1:00 |
| CMS content edit (1 Page) | 0:28 | 0:30 | 0:05 | 1:03 |

<!--kg-card-end: markdown-->
### Conclusions on Netlify builds

What Netlify praises as _incremental builds_ should really be called _cashed builds_ — it has nothing to to with the incremental build feature on Gatsby Cloud where you can achieve sub-ten seconds build times.

## Summary

**With Jamify**, you can try the _Gatsby Cloud Incremental Builds feature_ today and achieve **stunning sub-ten seconds** build times on simple CMS page edits. As content changes are the most frequent changes on a live site, this is really great news for publishers.

While the preview branch showed extremely stable results, _subsequent incremental builds_ on the production branch sometimes silently refused to apply expected content changes. I hope, these annoyances will vanish once incremental builds becomes more mature.

Conditional page builds turned out to be extremely experimental and I did not see any substantial build time reductions: neither when built locally nor when built on Netlify. On the positive side, caching always reduces build times for _subsequent_ builds. That's not a surprise, but something to keep in mind when developing plugins for Gatsby.

With Gatsby Cloud being the first to show what's possible when it comes to accelerating build times, I am confident that other vendors will follow soon. We are just at the beginning of a new Jamstack revolution!

> [Do you want early access to Blogody, the brand new blogging platform that I am creating? Just sign-up on the new Blogody landing page and be among the first to get notified!](https://www.blogody.com/landing)

***

This post was originally published at [jamify.org](https://www.jamify.org/2020/06/13/incremental-builds-with-jamify/) on June 06, 2020.