Design an Effective Build Stage for Continuous Integration
Published: 2021-04-08 12:18:05

The first move towards releasing software is **the build stage**. This practical guide will show you the general principles of build automation and reference implementations using [Semaphore](https://semaphoreci.com/). Optimizing your build stage will accelerate your development workflow and cut down costs.

## What happens in a build stage

In [continuous integration](https://semaphoreci.com/continuous-integration) (CI), this is where we build the application for the first time. The build stage is the first stretch of a [CI/CD pipeline](https://semaphoreci.com/blog/cicd-pipeline), and it automates steps like downloading dependencies, installing tools, and compiling.

Besides building code, build automation includes using tools to check that the code is safe and follows best practices. The build stage usually ends in the artifact generation step, where we create a production-ready package. Once this is done, the testing stage can begin.

![build stage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/3sjvy7uyltihqma7931g.png)

The build stage starts from code commit and runs from the beginning up to the test stage

We’ll be covering testing in-depth in future articles ([subscribe to the newsletter](https://semaphoreci.com/newsletter) so you don’t miss them). Today, we’ll focus on build automation.

Build automation verifies that the application, at a given code commit, can qualify for further testing. We can divide it into three parts:

1. **Compilation**: the first step builds the application.
2. **Linting**: checks the code for programmatic and stylistic errors.
3. **Code analysis**: using automated source-checking tools, we control the code’s quality.
4. **Artifact generation**: the last step packages the application for release or deployment.

### **Step 1: Compilation**

Step one of the build stage compiles the application. On compiled languages, it means that we can generate a working binary, whereas on interpreted languages, we confirm that we have the required dependencies and tools to successfully build the application.

The result can be a binary file, a code tarball, an installable package, a website, a container image — **the main thing is that we have something that we can run and test**.

### **Step 2: Analyzing Your Code**

Programming requires discipline. We must solve hard problems while following good practices and abiding by a common coding style. Automated analysis tools help you keep out gnarly code at bay. In general terms, there are three classes of code analysis tools:

- **Linting**: linters improve code quality by pointing problematic and hard-to-maintain bits. Linters use a set of rules to enforce a unified standard that improves readability for the team.
- **Quality**: this class of tools uses metrics to find places where code can be improved. The collected metrics include the number of lines, documentation coverage, and complexity level.
- **Security**: security analysis tools scan the code, flag parts that may cause vulnerabilities, and check that dependencies don’t have known security issues.

Since no one wants to ship unsafe or buggy code, we must design a CI pipeline that stops when errors are found. When the CI pipeline runs the right code analysis tools, **developers who review code can focus on more valuable questions** such as “_Is this code solving the right problem?_” or “_Does this code increase our technical debt?_“.

### **Step 3: Preparing your Application for Release**

The released package will include everything needed to run or install the application, including:

- Application code.
- Installation scripts.
- Dependencies and libraries.
- Application metadata.
- Documentation.
- License information.

The final package can be published on a website, in a package database such as [npmjs.com](https://www.npmjs.com/), in a container registry, or can be directly deployed into a QA or production environment.

## Useful Semaphore commands for build automation

A [CI/CD](https://semaphoreci.com/cicd) pipeline consists of blocks and jobs. [In Semaphore](https://docs.semaphoreci.com/guided-tour/concepts/), jobs are where your commands and code run, so it’s here where we define the commands for each step in the build stage. While setting up build automation, you’ll find the following Semaphore commands useful:

- [checkout](https://docs.semaphoreci.com/reference/toolbox-reference/#checkout): clones and CDs into the Git repository. It usually belongs near the top of the job.
- [sem-version](https://docs.semaphoreci.com/ci-cd-environment/sem-version-managing-language-versions-on-linux/): activates a specific version of a language. Run it to ensure the CI environment uses the same version you’re coding with.
- [cache](https://docs.semaphoreci.com/reference/toolbox-reference/#cache): stores your downloaded libraries, intermediate, and compiled files. The cache is a short-lived space used to pass files between jobs and workflows.
- [artifact](https://docs.semaphoreci.com/essentials/artifacts/): the `artifact` command lets store files on a more permanent basis. You can access these directly from the website. Semaphore provides three levels for artifacts:
    - **job**: used to store logs and other debugging details.
    - **workflow**: used to persist files between jobs.
    - **project**: used to store the final project output.

## How to implement a build stage with Semaphore

Shall we see a few practical examples of build automation in a CI pipeline? Let’s roll up our sleeves and write some code.

But first, a note about how I’m writing the commands:

- Commands starting with `$` are meant to run in your local development machine.
- Commands _without the dollar sign_ are for you to copy into your CI jobs.

👉 Jump to the example that interests you the most:

1. [A build stage for Java Applications](https://semaphoreci.com/blog/build-stage#example-1)
2. [A build stage for JavaScript applications](https://semaphoreci.com/blog/build-stage#example-2)
3. [A build stage for Python Applications](https://semaphoreci.com/blog/build-stage#example-3)
4. [Creating Docker images in the build stage](https://semaphoreci.com/blog/build-stage#example-4)

### Example 1: A build stage for Java applications

Java is a compiled language, so the build stage must compile the source into JVM bytecode. Here’s a starter pipeline:

![java build stage](https://wpblog.semaphoreci.com/wp-content/uploads/2021/03/pipe-java.png)

A Java build stage. Linting and code analysis run in parallel in the same block.

#### **Compilation block**

The leading popular build tools in Java are [Apache Maven](https://maven.apache.org/) and [Gradle](https://gradle.org/), both are included in Semaphore’s [machines](https://docs.semaphoreci.com/ci-cd-environment/ubuntu-18.04-image/). Let’s go with Maven; it handles dependencies and compiles the source with a single command: `mvn compile`. Use these commands in your CI job to build the application:

```bash
sem-version java 11
checkout
cache restore
mvn compile
cache store
```

The `cache` command will detect the compiled output in the `target` directory and store it for future use. Ideally, you should set the [environment variable](https://docs.semaphoreci.com/ci-cd-environment/environment-variables/) `MAVEN_OPTS` to `-Dmaven.repo.local=.m2`, so downloaded dependencies are also cached.

![Java build](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xvkel1y03ldykwavefi8.png)

Setting up environment variables in the build block.

#### **Linting with style**

Java developers use powerful IDEs that lint code in realtime. Powerful as they are, they are not enough. Quality checks must also be part of the CI pipeline. We can use [checkstyle](https://checkstyle.sourceforge.io/) to add a linting job.

These commands download checkstyle and run it on a file called `MyFile.java`:

```bash
sem-version java 11
wget https://github.com/checkstyle/checkstyle/releases/download/checkstyle-8.41/checkstyle-8.41-all.jar
java -jar checkstyle-8.41-all.jar -c /sun_checks.xml MyFile.java
```

Or, if you are using Maven, there’s a plugin for automatic linting. To enable it, add the `maven.checkstyle.plugin` to properties in your `pom.xml`.

```xml
<properties>
    <maven.checkstyle.plugin.version>3.1.2</maven.checkstyle.plugin.version>
</properties>
```

Then configure the plugin to fail on error. Add the following snippet to the build section in your `pom.xml`:

```xml
<build>
    <plugins>
        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-checkstyle-plugin</artifactId>
            <version>${maven.checkstyle.plugin.version}</version>
            <configuration>
                <failsOnError>true</failsOnError>
                <failOnViolation>true</failOnViolation>
            </configuration>
        </plugin>
    </plugins>
</build>
```

Now you can run checkstyle with:

```bash
sem-version java 11
checkout
mvn checkstyle:checkstyle
```

#### **Finding Mistakes with PMD**

Another nice checker to use in your CI environment is [PMD](https://pmd.github.io/). PMD works with Java, JavaScript, and Oracle’s SQL. You can scan code with the default rules for Java with the following:

```bash
sem-version java 11
checkout
wget https://github.com/pmd/pmd/releases/download/pmd_releases%2F6.32.0/pmd-bin-6.32.0.zip
unzip pmd-bin-6.32.0.zip
./pmd-bin-6.32.0/bin/run.sh pmd -d . -R rulesets/java/quickstart.xml -f text
```

#### **Security analysis with Infer**

[Infer](https://fbinfer.com/) was created by Facebook to find code that can lead to runtime errors, things such as race conditions and resource leaks. Infer works with Java and Android. Facebook has open-sourced this tool and uses it to fix their mobile client and the main Facebook app.

To run Infer on your CI, download the binary and build the application. Infer will output a report on the `infer-out` folder, which you can store for later analysis. The following command will make the pipeline stop when issues are found:

```bash
checkout
cache restore
mvn clean
curl -sSL "https://github.com/facebook/infer/releases/download/v1.0.0/infer-linux64-v1.0.0.tar.xz" | tar -xJ
./infer-linux64-v1.0.0/bin/infer run -- mvn package
artifact push job --expire-in 1w infer-out
./infer-linux64-v1.0.0/bin/infer analyze --fail-on-issue
```

#### **Finding bugs with Find Security Bugs**

[Find Security Bugs](https://find-sec-bugs.github.io/) uses a security database to detect almost 140 different vulnerability types in Java web applications.

To use this tool, add this snippet to the plugin section in your `pom.xml`:

```xml
<plugin>
     <groupId>com.github.spotbugs</groupId>
     <artifactId>spotbugs-maven-plugin</artifactId>
     <version>4.0.4</version>
     <configuration>
         <effort>Max</effort>
         <threshold>Low</threshold>
         <failOnError>true</failOnError>
         <includeFilterFile>${session.executionRootDirectory}/spotbugs-security-include.xml</includeFilterFile>
         <excludeFilterFile>${session.executionRootDirectory}/spotbugs-security-exclude.xml</excludeFilterFile>
         <plugins>
             <plugin>
                 <groupId>com.h3xstream.findsecbugs</groupId>
                 <artifactId>findsecbugs-plugin</artifactId>
                 <version>1.10.1</version>
             </plugin>
         </plugins>
     </configuration>
</plugin>
```

Then, create two files: `spotbugs-include.xml` and `spotbugs-exclude.xml`. The contents of the first one are:

```xml
<FindBugsFilter>
     <Match>
         <Bug category="SECURITY"/>
     </Match>
</FindBugsFilter>
```

You can exclude rules in the exclude file:

```xml
<FindBugsFilter></FindBugsFilter>
```

Commit the changes into your repository. Then, create a job that generates the report and pushes it to the artifact store:

```bash
checkout
cache restore 
mvn spotbugs:spotbugs 
artifact push job target/spotbugsXml.xml  --expire-in 1w 
mvn spotbugs:check
```

#### **Building a JAR**

We can generate a JAR file with: `mvn package`. Since we already tested the application, we can skip tests.

```bash
sem-version java 11
checkout 
cache restore 
mvn package 
cache store
```

The output JAR is uploaded to the artifact store, where it can be downloaded or used in [continuous delivery and deployment](https://semaphoreci.com/cicd) pipelines.

👉 For a complete example of setting up a CI/CD pipeline for Java applications, see _[**CI/CD for Spring Boot Microservices**](https://semaphoreci.com/blog/spring-boot-microservices-cicd)_.

### Example 2: A build stage for JavaScript applications

JavaScript code doesn’t need compiling. The build stage will only download dependencies, then lint the code, and finally, build a package.

![JavaScript build stage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/11flrknut86tmie6hyci.png)

JavaScript build stage

#### **Filling up node\_modules**

We can install JavaScript modules with [npm](https://docs.npmjs.com/cli/v7/commands/npm/):

```bash
checkout 
npm ci
```

The `npm ci` command creates a clean copy of the `node_modules` folder. Unfortunately, as a result we can’t use the cache at this stage. `npm ci` is the [cleanest and safest way](https://semaphoreci.com/blog/reproducible-node-builds-with-npm-ci) of installing modules.

But if you want or need to use the cache, you’ll need to run `npm install` instead of `npm ci`:

```bash
sem-version node 14.16 
checkout 
cache restore 
npm install 
cache store
```

If the project includes a [.nvmrc](https://github.com/nvm-sh/nvm) config file, you can replace `sem-version` with `nvm use`:

```bash
checkout 
nvm use 
cache restore 
npm install 
cache store
```

The CI machine includes [yarn](https://yarnpkg.com/) as well:

```bash
sem-version node 14.16 
checkout 
cache restore 
yarn install 
cache store
```

#### **Linting with ESLint and JSHint**

JavaScript has come a long way since the Netscape years. It grew from a language whipped up in [less than ten days](https://thenewstack.io/brendan-eich-on-creating-javascript-in-10-days-and-what-hed-do-differently-today/) into the closest thing we have to a lingua franca — almost all devices have JavaScript support.

As you can imagine, the language changed a lot over time, and not all its parts are good. Using a linter will help us stay away from the bad parts of JavaScript. In my experience, the [ESlint](https://eslint.org/) and [JSHint](https://jshint.com/) linters integrate very well into the CI environment. Any of these can be installed with `npm install --save-dev`.

For ESLint, you’ll need to create a configuration with:

```bash
$ npx eslintrc --init
```

Next, add it to the script section of `package.json`:

```json
"scripts": {
     "lint": "eslint '**/*.js' --ignore-pattern node_modules",
     ...
 }
```

After committing the changes, use the following to run the linter:

```bash
sem-version node 14.16 
checkout 
npm run lint
```

#### **Security audits: npm and AuditJS**

Security audits are built-in npm. You can check your dependencies with `npm audit`. Note, however, that you must set a threshold level in order for the command to fail on error:

```bash
sem-version node 14.16 
checkout 
cache restore 
npm audit --audit-level=high
```

If you find security issues, try running `npm audit fix` to attempt automatic resolution.

An alternative to `npm audit` is [AuditJS](https://www.npmjs.com/package/auditjs). This tool uses the [OSS index of vulnerabilities](https://ossindex.sonatype.org/), which feeds from many security databases, so results are more accurate.

You can add AuditJS to your project with:

```bash
$ npm install --save-dev auditjs
```

Then use the following command to run the test:

```bash
sem-version node 14.16
checkout
npx auditjs ossi
```

One nice feature AuditJS has is that it allows us to whitelist security issues. This is great, for instance, when we want to ignore a vulnerability because we know it doesn’t affect us.

For that, create a JSON file detailing the vulnerability ID and a reason for ignoring it:

```json
{
  "ignore": [
    {
        "id": "78a61524-80c5-4371-b6d1-6b32af349043",
        "reason": "Vulnerability doesn't affect our code"
    }
  ]
}
```

Then run AuditJS with the whitelist enabled:

```bash
sem-version node 14.16
checkout
auditjs ossi --whitelist my-whitelist.json
```

#### **Creating Node packages**

Use `npm pack` to create a tarball you can distribute, or use `npm publish` to upload the module to npmjs.com.

```bash
checkout
tarfile=$(npm pack)
artifact push project --force "$tarfile"
```

👉 For a complete example of setting up a CI/CD pipeline for a JavaScript application, see _**[How To Build and Deploy a Node.js Application To Kubernetes](https://semaphoreci.com/blog/nodejs-digitalocean-kubernetes)**_ or _[**Continuous Integration with Deno**](https://semaphoreci.com/blog/continuous-integration-with-deno)_.

### Example 3: A build stage for Python applications

We don’t need to run any compile commands in Python. As a result, the build job, like JavaScript’s, will only download and cache dependencies.

![python build stage](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/gq40zmqh6ssqps174v08.png)
Python build stage

#### **Installing modules**

In this example, we run [pip](https://pypi.org/project/pip/) to install the project dependencies in the `.pip_cache` folder:

```bash
sem-version python 3.9
checkout
cache restore
pip download --cache-dir .pip-cache -r requirements.txt
cache store
```

#### **Linting with Pylint**

Python developers are used to following strict coding standards. One good way of checking for good style is [Pylint](https://www.pylint.org/). Pylint ruleset includes checks for unused modules, variable name formats, and line length. It strictly adheres to Python’s [PEP 8](https://www.python.org/dev/peps/pep-0008/) style guide.

Like most linters, Pylint can be integrated into the IDE. But, in this case, we’ll use it in the pipeline to check commit quality. To show lint errors in a single package or file use:

```bash
sem-version python 3.9
checkout
pylint -E [MODULE|PYTHON_FILE]
```

You can scan every Python file in the repository with the following command:

```bash
sem-version python 3.9
checkout
git ls-files | grep -E '.py$' | xargs pylint -E
```

#### **Complexity analysis with Xenon**

One thing PEP 8 style guide cannot capture is cyclomatic complexity, a measure of how many independent paths there are in the codebase. To calculate it, you can use a tool like [Xenon](https://pypi.org/project/xenon/).

To get started with Xenon, add it to your Python project with:

```bash
$ pip install xenon
$ pip freeze > requirements.txt
```

Xenon uses a [ranking system](https://radon.readthedocs.io/en/latest/commandline.html#the-cc-command) that goes from A to F, where A is the lowest complexity level. In the job, you can supply the minimum level permitted before failing.

```bash
sem-version python 3.9
checkout
cache restore
xenon --max-absolute C --max-modules B --max-average A
```

The example will make the job fail if the average complexity is lower than A, the complexity in the modules goes lower than B, or the complexity of a block is lower than C.

#### **Packing for distribution**

Python’s package [format](https://packaging.python.org/tutorials/packaging-projects/) allows you to upload modules to the official [PyPI index](https://pypi.org/) or distribute it on your website.

To create a package, use the following commands:

```bash
sem-version python 3.9
python -m pip install --upgrade build
checkout
python -m build
artifact push project --force dist
```

👉 For a complete Python pipeline see _**[Python Continuous Integration and Deployment From Scratch](https://semaphoreci.com/blog/python-continuous-integration-continuous-delivery)**_.

### Example 4: Creating Docker images in the build stage

[Docker](https://semaphoreci.com/product/docker) is a container platform that standardizes build automation by letting us package any software regardless of the language. By building a Docker image, we get a portable application we can run on any machine, as long as it has a container runtime installed. It also allows us to run the application at scale on orchestration platforms like Docker Swarm or [Kubernetes](https://semaphoreci.com/blog/kubernetes-vs-docker).

Semaphore includes everything you need to build a Docker image. You only need to supply a `Dockerfile` with the build instructions.

The easiest way of writing a Dockerfile is by starting from a base image. Semaphore provides images for popular languages in the [Semaphore Container Registry](https://docs.semaphoreci.com/ci-cd-environment/semaphore-registry-images/). For example, this Dockerfile is for packaging a Python application:

```docker
# use image in Semaphore container registry
FROM registry.semaphoreci.com/python:3.9

# copy source files
ADD . ./opt/
WORKDIR /opt/

# install dependencies
RUN pip install --upgrade pip
RUN pip install -r requirements.txt

# start application
CMD ["python","app.py"]
```

Use these commands in the CI job to build the image:

```bash
checkout
docker build -t my-image .
```

It works, but the image is lost as soon as the job completes. We must upload it into a registry in order to keep it. And for that, you have to supply login information for an image registry with a [secret](https://docs.semaphoreci.com/essentials/using-secrets/).

To create a secret, click on your account badge on the top right corner and then **Settings**.

![Semaphore Settings menu](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/d0t7tad0qa16i6424slg.png)

Where to find Settings menu.

Then click on **Secrets** > **New Secret**.

Create a secret called `dockerhub` with your Docker Hub username and password:

![create a secret on Semaphore](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/hjc7l59qv6sfau7lpber.png)

Create a secret to store Docker Hub credentials

Now you can access the username and password in the job:

```bash
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
```

Next, you can pull the last image to speed up the build process:

```bash
docker pull $DOCKER_USERNAME/awesome-application:latest || true
```

Finally, build and push the image:

```bash
docker build -t $DOCKER_USERNAME/awesome-application:latest --cache-from=$DOCKER_USERNAME/awesome-application:latest .
docker push $DOCKER_USERNAME/awesome-application:latest
```

The complete job commands are:

```bash
echo "${DOCKER_PASSWORD}" | docker login -u "${DOCKER_USERNAME}" --password-stdin
docker pull $DOCKER_USERNAME/awesome-application:latest || true
docker build -t $DOCKER_USERNAME/awesome-application:latest --cache-from=$DOCKER_USERNAME/awesome-application:latest .
docker push $DOCKER_USERNAME/awesome-application:latest
```

Don’t forget to activate the secret in your job:

![activate the secret](https://dev-to-uploads.s3.amazonaws.com/uploads/articles/xsrvtd4ao0hgedpbxfkr.png)

Activate the secret in the block

What we have seen are the basic building blocks for a Docker image. The process is mostly the same for any language. The only thing that changes are the contents of the Dockerfile. We have a few in-depth tutorials and example Dockerfiles for you to peruse below:

- [Dockerizing Java Applications](https://semaphoreci.com/community/tutorials/dockerizing-a-java-play-application)
- [Dockerizing JavaScript and Node](https://semaphoreci.com/community/tutorials/dockerizing-a-node-js-web-application)
- [Dockerizing Python Applications](https://semaphoreci.com/community/tutorials/dockerizing-a-python-django-web-application)
- [Dockerizing Ruby on Rails](https://semaphoreci.com/community/tutorials/dockerizing-a-ruby-on-rails-application)
- [Dockerizing PHP Applications](https://semaphoreci.com/community/tutorials/dockerizing-a-php-application)
- [Dockerizing Elixir Applications](https://semaphoreci.com/community/tutorials/dockerizing-elixir-and-phoenix-applications)

## What’s Next

We learned what build automation is and saw a few examples. The principles always remain the same, regardless of the language or framework used.

The principle used in all examples is that **the build stage runs in a pipeline**. By running fast code checks right after compilation— and before the automated test suite—you optimize your continuous integration process for fast feedback.

I hope these recommendations help you optimize your build process. Happy building!