My Autoconf Primer
Tags: autoconf, autotools, automake
Published: 2022-02-09 20:43:23

Since a young age I've learned that build systems and dependency management range from difficult to mind-bogglingly frustrating. My best experiences have been with Autotools, which might surprise many. However, I had never made anything serious from scratch using Autotools myself, but rather only modified things setup by others. Lately I had some positive experiences building C programs linking `libguile` with Autoconf and this prompted me to try something bigger. I setup the build toolchain for a C++ program using Autotools. In this blog post I attempt to document what I learned along the way.

## Autotools, Automake and Autoconf

The first question I wanted answered is if I should be using Autotools, Autoconf or even Automake. The answer is that Autotools is not a binary or similar, but rather it's an umbrella term for Autoconf, Automake and everything related. On the other hand, `automake` is a binary that converts a `Makefile.am` into `Makefile.in`.

In my brand new program, I needed to learn very little about Automake. I created a top-level `Makefile.am` containing a pointer to the related documentation and an instruction to recurse into the subdirectory `src`:

    $ cat Makefile.am
    SUBDIRS = src
    dist_doc_DATA = README.md

The file in `src/Makefile.am`  contains information of the sources to compile into objects and the resulting binaries:

    $ cat src/Makefile.am 
    bin_PROGRAMS = test_algorithm
    test_algorithm_SOURCES = \
        algorithm.cpp \
        exception.cpp \
        job.cpp \
        warning.cpp \
        test_algorithm.cpp

You can run `automake` as a standalone program. However, I found it far more convenient to call it using `autoreconf`. `autoreconf` is a program that runs `automake` and `autoconf` (and more) conveniently for you. Typically, I run `autoreconf` with the following switches: `-i` to generate missing auxiliary files, `-v` for verbose output and `-f` to force regeneration of all configuration files.

    $ autoreconf -vif

But now I'm getting ahead of myself. Before we can run `autoreconf`, we need to write the input for `autoconf`, namely, `configure.ac`.

## Introduction to Autoconf

The role of Autoconf is to generate the `configure` to ship with a software distribution. The input file `configure.ac` is formatted in the macro language [M4](https://www.gnu.org/software/m4/m4.html).

Now, this might be a bit confusing, but `configure` (and hence `autoconf`) is not meant to work with software dependency management. Instead, the main design principle of Autoconf is to work with features and capabilities of the installed system where the distributed software is compiled.

## config.h

Although I didn't use it myself, I need to mention `config.h`. (for what it is worth, I should probably try it out and update this part of the blog post later)

As a result of running `configure`, a header file `config.h` is typically generated. This header file contains detected capabilities of the system we are running the compilation on. For example, if we look for `pthread`, `config.h` will contain suitable `#define`'s to indicate if `pthread` is found and linkable or not.

As stated previously, I did not utilize `config.h` and therefore no such file is generated in my case.

## Structure of configure.ac

The first statement of `configure.ac` must be `AC_INIT` and the last must be `AC_OUTPUT`. The rest of the statements can be placed anywhere, except for when they depend on each other. This dynamic seems to have created a de facto standard structure for `configure.ac` files. Even the Autoconf manual has a section [Standard `configure.ac` Layout](https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.71/html_node/Autoconf-Input-Layout.html). However, I find it quite confusing and instead I follow a mental picture, which is as follows:

* Initialization
* Compiler settings
* Configuration Files
* Dependencies
* Output

As previously stated, `AC_INIT` is the first statement. It accepts parameters such as program name, program version, maintainer e-mail address etc. In my case it is:

    AC_INIT([algorithm], [1.0])

As part of the initialization phase, I also put a statement to initialize `automake` with `AM_INIT_AUTOMAKE`. I found this in the [`automake` manual](https://www.gnu.org/software/automake/manual/html_node/amhello_0027s-configure_002eac-Setup-Explained.html). Note that statements starting with `AC_` are for Autoconf and statements starting with `AM_` are macros for Automake. However, `AM_INIT_AUTOMAKE` is the only statement for Automake in my case.

Now, this is really important: We give some parameters that look just like compiler flags, but they are for Automake when it generates `Makefile.in` etc. I used `-Wall`, `-Werror` and `foreign`. I needed `foreign`, because otherwise Automake requires all the files in a standard GNU package, i.e. `NEWS`, `README`, `AUTHORS` and `ChangeLog`. My `AM_INIT_AUTOMAKE` is as follows:

    AM_INIT_AUTOMAKE([-Wall -Werror foreign])

## Compiler settings

Next we need to select a compiler. When compiling a standard C program, we use `AC_PROG_CC`. In my case, I'm compiling C++ so I use `AC_PROG_CXX` instead. Note that the macro `AC_PROG_CPP` is related to the C preprocessor and has nothing to do with C++ (for the C++ preprocessor command, use the macro `AC_PROG_CXXCPP`).

The only parameter that `AC_PROG_CC` and `AC_PROG_CXX` accept are a compiler search list, but no compiler flags. Instead, if you want to specify compiler flags, you need to set `CFLAGS` or `CXXFLAGS` before calling the macro. As per the Autoconf manual, by default `AC_PROG_{CC,CXX}` sets `{C,CXX}FLAGS}` to `-g -O2` (or `-g` if the detected compiler is not a GNU compiler). But if `{C,CXX}FLAGS` was already set, it's kept unchanged.

In my case, I call the macro `AC_PROG_CXX` with the following construct:

    : ${CXXFLAGS="-g -O2 -Wall -Werror"}
    AC_PROG_CXX

Furthermore, as my source code uses C++17 features, I use an extra macro for that. But it's a little special and I will come back to it later in the section Autoconf Archive.

## Configuration files

As explained earlier, Automake compiles an `Makefile.in` from `Makefile.am`. However, this `Makefile.in` is not yet a `Makefile`. To specify which makefiles need to be compiled, we use the macro `AC_CONFIG_FILES`. In my case:

    AC_CONFIG_FILES([
        Makefile
        src/Makefile
    ])

Note that we will still distribute the `Makefile.in` and the `Makefile` will be generated only on site. The instructions to build the file are contained in `config.status` while the `configure` will use.

Furthermore, the utility of `AC_CONFIG_FILES` is not retricted to makefiles, but other files can be specified, too. However, I had no such use.

## Dependencies

As explained earlier, Autoconf doesn't work with software dependency managers in the conventional sense. I'll call this section *Dependencies* anyhow, as that's how I think about it. There are various things you can do here, and in my case I was happy I needed 3 different ways to search for libraries and headers, as that really made me learn.

### Dependencies: AC_CHECK_LIB

The most typical thing you would do is `AC_CHECK_LIB`. To `AC_CHECK_LIB` you give the library name, a symbol in the library, and, optionally, a code segment to execute on success and a code segment to execute on failure. For example, in my scenario I ended up doing

    AC_CHECK_LIB([pthread], [pthread_create], [], [AC_MSG_ERROR([pthread is not installed.])])

My first surprise was that a symbol from the library is needed. In fact, the macro will expand into a C program that will be compiled. The source of the C program will contain the symbol we indicated, i.e. `pthread_create` and the program will be linked against the library, i.e. with the flag `-lpthread`. If compilation succeeds, the third parameter will be executed, but since its `[]` the default action will be taken. Note that to disable the default action you can do `[ ]`. If compilation fails, we run the fourth parameter which is a macro to signal an error and halt operation.

The default action is to add the specified library to `LIBS`, i.e. in my case `LIBS="-lpthread  $LIBS" (yes, I double-checked from the generated `configure`). Furthermore, the define `HAVE_LIBlibrary` will be set. In my case, it's `#define HAVE_LIBPTHREAD 1`.

I think there's at least two takeaways here. First of all, we don't match libraries with any kind of versioning. Instead, we look for a symbol. I would assume this can cause headache with missing or deprecated features. Furthermore, the macro `AC_CHECK_LIB` uses *always* C linking. This means that C++ libraries cannot be added using this construct, unless the library also has some symbols using C linking (i.e. `extern "C"`).

### Dependencies: Linking C++ libraries

Since we can't use `AC_CHECK_LIB` for linking C++ code, what other options do we have? There's a macro `AX_CXX_CHECK_LIB`, but that is not distributed as part of Autoconf and not even the Autoconf Archive, so I have decided against using it.

Instead, I found a [blog post](https://nerdland.net/2009/07/detecting-c-libraries-with-autotools/) that suggests writing your own check. The basic idea is to use the macro `AC_LINK_IFELSE` (this is the same macro as `AC_CHECK_LIB` uses). To `AC_LINK_IFELSE` you give a source to compile and a segment to run on success and a segment to run on failure. In my case:

    AC_LINK_IFELSE(
        <source here>,
        [HAVE_<library>=1],
        [AC_MSG_ERROR([<library> is not installed.])])

To generate the source code that is used to verify linking, the blog post suggests using the macro `AC_LANG_PROGRAM`. The macro takes to parameters, a prologue and a body. In my case:

    AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])

We are still missing two pieces here. First, even though we set `AC_PROG_CXX`, we have never set the language to use when running compilation tests. This is done using the macro `AC_LANG`. By setting `AC_LANG(C++)`, `AC_LINK_IFELSE` will use a C++ compiler (and linker) rather than the default choice of a C compiler.

Furthermore, on success, `AC_CHECK_LIB` will add the specified library to the variable `LIBS`. Therefore, we have to do it manually. If we would want to be really fancy, we could unset the library from `LIBS`, if the test fails. Since we exit on failure, we don't unset anything.

Here's the whole block of code:

    AC_LANG(C++)
    LIBS="-lgtest $LIBS"
    AC_LINK_IFELSE(
        [AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])],
        [HAVE_GTEST=1],
        [AC_MSG_ERROR([gtest is not installed.])])

### Dependencies: Headers

Some dependencies are merely headers. To check the availability of a header, there's the macro `AC_CHECK_HEADER`. This macro takes the header as it would be used in a `#include` preprocessor directive. Furthermore, you can specify an action to take on success and on failure, respectively. In my case, I'm using the header-only library `nlohmann-json` and only headers from `gmock`, and thus I'm using the following two statements:

    AC_CHECK_HEADER([nlohmann/json.hpp], [], [AC_MSG_ERROR([nlohmann-json is not installed.])])
    AC_CHECK_HEADER([gmock/gmock.h], [], [AC_MSG_ERROR([gmock is not installed.])])

### Dependencies: pkg-config

The idea behind `pkg-config` seems simple enough: it's a program that can be used to query for installed software and their versions, respectively. `pkg-config` requires software to be distributed with package metadata contained in a `.pc` file. The metadata file will contain various information of the package, such as `CFLAGS` and `LIBS`. Sounds like a replacement for `AC_CHECK_LIB`, right?

The Autoconf Archives provides the macro `AX_PKG_CHECK_MODULES`. However, on various online forums and blogs, I found warnings about how Autoconf and `pkg-config` have a fundamentally different approach. Therefore, I decided against using `pkg-config` in my setup.

As I understand it, one problem is that merely the success or failure of running `pkg-config` does not adequately indicate if linking the package works. For example, if `pkg-config` is not installed, the test may fail, even though the dependency we are looking for exists and is linkable. Likewise, there can be a discrepancy between how `pkg-config` and our compiler/linker of choice works. Lastly, the information in the `.pc` file can be wrong. The last point is probably relevant when chrooting or similar.

## All of configure.ac

In the previous sections, I've explained all of my `configure.ac`, with the exception of `AC_OUTPUT`. There's not much to it: we just have to call it in the end of `configure.ac`, as it will generate the files needed for Automake.

My whole `configure.ac` is as follows (comments included):

    $ cat configure.ac
    AC_INIT([algorithm], [1.0])
    AM_INIT_AUTOMAKE([-Wall -Werror foreign])
    
    : ${CXXFLAGS="-g -O2 -Wall -Werror"}
    AC_PROG_CXX
    
    # check that compiler supports c++17 and set -std=c++17
    AX_CXX_COMPILE_STDCXX_17([noext], [mandatory])
    
    AC_CONFIG_FILES([
        Makefile
        src/Makefile
    ])
    
    AC_CHECK_LIB([pthread], [pthread_create], [], [AC_MSG_ERROR([pthread is not installed.])])
    
    # AC_CHECK_LIB doesnt work well with C++ linking. Instead, we use a test program to see if linking of gtest works.
    AC_LANG(C++)
    LIBS="-lgtest $LIBS"
    AC_LINK_IFELSE(
        [AC_LANG_PROGRAM([#include <gtest/gtest.h>], [EXPECT_EQ(1,1)])],
        [HAVE_GTEST=1],
        [AC_MSG_ERROR([gtest is not installed.])])
    
    # nlohmann-json is a header only library.
    AC_CHECK_HEADER([nlohmann/json.hpp], [], [AC_MSG_ERROR([nlohmann-json is not installed.])])
    
    # we need gmock headers even though we don't link it
    AC_CHECK_HEADER([gmock/gmock.h], [], [AC_MSG_ERROR([gmock is not installed.])])
    
    AC_OUTPUT

## Recap on Usage

Alright, so you've created all the needed source files and now you wonder what commands to use? First, we do
`autoreconf` to generate `configure`, `Makefile.in` and many more files:

    $ autoreconf -vif

Note: This step is only done by package maintainers.

To the end-user, we ship all of `configure`, `Makefile.in`'s etc. The first command the end-user does is to run the
`configure` script:

    $ ./configure

If we, as the package maintainers, did everything correctly, running `configure` at the end-user will either fail, if
the program (or library) would never be able to compile or run properly. Likewise, assuming we did everything correctly, 
if `configure` succeeds, then also compiling and running the program (or library) will work.

To build the source:

    $ make

Note that if we did everything correctly, overriding compiler flags with `CXXFLAGS` (or `CFLAGS` if C program), linkable
libraries with `LIBS` etc will work properly.

## The Autoconf Archives

The Autoconf Archives is a separate software distribution from Autoconf itself. It contains a multitude of macros that can be used when generating the `configure`. Note that it's not an issue to depend on the Autoconf Archives, as we don't distribute the `configure.ac`, but instead we distribute `configure`. Hence, the end-user needs neither Autoconf nor the Autoconf Archives.

## Conclusion

In this post I went through what I learned as I setup a C++ project which is built with Autoconf (and Automake). Autoconf is quite different from CMake and other build systems. For this reason it's probably off-putting for newcomers as many things seem unnecessarily complicated. However, in my experience, well-written `configure` files tend to work far better on various systems compared to many other solutions. I doubt I've reached that level of understanding, but everyone needs to start somewhere :-)

While it was definitely frustrating to figure out many things about Autoconf, I'm happy I did it. I feel I understand the world of package management and software distribution a tad better.