Dealing with dependence on generated source files in a Makefile
Tags: makefile
Published: 2018-03-31 07:34:26

---
title: "Dealing with dependence on generated source files in a Makefile"
published: true
tags: makefile
canonical_url: https://blog.tznvy.eu/2017-07-27-dealing-with-dependence-on-generated-source-files-in-a-makefile
---

I think I finally got it right today.

I have a static website, with pages generated from markdown documents. Some of them are written by hand, but most are generated with a python script from a flat database in a json file. The build process is automated with a Makefile.

I wanted to trigger the generator only when necessary, that is, when either the json or the script were changed. There are too many markdown docs, however, to hardcode their names in the Makefile. Normally, this is solved using a wildcard filemask, which enumerates relevant file names in a source directory, and prepending the path to the target directory:

```
ALL_MDS_IN_OBJ = $(wildcard $(OBJ)/*.md)
ALL_HTML_IN_BIN = $(addprefix $(BIN)/, $(notdir $(ALL_MDS_IN_OBJ:.md=.html)))

default: $(ALL_HTML_IN_BIN)

$(BIN)/%.html: \
    $(OBJ)/%.md \
    | $(BIN)

    ... convert mds to html with pandoc ...

$(OBJ)/%.md: \
    $(SRC)/source_data.json \
    generate-mds-from-json.py \
    | $(OBJ)

    ./generate-mds-from-json.py
```

The problem is, when `make` is run for the for the first time after cloning the repo or doing `make clean`, there are no markdown files in `$(OBJ)` yet. The wildcard doesn't match anything, so `make` happily announces that nothing needs to be done and exits.

To cope with that, I was running the generator manually before `make` each time I changed the json file or the script, which, of course, in the long run turned out to be tedious and error-prone.

Then I saw [this question on stackoverflow](https://stackoverflow.com/questions/991841/makefile-dependency-for-unknown-files-in-known-directory-for-docbook) and was finally enlightened.

As [ChrisW](https://stackoverflow.com/questions/991841/makefile-dependency-for-unknown-files-in-known-directory-for-docbook#answer-992029) explained, making the generator run for the first time could be forced by introducing a dependency on a sentinel file kept separately from the build artifacts, whose sole purpose was to contain the timestamp of the last generator run; The file would be touched if the generation was successful, and deleted on `make clean`.

```
MDS_SENTINEL = .mds_sentinel

default: \
    $(MDS_SENTINEL) \
    $(ALL_HTML_IN_BIN) \
    ... other deps ...

$(MDS_SENTINEL): \
    $(SRC)/source_data.json \
    generate-mds-from-json.py

    ./generate-mds-from-json.py \
        && touch $(MDS_SENTINEL)

clean:
    rm -r $(MDS_SENTINEL) $(OBJ)/* $(BIN)/*
```

But this was still not sufficient; As mentioned above, the wildcard function would still be evaluated before running the generator, resulting in empty files list and an early exit. It would be necessary to invoke `make` twice - first time to generate the intermediate files, second time to process them further. And compared to `./generate-mds-from-json.py && make`, having to do `make && make` was not that huge win.

Luckily, this problem was solved in the same thread by [Paul Roub](https://stackoverflow.com/questions/991841/makefile-dependency-for-unknown-files-in-known-directory-for-docbook#answer-994248), who suggested running `make` recursively from the recipe. This _inner_ make would have wildcards expanded after all files are generated and process the files correctly.

So the final solution looked something like:

```
MDS_SENTINEL = .mds_sentinel

ALL_MDS_IN_OBJ = $(wildcard $(OBJ)/*.md)
ALL_HTML_IN_BIN = $(addprefix $(BIN)/, $(notdir $(ALL_MDS_IN_OBJ:.md=.html)))

default: \
    $(MDS_SENTINEL) \
    $(ALL_HTML_IN_BIN) \
    ... other deps ...

$(MDS_SENTINEL): \
    $(SRC)/source_data.json \
    generate-mds-from-json.py

    ./generate-mds-from-json.py \
        && touch $(MDS_SENTINEL) \
        && make mds

mds: $(ALL_HTML_IN_BIN)

clean:
    rm -r $(MDS_SENTINEL) $(OBJ)/* $(BIN)/*
```



_This post was originally published on [blog.tznvy.eu](https://blog.tznvy.eu/2017-07-27-dealing-with-dependence-on-generated-source-files-in-a-makefile)_