Using Shake Oracles
Tags: build, shake, haskell
Published: 2020-05-15 10:18:49

---
title: Using Shake Oracles
published: true
description: How to use some useful helper functions provided by the Shake build system.
tags: build,shake,haskell
series: The Shake build system
---

In a [previous blogpost][rules-as-recipes] I argued for the use of the Shake build system, and for writing Shake rules like they are recipes. I also recommended the use of `newCache`, `addOracle`, and `addOracleCache` in situations where it's tricky to do recipes. `newCache`, `addOracle`, and `addOracleCache` are super useful, but they have imposing types, not the most helpful names, and they look similar, making it hard to know which one we need. In this post we're going to look at each in a little more detail. Let's start by briefly introducing these functions.

## Three functions with different uses

`newCache` allows us to memoize a function. A memoized function stores all the results it calculates. When a memoized function gets called with arguments it has seen before, it returns an earlier calculated result rather than repeating work. We can use `newCache` to prevent duplicate work during a build but not across builds, because Shake throws away memoized results after each run.

`addOracle` allows us to define dependencies that aren't files. Say that we want a rule to rebuild when the date changes. We can write some Haskell to get the current date and make it available as a dependency using `addOracle`. These oracles run on every build because Shake needs to know if the values they return have changed, in which case the rules that depend on them need to be re-evaluated.

`addOracleCache`, whatever its name implies, has a use case different from either `newCache` or `addOracle`. We can use it to create a rule that produces a Haskell value instead of a file. Suppose we have an expensive calculation, like recursively finding all the dependencies of a source file. We could define a regular build rule using `%>` that performs the calculation and stores the result in a `sources.json` file. Other rules could `need ["sources.json"]`, decode its contents and use the result. `addOracleCache` allows us to do the same thing without the encoding and decoding steps. Like other build rules and unlike `addOracle`, a rule defined using `addOracleCache` reruns if any of its dependencies changes.

To summarize what these three functions do before looking at each in more detail:

- `newCache` memoizes a function call within the current build.
- `addOracle` defines dependencies that aren’t files.
- `addOracleCache` creates a rule that produces a Haskell value instead of a file.

Now let's look at these in more detail.

## How to use `newCache`

With `newCache` we can create a memoized function. This is a function that when called several times with the same argument will run just once. The memoized function stores the result of the first run for use in future calls.

`newCache` has the following type.

```hs
newCache :: (Eq k, Hashable k) =>
            (k -> Action v) -> Rules (k -> Action v)
             ^^^^^^^^^^^^^            ^^^^^^^^^^^^^
             the function             the memoized
             to memoize               function
```

If we skip over the type constraints before the `=>` first (we'll get to them in a moment!), then we can see that `newCache` takes a function and then returns a memoized version of it. Wherever we were using the original function we can use the memoized version too, because they have the same type.

To add memoization behavior Shake needs to know whether we called the function with a particular argument before. That requires comparing the input argument with input arguments from previous calls, and so we need to insist the input argument is of a type that allows such comparisons. Shake does this by requiring the input type to meet the `Eq` and `Hashable` constraints.

One tricky situation where `newCache` can help us out is when writing a rule for a command that produces more than one file, but where we don't want to hardcode which files it produces. Suppose we have a script `generate-schemas.sh` that generates JSON schemas for common types.

```hs
rules :: Rules ()
rules = do
  "schemas/*.schema.json" %> \_ -> do
    schemaSrcs <- getDirectoryFiles "" ["elm/src/ApiTypes//*.elm"]
    need ("generate-schemas.sh" : schemaSrcs)
    cmd_ "generate-schemas.sh" schemaSrcs
```

The rule above works but is pretty inefficient. `generate-schemas.sh` creates all the schema files in a single run, but the rule above will rerun it every time we `need` a different schema. Let's use `newCache` to remove this duplication of work.

```hs
rules :: Rules ()
rules = do
  generateSchemasCached <- newCache $ \() -> do
    schemaSrcs <- getDirectoryFiles "" ["elm/src/ApiTypes//*.elm"]
    need ("generate-schemas.sh" : schemaSrcs)
    cmd_ "generate-schemas.sh" schemaSrcs

  "schemas/*.schema.json" %> \_ -> generateSchemasCached ()
```

Now if more than one schema files is `need`ed during a run, `generate-schemas.sh` is ran just once. Later runs might run `generate-schema.sh` again, because `newCache` doesn't save results across builds. That's good though, because we might have deleted schema files in the interim.

## How to use `addOracle`

Rules can signal they have a dependency on one or more files using `need`. This causes the rule to rebuild if the file they depend on changes. But what if you'd like your rule to depend on the day of the week, or the version of a tool? Using `addOracle` you can define such dependencies.

Say we have a rule that creates a letter from a template.

```hs
rules :: Rules ()
rules = do
  "letter.txt" %> \out -> do
    date <- liftIO Date.current
    need ["letter.template"]
    cmd_
      "templateer letter.template"
      "--out" [out]
      "--options" ["date=" <> date]
```

Imagine we're working on this until late in the evening. The next day we want to post it, so we run Shake one more time to get today's date on there. Shake skips the update though because it doesn't know it should rebuild the letter when the date changes.

If we had a file somewhere on our computer that always contained the current date then we could `need` that. As it stands the current date is not a file, it's the result of a call to the (made up) `Date.current` function. We can use `addOracle` to turn it into a dependency.

Let's start again by looking at `addOracle`'s type. Notice how apart from the functions' constraints (the part of the type before the `=>`) `addOracle` has the exact same type as `newCache`.

```hs
addOracle :: (RuleResult q ~ a, ShakeValue q, ShakeValue a) =>
             (q -> Action a) -> Rules (q -> Action a)
              ^^^^^^^^^^^^^            ^^^^^^^^^^^^^

type ShakeValue a = (Show a, Typeable a, Eq a, Hashable a, Binary a, NFData a)
```

Like `newCache` `addOracle` takes a function and returns a function of the same type. This new function adds a dependency in any rule it gets called in. Shake requires a bunch of constraints on the function's argument type `q` and return type `a` to pull this off. Check out the [`ShakeValue` documentation][shakevalue-docs] if you're interested in learning what these constraints are for. We'll see in a moment what `RuleResult q ~ a` is about.

We can use `addOracle` to fix our letter templating rule like so.

```hs
rules :: Rules ()
rules = do
  currentDate <- addOracle $ \CurrentDate -> liftIO Date.current

  "letter.txt" %> \out -> do
    date <- currentDate CurrentDate
    need ["letter.template"]
    cmd_
      "templateer letter.template"
      "--out" [out]
      "--options" ["date=" <> date]

data CurrentDate = CurrentDate
  deriving (Show, Eq, Generic)

instance Hashable CurrentDate

instance Binary CurrentDate

instance NFData CurrentDate

type instance RuleResult CurrentDate = String
```

We made a similar change when we introduced `newCache`: We extract into a function the lines we want to add special behavior to (memoization behavior in case of `newCache`, dependency-tracking behavior now). We wrap our extracted function using the right helper, and use function this returns in our rule.

What's different this time is that Shake requires each oracle created using `addOracle` or `addOracleCache` to have a unique type (we'll see why in a moment). We create one called `CurrentDate` and use `Generic` to generate all the instances Shake requires of the type. We also have to tell Shake that the oracle associated with the `CurrentDate` input type always returns a `String` result type (the return value of our imaginary `Date.current` function).

## Intermezzo: What's up with these boilerplate types?

The reason oracles argument types need to be unique and the reason we for the `RuleResult q ~ a` constraint is to support the `askOracle` function in Shake's APIs. Using it our letter templating example looks like this:

```hs
rules :: Rules ()
rules = do
  void . addOracle $ \CurrentDate -> liftIO Date.current

  "letter.txt" %> \out -> do
    date <- askOracle CurrentDate
    need ["letter.template"]
    cmd_
      "templateer letter.template"
      "--out" [out]
      "--options" ["date=" <> date]

data CurrentDate deriving (Show, Eq, Generic)

instance Hashable CurrentDate

instance Binary CurrentDate

instance NFData CurrentDate

type instance RuleResult CurrentDate = String
```

We can pass `askOracle` any of the types we have defined oracles for. Because all our oracles have unique input types `askOracle` can figure out which one to call. And because we have explicitly defined the return types for each oracle input type using the `RuleResult` type family the type checker knows the type of `askOracle SomeOracleType`.

The whole thing is pretty magical, so I like to wrap up oracles in an API that exposes regular functions. As an example, we could wrap up the date oracle like this:

```hs
module Rules.Date (rules, current)

import Development.Shake
import qualified Date

current :: Action String
current = askOracle Current

rules :: Rules ()
rules =
  void . addOracle $ \Current -> currentOracle

currentOracle :: Action String
currentOracle = liftIO Date.current

data Current deriving (Show, Eq, Generic)

instance Hashable Current

instance Binary Current

instance NFData Current

type instance RuleResult Current = String
```

Our letter templating example could use this module like so.

```hs
rules :: Rules ()
rules = do
  Rules.Date.rules

  "letter.txt" %> \out -> do
    date <- Rules.Date.current
    need ["letter.template"]
    cmd_
      "templateer letter.template"
      "--out" [out]
      "--options" ["date=" <> date]
```

## How to use `addOracleCache`

Suppose we have a project that contains several Elm applications. The Elm applications share some modules between them. If we change an Elm module we'd like Shake to recompile just those projects that use the module. We could write a rule like this:

```hs
rules :: Rules ()
rules = do
  "assets/*.elm.js" %> \out -> do
    let (Just [name]) = filePattern "assets/*.elm.js" out
    let main = name <.> "elm"
    let srcFiles = recursiveDependencies main
    need ("elm.json" : srcFiles)
    cmd_ "elm make --output" [out] [main]

recursiveDependencies :: FilePath -> Action [FilePath]
recursiveDependencies src = do
  direct <- directDependencies src
  recursive <- traverse recursiveDependencies direct
  pure (src : mconcat recursive)

directDependencies :: FilePath -> Action [FilePath]
directDependencies src =
  need [src]
  contents <- readFile src
  pure $ Elm.imports (Elm.parse contents)
```

This works, but it's not super efficient. For each Elm entrypoint it recalculates the full dependency tree. It would be nice if after calculating the dependency for a particular Elm module we could reuse that result in future builds, until any of the recursive dependencies of a module change.

The function `recursiveDependencies` looks a lot like a rule. The result it produces is a list of file paths corresponding to Elm modules. Its dependencies are the contents of those Elm modules, because a change in an Elm module might mean that its imports have changed, requiring a recalculation of the dependency tree. Let's use `addOracleCache` to write it as a rule.

```hs
rules :: Rules ()
rules = do
  void $ addOracleCache recursiveDependencies

  "assets/*.elm.js" %> \out -> do
    let (Just [name]) = filePattern "assets/*.elm.js" out
    let main = name <.> "elm"
    let srcFiles = askOracle (RecursiveDependenciesFor main)
    need ("elm.json" : srcFiles)
    cmd_ "elm make --output" [out] [main]

recursiveDependencies :: RecursiveDependencies -> Action [FilePath]
recursiveDependencies (RecursiveDependenciesFor src) = do
  direct <- directDependencies src
  recursive <- traverse (askOracle . RecursiveDependenciesFor) direct
  pure (src : mconcat recursive)

directDependencies :: FilePath -> Action [FilePath]
directDependencies src = do
  contents <- readFile' src -- This takes a dependency on `src`.
  pure $ Elm.imports (Elm.parse contents)

newtype RecursiveDependencies
  = RecursiveDependenciesFor FilePath
  deriving (Show, Eq, Hashable, Binary, NFData)

type instance RuleResult RecursiveDependenciesFor = [FilePath]
```

Done! Now we'll cache the calculation of module dependencies between builds. One further possible optimization would be to use `addOracleCache` to turn `directDependencies` into a build rule as well. That way changes to Elm modules that don't touch imports won't trigger recalculation of a module's dependencies. Give it a try!

It's worth emphasizing that although `addOracleCache` has an identical type to `addOracle`, it behaves quite differently. Remember that `addOracle` is for defining dependencies. Shake runs `addOracle` functions pre-emptively to check if their return values have changed. Had we used `addOracle` here performance would be worse than the non-oracle-based version of the code we started with, because Shake would rerun it even if none of the Elm source files in the entire project had changed.

## Closing thoughts

I hope this post has been helpful in understanding when and how to use Shake's `newCache`, `addOracle`, and `addOracleCached` functions. One final tip: make your oracle types nice and verbose because Shake uses them in its logs. It will make debugging oracles easier.

That's it. Happy shaking!

[rules-as-recipes]: https://dev.to/jwoudenberg/shake-rules-as-recipes-2el3
[shakevalue-docs]: https://www.stackage.org/haddock/lts-15.11/shake-0.18.5/Development-Shake.html#t:ShakeValue