Idempotent seeds in Elixir
Tags: elixir, seeds, phoenix
Published: 2024-03-14 09:36:37

A standard Phoenix app contains a [`priv/repo/seeds.exs`](https://github.com/phoenixframework/phoenix/blob/3c5993e459fef0d2e66259722e27975bd651979d/installer/templates/phx_ecto/seeds.exs) script file, which populates a database when it is run, so that developers can work with a conveniently prepared environment.

At Bitcrowd, we like our seeds to be idempotent. In practice, this means that running `mix run priv/repo/seeds.exs` multiple times will not create more rows each time, but rather upsert the existing data. While it is always possible to drop the local database with `mix ecto.reset`, we might want to keep the current state of our development database.  

## Upsert all the things!

Our strategy here is to use [`insert!`](https://hexdocs.pm/ecto/Ecto.Repo.html#c:insert/2) with a `conflict_target` on the `:id`. Let's take a classic blog app with `Post` and `Comment` schemas for the sake of the example. In our `seeds.exs` file we add this little helper:

```elixir
def insert_idempotently(schema) do
  Repo.insert!(schema, on_conflict: :replace_all, conflict_target: :id)
end
```

And instead of inserting a seed record like so:

```elixir
Repo.insert!(%Post{title: "Hello World!"})
```

We wrap our insert in the helper:

```elixir
@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
insert_idempotently(%Post{id: @id, title: "Hello World!"})
```

Nice 🎉

Next time we run the seeds, we will not get a second `Post` row in the database if a post with this `:id` already exists. This keeps our development environment neat and clean. 

## Idempotency for `has_many` associations

While the previous example is fairly simple, hardcoding UUIDs has its limitations when seeding `has_many` associations. For example, let's say we want to insert 50 `Comments` associated to the `Post`:

```elixir
@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
post = insert_idempotently(%Post{id: @id, title: "Hello World!"})

Enum.each(1..50, fn index -> 
  insert_idempotently(%Comment{
    id: ???, 
    message: "Comment #{index}", 
    post_id: post.id
  })
end)
```

Of course one could add 50 lines of hardcoded UUIDs… Or could we generate deterministic UUIDs from the `index` value? Yes we can ⚡️!

### Deterministic UUID v4 from a string

Our deterministic UUID generator should take a string as an argument, and always return the same UUID for the same argument. We first need to hash our string, and then to extract the number of bits that we need. UUIDs have a consistent structure: `"64af9d13-0f60-45fc-971f-07e6b490c059"`: one group of 8 characters, then three groups of 4, and finally a group of 12, all separated by a `-`.

This is how it works:

```elixir
def deterministic_uuid4(string) do
  # Hash the string and extract the 128 bits,
  # and match on the length of our characters group
  <<a::size(32), b::size(16), c::size(16), d::size(16), e::size(48), _rest::binary>> =
    :crypto.hash(:sha256, string)

  # Override some bits (necessary to create valid UUID v4)
  c = bor(band(c, 0x0FFF), 0x4000)
  d = bor(band(d, 0x0FFF), 0x8000)

  # Glue all of the chunks together and turn it into a string
  Enum.map_join([a, b, c, d, e], "-", &Base.encode16(:binary.encode_unsigned(&1)))
end
```

Let's see it in action:

```sh
iex(1)> MyApp.Seeds.deterministic_uuid4("foo")
"2C26B46B-68FF-468F-899B-453C1D304134"

iex(2)> MyApp.Seeds.deterministic_uuid4("foo")
"2C26B46B-68FF-468F-899B-453C1D304134"

iex(3)> MyApp.Seeds.deterministic_uuid4("bar")
"FCDE2B2E-DBA5-4BF4-8860-1FB721FE9B5C"
```

Finally, let's validate that our generated UUID is valid with the [uuid](https://hexdocs.pm/uuid/readme.html) utility:

```sh
iex(1)> UUID.info("2C26B46B-68FF-468F-899B-453C1D304134")
{:ok,
 [
   uuid: "2C26B46B-68FF-468F-899B-453C1D304134",
   binary: <<44, 38, 180, 107, 104, 255, 70, 143, 137, 155, 69, 60, 29, 48, 65,
     52>>,
   type: :default,
   version: 4,
   variant: :rfc4122
 ]}
```

Let's rewrite our seeds to make use of our brand new function:

```elixir
@id = "64af9d13-0f60-45fc-971f-07e6b490c059"
post = insert_idempotently(%Post{id: @id, title: "Hello World!"})

Enum.each(1..50, fn index -> 
  insert_idempotently(%Comment{
    id: deterministic_uuid4("comment-#{index}"), 
    message: "Comment #{index}", 
    post_id: post.id
  })
end)
```

Amazing! We won't get 50 new rows of `Comment` each time we run the seeds script. Our development database is clean and we made our developers happy ☕️.

### UUID Version-5

To ruin the party, deterministic UUID generation is exactly what UUID v5 is designed for. And since [Ecto does not validate UUIDs against their specs](https://github.com/elixir-ecto/ecto/issues/2147), you might as well use [uuid](https://hexdocs.pm/uuid/readme.html) again and do:

```elixir
iex(7)> UUID.uuid5(:nil, "foo")
"aa752cea-8222-5bc8-acd9-555b090c0ccb"

iex(8)> UUID.uuid5(:nil, "foo")
"aa752cea-8222-5bc8-acd9-555b090c0ccb"
```

But what's the fun in that 🤷‍♀️.

## Additional resources

- [UUID Versions Explained](https://www.uuidtools.com/uuid-versions-explained)
- [rfc4122](https://datatracker.ietf.org/doc/html/rfc4122#section-4.1.1)
- [More about how UUIDs are structured](https://www.sohamkamani.com/uuid-versions-explained/)

---

> If you enjoyed reading this, you might be interested in working with Elixir at bitcrowd. Check our [job offerings](https://bitcrowd.net/jobs/backend-developer)!