Efficient file hashing in Elixir
Tags: elixir, performance
Published: 2021-03-16 18:07:30

_You can download the code from this article [from GitHub](https://github.com/diogoko/hasher)_

Elixir in itself does not have functions to calculate hashes of files or data, but as usual you can use Erlang modules to do that. The [crypto](https://erlang.org/doc/man/crypto.html) module offers several cryptographic services, including many hashing algorithms.

# Naive approach

The easiest option is the [hash](https://erlang.org/doc/man/crypto.html#hash-2) function. It takes an atom indicating one of the supported algorithms and the data to be hashed. When hashing a file, you need to call [File.read!](https://hexdocs.pm/elixir/File.html#read!/1) or similar to read the data before calling `hash`.

```elixir
data = File.read!("sample.pdf")
sha256 = :crypto.hash(:sha256, data)
```

# Streaming approach

The problem with using the `hash` function is that it only works if the whole file is present in the memory. When working with large files this can quickly degrade the performance of the application or even crash it.

An alternative is using the "streaming mode" of the hashing functions. Instead of feeding the data to the hashing function at once, you read the data in pieces and apply the hashing algorithm to each piece in sequence, updating its internal state until all data has been processed and the hashing algorithm has its final result. This is how these hashing algorithms actually work and this mode is available in other programming languages too.

```
Initialize hashing algorithm context
While there is more data:
    Feed the algorithm a piece of data
Get the final result from the hashing algorithm
```

In Elixir, this can be implemented using the [File.stream!](https://hexdocs.pm/elixir/File.html#stream!/3) function and `Enum.reduce`.

```elixir
initial_hash_state = :crypto.hash_init(:sha256)

sha256 =
  File.stream!("sample.pdf", [], 2048)
  |> Enum.reduce(initial_hash_state, &:crypto.hash_update(&2, &1))
  |> :crypto.hash_final()
```

The function `hash_init` creates a "hash state" object that is updated by the hashing algorithm as new data is processed.  At this point, it's state is equivalent to hashing an empty file.

`File.stream!` produces an enumerable in which each item is a binary with length of up to 2048 bytes (in this example). This parameter can be tuned according to memory usage and performance requirements: larger buffers are faster but use more memory.

The enumerable returned by `File.stream!` is lazy and sometimes you need to explicitly execute it by calling [Stream.run](https://hexdocs.pm/elixir/Stream.html#run/1). Alternatively, most functions from the `Enum` module will trigger the execution of the file stream.

Inside `Enum.reduce` we call `hash_update`, passing the current hash state and the data to be processed. It returns the new state of the hasher, to be updated with the next item from the file stream or returned as the final result of the `Enum.reduce` call.

Having the hash state after processing the last data piece, we call `hash_final` to get the calculated digest as a binary.

# Bonus: formatting the hash

The result of hashing algorithms is a sequence of bytes (16 for MD5, 20 for SHA1, 32 for SHA256, etc.), but usually we present them in hexadecimal format. To do that, use the [Base.encode16](https://hexdocs.pm/elixir/Base.html#encode16/2) function.

```elixir
formatted_sha256 = Base.encode16(sha256, case: :lower)
```