Back to the Future in Clojure
Tags: clojure, functional
Published: 2020-02-09 01:05:44

---
title: Back to the Future in Clojure
published: true
description: 
tags: clojure,functionalprogramming
cover_image: https://dev-to-uploads.s3.amazonaws.com/i/i2pxwiw1qi0vv88l78o0.jpg
---

Parallel processing is an exciting topic, especially when talking about heavy processes that can be split into smaller tasks, executed in isolation, aggregated, and presented as a unique outcome.

Clojure provides different ways to accomplish this kind of job, and *futures* are a handy approach to split a big task into parallel threads that will eventually be gathered by the process that started them up, allowing it to aggregate the results to provide the same outcome in a faster way.

# Talking Futures

You can think about a *future* as a piece of code that runs on its own and independent thread. More specifically, a *future* is a Clojure macro that takes a set of expressions in order to execute them in another thread. The macro returns a reference in memory to the triggered *future*. This allows to communicate with it via some helper functions:

- [future?](https://clojuredocs.org/clojure.core/future_q): Verifies the provided argument is a *future*.
- [future-done?](https://clojuredocs.org/clojure.core/future-done_q): Takes a *future* as argument and verifies whether it has been completed.
- [future-cancel](https://clojuredocs.org/clojure.core/future-cancel): Attempts to cancel the given *future*.
- [future-cancelled?](https://clojuredocs.org/clojure.core/future-cancelled_q): Verifies if the *future* passed as argument was cancelled.


The result of the *future* is stored and available somewhere in the cache, we can attempt to get its outcome even when not finished yet by calling `(deref my-future)`. In this case, the thread that tries to `deref` is going to get blocked until the *future* is done.

# Getting the Stargazers from Github

We are interested in getting the total amount of stars that [Clojure](https://github.com/clojure) has on Github. That is, the sum of stars from each repository that belongs to the Clojure account.

This information can be readily fetched by the [Github API](https://developer.github.com/v3/). The `GET /users/:username/repos` [endpoint](https://developer.github.com/v3/repos/#list-user-repositories) already responds with the star count of each repository. We can just loop over the returned JSON array, get the star counts and sum them up.

For the sake of this article, let's assume the endpoint above does not return the star counts, but the name of the repositories, so that we can hit the `GET /repos/:owner/:repo` [endpoint](https://developer.github.com/v3/repos/#get) to get the star count. This means that for each repository name, a request to `GET /repos/:owner/:repo` will be launched.

![](https://dev-to-uploads.s3.amazonaws.com/i/qokm249rxkosa3u6fi94.png "Aggregate Repositories Synchronously")

As the image above suggests, getting the stars from each repository is a sequential and synchronous task, which might take a considerable amount of time for each request to complete. Once all requests have successfully responded, the `reduce` function comes into play to sum up all the stars.

The following function, `get-star-count`, executes the HTTP request synchronously to get the repository details and then extracts the `stargazers-count` attribute from the response, which represents the star count.

{% gist https://gist.github.com/jovannypcg/a29856e6d6c53f27e1b1f4e7b701e264 %}

Now, let’s take a look at the `sum-stars` function. It provides two arities, one that takes the username exclusively and another one that requires a sequence of repository names:

{% gist https://gist.github.com/jovannypcg/b739a00bc67efdab66034e57972f1cfd %}

Note that the first arity in line 3 calls the 2-arity function and sends the result of `github/repo-names` as the second argument (`repos`). This involves an additional HTTP call that prevents us from manually entering the repo names using a previously defined sequence. We can avoid this additional HTTP call interfering with `sum-stars` by either providing a defined sequence containing the repository names or storing the result of `github/repo-names` in a symbol and then pass it in to the function.

Each repository is passed in to `get-star-count` in line 6, blocking the main thread until each single request is finished synchronously. Then the results are aggregated by `reduce`.

# Futures in Action

Now we are going to implement the same functionality using *futures*. This time, for each repository a *future* that executes `GET /repos/:owner/:repo` will be launched.

![](https://dev-to-uploads.s3.amazonaws.com/i/q0e1reu017gtwbsuikks.png "Aggregate Repositories Asynchronously")

Every HTTP request is working in parallel, as shown by the image above. The *futures* are triggered in point A using the `future` macro. Once they all are ready with their response, a *list comprehension* can be utilized to make them meet in point B in order to be reduced to a unique result.

Note that some requests might take longer than others, as they depend on the network throughput, latency of the Github server or even the speed of the processor running the threads. So, the worst case for this scenario is the request that takes more time to complete, as that will be the time for `sum-stars` to be done.

{% gist https://gist.github.com/jovannypcg/0ee4009043cb170d60ac4c19306505f3 %}

The 1-arity function remains pretty much the same as in the synchronous version. All the repos of the given username are passed in to the 2-arity function.

The interesting part begins in line 5, with the `for` statement, which creates a list comprehension of *futures*. That is, a sequence of *futures* (line 6) that represents the location in memory of the threads working independently. Having this sequence of *futures* allows us to communicate with them using the helper functions explained earlier or simply try to `deref` and get the result.

The pipeline in line 7 shows up the process to aggregate each result. The sequence of *futures* is passed in to `map` using `deref`. This is going to block the main thread until each parallel thread is done. Once derefed, the list comprehension of *futures* becomes a sequence of responses with the actual star counts. Then they are passed in to the `reduce` function that simply sums them up.

# Benchmarking

Let’s see how fast the sync and async versions work. In order to do that, we are going to use the handy `time` function that is part of `clojure.core`.

## Considerations

- Executed on MacBook Pro, macOS Catalina 10.15.1, 2.2GHz 6-Core Intel i7, 16 GB 2400 MHz DDR4.
- The [Clojure](https://github.com/clojure) account on Github is used for the tests.
- `github/repo-names` is limited to return 30 repos, despite [Clojure](https://github.com/clojure) owns 85+.
- Based on the point above, the tests consider 30 repositories from the [Clojure](https://github.com/clojure) account.

```clojure
user=> (def username "clojure")
user=> (def repos (github/repo-names username))
user=>
user=> (time (core/sum-stars username repos))
"Elapsed time: 13841.095288 msecs"
20263
user=>
user=> (time (core/sum-stars-async username repos))
"Elapsed time: 587.558128 msecs"
20263
```

As you can see, the result of both versions is the same `20263`, which represents the total stars of the first 30 repositories of Clojure on Github. But the elapsed time for the synchronous version is almost 14 seconds, unlike the 587.55 ms of the asynchronous call.

# Conclusion

Parallel processing provides a powerful way to split a heavy task into smaller and independent problems that can be aggregated once they all are done. *Futures* in Clojure are a powerful and easy-to-use approach to accomplish this kind of task.

Next time you are dealing with a function that is taking so long to complete, consider using *futures*.

Do not forget to clone and play around with the [code of this article](https://github.com/jovannypcg/parallel-processing).

Happy coding!

The cover image is a Photo by [Jamie Street](https://unsplash.com/@jamie452?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText) on [Unsplash](https://unsplash.com/s/photos/parallel?utm_source=unsplash&utm_medium=referral&utm_content=creditCopyText)