An alternative to using Thread.sleep in your Java tests
Tags: java, testing
Published: 2022-07-04 09:57:30

At times the code you are testing runs in a separate thread than the thread executing the test itself. In this case, knowing _when_ to perform the assertions can be troublesome as you must exercise some sort of synchronization between the threads.

A solution is to use `Thread.sleep`. While I personally do not not consider it wrong, using `Thread.sleep` has some drawbacks:

- sleep for too long and your test spends a lot of time doing nothing;
- not enough sleep and your test will fail; and
- sleep slight above the "necessary" and your tests might fail intermittently.

In this blog post I will show you an alternative to using `Thread.sleep` in your Java tests. This post assumes you can refactor the code you are testing.

## The problem

To illustrate the problem let's jump right into our running example:

```java
class AsyncCounter implements Runnable {
  static final int COUNT_TO = 1_000_000;

  private volatile int value;

  public int get() {
    return value;
  }

  @Override
  public void run() {
    while (value < COUNT_TO) {
      value++;
    }
  }

  public void startCounting() {
    var t = new Thread(this);

    t.start();
  }
}
```

It is a counter or incrementer that runs in a separate thread. You use it by invoking the `startCounting()` method; and you can query the current value by invoking the `get()` method. I know... it is of no practical use and has some warning signs, such as updating a `volatile` in a loop, but it should be enough as our example.

Let's write a test for it. The counter is fixed to count to one million. So, after it has finished its work, we expect the `value` to be one million.

Of course we could just invoke the `run()` method straight from our test. But bear with me, we want the counting to occur asynchronously.

An initial solution attempt could be:

```java
public class AsyncCounterTest {
  @Test
  public void test() {
    var counter = new AsyncCounter();

    counter.startCounting();

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);
  }
}
```

But running this test fails:

```
FAILED: test
java.lang.AssertionError: expected [1000000] but found [0]
```

This is expected. Our assertion runs right after the thread starts and before the thread has a chance to do any work.

## Using Thread.sleep

A solution could be to add a `Thread.sleep` after the counting has started and before the assertion, like so:

```java
public class AsyncCounterTest {
  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter();

    counter.startCounting();

    Thread.sleep(1000);

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}
```

Apart from the `Thread.sleep`, I have also modified the test so it prints the total time (in ms) it took running the test. Running this version of the test in my machine gives:

```
totalTime=1003
PASSED: test
```

The question is: how much of the time sleeping is actual waiting for the counting to be over and how much is just sleeping and not doing much?

If we reduce the sleeping time from 1s to, say, 5ms and we ask TestNG to run the test three times the test fails (on my machine):

```
FAILED: test
java.lang.AssertionError: expected [1000000] but found [478484]

FAILED: test
java.lang.AssertionError: expected [1000000] but found [647833]

FAILED: test
java.lang.AssertionError: expected [1000000] but found [655242]
```

Finding an optimal waiting time is troublesome. Let's analyze some alternatives.

## Alternative: use `wait()` and `notify()`

The first thing we need to do is refactor our `AsyncCounter` so it can send a signal when the counting is over. Let's create a listener interface with a single `countingOver()` method:

```java
interface Listener {
  void countingOver();
}
```

And we invoke this method at the end of the `run()` execution: when the counting is over. The full modified code is listed below:

```java
class AsyncCounter implements Runnable {
  static final int COUNT_TO = 1_000_000;

  private final Listener listener;

  private volatile int value;

  public AsyncCounter(Listener listener) {
    this.listener = listener;
  }

  public int get() {
    return value;
  }

  @Override
  public void run() {
    while (value < COUNT_TO) {
      value++;
    }

    listener.countingOver();
  }

  public void startCounting() {
    var t = new Thread(this);

    t.start();
  }

  interface Listener {
    void countingOver();
  }
}
```

Now we have to refactor our test as well.

After we start the counter we need to block the thread running the test until the counting is over. Remember, the counting happens in a different thread.

In Java we can do this by invoking the `wait()` method on a object that will be used for synchronization between threads. But before a thread can invoke `wait()` on an object, it must first acquire the lock on that object. We will use the test object itself for synchronization, like so:

```java
counter.startCounting();

synchronized (this) {
  wait();
}
```

So the thread running the test:

- starts the counter and, therefore, starts the counting thread
- acquires the lock on the test object
- invokes `wait()` on the test object

__Before__ it returns from the `wait()` method, in other words, _inside_ the `wait()` method, the thread running the test:

- releases the lock on the test object
- blocks (stops executing code) and waits until notified by another thread

_(to be exact the blocked thread can also stop waiting when interrupted. In this case the `wait()` throws an `InterruptedException`)_

Next we have to ask the counting thread to notify the blocked thread when the counting is over. In Java we can do this by invoking the `notify()` (or `notifyAll()`) method on the object being used for synchronization. We are using the test object itself in our case.

So let's have our test class implement the `Listener` interface and let's code the `countingOver()` method:

```java
@Override
public void countingOver() {
  synchronized (this) {
    notify();
  }
}
```

Remember, this method is invoked by the counting thread when the counting is over.

Just like the `wait()` method, before a thread can invoke the `notify()` method on an object, it must first acquire the lock on that object.

So the counting thread:

- acquires the lock on the test object. This lock is available. The testing thread released it as a consequence of invoking `wait()`
- invokes `notify()` on the test object. This notifies a single thread waiting on this object.
- releases the lock on the test object by exiting the `synchronized` block.

Notice that, since we know there is a single thread waiting, we invoke the `notify()` method and not the `notifyAll()` method.

Since now:

- the lock on the test object is available; and
- the testing thread has been notified by the counting thread

The testing thread will try to acquire the lock on the test object again. Once the lock is acquired, it will continue executing by exiting the `wait()` method.

The full version of the test is listed below:

```java
public class AsyncCounterTest implements AsyncCounter.Listener {
  @Override
  public final void countingOver() {
    synchronized (this) {
      notify();
    }
  }

  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter(this);

    counter.startCounting();

    synchronized (this) {
      wait();
    }

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}
```

Running this test results:

```
totalTime=13
PASSED: test
```

## Variation: use a `Semaphore`

I personally think the previous solution is enough. In particular because it is running in a testing environment.

Having said that, explicitly using both `synchronized` and `Object.wait()` has implications regarding virtual threads. Both will block the carrier thread of the virtual. But scheduler's behavior for each case is slightly different:

- a `synchronized` block (or method) causes the virtual thread to be pinned to its carrier.
The scheduler does not compensate for that;
- on the other hand, `Object.wait()` is compensated by the scheduler: it may increase the number of carrier threads.

You can read more on the [JEP 425 page](https://openjdk.java.net/jeps/425).

The previous version is not wrong. We are not using virtual threads. Still, I think it would be nice to make the example more "loom-friendly". Or "forward compatible".

So let's write a variation of the alternative using a `Semaphore` instead. [JEP 425](https://openjdk.java.net/jeps/425) gives recommendations on how to mitigate the `synchronized` issue. It does not mention the `Semaphore` class explicitly. I am assuming the recommendations can be extended to it. I must stress though: __this is an assumption__.

First we need to get a `Semaphore` instance:

```java
private final Semaphore semaphore = new Semaphore(0);
```

We initialized the semaphore to zero as we want the semaphore to block in the first `acquire()` invocation. So, right after the counting starts, we call `acquire()` on the semaphore:

```java
counter.startCounting();

semaphore.acquire();
```

Once the counting is over, the counting thread must signal the waiting thread. It does so by calling `release()` on the semaphore:

```java
@Override
public void countingOver() {
  semaphore.release();
}
```

The full test is listed below:

```java
public class AsyncCounterTest implements AsyncCounter.Listener {
  private final Semaphore semaphore = new Semaphore(0);

  @Override
  public void countingOver() {
    semaphore.release();
  }

  @Test
  public void test() throws InterruptedException {
    var startTime = System.currentTimeMillis();
    var counter = new AsyncCounter(this);

    counter.startCounting();

    semaphore.acquire();

    assertEquals(counter.get(), AsyncCounter.COUNT_TO);

    var totalTime = System.currentTimeMillis() - startTime;
    System.out.println("totalTime=" + totalTime);
  }
}
```

Running it results in:

```
totalTime=12
PASSED: test
```

This variation is less verbose than the previous one.

It requires a new object instance and understanding of the `Semaphore` class though. In particular why it must be initialized with a zero value in this case.

Additionally, I must add that, apart from this blog post, I have never used a `Semaphore` before. I suspect this is not quite the use-case the designers of the class had in mind when creating it.

## Test time-out

In both alternatives the testing thread will block until signaled by the counting thread. If for any reason the counting thread fails to signal then your test will block indefinitely.

Therefore I highly recommend that you add a time-out to your test. In other words, you should make sure your test fails if it does not complete after a specific amount of time.

How much time? Ironically, we got ourselves back to our initial problem, didn't we? Meaning that, if we decide on a high time-out value and in the case of a wrong implementation, our test suite will spend a lot of time doing nothing. Use a low time-out value and the test will fail before it can complete its work.

But, unlike the `Thread.sleep` situation, this will only happen when our implementation is wrong.

I am using TestNG. It provides a `timeOut` in its `Test` annotation. If the test does not complete in the specified value in milliseconds then TestNG fails the test.

```java
@Test(timeOut = 100)
public void test() {
  // our test
}
```

I used 100 milliseconds. It is above the 12 milliseconds average I was getting running the tests.

## Conclusion

In this blog post I presented two variations of an alternative to using `Thread.sleep` in your tests. It assumes you can refactor the code you are testing so a listener can be introduced.

All solutions (`Thread.sleep` included) have pros and cons. Which solution you choose depend on your requirements, personal preferences or comfort using one API or the other.

You can find the source code for all of the example in this [GitHub repository](https://github.com/objectos/blog-examples/tree/master/p004-thread-sleep).

_Originally published at the [Objectos Software Blog](https://www.objectos.com.br/blog/alternative-thread-sleep-in-your-java-tests.html) on May 23<sup>th</sup>, 2022._

Follow me on [twitter](https://twitter.com/marcioendo).