Repository: erikbern/ann-benchmarks
Issue: Support Multi-threaded Benchmarks for Production-Realistic Performance Metrics
Root cause: Unimplemented Feature Gap

Currently, `ANN-Benchmarks` enforces single-CPU execution during experimentation, disabling multi-threading capabilities at the hardware level (AWS single-cpu mode) so that even libraries supporting multi-core CPU/threading can't use it. While batch mode exists, it doesn't address the potential performance benefits that multi-threading could offer.

There are ANN libraries supporting multi-core/multi-threading processing to improve performance. However, the single-threaded benchmark is unlikely to project their true potential in the production environment. This is especially true since most end users are likely to run ANN on multi-core CPUs, and there is no reason they should choose the wrong implementation based on single CPU benchmarks. Running and providing both single- and multi-threaded benchmarks will allow users to view more realistic benchmarks, matching production setup.

We request to consider adding multi-threading benchmarks alongside existing single-threaded tests.
- Optionally, Add a new flag (e.g., --threading) to enable multi-threading evaluation
- Create separate performance plots for single-threaded vs multi-threaded execution

We're happy to contribute to the implementation if needed and have already made some changes for local benchmarking to test ANN in multi-threaded mode.

Looking forward to your thoughts on this.

@mesibo thanks for the initiative, I agree that reporting on non-single-thread results is interesting. 

Could you detail what you are missing with the present `--batch` mode? The [default implementation](https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/algorithms/base/module.py#L48-L60) is of course not very interesting, but each implementation can specify its own behaviour. 

Actually, the default `ann-benchmarks` implementation is very well thought out and already supports multi-threading on local systems (e.g., `HNSWLib` module uses multiple threads; as a side note, the `set_num_threads()` line in `fit()` has possibly no effect). So, I don’t think major changes are required. The main point is to publish benchmarks for multi-threaded scenarios rather than restricting them to single-CPU. Given the changing landscape, this would be more meaningful and aligned with production use cases.

> Could you detail what you are missing with the present --batch mode? 

You are correct that the `--batch` mode could work—we’re still exploring it (FAISS GPU implements batch). However, it may not be ideal in all cases, let me explain our scenario:

Unlike most synchronous algorithms, our algorithm is asynchronous and analyzes patterns in vectors before indexing them. It uses multi-core systems for this pre-processing and hence has to be asynchronous. However, this poses no issue during indexing as it’s a one-time operation, and we only need to wait in the `fit()` function until the indexing is complete. Hence, no changes are required on the indexing side.

However, querying is also asynchronous, and the algorithm can handle multiple query vectors in parallel. This is where integrating it into the `ann-benchmarks` framework becomes slightly challenging. Running `single_query()` benchmarks doesn’t work because the query quickly returns without processing (due to its asynchronous nature). Also, unlike the `fit()` case, `single_query()` can’t wait for the query to finish, as it would negate the benefits of parallel processing.

While `--batch` mode could work, it introduces unnecessary overhead by maintaining a large queue, even though the algorithm typically processes 8–10 vectors at a time, leading to higher memory usage and additional overhead.

We are still investigating potential solutions, but any suggestions or guidance you can provide would be greatly appreciated.

Thanks for making such a strong case, @mesibo.  I'd be very interested to see the work that you are doing and how the current framework stands in your way to achieve the best performance for your implementation. 

I'm not entirely sure I understand what you mean with 

> While --batch mode could work, it introduces unnecessary overhead by maintaining a large queue, even though the algorithm typically processes 8–10 vectors at a time, leading to higher memory usage and additional overhead.

Batch-mode will bypass the `single_query` method and instead call https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/runner.py#L86-L121. This will pass the whole set of query vectors at once to the `batch_query` method that you are supposed to support with your implementation. 

The only part where we are using a multiprocessing queue is our implementation of `--parallelism`, which allows to carry out independent single-threaded experiments in parallel. 


Batch-mode should work, the concern was memory, but as we’ve learned, `ann_benchmarks` loads everything in memory anyway, so it shouldn’t make a difference. We should have our multithreaded integration ready by next week and will update you then. Thanks for the clarification!

We have implemented the ann-benchmarks module for our implementation and are currently testing it. As suggested by you, we are using --batch mode for parallel indexing and queries. 

Our main challenge now is with result presentation: When running benchmarks using --batch, we notice that results are shown separately as:

- Benchmarks for Single Queries
- Benchmarks for Batched Queries

Which will make the comparison ineffective. We can't use the single query approach as our implementation runs multiple queries in parallel asynchronously. The single query implementation in `ann-benchmarks` is expecting results synchronously:

`candidates = algo.query(v, count)`

It would be great if the results were combined into a unified view. This would make it easier to compare performance across different implementations.

Your input would be greatly appreciated.