Repository: erikbern/ann-benchmarks
Issue: 10X performance difference between multi-thread and single thread tests
Root cause: Resource Mishandling

Good day!

I have measured the difference in performance when switching the use_threads parameter from True to False on hnsw (nmslib). As it turns out, the difference is really huge, ~ 10X on 1M SIFT for low recalls (see enclosed plots for a 6-core intel cpu), and likely is caused by the python-part overhead of the tester scripts.

Multi-thread (default):
![sift_mt](https://user-images.githubusercontent.com/16265186/34437153-f7afe4fa-ecac-11e7-8f1c-cf382ef9a4b2.png)
Single-thread (use_threads=False):
![sift_st](https://user-images.githubusercontent.com/16265186/34437154-f7cada12-ecac-11e7-9109-a8377da007d1.png)

This makes the current results questionable, especially when compared to FALCONN (which seems to be the only algorithm which takes advantage of not using multi-thread tests). Multithread tests also significantly change the shape of the recall-QPS curve (flat vs rapidly decreasing QPS at low recall).

I suggest using the single thread version for all algorithms. Otherwise, for the case of relatively high QPS (~>2 kps) we just measure the performance of the tester script.

Here are the plots for the set of the fastest algorithms on SIFT 1M in two cases:
1. All algorithms are processed in the single thread-regime (by commenting all other options in runner.py)
![sift_st](https://user-images.githubusercontent.com/16265186/34526989-10183c16-f0b5-11e7-93fe-ec5481f97922.png)
2. Default, nothing changed - most of the algorithms are processed in the multi-thread regime.
![sift_mt_old](https://user-images.githubusercontent.com/16265186/34526988-0ff40864-f0b5-11e7-8a79-a94a983415b2.png)

As one can see, the absolute difference in speed is significant and at least three of algorithms have almost the same performance at low recalls in the multi threaded test, which is not adequate to reality.



Hi @yurymalkov – sorry I forgot about this one. This is really weird, not sure what's going on. I'll take a look. @maumueller any idea?

I have a theory what's going on here. Is nmslib/hnsw using threads under the hood, even if queried with a single thread?

The time consumption is measured inside each thread. So if you query a library from a single thread, but the library uses multiple threads under the hood, it will benefit from speedups and make the QPS number great. However, if you saturate all CPU cores by querying from *multiple* threads at the same time, its QPS number will go down. For libraries that don't use threads under the hood, the QPS will stay roughly the same. So I think this is actually the expected behavior (just a bit counterintuitive)

Btw one thing we could do to eliminate the Python overhead is to swap out the thread pool for a process pool...however that could cause issues for any algorithm that mutates the index during queries (the page cache would then force the index to be duplicated for each process). Not sure if that's a problem in practice.

@erikbern nmslib/hnsw search does **not** uses multi-threading under the hood, so this can be ruled out. It does use multi-threading for index construction though.

Are you sure it's not using eg a multithreaded linear algebra package or something similar?

It's possible that it's the Python overhead then, like GIL locking or something similar. Have you seen the macro `Py_BEGIN_ALLOW_THREADS`? I'm using it inside Annoy: https://github.com/spotify/annoy/blob/master/src/annoymodule.cc#L256 – it doesn't seem to be used in nmslib

@erikbern 
Yes, I am sure. It does not use any packages. C++ tests are also fine, I got 200-300k QPS on low recalls using 1M SIFT on a 4-core 6700k - way more than in current benchmark, even if divided by the number of cores.
The GIL macros is used in nmslib. We discussed the GIL in https://github.com/erikbern/ann-benchmarks/pull/28


Probably a way to solve it is measuring search quality and time in different runs. I.e. measuring the time with minimal overhead.
Another option is using batch queries, allowing the algorithm to use multi-threading without the evaluation part.
Also one can set a single-cpu affinity of the search thread and run tests in single thread. This would prevent using multi-threading under the hood.

How do you feel about swapping out the thread pool for a process pool? It's a one line change and will support single/threaded libraries alike

I am not sure about that. I thought that multiproccessing adds significant overhead compared to GIL-aware lib calls. Anyway, we can try and see if the results on SIFT 1M are reasonable.
Also. What libraries now do not support multi-threading? FALCONN does support it now  https://github.com/FALCONN-LIB/FALCONN/issues/6


Also we can add a dummy algo which would just return random items to measure the overhead.

multiprocessing should add no CPU overhead since GIL doesn't matter any more. The only issue is since it's using fork(), the memory usage might go up (potentially a lot, if the index mutates)

I'm skeptical about batch queries, would rather not let algorithms use that.

I can switch back to single threads if you can give me some pointers about how to implement CPU affinity? Let me know. Otherwise some algorithms (FALCONN) will have an unfair advantage. I also think some algos like kgraph might use linear algebra packages that use multithreading under the hood (eigen?) not sure

Sure, I'll look up for solutions on affinity in a few days. 
Probably using https://github.com/giampaolo/psutil will do, but I am not sure.

@yurymalkov I noticed nmslib doesn't use the Py_BEGIN_ALLOW_THREADS macro. Why not?

If you enable that macro (or maybe it's called something else when using pybind11) then you nmslib should be much faster. I honestly doubt the Python overhead is substantial.

I'd rather just have nmslib enable that macro than trying to change anything about ann-benchmarks honestly. That will benefit *all* users of nmslib, not just ann-benchmarks

@erikbern I see your point, but, yes, nmslib is using pybind11 c++ objects for GIL https://github.com/searchivarius/nmslib/blob/master/python_bindings/nmslib.cc#L112 . Anyway, I will ask @searchivarius if he checked the correctness of the new bindings.

There is another marker that points out that the problem is in the test - flattening of the QPS curve at low recalls. It didn't happen even when nmslib didn't release GIL https://github.com/erikbern/ann-benchmarks/pull/28#issuecomment-230346768. It looks like now it measures some internal time of the testing system.

Also, I suspect hyperthreading (or virtual cores on AWS) is also contributing to the problem of overall slowdown, at least 2X can attributed to that, which is strange.



@erikbern @yurymalkov the new bindings seem to release the GIL lock. However, there might be some subtle bug that is not obvious on a first look. I will double check. 

@erikbern @searchivarius I have tested the latest nmslib python bindings in the multithreaded regime and checked that the GIL is released. For high accuracy setting I get 5.3X increase in total QPS compared to a single thread version on a 6-core CPU with HT. 
For low accuracy setting the speedup is much less prominent, probably due to python thread pool (the nmslib python batch mode is working ok).

Sounds good. When was this released? I just ran a benchmark for cosine distance and I'm running one for Euclidean right now (see #50 for preliminary stats) but let me know if you want me to re-run for nmslib

@erikbern It seems my last comment was misleading. 
The "new" bindings are the current ones. There were also old bindings, which were known to parallelize well. I thought that it might be that the updated bindings have caused the problems, but it is not the case. The GIL release is working fine.

ok weird. i'll keep the benchmarks for now then. nmslib is still like 3x faster than anything else :)

Ok) I believe all algorithms that are tested in the multi-thread regime are also affected by this. The exception is FALCONN which runs in a single thread)
In a few days, I think I'll test a dummy search algo (in python, just return a random chuck of dataset) to test the maximum QPS the framework can measure. 

Hi @erikbern a somewhat unrelated question. Is the new version of Annoy using 2-means to build trees? The previous stable one (I checked a month ago) was still based on random projections. The previous ann-benchmark apparently used a 2-means tree, right?

The version in ann-benchmarks is the latest version of git which is also the latest version in PyPI: 1.11.1 (with AVX optimizations added)

Hi all, that‘s indeed a very interesting observation. I noticed that behavior a longer time ago as well. (see below).  

Note that ann-benchmarks already has a „batched mode“. This mode is for example necessary to test the FAISS-GPU instance. The default implementation (https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/algorithms/base.py#L11) just calls the usual query for every data point in the query set. We don‘t use a ThreadPool there, because the implementation should take measures to parallize queries.

However, note that time is measured differently between batched and non-batched queries: in batch mode, the start and end time is measured once for answering all the queries, in non-batched mode each individual query is measured and all timings are summed up together. This behavior was described and discussed here https://github.com/erikbern/ann-benchmarks/pull/37.

From my measurements, I always found that the ThreadPool adds too much overhead in the lower acc regime. The behavior described here seems particular unusual, though. 

Let me see what happens if I swap out the thread pool for a process pool instead. Might also help with L1/L2 thrashing.