Repository: erikbern/ann-benchmarks
Issue: Issue regarding parallelism in FAISS and default behaviour
Root cause: Configuration Error

Issue
----
**FAISS (& possibly others) seem to run on all cpus even though `--parallelism` is set to 1, and I am not using `--local`**

Machine has 80 cores with avx (no avx2 or avx512). This behaviour is affecting the final numbers, especially when trying to run multiple docker containers in parallel, I tried `--parallelism = 48`.

Could this be behaviour caused from rootless docker in my setup? Checking docker inspect "cpusetcpus", it is not set correctly to `cpu_limit`.

Any insight as to what might be happening, or what the default behaviour should be, would be very valuable. 

Behaviour Example :
----
In the following example I am running `python run.py --algorithm faiss-ivf --dataset sift-128-euclidean --parallelism 1`. The other shells are showing:
- `docker ps` : to verify running container
- `docker inspect <container_id>` : to check "CpusetCpus" parameter which should be "1"
- `docker stats <container_id>` : to check resource usage of container
- `htop` : to check resource usage of host machine

First screenshot - Force 1 thread for FAISS: 
- `faiss.omp_set_num_threads(1)` :  FORCE Faiss to only use a single thread. Here indeed only single cpu used for benchmark
-  "cpusetcpus" is not set to "1" even though argument `cpuset_cpus=cpu_limit` in client.contaners.run() under runner.py

<img width="1476" alt="image" src="https://github.com/user-attachments/assets/1754be32-23ed-4a0d-a12a-54f4838d86ea">


---

Second screenshot -  The default behaviour: 
- Here I am not using faiss.omp_set_num_threads() in the fit() method of FaissIVF, which defaults to using the entire 80 cores.
<img width="1316" alt="image" src="https://github.com/user-attachments/assets/ce5865e5-fd05-4561-9437-81d7fba955e5">

Note:
- As it's visible top right in the screenshots, I edited run_docker() to work with rootless docker by changing the socket path.


How results are affected:
-----
dataset = sift-128-euclidean
1) I had run all algorithms with `--parallelism 48` and QPS seems low (could be missing avx2/512) but the numbers especially for FAISS  seemed way off. 
    - when the benchmark was running with `--parallelism 48` I could see all 80 cores being used through htop.
2) I rerun only the faiss-ivf forcing it to use single thread as in the above example and the QPS is improved

This behaviour probably affects the rest of the algorithms as well, which is why the plot isn't smooth and many of them never finished (only one data point visible in the dataset )

<img width="1336" alt="image" src="https://github.com/user-attachments/assets/89b8ad0a-2545-4935-8b23-dfbc4e974214">



Thanks @mageirakos for the very detailed bug report. This is indeed a potential dangerous behavior for running the benchmark. Could you try the steps mentioned under [26) here](https://github.com/containers/podman/blob/main/troubleshooting.md#26-running-containers-with-resource-limits-fails-with-a-permissions-error) to verify that your user has the rights to set the cpu limit?

Thank you @maumueller for the response. Indeed this was a privileges issue with the user, which might extend to all rootless docker installations, but is fixed by giving the rights to set the cpu limit. Feel free to close the issue