Repository: erikbern/ann-benchmarks
Issue: [bug report] about the mem_limit of dockers
Root cause: Algorithmic Error

Hi! Team:

I've been trying to test several algorithms on the benchmark and used the following command:

`python3 run.py --parallelism 31 --dataset gist-960-euclidean --runs 5 --force`

I found that many algorithms failed and returned error 137. Upon checking the log, it showed that some algorithms were allocated less memory compared to others. The machine I used is the same as Erikbern's (i.e., an r6i.16xlarge machine on AWS, with 512GB of memory). Specifically, part of the log reads as follows:

<img width="950" alt="Snipaste_2024-10-16_16-13-16" src="https://github.com/user-attachments/assets/db2ebbaa-6a49-4495-9f35-751b2d1fbad9">

Actually, we expect that each algorithm is limited to about 512 GB / 32 = 16 GB of memory. However, you can see that these algorithms are only allocated about 11 GB, which is far less than our expectation.

**About Fix**

Checking the code, I found that there is a bug when setting the mem_limit at Line 73 in ann_benchmarks/main.py:

`mem_limit = int((psutil.virtual_memory().available - memory_margin) / args.parallelism)`

When using "available," the algorithms in the first batch will get 16 GB of memory, while the algorithms in the latter batches will get less than 16 GB of memory.

So, I think this line should be modified to return the correct memory limit:

`mem_limit = int((psutil.virtual_memory().total - memory_margin) / args.parallelism)`

Or is it my misunderstanding about the setting of mem_limit? Thanks!

**Environment**
a r6i.16xlarge machine on AWS

Thanks @ShawnShawnYou . I'm a bit split here, because that change only works for machines that are completely devoted to running the benchmark. In a setting where you share the machine, it would be wrong to split it up like you propose.

The problem only appears when a container is done and a new one is spawned, does it? 

Thanks @maumueller. 

For your first question, I think that when running the benchmark, there should not be any other processes running on the machine. Since the machine is equipped with 32 cores and we use `--parallelism 31`.

As for the second question, I'm a little confused about exactly how to run the benchmark to obtain results aligned with the ANN Benchmark website. Could you please let me know if you run the benchmark on a machine that is fully dedicated to this task? Also, do you run the algorithms and datasets independently or together? From my understanding, I run the datasets independently, and for each dataset, I run all the algorithms using a command like this:

`python3 run.py --parallelism 31 --dataset gist-960-euclidean --runs 5 --force`

Thus, there must be more than 32 configs of algorithms and containers. Therefore, a new container must be spawned.

And this modification is used to ensure fairness between each algorithm.

@erikbern ran them, so he should confirm, but it is my understanding that he ran it exactly as you have described. (So in particular implementation will have run into the same issue?) 

I don't have an ec2 instance running at the moment, but I think we should move <https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/main.py#L73> to <https://github.com/erikbern/ann-benchmarks/blob/main/ann_benchmarks/main.py#L250> and pass it to `run_worker` so that it is fixed for all containers.