Repository: erikbern/ann-benchmarks
Issue: Can you give an example command to run locally with 2 algorithms


I just do it via bash, e.g., `for algo in annoy faiss-ivf; do python run.py --algorithm $algo --dataset glove-100-angular; done`

Do you need to specify --local?

Sorry, missed that. Yes, it has to look like `for algo in annoy faiss-ivf; do python run.py --local --algorithm $algo --dataset glove-100-angular; done`

Thank you, it helped. I got Exception: Nothing to run. Do you also need to specify --definitions?

It uses the `algos.yaml` file by default, so the algorithms you want to test ought to have parameter settings in there. It would help if you could provide a MWE. 

This is the command I use: `python run.py --local --algorithm  hnsw --dataset sift-128-euclidean'
BTW, what is query-args values below in the algos.yaml file?

  hnsw(faiss):
     docker-tag: ann-benchmarks-faiss
     module: ann_benchmarks.algorithms.faiss_hnsw
     constructor: FaissHNSW
     base-args: ["@metric"]
     run-groups:
       M-4:
         arg-groups:
           - {"M": 4,  "efConstruction": 500}
         query-args: [[10, 20, 40, 80, 120, 200, 400, 600, 800]]

The implementation is `hnsw(faiss)`, so it's supposed to be `python run.py --local --algorithm hnsw\(faiss\) --dataset sift-128-euclidean`. 

Most implementations have a computationally expensive index building step (the `train` method), which can then be searched in many different ways. Here, it means that it constructs one index, and then uses 9 different parameters for searching for nearest neighbors.

for using hnsw(faiss), are query-args the efSearch values? For calculating queries/s, do you take 1 divided by the minimum of 9 query times in one index?

Yes, those are the efSearch values, see https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/algorithms/faiss_hnsw.py#L30.

I don't exactly follow on your QPS discussion. One instantiation of the algorithm with a fixed set of parameters gives exactly one measurement (say, the nearest neighbors returned, the time it took to answer these queries, ...). If T is the time it took to answer all queries in nanoseconds, we report the value #queries * 1e9/T. If there are 9 different efSearch values, there are going to be 9 different QPS values. The plots only show the pareto frontier of 'dominating' measurements, but you can find scatterplots with _all_ measurements on the website,e.g.,  http://ann-benchmarks.com/glove-100-angular_10_angular.html (bottom).

> Sorry, missed that. Yes, it has to look like `for algo in annoy faiss-ivf; do python run.py --local --algorithm $algo --dataset glove-100-angular; done`

@maumueller Could you please kindly explain when to add / why not to add the parameter `--batch`?

It's a different query mode: Without batch (the default mode), queries are presented one by one and the docker container is constrained to use a single CPU/thread. With `--batch`, all queries are presented at once, the docker container uses all resources available on the host, and result conversion is done in a separate method.

> It's a different query mode: Without batch (the default mode), queries are presented one by one and the docker container is constrained to use a single CPU/thread. With `--batch`, all queries are presented at once, the docker container uses all resources available on the host, and result conversion is done in a separate method.

How about batch mode using single CPU? Does that make any sense?