Repository: erikbern/ann-benchmarks
Issue: Plotting different versions of the same algorithm
Root cause: Documentation Deficiency

I would like to compare different versions (e.g. scann with different n_leaves) where the value of a particular single parameter is different. The plot.py file seems to somehow average all the settings for the same algorithm even if I specify multiple run groups (algos.yaml shown below). I was wondering how, for example, I get each run group plot as a separate curve in the qps vs. recall plot?

float:
  angular:
    scann:
      docker-tag: ann-benchmarks-scann
      module: ann_benchmarks.algorithms.scann
      constructor: Scann
      run-groups:
        scann1:
          args: [[500], [100], [0.2], [2], ["dot_product"]]  
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
        scann2:
          args: [[1000], [100], [0.2], [2], ["dot_product"]]  
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
        scann3:
          args: [[2000], [100], [0.2], [2], ["dot_product"]]  
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
        scann4:
          args: [[3000], [100], [0.2], [2], ["dot_product"]]  
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
          # args  n_leaves, n_leaves_to_search, avq_threshold, dims_per_block, dist
          # query-args [leaves_to_search, pre_reorder_num_neighbors]

Hi @Bollegala 

You should be able to do achieve this by replicating the entries, e.g.,

```
    scann:
      docker-tag: ann-benchmarks-scann
      module: ann_benchmarks.algorithms.scann
      constructor: Scann
      run-groups:
        scann1:
          args: [[600], [.nan], [2], ["squared_l2"]]
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
        scann2:
          args: [[2000], [.nan], [4], ["squared_l2"]]
          query-args: [[[10, 100], [15, 140], [25, 160], [35, 190], [40, 200], [45, 220], [50, 240], [60, 250], [70, 300], [80, 400], [100, 500], [120, 600], [150, 800], [200, 900]]]
        scann3:
          args: [[100], [.nan], [4], ["squared_l2"]]
          query-args: [[[2, 20], [3, 20], [3, 30], [4, 30], [5, 40], [8, 80]]]
```

could become 

```
scann1:
      docker-tag: ann-benchmarks-scann
      module: ann_benchmarks.algorithms.scann
      constructor: Scann
      run-groups:
        scann1:
          args: [[600], [.nan], [2], ["squared_l2"]]
          query-args: [[[4, 40], [3, 30], [6, 60], [8, 74], [9, 78], [10, 82], [11, 85], [13, 100], [16, 120], [20, 140], [30, 180], [35, 240], [50, 360]]]
scann2:
      docker-tag: ann-benchmarks-scann
      module: ann_benchmarks.algorithms.scann
      constructor: Scann
      run-groups:
        scann2:
          args: [[2000], [.nan], [4], ["squared_l2"]]
          query-args: [[[10, 100], [15, 140], [25, 160], [35, 190], [40, 200], [45, 220], [50, 240], [60, 250], [70, 300], [80, 400], [100, 500], [120, 600], [150, 800], [200, 900]]]
```

Then you just need to run `scann1` and `scann2` individually. 
       

Great! Thank you very much. Works like a charm. Much appreciate your quick response.