Repository: erikbern/ann-benchmarks
Issue: Is there a manual for providing a new dataset for ann-benchmarks?
Root cause: Documentation Deficiency

I have a new dataset which has 1 billion data points, and I want it to be used by ann-benchmarks, what do I need to do?

I'm not sure if we're "ready" for 1B data points. Which dataset is it?

It's an internal data points which isn't ready for open source by now. Basically it's 100 dims, signed int8.
Change the question, what if 1M data points? I'd like to sample a subset of 1B data points.

I'm a bit hesitant to open source proprietary data – I'd like for it to be some existing data set that's already public.

I was also in need of running it on a custom dataset and read through the code to make the changes. 

As far as I can see. There are 4 changes:
1. Input hdf5 file of the following format saved in a new ann-benchmarks/data directory. Train, test, neighbors, and distances are pandas data frames. Generating these data frames is a task in itself.
```
f = h5py.File('custom-dataset.hdf5', 'w')
f.attrs['distance'] = distance_metic
f.create_dataset('train', (len(train), len(train[0])), dtype=train.dtype)[:] = train
f.create_dataset('test', (len(test), len(test[0])), dtype=test.dtype)[:] = test
f.create_dataset('neighbors', (len(neighbors), len(neighbors[0])), dtype=neighbors.dtype)[:] = neighbors
f.create_dataset('distances', (len(distances), len(distances[0])), dtype=distances.dtype)[:] = distances
```

2. Update ann-benchmarks/ann-benchmarks/main.py to take in the new local_file argument and pass it to get_dataset
```
    parser.add_argument(
        '--local_file',
        type=bool,
        help='Use local hdf5 dataset',
        default=False)
        
        ...
        
    dataset = get_dataset(args.dataset, args.local_file)

```

3. Update get_dataset function in ann-benchmarks/ann-benchmarks/datasets.py to add a new argument local 
```
def get_dataset(which, local=False):
    hdf5_fn = get_dataset_fn(which)
    try:
        url = 'http://ann-benchmarks.com/%s.hdf5' % which
        download(url, hdf5_fn)
    except:
        if local:
            pass
            
        else:
            print("Cannot download %s" % url)
            if which in DATASETS:
                print("Creating dataset locally")
                DATASETS[which](hdf5_fn)

    hdf5_f = h5py.File(hdf5_fn, 'r')
    return hdf5_f
```

4. add the following line to the end of the ann-benchmarks/ann-benchmarks/datasets.py:
```
DATASETS['custom-dataset'] = lambda out_fn: random_float(out_fn, n_dims=num_dimensions, n_samples=num_neighbors_rows, centers=num_neighbors_columns, distance=distance_metic)
```

Do these seem about right @erikbern ?

It's a bit easier than that: 

1. Add a new function that generates the data in the dataset however you see fit, e.g., as in https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/datasets.py#L247-L254. The last line of the function is probably the call to `write_output` that takes care of generating the hdf5 file.

2. Add the dataset name/function to create it to the DATASETS dictionary, the value is the function that you've defined in 1., e.g., https://github.com/erikbern/ann-benchmarks/blob/master/ann_benchmarks/datasets.py#L424. (This is your step 4, which is fine.)

You don't have to concern yourself with updating any of the download code. It will automatically fall back to creating the local file if it doesn't find the hdf5 (which it won't for a custom dataset). Hope that helps.


Perfect! It works and is much simpler. Thanks!

I wrote up a medium post with the detailed code if anyone needs this [Link](https://medium.com/towards-artificial-intelligence/how-to-choose-the-best-nearest-neighbors-algorithm-8d75d42b16ab?sk=889bc0006f5ff773e3a30fa283d91ee7).

I'm not sure how you guys feel about Medium but the link I attached here is the medium's friend link which allows for viewing the story without the paywall. Hopefully, that is ok with you!

nice, that's pretty cool!