Repository: erikbern/ann-benchmarks
Issue: Unable to download mnist-784-euclidean
Root cause: External Dependency Breakage

I can download fashion-mnist-784-euclidean, sift-128-euclidean and gist-960-euclidean. 
But when I try to download mnist-784-euclidean I get an error.
```
python3 run.py --dataset mnist-784-euclidean

downloading https://ann-benchmarks.com/mnist-784-euclidean.hdf5 -> data/mnist-784-euclidean.hdf5...
Cannot download https://ann-benchmarks.com/mnist-784-euclidean.hdf5
Creating dataset locally
downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz -> mnist-train.gz...
Traceback (most recent call last):
  File "/home/mdcallag/git/ann_vuvova/ann_benchmarks/datasets.py", line 55, in get_dataset
    download(dataset_url, hdf5_filename)
  File "/home/mdcallag/git/ann_vuvova/ann_benchmarks/datasets.py", line 21, in download
    urlretrieve(source_url, destination_path)
  File "/usr/lib/python3.10/urllib/request.py", line 241, in urlretrieve
    with contextlib.closing(urlopen(url, data)) as fp:
  File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.10/urllib/request.py", line 525, in open
    response = meth(req, response)
  File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.10/urllib/request.py", line 563, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
```

I also get an error if I try to fetch that file via wget
```
$ wget  http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz

--2024-12-20 23:52:05--  http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Resolving yann.lecun.com (yann.lecun.com)... 216.165.22.6
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:80... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz [following]
--2024-12-20 23:52:05--  https://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Connecting to yann.lecun.com (yann.lecun.com)|216.165.22.6|:443... connected.
HTTP request sent, awaiting response... 403 Forbidden
2024-12-20 23:52:05 ERROR 403: Forbidden.
```

weird. I just migrated all the datasets. Let me check what happened.

I'm able to download it from my laptop... maybe it was just a glitch? Closing but let me know if it comes back!

From the error message the file is: http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
```
Traceback (most recent call last):
  File "/home/mdcallag/git/ann-benchmarks/run.py", line 7, in <module>
    main()
  File "/home/mdcallag/git/ann-benchmarks/ann_benchmarks/main.py", line 312, in main
    dataset, dimension = get_dataset(args.dataset)
  File "/home/mdcallag/git/ann-benchmarks/ann_benchmarks/datasets.py", line 60, in get_dataset
    DATASETS[dataset_name](hdf5_filename)
  File "/home/mdcallag/git/ann-benchmarks/ann_benchmarks/datasets.py", line 297, in mnist
    download("http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz", "mnist-train.gz")  # noqa
  File "/home/mdcallag/git/ann-benchmarks/ann_benchmarks/datasets.py", line 21, in download
    urlretrieve(source_url, destination_path)
```

When I visit http://yann.lecun.com/exdb/mnist and then click on any of the 4 files, including train-images-idx3-ubyte.gz, I get
```
Forbidden
You don't have permission to access this resource.
```


The error continues to occur for me using a home server and a server in the Hetzner cloud with this command line:
```
python3 run.py  --dataset mnist-784-euclidean
```

Oh I see so it's the original data that's broken. Out of curiosity, why do you need it?

I am trying to reproduce the results for MariaDB that were shared here:
https://mariadb.com/resources/blog/how-fast-is-mariadb-vector

Those results used fashion-mnist-784-euclidean, mnist-784-euclidean, sift-128-euclidean and gist-960-euclidean so i assume I need to run with --dataset $foo where $foo is each of fashion-mnist-784-euclidean, mnist-784-euclidean, sift-128-euclidean and gist-960-euclidean to download them.

I also don't have great download speed so I prefer to download just once.

is the link to the prebuilt index in the readme sufficient for you? http://ann-benchmarks.com/mnist-784-euclidean.hdf5

Ok, I'm seeing now that the run script is falling back to building the dataset locally. Is that still happening for you? Let me try to reproduce.

Think I figured it out!

Downloads from http://yann.lecun.com/exdb/mnist are throttled, or in my case blocked.
I assume I have to download the file once to get started, but I am new to ann-benchmarks.

This claims to be a mirror: https://github.com/fgnt/mnist


Yeah, we should fix the problem creating the original data. But the problem here was also that downloading the prepared dataset was failing, and it would fall back to creating the original data. I fixed the latter at least.