Title: `datasets` can't read a Parquet file in Python 3.9.13
Author: loubnabnl (NONE)
State: closed
Created: 2022-11-18T14:44:01Z
Labels: bug

### Describe the bug

I have an error when trying to load this [dataset](https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj) (it's private but I can add you to the bigcode org). `datasets` can't read one of the parquet files in the Java subset

```python
from datasets import load_dataset

ds = load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", split="train", revision="v1.1.a1", use_auth_token=True)
````
```
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```
It seems to be an issue with new Python versions, Because it works in these two environements:
```
- `datasets` version: 2.6.1
- Platform: Linux-5.4.0-131-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```

```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-debian-10.13
- Python version: 3.7.12
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```
But not in this:
```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```

### Steps to reproduce the bug

Load the dataset in python 3.9.13

### Expected behavior

Load the dataset without the pyarrow error.

### Environment info

```
- `datasets` version: 2.6.1
- Platform: Linux-4.19.0-22-cloud-amd64-x86_64-with-glibc2.28
- Python version: 3.9.13
- PyArrow version: 9.0.0
- Pandas version: 1.3.4
```

Comments:
Could you share the full stack trace please ?


Can you also try running this code ? It can be useful to determine if the issue comes from `datasets` or `fsspec` (streaming) or `pyarrow` (parquet reading):
```python
ds = load_dataset("parquet", data_files=a_parquet_file_url, use_auth_token=True)
```

Here's the full trace
```
Traceback (most recent call last):
  File "/home/loubna_huggingface_co/load.py", line 15, in <module>
    ds_all = load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java",use_auth_token=True, split="train", revision="v1.1.a1")
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1742, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 814, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 905, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 1502, in _prepare_split
    for key, table in logging.tqdm(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 67, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 286, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1227, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

when running
```python
ds = load_dataset("parquet", data_files="https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj/blob/v1.1.a1/data/java/data_0000.parquet", use_auth_token=True)
```
I get 401 error, but that's the case for the python subset too which I can load properly
```
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1719, in load_dataset
    builder_instance = load_dataset_builder(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1497, in load_dataset_builder
    dataset_module = dataset_module_factory(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1134, in dataset_module_factory
    return PackagedDatasetModuleFactory(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 707, in get_module
    data_files = DataFilesDict.from_local_or_remote(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/data_files.py", line 795, in from_local_or_remote
    DataFilesList.from_local_or_remote(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/data_files.py", line 764, in from_local_or_remote
    origin_metadata = _get_origin_metadata_locally_or_by_urls(data_files, use_auth_token=use_auth_token)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/data_files.py", line 710, in _get_origin_metadata_locally_or_by_urls
    return thread_map(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 94, in thread_map
    return _executor_map(ThreadPoolExecutor, fn, *iterables, **tqdm_kwargs)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/contrib/concurrent.py", line 76, in _executor_map
    return list(tqdm_class(ex.map(fn, *iterables, **map_args), **kwargs))
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/std.py", line 1183, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/venv/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/opt/conda/envs/venv/lib/python3.9/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/opt/conda/envs/venv/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/opt/conda/envs/venv/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/data_files.py", line 701, in _get_single_origin_metadata_locally_or_by_urls
    return (request_etag(data_file, use_auth_token=use_auth_token),)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/utils/file_utils.py", line 411, in request_etag
    response.raise_for_status()
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/requests/models.py", line 960, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj/blob/v1.1.a1/data/python/data_0000.parquet```

Can you check you used the right token ? You shouldn't get a 401 using your token

I checked it’s the right token, when loading the full dataset I get the error after data extraction so I can access the files. 
```
Downloading and preparing dataset parquet/bigcode--the-stack-dedup-pjj to /home/loubna_huggingface_co/.cache/huggingface/datasets/bigcode___parquet/bigcode--the-stack-dedup-pjj-872ffac7f4bb46ca/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec...
Downloading data files: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 22.38it/s]
Extracting data files: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 49.91it/s]
Traceback (most recent call last):
  File "/home/loubna_huggingface_co/load_ds.py", line 5, in <module>
    ds = load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", use_auth_token=True,split="train", revision="v1.1.a1")
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py", line 1742, in load_dataset
    builder_instance.download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 814, in download_and_prepare
    self._download_and_prepare(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 905, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py", line 1502, in _prepare_split
    for key, table in logging.tqdm(
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py", line 67, in _generate_tables
    parquet_file = pq.ParquetFile(f)
  File "/opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/parquet/__init__.py", line 286, in __init__
    self.reader.open(
  File "pyarrow/_parquet.pyx", line 1227, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```
Could it be that I'm using a wrong url, I just copied it from the address bar

The URL is wrong indeed, the right one is the one with "resolve" (the one you get when clicking on "download")- otherwise you try to download an html page ;)
```
https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj/resolve/v1.1.a1/data/java/data_0000.parquet
```

Ah thanks! So I tried it with the first parquet file and it works, is there a way to know which parquet file was causing the issue since there are a lot of shards?

I think you have to try them all :/

Alternatively you can add a try/catch in `parquet.py` in `datasets` to raise the name of the file that fails at doing `parquet_file = pq.ParquetFile(f)` when you run your initial code
```python
load_dataset("bigcode/the-stack-dedup-pjj", data_dir="data/java", split="train", revision="v1.1.a1", use_auth_token=True)
```
but it will still iterate on all the files until it fails

Ok I will do that

I did find the file, and I get the same error as before 
```
Downloading data files: 100%|███████████████████| 1/1 [00:00<00:00, 8160.12it/s]
Extracting data files: 100%|████████████████████| 1/1 [00:00<00:00, 1447.81it/s]
                            
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
Input In [22], in <cell line: 7>()
      4 data_features = (data["train"].features)
      6 url = "/home/loubna_huggingface_co/.cache/huggingface/datasets/downloads/93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7"
----> 7 data = load_dataset("parquet", 
      8                     data_files=url,
      9                     split="train",
     10                     features=data_features,
     11                     use_auth_token=True)

File /opt/conda/envs/venv/lib/python3.9/site-packages/datasets/load.py:1742, in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, keep_in_memory, save_infos, revision, use_auth_token, task, streaming, **config_kwargs)
   1739 try_from_hf_gcs = path not in _PACKAGED_DATASETS_MODULES
   1741 # Download and prepare data
-> 1742 builder_instance.download_and_prepare(
   1743     download_config=download_config,
   1744     download_mode=download_mode,
   1745     ignore_verifications=ignore_verifications,
   1746     try_from_hf_gcs=try_from_hf_gcs,
   1747     use_auth_token=use_auth_token,
   1748 )
   1750 # Build dataset for splits
   1751 keep_in_memory = (
   1752     keep_in_memory if keep_in_memory is not None else is_small_dataset(builder_instance.info.dataset_size)
   1753 )

File /opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py:814, in DatasetBuilder.download_and_prepare(self, output_dir, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, base_path, use_auth_token, file_format, max_shard_size, storage_options, **download_and_prepare_kwargs)
    808 if not downloaded_from_gcs:
    809     prepare_split_kwargs = {
    810         "file_format": file_format,
    811         "max_shard_size": max_shard_size,
    812         **download_and_prepare_kwargs,
    813     }
--> 814     self._download_and_prepare(
    815         dl_manager=dl_manager,
    816         verify_infos=verify_infos,
    817         **prepare_split_kwargs,
    818         **download_and_prepare_kwargs,
    819     )
    820 # Sync info
    821 self.info.dataset_size = sum(split.num_bytes for split in self.info.splits.values())

File /opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py:905, in DatasetBuilder._download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    901 split_dict.add(split_generator.split_info)
    903 try:
    904     # Prepare split will record examples associated to the split
--> 905     self._prepare_split(split_generator, **prepare_split_kwargs)
    906 except OSError as e:
    907     raise OSError(
    908         "Cannot find data file. "
    909         + (self.manual_download_instructions or "")
    910         + "\nOriginal error:\n"
    911         + str(e)
    912     ) from None

File /opt/conda/envs/venv/lib/python3.9/site-packages/datasets/builder.py:1502, in ArrowBasedBuilder._prepare_split(self, split_generator, file_format, max_shard_size)
   1500 total_num_examples, total_num_bytes = 0, 0
   1501 try:
-> 1502     for key, table in logging.tqdm(
   1503         generator,
   1504         unit=" tables",
   1505         leave=False,
   1506         disable=not logging.is_progress_bar_enabled(),
   1507     ):
   1508         if max_shard_size is not None and writer._num_bytes > max_shard_size:
   1509             num_examples, num_bytes = writer.finalize()

File /opt/conda/envs/venv/lib/python3.9/site-packages/tqdm/std.py:1195, in tqdm.__iter__(self)
   1192 time = self._time
   1194 try:
-> 1195     for obj in iterable:
   1196         yield obj
   1197         # Update and possibly print the progressbar.
   1198         # Note: does not call self.update(1) for speed optimisation.

File /opt/conda/envs/venv/lib/python3.9/site-packages/datasets/packaged_modules/parquet/parquet.py:67, in Parquet._generate_tables(self, files)
     65 for file_idx, file in enumerate(itertools.chain.from_iterable(files)):
     66     with open(file, "rb") as f:
---> 67         parquet_file = pq.ParquetFile(f)
     68         try:
     69             for batch_idx, record_batch in enumerate(
     70                 parquet_file.iter_batches(batch_size=self.config.batch_size, columns=self.config.columns)
     71             ):

File /opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/parquet/__init__.py:286, in ParquetFile.__init__(self, source, metadata, common_metadata, read_dictionary, memory_map, buffer_size, pre_buffer, coerce_int96_timestamp_unit, decryption_properties, thrift_string_size_limit, thrift_container_size_limit)
    280 def __init__(self, source, *, metadata=None, common_metadata=None,
    281              read_dictionary=None, memory_map=False, buffer_size=0,
    282              pre_buffer=False, coerce_int96_timestamp_unit=None,
    283              decryption_properties=None, thrift_string_size_limit=None,
    284              thrift_container_size_limit=None):
    285     self.reader = ParquetReader()
--> 286     self.reader.open(
    287         source, use_memory_map=memory_map,
    288         buffer_size=buffer_size, pre_buffer=pre_buffer,
    289         read_dictionary=read_dictionary, metadata=metadata,
    290         coerce_int96_timestamp_unit=coerce_int96_timestamp_unit,
    291         decryption_properties=decryption_properties,
    292         thrift_string_size_limit=thrift_string_size_limit,
    293         thrift_container_size_limit=thrift_container_size_limit,
    294     )
    295     self.common_metadata = common_metadata
    296     self._nested_paths_by_prefix = self._build_nested_paths()

File /opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/_parquet.pyx:1227, in pyarrow._parquet.ParquetReader.open()

File /opt/conda/envs/venv/lib/python3.9/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

Can you check the JSON file associated to `/home/loubna_huggingface_co/.cache/huggingface/datasets/downloads/93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7` ? In the JSON file we can know from where it was downloaded

You can find it at `/home/loubna_huggingface_co/.cache/huggingface/datasets/downloads/93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7.json`

It's this file `https://huggingface.co/datasets/bigcode/the-stack-dedup-pjj/resolve/f48656daa9f3a3607dacf8b57a65810a6a7a7f73/data/java/data_0022.parquet`  loading it gives the same error

I'm able to load it properly using
```python
ds = load_dataset("parquet", data_files=a_parquet_file_url, use_auth_token=token)
```

My guess is that your download was corrupted. Please delete `93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7` and `93431bc4380de07de8b0ab533666cb5a6120cbe266779e0a63c86bf7717475d7.json` locally and try again

That worked, thanks! But I thought if something went wrong with a download `datasets` creates new cache for all the files, that's not the case? (at some point I even changed dataset versions so it was still using that cache?)

Cool !

> But I thought if something went wrong with a download datasets creates new cache for all the files

We don't perform integrity verifications if we don't know in advance the hash of the file to download.

> at some point I even changed dataset versions so it was still using that cache?

`datasets` caches the files by URL and ETag. If the content of a file changes, then the ETag changes and so it redownloads the file

I see, thank you!


I experience the same error in v 2.12.0. But found out it was due to one column from polars was a categorical dtype (related to the error from #5706. Temporarily resolved it by casting the column to str instead.