sunny2309/githug-issues huggingface dataset:
Errror when saving to disk a dataset of images

### Describe the bug

Hello!

I have an issue when I try to save on disk my dataset of images. The error I get is:

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1442, in save_to_disk
    for job_id, done, content in Dataset._save_to_disk_single(**kwargs):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 1473, in _save_to_disk_single
    writer.write_table(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/arrow_writer.py", line 570, in write_table
    pa_table = embed_table_storage(pa_table)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2268, in embed_table_storage
    arrays = [
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2269, in <listcomp>
    embed_array_storage(table[name], feature) if require_storage_embed(feature) else table[name]
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in wrapper
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 1817, in <listcomp>
    return pa.chunked_array([func(chunk, *args, **kwargs) for chunk in array.chunks])
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/table.py", line 2142, in embed_array_storage
    return feature.embed_storage(array)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/features/image.py", line 269, in embed_storage
    storage = pa.StructArray.from_arrays([bytes_array, path_array], ["bytes", "path"], mask=bytes_array.is_null())
  File "pyarrow/array.pxi", line 2766, in pyarrow.lib.StructArray.from_arrays
  File "pyarrow/array.pxi", line 2961, in pyarrow.lib.c_mask_inverted_from_obj
TypeError: Mask must be a pyarrow.Array of type boolean
```

My dataset is around 50K images, is this error might be due to a bad image?

Thanks for the help.

### Steps to reproduce the bug

```python
from datasets import load_dataset
dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset["train"].save_to_disk("./myds", num_shards=40)
```

### Expected behavior

Having my dataset properly saved to disk.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
Return Numpy Array (fixed length) Mode, in __get_item__, Instead of List

### Feature request

There are old known issues, but they can be easily forgettable problems in multiprocessing with pytorch-dataloader:

Too high usage of RAM or shared-memory in pytorch when we set num workers > 1 and returning type of dataset or dataloader is "List" or "Dict".
https://github.com/pytorch/pytorch/issues/13246 

With huggingface datasets, unfortunately, the default return type is the list, so the problem is raised too often if we do not set anything for the issue.


However, this issue can be released when the returning output is fixed in length.
Therefore, I request the mode, returning outputs with fixed length (e.g. numpy array) rather than list.

The design would be good when we load datasets as
```python
load_dataset(..., with_return_as_fixed_tensor=True)
```

### Motivation

The general solution for this issue is already in the comments:  https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662 

: Numpy or Pandas seems not to have problems, while both have the string type.
(I'm not sure that the sequence of huggingface datasets can solve this problem as well)


### Your contribution

 I'll read it ! thanks
Fix xnumpy_load for .npz files

PR:
- #5626 

implemented support for streaming `.npy` files by using `numpy.load`.

However, it introduced a bug when used with `.npz` files, within a context manager:
```
ValueError: seek of closed file
```
or in streaming mode:
```
ValueError: I/O operation on closed file.
```

This PR fixes the bug and tests for both `.npy` and `.npz` files.

Fix #5711.
ArrowNotImplementedError when loading dataset from the hub

### Describe the bug

Hello,

I have created a dataset by using the image loader. Once the dataset is created I try to download it and I get the error:
```
Traceback (most recent call last):
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1860, in _prepare_split_single
    for _, table in generator:
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/packaged_modules/parquet/parquet.py", line 69, in _generate_tables
    for batch_idx, record_batch in enumerate(
  File "pyarrow/_parquet.pyx", line 1323, in iter_batches
  File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Nested data conversions not implemented for chunked array outputs

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/load.py", line 1791, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 891, in download_and_prepare
    self._download_and_prepare(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 986, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1748, in _prepare_split
    for job_id, done, content in self._prepare_split_single(
  File "/home/jplu/miniconda3/envs/image-xp/lib/python3.10/site-packages/datasets/builder.py", line 1893, in _prepare_split_single
    raise DatasetGenerationError("An error occurred while generating the dataset") from e
datasets.builder.DatasetGenerationError: An error occurred while generating the dataset
```

### Steps to reproduce the bug

Create the dataset and push it to the hub:
```python
from datasets import load_dataset

dataset = load_dataset("imagefolder", data_dir="/path/to/dataset")
dataset.push_to_hub("org/dataset-name", private=True, max_shard_size="1GB")
```

Then use it:
```python
from datasets import load_dataset

dataset = load_dataset("org/dataset-name")
```

### Expected behavior

To properly download and use the pushed dataset.

Something else to note is that I specified to have shards of 1GB max, but at the end, for the train set, it is an almost 7GB single file that is pushed.

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
OSError: Memory mapping file failed: Cannot allocate memory

### Describe the bug

Hello, I have a series of datasets each of 5 GB, 600 datasets in total. So together this makes 3TB. 

When I trying to load all the 600 datasets into memory, I get the above error message. 

Is this normal because I'm hitting the max size of memory mapping of the OS? 


Thank you

```terminal
0_21/cache-e9c42499f65b1881.arrow
load_hf_datasets_from_disk:  82%|████████████████████████████████████████████████████████████████████████████████████████████████████▍                     | 494/600 [07:26<01:35,  1.11it/s]
Traceback (most recent call last):
  File "example_load_genkalm_dataset.py", line 35, in <module>
    multi_ds.post_process(max_node_num=args.max_node_num,max_seq_length=args.max_seq_length,delay=args.delay)
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 142, in post_process
    genkalm_dataset = GenKaLM_Dataset.from_hf_dataset(path_or_name=ds_path, max_seq_length=self.max_seq_length,
  File "/home/geng/GenKaLM/src/dataloader/dataset.py", line 47, in from_hf_dataset
    hf_ds = load_from_disk(path_or_name)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/load.py", line 1848, in load_from_disk
    return Dataset.load_from_disk(dataset_path, keep_in_memory=keep_in_memory, storage_options=storage_options)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1549, in load_from_disk
    arrow_table = concat_tables(
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1805, in concat_tables
    tables = list(tables)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1550, in <genexpr>
    table_cls.from_file(Path(dataset_path, data_file["filename"]).as_posix())
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 1065, in from_file
    table = _memory_mapped_arrow_table_from_file(filename)
  File "/home/geng/.conda/envs/genkalm/lib/python3.8/site-packages/datasets/table.py", line 50, in _memory_mapped_arrow_table_from_file
    memory_mapped_stream = pa.memory_map(filename)
  File "pyarrow/io.pxi", line 950, in pyarrow.lib.memory_map
  File "pyarrow/io.pxi", line 911, in pyarrow.lib.MemoryMappedFile._open
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Memory mapping file failed: Cannot allocate memory
```








### Steps to reproduce the bug

Sorry I can not provide a reproducible code as the data is stored on my server and it's too large to share.

### Expected behavior

I expect the 3TB of data can be fully mapped to memory

### Environment info


- `datasets` version: 2.9.0
- Platform: Linux-4.15.0-204-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.6
- PyArrow version: 11.0.0
- Pandas version: 1.0.1
Manually dataset info made not taken into account

### Describe the bug

Hello,

I'm manually building an image dataset with the `from_dict` approach. I also build the features with the `cast_features` methods. Once the dataset is created I push it on the hub, and a default `dataset_infos.json` file seems to have been automatically added to the repo in same time. Hence I update it manually with all the missing info, but when I download the dataset the info are never updated.

Former `dataset_infos.json` file:
```
{"default": {
    "description": "",
    "citation": "",
    "homepage": "",
    "license": "",
    "features": {
        "image": {
            "_type": "Image"
        },
        "labels": {
            "names": [
                "Fake",
                "Real"
            ],
            "_type": "ClassLabel"
        }
    },
    "splits": {
        "validation": {
            "name": "validation",
            "num_bytes": 901010094.0,
            "num_examples": 3200,
            "dataset_name": null
        },
        "train": {
            "name": "train",
            "num_bytes": 901010094.0,
            "num_examples": 3200,
            "dataset_name": null
        }
    },
    "download_size": 1802008414,
    "dataset_size": 1802020188.0,
    "size_in_bytes": 3604028602.0
}}
```

After I update it manually it looks like:
```
{
   "bstrai--deepfake-detection":{
      "description":"",
      "citation":"",
      "homepage":"",
      "license":"",
      "features":{
         "image":{
            "decode":true,
            "id":null,
            "_type":"Image"
         },
         "labels":{
            "num_classes":2,
            "names":[
               "Fake",
               "Real"
            ],
            "id":null,
            "_type":"ClassLabel"
         }
      },
      "supervised_keys":{
         "input":"image",
         "output":"labels"
      },
      "task_templates":[
         {
            "task":"image-classification",
            "image_column":"image",
            "label_column":"labels"
         }
      ],
      "config_name":null,
      "splits":{
         "validation":{
            "name":"validation",
            "num_bytes":36627822,
            "num_examples":123,
            "dataset_name":"deepfake-detection"
         },
         "train":{
            "name":"train",
            "num_bytes":901023694,
            "num_examples":3200,
            "dataset_name":"deepfake-detection"
         }
      },
      "download_checksums":null,
      "download_size":937562209,
      "dataset_size":937651516,
      "size_in_bytes":1875213725
   }
}
```

Anything I should do to have the new infos in the `dataset_infos.json` to be taken into account? Or it is not possible yet?

Thanks!

### Steps to reproduce the bug

-

### Expected behavior

-

### Environment info

- `datasets` version: 2.11.0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.10
- Huggingface_hub version: 0.13.3
- PyArrow version: 11.0.0
- Pandas version: 2.0.0
Dataset sizes are in MiB instead of MB in dataset cards

As @severo reported in an internal discussion (https://github.com/huggingface/moon-landing/issues/5929):

Now we show the dataset size:
- from the dataset card (in the side column)
- from the datasets-server (in the viewer)

But, even if the size is the same, we see a mismatch because the viewer shows MB, while the info from the README generally shows MiB (even if it's written MB -> https://huggingface.co/datasets/blimp/blob/main/README.md?code=true#L1932)

<img width="664" alt="Capture d’écran 2023-04-04 à 10 16 01" src="https://user-images.githubusercontent.com/1676121/229730887-0bd8fa6e-9462-46c6-bd4e-4d2c5784cabb.png">

TODO: Values to be fixed in: `Size of downloaded dataset files:`, `Size of the generated dataset:` and `Total amount of disk used:`
- [x] Bulk edit on the Hub to fix this in all canonical datasets
- [x] Bulk PR on the Hub to fix ancient canonical datasets that were moved to organizations
Support categorical data types for Parquet

### Feature request

Huggingface datasets does not seem to support categorical / dictionary data types for Parquet as of now. There seems to be a `TODO` in the code for this feature but no implementation yet. Below you can find sample code to reproduce the error that is currently thrown when attempting to read a Parquet file with categorical columns:

```python
import pandas as pd
import pyarrow.parquet as pq

from datasets import load_dataset

# Create categorical sample DataFrame
df = pd.DataFrame({'type': ['foo', 'bar']}).astype('category')
df.to_parquet('data.parquet')

# Read back as pyarrow table
table = pq.read_table('data.parquet')
print(table.schema)
# type: dictionary<values=string, indices=int32, ordered=0>

# Load with huggingface datasets
load_dataset('parquet', data_files='data.parquet')
```

Error:
```
Traceback (most recent call last):                                                                                                                                                                                                        
  File ".venv/lib/python3.10/site-packages/datasets/builder.py", line 1875, in _prepare_split_single
    writer.write_table(table)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 566, in write_table
    self._build_writer(inferred_schema=pa_table.schema)
  File ".venv/lib/python3.10/site-packages/datasets/arrow_writer.py", line 379, in _build_writer
    inferred_features = Features.from_arrow_schema(inferred_schema)                                                                                     
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in from_arrow_schema
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1622, in <dictcomp>
    obj = {field.name: generate_from_arrow_type(field.type) for field in pa_schema}
  File ".venv/lib/python3.10/site-packages/datasets/features/features.py", line 1361, in generate_from_arrow_type
    raise NotImplementedError  # TODO(thom) this will need access to the dictionary as well (for labels). I.e. to the py_table
NotImplementedError
```

### Motivation

Categorical data types, as offered by Pandas and implemented with the `DictionaryType` dtype in `pyarrow` can significantly reduce dataset size and are a handy way to turn textual features into numerical representations and back. Lack of support in Huggingface datasets greatly reduces compatibility with a common Pandas / Parquet feature.

### Your contribution

I could provide a PR. However, it would be nice to have an initial complexity estimate from one of the core developers first.
Getting next item from IterableDataset took forever.

### Describe the bug

I have a large dataset, about 500GB. The format of the dataset is parquet. 

I then load the dataset and try to get the first item
```python
def get_one_item():
    dataset = load_dataset("path/to/datafiles", split="train", cache_dir=".", streaming=True)
    dataset = dataset.filter(lambda example: example['text'].startswith('Ar'))
    print(next(iter(dataset)))
```

However, this function never finish. I waited ~10mins, the function was still running so I killed the process. I'm now using `line_profiler` to profile how long it would take to return one item. I'll be patient and wait for as long as it needs. 

I suspect the filter operation is the reason why it took so long. Can I get some possible reasons behind this?

### Steps to reproduce the bug

Unfortunately without my data files, there is no way to reproduce this bug.

### Expected behavior

With `IteralbeDataset`, I expect the first item to be returned instantly.

### Environment info

- datasets version: 2.11.0 
- python: 3.7.12
5537 speedup load

I reimplemented fsspec.spec.glob() in `hffilesystem.py` as `_glob`,  used it in `_resolve_single_pattern_in_dataset_repository` only, and saw a 20% speedup in times to load the config, on average. 

That's not much when usually this step takes only 2-3 seconds for most datasets, but in this particular case,  `bigcode/the-stack-dedup` , the loading time to get the config (not download the entire 6tb dataset, of course), went from ~170 secs to ~20 secs. 

What makes this work is this code in `_glob`:

```  
      if self.dir_cache is not None:
            allpaths = self.dir_cache
        else:
            allpaths = self.find(root, maxdepth=depth, withdirs=True, detail=True, **kwargs)
```

I also had to `import glob.has_magic( )` for `_glob()` (confusing, I know).

I hope there is no issue with copying most of the code from `fsspec.spec.glob`, as it is a BSD 3-Clause License, 
and I left a comment about this in the docstring of` _glob()`, that we may want to delete.

As mentioned, I evaluated the speedup across a random selection of about 1000 datasets (not all 27k+), and verified that old_config.eq(new_method_config) with the build in method, but deleted this test and related code changes on the subsequent commit. It's in the commit history if anyone wants to see it. (Note this does not include the outlier of  `bigcode/the-stack-dedup`

|   |   old_time     |    new _time |     diff   |  pct_diff |
| -- | -- | -- | -- | -- |
| mean  |    3.340 |    2.642 |    0.698 |   18.404 |
| min   |    2.024   |  1.976   |  -0.840  | -37.634 |
| max   |  66.582  |   41.517 |   30.927  |  85.538 |
Is it possible or how to define a `datasets.Sequence` that could potentially be either a dict, a str, or None?

### Feature request

Hello! Apologies if my question sounds naive:

I was wondering if it’s possible, or how one would go about defining a 'datasets.Sequence' element in datasets.Features that could potentially be either a dict, a str, or None?

Specifically, I’d like to define a feature for a list that contains 18 elements, each of which has been pre-defined as either a `dict or None` or `str or None` - as demonstrated in the slightly misaligned data provided below:

```json
[
  [
    {"text":"老妇人","idxes":[0,1,2]},null,{"text":"跪","idxes":[3]},null,null,null,null,{"text":"在那坑里","idxes":[4,5,6,7]},null,null,null,null,null,null,null,null,null,null],
  [
    {"text":"那些水","idxes":[13,14,15]},null,{"text":"舀","idxes":[11]},null,null,null,null,null,{"text":"在那坑里","idxes":[4,5,6,7]},null,{"text":"出","idxes":[12]},null,null,null,null,null,null,null],
  [
    {"text":"水","idxes":[38]},
    null,
    {"text":"舀","idxes":[40]},
    "假",  // note this is just a standalone string
    null,null,null,{"text":"坑里","idxes":[35,36]},null,null,null,null,null,null,null,null,null,null]]
```

### Motivation

I'm currently working with a dataset of the following structure and I couldn't find a solution in the [documentation](https://huggingface.co/docs/datasets/v2.11.0/en/package_reference/main_classes#datasets.Features).

```json
{"qid":"3-train-1058","context":"桑桑害怕了。从玉米地里走到田埂上，他遥望着他家那幢草房子里的灯光，知道母亲没有让他回家的意思，很伤感，有点想哭。但没哭，转身朝阿恕家走去。","corefs":[[{"text":"桑桑","idxes":[0,1]},{"text":"他","idxes":[17]}]],"non_corefs":[],"outputs":[[{"text":"他","idxes":[17]},null,{"text":"走","idxes":[11]},null,null,null,null,null,{"text":"从玉米地里","idxes":[6,7,8,9,10]},{"text":"到田埂上","idxes":[12,13,14,15]},null,null,null,null,null,null,null,null],[{"text":"他","idxes":[17]},null,{"text":"走","idxes":[66]},null,null,null,null,null,null,null,{"text":"转身朝阿恕家去","idxes":[60,61,62,63,64,65,67]},null,null,null,null,null,null,null],[{"text":"灯光","idxes":[30,31]},null,null,null,null,null,null,{"text":"草房子里","idxes":[25,26,27,28]},null,null,null,null,null,null,null,null,null,null],[{"text":"他","idxes":[17]},{"text":"他家那幢草房子","idxes":[21,22,23,24,25,26,27]},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"远"],[{"text":"他","idxes":[17]},{"text":"阿恕家","idxes":[63,64,65]},null,null,null,null,null,null,null,null,null,null,null,null,null,null,null,"变近"]]}
```

### Your contribution

I'm going to provide the dataset at https://huggingface.co/datasets/2030NLP/SpaCE2022 .
Add Dataset.from_spark

Adds static method Dataset.from_spark to create datasets from Spark DataFrames.

This approach alleviates users of the need to materialize their dataframe---a common use case is that the user loads their dataset into a dataframe, uses Spark to apply some transformation to some of the columns, and then wants to train on the dataset.

Related issue: https://github.com/huggingface/datasets/issues/5678
Add Qdrant as another search index

### Feature request

I'd suggest adding Qdrant (https://qdrant.tech) as another search index available, so users can directly build an index from a dataset. Currently, FAISS and ElasticSearch are only supported: https://huggingface.co/docs/datasets/faiss_es 

### Motivation

ElasticSearch is a keyword-based search system, while FAISS is a vector search library. Vector database, such as Qdrant, is a different tool based on similarity (like FAISS) but is not limited to a single machine. It makes the vector database well-suited for bigger datasets and collaboration if several people want to access a particular dataset.

### Your contribution

I can provide a PR implementing that functionality on my own.