rachel521/github-issues-simple huggingface dataset:
Add more compression types for `to_json`
Opened by bhavitvyamalik (closed)

This PR adds `bz2`, `xz`, and `zip` (WIP) for `to_json`. I also plan to add `infer` like how `pandas` does it

Comments: @lhoestq, I looked into how to compress with `zipfile` for which few methods exist, let me know which one looks good:
1. create the file in normal `wb` mode and then zip it separately
2. use `ZipFile.write_str` to write file into the archive. For this we'll need to change how we're writing files from `_write` method  

How `pandas` handles it is that they have created a wrapper for standard library class `ZipFile` and allow the returned file-like handle to accept byte strings via `write` method instead of `write_str` (purpose was to change the name of function by creating that wrapper)

1. sounds not ideal since it creates an intermediary file.
I like pandas' approach. Is it possible to implement 2. using the pandas class ? Or maybe we can have something similar ?

Definitely, @lhoestq! I've adapted that from original code and turns out it is faster than `gz` compression. Apart from that I've also added `infer` option to automatically infer compression type from `path_or_buf` given

One small thing, currently I'm assuming that user will provide compression extension in `path_or_buf`. Is it this also possible?
`dataset.to_json("from_dataset.json", compression="zip")`? 
Should I put an `assert` to ensure the file name provided always has a compression extension?

Thanks !

> One small thing, currently I'm assuming that user will provide compression extension in path_or_buf. Is it this also possible?
>dataset.to_json("from_dataset.json", compression="zip")?
>Should I put an assert to ensure the file name provided always has a compression extension?

I think it's fine as it is right now :) No need to check the extension of the filename passed to `path_or_buf`.


> turns out it is faster than gz compression

I think the default compression level of `gzip` is 9 in python, which is very slow. Maybe we can switch to compression level 6 instead which is faster, like the `gzip` command on unix

I found that `fsspec` has something that may interest you: [fsspec.open(..., compression=...)](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.open). I don't remember if we've already mentioned it or not

It also has `zip` if I understand correctly ! see https://github.com/fsspec/filesystem_spec/blob/master/fsspec/compression.py#L70

Since `fsspec` is a dependency of `datasets` we can use all this :)

Let me know if you prefer using `fsspec` instead (I haven't tested this yet to write compressed files). IMO it sounds pretty easy to use and it would make the code base simpler

Just tried `fsspec` but I'm not able to write compressed `zip` files :/
`gzip`, `xz`, `bz2` are all working fine and it's really simple (no need for `FileWriteHandler` now!)
Dataset.from_pandas preserves useless index
Opened by Sorrow321 (closed)

## Describe the bug
Let's say that you want to create a Dataset object from pandas dataframe. Most likely you will write something like this:

```
import pandas as pd
from datasets import Dataset


df = pd.read_csv('some_dataset.csv')
# Some DataFrame preprocessing code...
dataset = Dataset.from_pandas(df)
```
If your preprocessing code contain indexing operations like this:
```
df = df[df.col1 == some_value]
```
then your df.index can be changed from (default) ```RangeIndex(start=0, stop=16590, step=1)``` to something like this ```Int64Index([    0,     1,     2,     3,     4,     5,     6,     7,     8,
                9,
            ...
            83979, 83980, 83981, 83982, 83983, 83984, 83985, 83986, 83987,
            83988],
           dtype='int64', length=16590)```

In this case, PyArrow (by default) will preserve this non-standard index. In the result, your dataset object will have the extra field that you likely don't want to have: '__index_level_0__'.

You can easily fix this by just adding extra argument ```preserve_index=False``` to call of ```InMemoryTable.from_pandas``` in ```arrow_dataset.py```.

If you approve that this isn't desirable behavior, I can make a PR fixing that.

## Environment info
- `datasets` version: 1.16.1
- Platform: Linux-5.11.0-44-generic-x86_64-with-glibc2.31
- Python version: 3.9.7
- PyArrow version: 6.0.1



Comments: Hi! That makes sense. Sure, feel free to open a PR! Just a small suggestion: let's make `preserve_index` a parameter of `Dataset.from_pandas` (which we then pass to `InMemoryTable.from_pandas`) with `None` as a default value to not have this as a breaking change.
Can't import `datasets` in python 3.10
Opened by lhoestq (closed)

When importing `datasets` I'm getting this error in python 3.10:
```python
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/__init__.py", line 34, in <module>
    from .arrow_dataset import Dataset, concatenate_datasets
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_dataset.py", line 47, in <module>
    from .arrow_reader import ArrowReader
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/arrow_reader.py", line 33, in <module>
    from .table import InMemoryTable, MemoryMappedTable, Table, concat_tables
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 334, in <module>
    class InMemoryTable(TableBlock):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 361, in InMemoryTable
    def from_pandas(cls, *args, **kwargs):
  File "/Users/quentinlhoest/Desktop/hf/nlp/src/datasets/table.py", line 24, in wrapper
    out = wraps(arrow_table_method)(method)
  File "/Users/quentinlhoest/.pyenv/versions/3.10.0/lib/python3.10/functools.py", line 61, in update_wrapper
    wrapper.__wrapped__ = wrapped
AttributeError: readonly attribute
```

This makes the conda build fail.
I'm opening a PR to fix this and do a patch release 1.16.1
How to choose proper download_mode in function load_dataset?
Opened by jzq2000 (closed)

Hi, I am a beginner to datasets and I try to use datasets to load my csv file.
my csv file looks like this

``` 
text,label
"Effective but too-tepid biopic",3
"If you sometimes like to go to the movies to have fun , Wasabi is a good place to start .",4
"Emerges as something rare , an issue movie that 's so honest and keenly observed that it does n't feel like one .",5
```

First I try to use this command to load my csv file .  

``` python
dataset=load_dataset('csv', data_files=['sst_test.csv'])
```

It seems good, but when i try to overwrite the convert_options to convert  'label' columns from int64 to float32 like this.

``` python
import pyarrow as pa
from pyarrow import csv
read_options = csv.ReadOptions(block_size=1024*1024)
parse_options = csv.ParseOptions()
convert_options = csv.ConvertOptions(column_types={'text': pa.string(), 'label': pa.float32()})
dataset = load_dataset('csv', data_files=['sst_test.csv'], read_options=read_options,
                       parse_options=parse_options, convert_options=convert_options)
```

It keeps the same:

```shell
Dataset(features: {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}, num_rows: 2210)
```

I think this issue is caused by the parameter "download_mode" Default to REUSE_DATASET_IF_EXISTS because after I delete the cache_dir, it seems right.

Is it a bug? How to choose proper download_mode to avoid this issue?


Comments: `download_mode=datasets.GenerateMode.FORCE_REDOWNLOAD` should work.
This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

Can we just use `features=...` in `load_dataset` for this @lhoestq?

Indeed you should use `features` in this case. 
```python
features = Features({'text': Value('string'), 'label': Value('float32')})
dataset = load_dataset('csv', data_files=['sst_test.csv'], features=features)
```
Note that because of an issue with the caching when you change the features (see #750 ) you still need to specify the `FORCE_REDOWNLOAD ` flag. I'm working on a fix for this one

https://github.com/huggingface/datasets/issues/769#issuecomment-717837832
> This makes me think we we should rename this to DownloadMode.FORCE_REDOWNLOAD. Currently that's confusing

@lhoestq do you still think we should rename it?


It's no big deal, but since it can be confusing to users I think it's worth renaming it, and deprecate `GenerateMode` until `datasets` 2.0 at least. IMO it's confusing to have `download_mode=GenerateMode.something`
push_to_hub does not upload videos
Opened by DominikVincent (open)

### Describe the bug

Hello,

I would like to upload a video dataset (some .mp4 files and some segments within them), i.e. rows correspond to subsequences from videos. Videos might be referenced by several rows.

I created a dataset locally and it references the videos and the video readers can read them correctly. I use push_to_hub() to upload the dataset to the hub.

Expectation: A user uses `load_dataset` and can load the videos.

However, the videos seem to be just referenced via paths on the computer and not uploaded to the hub. Therefore a target user cannot load the videos in the dataset.

### Steps to reproduce the bug

1. create a video dataset with paths e.g. { ["videos"]: [path1, path2, ...]}
2. dataset.push_to_hub
3. on a different computer (or same pc if relative paths are used in a different folder):
```
dataset = load_dataset("siplab/egosim", split="train")
video = dataset[0]["video_head"]
```

3. will fail

### Expected behavior

Expectation: A user uses `load_dataset` and can load the videos.

### Environment info

datasets                       3.1.0
Python 3.8.18

Comments: Hi ! the `Video` type is still experimental, and in particular `push_to_hub` doesn't upload videos at the moment (only the paths).

There is an open question to either upload the videos inside the Parquet files, or rather have them as separate files (which is great to enable remote seeking/streaming)

im having the same issue (btw i mistook this to be xet error https://huggingface.co/spaces/xet-team/README/discussions/4 )

@jsulz suggested me to use `upload_folder` but it exceeds hf limits (>10k files per folder and >100k files in total)

from my reading of the docs, in my case i have to save as either parquet or webdataset and then use `upload_folder`

i tried `ds.to_parquet("...")` but the parquet file also doesnt contain video, as of `datasets` v4.0

so i think the only workaround for my case is webdataset
Re-add download_manager module in utils
Opened by lhoestq (closed)

https://github.com/huggingface/datasets/pull/4384 moved `datasets.utils.download_manager` to `datasets.download.download_manager`

This breaks `evaluate` which imports `DownloadMode` from `datasets.utils.download_manager` 

This PR re-adds `datasets.utils.download_manager` without circular imports.

We could also show a message that says that accessing it is deprecated, but I think we can do this in a subsequent PR, and just focus on doing a patch release for now

Comments: _The documentation is not available anymore as the PR was closed or merged._

Thanks for the fix.

I'm wondering how this fixes backward compatibility...

Executing this code:
```python
from datasets.utils.download_manager import DownloadMode
```
we will have
```python
DownloadMode = None
```

If afterwards we use something like:
```python
if download_mode == DownloadMode.FORCE_REDOWNLOAD
```
that will raise an exception.

It works fine on my side:
```python
>>> from datasets.utils.download_manager import DownloadMode
>>> DownloadMode is not None
True
```

As reported in https://github.com/huggingface/evaluate/pull/143
```python
from datasets.utils import DownloadConfig
```
is also missing, I'm re-adding it

Took the liberty of merging this one, to do a patch release soon. If we think of a better approach we can improve it later
Choose columns to stream parquet data in streaming mode
Opened by lhoestq (closed)

Currently passing columns= to load_dataset in streaming mode fails

```
Tried to load parquet data with columns '['link']' with mismatching features '{'caption': Value(dtype='string', id=None), 'image': {'bytes': Value(dtype='binary', id=None), 'path': Value(dtype='null', id=None)}, 'link': Value(dtype='string', id=None), 'message_id': Value(dtype='string', id=None), 'timestamp': Value(dtype='string', id=None)}'
```

similar to https://github.com/huggingface/datasets/issues/6039

reported at https://huggingface.co/datasets/laion/dalle-3-dataset/discussions/3#65259a09617407d4520f4ad9
fix variable name typo
Opened by stas00 (closed)


Comments: Good catch :)
I think there is another occurence that needs to be fixed in the second gist (line 4924 of the notebook file):
```python
bleu = nlp.load_metric(...)
```

Was fixed in e16f79b5f7fc12a6a30c777722be46897a272e6f
Closing it.
Stream data concatenation issue
Opened by krishnapriya-18 (closed)

### Describe the bug

I am not able to concatenate the augmentation of the stream data. I am using the latest version of dataset.

ValueError: The features can't be aligned because the key audio of features {'audio_id': Value(dtype='string', 
id=None), 'audio': {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 'path': 
Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)}, 'transcript': Value(dtype='string', 
id=None)} has unexpected type - {'array': Sequence(feature=Value(dtype='float32', id=None), length=-1, id=None), 
'path': Value(dtype='null', id=None), 'sampling_rate': Value(dtype='int64', id=None)} (expected either 
Audio(sampling_rate=16000, mono=True, decode=True, id=None) or Value("null").

### Steps to reproduce the bug

dataset = load_dataset("tobiolatunji/afrispeech-200", "all", streaming=True).shuffle(seed=42)
dataset_cln = dataset.remove_columns(['speaker_id', 'path', 'age_group', 'gender', 'accent', 'domain', 'country', 'duration'])
dataset_cln = dataset_cln.cast_column("audio", Audio(sampling_rate=16000))
from audiomentations import AddGaussianNoise,Compose,Gain,OneOf,PitchShift,PolarityInversion,TimeStretch

augmentation = Compose([
    AddGaussianNoise(min_amplitude=0.005, max_amplitude=0.015, p=0.2)
])

def augment_dataset(batch):
    audio = batch["audio"]
    audio["array"] = augmentation(audio["array"], sample_rate=audio["sampling_rate"])    
    return batch

augmented_dataset_cln = dataset_cln['train'].map(augment_dataset)
dataset_cln['train'] = interleave_datasets([dataset_cln['train'], augmented_dataset_cln])
dataset_cln['train'] = dataset_cln['train'].shuffle(seed=42)

### Expected behavior

I should be able to merge as sampling rate is same.

### Environment info

import datasets
import transformers
import accelerate
print(datasets.__version__)
print(transformers.__version__)
print(torch.__version__)
print(evaluate.__version__)
print(accelerate.__version__)

2.12.0
4.28.1
2.0.0
0.4.0
0.18.0

Comments: Hi! 

You can call `map` as follows to avoid the error:
```python
augmented_dataset_cln = dataset_cln['train'].map(augment_dataset, features=dataset_cln['train'].features)
```

Thanks it is solved

Hi! 
I have run into the same problem with you. Could you please let me know how you solve it? Thanks!
load_dataset is case sentitive?
Opened by haonan-li (open)

### Describe the bug

load_dataset() function is case sensitive?

### Steps to reproduce the bug

The following two code, get totally different behavior.

1. load_dataset('mbzuai/bactrian-x','en')

2. load_dataset('MBZUAI/Bactrian-X','en')

### Expected behavior

Compare 1 and 2.
1 will download all 52 subsets, shell output:
```Downloading and preparing dataset json/MBZUAI--bactrian-X to xxx```
2 will only download single subset, shell output
```Downloading and preparing dataset bactrian-x/en to xxx```


### Environment info

Python 3.10.11
datasets Version: 2.11.0

Comments: Hi @haonan-li , thank you for the report! It seems to be a bug on the [`huggingface_hub`](https://github.com/huggingface/huggingface_hub) site, there is even no such dataset as `mbzuai/bactrian-x` on the Hub.  I opened and [issue](https://github.com/huggingface/huggingface_hub/issues/1453) there.

I think `load_dataset("mbzuai/bactrian-x")` shouldn't be loaded at all and raise an error but because of [this fallback](https://github.com/huggingface/datasets/blob/main/src/datasets/load.py#L1194) to packaged loaders when no other options are applicable, it loads the dataset with standard `json` loader instead of the custom loading script.
datasets-cli: convertion of a tfds dataset to a huggingface one.
Opened by vitalyshalumov (open)

### Discussed in https://github.com/huggingface/datasets/discussions/3079

<div type='discussions-op-text'>

<sup>Originally posted by **vitalyshalumov** October 14, 2021</sup>
I'm trying to convert a tfds dataset to a huggingface one.

I've tried:

1.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/mnist/3.0.1/

2.       datasets-cli convert  --tfds_path ~/tensorflow_datasets/mnist/3.0.1/ --datasets_directory ~/.cache/huggingface/datasets/


and other permutations.
The script appears to be running and finishing without an error but when looking in the  huggingface/datasets/ folder nothing is created.


</div>

Comments: Hi,

the MNIST dataset is already available on the Hub. You can use it as follows:
```python
import datasets
dataset_dict = datasets.load_dataset("mnist")
```

As for the conversion of TFDS datasets to HF datasets, we will be working on it in the coming months, so stay tuned.
Update fsspec max version to current release 2025.7.0
Opened by rootAvish (closed)

Diffusers currently asks for a max fsspec version of `2025.3.0`. This change updates it to the current latest version. This change is mainly required to resolve conflicts with other packages in an environment. In my particular case, `aider-chat` which is a part of my environment installs `2025.5.1` which is incompatible with `datasets`.

Comments: @lhoestq I ran the test suite locally and while some tests were failing those failures are present on the main branch too. Could you please review and trigger the CI?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_7701). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Which release will this be available in ? I'm running into this issue with `datasets=3.6.0`
[Datasets] add ted_hrlr
Opened by patrickvonplaten (closed)

@thomwolf - After looking at `xnli` I think it's better to leave the translation features and add a `translation` key to make them work in our framework. 

The result looks like this:
![Screenshot from 2020-05-12 18-34-43](https://user-images.githubusercontent.com/23423619/81721933-ee1faf00-9480-11ea-9e95-d6557cbd0ce0.png)

you can see that each split has a `translation` key which value is the nlp.features.Translation object. 

That's a simple change. If it's ok for you, I will add dummy data for the other configs and treat the other translation scripts in the same way.
Add Fake News Detection in Filipino dataset
Opened by anaerobeth (closed)

This PR adds the Fake News Filipino Dataset, a low-resource fake news detection corpora in Filipino. Contains 3,206 expertly-labeled news samples, half of which are real and half of which are fake.

Link to the paper: http://www.lrec-conf.org/proceedings/lrec2020/index.html

Link to the dataset/repo: https://github.com/jcblaisecruz02/Tagalog-fake-news

Comments: Hi! I'm the author of this paper (surprised to see our datasets have been added already).

That paper link only leads to the conference index, here's a link to the actual paper: https://www.aclweb.org/anthology/2020.lrec-1.316/

Would it be fine if I also edited your gsheet entry to reflect this change?

Hi Jan, please go ahead and update. I see you are also in the sprint slack channel. Let me know if what else needs updating. Thanks.
In v1.4.1, all TIMIT train transcripts are "Would such an act of refusal be useful?"
Opened by rcgale (closed)

## Describe the bug
Using version 1.4.1 of `datasets`, TIMIT transcripts are all the same.

## Steps to reproduce the bug
I was following this tutorial
- https://huggingface.co/blog/fine-tune-wav2vec2-english

But here's a distilled repro:
```python
!pip install datasets==1.4.1
from datasets import load_dataset
timit = load_dataset("timit_asr", cache_dir="./temp")
unique_transcripts = set(timit["train"]["text"])
print(unique_transcripts)
assert len(unique_transcripts) > 1
```
## Expected results
Expected the correct TIMIT data. Or an error saying that this version of `datasets` can't produce it.

## Actual results
Every train transcript was "Would such an act of refusal be useful?" Every test transcript was "The bungalow was pleasantly situated near the shore."

## Environment info
- `datasets` version: 1.4.1
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.9
- PyTorch version (GPU?): 1.9.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: tried both
- Using distributed or parallel set-up in script?: no
- 



Comments: Hi @rcgale, thanks for reporting.

Please note that this bug was fixed on `datasets` version 1.5.0: https://github.com/huggingface/datasets/commit/a23c73e526e1c30263834164f16f1fdf76722c8c#diff-f12a7a42d4673bb6c2ca5a40c92c29eb4fe3475908c84fd4ce4fad5dc2514878

If you update `datasets` version, that should work.

On the other hand, would it be possible for @patrickvonplaten to update the [blog post](https://huggingface.co/blog/fine-tune-wav2vec2-english) with the correct version of `datasets`?

I just proposed a change in the blog post.

I had assumed there was a data format change that broke a previous version of the code, since presumably @patrickvonplaten tested the tutorial with the version they explicitly referenced. But that fix you linked suggests a problem in the code, which surprised me.

I still wonder, though, is there a way for downloads to be invalidated server-side? If the client can announce its version during a download request, perhaps the server could reject known incompatibilities? It would save much valuable time if `datasets` raised an informative error on a known problem ("Error: the requested data set requires `datasets>=1.5.0`."). This kind of API versioning is a prudent move anyhow, as there will surely come a time when you'll need to make a breaking change to data.

Also, thank you for a quick and helpful reply!
When will the remaining math_dataset modules be added as dataset objects
Opened by tylerroost (closed)

Currently only the algebra_linear_1d is supported. Is there a timeline for making the other modules supported. If no timeline is established, how can I help?

Comments: On a similar note it would be nice to differentiate between train-easy, train-medium, and train-hard

Hi @tylerroost, we don't have a timeline for this at the moment.
If you want to give it a look we would be happy to review a PR on it.
Also, the library is one week old so everything is quite barebones, in particular the doc.
You should expect some bumps on the road.

To get you started, you can check the datasets scripts in the `./datasets` folder on the repo and find the one on math_datasets that will need to be modified. Then you should check the original repository on the math_dataset to see where the other files to download are located and what is the expected format for the various parts of the dataset.

To get a general overview on how datasets scripts are written and used, you can read the nice tutorial on how to add a new dataset for TensorFlow Dataset [here](https://www.tensorflow.org/datasets/add_dataset), our API is not exactly identical but it can give you a high-level overview.

Thanks I'll give it a look