rIsHu009/github-issues-updated huggingface dataset:
Title: Support webdataset format
Author: lhoestq (MEMBER)
State: closed
Created: 2022-12-07T11:32:25Z

Webdataset is an efficient format for iterable datasets. It would be nice to support it in `datasets`, as discussed in https://github.com/rom1504/img2dataset/issues/234.

In particular it would be awesome to be able to load one using `load_dataset` in streaming mode (either from a local directory, or from a dataset on the Hugging Face Hub). Some datasets on the Hub are already in webdataset format.

It terms of implementation, we can have something similar to the Parquet loader.
I also think it's fine to have webdataset as an optional dependency.

Comments:
I like the idea of having `webdataset` as an optional dependency to ensure our loader generates web datasets the same way as the main project.

Webdataset is the one of the most popular dataset formats for large scale computer vision tasks. Upvote for this issue. 

Any updates on this?

We haven't had the bandwidth to implement it so far, but if someone wants to give it a shot please don't hesitate ^^

Done in #6391
Title: Set `IterableDataset.map` param `batch_size` typing as optional
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-12-06T17:08:10Z

This PR solves #5325 

~Indeed we're using the typing for optional values as `Union[type, None]` as it's similar to how Python 3.10 handles optional values as `type | None`, instead of using `Optional[type]`.~

~Do we want to start using `Union[type, None]` for type-hinting optional values or just keep on using `Optional`?~ -> Keeping `Optional` still for consistency with the rest of the code in `datasets`

Also we now allow `batch_size` to be `None` for `IterableDataset.map` and `IterableDataset.filter`e.g. `MappedExamplesIterable` as `map` is internally instantiating those and propagating the `batch_size` param so if it can be `None` for `map` it should also do so for `MappedExamplesIterable`, as well as for `FilteredExamplesIterable` when calling `IterableDataset.filter`.

## TODOs

- [x] Add integration tests
- [x] Handle scenario where `batched=True` and `batch_size=None` or `batch_size<=0`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5336). All of your documentation changes will be reflected on that endpoint.

Hi @mariosasko, @lhoestq I was wondering whether we should include `batched` as a `pytest.mark` param for the functions testing `IterableDataset.map` so as to ensure that the changes done in this PR work fine without breaking anything of the actual functionality.

I've pushed updated tests just for one of the unit testing functions to be run as `pytest tests/test_iterable_dataset.py::test_mapped_examples_iterable -s --durations 0`, but some are still missing `batched` param, it was just to ask you whether we're supposed to do this for the rest of the functions or not, if it's a yes I'll push the commit as it's ready, but didn't want to push extra stuff that may be discarded later!

Thanks :hugs:

Thanks for the feedback @lhoestq, I agree with keeping `Optional` instead of `Union[type, None]` for now 👍🏻
Title: Update tasks.json
Author: sayakpaul (MEMBER)
State: closed
Created: 2022-12-06T11:37:57Z

Context:

* https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195

Cc: @osanseviero

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts

And I think we can remove tasks.json completely from this repo

Isn't tasks.json used anymore in this repo?

> I think the only place where we need to add it is here https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts
> 
> And I think we can remove tasks.json completely from this repo

What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts. 

The update is based on what I received in the output of the export job (c.f. https://github.com/huggingface/datasets/issues/5255#issuecomment-1339107195). 

Edit: Oh, are you referring to the dataset card of NYU Depth V2?

Yes, my suggestion was for the dataset card: you got the error message because you tried to set `depth-estimation` in `class_ids` instead of `class_categories`.

> What about the warning I mentioned in https://github.com/huggingface/datasets/issues/5255#issuecomment-1339013527? Also, the depth estimation entry is already present in https://github.com/huggingface/hub-docs/blob/main/js/src/lib/interfaces/Types.ts.

if you place it in `task_categories` you should be good :)

yes i would suggest rm'ing tasks.json here for clarity

Closing it. 

It's not clear if we can remove it btw, since old versions of `evaluate` rely on it (see https://github.com/huggingface/evaluate/pull/309)

cc @lvwerra 

Actually it can be removed without incidence in old versions of evaluate since we kept an hardcoded `known_task_ids` that is marked "DEPRECATED"
Title: Clean up docstrings
Author: stevhliu (MEMBER)
State: closed
Created: 2022-12-05T20:56:08Z
Labels: documentation

As raised by @polinaeterna in #5324, some of the docstrings are a bit of a mess because it has both Markdown and Sphinx syntax. This PR fixes the docstring for `DatasetBuilder`. 

I'll start working on cleaning up the rest of the docstrings and removing the old Sphinx syntax (let me know if you prefer one big PR with all the cleaned changes or multiple smaller ones)! 🧼

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! Let us know if we can help :)

Small pref for having multiple PRs

Awesome, thanks! Sorry this one is a little big, I'll open some smaller ones next :)
Title: Clarify imagefolder is for small datasets
Author: stevhliu (MEMBER)
State: closed
Created: 2022-12-01T21:47:29Z

Based on feedback from [here](https://github.com/huggingface/datasets/issues/5317#issuecomment-1334108824), this PR adds a note to the `imagefolder` loading and creating docs that `imagefolder` is designed for small scale image datasets.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I think it's also reasonable to add the same note to the AudioFolder decription

Thank you ! I think "regular" is more appropriate than "small". It can easily scale to a few thousands of images - just not millions x)

Replaced "small" with "several thousand" since what is considered "regular" and even "small" can be kind of vague!
Title: Fix docs building for main
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-12-01T17:07:45Z

This PR reverts the triggering event for building documentation introduced by:
- #5250

Fix #5326.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

EDIT
At least the docs for ~~main~~ PR branch are now built:
- https://github.com/huggingface/datasets/actions/runs/3594847760/jobs/6053620813

Build documentation for main branch was triggered after this PR being merged: https://github.com/huggingface/datasets/actions/runs/3603370082/jobs/6071482470
Title: Avoid unwanted behaviour when splits from script and metadata are not matching because of outdated metadata
Author: polinaeterna (CONTRIBUTOR)
State: open
Created: 2022-12-01T17:05:23Z

will fix  #5315

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5327). All of your documentation changes will be reflected on that endpoint.
Title: No documentation for main branch is built
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-12-01T16:50:58Z
Labels: bug

Since:
- #5250
  - Commit: 703b84311f4ead83c7f79639f2dfa739295f0be6

the docs for main branch are no longer built.

The change introduced only triggers the docs building for releases.

Comments:
Title: map(...batch_size=None) for IterableDataset
Author: frankier (CONTRIBUTOR)
State: closed
Created: 2022-12-01T15:43:42Z
Labels: enhancement

good first issue

### Feature request

Dataset.map(...) allows batch_size to be None. It would be nice if IterableDataset did too.

### Motivation

Although it may seem a bit of a spurious request given that `IterableDataset` is meant for larger than memory datasets, but there are a couple of reasons why this might be nice.

One is that load_dataset(...) can return either IterableDataset or Dataset. mypy will then complain if batch_size=None even if we know it is Dataset. Of course we can do:

    assert isinstance(d, datasets.DatasetDict)

But it is a mild inconvenience. What's more annoying is that whenever we use something like e.g. `combine_datasets(...)`, we end up with the union again, and so have to do the assert again.

Another is that we could actually end up with an IterableDataset small enough for memory in normal/correct usage, e.g. by filtering a massive IterableDataset.

For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

### Your contribution

Not this time.

Comments:
Hi! I agree it makes sense for `IterableDataset.map` to support the `batch_size=None` case. This should be super easy to fix.

@mariosasko as this is something simple maybe I can include it as part of https://github.com/huggingface/datasets/pull/5311? Let me know :+1:

#self-assign

Feel free to close this @lhoestq as part of https://github.com/huggingface/datasets/pull/5336 :hugs:

Thanks again :)

> For practical usages, an alternative to this would be to convert from an iterable dataset to a map-style dataset, but it is not obvious how to do this.

This is interesting as well, if anyone wants to explore
Title: Fix docstrings and types in documentation that appears on the website
Author: polinaeterna (CONTRIBUTOR)
State: open
Created: 2022-12-01T15:34:53Z
Labels: documentation

While I was working on https://github.com/huggingface/datasets/pull/5313 I've noticed that we have a mess in how we annotate types and format args and return values in the code. And some of it is displayed in the [Reference section](https://huggingface.co/docs/datasets/package_reference/builder_classes) of the documentation on the website.
 
Would be nice someday, maybe before releasing datasets 3.0.0, to unify it......

Comments:
I agree we have a mess with docstrings...

Ok, I believe we've cleaned up most of the old syntax we were using for the user-facing docs! There are still a couple of `:obj:`'s and `:class:` floating around in the docstrings we don't expose that I'll track down :)

Hi @polinaeterna @albertvillanova @stevhliu,  I hope you all are doing well.

Is this issue still unresolved as I am interested in it?

It should be mostly fixed for the user-facing APIs, but there may be some Sphinx syntax still lurking around in the non-public APIs. Feel free to open a PR to fix those if you catch any! 🤗 

Thanks for your reply @stevhliu  :)
Sure, I will try to find out the remaining and fix that.
Title: Raise error for `.tar` archives in the same way as for `.tar.gz` and `.tgz` in `_get_extraction_protocol`
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2022-12-01T15:19:28Z

Currently `download_and_extract` doesn't throw an error when it is used with files with `.tar` extension in streaming mode because `_get_extraction_protocol` doesn't do it (like it does for  `tar.gz` and `tgz`). `_get_extraction_protocol` returns formatted url as if we support tar protocol but we don't.
That means that in dataset scripts `.tar` files would be attempted to load and fail during examples generation (after `download_and_extract` execution). So this PR raises error for `tar` files too.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Fix loading from HF GCP cache
Author: lhoestq (MEMBER)
State: closed
Created: 2022-12-01T14:39:06Z

As reported in https://discuss.huggingface.co/t/error-loading-wikipedia-dataset/26599/4 it's not possible to download a cached version of Wikipedia from the HF GCP cache

I fixed it and added an integration test (runs in 10sec)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Do you know why this stopped working?

It comes from the changes in https://github.com/huggingface/datasets/pull/5107/files#diff-355ae5c229f95f86895404b72378ecd6e966c41cbeebb674af6fe6e9611bc126
Title: [Extract] Place the lock file next to the destination directory
Author: lhoestq (MEMBER)
State: closed
Created: 2022-12-01T13:55:49Z

Previously it was placed next to the archive to extract, but the archive can be in a read-only directory as noticed in https://github.com/huggingface/datasets/issues/5295

Therefore I moved the lock location to be next to the destination directory, which is required to have write permissions

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Origin/fix missing features error
Author: eunseojo (CONTRIBUTOR)
State: closed
Created: 2022-12-01T06:18:39Z

This fixes the problem of when the dataset_load function reads a function with "features" provided but some read batches don't have columns that later show up. For instance, the provided "features" requires columns A,B,C but only columns B,C show. This fixes this by adding the column A with nulls.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

please review :) @lhoestq @ola13 thankoo

Thanks :) I just updated the test to make sure it works even when there's a column missing, and did a minor change to json.py to add the missing columns for the other kinds of JSON files as well (I moved the code to`self._cast_table`)

Thanks Unso! If @lhoestq is happy then I'm also happy :D

When I noticed the ping, this PR had already been merged...

Luckily, PyArrow's `read_json` behaves the same when `explicit_schema` is given via `ParseOptions`, so I'm okay with this change (our JSON loader doesn't use `read_json` for decoding JSON in some scenarios, so this manual approach is the right one).
Title: Bug in sample_by="paragraph"
Author: adampauls (NONE)
State: closed
Created: 2022-11-30T19:24:13Z

### Describe the bug

I think [this line](https://github.com/huggingface/datasets/blob/main/src/datasets/packaged_modules/text/text.py#L96) is wrong and should be `batch = f.read(self.config.chunksize)`. Otherwise it will never terminate because even when `f` is finished reading, `batch` will still be truthy from the last iteration.

### Steps to reproduce the bug

```
> cat test.txt
a b c

d e f
````

```python
>>> import datasets
>>> datasets.load_dataset("text", data_files={"train":"test.txt"}, sample_by="paragraph")
```

This will go on forever.

### Expected behavior

Terminates very quickly. 

### Environment info

`version = "2.6.1"` but I think the bug is still there on main.

Comments:
Thanks for reporting, @adampauls.

We are having a look at it.
Title: Datasets: classification_report() got an unexpected keyword argument 'suffix'
Author: JonathanAlis (NONE)
State: closed
Created: 2022-11-30T14:01:03Z

https://github.com/huggingface/datasets/blob/main/metrics/seqeval/seqeval.py


> import datasets
predictions = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = datasets.load_metric("seqeval")
results = seqeval.compute(predictions=predictions, references=references)
print(list(results.keys()))
print(results["overall_f1"])
print(results["PER"]["f1"])

It raises the error:

> TypeError: classification_report() got an unexpected keyword argument 'suffix'

For context, versions on my pip list -v

> datasets                 1.12.1  
seqeval                  1.2.2

Comments:
This seems similar to https://github.com/huggingface/datasets/issues/2512 Can you try to update seqeval ? 

@JonathanAlis also note that the metrics are deprecated in our `datasets` library.

Please, use the new library 🤗 Evaluate instead: https://huggingface.co/docs/evaluate
Title: Fix description of streaming in the docs
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2022-11-29T18:00:28Z

We say that "the data is being downloaded progressively" which is not true, it's just streamed, so I fixed it. Probably I missed some other places where it is written? 

Also changed docstrings for `StreamingDownloadManager`'s `download` and `extract` to reflect the same, as these docstrings are displayed in the documentation cc @lhoestq

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add `features` param to `IterableDataset.map`
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-11-29T11:08:34Z

## Description

As suggested by @lhoestq in #3888, we should be adding the param `features` to `IterableDataset.map` so that the features can be preserved (not turned into `None` as that's the default behavior) whenever the user passes those as param, so as to be consistent with `Dataset.map`, as it provides the `features` param so that those are not inferred by default, but specified by the user, and later validated by `ArrowWriter`.

This is internally handled already by the functions relying on `IterableDataset.map` such as `rename_column`, `rename_columns`, and `remove_columns` as described in #5287.

## Usage Example

```python
from datasets import load_dataset, Features

ds = load_dataset("rotten_tomatoes", split="validation", streaming=True)
print(ds.info.features)
ds = ds.map(
    lambda x: {"target": x["label"]},
    features=Features(
        {"target": ds.info.features["label"], "label": ds.info.features["label"], "text": ds.info.features["text"]}
    ),
)
print(ds.info.features)
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Support xPath for Windows pathnames
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-29T09:20:47Z

This PR implements a string representation of `xPath`, which is valid for local paths (also windows) and remote URLs.

Additionally, some `os.path` methods are fixed for remote URLs on Windows machines.

Now, on Windows machines:
```python
In [2]: str(xPath("C:\\dir\\file.txt"))
Out[2]: 'C:\\dir\\file.txt'
In [3]: str(xPath("http://domain.com/file.txt"))
Out[3]: 'http://domain.com/file.txt'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Close stream in `ArrowWriter.finalize` before inference error
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-11-28T16:59:39Z

Ensure the file stream is closed in `ArrowWriter.finalize` before raising the `SchemaInferenceError` to avoid the `PermissionError` on Windows in `incomplete_dir`'s `shutil.rmtree`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Support `topdown` parameter in `xwalk`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-11-28T14:42:41Z

Add support for the `topdown` parameter in `xwalk` when `fsspec>=2022.11.0` is installed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I like the `kwargs` approach, thanks!
Title: timit_asr doesn't load the test split.
Author: seyong92 (NONE)
State: closed
Created: 2022-11-26T10:18:22Z

### Describe the bug

When I use the function ```timit = load_dataset('timit_asr', data_dir=data_dir)```, it only loads train split, not test split.

I tried to change the directory and filename to lower case to upper case for the test split, but it does not work at all.

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 0
    })
})
```

The directory structure of both splits are same. (DIALECT_REGION / SPEAKER_CODE / DATA_FILES)

### Steps to reproduce the bug

1. just use ```timit = load_dataset('timit_asr', data_dir=data_dir)```

### Expected behavior

```python
DatasetDict({
    train: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 4620
    })
    test: Dataset({
        features: ['file', 'audio', 'text', 'phonetic_detail', 'word_detail', 'dialect_region', 'sentence_type', 'speaker_id', 'id'],
        num_rows: 1680
    })
})
```

### Environment info

- ubuntu 20.04
- python 3.9.13
- datasets 2.7.1

Comments:
The [timit_asr.py](https://huggingface.co/datasets/timit_asr/blob/main/timit_asr.py) script iterates over the WAV files per split directory using this:
```python
wav_paths = sorted(Path(data_dir).glob(f"**/{split}/**/*.wav"))
wav_paths = wav_paths if wav_paths else sorted(Path(data_dir).glob(f"**/{split.upper()}/**/*.WAV"))
```

Can you check that there is a directory named "test" somewhere in your timit data directory ?
Title: Improve `use_auth_token` docstring and deprecate `use_auth_token` in `download_and_prepare`
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-11-25T17:09:21Z

Clarify in the docstrings what happens when `use_auth_token` is `None` and deprecate the `use_auth_token` param in `download_and_prepare`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Return a split Dataset in load_dataset
Author: lhoestq (MEMBER)
State: closed
Created: 2022-11-25T16:35:54Z

...instead of a DatasetDict.

```python
# now supported
ds = load_dataset("squad")
ds[0]  
for example in ds:
    pass

# still works
ds["train"]
ds["validation"]

# new
ds.splits  # Dict[str, Dataset] | None

# soon to be supported (not in this PR)
ds = load_dataset("dataset_with_no_splits")
ds[0]
for example in ds:
    pass
```

I implemented `Dataset.__getitem__` and `IterableDataset.__getitem__` to be able to get a split from a dataset.
The splits are defined by the `ds.info.splits` dictionary.

Therefore a dataset is a table that optionally has some splits defined in the dataset info. And a split dataset is the concatenation of all its splits.

I made as little breaking changes as possible. Notable breaking changes:
- `load_dataset("potato").keys() / .items() / .values() /` don't work anymore, since we don't return a dict
- same for `for split_name in load_dataset("potato")`, since we now iterate on the examples
- ..

TODO:
- [x] Update push_to_hub
- [x] Update save_to_disk/load_from_disk
- [ ] check for other breaking changes
- [ ] fix existing tests
- [ ] add new tests
- [ ] docs

This is related to https://github.com/huggingface/datasets/issues/5189, to extend `load_dataset` to return datasets without splits

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_5301). All of your documentation changes will be reflected on that endpoint.

Just noticed that now we have to deal with indexed & split datasets. The remaining tests are failing because one should be able to get an indexed dataset when accessing the split of a dataset made of indexed splits (right now the index is just trashed)
Title: Use same `num_proc` for dataset download and generation
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-11-25T15:37:42Z

Use the same `num_proc` value for data download and generation. Additionally, do not set `num_proc` to 16 in `DownloadManager` by default (`num_proc` now has to be specified explicitly).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I noticed this bug the other day and was going to look into it! "Where are these processes coming from?" ;-)
Title: Bug in xopen with Windows pathnames
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-25T15:21:32Z
Labels: bug

Currently, `xopen` function has a bug with local Windows pathnames:

From its implementation:
```python
def xopen(file: str, mode="r", *args, **kwargs):
    file = _as_posix(PurePath(file))
    main_hop, *rest_hops = file.split("::")
    if is_local_path(main_hop):
        return open(file, mode, *args, **kwargs)
```

On a Windows machine, if we pass the argument:
```python
xopen("C:\\Users\\USERNAME\\filename.txt")
```
it returns
```python
open("C:/Users/USERNAME/filename.txt")
```

Comments:
Title: Bug in xjoin with Windows pathnames
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-25T13:29:33Z
Labels: bug

Currently, `xjoin` function has a bug with local Windows pathnames: instead of returning the OS-dependent join pathname, it always returns it in POSIX format.

```python
from datasets.download.streaming_download_manager import xjoin

path = xjoin("C:\\Users\\USERNAME", "filename.txt")
```

Join path should be:
```python
"C:\\Users\\USERNAME\\filename.txt"
```
However it is:
```python
"C:/Users/USERNAME/filename.txt"
```

Comments:
Title: Missing documentation build for versions 2.7.1 and 2.6.2 
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-24T09:42:10Z
Labels: maintenance

After the patch releases [2.7.1](https://github.com/huggingface/datasets/releases/tag/2.7.1) and [2.6.2](https://github.com/huggingface/datasets/releases/tag/2.6.2), the online docs were not properly built (the build_documentation workflow was not triggered).

There was a fix by:
- #5291

However, both documentations were built from main branch, instead of their corresponding version branch.

We are rebuilding them.

Comments:
- Build docs for 2.6.2:
  - Commit: a6a5a1cf4cdf1e0be65168aed5a327f543001fe8
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539470622/jobs/5941404044
- Build docs for 2.7.1:
  - Commit: 5ef1ab1cc06c2b7a574bf2df454cd9fcb071ccb2
  - Build docs GH Action: https://github.com/huggingface/datasets/actions/runs/3539574442/jobs/5941636792
Title: [build doc] for v2.7.1 & v2.6.2
Author: mishig25 (NONE)
State: closed
Created: 2022-11-24T08:54:47Z

Do NOT merge. Using this PR to build docs for [v2.7.1](https://github.com/huggingface/datasets/pull/5291/commits/f4914af20700f611b9331a9e3ba34743bbeff934) & [v2.6.2](https://github.com/huggingface/datasets/pull/5291/commits/025f85300a0874eeb90a20393c62f25ac0accaa0)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

doc versions are built https://huggingface.co/docs/datasets/index
Title: Lossy json serialization - deserialization of dataset info 
Author: anuragprat1k (NONE)
State: open
Created: 2022-11-23T17:20:15Z

### Describe the bug

Saving a dataset to disk as json (using `to_json`) and then loading it again (using `load_dataset`) results in features whose labels are not type-cast correctly. In the code snippet below, `features.label` should have a label of type `ClassLabel` but has type `Value` instead. 

### Steps to reproduce the bug

```
from datasets import load_dataset

def test_serdes_from_json(d):
    dataset = load_dataset(d, split="train")
    dataset.to_json('_test')
    dataset_loaded = load_dataset("json", data_files='_test', split='train')
    try:
        assert dataset_loaded.info.features == dataset.info.features, "features unequal!"
    except Exception as ex:
        print(f'{ex}')
        print(f'expected {dataset.info.features}, \nactual { dataset_loaded.info.features }')
    
test_serdes_from_json('rotten_tomatoes')
```

Output
```
features unequal!
expected {'text': Value(dtype='string', id=None), 'label': ClassLabel(names=['neg', 'pos'], id=None)}, 
actual {'text': Value(dtype='string', id=None), 'label': Value(dtype='int64', id=None)}
```


### Expected behavior

The deserialized `features.label` should have type `ClassLabel`.

### Environment info

- `datasets` version: 2.6.1
- Platform: Linux-5.10.144-127.601.amzn2.x86_64-x86_64-with-glibc2.17
- Python version: 3.7.13
- PyArrow version: 7.0.0
- Pandas version: 1.2.3

Comments:
Hi ! JSON is a lossy format indeed. If you want to keep the feature types or other metadata I'd encourage you to store them as well. For example you can use `dataset.info.write_to_directory` and `DatasetInfo.from_directory` to store the feature types, split info, description, license etc.
Title: Save file name in embed_storage
Author: lhoestq (MEMBER)
State: closed
Created: 2022-11-23T10:55:54Z

Having the file name is useful in case we need to check the extension of the file (e.g. mp3), or in general in case it includes some metadata information (track id, image id etc.)

Related to https://github.com/huggingface/datasets/issues/5276

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I updated the tests, met le know if it sounds good to you now :)
Title: Import error
Author: feketedavid1012 (NONE)
State: closed
Created: 2022-11-22T12:56:43Z

https://github.com/huggingface/datasets/blob/cd3d8e637cfab62d352a3f4e5e60e96597b5f0e9/src/datasets/__init__.py#L28

Hy, 
I have error at the above line. I have python version 3.8.13, the message says I need python>=3.7, which is True, but I think the if statement not working properly (or the message wrong)

Comments:
Hi ! Can you 
```python
import platform
print(platform.python_version())
```
to see that it returns ?

Hi,

3.8.13

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:37:02 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Hi ! Can you

import platform
print(platform.python_version())

to see that it returns ?

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323691385>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F5YGG32W6WABYC25NJTWJTD75ANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


Then it should work as expected if you use the same python when using `datasets`

Please make sure you're running your code in the right environment

It's the right environment. But in if statement I have
"3.8.13" < 3.7
And in the error message is Python>=3.7 which is true in my case (3.8.13 is greater then 3.7), so I don't understand my python should be below the 3.7 which case the if statement is right, but the message is wrong, or above 3.7 which case if statement is wrong, error message is right.

Get Outlook for Android<https://aka.ms/AAb9ysg>
________________________________
From: Quentin Lhoest ***@***.***>
Sent: Tuesday, November 22, 2022 2:41:43 PM
To: huggingface/datasets ***@***.***>
Cc: feketedavid1012 ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/datasets] Import error (Issue #5280)


Then it should work as expected if you use the same python when using datasets

Please make sure you're running your code in the right environment

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/datasets/issues/5280#issuecomment-1323697094>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AJW7F54JURTAJJWWDO2QGI3WJTERPANCNFSM6AAAAAASHZJ2AU>.
You are receiving this because you authored the thread.Message ID: ***@***.***>


If you're having an error then you're not running your code in the right environment.
Title: Warn about checksums
Author: lhoestq (MEMBER)
State: closed
Created: 2022-11-22T10:58:48Z

It takes a lot of time on big datasets to compute the checksums, we should at least add a warning to notify the user about this step. I also mentioned how to disable it, and added a tqdm bar (delay=5 seconds)

cc @ola13

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I'm also in favor of disabling this by default - it's kinda impractical

Great, thanks for the quick turnaround on this!
Title: load_dataset does not read jsonl metadata file properly
Author: 065294847 (NONE)
State: closed
Created: 2022-11-22T10:24:46Z

### Describe the bug

Hi, I'm following [this page](https://huggingface.co/docs/datasets/image_dataset) to create a dataset of images and captions via an image folder and a metadata.json file, but I can't seem to get the dataloader to recognize the "text" column. It just spits out "image" and "label" as features. 

Below is code to reproduce my exact example/problem.

### Steps to reproduce the bug

```ruby 
dataset_link="19Unu89Ih_kP6zsE7f9Mkw8dy3NwHopRF"
id = dataset_link
output = 'Godardv01.zip'
gdown.download(id=id, output=output, quiet=False)

ds = load_dataset("imagefolder", data_dir="/kaggle/working/Volumes/TOSHIBA/Godard_imgs/Volumes/TOSHIBA/Godard_imgs/Full/train", split="train", drop_labels=False)
print(ds)
```

### Expected behavior

I would expect that it returned "image" and "text" columns from the code above.

### Environment info

- `datasets` version: 2.1.0
- Platform: Linux-5.15.65+-x86_64-with-debian-bullseye-sid
- Python version: 3.7.12
- PyArrow version: 5.0.0
- Pandas version: 1.3.5

Comments:
Can you try to remove "drop_labels=false" ? It may force the loader to infer the labels instead of reading the metadata

Hi, thanks for responding. I tried that, but it does not change anything.

Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Probably the issue, will report back asap!

Okay, now it seems to actually load the metadata and create the train_split, but it still says only returns "image" and "label", which is always 0 since all images are from same folder

> Can you try updating `datasets` ? Metadata support was added in `datasets` 2.4

Update: This was the issue.
Title: Remove YAML integer keys from class_label metadata
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-22T08:34:07Z

Fix partially #5275.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Also note that this approach is valid when metadata keys are str, but also if they are int.
- This will be helpful for any community dataset using old integer keys in their metadata

perfect !
Title: load_dataset possibly broken for gated datasets?
Author: TristanThrush (CONTRIBUTOR)
State: closed
Created: 2022-11-21T21:59:53Z

### Describe the bug

When trying to download the [winoground dataset](https://huggingface.co/datasets/facebook/winoground), I get this error unless I roll back the version of huggingface-hub:

```
[/usr/local/lib/python3.7/dist-packages/huggingface_hub/utils/_validators.py](https://localhost:8080/#) in validate_repo_id(repo_id)
    165     if repo_id.count("/") > 1:
    166         raise HFValidationError(
--> 167             "Repo id must be in the form 'repo_name' or 'namespace/repo_name':"
    168             f" '{repo_id}'. Use `repo_type` argument if needed."
    169         )

HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'datasets/facebook/winoground'. Use `repo_type` argument if needed
```

### Steps to reproduce the bug

Install requirements:

```
pip install transformers
pip install datasets
# It works if you uncomment the following line, rolling back huggingface hub:
# pip install huggingface-hub==0.10.1
```

Then:

```
from datasets import load_dataset
auth_token = ""  # Replace with an auth token, which you can get from your huggingface account: Profile -> Settings -> Access Tokens -> New Token
winoground = load_dataset("facebook/winoground", use_auth_token=auth_token)["test"]
```





### Expected behavior

Downloading of the datset

### Environment info

Just a google colab; see here: https://colab.research.google.com/drive/15wwOSte2CjTazdnCWYUm2VPlFbk2NGc0?usp=sharing

Comments:
@BradleyHsu

Btw, thanks very much for finding the hub rollback temporary fix and bringing the issue to our attention @KhoomeiK!

I see the same issue when calling `load_dataset('poloclub/diffusiondb', 'large_random_1k')` with `datasets==2.7.1` and `huggingface-hub=0.11.0`. No issue with `datasets=2.6.1` and `huggingface_hub==0.10.1`.

https://github.com/poloclub/diffusiondb/issues/7

I fixed my issue by specifying `repo_type` in `hf_hub_url()`. https://github.com/poloclub/diffusiondb/commit/9eb91c79aaca98b0515a0ce45778b8af65b84652

I opened a PR on the Winoground's repo: https://huggingface.co/datasets/facebook/winoground/discussions/2

This is a bug in the script, indeed. The most robust fix is to use a relative path instead of `hf_hub_url`, which does not depend on `huggingface_hub`'s version 🙂. I've opened a PR here: https://huggingface.co/datasets/facebook/winoground/discussions/3.

Awesome, big thanks to both @xiaohk and @mariosasko!

so, if i reproduce the bug, what should i do ? with huggingface_hub0.13.3  dataset2.6.1

huggingface_hub.utils._validators.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name':

    tokenizer = AutoTokenizer.from_pretrained(ARGS.model_path, trust_remote_code=True)

Please handle automatically for local path and repo name inside, otherwise users always get confused about this

I think I'm also hitting this error, trying to load a model from a local path.
Title: download_mode="force_redownload" does not refresh cached dataset
Author: nomisto (NONE)
State: open
Created: 2022-11-21T14:12:43Z

### Describe the bug

`load_datasets` does not refresh dataset when features are imported from external file, even with `download_mode="force_redownload"`. The bug is not limited to nested fields, however it is more likely to occur with nested fields.

### Steps to reproduce the bug

To reproduce the bug 3 files are needed: `dataset.py` (contains dataset loading script), `schema.py` (contains features of dataset) and `main.py` (to run `load_datasets`)

`dataset.py`
```python
import datasets
from schema import features

class NewDataset(datasets.GeneratorBasedBuilder):

    def _info(self):
        return datasets.DatasetInfo(
            features=features
        )

    def _split_generators(self, dl_manager):
        return [
            datasets.SplitGenerator(
                name=datasets.Split.TRAIN
            )
        ]

    def _generate_examples(self):
        data = [
            {"id": 0, "nested": []},
            {"id": 1, "nested": []}
        ]
        for key, example in enumerate(data):
            yield key, example
```

`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"text": datasets.Value("string")}
        ]
    }
)
```

`main.py`
```python
import datasets
a = datasets.load_dataset("dataset.py")
print(a["train"].info.features)
```

Now if `main.py` is run it prints the following correct output: `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`. However, if f.e. the label of the feature "text" is changed to something else, f.e. to


`schema.py`
```python
import datasets
features = datasets.Features(
    {
        "id": datasets.Value("int32"),
        "nested": [
            {"textfoo": datasets.Value("string")}
        ]
    }
)
```

 `main.py` still prints `{'id': Value(dtype='int32', id=None), 'nested': [{'text': Value(dtype='string', id=None)}]}`, even if run with `download_mode="force_redownload"`. The only fix is to delete the folder in the cache.

### Expected behavior

The cached dataset is deleted and refreshed when using `load_datasets` with `download_mode="force_redownload"`.

### Environment info

- `datasets` version: 2.7.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.7.9
- PyArrow version: 10.0.0
- Pandas version: 1.3.5

Comments:
Title: Fix #5269
Author: Freed-Wu (NONE)
State: closed
Created: 2022-11-20T07:50:49Z

```
$ datasets-cli convert --datasets_directory <TAB>
datasets_directory
benchmarks/  docs/        metrics/     notebooks/   src/         templates/   tests/       utils/
```

Comments:
See <https://github.com/huggingface/datasets/issues/5269>
Title: Shell completions
Author: Freed-Wu (NONE)
State: closed
Created: 2022-11-19T13:48:59Z
Labels: enhancement

### Feature request

Like <https://github.com/huggingface/huggingface_hub/issues/1197>, datasets-cli maybe need it, too.

### Motivation

See above.

### Your contribution

Maybe.

Comments:
I don't think we need completion on the datasets-cli, since we're mainly developing huggingface-cli

I see.
Title: Sharded save_to_disk + multiprocessing
Author: lhoestq (MEMBER)
State: closed
Created: 2022-11-18T18:50:01Z

Added `num_shards=` and `num_proc=` to `save_to_disk()`

EDIT: also added `max_shard_size=` to `save_to_disk()`, and also `num_shards=` to `push_to_hub`

I also:
- deprecated the fs parameter in favor of storage_options (for consistency with the rest of the lib) in save_to_disk and load_from_disk
- always embed the image/audio data in arrow when doing `save_to_disk`
- added a tqdm bar in `save_to_disk`
- Use the MockFileSystem in tests for `save_to_disk` and `load_from_disk`
- removed the unused integration tests with S3, since we can now test with `mockfs` instead of `s3fs`

TODO:
- [x] implem save_to_disk for dataset dict
- [x] save_to_disk for dataset dict tests
- [x] deprecate fs in dataset dict load_from_disk as well
- [x] update docs

Close #5263 
Close https://github.com/huggingface/datasets/issues/4196
Close https://github.com/huggingface/datasets/issues/4351

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Added both num_shards and max_shard_size in push_to_hub/save_to_disk. Will take care of updating the tests later

It's ready for a final review @mariosasko and @albertvillanova, let me know what you think :)

Took your comments into account, and also changed `iflatmap_unordered` to take an iterable of kwargs to make the code more redable :)
Title: Specify arguments as keywords in librosa.reshape to avoid future errors 
Author: polinaeterna (CONTRIBUTOR)
State: closed
Created: 2022-11-18T14:58:47Z

Fixes a warning and future deprecation from `librosa.reshape`:
```
FutureWarning: Pass orig_sr=16000, target_sr=48000 as keyword args. From version 0.10 passing these as positional arguments will result in an error

   array = librosa.resample(array, sampling_rate, self.sampling_rate, res_type="kaiser_best")
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Get an IterableDataset from a map-style Dataset
Author: lhoestq (MEMBER)
State: closed
Created: 2022-11-18T14:54:40Z
Labels: enhancement

streaming

This is useful to leverage iterable datasets specific features like:
- fast approximate shuffling
- lazy map, filter etc.

Iterating over the resulting iterable dataset should be at least as fast at iterating over the map-style dataset.

Here are some ideas regarding the API:
```python
# 1. 
# - consistency with load_dataset(..., streaming=True)
# - gives intuition that map/filter/etc. are done on-the-fly
ids = ds.stream()

# 2.
# - more explicit on the output type
# - but maybe sounds like a conversion tool rather than a step in a processing pipeline
ids = ds.as_iterable_dataset()
```

Comments:
I think `stream` could be misleading since the data is not being streamed from remote endpoints (one could think that's the case when they see `load_dataset` followed by `stream`). Hence, I prefer the second option.

PS: When we resolve https://github.com/huggingface/datasets/issues/4542, we could add `as_tf_dataset` to the API for consistency and deprecate `to_tf_dataset`.
Title: AttributeError: 'Value' object has no attribute 'names'
Author: emnaboughariou (NONE)
State: closed
Created: 2022-11-18T13:58:42Z

Hello

I'm trying to build a model for custom token classification 
I already followed the token classification course on huggingface

while adapting the code to my work, this message occures : 
'Value' object has no attribute 'names'

Here's my code:
`raw_datasets` 
generates
DatasetDict({
    train: Dataset({
        features: ['isDisf', 'pos', 'tokens', 'id'],
        num_rows: 14
    })
})

`raw_datasets["train"][3]["isDisf"]`
generates 
['B_RM', 'I_RM', 'I_RM', 'B_RP', 'I_RP', 'O', 'O']

`dis_feature = raw_datasets["train"].features["isDisf"]
dis_feature`
generates
Sequence(feature=Value(dtype='string', id=None), length=-1, id=None)

and
`label_names = dis_feature.feature.names
label_names`
generates
AttributeError                            Traceback (most recent call last)
[<ipython-input-28-972fd54a869a>](https://localhost:8080/#) in <module>
----> 1 label_names = dis_feature.feature.names
      2 label_names

AttributeError: 'Value' object has
AttributeError: 'Value' object has no attribute 'names'

Thank you for your help

Comments:
Hi ! It looks like your "isDif" column is a Sequence of Value("string"), not a Sequence of ClassLabel.

You can convert your Value("string") feature type to a ClassLabel feature type this way:
```python
from datasets import ClassLabel, Sequence

# provide the label_names yourself
label_names = [...]
# OR get them from the dataset
label_names = sorted(set(label for labels in raw_datasets["train"]["isDif"] for label in labels))

# Cast to ClassLabel
raw_datasets = raw_datasets.cast_column("isDif", Sequence(ClassLabel(names=label_names)))
```


thank you 
it works 💯
Title: Add PubTables-1M
Author: NielsRogge (CONTRIBUTOR)
State: open
Created: 2022-11-18T07:56:36Z
Labels: dataset request

### Name

PubTables-1M

### Paper

https://openaccess.thecvf.com/content/CVPR2022/html/Smock_PubTables-1M_Towards_Comprehensive_Table_Extraction_From_Unstructured_Documents_CVPR_2022_paper.html

### Data

https://github.com/microsoft/table-transformer

### Motivation

Table Transformer is now available in 🤗 Transformer, and it was trained on PubTables-1M. It's a large dataset for table extraction and structure recognition in unstructured documents.

Comments:
cc @albertvillanova the author would like to add this dataset to the hub: https://github.com/microsoft/table-transformer/issues/68#issuecomment-1319114621. Could you help him out?
Title: Restore order of split names in dataset_info for canonical datasets
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-11-17T15:13:15Z
Labels: dataset contribution

After a bulk edit of canonical datasets to create the YAML `dataset_info` metadata, the split names were accidentally sorted alphabetically. See for example:
- https://huggingface.co/datasets/bc2gm_corpus/commit/2384629484401ecf4bb77cd808816719c424e57c

Note that this order is the one appearing in the preview of the datasets.

I'm making a bulk edit to align the order of the splits appearing in the metadata info with the order appearing in the loading script.

Related to:
- #5202

Comments:
The bulk edit is running...

See for example: 
- A single config: https://huggingface.co/datasets/acronym_identification/discussions/2
- Multiple configs: https://huggingface.co/datasets/babi_qa/discussions/1

TODO: Add "dataset_info" YAML metadata to:
- [x] "chr_en" has no metadata JSON file, nor "dataset_info" YAML tag in its card
  - Fixing PR: https://huggingface.co/datasets/chr_en/discussions/1 
- [x] "conll2000" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "crime_and_punish" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "dart" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [x] "iwslt2017" has no metadata JSON file, but it has "dataset_info" YAML tag in its card
- [ ] "mc4" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "the_pile" has no metadata JSON file, nor "dataset_info" YAML tag in its card
- [ ] "timit_asr" has no metadata JSON file, nor "dataset_info" YAML tag in its card

The bulk edit is finished.