rIsHu009/github-issues-updated huggingface dataset:
Title: Fix hashing for python 3.9
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-16T16:42:31Z

In python 3.9, pickle hashes the `glob_ids` dictionary in addition to the `globs` of a function.

Therefore the test at `tests/test_fingerprint.py::RecurseDumpTest::test_recurse_dump_for_function_with_shuffled_globals` is currently failing for python 3.9

To make hashing deterministic when the globals are not in the same order, we also need to make the order of `glob_ids` deterministic.

Right now we don't have a CI to test python 3.9 but we should definitely have one. For this PR in particular I ran the tests locally using python 3.9 and they're passing now.

Fix https://github.com/huggingface/datasets/issues/4506

Comments:
_The documentation is not available anymore as the PR was closed or merged._

What do you think @albertvillanova ?
Title: Add uppercased versions of image file extensions for automatic module inference
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-06-16T14:14:49Z

Adds the uppercased versions of the image file extensions to the supported extensions. 

Another approach would be to call `.lower()` on extensions while resolving data files, but uppercased extensions are not something we want to encourage out of the box IMO unless they are commonly used (as they are in the vision domain)

Note that there is a slight discrepancy between the image file resolution and `imagefolder` as the latter calls `.lower()` on file extensions leading to some image file extensions being ignored by the resolution but not by the loader (e.g. `pNg`). Such extensions should also be discouraged, so I'm ignoring that case too.

Fix #4514.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Allow .JPEG as a file extension
Author: DiGyt (NONE)
State: closed
Created: 2022-06-16T12:36:20Z
Labels: bug

## Describe the bug
When loading image data, HF datasets seems to recognize `.jpg` and `.jpeg` file extensions, but not e.g. .JPEG. As the naming convention .JPEG is used in important datasets such as imagenet, I would welcome if according extensions like .JPEG or .JPG would be allowed.

## Steps to reproduce the bug
```python
# use bash to create 2 sham datasets with jpeg and JPEG ext
!mkdir dataset_a
!mkdir dataset_b
!wget https://upload.wikimedia.org/wikipedia/commons/7/71/Dsc_%28179253513%29.jpeg -O example_img.jpeg
!cp example_img.jpeg ./dataset_a/
!mv example_img.jpeg ./dataset_b/example_img.JPEG

from datasets import load_dataset

# working
df1 = load_dataset("./dataset_a", ignore_verifications=True)

#not working
df2 = load_dataset("./dataset_b", ignore_verifications=True)

# show
print(df1, df2)
```

## Expected results
```
DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
}) DatasetDict({
    train: Dataset({
        features: ['image', 'label'],
        num_rows: 1
    })
})
```

## Actual results
```
FileNotFoundError: Unable to resolve any data file that matches '['**']' at /..PATH../dataset_b with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I know that it can be annoying to allow seemingly arbitrary numbers of file extensions. But I think this one would be really welcome.

Comments:
Hi, thanks for reporting!  I've opened a PR with the fix.

Wow, that was quick! Thank you very much 🙏
Title: Update Google Cloud Storage documentation and add Azure Blob Storage example
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-06-16T11:46:09Z
Labels: documentation

While I was going through the 🤗 Datasets documentation of the Cloud storage filesystems at https://huggingface.co/docs/datasets/filesystems, I realized that the Google Cloud Storage documentation could be improved e.g. bullet point says "Load your dataset" when the actual call was to "Save your dataset", in-line code comment was mentioning "s3 bucket" instead of "gcs bucket", and some more in-line comments could be included.

Also, I think that mixing Google Cloud Storage documentation with AWS S3's one was a little bit confusing, so I moved all those to the end of the document under an h2 tab named "Other filesystems", with an h3 for "Google Cloud Storage".

Besides that, I was currently working with Azure Blob Storage and found out that the URL to [adlfs](https://github.com/fsspec/adlfs) was common for both filesystems Azure Blob Storage and Azure DataLake Storage, as well as the URL, which was updated even though the redirect was working fine, so I decided to group those under the same row in the column of supported filesystems.

And took also the change to add a small documentation entry as for Google Cloud Storage but for Azure Blob Storage, as I assume that AWS S3, GCP Cloud Storage, and Azure Blob Storage, are the most used cloud storage providers.

Let me know if you're OK with these changes, or whether you want me to roll back some of those! :hugs:

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi @stevhliu, I've kept the `>>>` before all the in-line code comments as it was done like that in the default S3 example that was already there, I assume that it's done like that just for readiness, let me know whether we should remove the `>>>` in the Python blocks before the in-line code comments or keep them.

![image](https://user-images.githubusercontent.com/36760800/174254663-b68d28d2-eae1-40f3-8695-dc4b0c3b479a.png)


Comments are ignored by doctest, so I think we can remove the `>>>` :)

Cool I'll remove those now 👍🏻

Sure @lhoestq, I just kept that structure as that was the more similar one to the one that was already there, but we can go with that approach, just let me know whether I should change the headers so as to leave all those providers in the same level (`h2`). Thanks!
Title: Add links to vision tasks scripts in ADD_NEW_DATASET template
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-06-16T10:35:35Z

Add links to vision dataset scripts in the ADD_NEW_DATASET template.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The CI failure is unrelated to the PR's changes. Merging.
Title: Support all negative values in ClassLabel
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-16T09:59:39Z

We usually use -1 to represent a missing label, but we should also support any negative values (some users use -100 for example). This is a regression from `datasets` 2.3

Fix https://github.com/huggingface/datasets/issues/4508

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for this fix! I'm not sure what the release timeline is, but FYI #4508 is a breaking issue for transformer token classification using Trainer and PyTorch. PyTorch defaults to -100 as the ignored label for [negative log loss](https://pytorch.org/docs/stable/generated/torch.nn.NLLLoss.html?highlight=nllloss#torch.nn.NLLLoss), so switching labels to -1 leads to index errors using Trainer defaults.

As a workaround, I'm using master branch directly (`pip install git+https://github.com/huggingface/datasets.git@master` for anyone who needs to do the same) until this gets released.

The new release `2.4` fixes the issue, feel free to update `datasets` :) 
```
pip install -U datasets
```
Title: Add regression test for `ArrowWriter.write_batch` when batch is empty
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-06-16T08:53:51Z

As spotted by @cccntu in #4502, there's a logic bug in `ArrowWriter.write_batch` as the if-statement to handle the empty batches as detailed in the docstrings of the function ("Ignores the batch if it appears to be empty, preventing a potential schema update of unknown types."), the current if-statement is not handling properly `writer.write_batch({})` as an error is triggered.

Also, if we add a regression test in `test_arrow_writer.py::test_write_batch` before applying the fix, the test will fail as when trying to write an empty batch as follows:

```
=================================================================================== short test summary info ===================================================================================
FAILED tests/test_arrow_writer.py::test_write_batch[None-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[None-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields1-10] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-None] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-1] - ValueError: Schema and number of arrays unequal
FAILED tests/test_arrow_writer.py::test_write_batch[fields2-10] - ValueError: Schema and number of arrays unequal
======================================================================== 9 failed, 73 deselected, 7 warnings in 0.81s =========================================================================
```

So the batch is not ignored when empty, as `batch_examples={}` won't match the condition `if batch_examples: ...`.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

As mentioned by @lhoestq, the current behavior is correct and we should not expect batches with different columns, in that case, the if should fail, as the values of the batch can be empty, but not the actual `batch_examples` value.
Title: Support skipping Parquet to Arrow conversion when using Beam
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-16T08:25:38Z


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4509). All of your documentation changes will be reflected on that endpoint.

When #4724 is merged, we can just pass `file_format="parquet"` to `download_and_prepare` and it will output parquet fiels without converting to arrow

I think we can close this one
Title: cast_storage method from datasets.features
Author: romainremyb (NONE)
State: closed
Created: 2022-06-15T20:47:22Z
Labels: bug

## Describe the bug
A bug occurs when mapping a function to a dataset object. I ran the same code with the same data yesterday and it worked just fine. It works when i run locally on an old version of datasets.

## Steps to reproduce the bug
Steps are:
- load whatever datset
- write a preprocessing function such as "tokenize_and_align_labels" written in https://huggingface.co/docs/transformers/tasks/token_classification
- map the function on dataset and get "ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

# Sample code to reproduce the bug
def tokenize_and_align_labels(examples):
    tokenized_inputs = tokenizer(examples["tokens"], truncation=True, is_split_into_words=True, max_length=38,padding="max_length")

    labels = []
    for i, label in enumerate(examples[f"labels"]):
        word_ids = tokenized_inputs.word_ids(batch_index=i)  # Map tokens to their respective word.
        previous_word_idx = None
        label_ids = []
        for word_idx in word_ids:  # Set the special tokens to -100.
            if word_idx is None:
                label_ids.append(-100)
            elif word_idx != previous_word_idx:  # Only label the first token of a given word.
                label_ids.append(label[word_idx])
            else:
                label_ids.append(-100)
            previous_word_idx = word_idx
        labels.append(label_ids)

    tokenized_inputs["labels"] = labels
    return tokenized_inputs

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")
dt = dataset.map(tokenize_and_align_labels, batched=True)

## Expected results
New dataset objects should load and do on older versions.

## Actual results
"ValueError: Class label -100 less than -1" from cast_storage method from datasets.features

## Environment info
everything works fine on older installations of datasets/transformers

Issue arises when installing datasets on google collab under python3.7
I can't manage to find the exact output you're requirering but version printed is datasets-2.3.2

Comments:
Hi! We've recently added a check to the `ClassLabel` type to ensure the values are in the valid label range `-1, 0, ..., num_classes-1` (-1 is used for missing values). The error in your case happens only if the `labels` column is of type `Sequence(ClassLabel(...))` before the `map` call and can be avoided by calling `dataset = dataset.cast_column("labels", Sequence(Value("int")))` beforehand. The token-classification examples in Transformers introduce a new `labels` column, so their type is also `Sequence(Value("int"))`, which doesn't lead to an error as this type unbounded. 

I'm fine with re-adding support for all negative values for unknown/missing labels @mariosasko, wdyt ?
Title: How to let `load_dataset` return a `Dataset` instead of `DatasetDict` in customized loading script
Author: liyucheng09 (NONE)
State: closed
Created: 2022-06-15T18:56:34Z
Labels: enhancement

If the dataset does not need splits, i.e., no training and validation split, more like a table. How can I let the `load_dataset` function return a `Dataset` object directly rather than return a `DatasetDict` object with only one key-value pair.

Or I can paraphrase the question in the following way: how to skip `_split_generators` step in `DatasetBuilder` to let `as_dataset` gives a single `Dataset` rather than a list`[Dataset]`?

Many thanks for any help.

Comments:
Hi @liyucheng09.

Users can pass the `split` parameter to `load_dataset`. For example, if your split name is "train",
```python
ds = load_dataset("dataset_name", split="train")
```
will return a `Dataset` instance.

@albertvillanova Thanks! I can't believe I didn't know this feature till now.
Title: Fix double dots in data files
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-15T16:31:04Z

As mentioned in https://github.com/huggingface/transformers/pull/17715 `data_files` can't find a file if the path contains double dots `/../`. This has been introduced in https://github.com/huggingface/datasets/pull/4412, by trying to ignore hidden files and directories (i.e. if they start with a dot)

I fixed this and added a test

cc @sgugger @ydshieh

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The CI fails are unrelated to this PR (apparently something related to `seqeval` on windows) - merging :)
Title: Refactor and add metadata to fever dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-15T14:59:47Z

Related to: #4452 and #3792.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

But this is somehow fever v3 dataset (see this link https://fever.ai/ under the dropdown menu called Datasets). Our fever dataset already contains v1 and v2 configs. Then, I added this as if v3 config (but named feverous instead of v3 to align with the original naming by data owners).

In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

> In any case, if you really think this should be a new dataset, then I would propose to create it on the Hub instead, as "fever/feverous".

Yea makes sense ! thanks :) let's push more datasets on the hub rather than on github from now on

I have added "feverous" dataset to the Hub: https://huggingface.co/datasets/fever/feverous

I change the name of this PR accordingly, as now it only:
- Refactors code and include for both Fever v1.0 and v2.0 specific:
  - Descriptions
  - Citations
  - Homepages
- Updates documentation card aligned with above:
  - It was missing v2.0 description and citation.
- Update metadata JSON
Title: Add `concatenate_datasets` for iterable datasets
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-15T13:58:50Z

`concatenate_datasets` currently only supports lists of `datasets.Dataset`, not lists of `datasets.IterableDataset` like `interleave_datasets`

Fix https://github.com/huggingface/datasets/issues/2564

I also moved `_interleave_map_style_datasets` from combine.py to arrow_dataset.py, since the logic depends a lot on the `Dataset` object internals

And I moved `concatenate_datasets` from arrow_dataset.py to combine.py to have it with `interleave_datasets` (though it's also copied in arrow_dataset module for backward compatibility for now)

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks ! I addressed your comments :)

> There is a slight difference in concatenate_datasets between the version for map-style datasets and the one for iterable datasets

Indeed, here is what I did to fix this:

- axis 0: fill missing columns with None.
(I first iterate over the input datasets to infer their columns from the first examples, then I set the features of the resulting dataset to be the merged features)
This is consistent with non-streaming concatenation

- axis 1: **fill the missing rows with None**, for consistency with axis 0
(but let me know what you think, I can still revert this behavior and raise an error when one of the dataset runs out of examples)
We might have to align the non-streaming concatenation with this behavior though, for consistency. What do you think ?

Added more comments as suggested, and some typing

While factorizing _apply_features_types for both IterableDataset and TypedExamplesIterable, I  fixed a missing `token_per_repo_id` that was not passed to TypedExamplesIteable

Let me know what you think now @mariosasko
Title: fix ETT m1/m2 test/val dataset
Author: kashif (CONTRIBUTOR)
State: closed
Created: 2022-06-15T11:51:02Z

https://huggingface.co/datasets/ett/discussions/1

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thansk for the fix ! Can you regenerate the datasets_infos.json please ? This way it will update the expected number of examples in the test and val splits

ah yes!
Title: WER and CER > 1
Author: sadrasabouri (NONE)
State: closed
Created: 2022-06-15T11:35:12Z
Labels: bug

## Describe the bug
It seems that in some cases in which the `prediction` is longer than the `reference` we may have word/character error rate higher than 1 which is a bit odd.

If it's a real bug I think I can solve it with a PR changing [this](https://github.com/huggingface/datasets/blob/master/metrics/wer/wer.py#L105) line to
```python
return min(incorrect / total, 1.0)
```

## Steps to reproduce the bug
```python
from datasets import load_metric
wer = load_metric("wer")
wer_value = wer.compute(predictions=["Hi World vka"], references=["Hello"])
print(wer_value)
```

## Expected results
```
1.0
```

## Actual results
```
3.0
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 2.3.0
- Platform: Linux-5.4.188+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.13
- PyArrow version: 6.0.1
- Pandas version: 1.3.5

Comments:
WER can have values bigger than 1.0, this is expected when there are too many insertions

From [wikipedia](https://en.wikipedia.org/wiki/Word_error_rate):
> Note that since N is the number of words in the reference, the word error rate can be larger than 1.0
Title: Re-add download_manager module in utils
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-15T09:44:33Z

https://github.com/huggingface/datasets/pull/4384 moved `datasets.utils.download_manager` to `datasets.download.download_manager`

This breaks `evaluate` which imports `DownloadMode` from `datasets.utils.download_manager` 

This PR re-adds `datasets.utils.download_manager` without circular imports.

We could also show a message that says that accessing it is deprecated, but I think we can do this in a subsequent PR, and just focus on doing a patch release for now

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the fix.

I'm wondering how this fixes backward compatibility...

Executing this code:
```python
from datasets.utils.download_manager import DownloadMode
```
we will have
```python
DownloadMode = None
```

If afterwards we use something like:
```python
if download_mode == DownloadMode.FORCE_REDOWNLOAD
```
that will raise an exception.

It works fine on my side:
```python
>>> from datasets.utils.download_manager import DownloadMode
>>> DownloadMode is not None
True
```

As reported in https://github.com/huggingface/evaluate/pull/143
```python
from datasets.utils import DownloadConfig
```
is also missing, I'm re-adding it

Took the liberty of merging this one, to do a patch release soon. If we think of a better approach we can improve it later
Title: Replace `assertEqual` with `assertTupleEqual` in unit tests for verbosity
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-06-15T09:29:16Z

As detailed in #4419 and as suggested by @mariosasko, we could replace the `assertEqual` assertions with `assertTupleEqual` when the assertion is between Tuples, in order to make the tests more verbose.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

FYI I used the following regex to look for the `assertEqual` statements where the assertion was being done over a Tuple: `self.assertEqual(.*, \(.*,)(\)\))$`, hope this is useful!
Title: Fix patching module that doesn't exist
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-15T08:17:50Z

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

Bug introduced by #4375

Fix https://github.com/huggingface/datasets/issues/4494

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Patching fails for modules that are not installed or don't exist
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-15T08:17:29Z

Reported in https://github.com/huggingface/huggingface_hub/runs/6894703718?check_suite_focus=true

When trying to patch `scipy.io.loadmat`:

```python
ModuleNotFoundError: No module named 'scipy'
```

Instead it shouldn't raise an error and do nothing

We use patching to extend such functions to support remote URLs and work in streaming mode

Comments:
Title: Add `@transmit_format` in `flatten`
Author: alvarobartt (MEMBER)
State: closed
Created: 2022-06-14T20:09:09Z

As suggested by @mariosasko in https://github.com/huggingface/datasets/pull/4411, we should include the `@transmit_format` decorator to `flatten`, `rename_column`, and `rename_columns` so as to ensure that the value of `_format_columns` in an `ArrowDataset` is properly updated.

**Edit**: according to @mariosasko comment below, the decorator `@transmit_format` doesn't handle column renaming, so it's done manually for those instead.

Comments:
@mariosasko please let me know whether we need to include some sort of tests to make sure that the decorator is working as expected. Thanks! 🤗 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/datasets/pr_4493). All of your documentation changes will be reflected on that endpoint.

Hi, thanks for working on this! Yes, please add (simple) tests so we can avoid any unexpected behavior in the future.

`@transmit_format` doesn't handle column renaming, so I removed it from `rename_column` and `rename_columns` and added a comment to explain this.

Oops, I thought this PR was already merged and deleted from the source repository, I'll be creating a new branch out of `main` so as to re-create this PR... My bad :weary:
Title: Pin the revision in imagenet download links
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-14T17:15:17Z

Use the commit sha in the data files URLs of the imagenet-1k download script, in case we want to restructure the data files in the future. For example we may split it into many more shards for better paralellism.

cc @mariosasko

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Dataset Viewer issue for Pavithree/test
Author: Pavithree (NONE)
State: closed
Created: 2022-06-14T13:23:10Z
Labels: dataset-viewer

### Link

https://huggingface.co/datasets/Pavithree/test

### Description

I have extracted the subset of original eli5 dataset found at hugging face. However, while loading the dataset It throws ArrowNotImplementedError: Unsupported cast from string to null using function cast_null error. Is there anything missing from my end? Kindly help.

### Owner

_No response_

Comments:
This issue can be resolved according to this post https://stackoverflow.com/questions/70566660/parquet-with-null-columns-on-pyarrow. It looks like first data entry in the json file must not have any null values as pyarrow uses this first file to infer schema for entire dataset.
Title: Use `torch.nested_tensor` for arrays of varying length in torch formatter
Author: mariosasko (COLLABORATOR)
State: open
Created: 2022-06-14T12:19:40Z
Labels: enhancement

Use `torch.nested_tensor` for arrays of varying length in `TorchFormatter`.

The PyTorch API of nested tensors is in the prototype stage, so wait for it to become more mature.

Comments:
What's the current behavior?

Currently, we return a list of Torch tensors if their shapes don't match. If they do, we consolidate them into a single Torch tensor.
Title: Add SV-Ident dataset
Author: e-tornike (NONE)
State: closed
Created: 2022-06-14T12:09:00Z


Comments:
Hi @e-tornike, thanks a lot for adding this interesting dataset.

Recently at Hugging Face, we have decided to give priority to adding datasets directly on the Hub. Would you mind to transfer your loading script to the Hub? You could create a dedicated org namespace, so that your dataset would be accessible using `vadis/sv_ident` or `sdproc/sv_ident` or `coling/sv_ident` (as you prefer).

You have an example here: https://huggingface.co/datasets/projecte-aina/catalan_textual_corpus

Additionally, please feel free to ping us if you need assistance/help in creating this dataset.

You could put the link to your Hub dataset here in this Issue discussion page, so that we can follow the progress. :)

Hi @albertvillanova, thanks for the feedback! Uploading via the Hub is a lot easier! 

I've uploaded the dataset here: https://huggingface.co/datasets/vadis/sv-ident, but it's reporting a "Status400Error". Is there any way to see the logs of the dataset script and what is causing the error?

Hi @e-tornike, good job at https://huggingface.co/datasets/vadis/sv-ident.

Normally, you can run locally the loading of the dataset by passing `streaming=True` (as the previewer does):
```python
ds = load_dataset("path/to/sv_ident.py, split="train", streaming=True)
item = next(iter(ds))
item
```

Let me have a look and open a discussion on your Hub repo! ;)

I've opened an Issue: 
- #4527
Title: Update PASS dataset version
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-06-14T10:47:14Z

Update the PASS dataset to version v3 (the newest one) from the [version history](https://github.com/yukimasano/PASS/blob/main/version_history.txt).

PS: The older versions are not exposed as configs in the script because v1 was removed from Zenodo, and the same thing will probably happen to v2.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add CCAgT dataset
Author: johnnv1 (NONE)
State: closed
Created: 2022-06-13T14:20:19Z

As described in #4075 

I could not generate the dummy data. Also, on the data repository isn't provided the split IDs, but I copy the functions that provide the correct data split. In summary, to have a better distribution, the data in this dataset should be separated based on the amount of NORs in each image.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hi! Excellent job @johnnv1! There were typos/missing words in the card, so I took the liberty to rewrite some parts to make them easier to understand. Let me know if you are ok with the changes. Also, feel free to add some info under the `Who are the annotators?` section.

Additionally, I fixed the issue with streaming and renamed the `digits` feature to `objects`.

@lhoestq Are you ok with skipping the dummy data test here as it's tricky to generate it due to the splits separation logic?

I think I can also add instance segmentation: by exposing the segment of each instance, so it will be similar with object detection:

- `instances`: a dictionary containing bounding boxes, segments, and labels of the cell objects 
  - `bbox`: a list of bounding boxes
  - `segment`: a list of segments in format of `[polygon]`, where each polygon is `[x0, y0, ..., xn, yn]`
  - `label`: a list of integers representing the category

Do you think it would be ok?

Don't you think it makes sense to keep the same category IDs for all approaches? 

Now we have:
 - nucleus category ID equals 0 for object detection and instance segmentation
 - background category ID equals 0 (on the masks) for semantic segmentation

I find it weird to have a dummy label in object detection just to align the mapping with semantic segmentation. Instead, let's explain in the card (under Data Fields -> annotation) what the pixel values mean (background + object detection labels)

Ok, I can do that in the next few days. I will create a `lapix` organization, and I will add this dataset and also #4565

So, I think we can close this PR? I have already moved these files there.

The link of CCAgT dataset is: https://huggingface.co/datasets/lapix/CCAgT

🤗 

Woohoo awesome !

closing this PR :)
Title: Fix cast to null
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-13T13:44:32Z

It currently fails with `ArrowNotImplementedError` instead of `TypeError` when one tries to cast integer to null type.

Because if this, type inference breaks when one replaces null values with integers in `map` (it first tries to cast to the previous type before inferring the new type).

Fix https://github.com/huggingface/datasets/issues/4483

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Better ImportError message when a dataset script dependency is missing
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-13T12:44:37Z

When a depenency is missing for a dataset script, an ImportError message is shown, with a tip to install the missing dependencies. This message is not ideal at the moment: it may show duplicate dependencies, and is not very readable.

I improved it from
```
ImportError: To be able to use bigbench, you need to install the following dependencies['bigbench', 'bigbench', 'bigbench', 'bigbench'] using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz" bigbench bigbench bigbench' for instance'
```
to
```
ImportError: To be able to use bigbench, you need to install the following dependency: bigbench.
Please install it using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz"' for instance'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Discussed offline with @mariosasko, merging :)

Fwiw, i think this same issue is occurring on the datasets website page, where preview isn't available due to the `bigbench` import error

For the preview of BigBench datasets, we're just waiting for bigbench to have a stable version on PyPI, instead of the one hosted on GCS ;)
Title: Test that TensorFlow is not imported on startup
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-13T10:33:49Z

TF takes some time to be imported, and also uses some GPU memory.

I just added a test to make sure that in the future it's never imported by default when
```python
import datasets
```
is called.

Right now this fails because `huggingface_hub` does import tensorflow (though this is fixed now on their `main` branch)

I'll mark this PR as ready for review once `huggingface_hub` has a new release

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Should we close this PR?

I'm closing this PR. Feel free to reopen it if necessary.
Title: Fix iwslt2017
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-13T09:51:21Z

The files were moved to google drive, I hosted them on the Hub instead (ok according to the license)

I also updated the `datasets_infos.json`

Comments:
_The documentation is not available anymore as the PR was closed or merged._

CI fails are just abut missing tags in the dataset card, merging !

FYI, 

The checksums have not been edited from the changes of .tgz to .zip files, and as a result a `ExpectedMoreDownloadedFiles` error occurs. Updating them in the `dataset_infos.json` should fix the error. 

Thanks for reporting and sorry for the delay, I opened https://huggingface.co/datasets/iwslt2017/discussions/2 to fix this
Title: Include entity positions as feature in ReCoRD
Author: richarddwang (CONTRIBUTOR)
State: closed
Created: 2022-06-12T11:56:28Z

https://huggingface.co/datasets/super_glue/viewer/record/validation

TLDR: We need to record entity positions, which are included in the source data but excluded by the loading script, to enable efficient and effective training for ReCoRD.

Currently, the loading script ignores the entity positions ("entity_start", "entity_end") and only records entity text. This might be because the training method of the official baseline is to make n training instance from a datapoint by replacing \"\@ placeholder\" in query with each entity individually.

But it increases the already heavy computation by multiple folds. So DeBERTa uses a method that take entity embeddings by their positions in the passage, and thus makes one training instance from one data point. It is way more efficient and proved effective for the ReCoRD task.

Can anybody help me with the dataset card rendering error? Maybe @lhoestq ?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thanks for the reply @lhoestq !

I have sucessed on `datasets-cli test ./datasets/super_glue --name record --save_infos`,
But as you can see, the check ran into `FAILED tests/test_dataset_cards.py::test_changed_dataset_card[super_glue] - V...`.
How can we solve it?

That would be neat! Let me implement it.
Title: Dataset Viewer issue for fgrezes/WIESP2022-NER
Author: AshTayade (NONE)
State: closed
Created: 2022-06-11T15:49:17Z

### Link

_No response_

### Description

_No response_

### Owner

_No response_

Comments:
https://huggingface.co/datasets/fgrezes/WIESP2022-NER

The error:

```
Message:       Couldn't find a dataset script at /src/services/worker/fgrezes/WIESP2022-NER/WIESP2022-NER.py or any data file in the same directory. Couldn't find 'fgrezes/WIESP2022-NER' on the Hugging Face Hub either: FileNotFoundError: Unable to resolve any data file that matches ['**test*', '**eval*'] in dataset repository fgrezes/WIESP2022-NER with any supported extension ['csv', 'tsv', 'json', 'jsonl', 'parquet', 'txt', 'blp', 'bmp', 'dib', 'bufr', 'cur', 'pcx', 'dcx', 'dds', 'ps', 'eps', 'fit', 'fits', 'fli', 'flc', 'ftc', 'ftu', 'gbr', 'gif', 'grib', 'h5', 'hdf', 'png', 'apng', 'jp2', 'j2k', 'jpc', 'jpf', 'jpx', 'j2c', 'icns', 'ico', 'im', 'iim', 'tif', 'tiff', 'jfif', 'jpe', 'jpg', 'jpeg', 'mpg', 'mpeg', 'msp', 'pcd', 'pxr', 'pbm', 'pgm', 'ppm', 'pnm', 'psd', 'bw', 'rgb', 'rgba', 'sgi', 'ras', 'tga', 'icb', 'vda', 'vst', 'webp', 'wmf', 'emf', 'xbm', 'xpm', 'zip']
```

I understand the issue is not related to the dataset viewer in itself, but with the autodetection of the data files without a loading script in the datasets library. cc @lhoestq @albertvillanova @mariosasko 

Apparently it finds `scoring-scripts/compute_seqeval.py` which matches `**eval*`, a regex that detects a test split. We should probably improve the regex because it's not supposed to catch this kind of files. It must also only check for files with supported extensions: txt, csv, png etc.
Title: `to_pandas` doesn't take into account format.
Author: Dref360 (CONTRIBUTOR)
State: closed
Created: 2022-06-10T20:25:31Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**

I have a large dataset that I need to convert part of to pandas to do some further analysis. Calling `to_pandas` directly on it is expensive. So I thought I could simply select the columns that I want and then call `to_pandas`.

**Describe the solution you'd like**

```python
from datasets import Dataset

ds = Dataset.from_dict({'a': [1,2,3], 'b': [5,6,7], 'c': [8,9,10]})
pandas_df = ds.with_format(columns=['a', 'b']).to_pandas()

# I would expect `pandas_df` to only include a,b as column.
```


**Describe alternatives you've considered**
I could remove all columns that I don't want? But I don't know all of them in advance. 

**Additional context**
I can probably make a PR with some pointers.

Comments:
Thanks for opening a discussion :)

Note that you can use `.remove_columns(...)` to keep only the ones you're interested in before calling `.to_pandas()`

Yes I can do that thank you!

Do you think that conceptually my example should work? If not, I'm happy to close this issue. 

If yes, I can start working on it.

Hi! Instead of `with_format(columns=['a', 'b']).to_pandas()`, use `with_format("pandas", columns=["a", "b"])` for easy conversion of the parts of the dataset to pandas via indexing/slicing.

The full code:
```python
from datasets import Dataset

ds = Dataset.from_dict({'a': [1,2,3], 'b': [5,6,7], 'c': [8,9,10]})
pandas_df = ds.with_format("pandas", columns=['a', 'b'])[:]
```

Ahhhh Thank you!

closing then :)
Title: Improve error message for missing pacakges from inside dataset script
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2022-06-10T16:59:36Z

Improve the error message for missing packages from inside a dataset script:

With this change, the error message for missing packages for `bigbench` looks as follows:
```
ImportError: To be able to use bigbench, you need to install the following dependencies:
        - 'bigbench' using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz"'
```

And this is how it looked before:
```
ImportError: To be able to use bigbench, you need to install the following dependencies['bigbench', 'bigbench', 'bigbench', 'bigbench'] using 'pip install "bigbench @ https://storage.googleapis.com/public_research_data/bigbench/bigbench-0.0.1.tar.gz" bigbench bigbench bigbench' for instance'
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

I opened a PR before I noticed yours ^^' You can find it here: https://github.com/huggingface/datasets/pull/4484

The only comment I have regarding your message is that it possibly shows several `pip install` commands, whereas one can run one single `pip install` command with the list of missing dependencies, which is maybe simpler.

Let me know which one your prefer

Closing in favor of #4484.
Title: [Docs] How to use with PyTorch page
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-10T16:25:49Z
Labels: documentation

Currently the docs about PyTorch are scattered around different pages, and we were missing a place to explain more in depth how to use and optimize a dataset for PyTorch. This PR is related to #4457 which is the TF counterpart :)

cc @Rocketknight1 we can try to align both documentations contents now I think

cc @stevhliu let me know what you think !

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Add SST-2 dataset
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-10T13:37:26Z

Add SST-2 dataset.

Currently it is part of GLUE benchmark.

This PR adds it as a standalone dataset.

CC: @julien-c

Comments:
_The documentation is not available anymore as the PR was closed or merged._

on the hub this dataset is referenced as `sst-2` not `sst2` – is there a canonical orthography? If not, could we name it `sst-2`?

@julien-c, we normally do not use hyphens for dataset names: whenever the original dataset name contains a hyphen, we usually:
- either suppress it: CoNLL-2000 (`conll2000`), CORD-19 (`cord19`)
- or replace it with underscore: CC-News (`cc_news`), SQuAD-es (`squad_es`)

There are some exceptions though... (I wonder why)

I think, the reason is there was a 1-to-1 relation with the corresponding Python module name.

I personally find confusing not having a rule and using both hyphens and underscores indistinctly: you never know which is the right orthography.

Whichever the decision we make, I would prefer to be applied consistently.

Also note that we already implemented this dataset as part of GLUE: https://github.com/huggingface/datasets/blob/master/datasets/glue/glue.py#L163
- dataset name: `glue`
- config name: `sst2`

On the other hand, let's see how other libraries name it:
- torchtext: `SST2` https://pytorch.org/text/stable/datasets.html#sst2
- OpenAI CLIP: `rendered-sst2` https://github.com/openai/CLIP/blob/main/data/rendered-sst2.md
- Kaggle: `SST2` https://www.kaggle.com/datasets/atulanandjha/stanford-sentiment-treebank-v2-sst2/version/22
- TensorFlow Datasets: `glue/sst2` https://www.tensorflow.org/datasets/catalog/glue#gluesst2

Ok, another option is to open PRs against the models in https://huggingface.co/models?datasets=sst-2 to change their dataset reference to `sst2`

(BTW some models refer to `sst2` already – but they're less popular: https://huggingface.co/models?datasets=sst2)

OK, I'm taking care of the subsequent PRs on models to align with this dataset name.
Title: Fix 401 error for unauthticated requests to non-existing repos
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-10T12:38:11Z

The hub now returns 401 instead of 404 for unauthenticated requests to non-existing repos.
This PR add support for the 401 error and fixes the CI fails on `master`

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: CI error with repo lhoestq/_dummy
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-10T12:26:06Z
Labels: bug

## Describe the bug
CI is failing because of repo "lhoestq/_dummy". See: https://app.circleci.com/pipelines/github/huggingface/datasets/12461/workflows/1b040b45-9578-4ab9-8c44-c643c4eb8691/jobs/74269
```
requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: https://huggingface.co/api/datasets/lhoestq/_dummy?full=true
```

The repo seems to no longer exist: https://huggingface.co/api/datasets/lhoestq/_dummy
```
error: "Repository not found"
```

CC: @lhoestq

Comments:
fixed by https://github.com/huggingface/datasets/pull/4472
Title: Replace data URLs in wider_face dataset once hosted on the Hub
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-10T08:13:25Z

This PR replaces the URLs of data files in Google Drive with our Hub ones, once the data owners have approved to host their data on the Hub.

They also informed us that their dataset is licensed under CC BY-NC-ND.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Generalize tutorials for audio and vision
Author: stevhliu (MEMBER)
State: closed
Created: 2022-06-09T22:00:44Z
Labels: documentation

This PR updates the tutorials to be more generalizable to all modalities. After reading the tutorials, a user should be able to load any type of dataset, know how to index into and slice a dataset, and do the most basic/common type of preprocessing (tokenization, resampling, applying transforms) depending on their dataset.

Other changes include:

- Removed the sections about a dataset's metadata, features, and columns because we cover this in an earlier tutorial about inspecting the `DatasetInfo` through the dataset builder.
- Separate the sharing dataset tutorial into two sections: (1) uploading via the web interface and (2) using the `huggingface_hub` library.
- Renamed some tutorials in the TOC to be more clear and specific.
- Added more text to nudge users towards joining the community and asking questions on the forums.
- If it's okay with everyone, I'd also like to remove the section about loading and using metrics since we have the `evaluate` docs now.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Extend support for streaming datasets that use xml.dom.minidom.parse
Author: albertvillanova (MEMBER)
State: closed
Created: 2022-06-09T06:58:25Z

This PR extends the support in streaming mode for datasets that use `xml.dom.minidom.parse`, by patching that function.

This PR adds support for streaming datasets like "Yaxin/SemEval2015".

Fix #4453.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Title: Use config_id to check split sizes instead of config name
Author: lhoestq (MEMBER)
State: closed
Created: 2022-06-08T17:45:24Z

Fix https://github.com/huggingface/datasets/issues/4462

Comments:
_The documentation is not available anymore as the PR was closed or merged._

closing in favor of https://github.com/huggingface/datasets/pull/4465
Title: BigBench: NonMatchingSplitsSizesError when passing a dataset configuration parameter
Author: lhoestq (MEMBER)
State: open
Created: 2022-06-08T17:31:24Z
Labels: bug

As noticed in https://github.com/huggingface/datasets/pull/4125 when a dataset config class has a parameter that reduces the number of examples (e.g. named `max_examples`), then loading the dataset and passing `max_examples` raises `NonMatchingSplitsSizesError`.

This is because it will check for expected the number of examples of the config with the same name without taking into account the `max_examples` parameter. This can be fixed by checking the expected number of examples using the **config id** instead of name. Indeed the config id corresponds to the config name + an optional suffix that depends on the config parameters

Comments:
Why not adding `max_examples` as part of the config name?

Yup it can also work, and maybe it's simpler this way. Opening a PR to fix bigbench instead of https://github.com/huggingface/datasets/pull/4463

Hi @lhoestq,

Thank you for taking a look at this issue, and proposing a solution. 
Unfortunately, after trying the fix in #4465 I still see the same issue.

I think there is some subtlety where the config name gets overwritten somewhere when `BUILDER_CONFIGS`[(link)](https://github.com/huggingface/datasets/blob/master/datasets/bigbench/bigbench.py#L126) is defined. 

If I print out the `self.config.name` in the current version (with the fix in #4465), I see just the task name, but if I comment out `BUILDER_CONFIGS`, the `num_shots` and `max_examples` gets appended as was meant by #4465.

I haven't managed to track down where this happens, but I thought you might know? 

(Another comment on your fix: the `name` variable is used to fetch the task from the bigbench API, so modifying it causes an error if it's actually called. This can easily be fixed by having `config_name` variable in addition to the `task_name`)