rIsHu009/github-issues-updated huggingface dataset:
Title: SNLI dataset has labels of -1 
Author: puzzler10 (NONE)
State: closed
Created: 2021-04-28T00:32:25Z

There are a number of rows with a label of -1 in the SNLI dataset. The dataset descriptions [here](https://nlp.stanford.edu/projects/snli/) and [here](https://github.com/huggingface/datasets/tree/master/datasets/snli) don't list  -1 as a label possibility, and neither does the dataset viewer. As examples, see index 107 or 124 of the test set.

It isn't clear what these labels mean. I found a [line of code](https://github.com/huggingface/datasets/blob/80e59ef178d3bb2090d091bc32315c655eb0633d/datasets/snli/snli.py#L94) that seems to put them in but it seems still unclear why they are there. The current workaround is to just drop the rows from any model being trained. 

Perhaps the documentation should be updated.

Comments:
Hi @puzzler10, 
Those examples where `gold_label` field was empty, -1 label was alloted to it. In order to remove it you can filter the samples from train/val/test splits. Here's how you can drop those rows from the dataset:
`dataset = load_dataset("snli")`
`dataset_test_filter = dataset['test'].filter(lambda example: example['label'] != -1)`

I agree it should have been mentioned in the documentation. I'll raise a PR regarding the same. Thanks for pointing out!
Title: Always update metadata in arrow schema
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-27T19:21:57Z

We store a redundant copy of the features in the metadata of the schema of the arrow table. This is used to recover the features when doing `Dataset.from_file`. These metadata are updated after each transfor, that changes the feature types.

For each function that transforms the feature types of the dataset, I added a step in the tests to make sure the metadata in the arrow schema are up to date.

I also added a line to update the metadata directly in the Dataset.__init__ method.
This way even a dataset instantiated with __init__ will have a table with the right metadata.

Fix #2271.

cc @mariosasko

Comments:
Title: Bug in Dataset.class_encode_column
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-27T16:13:18Z
Labels: bug

## Describe the bug

All the rest of the columns except the one passed to `Dataset.class_encode_column` are discarded.

## Expected results

All the original columns should be kept.

This needs regression tests.

Comments:
This has been fixed in this commit: https://github.com/huggingface/datasets/pull/2254/commits/88676c930216cd4cc31741b99827b477d2b46cb6

It was introduced in #2246 : using map with `input_columns` doesn't return the other columns anymore
Title: Synchronize table metadata with features
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-27T15:55:13Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**

As pointed out in this [comment](https://github.com/huggingface/datasets/pull/2145#discussion_r621326767):
> Metadata stored in the schema is just a redundant information regarding the feature types.
It is used when calling Dataset.from_file to know which feature types to use.
These metadata are stored in the schema of the pyarrow table by using `update_metadata_with_features`.
However this something that's almost never tested properly.

**Describe the solution you'd like**

We should find a way to always make sure that the metadata (in `self.data.schema.metadata`) are synced with the actual feature types (in `self.info.features`).

Comments:
See PR #2274
Title: Fix iterable interface expected by numpy
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-27T14:35:56Z

Numpy expects the old iterable interface with `__getitem__` instead of `__iter__`.

Comments:
It's been fixed in this commit: https://github.com/huggingface/datasets/commit/549110e08238b3716a5904667095fb003acda54e

Basically #2246 broke querying an index with a simple iterable.
With the fix, it's again possible to use iterables and we can keep RandIter as it is.

Closing since the fix is already on master
Title: Don't use pyarrow 4.0.0 since it segfaults when casting a sliced ListArray of integers
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-27T11:58:28Z

This test `tests/test_table.py::test_concatenation_table_cast` segfaults with the latest update of pyarrow 4.0.0.
Setting `pyarrow<4.0.0` for now. I'll open an issue on JIRA once I know more about the origin of the issue

Comments:
@lhoestq note that the segfault also occurs on Linux.

Created the ticket at
https://issues.apache.org/jira/browse/ARROW-12568

@lhoestq the ticket you mentioned is now in state resolved. Pyarrow supports AArch64 after version 4.0.0. Because of this restriction `datasets` is not installing in AArch64 systems.
Title: Make tests run faster
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-26T15:55:40Z

From 7min to 2min to run pytest.
Ideally we should keep the whole CI run time below 10min.

In this PR I removed the remote tests that were never used.
I also replaced nested parametrized tests with unit tests.
This makes me think that we could still add more high level tests to check for a few combinations of parameters (but not all of them since there are too many of them).
Let me know what you think

Finally in another PR we can also separate in two circleci jobs:
- the tests of the code code of the lib
- the tests of the all the dataset/metric scripts.

Comments:
LOL, I was also working on something similar 😅. I'm gonna have a look!!!

Sorry I didn't know you were also working on it ^^'
And yes I 100% agree with you on the points you mentioned. We should definitely improve the coverage. It would be nice to have a clearer separation to know which tests in the suite are unit tests and which ones are integration tests


Never mind: we both noticed tests can be improved. More PRs to come... 😉 

According to the literature, unit tests are those that test a behavior unit, isolated from the other components and must be very fast: for me, this last requirement implies that they must be performed completely _in memory_.

As opposed, integration tests are those which also test interactions with _external_ components, like web services, databases, file system, etc.

The problem I see is that our code is still too coupled and it is difficult to isolate components for testing. Therefore, I would suggest acting iteratively, by refactoring to decouple components and then implement unit tests for each component in isolation.
Title: Fix memory issue in multiprocessing: Don't pickle table index
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-26T09:21:35Z

The table index is currently being pickled when doing multiprocessing, which brings all the record batches of the dataset in memory.

I fixed that by not pickling the index attributes. Therefore each process has to rebuild the index when unpickling the table.

Fix issue #2256

We'll do a patch release asap !

Comments:
The code quality check is going to be fixed by #2265 

The memory issue didn't come from `self.__dict__.copy()` but from the fact that this dict contains `_batches` which has all the batches of the table in it.
Therefore for a MemoryMappedTable all the data in `_batches` were copied in memory when pickling and this is the issue.

I'm still investigating why we didn't catch this issue in the tests.
This test should have caught it but didn't:

https://github.com/huggingface/datasets/blob/3db67f5ff6cbf807b129d2b4d1107af27623b608/tests/test_table.py#L350-L353

I'll focus on the patch release and fix the test in another PR after the release

Yes, I think it is better that way...
Title: GooAQ dataset added
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-04-25T09:26:48Z

@lhoestq here the dataset is stored with Git LFS. Should I add option for manual downloading of dataset using `git lfs pull` post repo cloning or can we accommodate this in the current `download_and_extract`?

Comments:
Thanks for adding this one !
The download manager does support downloading files on git lfs via their github url. No need for a manual download option ;)
Title: Add support for Split.ALL
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-04-25T01:45:42Z

The title says it all.

Comments:
Honestly, I think we should fix some other issues in Split API before this change. E. g. currently the following will not work, even though it should:
```python
import datasets
datasets.load_dataset("sst", split=datasets.Split.TRAIN+datasets.Split.TEST)  # AssertionError
```

EDIT:
Actually, think it's OK to merge this PR because the fix will not touch this PR's code.
Title: Fix incorrect update_metadata_with_features calls in ArrowDataset
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-04-25T00:48:38Z

Fixes bugs in the `unpdate_metadata_with_features` calls (caused by changes in #2151)

Comments:
@lhoestq Maybe a test that runs the functions that call `update_metadata_with_features` and checks if metadata was updated would be nice to prevent this from happening in the future.
Title: added metrics for CUAD
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-04-24T14:09:54Z

For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here

Comments:
> For now I've added F1, AUPR, Precision at 80% recall, and Precision at 90%. Last 3 metrics were reported in the [paper](https://arxiv.org/pdf/2103.06268.pdf). Please let me know if we require `exact_match` metric too here

@bhavitvyamalik  I guess the mentioned metrics are enough but it would be better if exact match is also added since the standard SQUAD dataset also has it.

I would like to quote it from the website that I am following to learn
these things.
Exact Match:
This metric is as simple as it sounds. For each question+answer pair, if
the characters of the model's prediction exactly match the characters of
*(one
of) the True Answer(s)*, EM = 1, otherwise EM = 0. This is a strict
all-or-nothing metric; being off by a single character results in a score
of 0. When assessing against a negative example, if the model predicts any
text at all, it automatically receives a 0 for that example.

So, I guess you need to ensure at least 1 predicted answer matches for EM
to be 1.
Source:
https://qa.fastforwardlabs.com/no%20answer/null%20threshold/bert/distilbert/exact%20match/f1/robust%20predictions/2020/06/09/Evaluating_BERT_on_SQuAD.html

You can go to their homepage and read the other links. They have detailed
explanations on evaluation metrics. You can also have a look at the
squad_v2 metric file for further clarification.

Regards,
Mohammed Rakib

On Sun, 25 Apr 2021 at 15:20, Bhavitvya Malik ***@***.***>
wrote:

> I'm a little confused when it comes to 2 ground truths which can be a
> possible answer. Like here for eg.
>
> predictions = [{'prediction_text': ['The seller:', 'The buyer/End-User:
> Shenzhen LOHAS Supply Chain Management Co., Ltd.'], 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> references = [{'answers': {'answer_start': [143, 49], 'text': ['The
> seller:', 'The buyer/End-User: Shenzhen LOHAS Supply Chain Management Co.,
> Ltd.']}, 'id':
> 'LohaCompanyltd_20191209_F-1_EX-10.16_11917878_EX-10.16_Supply
> Agreement__Parties'}]
>
> Should I ensure at least 1 predicted answer matches or both predicted
> answers should match (like in this case) for EM to be 1?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/pull/2257#issuecomment-826289753>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AHMYZAZSAEZNFWEMVAPK6M3TKPNHLANCNFSM43QFZVPQ>
> .
>


Updated the same @MohammedRakib! Even if a single answer matches I'm returning 1 in that case for EM (not traversing all predictions once we have one `exact_match` from prediction)
Title: Running `datase.map` with `num_proc > 1` uses a lot of memory
Author: roskoN (NONE)
State: closed
Created: 2021-04-24T09:56:20Z
Labels: bug

## Describe the bug
Running `datase.map` with `num_proc > 1`  leads to a tremendous memory usage that requires swapping on disk and it becomes very slow.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dstc8_datset = load_dataset("roskoN/dstc8-reddit-corpus", keep_in_memory=False)


def _prepare_sample(batch):
    return {"input_ids": list(), "attention_mask": list()}


for split_name, dataset_split in list(dstc8_datset.items()):
    print(f"Processing {split_name}")
    encoded_dataset_split = dataset_split.map(
        function=_prepare_sample,
        batched=True,
        num_proc=4,
        remove_columns=dataset_split.column_names,
        batch_size=10,
        writer_batch_size=10,
        keep_in_memory=False,
    )
    print(encoded_dataset_split)

    path = f"./data/encoded_{split_name}"

    encoded_dataset_split.save_to_disk(path)
```

## Expected results
Memory usage should stay within reasonable boundaries.


## Actual results
This is htop-output from running the provided script.

![image](https://user-images.githubusercontent.com/8143425/115954836-66954980-a4f3-11eb-8340-0153bdc3a475.png)

## Versions
```
- Datasets: 1.6.0
- Python: 3.8.8 (default, Apr 13 2021, 19:58:26)
[GCC 7.3.0]
- Platform: Linux-4.19.128-microsoft-standard-x86_64-with-glibc2.10
```
Running on WSL2

Comments:
Thanks for reporting ! We are working on this and we'll do a patch release very soon.

We did a patch release to fix this issue.
It should be fixed in the new version 1.6.1

Thanks again for reporting and for the details :)
Title: Perform minor refactoring: use config
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-23T11:45:47Z
Labels: refactoring

Perform minor refactoring related to `config`.

Comments:
@lhoestq is there a problem in the master branch? I got a segmentation fault...
```
tests/test_table.py::test_concatenation_table_cast[in_memory] Fatal Python error: Segmentation fault
```

Oh wow. Let me re-run the CI just to make sure

Hmm interesting, the segfault is still there. I'm investigating this issue on my windows machine

Feel free to merge master into this branch to fix the CI :)
Title: while running run_qa.py, ran into a value error
Author: nlee0212 (NONE)
State: open
Created: 2021-04-23T07:51:03Z

command:

python3 run_qa.py   --model_name_or_path hyunwoongko/kobart    --dataset_name squad_kor_v2   --do_train   --do_eval   --per_device_train_batch_size 8   --learning_rate 3e-5   --num_train_epochs 3   --max_seq_length 512   --doc_stride 128   --output_dir /tmp/debug_squad/

error: 

ValueError: External features info don't match the dataset:
Got
{'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answer': {'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None), 'html_answer_start': Value(dtype='int32', id=None)}, 'url': Value(dtype='string', id=None), 'raw_html': Value(dtype='string', id=None)}
with type
struct<answer: struct<text: string, answer_start: int32, html_answer_start: int32>, context: string, id: string, question: string, raw_html: string, title: string, url: string>

but expected something like
{'answer': {'answer_start': Value(dtype='int32', id=None), 'html_answer_start': Value(dtype='int32', id=None), 'text': Value(dtype='string', id=None)}, 'context': Value(dtype='string', id=None), 'id': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'raw_html': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'url': Value(dtype='string', id=None)}
with type
struct<answer: struct<answer_start: int32, html_answer_start: int32, text: string>, context: string, id: string, question: string, raw_html: string, title: string, url: string>

I didn't encounter this error 4 hours ago. any solutions for this kind of issue?
looks like gained dataset format refers to 'Data Fields', while expected refers to 'Data Instances'.

Comments:
Title: Faster map w/ input_columns & faster slicing w/ Iterable keys
Author: norabelrose (CONTRIBUTOR)
State: closed
Created: 2021-04-21T19:49:07Z

@lhoestq Fixes #2193 

- `map` now uses `with_format` to only load needed columns in memory when `input_columns` is set
- Slicing datasets with Iterables of indices now uses a new `Table.fast_gather` method, implemented with `np.searchsorted`, to find the appropriate batch indices all at once. `pa.concat_tables` is no longer used for this; we just call `pa.Table.from_batches` with a list of all the batch slices.

Together these changes have sped up batched `map()` calls over subsets of columns quite considerably in my initial testing.

Comments:
@lhoestq Just fixed the code style issues— I think it should be good to merge now :)
Title: Set specific cache directories per test function call
Author: albertvillanova (MEMBER)
State: open
Created: 2021-04-20T17:06:22Z

Implement specific cache directories (datasets, metrics and modules) per test function call.

Currently, the cache directories are set within the temporary test directory, but they are shared across all test function calls.

This PR implements specific cache directories for each test function call, so that tests are atomic and there are no side effects.

Comments:
@lhoestq, I think this reaches some memory limit on Linux instances... (?)

It looks like the `comet` metric test fails because it tries to load a model in memory.
In the tests I think we have `patch_comet` that mocks the model download + inference. Not sure why it didn't work though.
I can take a look tomorrow (this afternoon is the pytorch ecosystem day)

@lhoestq thanks for the hint: I'm going to have a look at that mock... ;)

@lhoestq finally I did not find out why the mock is not used... If you can give me some other hint tomorrow...
Title: Map is slow and processes batches one after another
Author: villmow (NONE)
State: closed
Created: 2021-04-20T14:58:20Z
Labels: bug

## Describe the bug

I have a somewhat unclear bug to me, where I can't figure out what the problem is. The code works as expected on a small subset of my dataset (2000 samples) on my local machine, but when I execute the same code with a larger dataset (1.4 million samples) this problem occurs. Thats why I can't give exact steps to reproduce, I'm sorry. 

I process a large dataset in a two step process. I first call map on a dataset I load from disk and create a new dataset from it. This works like expected and `map` uses all workers I started it with. Then I process the dataset created by the first step, again with `map`, which is really slow and starting only one or two process at a time. Number of processes is the same for both steps.

pseudo code:
```python
ds = datasets.load_from_disk("path")
new_dataset = ds.map(work, batched=True, ...)  # fast uses all processes
final_dataset = new_dataset.map(work2, batched=True, ...)  # slow starts one process after another
```

## Expected results
Second stage should be as fast as the first stage.

## Versions
Paste the output of the following code:
- Datasets: 1.5.0
- Python: 3.8.8 (default, Feb 24 2021, 21:46:12)
- Platform: Linux-5.4.0-60-generic-x86_64-with-glibc2.10    

Do you guys have any idea? Thanks a lot!

Comments:
Hi @villmow, thanks for reporting.

Could you please try with the Datasets version 1.6? We released it yesterday and it fixes some issues about the processing speed. You can see the fix implemented by @lhoestq here: #2122.

Once you update Datasets, please confirm if the problem persists.

Hi @albertvillanova, thanks for the reply. I just tried the new version and the problem still persists. 

Do I need to rebuild the saved dataset (which I load from disk) with the 1.6.0 version of datasets? My script loads this dataset and creates new datasets from it. I tried it without rebuilding.

See this short video of what happens. It does not create all processes at the same time:

https://user-images.githubusercontent.com/2743060/115720139-0da3a500-a37d-11eb-833a-9bbacc70868d.mp4



There can be a bit of delay between the creations of the processes but this delay should be the same for both your `map` calls. We should look into this.
Also if you hav some code that reproduces this issue on google colab that'd be really useful !

Regarding the speed differences:
This looks like a similar issue as https://github.com/huggingface/datasets/issues/1992 who is experiencing the same speed differences between processes.
This is a known bug that we are investigating. As of now I've never managed to reproduce it on my machine so it's pretty hard for me to find where this issue comes from.


Upgrade to 1.6.1 solved my problem somehow. I did not change any of my code, but now it starts all processes around the same time.

Nice ! I'm glad this works now.
Closing for now, but feel free to re-open if you experience this issue again.
Title: Link to datasets viwer on Quick Tour page returns "502 Bad Gateway"
Author: martavillegas (NONE)
State: closed
Created: 2021-04-20T14:19:51Z
Labels: bug

Link to datasets viwer (https://huggingface.co/datasets/viewer/)  on Quick Tour page  (https://huggingface.co/docs/datasets/quicktour.html) returns "502 Bad Gateway"

The same error with https://huggingface.co/datasets/viewer/?dataset=glue&config=mrpc

Comments:
This should be fixed now!

cc @srush
Title: Add SLR32 to OpenSLR
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-04-20T11:02:45Z

I would like to add SLR32 to OpenSLR. It contains four South African languages: Afrikaans, Sesotho, Setswana and isiXhosa

Comments:
> And yet another one ! Thanks a lot :)

I just hope you don’t get fed up with openslr PR 😊 there are still few other datasets created by google in openslr that is not in hf dataset yet
Title: Update Dataset.dataset_size after transformed with map
Author: albertvillanova (MEMBER)
State: open
Created: 2021-04-19T15:19:38Z
Labels: enhancement

After loading a dataset, if we transform it by using `.map` its `dataset_size` attirbute is not updated.

Comments:
@albertvillanova I would like to take this up. It would be great if you could point me as to how the dataset size is calculated in HF. Thanks!
Title: Request to add StrategyQA dataset
Author: sarahwie (NONE)
State: open
Created: 2021-04-19T14:46:26Z
Labels: dataset request

## Request to add StrategyQA dataset
- **Name:** StrategyQA
- **Description:** open-domain QA [(project page)](https://allenai.org/data/strategyqa)
- **Paper:** [url](https://arxiv.org/pdf/2101.02235.pdf)
- **Data:** [here](https://allenai.org/data/strategyqa)
- **Motivation:** uniquely-formulated dataset that also includes a question-decomposition breakdown and associated Wikipedia annotations for each step. Good for multi-hop reasoning modeling.

Comments:
Title: Fix `xnli` dataset tuple key
Author: NikhilBartwal (CONTRIBUTOR)
State: closed
Created: 2021-04-16T19:12:42Z

Closes #2229 
The `xnli` dataset yields a tuple key in case of `ar` which is inconsistant with the acceptable key types (str/int).
The key was thus ported to `str` keeping the original information intact.

Comments:
Title: Start filling GLUE dataset card
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-16T18:37:37Z

The dataset card was pretty much empty.

I added the descriptions (mainly from TFDS since the script is the same), and I also added the tasks tags as well as examples for a subset of the tasks.

cc @sgugger

Comments:
I replaced all the "we" and applied your suggestion

Merging this for now, we can continue improving this card in other PRs :)
Title: `xnli` dataset creating a tuple key while yielding instead of `str` or `int`
Author: NikhilBartwal (CONTRIBUTOR)
State: closed
Created: 2021-04-16T13:21:53Z

When using  `ds = datasets.load_dataset('xnli', 'ar')`, the dataset generation script uses the following section of code in the egging, which yields a tuple key instead of the specified `str` or `int` key:
https://github.com/huggingface/datasets/blob/56346791aed417306d054d89bd693d6b7eab17f7/datasets/xnli/xnli.py#L196

Since, community datasets in Tensorflow Datasets also use HF datasets, this causes a Tuple key error while loading HF's `xnli` dataset. 
I'm up for sending a fix for this, I think we can simply use `file_idx + "_" + row_idx` as a unique key instead of a tuple.

Comments:
Hi ! Sure sounds good. Also if you find other datasets that use tuples instead of str/int, you can also fix them !
thanks :)

@lhoestq I have sent a PR for fixing the issue. Would be great if you could have a look! Thanks!
Title: [WIP] Add ArrayXD support for fixed size list.
Author: jblemoine (NONE)
State: open
Created: 2021-04-16T13:04:08Z

Add support for fixed size list for ArrayXD when shape is known . See https://github.com/huggingface/datasets/issues/2146
Since offset are not stored anymore, the file size is now roughly equal to the actual data size.

Comments:
Awesome thanks ! To fix the CI you just need to merge master into your branch.
The error is unrelated to your PR
Title: fixed one instance of 'train' to 'test'
Author: alexwdong (CONTRIBUTOR)
State: closed
Created: 2021-04-15T04:26:40Z

I believe this should be 'test' instead of 'train'

Comments:
Thanks ! good catch

Could you also update the metadata of this dataset ?
You can do so by running
```
datasets-cli test ./datasets/newsgroup --all_configs --save_infos --ignore_verifications
```
This should update the dataset_infos.json file that contains the size of all the splits for example.

Hi,
`dataset_infos.json` should be updated now.
Title: Set test cache config
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-14T12:55:24Z

Currently, running the tests populates the default cache directory `"~/.cache"`.

This PR monkey-patches the config to set the cache directory within the temporary test directory, avoiding side effects.

Comments:
>  why a cache dir per test function does not work?

Probably because we end up with multiple `datasets_module` in the python path. This breaks the import of all the datasets/metrics modules.
If you want to use one modules cache per test, you may need remove the `datasets_module` that was added to the python path during the test.
Indeed if the module cache hasn't been initialized, then it's added to the python path by calling `init_dynamic_modules`:

https://github.com/huggingface/datasets/blob/ba76012a19193a35053b9e20243ff40c2b4204ab/src/datasets/load.py#L291-L291

@lhoestq, for the moment, this PR avoids populating the `~/.cache` dir during training, which is already an improvement, isn't it?

Yes we can merge it this way if you're fine with it !
This is a good improvement

I will eventually try to implement a `cache_dir` per test function in another PR, but I think I should first fix some side effects in tests: each test function should be atomic and able to have its own `cache_dir` without being affected by the `cache_dir` set in other test functions.

Yes this would be ideal !
Title: Fix too long WindowsFileLock name
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-14T12:26:52Z
Labels: wontfix

Fix WindowsFileLock name longer than allowed MAX_PATH by shortening the basename.

Comments:
Windows users should disable the max path length limit. It's a nightmare to handle it.
Also the lock path must not be changed in a random way. Otherwise from another process the lock path might not be the same and the locking mechanism won't work.

Do you agree with handling the case where MAX_PATH is not disabled? If not, we can close this PR.

If so, would it work a deterministic lock path instead of random?

I'd rather not handle this at all, since there will be other places in the code where the limit will break things
Title: Add SLR70 - SLR80 and SLR86 to OpenSLR dataset
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-04-14T12:09:18Z

I would like to add SLR70, SLR71, SLR72, SLR73, SLR74, SLR75, SLR76, SLR77, SLR78, SLR79, SLR80 and SLR86 to OpenSLR dataset. The languages are:
Nigerian English, Chilean Spanish, Columbian Spanish, Peruvian Spanish, Puerto Rico Spanish, Venezuelan Spanish, Basque, Galician, Gujarati and Kannada.

Comments:
Title: Fix infinite loop in WindowsFileLock
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-14T10:49:58Z
Labels: wontfix

Raise exception to avoid infinite loop.

Comments:
How is it possible to get an infinite loop ? Can you add more details ?

Yes, in Windows, if the filename is too long, a `FileNotFoundError` is raised. The exception should be raised in this case. Otherwise, we get into an infinite loop.

If other process has the file locked, then `PermissionError` is raised. In this case, `pass` is OK.

Note that the filelock module comes from this project that hasn't changed in years - while still being used by ten of thousands of projects:
https://github.com/benediktschmitt/py-filelock

Unless we have proper tests for this, I wouldn't recommend to change it

I'm pretty sure many things from the library could break for windows users that haven't disabled the max path length limit.
Maybe it would be simpler to simply raise an error on startup. For exampe, for windows users the error could ask them to disable the limit if it's not been disabled yet ?
Title: Added CUAD dataset
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-04-13T21:05:03Z

Dataset link : https://github.com/TheAtticusProject/cuad/

Working on README.md currently.

Closes #2084 and [#1](https://github.com/TheAtticusProject/cuad/issues/1).

Comments:
1) Changed the language in a few places apart from those you mentioned in README
2) Reduced the size of dummy data folder by removing all other entries except the first
3) Updated YAML tags by using to the past version of `datasets-tagging` app. Will update the quick fix on that repository too in a while

@bhavitvyamalik  Thanks for adding the dataset on huggingface! Can you please add a metric also for the dataset using the squad_v2 metric file? 

@MohammedRakib you can check [#2257](https://github.com/huggingface/datasets/pull/2257)
Title: Revert breaking change in cache_files property
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-13T14:20:04Z

#2025 changed the format of `Dataset.cache_files`.
Before it was formatted like
```python
[{"filename": "path/to/file.arrow", "start": 0, "end": 1337}]
```
and it was changed to
```python
["path/to/file.arrow"]
```
since there's no start/end offsets available anymore.

To make this less breaking, I'm setting the format back to a list of dicts:
```python
[{"filename": "path/to/file.arrow"}]
```

Comments:
Title: Add datasets SLR35 and SLR36 to OpenSLR 
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-04-13T08:24:07Z

I would like to add [SLR35](https://openslr.org/35/) (18GB) and [SLR36](https://openslr.org/36/)  (22GB) which are Large Javanese and Sundanese ASR training data set collected by Google in collaboration with Reykjavik University and Universitas Gadjah Mada in Indonesia.

Comments:
Hi @lhoestq,
Could you please help me, I got this error message in all "ci/circleci: run_dataset_script_tests_pyarrow*" tests:
```
...
    """Wrapper classes for various types of tokenization."""
    
    from bleurt.lib import bert_tokenization
    import tensorflow.compat.v1 as tf
>   import sentencepiece as spm
E   ModuleNotFoundError: No module named 'sentencepiece'
...
```
I am not sure why I do get it. Thanks.


Hi ! This issue appeared on master since the last update of `BLEURT`.
I'm working on a fix. You can ignore this issue for this PR

> Hi ! This issue appeared on master since the last update of `BLEURT`.
> I'm working on a fix. You can ignore this issue for this PR

Thanks for the info

Merging since the CI is fixed on master
Title: load_metric error: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'
Author: nsaphra (NONE)
State: closed
Created: 2021-04-12T20:26:01Z
Labels: bug

I'm having the same problem as [Notebooks issue 10](https://github.com/huggingface/notebooks/issues/10) on datasets 1.2.1, and it seems to be an issue with the datasets package.

```python
>>> from datasets import load_metric
>>> metric = load_metric("glue", "sst2")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/ext3/miniconda3/lib/python3.8/site-packages/datasets-1.2.1-py3.8.egg/datasets/load.py", line 502, in load_metric
  File "/ext3/miniconda3/lib/python3.8/site-packages/datasets-1.2.1-py3.8.egg/datasets/load.py", line 66, in import_main_class
  File "/ext3/miniconda3/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/ns4008/.cache/huggingface/modules/datasets_modules/metrics/glue/e4606ab9804a36bcd5a9cebb2cb65bb14b6ac78ee9e6d5981fa679a495dd55de/glue.py", line 105, in <module>
    @datasets.utils.file_utils.add_start_docstrings(_DESCRIPTION, _KWARGS_DESCRIPTION)
AttributeError: module 'datasets.utils.file_utils' has no attribute 'add_start_docstrings'
```

Comments:
Hi @nsaphra, thanks for reporting.

This issue was fixed in `datasets` version 1.3.0. Could you please update `datasets` and tell me if the problem persists?
```shell
pip install -U datasets
```

There might be a bug in the conda version of `datasets` 1.2.1 where the datasets/metric scripts are downloaded from `master` instead of the `1.2.1` repo.

You can try setting the env var `HF_SCRIPTS_VERSION="1.2.1"` as a workaround. Let me know if that helps.

I just faced the same issue. I was using 1.2.1 from conda and received the same AttributeError complaining about 'add_start_docstrings'. Uninstalling the conda installed datasets and then installing the latest datasets (version 1.5.0) using pip install solved the issue for me. I don't like mixing up conda and pip installs in the same environments but this will have to do for now, until 1.5.0 is made available through conda.

Yep, seems to have fixed things! The conda package could really do with an update. Thanks!
Title: Can't reach "https://storage.googleapis.com/illuin/fquad/train.json.zip" when trying to load fquad dataset
Author: hanss0n (NONE)
State: closed
Created: 2021-04-12T13:49:56Z

I'm trying to load the [fquad dataset](https://huggingface.co/datasets/fquad) by running: 

```Python
fquad = load_dataset("fquad")
```

which produces the following error:

```
Using custom data configuration default

Downloading and preparing dataset fquad/default (download: 3.14 MiB, generated: 6.62 MiB, post-processed: Unknown size, total: 9.76 MiB) to /root/.cache/huggingface/datasets/fquad/default/0.1.0/778dc2c85813d05ddd0c17087294d5f8f24820752340958070876b677af9f061...

---------------------------------------------------------------------------

ConnectionError                           Traceback (most recent call last)

<ipython-input-48-a2721797e23b> in <module>()
----> 1 fquad = load_dataset("fquad")

11 frames

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in get_from_cache(url, cache_dir, force_download, proxies, etag_timeout, resume_download, user_agent, local_files_only, use_etag, max_retries, use_auth_token)
    614             raise FileNotFoundError("Couldn't find file at {}".format(url))
    615         _raise_if_offline_mode_is_enabled(f"Tried to reach {url}")
--> 616         raise ConnectionError("Couldn't reach {}".format(url))
    617 
    618     # Try a second time

ConnectionError: Couldn't reach https://storage.googleapis.com/illuin/fquad/train.json.zip
```

Does anyone know why that is and how to fix it?

Comments:
Hi ! Apparently the data are not available from this url anymore. We'll replace it with the new url when it's available

I saw this on their website when we request to download the dataset:
![image](https://user-images.githubusercontent.com/19718818/114879600-fa458680-9e1e-11eb-9e05-f0963d68ff0f.png)

Can we still request them link for the dataset and make a PR? @lhoestq @yjernite 

I've contacted Martin (first author of the fquad paper) regarding a possible new url. Hopefully we can get one soon !

They now made a website to force people who want to use the dataset for commercial purposes to seek a commercial license from them ...

The script has been adopted to support manual download from the website, so I'm closing this issue.
Title: Getting checksum error when trying to load lc_quad dataset
Author: hanss0n (NONE)
State: closed
Created: 2021-04-12T13:38:58Z

I'm having issues loading the [lc_quad](https://huggingface.co/datasets/fquad) dataset by running:

```Python
lc_quad = load_dataset("lc_quad")
```

which is giving me the following error:

``` 
Using custom data configuration default

Downloading and preparing dataset lc_quad/default (download: 3.69 MiB, generated: 19.77 MiB, post-processed: Unknown size, total: 23.46 MiB) to /root/.cache/huggingface/datasets/lc_quad/default/2.0.0/5a98fe174603f5dec6df07edf1c2b4d2317210d2ad61f5a393839bca4d64e5a7...

---------------------------------------------------------------------------

NonMatchingChecksumError                  Traceback (most recent call last)

<ipython-input-42-404ace83f73c> in <module>()
----> 1 lc_quad = load_dataset("lc_quad")

3 frames

/usr/local/lib/python3.7/dist-packages/datasets/utils/info_utils.py in verify_checksums(expected_checksums, recorded_checksums, verification_name)
     37     if len(bad_urls) > 0:
     38         error_msg = "Checksums didn't match" + for_verification_name + ":\n"
---> 39         raise NonMatchingChecksumError(error_msg + str(bad_urls))
     40     logger.info("All the checksums matched successfully" + for_verification_name)
     41 

NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://github.com/AskNowQA/LC-QuAD2.0/archive/master.zip']
```

Does anyone know why this could be and how I fix it?

Comments:
Hi,

I've already opened a PR with the fix. If you are in a hurry, just build the project from source and run:
```bash
datasets-cli test datasets/lc_quad --save_infos --all_configs --ignore_verifications
```



Ah sorry, I tried searching but couldn't find any related PR. 

Thank you!
Title: making labels consistent across the datasets
Author: dorost1234 (NONE)
State: closed
Created: 2021-04-11T10:03:56Z

Hi
For accessing the labels one can type 
```
>>> a.features['label']
ClassLabel(num_classes=3, names=['entailment', 'neutral', 'contradiction'], names_file=None, id=None)
```
The labels however are not consistent with the actual labels sometimes, for instance in case of XNLI, the actual labels are 0,1,2, but if one try to access as above they are entailment, neutral,contradiction,
it would be great to have the labels consistent.

thanks

Comments:
Hi ! The ClassLabel feature type encodes the labels as integers.
The integer corresponds to the index of the label name in the `names` list of the ClassLabel.
Here that means that the labels are 'entailment' (0), 'neutral' (1), 'contradiction' (2).

You can get the label names back by using `a.features['label'].int2str(i)`.


Hi! You can also easily reorder the label with the [`Dataset.align_labels_with_mapping`](https://huggingface.co/docs/datasets/master/en/process#align) method.
Title: Updating citation information on LinCE readme
Author: gaguilar (CONTRIBUTOR)
State: closed
Created: 2021-04-11T03:18:05Z

Hi!

I just updated the citation information in this PR. It had an additional bibtex from one of the datasets used in LinCE and then the LinCE bibtex. I removed the former and added a link that shows the full list of citations for each dataset. 

Thanks!

Comments:
Title: Add configurable options to `seqeval` metric
Author: marrodion (CONTRIBUTOR)
State: closed
Created: 2021-04-10T19:58:19Z

Fixes #2148

Adds options to use strict mode, different schemes of evaluation, sample weight and adjust zero_division behavior, if encountered.

`seqeval` provides schemes as objects, hence dynamic import from string, to avoid making the user do the import (thanks to @albertvillanova for the `importlib` idea).

Comments:
Title: Fix ArrowWriter overwriting features in ArrowBasedBuilder
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-09T12:56:19Z

This should fix the issues with CSV loading experienced in #2153 and #2200.

The CSV builder is an ArrowBasedBuilder that had an issue with its ArrowWriter used to write the arrow file from the csv data.
The writer wasn't initialized with the features passed by the user. Therefore the writer was inferring the features from the arrow data, discarding the features passed by the user.

I fixed that and I updated the tests

Comments:
Title: _prepare_split will overwrite DatasetBuilder.info.features
Author: Gforky (NONE)
State: closed
Created: 2021-04-09T11:47:13Z

Hi, here is my issue:
I initialized a Csv datasetbuilder with specific features:
```
def get_dataset_features(data_args):
    features = {}
    if data_args.text_features:
        features.update({text_feature: hf_features.Value("string") for text_feature in data_args.text_features.strip().split(",")})
    if data_args.num_features:
        features.update({text_feature: hf_features.Value("float32") for text_feature in data_args.num_features.strip().split(",")})
    if data_args.label_classes:
        features["label"] = hf_features.ClassLabel(names=data_args.label_classes.strip().split(","))
    else:
        features["label"] = hf_features.Value("float32")
    return hf_features.Features(features)

datasets = load_dataset(extension,
                                data_files=data_files,
                                sep=data_args.delimiter,
                                header=data_args.header,
                                column_names=data_args.column_names.split(",") if data_args.column_names else None,
                                features=get_dataset_features(data_args=data_args))
```
The `features` is printout as below before `builder_instance.as_dataset` is called:
```
{'label': ClassLabel(num_classes=2, names=['unacceptable', 'acceptable'], names_file=None, id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
````

But after the `builder_instance.as_dataset` is called for Csv dataset builder, the `features` is changed to:
```
{'label': Value(dtype='int64', id=None), 'notated': Value(dtype='string', id=None), 'sentence': Value(dtype='string', id=None), 'src_code': Value(dtype='string', id=None)}
```

After digged into the code, I releazed that in `ArrowBasedBuilder._prepare_split`, the DatasetBuilder's info's features will be overwrited by `ArrowWriter`'s `_features`. 
But `ArrowWriter` is initailized without passing `features`.
So my concern is:
It's this overwrite must be done, or, should it be an option to pass features in `_prepare_split` function?

Comments:
Hi ! This might be related to #2153 

You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
I'm opening a PR to fix this and also to figure out how it was not caught in the tests

EDIT: opened #2201

> Hi ! This might be related to #2153
> 
> You're right the ArrowWriter should be initialized with `features=self.info.features` ! Good catch
> I'm opening a PR to fix this and also to figure out how it was not caught in the tests
> 
> EDIT: opened #2201

Glad to hear that! Thank you for your fix, I'm new to huggingface, it's a fantastic project 😁
Title: Fix backward compatibility in Dataset.load_from_disk
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-09T11:01:10Z

Fix backward compatibility when loading from disk an old dataset saved to disk with indices using key "_indices_data_files".

Related to #2195.

Comments:
Hi @lhoestq, could you please check if this makes sense? Thanks.

What about using `_indices_data_files` field in save_to_disk instead of `_indices_files` ?
This way future datasets can also be reloaded from older versions of the lib

`_indices_files` was introduced in a recent PR and was not released

Yes, I have seen it is not released yet...

You are right! It was your awesome PR on Tables which renamed this. If there is no particular reason for this renaming, yes, we could switch it back to the previous `_indices_data_files`. ;)
Title: added file_permission in load_dataset
Author: bhavitvyamalik (CONTRIBUTOR)
State: closed
Created: 2021-04-09T09:39:06Z

As discussed in #2065 I've added `file_permission` argument in `load_dataset`. 

Added mainly 2 things here:
1) Permission of downloaded datasets when converted to .arrow files  can be changed with argument `file_permission` argument in `load_dataset` (default is 0o644 only)
2) Incase the user uses `map` later on to generate another cache file of dataset, it ensures the permissions of newly generated file are similar to that of` *-train.arrow` file inside cache_dir for that dataset.

Comments:
From offline discussions: we want to make the permissions handling consistent with `transformers`. However from discussion  in https://github.com/huggingface/transformers/pull/11119 it looks like it might not be a good solution to provide this argument. Users should use umask for now, and we'll see how things evolve.

@bhavitvyamalik I'm closing the PR for now if you don't mind
Title: `load_dataset` caches two arrow files?
Author: hwijeen (NONE)
State: closed
Created: 2021-04-09T03:49:19Z
Labels: question

Hi,

I am using datasets to load large json file of 587G.
I checked the cached folder and found that there are two arrow files created:
* `cache-ed205e500a7dc44c.arrow` - 355G
*  `json-train.arrow` - 582G

Why is the first file created?
If I delete it, would I still be able to `load_from_disk`?

Comments:
Hi ! Files that starts with `cache-*` are cached computation files, i.e. they are the cached results of map/filter/cast/etc. operations. For example if you used `map` on your dataset to transform it, then the resulting dataset is going to be stored and cached in a `cache-*` file. These files are used to avoid having to load the dataset in RAM, even after many transforms

Thanks @lhoestq! Hmm.. that's strange because I specifically turned off auto caching, and saved mapped result, using `save_to_disk`, to another location. At this location, the following file is created:`355G	cache-ed205e500a7dc44c.arrow`

To my observation, both `load_dataset` and `map` creates `cache-*` files, and I wonder what the `cache-*` file from `load_dataset` is for (as I believe the same information is stored in `json-train.arrow`.

This is a wrong report -- `cache-*` files are created only my `map`, not by `load_dataset`.