rIsHu009/github-issues-updated huggingface dataset:
Title: Optimize int precision
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-03-04T14:12:23Z

Optimize int precision to reduce dataset file size.

Close #1973, close #1825, close #861.

Comments:
@lhoestq, are the tests OK? Some other cases I missed? Do you agree with this approach?

I just tested this and it works like a charm :) 

However tokenizing and then setting the format to "torch" to feed the tokens into a model doesn't seem to work anymore, since the pytorch tensors have the int32/int8 precisions instead of int64 that is required as model inputs.

For example:

```python
import torch
from datasets import Dataset
from transformers import BertModel, BertTokenizer

torch.set_grad_enabled(False)

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertModel.from_pretrained("bert-base-uncased")

dataset = Dataset.from_dict({"text": ["hello there !"]})
dataset = dataset.map(tokenizer, input_columns="text", remove_columns=dataset.column_names)
dataset = dataset.with_format("torch")

print(dataset.features)
# {'attention_mask': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None),
#  'input_ids': Sequence(feature=Value(dtype='int64', id=None), length=-1, id=None),  # this should be int32 though
#  'token_type_ids': Sequence(feature=Value(dtype='int8', id=None), length=-1, id=None)}

model(**dataset[:1])
# RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.CharTensor instead (while checking arguments for embedding)

dataset = dataset.with_format("torch", dtype=torch.int64)

model(**dataset[:1])
# works as expected
```

Pinging @sgugger here to make sure we take the right decision here.

Do we want the "torch" format to always return int64 ? Or does it have to keep the precision defined by the `dataset.features` 
 and therefore we would need to specify "torch" with `dtype=torch.int64` ?

From a user perspective, I think it's fine if the "torch" format converts all ints types to `torch.int64` by default since it's what the model will need almost all the time. I don't see a case where you would want to keep the low precision at the top of my head, and one can always write a custom transform for an edge case.

Sounds good to me !
For consistency maybe we should make the float precision fixed as well (float32, I guess)

Yes, that would be the one used by default.

Do we have the same requirements for TensorFlow?

Yes I we should do the same for tensorflow as well since tf models would have the same issue

Thanks for adding this :)

@lhoestq I think this PR is ready... :)
Title: The size of CoNLL-2003 is not consistant with the official release.
Author: h-peng17 (NONE)
State: closed
Created: 2021-03-04T04:41:34Z

Thanks for the dataset sharing! But when I use conll-2003, I meet some questions.
The statistics of conll-2003 in this repo is  : 
\#train 14041  \#dev 3250 \#test 3453
While the official statistics is:
\#train 14987 \#dev 3466 \#test 3684
Wish for your reply~

Comments:
Hi,

if you inspect the raw data, you can find there are 946 occurrences of `-DOCSTART- -X- -X- O` in the train split and `14041 + 946 = 14987`, which is exactly the number of sentences the authors report. `-DOCSTART-` is a special line that acts as a boundary between two different documents and is filtered out in our implementation.

@lhoestq What do you think about including these lines? ([Link](https://github.com/flairNLP/flair/issues/1097) to a similar issue in the flairNLP repo)

We should mention in the Conll2003 dataset card that these lines have been removed indeed.

If some users are interested in using these lines (maybe to recombine documents ?) then we can add a parameter to the conll2003 dataset to include them.

But IMO the default config should stay the current one (without the `-DOCSTART-` stuff), so that you can directly train NER models without additional preprocessing. Let me know what you think

@lhoestq Yes, I agree adding a small note should be sufficient.

Currently, NLTK's `ConllCorpusReader` ignores the `-DOCSTART-` lines so I think it's ok if we do the same. If there is an interest in the future to use these lines, then we can include them.

I added a mention of this in conll2003's dataset card:
https://github.com/huggingface/datasets/blob/fc9796920da88486c3b97690969aabf03d6b4088/datasets/conll2003/README.md#conll2003

Edit: just saw your PR @mariosasko (noticed it too late ^^)
Let me take a look at it :)
Title: Fix NestedDataStructure.data for empty dict
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-03-03T20:16:51Z

Fix #1981

Comments:
I validated that this fixed the problem, thank you, @albertvillanova!


still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~\.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
    758         # Extract manually downloaded files.
    759         manual_files = dl_manager.extract(manual_paths_dict)
--> 760         extraction_map = dict(downloaded_files, **manual_files)
    761 
    762         for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list

Hi @sabania 
We released a patch version that fixes this issue (1.4.1), can you try with the new version please ?
```
pip install --upgrade datasets
```

I re-validated with the hotfix and the problem is no more.

It's working. thanks a lot.
Title: wmt datasets fail to load
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-03-03T19:21:39Z

on master:
```
python -c 'from datasets import load_dataset; load_dataset("wmt14", "de-en")'
Downloading and preparing dataset wmt14/de-en (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/stas/.cache/huggingface/datasets/wmt14/de-en/1.0.0/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e...
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/load.py", line 740, in load_dataset
    builder_instance.download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 578, in download_and_prepare
    self._download_and_prepare(
  File "/mnt/nvme1/code/huggingface/datasets-master/src/datasets/builder.py", line 634, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/stas/.cache/huggingface/modules/datasets_modules/datasets/wmt14/43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e/wmt_utils.py", line 760, in _split_generators
    extraction_map = dict(downloaded_files, **manual_files)
```

it worked fine recently. same problem if I try wmt16.

git bisect points to this commit from Feb 25 as the culprit https://github.com/huggingface/datasets/commit/792f1d9bb1c5361908f73e2ef7f0181b2be409fa

@albertvillanova

Comments:
@stas00 Mea culpa... May I fix this tomorrow morning?

yes, of course, I reverted to the version before that and it works ;)

but since a new release was just made you will probably need to make a hotfix.

and add the wmt to the tests?

Sure, I will implement a regression test!

@stas00 it is fixed. @lhoestq are you releasing the hot fix or would you prefer me to do it?

I'll do a patch release for this issue early tomorrow.

And yes we absolutly need tests for the wmt datasets: The missing tests for wmt are an artifact from the early development of the lib but now we have tools to generate automatically the dummy data used for tests :)

still facing the same issue or similar:
from datasets import load_dataset
wtm14_test = load_dataset('wmt14',"de-en",cache_dir='./datasets')

~.cache\huggingface\modules\datasets_modules\datasets\wmt14\43e717d978d2261502b0194999583acb874ba73b0f4aed0ada2889d1bb00f36e\wmt_utils.py in _split_generators(self, dl_manager)
758 # Extract manually downloaded files.
759 manual_files = dl_manager.extract(manual_paths_dict)
--> 760 extraction_map = dict(downloaded_files, **manual_files)
761
762 for language in self.config.language_pair:

TypeError: type object argument after ** must be a mapping, not list
Title: Loading all answers from drop
Author: KaijuML (CONTRIBUTOR)
State: closed
Created: 2021-03-03T17:13:07Z

Hello all,

I propose this change to the DROP loading script so that all answers are loaded no matter their type. Currently, only "span" answers are loaded, which excludes a significant amount of answers from drop (i.e. "number" and "date").

I updated the script with the version I use for my work. However, I couldn't find a way to verify that all is working when integrated with the datasets repo, since the `load_dataset` method seems to always download the script from github and not local files.

Note that 9 items from the train set have no answers, as well as 1 from the validation set. The script I propose simply do not load them.

Let me know if there is anything else I can do,
Clément

Comments:
Nice thanks for the change !
This looks all good to me

Before we merge can you just update the dataset_infos.json file of drop ? You can do it by running
```
datasets-cli test ./datasets/drop --all_configs --save_infos --ignore_verifications
```

Done!
Title: Add article_id and process test set template for semeval 2020 task 11…
Author: hemildesai (CONTRIBUTOR)
State: closed
Created: 2021-03-03T10:34:32Z

… dataset

- `article_id` is needed to create the submission file for the task at https://propaganda.qcri.org/semeval2020-task11/
- The `technique classification` task provides the span indices in a template for the test set that is necessary to complete the task. This PR implements processing of that template for the dataset.

Comments:
Thanks !
Now to fix the CI the only thing left is to add a dummy `test-task-tc-template.out` file inside the `dummy_data.zip` at `./datasets/sem_eval_2020_task_11/dummy/1.1.0`
It must contain the labels template for each dummy article of the test set included in `dummy_data.zip`

After that we should be good to merge this one :)

@lhoestq Made the changes! The failure now seems to be unrelated to the changes. Any idea what's going on?

This is a bug on master that we're investigating. You can ignore it
Title: Adding ro sts dataset
Author: lorinczb (CONTRIBUTOR)
State: closed
Created: 2021-03-03T10:08:53Z

Adding [RO-STS](https://github.com/dumitrescustefan/RO-STS) dataset

Comments:
@lhoestq thank you very much for the quick review and useful comments! 

I have tried to address them all, and a few comments that you left for ro_sts I have applied to the ro_sts_parallel as well (in read-me: fixed source_datasets, links to homepage, repository, leaderboard, thanks to me message, in ro_sts_parallel.py changed to camel case as well). In the ro_sts_parallel I have changed the order on the languages, also in the example, as you said order doesn't matter, but just to have them listed in the readme in the same order.

I have commented above on why we would like to keep them as separate datasets, hope it makes sense.

If there is anything else I should change please let me know.

Thanks again!

@lhoestq I tried to adjust the ro_sts_parallel, locally when I run the tests they are passing, but somewhere it has the old name of rosts-parallel-ro-en which I am trying to change to ro_sts_parallel. I don't think I have left anything related to rosts-parallel-ro-en, but when the dataset_infos.json is regenerated it adds it. Could you please help me out, how can I fix this? Thanks in advance!

Great, thanks for all your help!
Title: Add datasets full offline mode with HF_DATASETS_OFFLINE
Author: lhoestq (MEMBER)
State: closed
Created: 2021-03-02T17:26:59Z

Add the HF_DATASETS_OFFLINE environment variable for users who want to use `datasets` offline without having to wait for the network timeouts/retries to happen. This was requested in https://github.com/huggingface/datasets/issues/1939

cc @stas00

Comments:
Title: feat(docs): navigate with left/right arrow keys
Author: ydcjeff (NONE)
State: closed
Created: 2021-03-02T15:24:50Z

Enables docs navigation with left/right arrow keys. It can be useful for the ones who navigate with keyboard a lot.
More info : https://github.com/sphinx-doc/sphinx/pull/2064

You can try here : https://29353-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html

Comments:
Title: Question: what gets stored in the datasets cache and why is it so huge?
Author: ioana-blue (NONE)
State: closed
Created: 2021-03-02T14:35:53Z

I'm running several training jobs (around 10) with a relatively large dataset (3M samples). The datasets cache reached 178G and it seems really large. What is it stored in there and why is it so large? I don't think I noticed this problem before and seems to be related to the new version of the datasets library. Any insight? Thank you!

Comments:
Echo'ing this observation: I have a few datasets in the neighborhood of 2GB CSVs uncompressed, and when I use something like `Dataset.save_to_disk()` it's ~18GB on disk.

If this is unexpected behavior, would be happy to help run debugging as needed.

Thanks @ioana-blue for pointing out this problem (and thanks also @justin-yan). You are right that current implementation of the datasets caching files take too much memory. We are definitely changing this and optimizing the defaults, so that the file sizes are considerably reduced. I will come back to you as soon as this is fixed.

Thank you! Also I noticed that the files don't seem to be cleaned after the jobs finish. Last night I had only 3 jobs running, but the cache was still at 180GB. 

And to clarify, it's not memory, it's disk space. Thank you!

Hi ! As Albert said they can sometimes take more space that expected but we'll fix that soon.

Also, to give more details about caching: computations on a dataset are cached by default so that you don't have to recompute them the next time you run them.

So by default the cache files stay on your disk when you job is finished (so that if you re-execute it, it will be reloaded from the cache).
Feel free to clear your cache after your job has finished, or disable caching using
```python
import datasets

datasets.set_caching_enabled(False)
```

Thanks for the tip, this is useful. 

Hi @ioana-blue, we have optimized Datasets' disk usage in the latest release v1.5.

Feel free to update your Datasets version
```shell
pip install -U datasets
```
and see if it better suits your needs.

Thank you!
Title: Fix ArrowWriter closes stream at exit
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-03-02T07:12:34Z

Current implementation of ArrowWriter does not properly release the `stream` resource (by closing it) if its `finalize()` method is not called and/or an Exception is raised before/during the call to its `finalize()` method.

Therefore, ArrowWriter should be used as a context manager that properly closes its `stream` resource at exit.

Comments:
Oh nice thanks for adding the context manager ! All the streams and RecordBatchWriter will be properly closed now. Hopefully this gives a better experience on windows on which it's super important to close stuff.

Not sure about the error, it looks like a process crashed silently.
Let me take a look

> Hopefully this gives a better experience on windows on which it's super important to close stuff.

Exactly! On Windows, you got:
> PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

when trying to access the unclosed `stream` file, e.g. by `with incomplete_dir(self._cache_dir) as tmp_data_dir`: `shutil.rmtree(tmp_dir)`

The reason is: https://docs.python.org/3/library/os.html#os.remove

> On Windows, attempting to remove a file that is in use causes an exception to be raised; on Unix, the directory entry is removed but the storage allocated to the file is not made available until the original file is no longer in use.




The test passes on my windows. This was probably a circleCI issue. I re-ran the circleCI tests

NICE! It passed!

Maybe you can merge master into this branch and check the CI before merging ?

@lhoestq done! ;)

Thanks ! merging
Title: Add Turkish News Category Dataset - 270K - Lite Version
Author: yavuzKomecoglu (CONTRIBUTOR)
State: closed
Created: 2021-03-01T18:21:59Z

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me, @basakbuluz and @serdarakyol.
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

Comments:
Thanks for the change, merging now !
Title: Fix metrics collision in separate multiprocessed experiments
Author: lhoestq (MEMBER)
State: closed
Created: 2021-03-01T17:45:18Z

As noticed in #1942 , there's a issue with locks if you run multiple separate evaluation experiments in a multiprocessed setup.

Indeed there is a time span in Metric._finalize() where the process 0 loses its lock before re-acquiring it. This is bad since the lock of the process 0 tells the other process that the corresponding cache file is available for writing/reading/deleting: we end up having one metric cache that collides with another one. This can raise FileNotFound errors when a metric tries to read the cache file and if the second conflicting metric deleted it.

To fix that I made sure that the lock file of the process 0 stays acquired from the cache file creation to the end of the metric computation. This way the other metrics can simply sample a new hashing name in order to avoid the collision.

Finally I added missing tests for separate experiments in distributed setup.

Comments:
Since the failure was originally intermittent, there is no 100% telling that the problem is gone. 
But if my artificial race condition setup https://github.com/huggingface/datasets/issues/1942#issuecomment-787124529 is to be the litmus test then the problem has been fixed, as with this PR branch that particular race condition is taken care of correctly.

Thank you for taking care of this, @lhoestq  - locking can be very tricky to do right!
Title: Can we parallelized the add_faiss_index process over dataset shards ?
Author: shamanez (NONE)
State: closed
Created: 2021-03-01T12:47:34Z

I am thinking of making the  **add_faiss_index** process faster. What if we run the add_faiss_index process on separate dataset shards and then combine them before (dataset.concatenate) saving the faiss.index file ?

I feel theoretically this will reduce the accuracy of retrieval since it affects the indexing process.

@lhoestq

Comments:
Hi !
As far as I know not all faiss indexes can be computed in parallel and then merged. 
For example [here](https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes#splitting-and-merging-indexes) is is mentioned that only IndexIVF indexes can be merged.
Moreover faiss already works using multithreading to parallelize the workload over your different CPU cores. You can find more info [here](https://github.com/facebookresearch/faiss/wiki/Threads-and-asynchronous-calls#internal-threading)
So I feel like the gains we would get by implementing a parallel `add_faiss_index` would not be that important, but let me know what you think.


Actually, you are right. I also had the same idea. I am trying this in the context of end-ton-end retrieval training in RAG.  So far I have parallelized the embedding re-computation within the training loop by using datasets shards. 

Then I was thinking of can I calculate the indexes for each shard and combined them with **concatenate**  before I save.

@lhoestq  As you mentioned faiss is already using multiprocessing. I tried to do the add_index with faiss for a dataset object inside a RAY actor and the process became very slow... if fact it takes so much time. It is because a ray actor comes with a single CPU core unless we assign it more. I also tried assigning more cores but still running add_index in the main process is very fast.
Title: bug in SNLI dataset 
Author: dorost1234 (NONE)
State: closed
Created: 2021-02-28T19:36:20Z

Hi
There is label of -1 in train set of SNLI dataset, please find the code below:

```
import numpy as np 
import datasets 
data = datasets.load_dataset("snli")["train"]
labels = []
for d in data:
   labels.append(d["label"])
print(np.unique(labels))
```

and results:

`[-1  0  1  2]`

version of datasets used:
`datasets                  1.2.1                     <pip>
`

thanks for your help. @lhoestq

Comments:
Hi ! The labels -1 correspond to the examples without gold labels in the original snli dataset.
Feel free to remove these examples if you don't need them by using
```python
data = data.filter(lambda x: x["label"] != -1)
```
Title: Fix unused arguments
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-02-28T02:47:07Z

Noticed some args in the codebase are not used, so managed to find all such occurrences with Pylance and fix them.

Comments:
@lhoestq Re-added the arg. The ConnectionError in CI seems unrelated to this PR (the same test fails on master as well).

Thanks !
I'm re-running the CI, maybe this was an issue with circleCI

Looks all good now, merged :)
Title: Allow stateful function in dataset.map
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-02-28T01:29:05Z

Removes the "test type" section in Dataset.map which would modify the state of the stateful function. Now, the return type of the map function is inferred after processing the first example.

Fixes #1940 

@lhoestq Not very happy with the usage of `nonlocal`. Would like to hear your opinion on this.

Comments:
@lhoestq Added a test. If you can come up with a better stateful callable, I'm all ears 😄.  

Sorry I said earlier that it was good to have it inside the loop, my mistake !

@lhoestq Okay, did some refactoring and now the "cache" part comes before the for loop. Thanks for the guidance.

Think this is ready for the final review.
Title: Bug in skip_rows argument of load_dataset function ?
Author: LedaguenelArthur (NONE)
State: closed
Created: 2021-02-27T23:32:54Z

Hello everyone,

I'm quite new to Git so sorry in advance if I'm breaking some ground rules of issues posting... :/
I tried to use the load_dataset function, from Huggingface datasets library, on a csv file using the skip_rows argument described on Huggingface page to skip the first row containing column names

`test_dataset = load_dataset('csv', data_files=['test_wLabel.tsv'], delimiter='\t', column_names=["id", "sentence", "label"], skip_rows=1)`

But I got the following error message

`__init__() got an unexpected keyword argument 'skip_rows'`

Have I used the wrong argument ? Am I missing something or is this a bug ?

Thank you very much for your time,
Best regards,
Arthur

Comments:
Hi,

try `skiprows` instead. This part is not properly documented in the docs it seems.

@lhoestq I'll fix this as part of a bigger PR that fixes typos in the docs.
Title: XSum dataset download link broken
Author: himat (NONE)
State: closed
Created: 2021-02-27T21:47:56Z

I did 
```
from datasets import load_dataset

dataset = load_dataset("xsum")
```

This returns
`ConnectionError: Couldn't reach http://bollin.inf.ed.ac.uk/public/direct/XSUM-EMNLP18-Summary-Data-Original.tar.gz`

Comments:
Never mind, I ran it again and it worked this time. Strange.
Title: [distributed env] potentially unsafe parallel execution
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-02-27T20:38:45Z

```
metric = load_metric('glue', 'mrpc', num_process=num_process, process_id=rank)
```

presumes that there is only one set of parallel processes running - and will intermittently fail if you have multiple sets running as they will surely overwrite each other. Similar to https://github.com/huggingface/datasets/issues/1942 (but for a different reason).
That's why dist environments use some unique to a group identifier so that each group is dealt with separately. 

e.g. the env-way of pytorch dist syncing is done with a unique per set `MASTER_ADDRESS+MASTER_PORT`

So ideally this interface should ask for a shared secret to do the right thing.

I'm not reporting an immediate need, but am only flagging that this will hit someone down the road.

This problem can be remedied by adding a new optional `shared_secret` option, which can then be used to differentiate different groups of processes. and this secret should be part of the file lock name and the experiment.

Thank you

Comments:
You can pass the same `experiment_id` for all the metrics of the same group, and use another `experiment_id` for the other groups.
Maybe we can add an environment variable that sets the default value for `experiment_id` ? What do you think ?

Ah, you're absolutely correct, @lhoestq - it's exactly the equivalent of the shared secret. Thank you!
Title: typos + grammar
Author: stas00 (CONTRIBUTOR)
State: closed
Created: 2021-02-27T20:21:43Z

This PR proposes a few typo + grammar fixes, and rewrites some sentences in an attempt to improve readability.

N.B. When referring to the library `datasets` in the docs it is typically used as a singular, and it definitely is a singular when written as "`datasets` library", that is "`datasets` library is ..." and not "are ...".

Comments:
Title: add a new column 
Author: dorost1234 (NONE)
State: closed
Created: 2021-02-26T18:17:27Z

Hi
I'd need to add a new column to the dataset, I was wondering how this can be done? thanks 
@lhoestq

Comments:
Hi
not sure how change the lable after creation, but this is an issue not dataset request. thanks 

Hi ! Currently you have to use `map` . You can see an example of how to do it in this comment: https://github.com/huggingface/datasets/issues/853#issuecomment-727872188

In the future we'll add support for a more native way of adding a new column ;)
Title: Handle timeouts
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-26T15:02:07Z

As noticed in https://github.com/huggingface/datasets/issues/1939, timeouts were not properly handled when loading a dataset.
This caused the connection to hang indefinitely when working in a firewalled environment cc @stas00 

I added a default timeout, and included an option to our offline environment for tests to be able to simulate both connection errors and timeout errors (previously it was simulating connection errors only).

Now networks calls don't hang indefinitely.
The default timeout is set to 10sec (we might reduce it).

Comments:
I never said the calls were hanging indefinitely, what we need is quite different - in the firewalled env with a network, there should be no network calls or they should fail instantly.

To make this work I suppose on top of this PR we need:
1. `DATASETS_OFFLINE` env var to force set timeout to 0 globally (or to 0.0001 if 0 has a special meaning of no timeout)
2. `DATASETS_OFFLINE` should guard against failing network calls and not fail the program if it has all the data it needs locally.

Bottom line - if the logic wants to check online if the local file matches online dataset name, let it go wild, but it should fail instantly, recover and use the local file - if one is specified explicitly or cache if there is one. And only if neither was found only then assert.

I hope this makes sense and is doable.

I have started on the same approach for transformers https://github.com/huggingface/transformers/pull/10407

Thank you, @lhoestq 

Yes that was the first step to add DATASETS_OFFLINE :)

With this PR, if a request times out (which couldn't happen before because no time out was set), it falls back on the local files with no error.

As you said, setting the timeout to something like 1e-16 makes the requests fail instantly, which is one step forward. One last thing left is to disable request retries and everything will be instant !

Ah, fantastic. Thank you for elucidating that this PR is part of a bigger master plan!  

Merging this one, then I'll open a new PR for the `DATASETS_OFFLINE` env var :)
Title: Add cross-platform support for datasets-cli
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-02-26T14:56:25Z

One thing I've noticed while going through the codebase is the usage of `scripts` in `setup.py`. This [answer](https://stackoverflow.com/a/28119736/14095927) on SO explains it nicely why it's better to use `entry_points` instead of `scripts`. To add cross-platform support to the CLI, this PR replaces `scripts` with `entry_points` in `setup.py` and moves datasets-cli to src/datasets/commands/datasets_cli.py. All *.md and *.rst files are updated accordingly. The same changes were made in the transformers repo to add cross-platform ([link to PR](https://github.com/huggingface/transformers/pull/4131)).

Comments:
@mariosasko This is kinda cool!
Title: Enable Fast Filtering using Arrow Dataset
Author: gchhablani (CONTRIBUTOR)
State: open
Created: 2021-02-26T02:53:37Z

Hi @lhoestq,

As mentioned in Issue #1796, I would love to work on enabling fast filtering/mapping. Can you please share the expectations? It would be great if you could point me to the relevant methods/files involved. Or the docs or maybe an overview of `arrow_dataset.py`. I only ask this because I am having trouble getting started ;-;

Any help would be appreciated.

Thanks,
Gunjan

Comments:
Hi @gchhablani :)
Thanks for proposing your help !

I'll be doing a refactor of some parts related to filtering in the scope of https://github.com/huggingface/datasets/issues/1877
So I would first wait for this refactor to be done before working on the filtering. In particular because I plan to make things simpler to manipulate.

Your feedback on this refactor would also be appreciated since it also aims at making the core code more accessible (basically my goal is that no one's ever "having troubles getting started" ^^)

This will be available in a few days, I will be able to give you more details at that time if you don't mind waiting a bit !

Sure! I don't mind waiting. I'll check the refactor and try to understand what you're trying to do :)
Title: Update documentation with not in place transforms and update DatasetDict
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-25T16:23:18Z

In #1883 were added the not in-place transforms `flatten`, `remove_columns`, `rename_column` and `cast`.

I added them to the documentation and added a paragraph on how to use them

You can preview the documentation [here](https://28862-250213286-gh.circle-artifacts.com/0/docs/_build/html/processing.html#renaming-removing-casting-and-flattening-columns)

I also added these methods to the DatasetDict class.

Comments:
Title: Implement Dataset from CSV
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-02-25T15:10:13Z

Implement `Dataset.from_csv`.

Analogue to #1943.

If finally, the scripts should be used instead, at least we can reuse the tests here.

Comments:
@lhoestq question about public API: `keep_in_memory` or just `in_memory`?

For consistence I'd say `keep_in_memory`, but no strong opinion.

@lhoestq done!
Title: AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
Author: dorost1234 (NONE)
State: closed
Created: 2021-02-25T13:09:45Z

Hi
I am trying to concatenate a list of huggingface datastes as:

` train_dataset = datasets.concatenate_datasets(train_datasets)
`
Here is the `train_datasets` when I print:

```
[Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 120361
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2670
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 6944
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 38140
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 173711
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 1655
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 4274
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2019
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 2109
}), Dataset({
    features: ['attention_mask', 'idx', 'input_ids', 'label', 'question1', 'question2', 'token_type_ids'],
    num_rows: 11963
})]
```

I am getting the following error:

`AttributeError: 'DatasetDict' object has no attribute 'concatenate_datasets'
`

I was wondering if you could help me with this issue, thanks a lot

Comments:
sorry my mistake, datasets were overwritten closing now, thanks a lot
Title: Add Turkish News Category Dataset (270K - Lite Version)
Author: yavuzKomecoglu (CONTRIBUTOR)
State: closed
Created: 2021-02-25T09:45:22Z

This PR adds the Turkish News Categories Dataset (270K - Lite Version) dataset which is a text classification dataset by me,  @basakbuluz and @serdarakyol. 
This dataset contains the same news from the current [interpress_news_category_tr dataset](https://huggingface.co/datasets/interpress_news_category_tr) but contains less information, OCR errors are reduced, can be easily separated, and can be divided into 10 classes ("kültürsanat", "ekonomi", "siyaset", "eğitim", "dünya", "spor", "teknoloji", "magazin", "sağlık", "gündem") were rearranged.

@SBrandeis @lhoestq, can you please review this PR?

Comments:
I updated your suggestions. Thank you very much for your support. @lhoestq 

> Thanks for changing to ClassLabel :)
> This is all good now !
> 
> However I can see changes in other files than the ones for interpress_news_category_tr_lite, can you please fix that ?
> To do so you can create another branch and another PR to only include the interpress_news_category_tr_lite files.
> 
> Maybe this happened because of a git rebase ? Once you've already pushed your code, please use git merge instead of rebase in order to avoid this.

Thanks for the feedback.
New PR https://github.com/huggingface/datasets/pull/1967
Title: Implement Dataset from JSON and JSON Lines
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-02-25T07:17:33Z

Implement `Dataset.from_jsonl`.

Comments:
Thanks @lhoestq. I was trying to follow @thomwolf suggestion about integrating that script but as `from_json` method...
> Note that I don't think this is necessary a breaking change, we can still keep the old scripts around

Do you think there is a better way of doing it?

I was trying to implement more or less the same logic as in the script, but I confess I assumed the target was in-memory only...

Basically, I was trying to reimplement `Json(datasets.ArrowBasedBuilder)._generate_tables`, and no writing to arrow file (I assumed only in-memory usage). I started with the first "else" clause... 

I was planning to remove my `_cast_table_to_info_features` and use `paj.read_json(parse_options=...)` instead (like in the script).

@lhoestq I am wondering why `keep_in_memory` has no effect for JSON...

What's the issue exactly ? Apparently it's correctly passed to as_dataset so I don't find the issue

Nevermind @lhoestq, I found where the problem was in my code... I push!


merging master into this branch should fix the CI issue :)


Oops I didn't refresh the page sorry ^^'

Looks all good !

Good job ! I think we can merge after the last changes regarding the error message and the docstring above :)

@lhoestq Done! And I have also added some tests for the `field` parameter.

Let me add some more tests for dict of lists JSON file, please.

@lhoestq done! ;)

We can merge. Additional work will be done in another PR. ;)
Title: CommonGen dataset page shows an error OSError: [Errno 28] No space left on device
Author: yuchenlin (CONTRIBUTOR)
State: closed
Created: 2021-02-24T06:47:33Z
Labels: nlp-viewer

The page of the CommonGen data https://huggingface.co/datasets/viewer/?dataset=common_gen  shows 
![image](https://user-images.githubusercontent.com/10104354/108959311-1865e600-7629-11eb-868c-cf4cb27034ea.png)

Comments:
Facing the same issue for [Squad](https://huggingface.co/datasets/viewer/?dataset=squad) and [TriviaQA](https://huggingface.co/datasets/viewer/?dataset=trivia_qa) datasets as well.

We just fixed the issue, thanks for reporting !
Title: add CoVoST2
Author: patil-suraj (CONTRIBUTOR)
State: closed
Created: 2021-02-23T16:28:16Z

This PR adds the CoVoST2 dataset for speech translation and ASR.
https://github.com/facebookresearch/covost#covost-2

The dataset requires manual download as the download page requests an email address and the URLs are temporary.

The dummy data is a bit bigger because of the mp3 files and 36 configs.

Comments:
@patrickvonplaten 
I removed the mp3 files, dummy_data is much smaller now!
Title: Add Stanford Sentiment Treebank (SST)
Author: patpizio (CONTRIBUTOR)
State: closed
Created: 2021-02-23T12:53:16Z
Labels: dataset request

I am going to add SST:

- **Name:** The Stanford Sentiment Treebank
- **Description:** The first corpus with fully labeled parse trees that allows for a complete analysis of the compositional effects of sentiment in language
- **Paper:** [Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank](https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf)
- **Data:** https://nlp.stanford.edu/sentiment/index.html
- **Motivation:** Already requested in #353, SST is a popular dataset for Sentiment Classification

What's the difference with the [_SST-2_](https://huggingface.co/datasets/viewer/?dataset=glue&config=sst2) dataset included in GLUE? Essentially, SST-2 is a version of SST where:
- the labels were mapped from real numbers in [0.0, 1.0] to a binary label: {0, 1}
- the labels of the *sub-sentences* were included only in the training set
- the labels in the test set are obfuscated

So there is a lot more information in the original SST. The tricky bit is, the data is scattered into many text files and, for one in particular, I couldn't find the original encoding ([*but I'm not the only one*](https://groups.google.com/g/word2vec-toolkit/c/QIUjLw6RqFk/m/_iEeyt428wkJ) 🎵). The only solution I found was to manually replace all the è, ë, ç and so on into an `utf-8` copy of the text file. I uploaded the result in my Dropbox and I am using that as the main repo for the dataset.

Also, the _sub-sentences_ are built at run-time from the information encoded in several text files, so generating the examples is a bit more cumbersome than usual. Luckily, the dataset is not enormous.

I plan to divide the dataset in 2 configs: one with just whole sentences with their labels, the other with sentences _and their sub-sentences_ with their labels. Each config will be split in train, validation and test. Hopefully this makes sense, we may discuss it in the PR I'm going to submit.

Comments:
Dataset added in release [1.5.0](https://github.com/huggingface/datasets/releases/tag/1.5.0), I think I can close this.
Title: Use arrow ipc file format
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-23T10:38:24Z

According to the [documentation](https://arrow.apache.org/docs/format/Columnar.html?highlight=arrow1#ipc-file-format), it's identical to the streaming format except that it contains the memory offsets of each sample:

> We define a “file format” supporting random access that is build with the stream format. The file starts and ends with a magic string ARROW1 (plus padding). What follows in the file is identical to the stream format. At the end of the file, we write a footer containing a redundant copy of the schema (which is a part of the streaming format) plus memory offsets and sizes for each of the data blocks in the file. This enables random access any record batch in the file. See File.fbs for the precise details of the file footer.

Since it stores more metadata regarding the positions of the examples in the file, it should enable better example retrieval performances. However from the discussion in https://github.com/huggingface/datasets/issues/1803 it looks like it's not the case unfortunately. Maybe in the future this will allow speed gains.

I think it's still a good idea to start using it anyway for these reasons:
- in the future we may have speed gains
- it contains the arrow streaming format data
- it's compatible with the pyarrow Dataset implementation (it allows to load remote dataframes for example) if we want to use it in the future
- it's also the format used by arrow feather if we want to use it in the future
- it's roughly the same size as the streaming format
- it's easy to have backward compatibility with the streaming format

Comments:
Should we close this PR?

Yes, this one was mostly related to https://github.com/huggingface/datasets/issues/4542 but now I think the TF support is not needed at the moment.

What about enabling the Arrow IPC format through an environment variable/config? Would be very helpful to better interop. with other libraries (e.g. [polars](https://pola-rs.github.io/polars/py-polars/html/reference/api/polars.scan_ipc.html)) + supporting reads through pyarrow datasets.
Title: Fix builder config creation with data_dir
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-23T10:26:02Z

The data_dir parameter wasn't taken into account to create the config_id, therefore the resulting builder config was considered not custom. However a builder config that is non-custom must not have a name that collides with the predefined builder config names. Therefore it resulted in a `ValueError("Cannot name a custom BuilderConfig the same as an available...")`

I fixed that by commenting the line that used to ignore the data_dir when creating the config.

It was previously ignored before the introduction of config id because we didn't want to change the config name. Now it's fine to take it into account for the config id.

Now creating a config with a data_dir works again @patrickvonplaten

Comments:
Title: add m_lama (multilingual lama) dataset
Author: pdufter (CONTRIBUTOR)
State: closed
Created: 2021-02-23T08:11:57Z

Add a multilingual (machine translated and automatically generated) version of the LAMA benchmark. For details see the paper https://arxiv.org/pdf/2102.00894.pdf

Comments:
Hi, it seems I am somewhat stuck here. The failed test `ci/circleci: run_dataset_script_tests_pyarrow_1_WIN` seems to be caused by some broken connection (`ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host`). Any help on this is appreciated. 

Edit: Seems to be resolved now.

I guess the `dummy_data.zip` is too large. I can reduce the languages that are contained there, but when testing it, it obviously throws an error, as not all files can be found. I guess I can either i) change the default value regarding which languages are loaded or ii) let the `_generate_examples` silently skip any language for which it cannot find files. Both solutions are not really pretty - is there another way around this?

Thanks for the review and the constructive comments :) ! I tried to address them, and reduced the number of lines in the dummy data to 1 to reduce its size.
Title: updated the wino_bias dataset
Author: JieyuZhao (CONTRIBUTOR)
State: closed
Created: 2021-02-23T03:07:40Z

Updated the wino_bias.py script.
- updated the data_url
- added different configurations for different data splits
- added the coreference_cluster to the data features

Comments:
Hi @JieyuZhao ! Have you had a chance to add the different configurations ?
Thanks again for your help on this !

> Hi @JieyuZhao ! Have you had a chance to add the different configurations ?
> Thanks again for your help on this !

Hi @lhoestq  Yes, I've updated the code. Now the configuration will have dev/test splits.

> Cool thanks !
> This looks perfect this way.
> 
> Now we just need to update the dataset_infos.json (it contains the metadata of the dataset) and add dummy data to be able to test this script automatically.
> 
> To update the dataset_infos.json you just need delete the current one at `./datasets/wino_biais/dataset_infos.json`, and then run this command:
> 
> ```
> datasets-cli test ./datasets/wino_biais --save_infos --all_configs --ignore_verifications
> ```
> 
> To add the dummy data there's also a tool to add them automatically.
> First delete the folder at `./datasets/wino_biais/dummy` and then run
> 
> ```
> datasets-cli dummy_data ./datasets/wino_biais --auto_generate --match_text_files "*conll" --n_lines 15
> ```
> 
> Let me know if you have questions :)
> Also don't forget to run `make style` to format the code properly.

Thanks for the instruction! I've updated the metadata and the dummy data and also do the formatting. Please let me know if more is needed. :)
Title: Improve typing and style and fix some inconsistencies
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-02-22T22:47:41Z

This PR:
* improves typing (mostly more consistent use of `typing.Optional`)
* `DatasetDict.cleanup_cache_files` now correctly returns a dict 
* replaces `dict()` with the corresponding literal
* uses `dict_to_copy.copy()` instead of `dict(dict_to_copy)` for shallow copying

Comments:
@lhoestq Thanks for the quick review.

I merged master to this branch to re-run the CI before merging :)
Title: Updating old cards
Author: mcmillanmajora (CONTRIBUTOR)
State: closed
Created: 2021-02-22T19:26:04Z

Updated the cards for [Allocine](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/allocine), [CNN/DailyMail](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/cnn_dailymail), and [SNLI](https://github.com/mcmillanmajora/datasets/tree/updating-old-cards/datasets/snli). For the most part, the information was just rearranged or rephrased, but the social impact statements are new.

Comments:
Title: Update dataset card of wino_bias
Author: JieyuZhao (CONTRIBUTOR)
State: closed
Created: 2021-02-22T18:51:34Z
Labels: dataset contribution

Updated the info for the wino_bias dataset.

Comments:
Thanks @JieyuZhao.

I think this PR was superseded by your other PRs:
- #1930
- #2152 

I'm closing this.
Title: Fix: Wiki_dpr - add missing scalar quantizer
Author: lhoestq (MEMBER)
State: closed
Created: 2021-02-22T15:32:05Z

All the prebuilt wiki_dpr indexes already use SQ8, I forgot to update the wiki_dpr script after building them. Now it's finally done.

The scalar quantizer SQ8 doesn't reduce the performance of the index as shown in retrieval experiments on RAG.
The quantizer reduces the size of the index a lot but increases index building time.

Comments:
Title: Anonymous Dataset Addition (i.e Anonymous PR?)
Author: PierreColombo (CONTRIBUTOR)
State: closed
Created: 2021-02-22T15:22:30Z

Hello,
Thanks a lot for your librairy.
We plan to submit a paper on OpenReview using the Anonymous setting. Is it possible to add a new dataset without breaking the anonimity, with a link to the paper ? 
Cheers 
@eusip

Comments:
Hi !
I guess you can add a dataset without the fields that must be kept anonymous, and then update those when the anonymity period is over.
You can also make the PR from an anonymous org.
Pinging @yjernite just to make sure it's ok

Hello,
I would prefer to do the reverse: adding a link to an anonymous paper without the people names/institution in the PR. Would it be conceivable ?
Cheers


Sure, I think it's ok on our side

Yup, sounds good!