rIsHu009/github-issues-updated huggingface dataset:
Title: KeyError: '_indices_files' in `arrow_dataset.py`
Author: samsontmr (NONE)
State: closed
Created: 2021-04-09T01:37:12Z
Labels: bug

After pulling the latest master, I'm getting a crash when `load_from_disk` tries to load my local dataset.

Trace:
```
Traceback (most recent call last):
  File "load_data.py", line 11, in <module>
    dataset = load_from_disk(SRC)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/load.py", line 784, in load_from_disk
    return DatasetDict.load_from_disk(dataset_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/dataset_dict.py", line 692, in load_from_disk
    dataset_dict[k] = Dataset.load_from_disk(dataset_dict_split_path, fs, keep_in_memory=keep_in_memory)
  File "/opt/conda/envs/py38/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 634, in load_from_disk
    if state["_indices_files"]:
KeyError: '_indices_files'
```

I believe this is the line causing the error since there may not be a `_indices_files` key in the older versions:
https://github.com/huggingface/datasets/blob/b70141e3c5149430951773aaa0155555c5fb3e76/src/datasets/arrow_dataset.py#L634

May I suggest using `state.get()` instead of directly indexing the dictionary?

@lhoestq

Comments:
Thanks for reporting @samsontmr.

It seems a backward compatibility issue...

Thanks @samsontmr this should be fixed on master now

Feel free to reopen if you're still having issues
Title: Refactorize tests to use Dataset as context manager
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-08T11:21:04Z
Labels: refactoring

Refactorize Dataset tests to use Dataset as context manager.

Comments:
I find very interesting that idea of using a fixture instead!

Let me rework a little bit this PR, @lhoestq.

@lhoestq, as this is a big refactoring, I had many problems to solve the conflicts with the master branch...

Therefore, I think it is better to merge this as it is, and then to make other PRs with additional refactorings, before I get conflicts again with the master branch...

There are still some conflicts that prevent merging.
Moreover I noticed that you added one fixture per method of the Dataset object to be mocked. The code of all these fixtures is pretty much the same, feel free to factorize them into one fixture.

Also feel free to create another branch from `master` if you don't want to fix the conflicts of this branch.
Let me know if I can help you on this

@lhoestq, yes, the new conflicts appeared after today merge commits on master...

I am definitely going to split this PR into smaller ones in order to avoid having to resolve many conflicts after each commit on master. There are lots of conflicts and these are painful to resolve.
Title: News_commentary Dataset Translation Pairs are of Incorrect Language Specified Pairs
Author: anassalamah (NONE)
State: closed
Created: 2021-04-08T07:53:43Z

I used load_dataset to load the news_commentary dataset for "ar-en" translation pairs but found translations from Arabic to Hindi.  

```
train_ds = load_dataset("news_commentary", "ar-en", split='train[:98%]')
val_ds = load_dataset("news_commentary", "ar-en", split='train[98%:]')

# filtering out examples that are not ar-en translations but ar-hi
val_ds = val_ds.filter(lambda example, indice: indice not in chain(range(1312,1327) ,range(1384,1399), range(1030,1042)), with_indices=True)
```

* I'm fairly new to using datasets so I might be doing something wrong

Comments:
Hi @anassalamah,

Could you please try with this:
```python
train_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[:98%]')
val_ds = load_dataset("news_commentary", lang1="ar", lang2="en", split='train[98%:]')
```

Hello @albertvillanova, 

Thanks for the suggestion. I didn't know you could do that. however, it didn't resolve the issue

![image](https://user-images.githubusercontent.com/8571003/114169966-ec819400-993a-11eb-8a67-930f9a9b2290.png)
Title: save_to_disk doesn't work when we use concatenate_datasets function before creating the final dataset_object.
Author: shamanez (NONE)
State: closed
Created: 2021-04-08T04:42:53Z

As you can see, it saves the entire dataset.

@lhoestq 

You can  check by going through the following example,

```
from datasets import load_from_disk,concatenate_datasets

loaded_data=load_from_disk('/home/gsir059/HNSW-ori/my_knowledge_dataset')
n=20
kb_list=[loaded_data.shard(n, i, contiguous=True) for i in range(n)]
final_dataset=concatenate_datasets([kb_list[1],kb_list[2]])
final_dataset.save_to_disk('/home/gsir059/haha/k.arrow')
```

Comments:
Hi ! We refactored save_to_disk in #2025 so this doesn't happen.
Feel free to try it on master for now
We'll do a new release soon
Title: Duplicate data in Timit dataset
Author: thanh-p (NONE)
State: closed
Created: 2021-04-08T04:21:54Z

I ran a simple code to list all texts in Timit dataset and the texts were all the same.
Is this dataset corrupted?
**Code:**
timit = load_dataset("timit_asr")
print(*timit['train']['text'], sep='\n')
**Result:**
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
Would such an act of refusal be useful?
...
...
Would such an act of refusal be useful?

Comments:
Hi ! Thanks for reporting
If I recall correctly this has been recently fixed #1995
Can you try to upgrade your local version of `datasets` ?
```
pip install --upgrade datasets
```

Hi Ihoestq,

Thank you. It works after upgrading the datasets
Title: GEM: new challenge sets
Author: yjernite (MEMBER)
State: closed
Created: 2021-04-07T21:39:07Z

This PR updates the GEM dataset to:
- remove extraneous fields in WikiAuto after https://github.com/huggingface/datasets/pull/2171 fixed the source
- add context and services to Schema Guided Dialog
- Add new or update challenge sets for MLSUM ES and DE, XSUM, and SGD

Comments:
cc @sebastiangehrmann
Title: Fix s3fs tests for py36 and py37+
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-07T15:17:11Z

Recently several changes happened:
1. latest versions of `fsspec` require python>3.7 for async features
2. `s3fs` added a dependency on `aiobotocore`, which is not compatible with the `moto` s3 mock context manager

This PR fixes both issues, by pinning `fsspec` and `s3fs` for python 3.6, and by using `moto` in server mode to support running the tests on python>=3.7 with the latest version of `fsspec` and `s3fs`.

cc @philschmid

Comments:
Title: Set default in-memory value depending on the dataset size
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-07T13:00:18Z
Labels: enhancement

Set a default value for `in_memory` depending on the size of the dataset to be loaded.

Close #2179.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

Comments:
I ping @krandiash to keep him up to date.

TODO:
- [x] Add a section in the docs about this.
- ~Add a warning if someone tries to specify `cache_file_name=` in `map`, `filter` etc. on a dataset that is in memory, since the computation is not going to be cached in this case.~

@lhoestq I have a question, regarding:
> Also maybe we should add a warning if someone tries to specify cache_file_name= in map, filter etc. on a dataset that is in memory, since the computation is not going to be cached in this case.

- It might be the case that the user has an in-memory dataset and might want to use `map` and cache it, by passing `cache_file_name=`
- This is indeed allowed by the library and works as expected: the dataset is cached.

Why adding a warning?

Yes right, I meant if `load_from_cache_file` is set to True and `cache_file_name ` is None. my bad :p
Title: Load small datasets in-memory instead of using memory map
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-07T09:58:16Z
Labels: enhancement

generic discussion

Currently all datasets are loaded using memory mapping by default in `load_dataset`.
However this might not be necessary for small datasets. If a dataset is small enough, then it can be loaded in-memory and:
- its memory footprint would be small so it's ok
- in-memory computations/queries would be faster
- the caching on-disk would be disabled, making computations even faster (no I/O bound because of the disk)
- but running the same computation a second time would recompute everything since there would be no cached results on-disk. But this is probably fine since computations would be fast anyway + users should be able to provide a cache filename if needed.

Therefore, maybe the default behavior of `load_dataset` should be to load small datasets in-memory and big datasets using memory mapping.

Comments:
Title: Fix cast memory usage by using map on subtables
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-07T09:30:50Z
Labels: enhancement

The `cast` operation on a pyarrow Table may create new arrays in memory.
This is an issue since users expect memory mapped datasets to not fill up the RAM.

To fix that I used `map` to write a new arrow file on disk when cast is used.
To make things more convenient I introduced the `arrow` formatting of a dataset, to make it return pyarrow tables instead of python dicts. This way one can use pyarrow transforms directly when using `map`.

edit: we'll use the same mechanism for `filter`

Comments:
I addressed your comments about the docstrings and the output validation :)

I updated the bleurt mocking method and bleurt test is passing now.
I also ran the slow tests and they are passing for bleurt.

Thanks @lhoestq and @albertvillanova !
Title: add social thumbnial
Author: philschmid (CONTRIBUTOR)
State: closed
Created: 2021-04-07T06:40:06Z

# What does this PR do?

I added OpenGraph/ Twitter Card support to the docs to create nice social thumbnails.

![Bildschirmfoto 2021-04-07 um 08 36 50](https://user-images.githubusercontent.com/32632186/113821698-bac2ce80-977c-11eb-81aa-d8f16355857e.png)

To be able to add these I needed to install `sphinxext-opengraph`. I came across this [issue](https://github.com/readthedocs/readthedocs.org/issues/1758) on the readthedocs repo saying that since someone has built this plugin they are not integrating and providing documentation to it. That's why I added it for creating the documentation. The repository can be found [here](https://github.com/wpilibsuite/sphinxext-opengraph/tree/main).

P.S. It seemed that `make style` never ran for `docs/` i hope the changes are okay otherwise I'll revert it.

Comments:
Title: Converting a Value to a ClassLabel
Author: nelson-liu (NONE)
State: closed
Created: 2021-04-06T22:54:16Z
Labels: enhancement

Hi!

In the docs for `cast`, it's noted that `For non-trivial conversion, e.g. string <-> ClassLabel you should use map() to update the Dataset.`

Would it be possible to have an example  that demonstrates such a string <-> ClassLabel conversion using `map`? Thanks!

Comments:
Hi @nelson-liu!
Here is what I do to convert a string to class label:

```python
from datasets import load_dataset, features


dset = load_dataset(...)
col_name = "the string column name"

class_names  = dset.unique(col_name)
class_feature = features.ClassLabel(names=sorted(class_names))
dset = dset.map(lambda str_value: {col_name: class_feature.str2int(str_value)}, input_columns=col_name)

dset = dset.cast(features.Features({
    ...
    col_name: class_feature
})
```


Hi! You can use `Dataset.class_encode_column` for this. And in the next release of `datasets` (this feature is only available on `master`), you'll also be able to use `cast` to do the conversion. 

An example of conversion via `cast`: 
```python
from datasets import Dataset, Features, ClassLabel
d = Dataset.from_dict({"a": ["no", "yes", "no"]})
d = d.cast(Features({"a": ClassLabel(names=["yes", "no"])}))
```
Title: Pin docutils for better doc
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2021-04-06T12:40:20Z

The latest release of docutils make the navbar in the documentation weird and the Markdown wrongly interpreted:

![image](https://user-images.githubusercontent.com/35901082/113711773-5be55280-96b3-11eb-9b3b-9794f17709aa.png)

We had the same problem in Transformers and solved it by pinning docutils (a dep of sphinx).

You can see the version after the change [here](https://32769-250213286-gh.circle-artifacts.com/0/docs/_build/html/index.html).

Comments:
Title: Add OpenSLR dataset
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-04-06T12:08:34Z

OpenSLR (https://openslr.org/) is a site devoted to hosting speech and language resources, such as training corpora for speech recognition, and software related to speech recognition. There are around 80 speech datasets listed in OpenSLR, currently this PR includes only 9 speech datasets SLR41, SLR42, SLR43, SLR44, SLR63, SLR64, SLR65, SLR66 and SLR69 (Javanese, Khmer, Nepali and Sundanese, Malayalam, Marathi, Tamil, Telugu and Catalan). I can add other speech datasets gradually next time.

Comments:
Title: Pin fsspec lower than 0.9.0
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-06T09:19:09Z

Today's release of `fsspec` 0.9.0 implied a new release of `s3fs` 0.6.0 but this version breaks the CI (see [here](https://app.circleci.com/pipelines/github/huggingface/datasets/5312/workflows/490f3240-cd1c-4dd1-bb60-b416771c5584/jobs/32734) for example)

I'm pinning `fsspec` until this has been resolved

Comments:
Title: Fixed the link to wikiauto training data.
Author: mounicam (CONTRIBUTOR)
State: closed
Created: 2021-04-06T07:13:11Z


Comments:
Also you can ignore the CI failing on `docs`, this has been fixed on master :)

@lhoestq I need to update other stuff on GEM later today too, so will merge this one and remove columns in the next PR!

Ok !
Title: Updated WER metric implementation to avoid memory issues
Author: diego-fustes (NONE)
State: closed
Created: 2021-04-05T15:43:20Z

This is in order to fix this issue:

https://github.com/huggingface/datasets/issues/2078

Comments:
Hi ! Thanks for suggesting this fix 
Unfortunately it looks like it's already been fixed by #2111 

Feel free to share your thoughts about this PR !

I'm closing this one if you don't mind.
Title:  Split type not preserved when reloading the dataset
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-04-04T19:29:54Z

A minimal reproducible example:
```python
>>> from datasets import load_dataset, Dataset
>>> dset = load_dataset("sst", split="train")
>>> dset.save_to_disk("sst")
>>> type(dset.split)
<class 'datasets.splits.NamedSplit'>
>>> dset = Dataset.load_from_disk("sst")
>>> type(dset.split)  # NamedSplit expected
<class 'str'>
```

It seems like this bug was introduced in #2025.

Comments:
Title: Regarding Test Sets for the GEM datasets
Author: vyraun (NONE)
State: closed
Created: 2021-04-04T02:02:45Z
Labels: Dataset discussion

@yjernite Hi, are the test sets for the GEM datasets scheduled to be [added soon](https://gem-benchmark.com/shared_task)? 

e.g.

```
from datasets import load_dataset
DATASET_NAME="common_gen"
data = load_dataset("gem", DATASET_NAME)
```

The test set doesn't have the target or references.

```
data['test'][0]
{'concept_set_id': 0, 'concepts': ['drill', 'field', 'run', 'team'], 'gem_id': 'common_gen-test-0', 'gem_parent_id': 'common_gen-test-0', 'references': [], 'target': ''}
```

Comments:
Hi @vyraun ! The test references for CommonGen are not publicly available: you can reach out to the original dataset authors if you would like to ask for them, but we will not be releasing them as part of GEM (March 31st was the release date for the test set inputs, references are incidentally released for some of the test sets but shouldn't really be used for benchmark submissions)

cc @sebastiangehrmann

Oh okay, thanks @yjernite !
Title: Concat only unique fields in DatasetInfo.from_merge
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-04-03T14:31:30Z

I thought someone from the community with less experience would be interested in fixing this issue, but that wasn't the case.

Fixes #2103

Comments:
Hi @mariosasko,
Just came across this PR and I was wondering if we can use
`description = "\n\n".join(OrderedDict.fromkeys([info.description for info in dataset_infos]))`

This will obviate the need for `unique` and is almost as fast as `set`. We could have used `dict` inplace of `OrderedDict` but it's available 3.7+ onwards

Hi,

let's see what @lhoestq thinks. Although my approach adds more code, it's more readable IMO.

Yeah, that's true. Your approach is more readable.
Title: visualization for cc100 is broken 
Author: dorost1234 (NONE)
State: closed
Created: 2021-04-02T10:11:13Z
Labels: nlp-viewer

Hi
visualization through dataset viewer for cc100 is broken
https://huggingface.co/datasets/viewer/

thanks a lot

Comments:
This looks like an issue with the cc100 dataset itself but not sure
Did you try loading cc100 on your machine ?

Hi
loading works fine, but the viewer only is broken
thanks

On Wed, Apr 7, 2021 at 12:17 PM Quentin Lhoest ***@***.***>
wrote:

> This looks like an issue with the cc100 dataset itself but not sure
> Did you try loading cc100 on your machine ?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2162#issuecomment-814793809>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMRUO33JSOYGT6RETWLTHQWNLANCNFSM42IUOR6Q>
> .
>


Hi! This visualization tool is deprecated now. The viewer at https://huggingface.co/datasets/cc100 works fine, so I'm closing this issue.
Title: any possibility to download part of large datasets only?
Author: dorost1234 (NONE)
State: closed
Created: 2021-04-02T10:06:46Z

Hi
Some of the datasets I need like cc100 are very large, and then I wonder if I can download first X samples of the shuffled/unshuffled data without going through first downloading the whole data then sampling? thanks

Comments:
Not yet but it’s on the short/mid-term roadmap (requested by many indeed).

oh, great, really awesome feature to have, thank you very much for the great, fabulous work

We'll work on dataset streaming soon. This should allow you to only load the examples you need ;)

thanks a lot Quentin, this would be really really a great feature to have

On Wed, Apr 7, 2021 at 12:14 PM Quentin Lhoest ***@***.***>
wrote:

> We'll work on dataset streaming soon. This should allow you to only load
> the examples you need ;)
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2161#issuecomment-814791922>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMROD62QAKIJMAKWISTTHQWBVANCNFSM42IUI5JQ>
> .
>


Is streaming completed? On the 1.8.0 docs it is mentioned (https://huggingface.co/docs/datasets/dataset_streaming.html), but when following the example I get the following error:

```
>>> dataset2 = load_dataset("amazon_us_reviews", "Pet_Products_v1_00", split='train', streaming=True)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-21-1eedab26cff1> in <module>()
----> 1 en_dataset = load_dataset('oscar', "unshuffled_deduplicated_en", split='train', streaming=True)

3 frames
/usr/local/lib/python3.7/dist-packages/datasets/builder.py in _create_builder_config(self, name, custom_features, **config_kwargs)
    339                 if value is not None:
    340                     if not hasattr(builder_config, key):
--> 341                         raise ValueError(f"BuilderConfig {builder_config} doesn't have a '{key}' key.")
    342                     setattr(builder_config, key, value)
    343 

ValueError: BuilderConfig OscarConfig(name='unshuffled_deduplicated_en', version=1.0.0, data_dir=None, data_files=None, description='Unshuffled and deduplicated, English OSCAR dataset') doesn't have a 'streaming' key.
```

UPDATE: Managed to get streaming working by building from source and installing the additional `datasets[streaming]` package:

```
!pip install git+https://github.com/huggingface/datasets.git
!pip install datasets[streaming]
```

Hi ! Streaming is available on `master` only right now. We'll make a new release 1.9.0 on Monday :)
Title: data_args.preprocessing_num_workers almost freezes 
Author: dorost1234 (NONE)
State: closed
Created: 2021-04-02T07:56:13Z

Hi @lhoestq 

I am running this code from huggingface transformers https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py 

to speed up tokenization, since I am running on multiple datasets, I am using data_args.preprocessing_num_workers = 4 with opus100 corpus but this moves on till a point and then this freezes almost for sometime during  tokenization steps and then this is back again, overall to me taking more time than normal case, I appreciate your advice on how I can use this option properly to speed up.

thanks

Comments:
Hi.
I cannot always reproduce this issue, and on later runs I did not see it so far. Sometimes also I set 8 processes but I see less being showed, is this normal, here only 5 are shown for 8 being set, thanks

```
#3:  11%|███████████████▊                                                                                                                                  | 172/1583 [00:46<06:21,  3.70ba/s]
#4:   9%|█████████████▏                                                                                                                                    | 143/1583 [00:46<07:46,  3.09ba/s]
#7:   6%|█████████                                                                                                                                          | 98/1583 [00:45<11:34,  2.14ba/s]
#5:   8%|███████████▍                                                                                                                                      | 124/1583 [00:46<09:03,  2.68ba/s]
#6:   7%|██████████▏  
```

closing since I cannot reproduce it again, thanks
Title: adding ccnet dataset
Author: dorost1234 (NONE)
State: closed
Created: 2021-04-01T23:28:36Z
Labels: dataset request

## Adding a Dataset
- **Name:**  ccnet

- **Description:** 
Common Crawl

- **Paper:** 
https://arxiv.org/abs/1911.00359

- **Data:** 
https://github.com/facebookresearch/cc_net

- **Motivation:**
this is one of the most comprehensive clean monolingual datasets across a variety of languages. Quite important for cross-lingual reseach

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

thanks

Comments:
closing since I think this is cc100, just the name has been changed. thanks
Title: viewer "fake_news_english" error
Author: emanuelevivoli (NONE)
State: closed
Created: 2021-04-01T14:13:20Z
Labels: nlp-viewer

When I visit the [Huggingface - viewer](https://huggingface.co/datasets/viewer/) web site, under the dataset "fake_news_english" I've got this error:

> ImportError: To be able to use this dataset, you need to install the following dependencies['openpyxl'] using 'pip install # noqa: requires this pandas optional dependency for reading xlsx files' for instance'

as well as the error Traceback.

Comments:
Thanks for reporting !
The viewer doesn't have all the dependencies of the datasets. We may add openpyxl to be able to show this dataset properly

This viewer tool is deprecated now and the new viewer at https://huggingface.co/datasets/fake_news_english works fine, so I'm closing this issue
Title: Add table classes to the documentation
Author: lhoestq (MEMBER)
State: closed
Created: 2021-03-31T14:36:10Z

Following #2025 , I added the table classes to the documentation

cc @albertvillanova

Comments:
Just note that docstrings injected from PyArrow do not follow the same convention for formatting types in `Args` or `Returns` as we do... Not a big problem, anyway! 😄
Title: Adding the NorNE dataset for Norwegian POS and NER
Author: versae (CONTRIBUTOR)
State: closed
Created: 2021-03-31T14:22:50Z

NorNE is a manually annotated corpus of named entities which extends the annotation of the existing Norwegian Dependency Treebank. Comprising both of the official standards of written Norwegian (Bokmål and Nynorsk), the corpus contains around 600,000 tokens and annotates a rich set of entity types including persons, organizations, locations, geo-political entities, products, and events, in addition to a class corresponding to nominals derived from names.

See #1720.

Comments:
Awesome!
Title: load_dataset ignoring features
Author: GuillemGSubies (NONE)
State: closed
Created: 2021-03-31T08:30:09Z
Labels: bug

First of all, I'm sorry if it is a repeated issue or the changes are already in master, I searched and I didn't find anything. 

I'm using datasets 1.5.0

![image](https://user-images.githubusercontent.com/37592763/113114369-8f376580-920b-11eb-900d-94365b59f04b.png)

As you can see, when I load the dataset, the ClassLabels are ignored, I have to cast the dataset in order to make it work.

Code to reproduce:

```python
import datasets
data_location = "/data/prueba_multiclase"
features = datasets.Features(
        {"texto": datasets.Value("string"), "label": datasets.features.ClassLabel(names=["false", "true"])}
    )
dataset = datasets.load_dataset(
        "csv", data_files=data_location, delimiter="\t", features=features
    )
```

Dataset I used:


[prueba_multiclase.zip](https://github.com/huggingface/datasets/files/6235022/prueba_multiclase.zip) (it has to be unzipped)


Thank you! ❤️

Comments:
Hi ! Thanks for reporting. I opened a PR to fix this issue: #2201

Nice question which helped me a lot! I have wasted a lot of time to the `DatasetDict` creation from a csv file. Hope the document of this module add some simple examples.

Hi :) We're indeed working on tutorials that we will add to the docs !
Title: Add support for axis in concatenate datasets
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-03-30T16:58:44Z
Labels: enhancement

Add support for `axis` (0 or 1) in `concatenate_datasets`.

Close #853.

Comments:
@lhoestq I am going to implement the consolidation step you mentioned in #1870.

@lhoestq I was thinking that the order of the TableBlocks is not relevant, isn't it?

I mean, in order to consolidate _consecutive_ in-memory table blocks, in this case:
```
blocks = [in_memory_1, memory_mapped, in_memory_2]
```
I could reorder the list:
```
blocks = [in_memory_1, in_memory_2, memory_mapped]
```
so that the first 2 can be consolidated into a single one:
```
blocks = [in_memory_3, memory_mapped]
```

I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

> I think the order is important, users won't expect the dataset to be "shuffled" when they add a new item

OK, therefore I leave `_consolidate_blocks` as it is, which currently keeps the order of the blocks (no shuffling).

Thank you guys for implementing this. Minor thing I noticed in the [documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.concatenate_datasets): it says "Converts a list of Dataset with **the same schema** into a single Dataset". With the addition of the axis parameter, perhaps this should be reworded, no?
Title: Allow pickling of big in-memory tables
Author: lhoestq (MEMBER)
State: closed
Created: 2021-03-30T15:51:56Z

This should fix issue #2134 

Pickling is limited to <4GiB objects, it's not possible to pickle a big arrow table (for multiprocessing for example).
For big tables, we have to write them on disk and only pickle the path to the table.

Comments:
Title: Telugu subset missing for xtreme tatoeba dataset
Author: cosmeowpawlitan (CONTRIBUTOR)
State: closed
Created: 2021-03-30T15:26:34Z

from nlp import load_dataset
train_dataset = load_dataset('xtreme', 'tatoeba.tel')['validation']
ValueError: BuilderConfig tatoeba.tel not found.

but language tel is actually included in xtreme:
https://github.com/google-research/xtreme/blob/master/utils_preprocess.py
def tatoeba_preprocess(args):
  lang3_dict = {
    'afr':'af', 'ara':'ar', 'bul':'bg', 'ben':'bn',
    'deu':'de', 'ell':'el', 'spa':'es', 'est':'et',
    'eus':'eu', 'pes':'fa', 'fin':'fi', 'fra':'fr',
    'heb':'he', 'hin':'hi', 'hun':'hu', 'ind':'id',
    'ita':'it', 'jpn':'ja', 'jav':'jv', 'kat':'ka',
    'kaz':'kk', 'kor':'ko', 'mal':'ml', 'mar':'mr',
    'nld':'nl', 'por':'pt', 'rus':'ru', 'swh':'sw',
    'tam':'ta', **_'tel':'te'_**, 'tha':'th', 'tgl':'tl', <----here
    'tur':'tr', 'urd':'ur', 'vie':'vi', 'cmn':'zh',
    'eng':'en',
  }

Comments:
Good catch ! Thanks for reporting

I just opened #2180 to fix this

Fixed in #2180
Title: Add configurable options to `seqeval` metric
Author: marrodion (CONTRIBUTOR)
State: closed
Created: 2021-03-30T15:04:06Z

Right now `load_metric("seqeval")` only works in the default mode of evaluation (equivalent to conll evaluation).

However, seqeval library [supports](https://github.com/chakki-works/seqeval#support-features) different evaluation schemes (IOB1, IOB2, etc.), which can be plugged in just by supporting additional kwargs in `Seqeval._compute`
https://github.com/huggingface/datasets/blob/85cf7ff920c90ca2e12bedca12b36d2a043c3da2/metrics/seqeval/seqeval.py#L109

Things that would be relevant are, for example, supporting `mode="strict", scheme=IOB2` to count only full entity match as a true positive and omit partial matches.

The only problem I see is that the spirit of `metrics` seems to not require additional imports from user. `seqeval` only supports schemes as objects, without any string aliases. 

It can be solved naively with mapping like `{"IOB2": seqeval.scheme.IOB2}`. Or just left as is and require user to explicitly import scheme from `seqeval` if he wants to configure it past the default implementation.

If that makes sense, I am happy to implement the change.

Comments:
Hi @marrodion. 

Thanks for pointing this out. It would be great to incorporate this metric-specific enhancement.

Another possibility would be to require the user to input the scheme as a string `mode="strict", scheme="IOB2"` and then dynamically import the corresponding module using Python `importlib`:
```python
if scheme:
    scheme = importlib.import_module(f"seqeval.scheme.{scheme}")
```

Feel free to create a Pull Request to make this contribution.
Title: Implement Dataset add_column
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-03-30T14:02:14Z
Labels: enhancement

Implement `Dataset.add_column`.

Close #1954.

Comments:
#2274 has been merged. You can now merge master into this branch and use `assert_arrow_metadata_are_synced_with_dataset_features(dset)` to make sure that the metadata are good :)
Title: Gem V1.1
Author: yjernite (MEMBER)
State: closed
Created: 2021-03-29T23:47:02Z

This branch updates the GEM benchmark to its 1.1 version which includes:
- challenge sets for most tasks
- detokenized TurkCorpus to match the rest of the text simplification subtasks
- fixed inputs for TurkCorpus and ASSET test sets
- 18 languages in WikiLingua

cc @sebastianGehrmann

Comments:
Title: added spans field for the wikiann datasets
Author: rabeehk (CONTRIBUTOR)
State: closed
Created: 2021-03-29T23:38:26Z

Hi @lhoestq 
I tried to add spans to the wikiann datasets.
Thanks a lot for kindly having a look.
This addresses https://github.com/huggingface/datasets/issues/2130. 
Best regards
Rabeeh

Comments:
Hi @lhoestq 
Thanks a lot for taking time checking it. I update "dataset_infos.json", I added description to the function of _generate_samples  in wikiann.py but I was not sure about the format to write in README. thanks. 

Thanks !

For the fields description in the dataset card, something like this does the job:
```
- `tokens`: a `list` of `string` features.
- `langs`: a `list` of `string` features that correspond to the language of each token.
- `ner_tags`: a `list` of classification labels, with possible values including `O` (0), `B-PER` (1), `I-PER` (2), `B-ORG` (3), `I-ORG` (4), `B-LOC` (5), `I-LOC` (6).
- `spans`: a `list` of `string` features, that is the list of named entities in the input text formatted as ``<TAG>: <mention>``
```

Also for information, I think the trailer of rick and morty season 5 is out now :)

Hi @lhoestq 
thank you! This is updated now, please feel free to let me know if I need to modify something :)  thanks
Title: TypeError when using save_to_disk in a dataset loaded with ReadInstruction split
Author: PedroMLF (NONE)
State: closed
Created: 2021-03-29T18:23:54Z

Hi,

Loading a dataset with `load_dataset` using a split defined via `ReadInstruction` and then saving it to disk results in the following error: `TypeError: Object of type ReadInstruction is not JSON serializable`.

Here is the minimal reproducible example:

```python
from datasets import load_dataset
from datasets import ReadInstruction

data_1 = load_dataset(
    "wikiann",
    "en",
    split="validation",
)

data_1.save_to_disk("temporary_path_1")

print("Save with regular split works.")

data_2 = load_dataset(
    "wikiann",
    "en",
    split=ReadInstruction("validation", to=50, unit="%"),
)

data_2.save_to_disk("temporary_path_2")
```

and the corresponding output:

```
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Save with regular split works.
Reusing dataset wikiann (/xxxxx/.cache/huggingface/datasets/wikiann/en/1.1.0/0b11a6fb31eea02f38ca17610657bfba3206100685283014daceb8da291c3be9)
Traceback (most recent call last):
  File "bug.py", line 20, in <module>
    data_2.save_to_disk("temporary_path_2")
  File "/xxxxx/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 645, in save_to_disk
    json.dump(state, state_file, indent=2, sort_keys=True)
  File "/usr/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type ReadInstruction is not JSON serializable
```

Let me know if there is some misuse from my end.

Thanks in advance.

Comments:
Hi !
I think this has been fixed recently on `master`.
Can you try again by installing `datasets` from `master` ?
```
pip install git+https://github.com/huggingface/datasets.git
```

Hi!

Using that version of the code solves the issue. Thanks!
Title: en language data from MLQA dataset is missing
Author: rabeehk (CONTRIBUTOR)
State: closed
Created: 2021-03-29T10:47:50Z

Hi
I need mlqa-translate-train.en dataset, but it is missing from the MLQA dataset. could you have a look please? @lhoestq  thank you for your help to fix this issue.

Comments:
Hi ! Indeed only the languages of the `translate-train` data are included...
I can't find a link to download the english train set on https://github.com/facebookresearch/MLQA though, do you know where we can download it ?

Hi @lhoestq 
thank you very much for coming back to me, now I see, you are right, in the link you sent I see split of  {split}-context-{context_language}-question-{question_language}.json with context_language=question_language=en, TFDS most probably has extracted english ones from these files as en language files, but translate-train/test do not have en indeed. thanks a lot for the great explanations

I close the ticket, since I do not see any en existing, they have trained on "SQuAD V1.1" instead. Thanks.
Title: TydiQA dataset is mixed and is not split per language 
Author: dorost1234 (NONE)
State: open
Created: 2021-03-29T08:56:21Z

Hi @lhoestq 
Currently TydiQA is mixed and user can only access the whole training set of all languages:
https://www.tensorflow.org/datasets/catalog/tydi_qa

for using this dataset, one need to train/evaluate in each separate language, and having them mixed, makes it hard to use this dataset. This is much convenient for user to have  them split and I appreciate your help on this. 

Meanwhile, till hopefully this is split per language, I greatly appreciate telling me how I can preprocess and get data per language. thanks a lot

Comments:
You can filter the languages this way:
```python
tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
```

Otherwise maybe we can have one configuration per language ?
What do you think of this for example ?

```python
load_dataset("tydiqa", "primary_task.en")
```

Hi
thank you very much for the great response, this will be really wonderful
to have one configuration per language, as one need the dataset in majority
of case per language for cross-lingual evaluations.
This becomes also then more close to TFDS format, which is separated per
language https://www.tensorflow.org/datasets/catalog/tydi_qa which will be
really awesome to have.
thanks

On Mon, Mar 29, 2021 at 6:17 PM Quentin Lhoest ***@***.***>
wrote:

> You can filter the languages this way:
>
> tydiqa_en = tydiqa_dataset.filter(lambda x: x["language"] == "english")
>
> Otherwise maybe we can have one configuration per language ?
> What do you think of this for example ?
>
> load_dataset("tydiqa", "primary_task.en")
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/2132#issuecomment-809516799>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AS37NMXPW2PWSQ2RHG73O7TTGCY4LANCNFSM4Z7ER7IA>
> .
>


@lhoestq  I greatly appreciate any updates on this. thanks a lot
Title: When training with Multi-Node Multi-GPU the worker 2 has TypeError: 'NoneType' object
Author: andy-yangz (NONE)
State: closed
Created: 2021-03-29T08:45:58Z
Labels: bug

version: 1.5.0
met a very strange error, I am training large scale language model, and need train on 2 machines(workers).
And sometimes I will get this error `TypeError: 'NoneType' object is not iterable`
This is traceback
```

71 |   | Traceback (most recent call last):
-- | -- | --
72 |   | File "run_gpt.py", line 316, in <module>
73 |   | main()
74 |   | File "run_gpt.py", line 222, in main
75 |   | delimiter="\t", column_names=["input_ids", "attention_mask", "chinese_ref"])
76 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/load.py", line 747, in load_dataset
77 |   | use_auth_token=use_auth_token,
78 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 513, in download_and_prepare
79 |   | self.download_post_processing_resources(dl_manager)
80 |   | File "/data/miniconda3/lib/python3.7/site-packages/datasets/builder.py", line 673, in download_post_processing_resources
81 |   | for split in self.info.splits:
82 |   | TypeError: 'NoneType' object is not iterable
83 |   | WARNING:datasets.builder:Reusing dataset csv (/usr/local/app/.cache/huggingface/datasets/csv/default-1c257ebd48e225e7/0.0.0/2960f95a26e85d40ca41a230ac88787f715ee3003edaacb8b1f0891e9f04dda2)
84 |   | Traceback (most recent call last):
85 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
86 |   | "__main__", mod_spec)
87 |   | File "/data/miniconda3/lib/python3.7/runpy.py", line 85, in _run_code
88 |   | exec(code, run_globals)
89 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 340, in <module>
90 |   | main()
91 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 326, in main
92 |   | sigkill_handler(signal.SIGTERM, None)  # not coming back
93 |   | File "/data/miniconda3/lib/python3.7/site-packages/torch/distributed/launch.py", line 301, in sigkill_handler
94 |   | raise subprocess.CalledProcessError(returncode=last_return_code, cmd=cmd)

```
On worker 1 it loads the dataset well, however on worker 2 will get this error. 
And I will meet this error from time to time, sometimes it just goes well.

Comments:
Hi ! Thanks for reporting
I was able to reproduce this issue. This was caused by missing split infos if a worker reloads the cache of the other worker.

I just opened https://github.com/huggingface/datasets/pull/2137 to fix this issue

The PR got merged :)
Feel free to try it out on the `master` branch

Sorry for the late reply.  
Now everything just works well XD
Title: How to train BERT model with next sentence prediction?
Author: jnishi (NONE)
State: closed
Created: 2021-03-29T06:48:03Z

Hello.

I'm trying to pretrain the BERT model with next sentence prediction. Is there any function that supports next sentence prediction 
like ` TextDatasetForNextSentencePrediction` of `huggingface/transformers` ?

Comments:
Hi !
We're not using `TextDatasetForNextSentencePrediction` in `datasets`.
Although you can probably use the `TextDatasetForNextSentencePrediction.create_examples_from_document` on a dataset to prepare it for next sentence prediction.

Thanks.

Do you mean that `TextDatasetForNextSentencePrediction.create_exapmles_from_document` can be applied to dataset object other than `TextDatasetForNextSentencePrediction`  e.g. a `Dataset` object which is  loaded by  `datasets.load_dataset`?

It would probably require a bit of tweaking, but you can apply it to a dataset, yes.
This should give you a new dataset with sentence pairs you can train a model on.

You can find the documentation about dataset processing here:
https://huggingface.co/docs/datasets/processing.html#processing-data-with-map

Thank you for detail information.

I'll try to apply `create_examples_from_document` to `Dataset` object.
Title: Dialogue action slot name and value are reversed in MultiWoZ 2.2
Author: adamlin120 (CONTRIBUTOR)
State: closed
Created: 2021-03-29T06:34:02Z
Labels: dataset bug

Hi @yjernite, thank you for adding MultiWoZ 2.2 in the huggingface datasets platform. It is beneficial!

I spot an error that the order of Dialogue action slot names and values are reversed.

https://github.com/huggingface/datasets/blob/649b2c469779bc4221e1b6969aa2496d63eb5953/datasets/multi_woz_v22/multi_woz_v22.py#L251-L262

Comments:
Hi
Good catch ! Thanks for reporting

If you are interested in contributing, feel free to open a PR to fix this :)
Title: Is dataset timit_asr broken?
Author: kosuke-kitahara (NONE)
State: closed
Created: 2021-03-28T08:30:18Z

Using `timit_asr` dataset, I saw all records are the same.

``` python
from datasets import load_dataset, load_metric

timit = load_dataset("timit_asr")

from datasets import ClassLabel
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)

    df = pd.DataFrame(dataset[picks])
    display(HTML(df.to_html()))


show_random_elements(timit['train'].remove_columns(["file", "phonetic_detail", "word_detail", "dialect_region", "id", 
                                                    "sentence_type", "speaker_id"]), num_examples=20)

```

`output`

<img width="312" alt="Screen Shot 2021-03-28 at 17 29 04" src="https://user-images.githubusercontent.com/42398050/112746646-21acee80-8feb-11eb-84f3-dbb5d4269724.png">


I double-checked it [here](https://huggingface.co/datasets/viewer/), and met the same problem.

<img width="1374" alt="Screen Shot 2021-03-28 at 17 32 07" src="https://user-images.githubusercontent.com/42398050/112746698-9bdd7300-8feb-11eb-97ed-5babead385f4.png">

Comments:
Hi,

thanks for the report, but this is a duplicate of #2052. 

@mariosasko 
Thank you for your quick response! Following #2052, I've fixed the problem.
Title: Adding ScaNN library to do MIPS?
Author: shamanez (NONE)
State: open
Created: 2021-03-28T00:07:00Z

@lhoestq Hi I am thinking of adding this new google library to do the MIPS similar to **add_faiss_idex**. As the paper suggests, it is really fast when it comes to retrieving the nearest neighbors. 

https://github.com/google-research/google-research/tree/master/scann

![image](https://user-images.githubusercontent.com/16892570/112738294-78ec9800-8fc6-11eb-9a5f-3d7ee5818e76.png)

Comments:
I haven't played with it (yet) but it sounds really cool !
Title: Problem downloading GEM wiki_auto_asset_turk dataset
Author: mille-s (NONE)
State: closed
Created: 2021-03-27T18:41:28Z

@yjernite 

### Summary

I am currently working on the GEM datasets and do not manage to download the wiki_auto_asset_turk data, whereas all other datasets download well with the same code.

### Steps to reproduce
Code snippet:

from datasets import load_dataset
#dataset = load_dataset('gem', 'web_nlg_en')
dataset = load_dataset('gem', 'wiki_auto_asset_turk')

```

**Expected behavior:**

I expect the dataset to start downloading (download bar appears and progresses toward 100%)

**Actual behavior:**
Instead of seeing the download bar appearing, nothing happens; the following appears in the console as expected, but nothing more:

Downloading: 36.6kB [00:00, 37.2MB/s]
Downloading: 41.7kB [00:00, ?B/s]
Downloading and preparing dataset gem/wiki_auto_asset_turk (download: 121.37 MiB, generated: 145.69 MiB, post-processed: Unknown size, total: 267.07 MiB) to C:\Users\sfmil\.cache\huggingface\datasets\gem\wiki_auto_asset_turk\1.0.0\f252756d7f1b8f019aac71a1623b2950acfe10d25d956668ac4eae4e93c58b8d...

### Is this a regression?
No, it was the first time I was trying to download this dataset (same for the other ones).

### Debug info
- Python version: Python 3.8.2
- OS version: Windows 10 Family

Comments:
Hi,

sadly I can't replicate the problem on my Windows machine. Try to update the library to the newest version with:
```bash
pip install git+https://github.com/huggingface/datasets
``` 

Thanks for the answer! I updated the library but unfortunately it didn't solve the problem.

Is there an error message ?
What stacktrace do you get if you interrupt the execution of the program while downloading ?

Sorry for the long time since my last comment, I tried again and don't seem to have the problem anymore, thanks for your support!

Great ! I'm closing the issue then. Feel free to re-open if you experience this issue again
Title: dataset viewer does not work anymore 
Author: dorost1234 (NONE)
State: closed
Created: 2021-03-26T13:22:13Z
Labels: nlp-viewer

Hi
I normally use this link to see all datasets and how I can load them 


https://huggingface.co/datasets/viewer/

Now I am getting 

502 Bad Gateway
nginx/1.18.0 (Ubuntu)

could you bring this webpage back ? this was very helpful @lhoestq 
thanks for your help

Comments:
Thanks for reporting :) We're looking into it

Back up.
Title: copy.deepcopy os.environ instead of copy
Author: NihalHarish (CONTRIBUTOR)
State: closed
Created: 2021-03-26T03:58:38Z

Fixes: https://github.com/huggingface/datasets/issues/2115

- bug fix: using envrion.copy() returns a dict.
- using deepcopy(environ) returns an `_environ` object
- Changing the datatype of the _environ object can break code, if subsequent libraries perform operations using apis exclusive to the environ object, like `environ.getenv()` for example.


Testing:

Tested the change on my terminal:

```
>>> import os
>>> x = deepcopy(os.environ)
>>> y = os.environ
>>> x is y
False
>>> isinstance(x, type(os.environ))
True
>>> z = os.environ.copy()
>>> isinstance(z, type(os.environ))
False
>>> isinstance(z, dict)
True
```

Comments: