rIsHu009/github-issues-updated huggingface dataset:
Title: normalized TOCs and titles in data cards
Author: yjernite (MEMBER)
State: closed
Created: 2021-05-12T20:59:59Z

I started fixing some of the READMEs that were failing the tests introduced by @gchhablani but then realized that there were some consistent differences between earlier and newer versions of some of the titles (e.g. Data Splits vs Data Splits Sample Size, Supported Tasks vs Supported Tasks and Leaderboards). We also had different versions of the Table of Content

This PR normalizes all of them to the newer version

Comments:
Oh right! I'd be in favor of still having the same TOC across the board, we can either leave it as is or add a `[More Info Needed]` `Contributions` Section wherever it's currently missing, wdyt?

(I thought those were programmatically updated based on git history :D )

Merging for now to avoid conflict since there are so many changes but let's figure out the contributions section next ;)
Title: Document DatasetInfo attributes
Author: lewtun (MEMBER)
State: closed
Created: 2021-05-12T20:01:29Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
As noted in PR #2255, the attributes of `DatasetInfo` are not documented in the [docs](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=datasetinfo#datasetinfo). It would be nice to do so :)

Comments:
Title: Set to_json default to JSON lines
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-05-12T08:19:25Z

With this PR, the method `Dataset.to_json`:
- is added to the docs
- defaults to JSON lines

Comments:
This is perfect, @albertvillanova - thank you! Tested it to work.

Might it be a good idea to document the args to `to_json`?

and also even a very basic progress bar? took 10min for 8M large records for `openwebtext` so perhaps some indication of it's being alive every min or so?

@lhoestq I added tests for both `lines` and `orient`.
Title: simpllify faiss index save
Author: Guitaricet (CONTRIBUTOR)
State: closed
Created: 2021-05-12T03:54:10Z

Fixes #2350

In some cases, Faiss GPU index objects do not have neither "device" nor "getDevice". Possibly this happens when some part of the index is computed on CPU.

In particular, this would happen with the index `OPQ16_128,IVF512,PQ32` (issue #2350). I did check it, but it is likely that `OPQ` or `PQ` transforms cause it.

I propose, instead of using the index object to get the device, to infer it form the `FaissIndex.device` field as it is done in `.add_vectors`. Here we assume that `.device` always corresponds to the index placement and it seems reasonable.

Comments:
Title: `FaissIndex.save` throws error on GPU
Author: Guitaricet (CONTRIBUTOR)
State: closed
Created: 2021-05-12T03:41:56Z
Labels: bug

## Describe the bug

After training an index with a factory string `OPQ16_128,IVF512,PQ32` on GPU, `.save_faiss_index` throws this error.

```
  File "index_wikipedia.py", line 119, in <module>
    data["train"].save_faiss_index("text_emb", index_save_path)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 470, in save_faiss_index
    index.save(file)
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/datasets/search.py", line 334, in save
    faiss.write_index(index, str(file))
  File "/home/vlialin/miniconda3/envs/cat/lib/python3.8/site-packages/faiss/swigfaiss_avx2.py", line 5654, in write_index
    return _swigfaiss.write_index(*args)
RuntimeError: Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) at /root/miniconda3/conda-bld/faiss-pkg_1613235005464/work/faiss/impl/index_write.cpp:453: don't know how to serialize this type of index
```

## Steps to reproduce the bug

Any dataset will do, I just selected a familiar one.

```python
import numpy as np
import datasets
INDEX_STR = "OPQ16_128,IVF512,PQ32"
INDEX_SAVE_PATH = "will_not_save.faiss"

data = datasets.load_dataset("Fraser/news-category-dataset", split=f"train[:10000]")

def encode(item):
    return {"text_emb": np.random.randn(768).astype(np.float32)}

data = data.map(encode)

data.add_faiss_index(column="text_emb", string_factory=INDEX_STR, train_size=10_000, device=0)
data.save_faiss_index("text_emb", INDEX_SAVE_PATH)
```

## Expected results
Saving the index

## Actual results
Error in void faiss::write_index(const faiss::Index*, faiss::IOWriter*) ... don't know how to serialize this type of index

## Environment info
- `datasets` version: 1.6.2
- Platform: Linux-4.15.0-142-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.8.1+cu111 (True)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


I will be proposing a fix in a couple of minutes

Comments:
Just in case, this is a workaround that I use in my code and it seems to do the job.

```python
if use_gpu_index:
    data["train"]._indexes["text_emb"].faiss_index = faiss.index_gpu_to_cpu(data["train"]._indexes["text_emb"].faiss_index)
```
Title: Add tests for dataset cards
Author: gchhablani (CONTRIBUTOR)
State: closed
Created: 2021-05-11T17:14:27Z

Adding tests for dataset cards

This PR will potentially remove the scripts being used for dataset tags and readme validation.

Additionally, this will allow testing dataset readmes by providing the name as follows:

```bash
pytest tests/test_dataset_cards.py::test_dataset_tags[fashion_mnist]
```
and

```bash
pytest tests/test_dataset_cards.py::test_readme_content[fashion_mnist]
```
or a combined test as:

```bash
pytest tests/test_dataset_cards.py::test_dataset_card[fashion_mnist]
```
@lhoestq

Comments:
@lhoestq

Should I remove the scripts? or atleast remove running them from the CircleCI config?

Also, I hope it is okay that the combined method (metadata+content) is only a slow test, and for the Circle CI, I assume only non-slow tests are run? If yes, this would mean separate tests for content and metadata.

Also feel free to remove the scripts from the CI and also remove the scripts files :)
Title: Add an API to access the language and pretty name of a dataset
Author: sgugger (CONTRIBUTOR)
State: closed
Created: 2021-05-11T14:10:08Z
Labels: enhancement

It would be super nice to have an API to get some metadata of the dataset from the name and args passed to `load_dataset`. This way we could programmatically infer the language and the name of a dataset when creating model cards automatically in the Transformers examples scripts.

Comments:
Hi ! With @bhavitvyamalik we discussed about having something like
```python
from datasets import load_dataset_card

dataset_card = load_dataset_card("squad")
print(dataset_card.metadata.pretty_name)
# Stanford Question Answering Dataset (SQuAD)
print(dataset_card.metadata.languages)
# ["en"]

```
What do you think ?

I don't know if you already have a way to load the model tags in `transformers` but we can agree on the API to have something consistent.

Also note that the pretty name would only be used to show users something prettier than a dataset id, but in the end the source of truth will stay the dataset id (here `squad`).

That works for me!

maybe use the hub-backed dataset_info method? (so there's only one parser of README.md metadata)?

What dataset_info method are you talking about @julien-c ? In `huggingface_hub` I can only see `model_info`.

hmm the equivalent method in `datasets` (which could go into `huggingface_hub` at some point)

Indeed, this info can now be fetched with `huggingface_hub.dataset_info`, so I think we can close this issue.
Title: Add Qasper Dataset
Author: cceyda (CONTRIBUTOR)
State: closed
Created: 2021-05-11T09:25:44Z

[Question Answering on Scientific Research Papers](https://allenai.org/project/qasper/home)

Doing NLP on NLP papers to do NLP ♻️ I had to add it~

- [x] Add README (just gotta fill out some more )
- [x] Dataloader code
- [x] Make dummy dataset
- [x] generate dataset infos
- [x] Tests

Comments:
I saw that the README [template](https://github.com/huggingface/datasets/blob/master/templates/README.md) changed while I was working on this 😅 Some TOC titles may be different but I filled it to the best of my knowledge & readme quality check passes now.
ready for review @lhoestq
Title: [Question] How to move and reuse preprocessed dataset? 
Author: AtmaHou (NONE)
State: closed
Created: 2021-05-11T09:09:17Z

Hi, I am training a gpt-2 from scratch using run_clm.py.

I want to move and reuse the preprocessed dataset (It take 2 hour to preprocess),

I tried to :

copy path_to_cache_dir/datasets to new_cache_dir/datasets
set export HF_DATASETS_CACHE="new_cache_dir/"
but the program still re-preprocess the whole dataset without loading cache.

I also tried to torch.save(lm_datasets, fw), but the saved file is only 14M.

What is the proper way to do this?

Comments:
@lhoestq @LysandreJik


Hi :) Can you share with us the code you used ?


EDIT: from https://github.com/huggingface/transformers/issues/11665#issuecomment-838348291 I understand you're using the run_clm.py script. Can you share your logs ?


Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

> Also note that for the caching to work, you must reuse the exact same parameters as in the first run. Did you change any parameter ? The `preprocessing_num_workers` should also stay the same

I only changed the `preprocessing_num_workers` maybe it is the problem~ I will try again~
Title: Is there a way to join multiple datasets in one?
Author: avacaondata (NONE)
State: open
Created: 2021-05-10T23:16:10Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
I need to join 2 datasets, one that is in the hub and another I've created from my files. Is there an easy way to join these 2? 

**Describe the solution you'd like**
Id like to join them with a merge or join method, just like pandas dataframes. 

**Additional context**
If you want to extend an existing dataset with more data, for example for training a language model, you need that functionality. I've not found it in the documentation.

Comments:
Hi ! We don't have `join`/`merge` on a certain column as in pandas.
Maybe you can just use the [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets) function.


Hi! You can use `datasets_sql` for that now. As of recently, PyArrow also supports querying tables via Substrait, so I think we can start adding these methods to the API soon.
Title: Columns are removed before or after map function applied?
Author: taghizad3h (NONE)
State: open
Created: 2021-05-10T02:36:20Z
Labels: bug

## Describe the bug
According to the documentation when applying map function the [remove_columns ](https://huggingface.co/docs/datasets/processing.html#removing-columns) will be removed after they are passed to the function, but in the [source code](https://huggingface.co/docs/datasets/package_reference/main_classes.html#datasets.Dataset.map) it's documented that they are removed before applying function. I thinks the source code doc is more accurate, right?

Comments:
Hi! Columns are removed **after** applying the function and **before** updating the examples with the function's output (as per the docs [here](https://huggingface.co/docs/datasets/package_reference/main_classes#datasets.Dataset.map.remove_columns)). I agree the docs on this should be more clear.
Title: Added the Ascent KB
Author: phongnt570 (CONTRIBUTOR)
State: closed
Created: 2021-05-09T14:17:39Z

Added the Ascent Commonsense KB of 8.9M assertions.

- Paper: [Advanced Semantics for Commonsense Knowledge Extraction (WWW'21)](https://arxiv.org/abs/2011.00905)
- Website: https://ascent.mpi-inf.mpg.de/

(I am the author of the dataset)

Comments:
Thanks for approving it!
Title: NonMatchingChecksumError for web_of_science dataset
Author: nbroad1881 (NONE)
State: closed
Created: 2021-05-09T02:02:02Z
Labels: bug

NonMatchingChecksumError when trying to download the web_of_science dataset. 

>NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://data.mendeley.com/datasets/9rw3vkcfy4/6/files/c9ea673d-5542-44c0-ab7b-f1311f7d61df/WebOfScience.zip?dl=1']

Setting `ignore_verfications=True` results in OSError.

>OSError: Cannot find data file. 
Original error:
[Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/37ab2c42f50d553c1d0ea432baca3e9e11fedea4aeec63a81e6b7e25dd10d4e7/WOS5736/X.txt'

```python
dataset = load_dataset('web_of_science', 'WOS5736')
```
There are 3 data instances and they all don't work. 'WOS5736', 'WOS11967', 'WOS46985'

datasets 1.6.2
python 3.7.10
Ubuntu 18.04.5 LTS

Comments:
I've raised a PR for this. Should work with `dataset = load_dataset("web_of_science", "WOS11967", ignore_verifications=True)`once it gets merged into the main branch. Thanks for reporting this!
Title: Fix overflow issue in interpolation search
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-05-08T20:51:36Z

Fixes #2335 

More info about this error can be found [here](https://stackoverflow.com/questions/53239890/why-do-i-keep-getting-this-error-runtimewarning-overflow-encountered-in-int-sc/53240100).

Comments:
~~Seems like the CI failure is unrelated to this PR~~ (fixed with the merge). 

@lhoestq Can you please verify that everything is OK in terms of speed? Another solution is to change the offsets array dtype to np.int64 (but this doesn't scale in theory compared to Python integer which is unbound). I'm not sure why on my 64-bit machine the default numpy dtype is np.int32 tho.

Hi ! Thanks for the fix.
Unfortunately in terms of speed this is not acceptable :/
The `get_batch_of_1024_random_rows` metric or the `benchmark_getitem_100B ` benchmark is almost at 1sec instead of a few milliseconds.

Would it be possible to avoid the overflow by simply passing `dtype=np.int64` to `np.cumsum` ?
On windows machines the default is int32 unfortunately so we have to force the dtype to be int64



Yes, casting the array to np.int64 should work as well. Another option would be to cast the array elements (`arr[i], arr[j]`) in interpolation search to Python integers (bound only with memory) before multiplication (the error stems from this part: `(j - i) * (x - arr[i])`) when working with big values. But for now, the first option is OK for the sake of simplicity.
Title: Index error in Dataset.map
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-05-08T20:44:57Z
Labels: bug

The following code, if executed on master, raises an IndexError (due to overflow):
```python
>>> from datasets import *
>>> d = load_dataset("bookcorpus", split="train")
Reusing dataset bookcorpus (C:\Users\Mario\.cache\huggingface\datasets\bookcorpus\plain_text\1.0.0\44662c4a114441c35200992bea923b170e6f13f2f0beb7c14e43759cec498700)
2021-05-08 21:23:46.859818: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library cudart64_101.dll
>>> d.map(lambda ex: ex)
  0%|▎                                                                    | 289430/74004228 [00:13<58:41, 20935.33ex/s]c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py:84: RuntimeWarning: overflow encountered in int_scalars
  k = i + ((j - i) * (x - arr[i]) // (arr[j] - arr[i]))
  0%|▎                                                                    | 290162/74004228 [00:13<59:11, 20757.23ex/s]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1498, in map
    new_fingerprint=new_fingerprint,
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 174, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\fingerprint.py", line 340, in wrapper
    out = func(self, *args, **kwargs)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1799, in _map_single
    for i, example in enumerate(pbar):
  File "C:\Users\Mario\Anaconda3\envs\hf-datasets\lib\site-packages\tqdm\std.py", line 1133, in __iter__
    for obj in iterable:
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1145, in __iter__
    format_kwargs=format_kwargs,
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\arrow_dataset.py", line 1337, in _getitem
    pa_subtable = query_table(self._data, key, indices=self._indices if self._indices is not None else None)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\formatting\formatting.py", line 368, in query_table
    pa_subtable = _query_table(table, key)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\formatting\formatting.py", line 79, in _query_table
    return table.fast_slice(key % table.num_rows, 1)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py", line 128, in fast_slice
    i = _interpolation_search(self._offsets, offset)
  File "c:\users\mario\desktop\projects\datasets-1\src\datasets\table.py", line 91, in _interpolation_search
    raise IndexError(f"Invalid query '{x}' for size {arr[-1] if len(arr) else 'none'}.")
IndexError: Invalid query '290162' for size 74004228.
```
Tested on Windows, can run on Linux if needed.

EDIT:
It seems like for this to happen, the default NumPy dtype has to be np.int32.

Comments:
Title: Updating the DART file checksums in GEM
Author: yjernite (MEMBER)
State: closed
Created: 2021-05-07T21:53:44Z

The DART files were just updated on the source GitHub

https://github.com/Yale-LILY/dart/commit/34b3c872da4811523e334f1631e54ca8105dffab

Comments:
@sebastianGehrmann
Title: Add Topical-Chat
Author: ktangri (NONE)
State: open
Created: 2021-05-07T13:43:59Z
Labels: dataset request

## Adding a Dataset
- **Name:** Topical-Chat
- **Description:** a knowledge-grounded human-human conversation dataset where the underlying knowledge spans 8 broad topics and conversation partners don’t have explicitly defined roles
- **Paper:** https://www.isca-speech.org/archive/Interspeech_2019/pdfs/3079.pdf
- **Data:** https://github.com/alexa/Topical-Chat
- **Motivation:** Good quality, knowledge-grounded dataset that spans a broad range of topics

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Allow passing `desc` to `tqdm` in `Dataset.map()`
Author: changjonathanc (CONTRIBUTOR)
State: closed
Created: 2021-05-07T05:52:54Z
Labels: enhancement

good first issue

It's normal to have many `map()` calls, and some of them can take a few minutes,
it would be nice to have a description on the progress bar.

Alternative solution:
Print the description before/after the `map()` call.

Comments:
Hi @lhoestq,
Should we change `desc` in [pbar](https://github.com/huggingface/datasets/blob/81fcf88172ed5e3026ef68aed4c0ec6980372333/src/datasets/arrow_dataset.py#L1860) to something meaningful?

I think the user could pass the `desc` parameter to `map` so that it can be displayed in the tqdm progress bar, as suggested by @cccntu.

When there's no multiprocessing, the `desc` of the progress bar could be the `desc` passed by the user.
In multiprocessing, we were already using a `desc` equal to `"#" + str(rank)`.
We can change it to be `(desc or "") + "#" + str(rank)` instead.

In the end, since both `desc` and `rank` could be None, we can have:
```python
pbar_desc = (desc or "") + "#" + str(rank) if rank is not None else desc
```

Finally let's remember that if we add `desc` as a new parameter to `map`, we should add it to the `ignore_kwargs` list of the `@fingerprint_transform` decorator of `Dataset._map_single` since we don't want this parameter to affect the fingerprint of the resulting dataset.
Title: Add cache dir for in-memory datasets
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-05-06T19:35:32Z

Adds the cache dir attribute to DatasetInfo as suggested by @lhoestq.

Should fix #2322

Comments:
Yes, having `cache_dir` as an attribute looks cleaner.





Good job! Looking forward to this new feature! 🥂

@lhoestq Sorry for the late reply. Yes, I'll start working on tests. Thanks for the detailed explanation of the current issues with caching (like the idea of adding the `use_caching` parameter to `load_dataset`) 

@lhoestq Sure. I'm aware this is a high-priority issue to some extent, so feel free to take over.

Few suggestions I have:
* there is a slight difference between setting `use_caching` to `False` in `load_dataset` and disabling caching globally with `set_caching_enabled(False)` because the former will never execute the following code (`self._cache_dir` is always `False`): 
https://github.com/huggingface/datasets/blob/c231abdb174987419bbde3360b5b9d6a4672c736/src/datasets/arrow_dataset.py#L1807-L1824
, so I'm just checking whether this is intended (if yes, maybe the docs should mention this) or not?
* think we should add the `use_caching` parameter to every method that has the `keep_in_memory` (and `in_memory` 😃) parameter in its signature for better consistency, but I say let's address this in a separate PR. IMO we need one more PR that will deal exclusively with consistency in the caching logic.

Hi @mariosasko 
We discussed internally and we think that this feature might not be the direction we're doing to take for these reasons:

- it goes against our simple definition of caching: on-disk == uses file cache, and in-memory == nothing is written to disk. I think it adds too much complexity just for a minimal flexibility addition
- there are a few edge cases which are really confusing:
    - map on an in memory dataset with a cache_file_name specified by the user -> should the result be in memory or from disk ?
    - it would require a special cache directory just for in memory datasets, since they don’t have a preferred directory for caching
- it would break a lot of stuff and would require to rewrite significant parts of the core code and the tests


So in the end we're probably going to close this PR.
Let me know what you think, and thank you anyway for your help on this !

Hi,

I'm fine with that. I agree this adds too much complexity. Btw, I like the idea of reverting default in-memory for small datasets that led to this PR.

Superseded by #2460 (to close issue #2458).
Title: A syntax error in example
Author: mymusise (NONE)
State: closed
Created: 2021-05-06T14:34:44Z
Labels: bug

![image](https://user-images.githubusercontent.com/6883957/117315905-b47a5c00-aeba-11eb-91eb-b2a4a0212a56.png)

Sorry to report with an image, I can't find the template source code of this snippet.

Comments:
cc @beurkinger but I think this has been fixed internally and will soon be updated right ?

This issue has been fixed.
Title: Enable auto-download for PAN-X / Wikiann domain in XTREME
Author: lewtun (MEMBER)
State: closed
Created: 2021-05-05T20:58:38Z

This PR replaces the manual download of the `PAN-X.lang` domains with an auto-download from a Dropbox link provided by the Wikiann author. We also add the relevant dummy data for these domains.

While re-generating `dataset_infos.json` I ran into a `KeyError` in the `udpos.Arabic` domain so have included a fix for this as well.

Comments:
Title: Added the HLGD dataset
Author: tingofurro (CONTRIBUTOR)
State: closed
Created: 2021-05-05T16:53:29Z

Added the Headline Grouping Dataset (HLGD), from the NAACL2021 paper: News Headline Grouping as a Challenging NLU Task
Dataset Link: https://github.com/tingofurro/headline_grouping
Paper link: https://people.eecs.berkeley.edu/~phillab/pdfs/NAACL2021_HLG.pdf

Comments:
Is there anything else needed from my end?

Thanks Bhavitvya and Quentin, this was very streamlined!
Title: load_dataset("timit_asr") gives back duplicates of just one sample text
Author: ekeleshian (NONE)
State: closed
Created: 2021-05-05T13:14:48Z
Labels: bug

## Describe the bug
When you look up on key ["train"] and then ['text'], you get back a list  with just one sentence duplicated 4620 times. Namely, the sentence "Would such an act of refusal be useful?". Similarly when you look up ['test'] and then ['text'], the list is one sentence repeated "The bungalow was pleasantly situated near the shore." 1680 times. 

I tried to work around the issue by downgrading to datasets version 1.3.0, inspired by [this post](https://www.gitmemory.com/issue/huggingface/datasets/2052/798904836) and removing the entire huggingface directory from ~/.cache, but I still get the same issue.  

## Steps to reproduce the bug
```python
from datasets import load_dataset
timit = load_dataset("timit_asr")
print(timit['train']['text'])
print(timit['test']['text'])
```

## Expected Result
Rows of diverse text, like how it is shown in the [wav2vec2.0 tutorial](https://colab.research.google.com/github/patrickvonplaten/notebooks/blob/master/Fine_tuning_Wav2Vec2_for_English_ASR.ipynb)
<img width="485" alt="Screen Shot 2021-05-05 at 9 09 57 AM" src="https://user-images.githubusercontent.com/33647474/117146094-d9b77f00-ad81-11eb-8306-f281850c127a.png">


## Actual results
Rows of repeated text.
<img width="319" alt="Screen Shot 2021-05-05 at 9 11 53 AM" src="https://user-images.githubusercontent.com/33647474/117146231-f8b61100-ad81-11eb-834a-fc10410b0c9c.png">


## Versions
- Datasets: 1.3.0
- Python: 3.9.1
- Platform: macOS-11.2.1-x86_64-i386-64bit}

Comments:
Upgrading datasets to version 1.6 fixes the issue

This bug was fixed in #1995. Upgrading the `datasets` should work! 

Thanks @ekeleshian for having reported.

I am closing this issue once that you updated `datasets`. Feel free to reopen it if the problem persists.
Title: [api request] API to obtain "dataset_module" dynamic path?
Author: richardliaw (NONE)
State: closed
Created: 2021-05-05T08:40:48Z
Labels: enhancement

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is.

This is an awesome library. 

It seems like the dynamic module path in this library has broken some of hyperparameter tuning functionality: https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/34

This is because Ray will spawn new processes, and each process will load modules by path. However, we need to explicitly inform Ray to load the right modules, or else it will error upon import. 

I'd like an API to obtain the dynamic paths. This will allow us to support this functionality in this awesome library while being future proof.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

`datasets.get_dynamic_paths -> List[str]` will be sufficient for my use case.

By offering this API, we will be able to address the following issues (by patching the ray integration sufficiently):

https://github.com/huggingface/blog/issues/106
https://github.com/huggingface/transformers/issues/11565
https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/34
https://discuss.huggingface.co/t/using-hyperparameter-search-in-trainer/785/35

Comments:
Hi @richardliaw, 

First, thanks for the compliments.

In relation with your request, currently, the dynamic modules path is obtained this way:
```python
from datasets.load import init_dynamic_modules, MODULE_NAME_FOR_DYNAMIC_MODULES

dynamic_modules_path = init_dynamic_modules(MODULE_NAME_FOR_DYNAMIC_MODULES)
```

Let me know if it is OK for you this way. 

I could set `MODULE_NAME_FOR_DYNAMIC_MODULES` as default value, so that you could instead obtain the path with:
```
dynamic_modules_path = datasets.load.init_dynamic_modules()
```

Hi @albertvillanova, the default value proposal seems great :) Looking forward to this!

I like the idea as well ! thanks @albertvillanova 

Hi @richardliaw, the feature is on the master branch and will be included in the next release in a couple of weeks.

awesome work @albertvillanova !
Title: Incorrect version specification for pyarrow
Author: cemilcengiz (CONTRIBUTOR)
State: closed
Created: 2021-05-04T19:15:11Z
Labels: bug

## Describe the bug
The pyarrow dependency is incorrectly specified in setup.py file, in [this line](https://github.com/huggingface/datasets/blob/3a3e5a4da20bfcd75f8b6a6869b240af8feccc12/setup.py#L77).
Also as a snippet:
```python
 "pyarrow>=1.0.0<4.0.0",
```

## Steps to reproduce the bug
```bash
 pip install "pyarrow>=1.0.0<4.0.0"
```

## Expected results
It is expected to get a pyarrow version between 1.0.0 (inclusive) and 4.0.0 (exclusive).

## Actual results
pip ignores the specified versions since there is a missing comma between the lower and upper limits. Therefore, pip installs the latest pyarrow version from PYPI, which is 4.0.0.
This is especially problematic since "conda env export" fails due to incorrect version specification. Here is the conda error as well:
```bash
conda env export
InvalidVersionSpec: Invalid version '1.0.0<4.0.0': invalid character(s)
```


## Fix suggestion
Put a comma between the version limits which means replacing the line in setup.py file with the following:
```python
 "pyarrow>=1.0.0,<4.0.0",
```

## Versions
Paste the output of the following code:
```python
- Datasets: 1.6.2
- Python: 3.7.10 (default, Feb 26 2021, 18:47:35) 
[GCC 7.3.0]
- Platform: Linux-5.4.0-42-generic-x86_64-with-debian-buster-sid
```

Comments:
Fixed by #2317.
Title: Datasets cli improvements
Author: mariosasko (COLLABORATOR)
State: closed
Created: 2021-05-04T18:55:11Z

This PR:
* replaces the code from the `bug_report.md` that was used to get relevant system info with a dedicated command (a more elegant approach than copy-pasting the code IMO)
* removes the `download` command (copied from the transformers repo?)
* adds missing help messages to the cli commands

Comments:
Additionally, I've deleted the points that are not very relevant for this repo (I guess the deleted points originate from the transformers repo). With this change, running `datasets-cli` is identical to copy-pasting the code from `bug_report.md`, but is more elegant because it doesn't require launching the REPL and copy-pasting the code.
Title: Minor refactor prepare_module
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-05-04T18:37:26Z

Start to refactor `prepare_module` to try to decouple functionality.

This PR does:
- extract function `_initialize_dynamic_modules_namespace_package`
- extract function `_find_module_in_github_or_s3`
- some renaming of variables
- use of f-strings

Comments:
@lhoestq this is the PR that I mentioned to you, which can be considered as a first step in refactoring `prepare_module`.

closing in favor of #2986
Title: Add SLR52, SLR53 and SLR54 to OpenSLR
Author: cahya-wirawan (CONTRIBUTOR)
State: closed
Created: 2021-05-04T09:08:03Z

Add large speech datasets for Sinhala, Bengali and Nepali.

Comments:
Hi @lhoestq , I am not sure about the error message:
```
#!/bin/bash -eo pipefail
./scripts/datasets_metadata_validator.py
WARNING:root:❌ Failed to validate 'datasets/openslr/README.md':
__init__() got an unexpected keyword argument 'SLR32'
INFO:root:❌ Failed on 1 files.

Exited with code exit status 1
CircleCI received exit code 1 
```
Could you have a look please? Thanks.

Hi ! The error is unrelated to your PR and has been fixed on master
Next time feel free to merge master into your branch to fix the CI error ;)
Title: Update README.md
Author: cryoff (CONTRIBUTOR)
State: closed
Created: 2021-05-04T04:38:01Z

Provides description of data instances and dataset features

Comments:
Hi @cryoff, thanks for completing the dataset card.

Now there is an automatic validation tool to assure that all dataset cards contain all the relevant information. This is the cause of the non-passing test on your Pull Request:
```
Found fields that are not non-empty list of strings: {'annotations_creators': [], 'language_creators': []}
```
Title: Fix conda release
Author: lhoestq (MEMBER)
State: closed
Created: 2021-05-03T14:52:59Z

There were a few issues with conda releases (they've been failing for a while now).
To fix this I had to:
- add the --single-version-externally-managed tag to the build stage (suggestion from [here](https://stackoverflow.com/a/64825075))
- set the python version of the conda build stage to 3.8 since 3.9 isn't supported
- sync the evrsion requirement of `huggingface_hub`

With these changes I'm working on uploading all missing versions until 1.6.2 to conda

EDIT: I managed to build and upload all missing versions until 1.6.2 to conda :)

Comments:
Title: Add SubjQA dataset
Author: lewtun (MEMBER)
State: closed
Created: 2021-05-02T14:51:20Z

Hello datasetters 🙂!

Here's an interesting dataset about extractive question-answering on _subjective_ product / restaurant reviews. It's quite challenging for models fine-tuned on SQuAD and provides a nice example of domain adaptation (i.e. fine-tuning a SQuAD model on this domain gives better performance).

I found a bug in the start/end indices that I've proposed a fix for here: https://github.com/megagonlabs/SubjQA/pull/2

Unfortunately, the dataset creators are unresponsive, so for now I am using my fork as the source. Will update the URL if/when the creators respond.

Comments:
I'm not sure why the windows test fails, but looking at the logs it looks like some caching issue on one of the metrics ... maybe re-run and 🤞 ?

Hi @lewtun, thanks for adding this dataset!

If the dataset is going to be referenced heavily, I think it's worth spending some time to make the dataset card really great :) To start, the information that is currently in the `Data collection` paragraph should probably be organized in the `Dataset Creation` section.

Here's a link to the [relevant section of the guide](https://github.com/huggingface/datasets/blob/master/templates/README_guide.md#dataset-creation), let me know if you have any questions!

> If the dataset is going to be referenced heavily, I think it's worth spending some time to make the dataset card really great :) To start, the information that is currently in the `Data collection` paragraph should probably be organized in the `Dataset Creation` section.

great idea @yjernite! i've added some extra information / moved things as you suggest and will wrap up the rest tomorrow :)

hi @yjernite and @lhoestq, i've fleshed out the dataset card and think this is now ready for another round of review!
Title: Add VoxPopuli
Author: patrickvonplaten (CONTRIBUTOR)
State: closed
Created: 2021-05-02T12:17:40Z
Labels: dataset request

speech

## Adding a Dataset
- **Name:** Voxpopuli
- **Description:** VoxPopuli is raw data is collected from 2009-2020 European Parliament event recordings
- **Paper:** https://arxiv.org/abs/2101.00390
- **Data:** https://github.com/facebookresearch/voxpopuli
- **Motivation:** biggest unlabeled speech dataset

**Note**: Since the dataset is so huge, we should only add the config `10k` in the beginning.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
I'm happy to take this on:) One question: The original unlabelled data is stored unsegmented (see e.g. https://github.com/facebookresearch/voxpopuli/blob/main/voxpopuli/get_unlabelled_data.py#L30), but segmenting the audio in the dataset would require a dependency on something like soundfile or torchaudio. An alternative could be to provide the segments start and end times as a Sequence and then it's up to the user to perform the segmentation on-the-fly if they wish?

Hey @jfainberg,

This sounds great! I think adding a dependency would not be a big problem, however automatically segmenting the data probably means that it would take a very long time to do:

```python
dataset = load_dataset("voxpopuli", "french")
```

=> so as a start I think your option 2 is the way to go!

@polinaeterna VoxPopuli is available [here](https://huggingface.co/datasets/facebook/voxpopuli), so we can close this issue, right?


@mariosasko yes, sure, closing it
Title: My iPhone
Author: Jasonbuchanan1983 (NONE)
State: closed
Created: 2021-05-02T11:11:11Z

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: 1
Author: zinnyi (NONE)
State: closed
Created: 2021-04-30T17:53:49Z
Labels: dataset request

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).

Comments:
Title: Create ExtractManager
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-30T17:13:34Z
Labels: refactoring

Perform refactoring to decouple extract functionality.

Comments:
Hi @lhoestq,

Once that #2578 has been merged, I would like to ask you to have a look at this PR: it implements the same logic as the one in #2578 but for all the other file compression formats.

Thanks.

I think all is done @lhoestq ;)
Title: Slow #0 when using map to tokenize.
Author: VerdureChen (NONE)
State: open
Created: 2021-04-30T08:00:33Z

Hi, _datasets_ is really amazing! I am following [run_mlm_no_trainer.py](url) to pre-train BERT, and it uses `tokenized_datasets = raw_datasets.map(
            tokenize_function,
            batched=True,
            num_proc=args.preprocessing_num_workers,
            remove_columns=column_names,
            load_from_cache_file=not args.overwrite_cache,
        )` to tokenize by multiprocessing. However, I have found that when `num_proc`>1，the process _#0_ is much slower than others.
It looks like this:
![image](https://user-images.githubusercontent.com/31714566/116665555-81246280-a9cc-11eb-8a37-6e608ab310d0.png)
It takes more than 12 hours for #0, while others just about half an hour. Could anyone tell me it is normal or not, and is there any methods to speed up it?

Comments:
Hi ! Have you tried other values for `preprocessing_num_workers` ? Is it always process 0 that is slower ?
There are no difference between process 0 and the others except that it processes the first shard of the dataset.

Hi, I have found the reason of it. Before using the map function to tokenize the data, I concatenate the wikipedia and bookcorpus first, like this:
```if args.dataset_name1 is not None:
        dataset1 = load_dataset(args.dataset_name1, args.dataset_config_name1, split="train")
        dataset1 = dataset1.remove_columns('title')
        if args.dataset_name2 is not None:
            dataset2 = load_dataset(args.dataset_name2, args.dataset_config_name2,split="train")
            assert dataset1.features.type == dataset2.features.type, str(dataset1.features.type)+';'+str(dataset2.features.type)
           datasets12 = concatenate_datasets([dataset1, dataset2], split='train')
```
When I just use one datasets, e.g. wikipedia, the problem seems no longer exist:
![image](https://user-images.githubusercontent.com/31714566/116967059-13d24380-ace4-11eb-8d14-b7b9c9a275cc.png)

Bookcorpus has more row numbers than Wikipedia, however, it takes much more time to process each batch of wiki than that of bookcorpus. When we first concatenate two datasets and then use _map_ to process the concatenated datasets,  e.g. `num_proc=5`, process 0 has to process all of the wikipedia data, causing the problem that #0 takes a longer time to finish the job. 

The problem is caused by the different characteristic of different datasets. One solution might be using _map_ first to process two datasets seperately, then concatenate the tokenized and processed datasets before input to the `Dataloader`.



That makes sense ! You can indeed use `map` on both datasets separately and then concatenate.
Another option is to concatenate, then shuffle, and then `map`.
Title: Don't copy recordbatches in memory during a table deepcopy
Author: lhoestq (MEMBER)
State: closed
Created: 2021-04-29T16:26:05Z

Fix issue #2276 and hopefully #2134

The recordbatches of the `IndexedTableMixin` used to speed up queries to the table were copied in memory during a table deepcopy.
This resulted in `concatenate_datasets`, `load_from_disk` and other methods to always bring the data in memory.

I fixed the copy similarly to #2287 and updated the test to make sure it doesn't happen again (added a test for deepcopy + make sure that the immutable arrow objects are passed to the copied table without being copied).

The issue was not caught by our tests because the total allocated bytes value in PyArrow isn't updated when deepcopying recordbatches: the copy in memory wasn't detected. This behavior looks like a bug in PyArrow, I'll open a ticket on JIRA.

Thanks @samsontmr , @TaskManager91 and @mariosasko for the help

Comments:
Title: Bbaw egyptian
Author: phiwi (CONTRIBUTOR)
State: closed
Created: 2021-04-29T15:27:58Z

This is the "hieroglyph corpus" that I could unfortunately not contribute during the marathon. I re-extracted it again now, so that it is in the state as used in my paper (seee documentation). I hope it satiesfies your requirements and wish every scientist out their loads of fun deciphering a 5.000 years old language :-)

Comments:
Hi @phiwi,

Thanks for contributing this nice dataset. If you have any blocking problem or question, do not hesitate to ask here. We are pleased to help you.

Could you please first synchronize with our master branch? From your branch `bbaw_egyptian`, type:
```
git fetch upstream master
git merge upstream/master
```

Thanks ! Can you check that you have `black==21.4b0` and run `make style` again ? This should fix the "check_code_quality" CI issue

Reformatted with black.

Hi @phiwi, there are still some minor problems in relation with the tags you used in the dataset card (README.md).

Here you can find the output of the metadata validator:
```
WARNING:root:❌ Failed to validate 'datasets/bbaw_egyptian/README.md':
Could not validate the metada, found the following errors:
* field 'size_categories':
	['100K<n<1000K'] are not registered tags for 'size_categories', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/size_categories.json
* field 'task_ids':
	['machine translation'] are not registered tags for 'task_ids', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/tasks.json
* field 'languages':
	['eg'] are not registered tags for 'languages', reference at https://github.com/huggingface/datasets/tree/master/src/datasets/utils/resources/languages.json

``` 

@albertvillanova corrected :-)

Thanks, @phiwi. Now all tests should pass green.

However, I think there is still an issue with the language code:
- the code for the Ancient Egyptian is not `ar-EG`
- there is no ISO 639-1 code for the Ancient Egyptian
- there is an ISO 639-2 code: `egy`; but this code will not pass the validation test because it is not in the list of valid codes

I am not sure what to do in this case... Maybe @lhoestq has an idea? Maybe adding the code to the list? https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/languages.json

I have just checked that in the [list of valid codes](https://github.com/huggingface/datasets/blob/master/src/datasets/utils/resources/languages.json) there are already ISO 639-2 codes. Therefore, I would suggest you to add it to the list:
```
"egy": "Egyptian (Ancient)",
```
and change it in the dataset card.

Done.

Hope, everything is okay right now.
Title: Allow collaborators to self-assign issues
Author: albertvillanova (MEMBER)
State: closed
Created: 2021-04-29T15:07:06Z

Allow collaborators (without write access to the repository) to self-assign issues.

In order to self-assign an issue, they have to comment it with the word: `#take` or `#self-assign`.

Comments:
What do you think, @lhoestq? 😉 

I think this could be another step to facilitate community contributions.

@lhoestq, it doesn't exist in `transformers`... I picked the idea from `scikit-learn`, where I have previously collaborated...

And sure, this must be documented! I just wanted first to know your opinion...
Title: Load_dataset for local CSV files
Author: sstojanoska (NONE)
State: closed
Created: 2021-04-29T15:01:10Z
Labels: bug

The method load_dataset fails to correctly load a dataset from csv. 

Moreover, I am working on a token-classification task ( POS tagging) , where each row in my CSV contains two columns each of them having a list of strings.
row example:
```tokens  |  labels
['I' , 'am', 'John']  |  ['PRON', 'AUX', 'PROPN' ] 
```
The method, loads each list as a string:  (i.g "['I' , 'am', 'John']").
To solve this issue, I copied the Datasets.Features, created Sequence types ( instead of Value)  and tried to cast the features type
```
new_features['tokens'] = Sequence(feature=Value(dtype='string', id=None))
new_features['labels'] = Sequence(feature=ClassLabel(num_classes=len(tag2idx), names=list(unique_tags)))
dataset = dataset.cast(new_features)
```
but I got the following error 
```
ArrowNotImplementedError: Unsupported cast from string to list using function cast_list
```
Moreover, I tried to set feature parameter in load_dataset method, to my new_features, but this fails as well.
How can this be solved ?

Comments:
Hi,

this is not a standard CSV file (requires additional preprocessing) so I wouldn't label this as s bug. You could parse the examples with the regex module or the string API to extract the data, but the following approach is probably the easiest (once you load the data):
```python
import ast
# load the dataset and copy the features
def process(ex):
    return {"tokens": ast.literal_eval(ex["tokens"]), "labels": ast.literal_eval(ex["labels"])}
dataset = dataset.map(process, features=new_features)
```


Hi,

Thanks for the reply.
I  have already used  ```ast.literal_eval``` to evaluate the string into list, but I was getting another error:
```
ArrowInvalid: Could not convert X with type str: tried to convert to int
```
Why this happens ? Should labels be mapped to their ids and use int instead of str ?

Yes, just map the labels to their ids.
Title: Fixed typo  seperate->separate
Author: laksh9950 (CONTRIBUTOR)
State: closed
Created: 2021-04-29T08:55:46Z


Comments:
Hi ! Thanks for the fix :)
The CI fail isn't related to your PR. I opened a PR #2286 to fix the CI.
We'll wait for #2286 to be merged to master first if you don't mind

The PR has been merged ! Feel free to merge master into your branch to fix the CI
Title: Loss result inGptNeoForCasual
Author: Yossillamm (NONE)
State: closed
Created: 2021-04-28T15:39:52Z
Labels: enhancement

Is there any way you give the " loss" and "logits" results in the gpt neo api?

Comments:
Hi ! I think you might have to ask on the `transformers` repo on or the forum at https://discuss.huggingface.co/

Closing since it's not related to this library