t20062443/AT1ds huggingface dataset:
kandorm (NONE) commented on 2020-09-11 11:39:45:
Hi,

when I use the multiprocessing in ```.map```:
```
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                      truncation=True, max_length=args.block_size), batched=True, num_proc=16)
dataset.set_format(type='torch', columns=['input_ids'])
```
I get the following error:
```
Traceback (most recent call last):
  File "src/run.py", line 373, in <module>
    main()
  File "src/run.py", line 295, in main
    get_dataset(data_args, tokenizer=tokenizer, cache_dir=model_args.cache_dir) if training_args.do_train else None
  File "src/run.py", line 153, in get_dataset
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
  File "/root/miniconda3/envs/py3.8/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1287, in map
    transformed_shards = [r.get() for r in results]
  File "/root/miniconda3/envs/py3.8/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1287, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
    put(task)
  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/root/miniconda3/envs/py3.8/lib/python3.8/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'get_dataset.<locals>.<lambda>'
```
I think you should use [pathos](https://github.com/uqfoundation/pathos) to pickle the lambda function and some others!
I change the 30 line of src/datasets/arrow_dataset.py as following:
```
# 30 line: from multiprocessing import Pool, RLock
import pathos
from pathos.multiprocessing import Pool
from multiprocessing import RLock
```
and it works!
lhoestq (MEMBER) commented on 2020-09-11 15:25:03:
It looks like a Pyarrow limitation.
I was able to reproduce the error with 

```python
import pandas as pd
import numpy as np
import pyarrow as pa

 n = 1713614
df = pd.DataFrame.from_dict({"a": list(np.zeros((n, 128))), "b": range(n)})
pa.Table.from_pandas(df)
```

I also tried with 50% of the dataframe and it actually works.
I created an issue on Apache Arrow's JIRA [here](https://issues.apache.org/jira/browse/ARROW-9976)

One way to fix that would be to chunk the dataframe and concatenate arrow tables.
richarddwang (CONTRIBUTOR) commented on 2020-09-12 05:18:18:
It seems that I ran into the same problem
```
def tokenize(cols, example):
  for in_col, out_col in cols.items():
    example[out_col] = hf_tokenizer.convert_tokens_to_ids(hf_tokenizer.tokenize(example[in_col]))
  return example
cola = datasets.load_dataset('glue', 'cola')
tokenized_cola = cola.map(partial(tokenize, {'sentence': 'text_idxs'}),
                             num_proc=2,)
```
and it outpus (exceprts)
```
Concatenating 2 shards from multiprocessing
Set __getitem__(key) output type to python objects for ['idx', 'label', 'sentence', 'text_idxs'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Testing the mapped function outputs
Testing finished, running the mapping function on the dataset
Done writing 532 indices in 4256 bytes .
Done writing 531 indices in 4248 bytes .
Process #0 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00000_of_00002.arrow
Process #1 will write at /home/yisiang/.cache/huggingface/datasets/glue/cola/1.0.0/930e9d141872db65102cabb9fa8ac01c11ffc8a1b72c2e364d8cdda4610df542/tokenized_test_00001_of_00002.arrow
Spawning 2 processes
```
and then the program never stop.
BramVanroy (CONTRIBUTOR) commented on 2020-09-12 13:15:41:
@thomwolf Sure. I'll try downgrading to 3.7 now even though Arrow say they support >=3.5.

Linux (Ubuntu 18.04) - Python 3.8
======================
Package   -   Version
---------------------
certifi               2020.6.20
chardet               3.0.4
click                 7.1.2
datasets              1.0.1
dill                  0.3.2
fasttext              0.9.2
filelock              3.0.12
future                0.18.2
idna                  2.10
joblib                0.16.0
nltk                  3.5
numpy                 1.19.1
packaging             20.4
pandas                1.1.2
pip                   20.0.2
protobuf              3.13.0
pyarrow               1.0.1
pybind11              2.5.0
pyparsing             2.4.7
python-dateutil       2.8.1
pytz                  2020.1
regex                 2020.7.14
requests              2.24.0
sacremoses            0.0.43
scikit-learn          0.23.2
scipy                 1.5.2
sentence-transformers 0.3.6
sentencepiece         0.1.91
setuptools            46.1.3
six                   1.15.0
stanza                1.1.1
threadpoolctl         2.1.0
tokenizers            0.8.1rc2
torch                 1.6.0+cu101
tqdm                  4.48.2
transformers          3.1.0
urllib3               1.25.10
wheel                 0.34.2
xxhash                2.0.0

Windows 10 - Python 3.8
================
Package       -        Version
----------------------------
certifi               2020.6.20
chardet               3.0.4
click                 7.1.2
datasets              1.0.1
dill                  0.3.2
fasttext              0.9.2
filelock              3.0.12
future                0.18.2
idna                  2.10
joblib                0.16.0
nlp                   0.4.0
nltk                  3.5
numpy                 1.19.1
packaging             20.4
pandas                1.1.1
pip                   20.0.2
protobuf              3.13.0
pyarrow               1.0.1
pybind11              2.5.0
pyparsing             2.4.7
python-dateutil       2.8.1
pytz                  2020.1
regex                 2020.7.14
requests              2.24.0
sacremoses            0.0.43
scikit-learn          0.23.2
scipy                 1.5.2
sentence-transformers 0.3.5.1
sentencepiece         0.1.91
setuptools            46.1.3
six                   1.15.0
stanza                1.1.1
threadpoolctl         2.1.0
tokenizers            0.8.1rc1
torch                 1.6.0+cu101
tqdm                  4.48.2
transformers          3.0.2
urllib3               1.25.10
wheel                 0.34.2
xxhash                2.0.0
BramVanroy (CONTRIBUTOR) commented on 2020-09-12 13:26:31:
Downgrading to 3.7 does not help. Here is a dummy text file:

```text
Verzekering weigert vaker te betalen
Bedrijven van verzekeringen erkennen steeds minder arbeidsongevallen .
In 2012 weigerden de bedrijven te betalen voor 21.055 ongevallen op het werk .
Dat is 11,8 % van alle ongevallen op het werk .
Nog nooit weigerden verzekeraars zoveel zaken .
In 2012 hadden 135.118 mensen een ongeval op het werk .
Dat zijn elke werkdag 530 mensen .
Bij die ongevallen stierven 67 mensen .
Bijna 12.000 hebben een handicap na het ongeval .
Geen echt arbeidsongeval Bedrijven moeten een verzekering hebben voor hun werknemers .
```

A temporary work around for the "text" type, is

```python
dataset = Dataset.from_dict({"text": Path(dataset_f).read_text().splitlines()})
```
banunitte (NONE) commented on 2020-09-12 14:32:50:
![image](https://user-images.githubusercontent.com/6847024/92997714-d2add900-f532-11ea-83d4-e3473c2d94d7.png)
![image](https://user-images.githubusercontent.com/6847024/92997724-e22d2200-f532-11ea-951d-b1d8f4582ea3.png)
even i am facing the same issue.
thomwolf (MEMBER) commented on 2020-09-12 18:33:58:
You need to tokenize the string inputs to convert them in integers before you can feed them to a pytorch dataloader.

You can read the quicktour of the datasets or the transformers libraries to know more about that:
- transformers: https://huggingface.co/transformers/quicktour.html
- dataset: https://huggingface.co/docs/datasets/quicktour.html
mojave-pku (NONE) commented on 2020-09-13 03:22:10:
same problem.
`encoded_dataset = core_data.map(lambda examples: tokenizer(examples["query"], examples["document"], padding=True, truncation='longest_first', return_tensors="pt", max_length=384), num_proc=16, keep_in_memory=True)`
it outputs:
```
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787500 indices in 25568400000 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Done writing 1787499 indices in 25568385696 bytes .
Set __getitem__(key) output type to python objects for ['document', 'is_random', 'query'] columns  (when key is int or slice) and don't output other (un-formatted) columns.
Spawning 16 processes
```
ibeltagy (NONE) commented on 2020-09-13 05:10:53:
Updates - the differences between the following three
(1) https://github.com/bheinzerling/pyrouge (previously popular. The one I trust the most)
(2) https://github.com/google-research/google-research/tree/master/rouge
(3) https://github.com/pltrdy/files2rouge (used in fairseq)
can be explained by two things, stemming and handling multiple sentences.

Stemming: 
(1), (2): default is no stemming. (3): default is with stemming ==> No stemming is the correct default as you did [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L84)

Multiple sentences:
(1) `rougeL` splits text using `\n`
(2) `rougeL` ignores `\n`. 
(2) `rougeLsum` splits text using `\n`
(3) `rougeL` splits text using `.`

For (2), `rougeL` and `rougeLsum` are identical if the sequence doesn't contain `\n`. With `\n`, it is `rougeLsum` that matches (1) not `rougeL`. 

Overall, and as far as I understand, for your implementation here https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L65 to match the default, you only need to change `rougeL` [here](https://github.com/huggingface/datasets/blob/master/metrics/rouge/rouge.py#L86) to `rougeLsum` to correctly compute metrics for text with newlines.

Tagging @sshleifer who might be interested.
sipah00 (NONE) commented on 2020-09-13 13:41:03:
Hey @chiyuzhang94, I was also having trouble in loading a large text file (11GB).
But finally got it working. This is what I did after looking into the documentation.

1. split the whole dataset file into smaller files
```bash
mkdir ./shards
split -a 4 -l 256000 -d full_raw_corpus.txt ./shards/shard_
````
2. Pass paths of small data files to `load_dataset`
```python
files = glob.glob('shards/*')
from datasets import load_dataset
dataset = load_dataset('text', data_files=files, split='train')
```
(On passing the whole dataset file (11GB) directly to `load_dataset` was resulting into RAM issue)

3. Tokenization
```python
def encode(examples):
  return tokenizer(examples['text'], truncation=True, padding='max_length')
dataset = dataset.map(encode, batched=True)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
```
 Now you can pass `dataset` to `Trainer` or `pytorch DataLoader`
```python
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4)
next(iter(dataloader))
```
Hope this helps
lhoestq (MEMBER) commented on 2020-09-14 07:55:04:
I wasn't able to reproduce on google colab (python 3.6.9 as well) with 

pickle==4.0
dill=0.3.2
transformers==3.1.0
datasets=1.0.1 (also tried nlp 0.4.0)

If I try

```python
from datasets import load_dataset  # or from nlp
from transformers import BertTokenizer

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=512), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
```
It runs without error
stas00 (CONTRIBUTOR) commented on 2020-09-14 16:28:14:
The problem is that `transformers` imports `datasets` and the latter starts logging on `import`: at least 3 info messages - apache beam/torch/tf available - so it injects noise whether one uses the library or not - i.e. no choice given to the user.

Would you be open for me to changing this PR, to keep the initial level at INFO, but to keep the `DATASETS_VERBOSITY` env var it introduces, to let the user control the verbosity?
lhoestq (MEMBER) commented on 2020-09-14 16:43:09:
> The problem is that `transformers` imports `datasets` and the latter starts logging on `import`: at least 3 info messages - apache beam/torch/tf available - so it injects noise whether one uses the library or not - i.e. no choice given to the user.
> 
> Would you be open for me to changing this PR, to keep the initial level at INFO, but to keep the `DATASETS_VERBOSITY` env var it introduces, to let the user control the verbosity?

For now we can do that, then I'll change some messages to warnings and set the default verbosity at warning as well at that point. Does it sound good to you ?

> If you're referring to tqdm progress reports, it's not affected by changing the logging levels. It's not using logging.

Actually we configured some progress bars to be disabled depending on the logging level ^^'
stas00 (CONTRIBUTOR) commented on 2020-09-14 16:48:50:
> For now we can do that, then I'll change some messages to warnings and set the default verbosity at warning as well at that point. Does it sound good to you ?

If it is logical then by all means. 

> > If you're referring to tqdm progress reports, it's not affected by changing the logging levels. It's not using logging.
> 
> Actually we configured some progress bars to be disabled depending on the logging level ^^'

This is very smart!

I reverted s/WARNINGS/INFO/.

Thank you!
stas00 (CONTRIBUTOR) commented on 2020-09-14 18:11:51:
> Note that it’s the same in `transformers` @stas00, tdqm are also controlled by the logging level there.

That's good to know, @thomwolf - thank you!

I see that it's controlled in `trainer.py`, but in `examples` it's not - since that's where I usually see the progressbars (and they are great!). But I suppose they aren't API, so `examples` can behave differently.
abarbosa94 (CONTRIBUTOR) commented on 2020-09-14 19:38:09:
Hi @lhoestq sorry for the delay in addressing your comments. Thanks for your assistance :)

You were correct as well, as I was using the script without the `datasets/race/dataset_infos.json` file, it did not verify the checksum. I already fix it as well :)

I managed to get everything running smoothly by now. Please let me know if you think that I could improve my solution
chiyuzhang94 (NONE) commented on 2020-09-15 00:51:47:
Thanks, @thomwolf  and @sipah00 ,

I tried to implement your suggestions in my scripts. 
Now, I am facing some connection time-out error. I am using my local file, I have no idea why the module request s3 database.

The log is:
```
Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    raise err
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    timeout=timeout
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

Traceback (most recent call last):
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    chunked=chunked,
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='s3.amazonaws.com', port=443): Max retries exceeded with url: /datasets.huggingface.co/datasets/datasets/text/text.py (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection obj
ect at 0x7fff401e0e48>: Failed to establish a new connection: [Errno 110] Connection timed out',))

Traceback (most recent call last):
  File "/scratch/roberta_emohash/run_language_modeling.py", line 1019, in <module>
    main()
  File "/scratch/roberta_emohash/run_language_modeling.py", line 962, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 177, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/roberta_emohash/run_language_modeling.py", line 117, in HG_Datasets
    dataset = load_dataset('text', data_files=files, cache_dir = args.data_cache_dir, split="train")
  File "/arc/project/evn_py36/datasets/datasets/src/datasets/load.py", line 590, in load_dataset
    self._validate_conn(conn)
  File "/home/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 300, in connect
    conn = self._new_conn()
  File "/home/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fff401e0da0>: Failed to establish a new connection: [Errno 110] Connection timed out

``` 

Do you have any experience on this issue?
stas00 (CONTRIBUTOR) commented on 2020-09-15 04:02:26:
BTW, this is what I'm talking about:
```
python -c "import transformers"
2020-09-14 21:00:58.032658: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
PyTorch version 1.7.0.dev20200910 available.
TensorFlow version 2.3.0 available.
Apache Beam available.
```
why does the user need to see this? Especially, if they aren't even using `datasets` directly?
thomwolf (MEMBER) commented on 2020-09-15 07:08:28:
Yes you are right, we should re-think the logging level of various elements.
I also think that the `set_format` messages are confusing when they are the results of our internal operations (as mentioned [here](https://discuss.huggingface.co/t/pipeline-with-custom-dataset-tokenizer-when-to-save-load-manually/1084/7?u=thomwolf))
thomwolf (MEMBER) commented on 2020-09-15 08:59:50:
To complete what @lhoestq is saying, I think that to use the new version of the `text` processing script (which is on master right now) you need to either specify the version of the script to be the `master` one or to install the lib from source (in which case it uses the `master` version of the script by default):
```python
dataset = load_dataset('text', script_version='master', data_files=XXX)
```
We do versioning by default, i.e. your version of the dataset lib will use the script with the same version by default (i.e. only the `1.0.1` version of the script if you have the PyPI version `1.0.1` of the lib).
ksjae (NONE) commented on 2020-09-15 22:40:09:
Basically ~600MB txt files(UTF-8) * 59. 
contents like ```안녕하세요, 이것은 예제로 한번 말해보는 텍스트입니다. 그냥 이렇다고요.
\n```

Also, it gets stuck for a loooong time at ```Testing the mapped function outputs```, for more than 12 hours(currently ongoing)
ScottishFold007 (NONE) commented on 2020-09-16 06:36:28:
![image](https://user-images.githubusercontent.com/36957508/93300760-fa9a8680-f829-11ea-9105-7a6f67ad8373.png)
win10, py3.6


```
from datasets import Features, Value, ClassLabel, load_dataset


features = Features({'text': Value('string'), 'ctext': Value('string')})
file_dict = {'train': PATH/'summary.csv'}

dataset = load_dataset('csv', data_files=file_dict, script_version='master', delimiter='\t', column_names=['text', 'ctext'], features=features)
```
chiyuzhang94 (NONE) commented on 2020-09-16 23:29:26:
I noticed this is because I use a cloud server where does not provide for connections from our standard compute nodes to outside resources. 

For the `datasets` package, it seems that if the loading script is not already cached in the library it will attempt to connect to an AWS resource to download the dataset loading script. 

I am wondering why the package works in this way. Do you have any suggestions to solve this issue?
moscow25 (NONE) commented on 2020-09-17 03:17:09:
Great list. Any idea if Amazon Reviews has been added?

- ~40 GB of text (sadly no emoji)
- popular MLM pre-training dataset before bigger datasets like WebText https://arxiv.org/abs/1808.01371
- turns out that binarizing the 1-5 star rating leads to great Pos/Neg/Neutral dataset, T5 paper claims to get very high accuracy (98%!) on this with small amount of finetuning https://arxiv.org/abs/2004.14546

Apologies if it's been included (great to see where) and if not, it's one of the better medium/large NLP dataset for semi-supervised learning, albeit a bit out of date. 

Thanks!! 

cc @sshleifer