t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-09-22 10:17:21:
Thanks for reporting !
There indeed a mismatch between the features and the kwargs description

I believe `answer_start` was added to match the squad dataset format for consistency, even though it is not used in the metric computation. I think I'd rather keep it this way, so that you can just give `references=squad["answers"]` to `.compute()`.
Maybe we can just fix the description then.
lhoestq (MEMBER) commented on 2020-09-22 10:21:24:
> I installed datasets at /project/chiyuzh/evn_py36/datasets/src where is a writable directory.
> I also tried change the environment variables to the writable directory:
> `export HF_MODULES_PATH=/project/chiyuzh/evn_py36/datasets/cache_dir/`

I think it is `HF_MODULES_CACHE` and not `HF_MODULES_PATH` @chiyuzhang94 .
Could you try again and let me know if it fixes your issue ?
lhoestq (MEMBER) commented on 2020-09-22 10:41:36:
Hi. Closing this one since #626 updated the glue urls.

> 1. Why is it still blocking? Is it still downloading?

After downloading it generates the arrow file by iterating through the examples.
The number of examples processed by second is shown during the processing (not sure why it was not the case for you)

> 2. I specified split as train, so why is the test folder being populated?

It downloads every split
lhoestq (MEMBER) commented on 2020-09-22 14:41:47:
It looks like a pyarrow issue with google colab.
For some reason this code increases the disk usage of google colab while it actually writes into google drive:

```python
import pyarrow as pa

stream = pa.OSFile("/content/drive/My Drive/path/to/file.arrow", "wb")
writer = pa.RecordBatchStreamWriter(stream, schema=pa.schema({"text": pa.string()}))
writer.write_table(pa.Table.from_pydict({"text": ["a"*511 + "\n"] * ((1 << 30) // 512)}))  # 1GiB
writer.close()
stream.close()
```

Moreover if I `rm` the file on google drive, it frees disk space on google colab.
abhi1nandy2 (NONE) commented on 2020-09-22 14:47:21:
> Which OS are you using ?

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
lhoestq (MEMBER) commented on 2020-09-22 15:26:42:
Actually I also have the issue when writing a regular text file

```python
f = open("/content/drive/My Drive/path/to/file", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```

Is that supposed to happen ?
n1t0 (NONE) commented on 2020-09-22 20:48:50:
The parallelism is automatically disabled on `tokenizers` when the process gets forked, while we already used the parallelism capabilities of a tokenizer. We have to do it in order to avoid having the process hang, because we cannot safely fork a multithreaded process (cf https://github.com/huggingface/tokenizers/issues/187).
So if possible, the tokenizers shouldn't be used before the fork, so that each process can then make use of the parallelism. Otherwise using `TOKENIZERS_PARALLELISM=false` is the way to go.
lhoestq (MEMBER) commented on 2020-09-23 09:23:01:
> Thanks for taking a look. I pulled the master but I still see the key error.

I am no longer able to get the error since #659 was merged. Not sure why you still have it @timothyjlaurent 
Maybe it is a cache issue ? Could you try to use `load_from_cache_file=False` in your `.map()` calls ?
lhoestq (MEMBER) commented on 2020-09-23 09:27:38:
> The parallelism is automatically disabled on `tokenizers` when the process gets forked, while we already used the parallelism capabilities of a tokenizer. We have to do it in order to avoid having the process hang, because we cannot safely fork a multithreaded process (cf [huggingface/tokenizers#187](https://github.com/huggingface/tokenizers/issues/187)).
> So if possible, the tokenizers shouldn't be used before the fork, so that each process can then make use of the parallelism. Otherwise using `TOKENIZERS_PARALLELISM=false` is the way to go.

Ok thanks :)

Is there something we should do on the `datasets` side to avoid that that the program hangs ?

Also when doing `.map` with a tokenizer, the tokenizer is called once on the first examples of the dataset to check the function output before spawning the processes. Is that compatible with how tokenizers are supposed to be used with multiprocessing ?
yjernite (MEMBER) commented on 2020-09-23 22:56:24:
It would be amazing if we ended up with this much information on all of our datasets :) 

I don't think there's too much repetition, everything that is in here is relevant. The main challenge will be to figure out how to structure the sheet so that all of the information can be presented without overwhelming the reader. We'll also want to have as much of it as possible in structured form so it can be easily navigated.
wangcongcong123 (NONE) commented on 2020-09-23 23:49:42:
@saareliad I got the same issue that troubled me quite a while. Unfortunately, there are no good answers to this issue online, I tried it on Linux and that's absolutely fine. After hacking the source code, I solved this problem as follows.

In the source code file: arrow_dataset.py -> _map_single(...)

change
```python
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```
to
```python
tmp_file.close()
if update_data and tmp_file is not None:
    shutil.move(tmp_file.name, cache_file_name)
```

Then it works without needing multiple times runs to avoid the permission error.
I know this solution is unusual since it changes the source code. Hopefully, the lib's contributors can have better solutions in the future.
chiyuzhang94 (NONE) commented on 2020-09-24 00:59:49:
Hi @thomwolf , @lhoestq ,

Thanks for your suggestions. With the latest version of this package, I can load text data without Internet. 

But I found the speed of dataset loading is very slow. 

My scrips like this: 
```
    def token_encode(examples):
        tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=args.block_size)
        return tokenizer_out
    
    path = Path(file_path)
    files = sorted(path.glob('*'))
    dataset = load_dataset('./text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
    dataset = dataset.map(token_encode, batched=True)

    dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
```

I have 1,123,870,657 lines in my input directory. 
I can find the processing speed as following. It is very slow.  
```
| 13/1123871 [00:02<62:37:39,  4.98ba/s]^M  0%|   
| 14/1123871 [00:03<61:27:31,  5.08ba/s]^M  0%|          
| 15/1123871 [00:03<66:34:19,  4.69ba/s]^M  0%|         
| 16/1123871 [00:03<68:25:01,  4.56ba/s]^M  0%|          
| 17/1123871 [00:03<72:00:03,  4.34ba/s]^M  0%|       
```
Do you have any suggestions to accelerate this loading process?
leethu2012 (NONE) commented on 2020-09-24 11:30:38:
@lhoestq @sgugger Thanks for your comments. I have install from source code as you told, but the problem is still there.
To reproduce the issue, just replace [these lines](https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_language_modeling.py#L241-L258) with: 
(load_dataset and DataCollatorForDatasetsLanguageModeling as [above mentioned](https://github.com/huggingface/datasets/issues/633#issue-702440484))
```python
    dataset = load_dataset("bookcorpus")
    dataset = dataset.train_test_split(test_size=0.1)
    train_dataset = dataset['train']
    eval_dataset = dataset['test'] if training_args.do_eval else None

    data_collator = DataCollatorForDatasetsLanguageModeling(
        tokenizer=tokenizer,
        mlm=data_args.mlm,
        mlm_probability=data_args.mlm_probability,
        block_size=data_args.block_size
    )
```
and run by:
```bash
python run_language_modeling.py
--output_dir=output \
--model_type=bert \
--model_name_or_path=bert-base-uncased \
--do_train \
--do_eval \
--mlm 
```
thomwolf (MEMBER) commented on 2020-09-25 12:00:01:
Hi @xixiaoyao,
Depending on what you want to do you can:
- use a first step of `filter` to filter out the invalid examples: https://huggingface.co/docs/datasets/processing.html#filtering-rows-select-and-filter
- or directly detect the invalid examples inside the callable used with `map` and return them unchanged or even remove them at the same time if you are using `map` in batched mode. Here is an example where we use `map` in batched mode to add new rows on the fly but you can also use it to remove examples on the fly (that's what `filter` actually do under-the-hood): https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset
lhoestq (MEMBER) commented on 2020-09-25 15:14:55:
It looks like it's going to be fixed in pyarrow 2.0.0 :)

In the meantime I suggest to chunk big dataframes to create several small datasets, and then concatenate them using [concatenate_datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets)
lhoestq (MEMBER) commented on 2020-09-25 15:44:33:
> ![image](https://user-images.githubusercontent.com/36957508/93300760-fa9a8680-f829-11ea-9105-7a6f67ad8373.png)
> win10, py3.6
> 
> ```
> from datasets import Features, Value, ClassLabel, load_dataset
> 
> 
> features = Features({'text': Value('string'), 'ctext': Value('string')})
> file_dict = {'train': PATH/'summary.csv'}
> 
> dataset = load_dataset('csv', data_files=file_dict, script_version='master', delimiter='\t', column_names=['text', 'ctext'], features=features)
> ```

Since #644 it should now work on windows @ScottishFold007 

> Trying the following snippet, I get different problems on Linux and Windows.
> 
> ```python
> dataset = load_dataset("text", data_files="data.txt")
> # or 
> dataset = load_dataset("text", data_files=["data.txt"])
> ```
>
> Windows just seems to get stuck. Even with a tiny dataset of 10 lines, it has been stuck for 15 minutes already at this message:
> 
> ```
> Checking C:\Users\bramv\.cache\huggingface\datasets\b1d50a0e74da9a7b9822cea8ff4e4f217dd892e09eb14f6274a2169e5436e2ea.30c25842cda32b0540d88b7195147decf9671ee442f4bc2fb6ad74016852978e.py for additional imports.
> Found main folder for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text
> Found specific version folder for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7
> Found script file from https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py to C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7\text.py
> Couldn't find dataset infos file at https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text\dataset_infos.json
> Found metadata file for dataset https://raw.githubusercontent.com/huggingface/datasets/1.0.1/datasets/text/text.py at C:\Users\bramv\.cache\huggingface\modules\datasets_modules\datasets\text\7e13bc0fa76783d4ef197f079dc8acfe54c3efda980f2c9adfab046ede2f0ff7\text.json
> Using custom data configuration default
> ```

Same for you @BramVanroy .

Not sure about the one on linux though
lhoestq (MEMBER) commented on 2020-09-25 15:51:17:
> Ah, it’s much faster now(Takes around 15~20min).

Glad to see that it's faster now. What did you change exactly ?

> BTW, any way to set default tensor output as plain tensors with distributed training? The ragged tensors are incompatible with tpustrategy :(

Oh I didn't know about that. Feel free to open an issue to mention that.
I guess what you can do for now is set the dataset format to numpy instead of tensorflow, and use a wrapper of the dataset that converts the numpy arrays to tf tensors.
chiyuzhang94 (NONE) commented on 2020-09-25 16:59:32:
> You can use multiprocessing by specifying `num_proc=` in `.map()`
> 
> Also it looks like you have `1123871` batches of 1000 elements (default batch size), i.e. 1,123,871,000 lines in total.
> Am I right ?

Hi @lhoestq ,

Thanks. I will try it.

You are right. I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines.

I have another question. Because I am using a cloud server where only allows running a job up to 7 days. Hence, I need to resume my model every week. If the script needs to load and process the dataset every time. It is very low efficient based on the current processing speed. Is it possible that I process the dataset once and use the process cache to in the future work?
chiyuzhang94 (NONE) commented on 2020-09-26 03:44:07:
Hi @lhoestq ,

I tried to use multi-processor, but I got errors as follow: 
Because I am using python distributed training, it seems some conflicts with the distributed job. 

Do you have any suggestions?
```
I0925 10:19:35.603023 140737353971520 filelock.py:318] Lock 140737229443368 released on /tmp/pbs.1120510.pbsha.ib.sockeye/cache/_tmp_pbs.1120510.pbsha.ib.sockeye_cache_text_default-7fb934ed6fac5d01_0.0.0_512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7
fcc649178b014.lock
Traceback (most recent call last):
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 1024, in <module>
    main()
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 967, in main
    train_dataset = load_and_cache_examples(args, tokenizer, evaluate=False)
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 180, in load_and_cache_examples
    return HG_Datasets(tokenizer, file_path, args)
  File "/scratch/chiyuzh/roberta/run_language_modeling.py", line 119, in HG_Datasets
    dataset = dataset.map(token_encode, batched=True, batch_size = 10000, num_proc = 16)
  File "/project/chiyuzh/evn_py36/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1287, in map
    transformed_shards = [r.get() for r in results]
  File "/project/chiyuzh/evn_py36/lib/python3.6/site-packages/datasets/arrow_dataset.py", line 1287, in <listcomp>
    transformed_shards = [r.get() for r in results]
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/pool.py", line 424, in _handle_tasks
    put(task)
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
  File "/project/chiyuzh/evn_py36/lib/python3.6/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'HG_Datasets.<locals>.token_encode'
```
lhoestq (MEMBER) commented on 2020-09-26 14:39:39:
> I have another question. Because I am using a cloud server where only allows running a job up to 7 days. Hence, I need to resume my model every week. If the script needs to load and process the dataset every time. It is very low efficient based on the current processing speed. Is it possible that I process the dataset once and use the process cache to in the future work?

Feel free to save your processed dataset using `dataset.save_to_disk("path/to/save/directory")`.

Then you'll be able to reload it again using
```python
from datasets import load_from_disk

dataset = load_from_disk("path/to/save/directory")
```
lhoestq (MEMBER) commented on 2020-09-28 08:34:39:
Also not that C4 is a dataset that needs an Apache Beam runtime to be generated.
For example Dataflow, Spark, Flink etc.

Usually we generate the dataset on our side once and for all, but we haven't done it for C4 yet.
More info about beam datasets [here](https://huggingface.co/docs/datasets/beam_dataset.html)

Let me know if you have any questions
lhoestq (MEMBER) commented on 2020-09-28 08:52:37:
> BTW, is there a one-line command to make our building scripts pass flake8 test? (included code quality test), I got like trailing space or mixed space and tab warning and error, and fixed them manually.

I don't think so.
We have a command for black and isort but not flake8 as far as I know.
raruidol (NONE) commented on 2020-09-28 09:54:59:
> To complete what @lhoestq is saying, I think that to use the new version of the `text` processing script (which is on master right now) you need to either specify the version of the script to be the `master` one or to install the lib from source (in which case it uses the `master` version of the script by default):
> 
> ```python
> dataset = load_dataset('text', script_version='master', data_files=XXX)
> ```
> 
> We do versioning by default, i.e. your version of the dataset lib will use the script with the same version by default (i.e. only the `1.0.1` version of the script if you have the PyPI version `1.0.1` of the lib).

Linux here:

I was using the 0.4.0 nlp library load_dataset to load a text dataset of 9-10Gb without collapsing the RAM memory. However, today I got the csv error message mentioned in this issue. After installing the new (datasets) library from source and specifying the script_verson = 'master' I'm still having this same error message. Furthermore, I cannot use the dictionary "trick" to load the dataset since the system kills the process due to a RAM out of memory problem. Is there any other solution to this error? Thank you in advance.
lhoestq (MEMBER) commented on 2020-09-28 10:00:30:
Hi @raruidol 
To fix the RAM issue you'll need to shard your text files into smaller files (see https://github.com/huggingface/datasets/issues/610#issuecomment-691672919 for example)

I'm not sure why you're having the csv error on linux.
Do you think you could to to reproduce it on google colab for example ?
Or send me a dummy .txt file that reproduces the issue ?
lhoestq (MEMBER) commented on 2020-09-28 10:40:07:
Hi :)

I think what's left to do is
1- rebase from master, since we changed the name of the library
2- update the metadata file of the dataset using the command 
```
datasets-cli test ./datasets/compguesswhat --save_infos --all_configs --ignore_verifications
```

This command should update the checksum of the dropbox file
raruidol (NONE) commented on 2020-09-28 10:42:34:
@lhoestq 

The crash message shows up when loading the dataset:
```
print('Loading corpus...') 
files = glob.glob('corpora/shards/*') 
-> dataset = load_dataset('text', script_version='master', data_files=files) 
print('Corpus loaded.')
```
And this is the exact message:
```
Traceback (most recent call last):
  File "run_language_modeling.py", line 27, in <module>
    dataset = load_dataset('text', script_version='master', data_files=files)
  File "/home/jupyter-raruidol/DebatAnalyser/env/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/home/jupyter-raruidol/DebatAnalyser/env/lib/python3.7/site-packages/datasets/builder.py", line 471, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/jupyter-raruidol/DebatAnalyser/env/lib/python3.7/site-packages/datasets/builder.py", line 548, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/jupyter-raruidol/DebatAnalyser/env/lib/python3.7/site-packages/datasets/builder.py", line 892, in _prepare_split
    for key, table in utils.tqdm(generator, unit=" tables", leave=False, disable=not_verbose):
  File "/home/jupyter-raruidol/DebatAnalyser/env/lib/python3.7/site-packages/tqdm/std.py", line 1130, in __iter__
    for obj in iterable:
  File "/home/jupyter-raruidol/.cache/huggingface/modules/datasets_modules/datasets/text/512f465342e4f4cd07a8791428a629c043bb89d55ad7817cbf7fcc649178b014/text.py", line 107, in _generate_tables
    convert_options=self.config.convert_options,
  File "pyarrow/_csv.pyx", line 714, in pyarrow._csv.read_csv
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 84, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: CSV parse error: Expected 1 columns, got 2
```

And these are the pip packages I have atm and their versions:

```
Package         Version   Location                                                     
--------------- --------- -------------------------------------------------------------
certifi         2020.6.20 
chardet         3.0.4     
click           7.1.2     
datasets        1.0.2     
dill            0.3.2     
filelock        3.0.12    
future          0.18.2    
idna            2.10      
joblib          0.16.0    
numpy           1.19.1    
packaging       20.4      
pandas          1.1.1     
pip             19.0.3    
pyarrow         1.0.1     
pyparsing       2.4.7     
python-dateutil 2.8.1     
pytz            2020.1    
regex           2020.7.14 
requests        2.24.0    
sacremoses      0.0.43    
sentencepiece   0.1.91    
setuptools      40.8.0    
six             1.15.0    
tokenizers      0.8.1rc2  
torch           1.6.0     
tqdm            4.48.2    
transformers    3.0.2     /home/jupyter-raruidol/DebatAnalyser/env/src/transformers/src
```
mrm8488 (CONTRIBUTOR) commented on 2020-09-28 16:14:40:
To reproduce it: 
```bash
!df -h | grep sda1
```
```python
f = open("/content/drive/My Drive/test_to_remove.txt", "w")
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.write(("a"*511 + "\n") * ((1 << 30) // 512))  # 1GiB
f.close()
```
```bash
!ls -lh /content/drive/My\ Drive/test_to_remove.txt

!df -h | grep sda1

!rm -rf /content/drive/My\ Drive/test_to_remove.txt

```
[Colab](https://colab.research.google.com/drive/1D0UiweCYQwwWZ65EEhuqqbaDDbhJYXfm?usp=sharing)
lhoestq (MEMBER) commented on 2020-09-28 16:39:32:
I tested on google colab which is also linux using this code:

- first download an arbitrary text file
```bash
wget https://raw.githubusercontent.com/abisee/cnn-dailymail/master/url_lists/all_train.txt
```
- then run
```python
from datasets import load_dataset

d = load_dataset("text", data_files="all_train.txt", script_version='master')
```
And I don't get this issue.

\> Could you test on your side if these lines work @raruidol ?

also cc @Skyy93 as it seems you have the same issue

If it works:
It could mean that the issue could come from unexpected patterns in the files you want to use.
In that case we should find a way to handle them.

And if it doesn't work:
It could mean that it comes from the way pyarrow reads text files on linux.
In that case we should report it to pyarrow and find a workaround in the meantime

Either way it should help to find where this bug comes from and fix it :)

Thank you in advance !
raruidol (NONE) commented on 2020-09-28 17:00:43:
It looks like with your text input file works without any problem. I have been doing some experiments this morning with my input files and I'm almost certain that the crash is caused by some unexpected pattern in the files. However, I've not been able to spot the main cause of it. What I find strange is that this same corpus was being loaded by the nlp 0.4.0 library without any problem... Where can I find the code where you structure the input text data in order to use it with pyarrow?