t20062443/AT1ds huggingface dataset:
chiyuzhang94 (NONE) commented on 2020-10-07 20:30:36:
Hi @lhoestq ,

Thanks for your suggestion. 
I tried to process the dataset and save it to disk. 
I have 1.12B samples in the raw dataset. I used 16 processors.
I run this process job for 7 days. But it didn't finish. I don't why the processing is such slow. 

The log shows that some processors (\#12, \#14, \#15) are very slow. The different processor has a different speed. These slow processors look like a bottleneck. 

Could you please give me any suggestion to improve the processing speed? 

Thanks. 
Chiyu

Here is my code:
```
def token_encode(examples):
        tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=args.block_size)
        return tokenizer_out

path = Path(file_path)
files = sorted(path.glob('*'))
dataset = load_dataset('./text.py', data_files=files, cache_dir = args.data_cache_dir, split="train")
dataset = dataset.map(token_encode, batched=True, batch_size = 16384, num_proc = 16)
dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
dataset.save_to_disk(output_dir)
```
Here is the log. 
```
^M#6:   1%|▏         | 59/4288 [55:10<66:11:58, 56.35s/ba]
^M#1:   8%|▊         | 356/4288 [55:39<10:40:02,  9.77s/ba]
^M#2:   5%|▍         | 210/4288 [55:33<17:47:19, 15.70s/ba]
^M#0:  19%|█▉        | 836/4288 [55:53<4:08:56,  4.33s/ba]
^M#0:  20%|█▉        | 837/4288 [55:57<4:01:52,  4.21s/ba]
^M#1:   8%|▊         | 357/4288 [55:48<10:38:09,  9.74s/ba]
^M#0:  20%|█▉        | 838/4288 [56:01<4:02:56,  4.23s/ba]
^M#3:   4%|▎         | 155/4288 [55:43<24:41:20, 21.51s/ba]
^M#0:  20%|█▉        | 839/4288 [56:05<4:04:48,  4.26s/ba]
^M#12:   1%|          | 29/4288 [54:50<133:20:53, 112.72s/ba]
^M#2:   5%|▍         | 211/4288 [55:48<17:40:33, 15.61s/ba]
^M#14:   0%|          | 2/4288 [04:24<157:17:50, 132.12s/ba]
^M#15:   0%|          | 1/4288 [02:24<172:11:37, 144.60s/ba]
```
sshleifer (CONTRIBUTOR) commented on 2020-10-08 02:02:18:
This is a real issue, sorry for missing the mention @ibeltagy

We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines. 

Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

#### Sidebar: Wouldn't Deterministic Be Better?

`rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.

I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.
lhoestq (MEMBER) commented on 2020-10-08 08:26:37:
Hi !

As far as I can tell, there could be several reasons for your processes to have different speeds:
- some parts of your dataset have short passages while some have longer passages, that take more time to be processed
- OR there are other processes running that prevent some of them to run at full speed
- OR the value of `num_proc` is higher than the number of actual processes that you can run in parallel at full speed.

So I'd suggest you to check that you have nothing else running in parallel to your processing job, and also maybe take a look at the slow parts of the datasets.
When doing multiprocessing, the dataset is sharded in `num_proc` contiguous parts that are processed individually in each process. If you want to take a look at the dataset processed in the 12th shard of 16 for example, you can do:

```python
my_shard = dataset.shard(num_shards=16, index=12, contiguous=True)
```

Hope this helps, let me know if you find what is causing this slow down.
lhoestq (MEMBER) commented on 2020-10-08 08:58:40:
> This is a real issue, sorry for missing the mention @ibeltagy
> 
> We implemented a more involved [solution](https://github.com/huggingface/transformers/blob/99cb924bfb6c4092bed9232bea3c242e27c6911f/examples/seq2seq/utils.py#L481) that enforces that sentences are split with `\n` so that rougeLsum scores match papers even if models don't generate newlines.
> 
> Unfortunately, the best/laziest way I found to do this introduced an `nltk` dependency (For sentence splitting, all sentences don't end in `.`!!!), but this might be avoidable with some effort.

Thanks for the details, I didn't know about that. Maybe we should consider adding this processing step or at least mention it somewhere in the library or the documentation

> #### Sidebar: Wouldn't Deterministic Be Better?
> `rouge_scorer.scoring.BootstrapAggregator` is well named but is not deterministic which I would like to change for my mental health, unless there is some really good reason to sample 500 observations before computing f-scores.
> 
> I have a fix on a branch, but I wanted to get some context before introducting a 4th way to compute rouge. Scores are generally within .03 Rouge2 of boostrap after multiplying by 100, e.g 22.05 vs 22.08 Rouge2.

I think the default `n_samples` of the aggregator is 1000. We could increase it or at least allow users to change it if they want more precise results.
chiyuzhang94 (NONE) commented on 2020-10-08 19:00:39:
> Do you use a fast or a slow tokenizer from the `transformers` library @chiyuzhang94?

Hi @thomwolf ,
 I use this: 
```
from transformers import
AutoTokenizer.from_pretrained(args.model_name_or_path, cache_dir=args.cache_dir)
```

I guess this is a slow one, let me explore the fast tokenizer.
chiyuzhang94 (NONE) commented on 2020-10-08 19:12:59:
> Hi !
> 
> As far as I can tell, there could be several reasons for your processes to have different speeds:
> 
> * some parts of your dataset have short passages while some have longer passages, that take more time to be processed
> * OR there are other processes running that prevent some of them to run at full speed
> * OR the value of `num_proc` is higher than the number of actual processes that you can run in parallel at full speed.
> 
> So I'd suggest you to check that you have nothing else running in parallel to your processing job, and also maybe take a look at the slow parts of the datasets.
> When doing multiprocessing, the dataset is sharded in `num_proc` contiguous parts that are processed individually in each process. If you want to take a look at the dataset processed in the 12th shard of 16 for example, you can do:
> 
> ```python
> my_shard = dataset.shard(num_shards=16, index=12, contiguous=True)
> ```
> 
> Hope this helps, let me know if you find what is causing this slow down.

Hi @lhoestq ,

Thanks for your suggestions. 
I don't think my problem is due to any one of these seasons. 

1.  I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines. The last file is smaller a little bit. But they are similar. I randomly shuffled all the 1,123,870,657 lines. Hence, the sequences should also be similar across all the files. 

2. I run this script on the entire node. I requested all the resources on the nodes (40 CPUs, 384GB memory). Hence, these were not any other processes. 

 3. As I say, the node has 40 CPUs, but I set num_proc = 16. This should not be a problem.
chiyuzhang94 (NONE) commented on 2020-10-10 20:24:11:
Hi @thomwolf 
I am using `RobertaTokenizerFast` now. 

But the speed is still imbalanced, some processors are still slow. 
Here is the part of the log. #0 is always much fast than lower rank processors. 

```
#15:   3%|▎         | 115/3513 [3:18:36<98:01:33, 103.85s/ba]
#2:  24%|██▍       | 847/3513 [3:20:43<11:06:49, 15.01s/ba]
#1:  37%|███▋      | 1287/3513 [3:20:52<6:19:02, 10.22s/ba]
#0:  72%|███████▏  | 2546/3513 [3:20:52<1:51:03,  6.89s/ba]
#3:  18%|█▊        | 617/3513 [3:20:36<15:50:30, 19.69s/ba]
#0:  73%|███████▎  | 2547/3513 [3:20:59<1:50:25,  6.86s/ba]
#1:  37%|███▋      | 1288/3513 [3:21:02<6:21:13, 10.28s/ba]
#7:   7%|▋         | 252/3513 [3:20:09<44:09:03, 48.74s/ba]
#12:   4%|▍         | 144/3513 [3:19:19<78:00:54, 83.36s/ba]
#4:  14%|█▍        | 494/3513 [3:20:37<20:46:06, 24.77s/ba]
#0:  73%|███████▎  | 2548/3513 [3:21:06<1:49:26,  6.80s/ba]
#2:  24%|██▍       | 848/3513 [3:20:58<11:06:17, 15.00s/ba]
```
Here is my script related to the datasets processing, 

```
tokenizer = RobertaTokenizerFast.from_pretrained(args.model_name_or_path, cache_dir=args.cache_dir)

def token_encode(examples):
    tokenizer_out = tokenizer(examples['text'], truncation=True,  padding="max_length", add_special_tokens=True, max_length=128)
    return tokenizer_out

def HG_Datasets(tokenizer, file_path, args):
    
    path = Path(file_path)
    files = sorted(path.glob('*'))
    dataset = load_dataset('./text.py', data_files=files, cache_dir = ""./, split="train")
    dataset = dataset.map(token_encode, batched=True, batch_size = 20000, num_proc = 16)

    dataset.set_format(type='torch', columns=['input_ids', 'attention_mask'])
    return dataset

```
I have 1,123,870,657 lines totally in the path. I split the large file into 440 small files. Each file has 2,560,000 lines.

Could you please give any suggestion? Thanks very much!!
lhoestq (MEMBER) commented on 2020-10-12 09:29:05:
We only support http by default for downloading.
If you really need to use ftp, then feel free to use a library that allows to download through ftp in your dataset script (I see that you've started working on #722 , that's awesome !). The users will get a message to install the extra library when they load the dataset.

To make the download_manager work with a custom downloader, you can call `download_manager.download_custom` instead of `download_manager.download_and_extract`. The expected arguments are the following:
```
url_or_urls: url or `list`/`dict` of urls to download and extract. Each
        url is a `str`.
custom_download: Callable with signature (src_url: str, dst_path: str) -> Any
        as for example `tf.io.gfile.copy`, that lets you download from google storage
```
SparkJiao (NONE) commented on 2020-10-12 12:33:42:
Hi, I have manually downloaded the compressed dataset `openwebtext.tar.xz' and use the following command to preprocess the examples:
```
>>> dataset = load_dataset('/home/admin/workspace/datasets/datasets-master/datasets-master/datasets/openwebtext', data_dir='/home/admin/workspace/datasets')
Using custom data configuration default
Downloading and preparing dataset openwebtext/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02...
Dataset openwebtext downloaded and prepared to /home/admin/.cache/huggingface/datasets/openwebtext/default/0.0.0/5c636399c7155da97c982d0d70ecdce30fbca66a4eb4fc768ad91f8331edac02. Subsequent calls will reuse this data.
>>> len(dataset['train'])
74571
>>>
```
The size of the pre-processed example file is only 354MB, however the processed bookcorpus dataset is 4.6g. Are there any problems?
sleepinyourhat (NONE) commented on 2020-10-12 23:04:17:
Question: Where does this standard come from? It looks similar to both
'Data Statements' and 'Datasheets for Datasets', but it doesn't look quite
like either.

On Mon, Oct 12, 2020 at 4:27 PM Yacine Jernite <notifications@github.com>
wrote:

> Merged #663
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=s4_X-BSEnTKgGg9rPLBt3cyVptyMX_iWD5Ql3UMBi-I&e=>
> into master.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23event-2D3868180429&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=elcM4umqReQfIrgHhpey9W_wPaq5QRgq7xNlubM47QI&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWJVGQRCR4OTTV27VTTSKNRBXANCNFSM4RWKAZRA&d=DwMCaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=D34WbiHBTYHOdXsI9JV9wJqSieP6zAPGqGKDziM5uKU&s=NB6nEROnTPgwNyF3ZklOmHnvP7kOkOm7sEa740KbVCs&e=>
> .
>
lhoestq (MEMBER) commented on 2020-10-13 07:57:50:
A new configuration for those datasets should do the job then.
Note that until now datasets like xsum only had one configuration. It means that users didn't have to specify the configuration name when loading the dataset. If we add new configs, users that update the lib will have to update their code to specify the default/standard configuration name (not the one with pseudo labels).
stas00 (CONTRIBUTOR) commented on 2020-10-13 21:50:06:
Thank you!

> Not sure I understand what you mean by the second part?

Compare the 2:
* https://huggingface.co/datasets/wikihow
* https://huggingface.co/nlp/viewer/?dataset=wikihow&config=all
Can you see the difference? 2nd has formatting, 1st doesn't.
danyaljj (CONTRIBUTOR) commented on 2020-10-16 01:35:36:
@lhoestq @sshleifer like @jbragg said earlier, the main issue for us is that the current XSUM dataset (in your package) does not have IDs suggested by the original dataset ([here is the file](https://raw.githubusercontent.com/EdinburghNLP/XSum/master/XSum-Dataset/XSum-TRAINING-DEV-TEST-SPLIT-90-5-5.json).) Would appreciate if you update the XSUM dataset to include the instance IDs. 

The missing instances is also a problem, but likely not worth pursuing given its relatively small scale.
lhoestq (MEMBER) commented on 2020-10-16 08:21:38:
> Thanks @lhoestq
> Are there any guidelines for the dummy data?
> In this particular case for example, the dataset fetches from two hardcoded URLs.
> Do I just `head -n 10` both files and zip them?

Yes the idea is just to have a few examples to properly test the script and make sure it keeps working in the long run.

And FYI there's a command to help you name the dummy data files correctly. More info in the documentation [here](https://huggingface.co/docs/datasets/share_dataset.html#adding-dummy-data)
stas00 (CONTRIBUTOR) commented on 2020-10-17 02:57:26:
Great, as you found it useful I improved the code a bit to automate indentation in the parent class, so that the child repr doesn't need to guess the indentation level, while repr'ing nicely on its own.

- do we want indent=4 or 2?
- do we want `{` ... `}` or w/o?

currently it's indent4 and w/ curly braces, so it looks:

```
DatasetDict({
    train: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 157252
    })
    validation: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5599
    })
    test: Dataset({
        features: ['text', 'headline', 'title'],
        num_rows: 5577
    })
})
```
just child:
```
Dataset({
    features: ['text', 'headline', 'title'],
    num_rows: 5577
})
```
iliemihai (CONTRIBUTOR) commented on 2020-10-19 13:45:15:
The same problem happens with "csv".
`
from datasets import load_dataset

dataset = load_dataset("csv", data_files="custom_data.csv", delimiter="\t", column_names=["title", "text"], script_version="master")
`
ArrowInvalid: CSV parse error: Expected 1 columns, got 2
lhoestq (MEMBER) commented on 2020-10-19 13:49:10:
> The same problem happens with "csv".
> `
> from datasets import load_dataset
> 
> dataset = load_dataset("csv", data_files="custom_data.csv", delimiter="\t", column_names=["title", "text"], script_version="master")
> `
> ArrowInvalid: CSV parse error: Expected 1 columns, got 2

Could you open a new issue for CSV please ?
Text and CSV datasets used to share some similarities in the way the file are loaded and parsed but it's no longer the case since 1.1.2
Hopefully we can find a fix quicker now for such errors :) 
Also if you can provide a google colab to reproduce the issue that would be awesome !
moscow25 (NONE) commented on 2020-10-20 03:41:23:
On the Amazon Reviews dataset, the original UCSD website has noted these are now updated to include product reviews through 2018 -- actually quite recent compared to many other datasets. Almost certainly the largest NLP dataset out there with labels!
https://jmcauley.ucsd.edu/data/amazon/ 

Any chance someone has time to onboard this dataset in a HF way?

cc @sshleifer
lhoestq (MEMBER) commented on 2020-10-20 08:59:49:
Thanks for reporting.
In theory since the dataset script is just made to yield examples to write them into an arrow file, it's not supposed to create memory issues.

Could you please try to run this exact same loop in a separate script to see if it's not an issue with `PIL` ?
You can just copy paste what's inside `_generate_examples` and remove all the code for `datasets` (remove yield).

If the RAM usage stays low after 600 examples it means that it comes from some sort of memory leak in the library, or with pyarrow.
iliemihai (CONTRIBUTOR) commented on 2020-10-20 09:26:32:
I think another good example is the following:
`
from datasets import load_dataset
`
`
dataset = load_dataset("csv", data_files=["./sts-dev.csv"], delimiter="\t", column_names=["one", "two", "three", "four",  "score", "sentence1", "sentence2"], script_version="master")`
`

Displayed error `CSV parse error: Expected 7 columns, got 6` even tough I put 7 columns. First four columns from the csv don't have a name, so I've named them by default. The csv file is the .dev file from STSb benchmark dataset.
AmitMY (CONTRIBUTOR) commented on 2020-10-20 14:14:29:
Here's an equivalent loading code:
```python
images_path = "PHOENIX-2014-T-release-v3/PHOENIX-2014-T/features/fullFrame-210x260px/train"

for dir_path in tqdm(os.listdir(images_path)):
    frames_path = os.path.join(images_path, dir_path)
    np_frames = []
    for frame_name in os.listdir(frames_path):
        frame_path = os.path.join(frames_path, frame_name)
        im = Image.open(frame_path)
        np_frames.append(np.asarray(im))
        im.close()
```

The process takes 0.3% of memory, even after 1000 examples on the small machine with 120GB RAM.

I guess something in the datasets library doesn't release the reference to the objects I'm yielding, but no idea how to test for this
akhilkapil (NONE) commented on 2020-10-20 19:09:43:
Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error


---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-145-ca218223c9fc> in <module>()
----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
      2 val_f1 = f1_score_func(predictions, true_val)
      3 tqdm.write(f'Validation loss: {val_loss}')
      4 tqdm.write(f'F1 Score (Weighted): {val_f1}')

6 frames
/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in <genexpr>(.0)
    160 
    161     def __getitem__(self, index):
--> 162         return tuple(tensor[index] for tensor in self.tensors)
    163 
    164     def __len__(self):

TypeError: new(): invalid data type 'str'
AmitMY (CONTRIBUTOR) commented on 2020-10-21 14:12:53:
If it's just 1-2 weeks, I think it's best if we wait. I don't think it is very urgent to add it, and it will be much more useful with the images loaded rather than not (the images are low resolution, and thus papers using this dataset actually fit the entire video into memory anyway)

I'll keep working on other datasets in the meanwhile :)
Doragd (NONE) commented on 2020-10-22 08:33:56:
> Hi, actually the thing is I am getting the same error and even after tokenizing them I am passing them through batch_encode_plus.
> I dont know what seems to be the problem is. I even converted it into 'pt' while passing them through batch_encode_plus but when I am evaluating my model , i am getting this error
> 
> TypeError Traceback (most recent call last)
> in ()
> ----> 1 val_loss, predictions, true_val = evaluate(dataloader_validation)
> 2 val_f1 = f1_score_func(predictions, true_val)
> 3 tqdm.write(f'Validation loss: {val_loss}')
> 4 tqdm.write(f'F1 Score (Weighted): {val_f1}')
> 
> 6 frames
> /usr/local/lib/python3.6/dist-packages/torch/utils/data/dataset.py in (.0)
> 160
> 161 def **getitem**(self, index):
> --> 162 return tuple(tensor[index] for tensor in self.tensors)
> 163
> 164 def **len**(self):
> 
> TypeError: new(): invalid data type 'str'

I got the same error and fix it .
you can check your input where there may be string contained.
such as
```
a = [1,2,3,4,'<unk>']
torch.tensor(a)
```
AmitMY (CONTRIBUTOR) commented on 2020-10-24 05:16:23:
@hoanganhpham1006 yes.
See pull request https://github.com/huggingface/datasets/pull/722 , it has a loader for this dataset, mostly ready.
There's one issue that delays it being merged - https://github.com/huggingface/datasets/issues/741 - regarding memory consumption.
AmitMY (CONTRIBUTOR) commented on 2020-10-24 07:26:10:
The dataset loader is not yet ready, because of that issue.
If you want to just download the dataset the old-fashioned way, just go to: https://www-i6.informatik.rwth-aachen.de/ftp/pub/rwth-phoenix/2016/phoenix-2014-T.v3.tar.gz (the ftp link is now broken, and its available over https)
li1117heex (NONE) commented on 2020-10-24 10:49:09:
```python
tokenizer = FunnelTokenizer.from_pretrained('funnel-transformer/small')

def tokenize(batch):
    return tokenizer(batch['text'], padding='max_length', truncation=True,max_length=512)

dataset = load_dataset("bookcorpus",split='train[:1000]').shuffle()
dataset = dataset.map(tokenize, batched=True, batch_size=512)

# dataset = LineByLineTextDataset(
#     tokenizer=tokenizer,
#     file_path="./wiki1000.txt",
#     block_size=128
# )

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

config=FunnelConfig(
    return_dict=True
)

model= FunnelForMaskedLM(config=config)

training_args = TrainingArguments(
    output_dir="./checkpoints",
    overwrite_output_dir=True,
    do_train=True,
    num_train_epochs=1,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=16,
    save_steps=10000,
    logging_dir='./ptlogs'
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
)
trainer.train()
```
li1117heex (NONE) commented on 2020-10-24 10:51:57:
`RuntimeError: CUDA out of memory. Tried to allocate 954.00 MiB (GPU 0; 15.90 GiB total capacity; 14.35 GiB already allocated; 753.75 MiB free; 14.39 GiB reserved in total by PyTorch)
Exception raised from malloc at /pytorch/c10/cuda/CUDACachingAllocator.cpp:272 (most recent call first):`

part of error output
lhoestq (MEMBER) commented on 2020-10-26 09:20:48:
Hi ! Thanks for reporting.
Is the distribution of text length of your data evenly distributed across your dataset ? I mean, could it be because the examples in the first part of your dataset are slower to process ?
Also could how many CPUs can you use for multiprocessing ?
```python
import multiprocessing
print(multiprocessing.cpu_count())
```
Which tokenizer are you using ?