t20062443/AT1ds huggingface dataset:
gaceladri (NONE) commented on 2021-02-10 10:25:10:
My workaround is to shard the dataset into splits in my ssd disk and feed the data in different training sessions. But it is a bit of a pain when we need to reload the last training session with the rest of the split with the Trainer in transformers.

I mean, when I split the training and then reloads the model and optimizer, it not gets the correct global_status of the optimizer, so I need to hardcode some things. I'm planning to open an issue in transformers and think about it.
```
from datasets import load_dataset

book_corpus = load_dataset("bookcorpus", split="train[:25%]")
wikicorpus = load_dataset("wikicorpus", split="train[:25%]")
openwebtext = load_dataset("openwebtext", split="train[:25%]")

big_dataset = datasets.concatenate_datasets([wikicorpus, openwebtext, book_corpus])
big_dataset.shuffle(seed=42)
big_dataset = big_dataset.map(encode, batched=True, num_proc=20, load_from_cache_file=True, writer_batch_size=5000)
big_dataset.set_format(type='torch', columns=["text", "input_ids", "attention_mask", "token_type_ids"])


training_args = TrainingArguments(
    output_dir="./linear_bert",
    overwrite_output_dir=True,
    per_device_train_batch_size=71,
    save_steps=500,
    save_total_limit=10,
    logging_first_step=True,
    logging_steps=100,
    gradient_accumulation_steps=9,
    fp16=True,
    dataloader_num_workers=20,
    warmup_steps=24000,
    learning_rate=0.000545205002870214,
    adam_epsilon=1e-6,
    adam_beta2=0.98,
    weight_decay=0.01,
    max_steps=138974,  # the total number of steps after concatenating 100% datasets
    max_grad_norm=1.0,
)

trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=big_dataset,
    tokenizer=tokenizer))
```

I do one training pass with the total steps of this shard and I use len(bbig)/batchsize to stop the training (hardcoded in the trainer.py) when I pass over all the examples in this split.

Now Im working, I will edit the comment with a more elaborated answer when I left the work.
lhoestq (MEMBER) commented on 2021-02-10 13:42:41:
Hi @sshleifer :) 

We don't have methods like `Dataset.add_batch` or `Dataset.add_entry/add_item` yet.
But that's something we'll add pretty soon. Would an API that looks roughly like this help ? Do you have suggestions ?
```python
import numpy as np
from datasets import Dataset

tokenized = [np.array([4,4,2]), np.array([8,6,5,5,2]), np.array([3,3,31,5])

# API suggestion (not available yet)
d = Dataset()
for input_ids in tokenized:
    d.add_item({"input_ids": input_ids})

print(d[0]["input_ids"])
# [4, 4, 2]
```

Currently you can define a dataset with what @albertvillanova suggest, or via a generator using dataset builders. It's also possible to [concatenate datasets](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=concatenate#datasets.concatenate_datasets).
lhoestq (MEMBER) commented on 2021-02-10 15:11:40:
Just some addition about loading only a subset of the data:
Currently if even you specify `split='train[:10]'`, it downloads and generate the full dataset, so that you can pick another part afterward if you want to. We may change that in the future and use streaming.

And about your open files issue, you can try to close each image file after reading its content.
lhoestq (MEMBER) commented on 2021-02-10 16:18:25:
Hi ! You're right this can can useful.
This should be easy to add, so feel free to give it a try if you want to contribute :)
I think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)
lhoestq (MEMBER) commented on 2021-02-10 16:42:34:
Hi @wumpusman 
`datasets` has a caching mechanism that allows to cache the results of `.map` so that when you want to re-run it later it doesn't recompute it again.
So when you do `.map`, what actually happens is:
1. compute the hash used to identify your `map` for the cache
2. apply your function on every batch

This can explain the time difference between your different experiments.

The hash computation time depends of how complex your function is. For a tokenizer, the hash computation scans the lists of the words in the tokenizer to identify this tokenizer. Usually it takes 2-3 seconds.

Also note that you can disable caching though using
```python
import datasets

datasets.set_caching_enabled(False)
```
Valahaar (CONTRIBUTOR) commented on 2021-02-10 16:58:07:
Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.
lhoestq (MEMBER) commented on 2021-02-10 17:35:10:
Hi ! We're in the process of switching the community datasets to git repos, exactly like what we're doing for models.
You can find an example here:
https://huggingface.co/datasets/lhoestq/custom_squad/tree/main

We'll update the CLI in the coming days and do a new release :)

Also cc @julien-c maybe we can make improve the error message ?
Valahaar (CONTRIBUTOR) commented on 2021-02-10 17:39:47:
Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))

```python
result = (
  sentence_counter,
    {
      "id": str(sentence_counter),
      "meta": {
        "year": year,
        "imdbId": imdb_id,
        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},
        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},
        # or maybe src/tgt? I'd go with the first one for consistency with 'translation'
        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},
        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},
      },
      "translation": {l1: x, l2: y},
    },
  )
```
Or at top level, avoiding nesting into 'meta'?
aclifton314 (CONTRIBUTOR) commented on 2021-02-10 18:33:11:
Closer still!
```python
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git commit
[logiqa_en 2664fe7f] fixed several issues with logiqa_en.
 4 files changed, 324 insertions(+)
 create mode 100644 datasets/logiqa_en/README.md
 create mode 100644 datasets/logiqa_en/dataset_infos.json
 create mode 100644 datasets/logiqa_en/dummy/1.1.0/dummy_data.zip
 create mode 100644 datasets/logiqa_en/logiqa_en.py
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git fetch upstream
remote: Enumerating objects: 1, done.
remote: Counting objects: 100% (1/1), done.
remote: Total 1 (delta 0), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (1/1), 590 bytes | 590.00 KiB/s, done.
From https://github.com/huggingface/datasets
   6e114a0c..318b09eb  master     -> upstream/master
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git rebase upstream/master 
error: cannot rebase: You have unstaged changes.
error: Please commit or stash them.
aclifton@pop-os:~/data/hf_datasets_sprint/datasets$ git push -u origin logiqa_en
Username for 'https://github.com': aclifton314
Password for 'https://aclifton314@github.com': 
To https://github.com/aclifton314/datasets
 ! [rejected]          logiqa_en -> logiqa_en (non-fast-forward)
error: failed to push some refs to 'https://github.com/aclifton314/datasets'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
```
jguoxu (NONE) commented on 2021-02-10 22:39:48:
@lhoestq Hi! I encounter the same error when loading `yelp_review_full`.

```
from datasets import load_dataset
dataset_yp = load_dataset("yelp_review_full")
```

When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?
lhoestq (MEMBER) commented on 2021-02-11 10:18:06:
> When you say the "Quota Exceeded from Google drive". Is this a quota from the dataset owner? or the quota from our (the runner) Google Drive?

Each file on Google Drive can be downloaded only a certain amount of times per day because of a quota. The quota is reset every day. So if too many people download the dataset the same day, then the quota is likely to exceed.
That's a really bad limitations of Google Drive and we should definitely find another host for these dataset than Google Drive.
For now I would suggest to wait and try again later..

So far the issue happened with CNN DailyMail, Amazon Polarity and Yelp Reviews. 
Are you experiencing the issue with other datasets ? @calebchiam @dtch1997
lhoestq (MEMBER) commented on 2021-02-11 14:21:59:
Reopening since @bhavitvyamalik started looking into it !

Also I'm posting here a function that could be helpful to support preserving the dtype of tensors.

It's used to build a pyarrow array out of a numpy array and:
- it doesn't convert the numpy array to a python list
- it keeps the precision of the numpy array for the pyarrow array
- it works with multidimensional arrays (while `pa.array` can only take a 1D array as input)
- it builds the pyarrow ListArray from offsets created on-the-fly and values that come from the flattened numpy array

```python
from functools import reduce
from operator import mul

import numpy as np
import pyarrow as pa

def pa_ndarray(a):
    """Build a PyArrow ListArray from a multidimensional NumPy array"""
    values = pa.array(a.flatten()) 
    for i in range(a.ndim - 1): 
        n_offsets = reduce(mul, a.shape[:a.ndim - i - 1], 1) 
        step_offsets = a.shape[a.ndim - i - 1] 
        offsets = pa.array(np.arange(n_offsets + 1) * step_offsets, type=pa.int32()) 
        values = pa.ListArray.from_arrays(offsets, values) 
    return values 

narr = np.arange(42).reshape(7, 2, 3).astype(np.uint8)
parr = pa_ndarray(narr)
assert isinstance(parr, pa.Array)
assert parr.type == pa.list_(pa.list_(pa.uint8()))
assert narr.tolist() == parr.to_pylist()
```

The only costly operation is the offsets computations. Since it doesn't iterate on the numpy array values this function is pretty fast.
gchhablani (CONTRIBUTOR) commented on 2021-02-11 17:00:11:
Hi @lhoestq,
Thanks for your response.

I used `gc.collect()` inside the loop and that worked for me. I think since we are using a generator, and if I have something like `train[100000:100002]`, we will need to generate the first 1000001 examples and store. Ofcourse, this feature isn't a necessity right now, I suppose.
johncookds (NONE) commented on 2021-02-12 04:34:43:
I believe this is an actual issue, tokenizing a ~4GB txt file went from an hour and a half to ~10 minutes when I switched from my pre-trained tokenizer(on the same dataset) to the default gpt2 tokenizer.
Both were loaded using:
```
AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)
```
I trained the tokenizer using ByteLevelBPETokenizer from the Tokenizers library and save it to a tokenizer.json file.

I have tested the caching ideas above, changing the number of process, the TOKENIZERS_PARALLELISM env variable, keep_in_memory=True and batching with different sizes.

Apologies I can't really upload much code, but wanted to back up the finding and hopefully a fix/the problem can be found.
I will comment back if I find a fix as well.
lhoestq (MEMBER) commented on 2021-02-12 14:34:30:
Hi @alejandrocros it looks like an incompatibility with the current Trainer @sgugger 
Indeed currently the Trainer of `transformers` doesn't support a dataset with a transform

It looks like it comes from this line: https://github.com/huggingface/transformers/blob/f51188cbe74195c14c5b3e2e8f10c2f435f9751a/src/transformers/trainer.py#L442

This line sets the format to not return certain unused columns. But this has two issues:
1. it forgets to also set the format_kwargs (this causes the error you got):
```python
dataset.set_format(type=dataset.format["type"], columns=columns, format_kwargs=dataset.format["format_kwargs"])
```
2. the Trainer wants to keep only the fields that are used as input for a model. However for a dataset with a transform, the output fields are often different from the columns fields. For example from a column "text" in the dataset, the strings can be transformed on-the-fly into "input_ids". If you want your dataset to only output certain fields and not other you must change your transform function.
lhoestq (MEMBER) commented on 2021-02-12 16:03:42:
I just added the steps to share a dataset on the datasets hub. It's highly inspired by the steps to share a model in the `transformers` doc.

Moreover once the new huggingface_hub is released we can update the version in the setup.py. We also need to update the command to create a dataset repo in the documentation

I added a few more tests with the "lhoestq/test" dataset I added on the hub and it works fine :)
johncookds (NONE) commented on 2021-02-12 16:11:29:
Hi yes, I'm closing the loop here with some timings below. The issue seems to be at least somewhat/mainly with the tokenizer's themselves. Moreover legacy saves of the trainer tokenizer perform faster but differently than the new tokenizer.json saves(note nothing about the training process/adding of special tokens changed between the top two trained tokenizer tests, only the way it was saved). This is only a 3x slowdown vs like a 10x but I think the slowdown is most likely due to this.

```
trained tokenizer - tokenizer.json save (same results for AutoTokenizer legacy_format=False):
Tokenizer time(seconds): 0.32767510414123535
Tokenized avg. length: 323.01

trained tokenizer - AutoTokenizer legacy_format=True:
Tokenizer time(seconds): 0.09258866310119629
Tokenized avg. length: 301.01

GPT2 Tokenizer from huggingface
Tokenizer time(seconds): 0.1010282039642334
Tokenized avg. length: 461.21
```
lhoestq (MEMBER) commented on 2021-02-12 16:42:17:
Hi !
About the problems you mentioned:
- Saving the infos is only done for the configurations inside the BUILDER_CONFIGS. Otherwise you would need to run the scripts on ALL language pairs, which is not what we want.
- Moreover when you're on your branch, please specify the path to your local version of the dataset script, like "./datasets/open_subtitles". Otherwise the dataset is loaded from the master branch on github.
Hope that clarifies things a bit

And of course feel free to add methods or classmethods to your builder.
wumpusman (NONE) commented on 2021-02-12 16:45:35:
@lhoestq ,

Hi, which version of datasets has datasets.set_caching_enabled(False)? I get 
module 'datasets' has no attribute 'set_caching_enabled'. To hopefully get around this, I reran my code on a new set of data, and did so only once.

@johncookds , thanks for chiming in, it looks this might be an issue of Tokenizer.

**Tokenizer**: The runtime of GPT2TokenizerFast.from_pretrained("gpt2") on 1000 chars is: **143 ms**
**SlowTokenizer**: The runtime of a locally saved and loaded Tokenizer using the same vocab  on 1000 chars is:  **4.43 s**

That being said, I compared performance on the map function:

Running Tokenizer versus using it in the map function for 1000 chars goes from **141 ms** to **356 ms** 
Running SlowTokenizer versus using it in the map function for 1000 chars with a single element goes from **4.43 s** to **9.76 s**

I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

@lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Regards,

Michael
lhoestq (MEMBER) commented on 2021-02-12 16:58:29:
Thanks for the experiments @johncookds and @wumpusman ! 

> Hi, which version of datasets has datasets.set_caching_enabled(False)?

Currently you have to install `datasets` from source to have this feature, but this will be available in the next release in a few days.

> I'm trying to figure out why the overhead of map would increase the time by double (figured it would be a fixed increase in time)? Though maybe this is expected behavior.

Could you also try with double the number of characters ? This should let us have an idea of the fixed cost (hashing) and the dynamic cost (actual tokenization, grows with the size of the input)

> @lhoestq, do you by chance know how I can redirect this issue to Tokenizer?

Feel free to post an issue on the `transformers` repo. Also I'm sure there should be related issues so you can also look for someone with the same concerns on the `transformers` repo.
wumpusman (NONE) commented on 2021-02-12 17:14:46:
@lhoestq,

I just checked that previous run time was actually  3000 chars. I increased it to 6k chars, again, roughly double.

SlowTokenizer **7.4 s** to **15.7 s**
Tokenizer: **276 ms** to **616 ms**

I'll post this issue on Tokenizer, seems it hasn't quite been raised (albeit I noticed a similar issue that might relate).

Regards,

Michael
sgugger (CONTRIBUTOR) commented on 2021-02-12 23:14:04:
FYI that option can be removed with `remove_unused_columns = False` in your `TrainingArguments`, so there is a workaround @alexvaca0 while the fix in `Trainer` is underway.

@lhoestq I think I will just use the line you suggested and if someone is using the columns that are removed in their transform they will need to change `remove_unused_columns` to `False`. We might switch the default of that argument in the next version if that proves too bug-proof.
patrickvonplaten (CONTRIBUTOR) commented on 2021-02-15 05:06:24:
Hey @BirgerMoell - awesome that you started working on Common Voice. Common Voice is a bit special since, there is no direct download link to download the data. In these cases we usually consider two options:

1) Find a hacky solution to extract the download link somehow from the XLM tree of the website 
2) If this doesn't work we force the user to download the data himself and add a `"data_dir"` as an input parameter. E.g. you can take a look at how it is done for [this](https://github.com/huggingface/datasets/blob/66f2a7eece98d2778bd22bb5034cb7c2376032d4/datasets/arxiv_dataset/arxiv_dataset.py#L66) 

Also the documentation here: https://huggingface.co/docs/datasets/add_dataset.html?highlight=data_dir#downloading-data-files-and-organizing-splits (especially the "note") might be helpful.
lhoestq (MEMBER) commented on 2021-02-15 10:46:44:
Thanks for the investigation @gaceladri 

Apparently this happens when `num_workers>0` and has to do with objects being copied-on-write.
Did you try setting num_workers to 0 @gaceladri ?
If the issue doesn't happen with `num_workers=0` then this would confirm that it's indeed related to this python/pytorch issue.

Since a `Dataset` object is a wrapper of a pyarrow Table, we should investigate if the data being copied comes from the Table itself or from metadata in the `Dataset` object. If it comes from the metadata in the `Dataset` object, we should be able to implement a workaround. But if it comes from the Table, we'll need to see with the pyarrow team what we can do...
gaceladri (NONE) commented on 2021-02-15 12:04:38:
@lhoestq But despite this, I got lost into the [class Dataset()](https://huggingface.co/docs/datasets/_modules/datasets/arrow_dataset.html#Dataset) reading the pyarrow files.

Edit: but you should be rigth, that it does not have to be related to multiprocessing since it keeps happening when `num_workers=0`
lhoestq (MEMBER) commented on 2021-02-15 13:29:26:
There could be several situations:
- download a file with no extraction
- download a file and extract it
- download a file, extract it and then inside the output folder extract some more files
- extract a local file (for datasets with data that are manually downloaded for example)
- extract a local file, and then inside the output folder extract some more files

So I think it's ok to have `downloaded_paths` as a dict url -> downloaded_path and `extracted_paths` as a dict local_path -> extracted_path.
gaceladri (NONE) commented on 2021-02-15 13:33:56:
@lhoestq It keeps happening. I have uninstalled wandb from my env, setted `%env WANDB_DISABLED=true` on my notebook, and commented this func:

```
def get_available_reporting_integrations():
    integrations = []
    if is_azureml_available():
        integrations.append("azure_ml")
    if is_comet_available():
        integrations.append("comet_ml")
    if is_mlflow_available():
        integrations.append("mlflow")
    if is_tensorboard_available():
        integrations.append("tensorboard")
    # if is_wandb_available():
    #     integrations.append("wandb")
    return integrations
```
As a fast test and it keeps increasing the ram memory. Wandb could not be the blameworthy here.
patrickvonplaten (CONTRIBUTOR) commented on 2021-02-15 13:34:52:
Hey @anton-l,

Thanks a lot for the very clean integration!

1) I think we should now start having "automatic-speech-recognition" as a label in the dataset tagger (@yjernite is it easy to add?). But we can surely add this dataset with the tag you've added and then later change the label to `asr` 

2) That's perfect! Yeah good question - we're currently thinking about a better design with @lhoestq 

3) Again tagging @yjernite & @lhoestq here - guess we should add this license though!
gaceladri (NONE) commented on 2021-02-15 13:43:01:
@lhoestq sure. Here you have https://colab.research.google.com/drive/1ba09ZOpyHGAOQLcsxiQAHRXl10qnMU5o?usp=sharing let me know if the link works and it reproduces the issue. To me, it reproduces the issue, since if you start the training the ram memory keeps increasing.

Let me know. Thanks!
lhoestq (MEMBER) commented on 2021-02-15 13:52:48:
Hi ! Indeed if you add a column to a formatted dataset, then the new dataset gets a new formatting in which:
```
new formatted columns = (all columns - previously unformatted columns)
```
Therefore the new column is going to be formatted using the `torch` formatting.

If you want your new column to be unformatted you can re-run this line:
```python
data.set_format("torch", columns=["some_integer_column1", "some_integer_column2"], output_all_columns=True)
```