t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-08-20 17:18:17:
Ok this is now ready for review ! Thanks for your awesome work in this @eltoto1219 

Summary of the changes:
- added new feature type `Array2D`, that can be instantiated like `Array2D("float32")` for example
- added pyarrow extension type `Array2DExtensionType` and array `Array2DExtensionArray` that take care of converting from and to arrow. `Array2DExtensionType`'s storage is a list of list of any pyarrow array.
- added pandas extension type `PandasArrayExtensionType` and array `PandasArrayExtensionArray` for conversion from and to arrow/python objects
- refactor of the `ArrowWriter` write and write_batch functions to support extension types while preserving the type inference behavior.
- added a utility object `TypedSequence` that is helpful to combine extension arrays and type inference inside the writer's methods.
- added speed test for sequences writing (printed as warnings in pytest)
- breaking: set disable_nullable to False by default as pyarrow's type inference returns nullable fields

And there are plenty of new tests, mainly in `test_array2d.py` and `test_arrow_writer.py`.

Note that there are some collisions in `arrow_dataset.py` with #513 so let's be careful when we'll merge this one.

I know this is a big PR so feel free to ask questions
lhoestq (MEMBER) commented on 2020-08-21 15:40:18:
I took your comments into account and I added Array[3-5]D.
I changed the storage type to fixed lengths lists. I had to update the `to_numpy` function because of that. Indeed slicing a FixedLengthListArray returns a view a of the original array, while in the previous case slicing a ListArray copies the storage.
cosmeowpawlitan (CONTRIBUTOR) commented on 2020-08-23 07:57:17:
> This is probably a Windows issue, we need to specify the encoding when `load_dataset()` reads the original CSV file.
> Try to find the `open()` statement called by `load_dataset()` and add an `encoding='utf-8'` parameter.
> See issues #242 and #307

Sorry for not responding for about a month.
I have just found that it is necessary to change / add the environment variable as what was told in #242.
Everything works after I add the new environment variable and restart my PC.

I think the encoding issue for windows isn't limited to the open() function call specific to few dataset, but actually in the entire library, depends on the machine / os you use.
lhoestq (MEMBER) commented on 2020-08-24 10:29:49:
The slow test doesn't support dataset that require config parameters that don't have default values.

To improve that we can replace it by two tests:
- one test that loads the default config (it can simply be the first config of the config lists for example)
- one tests that iterate over all configs and load them all one by one

By using the configs inside the builder config lists, there is no need to instantiate new configs, so the missing parameter error doesn't happen.

Does that sound good to you ?
lhoestq (MEMBER) commented on 2020-08-24 10:33:22:
Thanks for creating the issue :)
The download link for wmt-en-de raw looks like a mirror. We should use that instead of the current url.
Is this mirror official ?

Also it looks like for `ro-en` it tried to download other languages. If we manage to only download the one that is asked it'd be cool

Also cc @patrickvonplaten
lhoestq (MEMBER) commented on 2020-08-24 15:22:06:
I think you're having this issue because you try to format strings as pytorch tensors, which is not possible.
Indeed by having "Text" in `columns=['Text','Label']`, you try to convert the text values to pytorch tensors.

Instead I recommend you to first tokenize your dataset using a tokenizer from transformers. For example

```python
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")

train_dataset.map(lambda x: tokenizer(x["Text"]), batched=True)
train_dataset.set_format("torch", column=["input_ids"])
```

Another way to fix your issue would be to not set the format to pytorch, and leave the dataset as it is by default. In that case, the strings are returned normally when you get examples from your dataloader. It means that you would have to tokenize the examples in the training loop (or using a data collator) though.

Let me know if you have other questions
marcotcr (NONE) commented on 2020-08-25 20:47:46:
> Also, a little out of my depth there, but would there be a way to have the default pip install checklist command not require mysql and mariadb to be installed? Feels like that might be a source of confusion for users.

I removed the pattern dependency, mysql is not a requirement anymore. I'm not sure where mariadb is coming from.
lhoestq (MEMBER) commented on 2020-08-26 08:42:00:
Do you want to convert a dataset script to the tfds format ?
If so, we currently have a comversion script nlp/commands/convert.py but it is a conversion script that goes from tfds to nlp.
I think it shouldn't be too hard to do the changes in reverse (at some manual adjustments).
If you manage to make it work in reverse, feel free to open a PR to share it with the community :)
go-inoue (NONE) commented on 2020-08-26 09:42:33:
Thanks! I tested on 328MB text data on `n1-standard-8 (8 vCPUs, 30 GB memory)`. The main script ran without any issue, but it seems to require a huge space in the drive.

As suggested, I ran the following script before running the pre-training command with `xla_spawn.py`.

```python
from nlp import load_dataset

file_path="your_file_name"
load_dataset("text", data_files=file_path, split="train")
```
This will create `text-train.arrow` under the default cache directory. Then, I run the script with `xla_spawn.py`. It will load data from the cached file. My understanding is that there's no other way but to do this two-step process with the current version (0.4) of `nlp`.

During another caching process that happens in the main script:

```
08/26/2020 09:19:51 - INFO - nlp.utils.info_utils -   All the checksums matched successfully for post processing resources
08/26/2020 09:19:53 - INFO - nlp.arrow_dataset -   Caching processed dataset at /home/*****/.cache/huggingface/datasets/text/default-b0932b2bdbb63283/0.0.0/447f2bcfa2a721a37bc8fdf23800eade1523cf07f7eada6fe661fe4d070d380d/cache-f90f341e5308a7469
8d872bcc88f9c0e.arrow
```

`nlp` generates a temporary file per core, each of which is three times larger than the original text data. If each process is actually writing on the disk, you will need a huge amount of space in your drive. (Maybe I'm missing something.)

```
-rw-r--r-- 1 ***** *****  674 Aug 26 09:19 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 09:19 LICENSE
-rw-r--r-- 1 ***** ***** 332M Aug 26 09:10 text-train.arrow
-rw------- 1 ***** ***** 940M Aug 26 09:31 tmp0k43sazw
-rw------- 1 ***** ***** 940M Aug 26 09:31 tmp7sxs9mj5
-rw------- 1 ***** ***** 939M Aug 26 09:31 tmpbbiqw2vp
-rw------- 1 ***** ***** 937M Aug 26 09:31 tmpjxb5ptyu
-rw------- 1 ***** ***** 933M Aug 26 09:31 tmpk3hkdh0e
-rw------- 1 ***** ***** 944M Aug 26 09:31 tmpnoalwftz
-rw------- 1 ***** ***** 931M Aug 26 09:31 tmpuxdr_dz3
-rw------- 1 ***** ***** 945M Aug 26 09:31 tmpxjyuy6dk
```
After the caching process, they seem to be merged into one file.

```
-rw------- 1  ***** ***** 989M Aug 26 09:32 cache-f90f341e5308a74698d872bcc88f9c0e.arrow
-rw-r--r-- 1  ***** *****  674 Aug 26 09:19 dataset_info.json
-rw-r--r-- 1  ***** *****    0 Aug 26 09:19 LICENSE
-rw-r--r-- 1  ***** ***** 332M Aug 26 09:10 text-train.arrow
```
lhoestq (MEMBER) commented on 2020-08-26 09:53:47:
Again it looks like every process tries to tokenize the full dataset at the same time.
If you do the tokenization before calling `xla_spawn.py` once, then each process will then use the tokenized cached file `cache-f90f341e5308a74698d872bcc88f9c0e.arrow` and not recompute it.

Not sure if there's a better way to do that cc @julien-c @thomwolf
lhoestq (MEMBER) commented on 2020-08-26 12:36:09:
It should be good now. I was able to run

```python
>>> from nlp import concatenate_datasets, load_dataset
>>>
>>> bookcorpus = load_dataset("bookcorpus", split="train")
>>> wiki = load_dataset("wikipedia", "20200501.en", split="train")
>>> wiki.remove_columns_("title")  # only keep the text
>>>
>>> assert bookcorpus.features.type == wiki.features.type
>>> bert_dataset = concatenate_datasets([bookcorpus, wiki])
```
go-inoue (NONE) commented on 2020-08-26 14:14:31:
I wrote a separate script just for preparing a cached file, including tokenization. Each process did use the tokenized cached file.

Currently I'm testing the pipeline on 24GB text data. It took about 1.5 hour to create a cached file on `n1-highmem-16 (16 vCPUs, 104 GB memory)`. I assume loading this cached file in the main script with `xla_spawn.py` won't be an issue (even if there are 8 processes).

```
total 98G
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 13:38 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 12:24 ..
-rw------- 1 ***** *****  74G Aug 26 13:38 cache-a7aa04134ba7b1aff5d9710f14a4e334.arrow
-rw-r--r-- 1 ***** *****  681 Aug 26 12:24 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 12:24 LICENSE
-rw-r--r-- 1 ***** *****  25G Aug 26 12:24 text-train.arrow
```
go-inoue (NONE) commented on 2020-08-26 15:43:30:
Sorry, I thought it was working, but actually the second call doesn't use the cached file that was generated separately, and it will generate another cache-****.arrorw file with a different name. If I run the training script again (with `xla_spawn.py`), it will use the second cached file, which was generated by the training script itself in the previous run.

```
drwxr-xr-x 2 ***** ***** 4.0K Aug 26 15:35 .
drwxr-xr-x 3 ***** ***** 4.0K Aug 26 15:29 ..
-rw------- 1 ***** *****  99M Aug 26 15:35 cache-0d77dfce704493dbe63f071eed6a5431.arrow
-rw------- 1 ***** *****  99M Aug 26 15:29 cache-69633651476e943b93c89ace715f9487.arrow
-rw-r--r-- 1 ***** *****  670 Aug 26 15:33 dataset_info.json
-rw-r--r-- 1 ***** *****    0 Aug 26 15:33 LICENSE
-rw-r--r-- 1 ***** *****  33M Aug 26 15:29 text-train.arrow
```
go-inoue (NONE) commented on 2020-08-26 15:50:11:
Yes.

1. `cache-69633651476e943b93c89ace715f9487.arrow` was generated with a separate script. 
2. I ran the entire script with `xla_spawn.py`.
3. `cache-69633651476e943b93c89ace715f9487.arrow` is not used.
4. `cache-0d77dfce704493dbe63f071eed6a5431.arrow` is created.
5. training starts...

Now, if I kill the process at step 5, and do the step 2 again, it will use `cache-0d77dfce704493dbe63f071eed6a5431.arrow` (cached file created at step 4) without any issue.

I used the following to generate the first cached file.
```python
dataset = load_dataset("text", data_files=file_path, split="train")
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                        truncation=True, max_length=args.block_size), batched=True)
dataset.set_format(type='torch', columns=['input_ids'])
```
go-inoue (NONE) commented on 2020-08-27 08:32:17:
I trained a byte-level BPE tokenizer on my data with `tokenziers` library following this [example](https://github.com/huggingface/tokenizers/blob/master/bindings/python/examples/train_bytelevel_bpe.py).

And I put these model files in a directory named `"model_name"`. I also put config.json, which is the original RoBERTa config file.

```bash
%ls  model_name
config.json     merges.txt      vocab.json
```

[This](https://github.com/huggingface/transformers/blob/4bd7be9a4268221d2a0000c7e8033aaeb365c03b/examples/language-modeling/run_language_modeling.py#L196) is the line where `run_language_modeling.py` loads the tokenier.

```python
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
```

I use `"model_name"` for `model_args.tokenizer_name`. I don't specify `model_args.cache_dir`. It is 'None' by default.
thomwolf (MEMBER) commented on 2020-08-27 08:35:16:
Should we add an argument in `load_dataset` to override some URL with a custom URL (e.g. mirror) or a local path?

This could also be used to provide local files instead of the original files as requested by some users (e.g. when you made a dataset with the same format than SQuAD and what to use it instead of the official dataset files).
go-inoue (NONE) commented on 2020-08-27 08:58:36:
It still creates a new file even if I remove `use_fast=True`... 

Here's the script used to create a cached file.
```python 
#!/usr/bin/env python3

import argparse

from transformers import AutoTokenizer

from nlp import load_dataset


def main():
    parser = argparse.ArgumentParser(description='description')
    parser.add_argument('--config_name', type=str, help='Pretrained config name or path if not the same as model_name')
    parser.add_argument('--data_file', type=str, help='The input data file (a text file).')
    parser.add_argument('--block_size', type=int, default=-1, help='The training dataset will be truncated in block of this size for training')
    args = parser.parse_args()

    tokenizer = AutoTokenizer.from_pretrained(args.config_name)

    dataset = load_dataset("text", data_files=args.data_file, split="train")
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                truncation=True, max_length=args.block_size), batched=True)
    dataset.set_format(type='torch', columns=['input_ids'])


if __name__ == "__main__":
    main()
```

Here's how the data is loaded in the modified `run_language_modeling.py`. [[original function](https://github.com/huggingface/transformers/blob/971d1802d009d9996b36a34a34477cee849ef39f/examples/language-modeling/run_language_modeling.py#L128-L135)]

```python
def get_dataset(args: DataTrainingArguments, tokenizer: PreTrainedTokenizer, evaluate=False):
    file_path = args.eval_data_file if evaluate else args.train_data_file
    split = "validation" if evaluate else "train"
    if args.line_by_line:
        # return LineByLineTextDataset(tokenizer=tokenizer, file_path=file_path, block_size=args.block_size)
        dataset = load_dataset("text", data_files=file_path, split="train")
        dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                              truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
        return dataset

    else:
        return TextDataset(
            tokenizer=tokenizer, file_path=file_path, block_size=args.block_size, overwrite_cache=args.overwrite_cache
        )
```

Probably I don't need this part in the main script,

```python
dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                              truncation=True, max_length=args.block_size), batched=True)
        dataset.set_format(type='torch', columns=['input_ids'])
```
and simply do this?
```python
dataset = load_dataset("text", data_files=file_path, split="train")
return dataset
```
lhoestq (MEMBER) commented on 2020-08-27 09:11:11:
I can see that the tokenizer in `run_language_modeling.py` is not instantiated the same way as in your separated script.
Indeed we can see L196:
```python
tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)
```
Could you try to make it so they are instantiated the exact same way please ?
go-inoue (NONE) commented on 2020-08-27 09:32:29:
I updated my separated script, but it's creating a cached file again. If I don't use the `model_args.cache_dir`, both will get `None`, so they should be the same.

```python
#!/usr/bin/env python3
import argparse

from transformers import AutoTokenizer
from nlp import load_dataset

def main():
    parser = argparse.ArgumentParser(description='description')
    parser.add_argument('--tokenizer_name', type=str, help='Pretrained tokenizer name or path if not the same as model_name')
    parser.add_argument('--data_file', type=str, help='The input data file (a text file).')
    parser.add_argument('--cache_dir', type=str, default=None, help='Where do you want to store the pretrained models downloaded from s3')
    parser.add_argument('--block_size', type=int, default=-1, help='The training dataset will be truncated in block of this size for training')

    model_args = parser.parse_args()

    tokenizer = AutoTokenizer.from_pretrained(model_args.tokenizer_name, cache_dir=model_args.cache_dir)

    dataset = load_dataset("text", data_files=model_args.data_file, split="train")
    dataset = dataset.map(lambda ex: tokenizer(ex["text"], add_special_tokens=True,
                                truncation=True, max_length=model_args.block_size), batched=True)
    dataset.set_format(type='torch', columns=['input_ids'])

if __name__ == "__main__":
    main()
```

Is there a way to specify the cache file to load, and skip the re-computation?
thomwolf (MEMBER) commented on 2020-08-27 22:33:57:
Warning from pytorch that we should maybe consider at some point @lhoestq:
```
/__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
and PyTorch does not support non-writeable tensors. This means you can write to the underlying
(supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
protect its data or make it writeable before converting it to a tensor. This type of warning will be
suppressed for the rest of this program.
(Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
532
  return torch.tensor(x, **format_kwargs)
```
lhoestq (MEMBER) commented on 2020-08-28 07:58:07:
> Warning from pytorch that we should maybe consider at some point @lhoestq:
> 
> ```
> /__w/nlp/nlp/src/nlp/arrow_dataset.py:648: UserWarning: The given NumPy array is not writeable,
> and PyTorch does not support non-writeable tensors. This means you can write to the underlying
> (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to
> protect its data or make it writeable before converting it to a tensor. This type of warning will be
> suppressed for the rest of this program.
> (Triggered internally at  /pytorch/torch/csrc/utils/tensor_numpy.cpp:141.)
> 532
>   return torch.tensor(x, **format_kwargs)
> ```

Not sure why we have that, it's probably linked to zero copy from arrow to numpy
moskomule (NONE) commented on 2020-08-28 14:35:45:
Hi, I got another error (on Colab):

```python
# You can read a few attributes of the datasets before loading them (they are python dataclasses)
from dataclasses import asdict

for key, value in asdict(datasets[6]).items():
    print('👉 ' + key + ': ' + str(value))

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-6-b8ace6c227a2> in <module>()
      2 from dataclasses import asdict
      3 
----> 4 for key, value in asdict(datasets[6]).items():
      5     print('👉 ' + key + ': ' + str(value))

/usr/local/lib/python3.6/dist-packages/dataclasses.py in asdict(obj, dict_factory)
   1008     """
   1009     if not _is_dataclass_instance(obj):
-> 1010         raise TypeError("asdict() should be called on dataclass instances")
   1011     return _asdict_inner(obj, dict_factory)
   1012 

TypeError: asdict() should be called on dataclass instances
```
lhoestq (MEMBER) commented on 2020-08-31 10:19:07:
I changed the way I implemented fingerprint updates to use decorator functions.

I also added a new attribute called `_inplace_history` that stores the in-place history of transforms (like cast_, rename_columns, etc.). This history is useful to replay the changes that were done in-place when unpickling a dataset that is memory mapped from a file.

Let me know what you think @thomwolf
lhoestq (MEMBER) commented on 2020-08-31 10:51:09:
`NonMatchingChecksumError` means that the checksum of the downloaded file is not the expected one.
Either the file you downloaded was corrupted along the way, or the host updated the file.
Could you try to clear your cache and run `load_dataset` again ? If the error is still there, it means that there was an update in the data, and we may have to update the expected checksum value.
thomwolf (MEMBER) commented on 2020-08-31 12:58:58:
Hi @gaguilar 

If you want to take care of this, it very simple, you just need to regenerate the `dataset_infos.json` file as indicated [in the doc](https://huggingface.co/nlp/share_dataset.html#adding-metadata) by [installing from source](https://huggingface.co/nlp/installation.html#installing-from-source) and running the following command from the root of the repo:
```bash
python nlp-cli test ./datasets/lince --save_infos --all_configs
```
And then you can open a pull-request with the updated json file.

Otherwise we'll do it sometime this week.
lhoestq (MEMBER) commented on 2020-08-31 13:04:10:
When you load a text file for the first time with `nlp`, the file is converted into Apache Arrow format. Arrow allows to use memory-mapping, which means that you can load an arbitrary large dataset.

Note that as soon as the conversion has been done once, the next time you'll load the dataset it will be much faster.

However for a 1TB dataset, the conversion can indeed take time. You could try to load parts of it in parallel, and then use `nlp.concatenate_datasets` to get your full dataset.
agemagician (NONE) commented on 2020-08-31 13:57:16:
@lhoestq Yes, I understand that the first time requires more time. The concatenate_datasets seems to be a workaround, but I believe a multi-processing method should be integrated into load_dataset to make it easier and more efficient for users.

@thomwolf Sure, here are the statistics:
Number of lines: 4.2 Billion
Number of files: 6K
Number of tokens: 800 Billion
The number of lines is distributed equally across these 6k files.
The line length varies between 100 tokens to 40k tokens.
lhoestq (MEMBER) commented on 2020-08-31 15:40:54:
Do you manage to run a dummy beam pipeline with python on windows ? 
You can test a dummy pipeline with [this code](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/examples/wordcount_minimal.py)

If you get the same error, it means that the issue comes from apache beam.
Otherwise we'll investigate what went wrong here
lhoestq (MEMBER) commented on 2020-08-31 15:52:02:
Yes indeed it looks like some `'` and spaces are missing (for example in `dont` or `didnt`).
Do you know if there exist some copies without this issue ?
How would you fix this issue on the current data exactly ? I can see that the data is raw text (not tokenized) so I'm not sure I understand how you would do it. Could you provide more details ?
lhoestq (MEMBER) commented on 2020-08-31 18:15:05:
Awesome !
Also I was wondering if we should try to make the hashing of the `data_files` faster (it is used to build the cache directory of datasets like `text` or `json`). Right now it reads each file and hashes all of its data. We could simply hash the path and some metadata including the `time last modified` tag no ? Apparently we can get this tag with `os.path.getmtime(path)`