t20062443/AT1ds huggingface dataset:
shexuan (NONE) commented on 2020-11-10 06:11:57:



> > Hi !
> > The url works on my side.
> > Is the url working in your navigator ?
> > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> 
> I tried another server, it's working now. Thanks a lot.
> 
> And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?

I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？ 

Thanks :D
lhoestq (MEMBER) commented on 2020-11-10 09:23:07:
Hi ! I'm not sure we should make this split decision arbitrarily on our side. Users can split it afterwards to whatever they want using `dataset.train_test_split` for example.
Moreover it looks like there's already papers that use this dataset and propose their own splits ([here](http://xanthippi.ceid.upatras.gr/HealthSign/resources/Publications/sitis_paper_25_10.pdf) 80-20) 
What do you think ?
rabeehkarimimahabadi (NONE) commented on 2020-11-10 10:20:34:
Hi there, 
Here is what I am trying, this is not working for me in map-style datasets, could you please tell me how to use datasets with being able to access ___getitem__ ? could you assist me please correcting this example? I need map-style datasets which is formed from concatenation of two datasets from your library. thanks 


```
import datasets
dataset1 = load_dataset("squad", split="train[:10]")
dataset1 = dataset1.map(lambda example: {"src_texts": "question: {0} context: {1} ".format(
    example["question"], example["context"]),
    "tgt_texts": example["answers"]["text"][0]}, remove_columns=dataset1.column_names)
dataset2 = load_dataset("imdb", split="train[:10]")
dataset2 = dataset2.map(lambda example: {"src_texts": "imdb: " + example["text"],
            "tgt_texts": str(example["label"])}, remove_columns=dataset2.column_names)
train_dataset = datasets.concatenate_datasets([dataset1, dataset2])
train_dataset.set_format(type='torch', columns=['src_texts', 'tgt_texts'])
dataloader = torch.utils.data.DataLoader(train_dataset, batch_size=32)
for id, batch in enumerate(dataloader):
    print(batch)

```
rabeehkarimimahabadi (NONE) commented on 2020-11-10 12:55:25:
Hi Thomas,
what I do not get from documentation is that why when you set batched=True,
this is processed in batch, while data is not divided to batched
beforehand, basically this is a question on the documentation and I do not
get the batched=True, but sure, if you think this is more appropriate in
forum I will post it there.
thanks
Best
Rabeeh

On Tue, Nov 10, 2020 at 12:21 PM Thomas Wolf <notifications@github.com>
wrote:

> Hi I don’t think this is a request for a dataset like you labeled it.
>
> I also think this would be better suited for the forum at
> https://discuss.huggingface.co. we try to keep the issue for the repo for
> bug reports and new features/dataset requests and have usage questions
> discussed on the forum. Thanks.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/823#issuecomment-724639476>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ARPXHH4FIPFHVVUHANAE4F3SPEO2JANCNFSM4TQQVEXQ>
> .
>
yjernite (MEMBER) commented on 2020-11-10 17:31:06:
Hi @bminixhofer ! Parsing WikiMedia is notoriously difficult: this processing used [mwparserfromhell](https://github.com/earwig/mwparserfromhell) which is pretty good but not perfect.

As an alternative, you can also use the Wiki40b dataset which was pre-processed using an un-released Google internal tool
thomwolf (MEMBER) commented on 2020-11-11 17:37:34:
We should expose the [`block_size` argument](https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions) of Apache Arrow csv `ReadOptions` in the [script](https://github.com/huggingface/datasets/blob/master/datasets/csv/csv.py).


In the meantime you can specify yourself the `ReadOptions` config like this:
```python
import pyarrow.csv as pac   # PyArrow is installed with `datasets`

read_options = pac.ReadOptions(block_size=1e9)  # try to find the right value for your use-case
dataset = load_dataset('csv', data_files=files, read_options=read_options)
```
randubin (NONE) commented on 2020-11-11 19:43:12:
This did help to load the data. But the problem now is that I get:
ArrowInvalid: CSV parse error: Expected 5 columns, got 187

It seems that this change the parsing so I changed the table to tab-separated and tried to load it directly from pyarrow
But I got a similar error, again it loaded fine in pandas so I am not sure what to do.
phtephanx (NONE) commented on 2020-11-12 20:56:52:
I might have a related issue:

My custom arrow file created by `load_dataset` is around 260 GB large. The corresponding dataset is wrapped in a `torch.utils.data.Dataset`.

When spawning ``8`` processes with `xla_multiprocessing` but ``0`` `num_workers` for the `DataLoader`, the memory peaks at around 300 GB RAM in the beginning and then levels off at 250 GB RAM. Since the number of workers for the data loading is `0`, there shouldn't be prefetched batches AFAIK. 

When looping over the same dataset without the `DataLoader` in a single process and fetching the same batch size, less than 1 GB of RAM is requested (certainly due to some additional tensor creation in `__getitem__`).

_Is this expected?_
lhoestq (MEMBER) commented on 2020-11-12 23:09:52:
The library does not load the dataset in memory. Not sure why your script using xla_multiprocessing seems to load everything unfortunately :/
Pytorch DataLoader use `__getitem__` to iterate through the dataset batch by batch without loading everything indeed.

Do you think you can share a script that reproduces the issue with a smaller dataset (a few GB) to experiment/debug with ?
lhoestq (MEMBER) commented on 2020-11-12 23:28:27:
Right now multiprocessing only runs on single node.

However it's probably possible to extend it to support multi nodes. Indeed we're using the `multiprocess` library from the `pathos` project to do multiprocessing in `datasets`, and `pathos` is made to support parallelism on several nodes. More info about pathos [on the pathos repo](https://github.com/uqfoundation/pathos).

If you're familiar with pathos or if you want to give it a try, it could be a nice addition to the library :)
lhoestq (MEMBER) commented on 2020-11-13 14:36:17:
As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.

The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.

Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.
shexuan (NONE) commented on 2020-11-13 15:55:10:
> As you can see in the error the test checks if `indices_mappings_in_memory` is True or not, which is different from the test you do in your script. In a dataset, both the data and the indices mapping can be either on disk or in memory.
> 
> The indices mapping correspond to a mapping on top of the data table that is used to re-order/select a sample of the original data table. For example if you do `dataset.train_test_split`, then the resulting train and test datasets will have both an indices mapping to tell which examples are in train and which ones in test.
> 
> Before saving your datasets on disk, you should call `dataset.flatten_indices()` to remove the indices mapping. It should fix your issue. Under the hood it will create a new data table using the indices mapping. The new data table is going to be a subset of the old one (for example taking only the test set examples), and since the indices mapping will be gone you'll be able to concatenate your datasets.

`dataset.flatten_indices()` solved my problem, thanks so much!
phtephanx (NONE) commented on 2020-11-14 13:01:33:
I minimalistically put together the relevant steps in this [colab](https://colab.research.google.com/drive/1LclO69NVPBOO4hzWtW8uIjaTgVCR-MDv?usp=sharing). Maybe you spot a problem? I'm really wondering what causes this huge memory consumption. The `DataLoader` just produces tensors of shape `(batch_size, seq_len) = (128, 128)` which accounts for < 1 GB. Apart from the dataset, the only large component is the model which consumes around 12 GB per process but of the devices' (TPUs') memory.

Multiprocess-training with TPU leads to another problem: `out of space in memory space smem` (https://github.com/pytorch/xla/issues/2628). I'm not sure, though, whether this is related.
wuaibo (NONE) commented on 2020-11-16 08:43:38:
hello, how did you solve this problems?

> > > Hi !
> > > The url works on my side.
> > > Is the url working in your navigator ?
> > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > 
> > 
> > I tried another server, it's working now. Thanks a lot.
> > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> 
> I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> 
> Thanks :D

hello, I tried this. but it still failed. how do you fix this error?
lhoestq (MEMBER) commented on 2020-11-16 10:00:00:
Unfortunately `concatenate_datasets` only supports concatenating the rows, while what you want to achieve is concatenate the columns.
Currently to add more columns to a dataset, one must use `map`.
What you can do is somehting like this:
```python
# suppose you have datasets d1, d2, d3
def add_columns(example, index):
    example.update(d2[index])
    example.update(d3[index])
    return example

full_dataset = d1.map(add_columns, with_indices=True)
```
rabeehk (CONTRIBUTOR) commented on 2020-11-16 10:12:19:
Hi
thanks, but I did tried to install from the pip install git+... and it does
not work for me,. thanks for the help. I have the same issue with wmt16,
"ro-en"
thanks.
Best
Rabeeh

On Mon, Nov 16, 2020 at 10:29 AM Quentin Lhoest <notifications@github.com>
wrote:

> Actually it's already fixed on the master branch since #740
> <https://github.com/huggingface/datasets/pull/740>
> I'll do the 1.1.3 release soon
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/datasets/issues/798#issuecomment-727854736>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/ABP4ZCEUBJKPOCLABXCKMPDSQDWH3ANCNFSM4TJBUKSA>
> .
>
lhoestq (MEMBER) commented on 2020-11-16 10:18:25:
I just tested on google colab using
```python
!pip install git+https://github.com/huggingface/datasets.git
from datasets import load_dataset
load_dataset("trec")
```
and it works.
Can you detail how you got the issue even when using the latest version on master ?

Also about wmt we'll look into it, thanks for reporting !
lhoestq (MEMBER) commented on 2020-11-16 10:52:45:
It looks like the official OPUS server for WMT16 doesn't provide the data files anymore (503 error).
I searched a bit and couldn't find a mirror except maybe http://nlp.ffzg.hr/resources/corpora/setimes/ (the data are a cleaned version of the original ones though)
Should we consider replacing the old urls with these ones even though it's not the exact same data ?
vblagoje (CONTRIBUTOR) commented on 2020-11-16 18:04:15:
@lhoestq I fixed issues except for the dummy_data zip file. But I think I know why is it happening. So when unzipping dummy_data.zip it gets save in /tmp directory where glob doesn't pick it up. For regular downloads, the archive gets unzipped in ~/.cache/huggingface. Could that be a reason?
vblagoje (CONTRIBUTOR) commented on 2020-11-16 18:56:44:
@lhoestq I understand but for some reason, it is not happening. I added logs to see where dummy_data.zip gets unzipped in /tmp but I suppose when the test process finishes that tmp is gone. I also tried to glob anything in _generate_examples from that directory using /* instead of **/*.epub.txt and nothing is being returned. Always an empty array.
shexuan (NONE) commented on 2020-11-17 06:08:02:
> hello, how did you solve this problems?
> 
> > > > Hi !
> > > > The url works on my side.
> > > > Is the url working in your navigator ?
> > > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > > 
> > > 
> > > I tried another server, it's working now. Thanks a lot.
> > > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> > 
> > 
> > I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> > Thanks :D
> 
> hello, I tried this. but it still failed. how do you fix this error?

你把那个脚本下载到你本地安装目录下，然后 `load_dataset(csv_script_path, data_fiels)`
wuaibo (NONE) commented on 2020-11-17 06:49:32:
> > hello, how did you solve this problems?
> > > > > Hi !
> > > > > The url works on my side.
> > > > > Is the url working in your navigator ?
> > > > > Are you connected to internet ? Does your network block access to `raw.githubusercontent.com` ?
> > > > 
> > > > 
> > > > I tried another server, it's working now. Thanks a lot.
> > > > And I'm curious about why download things from "github" when I load dataset from local files ? Dose datasets work if my network crashed?
> > > 
> > > 
> > > I download the scripts `https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/csv/csv.py` and move it to the package dir `*/datasets/` solved the problem. Could you please put the file `datasets/datasets/csv/csv.py` to `datasets/src/datasets/`？
> > > Thanks :D
> > 
> > 
> > hello, I tried this. but it still failed. how do you fix this error?
> 
> 你把那个脚本下载到你本地安装目录下，然后 `load_dataset(csv_script_path, data_fiels)`

好的好的！解决了，感谢感谢！！！
vblagoje (CONTRIBUTOR) commented on 2020-11-17 10:18:06:
In _generate_examples_ I wrote the following:
```
glob_target = os.path.join(directory, "**/*.epub.txt")
print(f"Glob target {glob_target    }")
```

And here is the test failure:


========================================================================================== FAILURES ===========================================================================================
________________________________________________________________ LocalDatasetTest.test_load_dataset_all_configs_bookcorpusopen ________________________________________________________________

self = <tests.test_dataset_common.LocalDatasetTest testMethod=test_load_dataset_all_configs_bookcorpusopen>, dataset_name = 'bookcorpusopen'

    @slow
    def test_load_dataset_all_configs(self, dataset_name):
        configs = self.dataset_tester.load_all_configs(dataset_name, is_local=True)
>       self.dataset_tester.check_load_dataset(dataset_name, configs, is_local=True)

tests/test_dataset_common.py:232: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_dataset_common.py:193: in check_load_dataset
    self.parent.assertTrue(len(dataset[split]) > 0)
E   AssertionError: False is not true
------------------------------------------------------------------------------------ Captured stdout call -------------------------------------------------------------------------------------
Downloading and preparing dataset book_corpus_open/plain_text (download: 1.00 MiB, generated: 1.00 MiB, post-processed: Unknown size, total: 2.00 MiB) to /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpmuu0_ln2/book_corpus_open/plain_text/1.0.0...
Glob target /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpm6tpvb3f/extracted/d953b414cceb4fe3985eeaf68aec2f4435f166b2edf66863d805e3825b7d336b/dummy_data/**/*.epub.txt
Dataset book_corpus_open downloaded and prepared to /var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpmuu0_ln2/book_corpus_open/plain_text/1.0.0. Subsequent calls will reuse this data.
------------------------------------------------------------------------------------ Captured stderr call -------------------------------------------------------------------------------------
vblagoje (CONTRIBUTOR) commented on 2020-11-17 10:22:32:
And when I do os.listdir on the given directory I get:

        glob_target = os.path.join(directory, "**/*.epub.txt")
        print(f"Glob target {glob_target    }")
>       print(os.listdir(path=directory))
E       FileNotFoundError: [Errno 2] No such file or directory: '/var/folders/y_/6k6zhblx0k9dsdz5nd_z9x5c0000gp/T/tmpbu_aom5q/extracted/d953b414cceb4fe3985eeaf68aec2f4435f166b2edf66863d805e3825b7d336b/dummy_data'
AmitMY (CONTRIBUTOR) commented on 2020-11-17 10:25:17:
I added it, but still it consumes as much memory

https://github.com/huggingface/datasets/pull/722/files#diff-2e0d865dd4a60dedd1861d6f8c5ed281ded71508467908e1e0b1dbe7d2d420b1R66

Did I not do it correctly?
lhoestq (MEMBER) commented on 2020-11-17 13:40:21:
Ok found the issue !

The dummy_data.zip file must be an archive of a folder named dummy_data. Currently the dummy_data.zip is an archive of a folder named book1. In order to have a valid dummy_data.zip file you must first take the dummy book1 folder, place it inside a folder named dummy_data and then compress the dummy_data folder to get dummy_data.zip
lhoestq (MEMBER) commented on 2020-11-17 14:10:58:
The preprocessing tokenizes the input text. Tokenization outputs `input_ids`, `attention_mask`, `token_type_ids` and `special_tokens_mask`. All those are of length`max_seq_length` because of padding. Therefore for each sample it generate 4 *`max_seq_length` integers. Currently they're all saved as int64. This is why the tokenization takes so much space.

I'm sure we can optimize that though
What do you think @sgugger ?
lhoestq (MEMBER) commented on 2020-11-17 15:16:04:
First I think we should disable padding in the dataset processing and let the data collator do it.

Then I'm wondering if you need attention_mask and token_type_ids at this point ?

Finally we can also specify the output feature types at this line https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py#L280 to use more optimized integer precisions for the output. Maybe something like:
- input_ids: uint16 or uint32
- token_type_ids: uint8 or bool
- attention_mask: bool
- special_tokens_mask: bool

Also IMO these changes are all on the `transformers` side. Maybe we should discuss on the `transformers` repo
sgugger (CONTRIBUTOR) commented on 2020-11-17 15:38:15:
> First I think we should disable padding in the dataset processing and let the data collator do it.

No, you can't do that on TPUs as dynamic shapes will result in a very slow training. The script can however be tweaked to use the `PaddingDataCollator` with a fixed max length instead of dynamic batching.

For the other optimizations, they can be done by changing the script directly for each user's use case. Not sure we can find something that is general enough to be in transformers or the examples script.
shawwn (NONE) commented on 2020-11-17 15:59:26:
I just wanted to say thank you to everyone involved in making this happen! I was certain that I would have to add bookcorpusnew myself, but then @vblagoje came along and did it, and @lhoestq gave some great support in a timely fashion.

By the way @vblagoje, are you on Twitter? I'm https://twitter.com/theshawwn if you'd like to DM and say hello. Once again, thanks for doing this!

I'll mention over at https://github.com/soskek/bookcorpus/issues/27 that this was merged.
shawwn (NONE) commented on 2020-11-17 16:52:49:
@vblagoje Would you be interested in adding books3 as well? https://twitter.com/theshawwn/status/1320282149329784833

Huggingface is interested and asked me to add it, but I had a bit of trouble during setup (https://github.com/huggingface/datasets/issues/790) and never got around to it. At this point you have much more experience than I do with the datasets lib.

It *seems* like it might simply be a matter of copy-pasting this PR, changing books1 to books3, and possibly trimming off the leading paths -- each book is at e.g. the-eye/Books/Bibliotok/J/Jurassic Park.epub.txt, which is rather lengthy compared to just the filename -- but the full path is probably fine, so feel free to do the least amount of work that gets the job done. Otherwise I suppose I'll get around to it eventually; thanks again!
vblagoje (CONTRIBUTOR) commented on 2020-11-18 12:03:46:
@shawwn I'll take a look as soon as I clear my work queue. TBH, I would likely work on making sure HF datasets has all the datasets used to train https://github.com/alexa/bort/ and these are: Wikipedia, Wiktionary, OpenWebText (Gokaslan and Cohen, 2019), UrbanDictionary, Onel Billion Words (Chelba et al., 2014), the news subset of Common Crawl (Nagel, 2016)10, and Bookcorpus. cc @lhoestq
yjernite (MEMBER) commented on 2020-11-19 16:35:21:
@lhoestq agreed on having one config per language, but we also need to be able to have different split names and people are going to want to use hyphens, so we should at the very least warn them why it's failing :) E.g. for ANLI with different stages of data (currently using underscores) or https://www.tau-nlp.org/commonsenseqa with their train-sanity or dev-sanity splits
lhoestq (MEMBER) commented on 2020-11-19 17:30:41:
I couldn't reproduce unfortunately. I tried
```python
from datasets import load_dataset

load_dataset("cnn_dailymail", "3.0.0", download_mode="force_redownload")
```
and it worked fine on both my env (python 3.7.2) and colab (python 3.6.9)

Maybe there was an issue with the google drive download link of the dataset ?
Are you still having the issue ? If so could your give me more info about your python and requests version ?
thomwolf (MEMBER) commented on 2020-11-20 08:05:27:
I think the only way to avoid this warning would be to do a copy of the numpy array before providing it.

This would slow down a bit the iteration over the dataset but maybe it would be safer. We could disable the copy with a flag on the `set_format` command.

In most typical cases of training a NLP model, PyTorch shouldn't modify the input so it's ok to have a non-writable array but I can understand the warning is a bit scary so maybe we could choose the side of non-warning/slower by default and have an option to speedup.

What do you think @lhoestq ?
phtephanx (NONE) commented on 2020-11-22 10:11:16:
I spotted the problem: it's the sampler's list of indices (e.g. [RandomSampler](https://github.com/pytorch/pytorch/blob/master/torch/utils/data/sampler.py#L125)) which consumes lots of memory.

Try out and watch your memory, `indices` takes about 3.8 G RAM. Compare it to e.g. `n=int(1e7)`.
```python
indices = torch.randperm(int(1e8), generator=None).tolist()
```
Not converting to a list is much more [memory-efficient](https://github.com/pytorch/pytorch/issues/45427). 

This might be interesting for the TO @chiyuzhang94.