t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-09-28 17:14:49:
Under the hood it does
```python
import pyarrow as pa
import pyarrow.csv

# Use csv reader from Pyarrow with one column for text files

# To force the one-column setting, we set an arbitrary character
# that is not in text files as delimiter, such as \b or \v.
# The bell character, \b, was used to make beeps back in the days
parse_options = pa.csv.ParseOptions( 
    delimiter="\b", 
    quote_char=False, 
    double_quote=False, 
    escape_char=False, 
    newlines_in_values=False, 
    ignore_empty_lines=False, 
)

read_options= pa.csv.ReadOptions(use_threads=True, column_names=["text"])

pa_table = pa.csv.read_csv("all_train.txt", read_options=read_options, parse_options=parse_options)
```

Note that we changed the parse options with datasets 1.0
In particular the delimiter used to be `\r` but this delimiter doesn't work on windows.
BramVanroy (CONTRIBUTOR) commented on 2020-09-29 10:09:32:
@lhoestq Can you ever be certain that a delimiter character is not present in a plain text file? In other formats (e.g. CSV) , rules are set of what is allowed and what isn't so that it actually constitutes a CSV file. In a text file you basically have "anything goes", so I don't think you can ever be entirely sure that the chosen delimiter does not exist in the text file, or am I wrong? 

If I understand correctly you choose a delimiter that we hope does not exist in the file, so that when the CSV parser starts splitting into columns, it will only ever create one column? Why can't we use a newline character though?
lhoestq (MEMBER) commented on 2020-09-29 10:12:45:
So we're using the csv reader to read text files because arrow doesn't have a text reader.
To workaround the fact that text files are just csv with one column, we want to set a delimiter that doesn't appear in text files.
Until now I thought that it would do the job but unfortunately it looks like even characters like \a appear in text files.

So we have to option:
- find another delimiter that does the job (maybe `\x1b` esc or `\x18` cancel)
- don't use the csv reader from arrow but the text reader from pandas instead (or any other reader). The only important thing is that it must be fast (arrow's reader has a nice and fast multithreaded for csv that we're using now but hopefully we can find an alternative)



> @lhoestq Can you ever be certain that a delimiter character is not present in a plain text file? In other formats (e.g. CSV) , rules are set of what is allowed and what isn't so that it actually constitutes a CSV file. In a text file you basically have "anything goes", so I don't think you can ever be entirely sure that the chosen delimiter does not exist in the text file, or am I wrong?

As long as the text file follows some encoding it wouldn't make sense to have characters such as the bell character. However I agree it can happen.

> If I understand correctly you choose a delimiter that we hope does not exist in the file, so that when the CSV parser starts splitting into columns, it will only ever create one column? Why can't we use a newline character though?

Exactly. Arrow doesn't allow the newline character unfortunately.
lhoestq (MEMBER) commented on 2020-09-29 10:22:37:
> Okay, I have splitted the crashing shards into individual sentences and some examples of the inputs that are causing the crashes are the following ones

Thanks for digging into it !

Characters like \a or \b are not shown when printing the text, so as it is I can't tell if it contains unexpected characters.
Maybe could could open the file in python and check if `"\b" in open("path/to/file", "r").read()` ?

> I think that the main difference between the crashing samples and the rest is their length. Therefore, couldn't the length be causing the message errors? I hope with these samples you can identify what is causing the crashes considering that the 0.4.0 nlp library was loading them properly.

To check that you could try to run 

```python
import pyarrow as pa
import pyarrow.csv

open("dummy.txt", "w").write((("a" * 10_000) + "\n") * 4)  # 4 lines of 10 000 'a'

parse_options = pa.csv.ParseOptions( 
    delimiter="\b", 
    quote_char=False, 
    double_quote=False, 
    escape_char=False, 
    newlines_in_values=False, 
    ignore_empty_lines=False, 
)

read_options= pa.csv.ReadOptions(use_threads=True, column_names=["text"])

pa_table = pa.csv.read_csv("dummy.txt", read_options=read_options, parse_options=parse_options)
```

on my side it runs without error though
lhoestq (MEMBER) commented on 2020-09-29 12:17:06:
> In general, I don't think there is any hard reason we don't allow to use `features` in the csv script, right @lhoestq?
> 
> Should I add it?

Sure let's add it. Setting the convert options should do the job

> Hi @lhoestq we've tried out your suggestion but are now running into the following error:
> 
> ```
> ---------------------------------------------------------------------------
> ValueError                                Traceback (most recent call last)
> <ipython-input-163-81ffd5ac18c9> in <module>
> ----> 1 dataset.cast_(emotion_features)
>
>  /usr/local/lib/python3.6/dist-packages/pyarrow/table.pxi in pyarrow.lib.Table.cast()
> 
> ValueError: Target schema's field names are not matching the table's field names: ['text', 'label'], ['label', 'text']
> ```
>
> Did we define the `emotion_features` incorrectly? We just followed the instructions from the [docs](https://huggingface.co/docs/datasets/features.html?highlight=features#dataset-features), but perhaps we misunderstood something 😬

Thanks for reporting, that's a bug :) I'm fixing it right now
lhoestq (MEMBER) commented on 2020-09-29 12:25:13:
> That's true, It was my error printing the text that way. Maybe as a workaround, I can force all my input samples to have "\b" at the end?

I don't think it would work since we only want one column, and "\b" is set to be the delimiter between two columns, so it will raise the same issue again. Pyarrow would think that there is more than one column if the delimiter is found somewhere.

Anyway, I I'll work on a new text reader if we don't find the right workaround about this delimiter issue.
lhoestq (MEMBER) commented on 2020-09-29 16:40:56:
> I have the same issue with `transformers/BertJapaneseTokenizer`.

It looks like it this tokenizer is not supported unfortunately.
This is because `t.word_tokenizer.mecab` is a `fugashi.fugashi.GenericTagger` which is not compatible with pickle nor dill.

We need objects passes to `map` to be picklable for our caching system to work properly.
Here it crashes because the caching system is not able to pickle the GenericTagger.

\> Maybe you can create an issue on [fugashi](https://github.com/polm/fugashi/issues) 's repo and ask to make `fugashi.fugashi.GenericTagger` compatible with pickle ?

What you can do in the meantime is use a picklable wrapper of the tokenizer:


```python
from transformers import BertJapaneseTokenizer, MecabTokenizer

class PicklableTokenizer(BertJapaneseTokenizer):

    def __getstate__(self):
        state = dict(self.__dict__)
        state["do_lower_case"] = self.word_tokenizer.do_lower_case
        state["never_split"] = self.word_tokenizer.never_split 
        del state["word_tokenizer"]
        return state

    def __setstate__(self, state):
        do_lower_case = state.pop("do_lower_case")
        never_split = state.pop("never_split")
        self.__dict__ = state
        self.word_tokenizer = MecabTokenizer(
            do_lower_case=do_lower_case, never_split=never_split)
        )

t = PicklableTokenizer.from_pretrained("cl-tohoku/bert-base-japanese-whole-word-masking")
encoded = train_ds.map(lambda examples: {'tokens': t.encode(examples['title'])}, batched=True)  # it works
```
thomwolf (MEMBER) commented on 2020-09-29 16:52:32:
We can also update the `BertJapaneseTokenizer` in `transformers` as you just shown @lhoestq to make it compatible with pickle. It will be faster than asking on fugashi 's repo and good for the other users of `transformers` as well.

I'm currently working on `transformers` I'll include it in the https://github.com/huggingface/transformers/pull/7141 PR and the next release of `transformers`.
lhoestq (MEMBER) commented on 2020-09-30 09:11:07:
Hi !

This is because `encode` expects one single text as input (str), or one tokenized text (List[str]).
I believe that you actually wanted to use `encode_batch` which expects a batch of texts.
However this method is only available for our "fast" tokenizers (ex: BertTokenizerFast).
BertJapanese is not one of them unfortunately and I don't think it will be added for now (see https://github.com/huggingface/transformers/pull/7141)...
cc @thomwolf for confirmation.

Therefore what I'd suggest for now is disable batching and process one text at a time using `encode`.
Note that you can make it faster by using multiprocessing:

```python
num_proc = None  # Specify here the number of processes if you want to use multiprocessing. ex: num_proc = 4
encoded = train_ds.map(
    lambda example: {'tokens': t.encode(example['title'], max_length=1000)}, num_proc=num_proc
)
```
peinan (NONE) commented on 2020-09-30 09:52:42:
Thank you very much for the kind and precise suggestion!
I'm looking forward to seeing BertJapaneseTokenizer built into the "fast" tokenizers.

I tried `map` with multiprocessing as follows, and it worked!

```python
# There was a Pickle problem if I use `lambda` for multiprocessing
def encode(examples):
    return {'tokens': t.encode(examples['title'], max_length=1000)}

num_proc = 8
encoded = train_ds.map(encode, num_proc=num_proc)
```

Thank you again!
richarddwang (CONTRIBUTOR) commented on 2020-09-30 11:58:11:
Yes actually the BookCorpus on hugginface is based on [this](https://github.com/soskek/bookcorpus/issues/24#issuecomment-643933352). And I kind of regret naming it as "BookCorpus" instead of something like "BookCorpusLike".

But there is a good news ! @shawwn has replicated BookCorpus in his way, and also provided a link to download the plain text files. see [here](https://github.com/soskek/bookcorpus/issues/27). There is chance we can have a "OpenBookCorpus" !
lhoestq (MEMBER) commented on 2020-09-30 16:58:09:
I just merged a new text reader based on pandas.
Could you try again on your data @raruidol ?

Until we do a new release you can experiment with it using

```python
from datasets import load_dataset

d = load_dataset("text", data_files=..., script_version="master")
```
(`script_version` is `master` by default if you installed from source)
ldong87 (NONE) commented on 2020-09-30 17:34:34:
> @zphang is having different batch sizes per task actually helpful? Would be interesting to know as it's not something I've come across as a technique used by any MTL papers.

I think having different batch sizes per task is particularly helpful in some scenarios where each task has different amount of data. For example, the problem I'm currently facing is one task has tens of thousands of samples while one task has a couple hundreds. I think in this case different batch size could help. But if using the same batch size is a lot simpler to implement, I guess it makes sense to go with that.
lhoestq (MEMBER) commented on 2020-10-01 09:15:39:
Hi @timothyjlaurent 

The hanging fix just got merged, that why you still had it.

For the key error it's possible that the code you ran reused cached datasets from where the KeyError bug was still there.
Could you try to clear your cache or make sure that it doesn't reuse cached data with `.map(..., load_from_cache=False)` ?
Let me know if it it helps
ghomasHudson (CONTRIBUTOR) commented on 2020-10-01 10:55:21:
Are there any references for people doing different batch sizes per task in the literature? I've only seen constant batch sizes with differing numbers of batches for each task which seems sufficient to prevent the impact of large datasets (Read 3.5.3 of the [T5 paper](https://arxiv.org/pdf/1910.10683.pdf) for example).
lhoestq (MEMBER) commented on 2020-10-01 14:05:12:
I'll do a release in the next few days to make the fix available for everyone.
In the meantime you can load `xnli` with
```
xnli = load_dataset('xnli', script_version="master")
```
This will use the latest version of the xnli script (available on master branch), instead of the old one.
sleepinyourhat (NONE) commented on 2020-10-01 14:12:17:
Be careful to keep the ‘sa’ term in the license. It’s something we
inherited from the Flickr captions.

On Thu, Oct 1, 2020 at 10:09 AM Julien Chaumond <notifications@github.com>
wrote:

> *@julien-c* commented on this pull request.
> ------------------------------
>
> In datasets/snli/README.md
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23discussion-5Fr498273172&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=WbEkKXCbL6j5Ui3sox_WqvzrbShbJn2WW-51SENL2ZQ&e=>
> :
>
> > +---
> +language:
> +- en
> +task:
> +- text-classification
> +purpose:
> +- NLI
> +size:
> +- ">100k"
> +language producers:
> +- crowdsourced
> +annotation:
> +- crowdsourced
> +tags:
> +- extended-from-other-datasets
> +license: "CC BY-SA 4.0"
>
> ⬇️ Suggested change
>
> -license: "CC BY-SA 4.0"
> +license: cc-by-4.0
>
> For models (documented at
> https://huggingface.co/docs#what-metadata-can-i-add-to-my-model-card
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__huggingface.co_docs-23what-2Dmetadata-2Dcan-2Di-2Dadd-2Dto-2Dmy-2Dmodel-2Dcard&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=ck3x8c_ujrwKReDTSGuWWgD9W6REHEPbZaO7S4GFRd4&e=>)
> we use the License keywords listed by GitHub at
> https://docs.github.com/en/free-pro-team@latest/github/creating-cloning-and-archiving-repositories/licensing-a-repository#searching-github-by-license-type
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.github.com_en_free-2Dpro-2Dteam-40latest_github_creating-2Dcloning-2Dand-2Darchiving-2Drepositories_licensing-2Da-2Drepository-23searching-2Dgithub-2Dby-2Dlicense-2Dtype&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=dWBP-ZvtMErD-egoBiBTCKA4500mjDXVSk03oW1g16U&e=>
>
> (Hopefully we'll plug some sort of form validation for users at some point)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_huggingface_datasets_pull_663-23pullrequestreview-2D500386385&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=HU2Hwi7HH9W2NtMoCIiQlhXxxEULLi8L9gnWU5PBAPY&e=>,
> or unsubscribe
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAJZSWL63W2LB7SBICA2GMTSISEPZANCNFSM4RWKAZRA&d=DwMFaQ&c=slrrB7dE8n7gBJbeO0g-IQ&r=sCzLyHdE8zgQwk2-sKwA1w&m=PHPCew9Xj3CBQrudcaii70ln-wpRtbngE_tj3Ioy3NI&s=086__lKQLxTanHfjE8kOIpaJbaWPzBB9gGIt_prWeH8&e=>
> .
>
imadarsh1001 (NONE) commented on 2020-10-02 07:07:15:
also i tried below code to solve checksum error 
`datasets-cli test ./datasets/xnli --save_infos --all_configs`

and it shows 

```
2020-10-02 07:06:16.588760: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 268, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 279, in prepare_module
    local_path = cached_path(file_path, download_config=download_config)
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/opt/conda/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 474, in get_from_cache
    raise FileNotFoundError("Couldn't find file at {}".format(url))
FileNotFoundError: Couldn't find file at https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/bin/datasets-cli", line 36, in <module>
    service.run()
  File "/opt/conda/lib/python3.7/site-packages/datasets/commands/test.py", line 76, in run
    module_path, hash = prepare_module(path)
  File "/opt/conda/lib/python3.7/site-packages/datasets/load.py", line 283, in prepare_module
    combined_path, github_file_path, file_path
FileNotFoundError: Couldn't find file locally at ./datasets/xnli/xnli.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.0.2/datasets/./datasets/xnli/xnli.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/./datasets/xnli/xnli.py
```
lhoestq (MEMBER) commented on 2020-10-02 10:05:07:
I like the idea of returning all the scores for two reason:
- Rouge's aggregator does sampling and therefore it returns "low" "mid" and "high" scores
- It is interesting to have the precision and recall to see how the F1 score was computed
But I understand your point that returning only the F1 score makes sense since it's the one that's always used
Shashi456 (NONE) commented on 2020-10-02 10:15:00:
@thomwolf the scores now returned look like this:
```
{'rouge1': AggregateScore(low=Score(precision=0.16620308156871524, recall=0.18219819615984395, fmeasure=0.16226017699359463), mid=Score(precision=0.17274338501705871, recall=0.1890957812369246, fmeasure=0.16823877588620403), high=Score(precision=0.17934569582981455, recall=0.1965626706042028, fmeasure=0.17491509794856058)), 
'rouge2': AggregateScore(low=Score(precision=0.12478835737689957, recall=0.1362113231755514, fmeasure=0.12055941950062395), mid=Score(precision=0.1303967602691664, recall=0.1423747229852964, fmeasure=0.1258363976151122), high=Score(precision=0.13654527560789362, recall=0.1488071465116122, fmeasure=0.13184989406704056)), 
'rougeL': AggregateScore(low=Score(precision=0.16568068818352072, recall=0.1811919016674486, fmeasure=0.1614784523482225), mid=Score(precision=0.17156684723552357, recall=0.1879777628247058, fmeasure=0.16720699286250762), high=Score(precision=0.17788847350584547, recall=0.1948899838530898, fmeasure=0.17316501523379826))}
```

While when computed through the perl rouge script, it looks like:
```
ROUGE-1 Average_R: 0.34775 (95%-conf.int. 0.34546 - 0.35025)
ROUGE-1 Average_P: 0.19381 (95%-conf.int. 0.19246 - 0.19538)
ROUGE-1 Average_F: 0.24070 (95%-conf.int. 0.23925 - 0.24230)
---------------------------------------------
ROUGE-2 Average_R: 0.07160 (95%-conf.int. 0.07010 - 0.07298)
ROUGE-2 Average_F: 0.04845 (95%-conf.int. 0.04741 - 0.04942)
---------------------------------------------
ROUGE-L Average_R: 0.26404 (95%-conf.int. 0.26215 - 0.26598)
ROUGE-L Average_P: 0.14696 (95%-conf.int. 0.14576 - 0.14815)
ROUGE-L Average_F: 0.18245 (95%-conf.int. 0.18120 - 0.18367)
```
while the wrapper returns the much more readable:
```
[2020-07-30 18:13:38,556 INFO] Rouges at step 13000                                       
>> ROUGE-F(1/2/3/l): 43.43/20.42/39.78              
ROUGE-R(1/2/3/l): 53.91/25.34/49.32
```

The formatting allows for easy reading, and although "low", "mid", "high" make sense, this is more concise and effective. 

One way of changing this might be to return a dictionary that returns values like `rouge_1_precision`, `rouge_1_F1`, `rouge_1_recall`, and maybe also having the ability to get the values you are interested in and keeping `recall` and `F1` as default.
timothyjlaurent (NONE) commented on 2020-10-03 18:27:32:
Hi @lhoestq , 

Thanks for letting me know about the update.

So I don't think it's the caching - because hashing mechanism isn't stable for me -- but that's a different issue. In any case I `rm -rf ~/.cache/huggingface` to make a clean slate.

I synced with master and I see the key error has gone away, I tried with and without the `TOKENIZERS_PARALLELISM` variable set and see the log line for setting the value false before the map.

Now I'm seeing an issue with `.train_test_split()` on datasets that are the product of a multiprocess map.

Here is the stack trace

```
  File "/Users/timothy.laurent/src/inv-text2struct/text2struct/model/dataset.py", line 451, in load_prodigy_arrow_datasets
    ner_ds_dict = ner_ds.train_test_split(test_size=test_pct, shuffle=True, seed=seed)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/arrow_dataset.py", line 168, in wrapper
    dataset.set_format(**new_format)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/fingerprint.py", line 163, in wrapper
    out = func(self, *args, **kwargs)
  File "/Users/timothy.laurent/.virtualenvs/inv-text2struct/src/datasets/src/datasets/arrow_dataset.py", line 794, in set_format
    list(filter(lambda col: col not in self._data.column_names, columns)), self._data.column_names
ValueError: Columns ['train', 'test'] not in the dataset. Current columns in the dataset: ['_input_hash', '_task_hash', '_view_id', 'answer', 'encoding__ids', 'encoding__offsets', 'encoding__overflowing', 'encoding__tokens', 'encoding__words', 'ner_ids', 'ner_labels', 'relations', 'spans', 'text', 'tokens']
```
punitaojha (NONE) commented on 2020-10-04 18:52:50:
Hello @mathcass 
1. I did fork the repository and clone the same on my local system. 

2. Then learnt about how we can publish our package on pypi.org. Also, found some instructions on same in setup.py documentation.

3. Then I Perplexity document link that you shared above. I created a colab link from there keep both tensorflow and pytorch means a mixed option and tried to run it in colab but I encountered no errors at a point where you mentioned. Can you help me to figure out the issue. 

4.Here is the link of the colab file with my saved responses. 
https://colab.research.google.com/drive/1hfYz8Ira39FnREbxgwa_goZWpOojp2NH?usp=sharing
Skyy93 (NONE) commented on 2020-10-05 07:05:53:
I had the same error but managed to fix it with adding `lineterminator='\n'` to the pd.read_csv() in the text.py

Have a look at [Stackoverflow](https://stackoverflow.com/questions/33998740/error-in-reading-a-csv-file-in-pandascparsererror-error-tokenizing-data-c-err) 

EDIT:
I tested it with my 1,5tb dataset. With the lineterminator added it works fine!
lhoestq (MEMBER) commented on 2020-10-05 09:48:26:
I found a way to implement it without third party lib and without separator/delimiter logic.
Creating a PR now.

I'd love to have your feedback on the PR @Skyy93 , hopefully this is the final iteration of the text dataset :) 

Let me know if it works on your side !

Until there's a new release you can test it with 

```python
from datasets import load_dataset

d = load_dataset("text", data_files=..., script_version="master")
```
BramVanroy (CONTRIBUTOR) commented on 2020-10-05 11:23:57:
> I remember issue #546 where this was kinda requested (but maybe IO would bottleneck). What do you think?

It still would be interesting I think, especially in scenarios where IO is less of an issue (SSDs particularly) and where there are many smaller files. Wrapping this function in a `pool.map` is perhaps an easy thing to try.