t20062443/AT1ds huggingface dataset:
lhoestq (MEMBER) commented on 2020-06-30 10:02:57:
It's definitely going in the right direction ! Thanks for giving it a try

I really like the API.
IMO it's fine right now if we don't have all the dataset transforms (map, filter, etc.) as it can be done before building the multitask dataset, but it will be important to have them in the end.
All the formatting methods could easily be added though.

I think there are some parts that will require some work with apache arrow like slicing. I can find a way to do it using pyarrow tables concatenation (I did something similar when implementing `__getitem__` with an input that is a list of indices [here](https://github.com/huggingface/nlp/pull/322/files#diff-73270df8d7f08c62a27e40806e1a5fb0R463-R469)). It is very fast and it allows to have the same output format as a normal Dataset.

Also maybe we should check that not only the columns but also the schemas match ?
And maybe add the `seed` of the shuffling step as an argument ?
lhoestq (MEMBER) commented on 2020-06-30 15:43:12:
I'm pretty sure you can get some inspiration from the squad_v2 script. It looks like the dataset is quite big so it will take some time for the users to generate it, but it should be reasonable.

Also you are saying that you are still making the dataset grow in size right ?
It's probably good practice to let the users do their training/evaluations with the exact same version of the dataset.
We allow for each dataset to specify a version (ex: 1.0.0) and increment this number every time there are new samples in the dataset for example. Does it look like a good solution for you ? Or would you rather have one final version with the full dataset ?
lhoestq (MEMBER) commented on 2020-06-30 16:05:09:
It looks like it comes from `mwparserfromhell`.

Would it be possible to get the bad `section` that causes this issue ? The `section` string is from `datasets/wikipedia.py:L548` ? You could just add a `try` statement and print the section if the line `section_text.append(section.strip_code().strip())` crashes.

It will help us know if we have to fix it on our side or if it is a `mwparserfromhell` issue.
flozi00 (CONTRIBUTOR) commented on 2020-06-30 16:38:02:
It would also be good if there is any possibility for versioning, I think this way is much better than the dynamic way.
If you mean that part to put the tiles into one is the generation it would take up to 15-20 minutes on home computer hardware.
Are there any compression or optimization algorithms while generating the dataset ?
Otherwise the hardware limit is around 32 GB ram at the moment.
If everything works well we will add some more gigabytes of data in future what would make it pretty memory costly.
lhoestq (MEMBER) commented on 2020-06-30 17:01:23:
To be able to generate the Arrow dataset you need to either use our csv or json utilities `load_dataset("json", data_files=my_json_files)` OR write your own custom dataset script (you can find some inspiration from the [squad](https://github.com/huggingface/nlp/blob/master/datasets/squad/squad.py) script for example). Custom dataset scripts can be called locally with `nlp.load_dataset(path_to_my_script_directory)`.

This should help you get what you call "Dataset1".

Then using some dataset transforms like `.map` for example you can get to "DatasetNER" and "DatasetREL".
jarednielsen (CONTRIBUTOR) commented on 2020-06-30 19:56:04:
Environment is Ubuntu 18.04, Python 3.7.5, nlp==0.3.0, filelock=3.0.12.

The external volume is Amazon FSx for Lustre, and it by default creates files with limited permissions. My working theory is that FileLock creates a lockfile that isn't writable, and thus there's no way to acquire it by removing the .lock file. But Python is able to create new files and write to them outside of the FileLock package.

When I attempt to use FileLock within a Docker container by writing to `/root/.cache/hello.txt`, it succeeds. So there's some permissions issue. But it's not a Docker configuration issue; I've replicated it without Docker.
```bash
echo "hello world" >> hello.txt
ls -l

-rw-rw-r-- 1 ubuntu ubuntu 10 Jun 30 19:52 hello.txt
```
thomwolf (MEMBER) commented on 2020-06-30 20:57:41:
Oh yes, this is a backward compatibility feature with tensorflow_dataset in which a `Sequence` or `dict` is converted in a `dict` of `lists`, unfortunately it is not very intuitive, see here: https://github.com/huggingface/nlp/blob/master/src/nlp/features.py#L409

To avoid this behavior, you can just define the list in the feature with a simple list or a tuple (which is also simpler to write).
In your case, the features could be as follow:
``` python
...
features=nlp.Features({
    "title": nlp.Value("string"),
    "vertexSet": [[{
        "name": nlp.Value("string"),
        "sent_id": nlp.Value("int32"),
        "pos": nlp.features.Sequence(nlp.Value("int32")),
        "type": nlp.Value("string"),
    }]],
    ...
    }),
...
```
jarednielsen (CONTRIBUTOR) commented on 2020-06-30 21:23:52:
Looks like the `flock` syscall does not work on Lustre filesystems by default: https://github.com/benediktschmitt/py-filelock/issues/67.

I added the `-o flock` option when mounting the filesystem, as [described here](https://docs.aws.amazon.com/fsx/latest/LustreGuide/getting-started-step2.html), which fixed the issue.
timothyjlaurent (NONE) commented on 2020-07-02 05:49:28:
Thanks for the helpful advice, @lhoestq  -- I wasn't quite able to get the json recipe working - 

```
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.py in __init__(self, source)
     60 
     61     def __init__(self, source):
---> 62         self._open(source)
     63 
     64 
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/ipc.pxi in pyarrow.lib._RecordBatchStreamReader._open()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()
~/.virtualenvs/inv-text2struct/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Tried reading schema message, was null or length 0
```

But I'm going to give the generator_dataset_builder a try.

1 more quick question -- can .map be used to output different length mappings -- could I skip one, or yield 2, can you map_batch
jarednielsen (CONTRIBUTOR) commented on 2020-07-02 15:36:13:
This approach still seems quite slow. When using TFRecords with a similar training loop, I get ~3.0-3.5 it/s on multi-node, multi-GPU training. I notice a pretty severe performance regression when scaling, with observed performance numbers. Since the allreduce step takes less than 100ms/it and I've achieved 80% scaling efficiency up to 64 GPUs, it must be the data pipeline.

| Nodes | GPUs | Iterations/Second |
| --- | --- | --- |
| 1 | 2 | 2.01 |
| 1 | 8 | 0.81 |
| 2 | 16 | 0.37 |

Here are performance metrics over 10k steps. The iteration speed appears to follow some sort of caching pattern. I would love to use `nlp` in my project, but a slowdown from 3.0 it/s to 0.3 it/s is too great to stomach.

<img width="1361" alt="Screen Shot 2020-07-02 at 8 29 22 AM" src="https://user-images.githubusercontent.com/4564897/86378156-2f8d3900-bc3e-11ea-918b-c395c3df5377.png">
Shiro-LK (NONE) commented on 2020-07-03 14:07:42:
Hi, 

Thank you for you answer.
I have try to print the bad section using `try` and `except`, but it is a bit weird as the error seems to appear 3 times for instance, but the two first error does not print anything (as if the function did not go in the `except` part).
For the third one, I got that (I haven't display the entire text) :

> error : ==== Parque nacional Cajas ====
> {{AP|Parque nacional Cajas}}
> [[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
> El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
>  [[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
> leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
> Para acceder al parque desde la costa, la vía Molleturo-Cuenca es también la mejor opción.

How can I display the link instead of the text ? I suppose it will help you more
lhoestq (MEMBER) commented on 2020-07-03 14:23:01:
The error appears several times as Apache Beam retries to process examples up to 4 times irc.

I just tried to run this text into `mwparserfromhell` but it worked without the issue.

I used this code (from the `wikipedia.py` script):
```python
import mwparserfromhell as parser
import re
import six

raw_content = r"""==== Parque nacional Cajas ====
{{AP|Parque nacional Cajas}}
[[Archivo:Ecuador cajas national park.jpg|thumb|left|300px|Laguna del Cajas]]
El parque nacional Cajas está situado en los [[Cordillera de los Andes|Andes]], al sur del [[Ecuador]], en la provincia de [[Provincia de Azuay|Azuay]], a 33
[[km]] al noroccidente de la ciudad de [[Cuenca (Ecuador)|Cuenca]]. Los accesos más comunes al parque inician todos en Cuenca: Desde allí, la vía Cuenca-Mol
leturo atraviesa en Control de [[Surocucho]] en poco más de 30 minutos de viaje; más adelante, esta misma carretera pasa a orillas de la laguna La Toreadora donde están el Centro Administrativo y de Información del parque. Siguiendo de largo hacia [[Molleturo]], por esta vía se conoce el sector norte del Cajas y se serpentea entre varias lagunas mayores y menores.
"""

wikicode = parser.parse(raw_content)

# Filters for references, tables, and file/image links.
re_rm_wikilink = re.compile("^(?:File|Image|Media):", flags=re.IGNORECASE | re.UNICODE)

def rm_wikilink(obj):
    return bool(re_rm_wikilink.match(six.text_type(obj.title)))

def rm_tag(obj):
    return six.text_type(obj.tag) in {"ref", "table"}

def rm_template(obj):
    return obj.name.lower() in {"reflist", "notelist", "notelist-ua", "notelist-lr", "notelist-ur", "notelist-lg"}

def try_remove_obj(obj, section):
    try:
        section.remove(obj)
    except ValueError:
        # For unknown reasons, objects are sometimes not found.
        pass

section_text = []
for section in wikicode.get_sections(flat=True, include_lead=True, include_headings=True):
    for obj in section.ifilter_wikilinks(matches=rm_wikilink, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_templates(matches=rm_template, recursive=True):
        try_remove_obj(obj, section)
    for obj in section.ifilter_tags(matches=rm_tag, recursive=True):
        try_remove_obj(obj, section)

    section_text.append(section.strip_code().strip())
```
lhoestq (MEMBER) commented on 2020-07-03 16:27:29:
Because of the way Apache Beam works we indeed don't have access to the file name at this point in the code.
We'll have to use some tricks I think :p 

You can append `filepath` to `title` in `wikipedia.py:L512` for example. [[EDIT: it's L494 my bad]]
Then just do `try:...except:` on the  call of `_parse_and_clean_wikicode` L500 I guess.

Thanks for diving into this ! I tried it myself but I run out of memory on my laptop
As soon as we have the name of the file it should be easier to find what's wrong.
thomwolf (MEMBER) commented on 2020-07-03 21:52:32:
That's an interesting first draft, thanks a lot for that and the user facing API is really nice.

I think we should dive more into this and the questions of #217 before merging the first version though.

In particular, the typical way to do multi-tasking is usually to sample a task and then sample a batch within the selected task. I think we should probably stay be closer to this traditional approach, or at least make it very easy to do, rather than go to close to the T5 approach which is very specific to this paper.

In this regard, it seems important to find some way to address the remarks of @zphang. I'm still wondering if we should not adopt more of a sampling approach rather than an iteration approach.
ghomasHudson (CONTRIBUTOR) commented on 2020-07-04 19:30:37:
@thomwolf Thanks! I mainly wanted to get something working quickly for my own MTL research. I agree with a lot of the points you made so I'll convert this pull request back to a draft.

For your specific point about 'batch-level' multitask mixing, it would be a pretty trivial change to add a `batch_size` parameter and ensure every `batch_size` examples are from the same task. This would certainly work, but would add a notion of 'batches' to a Dataset, which does feel like a 'Sampler-level' concept and not a Dataset one. There's also the possibility of wanting some specific task-level sampling functionality (e.g. applying `SortishSampler` to each task) which would only work with this kind of 2 step sampling approach. My first proposal in the transformers repo was actually a Sampler https://github.com/huggingface/transformers/issues/4340. I wonder whether functionality at the sampler-level has a place in the vision for the `nlp` repo?

I imagine following a sampling approach you'd have to abandon maintaining the same user-facing API as a standard dataset (A shame because replacing a single dataset seamlessly with a multitask one is a really nice user-experience).

Random half-Idea: You could have a class which accepts a list of any iterables (either a Dataset or a DataLoader which already is doing the batching). Not sure what interface you'd present though. hmmm. 

There's definitely more discussion to have.
lhoestq (MEMBER) commented on 2020-07-05 12:23:52:
`dataset_infos.json` is still not up to date with the new urls (we can see that there are urls like `gs://deepmind-gutenberg/train/*` instead of `https://storage.googleapis.com/deepmind-gutenberg/train/*` in the json file)

Can you check that you re-ran the command to update the json file, and that you pushed the changes @lucidrains ?
lucidrains (CONTRIBUTOR) commented on 2020-07-05 16:46:27:
@lhoestq ohhh, I made the change in this commit https://github.com/lucidrains/nlp/commit/f3e23d823ad9942031be80b7c4e4212c592cd90c , that's interesting that the pull request didn't pick it up. maybe it's because I did it on another machine, let me check and get back to you!
lhoestq (MEMBER) commented on 2020-07-06 12:16:37:
For datasets with nested features we have two aspects to take into account:
1) There can be nested dict of features. What is done in tensorflow_datasets to make things work is to flatten the dictionaries to end up with one single dictionary. A dict like `{"column1": {"subfeature": ...}}` is converted to `{"column1/subfeature":...}`
2) There can be ragged tensors, i.e. lists of objects with non-fixed shapes. For example in squad there are often multiple possible answers per question. What is done in tensorflow_datasets to make things work is to concatenate everything and add ragged attributes (cf serialization code [here](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/core/example_serializer.py))
Shiro-LK (NONE) commented on 2020-07-06 16:30:58:
Thanks for your help.

I tried to print the "title" of the document inside the` except (mwparserfromhell.parser.ParserError) as e`,the title displayed was : "Campeonato Mundial de futsal de la AMF 2015". (Wikipedia ES) Is it what you were looking for ?
lhoestq (MEMBER) commented on 2020-07-06 18:22:55:
Thanks a lot @Shiro-LK !

I was able to reproduce the issue. It comes from [this table on wikipedia](https://es.wikipedia.org/wiki/Campeonato_Mundial_de_futsal_de_la_AMF_2015#Clasificados) that can't be parsed.

The file in which the problem occurs comes from the wikipedia dumps, and it can be downloaded [here](https://dumps.wikimedia.org/eswiki/20200501/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2)

Parsing the file this way raises the parsing issue:

```python
import mwparserfromhell as parser
from tqdm.auto import tqdm
import bz2
import six
import logging
import codecs
import xml.etree.cElementTree as etree

filepath = "path/to/eswiki-20200501-pages-articles-multistream6.xml-p6424816p7924815.bz2"

def _extract_content(filepath):
    """Extracts article content from a single WikiMedia XML file."""
    logging.info("generating examples from = %s", filepath)
    with open(filepath, "rb") as f:
        f = bz2.BZ2File(filename=f)
        if six.PY3:
            # Workaround due to:
            # https://github.com/tensorflow/tensorflow/issues/33563
            utf_f = codecs.getreader("utf-8")(f)
        else:
            utf_f = f
        # To clear root, to free-up more memory than just `elem.clear()`.
        context = etree.iterparse(utf_f, events=("end",))
        context = iter(context)
        unused_event, root = next(context)
        for unused_event, elem in tqdm(context, total=949087):
            if not elem.tag.endswith("page"):
                continue
            namespace = elem.tag[:-4]
            title = elem.find("./{0}title".format(namespace)).text
            ns = elem.find("./{0}ns".format(namespace)).text
            id_ = elem.find("./{0}id".format(namespace)).text
            # Filter pages that are not in the "main" namespace.
            if ns != "0":
                root.clear()
                continue
            raw_content = elem.find("./{0}revision/{0}text".format(namespace)).text
            root.clear()

            if "Campeonato Mundial de futsal de la AMF 2015" in title:
                yield (id_, title, raw_content)

for id_, title, raw_content in _extract_content(filepath):
    wikicode = parser.parse(raw_content)
```

The copied the raw content that can't be parsed [here](https://pastebin.com/raw/ZbmevLyH).

The minimal code to reproduce is:
```python
import mwparserfromhell as parser
import requests

raw_content = requests.get("https://pastebin.com/raw/ZbmevLyH").content.decode("utf-8")
wikicode = parser.parse(raw_content)

```

I will create an issue on mwparserfromhell's repo to see if we can fix that
yjernite (MEMBER) commented on 2020-07-06 20:12:39:
Hi @saverymax ! For licensing reasons, the original team was unable to release pre-processed CommonCrawl documents. Instead, they provided a script to re-create them from a CommonCrawl dump, but it unfortunately requires access to a medium-large size cluster:
https://github.com/facebookresearch/ELI5#downloading-support-documents-from-the-commoncrawl

In order to make the task accessible to people who may not have access to this kind of infrastructure, we suggest to use Wikipedia as a knowledge source rather than the full CommonCrawl. The following blog post shows how you can create Wikipedia support documents and get a performance that is on par with a system that uses CommonCrawl pages.
https://yjernite.github.io/lfqa.html#task_description

Hope that helps, using ElasticSearch to index Wiki40b and create the documents should take about 4 hours. Let us know if you have any trouble with the blog post though!
jarednielsen (CONTRIBUTOR) commented on 2020-07-06 20:33:56:
I added support for nested dictionaries. A few more design decisions popped up:

_Should we serialize from NumPy arrays or from tf.Tensors?_
- The [tfds example serializer](url) works from NumPy arrays.
- Calling `dset.set_format("tensorflow")` makes `__getitem__` return a tf.Tensor. So serializing from NumPy arrays would mean calling `dset.export()` before setting the format, which is confusing.
- NumPy arrays can be serialized as their underlying datatype (int, float), while tf.Tensors must be converted to strings before serialization. This adds another step when serializing and deserializing, and removes the static-typing advantages of the TFRecord format.

I think we should export directly from the underlying NumPy arrays into TFRecords, rather than using an intermediate step of tf.Tensor.

_Should we serialize lists of dictionaries?_
- The test_format_nested() test creates a list of dictionaries: https://github.com/huggingface/nlp/blob/911d5596f9b500e39af8642fe3d1b891758999c7/tests/test_arrow_dataset.py#L278-L288
- This is difficult to serialize effectively, and I'm not aware of any dataset that has this format. SQuAD has a dictionary of lists, such as the `answers` key. Is this necessary?
ghomasHudson (CONTRIBUTOR) commented on 2020-07-07 11:29:05:
I've tried it and am getting the same error as you.

You could use the text files rather than the pickle:
```
https://www.dropbox.com/s/ikkqxfdbdec3fuj/test.txt
https://www.dropbox.com/s/1pzkadrvffbqw6o/train.txt
https://www.dropbox.com/s/2mzialpsgf9k5l3/val.txt
```

Then you would get all 3 splits rather than just the train split.
saverymax (NONE) commented on 2020-07-07 15:00:52:
Hi, thanks for the quick response. The blog post is quite an interesting working example, thanks for sharing it.
Two follow-up points/questions about my original question:

1. Yes, I read that the facebook team could not share the CommonCrawl b/c of licensing reasons. They state "No, we are not allowed to host processed Reddit or CommonCrawl data," which indicates they could also not share the Reddit data for licensing reasons. But it seems that HuggingFace is able to share the Reddit data, so why not a subset of CommonCrawl?

2. Thanks for the suggestion about ElasticSearch and Wiki40b. This is good to know about performance. I definitely could do the indexing and querying myself. What I like about the ELI5 dataset though, at least what is suggested by the paper, is that to create the dataset they had already selected the top 100 web sources and made a single support document from those. Though it doesn't appear to be too sophisticated an approach, having a single support document pre-computed (without having to run the facebook code or a replacement with another dataset) is super useful for my work, especially since I'm not working on developing the latest and greatest retrieval model. Of course, I don't expect HF NLP datasets to be perfectly tailored to my use-case. I know there is overhead to any project, I'm just illustrating a use-case of ELI5 which is not possible with the data provided as-is. If it's for licensing reasons, that is perfectly acceptable a reason, and I appreciate your response.
johanneskiesel (NONE) commented on 2020-07-07 18:40:29:
Thank you so much for working on this! This is awesome!

How much would it help you if we would remove the manual request?

We are naturally interested in getting some broad idea of how many people and who are using our dataset. But if you consider hosting the dataset yourself, I would rather remove this small barrier on our side (so that we then still get the download count from your library).
thomwolf (MEMBER) commented on 2020-07-08 11:45:08:
An interesting alternative to investigate here would be to use the tf.io library which has some support for Arrow to TF conversion: https://www.tensorflow.org/io/api_docs/python/tfio/arrow/ArrowDataset

There are quite a few types supported, including lists so if the unsupported columns are dropped then we could maybe have a zero-copy mapping from Arrow to TensorFlow, including tokenized inputs and 1D tensors like the ones we mostly use in NLP: https://github.com/tensorflow/io/blob/322b3170c43ecac5c6af9e39dbd18fd747913e5a/tensorflow_io/arrow/python/ops/arrow_dataset_ops.py#L44-L72

Here is an introduction on Arrow to TF using tf.io: https://medium.com/tensorflow/tensorflow-with-apache-arrow-datasets-cdbcfe80a59f