Title: Adding Europarl Bilingual dataset
Author: lucadiliello (CONTRIBUTOR)
State: closed
Created: 2021-02-13T17:02:04Z

Implementation of Europarl bilingual dataset from described [here](https://opus.nlpl.eu/Europarl.php).

This dataset allows to use every language pair detailed in the original dataset. The loading script manages also the small errors contained in the original dataset (in very rare cases (1 over 10M) there are some keys that references to inexistent sentences).
I chose to follow the the style of a similar dataset available in this repository: `multi_para_crawl`.

Comments:
is there a way to check errors without subscribing to CircleCI? Because they want access to private repositories when logging.

I think you need to be logged in to check the errors unfortunately. Feel free to create an account with bitbucket maybe if you don't want it to access your private github repos

I've resolved some requirements, but I cannot create dummy data. The dataset works as follows: for each language pair `<lang1>-<lang2>` 3 files are downloaded:
- dataset for `<lang1>`
- dataset for `<lang2>`
- alignments between `<lang1>` and `<lang2>`

Suppose we work with the `bg-cs` language pair. Then, the dataset will download three `gzip` files which should be decompressed. I do not understand the relation between the folders created by the script to create dummy data and the original data provided by the download manager.

Hi ! Indeed the data files structure of this dataset looks very specific.
The command `datasets-cli dummy_data ./datasets/europarl_bilingual` shows some instructions for each split but let me add more details.

First things to know is that the dummy data files need to be uncompressed data, so for example for the file `bg.zip` you should actually have one folder with all the xml files in it instead. In the same way, `bg-cs.xml.gz` must be replaced by an actual uncompressed xml file.

Let's take the bg-cs config as an example. To make the dummy data you need to:
- go to `./datasets/europarl_bilingual/dummy/bg-cs/8.0.0` and create a folder named `dummy_data`. Then go inside this folder
- create a text file named `bg-cs.xml.gz` containing xml content (so without .gz compression). The xml content must have the same structure as the original `bg-cs.zml` but only include 1 `linkGrp` entry. You can pick one entry from the original `bg-cs.xml` file. Let's say this entry is about this file: `ep-06-01-16-003.xml`
- create a folder named `bg.zip` and inside this folder add one file Europarl/raw/bg/ep-06-01-16-003.xml. You can pick the xml file from the original `bg.zip` archive.
- create a folder named `cs.zip` and inside this folder add one file Europarl/raw/cs/ep-06-01-16-003.xml. You can pick the xml file from the original `cs.zip` archive.
- zip the `dummy_data` into `dummy_data.zip`

At this point you have dummy data files to generate 1 example which is what we want to be able to test the dataset script `europarl_bilingual.py` with pytest. 

In particular this will make this test pass:
```
pytest tests/test_dataset_common.py::LocalDatasetTest::test_load_dataset_europarl_bilingual
```

Ideally it would be awesome to have dummy data for all the different configs so if we manage to make a script that generates all of it automatically that would be perfect. However since the structure is not trivial, another option would be to only have the dummy data for only 1 or 2 configs, like what we do for [bible_para](https://github.com/huggingface/datasets/blob/master/datasets/bible_para/bible_para.py) for example. In `bible_para` only a few configurations are tested. As you can see there is only 6 configs in the `BUILDER_CONFIGS` attribute. All the other configs can still be used, here is what is said inside the dataset card of bible_para:
```
To load a language pair which isn't part of the config, all you need to do is specify the language code as pairs.
You can find the valid pairs in Homepage section of Dataset Description: http://opus.nlpl.eu/bible-uedin.php
E.g.

`dataset = load_dataset("bible_para", lang1="fi", lang2="hi")`
```
In this case the configuration "fi-hi" is simply created on the fly, instead of being picked from the `BUILDER_CONFIGS` list.

I hope this helps, let me know if you have questions or if I can help

I already created the scripts to create reduced versions of the data. What I didn't understand was how to put files in the dummy_data folder because, as you noticed, some file decompress to a nested tree structure. I will now try again with your suggestions!

Is there something else I should do? If not can this be integrated?

Thanks a lot !!
Since the set of all the dummy data files is quite big I only kept a few of them. If we had kept them all the size of the `datasets` repo would have increased too much :/
So I did the same as for `bible_para`: only keep a few configurations in BUILDER_CONFIGS and have all the other pairs loadable with the lang1 and lang2 parameters like this:

`dataset = load_dataset("europarl_bilingual", lang1="fi", lang2="fr")`