Title: Update Open Subtitles corpus with original sentence IDs
Author: Valahaar (CONTRIBUTOR)
State: closed
Created: 2021-02-08T13:55:13Z
Labels: good first issue

Hi! It would be great if you could add the original sentence ids to [Open Subtitles](https://huggingface.co/datasets/open_subtitles).

I can think of two reasons: first, it's possible to gather sentences for an entire document (the original ids contain media id, subtitle file id and sentence id), therefore somewhat allowing for document-level machine translation (and other document-level stuff which could be cool to have); second, it's possible to have parallel sentences in multiple languages, as they share the same ids across bitexts.

I think I should tag @abhishekkrthakur as he's the one who added it in the first place.

Thanks!

Comments:
Hi ! You're right this can can useful.
This should be easy to add, so feel free to give it a try if you want to contribute :)
I think we just need to add it to the _generate_examples method of the OpenSubtitles dataset builder [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L103)

Hey @lhoestq , absolutely yes! Just one question before I start implementing. The ids found in the zip file have this format: 
(the following is line `22497315` of the `ids` file of the `de-en` dump)


`de/2017/7006210/7063319.xml.gz  en/2017/7006210/7050201.xml.gz  335     339 340` (every space is actually a tab, aside from the space between `339` and `340`)


Where filenames encode the information like this: `lang/year/imdb_id/opensubtitles_id.xml.gz` whereas the numbers correspond to the sentence ids which are linked together (i.e. sentence `335` of the German subtitle corresponds to lines `339` and `340` of the English file)

That being said, do you think I should stick to the raw sentence id (and replace the current sequential id) or should I include more detailed metadata (or both things maybe)?

Going with raw ID is surely simpler, but including `year`, `imdbId` and `subtitleId` should save space as they're just integers; besides, any operation (like filtering or grouping) will be much easier if users don't have to manually parse the ids every time.
As for the language-specific sentenceIds, what could be the best option? A list of integers or a comma-separated string?

**Note:** I did not find any official information about this encoding, but it appears to check out:
https://www.imdb.com/title/tt7006210/, https://www.opensubtitles.org/en/subtitles/7063319 and https://www.opensubtitles.org/en/subtitles/7050201 all link to the same episode, so I guess (I hope!) it's correct.



I like the idea of having `year`, `imdbId` and `subtitleId` as columns for filtering for example.
And for the `sentenceIds` a list of integers is fine.

Thanks for improving it @Valahaar :) 

Something like this? (adapted from [here](https://github.com/huggingface/datasets/blob/master/datasets/open_subtitles/open_subtitles.py#L114))

```python
result = (
  sentence_counter,
    {
      "id": str(sentence_counter),
      "meta": {
        "year": year,
        "imdbId": imdb_id,
        "subtitleId": {l1: l1_sub_id, l2: l2_sub_id},
        "sentenceIds": {l1: [... source_sids ...], l2: [... target_sids ...]},
        # or maybe src/tgt? I'd go with the first one for consistency with 'translation'
        "subtitleId": {"src": l1_sub_id, "tgt": l2_sub_id},
        "sentenceIds": {"src": [... source_sids ...], "tgt": [... target_sids ...]},
      },
      "translation": {l1: x, l2: y},
    },
  )
```
Or at top level, avoiding nesting into 'meta'?

Merged in #1865, closing. Thanks :)