Add support for Tarred dataset for Transformer LM
State: closed | Pull request: yes

# Changelog
- Adds tarred dataset support for transformer based Language models
- Pre-tokenizes and chunks the dataset for more efficient external dataset handling
- Adds utility script to construct a tarred tokenized LM dataset
- Adds `cache_ids` flag support for transformer based language models

This pull request **introduces 2 alerts** when merging b7801269d9ee2ca213f5bd43e4302fd148ed66bc into 38c854da3f553b6577d6f8b08490e026b53b7ea3 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-fa359a48d5f95a10eb6e93d11f38e27ea636a1e6)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 2 alerts** when merging 4433d5a86d59373b9e0250d55e11efeed56a221f into 5dbe245d04d4a062aaf42bd414ce72dfcdcbb092 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-66b8a18e76d721f10280ed97e01d9f333f971244)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 2 alerts** when merging 49eb7b4238804bebe1bb4fee6a83f171160de066 into a711ca1ad2987797b1e80ac9b0ffd7479c0a4730 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-6a3ee9a8d9291ff992ba1a3e3e39e4465f4aaf94)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 2 alerts** when merging 325a9e15542900671730a0a78de7043244a315cd into 48bd21309145a6772fca7375d3ced377033dd3cf - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-acd1f51a693f4dccee1ce4e83bb806da8c454bdd)

**new alerts:**

* 1 for Unused local variable
* 1 for Unused import

This pull request **introduces 1 alert** when merging 61acf63635b6b437345a0123f4d3bcd70ffbd893 into 5b6dd3850129898be47cf0d65587897ec45a5b59 - [view on LGTM.com](https://lgtm.com/projects/g/NVIDIA/NeMo/rev/pr-ee293a8e9cfdfb6b110aed089b11822e23e2cab9)

**new alerts:**

* 1 for Unused local variable

@AlexGrinch The tokenizer can be either cased or uncased - for example LS normalized text corpus is all lower case chars, but other datasets may be cased. When training a LM with a custom vocabulary, it is worthwhile to have an option to lowercase the text prior to tokenization if required, so that additional processing is not required inside the tarred dataset. 

@titu1994 Please correct me if I am wrong. What I understood from the code is that in the process of creating tarred dataset, we take all text, transform it into token_ids via _tokenizer.text_to_ids(text)_, and save this numbers into tarred numpy array. If our tokenizer works with lower case text, text.lower() is called somewhere inside _tokenizer.text_to_ids(text)_. It seems that there is no need to do it in advance as we call _tokenizer.text_to_ids(text)_ either way during dataset creation.

For ASR datasets (as an example) - Librispeech LM corpus - the tokenizer is a bert base cased. The language corpus itself is in all Upper case, but the tokenizer is built to support only lower case. Since it is a cased tokenizer, it will convert all text to [UNK] instead of lowering it. 

Since this script directly uses a prebuilt tokenizer vocabulary therefore it is necessary to provide the logic to explicitly lowercase via the flag for compatibility between the text corpus (cases or uncased), the tokenizer vocabulary (cased or uncased) and the tokenizer construction (cased or uncased).

Okay, I think we can merge this PR after it passes all checks.

Sounds good, I'll fix up the error