Remove ending newlines when encoding strings w/ sentencepiece tokenizer
State: closed | Pull request: yes
Labels: name: NLP

Signed-off-by: Patrick Simianer <patrick@lilt.com>

# What does this PR do ?

SentencePiece models can be generated in a number of different ways in terms of handling whitespace. An important parameter is `normalization_rule_name`: Setting this to "identity" will cause newline characters `\n` to remain in the string when encoding. In the default setting (`nmt_nfkc`) any `\n` would be replaced by a single space. Since `remove_extra_whitespaces` is also on by default this'll cause it just to be removed.

The problem with the `identity` setting in conjunction with NeMo is that NeMo presents the input strings for preprocessing with the final `\n` character included, and since sentencepiece itself won't add it to the vocabulary (https://github.com/google/sentencepiece/issues/804), this will cause an unknown token at the end of every training example. A minimal example to reproduce this behavior is:

Train SentencePiece model:
```
#!/bin/bash

../sentencepiece_upstream/build/src/spm_train \
  --input ../europarl-v7.de-en.en,../europarl-v7.de-en.de \
  --input_sentence_size 9999 \
  --model_prefix "bpe.joint" \
  --model_type "bpe" \
  --pad_id 3 \
  --pad_piece "<pad>" \
  --normalization_rule_name "identity"
```

```
#!/usr/bin/env bash

CUDA_VISIBLE_DEVICES= python ../nemo/examples/nlp/machine_translation/create_tarred_parallel_dataset.py \
  --shared_tokenizer \
  --encoder_tokenizer_name sentencepiece \
  --decoder_tokenizer_name sentencepiece \
  --encoder_tokenizer_bpe_dropout 0.0 \
  --decoder_tokenizer_bpe_dropout 0.0 \
  --src_fname ../europarl-v7.de-en.en \
  --tgt_fname ../europarl-v7.de-en.de \
  --out_dir preproc_dir \
  --max_seq_length 512 \
  --min_seq_length 1 \
  --tokens_in_batch 8192 \
  --lines_per_dataset_fragment 1000000 \
  --num_batches_per_tarfile 100 \
  --encoder_tokenizer_model bpe.joint.model \
  --decoder_tokenizer_model bpe.joint.model \
  --n_preproc_jobs 32 \
  --clean
```

Observe that every training example has a unknown token (id 0) right before EOS (id 2):
```
> cd preproc_dir
> tar xfv parallel.batches.tokens.8192.0.tar
> cd preproc_dir
> python
>>> import pickle
>>> pickle.load(open("fragment-0-batch-1.pkl", "rb"))["src"][0]
[   1 1336    0    2]
```

As a minimal fix this PR just removes the final newline character before encoding w/ sentencepiece's methods.

(The other tokenization option, yttm, will by default remove any trailing whitespace and there's also not setting to change that. So there won't be any issues like that.)


**Collection**: `common`

# Changelog 
- Remove ending newlines when encoding strings w/ sentencepiece tokenizer 

# Usage
* See above.

# Before your PR is "Ready for review"
**Pre checks**:
- [x] Make sure you read and followed [Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md)
- [ ] Did you write any new necessary tests?
- [ ] Did you add or update any necessary documentation?
- [ ] Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  - [ ] Reviewer: Does the PR have correct import guards for all optional libraries?
  
**PR Type**:
- [ ] New Feature
- [x] Bugfix
- [ ] Documentation


## Who can review?

Anyone in the community is free to review the PR once the checks have passed. 
[Contributor guidelines](https://github.com/NVIDIA/NeMo/blob/main/CONTRIBUTING.md) contains specific people who can review PRs to various areas.

# Additional Information