ITN: Recursion Error for Long Texts
State: closed | Pull request: no
Labels: name: bug

name: TN/ITN

**Describe the bug**

When (inverse) normalizing text using the InverseNormalizer's `normalize` function, a recursion error is raised when the number words in the input surpasses a threshold.

**Steps/Code to reproduce bug**
Running the following script produces the error when `len(t)=994`.

```
from nemo_text_processing.inverse_text_normalization.inverse_normalize import InverseNormalizer
from pynini.lib import pynutil
import pynini


if __name__ == '__main__':
    itn = InverseNormalizer(lang='en')

    for length in range(990, 999):
        t = " ".join(['a' for i in range(0, length)])
        print(f"Testing input of length {len(t.split())} words.")
        itn.inverse_normalize(t, verbose=False)
```

A recursion error is raised once `length=994`:

```
>> python recursion_error.py
[NeMo I 2022-01-03 12:35:00 tokenize_and_classify:68] ClassifyFst.fst was restored from ./cache/_en_itn.far.
Testing input of length 990 words.
Testing input of length 991 words.
Testing input of length 992 words.
Testing input of length 993 words.
Testing input of length 994 words.
Traceback (most recent call last):
  File "recursion_error.py", line 12, in <module>
    itn.inverse_normalize(t, verbose=False)
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/inverse_text_normalization/inverse_normalize.py", line 91, in inverse_normalize
    return self.normalize(text=text, verbose=verbose)
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 143, in normalize
    for tagged_text in tags_reordered:
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 213, in _helper
    yield from _helper(prefix + token_option, tokens, idx + 1)
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 213, in _helper
    yield from _helper(prefix + token_option, tokens, idx + 1)
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 213, in _helper
    yield from _helper(prefix + token_option, tokens, idx + 1)
  [Previous line repeated 990 more times]
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 211, in _helper
    token_options = self._permute(tokens[idx])
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 178, in _permute
    rec = self._permute(v)
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 176, in _permute
    subl = ["".join(x) for x in itertools.product(subl, [f"{k}: \"{v}\" "])]
  File "/home/lhr/.conda/envs/nemo-text-process/lib/python3.8/site-packages/nemo_text_processing/text_normalization/normalize.py", line 176, in <listcomp>
    subl = ["".join(x) for x in itertools.product(subl, [f"{k}: \"{v}\" "])]
RecursionError: maximum recursion depth exceeded while calling a Python object
```

**Expected behavior**

Normalization supported for sequences of arbitrary length.

**Environment overview (please complete the following information)**

 - Environment location: venv on local machine
 - Method of NeMo install: pip install nemo-toolkit (v. 1.5.1)

**Environment details**

- OS version - Ubuntu 18.04.5 LTS
- PyTorch version - 1.10.1
- Python version - 3.8.11