Export of rules to sparrowhawk
State: closed | Pull request: no
Labels: name: feature

**Is your feature request related to a problem? Please describe.**

Paper for (inverse) text normalization describes that its easy to deploy developed text (de)normalization rules into prod using sparrowhawk. I havent found any documentation for that though. So far I figured how to export grammars as far files consumed by sparrowhawk:
```
    normalizer = Normalizer(input_case="cased")

    tag_exp = grm.Exporter(args.tagger)
    tag_exp['TOKENIZE_AND_CLASSIFY'] = normalizer.tagger.fst
    tag_exp.close()

    verb_exp = grm.Exporter(args.verbalizer)
    verb_exp['ALL'] = normalizer.verbalizer.fst
    verb_exp.close()
```

But when I try to apply those rules, verbalization fails.
here is normalization using default sparrowhawk grammars shipped as part of documentation:
```
# cat test.txt 
Hello 13 world
# normalizer_main --config=sparrowhawk_configuration.ascii_proto --multi_line_text < test.txt
[DEBUG:normalizer.cc:162] Verbalizing: ["13(r
13]
[DEBUG:normalizer.cc:195] Verbalize output: Words
Hello thirteen world
```
This is how it fails using grammars from NeMo:
```
[DEBUG:normalizer.cc:162] Verbalizing: ["13(r

thirteen]
[ERROR:rule_system.cc:121] Application of rule "ALL" failed[ERROR:normalizer.cc:213] Failed to verbalize "rthirteen"[WARNING:normalizer.cc:167] First-pass verbalization FAILED for ["13(rthirteen][ERROR:rule_system.cc:121] Application of rule "ALL" failed[ERROR:normalizer.cc:213] Failed to verbalize "�13"[ERROR:normalizer.cc:180] Verbalization FAILED for ["13(rthirteen][DEBUG:normalizer.cc:195] Verbalize output: Words
Hello world
```
 I think this might be due to fact that parsing of tokenize_classify output is somehow incompatible. Is it the case? is there any tutorial?

**Describe the solution you'd like**

Documentation on how to export NeMo grammars to sparrowhawk as paper suggests.

**Describe alternatives you've considered**

Any pointers to debug the issue, maybe some simple grammar that actually works in export.


important fix - to export not normalizer.verbalizer.fst (which is VerbalizeFinalFst), but VerbalizeFst. There are a lot of other inconsistencies, but they seem to be related to serialization indeed:

failed to verbalize: "time { hours: "three" minutes: "thirty" }", because Unable to convert string to int32. I guess this is typing in proto.

and another:

[ERROR:protobuf_parser.cc:183] Unknown field: [integer_part]
[ERROR:protobuf_parser.cc:342] Full input: [tokens { name: "mister" } tokens { name: "Snookums" } tokens { name: "was" } tokens { name: "on" } tokens { name: "the" } tokens { name: "train" } tokens { name: "carrying" } tokens { money { currency: "dollars" integer_part: "forty"  fractional_part: "two five" } } tokens { name: "(" } tokens { money { currency: "pounds" integer_part: "thirty"  fractional_part: "six o" } } tokens { name: ")" } tokens { name: "of" } tokens { name: "Belgian" } tokens { name: "chocolate" } tokens { name: "in" } tokens { name: "a" } tokens { measure { cardinal { integer: "three" } units: "kilograms" } } tokens { name: "box" } tokens { name: "that" } tokens { name: "was" } tokens { measure { cardinal { integer: "twenty" } units: "centimeters" } } tokens { name: "wide." }]
[ERROR:normalizer.cc:131] Failed to parse tokens from FST for "Mr. Snookums was on the train carrying $40.25 (£30.60) of Belgian chocolate in a 3kg box that was 20cm wide."

The documentation could be found [here](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/v1.0.0/tools/text_processing_deployment.html) and tutorials: [Text normalization](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Text_Normalization.ipynb) and [ITN](https://colab.research.google.com/github/NVIDIA/NeMo/blob/main/tutorials/text_processing/Inverse_Text_Normalization.ipynb)

@balacoon Did the instructions @ekmb helped you solve your problem?

yes, perfect. thanks!