Byte-Level Massively Multilingual Semantic Parsing (2022)

Abstract: Token free approaches have been successfully applied to a series of word and span level tasks. In this work, we evaluate a byte-level sequence to sequence model (ByT5) on the 51 languages in the MASSIVE multilingual semantic parsing dataset. We examine multiple experimental settings: (i) zero-shot, (ii) full gold data and (iii) zero-shot with synthetic data. By leveraging a state-of-the-art label projection method for machine translated examples, we are able to reduce the gap in exact match to only 5 points with respect to a model trained on gold data from all the languages. We additionally provide insights on the cross-lingual transfer of ByT5 and show how the model compares with respect to mT5 across all parameter sizes.

## introduction semantic parsers map natural languages utterances into logical forms (lfs). in the context of conversational agents @xcite , robotics @xcite or question answering systems @xcite , task-oriented semantic parsers map user queries (e.g. "set an 8 am alarm") to machine readable lfs (e.g. [in:create_alarm [sl:time 8 am ]]), in the form of structured interpretations that can be understood and executed by downstream components. learning parsers requires training data in the form @xmath0, @xmath1. such data is costly to obtain especially at large scale @xcite , since expert annotators have to derive the correct lfs given an input utterance. this problem is exacerbated in a multilingual setting, where the availability of annotators, especially for non top-tier languages, is scarce and therefore even more expensive. with the release of massive @xcite , the research community has now access to a massively multilingual semantic parsing dataset that can be used to evaluate large language models fine-tuned on the task and to study cross-lingual transfer for numerous languages. on the multilinguality front, token-free models with byte or character based vocabularies have gained strength given their competitiveness with respect to traditional subword-based pretrained language models. models such as byt5 @xcite , canine @xcite and the charformer @xcite have been applied to popular multilingual benchmarks obtaining state-ofthe-art results. in this paper, we perform the first in-depth evaluation of a token-free model in the context of multilingual semantic parsing. we compare the byt5 and mt5 @xcite models across different parameter sizes and data regime settings. in addition to that, we build a map of the cross-lingual transfer for all the languages in massive. lastly, we show that with the use of machine translated synthetic data the accuracy of a state-of-the-art multilingual parser can be just 5 points lower than the same parser trained with all the available multilingual supervision. to incentivize research on synthetic data augmentation approaches, we release the massive english training utterances translated to 50 languages. 1 2 the massive dataset @xcite is a semantic parsing dataset covering 51 languages, 18 domains, 60 intents and 55 slots. the dataset was created by professional translators starting from the english slurp dataset @xcite . a significant portion of the translations have been localized too, following the recent trend in multilingual benchmarks of replacing western-centric entities with entities that are more relevant for the target languages @xcite @xcite . ## experiments we use massive as a test bed for two model families, byt5 and mt5, evaluating them at all sizes in three different data settings. we report intent accuracy (ia) and exact match (em) accuracy. we do not perform any hyper-parameter tuning: we train for 30k steps with a fixed learning rate of 0.0001 and a batch size of 128 for all models but xxl, for which batch size was reduced to 32. we run fine tuning on cloud tpu v3 with an input/target length of 1024/512 for byt5 and 512/512 for mt5. to minimize compute, all the reported results are from single runs. we experiment with three different settings, summarized below: training is performed on english data only, and the model selection is done on the english development set. results are reported in we can see a pattern that is common to all the experiments: at smaller sizes, byt5 has much better em accuracy then the corresponding mt5 models. as stated in @xcite , this may be explained by the fact that at these sizes less than 0.3% of byt5 parameters are locked in embedding tables and a larger amount of dense parameters is updated during training. mt5 parameters are instead dominated by the embedding tables, which are updated less often than the dense layers. in addition to that, byt5-large is worse than byt5-base at span labeling, which is a word level task. both our observations confirm the findings in @xcite . in the synthetic data setting ( ## additional experiments and results in zero-shot evaluations, english is the most studied language given the availability of labeled data. recent work has shown that this language may not be the best at cross-lingual transfer @xcite . since massive provides training and test data for all its languages, we can evaluate the zero-shot performance of each language. we train 51 byt5-base model for a fixed number of steps in the appendix, we report the accuracy for each individual intent on the union of the test set exbest to worst donor fr, de, es, nl, pl, • • •, mn, am, sw, km, cy receiver en, de, pt, fr, sv, • • •, zh, am, mn, sw, cy finally, we compare our nmt translations of the training set with the corresponding gold translations produced by professional translators. we summarize the most interesting information in ta multilingual models are architecturally similar to monolingual transformer-based models but they are pretrained on multilingual corpora. these models include xlm @xcite , xlm-r @xcite and mt5 @xcite , the multilingual version of t5 @xcite . they all use a subword vocabulary, a choice that may result in poor performance for languages with limited amount of data @xcite . token-free models such as byt5 @xcite , canine @xcite and charformer @xcite were designed to avoid this issue and have been applied to popular multilingual benchmarks obtaining state-of-the-art results. in this work, we compare the multilinguality and the generative capabilities of mt5 and byt5 in a massively multilingual semantic parsing task. data augmentation is the process of creating synthetic labeled data from available annotated examples. one approach in the multilinguality space is to translate annotated data in one language, e.g. english, to other languages. neural machine translation is a strong baseline as it has been shown in recent cross-lingual evaluation benchmarks @xcite . while translation works quite well for classification tasks where the label is at instance level, sequence tagging or parsing tasks require an annotation projection step because labels are at token level. translate-and-align methods use bilingual word aligners, statistical @xcite @xcite , and neural figure 2: zero-shot em accuracies of individual byt5-base models fine-tuned on a single language (y-axis) and evaluated on dev sets from all languages (x-axis). @xcite @xcite . more recent works removes this explicit alignment requirement @xcite @xcite . in our work, we use a label projection method based on pretrained language models @xcite that reconstructs a full semantic parse from an utterance and a signature of the same parse. ## conclusions in this paper, we evaluated byt5 and mt5 @xcite models in a massively multilingual semantic parsing task, showing that byt5 is particularly competitive at smaller sizes. we have provided a map of the cross-lingual transfer for all the languages in massive and demonstrated that synthetic examples created with nmt are effective for building accurate semantic parsers.