One-Size-Fits-All Multilingual Models (2020)

Abstract: This paper presents DeepSPIN’s submissions to Tasks 0 and 1 of the SIGMORPHON 2020 Shared Task. For both tasks, we present multilingual models, training jointly on data in all languages. We perform no language-specific hyperparameter tuning – each of our submissions uses the same model for all languages. Our basic architecture is the sparse sequence-to-sequence model with entmax attention and loss, which allows our models to learn sparse, local alignments while still being trainable with gradient-based techniques. For Task 1, we achieve strong performance with both RNN- and transformer-based sparse models. For Task 0, we extend our RNN-based model to a multi-encoder set-up in which separate modules encode the lemma and inflection sequences. Despite our models’ lack of language-specific tuning, they tie for first in Task 0 and place third in Task 1.

## introduction character transduction tasks such as grapheme-tophoneme conversion (g2p) and morphological inflection are important in many practical real-world applications. however, it is often difficult to train models for these tasks with deep learning techniques, due to the scarcity of labeled data for most of the world's languages. in these circumstances, it is common to use a non-neural method with a stronger inductive bias @xcite or to generate synthetic data that hopefully ameliorates the data scarcity problem. we find both of these choices unsatisfying. first, older non-neural techniques have a higher floor but also a lower ceilingprevious sigmorphon shared tasks have shown that neural methods outpace them in the presence of even moderate quantities of data @xcite . second, although data augmentation has proven helpful for morphological inflection @xcite , any data augmentation procedure makes implicit assumptions about language structure: techniques that work for western languages may fail when confronted with reduplication, vowel harmony, or non-concatenative morphology. the kinds of languages for which labeled data are scarce are precisely the languages for which nlp practitioners' assumptions are most suspect. therefore, our submissions to this shared task make use of a third alternative: multilingual training. similarly to hallucinated data, multilingual training improves results in low resource settings by acting as a regularizer. however, the models it yields are more versatile, as they are capable of good performance on several languages at the same time. we show that our technique is competitive with state-of-the-art monolingually trained models regardless of training data size for both g2p and morphological inflection. this is despite our approach having a significant disadvantage from a tuning perspective -while conventional monolingual models can tune their hyperparameters separately for each language, we use exactly the same model for each language within a submission. our contributions are as follows: ## models the common theme of the models we submit is their use of sparse functions for attention weights and output distributions, in place of the betterknown softmax @xcite . sparse functions have the following motivations: ## analysis next we consider a few questions that multilingual models raise. ## conclusion we showed that massively multilingual models are competitive with the individually-tuned state of the art for morphological inflection and g2p. we presented the first result applying entmax-based sparse attention and losses to g2p, showing that it performed with both rnn and transformer models. we release our code to facilitate further research.