Transformers on Multilingual Clause-Level Morphology (2022)

Abstract: This paper describes the KUIS-AI NLP team’s submission for the 1st Shared Task on Multilingual Clause-level Morphology (MRL2022). We present our work on all three parts of the shared task: inflection, reinflection, and analysis. We mainly explore two approaches: Trans- former models in combination with data augmentation, and exploiting the state-of-the-art language modeling techniques for morphological analysis. Data augmentation leads to a remarkable performance improvement for most of the languages in the inflection task. Prefix-tuning on pretrained mGPT model helps us to adapt reinflection and analysis tasks in a low-data setting. Additionally, we used pipeline architectures using publicly available open-source lemmatization tools and monolingual BERT- based morphological feature classifiers for rein- flection and analysis tasks, respectively. While Transformer architectures with data augmentation and pipeline architectures achieved the best results for inflection and reinflection tasks, pipelines and prefix-tuning on mGPT received the highest results for the analysis task. Our methods achieved first place in each of the three tasks and outperforms mT5-baseline with 89% for inflection, 80% for reflection, and 12% for analysis. Our code 1 is publicly available.

## introduction the shared task on multilingual clause-level morphology was designed to provide a benchmark for morphological analysis and generation at the level of clauses for various typologically diverse languages. the shared task is composed of three subtasks: inflection, reinflection and analysis. for the inflection task, participants are required to generate an output clause, given a verbal lemma and a specific set of morphological tags (features) as an input. in the reinflection task the input is an inflected clause, accompanied by its features (tags). participants need to predict the target word given a new set of tags (features). finally, the analysis task requires predicting the underlying lemma and tags (features) given the clauses. literature has examined morphology mainly at the word level, but morphological processes are not confined to words. phonetic, syntactic, or semantic relations can be studied at phrase-level to explain these processes. thus, this shared task examines phrase-level morphology and questions the generalization of the relations between the layers of language among languages with different morphological features. the shared task includes eight languages with different complexity and varying morphological characteristics: english, french, german, hebrew, russian, spanish, swahili, and turkish. in our work, we explored two main approaches: (1) training character-based transformer architectures from scratch with data augmentation, (2) adapting a recent prefix-tuning method for language models at multilingual morphological tasks. ## methods in this section, first we cover the model architectures and training strategies that we have used @xcite @xcite , and then discuss our data augmentation strategies in details @xcite . ## related work word-level morphological tasks have been studied to a great extent, with lstm @xcite @xcite , gru @xcite , variants of transformer @xcite ; @xcite and other neural mod-els (e.g., invertible neural networks (sahin and gurevych, 2020)). unlike word-level, there is limited work on clause-level morpho-syntactic modeling. @xcite presents a new dataset for clause-level morphology covering 4 typologically-different languages (english, german, turkish, and hebrew); motivates redefining the problem at the clause-level to enable the crosslinguistical study of neural morphological modeling; and derives clause-level inflection, reinflection, and analysis tasks together with baseline model results. pre-trained llms have been successfully applied to downstream tasks like sentiment analysis, question answering, named entity recognition, and part-of-speech (pos) tagging @xcite @xcite . even though, there is limited work on applications of llms to morphological tasks, it has been demonstrated that using pre-trained contextualized word embeddings can significantly improve the performance of models for downstream morphological tasks. @xcite explored bert-based classifiers for training morphosyntactic tagging models for arabic and its dialect. anastasyev (2020) explored the usage of elmo and bert embeddings to improve the performance of joint morpho-syntactic parser for russian. hofmann et al. (2020) used a finetuning approach to bert for the derivational morphology generation task. finally, @xcite presented a large pre-trained language model for modern hebrew that shows promising results at several tasks. on the other hand, since fine-tuning llms requires to modify and store all the parameters in a lm that results with a huge computational cost. rebuffi et al. (2017); houlsby et al. (2019) used adapter-tuning which adds task-specific layers (adapters) between the each layer of a pre-trained language model and tunes only the 2%-4% parameters of a lm. similarly, li and liang (2021) proposed prefix-tuning which is a light-weight alternative method for adapter-tuning that is inspired by prompting. ## conclusion in this paper, we described our winning methods multilingual clause-level morphology shared task for inflection, reinflection, and analysis. due to the different complexity between tasks and the varying morphological characteristics of languages, there is no single best model that achieves the best results for each task in each language. thus, we try to implement different types of systems with different objectives. for inflection we used a vanilla transformer adapted from @xcite and applying data hallucination substantially improves accuracy @xcite . the reinflection task is more challenging compared to the other tasks due to its complex input form. to overcome this issue, we have removed the original feature tags from the input. we only used the inflected clause and target features in the input. we again used a vanilla transformer as a model choice. finally, for the analysis task, we used the prefixtuning method based on mgpt. on average, we have achieved the best results for every three tasks among all participants.