The MRL 2022 Shared Task on Multilingual Clause-level Morphology (2022)

Abstract: The 2022 Multilingual Representation Learning (MRL) Shared Task was dedicated to clause-level morphology. As the first ever benchmark that defines and evaluates morphology outside its traditional lexical boundaries, the shared task on multilingual clause-level morphology sets the scene for competition across different approaches to morphological modeling, with 3 clause-level sub-tasks: morphological inflection, reinflection and analysis, where systems are required to generate, manipulate or analyze simple sentences centered around a single content lexeme and a set of morphological features characterizing its syntactic clause. This year’s tasks covered eight typologically distinct languages: English, French, German, Hebrew, Russian, Spanish, Swahili and Turkish. The tasks has received submissions of four systems from three teams which were compared to two baselines implementing prominent multilingual learning methods. The results show that modern NLP models are effective in solving morphological tasks even at the clause level. However, there is still room for improvement, especially in the task of morphological analysis.

## introduction universality is an important premise of many morphological datasets and shared tasks. recent shared tasks of sigmorphon have introduced the notion of comparative analysis in morphological studies by incorporating up to 100 languages @xcite in their evaluation benchmark, by providing all of them with data that is annotated according to a single universal schema @xcite . systems that succeed in these tasks ideally should boast in their ability to handle various morphological phenomena observed in almost any language family on earth @xcite . however, as pointed out recently by @xcite , the perceived universality of morphological tasks is impaired by the lack of a working definition of a morphosyntactic word @xcite . without such definition, the boundary between morphology and syntax is blurred and the assignment of linguistic phenomena to either morphological or syntactic data results in inconsistency across languages. thus, limiting the scope of morphological tasks to white-spaced words creates an undue advantage to some languages based on their grammarian traditions, typological characteristics, and some other arbitrary factors. for example, some languages, like english, are considered isolating and have word-level inflection tables of tiny size, while other languages, like turkish, are considered agglutinative and have huge inflection tables. however, isolating and agglutinative languages largely differ orthographically rather than linguistically, as both types concatenate pieces of text. the universal benchmark presented here allows testing of both models and theories while ignoring orthographic characteristics like whitespaces and treats equally languages with varying typological characteristics. in this shared task, we operationalize a more universally applicable and comprehensive approach to morphology by liberating the evaluated tasks from the ill-defined formal restrictions dictated by whitespaces. we start with a fix universal set of inflectional features foot_0 and inflect lemmas in all languages to all possible combinations of features, disregarding the number of white-spaced words required to express them orthographically. the features define fully-saturated clauses and the result is a data set of clauses organized in inflection tables and tasks that go beyond the word-level and include phenomena considered syntactic, such as word order manipulation and the like. we can thus test the submitted systems' ability to cope with these phenomena. the shared task includes 3 sub-tasks: inflection, where systems are to generate simple clauses from a lemma and a set of morphological features; reinflection, where systems should manipulate a source clause to a target clause; and analysis, where the task is to output a lemma and a set of features given a clause. see all three sub-tasks include evaluation data annotated in the following eight languages: english, french, german, hebrew, russian, spanish, swahili and turkish, from four different language families. the variety of languages induces a plethora of alternations to be modeled by the submitted systems, from pronoun incorporation in swahili to verb-splitting german, from ablautextensive semitic morphology to highly agglutinative turkish. however, in terms of dimensions of meaning, the data is extremely uniform across languages as all morphological features are implemented in our data if they are implemented in the language. the results included in this shared task compare 4 submitted systems and 2 baseline systems with various characteristics, from rule-based systems to systems based on a large pretrained language models. the best performing system outperforms the best baseline and reduces the error rates by 3 to 8 fold, depending on the sub-task. future editions of the task are intended to further expand the number of languages and the scope of the data for better alignment with real-world phenomena and distributions. ## the tasks this shared task consists of three sub-tasks which test the ability of systems to deal with clause-level morphological data in multiple languages. in this section we define and formalize the tasks, and in ## the data the data included in this task is based on the mightymorph data set presented by @xcite . the data for four of the languages was prepared in prior work, and in this shared task we have doubled the number of languages to include eight languages in total from four language families. for most languages the data was created by expanding the unimorph @xcite wordlevel inflection tables into respective clauses that saturate all the required arguments of the verbal lemma. this was done in two phases. initially, we used a language-specific rule-based grammar that included the inflection tables of any relevant auxiliaries in order to construct all possible periphrastic constructions of the inflected verb. for example, when constructing the future perfect form for the english verb receive, equivalent to the features ind;fut;prf, we used the past participle from the unimorph inflection table received and the auxiliaries will and have to construct will have received. we then manually determined which arguments each verb can take in order to generate a fullysaturated clause. to retain the tasks with a single lemma, all arguments are realized as pronominal features. for example, the english verb receive has 2 possible argument combinations: {nom, acc} and {nom, acc, abl}, equivalent to sentences like "i received it" and "i received it from you", respectively. for each argument combination we exhaustively generated all suitably cased pronouns without regarding the semantic plausibility of the resulted clause. turkish and swahili are somewhat exceptional to the process described above in the sense that the clause-level tables were constructed solely by grammars of morphemes without relying on the unimorph word-level tables. in addition to using unimorph we generated the french data based on the lefff @xcite , which is a large-coverage and freely available morphological and syntactic lexicon for french. in contrast with the other languages, the types of arguments and their combinations for each verb was not determined manually but automatically with the lefff. the auxiliary allowed for each verb was also decided using the lefff. the result is a fully-populated clause-level inflection table, where each entry in the table is structured as (lemma, f eatures, f orm). see ## results tables 4, 5 and 6 summarize the results per system for the inflection, reinflection and analysis tasks, 5 https://spacy.io/ all systems significantly outperformed the finetuned mt5 which is the strongest baseline. the systems submitted by the kuis-ai team rank first in all tasks, both in terms of the main evaluation metric used in each task(edit distance or f1) and in terms of the exact match accuracy. comparing the performance of all systems over all tasks in terms of exact match accuracy, it is clearly shown that inflection is the easier task of the three. since reinflection can be conceptually and practically decomposed to an analysis followed by an inflection operation, one can hypothesize that the under-performance in this task stems from the difficulty of the analysis operation. figures 1 and 2 average the performance over all systems to gain some insights into the relative difficulty of the tasks in the various languages. the trends in the different tasks point to different languages as being more or less difficult. for example, swahili was one the toughest languages in the analysis task but one of the easiest in inflection and reinflection, while the opposite is true for russian. systems also tended to under-perform in vocalized hebrew in both inflection and reinflection, pointing to the complexity of the semitic inflectional system. however, in the analysis task, performance over vocalized hebrew was actually better than that over the unvocalized version, probably due to the ease of disambiguation when vowels are written. interestingly, the inclusion of the swahili language drove down the overall result of the göttingen system in the analysis task, potentially depriving it from leading the table. this points both to the importance of inclusion of low-resourced languages in multilingual tasks, and also to the limits of rule-based systems that may be dependent on the knowledge of their designers of the languages at hand. ## conclusion and future directions the first shared task on multi-lingual clause-level morphology proposed novel means for modeling the evaluation of morphosyntactic representations in a more universally inclusive setting. the multilingual and typologically diverse nature of the data used in the construction of the benchmark allows its usage in comparative studies from different fields and schools. apart from including more languages, future shared tasks should take into account the overall good, even if not perfect, performance of the systems and try to tease apart the characteristics that make morphological tasks easier in order to figure out whether they are justified. an example for that may be the invariability in morphological data, compared to other nlp tasks such as translation. @xcite pointed to the remarkably different behavior of models in decoding language from morphological data, specifically to the sufficiency of greedy decoding. this is not surprising due to the conceptualization of morphological data as containing a single inflected form for every bundle of inflectional features. however, on the clause-level such one-to-one mapping is less justified, as speakers can vary the word order of a sentence or the grammatical construction chosen to pronounce the same meaning. hence, future shared tasks could allow multiple realizations of feature bundles, making the decoding more complicated. semantic plausibility is another factor that was largely ignored in creating the data for this shared task. this path was chosen in order to test the systems' ability to recreate the human grammar that is well able to produce implausible sentences. however, different settings can take this factor into account so systems will not be punished for failures to predict sentences are not used in practice. finally, while this task included only clauses with verbal head, future tasks may include nominal and adjectival clauses as well. however, different languages use different means to express tenses in this kind of clauses, so this requires a careful linguistic treatment of copulas in comparison to (partially) zero-copula languages like turkish and hebrew. to conclude, the shared task showed that modern nlp models, whether relying on pretrained models or not, are capable of solving clause-level morphological tasks to a large extent. still, there is room for improvement, both in the systems' ability to analyze data and in terms of the data included in these tasks.