SIGTYP 2020 Shared Task: Prediction of Typological Features (2020)

Abstract: Typological knowledge bases (KBs) such as WALS (Dryer and Haspelmath, 2013) contain information about linguistic properties of the world’s languages. They have been shown to be useful for downstream applications, including cross-lingual transfer learning and linguistic probing. A major drawback hampering broader adoption of typological KBs is that they are sparsely populated, in the sense that most languages only have annotations for some features, and skewed, in that few features have wide coverage. As typological features often correlate with one another, it is possible to predict them and thus automatically populate typological KBs, which is also the focus of this shared task. Overall, the task attracted 8 submissions from 5 teams, out of which the most successful methods make use of such feature correlations. However, our error analysis reveals that even the strongest submitted systems struggle with predicting feature values for languages where few features are known.

## introduction linguistic typology is the study of structural properties of languages @xcite @xcite . approaches to the categorisation of the languages of the world according to their linguistic properties are represented by, e.g., typological features in databases such as wals @xcite , uriel @xcite , and autotyp @xcite , e.g. in terms of their syntax, morphology, and phonology. one example of such a typological feature is the basic word order feature in wals. for instance, english is best described as a subject-verb-object (svo) language, whereas japanese is best described as a subject-object-verb (sov) language. once a relatively niche topic in the nlp community, studying typological features has recently risen in popularity and importance for a number of reasons. the field has seen considerable advances in cross-lingual transfer learning, whereby stable cross-lingual representations can be learned on massive amounts of data in an unsupervised way, be it for words @xcite or, more recently, sentences @xcite @xcite . this naturally raises the question of what these representations encode, and some have turned to typology for potential answers @xcite . in a similar vein, research has shown that these learned representations can be fine-tuned for supervised tasks, then applied to new languages in a few-or even zero-shot fashion with surprisingly high performance. this has raised the question of what causes such surprisingly high results, and to what degree typological similarities are exploited by such models @xcite @xcite . in addition to using typology for diagnostic purposes, prior work has also found that typology can guide cross-lingual sharing @xcite . finally, the relationship between typological knowledge bases (kbs) such as wals @xcite and language representations has been studied, which has shown that knowledge base population methods can be used to complete typological kbs @xcite @xcite , and that implications can be discovered in typological kbs @xcite . the latter stream of work has provided the inspiration for this shared task on typological feature prediction. as knowledge bases are notoriously incomplete and require manual labour from (in this case, linguistic) domain experts to create, populate and maintain, high-performance methods for automatic knowledge base population are highly desirable. while past approaches have shown the feasibility of typological feature prediction, the considered evaluation setups have some flaws which led to overestimated performance. some papers control for phylogenetic relationships between languages, e.g. not both training and testing on slavic languages, but little-to-no work has considered controlling for geographical proximity. this is corrected for in this shared task. the shared task attracted 8 system submissions from 5 teams for two subtasks (constrained and unconstrained resources). in general, the systems which make use of correlations between features, and exploit observed features during inference, perform better, whereas those that do not make use of observed features perform similarly to our baselines. ## task description the sigtyp 2020 shared task is concerned with predicting typological features from the world atlas of language structures (wals) @xcite . for the task, participants were invited to build systems to predict features for languages unseen at training time. the shared task consisted of two subtasks: 1) the constrained setting, for which only the provided training data may be used; 2) the unconstrained setting, for which training data may be extended with any external source of information (e.g. pre-trained embeddings, additional text, etc.) data format for each instance, the following information is provided: the language code, name, latitude, longitude, genus, family, country code, and features. at training time, both the feature names and feature values are given, while at test time, submitted systems are required to fill values for the requested features. an example of a test instance is given in ## results figure 2 shows the overall results and rankings for all shared task submissions. the rankings use macro-averaged accuracies as this equally weights the controlled genera (the exception is the comparison to micro-averaged accuracies in figure 3 ). this year's shared task was separated into two subtasks: constrained systems which used only the wals features and data provided, and unconstrained systems, open to use of any data or pre-trained models. accordingly, we have two winning systems: úfal for constrained, and crosslingference for unconstrained, with úfal producing the best results overall across both subtasks. results for each unobserved genus, shown in comparison to results across genera observed in training, may be found in wals feature value formatting is not standardized and, unfortunately, the test data was released containing additional tabs within the feature values for some features, which adversely affected teams who may have used tab-separation for data preprocessing. many teams accounted for this and submitted feature values for all 2417 features across the 149 languages in the test data, but for two teams this led to missing features in their submissions: crosslingference was missing 7 features across 7 languages, affecting their results by 1%; panlingua was missing 61 features across 15 languages in their rule-based submission and 57 across 11 for their additional two submissions, affecting their results by 2%. when evaluating without the affected features, rankings were not changed, nor were there significant differences between submitted systems. ## analysis our test data was constructed to enable comparison across controlled phylogenetic and geographic relationships, and randomly sampled features from languages covered in training as is common in previous work. ## related work previous work can be divided into research on predicting typological features automatically, crosslingual transfer learning which utilises typology to inform sharing, probing of representations for what typological knowledge they encode, and finally, work on how best to represent a language in terms of its typological features. ## conclusions this paper documents the first sigtyp shared task on prediction of typological features in wals. the 8 system submissions from 5 teams showed that a variety of different methods can be applied to the task. interestingly, the best system only achieved a macro-averaged accuracy of 75%, indicating that the task is far from solved. this further shows that the evaluation set-up in which we controlled for both phylogenetic relationships and geographic proximity is a challenging one. we expect that further exploration of unconstrained systems to have the most potential for predicting features in such cases, where little or nothing is known about a language. empirical methods in natural language processing, pages 4992-4997, brussels, belgium. association for computational linguistics. patrick littell, david r. mortensen, ke lin, katherine kairis, carlisle turner, and lori levin. 2017. uriel and lang2vec: representing languages as typological, geographical, and phylogenetic vectors. in proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pages 8-14. association for computational linguistics. chaitanya malaviya, graham neubig, and patrick littell. 2017. learning language representations for typology prediction. in proceedings of the 2017 conference on empirical methods in natural language processing, pages 2529-2535, copenhagen, denmark. association for computational linguistics. yugo murawaki. 2017. diachrony-aware induction of binary latent representations from typological features. in proceedings of the eighth international joint conference on natural language processing (volume 1: long papers), pages 451-461, taipei, taiwan. asian federation of natural language processing. tahira naseem, regina barzilay, and amir globerson. 2012. selective sharing for multilingual dependency parsing. in proceedings of the 50th annual meeting of the association for computational linguistics (volume 1: long papers), pages 629-637, jeju island, korea. association for computational linguistics.