Predicting Typological Features in WALS using Language Embeddings and Conditional Probabilities: ÚFAL Submission to the SIGTYP 2020 Shared Task (2020)

Abstract: We present our submission to the SIGTYP 2020 Shared Task on the prediction of typological features. We submit a constrained system, predicting typological features only based on the WALS database. We investigate two approaches. The simpler of the two is a system based on estimating correlation of feature values within languages by computing conditional probabilities and mutual information. The second approach is to train a neural predictor operating on precomputed language embeddings based on WALS features. Our submitted system combines the two approaches based on their self-estimated confidence scores. We reach the accuracy of 70.7% on the test data and rank first in the shared task.

## introduction the world atlas of language structures (wals) @xcite is a database of over 2,000 languages, which lists structural properties ('features') of each language, gathered from reference grammars. the properties/features are phonological (such as the number of distinct consonants), grammatical (such as morphological devices or dominant word order) and lexical (such as the inventory of lexemes for colors). wals can be browsed online 1 and the database is also available for download. it has been used in cross-lingual nlp to identify languages that are grammatically similar, and to cope with expected dissimilarities @xcite . unfortunately the wals database is sparse as many feature values for many languages are missing. the goal of the present shared task is to predict the missing features with the help of the information that is known. some typological properties are mutually dependent @xcite and im-1 https://wals.info/ plications among the feature values can be found @xcite . we try to learn these implications from the wals data using a combination of two machine learning approaches. after summarizing the task setup in section 2, we describe our systems in section 3, including models which we tested but did not use in the final submission. we thoroughly evaluate the systems in section 4. ## task and data the sigtyp 2020 shared task @xcite splits the wals data into training, development and test portion. in the blind test data, some feature values are masked and the participating system is supposed to predict them based on the remaining features that are left visible. the task specification envisions a constrained and an unconstrained track, where the constrained systems can use only the provided wals data, while an unconstrained system can use additional external resources, such as texts or pre-trained word vectors. we stay within the constrained track as for most languages we do not have any other data anyway. the training data consists of 1,125 languages, the development data has 83 and the test data has 149 languages. each language has 7 general properties, which are always filled in: the wals language code, language name, genus, family, latitude, longitude and country codes (more than one country can be listed for a language). then there are up to 185 linguistic properties. no language has all of them and no linguistic property has a known value for all languages. within the training data, the best described language is english with 159 linguistic properties; the least described languages have just four linguistic properties each, and the median is 28. the most covered linguistic property is 83a order of object and verb, filled in for 785 languages; the median is 168 languages, and the two least covered properties appear with 8 languages each. we refer to all language properties (general and linguistic) as features. to be able to use the development data for evaluation, we randomly masked about one half of the linguistic properties (equivalent to about 42% of all features) and let the system predict them. on average, a language had 26.9 non-empty features and 19.2 features to predict. minimum knowledge was 8 non-empty features (i.e., the 7 general properties plus one linguistic property), with 3 features to predict. note that the organizers did not reveal in advance what would be the proportion of known and unknown features in the blind test data; in the end it turned out to be very similar to the proportion we used in our development data. ## systems we tried several approaches to the problem, two of which yielded promising results. we refer to them as the probabilistic system (section 3.1) and the neural system (section 3.2). our final setup is a combination of the two (section 3.3); the output of this setup was submitted to the shared task evaluation. we also briefly mention other setups that we considered and abandoned because they did not fare well in preliminary evaluation (section 3.4). ## evaluation the evaluation metric is a simple accuracy: the number of correctly predicted feature values divided by the number of masked features. failure to predict a value of a masked feature would count as an incorrect prediction. nevertheless, our systems always predict all masked features, even if there are few reliable clues and the system's confidence is low. the accuracies of different setups are given in accuracies computed on individual languages are skewed because for some languages the system had to predict only one feature. if we look at development languages where 10 or more features were masked, our probabilistic system achieved 100% on four of them (ukrainian, uyghur, wan and venda); at the other end of the scale, accuracy on the south american language uru is only 30%. the system never failed on 34 features; the most frequent of them is unsurprisingly (see section 3.1) minor morphological means of signaling negation, followed by order of relative clause and noun and postnominal relative clauses. note however that these observations do not necessarily indicate that a feature is easy or difficult to predict. the success depends heavily on how frequently the feature co-occurs with a suitable correlated feature in wals, and whether our random masking selected the feature in presence or absence of the correlated feature. ## conclusion we have described three systems (two individual systems and one combined system) that use the known features of a language to predict its remaining features. the first system is based on conditional probability of feature values, log frequency and mutual information. it achieves an accuracy of 73.81% on the development data. the second system uses a neural network to train language and feature embeddings, and to predict a feature value's probability based on the embeddings. it achieves slightly better results than the probabilistic model on the development data, but it is slightly worse on the test data. finally, we use the scores produced by the two systems together with their predictions to decide which of the systems knows the correct answer in which situation. this combined system achieves 75.50% on the development data and 70.75% on the test data. our code is available at https://github.com/ ufal/st2020 .