NEMO : Frequentist Inference Approach to Constrained Linguistic Typology Feature Prediction in SIGTYP 2020 Shared Task (2020)

Abstract: This paper describes the NEMO submission to SIGTYP 2020 shared task (Bjerva et al., 2020) which deals with prediction of linguistic typological features for multiple languages using the data derived from World Atlas of Language Structures (WALS). We employ frequentist inference to represent correlations between typological features and use this representation to train simple multi-class estimators that predict individual features. We describe two submitted ridge regression-based configurations which ranked second and third overall in the constrained task. Our best configuration achieved the microaveraged accuracy score of 0.66 on 149 test languages.

## introduction the rapidly developing field of computational lin guistic typology @xcite is becoming in creasingly popular in natural language processing (nlp) research @xcite , where typological corpora such as the world at las of language structures (wals) @xcite and autotyp @xcite are seeing increased use @xcite . despite their popular ity, typological corpora are very sparse. accord ing to @xcite , less than 15% of the feature values are present for the 2,679 languages represented in wals. these databases are humancurated and depend on grammatical de scriptions for their development; sparsity is often due to linguistic sources lacking data on particular features for a given language. developing method ologies for accurately predicting missing typolog ical features on the basis of existing knowledge is therefore crucial for a wider adoption of typologi cal resources in nlp tasks and beyond @xcite . this paper presents the work done by the "nemo team" (google london and tokyo) on the constrained subtask for the sigtyp 2020 shared task @xcite . we experimented with a variety of machine learning models using only the features provided in the training, devel opment and test sets. our features included ge netic features (genus and family), areal features (clusters of languages within a radius of the tar get language), and derived implicational univer sals. originally introduced by @xcite and demonstrated to capture the syntactic typology well @xcite , sim ilarly to others @xcite we use the framework of universal implications to capture correlations between other types of typo logical features as well. we describe how these features were derived in detail in section 3. as we report below in section 4, the per formance of machine learning algorithms varied across different feature predictions, with some al gorithms working better for some features, and less well for others. on balance however we found that ridge regression @xcite , also known as tikhonov regularization @xcite , was the most useful approach. ## related work here we review the approaches corresponding to the constrained subtask of the shared task, where no external data, such as unlabeled texts, is al lowed. a popular approach to modeling the typologi cal diversity of the world's languages is based on bayesian probabilistic inference. despite recently drawing some criticism on linguistic grounds by @xcite , this approach possesses impressive explanatory power. in what possibly represents the earliest modelbased typological feature imputer, @xcite introduced the probabilistic bayesian model for uncovering uni versal implications in wals data by associating random variables with individual wals features and discovering the interfeature correlations from statistical dependence between random variables. the modeling power of the bayesian approach was further demonstrated by @xcite in a non parametric hierarchical bayesian model combin ing linguistic areas and phylogeny. @xcite proposed a deep learning ap proach to phylogenetic inference by mapping the language vectors to a latent space using an auto encoder trained using typologicallyinspired objec tive on wals data with missing values imputed us ing a regularized iterative variant of multiple cor respondence analysis (mca) @xcite . in later work, @xcite and @xcite abandoned an earlier model in favor of a bayesian autologistic approach and demonstrated the superi ority of bayesian predictor over mca. in this ap proach, the languages are represented as random variables that are explained in terms of other lan guages related to each other through phylogenetic and spatial neighborhood graphs. @xcite introduce a generative model inspired by the chomskyan principlesandparameters frame work, drawing on the correlations between typo logical features of languages to tackle the novel task of typological collaborative filtering, a con cept borrowed from the area of recommender sys tems. while most stateoftheart missing feature im putation methods are modelbased, recently @xcite employed the iterative tech nique based on singular value decomposition (svd) from the wellstudied area of lowrank ma trix completion and reported the performance on par with the prior art. although lacking explana tory power, similarly to mca, such techniques are attractive due to their simplicity and computational efficiency. @xcite took a stan dard machine learning approach by training multi nomial logistic regression classifiers for individ ual wals features based on other features present in the database under various experimental condi tions, hypothesizing that the classifier would cap ture feature correlations implicit in the data. the frequentist approach we take is similar in some respects to work of @xcite in that we also train vector space classifiers, but there are a few notable differences, we: (i) use typologicallymotivated "probabilistic" frequency based input space by explicitly representing areal and phylogenetic associations and implicational universals, and (ii) explore a wider range of clas sification approaches. although for this task there is a special inter est in the geographic aspect of the modeling, for our final submission we limited ourselves to the rather orthodox approach of representing language associations through fixed neighborhoods, in or der not to overcomplicate our method (described in the next section) that can be difficult to rec oncile with the more sophisticated models, such as the very promising model of language evolu tion from @xcite and the findings emerging from the fields of dialectology and di alectometry @xcite . ## method here we outline the details of our approach used to generate the final submission. the opensource implementation of our training and evaluation pipeline has been released in public domain. foot_0 ## experiments and discussion in what follows, we provide a brief overview of the datasets, introduce the baseline systems we evalu ated against during the development of our method, provide the evaluation results of miscellaneous ma chine learning algorithms on the development set that guided our final model selection, describe the two configurations submitted to the constrained subtask and, finally, mention the approaches that did not work well in our settings. ## conclusion we have presented the nemo submission to the 2020 sigtyp shared task. our system used ge netic, geographical and automatically deduced im plicational universals, and a range of classifiers of which ridge regression yielded the best overall performance. our method achieved 0.66 overall accuracy on the task, compared to the best base line provided by the organizers, which had an ac curacy of 0.51. our system tended to do better on test language families that were better represented in the training and development data, though in terestingly the correlation between the number of instances of features in the training/development data and performance on that feature was not strong. as discussed above, we include in the training package methods for ensembling classifiers so that one can find perfeature optimal classifiers, but we did not make use of this functionality for our sub mitted results. we do think, however, that further experimentation with this functionality would be useful. finally, as one of the reviewers notes, it would be interesting to ask what use work along the lines presented here could be to the field lin guist/typologist who is interested in testing po tential relationships between languages based on shared features. many of the systems reported in section 2 had this aim in mind. while we do not want to overstress the value of this sort of work compared to good linguistic intuition, we do think that knowing the bayesian priors of associations, as we discussed in section 3.1 could at least serve as a reminder that some feature settings may be shared simply because they are very common any way.