The Past, Present, and Future of Typological Databases in NLP (2023)

Abstract: Typological information has the potential to be beneficial in the development of NLP models, particularly for low-resource languages. Unfortunately, current large-scale typological databases, notably WALS and Grambank, are inconsistent both with each other and with other sources of typological information, such as linguistic grammars. Some of these inconsistencies stem from coding errors or linguistic variation, but many of the disagreements are due to the discrete categorical nature of these databases. We shed light on this issue by systematically exploring disagreements across typological databases and resources, and their uses in NLP, covering the past and present. We next investigate the future of such work, offering an argument that a continuous view of typological features is clearly beneficial, echoing recommendations from linguistics. We propose that such a view of typology has significant potential in the future, including in language modeling in low-resource scenarios.

## introduction linguistic typology is concerned with investigating languages based on their properties, including similarities and differences in grammar and meaning @xcite . this type of systematic investigation of language allows for comparing and contrasting languages, providing insights into language, human cognition, and allowing for computational linguistic investigations into language structures. furthermore, typological properties offer a promising avenue to enabling truly low-resource language technology -efficiently using what we know about languages may be the key to effective transfer learning, allowing language models to be as impactful for smaller communities as they are for the global elite. typological information is meticulously organised in resources such as the world atlas of language structures (wals) and grambank. the information in these databases is typically gathered from other resources, e.g., provided by field linguists in the form of grammars. however, these databases are severely limited by two factors: (i) they contain significant disagreements with one another, implying classification errors, and denoting a hurdle to investigation of language variation across languages and continents; and (ii) they explicitly enforce a categorical view of language, where a continuous view may be more appropriate. in this paper, we take a long-term view on typology in the context of computational linguistics, nlp, and large language models. we outline past work in the intersection of linguistic typology and nlp, lay out the current state of typological databases, and provide empirical evidence of their inconsistencies. finally, we take a look at the potential future of such work, highlighting cases where a continuous view of typological features is clearly appropriate. the core of our work is to argue for the awareness of these limitations in the field, and to encourage researchers to focus on the continuous scale view of typology. ## quantifying the mismatches in typological databases it is well-established that resources such as wals contain errors. for instance, @xcite carry out an in-depth investigation of the features described for german, noting that 'a non-negligible proportion of the values assigned is found to be problematic, in the sense of being arbitrary or uncertain in view of analytic alternatives, unappreciative of dialectal variation, unclear as to what has been coded, or factually erroneous' @xcite . we begin by building upon this work, attempting to identify further errors and problems in wals, by contrasting it with another existing database, grambank. this allows us to use a semi-automatic approach to quantify errors in typological databases. we match a subset of features from each database to one another. this is straightforward in many cases, as some features have exact counterparts in the other database, and can therefore be easily compared. other features do not have exact counterparts. as an example, feature 81a in wals uses seven different labels (sov, svo, vso, vos, ovs, osv, and 'no dominant order') to represent the most dominant word order in a given language. feature 81b is similar, but uses five different labels ('sov or svo', 'vso or vos', 'svo or vso', 'svo or vos', and 'sov or ovs') to represent the two most dominant word orders for languages with two dominant word orders. comparatively, the word order features in grambank (131, 132, and 133) have binary labels which represent whether or not verb-initial, verb-medial, or verb-final word orders are present in the language. given these differences, we use two methods to compare the word orders indicated in each database. in the first, we check whether or not the labels in wals exactly match the labels in grambank (exact match). in the second, we take into account the fact that wals represents only the dominant word order(s) in a language, while grambank represents all word orders that occur in that language (soft match). to do so, we count as matches the languages where the wals labels are a subset of the grambank labels. examples of exact match, soft match, and no match cases can be found in the results of this initial experiment, displayed in the main inconsistencies and errors in wals and grambank fall into one of three categories: factual errors, differences due to differing language varieties, or simplifications introduced by the discretization of data to fit the database formats. ## typological databases and large language models these data issues make it difficult to successfully incorporate typological features into language models. as a simplified example, we attempt to finetune large language models (llms) to predict a variety of languages' wals features from wikipedia articles about these languages. to do so, we extract the sections in these wikipedia articles related to word ordering, using string matching to verify that they contain sufficient word order-related informa-tion. we then fine-tune pretrained llms, with an added classification head, to predict a language's wals word order feature, given that language's wikipedia snippet as input. as hypothesised based on the above data issues, the fine-tuned models generally do not perform well. more often than not, the models output the same predictions as a baseline model that predicts the majority class for the relevant wals feature, implying the models are able to extract little to none of the input typological information. these results can be found in ## the past, present, and future of typology in nlp in spite of the inconsistencies and errors we remark upon in typological databases such as wals, there is a body of research in nlp which largely appears to be unaware of these issues. past considering the past years of research, typological databases have mainly been used in the context of feature predictions. methodologically speaking, features are typically predicted in the context of other features, and other languages @xcite @xcite @xcite @xcite @xcite @xcite . that is to say, given a language @xmath0, where l is the set of all languages contained in a specific database, and the features of that language f l , the setup is typically to attempt to predict some subset of features f ⊂ f l , based on the remaining features f l \ f . typically, this language may be (partially) held out during training, such that a typological feature prediction model is fine-tuned on l \ l, before being evaluated on language l. variations of this setup exist, with attempts to control for language relatedness in training/test sets, using genealogical, areal, or structural similarities @xcite . present a more recent trend is using typological feature prediction to interpret what linguistic information is captured in language representations of, for example, a neural language model @xcite @xcite @xcite . such work leans on the intuition that, if a model encapsulates typological properties, it should be possible to accurately predict these based on the model's internal representations. given the inconsistencies and errors in typological databases, it is difficult to firmly establish what may be learned about either language or about nlp models by this line of research. future recent work in linguistics has argued for a gradient approach to word order, or in other words, a continuous scale view on this type of typological features @xcite . we echo this recommendation, corroborated by the fact that languages typically lie on a continuum in terms of word orders. furthermore, we argue that the lack of this view is the root cause of some of the disagreements between established linguistic resources. we provide the added perspective that taking this continuous view will also potentially lead to improvements in incorporating typological features in nlp models. based on this line of argumentation, it is perhaps not so surprising that to date, work on incorporating typological features in nlp models has only led to limited effects on performance @xcite . ## conclusion typological information has the potential to improve downstream nlp tasks, particularly for lowresource languages. however, this can only happen if the typological data we have access to is reliable. we argue that a continuous view of typological features is necessary in order for this to be the case, and for typology to truly have an impact in lowresource nlp and language modelling. until this is amended, we argue that it is important that the community is aware of these inherent limitations when using typology for interpretability of models.