Language Diversity: Visible to Humans, Exploitable by Machines (2022)

Abstract: The Universal Knowledge Core (UKC) is a large multilingual lexical database with a focus on language diversity and covering over two thousand languages. The aim of the database, as well as its tools and data catalogue, is to make the abstract notion of linguistic diversity visually understandable for humans and formally exploitable by machines. The UKC website lets users explore millions of individual words and their meanings, but also phenomena of cross-lingual convergence and divergence, such as shared interlingual meanings, lexicon similarities, cognate clusters, or lexical gaps. The UKC LiveLanguage Catalogue, in turn, provides access to the underlying lexical data in a computer-processable form, ready to be reused in cross-lingual applications.

## introduction a recent challenge in computational linguistics has been the development of efficient multilingual and cross-lingual techniques for language understanding and processing. in terms of solutions, a mainstream, yet often implicit assumption has been that shared meaning unites languages beyond superficial differences in lexicon and grammar: after all, humankind on the whole has been successful in getting ideas across linguistic borders. hence the recent trend of massively multilingual resources-lexical databases, crosslingual transfer matrices, pre-trained multilingual language models-exploiting a common meaningbased mapping across linguistic units. linguistic diversity remains, nevertheless, a key concept insomuch as it refers to deep-running, irreducible, meaning-level differences across languages and underlying cultural concepts. to take real examples from state-of-the-art machine translation, syntactically correct but semantically absurd outputs such as 'my older brother is younger than me' or 'this raw rice is tasty' are not rare exceptions but recurrent consequences of diversity: the diverging ways languages express culturally significant concepts such as brother or rice. while phenomena such as lexical gaps @xcite , culturally diverse terminology, or the varying relevance of the notion of word itself across languages are not unfamiliar to us computational linguists, this intuitive and high-level understanding is hard to translate into actual 'diversity-aware' computational applications, not the least because of the lack of formal datasets that would provide such information. lexical typology has described and catalogued many of such phenomena @xcite . a few online databases also provide contrastive data and visualisations, sometimes over thousands of languages @xcite @xcite . the these databases are rarely used in the nlp community, probably because they are often targeted towards historical linguistics and use phonetic representations of words or are limited to a few hundred core concepts. yet, our position is that typological data can and should be reused for computational purposes, provided that they are meaningfully integrated with existing resources on contemporary language. computationally-oriented resources that address language diversity or linguistic typology have so far been concentrating on multilingual morphosyntax @xcite @xcite . on diversity in lexical semantics, only a few studies @xcite and sporadic data have been available for specific languages, such as a few hundred lexical gaps in hebrew (ordan and wintner, 2007) or in italian @xcite . large-scale multilingual lexical databases (mldb), such as babelnet @xcite or the open multilingual wordnet @xcite , have so far ignored phenomena related to language diversity and have concentrated on shared meaning. the universal knowledge core (ukc) database and system presented in this paper provides computer-readable cross-lingual lexical data, covering both the common and the diverse among more than a thousand lexicons. the data is being made available for download from the ukc livelanguage catalogue foot_0 , while the ukc website foot_1 provides a set of interactive tools that allow both highlevel visualisations and an in-depth exploration of diversity data. the rest of the paper provides an overview of the ukc database structure and contents, the online tools, and the data catalogue. 3 ## a multilingual lexical database on language diversity among existing large-scale mldbs, those with a published formal, computer-exploitable data model-such as the open multilingual word-net, babelnet, or eurowordnet (vossen, 1997)concentrate solely on language unity, i.e. shared supra-lingual meaning, through linking together words with the same meaning across languages. the ukc offers a richer, two-layered representation of language unity, as well as introducing language diversity as formal data, both in terms of lexical model and actual content. the ukc data model, the theoretical underpinnings of which have been exposed in @xcite , is illustrated in figure 1 . on the top of the figure, a supra-lingual concept layer contains hierarchies of concepts that represent lexical meaning shared across languages. concepts thus act as bridges across languages. the only criterion for a concept to be present in the concept layer is that it is lexicalised by at least one language. the bottom lexicon layer consists of languagespecific lexicons. as in other lexical databases, these provide lexicalisations for concepts, such as the english 'rice' and the italian 'riso' for the concept of rice in figure 1 . beyond lexicalisations, however, the ukc lexicons also provide rich crosslingual information on language unity and diversity through the additional constructs detailed below. lexical gaps. as evoked in the introduction, when unrecognised, lexical untranslatability can decrease the performance of cross-lingual applications. syntactically correct yet meaningless google translations, such as the hungarian sentence 'a bátyám három évvel fiatalabb nálam,' meaning 'my older brother is three years younger than me,' are systematically produced due to nonexistent equivalent translations, in this case for the word brother in hungarian. 4 likewise, as shown in figure 1 , english has no single word for raw, uncooked rice while swahili-and many other languages, cf. @xcite -has no equivalent for the general term rice. the ukc provides evidence of untranslatability by representing lexical gaps inside lexicons. such information can be used, among others, to indicate the absence of equivalent terms to downstream cross-lingual applications. ## exploring diversity data the website of the ukc database provides browseable online access to the full database contents, data visualisation tools, extensive information on related projects, publications, source materials, as well as example downstream services, such as word translation between any two languages or multilingual word sense disambiguation (to be released soon). a major feature of the website is the interactive exploration of lexicons and diversity data. the user can browse: (1) linguistic metadata of the 2k lexicons, selecting the language from an interactive map (figure 2 ) or by name; (2) within a language, all meanings of a word typed in by the user; and a screenshot of the last-and richest-concept exploration functionality, taking the example concept of blueness, is provided in figure 3 . on the left-hand side of the screen, details are provided on the lexicalisation of the concept in the current language, such as synonyms, definition, part of speech, as well as lexical relationships to other word senses (e.g. derivations, metonyms, cognates in other languages). the middle part of the screen shows an interactive clickable map of all languages that either lexicalise the concept in the database or, on the contrary, are known not to lexicalise it. the colour-coded dots (indicating the language family while 'black holes' stand for gaps) thus provide an instant global typological overview for the concept selected, e.g. from figure 3 one can see that a lot of languages in the americas do not lexicalise blue as a separate colour. some language do not appear on the map due to lexicon incompleteness: for those languages the ukc has no information whether they lexicalise blueness or not. the right-hand side, finally, shows the concept in the context of the lexico-semantic concept hierarchy, displayed as a graph. a detail of the full graph is shown, including the currently observed concept blueness, the parent (broader), and the child (narrower) concepts. other lexico-semantic relationships (e.g. meronymy and metonymically related concepts) are also shown when they exist. while for usability reasons the graph only displays a part of the full hierarchy, it is interactive, allowing the entire concept graph to be explored in the currently selected language. changing the language is as simple as clicking on the map or selecting it from the drop-down in the upper left corner of the screen. colours in the graph are indicative of language diversity: they show whether a concept is lexicalised in the current language (dark-coloured nodes), are missing from its lexicon (light-coloured nodes), or are lexical gaps (black nodes). ## visualising language diversity beyond the fine-grained word and concept exploration presented in the previous section, the ukc website also offers visualisation tools that allow humans to grasp diversity both in its globality and from different angles. currently the following tools are provided, three of which we present below: (1) cognate diversity clusters; (2) colexifications; ## the livelanguage data catalogue as a complement to online exploration, we are also making available the contents of the ukc for computational applications. while an open, finegrained, api-based access to the data is planned as future work, we are already in the process of publishing data for download through the ukc livelanguage data catalogue. the catalogue, accessible from the website through any of the numerous download links, provides access to the ukc data through multiple modalities. accordingly, the structure of the catalogue, shown in figure 7 , consists of (1) cross-lingual datasets; (2) individual lexicons that aggregate all types of data about a single language; and (3) lexicon sets that also provide cross-lingual information. all datasets are published in full respect of the licensing constraints of their constituting resources; data that disallow redistribution are excluded from the catalogue. raw data on cross-lingual diversity. these datasets originate from projects on diversity and cover domain-specific lexical gaps, multilingual morphology, lexical similarity, and cognate relationships. datasets are distributed in their original formats, with concepts linked to princeton word-net 3.0 identifiers for interoperability with thirdparty data. individual lexicons. these datasets are produced as language-specific 'cross-sections' of the full ukc data. their added value lies in the integration of multiple sources-words from wordnets and wiktionary, language-specific morphological and lexico-semantic relationships, gaps-into a single formal representation. these datasets will be provided in multiple formats (under development), including the iso standard lexical markup framework (lmf) format as well as ontolex. lexicon sets. the notion of language diversity gains full significance across languages. consequently, the development of an online service is underway to allow the simultaneous download of multiple concept-aligned lexicons as a single multilingual resource. the service will export multilingual data from the ukc database in real time. such datasets will be directly exploitable in cross-lingual training and evaluation tasks. ## conclusions and future work we see a huge research potential both in the creation and in the reuse of diversity-aware language resources. exploiting diversity to improve stateof-the-art cross-lingual applications is a direction that we expect to gain importance and popularity in the community. the ukc database, website, and data catalogue aims to contribute to such efforts. all components of the system are going through a rapid evolution: the database contents in terms of language support, lexicon correctness and completeness, the data exploration and visualisation tools, new demonstrators, and apis are continually being created and extended. we are also deploying regional instances of the ukc-e.g. for the middle-east or the indian subcontinent-where diversity among local languages and dialects is studied in fine detail by local communities of linguists.