Creating and enriching a repository of 177k interlinearized examples in 1611 mostly lesser-resourced languages (2025)

Abstract: Much of NLP is concerned with languages for which dictionaries, thesauri, word nets or treebanks are available. This contribution focuses on languages for which all we have might be some isolated examples with word-to-word translation. We detail the collection, aggregation, storage and querying of this database of 177k examples from 1611 languages with a special eye on enrichment via Named Entity Recognition and links to the Wikidata ontology. We also discuss pitfalls of the approach and discuss the legal status of interlinear examples.

## interlinear glossed text in the context of linguistic typology and language documentation, the typical format is so called interlinear glossed text (igt). an example is given in (1) (1) tayaĝu-x man-sg qa-x fish-sg qa-ku-x. eat-pres-3sg. the first line contains the vernacular text, in this case in aleut. the second line contains a wordto-word (or morpheme-to-morpheme) translation. the third line contains a free translation of the whole sentence. while this is only very little information, some insights can readily be obtained: x for instance marks singular both on nouns and verbs, and the lexical items tayaĝu 'man' qa 'fish' and qa 'eat' can also be extracted. various formats have been proposed for the modeling of interlinear glossed text @xcite @xcite . for our purposes, we use the cldf format @xcite , which is csvbased and which has a whole ecology supporting a variety of websites and services, such as wals, apics or glottolog. a cldf rendering of example 1 is given in figure 1 . man-sg⇒fish-sg⇒eat-pres-3sg. the man is eating the fish. figure 1 : the cldf representiation of example (1). ⇒ stands for a tab. note that this tabular data is complemented by a json file describing the different column types. ## querying the site imtvault.org offers various querying facilities. the site runs elasticsearch, which can be accessed by humans through a responsive faceted search interface, or queried by machines using a well documented query language foot_0 . one main use case is the retrieval of examples based on strings found in the vernacular, the gloss, or the translation. this can be accomplished via a free text search for all three fields together, or via dedicated entry fields for the different lines. next to this string-based search, examples can also be filtered by length. for syntactic research, for instance, more than 3 words could be required in order to arrive at any meaningful conclusions regarding syntax. all glosses in allcaps are seen as grammatical categories and matched against the leipzig glossing rules @xcite . the leipzig glossing rules area a standardized set of common abbreviations, such as acc(usative) or fut(ure). additionally, any lists of abbreviations contained in a langsci book or a glossa article are also made available. the categories are taken as strings, at face value. no efforts are made to match them to an ontology or to merge/reconcile/disambiguate them. it is up to the reader to interpret whether the irr in an example would indeed match the reader's preferred definition of 'irrealis' for instance. these querying facilities are basic and work on the content already available in the original dataset. further querying possibilities are available via various enrichment procedures, which draw information from other datasets, link it, and make it available (section 5). figure 7 shows a complex query for the concept "vehicle", the category "past" and the language family "atlantic-congo". this query returns 4 hits in three languages (fwe, limbum, mossi) from three different publications. while the information about the category "past" is present in the source files, information about concepts and languoids has to be added via enrichment procedures, discussed in the section 5. next to the html view given in figure 7 , the knowledge base can also be queried via the normal elasticsearch api. ## comparison imtvault is not the only aggregator for information about lesser-resourced languages. the aggregators can be divided into aggregators for resource bundles above the sentence level (documents, corpora) on the one hand and aggregators on the sentence level on the other. for the sentence level, we can mention odin, for the document/corpus level, we can mention olac, pangloss, and vlo. ## legal aspects imtvault is hosted in germany, part of the eu. 8 we only use data which are available under a cclicence, but the legal status of igt data such as 'the man is eating the fish.' in example (1) above or its high tide; the canoes float. in (3) is interesting. in continental europe a text is a copyrightable work only if some creativity is involved. a sentence like 'the man is eating the fish' is clearly not creative, so the question arises to what extent it would be copyrightable in the first place. furthermore, it is the expression which is copyrighted, not the facts contained therein. mapping the morpheme -x to the meaning 'singular' is a factual assertion, and as such not copyrightable. even if we assume for the sake of argument that 'the man is eating the fish' fell under copy-right law, the question is who the copyright holder would be. the journal word is now owned by taylor and francis, who assert their copyright. it may be the case that knut bergsland transferred his copyright to t&f, but since he is not the original creator of the sentence tayaĝu-x qa-x qa-ku-x., he does not own any rights to this sentence in the first place, and hence, they could not be transferred to t&f (to the extent that it is copyrightable at all). this is only one example of 200k, but it shows that the aggregation of examples comes with its own kinds of legal problems. these problems may, however, be less serious than could be feared, at least for trivial examples. there are of course creative narratives, myths, and songs which clearly meet the threshold required by continental copyright law. this is the case for the examples provided by inel and otc for instance, and these contain clear creator information. this being said, the existence of a copyright framework in the eu jurisdiction will never exempt the individual researcher from making their own ethical evaluation of the circumstances under which a particular example can or cannot be used or distributed. ## outlook imtvault has grown from 40k examples to 177k examples. further data providers have been added and more facets for querying have been provided. given the legal analysis presented above, an inclusion of the examples found for instance in the 10 000 books included in the dream corpus @xcite ) should be possible. it can also not be excluded that publishers will actually be happy to find their data in imtvault, as a way to channel readers towards their publication and generate traffic and revenue. named entity extraction and linking do currently work, but a comparison of different algorithms and ontologies might lead to significant improvements here, both in terms of precision/recall as well as in terms of cultural appropriateness. a systematic evaluation is out-of-scope for this paper, but will be covered in future research.