CILI : the Collaborative Interlingual Index (2016)

Abstract: This paper introduces the motivation for and design of the Collaborative InterLingual Index (CILI). It is designed to make possible coordination between multiple loosely coupled wordnet projects. The structure of the CILI is based on the Interlingual index first proposed in the EuroWordNet project with several pragmatic extensions: an explicit open license, definitions in English and links to wordnets in the Global Wordnet Grid.

## introduction within 10 years of the release of wordnet @xcite researchers had started to extend it to other languages @xcite . currently, the open multilingual wordnet (omw: @xcite has brought together wordnets for 33 languages that have released open data, 1 and automatically produced data for 150. there are even more wordnets than this: some large projects have released non-open data, notably german @xcite and korean @xcite and many projects have yet to release any. this activity shows that the structure of wordnets is applicable to many languages. all the wordnets are based on the basic structure of the princeton wordnet (pwn: @xcite : synonyms grouped together into synsets and linked to each other by semantic relations. the majority of wordnets have been based on the expand approach, that is adding lemmas in new languages to existing pwn synsets @xcite , boot-strapping from the structure of english. 28 out of 33 of the wordnets in omw take this approach. a few wordnets are based on the merge approach, where independent language specific structures are built first and then some synsets linked to the pwn. in omw, only five projects take this approach: chinese (taiwan), danish, dutch, polish and swedish @xcite @xcite @xcite . to investigate meaning across languages, we need to link synsets cross-lingually. it is easy to link expand-style wordnets: they all link to pwn and it can be used as a pivot to link them together. this is one of the attractions of using the expand approach, you immediately gain multilingual links. the disadvantage is that concepts not in pwn (either because they are not lexicalized in english or just because they have not been covered yet) cannot be expressed. because of this, many expandstyle wordnets also define some new, languagespecific synsets, typically a few tens or hundreds (arabic, chinese, italian, japanese, catalan, spanish, galician, finnish, malay/indonesian, bulgarian, greek, romanian, serbian and turkish all do so) @xcite @xcite @xcite @xcite . it is harder to link merge-style wordnets. the projects need to somehow identify links to pwn, and as a result, only a small subset of the language specific synsets are linked to pwn. examining the unlinked synsets, this seems to be principally due to the lack of resources to link them than semantic incompatibility. for example, danish and polish @xcite have many synsets which can be linked but are not currently. currently, when projects create their own synsets, there is no coordination between these projects. this means that similar or even identical concepts may be introduced in multiple places. for example, most south east asian languages distinguish between cooked and uncooked rice: these concepts have been added independently to the korean and japanese wordnets. typically, clusters of projects have tried to coordinate, such as eurowordnet, the multilingual central repository for basque, catalan, galician and spanish @xcite , the multiwordnet for italian and hebrew, @xcite , balkanet @xcite , the wordnet bahasa for malay and indonesian @xcite , the in-dowordnet project @xcite . clearly, there is a need for a single shared repository of concepts. in this paper, we propose to build one: the collaborative interlingual index (cili). we base the index on the technical foundations laid down in eurowordnet: a single list that is the union of all the synsets in all the wordnets @xcite . to this we add ideas from the best-practice of the semantic web: a shared easily accessible resource with a well defined license; from open-source software: build a community of users who will co-develop the resource; and from experiences in many multilingual lexical projects: accept the de facto use of english as a common language of communication. in the following sections we discuss the motivation further ( § 2), then describe in detail the structure of the cili ( § 3), list some open issues ( § 4) and finally conclude. ## motivation wordnets have been built with different methods and from different starting points: expand or merge, manually or semi-automatically and based on pre-existing monolingual resources or using available bilingual resources to translate english synsets to words in the target language. furthermore, it is up to the wordnet builders to make decisions about which words are synonyms, what are the semantic relations between the synsets and how to interpret each semantic relation. we can observe very large synsets in one wordnet being linked through pwn to small synsets in another language. different granularities of synsets brings into questions the notion of the same concept existing across these wordnets. pwn uses 44 semantic relations (if separated by part-of-speech) but in eurowordnet 71 relations were defined that partially overlap. even if two wordnets use the same relation name, there is no guarantee that it is inter-preted in the same way. in fact, different wordnet editors and algorithms may interpret relations differently. even the symbols used for parts of speech differ in different projects (adverb is 'r' in pwn but 'b' in some projects). finally, one can observe large differences in coverage of the vocabulary and in the degree of polysemy. vocabularies and concepts differ in size but also in terms of genre, pragmatics, the inclusion of multiword expressions as ``phrase sets'' @xcite and specific domains and areas. choices for distinguishing senses lead to fine-grained and coarse-grained polysemy, where the latter may lead to multiple hypernyms that can be modeled as complex types @xcite . finally, the glosses for synsets play an underestimated role in addition to the synsets and the relations, but no formal structuring is defined for these glosses. as a result, glosses are not sufficiently descriptive to precisely identify the meaning of a concept. such differences across wordnets make it difficult to establish the proper relations to the ili and thus to compare and exploit wordnets across languages. further, if a synset is not realized in a language it is not clear if that is because the concept is not lexicalized in that language, or if it is merely not realized yet (the compilers may just not have got round to it). to solve these problems, we need to not just define an interlingual index, but also shared guidelines for relations, how to write definitions, standard data formats and so forth. ## the collaborative interlingual index in this section we describe the core properties of cili. to coordinate an index among all the different wordnet projects, we propose that it should, ideally, have the following properties (building on 1--5 from @xcite : property 7, persistent identifiers, is an important criterion for stability. if the ili changed its ids, projects without the resources to maintain compatibility would fall behind. if a project changes its hierarchy, then it will need to add new nodes and delink the old ones. to keep backwards compatibility, even if a concept is deemed problematic, it will remain in the cili, and marked as deprecated, preferably with a link to the concept that supercedes it. property 8, that all synsets should have a definition in english, recognizes that, in practice, the only language shared by all groups is english. here we are inspired by experience with the cicc project, a multilingual machine translation project linking thai, chinese, japanese, malay and indonesian (but not english) @xcite . no members spoke all five languages, but someone in each group spoke english, so all dictionary entries also had an english translation or definition. having a universally understood definition is a prerequisite in avoiding redundant creation of new senses. this creates a burden on non-english speakers, which we will try to lighten by giving clear guidelines for writing definitions (see section 3.3). note, that while the definition must be in english, the concept is not necessarily lexicalized in english, in contrast to princeton wordnet. properties 9 and 10 make sure that all new concepts link to something, there should be no orphaned concepts. exactly which links are acceptable is still a matter of research. 2 the final point ( 11 ) is about coordination. practically, it will not be possible to have a single moderator who can check new synsets in every language. we therefore propose that the burden of checking for duplication with existing synsets should be placed on the project wanting to add new synsets. as new concepts should be linked to existing concepts through relational links in a wordnet, and definitions in english will exist for all entries, checking for a compatible entry in the ili should not be too burdensome. project members with wordnets in the shared multilingual index would gain write privileges to the ili, of course anyone should be able to read it. we will build automated tools that warn if definitions are too similar (for details see @xcite . for the ili to be successful there will be an initial cost to combine all existing non-english synsets, adding english definitions for all and merging duplicates. it would also require buy-in from all participating projects, but fortunately most non-english wordnets contain few synsets that do not correspond to an english synset, so this first step should not be too burdensome. for wordnets built with the merge approach there will be many more new synsets, these should be checked carefully and validated against corpora before being included in the ili. we will support this with workshops at relevant conferences (such as the 16th global wordnet conference). in the long run, we hope that external resources will link to the ili's persistent ids (things like @xcite , the many sentiment wordnets @xcite . ## open issues there are a few cases where it was hard to decide whether a concept should be represented in the in-terlingual index. one example is named entities. roughly 6.6% of the entries in pwn are linked by the instance relation (including the names of people, places, planets, gods and many more). named entities are much more numerous than words and these concepts and their relations are better captured by other kinds of resources. however, some named entities can be considered part of the lexicon as well as names for objects, for example glaswegian 𝑎∶1 ``of or relating to or characteristic of glasgow or its inhabitants'', which is also used in the definition of other concepts. thus, we retain a small number of named entities, especially geographic terms but further discussion is required to refine an exact policy. it could also be argued that some of the derived forms (for example quickly 𝑟∶1 from quick 𝑎∶1 ) are unnecessary: as the meaning change is generative, there is no point in having two concepts. these kind of changes can be applied later by means of superceding other concepts, and for the moment we apply the distinctions made by princeton word-net. ## conclusions we have introduced and motivated the collaborative interlingual index (cili). its simple design allows us to link wordnets with a minimum of extra work. once concepts are added to the cili, they will get a persistent id and thereafter should not be deleted or change in meaning. we propose that the task of checking the validity of new concepts is taken up by the individual wordnet projects, with only a light layer of moderation.