Thinking globally, acting locally – Progress in the A frican W ordnet Project (2019)

Abstract: The African Wordnet Project (AWN) includes all nine indigenous South African languages, namely isiZulu, isiXhosa, Setswana, Sesotho sa Leboa, Tshivenda, Siswati, Sesotho, isiNdebele and Xitsonga. The AWN currently includes 61 000 synsets as well as definitions and usage examples for a large part of the synsets. The project recently received extended funding from the South African Centre for Digital Language Resources (SADiLaR) and aims to update all aspects of the current resource, including the seed list used for new development, software tools used and mapping the AWN to the latest version of PWN 3.1. As with any resource development project, it is essential to also include phases of focused quality assurance and updating of the basis on which the resource is built. The African languages remain under-resourced. This paper describes progress made in the development of the AWN as well as recent technical improvements.

## introduction the african wordnet project (awn) has seen various phases of development with different funding cycles and collaborators (see @xcite for a comprehensive breakdown of previous phases). the most recent cycle is funded by the south african centre for digital language resources (sadilar) 1 and will run from 2018 to the end of february 2020, with an extension to 2022 currently under consideration. the most notable change to the project in the past two years is the addition of four further languages to include the full range of nine indigenous south african languages, namely isizulu (zul), isixhosa (xho), setswana (tsn), sesotho sa leboa (nso), tshivenḓa (ven), siswati (ssw), sesotho (sot), isindebele (nde) and xitsonga (tso). the number of synsets, usage examples and definitions for all languages included in the awn have also been substantially increased. as 1 https://www.sadilar.org/ with any resource development project, it is essential to include phases of focused quality assurance and updating of the basis on which the resource is built. for the awn, this meant reassessing several core aspects, including the seed terms used for further development, software to assist linguists to develop and structure the wordnets, as well as the process by which development is managed. the african languages remain under-resourced despite progress being made with a resource catalogue hosted by the resource management agency of sadilar. currently there are still no freely available dictionaries for any of the languages and as oliver (2014:7) notes: "the most commonly used strategy within the expand model is the use of bilingual dictionaries". in this paper, key aspects pertaining to the development of a multilingual wordnet for such under-resourced languages will be highlighted and our solutions to challenges that emerged as a result of the growth in the scope of the project, will be discussed. the last section of the paper will mention smaller challenges and project specific matters that might be of interest to other projects with similar restrictions. ## recent progress the awn team first began the development of wordnets for south african languages in 2010 and has grown slowly but consistently. currently, the awn includes 61 000 synsets across the nine identified languages. the number of synsets per language varies from nearly 17 000 for setswana, to only 600 each for isindebele and xitsonga. this variation is due to the amount of time linguists have available to work on the project as well as the incremental addition of languages to the project (see below). in addition to the basic synsets, the awn also includes 26 500 definitions and 37 000 usage examples across the nine languages. one of the most significant expansions to the awn over the past two years has been the addition of four new languages. this means that all nine indigenous south african languages are now represented foot_0 , although not in equal numbers yet. the current funding phase will see isindebele and xitsonga also grow to 1 000 synsets each, with definitions and usage examples. in addition to this, the awn has also added definitions to the synsets already captured in previous phases. where the developers initially focussed only on synsets with their usage examples, feedback from the south african digital humanities and human language technology communities indicated that definitions would make the awn even more useful in language learning applicationsan ever-growing research and development area given the multilingual nature of the country. an initial experiment into this application is described in @xcite . in the second application, data from the awn has also been used experimentally in the kamusi gold project foot_1 to populate an online dictionary for which definitions and usage examples are important. section 3 describes another significant decision regarding the content of the awnmoving away from relying on the eurocentric core base concepts foot_2 (cbc) to a more localised wordlist to be used as seed terms for new synsets. ## visualisation of the awn in word-netloom developing data to populate a wordnet offline in spreadsheets has its advantages, most notably fast tracking of development because it is a familiar process for inexperienced linguists, ease of applying spell checking or other quality assurance, no delays due to interruptions in internet connectivity or access to a central server, etc. however, it is very difficult to see the true nature of a wordnet with connecting relations and multilingual similarities. the awn previously used the deb-visdic editor (see @xcite to facilitate development and align work across the different languages. at the onset of the current phase, however, it became clear that a focus on quality assurance of, especially the semantic relations, was needed and it was decided to port the awn to wordnetloom (wnl) (cf. naskret et al., 2018)an editor with advanced visualisation of wordnets. while preparing the data for use in this tool, the awn was also mapped to the pwn 3.0 to ensure the latest format and most up to date english equivalents. to move from debvisdic to wnl involved extracting the awn database in lmf format, whereafter a programmer could map the awn to pwn 3.0 using an in-house script. where there was no pwn 3.0 equivalent or id, the pwn 2.0 id was retained. advantages of this tool are that it speaks to the organic growth development style that the awn teams have always favoured (see @xcite and also adds the ability to perform more productive searches when working in a specific domain or looking for a specific semantic relation. the addition of a multilingual relation also means that specific senses in different languages can be connected to each other without having to connect the entire synset. the subtle differences between isizulu and sesotho sa leboa verbs and kinship terms mentioned in section 3.2 can, for instance, be represented more accurately. discussions with the development team behind this stateof-the-art software tool led to an intense two-day workshop in south africa, facilitated by members of the wnl and polish wordnet development team where linguists were introduced to wnl and its many advanced features. the workshop, which was hosted by sadilar, was attended by at least two linguists from each of the nine languages included in the awn and took on a very hands-on approach. figure 1 shows the "harvest" example from many challenges, including the low resource nature of the languages in the awn, restraints on funding, a part-time development team etc. were reported on extensively in @xcite . the awn team have managed to mitigate these risks to a large extent and porting development of the awn to wnl played a large part in this. the porting process described in section 4 did not come without some initial challenges and adaptations needed. most notable is that the visualisation of the awn now draws our attention to the lack of proper definition and application of the semantic relations between terms. relations were previously automatically carried over from the pwn as is common when following the expand model. the awn team was always aware that this method was not fool proof and that relations would need revision. wnl enables linguists to see immediately all synsets connected with any of the predefined relations as well as the lacking relations within the south african context. some terms also need to be moved from an independent synset to a more accurate embedded synonym position and vice versa. lexical gaps between the (american) english pwn and the african languages can now also be addressed more effectively by eliminating the need to link a synset in an african language to a synset in the pwn as synsets can either stand independently in wnl or be linked to another african language. again, the visualisation of the synsets within the larger structure is key in this process of identifying the lexical gaps, as can be seen in figure 2 , a representation of the isizulu marriage relations discussed in with continued research, collaboration with other developers and an invested interest in growing the african languages as digital language resources, we believe that this project will soon be of significant academic and industrial interest to members of the global wordnet community.