VISCOSE - a Kanji Dictionary Enriched with VIS ual, CO mpositional, and SE mantic Information (2022)

Abstract: In this paper, we present a novel approach for building kanji dictionaries by enriching the lexical data of 3,500 kanji with images, structural decompositions, and semantically based cross-media mappings from the textual to the visual dimension. Our kanji dictionary is part of a Web-based contextual language learning environment based on augmented browsing technology. We display our multimodal kanji information as kanji cards in the Web browser, offering a versatile representation that can be integrated into other advanced creative language learning applications, such as memorization puzzles, creative storytelling assignments, or educational games.

## introduction learning a foreign language can be interesting and exciting. however, when we begin to feel overwhelmed by difficulties, or are faced with uninteresting learning material, it can become tedious and frustrating @xcite . learning japanese as an english speaker poses a particular challenge @xcite . it requires extensive memorization due to the drastically different and complicated writing system @xcite . the logographic characters called kanji make up its core and are supplemented by two syllabic scripts: hiragana and katakana. kanji pose a significant challenge to students who are not aware of this concept @xcite . there are thousands of these characters and each of them has several possible meanings and pronunciations depending on the usage context @xcite . people born and educated in japan learn 80-200 kanji per school year, and more advanced characters after high school. therefore acquiring kanji is a lifelong process. someone who decides to learn japanese as a foreign language cannot fall back on this incremental and steady acquisition process, but has to memorize these characters quickly. ## related work in order to build effective methods for the study of kanji, extensive high quality digital resources are needed. the most predominant, freely available kanji dictionary is kanjidic2 @xcite , which contains detailed information about 13,108 characters. wiktionary data is even more comprehensive and accessible as llod (linguistic linked open data) via dbnary @xcite . recently, lexical resources have been increasingly enhanced with visual representations. prominent examples are wikipedia thumbnails and other illustrations, also accessible via dbpedia @xcite . however, the number of images provided varies widely across language versions. babelfy @xcite is another good example of such efforts, whereas wiktionary contains comparatively few images so far. on the other hand, wiktionary displays ideographic description sequences (ids) from the ids data set, which is derived from the chise project @xcite and is freely available on github foot_0 . we are aware of only one application that uses this compositional data for educational purposes: the kanjibuilder component included in kanshudo foot_1 . however, the entire content is proprietary, hence it is not available for academic use. at the semantic level, efforts are equally limited. so far, to the best of our knowledge, there exists no approach that maps kanji to wordnet @xcite synsets. however, there are ongoing efforts at the word level in the japanese wordnet @xcite . regarding the mapping of images to wordnet synsets, there is the well-known ima-genet collection, which maps about 1,000 images to each synset @xcite . while this is undoubtedly an excellent resource for deep learning applications, it is not ideal for educational use, where we often need one well-chosen, representative image. another effort to assign cliparts from openclipart foot_2 to synsets was discontinued after illustrating only 581 synsets @xcite . the use of flashcards has a long tradition in studying japanese kanji @xcite , e.g. available in anki foot_3 . we integrate the display of our kanji data as kanji cards into our contextual language learning environment by using augmented browsing technology, as opposed to the traditionally isolated presentation. this way, we can offer advanced creative learning @xcite solutions, such as storytelling @xcite or educational games @xcite . as stated very recently in (elm learning, 2021), creative learning is vital for several reasons: we included these insightful considerations and combine them with our experience from our longlasting and ongoing research efforts in refining the structure and presentation of enriched kanji cards @xcite . we considered the feedback, critique, and comments we received in the course of this research to develop the novel design of kanji cards presented in this paper. our enriched kanji dictionary viscose lays the foundation for a novel and innovative method that contributes a vital stimulus for second language acquisition of japanese. we believe that this will greatly alleviate the difficulty of memorizing kanji for beginners @xcite and keep advanced students motivated. ## building an enhanced kanji dictionary we have implemented our kanji dictionary in swi-prolog @xcite because it is an optimal choice for natural language processing tasks as well as the management and retrieval of linguistic data. it is suitable for efficient processing of large xml and rdf files and provides a scalable web server solution @xcite . we use the latter to generate dynamic web content for creative language learning scenarios via augmented browsing. we chose to render the kanji information in the web browser as kanji cards because this is an aesthetically pleasing representation, which can be flexibly adapted to many language learning applications, including quizzes and educational games. the kanji cards in fig. 1 contain the following information: the radical number, variant forms, on'yomi and kun'yomi readings (described below), glosses, an image, and the ideographic description sequence. the radical is the main component of the kanji used for lookup in a paper kanji dictionary. there are 214 radicals in total. the on'yomi readings are displayed in uppercase, they descend from approximations of original chinese pronunciations. in contrast, kun'yomi readings are based on pronunciations of native words approximating the meaning of the kanji when it was introduced. some kanji do not have on'yomi, they are so-called kokuji, having been newly created in japan. we extracted all this lexical data from kanjidic2. so far, we performed 460 corrections and additions using mainly wiktionary as supplementary resource for the 3,500 kanji in our dictionary. the images displayed on the kanji cards add essential visual information. they were all collected manually and exclusively from wikipedia pages due to licensing reasons and to guarantee access to valuable contextual and ontological data via dbpedia datasets. we have also added compositional information to the kanji cards, represented as an ideographic description sequence (ids). we spent extensive editorial work to meticulously check the ids data resulting in 581 corrections. one reason for this high number was our goal of creating consistent and complete decompositions of all kanji as shown in fig. 1 . this means that we recursively follow the ids data down to the level of radicals and some additional individual strokes. all the intermediary components have to be valid characters, i.e. they are included in our set of 3,500 kanji. in our visual representation, the kanji cards for radicals are red, same as the radical of the kanji if it occurs in the ids. additional radical kanji in the ids are highlighted in orange. during the arduous process of gathering suitable images for kanji cards, we soon identified successful strategies for finding visual representations of abstract concepts. we also observed that we often covered several aspects of the lexical data in one image, which sometimes happened even subconsciously. often these hidden associations were discovered in a review process by a second person. these experiences were the main motivation to start annotating this mapping process to be able to preserve a formalized representation of the image acquisition task. this semantic information is a valuable resource for many more advanced language learning applications. however, to reap its full benefit we also had to address the issue of word sense disambiguation by assigning each mapping to the correct wordnet synset. a graphical representation of the integrated resources and an example depiction of a kanji card are shown in fig. 2 . figure 3 shows an example of three mappings for a kanji card with verbal descriptions and wordnet synset definitions. we have converted this representation into a more formal annotation format as a set of septuples: {(lexicaldata,sourcetype,informationtype, targetregion,depictiontype,mappingtype,synset)} for instance, the mappings in fig. 3 would be translated into: {(chop,gloss,activity,main motif, real depiction,execution,chop.v.05), (tree,radical,tangible object,foreground, real depiction,direct,tree.n.01), (axe,component,tangible object,detail of main motif, real depiction,direct,ax.n.01)}. of course, we actually use one-letter codes for the values of the five features to guarantee the effi-cient storage and processing of the annotation data. the features are explained in detail in sect. 4. we have created annotations for all 3,500 kanji in our dictionary resulting in a total number of 4,282 annotated mappings. we used altogether 45 different feature values, which are introduced in the next section and summarized later in ## annotation examples in the following subsections, we present detailed examples of annotations for all values of the five features that are used to represent the mappings from the textual kanji information to its visual representation. ## conclusion in this paper we presented viscose, an enhanced kanji dictionary with detailed visual, compositional, and semantic annotations. we described our motivation and the steps involved in building this lexical resource. our main contribution is a formal representation of semantically grounded cross-media mappings from the textual to the visual dimension. we drew upon years of experience and many iterations of designs to optimize the presentation of the many details on the kanji cards. the current version of the annotation was done by the authors, who both have a higher educational background in computer science, linguistics, and japanese studies, as well as long research and teaching experience in natural language processing including several projects on lexicographic and terminological topics. future research will focus on editorial work and writing a comprehensive annotation guide. we will make a preliminary version available as soon as possible, before publishing our resource at lre map. we also plan to involve students in the process of refining and extending the annotations as class room assignments, and evaluate the annotation agreement. in this context we plan to measure the improvement regarding language acquisition in comparison with other approaches, as well as get feedback on usability, entertainment factors and the often related engagement level. we already envision many interesting future use cases. one example is the use of kanji cards to replace thematic cards in strategic games based on feature value agreements and synset similarities. another idea is to create cross-media analogies for memorization puzzles, such as guessing the correct gloss based on an image and an analogous kanji card. finally, we also want to put the structural information to good use by applying our decomposition diagrams to creative storytelling to collect mnemonic sentences for improved kanji retention. finally, we will continue our ambitious research efforts towards integrating the kanji cards into multifaceted annotations of japanese sentences at the lexical, syntactic, conceptual, and relational level @xcite . all these implementations will be evaluated in graduate courses with students in translation studies. we will follow recent suggestions in evaluation approaches @xcite by putting learners in the center of the evaluation process and giving them an active role in the further development of our environment. we will solicit feedback at the beginning, during, and at the end of the evaluation to incorporate criticism and suggestions into the continued development of viscose.