Towards an RDF Representation of the Infrastructure consisting in using Wordnets as a conceptual Interlingua between multilingual Sign Language Datasets (2023)

Abstract: We present ongoing work dealing with a Linked Data compliant representation of infrastructures using wordnets for connecting multilingual Sign Language data sets. We build for this on already existing RDF and OntoLex representations of Open Multilingual Wordnet (OMW) data sets and work done by the European EASIER research project on the use of the CSV files of OMW for linking glosses and basic semantic information associated with Sign Language data sets in two languages: German and Greek. In this context, we started the transformation into RDF of a Danish data set, which links Danish Sign Language data and the wordnet for Danish, DanNet. The final objective of our work is to include Sign Language data sets (and their conceptual cross-linking via wordnets) in the Linguistic Linked Open Data cloud.

## introduction a final goal of our work is to represent and publish sign language (sl) data sets in the linguistic linked data (llod) cloud, which is a subset of the linked data (ld) cloud. @xcite we can observe that sl data are not represented in the data sets currently included in the llod cloud. and looking at the "overview of datasets for the sign languages of europe" published by the "easier" european project (kopf et al., 2022) @xcite we do not see any mention of a data set being available in a linked data compliant format. this shortcoming is a problematic issue, as an important type of natural language is missing from the llod, while the motivation behind the creation of the llod is that it can ease the linking of all types of natural language resources. @xcite the prerequisite for publishing linguistic data in the llod cloud is to have it formally represented within the resource description framework (rdf). @xcite and as an rdf-based de facto standard for representing lexical information, the ontolex-lemon specifications, 5 already exists, we investigate as a first step the re-use of this model in order to accommodate the description of sign language data sets. but as we can see in figure 1 , the class ontolex:form covers only the representation of written languages (with the addition of the associated pronunciation information), so that there is a need to think about possible adaptations or extensions of ontolex-lemon. at the same time, the ontolex-lemon model supports the representation of wordnet data, which are typically encoded with the skos 6 vocabulary, where the synsets are represented as instances of the ontolex:lexicalconcept subclass of the skos:concept class. @xcite this feature is offering us a good starting point for transforming into rdf (and ontolex-lemon) recent work by the easier project dealing with the use of shared ids of the open multilingual wordnet (omw) 8 infrastructure for interlinking sl data sets for two languages: german and greek, as described in (bigeard et al., 2022) figure 2: a screenshot showing how german and greek sign language data are interlinked via a shared omw index, as proposed by the easier project. taken from https://www.fdr.uni-hamburg.de/ record/10169#.y01wxexbzme , screenshot4.jpg figure 2 , foot_4 displays the starting point of a video representing a sign, its related glosses (written using only capital letters, in order to distinguish them from keywords or potential lexical entries), foot_5 its phonetic transcription (in hamnosys, (hanke, 2004)), as well as its link to the omw version of the princeton wordnet (omw.01115162-n, with its lemma and definition). this link to omw is done for the german sign indexed with "dgs9292", foot_6 which points to the subtype (or the "sub-gloss") "to-sell1" of "to-distribute2ˆ". those english glosses are in fact translations of the german glosses "verkaufen1" and "verteilen2ˆ", which are used for accessing the princeton wordnet in omw (as the germanet resource (kunze and lemnitzer, 2002) used in easier is not included in omw). it can also be seen in figure 2 that a "gsl" box is being positively checked. "gsl" stands for "greek sign language", and the positively checked abbreviation in the screenshot means that there is a corresponding synset in the greek wordnet available in omw. this way, a dgs sign can be linked to a gsl sign, based on a shared omw id, which is much more accurate than linking only via translation of glosses. the links between the one omw id and the two signs/videos ids are available in excel files. @xcite the corresponding csv lines are displayed in figure 3 and figure 4 , where we can see that one omw id (omw.00377364-n, with the english lemma "explosion", translated to german "explosion", and with the greek lemma "έκρηξη") is associated with both the german and the greek sl resources, thus establishing a conceptual link between those. those elements: videos, glosses, phonetic transcriptions, links to omw, are the elements we are encoding in a unified and harmonized linked data compliant format. ## linked data compliant encoding of the infrastructure using shared omw ids as stated in the introduction, we need to transform into rdf the different types of data used for representing signs for their future publication in the llod cloud. we also make use of rdf(s) and owl representation languages, as those are constitutive parts of the ontolex-lemon specifications and of the building of ontologies. @xcite for the rdf representation of videos included in our data set, we just introduce a class and have all videos encoded as instances of this class. listing 1 displays the rdf-based encoding of a video containing a german sign. @xcite a partial view of the original web page is displayed in figure 5 . in the rdf representation of the sign, it can be seen that the video/sign is linked to two glosses, as this sign has more than one gloss related to it. listing 1: the rdf-based encoding of a video containing a sign listing 2 displays the corresponding glosses (as instances of a specific class). the subclass/subtype relation between the glosses displayed in listing 2 is encoded in a specific class, called "type", as can be seen in listing 3, which displays the subclass hierarchy between glosses (here class and subclass instances are linking to the same video), and linking the instance of the subclass to an omw element (as instance of the class ontolex:lexicalconcept), establishing thus the link to the wordnet world, and to the corresponding video(s), as we can collect more than one video representing a sign. the omw synset linked to in figure 3 has the internal organisation displayed in figure 6 . here we didn't include links to glosses or videos, as the relations to omw described in listing 3 are inverse. finally, the representation of the form(s) of the sign is performed for the time being as instances of ontolex:form (mediated, also for the time being, by an underspecified instance of ontolex:lexicalentry). this representation, displayed in figure 7 , includes the machinereadable transcription of the hamnosys code, in the so-called sigml xml format (neves et al., 2020). it also includes potential keywords or lexical entries. ## extending the easier approach with additional sign videos per language we searched for other sign language resources in order to extend the approach described in (bigeard et al., 2022), thus linking sl data and wordnets, and then transforming those sls-wordnets combinations into rdf and ontolex-lemon. we found a basic lexicon of 1000 concepts associated with sl data in 4 languages, english, french, german and greek, an outcome of the past dicta-sign project (matthes et al., 2012), which is available at the university of hamburg at https://www.sign-lang. uni-hamburg.de/dicta-sign/portal/ concepts/concepts_eng.html . this resource is directly relevant to our purposes, as the included videos are equipped with glosses and hamnosys transcriptions, as shown in figure 8 . in figure 8 , we observe that the gloss and the hamnosys transcription for the german video are identical with those deployed in the figure 7 : the encoding of the form of a sign data used by the easier project for linking sl data and wordnets, as can be seen at https: //www.sign-lang.uni-hamburg.de/ meinedgs/types/type13990_de.html, and which is also shown in figure 5 . this concordance of gloss and hamnosys transcriptions 16 not only allows for the association of two videos representing this german sign to one owm id, 17 but it also permits the addition of signs in two additional languages, english and french, extending thus the multilingual coverage of the approach described by (bigeard et al., 2022). we just need to introduce in our rdf representation new video instances (one per language) and to link them to the same omw id. @xcite but we can observe that in the one case the gloss is realised as a noun and in the second case as a verb. signs are often ambiguous with respect to pos, and in the future we will link the videos to both the nominal and verbal synsets, if both are available in the corresponding wordnet. @xcite as the page https://www.sign-lang. uni-hamburg.de/dicta-sign/portal/ concepts/cs/cs_688.html is linking to a more detailed lexical description of the sign, with the same gloss and hamnosys transcription (see https://www.sign-lang.uni-hamburg.de/ galex/glossen/g13990.html ), with another video for the sign, we can in fact have 3 videos for this german sign associated with one omw id. thus, the transformation of this additional data into our rdf and ontolex-lemon representation means organising those originally disparate and heterogeneous data sources in one harmonised representation format, with omw as the central component for the interlinking of the different data types and sources. ## extending the approach to danish wordnet and sign language data (troelsgård and kristoffersen, 2018) discusses approaches for ensuring consistency between (danish) sign language corpus data and the dictionary of danish signs that is described in (kristoffersen and troelsgård, 2010). this approach aims at getting a correspondence between the dictionary lemmas and the corpus lexicon, which consists of types introduced for lemmatising the tokens found in the annotations (glosses added to the signs) of the corpus. the strategy being to use words and their equivalents (also found in the dictionary) to search for signs in the corpus. in order to extend the list of potential danish equivalents that could be used for a word-based search of signs in the corpus, (troelsgård and kristoffersen, 2018) suggest us-ing the danish wordnet, dannet, which is briefly described below and in more details in (pedersen et al., 2009) and (pedersen et al., 2018). dannet was constructed using the merge approach where the wordnet is built on a monolingual resource, in this case on the corpus-based danish dictionary den danske ordbog (ddo, (lorentzen, 2004), see also https://ordnet. dk/ddo ), and subsequently linked to pwn. for dannet this linking was based on the princeton core wordnet, a subset containing 5,000 central concepts of english (see the file "corewordnet.txt" in the folder "lfglosses.standofffiles.zip " under https://wordnetcode. princeton.edu/morpholinks ) that were semi-automatically linked to dannet. these linked elements constitute the part of dannet available in omw. @xcite the relations between sign identifiers and lexical elements from both dannet and other dictionary sources are encoded in a database, out of which we got a csv export. in this export, we first have the signs, which are corresponding to entries in the dictionary of signs available at www.tegnsprog.dk . a second type of data available in the export holds video links and information about the sign form (hamnosys/sigml). the hamnosys notation, though, is rather coarse, as it is generated automatically from the dictionary's phonological descriptions. a third type of information included in the export is dealing with the senses associated with the signs and their (form) variants. lastly, we also have the information from dan-net and pwn. our work consists thus in porting all those (interlinked) resources to rdf and ontolex-lemon, as we did for the german and the greek data, as presented in section 2. in the omw version of dannet, we find for example the following information"00817680-n lemma beskyttelse", where the lemma corresponds to the omw english wordnet "00817680-n lemma protection", sharing thus the same id for the concept of "protection" as the english and greek wordnets we have in omw. so that we can add the danish sign id (and video), which we got from the database, to our infrastructure. the danish sign associated with the wordnet lemma "beskyttelse" is displayed in figure 9 . it is then straightforward to encode all those types of information on the relation between danish sl data and dannet into our rdf-based model. we need only to add an instance for the video displaying the sign, and its associated gloss (with language equivalents), as shown in figure 10 . the language equivalents are included, so that a danish sign can be cross-lingually searched for, using glosses in other languages. then we just have to add an ontolex:form instance for the danish sign, displayed in figure 11 and which is linked via its corresponding lexical entry to the corresponding omw instance, which are shown in figure 12 . finally, figure 13 displays (partially) the current encoding of the omw id, showing the central and pivotal role of this id for interlinking the various types of resources involved in our work. ## current results our encoding results in a harmonised representation of data that was originally stored in different formats in different locations. taking advantage of the work proposed by (bigeard et al., 2022), (troelsgård and kristoffersen, 2018) and others, we can include the links between sl data and wordnets in a harmonised representation, under the umbrella of rdf and by re-using elements of ontolex-lemon. the open multilingual wordnet infrastructure is playing a central role in this work, as the shared omw ids across various languages are at the core of the interlinking of the distinct data types and sources. the resulting unified representation supports a dense linking of different types of information. our model will be made available on github ( https://github.com/declerck/ sl-wn-rdf-ontolex ) ## future work the next steps of our work will consist in automatising the transformation into rdf and aspects of ontolex-lemon so that we have all the data in the harmonised representation space. we are also planning to investigate a transformation of aslnet (lualdi et al., 2021) into rdf. we continue to extend our work with more data in more languages, starting with maltese, 19 as a low resourced lan-figure 13 : the encoding of the owm id, linking to corresponding lexical entries, which again are linking to other elements of our data set, as can be seen in 12 for the danish case guage. finally, we aim at adding other types of visual lexical data, like pictograms, as the links between such data and wordnet have been already investigated, for example in (schwab et al., 2020).