A Linked Data Approach for linking and aligning Sign Language and Spoken Language Data (2023)

Abstract: We present work dealing with a Linked Open Data (LOD)-compliant representation of Sign Language (SL) data, with the goal of supporting the cross-lingual alignment of SL data and their linking to Spoken Language (SpL) data. The proposed representation is based on activities of groups of researchers in the field of SL who have investigated the use of Open Multilingual Wordnet (OMW) datasets for (manually) cross-linking SL data or for linking SL and SpL data. Another group of researchers is proposing an XML encoding of articulatory elements of SLs and (manually) linking those to an SpL lexical resource. We propose an RDF-based representation of those various data. This unified formal representation offers a semantic repository of information on SL and SpL data that could be accessed for supporting the creation of datasets for training or evaluating NLP applications dealing with SLs, thinking for example of Machine Translation (MT) between SLs and between SLs and SpLs.

## fair data and linguistic linked open data fair data plays a central role in a number of prominent initiatives which have recently been proposed for the promotion of open science and data by numerous organisations and research fund-ing bodies. we advocate that llod models can contribute to the creation of fair language resources. it should come as no surprise, given the growing importance of open science initiatives and in particular those promoting the fair guidelines, that shared models and standardized vocabularies have begun to take on an increasingly prominent role within numerous disciplines, not least in the fields of linguistics and language resources. although the linguistic linked data community has been active in advocating for the use of shared rdf-based vocabularies and models for quite some time now, this new emphasis on fair language resources is likely to have a considerable impact in several ways, in terms of the necessity for these models and vocabularies to demonstrate greater coverage with respect to the kinds of linguistic phenomena they can describe, and for them to be more interoperable with each other. in the fair guiding principles for scientific data management and stewardship (wilkinson et al., 2016), the article which first articulated the bynow ubiquitous fair principles, the authors state that the criteria proposed by those principles are intended both "for machines and people" and that they provide "'steps along a path' to machine actionability", where the latter is understood to describe structured data that would allow a "computational data explorer" to determine: the current popularity of the fair principles and, in particular, their promotion by governments, transnational organisations and research funding bodies, such as the european commission, 5 reflects a wider recognition of the potential of structured, interoperable, machine-actionable data to help effect a major shift in how research is carried out, and in particular, its potential to help underpin open science best practices. the fair ideal, in short, is to allow machines (non-human software agents) a greater level of autonomy in working with data by rendering as much of the semantics of that data explicit (in the sense of machineactionable) as possible. publishing data using a standardised, general purpose data model such as rdf foot_1 goes a long way towards facilitating the publication of datasets as fair data. rdf, taken together with the other standards proposed in the semantic web stack and the technical infrastructure which has been developed to support it, was specifically intended to facilitate interoperability and interlinking between datasets. in order to ensure the interoperability and reusability of datasets within a domain, however, it is vital that in addition to more generic data models such as rdf there also exist domain-specific vocabularies/terminologies/models and data category registries (compatible with the former). such resources serve to describe, ideally in a machineactionable way, the shared theoretical assumptions held by a community of domain experts as reflected in the terminology or terminologies in use within that community. we note here that the emphasis placed on machine actionability in fair resources (that is, recall, on enabling computational agents to find relevant datasets and resources and to take "appropriate action" when they find them) gives semantic web vocabularies/models/registries a substantial advantage over other (non-semantic webnative) standards in the fields of linguistics and language resources. the ontolex-lemon ecosystem is to be understood in this light, aiming at enhancing the interoperability and machine actionability of linguistic datasets. it is, therefore, crucial to overcome the one limitation we noticed: there are for now no sl datasets within the llod, if we ignore the ongoing experiments in porting to rdf/ontolex-lemon the sl datasets (and their linking to omw or other lexical resources) that are described in sections 4, 5, 6, and 7. ## the best practices for multilingual linked open data w3c group the bpmlod w3c community group foot_2 initially created in 2015 to propose community-sourced guidelines for multilingual linked open data, has recently been resurrected in order to actualise the previously proposed guidelines, as there have been major evolutions in the field. these renewed efforts have a much broader scope, covering topics such as neurosymbolic approaches to language processing, cross-lingual linking, multi-modality, and the representation of sign languages. the latter two specification efforts are central to establishing the foundational groundwork necessary for representing both sl and spl data as rdf under the ontolex family of models, as linked open data. the nature of semantic web technologies is conducive to easily enabling interlinking once both modalities can be represented in one harmonized formal model. the bpmlod community group has thus the potential of becoming a community nexus to channel work on semantic web models for sls, spls and their linking. we encourage the widespread involvement of both the sl and the spl communities in this initiative. as mentioned above, bpm-lod is currently working on a survey of existing best-practices to model linguistic (including sl) data as linked data. foot_3 ## aligning several sl resources via the open multilingual wordnet infrastructure the work reported on in this section is developed within a research project, which aims to ease the communication between deaf and hearing individuals with the help of mt technologies. as such, linking different sls through semantics is a priority. we chose to use the open multilingual wordnet (omw) infrastructure (bond and paik, 2012; bond et al., 2016) foot_4 as a (semantic) pivot between sl data. we are dealing with four languages (german, greek, english and dutch sign languages). the resources involved in our approach are the dgs corpus (prillwitz et al., 2008), noema+ gsl dictionary (efthimiou et al., 2016), bsl signbank (jordan et al., 2014), and the ngt global signbank (crasborn et al., 2020). these resources contain various types of spoken language words associated with each sign. they may be keywords, equivalents, or sl glosses. they are used as a starting point to match with the lemmas present in the corresponding (and aligned) language versions of omw. then, native signers manually validate the potential matches. by using the open multilingual wordnet, we aim to identify the signs with the same (or related) senses across languages. each resource involved has different structures, and so, the method must be flexible enough to exploit all the data available and avoid mistakes. as an example, the dgs corpus has a multi-level structure, where each sign can be a type, a subtype, or a variant. semantics are attached to the sub-type level. if a sense has been associated with a sub-type, it can be spread down to the variants associated with it, but not up to the type. the dgs corpus also contains synonymy links that can be exploited to spread senses to other signs. we describe in the following paragraphs elements of sls that need to and could be (semantically) aligned across languages and language types. phonological transcriptions: while in an ideal world, those transcriptions from videos displaying signs could be used for establishing links between sl data for different languages, different sl data sets are transcribed with different transcription systems, e.g. hamnosys (hanke, 2004), signwriting (sutton, 1991) or others, as in the case of the swedish sl data foot_5 or irish sl, for which an xml-based transcription is under development (see section 6 for more details). besides, even if two resources use the same transcription system, the level of accuracy or precision of the transcription is not the same for all data. in some cases the transcription can be either semi-automatically generated or produced by human transcribers with different skills and views on which phonological elements of a sign should be transcribed. @xcite we are aware of efforts being made toward analysing and processing the videos directly using machine learning, rather than comparing and aligning transcriptions, but those are not in the scope of our current work. glosses: many projects dealing with sl use glosses to identify signs. a gloss is, typically, a spoken language word optionally followed by a sequence of numbers or letters, to allow several signs to share the same word. the word is typically related to the meaning or iconicity of the sign, in the surrounding spl, for easier identification. but the used word is ultimately somewhat arbitrary. two unrelated projects working on the same sign language might have different glosses for the same sign, or the same gloss for different signs. this creates an obstacle toward linking resources together. while many sl resources use glosses for labelling their data, the low accuracy/precision of automated tagging and the low inter-annotator agreement (iaa) between human annotators for such tagging made the glosses difficult to use as a potential cross-language instrument for interlinking sl data in various languages. @xcite for linking to the ids in omw, we preferably use keywords and translations as a starting point to approximate the meaning of the sign, and only use glosses as a last resort. however, we use glosses as identifiers. ## cross-linking nordic sl and spl data we extended our rdf representation of the language coverage described in section 4 to three nordic languages: danish, icelandic and swedish. troelsgård and kristoffersen (2018) discuss approaches for ensuring consistency between (danish) sign language corpus data and the dictionary of danish signs. this approach aims at delivering a correspondence between the dictionary lemmas and the corpus lexicon, which consists of types introduced for lemmatising the tokens found in the corpus annotations (glosses added to the signs). the strategy is to use words and their equivalents (also found in the dictionary) to search for signs in the corpus. in order to extend the list of potential danish equivalents that could be used for a wordbased search of signs in the corpus, troelsgård and kristoffersen (2018) suggest using the danish wordnet, dannet, which is described in pedersen et al. (2009; pedersen et al. (2018). this approach is thus very similar to the one described in bigeard et al. (2022), but is 'limited' to the danish language. the relations between sign identifiers and lexical elements from both dannet and other dic-tionary sources are encoded in a database, from which we obtained a tsv export. luckily for us, the wordnet elements encoded in this tsv export are the subset of dannet entries that are contained in the danish section of omw. in this export, we first have the signs, which correspond to entries in the dictionary of danish signs (see figure 1 ). a second type of data available in the export holds video links and information about the sign form (hamnosys/sigml). the hamnosys notation, though, is rather coarse, as it is generated automatically from the dictionary's phonological descriptions, and it is not displayed at the web page. a third type of information included in the export concerns the senses associated with the signs and their (form) variants. our work consists thus in porting all those (interlinked) resources to rdf and ontolex-lemon, as we did for the data described in section 4. in the omw version of dannet, we find for example the following information "00817680-n lemma beskyttelse", where the lemma corresponds to the omw english wordnet "00817680-n lemma protection", thus sharing the same id for the concept of "protection" in omw (this holds also for french, etc.). we can therefore add the danish sign id (and video), which we obtained from the database, to our rdf-based infrastructure. figure 1 : the danish sign associated with the omw id "00817680-n", corresponding to the (highlighted) lemma "beskyttelse", here as one possible lexical realisation of the danish gloss "forsvare" (defend) using the same strategy of deploying omw as a pivot between concepts expressed in the videos, we extended our approach to icelandic and swedish. through omw we can find the lemmas for icelandic and swedish associated with the omw ids "1128193-v" and "00817680-n" (corresponding to the danish lemmas). we use these to search in the icelandic signwiki, 13 and in the swedish sign language dictionary, described in 13 https://is.signwiki.org/index.php/ mesch et al. (2012). @xcite icelandic and swedish glosses can be easily integrated in our rdf-based representation, as can be seen for example in listing 1, where the gloss for the danish sign depicted in figure 1 is augmented with glosses or lemmas from other languages. ## a new transcription system for the irish sign language building on work dealing with linguistic properties of the irish sign language (murtagh, 2019), a group of researchers was confronted with the question of what is needed for creating an sl lexicon entry, as they wanted to document or "write down" what was being signed or articulated in the videos. while spl and sl share fundamental properties in relation to linguistic structure, certain modalityspecific linguistic phenomena must be accommodated in computational terms, to allow for the modelling and processing of sls. a new transcription system was developed for this, which, contrary to hamnosys or signwriting, is not based on iconic symbols, but directly encoded in xml. the sign a framework (murtagh et al., 2022) was developed with a view to providing a definition of linguistically motivated lexicon entries, that were sufficiently robust to accommodate sign language, in particular irish sign language (isl). sign a provides a formal description for the computational phonological parameters of sl. a sign a xml specification is provided for manual features (mfs), non-manual features (nmfs), location (both spatial and body anchored) information, and also temporal information. mfs include parameters for hand 〈hand〉, handshape 〈hs〉, hand movement 〈hm〉, palm orientation 〈po〉, arm movement 〈am〉, forearm 〈fa〉and upper arm 〈ua〉. in figure 2 , we can see the sign a xml representation for the hands, where the "dominant hand" is defined as 〈dh〉, and the "non-dominant hand" as 〈ndh〉. the nmfs include parameters for eyebrow 〈eb〉, eyelid 〈el〉, eye gaze 〈eg〉, cheek 〈cheek〉, mouth 〈mouth〉, tongue 〈tng〉, nose 〈nose〉, shoulder 〈shoulder〉, mouthing 〈mouthing〉, and mouth gesture 〈mouthgesture〉. the head element 〈head〉, contains a 〈head-mode〉 attribute, which can accept various actions pertaining to the head, e.g. nod, shake, tilt, turn, etc. we provide the xml specification for nodding the head twice in figure 3 . sign a also includes parameters to accommodate the location in space where a sign is articulated. the location parameters can be mapped to spatial locations 〈loc〉 within the signing space and also to locations on the signer's body, referred to as body-anchored 〈ba〉 locations. finally, the formalism also includes an xml specification for temporal information, where each phonological parameter has timing information associated with it, referred to as event duration 〈ed〉. sign a also includes a timeline parameter 〈tl〉, which refers to the overall timing of an utterance. this parameter is used to synchronise the simultaneous and parallel articulation of any given phonological parameter 'event' across an entire sl utterance. while sign a offers a very detailed description (and a taxonomic structure) of articulatory elements of sls, its xml encoding also eases the conversion of the data into rdf, a task we are start-ing on now. another relevant aspect of the work pursued in the context of sign a is the attention given to linking the described sign to spl lexical resources, as can be seen in figure 4 , which is taken from murtagh et al. (2022). porting this cross-language type linking to rdf and ontolex-lemon will contribute to a full linking between sl and spl lexical data, beyond the work described in sections 4 and 5, which focus on the specific multilingual wordnet-based lexical resources for cross-linking sl data. we plan to link the sign a sl data to the dbnary resource (sérasset and tchechmedjiev, 2014; sérasset, 2015) which represents lexical information extracted for 23 language editions available from wiktionary in a way compliant with linked open data. ## signnets -wordnets for a specific type of natural language in the northern part of belgium (flanders), the official language is dutch; in the southern part (wallonia), it is french. there are also two officially acknowledged sign languages, vgt (flemish sign language) and lsfb (french belgian sign language). dutch is also the official spoken language in the netherlands, but the officially acknowledged sign language is ngt (dutch sign language). in this section, we concentrate on vgt and ngt, and the link with another natural language: spoken dutch. @xcite vgt and ngt are rather different sls, having themselves developed quite independently. vgt tends to share characteristics with lsfb, even though they are growing apart. nev-ertheless, similarities between vgt and ngt are noted especially when dealing with iconic signs, or when mouthing plays an important role, since in both cases the surrounding spl is dutch. when linking via wordnet (omw) it should be stressed that the glosses assigned to signs in fact represent a (semantic) concept instead of just words, i.e. they represent spl synsets instead of a word belonging to such a synset. the gloss can even represent several parts of speech. glosses used for specific concepts 16 may differ in ngt and vgt, but even the two providers of ngt data 17 may use different glosses for one and the same sign. in all these cases, even the part of speech of the chosen gloss may differ. @xcite in contrast, vgt and ngt may use the same gloss for different signs. within vgt, one and the same gloss often represents a series of signs, all expressing the same concept. this is due to the regional variations of a sign, a property of vgt explicitly preserved by the deaf community after the official recognition of vgt. note that especially older variants may disappear, while new ones pop up. in the netherlands, the situation was the reverse: one sign per concept was pursued. @xcite the vgt gloss will express the common concept. in both the ngt signbank and the vgt dictionary, indicative translations in spoken dutch are included to indicate the concept expressed. quite often these represent several parts of speech like nouns and verbs, nouns and adjectives, etc. @xcite we are linking these to the synsets per pos included in omw, but are also creating new, broader identifiers to link them to sl concepts, surpassing pos differences. in signnet (schuurman et al., to be published), 21 vgt thus comes with synsets of signs, @xcite concept, not sign! @xcite nederlands gebarencentrum https://www. gebarencentrum.nl and the ngt part of the global signbank https://signbank.cls.ru. nl/datasets/ngt . @xcite in ngt the gloss for the concept covering 'arm' (poor) is behoeftig (an adjective), in vgt it is armoede (a noun). @xcite when in ngt more signs are covered using variants of the same gloss (behoeftig-a, behoeftig-b), quite often the coverage of the semantic concept differs. behoeftig-b can also mean 'broke', not only 'poor', which does not hold for behoeftig-a. @xcite vossen (1999) refers to such words as being near-synonyms, referring to the eq near synonym relation between 'aardig' (adjective) in dutch and 'to like' (verb) in english. @xcite based on sl dictionaries, signbanks etc. whereas ngt usually does not. in signnet signs (concepts) and words in spoken language are linked, using omw, and adding hyponyms, hypernyms, homonyms, definitions of the concepts, etc. there are at least two issues in doing so: first, omw makes use of open dutch wordnet, and odw (and omw) often use the dutch meaning of a word, not the flemish one. for example, 'voormiddag' refers to the hours before lunchtime in flanders, and after lunch in the netherlands. so we have to adapt odw (and omw) to cover such differences. we intend to do so by adding in odw (and omw) a 'geography' label "belg" to words that only are used in a specific sense in flanders ('kleedje' instead of 'jurk' (dress)) or "ned" when the word is only used in the netherlands ('kinderkopje' instead of 'kassei' (cobblestone)). a second issue: quite often concepts labelled by one gloss in vgt (and ngt) cover more than one synset in the wordnet of the surrounding language, for example when several parts of speech are involved. however, sometimes also smaller sets are used: artists using voice taken together (singer, actor) vs artists not using voice (ballet dancer, painter, ...). ebling et al. (2012) describe similar cases for the swiss-german sl. and for example when the sign is rather iconic, showing a vertical versus a horizontal movement. in dutch, there is the verb 'aanhaken' (hook on), used both to express hooking a painting on a hook in a wall (vertical) and hooking a trailer on a car (horizontal). in vgt and ngt, there are two different signs that respectively show a more vertically or horizontally oriented movement. because this difference is not made in spl, it is neither represented in odw nor omw, so we may need to adapt odw in this respect as well. considerations of the similarities and differences between the two variants of the dutch sls and of the dutch spls point to the need to properly address linguistic variations, if one wants to adequately interlink or align those variants across languages and language types. it seems that the current status of the omw infrastructure cannot offer wordnet ids to serve as pivot in those cases. we thus need to address those issues in the next steps of our representation work in rdf, and to investigate whether the current "vartrans" module 22 of ontolex-lemon is adequately formulated for this task. an important lesson we can retain from this section is that the generation of parallel data for sl and spl language variations is a challenging task. ## a first implementation of linking and aligning strategies in rdf/ontolex listing 1 has already shown how we can encode in rdf a danish gloss and augment it with glosses or lemmas from other languages, which we extracted via the shared ids implemented in omw, pointing back to the danish video equipped with the corresponding gloss. with the next listings, we would like to give an idea of how the rdf and ontolex-lemon representation ensures the accurate linking of information in a standardized and interoperable way. listing 2 shows the encoding of the danish video already displayed in figure 1 above, and listing 3 shows the rdf-based representation of the corresponding gloss. <http ://example.org/dts# signvideos_dts-722.@xmath0:type sl:signvideos ; sl:hasgloss dts:gloss_dts-722 ; sl:hasvideoadresss " https://www . the full rdf code will be made available in a github repository, so that interested colleagues can contribute to future developments. ## conclusion we proposed in this paper to investigate the possibilities of a harmonised representation of data from both spoken and sign languages that were originally stored in different formats in different locations. basing ourselves on the works and issues presented in sections 4, 5, 6 and 7, we propose the use of rdf and associated standardized vocabularies or models (like ontolex-lemon) to support an interoperable encoding for constitutive elements of both sl and spl resources and their interlinking and alignment, whilst also stressing the importance of following the principles of fair data. we hope in this way to create a semantically organized repository of cross-lingual (both sls and spl) data, especially in the field of low-resource sls, which can be of help for supporting the creation of data sets for training or evaluating nlp applications, thinking in the first place of automated translation.