Defining meaningful units. Challenges in sign segmentation and segment-meaning mapping (short paper) (2021)

Abstract: This paper addresses the tasks of sign segmentation and segment-meaning mapping in the context of sign language (SL) recognition. It aims to give an overview of the linguistic properties of SL, such as coarticulation and simultaneity, which make these tasks complex. A better understanding of SL structure is the necessary ground for the design and development of SL recognition and segmentation methodologies, which are fundamental for machine translation of these languages. Based on this preliminary exploration, a proposal for mapping segments to meaning in the form of an agglomerate of lexical and non-lexical information is introduced.

## introduction the first steps for a machine translation (mt) pipeline which targets signed languages are: 1) defining a way to transcribe the sign stream that exhaustively describes all articulated features; 2) subdividing the transcriptions into units and 3) connecting these units to meaning. 1 in this work, we employ sign a @xcite to address the first step. sign a provides a detailed description of the (computational) phonological parameters that are essential to articulate the various phonemes, morphemes and lexemes of a sl utterance. in @xcite , sign a transcriptions are combined with role and reference grammar (rrg), a form of syntactic representation that considers semantic and communicative functions @xcite , i.e. addressing the third step; however, sign a is not automatically connected to rrg currently. so, how to connect sign a transcriptions to meaning is an open question at this stage; it largely depends on defining meaningful units that can be linked with meaning, i.e. step 2. in order to know what kind (or format) of meaning needs to be mapped to sign a (and vice-versa) and how, we need to know how the utterance is subdivided into parts. for example, if we consider a written utterance, the text is normally subdivided into tokens (words, punctuation 1 while current deep learning methods allow for efficient end-to-end approaches, the complexity of sls and the lack of annotated data makes the use of such methods infeasible. ## representations of signs to date, there is no tradition of writing signed languages @xcite . several sign notation systems have been developed, but none of them evolved into being widely accepted and used. in the 1960s @xcite defined a set of symbols to notate the components of each sign of american sign language (mostly intended for dictionary entries). later, in the 1970s, valerie sutton introduced a writing system for sl based on a dance notation, called 3 as in spoken languages, syntactic and prosodic constituents are non-isomorphic @xcite ; however, intonation and rhythm can provide useful information for sentence segmentation @xcite . 4 however, checking whether a coarticulation process only occurs within a prosodic domain and not across boundaries can be evidence of the existence of these boundaries and might be used for linguistic segmentation (in this respect, see @xcite . a limit to this approach is the optionality of coarticulation phenomena generally, which might prevent them from being reliable cues. signwriting. 5 it is made up of schematized iconic symbols for the hands, face and body, with additional notations for location and direction and intents to capture gestural behaviour in the flow of performance. more recently, the hamburg notation system (hamnosys) was created to transcribe signs from many different signed languages @xcite . it is a very detailed transcription system that was developed in conjunction with a standard computer font, mainly to be used for linguistic analysis. the notation of signs or a sl using any of these notation/writing systems results in a (more or less detailed) representation of the signs for their physical forms. representing the meaning of signs is most commonly done by using glosses consisting of words drawn from the spoken language of the surrounding community or in books and articles of the language of publication. glosses are most often used for representing the manual signs. typically, established signs are represented by capital letters glosses, and productive signs with several words. the use of glosses to annotate natural signed discourse is not without difficulty nor risk @xcite . for example, using words from a spoken language to represent the meaning of a sign can lead to an inappropriate semantic of grammatical analysis of that sign. another important problem is that there is no standardized way of glossing, and that gloss annotations differ between -and sometimes even within -corpora. the sign a framework @xcite was developed in the pursuit of defining a lexicon architecture that is sufficiently robust in nature to accommodate sl. the "a" in sign a refers to articulatory structure level. this level of lexical meaning aims to represent the essential (computational) phonological parameters of an object as defined by the lexical item. these parameters will be used to account for various linguistic phenomena pertaining to manual and non-manual features. rrg can be described as a structural functionalist theory of grammar and a functional model of language. rrg is a monostratal theory positing only one level of syntactic representation, the actual form of the sentence. therefore, there is only one syntactic representation for a sentence. this representation corresponds to the actual form of the sentence. leveraging rrg in combination with sign a allows for the development of a lexicon architecture capable of accommodating sl in computational linguistic terms. while our work focuses on sign a as a representation of a signed message, we acknowledge that the proposed method can be applied, after some adaptation, to signed messages represented in other notations. ## provisional proposal since nonmanual articulators add semantic information to manual articulators @xcite , it might be possible to use the manual articulators as bases for a segment, i.e. as a 'root' of an environment bigger than a word. we propose to map the sign a transcription to an 'enriched glosses' structure, where the lexical entry is enriched with the surrounding features conveying meaning (so having blocks like noun phrases or verb phrases). these enriched glosses can be compatible with and resemble glosses used for spoken languages (see the leipzig glossing rule foot_2 ). meaning can be implemented with rrg specifications or morphological information. these glosses (enriched with rrg) from one signed or spoken language could then be reconverted in either a spoken or a sl output through an mt approach (see, for instance, @xcite . using enriched glosses might prevent information loss that takes place when glosses for signed languages are used @xcite . glosses for signed languages are mostly used to transcribe lexemes only, while enriched glosses would include other pieces of information; for instance, the morphological suffixation that modifies the lexeme. with this approach, sl glosses would be similar to those of agglutinative lan-guages (e.g. turkish), which express grammatical information in an agglomerate of sub-units (i.e. many morphemes attach to one root); of course, a significant distinction remains: agglutinative languages units behave in a linear way like other spoken languages (i.e. one morpheme is attached next to the other, in a flat structure), while signs have simultaneous components. enriched glosses aim to address the structural complexity of these languages and to provide an exhaustive form of denoting meaning. being able to account for any meaningful element of the sign stream is a fundamental aspect for the preservation of the message, and for its efficient translation. ## conclusion in this paper we discussed the challenges to sign segmentation and to segment-meaning mapping. after an overview of the sl properties which need to be considered when addressing segmentation, we outlined a proposal, employing the sign a formalism, for connecting segments to meaning into an agglomerate of lexical and non-lexical information.