Signed Coreference Resolution (2021)

Abstract: Coreference resolution is key to many natural language processing tasks and yet has been relatively unexplored in Sign Language Processing. In signed languages, space is primarily used to establish reference. Solving coreference resolution for signed languages would not only enable higher-level Sign Language Processing systems, but also enhance our understanding of language in different modalities and of situated references, which are key problems in studying grounded language. In this paper, we: (1) introduce Signed Coreference Resolution (SCR), a new challenge for coreference modeling and Sign Language Processing; (2) collect an annotated corpus of German Sign Language with gold labels for coreference together with an annotation software for the task; (3) explore features of hand gesture, iconicity, and spatial situated properties and move forward to propose a set of linguistically informed heuristics and unsupervised models for the task; (4) put forward several proposals about ways to address the complexities of this challenge effectively.

## introduction while signed languages are fully-fledged natural languages with sophisticated grammatical systems that are fully comparable to those of spoken languages @xcite , they are also in a completely different modality with such extreme complexity that has yet to be thoroughly studied and understood. much of our current language technologies are not effective on signed languages, as natural language processing (nlp) modeling approaches are often based on linguistic theories of spoken languages, and expect either speech or written text as input. this results in technology that may be inaccessible to deaf people where signed languages are their primary mean of communication and who strongly prefer using their native language than a spoken language @xcite , thus it is essential to extend nlp to signed languages. on the other hand, most of the recent research in sign language processing (slp) mainly focus on the visual component of signed languages and fail to address its linguistic challenges, such as coreference resolution @xcite . coreference resolution is a critical component of natural language understanding and higher-level nlp applications including information extraction, text summarization, and machine translation, yet it is completely unexplored for signed languages. although coreference relations have been studied in sign linguistics, computational models fall short in this area. resolving coreference in signed languages presents novel challenges as the meaning of pronominal signs are highly dependent on discourse and spatial context @xcite . tackling this problem will help us gain a better understanding of how grounding is achieved across different types of natural languages and in multimodal communication, and broaden the ability of current nlp systems to handle multiple modalities. in addition, achieving automatic coreference resolution for signed languages will enable technologies for sign language translation or provide educational tools for sign language learners, among many more. in this paper, we introduce signed coreference resolution (scr) (figure 1 ) as a new challenge for coreference resolution and slp. we present how coreference is established in signed languages and explore its features of gesture, discourse and spatial grounding for modeling. we then develop a software to annotate signed coreference and release dgs-coref, a german sign language (dgs) dataset to evaluate scr models. we propose a novel architecture based on multigraphs and linguistically-informed heuristics to perform unsupervised scr, which we hope to extend to other signed languages, as all signed languages exhibit similar properties and methods to establish referents in space @xcite . finally, we discuss the complexities and important considerations to take into account for scr, as well as suggestions for future directions of research. we believe that the development of scr will provide an important stepping stone to sign language understanding. ## related work coreference resolution aims to identify all references to the same entity in discourse and forms a core component of nlp. while automatic coreference resolution has been widely studied for various spoken languages @xcite , no existing work to our knowledge attempts to resolve coreference in signed phrases automatically. we refer to @xcite for an overview of the early coreference resolution algorithms, @xcite for the mention-pair model, entity-mention model, and ranking models, and @xcite for a more recent survey of deep-learning based approaches. ## coreference in signed languages coreference is a core property of natural language @xcite and signed language is no exception. expressed in the visual modality, signed languages use space to maintain discourse coherence and refer back to previously mentioned entities @xcite . moreover, research suggests that the ability to use space to ground referents is innate to humans @xcite . therefore, studying coreference in signed languages will give us a better understanding of fundamental phenomena of natural language and help us build tools in various communication systems that are expressed in the visual modality. ## data to evaluate scr models, we develop a small dataset of a signed language with gold coreference labels. the public dgs corpus @xcite ) is a dataset comprising 50 hours of annotated dialogue between two native signers of dgs. we use this dataset for the following reasons: (i) it is the largest publicly available dataset of a signed language containing gloss annotations at the time, which enables the extraction of enough instances of pronominal pointing to train our models; (ii) it is an open-domain collection of natural signing by 330 native signers, which more closely portray signing in the real-world than other datasets @xcite ; (iii) its annotations include pose estimations, specific glosses for different indexing signs as well as english and german translations, which we use during our modeling. although our study is limited to dgs, primarily because of the lack of adequate resources in other signed languages, research suggests all (studied) signed languages use signing space similarly to ground discourse entities and establish pronominal references @xcite . thus, we believe that the task we define and the modeling approach we propose are easily generalizable to other signed languages. ## model in this initial study of scr, we use dgs glosses and spatial features extracted from pose estimations of the signed phrases to remove the overhead of visual processing and model the linguistic aspect of the task while adequate sign language recognition resources are lacking. we jointly model mention detection and coreference resolution on glosses. ## results in this section, we discuss the strengths and limitations of our approach. as scr is a new challenge with no existing baseline, our proposed unsupervised model presents a strong baseline for subsequent works. ## conclusions and future work we present a new challenge for automatically resolving and evaluating coreference in signed languages. we also release the first dataset in german sign language with gold labels for coreference resolution, as well as a web interface to annotate coreference in signed languages. finally, we propose a novel model to perform unsupervised coreference resolution that relies on a multigraph-based architecture with new, linguistically-informed heuristics which provides a strong baseline for this task. our paper performs coreference resolution on glosses to remove the overhead of visual processing and focus on the purely linguistic aspect of signed coreference. future work involves modeling approaches that process signing videos directly that may more closely reflect real-world applications. we also leave for future work the resolution of nonindexing signs that also may serve a pronominal function, such as body shift and facial markers. our work can additionally be extended to studying other types of ambiguous signs, such as directional verbs where the subject and/or object are not explicitly signed but grounded in space. this task also provides the opportunity to explore ways studying scr can benefit spoken lan-guage understanding, particularly multimodal communication where meaning in spoken languages can also be conveyed through the visual modality, such as co-speech indexing gestures. we also hope that future efforts towards scr and slp in general, through close collaboration with signing communities, result in assistive technology that can help deaf students in education, research, and everyday communication in their preferred language.