Automatic Annotation Elaboration as Feedback to Sign Language Learners (2024)

Abstract: Beyond enabling linguistic analyses, linguistic annotations may serve as training material for developing automatic language assessment models as well as for providing textual feedback to language learners. Yet these linguistic annotations in their original form are often not easily comprehensible for learners. In this paper, we explore the utilization of GPT-4, as an example of a large language model (LLM), to process linguistic annotations into clear and understandable feedback on their productions for language learners, specifically sign language learners.

## introduction annotating linguistic data is a complex task, presenting ongoing challenges such as interpreting ambiguities and accounting for annotators @xcite . in the context of sign languages, this complexity is increased by the absence of common writing systems and codified grammars @xcite , along with the challenge posed by the simultaneous production of manual and nonmanual components foot_0 in expressing information. annotating sign language data is still a humanbased and extremely time-consuming process. as evidenced by the ongoing german sign language (deutsche gebärdensprache, dgs) corpus project, the annotation task proves to be highly laborintensive, demanding approximately up to 600 minutes to transcribe and annotate a single minute of signing @xcite . these challenges slow down advancements in (semi-)automatic annotation of sign language data. no computational tools at a production stage are currently capable of supporting the process of generating (high-quality) annotations as part of a semiautomatic setting. recent years have seen the rise of large language models (llms), enabling the annotation of large textual datasets. llms have proven effective in reliably annotating data by supporting human annotators @xcite . this concept of llm-based annotation has extended to language teaching and assessment @xcite . in this area, efforts have been made to provide language learners with formative feedback by processing data annotations, although only tested with written learner data @xcite . in this paper, we align with this latter research area and explain our idea of using linguistic annotations for providing feedback to sign language learners with the assistance of a large language model. first, we introduce in section 2 the process of annotating sign language (learner) data as well as the first approaches to using llms for providing feedback to learners based on previous written annotation. in section 3, we outline the annotation process and the annotation scheme. we provide a brief explanation of how our annotation process works, to make it accessible to a non-expert audience. our goal is to illustrate the steps necessary to generate formative feedback. then, in section 4, we present an experiment by employing @xcite to process the linguistic annotations from data of our corpus of continuous sign language learner productions. leveraging gpt-4 as an instance of llms, our goal is to transform the linguistic annotations into clear and understandable feedback to sign language learners. an evaluation of the feedback with sign language learners is also presented. the findings suggest that our initial approach holds promise in aiding sign language learning. ## annotating the corpus data in this section, we offer an overview of our annotation process, outlining the annotation scheme. this scheme encompasses the features extracted from the annotated data, which are then used to input into the gpt-4 model for our experiments (section 4). for our experiment, we utilized data from a swiss german sign language (deutschschweizerische gebärdensprache, dsgs) corpus including a longitudinal sign language learner sub-corpus and a corresponding sub-corpus of native signers. the l2 data was gathered from the same learners at four collection points separated by six-month intervals between march 2022 and november 2023. both l1 and l2 videos underwent post-processing and were imported into ilex (section 2.1). ## using gpt-for sign language annotation elaboration the primary aim of this experiment was to leverage the ability of gpt-4 foot_5 to produce coherent textual feedback from keywords representing linguistic annotations, intended to be presented to sign language learners. our final scenario is to incorporate the feedback generated through this process into a prototype sign language assessment system, providing additional feedback alongside visual hints. figure 1 shows a predecessor prototype system giving feedback on lexical items, i.e., on individual signs. note that our aim is to provide feedback on the production of continuous sign sentences, thereby naturally also touching on the correctness of production of non-manual components of signing. ## conclusion and outlook in this paper, we have presented the multifaceted challenges associated with annotating sign language data. we have covered the steps included in the annotation process and summarized the main features in a sign language annotation scheme. we have also reported on an experiment using gpt-4 to elaborate on linguistic error annotations from the scheme to provide more comprehensible feedback to sign language learners (cf. example 2 in the evaluation results demonstrated the successful application of gpt-4 in this task. they also obviated a need for more concision, evidence, and specificity in the generated error elaborations. the level of details of our annotation scheme may have constrained the quality of the generated feedback, leading to inconsistencies in gpt-4's treatment of non-manual errors, as shown in examples 4 and 57 in in a practical scenario, gpt-4 can be guided to provide additional explanations or customized support by using clear prompts that incorporated human-written explications. within the linguistic annotations of the corpus, we offer insights into errors tied to specific grammatical constructs. a future enhancement could involve instructing gpt-4 to generate feedback comments explaining linguistic concepts and grammatical structure, moving beyond solely error-focused elaborations. this shift could contribute to a more comprehensive and educational feedback system, aiding sign language learners in grasping the underlying linguistic principles. this aspect could be evaluated by performing a study comparing feedback generated by sign language experts with that produced by gpt-4. in future research, we also aim to explore fewshot methods to improve the model's capability of elaboration. we tried different promptings and adjusted settings. even though we aimed for a clear prompt, the choice of the 0.2 temperature parameter was somewhat arbitrary. future work could explore different parameter settings. lastly, in this paper, the numbers related to annotators, the amount of annotated data and evaluators may seem small compared to the data commonly used in natural language processing (nlp) tasks employing deep learning methods. yet this paper aims at enhancing knowledge and awareness of annotation efforts in sign languages, presently recognized as under-resourced languages @xcite . it sheds light on how the community addresses the demands for large, high-quality, and annotated datasets required for technological progress.