Sign Language Translation: Ongoing Development, Challenges and Innovations in the S ign ON Project (2022)

Abstract: The SignON project (www.signon-project.eu) focuses on the research and development of a Sign Language (SL) translation mobile application and an open communications framework. SignON rectifies the lack of technology and services for the automatic translation between signed and spoken languages, through an inclusive, humancentric solution which facilitates communication between deaf, hard of hearing (DHH) and hearing individuals. We present an overview of the current status of the project, describing the milestones reached to date and the approaches that are being developed to address the challenges and peculiarities of Sign Language Machine Translation (SLMT).

## introduction signon 1 focuses on the research and development of a sign language (sl) translation mobile application and an open communications framework. signon addresses the lack of technology and services for the automatic translation between signed and spoken languages, through an inclusive, human-centric solution which facilitates communication between deaf, hard of hearing (dhh) and hearing individuals. we present an overview of the status of the project, describing the milestones and the approaches developed to address the challenges and peculiarities of sl machine translation (slmt). sls are the primary means of communication for over 70 million dhh individuals. @xcite despite this, they are rarely included in ongoing developments of natural-language processing (nlp) advancements (yin et al., 2021). machine translation (mt) research which targets sls is still in its infancy, due mainly to the lack of data and effective representation of signs (including the lack of a standardized written form for sls). both the low volume of available resources, as well as the linguistic properties of sls provide challenges for mt. furthermore, sls are visual languages, which presents yet another challenge: © 2022 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. @xcite signon is a horizon 2020 (research and innovation programme grant agreement no. 101017255) project that runs from 2021 until the end of 2023. ## the signon approach to slmt the objective of the signon project is mt between signed and spoken languages in all possible combinations, as well as the delivery of this service to the primary user groups: dhh and hearing users. the project revolves around 4 spoken (english, spanish, dutch, irish) and 5 sls, (isl, ngt, vgt, lse, and bsl -namely irish, dutch, flemish, spanish and british sl). addressing this many language pairs and directions on a pair-by-pair basis would require a substantial amount of time and effort, far beyond the scope of the project. signon employs an mt approach that (i) focuses on processing and understanding individual languages, (ii) employs a common multi-lingual representation (interl) to facilitate translation and (iii) uses symbolic as well as deep-learning methods for the synthesis of a 3d virtual signer. this approach involves automatic sl and speech recognition (slr and asr respectively), nlp, sign and speech synthesis, text generation and, most importantly, representation of utterances in a common frame of reference -an interlingual representation space based on embeddings and/or symbolic structures, the interl. the complexity and diversity of these processing steps require multidomain knowledge and expertise. furthermore, we chose this approach as there are only limited parallel resources available between signed and spoken/written languages. relying on techniques such as transfer learning, and pre-built nlp models (i.e. mbart (lewis et al., 2020)) will improve mt performance. we have built state-of-the-art models and components for slr , exploiting convolutional neural network-, recurrent neural network-and transformer-based models, natural-language understanding and mt based on mbart. we are developing approaches through wordnets and abstract semantic representation and synthesis based on language specific logical structures for sl, behavioral markup language and a 3d avatar rendering system. the asr component will tune to the use cases and to the speaker (including atypical speech from deaf speakers and speakers with cochlear implants). the asr addresses (i) privacy challenges (ii) adaption to communicative settings and (iii) extension to new data and languages. currently, english and dutch are ready; spanish is in progress. the transfer learning approach is adapted for irish. the asr works as a web service via a secure restful api. ## signon application and open framework the general architecture (figure 1 ) consists of a mobile application which connects users to the cloud-based mt platform. the signon app is the interface between the user and the signon framework which handles the internal data flow and processing. the framework executes the following steps. the source message (audio, video or text) and any relevant metadata coming from the mobile app is processed by an orchestrator which queues it towards the translation pipeline through a message broker. a dispatcher subscribed to the appropriate queue receives the message, invoking the relevant component depending on the type of input. after the required processing is complete, the message passes to the next stage of the pipeline until, finally, once the translation tasks are completed, the output message is produced in the requested format (text, audio or sign language avatar). the output is delivered to the app via the orchestrator. each component is encapsulated in a docker container and distributed over different machines. the first release of the signon mobile application is due in june 2022, and will then evolve to its final release at the end of the project (dec. 2023). the app will be available as open source and for free. ## societal impact along with the technological and academic innovations that come in terms of new models and methodsfor slmt, signon strives towards having a large societal impact. currently we face societal challenges such as clashes between the views of dhh and hearing people, with respect to usecases, technological importance and communication needs. we organized two sets of interviews with deaf participants, an online survey and we have two round tables planned. via workshops we inform both the research and user communities about the progress of signon and the state-of-theart in slmt. ## progress and next steps in the first 15 months of this project 8 academic papers were accepted for publication. these papers discuss slr, nlp, slmt as well as sl representations. at the time of writing more than 5 papers are under review. we have conducted focus group interviews with vgt, isl, lse and ngt signers as well as public and internal surveys.