sign.mt: Real-Time Multilingual Sign Language Translation Application (2024)

Abstract: This paper presents sign.mt , an open-source application for real-time multilingual bi-directional translation between spoken and signed languages. Harnessing state-of-the-art open-source models, this tool aims to address the communication divide between the hearing and the deaf, facilitating seamless translation in both spoken-to-signed and signed-to-spoken translation directions. To provide reliable and unrestricted communication, sign.mt offers offline functionality, crucial in areas with limited internet connectivity. It enhances user engagement by providing customizable photorealistic sign language avatars, encouraging a more personalized and authentic user experience. Licensed under CC BY-NC-SA 4.0, sign.mt signifies an important stride towards open, inclusive communication.The app can be used and modified for personal and academic purposes and even supports a translation API, fostering integration into a wider range of applications. However, it is by no means a finished product. We invite the NLP community to contribute towards the evolution of sign.mt . Whether it be the integration of more refined models, the development of innovative pipelines, or user experience improvements, your contributions can propel this project to new heights. Available at https://sign.mt , it stands as a testament to what we can achieve together, as we strive to make communication accessible to all.

## motivation sign language translation applications are crucial tools for enabling communication between individuals who are deaf or hard of hearing and those who communicate through spoken language. however, the complexity of developing sign language translation applications goes beyond handling mere text. these applications must be able to process and generate videos, demanding additional considerations such as compute capabilities, accessibility, usability, handling large files, and platform support. by providing a comprehensive application infrastructure that integrates the essential features around the translation process, sign.mt serves as a dynamic proof-of-concept. it aims to streamline the integration of new research findings into the application, sidestepping the overhead typically associated with implementing a full-stack application. when a research group develops a new model or improves a pipeline, they can integrate their advancements into the app swiftly, focusing only on their model. this approach allows researchers to deploy the app in a branch, testing their models in a practical environment. if the license allows and the models show an improvement, they can contribute their models to the main codebase. this is the first tool of its kind, diverging significantly from closed-source commercial applications. further, sign.mt serves as a multilingual platform, thus unifying the fragmented research landscape. it enables the concurrent running of models from different research groups for the supported languages, providing users with state-of-the-art translation capabilities for each language. through this, sign.mt not only enhances accessibility and communication but also fuels continuous innovation in sign language translation research. ## implementation sign language translation presents unique challenges that set it apart from text-based translation. while text-based translation operates entirely within the textual domain for both input and output, sign language translation involves cross-modal transformation -from text to video and vice versa. this demands distinct implementations not only in functionality but also in the user interface. it is essential to emphasize that the specific models utilized within various pipelines are deliberately modular and interchangeable. our current choice of models for each module or task is primarily opportunistic, driven by availability rather than performance metrics or user evaluations. the app serves as a dynamic orchestrator, seamlessly coordinating among these models to deliver an integrated user experience. the platform's design accommodates the likelihood that researchers or users may wish to experiment with different models or fine-tune existing pipelines, without being constrained by rigid implementation details. ## user engagement the impact of sign.mt can be measured by its widespread and consistent usage, highlighting the tremendous growth potential as the app continues to slowly improve. figure 3 depicts the global adoption of sign.mt, with users distributed across multiple countries. none of these top user countries are home to the core developer of the app. as shown in figure 4 , sign.mt demonstrates slow but consistent user growth (by google analytics), indicative of its reliability and sustained relevance. further validation of the community interest in sign.mt is evidenced by the increasing number of stars for its repository, reaching 470 stars as of october 6th, 2024 (figure 5 ). public interest in sign.mt is further supported by google search console metrics (figure 6 ), showing a significant increase in impressions and clicks over the past six months: 3.75k clicks (up from 1.56k), and 106k impressions (up from 24.4k). despite the absence of a marketing team and a single maintainer, sign.mt has managed to carve a niche for itself in the realm of nlp tools, reiterating its significance and impact. ## distribution the code for sign.mt is openly accessible and available for contribution on github at https: //github.com/sign/translate , under cc by-nc-sa 4.0. open sourcing with a permissive license encourages the continuous refinement and enhancement of the app through contributions from the wider developer and research communities. the web application is freely accessible at https://sign.mt , designed with a responsive layout to cater to both desktop and mobile devices. adhering to the design principles native to each platform, the application ensures an intuitive and user-friendly experience across all devices. with localization being a critical aspect of accessibility, the app interface supports 104 languages. contributors can add their language or enhance the support for existing languages. in addition to the web application, native builds for ios and android devices are also provided through the github repository. while these are currently in development, the plan is to make them available on the respective app stores as they reach stability, thereby extending the reach of sign.mt to a wider audience.