S wiss ADT : An Audio Description Translation System for S wiss Languages (2025)

Abstract: Audio description (AD) is a crucial accessibility service provided to blind persons and persons with visual impairment, designed to convey visual information in acoustic form. Despite recent advancements in multilingual machine translation research, the lack of well-crafted and time-synchronized AD data impedes the development of audio description translation (ADT) systems that address the needs of multilingual countries such as Switzerland. Furthermore, most ADT systems rely on text alone, and it is unclear whether incorporating visual information from video clips improves the quality of ADT outputs.In this work, we introduce SwissADT, an **emerging** ADT system for three main Swiss languages and English, designed for future use by our industry partners. By collecting well-crafted AD data augmented with video clips in German, French, Italian, and English, and leveraging the power of Large Language Models (LLMs), we aim to enhance information accessibility for diverse language populations in Switzerland by automatically translating AD scripts to the desired Swiss language. Our extensive experimental ADT results, composed of both automatic and human evaluations of ADT quality, demonstrate the promising capability of SwissADT for the ADT task. We believe that combining human expertise with the generation power of LLMs can further enhance the performance of ADT systems, ultimately benefiting a larger multilingual target population.

## introduction ad denotes the process of acoustically describing relevant visual information that renders streaming media content in television or movies and other art forms partly accessible to blind persons and persons with visual impairment @xcite @xcite . this service involves the creation of textual descriptions, so-called "ad scripts", of key visual elements of a scene, such as actions, environments, facial expressions, and other important details that are not conveyed through dialogue, sound effects, or music @xcite . they are typically inserted into natural pauses that do not interfere with the ongoing narration. ad scripts are voiced by a professional human speaker or synthesized by a computer and mixed with the original audio. despite recent advancements in multilingual machine translation @xcite and large language models (llms) research @xcite , two major challenges remain unsolved in developing well-performing adt systems. firstly, many adt systems are built on pre-trained machine translation models that need texts in both the source and target languages as inputs. training these adt systems requires large amounts of manually crafted data, leading to high operational costs @xcite . secondly, existing adt systems are predominantly text-only machine translation models, neglecting the visual modality which is paramount for the adt task and has proven to be useful as part of multimodal machine translation @xcite . in switzerland, the primary target group of ad users comprises approximately 55,000 blind persons and 327,000 persons with visual impairment @xcite . meeting the accessibility demands of switzerland's multilingual population requires high-quality translation solutions. in this work, we address the aforementioned challenges by developing an adt system specifically for the three main languages of switzerland, i.e., german, french, and italian. to create training data for llm-based adt models with minimal human effort, we utilize deepl foot_2 with english as an auxiliary language to generate ad scripts in the three swiss languages. to verify if llms are a potential solution to adt task, we conduct automatic and human evaluations of llm-generated ad scripts. additionally, to further improve the translation quality, we incorporate video clips as part of the inputs to the llm-based adt models. our contributions are: 1) we propose swissadt, the first multilingual and multimodal adt system for swiss languages; 2) we conduct extensive evaluations of our adt systems using both automatic and human quality assessments; 3) we highlight the system's emerging potential for real-world multilingual adt applications; and 4) we provide the source code for swissadt, which is easily installable for reproducibility. ## related work the automatic generation of ads from video clips has been explored by both the natural language processing (nlp) and computer vision (cv) communities. this research is often conducted as part of tasks such as video captioning (generating descriptive text for a video) or video grounding (temporally aligning a text query with video segments). a second line of research explores the feasibility and suitability of applying machine translation models for adt which was originally conceived as a human task. in the study conducted by @xcite , the creation, translation, and post-editing of english-catalan ad script pairs were extensively investigated to assess whether machine-translated ad scripts achieved satisfactory quality. the authors found that machine translation models can serve as a feasible solution. @xcite studied english-dutch ad script pairs and found that errors were prevalent in the machine-translated ad scripts, indicating that post-editing by human experts was necessary. in contrast to some of the above studies, we show that introducing visual inputs to adt systems can lead to improved results, as verified by our ad professionals during the human evaluation. ## swissadt: an adt system for swiss languages swissadt is a multilingual and multimodal llmbased adt system that translates ad scripts between english and the three main languages of switzerland with visual and textual input. it contains three basic components: moment retriever to identify the most relevant moment (that is, a sequence of consecutive frames) in a video clip for a given ad segment, we initially select a video segment that spans from ten seconds before the ad's start runtime (onset) to ten seconds after its end runtime (offset). 4 we then apply the video temporal grounder cg-detr @xcite , which takes in both the ad script and the selected video segment and outputs the most relevant moment of variable length by providing the start and end times, along with a grounding score. the final moment is retrieved by selecting the highestranked moment with the highest grounding score from the pool of candidate moments. frame sampler we linearly sample multiple video frames from the retrieved moment. 5 these frames are then utilized as visual inputs of the ad translator. we empirically report results on using four frames and every 50th frame. foot_5 ad translator we deploy multilingual and multimodal llms as the backbone ad translator of swissadt. we conduct experiments with the fundamental gpt-4 models gpt-4o and gpt-4-turbo. we decide to apply zero-shot learning as part of a cost-effective solution. our modularized implementation of swissadt streamlines the integration of state-of-the-art llm research outcomes. this design allows for the seamless incorporation of cutting-edge moment retrievers and ad translators with minimal effort. ## conclusions and future work in this work, we present swissadt, a multilingual and multimodal adt system designed to support three swiss languages and english. our findings demonstrate that leveraging llms to address the adt task represents a significant initial step towards achieving information accessibility, as validated by our experienced ad experts. this system provides a viable solution for enhancing accessibility for blind and visually impaired individuals in multilingual settings. future research will focus on fine-tuning llms for swiss languages, improving system robustness to real-world data variability, and deploying the system with our industry partners. additionally, we plan to conduct post-editing studies to further validate swissadt's potential for real-world applications, ensuring high-quality outputs that minimize human effort while supporting professional workflows. post-editing data will also be used to refine and improve the models over time. we believe that integrating human expertise into the llm pipeline for the adt task will more effectively meet end users' expectations and satisfaction. as with any accessibility technology, it is paramount that it serves the needs of the end users.