rr SDS 2.0: Incremental, Modular, Distributed, Multimodal Spoken Dialogue with Robotic Platforms (2025)

Abstract: This demo will showcase updates made to the ‘robot-ready spoken dialogue system’ built on the Retico framework. Updates include new modules, logging and real-time monitoring tools, integrations with the Coppelia Sim virtual robot platfrom, integrations with a benchmark, improved documentation, and pypi environment usage.

## introduction the 'robot-ready spoken dialogue system' (rrsds) was introduced in @xcite based on the five requirements of modularity, multimodality, distributiveness, incrementality and alignment. modularity ensures that the system components can be independently integrated. multimodality allows the system to process various input forms. distributiveness enables components to function across different systems. incrementality processes inputs quickly in real-time. alignment ensures that inputs from different modalities are synchronized.the rrsds architecture continues to use retico @xcite which adopts the incremental unit framework @xcite where incremental units (ius) are passed between modules. rrsds users can define their own ius or use ones that are pre-defined. we report updates with new modules that maintain these requirements. upgraded modules new modules yolov11 rt-detr yolov12 sam mistypy dinov2 mistyrobot whisper misty facedetection wav2vec objectpermanence handtracking posetracking huggingface ## modules in this section, we explain the perception, robot, simulation, language understanding and generation, and system-level modules that have been upgraded or newly added to form rrsds 2.0. object detection & feature extraction yolo models are known for their high detection accuracy with real-time inference speed. yolov11 has a cnn-based architecture @xcite , while yolov12 has an attention-centric design that achieves state-of-the-art object detection accuracy in real-time @xcite . rrsds now includes both yolov11 and yolov12. we have also added the real-time detection transformer (rt-detr) model, which uses a hybrid encoder and transformer decoder to perform object detection @xcite , and the segment anything model (sam) for object segmentation (object masking as well as bounding box information) without labels @xcite , and a module for the dinov2 vision model, which differs from other models because its training is largely unsupervised, which may have better theoretical alignment for some robot tasks and systems @xcite . all vision modules use the same iu types to make integration and replacement seamless. local speech recognition two local automatic speech recognition (asr) models have been added to rrsds. whisper is an asr trained on 680k hours of multilingual and multitask data. the model uses an encoder-decoder transformer architecture @xcite . wav2vec has been trained on 960 hours of unlabeled audio data. while other models, including whisper, use spectrograms, wav2vec uses raw audio as its input @xcite . prior work has evaluated wav2vec's incremental asr capabilities as part of rrsds @xcite . misty ii the misty modules enable rrsds to interface directly with the misty robot through two components. mistypy connects to the robot via its ip address using the robot class and is initialized separately from standard rrsds modules. misty-robot provides access to robot functionalities, including receiving camera input and sending commands for movement and speech. the facedetec-tionmodule detects human faces in single images or video streams and outputs bounding box coordinates. it requires a video input source, such as the videomodule or the mistyrobot camera, and supports applications such as face recognition and gaze tracking. ## example system implementations in this section, we present some systems that demonstrate different usages of modalities in rrsds. cozmo on coppeliasim we offer an example system for controlling a virtual cozmo robot in coppeliasim, depicted in figure 1 . the full system uses an asr module, and three coppeliasim modules that process the virtual environment, cozmo's internal states, cozmo's virtual camera, cozmo's actions, and debugging. the system is simple in that it is built as a "verbal joystick" (e.g., move forward, stop, turn right, turn left). ## conclusion & future work while integrating rrsds 2.0 across physical and simulated robots, we faced challenges balancing real-time, multimodal interaction. in the future, we are exploring integrations with other benchmarks and the remdis incremental processing framework for language models @xcite .