S cript B oard: Designing modern spoken dialogue systems through visual programming (2025)

Abstract: Implementation of spoken dialogue systems can be time-consuming, in particular for people who are not familiar with managing dialogue states and turn-taking in real-time. A GUI-based system where the user can quickly understand the dialogue flow allows rapid prototyping of experimental and real-world systems. In this demonstration we present ScriptBoard, a tool for creating dialogue scenarios which is independent of any specific robot platform. ScriptBoard has been designed with multi-party scenarios in mind and makes use of large language models to both generate dialogue and make decisions about the dialogue flow. This program promotes both flexibility and reproducibility in spoken dialogue research and provides everyone the opportunity to design and test their own dialogue scenarios.

## introduction in spoken dialogue system (sds) research the methodology or tools used to create interactions is often not fully described. a common approach is to use finite state machines, but these may grow unwieldy with more complex interactions because of the large number of states. behavior trees are another approach that is used for ai in video games @xcite , but programming these may require a steeper learning curve. researchers who are unfamiliar with implementing sdss often need time to learn how to manage aspects such as turn-taking and handling dialogue states in real-time systems, where human and robot turns are not neatly separated and system interruptions can be frequent. with no existing standards for the design of sdss, researchers need to create even simple interactions from scratch. furthermore, it is often difficult for these interactions to be reused and modified by others if they are written exclusively using code with no graphical interface. we propose that due to the above issues a gui should be used to assist in quickly designing replicable spoken dialogue interactions. there are several visual programming approaches which have been used in the literature, but these were developed before the rise of two research fields in sdss -large language models (llms) and multi-party interactions. an updated approach would need to accommodate these aspects in the system. llms drastically reduce the number of dialogue generation states needed in a visual programming interface since one llm node can handle both natural language understanding (nlu) and response generation. therefore the graphical design of complex interactions becomes more viable for novice users. in this demonstration we present scriptboard (script builder offering assistance with robot dialogue), a visual programming system which can be used to quickly develop sdss. scriptboard has the ability to handle multiple human participants, integrates prompt-based llms into the dialogue flow, and handles spoken dialogue features such as silence and barge-in. our program allows researchers to create spoken dialogue scenarios for both real-world implementation and experiments. the system is written in python using the pyqt package for the graphical interface. in this work we define the "user" as a person who designs the dialogue scenario and the "participant" as the person who is actually involved in the interaction. ## related work similar visual systems to manage robot interactions have been implemented in previous works @xcite @xcite @xcite @xcite . these often make use of a state-machine design in which the states of an interaction are set by the user and tracked during the interaction. state transitions in spoken dialogue systems are often triggered by au-tomatic speech recognition (asr) results, however this may not be appropriate for scenarios such as conversation where deciding when to speak (i.e. turn-taking) is crucial for smooth interactions. another issue is systems which are tightly coupled with a specific robot platform. for example, iristk is used to design interactions for furhat @xcite and choreographe @xcite is used for the nao robot. it would be preferable if users were able to reproduce interactions with different robots, particularly for comparative evaluation. existing systems are also usually made with the assumption of oneto-one interaction. iristk is made for multi-party interaction, however this is based on xml code and may be somewhat difficult for novice users. this issue of usability is also critical. some systems such as interaction composer @xcite and dialogos @xcite focus on making the interface understandable to novice users while more powerful systems such as rise @xcite are targeted towards researchers and may require more expert knowledge. recent technological advances mean that even novice users can use llms to generate dialogue without needing any low level programming as they can use prompt-based inputs to direct the system. a tool which could assist users to design sds scenarios in this way would be ideal for rapid prototyping and testing. frameworks such as retico @xcite and dialbb @xcite integrate llms into their systems although they are less focused on visual programming. scriptboard allows users to do everything within the gui. ## system architecture scriptboard is not a spoken dialogue system itself, but communicates with an external controller using tcp/ip messaging. the external controller handles speech recognition input from the user and turntaking and sends this information to scriptboard. this information is then processed using the scenario created in the gui by the user of scriptboard to decide the robot's behavior. scriptboard messages are then parsed by the controller and used for behavior generation. natural language processing and dialogue generation can be done through simple keyword comparators, but scriptboard allows users to use llms in their scenarios. this approach simplifies the creation of interaction scenarios because the user can hand off complex tasks to the llm rather than handcrafting individual dialogue states for the robot. ## user interface the user interface of scriptboard is based on the paradigm of dialogue states which has been used in previous visual programming systems @xcite . it uses a drag-and-drop mechanism in which the user can place states (also known as nodes) on the canvas. for a node, the leftmost connector represents an inbound connection to the node, while connectors on the right represent transitions out of the node. users connect these by dragging lines to other nodes to visually represent the dialogue flow. we use figure 2 as a reference interaction for this work. the toolbar at the top of the gui contains various nodes for the scenario. we create this interaction by connecting the following nodes which we will describe in more detail. human turn node which processes speech from the participant sets the target participant generates dialogue for the robot (handcrafted) ## additional features for more complex sdss, scriptboard has several other unique features for helping users customize their own scenarios. ## system usage once the user has created their dialogue scenario and it is connected to the external controller, they can simply push the "play" button in the top toolbar to start the interaction. the user can view the dialogue history and the variables in the scenario through a monitoring window. the gui also focuses the viewpoint on the current node to let the user know exactly where they are in the scenario, allowing them to visually track its progress. this is useful for debugging the logic of the scenario or identifying areas of improvement. the scenario should run autonomously until there are no more connected nodes, but it is possible for an interaction to run in a loop. scriptboard can be for a diverse number of systems and robots. we describe one such implementation in a multiparty setting. ## conclusion we demonstrate the scriptboard system, which we use to design and implement spoken dialogue systems using visual programming. scriptboard uses turn states as a basis for dialogue management and incorporates recent advances in llm technology and multi-party scenarios. it is independent of any agent or robot and we have used it in different types of scenarios to demonstrate its capabilities.