In this paper we present a data-driven model for detecting opportunities and obligations for a robot to take turns in multi-party discussions about objects. The data used for the model was collected in a public setting, where the robot head Furhat played a collaborative card sorting game together with two users. The model makes a combined detection of addressee and turn-yielding cues, using multi-modal data from voice activity, syntax, prosody, head pose, movement of cards, and dialogue context. The best result for a binary decision is achieved when several modalities are combined, giving a weighted F 1 score of 0.876 on data from a previously unseen interaction, using only automatically extractable features.
Using the resulting data REF creates a data driven classification after which utterance the robot should take the turn, utilizing voice activity, syntax, prosody, head pose of both persons, movement of cards, and dialogue context.
Opportunities and Obligations to Take Turns in Collaborative Multi-Party Human-Robot Interaction
SIGDIAL Conference
Computer Science