H aptic C ap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals (2025)

Abstract: Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningfully with users is challenging. To facilitate this, we introduce a multimodal dataset and task, of matching user descriptions to vibration haptic signals, and highlight two primary challenges: (1) lack of large haptic vibration datasets annotated with textual descriptions as collecting haptic descriptions is time-consuming, and (2) limited capability of existing tasks and models to describe vibration signals in text.To advance this area, we create HapticCap, the first fully human-annotated haptic-captioned dataset, containing 92,070 haptic-text pairs for user descriptions of sensory, emotional, and associative attributes of vibrations. Based on HapticCap, we propose the haptic-caption retrieval task and present the results of this task from a supervised contrastive learning framework that brings together text representations within specific categories and vibrations. Overall, the combination of language model T5 and audio model AST yields the best performance in the haptic-caption retrieval task, especially when separately trained for each description category. The dataset is available at https://huggingface.co/datasets/GuiminHu/HapticCap.

## introduction haptic signals can convey information and emotions through programmable feedback experienced by users via the sense of touch. from touchscreen and vr interactions @xcite to gaming @xcite , rehabilitation @xcite and healthcare @xcite , haptic feedback has gained significant attention, with the market expected to exceed $28 billion usd by 2026. however, little is known about how users perceive hap- ## hapticcap dataset we present our process for creating hapticcap over 11 months, which involves compiling diverse signals, collecting user descriptions, and performing manual checks and data validation to ensure the dataset's diversity and quality.