IMU 2 CLIP : Language-grounded Motion Sensor Translation with Multimodal Contrastive Learning (2023)

Abstract: We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos – while preserving the transitivity across these modalities. We introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data – all using text as the grounding platform. In addition, we show that IMU2CLIP significantly improves downstream performances when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource. Our code and models will be released publicly.

## introduction with the growing popularity of smart glasses or new-generation wearable devices, first-person or egocentric videos have recently become prevalent @xcite @xcite . these egocentric videos are often accompanied by the parallel head-mounted imu sensor readings, which record devices' linear and rotational movements and accelerations. given its low power consumption level, imu is regarded as an important modality for powering various always-on on-device models that require understanding of device wearer's movement patterns (e.g. exercise / activity recognition for health applications) -which would not work with more batteryconsuming camera sensors. the previous works on imu modeling typically focus on the purposebuilt datasets with manual annotations @xcite , which are limited in their scale. consequently, the utilization of imu models in real-world scenarios has been confined to a relatively small number of use cases. on the contrary, for the modalities that are widely studied (e.g. text, video), there are vast large-scale resources such as bert @xcite and gpt @xcite ) for text, or clip4clip @xcite for videos. these powerful pre-trained resources have driven the development of many application-oriented models, showing significant improvements when fine-tuned for each respective task @xcite . to our knowledge, however, the study on the equivalent resources for imu signals has been lacking. inspired by the recent works studied for other modalities, we present imu2clip, a new approach to pre-train an imu encoder by aligning the parallel video↔imu↔text data. specifically, we propose to use clip @xcite , which contains the video and language models pretrained on the large image-text parallel data. the imu encoder thus learns semantic representations of various scenes transferred from other modalities. note that anchoring onto the joint text-vision space essentially allows for translation across these modalities. this new model capability opens up novel applications in the real-world wearable ai, such as imu-based media search and lm-based multimodal reasoning (fig. 1 ) -serving as efficient low-power alternatives to a more power-consuming approach of video-based media retrieval, which can operate under the strict on-device hardware constraints of wearable devices. in our evaluation, we provide an in-depth analysis on these newly proposed applications, as well as its efficacy on the downstream applications when fine-tuned. ## related work contrastive learning is as an efficient selfsupervised framework applied across multiple domains, which learns similar/dissimilar representations from paired data. for instance, simclr @xcite is a unimodal application of contrastive learning in the data augmentation setting, which proposes to learn a vision encoder given a set of perturbed images. as an example in multimodal settings, contrastive language-image pre-training (clip) @xcite learns visual representations from natural language supervision using image and text pairs, achieving competitive results in e.g. zero-shot image classification, image retrieval via text, and image/caption generation. similarly, wav2clip @xcite proposes to learn audio representation by distilling it from clip. we extend this line of work to a unique multimodal setting that utilizes imu signals, which is specific to a new generation of devices (such as smart glasses) that are equipped with such sensors. pre-training resources: there are numerous pretrained resources for well-studied modalities such as text or image. popular language models (lm) include bert @xcite , @xcite , and gpt-3 @xcite , which typically use self-superivsion techniques such as next-word predictions or masked token predictions, thus not requiring any explicit task labels. studies report that these pre-trained resources achieve competitive zero-shot performance @xcite , and when fine-tuned, often outperform fully supervised models on several downstream tasks @xcite . to our knowledge, the equivalent resource for imu is not made publicly available. we perform large-scale pre-training for the unique wearble sensor signals dataset, and show that such pre-training significantly improves downstream performances. egocentric datasets: we focus on egocentric (first-person) datasets, for understanding of users' activities from head-mounted devices. these include ego4d @xcite , epic-kitchens @xcite , and aria @xcite . using these datasets, we propose various sub-tasks that can effectively evaluate diverse capabilities of imu2clip, and demonstrate the feasibility of future applications. in addition, we implement a universal multimodal dataloader to allow for easy cross-modality and cross-domain studies. imu modeling: imu signals are used in various motion recognition tasks, such as pose estimation and walking speed estimation @xcite . various architectures @xcite have been explored for modeling imu in downstream tasks, including transformer-cnn based imu models @xcite . our work proposes a new imu model architecture, and conducts ablation studies over other models above. different from prior works that train imu models for specific tasks, however, our work focuses on learning multimodal imu representations by aligning imu with text and image, which can enable a wider set of downstream applications. ## methods in a nutshell, we train an imu encoder such that the imu representation of a given clip resembles the representation of its corresponding textual descriptions (narrations), or corresponding video frames. fig. 5 illustrates the overall approach. cross-modal contrastive learning. we consider a batch of b ground-truth imu↔video↔text parallel windows: {(i 1 , v 1 , t 1 ), ..., (i b , v b , t b )}, where the embeddings of each modality lies on the unit hypersphere s d . since they are unitnormalized, the similarity can be calculated as their inner products: sim(i i , v j ) = ⟨i i , v j ⟩ and sim(i i , t j ) = ⟨i i , t j ⟩. we then train three flavors of imu2clip: (a) aligning imu↔text, (b) imu↔video, and (c) imu↔video↔text. specifically, we project the imu representations into the joint clip space @xcite to leverage the visual and textual knowledge already encoded in clip. similar to @xcite , we propose to minimize the symmetric cross-modal contrastive loss (ablations on loss choices in appendix). for the imu↔text alignment, we use the sum of imu-to-text (i2t) and text-to-imu (t2i) cross-entropy losses: 4 experiments dataset. we use ego4d @xcite and aria @xcite as the main datasets for the experiments below, both of which feature largescale parallel video and imu signals. for ego4d, a subset of the clips are also annotated with text narrations. we split the data into train, validation, and test sets (split by video ids). the statistics of the datasets are provided in ## conclusions with the growing popularity of wearable devices of diverse form factors (e.g. smart glasses), it is imperative to study ambient motion sensors such as imu. we thus make the following contributions: 1) we propose a large-scale pre-training method for imu↔text, and release the pre-trained multimodal encoders for future research. (2) we provide an indepth empirical analysis for both upstream and finetuning tasks. (3) most importantly, we demonstrate the feasibility of many multimodal nlp applications for ambient sensors via their translation to text, which can spur future research.