Learning Adverbs with Spectral Mixture Kernels (2024)

Abstract: For humans and robots to collaborate more in the real world, robots need to understand human intentions from the different manner of their behaviors. In our study, we focus on the meaning of adverbs which describe human motions. We propose a topic model, Hierarchical Dirichlet Process-Spectral Mixture Latent Dirichlet Allocation, which concurrently learns the relationship between those human motions and those adverbs by capturing the frequency kernels that represent motion characteristics and the shared topics of adverbs that depict such motions. We trained the model on datasets we made from movies about “walking” and “dancing”, and found that our model outperforms representative neural network models in terms of perplexity score. We also demonstrate our model’s ability to determine the adverbs for a given motion and confirmed that the model predicts more appropriate adverbs.

## introduction with technological innovations in artificial intelligence, the widespread use of household robots that collaborate with humans to assist them in their daily lives is becoming a reality. in order for these robots to collaborate with humans, it is important to share and understand their experiences through language, because language is the most convenient communication tool capable of conveying human experience and knowledge. with this background, research on language use by robots in the real world has been actively studied @xcite @xcite . significantly, within this domain, large-scale language models (llms) such as openai's chatgpt 1 and google's palm @xcite are also used to control robots. chatgpt is used to execute various types of robotics tasks @xcite , and palm-saycan @xcite ) and 1 https://chat.openai.com . palm-e @xcite have been developed based on palm @xcite . @xcite and @xcite have proposed methodologies for generating task plans for robots that employ llm. their approach conveys robot's motion plans through a chain-of-thought framework @xcite . though they are good at describing the general plan of action of a robot in language to accomplish a specific task, the language description does not capture the precise correspondence between nuanced expressions and the actual robot behaviors in the real world. furthermore, the focus of their studies is not on the verbal representation of the behaviors of the observed object by a robot, but on the robot's action plan. for the advancement of robotics, it becomes imperative to comprehensively and statistically grasp the repertoire of "motions" that humans genuinely exhibit, as well as discern the variations in individual characteristics and contextual nuances associated with them. these insights should be aptly assimilated within the robotic systems. building upon the aforementioned, we shall address this challenge by casting our focus on adverbs to mathematically establish a correspondence between motions and adverbs that represent them. ## related work research is being conducted to elucidate the relationship between motions and the natural language that describes them. bidirectional conversion models from natural language descriptions to motions, or vice versa, using sequence-to-sequence (seq2seq) @xcite learning have been proposed by @xcite ; @xcite ; @xcite . though these models can achieve bidirectional conversion between language and motion sequences, it lacks in learning the correspondence between the manner of motions and the language that represent them. furthermore, in the conventional research the fo-cus has predominantly revolved around finite motions, such as "take" and "put", which were preconceived by humans, thereby neglecting the pursuit of methodologies that facilitate the adaptable modulation of multiple motions contingent upon contextual cues. in this paper, we focus on capturing the relationship between verbs and adverbs to flexibly express actions using the function of adverbs. limited research has been conducted thus far to delve into the semantic comprehension of adverbs. notable instances within this domain include the three-stream hybrid model @xcite , which employs long short-term memory (lstm) @xcite and inceptionv3 @xcite to acquire knowledge related to adverbs. additionally, action modifiers @xcite , which employ an i3d network @xcite and scaled dot-product attention @xcite to discern the impact of adverbs on motion sequences. these models employ image features derived from videos, such as rgb and optical flow @xcite , as representations of motions. however, these representations fail to capture the intrinsic essence of the motions themselves because they are unable to discern the component of motions denoted by the adverb. therefore, unlike conventional research approaches, in this study, we focus on the frequency components that make up human motion and attempt to express the motion by those components. by doing so, we aim to enable the robot to understand the meaning of adverbs related to motions such as "cut roughly", "dance dynamically", and so on. ## joint topic model of motions and adverbs for this purpose, we propose a new topic model, hierarchical dirichlet process-spectral mixture latent dirichlet allocation (hdp-smlda) to capture the relationship between the frequency components of human motions and the adverbs that describe them. the proposed model makes it possible to establish a statistical correspondence between adverbs and nuances associated with motions. this enables the control of robot actions through verbal instructions, such as "handle with more caution" or "cut roughly", and it is also possible to make the robot understand human intentions due to slightly different manner of movement. on the contrary, from the perspective of natural language processing, it has been impossible to express the actual meaning behind words like "freely" or "flexibly". however, the integration with robotics makes it possible for the first time to represent their meaning, allowing not only the description of actions through language but also the generation of actions from language cues. ## experiments we begin by providing a description of the datasets utilized in our experiments. we then proceed to conduct an experiment involving hdp-smlda, where we examine the adverbs and frequency components, and generate adverbs based on the frequency components within the trained model. ## conclusion we have proposed a joint topic model named hdp-smlda, which aims to comprehend the semantic nuances of sensory adverbs pertaining to human motions by learning co-occurrence relationships between motion features and adverbs. within our framework, adverbs are modeled as a composite distribution within the frequency space of their kernels in a gaussian process that represents the latent trajectory of motions. consequently, it becomes feasible to estimate the constituents of sensory adverbial motions. when compared to the simple neural network model, our model exhibits superior performance on classification of adverbs. our approach considers motions as a mixture of diverse frequency components, leading to the successful generation of appropriate adverbs from motion features in our empirical investigations.