  We propose a promising neural network model with which to acquire a grounded
representation of robot actions and the linguistic descriptions thereof.
Properly responding to various linguistic expressions, including polysemous
words, is an important ability for robots that interact with people via
linguistic dialogue. Previous studies have shown that robots can use words that
are not included in the action-description paired datasets by using pre-trained
word embeddings. However, the word embeddings trained under the distributional
hypothesis are not grounded, as they are derived purely from a text corpus. In
this letter, we transform the pre-trained word embeddings to embodied ones by
using the robot's sensory-motor experiences. We extend a bidirectional
translation model for actions and descriptions by incorporating non-linear
layers that retrofit the word embeddings. By training the retrofit layer and
the bidirectional translation model alternately, our proposed model is able to
transform the pre-trained word embeddings to adapt to a paired
action-description dataset. Our results demonstrate that the embeddings of
synonyms form a semantic cluster by reflecting the experiences (actions and
environments) of a robot. These embeddings allow the robot to properly generate
actions from unseen words that are not paired with actions in a dataset.
