Motivated by psycholinguistic findings that eye gaze is tightly linked to human language production, we developed an unsupervised approach based on translation models to automatically learn the mappings between words and objects on a graphic display during human machine conversation. The experimental results indicate that user eye gaze can provide useful information to establish such mappings, which have important implications in automatically acquiring and interpreting user vocabularies for conversational systems.
In REF ), a translation model was developed to associate words with visual objects on a graphical display.
Automated Vocabulary Acquisition and Interpretation in Multimodal Conversational Systems
45th Annual Meeting of the Association of Computational Linguistics
Computer Science