An interface for mutual disambiguation of recognition errors in a multimodal navigational assistant
Users often have tasks that can be accomplished with the aid of multiple media-for example with text, sound and pictures. For example, communicating an urban navigation route can be expressed with pictures and text. Today's mobile devices have multimedia capabilities; cell phones have cameras, displays, sound output, and (soon) speech recognition. Potentially, these multimedia capabilities can be used for multimedia-intensive tasks, but two things stand in the way. First, recognition of visual input and speech recognition still remain unreliable. Second, the mechanics of integrating multiple media and recognition systems remains daunting for users. We address both these issues in a system, MARCO, multimodal agent for route construction. MARCO collects route information by taking pictures of landmarks, accompanied by verbal directions. We combine results from off-the-shelf speech recognition and optical character recognition to achieve better recognition of route landmarks than either recognition system alone. MARCO automatically produces an illustrated, step-by-step guide to the route.
speech recognition optical character recognition context-aware applications human-computer interaction scripting multimodal input