Worldly Wise ( W o W ) - Cross-Lingual Knowledge Fusion for Fact-based Visual Spoken-Question Answering (2021)

Abstract: Although Question-Answering has long been of research interest, its accessibility to users through a speech interface and its support to multiple languages have not been addressed in prior studies. Towards these ends, we present a new task and a synthetically-generated dataset to do Fact-based Visual Spoken-Question Answering (FVSQA). FVSQA is based on the FVQA dataset, which requires a system to retrieve an entity from Knowledge Graphs (KGs) to answer a question about an image. In FVSQA, the question is spoken rather than typed. Three sub-tasks are proposed: (1) speech-to-text based, (2) end-to-end, without speech-to-text as an intermediate component, and (3) cross-lingual, in which the question is spoken in a language different from that in which the KG is recorded. The end-to-end and cross-lingual tasks are the first to require world knowledge from a multi-relational KG as a differentiable layer in an end-to-end spoken language understanding task, hence the proposed reference implementation is called Worldly-Wise (WoW).WoW is shown to perform end-to-end cross-lingual FVSQA at same levels of accuracy across 3 languages - English, Hindi, and Turkish.

## introduction imagine being able to ask your voice assistant a question in any language, to learn some trivia about your favorite movie star. this task falls in the realm of knowledge-based question answering (qa). one such challenging qa task is that of fact-based visual question answering (fvqa) @xcite which seeks to imitate how humans leverage background common-sense knowledge when answering visual questions. this task ensures that answering each question about an image requires external knowledge not directly available within the image or the text of the question. (see fig. 1 ). the external information is provided in the form of knowledge graphs, which are multi-relational figure 1 : example of a fact-based visual question question -which object in this image can be found in a jazz club? supporting fact -you are likely to find @xcite in @xcite ] subject, predicate, object -(trumpet, atlocation, jazz club) answer -trumpet graphs, storing relational representations between entities. the entities could be single words or phrases of words that denote objects or concepts. such tasks, though widely studied, exist mostly for well-resourced languages @xcite . these languages generally also have mature automatic speech recognition (asr) systems and language models. the accompanying knowledge graphs (kgs) also tend to be limited to languages that are well-resourced @xcite @xcite . against this background, it is worthwhile to think of building end-to-end systems which directly use speech signals as input, that can readily harness huge knowledge repositories stored in another language, instead of requiring tabula rasa learning. with these motivations, the main contributions of this paper are two-fold: 1) a new task referred to as fact-based visual spoken-question answer-ing (fvsqa) along with the release of 5 hours of synthetic-speech data in each of the three languages -english, hindi, and turkish. 2) an endto-end architecture worldly-wise (wow) capable of answering questions trained directly on speech features in all three languages. to the best of our knowledge, this is the first work to perform kg knowledge acquisition using only a speech signal as input, without the requirement for a pre-trained automatic speech recognizer as a system component. worldly-wise (wow) is readily generalizable to other languages, even those without an asrsystem. this is possible because of two reasons -a) it obtains speech features as mel-frequency cepstral coefficients and does not require asr-based text-conversion or speech feature extraction from a language-specific pretrained network, and b) for knowledge acquisition, it does not require the entity label to be in the same language as the question, instead leveraging neuro-symbolic entity representations in the form of kg embeddings. these kg embedding methods, trained to remedy kg sparsity by performing missing-edge prediction, learn transferable entity-features that encode the local and global structures in kgs. this also permits the architecture to use an image representation technique called 'image-as-knowledge' (iak). this uses a co-attention mechanism that attends to important entities in the image and time-steps in a question, thus allowing for improved answer retrieval. the iak technique was first presented by @xcite for the goal of performing fvqa over incomplete kgs, but is applied to a speech signal as opposed to a textual question. we revisit its important details below in the relevant sections. we report experimental results on synthetic speech data in the aforementioned diverse languages to demonstrate its effectiveness. hindi and turkish are simulated as under-resourced languages by denying the system access to any text, asr, or machine translation to or from those languages, thereby requiring the system to learn the mapping from hindi and turkish speech signals to the kg knowledge stored in english. through this work, we hope to motivate research in expanding spoken language understanding (slu) in underresourced languages through models which circumvent the need for parallel text labelled resources. for most of its history, slu was developed in a pipelined fashion, with asr feeding text to a natural language understanding system, e.g., to the best of our knowledge, the only published uses of slu with knowledge graphs that fit this description is @xcite . recent research in end-to-end multimodal slu bypasses the need for asr by leveraging a parallel modality such as image @xcite or video @xcite , or a non-parallel corpus of text @xcite , to guide learning speech embeddings such that the speech input can be used in a downstream task. in speech-based vqa applications, the most common approach is a two-step approach which consists of an asr followed by text-based vqa @xcite . however, these systems are not generalizable to under-resourced or unwritten languages for which we cannot train an asr system. therefore, in this study, we will explore using neural speech embeddings, which are guided by the information in the kg, for achieving fvsqa. ## task formulation this section introduces a new task called fvsqa and presents a new dataset collected for this task. ## our approach fig. 2 depicts the architecture we use for fvsqa. as shown in the figure, co-attention fuses an image i and question q to form a query vector ν. this query vector is then used to retrieve the answer from the kg as ŷ(q|i) = argmax @xmath0, i) t e. the following sections address representations of the question, kg, and image, the information fusion function ν(q, i), and the loss function. the image and kg representations are identical to those considered in (ramnath and hasegawa-johnson, 2020), however, their goal is different from ours, as they perform monolingual text-fvqa over incomplete kgs. ## experimental setup apart from the mfcc feature generation, the rest of the experimental setup is similar to that described in seeing-is-knowing @xcite . it is briefly recapped in the sections below. ## conclusion a new task fvsqa is presented in this work, along with an architecture that can perform cross-lingual knowledge acquisition for question-answering. in the process, we demonstrate the first task to perform knowledge acquisition directly using a speech signal as an input. this knowledge acquisition for speech can be extended to other tasks such as audio caption-based scene identification @xcite and multi-modal word discovery @xcite . future work will include extending fvsqa to a multi-speaker setting, gathering spo-3 https://kaldi-asr.org/models/m1 ken data from real-world speakers, as well as extending it to languages without an asr system.