M o EM o E : Question Guided Dense and Scalable Sparse Mixture-of-Expert for Multi-source Multi-modal Answering (2025)

Abstract: Question Answering (QA) and Visual Question Answering (VQA) are well-studied problems in the language and vision domain. One challenging scenario involves multiple sources of information, each of a different modality, where the answer to the question may exist in one or more sources. This scenario contains richer information but is highly complex to handle. In this work, we formulate a novel question-answer generation (QAG) framework in an environment containing multi-source, multimodal information. The answer may belong to any or all sources; therefore, selecting the most prominent answer source or an optimal combination of all sources for a given question is challenging. To address this issue, we propose a question-guided attention mechanism that learns attention across multiple sources and decodes this information for robust and unbiased answer generation. To learn attention within each source, we introduce an explicit alignment between questions and various information sources, which facilitates identifying the most pertinent parts of the source information relative to the question. Scalability in handling diverse questions poses a challenge. We address this by extending our model to a sparse mixture-of-experts (sparse-MoE) framework, enabling it to handle thousands of question types. Experiments on T5 and Flan-T5 using three datasets demonstrate the model’s efficacy, supported by ablation studies.

## introduction the field of question-answer generation (qag) @xcite and visual question-answer generation (vqag) @xcite holds significant promise with extensive applications across various domains. recent advancements in large-scale language 1 equal contribution. 2 this work was done while author was in international machine learning team. models @xcite and vision models @xcite @xcite have demonstrated notable progress. however, current models are often constrained to qag tasks that use a single source of information, generating answers solely from either language or visual signals. in practical applications, handling multiple sources of information is crucial, as answer signals may exist in any or all sources. for example, when reading a paper or article, relying solely on textual content may be insufficient, requiring references to images for a more comprehensive understanding. similarly, in e-commerce platforms, questions related to attributes such as pattern, fabric, or material can be answered using diverse sources, including images, product descriptions, or external references. figure 1 illustrates this scenario: questions about attributes like pattern color can be inferred from images, while fabric and material are extracted from text descriptions. notably, certain attributes, such as pattern, may be present in both sources of information. recently, various models @xcite @xcite have emerged for answer generation, leveraging singlesource information from either images or text. to handle multi-source information, these models often rely on separate models for different sources, integrating their outputs through post-processing. novel frameworks, such as pam @xcite and mxt @xcite , have intro-duced multi-source, multi-modal generative approaches, showing promising results in attributerelated question answering. however, significant challenges remain in developing efficient mechanisms for training and integrating models to handle diverse sources of information. existing approaches face several limitations that hinder their effectiveness in answer generation tasks. firstly, reliance on textual data introduces language bias, potentially leading to skewed attribute generation. additionally, these models often neglect crucial visual information contained in product images, relying primarily on textual descriptions, which undermines the benefits of multisource and multi-modal data @xcite ). effective answer generation also requires selectively attending to the most relevant source, focusing on key visual or textual information within that source. furthermore, handling a diverse range of questions with a single model poses scalability challenges, necessitating expert models tailored to specific question types. unfortunately, models such as mxt @xcite and pam @xcite fail to address these limitations. the proposed model addresses the aforementioned limitations by incorporating a questionguided attention (qga) mechanism and a sparse mixture-of-experts (moe) model. the qga mechanism enables the model to autonomously discern attention patterns across multiple sources in scenarios involving diverse information streams. these attention patterns are tailored to the specific posed question. when the answer relies on visual information, the model focuses its attention on visual embeddings. conversely, when the answer is within the textual context, the model assigns higher weights to textual information. in cases where the answer is derived from all available sources, the model distributes attention appropriately across each source. while cross-modal attention aids in aligning different modalities, it is insufficient for acquiring robust attention patterns within a single source. to address this, we introduce separate embeddings for the question, context, and image, aligning question-image and question-context pairs by maximizing their correlation. this alignment process allows the model to learn precise attention patterns within individual sources based on the given question. given the diverse nature of the questions, a single model struggles to handle all question types effectively. to address this, we incorporate an moe strategy into our model, allowing experts to specialize in different question types. experiments on a large-scale multimodal dataset show state-of-the-art performance in attribute-based answer generation. ablation studies analyze the contribution of each model component. ## related work extensive work has been conducted on attribute answer extraction, which can be broadly categorized as extractive, predictive, and generative. extractive models tag each word in a description using named entity recognition (ner) and extract answers based on these tags. recent works such as opentag @xcite , latexnumeric @xcite , and mqmrc @xcite leverage ner for answer extraction. while effective for certain categories, these models face limitations in predicting novel entities, and defining entity classes remains challenging. furthermore, ner-based approaches rely solely on unimodal data, ignoring the richer context available in multi-modal sources such as text and images. predictive models form another popular category, where answers are predicted from predefined classes using classification models. these approaches accept unimodal or multimodal data with a question and predict attributes from a fixed set. cma-clip @xcite ) is a recent multimodal approach for attribute prediction. however, these models are limited to predefined attributes and cannot perform zero-shot inference. given the vast diversity and continuous growth of data, defining a fixed answer set is impractical, and managing large classifier sizes is challenging. generative models offer a more flexible solution by generating attributes rather than predicting or extracting them. these models take a question and unimodal or multimodal information as input. avgpt @xcite generates attribute answers using text data, while pam @xcite and mxt @xcite introduce multimodal generative frameworks. pam and mxt are closely related to our approach, both employing generative models in multimodal settings. however, mxt uses two image encoders @xcite and xception (chollet, 2017)), making image encoding computationally expensive, and relies on a joint encoder for questions and context. this design prevents direct interaction between the question and the image, limiting the model's ability to focus on relevant image regions. additionally, mxt uses a cross-modal proposed model addresses these limitations by using a single image encoder and learning patchwise attention, enabling efficient and precise focus on image regions. furthermore, question-guided attention facilitates attending to the most relevant mode across all sources of information. we also incorporate the moe @xcite , a recent advancement combining specialized expert models for specific tasks or modalities. moe has demonstrated significant performance improvements in decoder-only architectures, such as mixtral (ai, 2023) and moe-llava @xcite , compared to their non-moe counterparts. ## problem setting the proposed model solves the qag task, unlike standard vqa or qa tasks, our approach incorporates multi-source information, where the answer to a given question may originate from any of the available sources. we define the dataset as ## proposed model to obtain a robust and highly generalizable model, an approach is needed that can automatically attend to various sources in a multi-modal information scenario. to address this, our approach employs three encoders with unshared parameters for context, image, and question. we have developed a question-guided attention mechanism to automatically learn weights for different data sources. the following section provides a detailed discussion of the proposed model and its components. ## experiment and results this section, briefly discusses the datasets, baselines and the results obtained using the proposed model. ## ablations we conducted the extensive ablation over the various proposed components and figure -5 shows the results for the same. notably, we observed that the presence of question-guided attention and alignment loss had a substantial impact on the model's performance. in the absence of question guidance (woqg), the performance dropped from 59.9 to 58.1 for kurta pts and from 69.4 to 68.1 for shirt pts. when leveraging the joint encoder (s-enc), where the question and context are merged, the application of question-guided attention was not feasible, leading to a significant performance drop and the lowest results for both kurta and shirt product types. the alignment loss emerged as a crucial factor in directing attention within the source information, enabling the model to focus on the most relevant parts of the image or context. incorporating the alignment loss further enhanced the model's performance, raising it from 59.4 and 68.8 to 59.9 and 69.4 for the kurta and shirt pts, ## conclusions in this work, we introduce moemoe, a robust model designed for question answering from multisource, multi-modal information. our approach leverages automatic attention learning across diverse information sources, facilitating the identification of the most reliable source for robust answer generation. the proposed question-guided attention mechanism employs a dense-moe architecture combined with alignment loss and sparse-moe training in the intermediate layer, which significantly enhances the model's ability to extract robust features in a scalable manner. the moe-moe model achieves state-of-the-art results compared to recent baselines. the proposed attention mechanism, operating both between and within multiple sources, is versatile and applicable to various contexts. by incorporating alignment loss between question-context and question-image pairs, the model effectively explores attention within each source, enabling it to focus on the most pertinent parts of the image or context based on the given question. extensive experiments on a large-scale dataset, coupled with ablation studies, validate the efficacy of our approach.