Federated Meta-Learning for Emotion and Sentiment Aware Multi-modal Complaint Identification (2023)

Abstract: Automatic detection of consumers’ complaints about items or services they buy can be critical for organizations and online merchants. Previous studies on complaint identification are limited to text. Images along with the reviews can provide cues to identify complaints better, thus emphasizing the importance of incorporating multi-modal inputs into the process. Generally, the customer’s emotional state significantly impacts the complaint expression; thus, the effect of emotion and sentiment on complaint identification must also be investigated. Furthermore, different organizations are usually not allowed to share their privacy-sensitive records due to data security and privacy concerns. Due to these issues, traditional models find it hard to understand and identify complaint patterns, particularly in the financial and healthcare sectors. In this work, we created a new dataset - Multi-modal Complaint Dataset (MCD), a collection of reviews and images of the products posted on the website of the retail giant Amazon. We propose a federated meta-learning-based multi-modal multi-task framework for identifying complaints considering emotion recognition and sentiment analysis as two auxiliary tasks. Experimental results indicate that the proposed approach outperforms the baselines and the state-of-the-art approaches in centralized and federated meta-learning settings.

## introduction nowadays, social media platforms and online ecommerce websites provide users with the freedom to express their opinions and observations towards a product, an organization, or an event. customers who plan to buy a product usually base their decisions on customer reviews @xcite . as a result, commercial and retail firms regard product reviews as a significant source of 1 the dataset and code are available at https://github.com/appy1608/ emnlp2023-multimodal-complaint-detection knowledge, which they can use to design their advertising strategies and resolve any product-related concerns. this could also benefit customers by providing recommendations on the quality of goods or services they intend to buy. identifying complaint texts in natural language is critical for developers of downstream applications such as chatbots @xcite , commercial organizations to strengthen their customer service capabilities by identifying and resolving product-related issues (coussement and @xcite . the emotional state and sentiment of an individual have a significant impact on the intended content @xcite . however, emotion recognition is a far more subtle and fine-grained analysis than sentiment classification @xcite . the correlation between emotion and sentiment motivates us to consider customers' sentiment and emotion while analyzing complaints. we learn the tasks of complaint identification, emotion recognition, and sentiment classification in a multi-task setting to examine further the relationship between complaint, emotion, and sentiment. according to an analysis of relevant literature, text-based complaints have been previously analyzed based on semi-supervised strategies, different domains, degree of urgency, and feedback likelihood @xcite @xcite , @xcite . although multi-modal information sources (e.g., images in addition to text) could provide more information in identifying complaints, this has not been investigated to date, with one of the main reasons being the lack of multimodal datasets. deep-learning-based complaint detection techniques generally produce better outcomes. nonetheless, the datasets available for training in such a particular application area are often unbalanced, with the class of interest (complaints) being significantly underrepresented compared to the others. this decreases the efficacy of binary classifiers, thereby prejudicing the findings toward the dominant class (non-complaint) even though the minority class is of main interest. using more labeled data from different sources or domains is one possible option. however, in practical applications, such data are almost always stored in separate geographical locations or with different organizations and are unavailable to others due to individual privacy or legal considerations. federated learning (fl) @xcite helps organizations break down such data-related barriers by offering an expansive range of data available to them. the ability of learning to learn or meta-learning @xcite @xcite could be beneficial in data-scarce scenarios. organizations with recently launched products and a few training samples could benefit from using the meta-learning technique. in the healthcare domain, user health-record-related complaints often involve personal data gathered in providing healthcare services that may disclose details about the patient's medical history if made public; healthcare institutions are legally required to maintain privacy. in such cases, the proposed fl model safeguards end-user data and maintains privacy while providing healthcare services with a robust trained model. in this work, we extend a benchmark multimodal complaint dataset @xcite with additional review instances and manually annotate each review with the complaint, emotion, and sentiment labels. subsequently, we proposed a federated meta-learning-based framework for identifying complaints in a multi-modal multi-task framework. even though the dataset developed and used in our work is based on publicly available reviews and images, the proposed model could benefit organizations interested in collaborative learning while keeping their local data private. the key contributions of our proposed work are outlined as follows: ## multi-modal complaint dataset (mcd) publicly available complaint datasets, such as @xcite , only consider text-based complaints and exclude the associated sentiment and emotion classes of the complaints. for building the proposed framework, we use the benchmark multi-modal complaint dataset cesamard @xcite ) that consists of reviews and associated user-uploaded images from the e-commerce platform amazon, labeled with emotion and sentiment classes. we extend the cesamard dataset with an additional 3928 review instances, each annotated with emotion and sentiment in addition to complaint labels. here, we discuss the details of the extended multimodal complaint (mcd) dataset. ## proposed methodology in this section, we define our problem and go over the details of the proposed approach. the overall framework is shown in figure 2 . ## conclusion and future work in this work, we present a multi-task framework based on federated meta-learning for identifying complaints in a multi-modal context. the underlying system is a dual attention-based multi-modal multi-task framework for simultaneous optimization of complaint, emotion, and sentiment tasks. to encourage the study of multi-modal complaints, we extend the publically available multi-modal complaint dataset with 3928 reviews and user-uploaded images collected from the amazon website and annotated with the complaint, emotion, and sentiment classes. experimental results indicate that the proposed approach outperforms the baselines and the state-of-the-art approaches in centralized and federated meta-learning settings. in the future, we aim to extend this work with more fine-grained complaint severity annotation and identify complaints at the sentence level. we plan to work on the security aspect and resource usage in future research concerning federated metalearning.