Improving Federated Learning for Aspect-based Sentiment Analysis via Topic Memories (2021)

Abstract: Aspect-based sentiment analysis (ABSA) predicts the sentiment polarity towards a particular aspect term in a sentence, which is an important task in real-world applications. To perform ABSA, the trained model is required to have a good understanding of the contextual information, especially the particular patterns that suggest the sentiment polarity. However, these patterns typically vary in different sentences, especially when the sentences come from different sources (domains), which makes ABSA still very challenging. Although combining labeled data across different sources (domains) is a promising solution to address the challenge, in practical applications, these labeled data are usually stored at different locations and might be inaccessible to each other due to privacy or legal concerns (e.g., the data are owned by different companies). To address this issue and make the best use of all labeled data, we propose a novel ABSA model with federated learning (FL) adopted to overcome the data isolation limitations and incorporate topic memory (TM) proposed to take the cases of data from diverse sources (domains) into consideration. Particularly, TM aims to identify different isolated data sources due to data inaccessibility by providing useful categorical information for localized predictions. Experimental results on a simulated environment for FL with three nodes demonstrate the effectiveness of our approach, where TM-FL outperforms different baselines including some well-designed FL frameworks.

## introduction aspect-based sentiment analysis (absa) is one of the most popular natural language processing (nlp) tasks aiming to predict the sentiment polarity (i.e., "positive", "negative", and "neutral") for an aspect term in sentences. currently, methods based on deep learning have been widely utilized for absa and demonstrated excellent potentials @xcite @xcite @xcite @xcite . however, these methods still reach a bottleneck if there is no enough labeled training data. one feasible solution for it is to leverage extra labeled data from other sources or domains. however, in real applications, these data are always stored in different locations (nodes) and are inaccessible to each other owing to privacy or legal concerns. to address the data isolation issue, federated learning (fl) @xcite is proposed and has shown its great promises for many machine learning tasks, such as user-computer interaction @xcite , medical image analysis @xcite , and financial data analysis @xcite . in some cases, data in different nodes are encrypted and aggregated to the centralized model, and they are invisible to each other during the training stage @xcite . this property makes fl an essential technique for real applications with privacy and security requirements. recently, fl has been applied to many downstream natural language processing (nlp) applications @xcite such as mobile keyboard prediction @xcite , language model training @xcite , representation learning @xcite , spoken language understanding @xcite , medical relation extraction @xcite , medical named entity recognition @xcite , and news recommendation @xcite . however, conventional fl techniques are more suitable for nodes sharing homogeneous data, which is seldom the case for nlp tasks be-cause text data are usually heterogeneous in vocabularies and expression patterns. particularly for absa, it is sensitive to the domain information, where one particular token may suggest completely different sentiment polarity in different datasets. therefore, the restricted data access in traditional federated learning approaches could result in inferior performance for absa since they cannot update the model using all domain information. unfortunately, limited attentions have been paid to address this issue. most existing approaches with fl on nlp (e.g., for language modeling @xcite , named entity recognition @xcite , and text classification @xcite ) mainly focus on optimizing the learning process and ignore domain diversities. in this paper, we propose a neural model based on fl for absa in a distributed environment, namely tm-fl, with a topic memory to enhance fl by providing categorical (topic) information for localized predictions, which can address the difficulty of identifying text sources caused by data inaccessibility. specifically, the topic model serves as a server-side component to read different inputs from each node and respond with categorical weights to help the backbone absa classifier. compared with previous absa studies that leverage extra features, e.g., document information @xcite , commonsense knowledge @xcite , and word dependencies @xcite , our approach offers an alternative to improve absa by leveraging extra labeled data through the fl framework enhanced by tm. experimental results on a simulated environment with isolated data from laptop, restaurant reviews, and social media (i.e., tweets), demonstrate the effectiveness of our approach, where tm-fl outperforms different baselines including the ones with well designed fl framework. ## the proposed method we propose tm-fl for absa and the overall server-node architecture of our approach is illustrated in figure 2 . the centralized model is stored in the fl server and data from multiple sources (domains) are stored at different nodes (the i-th node is denoted by n i ), respectively. encrypted information (e.g., data, vectors, and loss) communicates between each node n i and the fl server. in this way, the original data stays in the local node and is not accessible to the other nodes. to encode categorical information to facilitate localized prediction, we incorporate tm into the centralized model (figure 1 ). herein, fl encodes the topic information from the encrypted input and uses the encoded information to guide the centralized model to make a localized prediction. in the following texts, we introduce fl for absa and then the centralized model with tm. ## conclusion in this paper, we present tm-fl, a domain-aware topic memory network under the federated learning framework to enhance absa under the restriction of data isolation issues. specifically, our approach offers an alternative to enhance absa by leveraging extra labeled data through the fl framework improved by tm. experimental results on three widely used english benchmark datasets demonstrate the effectiveness of our method, which outperforms all the baseline models trained under the federated learning framework and competes for state-of-the-art performance on all datasets.