Few-Shot (Dis)Agreement Identification in Online Discussions with Regularized and Augmented Meta-Learning (2022)

Abstract: Online discussions are abundant with opinions towards a common topic, and identifying (dis)agreement between a pair of comments enables many opinion mining applications. Realizing the increasing needs to analyze opinions for emergent new topics that however tend to lack annotations, we present the first meta-learning approach for few-shot (dis)agreement identification that can be quickly applied to analyze opinions for new topics with few labeled instances. Furthermore, we enhance the meta-learner’s domain generalization ability from two perspectives. The first is domain-invariant regularization, where we design a lexicon-based regularization loss to enable the meta-learner to learn domain-invariant cues. The second is domain-aware augmentation, where we propose domain-aware task augmentation for meta-training to learn domain-specific expressions. In addition to using an existing dataset, we also evaluate our approach on two very recent new topics, mask mandate and COVID vaccine, using our newly annotated datasets containing 1.5k and 1.4k SubReddits comment pairs respectively. Extensive experiments on three domains/topics demonstrate the effectiveness of our meta-learning approach.

## introduction as seen in many online forums and subreddits, people express different opinions and perspectives towards a common topic in online discussions, by posting their comments towards a given question or replying directly to another user's previous comments. generally, we call the comments replying to another comments as response and the comments being replied as quote @xcite . detecting agreement and disagreement relations between (quote, response) pairs addressing a shared topic will enable many opinion mining applications and inform policy making. however, realizing that new topics keep emerging, it is unrealistic to expect existing annotated datasets to cover every topics of interest. to avoid the time-consuming process to create a large annotated dataset for a new topic, we study few-shot agreement and disagreement identification that aims to quickly build a model on a new topic domain with few labeled instances. to tackle the difficulty of few-shot agreement and disagreement identification under a new topic domain, we present a metric-based meta-learning approach that trains a meta-learner with two key abilities: deriving the class embedding from few provided support examples, and comparing the relation between new instance and each class embedding to make the prediction. specifically, the meta-learner is trained on annotation-rich domains with the technique of episodic training @xcite : in each training episode, k examples per class are sampled as support set and n examples as query set, each query instance is compared with the class embeddings derived from the support set and the loss is minimized on the query set. thus, when adapting to a new domain with very few labeled instances, the meta-learner has the ability of deriving the class embedding on the new domain from the provided few examples, and comparing a test instance with each class embedding via a learner relation network @xcite . inspired by prior research @xcite that studied rich domain-independent indicators of agreement and denial in online discussions, we further encourage the meta-learning system to learn domain-invariant features and thus enhance its ability of quickly generalizing to a new test domain. specifically, guided by @xcite , we complied a lexicon of domainindependent (dis)agreement indicators consisting of several hundreds of words or short phrases, e.g., indicators expressing agreement like "yes", "make sense", or conveying denial like "no", "but" etc, which can help our human beings distinguish (dis)agreement relation. furthermore, we designed a regularization loss based on the lexicon and added it to the meta-learning system, so that the meta-learner pay more attention to the domainindependent cues. the designed lexicon-guided regularization loss can incorporate human knowledge into the meta-learner, encourage the metalearner to focus more on domain-invariant features, so as to enhance its domain generalization ability. meanwhile, we design domain-aware task augmentation which decomposes the entire training dataset into clusters for episodic training, with each cluster corresponding to a topic domain, in order to better train the meta-learner to recognize domainspecific expressions of agreement and disagreements. existing labeled datasets for agreement and disagreement identification usually contain data instances from multiple topic domains. if we randomly sample from an entire dataset for each metatraining episode, many episodes have sampled instances in the support set and the query set that do not match in domain and have divergent data distributions. the domain mismatch will lead to poor transfer between support and query sets @xcite , and thus hinders the meta-learner from learning to recognize domain-specific expressions of agreement and disagreements. therefore, we perform domain-aware task augmentation for meta-training to strengthen the few-shot adaptation ability of the meta-learner, where we sample instances from the same domain to form support set and query set for each episode. in addition to using an existing dataset for evaluation, we also evaluate our approach on very recent new topics. we collected and annotated 1.5k comment pairs from subreddit on the mask mandate topic foot_2 , as well as 1.4k comment pairs from subreddit on the covid vaccine topic foot_3 in the year of 2021. experiments of both binary and three-class classification on the three domains/topics, shows that compared to the conventional fine tuning method and prompt-tuning method, the meta-learning approach achieves consistent and noticeable performance gains across the three domains under the challenging few-shot setting for (dis)agreement identification. both of the two strategies for strengthening the adaptation abil-ity of the meta-learner further improve the performance of the meta-learner, by enabling it to learn both domain-invariant cues and domain-specific expressions for (dis)agreement identification. to summarize, our contributions are as follows: meta-learning has been studied for years for few-shot learning. metric-based meta-learning learns an embedding function mapping individual instances into a representation space and learns a similarity function to calculate distance between two instances. several metric-based meta-learners have been proposed, including siamese network @xcite , matching network @xcite , prototype network @xcite and relation network @xcite ). another direction is optimization-based meta-learning @xcite that aims to learn a good initialization to make a neural model reach the optimal for a new task quickly. we focus on developing a metric-based meta-learning model on the basis of prototype and relation network models. meta-learning for nlp has been studied for many nlp tasks under the few-shot setting, including topic classification @xcite , entity relation classification @xcite , word sense disambiguation @xcite and event detection @xcite . mostly, prior works used metalearning to identify unseen new classes and treat a class as a task, however, we aim to identify (dis)agreement in unseen new domains and treat a domain as a task @xcite . domain generalization has been studied long before the emergence of meta-learning, aiming to generalize from a set of seen domains to unseen domains without accessing any instance from the unseen domain during the training stage. as a strategy to achieve domain generalization, @xcite @xcite proposed extracting domain-invariant features from various seen domains to enhance generalization ability. to the best of our knowledge, we lead on using domain-invariant features together with metalearning to enhance few-shot generalization ability across domains. task augmentation for meta-learning was first studied in @xcite . lack of welldefined data distribution is a recognized obstacle of meta-learning for solving nlp problems, generating attempts in augmenting meta-training tasks. @xcite proposed the smlmt method to create new self-supervised tasks. most closely related to our work is the strategy mentioned in @xcite which clustered the entire dataset into several clusters by k-means and sampled support & query set from the same cluster to form training tasks. our idea of task augmentation is different from theirs in that we relied on domain information to decompose the entire dataset into different training domains, creating clearer boundaries for different types of tasks. ## the meta-learning approach in this section, we will elaborate our meta-learning approach in details. firstly, we introduced the structure of the basic meta-learning model. then we enhanced the model's domain generalization ability from two perspectives: (1) manually created a lexicon for domain-independent (dis)agreement in-dicators, and designed a regularization loss to make the meta-learner focus more on domain-invariant features, (2) decomposed the entire training dataset into several sub-datasets based on domain-specific info to augment the task distribution. fig. 1 illustrated the pipeline of our meta-learning approach. ## conclusion in this paper, we developed a metric-based metalearning model for few-shot (dis)agreement identification problem. furthermore, we enhance the meta-learner's domain generalization ability from two perspectives: domain-invariant regularization through a lexicon-based regularization loss to learn domain-invariant features, and domain-aware task augmentation to learn domain-specific expressions.