Meta-learning for Classifying Previously Unseen Data Source into Previously Unseen Emotional Categories (2021)

Abstract: In this paper, we place ourselves in a classification scenario in which the target classes and data type are not accessible during training. We use a meta-learning approach to determine whether or not meta-trained information from common social network data with fine-grained emotion labels can achieve competitive performance on messages labeled with different emotion categories. We leverage few-shot learning to match with the classification scenario and consider metric learning based meta-learning by setting up Prototypical Networks with a Transformer encoder, trained in an episodic fashion. This approach proves to be effective for capturing meta-information from a source emotional tag set to predict previously unseen emotional tags. Even though shifting the data type triggers an expected performance drop, our meta-learning approach achieves decent results when compared to the fully supervised one.

## introduction training a model for a classification task without having access to the target data nor the precise tag set is becoming a common problem in natural language processing (nlp). this is especially true for nlp tasks applied to company data, highly specialized, and which is most of the time raw data. annotating these data requires to set up a lengthy and costly annotation process, and annotators must have specific skills. it also raises some data privacy issues. our study is conducted in this context. it deals with private messages, that shall be annotated with emotions as labels. this task is highly difficult because of the subjective and ambiguous nature of the emotions, and because of the nature of the data. we tackle this problem in an emotion classification task from short texts. we assume that meta-learning can serve for emotion classification in different text structures along with a different tag set. predicting and classifying emotions in text is a widely spread research topic, going from polaritybased labels @xcite @xcite to more complex representations of emotion @xcite @xcite @xcite @xcite . in this paper, we place ourselves in a situation where we have no access to target data or models of target classes. therefore, we want to learn information from related data sets to predict labels on our target data, even though label sets differ. thus, we apply meta-learning using a few-shot learning approach to predict emotions in messages from daily conversations @xcite based on meta-information inferred from social media informal texts, i.e. reddit comments @xcite . with this setup, our goal is to investigate if combining few-shot learning and meta-learning can yield competitive performance on data of a different kind from those on which the model was trained. indeed, recent work already showed metalearning is useful when shifting to different topics on a classification task with the amazon data set @xcite or different entity relations on the dedicated few-rel data set @xcite . in this paper, we take another step forward by leveraging meta-learning when shifting not only emotional tag sets but also data sources, involving different topics, lexicons and phrasal structures. for instance, the "surprise" emotion is set for "wow you found the answer, wish you were on top, will link to you in my post" in goemotions @xcite and for "are you from south?" in dailydialog @xcite , varying both the lexicon used (post related vocabulary for goemotions) and the sentence structure (cleaner syntactic structures in dailydialog). our contribution relies on the implementation of a two-level meta-learning distinguishing data by their label set and data source at the same time. we also try to quantify the impact of switching data sources in this framework. after summarizing the related work (section 2), we present the data sets and labels (section 3) that we consider in our methodology and experiments (section 4). we then present the results (section 5) before discussing some key points (section 6) and conclude (section 7). the data preparation code and files, and the implementations are available in a public repository: ## related work emotion classification approaches @xcite @xcite @xcite @xcite @xcite usually benefit from using as many examples as possible when training the classifier. however, it is not always possible to obtain large data sets for a specific task: we need to learn from a few examples by applying specific strategies. few-shot learning @xcite @xcite is an approach dedicated to learn from a few examples per class and thus to create efficient models on a specific task. meta-learning. while they can be used for different purposes, few-shot learning frameworks are often used for meta-learning @xcite , defined as "learning to learn". like few-shot learning, meta-learning considers tasks for training but with the aim of being effective at a new task in the testing stage @xcite . to do so, metalearning can focus on different aspects such as learning a meta-optimizer (various gradient descent schemes, reinforcement learning, etc.), a metarepresentation (embedding by metric learning, hyper parameters, etc.), or a meta-objective (few-shot, multi-task, etc.), three aspects respectively represented as "how", "what" and "why" @xcite . both few-shot learning and metalearning approaches have mainly been developed in computer vision using different optimization schemes. the main meta-learning approaches use an episodic setting @xcite which consists in training on multiple random tasks with only a few examples per class. then, each task is an episode made of a number of shots (examples per class), a support set (set of examples to train from), a query set (set of examples to predict and compute a loss), and a number of ways (classes). optimization-based. optimization-based meta learning is an approach represented mainly by the model agnostic meta learning (maml) @xcite which learns parameters metainitialization and meta-regularization. it possesses multiple variations, such as first-order maml @xcite , which reduces computation; reptile @xcite , which considers all training tasks and requires target tasks to be close to training tasks; and minibatch proximal updates @xcite , which learns a prior hypothesis shared across tasks. another recent approach focuses on learning a dedicated loss @xcite . ## datasets and tag sets we consider two different english data sets to stay in line with our will to use a source data set on which the meta-model will be trained and a target data set on which we will evaluate the transferring capabilities of our model. goemotions @xcite is the data set we use to train and tune hyper-parameters. it is a corpus made of 58,000 curated reddit comments labeled with 27 emotion categories. we split it into 3 tag sets (emotagsets) for meta-training afterwards which detail later on. goemotions @xcite also comes with predefined train/val/test splits by ratio, ensuring the presence of all labels in each split. we use them to apply the fully supervised learning. dailydialog @xcite corresponds to the target data to be labeled using the meta-trained model. this corpus is initially structured as 13,118 human-written daily conversations, going through multiple topics; but for the purpose of our study, we only use it as individual utterances. we chose this corpus because of its propinquity with our case study: messages from conversational context are usually private and unlabeled. we retrieve utterances from the official test set with their associated emotion label, because studying the conversational context exceeds the scope of this paper. we only focus on utterances, language structure differences, and different emotion tag sets for meta-learning. this leads to a total of 1,419 utterances for 6 emotion labels (emotagset3). as for goemotions, dai-lydialog comes with official train/val/test splits that we use for comparison purposes while using supervised or meta learning approaches. tag sets. to apply meta-learning on emotion labels we consider 3 different tag sets named emo-tagsets. as previously said, we made these tag sets considering the different labels from each data set: let z g represent the set of goemotions' labels and z d the set of dailydialog's labels, we consider the intersection z d ∩ z g as the target labels named emotagset3. these target labels are the labels we want to hide from both training and validation phases to only use them during the test phase. the purpose of using the intersection is to enable results comparison on both data sets. the complement of the resulting intersection is then used to create emotagset1 and emotagset2, while taking into account class balance and polarity distribution to ensure each emotagset1 and 2 possesses a variety of classes. the resulting tag sets and their dedicated usage are visible in ## methodology and experimental protocol first, the objective is to retrieve label-level metainformation using reddit comments (goemotions) and the different label sets (emotagsets). then, we seek to transfer the meta-information to daily conversation-extracted utterances (dailydialog), hence varying in data structure and vocabulary. ## results ## discussions how do meta-trained models manage to perform on previously unseen tags? prototypical networks use the support set to compute a prototype for each class (i.e. way), hence new prototypes are computed for each episode. this means the trained encoder does not rely on predicting classes, but gathers representative information that will determine the position of the elements in the embedding space. because it is the relative proximity that serves to assign a query element to a specific prototype, having a different tag set that will be embedded "far away" should not hinder how well the model can classify data. ## conclusion in this paper, we are interested in a classification scenario where we only possess a certain kind of training data, with no guarantee that the testing data will be of the same type nor use the same labels. we choose our training data from common social media sources (reddit) with fine-grained emotion labels. we address this problem using meta-learning and few-shot learning, to evaluate our model on conversation utterances with a simpler emotion tag set. we consider metric learning based meta-learning by setting up prototypical networks with a transformer encoder, trained in an episodic fashion. we obtained encouraging results when comparing our meta-model with a supervised baseline. in this use-case scenario with a two-level meta-learning, our best meta-model outperforms both other encoder strategies and the baseline in terms of metalearning for nlp. moreover, our approach works well for learning emotion-related meta-information but still struggles while varying data types. for future work, we wish to investigate if this meta-learning approach could integrate the conversational context for classifying the utterances of the target dialog data. we also plan on applying this approach to another language than english.