On the Hidden Negative Transfer in Sequential Transfer Learning for Domain Adaptation from News to Tweets (2021)

Abstract: Transfer Learning has been shown to be a powerful tool for Natural Language Processing (NLP) and has outperformed the standard supervised learning paradigm, as it takes benefit from the pre-learned knowledge. Nevertheless, when transfer is performed between less related domains, it brings a negative transfer, i.e. hurts the transfer performance. In this research, we shed light on the hidden negative transfer occurring when transferring from the News domain to the Tweets domain, through quantitative and qualitative analysis. Our experiments on three NLP taks: Part-Of-Speech tagging, Chunking and Named Entity recognition, reveal interesting insights.

## introduction high performing nlp neural tools often require huge volumes of annotated data to produce powerful models and prevent over-fitting. consequently, in the case of social media content (informal texts) such as tweets, it is difficult to achieve the performance of state-of-the-art neural models on news (formal texts). the last few years have witnessed an escalated interest in studying transfer learning (tl) for neural networks to overcome the problem of the lack of annotated data. tl aims at performing a task on a target dataset using features learned from a source dataset @xcite . tl has been proven to be effective for a wide range of applications @xcite @xcite , especially for low-resourced domains. however, it has been shown in many works in the literature @xcite @xcite @xcite @xcite that, when source and target domains are less related (e.g. languages from different families), sequential transfer learning may lead to a negative effect on the performance, instead of improving it. this phenomenon is referred to as negative transfer. precisely, negative transfer is considered when transfer learning is harmful for the target task/dataset, i.e. the performance when using transfer learning algorithm is lower than that with a solely supervised training on in-target data @xcite . several works @xcite @xcite have shown that sequential transfer learning from the news resource-rich domain to the tweets low-resource domain enhances the performance of sequence labelling of tweets. hence, following the above definition of negative transfer, transfer learning from news to tweets does not beget a negative transfer. contrariwise, in this work, we rather consider the hidden negative transfer, i.e. the percentage of predictions which were correctly tagged by random initialisation, but using transfer learning falsified. in this work, we take a step towards identifying and analysing the impact of transfer from news to tweets. precisely, we perform an empirical analysis to investigate the hidden negative transfer. first, we show in section.5.1 that, the final gain brought by tl can be separated into two categories: positive transfer and negative transfer. we define positive transfer as the percentage of tokens that were wrongly predicted by random initialisation, but the tl changed to the correct ones. in comparison, negative transfer represents the percentage of words which were tagged correctly by random initialisation, but using tl gives wrong predictions. then, in section.5.2, we study the impact of pretraining state on negative and positive transfer. finally, in section.5.3, we provide some qualitative examples of negative transfer. our experiments on three nlp tasks (part-of-speech tagging (pos), chunking (ck) and named entity recognition (ner)) reveal interesting insights. the remainder of the paper is organised as follow. we first present, briefly, the sequence tagging neural model ( §2). then, we describe the sequential transfer learning method ( §3), followed by a short presentation of the involved datasets and tasks ( §4.1). then, we report the results of our analysis to highlight the hidden negative transfer occurring when transferring from news to tweets ( §5). finally, we wrap up with a conclusion and future work ( §6). ## sequence tagging neural architecture we perform experiments on 3 sequence tagging (st) tasks: part-of-speech tagging (pos), chunking (ck) and named entity recognition (ner). given an input sentence of n successive tokens @xmath0, . . . , w n ], the goal of a st model is to predict the tag c @xmath1, with c being the tag-set. we use a common st neural model. it includes three main components. first, we have a wre (word representation extractor) to build, for each word w i , a final representation x i combining two hybrid representations; a word-level embedding (denoted υ word ) and a character-level embedding based on a bidirectional-long short-term memory (bilstms) encoder (denoted υ char ). second, the x i representation is fed into a features extractor (fe) (denoted φ) based on a single-layer bilstms network, to produce a hidden representation h i which constitutes the input of the classifier (denoted ψ): a fully-connected (fc) layer used for classification. formally, given w i , the predictions are obtained using the following equation: with υ ensuring the concatenation of υ char and υ word . foot_0 ## transfer learning method we use a simple sequential tl method to transfer knowledge from the news domain to the tweetsdomain. it consists in learning a source model on the source task with enough data from the news domain, then transferring a part of the learned parameters to initialise the target model, which is further fine-tuned on the target task with few training examples from the tweets domain. specifically, in this work we perform the tl following three simple yet effective steps: 1) the source model is learnt using a large annotated dataset from the source domain. 2) we transfer to the target model the first set of parameters (υ and φ) of the source model, while the second set of parameters (ψ) of the target model is randomly initialised. then, 3) the target model is further fine-tuned on the small target data-set. ## analysis first, in order to have an idea about the final impact of tl compared to randomly initialised models, we provide in table .2 the performance of random initialisation and transfer learning. clearly, tl enhances the performance across all data-sets and tasks. in the following sub-sections, we attempt to analyse thoroughly these results by showing that the impact of tl is two fold, positive transfer and negative transfer. ## conclusion our analysis on the hidden negative transfer from news-domain to tweets-domain reveals interesting insights: 1) even if using tl improves the performance on tweets sequence labelling, an inherent negative transfer may minimise the final gain; and 2) the negative transfer increases with the number of pretraining epochs. this study opens a set of promising directions. we plan to 1) extend our experiments by investigating the impact of the model's hyper-parameters (size, activation functions, learning rate, etc.). 2) investigate the impact of the similarity between source and target datasets and source and target training datasets size on the negative transfer. 3) tackle the negative transfer problem, by identifying automatically biased neurons in the pretrained model and proceed to a pruning of the most biased ones before fine-tuning. 4) explore negative transfer on transformers-based pretrained models, such as bert, xlnet, etc.