D ata A nnealing for I nformal L anguage U nderstanding T asks (2020)

Abstract: There is a huge performance gap between formal and informal language understanding tasks. The recent pre-trained models that improved formal language understanding tasks did not achieve a comparable result on informal language. We propose data annealing transfer learning procedure to bridge the performance gap on informal natural language understanding tasks. It successfully utilizes a pre-trained model such as BERT in informal language. In the data annealing procedure, the training set contains mainly formal text data at first; then, the proportion of the informal text data is gradually increased during the training process. Our data annealing procedure is model-independent and can be applied to various tasks. We validate its effectiveness in exhaustive experiments. When BERT is implemented with our learning procedure, it outperforms all the state-of-the-art models on the three common informal language tasks.

## introduction and related work because of the noisy nature of the informal language and the shortage of labeled data, the progress on informal language is not as promising as in formal language. many tasks on formal data obtain a high performance due to deep neural models @xcite . however, these state-of-the-art models' excellent performance usually fails to transfer to informal data directly. for example, when a bert model is fine-tuned on informal data, its performance is less encouraging than on formal data. it is because of the domain discrepancy between the pre-training corpus used by bert and the target data. to solve the issues mentioned above, we propose a model-agnostic data annealing procedure. we set informal data as target data and set formal data as source data. the training data first contains mainly source data, when data annealing procedure takes the advantages of a proper parameter initialization from the clean nature of formal data. the proportion of source data keeps decreasing exponentially while the proportion of target data keeps increasing, which empowers the model with more freedom to explore the direction of its next update. the philosophy behind data annealing is shared with other commonly used annealing techniques. one popular usage of annealing is learning rate annealing. a gradually decayed learning rate enhances the model with more freedom of exploration at the beginning and leads to better model performance @xcite @xcite . another widespread implementation of annealing is simulated annealing @xcite . it reduces the probability of a model converging to a bad local optimal by introducing random noise in the training process. data annealing has similar functionality with simulated annealing but replaces random noise with source data. by doing this, the model explores more space at the beginning of the training process and is guided by the knowledge learned from the source domain. current state-of-the-art models on informal language tasks are usually designed specifically for a particular task and cannot generalize to different tasks @xcite . data annealing is model-independent and could be employed in various informal language tasks. we validate our learning procedure with two popular neural network models in nlp, lstm, and bert, on three popular natural language understanding tasks, i.e., named entity recognition (ner), partof-speech (pos) tagging and chunking on twitter. when bert is fine-tuned with data annealing procedure, it outperforms all three state-of-the-art models with the same structure. by doing this, we also set the new state-of-the-art result for the three informal language understanding tasks. experiments also validate our data annealing procedure's effectiveness when there are limited training resources in target data. ## data annealing a pre-trained model like bert is suggested to avoid over-training when implemented on downstream task @xcite . in transfer learning, it is not ideal to feed too much source data, as it not only prolongs the training time but also confuses the model. therefore, we propose data annealing, a transfer learning procedure that adjusts the ratio of the formal source data and the informal target data from large to small in the training process to solve the overfitting and the noisy initialization problems. at the first stage of data annealing, most of the training samples are source data. therefore the model obtains a proper initialization from the abundant clean source data. in the second stage, as we gradually increase the proportion of the target data and reduce the proportion of the source data, the model explores a larger parameter space. besides, the labeled source dataset works as an auxiliary task. at the third stage of the training process, most of the training data is target data so that the model focus on the target information more. we reduce the source data proportion exponentially. α represents the initial proportion of the source data. t represents the current training step, and m represents the number of batches in total. λ represents the exponential decay rate of α. r t s and r t t represent the proportion of the source data and proportion of target data at time step t. let d s represents the accumulated source data used to train the model, and let b represents the batch size. we have d s could be empirically decided based on the relation between source dataset and target dataset. for example, the higher the similarity between the source and the target data, the more knowledge the target task could borrow from the source task, and larger d s is. if researchers want to simplify the hyper-parameters tuning process or constrain the influence of source data, α can be set by d s : ## experimental design we validate it by two popular model lstm and bert on three tasks: named entity recognition (ner), part-of-speech tagging (pos), and chunking. these tasks have much better performance on formal text (such as news) than informal text (such as tweets). ## experiment results the result of the three tasks is shown in part-of-speech tagging (pos tagging). all the bert models and lstm models under our data annealing procedure outperform other transfer learning procedures. the improvement over the state-ofthe-art model dcnn @xcite ## error analysis we did an error analysis in ritter11-ner dataset. we randomly sampled 30 sentences that contain entities that are incorrectly predicted by da bert large and attached them in appendix a. we found that a relatively large proportion of sentences has a too strong noisy feature to be predicted correctly. this feature is embedded in the informal text, and we might need to explore more on the nature of informal language to solve it perfectly. we also calculated the f 1 score of the ten predefined entity types. we find that compared with vanilla bert large and init bert large , da bert large achieves higher f 1 score on two frequent entities, "person" and "other". "person" is a frequent concept in formal data. it shows our method learns to utilize formal data knowledge to improve "person" detection. besides, "other" means entities that are not in the ten pre-defined entity types. higher performance on "other" suggests da bert large has a better understanding of the general concept of an entity. init bert large achieves a higher f 1 score on "geo-loc". we did not find a clear difference in other entity types. besides, we found that if a word is of a rarely appeared entity type, all the three models are less likely to predict its entity type correctly. we suspect that a neural model implicitly learns to predict a word when it is trained to predict other words in the same entity type since these words could share a similar representation in the ner task. we plan to assign more penalty to infrequent entity types to tackle this issue in the future. ## conclusion in this paper, we propose data annealing, a modelindependent transfer learning procedure for informal language understanding tasks. it applies to various models such as lstm and bert. it has been proven as a good approach to utilizing knowledge from formal data to informal data by exhaustive experiments. when data annealing is applied with bert, it outperforms different state-of-the-art models on different informal language understanding tasks. since large pre-trained models have been widely used, it could also serve as an excellent fine-tuning method. data annealing is also useful when there are limited labeled resources. a mispredicted sentences examples on named entity recognition task