U o B at P rof NER 2021: Data Augmentation for Classification Using Machine Translation (2021)

Abstract: This paper describes the participation of the UoB-NLP team in the ProfNER-ST shared subtask 7a. The task was aimed at detecting the mention of professions in social media text. Our team experimented with two methods of improving the performance of pre-trained models: Specifically, we experimented with data augmentation through translation and the merging of multiple language inputs to meet the objective of the task. While the best performing model on the test data consisted of mBERT fine-tuned on augmented data using back-translation, the improvement is minor possibly because multi-lingual pre-trained models such as mBERT already have access to the kind of information provided through back-translation and bilingual data.

## introduction and motivation the increase of user-generated content online has allowed researchers to extract information for studies on a variety of subjects, namely tracking infectious diseases and promoting public health @xcite . consequently the emergence of covid-19 has resulted in a rapid increase of information related to the virus on social media platforms @xcite . profner, a task under social media mining for health applications (smm4h) workshop @xcite , requires the identification of occupations that might be particularly affected, either mentally or physically, by the exposure to covid-19. the task organisers give participants tweets in spanish and english. the english tweets were translated by means of a machine translation system. of the tweets provided, 24% contain a mention of an occupation @xcite . classifiers are dependent on the size and quality of the training data @xcite , and are sensitive to class imbalance. we hypothesise that increasing the number of examples in the positive class using data augmentation techniques will successfully increase the performance of trained models. this work describes the training of four classifiers using pre-trained bert models to detect the mention of occupations in tweets. in addition to training two baseline models, bert-base and mbert, we train one model on augmented textual data, mbert-aug, and another model on bilingual data. we compare these models to each other and to fine-tuned pre-trained bert models, described in section 3.2. the small increase in f1 scores over the baselines, which is inconsistent across our validation and test experiments leads us to conclude that back-translation and bilingual data input are ineffective as methods of addressing class imbalance in pre-trained models, especially multi-lingual models (see section 4). our models were trained using the data provided by the task organisers for subtask 7a. results are discussed in section 4. ## related work augmenting textual data is challenging because it can introduce label noise and must be done before training a model @xcite . among techniques developed for text augmentation is synonym replacement, random insertion, swap and deletion, as presented by @xcite . @xcite uses back-translation, to translate the data in a second language and then back to the source language. they train their model on a binary classification task in a setting where low amounts of labelled data are available. work continues to be done in back-translation for classification, as there is little research otherwise @xcite . in this work, we use back-translation as a tool for augmenting the text data for the positive class. this work contributes to the field of generating synthetic data for text classification. others have tried to add features to models to increase performance @xcite , we attempt to bring together representations in different languages so as to maximise the information available to the models. ## system overview and experimental set-up this section describes our experimental design. the code, models, and hyper-parameters are available on our team's github repository for the task 1 . ## results and discussion the bilingual model obtains the best results on the validation data, while mbert-aug is the best scoring model on the test data, with a f-1 score of 0.83. we perform experiments after the evaluation period to obtain results on the test data for bert-base and the bilingual model. we do this to compare the results of all models on the test data. we find that neither the addition of augmented data, nor combining representations in different languages significantly improves the results, with the bilingual model performing better on the validation data, and the mbert-aug performing better on the test data. we believe that a reason the bert-base and bilingual models have lower scores on the test data is due to the quality of the machine translation system that we used whereas the validation data was provided by the task organisers. for example, ultima hora in spanish was translated as last minute, when it should have been translated as breaking news in the context it was used in. another example is consellera which translates to advisor was not translated at all in some tweets. while these methods of data augmentation provide a small improvement, fine-tuned pre-trained bert models are quite robust. training on parallel corpora gave these models everything that could be extracted through back-translation and bilingual data. ## conclusion our work presents experiments with pre-trained transformer based models to perform binary classification on an imbalanced dataset. we hypothesised that the use of data augmentation and parallel inputs in multiple languages will provide a method of addressing class imbalance (section 1). however, our experiments showed that neither of these methods are particularly powerful in this regard (section 4). in the future, we will continue to experiment with other techniques to handle imbalanced classes, such as one-class classification and reinforcement learning-based networks to generate text.