CXP 949 at WNUT -2020 Task 2: Extracting Informative COVID -19 Tweets - R o BERT a Ensembles and The Continued Relevance of Handcrafted Features (2020)

Abstract: This paper presents our submission to Task 2 of the Workshop on Noisy User-generated Text. We explore improving the performance of a pre-trained transformer-based language model fine-tuned for text classification through an ensemble implementation that makes use of corpus level information and a handcrafted feature. We test the effectiveness of including the aforementioned features in accommodating the challenges of a noisy data set centred on a specific subject outside the remit of the pre-training data. We show that inclusion of additional features can improve classification results and achieve a score within 2 points of the top performing team.

## introduction identification of informative tweets in relation of coronavirus presents a text classification problem. pre-trained bidirectional transformer-based models such as such as bert @xcite and roberta @xcite have proven to be extremely successful on text classification tasks; the process of pre-training on a large corpora enables the generation of effective contextual embeddings during fine-tuning, which can be leveraged on the classification task through the addition of an output layer @xcite . the progression of the state-of-the-art that these models have facilitated is clearly demonstrable on performance against the general language understanding evaluation (glue) benchmark @xcite ; a collection of natural language understanding tasks with an associated online platform for evaluation and analysis. the corpora utilised during pre-training of transformer-based models typically consists of documents written in formal english. moreover, this data is highly unlikely to contain references to the coronavirus pandemic which has only just occurred. therefore, the noise inherent in social media data and subject specificity to coronavirus in the current data set @xcite , present challenges to conducting text classification with a pretrained transformer-based model alone. to this regard, supplementary information may be useful to improving performance; corpus level information may potentially capture notions of relevance to words which fall outside the pre-trained vocabulary, and handcrafted features may show additional distinctions between classes. in this paper, we describe the development of a system with includes such features through the use of an ensemble, and which in the final submission to the evaluation stage achieved an f 1 score of 0.8910. the rest of this paper is organised as follows, firstly a discussion of related work is presented at section 2. this is followed by a description of the methodological approach at section 3. we present our results and analysis at section 4 and present our conclusions in section 5. ## related work as the 6th workshop of noisy user-generated text presents the first time task 2 has been made available, we consider research utilising pre-trained bidirectional transformer-based models from similar tasks to better enable informed decision making. ## methodology the data for this work was provided by the workshop on noisy user-generated text @xcite and consisted of 10,000 english tweets in relation to covid-19. a 70/10/20 rate had been used to split the 10,000 tweets into training, validation and test sets. each tweet had been labelled either as 'uninformative' or 'informative' by three independent annotators with an inter-annotator agreement score of fleiss' kappa at 0.818 @xcite . at the onset of this work, only the training and validation data had been provided. the test set was retained as a holdout set for the workshop to evaluate the performance of models, discussed at section 5. in an attempt to build upon the success of leveraging tweet metadata in a classification model (see section 2.3) an initial exploration of the distribution of metadata and text features between the "uninformative" and "informative" classes in the training dataset was conducted. this processes highlighted little differentiation between the classes, suggesting the distinction was more nuanced and conveyed within the circumstantial and contextual meanings. however, it was found that the higher the probability any given character in tweet was a numeric character increased the probability of it being "informative". we present a visual representation of this at figure 1 . by virtue of the competitive element, this task presented an incentive to find the best performing model. in order to have a basis for comparison two initial models were trained on the raw training data and evaluated against the validation data: (1) a linear support vector machine (svm) using tf-idf features, and (2) fine-tuned bert base . as is the case with all the deep networks tested in this study, we test a range of hyperparameter values (details of which are available in the program code and model details released as part of this publication 1 ), and test each combination against five different random seeds. the effect of random seed on bert is emphasised in @xcite where varying only the random seed was shown to produce substantial improvements over previously published results. we present the highest performing results of these models at the svm model was chosen due to the established high accuracy this model can achieve on the task of text classification @xcite ; @xcite ). we observe that a fine-tuned bert model achieved a good performance against the validation data. the next logical progression was to experiment with a fine-tuned roberta implementation, given that the robustly optimised pre-training this 1 https://github.com/calumperrio/ wnut-2020 model employs improved performance over bert @xcite . we fine-tuned a roberta base model, and the results of this are presented against the bert and svm baselines in ## results and analysis we present the results of the three ensemble models in comparison to the processed roberta model from we observe that against the baseline roberta model, inclusion of the hand-crated feature improved performance. however, the increase is small, potentially reflecting that the number of tweets in the data set for which this features is a useful indicator is equally small. we also observe that the inclusion of tf-idf features on the validation data alone did not improve performance, however the inclusion of both the average metric and tf-idf produced an improved result over the baseline and roberta+prob. the former result does not support our hypothesis of improving performance with the inclusion of tf-idf features and is in contradiction to the latter result. between the roberta+tfidf model and roberta+tfidf+prob models, the highest performances were found on models featuring 9000 and 6000 tf-idf features respectively. it is notable that the total dimension of the stemmed documentterm matrix was over 15,000 features. as such, due to memory constraints it was not possible to use a vector of this entire dimension. we submit an interesting experiment would be to parse for domainspecific expressions as in @xcite and standardise these to dictionary representation to reduce the number of tf-idf features. additionally, standardisation of these expressions across the corpus could potentially enable greater accuracy in calculating the inverse document frequency, and by virtue of this better capture the amount of information a term provides. in our submission to the evaluation stage we submit the roberta+prob+tfidf ensemble model which achieved the highest performance of all models tested against the validation data (see ## conclusions and further work we have explored with providing supplementary information to improve the performance of roberta. we have observed that the addition of a handcrafted feature improved performance of a pre-trained bidirectional transformer-based language model, suggesting that for text classification tasks and noisy data sets the inclusion of additional features that distinguish the classes can be beneficial. we intend to explore this concept further, beginning with conducting a statistical test to observe whether the difference between the presence of numeric data in informative tweets is significant. additionally, we have experienced success with incorporating tf-idf features with bert to a degree, however we present that the size of the document-term matrix, the short length of tweets and domain-specific features of twitter present potentially a challenge in utilising this optimally. as the global coronavirus pandemic continues to develop, the nature of what becomes "informative" information will likely develop also. therefore, in the present task, an approach with a greater level of generalisation is arguably preferable. to this regard, initial testing of parsing the tweets and replacing the terms "coronavirus" and "covid-19" to "coronavirus disease", with a view to leveraging the existing embedding for "disease" to provide contextual information has shown promising results. the application of this new parsing objective presents an opportunity for future work. furthermore, we observed in section 2 that conducting additional in-domain pre-training was successfully utilised in relation to pre-training transformer-based models in a similar task. additionally, in @xcite the authors release a bert model pre-trained on a data set of tweets in relation to the coronavirus that shows a 10-30% marginal improvement on numerous classification data sets compared to bert large . exploration of these two concepts may provide insight into improving the base roberta model's performance (as discussed in section 4.1), and present further potential for future work and overcoming the challenges of a noisy data set centred around the topic of coronavirus.