Aambela at BLP -2023 Task 1: Focus on UNK tokens: Analyzing Violence Inciting B angla Text with Adding Dataset Specific New Word Tokens (2023)

Abstract: The BLP-2023 Task 1 aims to develop a Natural Language Inference system tailored for detecting and analyzing threats from Bangla YouTube comments. Bangla language models like BanglaBERT have demonstrated remarkable performance in various Bangla natural language processing tasks across different domains. We utilized BanglaBERT for the violence detection task, employing three different classification heads. As BanglaBERT’s vocabulary lacks certain crucial words, our model incorporates some of them as new special tokens, based on their frequency in the dataset, and their embeddings are learned during training. The model achieved the 2nd position on the leaderboard, boasting an impressive macro-F1 Score of 76.04% on the official test set. With the addition of new tokens, we achieved a 76.90% macro-F1 score, surpassing the top score (76.044%) on the test set.

## introduction in recent times, transformer models have gained popularity through pretraining on diverse text data @xcite . pretraining imparts context awareness, linguistic patterns, and word knowledge @xcite . combining it with fine-tuning vastly outperforms traditional models. these models have a vocabulary from pretraining, representing known words. during fine-tuning the pretrained in a task, there might be some words not in the vocabulary. tokenizer furhtermore tries to split the word into multiple subwords. still, if a word is missing or unsegmentable, they use a [unk] token @xcite . banglabert @xcite model is one of the most popular pretrained language models in bangla text classification which is trained on a large corpus of bangla text. it faces trouble because of [unk] tokens. this problem arises because the pretrianed bert model sometimes can't give representation to the words that are very important to understand the context. in this study, the main focus was to identify the most frequent words for which [unk] tokens are assigned and add these words to the pretrained vocabulary as a special token. it is shown in the study that this approach improved performance for text classification. to further improve the model's performance, three different classification heads were used. these heads improved the model's prediction by focusing on different words. three classification heads, along with the proposed approach, achieved better performance than the previous approach. ## result and analysis during the development phase, we conducted various experiments, all of which are detailed in appendix b. the experiment setup and hyperparameter specifics can be found in appendix a. our experiments for model selection encompassed a wide range, including machine learning models (svm, randomforest, xgboost) with tf-idf feature extraction, deep learning models (lstm, lstm+attention), and multilingual transformer models (mbert, mdeberta, xlm-roberta base). notably, mdeberta exhibited superior performance. additionally, we evaluated two bangla language models, with the csebuetnlp-banglabert model emerging as the top performer. for a concise summary of the experimental outcomes related to model selection, please refer to it's important to note that the lstm and lstm+attention models were trained for 5 epochs, while all transformer-based models underwent finetuning for 3 epochs with the utilization of the [cls]-based classification head, as detailed in section 3.2.1. the evaluation metric. three distinct scenarios were examined with different classification heads, as outlined in section 3.2. in the first scenario, the use of new token additions (described in section 3.1) was omitted. in this context, we observed that the cls+mlp configuration outperformed others in the development set. however, the dropouts enhanced mlp head demonstrated notable improvements, not only in cross-validation scores but also in the test set performance. the attention based head had showed significant enhancements in both cross-validation scores and test set results, despite a slightly lower performance in the development set. interestingly, incorporating the development dataset with the training dataset did not yield supe-rior results in the test dataset, despite achieving a better cross-validation score. fascinating findings emerged when we incorporated new token additions as new special tokens (described in section 3.1) into the pretrained banglabert vocabulary. the words that are considered as new tokens are mentioned in appendix c. in this experiment, we observed approximately a 1% improvement in both dev set and test set performance, measured by accuracy and macro-f1 metrics across all heads. remarkably, the attention pool combined with the addition of new tokens yielded the highest macro-f1 score. notably, the dropouts enhanced mlp model without the new tokens addition , which secured the 2nd position on the leaderboard, emerged as the topperforming model among the submissions. all heads with new token addition and attention pool head without new tokens addition beat the top leaderboard score, which was 76.044% macro-f1 score. unfortunately, those models weren't submitted during the competition. the models that beat the top leaderboard score are marked as underlined in ## conclusion in this study, an analysis is done when we add dataset-specific tokens (most frequent) to the pretrained vocabulary of banglabert for which the banglabert tokenizer gives the [unk] token. the addition tokens learn their embeddings during the finetuning. from the experiment, it has been seen that the addition of those type of tokens boosts the model's performance. to enhance the model prediction's further, different classification heads are applied.