CLPLM : Character Level Pretrained Language Model for E xtracting S upport Phrases for Sentiment Labels (2020)

Abstract: In this paper, we have designed a character-level pre-trained language model for extracting support phrases from tweets based on the sentiment label. We also propose a character-level ensemble model designed by properly blending Pre-trained Contextual Embeddings (PCE) models- RoBERTa, BERT, and ALBERT along with Neural network models- RNN, CNN and WaveNet at different stages of the model. For a given tweet and associated sentiment label, our model predicts the span of phrases in a tweet that prompts the particular sentiment in the tweet. In our experiments, we have explored various model architectures and configuration for both single as well as ensemble models. We performed a systematic comparative analysis of all the model’s performance based on the Jaccard score obtained. The best performing ensemble model obtained the highest Jaccard scores of 73.5, giving it a relative improvement of 2.4% over the best performing single RoBERTa based character-level model, at 71.5(Jaccard score).

## introduction sentiment analysis has been a trendy topic for the last some decades. whether its a graphical image or textual data, all types of an entity consists of something that conveys the sentiment. with the recent development in machinelearning methods, new innovative and powerful models have developed in the field on natural language processing such heavy pretrained language models. we designed a novel character-level pretrained language model framework which utilizes the transformers and character-level language models to extract sentiment phrases. the proposed model works in four steps-(i) token-level span prediction using transformer model, (ii) converting tokenlevel representation to character-level representa-tion, (iii) precise character-level span prediction using character-level neural network model, and (iv) retrieving selected sentiment phrases. we also proposed an ensemble character-level model that surpassed the single transformer models. despite of being a simple idea, ensemble learning has always been successful in several tasks @xcite . we conducted extensive performance analysis of various models and systematically presented our results in the paper. the primary challenge was in the dataset. as shown in the ## proposed transformer language models in this section, we have sequentially discussed and elaborated on the design of the proposed character level pre-trained language models. we first talked about the architecture and working of single pretrained contextual embeddings (pce) based character model. then we talked about the construction and functioning of the character level ensemble model. ## experiment in this section, we have discussed the dataset, and it's preprocessing step along with the evaluation matrix used in our experiment. we have sequentially explained the experiment setup and model configuration in detail. ## result and discussion in this section, we have discussed the experiment results and presented a performance analysis of character-level transformer models and ensemble models. during our experiment, we combined the level 1 model, with every level 2 model pairwise. as a result, we found out that the rnn based model surpassed other models by achieving a higher jaccard score during testing. as a result, table1 contains the jaccard score mean of each transformer combined with the rnn model as level 2 character-nn model. according to the our proposed ensemble model outperformed all the single transformer by an average of 2.4%. with a perfect blending of transformer models, the ensemble model was able to achieve a jaccard score mean of 73.5%. ## conclusion in this paper, we proposed a character-level language model consisting of a pre-trained language model and character-level neural network for extracting sentiment support phrases from humangenerated tweets based on the associated sentiment labels. we also designed a stacking ensemble model by carefully blending multiple pce transformer models like bert, roberta, albert, and character-level neural network models like cnn, rnn, and wavenet. we conducted a comparative performance analysis of all character-level transformer models and ensemble models. we found that with the optimal combination of transformer models and character-level neural network models, the ensemble architecture generalizes better and outperforms the single transformer models at every level. the ensemble model showed promising results, having a jaccard score of 73.5%, which is better than any single transformer model. text(given) sentiment(given) selected text(target) i really really likeeee the song love story by taylor swift positive likee i need to get my computer fixed neutral i need to get my computer fixed sooo sad i will miss you here in san diego negative sooo sad