Learnable Conjunction Enhanced Model for C hinese Sentiment Analysis (2023)

Abstract: “Sentiment analysis is a crucial text classification task that aims to extract, process, and analyzeopinions, sentiments, and subjectivity within texts. In current research on Chinese text, sentenceand aspect-based sentiment analysis is mainly tackled through well-designed models. However,despite the importance of word order and function words as essential means of semantic ex-pression in Chinese, they are often underutilized. This paper presents a new Chinese sentimentanalysis method that utilizes a Learnable Conjunctions Enhanced Model (LCEM). The LCEMadjusts the general structure of the pre-trained language model and incorporates conjunctionslocation information into the model’s fine-tuning process. Additionally, we discuss a variantstructure of residual connections to construct a residual structure that can learn critical informa-tion in the text and optimize it during training. We perform experiments on the public datasetsand demonstrate that our approach enhances performance on both sentence and aspect-basedsentiment analysis datasets compared to the baseline pre-trained language models. These resultsconfirm the effectiveness of our proposed method. Introduction”

## introduction sentiment analysis is a crucial area of research within the field of natural language processing. before the advent of transformer @xcite , recurrent neural networks (rnns) were the primary method used to model sequences in language modeling tasks @xcite @xcite . rnn, along with its variants lstm (long-short term memory) and gru (gated recurrent unit), are powerful models for processing sequences of varying lengths and addressing long-term dependencies. however, the sequential nature of rnns makes parallelization difficult. transformer introduces the attention mechanism to encode the context information, which can well capture the internal correlation and ease the problem of long-term dependencies. this allows for greater parallelization and improved performance on certain tasks. nevertheless, since self-attention discards sequential operations when processing sequences, the position information in the sequence cannot be fully utilized. in languages such as chinese, word order plays a crucial role in conveying grammatical meaning 0 , making it important to consider the sequential nature of the language when developing natural language processing models. word order refers to the sequence of words in a phrase or sentence, while chinese word order is relatively fixed, and the change of word order can make the phrase or sentence express different meanings. "speak well/说好 话", "easy to speak with/好说话", and "easier said/话好说" are three chinese phrases that demonstrate the importance of word order in conveying meaning. although these phrases share similar characters, their meanings differ greatly depending on how those characters are arranged. "speak well/说好话" means to speak positively or say good things about someone or something, while "easy to speak with/好 说话" describes someone who is easy to communicate with. lastly, "easier said/话好说" implies that something may sound simple or easy to do but can be more difficult in practice. it's essential to consider both the context and word order when interpreting or translating chinese phrases. in addition, function words in chinese play an important role in constructing the grammatical structure of a sentence and reflecting specific grammatical relationships. they are a crucial grammatical tool necessary for expressing meaning 1 . among them, conjunctions connect grammatical units at different levels, and their positions in sentences are significantly different @xcite , which can be used as an essential aspect of studying syntactic distribution. therefore, in this paper, we propose lcem, a learnable conjunctions augmentation model for chinese sentiment analysis. by adjusting the structure of the pre-trained language model, lcem introduces the conjunction position information into the fine-tuning process. the paper also explores variants of residual structure and constructs an enhanced model capable of learning critical information during training and optimization of the residual structure. the main contributions of this paper can be summarized as follows: ## learnable parameters analysis figure 4 and figure 5 show the changes of relative position parameter ω and α to β ratio over time. the x-axis represents the range of parameter values, while the y-axis on the right represents the number of training steps. each slice in the figure is a single histogram, representing the distribution of parameters in a training step. the number of training steps is gradually increased from back to front. according to figure 4 , the learnable parameter ω of the relative position feature rp is more evenly distributed in [0.919, 0.999], indicating that the relative position feature occupies a vital proportion of attention. moreover, combined with the ablation experiment results in section 5.4, relative location feature enhanced attention can capture both content and distance information and learn a richer context representation under the effect of location information. figure 5 shows that the ratio of α to β is evenly distributed in [0.919, 1.01]. in most cases, the proportion of input branches is smaller than that of residual branches. in each transformer encoder, the proportion of representations from the previous layer is smaller than that of expressions enhanced by the relative position of the conjunctions. it demonstrates the significance of the semantic representation obtained through enhanced attention in the network. moreover, input branches also play an important role in network. through layer-by-layer propagation, the semantic representation acquired by each layer can be preserved in the lower layers and will participate in the attention mechanism to further extract abstract semantics. the learnable parameters greatly help the information transfer and optimization of network structure.