Recognition Method of Important Words in K orean Text based on Reinforcement Learning (2020)

Abstract: The manual labeling work for constructing the Korean corpus is too time-consuming and laborious. It is difficult for low-minority languages to integrate resources. As a result, the research progress of Korean language information processing is slow. From the perspective of representation learning, reinforcement learning was combined with traditional deep learning methods. Based on the Korean text classification effect as a benchmark, and studied how to extract important Korean words in sentences. A structured model Information Distilled of Korean (IDK) was proposed. The model recognizes the words in Korean sentences and retains important words and deletes non-important words. Thereby transforming the reconstruction of the sentence into a sequential decision problem. So you can introduce the Policy Gradient method in reinforcement learning to solve the conversion problem. The results show that the model can identify the important words in Korean instead of manual annotation for representation learning. Furthermore, compared with traditional text classification methods, the model also improves the effect of Korean text classification.

## introduction the languages of ethnic minorities have created the diversity of chinese characters and are an important part of chinese characters, providing important support for the development of national culture. however, the research on korean natural language processing in my country is still in the development stage, and the related research is still relatively lagging behind south korea and north korea @xcite . for manual annotation of korean sentences, the structure division requires a lot of energy and time. so for this problem, we associate the method of representation learning. representation learning has been widely used in text classification, sentiment analysis, language reasoning and other fields in recent years. it is a basic problem in the field of artificial intelligence, and it is particularly important in the field of natural language processing. therefore, we use this method as the core logic of the model, aiming at korean text, identifying important words and performing sentence classification tasks on newly constructed sentences. the resulting structural representation does not require manual annotation, greatly reducing manpower and scientific research resources. in order to find important korean words in sentences, we use the effect of text classification as feedback in reinforcement learning. the current mainstream text classification models are roughly divided into four types: bag-of-words model, sequence model, structure representation model, and attention model. the bag-of-words representation model often ignores the order of words, such as deep average networks, self-encoders @xcite ; the sequence representation model often only considers the words themselves, but ignores the phrase structure, such as cnn, rnn and other neural network models @xcite ; structural representation models often rely on pre-specified parse trees to construct structured representations, such as tree-lstm, recursive autoencoders (zhu x, 2015); representation models based on attention mechanisms need to use input words or sentences the attention scoring function is used to construct a representation, such as self-attention (yang z, 2016), and the effect is very dependent on the reliability of scoring. in the existing structured representation model, the structure can be provided as input, or it can be predicted using the supervised method of explicit tree annotations, but few studies have studied the representation with automatically optimized structure. yogatama et al. proposed to construct a binary tree structure for sentence representation only under the supervision of downstream tasks, but this structure is very complicated and the depth is too large, resulting in unsatisfactory classification performance @xcite . chung et al. proposed a hierarchical representation model to capture the latent structure of sequences with latent semantics, but the structure can only be found in the hidden space @xcite . tianyang et al. proposed a method that combines the strategy gradient method in reinforcement learning with the lstm model in deep learning. the effect of text classification is used as the baseline for reinforcement learning to carry out unsupervised structuring, and its structuring effect is closer to human, and the classification effect is significantly better than other mainstream models @xcite . ## strategy network based on reinforcement learning combined with attention mechanism the core idea of the strategy network is the policy gradient method , which is different from the traditional method. instead of backpropagating through errors, the observation reward reward value is used to enhance or weaken the possibility of selecting actions. that is, the probability that a good action will be selected next time will increase; the probability that a bad action will be selected next time will decrease. a complete strategy represents a sequence of actions taken in each state in a round, the cumulative sum of the revenue generated by each action represents the round reward value. we use a random strategy π (a t |s t ; θ). and use the reward generated by each round delay to guide strategy learning, for each state of the structured representation model generated each time, different actions are sampled. first, all the words of the entire sentence must be sampled for action, so as to determine the actions of all states corresponding to a sentence ,secondly, the determined action sequence is passed into the representation model to generate a new structured representation; then the generated structured representation is passed into the classification network, and this representation model is used to calculate the classification accuracy p (y|x) , finally, the calculated reward is used for strategy learning. in the loop iteration, a better strategy is found, so that a better structured representation is obtained, and then a better classification effect is obtained. the strategy is defined as follows: which a t represents the probability of choosing at; σ represents the sigmoid function; θ represents the parameters of the strategy network. during the training, actions are sampled according to the probability c c l 2 0 2 0 in equation 1. during the test, the action with the highest probability will be selected to achieve a better classification effect. when all actions are sampled by the strategy network, the structured representation of the sentence is determined by the representation model, and the determined representation model is passed to the classification network to obtain, where y is the classification label, reward will be calculated from the predicted distribution, and there are also factors for thinking about the trend of structural choices. therefore, the policy gradient method in the reinforcement learning algorithm is used to optimize the parameters of the strategy network (keneshloo y, 2019), so as to maximize the expected return, as shown in equation 3. the reward calculation is only for one round, because the state at step t + 1 is completely determined by the state at step t, so the probability sum is 1. through the likelihood ratio technique, the following gradient update strategy network (sutton r s, 2000) is finally used, where n represents the round number. in the iterative process of reinforcement learning, the variance is generally large. if the loss value is always positive, the direction of the iteration is easy to move toward. it has been proceeding in the wrong direction, so the introduction of b as a baseline can accelerate convergence, as shown in equation 4. on this basis, the attention mechanism is introduced into the strategy network, and the encoder-decoder framework is adopted. use bi-lstm (graves a, 2005) as the encoder model, lstm as the decoder model, and the output of the structured representation model as the input of the strategy network, because the core logic of the attention model is from focusing on the whole to focusing on the core. the purpose of this article is the same, so the combination of reinforcement learning and soft attention mechanism to make up for the shortcomings of the predecessors in the traditional attention model in the text classification process, relying heavily on the scoring function @xcite . after introducing the attention mechanism, the corresponding actions in each state are output as shown in figure 1 . where h j is the hidden vector of the input, f is the tanh activation function; c is the attention distribution; and α tj is the attention obtained by each input. after introducing the attention mechanism, the global observation can be better, so that the generated action sequence is optimized in two aspects of the strategy gradient and the attention mechanism, thereby improving the model effect. ## conclusion we combined reinforcement learning methods to learn korean sentence representations by finding important words related to the task. in the framework of reinforcement learning and attention mechanism, this paper uses the idk model, which is used to extract task-related words and express them in purified sentences. among them, reinforcement learning uses the accuracy rate of text classification as a baseline to optimize the action sequence, and the action sequence can generate a text structure representation that is more suitable for classification. an attention mechanism is introduced in the process of action sequence generation to compensate for the variance of the reinforcement learning method. the disadvantage of being too large and difficult to fit, compared with the traditional attention model, not only has its advantages of taking into account the overall situation, but also adds a more ingenious way to improve the accuracy of downstream tasks, and the experimental results have performed well. experiments show that our method can find important words related to tasks without explicit structure annotation. the model not only improves the effect of korean text classification, but also works well in the task of processing korean text important word recognition.