Sentiment Analysis for Emotional Speech Synthesis in a News Dialogue System (2020)

Abstract: As smart speakers and conversational robots become ubiquitous, the demand for expressive speech synthesis has increased. In this paper, to control the emotional parameters of the speech synthesis according to certain dialogue contents, we construct a news dataset with emotion labels (“positive,” “negative,” or “neutral”) annotated for each sentence. We then propose a method to identify emotion labels using a model combining BERT and BiLSTM-CRF, and evaluate its effectiveness using the constructed dataset. The results showed that the classification model performance can be efficiently improved by preferentially annotating news articles with low confidence in the human-in-the-loop machine learning framework.

## introduction as smart speakers and conversational robots become ubiquitous, the demand for expressive speech synthesis has increased. speech synthesis is a technology that converts input text into a speech signal that represents its contents. in recent years, research has not only improved the quality so that a synthesized voice is similar to a real one, but also has diversified expressions @xcite @xcite . emotional speech synthesis is a technique for diversifying the expression of speech synthesis @xcite @xcite . it specifies both emotional parameters and the text input so that the speech reflects the designated emotion @xcite @xcite . in conversational robots and non-task-oriented dialogue systems, the emotions of speech synthesis are controlled not only by the contents of the text spoken by the system but also by the personality of the robot or environmental information @xcite @xcite . on the other hand, in audiobooks and news reading systems, the emotions when the system speaks are often determined by the content of the text. in such applications, it is desirable that the emotional parameters used in the speech synthesis system be automatically estimated from the input text @xcite @xcite @xcite . when news is transmitted by a synthesized voice, it is beneficial for listeners that news with positive content is transmitted with voices that are synthesized with positive emotion, whereas news with negative content is transmitted with voices that are synthesized with negative emotion @xcite . in our spoken dialogue system that delivers news @xcite , it is important to speak clearly with emotion according to the content of the news to improve the users' understanding. the structure of this paper is as follows. section 2 discusses related work. section 3 overviews the annotation method of emotion labels and the statistics of the constructed dataset. section 4 describes the proposed model. section 5 shows the performance results of the proposed model. section 6 reports the effects of active learning. section 7 provides conclusions and future work. ## related work sentiment analysis is a technique to analyze emotions contained in texts @xcite . most sentiment analysis studies target texts containing subjective opinions such as twitter tweets and review documents @xcite @xcite . many of these studies classify emotions from the perspective of writers. on the other hand, studies on news articles that mainly refer to objective events classify emotions from the readers' perspective. for example, lin et al. classified readers' emotions about news into happy, angry, sad, surprised, heartwarming, awesome, bored, and useful @xcite . they assumed that the most common emotion chosen by users who read the news article to be the correct answer and identified the emotions of news articles using svm @xcite . li et al. classified readers' emotions into touching, empathy, boredom, anger, amusement, sadness, surprise, and warmness @xcite . they formulated the sentiment analysis problem as a multi-label classification problem and proposed a topic model to estimate the weight of different documents for each emotion @xcite . ciptadi et al. classified readers' emotions into proud，angry，sad，happy，afraid，amused，inspired，and surprised @xcite . they showed that the classification performance of a naive bayes classifier and logistic regression was improved by applying the smote @xcite oversampling method to alleviate the problem of imbalanced data. in addition to studies classifying entire news articles @xcite @xcite @xcite @xcite , some research classified headlines of news articles @xcite , while others classified sentences of news articles @xcite @xcite @xcite . in studies on sentence classification, for example, bhowmick et al. classified readers' emotions into anger, disgust, fear, happiness, sadness, and surprise @xcite . they asked multiple people to annotate the sentences of news articles. they confirmed that the agreement rate could be improved by eliminating surprise as well as integrating anger and disgust. in addition, they evaluated the multi-label classification performance by adtboost.mh @xcite . similarly, eliminating surprise and integrating anger and disgust improved the model performance. li et al. modeled the label dependency by assuming a single sentence was likely to have similar labels such as hate and anger, and the context dependency of sentences by assuming that sentences in the same context were likely to have the same label using a factor graph @xcite . the model using two neighboring sentences rather than the document or paragraph as the context showed the best performance. although it was a study on review documents, zhang et al. formulated the sentiment analysis problem as a sentence sequence labeling problem @xcite . they proposed a method to identify emotion labels (positive, negative, and neutral) by crf @xcite . from the experimental results of active learning, the method labeling a document with the smallest average probability of the first half of the sentences in the document most effectively improved the model performance. in recent years, in the field of natural language processing, approaches for fine-tuning a language model pre-learned with huge unlabeled text data in downstream tasks have been attracting attention @xcite . bert @xcite is a typical method of pre-training language representations. the effectiveness of the method using bert has also been confirmed in the sentiment analysis task @xcite @xcite . however, these models identify the emotion of an entire review or a single sentence. when considering the classification of emotions in each sentence of a news article, it is necessary to consider not only each sentence but also the contextual information around them. most conventional studies on sentiment analysis for news articles classify emotions at a finer granularity from the readers' perspective. in this study, we adopt three classes of "positive," "negative," and "neutral" for the granularity of the classification that can be agreed upon by many listeners. similar to zhang et al., we formulate the problem of identifying emotion labels for each sentence of news articles as a sentence sequence labeling problem. we classified emotions in news content as "positive," "negative," or "neutral," and constructed a dataset for machine learning by annotating these emotion labels for each sentence in a news article. a positive label is annotated when the subject of the sentence considering the context is good or indicates that the subject is heading in a good direction. examples include social contribution, market expansion, and acquisition of interests. a negative label is annotated when the subject of the sentence considering the context is bad or indicates that the subject is heading in a bad direction. examples include a decline in business, acts of dishonesty, incidents, and accidents. a neutral label is annotated when neither positive nor negative content is included. articles containing sentences with both positive and negative content were excluded in this study. annotation was performed by a web news clipping expert for news articles with 5 to 20 sentences in the nihon keizai shimbun. the annotator was presented with lists of news articles ranked by category using a rule-based approach (see section 5.1). the annotator annotated high ranked articles (news expected to be positive), middle ranked articles (news expected to be neutral), and low ranked articles (news expected to be negative) so that they were evenly distributed into the list. to cover various topics, we instructed the annotator to avoid annotating similar topics as much as possible. in the annotation work, first, the annotator read the title of the news article and checked the summary of the article. next, the annotator assigned an emotion label to the title to understand the emotional tendency of the whole article. after reading all sentences of the article, the annotator assigned emotion labels to each sentence beginning from the first sentence. ## proposed model we formulated the emotion label identification problem as a sequence labeling problem. figure 1 shows a schematic diagram of the proposed model. bert @xcite , which is a model based on the transformer @xcite encoder, is used as the word encoder. self-attention @xcite is calculated for the representation that combines the embedding of the top layer of bert corresponding to each word and the embedding of the auxiliary features of each word. the obtained sentence vectors are given to bilstm-crf @xcite , and the emotion labels of each sentence are estimated. at the time of decoding, the labels are estimated by the viterbi algorithm. the following information is used as auxiliary features: morphological information (part of speech, inflectional form, inflected type, category, domain) of juman++ (ver.1.02) 1 @xcite , named entity classes and types of dependencies obtained by applying knp (4.19) 2 @xcite , distance from the top node of the dependency tree to the clause that contains the target word, the number of clauses whose destination is the clause that contains the target word, tf, idf, tf-idf, whether the word is included in the range of the corner bracket, clause position from the beginning of the sentence, position of the sentence in the article, position of the paragraph in the article, news category of the article, emotion polarity value of the word (using the "japanese sentiment polarity dictionary" 3 @xcite , the "semantic orientations of words" 4 @xcite ，the polarity dictionary included in "models for opinion extraction tool" 5 ), and whether the word is a polarity inversion expression (using the reverse expression dictionary included in "models for opinion extraction tool"). ## active learning in the human-in-the-loop machine learning framework @xcite , the model performance can be efficiently improved by preferentially annotating articles with low confidence. ## conclusion to enable emotional speech synthesis based on news content in a spoken dialogue system, we constructed a dataset for machine learning by annotating the emotion labels for each sentence in a news article. in addition, we proposed a model that identifies the emotion label of each sentence, and evaluated its effectiveness using the constructed dataset. the model performance can be efficiently improved by preferentially annotating articles with low confidence in the human-in-the-loop machine learning framework. in the future, we will develop a speech synthesis system that can control the emotional parameters using the emotion label estimated by the proposed model. we will also confirm whether speaking with emotion promotes users' understanding in news delivery tasks. it was revealed on the 2nd that the chinese trademark office revoked the registration of two brands, "mori izo" and "isami" due to the problem that potato shochus in kagoshima prefecture, which are very popular, were registered without a trademark in china.