Neural text normalization leveraging similarities of strings and sounds (2020)

Abstract: We propose neural models that can normalize text by considering the similarities of word strings and sounds. We experimentally compared a model that considers the similarities of both word strings and sounds, a model that considers only the similarity of word strings or of sounds, and a model without the similarities as a baseline. Results showed that leveraging the word string similarity succeeded in dealing with misspellings and abbreviations, and taking into account the sound similarity succeeded in dealing with phonetic substitutions and emphasized characters. So that the proposed models achieved higher F1 scores than the baseline.

## introduction non-standard words such as misspellings or abbreviations are often used in social media such as twitter. it is difficult to understand the meaning of such words without any extra knowledge, and it is challenging to perform natural language processing over sentences including them. text normalization plays an important role in dealing with these broken texts by correcting such a sentence into a standard one. the following is an example of text normalization. in text normalization, as shown in the example, we need to correct several types, including misspellings @xcite , abbreviations ('convo' to 'conversation'), phonetic substitutions @xcite , and emphasized characters @xcite . even though the similarities of character surfaces and phonemes is important for a human to understand non-standard words, current neural network-based text normalization methods do not consider this information. we assume that text normalization more intuitive to humans is possible by explicitly considering such features in a neural network-based method. based on this assumption, in this work, we propose neural text normalization models that leverage both string and sound similarities. experimental results show that our proposed models outperformed a baseline and achieved state-of-the-art results in the text normalization track on wnut-2015. to solve this problem, sequence-to-sequence (seq2seq) @xcite has been used for text normalization. @xcite performed highly accurate and fluent text normalization by using the attention-based seq2seq @xcite . @xcite used seq2seq in automatic speech recognition error correction, a task similar to text normalization. taking these trends into account, in this work, we propose a neural text normalization model that leverages both string and sound similarity. ## methodology figure 1 shows an overview of our proposed method. in this study, we perform text normalization based on the method of @xcite , which incorporates token embeddings as an input to the encoder. our method expands their work by utilizing features related to character strings and sounds. @xcite proposed deep levenshtein, a network that captures the feature of character strings based on the levenshtein edit distance @xcite in order to correct any character fluctuations of named entities in a text. in this study, we incorporate this mechanism into the seq2seq model to make it more robust to a broken text. deep levenshtein is a neural network that takes two words x and y as an input and then outputs hidden representations for the character strings of the words. two words x and y are fed into the word embedding layer, and then we obtain word embeddings e x and e y , respectively. each word embedding is an input to the bidirectional lstm @xcite to obtain hidden representations c x and c y , that capture the feature of the character string. here, -→ h x and ←h x represent the forward and backward paths of the bidirectional lstm, respectively. c x is obtained by concatenating them as ## conclusion and future work in this paper, we proposed a method that takes into account the similarities of word strings and sounds as features for text normalization. our evaluation results showed that incorporating such features to seq2seq improves the performance of text normalization compared to the baseline method in terms of f 1 score. we also found that the proposed methods managed to effectively consider both surface character and phonetic similarities. for future work, we will try data augmentation based on the finding by @xcite . they reported that synthetic data can be generated by substituting words commonly confused with each other to improve performance. on the contrary, we will use the dataset proposed by @xcite , a much larger dataset than the wnut-2015 shared task2, to empirically gain deep insights. we would also like to apply elmo @xcite , transformer @xcite and bert @xcite to seq2seq for better performance.