Semi-supervised URL Segmentation with Recurrent Neural Networks Pre-trained on Knowledge Graph Entities (2020)

Abstract: Breaking domain names such as openresearch into component words open and research is important for applications like Text-to-Speech synthesis and web search. We link this problem to the classic problem of Chinese word segmentation and show the effectiveness of a tagging model based on Recurrent Neural Networks (RNNs) using characters as input. To compensate for the lack of training data, we propose a pre-training method on concatenated entity names in a large knowledge database. Pre-training improves the model by 33% and brings the sequence accuracy to 85%.

## introduction word segmentation is a fundamental nlp analysis problem for written languages with no space delimiters between words such as chinese and japanese. in the age of digital communications, new urls (e.g. www.openresearch.org ) and hashtags (e.g. #photooftheday), which often include strings of concatenated words (openresearch, photooftheday) are being added every day to a growing set of tokens that an nlp system may need to deal with, and they pose challenges for language and speech applications. for example, a text-to-speech (tts) synthesis system will struggle to pronounce these concatenated tokens, since simply applying a grapheme-to-phoneme system out of the box to something like photooftheday will usually yield poor results. this suggests the need for a model that can split such tokens into the component words. so-called "end-to-end" neural tts systems @xcite , which learn to map directly from character sequences to speech might seem to hold out the hope of avoiding treating this problem separately. however, the fact that urls occur relatively rarely in most tts training data limits the promise of such models on this long-tail problem. the problem of analyzing urls does differ in one useful way from more general text normalization problems. for a token such as 123 in a text, one typically needs to know what context it occurs in in order to know how to read it: is it one hundred twenty three or one twenty three; see @xcite , inter alia. in the case of urls, these are largely context-independent since the output segmentation is usually unaffected by the surrounding words. hence the problem can be treated as a standalone one that does not require the system to be trained as part of broader text normalization training. our training data comes from camel case urls that naturally define the segment boundaries (e.g. nytimes.com maps to n y times . com) along with manual corrections for non-trivial boundaries. we release our training and evaluation data sets to promote research on this problem. by drawing an analogy with chinese word segmentation, we cast the url segmentation problem as a sequence tagging problem. we propose a simple recurrent neural network (rnn) based tagger with an encoder and a decoder. the model trained on the data set has a decent full sequence accuracy (64%) but fails to generalize to more rare words due to the size of the training data. inspired by the success of pre-training in many nlp tasks @xcite , we propose a pre-training recipe for the segmenter. based on the observation that urls are often compound entity names and so are knowledge graph entities this work is licensed under a creative commons attribution 4.0 international license. license details: http:// creativecommons.org/licenses/by/4.0/ . @xcite , we create a large synthetic training data set by concatenating the knowledge graph entity names. we observe 21% absolute (33% relative) improvement in sequence accuracy after applying pre-training followed by fine-tuning. ## related work every text-to-speech system has to be able to read urls and other electronic addresses, but there is very little in the published literature that discusses this problem specifically as a research topic, and most systems seem not to do much in the way of interesting analysis of the internal components of the address. for example, the kestrel text normalization system @xcite identifies urls and other electronic addresses using finite-state matchers, and parses the main components into separate tokens based on standard delimiters (/, :, etc.): thus www.google.com would be parsed into www . google . com. the individual components are then pronounced separately. common components such as ., www, nytimes and com are handled by table look-up, but other components, such as jpopasia in jpopasia.com are not otherwise broken down and if they do not match a lexicon entry, may end up being read letter-by-letter. the problem has applications beyond tts. in web search, analyzing urls and hashtags leads to better matching. @xcite termed the problem "url word breaking". they used a noisy channel model with an n-gram language model trained on word-segmented data and a word-synchronous beam search algorithm for inference. the model is essentially unsupervised. they found that the style of the text used to build the model played a crucial role and document titles yielded the best results. in our pre-training experiments, we also tried web queries and documents. none of them gave the same improvement as the knowledge graph entity names in title case. @xcite improved the model of @xcite by adding a supervised max-margin structured prediction model using individual unsupervised language models as features. fine-tuning on in-domain data is the counterpart in our system. both have created training or evaluation data sets of url domain names for their experiments, but these have not been publicly released. we contribute a data set of urls crawled from a public repository of web texts with their internal segments annotated by crowd sourcing. @xcite reported experiments on a related artificial problem of splitting of space-free english through bayesian inference for fsts following the work of @xcite . 1 a closely related problem is compound splitting. @xcite presented an unsupervised probabilistic model for splitting compound words into parts, with the compound part sub-model being a zero-order model to enable efficient dynamic programming inference. the model is optimized for the task of machine translation. they only reported results for seven (germanic, hellenic, and uralic) languages other than english. more recently, fully supervised letter sequence labelling models have been introduced for german compound splitting @xcite and sanskrit word splitting @xcite . pre-training can potentially further improve these models. there is a large body of research on chinese word segmentation. the best models are supervised ones using structured prediction @xcite , transition-based models @xcite , and most recently rnns @xcite and bert-based models @xcite . the superior results of the bert-based models demonstrate that pre-training is effective on word segmentation. unlike bert, we pre-train the entire model, not just the encoder. ## rnn tagging model we formulate the segmentation problem as a character sequence tagging model. given an input character sequence @xmath0, . . . , x i , the model predicts an output tag sequence @xmath1, . . . , y i , with y @xmath2, i} being the tag for the character we model p (y |x) with an encoder-decoder rnn. the encoder generates hidden state sequences of length i. at decoding step i, the decoder attends to position i in both the hidden state sequences and the input embedding sequence. figure 1 illustrates the architecture of the model. our architecture is a modification of the stacking lstm architecture of @xcite which can also be understood as using hard attention @xcite in the decoder. we do not use bigram character features. instead, we rely on forward and backward rnns for implicit input feature extraction. we make the decoder auto-regressive by feeding the previously predicted tag to the decoder rnn and applying beam search in inference. ## experiments we choose the hyper-parameters on the internal crawl dev set for the network in figure 1 .