More Than Words: Collocation Retokenization for L atent D irichlet A llocation Models (2022)

Abstract: Traditionally, Latent Dirichlet Allocation (LDA) ingests words in a collection of documents to discover their latent topics using word-document co-occurrences. Previous studies show that representing bigrams collocations in the input can improve topic coherence in English. However, it is unclear how to achieve the best results for languages without marked word boundaries such as Chinese and Thai. Here, we explore the use of retokenization based on chi-squared measures, t -statistics, and raw frequency to merge frequent token ngrams into collocations when preparing input to the LDA model. Based on the goodness of fit and the coherence metric, we show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those of unmerged models.

## introduction latent dirichlet allocation (lda) models provide useful insights into themes and trends in a large text collection through the unsupervised inference of topics, or probability distributions over unigram word types in the corpus @xcite . topics from these models are often interpreted based on their highest-probability words, with documents expressed as vectors of proportions of each topic. unfortunately, the context in which these tokens arise can be obscured in the bag-of-words rendering of text as unigram counts in documents. for instance, a topic with high probabilities of both "coffee" and "table" is tempting to interpret as focusing on the furniture item "coffee table", but both words could be frequent in a discussion of cafes containing no coffee tables. this problem is amplified in languages without marked word boundaries, such as chinese and thai: while existing tokenizers in these languages can segment characters into * corresponding author words, there is always a question about to what extent the tokenizers should group words together. words that have been segmented by tokenizers may not express the concept of the original text if they were found as parts of collocations. meaningful interpretation of topics can be lost without careful recombination of these words. we hypothesize that the morphology of the language should play an important role in determining the suitable pre-processing steps that would improve the results of topic models. the main morphological types we consider are synthetic language and analytic language. synthetic languages use many morphemes to compose a word and can be further divided into fusional and agglutinative languages. fusional languages such as german differ from agglutinative languages such as korean and japanese: a single morpheme in fusional languages can code for many morphosyntactic features. on the other hand, analytic languages such as thai and chinese convey meanings by relating many words together, and morphological devices are more rarely used. under our hypothesis, analytic languages should benefit from token merging, but synthetic languages might not because the meaning is conveyed by inflection (through bound morphemes) and agglutination (through free morphemes). in this project, we investigate the effects of token merging as a pre-processing step, and study how those effects vary based on the writing systems and the morphological features of the languages. we evaluate three measures to determine when to merge multiple adjacent words into conceptuallyunified phrasal tokens prior to lda model training: chi-squared statistics, t-statistics, and raw frequency counts of phrases. we test these merging strategies on english, german, chinese, japanese, korean, thai, and arabic. this set of languages is drawn from various writing systems and different morphological typology to see which type of language favors which type of merging strategy. the main contributions of this paper are as follows: ## related work pre-processing steps can substantially alter the results of the lda models even in languages with good tokenization heuristics such as english @xcite . we believe that languages that do not have clear tokenization standards deserve investigation into what kind of processing is appropriate. many works recognize that lda results can be improved when input are including phrases @xcite @xcite @xcite @xcite . we consider it valuable to specifically assess approaches to determining these phrases. despite their popularity in analyzing large amounts of text data, lda models are notoriously complex to evaluate. one must evaluate both the statistical fit of a model and the human-registered thematic coherence of the words found to arise in the high-probability words, or keys, of a topic, which may not correlate @xcite . analyses often combine evaluations of fit @xcite and automated approximations of human judgments of coherence @xcite based on mutual information, even with the expectation these may only somewhat correlate with true human judgments @xcite . a limitation of these existing approaches, however, is that they expect the vocabulary and tokenization to remain constant between the two models. for our evaluation, we use a normalized log-likelihood approach to capture fit while accounting for changes in vocabulary @xcite . ## collocations as lda token collocations consist of two or more words that express conventional meaning, which can convey information about multi-word entities, context, and word usage. we hypothesize that the introduction of multi-word tokens, which capture collocations as bigrams or trigrams by way of concatenation of adjacent tokens, can help achieve more useful and coherent topic models. for languages without clear word boundaries, there is a possible additional benefit to multi-word tokens: it can be hard to intuit whether inferred word boundaries will have a large impact on the final results. merging adjacent words into 'multi-word' tokens may help remedy the potential problem of a segmentation that is not optimal for topic modeling purposes. many methods are possible to select collocations to merge from tokenized text @xcite . in this paper, we evaluate the chisquared statistics (χ 2 ), the t-statistic and raw frequency as approaches to develop a threshold for merging collocations into multi-word tokens prior to topic model training. the chi-squared measure χ 2 (w 1 , w 2 ) and t(w 1 , w 2 ) t-statistic for two adjacent tokens w 1 and w 2 are defined as: we first compute the collocation measures for all bigrams on a large collocation training corpus. then we select the top bigrams that score the highest on the collocation measures and add those to our lexicon. after we tokenize and pre-process the collection of documents on which we would like to train lda, we retokenize the data based on the collocation training corpus. we find all of the bigrams in the lda training data that are also found in the top bigram lexicons that we obtain from the collocation training corpus. then, the lda training process proceeds as usual but with some of the original tokens merged into multi-word tokens as defined from the collocation training data. ## evaluation metrics we consider two primary evaluation metrics for exploring the effect of merging tokens: one based on log-likelihood, and one based on silhouette coefficients. held-out likelihood. when multi-word phrases are converted to individual tokens, the number of tokens in the document decreases while the size of the corpus vocabulary increases. it is therefore illogical to compare the likelihoods of the word-token model and collocation-token model directly. in order to normalize the scores between the two models that do not have the exact same vocabulary and tokens, we use the log-likelihood ratio between the lda model likelihood and the null (unigram) likelihood for each model. in other words, we normalize the lda model likelihood (l model ) by dividing it with the unigram likelihood (l unigram ) as introduced by @xcite . therefore, the normalized loglikelihood per token (ptll norm ) is a good topic should have all of its topic keys close to each other and away from other words that do not belong in the same topic. therefore, the word embeddings of these topic keys should have shorter cosine distances within the same topic, and longer distances to the topic keys in other topics. when words are represented as a vector, this is exactly what the silhouette coefficients measure. to compute them, we first compute the a(i), which is the mean cosine distance between topic-key i and other topic-keys in the same topic. after obtaining a(i) and b(i), the silhouette coefficient for topic-key i is defined as: the silhouette coefficient for the entire model is the average s(i) over all i. the larger silhouette coefficient means that topic-keys are relatively similar within their topic and different from other topics. in order to compare the distances among words merged by different criteria, all compared word embeddings must be in the same space. since merged tokens will modify the vocabulary of the corpus, we create four versions of the word embedding training corpus: the original version and the three other versions where tokens are merged based on χ 2 , t and frequency collocation measures. we train the word embeddings on these four versions of the corpus so we can then compare word embeddings on a consistent vocabulary in each retokenization scheme. ## experiments we hypothesize that morphology should play an important role in determining the suitable preprocessing steps. we test our methods on one fusional language (german), two agglutinative languages (japanese and korean), three analytic languages (chinese, thai, and arabic), and english, which can be thought of as either analytic or fusional. these languages also represent languages drawn from all writing systems: logograms (chinese), syllabic system (japanese), featural system (korean), abugida (thai), abjad (arabic), and true alphabets (english and german). the english corpora are drawn from the new york times @xcite , the yelp dataset 1 , and united states state of the union addresses (1790 to 2018) divided into paragraphs 2 . the german data come from ten thousand german news articles dataset 3 . the chinese data come from three corpora: the news articles from chinanews 4 , restaurant reviews from dianping 5 , and the movie reviews from douban 6 . the japanese data is from the webhose's free datasets 7 . the korean data come from the kaist corpus 8 . the thai data come from the news articles in prachathai 9 , the restaurant reviews from wongnai 10 , the best corpus 11 , and the thai national corpus @xcite . the arabic data come from the antcorpus @xcite . each corpus is separated into 75% training documents and 25% test documents ( we train the χ 2 , t, and frequency-based tokenizers for each language on wikipedia articles for that language. for all languages, we use the reduced version of wikipedia database, except for english we use the filtered wiki103 dataset @xcite . english, german, chinese, japanese, korean, thai and arabic documents are tokenized with nltk @xcite uhrig, 2016), stanford word segmenter @xcite , fugashi (mccann, 2020), konlpy (park and cho, 2014), attacut @xcite and camel-tools @xcite ) respectively. for each criterion, we create a list of 50,000 top bigrams that have the highest scores. these lists of top bigrams will be used to merge words in the input of the lda, effectively training a new tokenizer. to train word embeddings, we use the gensim @xcite implementation with the continuous bag-of-word (cbow) algorithm @xcite to obtain word embeddings. the training corpora and their collocation versions are prepared based on the tokenizers that we discuss above. we preprocess the word embedding training data and the lda training data the same way. for english, we lemmatize and lowercase the data. for korean, japanese, and arabic, we lemmatize the data. for german, chinese, and thai, we do not do any normalization. we use @xcite implementation of lda with the default hyperparameters to train and evaluate topic models in both word and multi-word (collocation) documents with 10, 50, 100 topics. we run the experiment 3 times for each combination of corpus, type of retokenization (no retokenization, χ 2 , t or frequency) and number of topics to compute the means of the normalized held-out likelihood and cbes, discussed in section 4. ## results and discussion the normalized log-likelihood per token of the t and frequency-based retokenization is significantly higher than the baseline for english, german, chinese, japanese, korean, and arabic for all text collections and the number of topics except en-yelp, th-best, and th-tnc ( similarly, we observe a general improvement in coherence for the t and frequency retokenization ( what could account for this discrepancy across languages and corpora? first, we observe a large variation of percentages of merged tokens across corpora. because we fix the number of bigrams types to merge during the tokenizer training process to 50,000 for all three criteria ( another possible effect these results may show is that the writing system or the morphology could account for this notable discrepancy in retokenization percentage across languages. for english, the top 20 χ 2 collocations are primarily specific named entities, but the t and frequency-based retokenizers yield more general compound nouns and common phrases (figure 1 ). as the top 50,000 χ 2 collocations contain primarily rare words, these are expected to co-occur rarely enough that even a few co-occurrences can trigger significance. therefore, when we use this truncated list of rarelyoccurring χ 2 collocations, we generally see a very low merged token percentage. the quality of retokenization impacts both the goodness of fit the model, as indicated by the normalized log-likelihood score, and the coherence of the model, as indicated by the cbes score. within the same language, news corpora have higher percentages of merged words when merged with t and frequency collocations, while corpora containing restaurant and movie reviews tend to see lower percentages ( with retokenization). pooling across all languages and corpora, we found the correlation coefficients of 0.41, 0.77, and 0.68 for the models with 10, 50, and 100 topics respectively for ptll. as for the coherence metric, we found the correlation coefficients of 0.73, 0.76, and 0.79 for the models with 10, 50, and 100 topics respectively for cbes. this means the models with higher merge percentages are better than their corresponding word models in reproducing the statistics of the held-out data. this suggests that the quality of the lda models depends on the generalizability of the retokenizers. the lda model results become more understandable when certain tokens are retokenized. we see merged tokens in the topic key sets of almost all topics in all corpora when retokenized based on t or raw frequency. many of these represent non-compositional meanings that might have been lost without retokenization: for example, the collocation "social security" is not fully represented by the individual tokens "social" or "security" separately. more strikingly, the collocation 'kōn sǔa dāng' refers to a political movement group in thailand. when it is separated into kōn (people) sǔa (shirt) dāng (red), the key meaning is totally lost. when we compare by looking at the topic-keys of the word and multi-word models, we can come up with similar topics because we as a human who understands english and has general knowledge of the world can make the connection based on surrounding topic-keys even though they are not explicitly merged. however, if we want to use these topic keys as input to other downstream tasks such as information retrieval or text classification, the merged tokens help retain the specificity of the "red shirt people" as a meaningful entity distinct from the phrase's constituting parts. ## conclusion in this work, we improve the quality of lda models by better processing the input text before training the model. we found that the retokenizers trained based on t statistics and raw frequency yield an improvement across all languages considered in this study, while the χ 2 approach was a less efficient approach that focuses more on rare named entities than common noun phrases. using retokenizers ensures that lda models can fit better to the data, the topic keys are more coherent, and the topics are more distinct. outputs from retokenization with t statistics and frequency approaches yield common noun phrases in the most frequent terms of topics that represent a significant aid to both direct topic interpretation and expected utility of these topics in downstream tasks.