S entence LDA : Discriminative and Robust Document Representation with Sentence Level Topic Model (2024)

Abstract: A subtle difference in context results in totally different nuances even for lexically identical words. On the other hand, two words can convey similar meanings given a homogeneous context. As a result, considering only word spelling information is not sufficient to obtain quality text representation. We propose SentenceLDA, a sentence-level topic model. We combine modern SentenceBERT and classical LDA to extend the semantic unit from word to sentence. By extending the semantic unit, we verify that SentenceLDA returns more discriminative document representation than other topic models, while maintaining LDA’s elegant probabilistic interpretability. We also verify the robustness of SentenceLDA by comparing the inference results on original and paraphrased texts. Additionally, we implement one possible application of SentenceLDA on corpus-level key opinion mining by applying SentenceLDA on an argumentative corpus, DebateSum.

## introduction what a word conveys can vary significantly in different contexts. it is not just a matter of polysemy. the word 'gas' in the energy-related context conveys different sentiments and implications compared to that in the environment-related context, though lexically identical. as a result, lexical property cannot fully explain the difference between different topics. to discriminate the differences, extending the semantic unit from word to sentence or paragraph seems natural and necessary. on the other hand, different words can convey similar meanings given similar contexts. synonym is a particular example. identifying similar words helps nlp models be robust to semantic information, not being biased to word spelling. to achieve sentencelda 1. the frequency of wars between states has diminished since world war ii (...). 2. the ability of countries to cooperate and resolve their differences through (...) direction taken by the worlds leading economy. 3. the worlds oceans have been shown to be less able to absorb and store carbon dioxide and other greenhouse gases (...). lda 1. states, war, world, power, china 2. fish, ocean, species, global, warming 3. would, one, even, years, world, said multiple researchers suggested contextualized word representation by pre-training neural language models (plm, @xcite @xcite to overcome these shortcomings. plm considers the contextual information by extending the semantic unit to the whole input text, not a separate word. plm showed great improvement in various nlp tasks for the last five years. meanwhile, topic models are useful tools for corpus-level analysis. latent dirichlet allocation @xcite is one of the most successful topic models which combines topic modeling and a probabilistic graphical model. lda proposed an elegant generative process by utilizing bayesian statistics to capture corpus-wide topics, as presented on in this paper, we propose sentencelda, a sentence-level topic model. we successfully combine lda and sentencebert @xcite while maintaining its probabilistic interpretation. our main research hypothesis is "does the semantic unit extension of a topic model from word to sentence return more discriminative and robust document representation?" we test the discriminativeness with a text classification task by comparing sentencelda with various topic models. we also test the robustness by comparing the representation of lexical and syntactic paraphrases. as a potential application, we apply sentencelda on the argumentative corpus to obtain corpus-level key opinions, as exemplified on ## related works various derivatives were proposed after @xcite proposed the lda. one stream of the research was to exploit continuous word embedding by exchanging the multinomial word distribution in the lda with gaussian distribution @xcite , utilizing linear classifier @xcite , or both @xcite . after the plms were introduced, e.g. elmo @xcite and bert @xcite , researchers have explored the possibility of combining the plms with classic topic models. they used the contextual information as a direct input @xcite , knowledge distillation (hoyle et al., 2020), or direct clustering @xcite @xcite . but this line of research focuses on improving the wordlevel topic model, not considering sentence-level information. another line of research explores the sentencelevel topic models. one way is to assign a topic to each sentence and sample words from the sentence topic distribution, not document topic distribution, @xcite @xcite . but these models still assume the exchangeability between words, which makes a model less effective in discriminating sentences sharing the same words but having different word orders. to bypass the exchangeability issue, some researchers applied a variational auto-encoder scheme based on rnn or lstm decoder @xcite @xcite . as an extension of this stream, researchers applied plms to obtain more useful sentence representation. but in the document-generating process, they usually depend on clustering @xcite or similarity metric between sentence and topic embedding @xcite . it is difficult to probabilistically interpret the topic embeddings, which are merely the center of each cluster. to the best of our knowledge, there is no sentence-level topic model, extending the lda while utilizing contextual information and maintaining its probabilistic interpretation. moreover, there is no work exploring the benefit of the sentence-level topic model from the view of discriminativeness and robustness. ## sentencelda: sentence-level topic model to capture a subtle nuance of a word, semantic unit extension from word to sentence is necessary. this extension would make a topic model more discriminative between documents sharing similar word distribution while containing different topics. on the other hand, this extension would make a topic model to be more robust to the semantics of a document, not being biased toward word spellings. to achieve these goals, we present our sentencelevel topic model sentencelda. but thanks to the simple modification, sen-tencelda fully utilizes a bayesian probabilistic framework. it is the key difference between sen-tencelda and other sentence-level topic models, which utilize clustering and similarity metrics. as a result, we can fully interpret the resulting topic distribution from a probabilistic perspective. lda gaussianlda sentencelda for example, we cannot probabilistically interpret the center of each topic cluster returned by other sentence-level topic models. in this perspective, each sentence is just 'a point which is close to the center of a topic cluster'. but sentencelda's topic embedding is the mean and mode of each gaussian topic distribution, and each sentence is a random sample from the topic distribution. we can also obtain the variance of each topic distribution, unlike other sentence-level topic models. as a result, sentencelda succeeds lda's elegant document-generative process while considering sentence-level information. moreover, sentencelda can fully utilize sentence-level information. by utilizing the sentence-level information, sentencelda is not biased toward word spelling. consequently, sen-tencelda can discriminate documents having different topics while sharing similar lexical distribution. moreover, since sentencelda does not assume exchangeability between words, it can capture the difference between two sentences sharing the same words but having different word order. on the other hand, sentencelda can robustly capture the semantics of two documents sharing the same meaning but having different lexical distributions or syntactic structures. as a result, sentencelda returns a more discriminative and robust document representation by considering sentence-level information. we evaluate these properties on section 4. we assume the existence of a feasible encoder f enc to encode each sentence into embedding space. also, we need a decoder f dec to decode the sen-tence embedding to a natural language sentence. we can improve the sentencelda by exchanging the encoder and decoder thanks to its modular structure. for implementation, we use sentencebert @xcite (all-mpnet-base-v2) as an f enc and fine-tuned gpt2-xl as f dec . for the topic model parameter inference, we utilize the collapsed gibbs sampling with cholesky decomposition as gaussianlda, which is presented in @xcite . ## application: key opinion mining from an argumentative corpus utilizing the discriminative power and robustness of sentencelda, we can robustly discriminate subtle nuances of a word in different contexts. one domain where capturing subtle nuances of words is important is argumentative corpus. in an argumentative corpus, word senses are complexly entangled. for example, the word 'korea' may contain totally different senses when the given context is economic development, democracy, human rights, or nuclear weapons. many researchers applied topic modeling to opinion mining-related tasks, e.g. sentiment classification @xcite , social community detection @xcite , and opinion summarization @xcite . however, the majority of these researches are limited to social media texts or product review datasets, which are relatively short and straightforward, unlike argumentative corpus. meanwhile, in the argument mining domain, researchers have utilized topic modeling techniques to access external knowledge through knowledge graph @xcite or to classify each sentence into argumentative unit categories (e.g. claims, and premises) in supervised @xcite and unsupervised manner @xcite extracted topics slda 1. with the war in ukraine, russia has not been able to count on the united states and europe to keep moscows feet firmly to the fire, much less to revive the stalled sinorussian economic cooperation. 6. as china grows more powerful, it is increasingly at odds with japan, which has a strong economic stake in the success of sinoamerican relations and is understandably nervous about beijings intentions in the south china sea. 10. the worlds oceans have been shown to be less able to absorb and store carbon dioxide and other greenhouse gases, and the number of species known to be experiencing reduced populations has been rising since the 1950s. lda 1. nuclear, would, weapons, iran, war, states, could, us, one, attack 3. fish, ocean, one, species, global, change, said, also, data, warming 5. energy, oil, gas, us, said, prices, also, years, new, industry 6. china, us, military, russia, trade, said, japan, security, new, would 8. states, war, world, power, china, conflict, economic, political, united, global to explore the possibility, we apply the sen-tencelda to debatesum dataset @xcite . debatesum dataset consists of 187,386 debate documents extracted from the national speech and debate association over a 7-year period. since it includes multiple heterogeneous debate topics from national defense to lgbt, we filter it with the keyword 'impact defense core'. as a result, we obtain 762 debate documents with 12,957 sentences. we apply sentencelda and lda with 10 topics, and the result is on though the lda captures 'war' (1), 'russia' (6), 'economic' (8) as separate topics, the first topic from the sentencelda captures a more nuanced topic related to the ukraine war and its economic consequences related to western countries. likewise, the lda captures 'ocean (sea)' (3), 'china' and 'japan' (6), and 'economic' (8) separately, while the sixth topic sentence from the sentencelda captures the complicated relationship between the words from economic history to current dispute on the south china sea between china and japan. the lda classifies 'gas' (5) into an energy-related topic and separates it from the environment-related topic (3), but the 10th topic sentence from the sentencelda shows that 'gas' can be used in the context of the earth's environment. in summary, the sentencelda enables us to obtain a more nuanced and comprehensive understanding of an argumentative corpus because it considers the relationships between words within the context of the sentence. while lda can provide insight into the most frequently occurring words and themes in the corpus, it may miss important nuances and variations in meaning that are present in longer text units. ## conclusion we introduce sentencelda, a sentence-level topic model combining modern sentence embedding techniques with a classic topic model. with a simple modification from lda and gaussianlda, sentencelda succeeds lda's elegant probabilistic interpretability. we demonstrate that the sentencelevel topic model can return more discriminative and robust document representation compared to word-level topic models. we evaluate the discriminativeness with a text classification task by comparing topic models on two classification datasets. as a result, sentencelda significantly outperforms other models by not being biased toward word spelling but considering word sense in context. by evaluating the robustness of each model with the paraphrased dataset, we observe that sentencelda predicts topic distribution for paraphrases more robustly than other topic models. especially, sen-tencelda returns more discriminative and robust document representation compared to topic mod-els utilizing the same contextual information, con-textualtm and senclu. also, we apply sen-tencelda to an argumentative corpus, debatesum, and demonstrate the applicability of sentencelda to corpus-wide key opinion mining. as a result, we show that sentencelda is a powerful tool to obtain discriminative and robust document representation.