Domain-Specific Word Embeddings with Structure Prediction (2023)

Abstract: Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.

## introduction word embeddings @xcite are a powerful tool for word-level representation in a vector space that captures semantic and syntactic relations between words. they have been successfully used in many applications such as text classification @xcite and machine translation @xcite . word embeddings highly depend on their training corpus. for example, technical terms used in scientific documents can have a different meaning in other domains, and words can change their meaning over time-''apple'' did not mean a tech company before apple inc. was founded. on the other hand, such local or domain-specific representations are also not independent of each other, because most words are expected to have a similar meaning across domains. there are many situations where a given target corpus is considered to have some structure. for example, when analyzing news articles, one can expect that articles published in 2000 and 2001 are more similar to each other than the ones from 2000 and 2010. when analyzing scientific articles, uses of technical terms are expected to be similar in articles on similar fields of science. this implies that the structure of a corpus can be a useful side resource for obtaining better word representation. various approaches to analyze semantic shifts in text have been proposed where typically first individual static embeddings are trained and then aligned afterwards (e.g., @xcite @xcite . as most word embeddings are invariant with respect to rotation and scaling, it is necessary to map word embeddings from different training procedures into the same vector space in order to compare them. this procedure is usually called alignment, for which orthogonal procrustes can be applied as has been used in @xcite . recently, new methods to train diachronic word embeddings have been proposed where the alignment process is integrated in the training process. @xcite propose a bayesian approach that extends the skip-gram model @xcite . @xcite analyze dynamic changes in word embeddings based on exponential family embeddings. @xcite propose dynamic word2vec where word embeddings for each year of the new york times corpus are trained based on individual positive point-wise information matrices and aligned simultaneously. we argue that apart from diachronic word embeddings there is a need to train dynamic word embeddings that not only capture temporal shifts in language but for instance also semantic shifts between domains or regional differences. it is important that those embeddings can be trained on small datasets. we therefore propose two generalizations of dynamic word2vec. our first method is called word2vec with structure constraint (w2vconstr), where domain-specific embeddings are learned under regularization with any kind of structure. this method performs well when a respective graph structure is given a priori. for more general cases where no structure information is given, we propose our second method, called word2vec with structure prediction (w2vpred), where domain-specific embeddings and sub-corpora structure are learned at the same time. w2vpred simultaneously solves three central problems that arise with word embedding representations: all our methods rely on static word embeddings as opposed to currently often used contextualized word embeddings. as we learn one representation per slice such as year or author, thus considering a much broader context than contextualized embeddings, we are able to find a meaningful structure-between corresponding slices. another main advantage comes from the fact that our methods do not require any pre-training and can be run on a single gpu. we test our methods on 4 different datasets with different structures (sequences, trees, and general graphs), domains (news, wikipedia, high literature), and languages (english and german). we show on numerous established evaluation methods that w2vconstr and w2vpred significantly outperform baseline methods with regard to general as well as domain-specific embedding quality. we also show that w2vpred is able to predict the structure of a given corpus, outperforming all baselines. additionally, we show robust heuristics to select hyperparameters based on proxy measurements in a setting where the true structure is not known. finally, we show how w2vpred can be used in an explorative setting to raise novel research questions in the field of digital humanities. our code is available at https://github .com/stephaniebrandl/domain-word -embeddings . ## related work various approaches to track, detect, and quantify semantic shifts in text over time have been proposed @xcite @xcite @xcite . this research is driven by the hypothesis that semantic shifts occur, for example, over time @xcite and viewpoints @xcite , in political debates @xcite , or caused by cultural developments @xcite . analysing those shifts can be crucial in political and social studies but also in literary studies, as we show in section 5. typically, methods first train individual static embeddings for different timestamps, and then align them afterwards (e.g., @xcite @xcite @xcite ; and a comprehensive survey by @xcite . other approaches, which deal with more general structure @xcite and more general applications @xcite , also rely on post-alignment of static word embeddings @xcite . with the rise of larger language models such as bert @xcite and, with that, contextualized embeddings, a part of the research question has shifted towards detecting language change in contextualized word embeddings (e.g., @xcite . recent methods directly learn dynamic word embeddings in a common vector space without post-alignment: @xcite proposed a bayesian probabilistic model that generalizes the skip-gram model @xcite to learn dynamic word embeddings that evolve over time. @xcite analyzed dynamic changes in word embeddings based on exponential family embeddings, a probabilistic framework that generalizes the concept of word embeddings to other types of data @xcite . @xcite proposed dynamic word2vec (dw2v) to learn individual word embeddings for each year of the new york times dataset while simultaneously aligning the embeddings in the same vector space. specifically, they solve the following problem for each timepoint @xmath0, . . . , t sequentially: ## methods by generalizing dw2v, we propose two methods, one for the case where sub-corpora structure is given as prior knowledge, and the other for the case where no structure is given a priori. we also argue that combining both methods can improve the performance in cases where some prior information is available but not necessarily reliable. ## experiments on benchmark data we conducted four experiments starting with well-known settings and datasets and incrementally moving to new datasets with different structures. the first experiment focuses on the general embedding quality, the second one presents results on domain-specific embeddings, the third one evaluates the method's ability to predict structure and the fourth one shows the method's performance on various word similarity tasks. in the following subsections, we will category #articles natural sciences 8536 chemistry 19164 computer science 11201 biology 10988 engineering & technology 20091 civil engineering 17797 electrical & electronic engineering 6809 mechanical engineering 4978 social sciences 17347 business & economics 14747 law 13265 psychology 5788 humanities 15066 literature & languages 24800 history & archaeology 16453 religion & philosophy & ethics 19356 ## application on digital humanities we propose an application of w2vpred to the field of digital humanities, and develop an example more specifically related to computational literary studies. in the renewal of literary studies brought by the development and implementation of computational methods, questions of authorship attribution and genre attribution are key to formulating a structured critique of the classical design of literary history, and of cultural heritage approaches at large. in particular, the investigation of historical person networks, knowledge distribution, and intellectual circles has been shown to benefit significantly from computational methods @xcite . hence, our method and its capability to reveal connections between sub-corpora (such as authors' works), can be applied with success to these types of research questions. here, the use of quantitative and statistical models can lead to new, hitherto unfathomed insights. a corpus-based statistical approach to literature also entails a form of emancipation from literary history in that it makes it possible to shift perspectives, e.g., to reconsider established author-based or genre-based approaches. to this end, we applied w2vpred to high literature texts (belletristik) from the lemmatized versions of dta (german text archive), a corpus selection that contains the 20 most represented authors of the dta text collection for the period 1770-1900. we applied w2vpred in order to predict the connections between those authors with λ = 512, τ = 1024 (same as wikifos). as a measure of comparison, we extracted the year of publication as established by dta, and identified the place of work for each author foot_9 and categorized each publication into one of three genre categories (ego document, verse, and fiction). ego documents are texts written in the first person that document personal experience in their historical context. they include letters, diaries, and memoirs and have gained momentum as a primary source in historical research and literary studies over the past decades. we created pairwise distance matrices for all authors based on the spatial, temporal, and genre information. temporal distance was defined as the absolute distance between the average publication year, the spatial distance as the geodesic distance between the average coordinates of the work places for each author and the genre difference as cosine distance between the genre proportions for each author. for each author, we correlated linear combinations of this (normalized) spatio-temporal-genre prior knowledge with the structure found by our method, which we show in figure 4 . ## conclusion we proposed novel methods to capture domainspecific semantics, which is essential in many nlp tasks: word2vec with structure constraint (w2vconstr) trains domain-specific word embeddings based on prior information on the affinity structure between sub-corpora; word2vec with structure prediction (w2vpred) goes one step further and predicts the structure while learning domain-specific embeddings simultaneously. both methods outperform baseline methods in benchmark experiments with respect to embedding quality and the structure prediction performance. specifically, we showed that embeddings provided by our methods are superior in terms of global and domain-specific analogy tests, word similarity tasks, and the qvec evaluation, which is known to highly correlate with downstream performance. the predicted structure is more accurate than the baseline methods including burrows' delta. we also proposed and successfully demonstrated a procedure, word2vec with denoised structure constraint (w2vden), to cope with the case where the prior structure information is not suitable for enhancing embeddings, by using both w2vconstr and w2vpred. overall, we showed the benefits of our methods, regardless of whether (reliable) structure information is given or not. finally, we were able to demonstrate how to use w2vpred to gain insight into the relation between 19th century authors from the german text archive and also how to raise further research questions for high literature. analogies each analogy consists of two word pairs (e.g., countrya -capitala; countryb -capi-talb). we estimate the vector for the last word by @xmath1, and check if capitalb is contained in the n nearest neighbors of the resulting vector v.