Evaluation of Word Embeddings for the Social Sciences (2022)

Abstract: Word embeddings are an essential instrument in many NLP tasks. Most available resources are trained on general language from Web corpora or Wikipedia dumps. However, word embeddings for domain-specific language are rare, in particular for the social science domain. Therefore, in this work, we describe the creation and evaluation of word embedding models based on 37,604 open-access social science research papers. In the evaluation, we compare domain-specific and general language models for (i) language coverage, (ii) diversity, and (iii) semantic relationships. We found that the created domain-specific model, even with a relatively small vocabulary size, covers a large part of social science concepts, their neighborhoods are diverse in comparison to more general models Across all relation types, we found a more extensive coverage of semantic relationships.

## introduction word embedding models learn word representations from large sets of text so that similar words have a similar representation. models can be used to find semantically related words, for example for applications such as natural language understanding. technically, word embeddings are distributed representations of words in a vector space @xcite so that related words are nearby in the space and can be found with distance measures such as the cosine similarity @xcite . in general, word embedding models are trained on large and general language text collections, e.g., on web corpora or on wikipedia dumps. however, there are some initiatives to create and evaluate word embeddings for specific domains on a smaller scale, for example, for computer science @xcite , finance @xcite , patents @xcite , oil & gas industry @xcite , and especially in the biomedical domain @xcite @xcite @xcite . word embeddings capture "precise syntactic and semantic word relationships" @xcite . however, general and domain-specific models can differ much in terms of included specialized vocabulary and semantic relationships @xcite @xcite . intrinsic evaluation methods are used to test models for these relationships @xcite . in this work, we focus on creating and evaluating word embeddings for the social science domain and comparing them to general language models. word embeddings are used in the social sciences domain for a number of nlp tasks. @xcite provide an overview of word embeddings techniques and applications in the social sciences based on a literature review. word embeddings are used, for example, for the extraction of trends of biases or culture from data @xcite , using vectors to define working variables that embody the concept or research questions @xcite , or use reference words and their semantic neighborhoods to analyze gender terms and its relation to specific occupations @xcite . other applications are the processing of scientific documents, for example the extraction of acknowledged entities from full texts @xcite . for the retrieval of specialized information, word embeddings can be used for query expansion @xcite . all these applications depend on meaningful word embeddings. the more precise the specialized language is available in the vector space and the better related terms are arranged in the vector space, the better the applications work. ## evaluation in what follows, we evaluate word embeddings trained on social science language versus those trained on general language. we want to understand the effects on domain-specific language coverage, diversity, and semantic relationships. the evaluation is performed with the german models, since a larger part of the source texts is written in german. these models are called ssoar.de in the remainder of this paper. ## conclusion in this work, we built domain-specific word embeddings for the social sciences and compared them to general language models. first, we checked for coverage of specialized language keywords. wiki.de performed best with 92%, but ssoar.de followed closely with 88% with only one-fifth of vocabulary size. we then analysed the diversity of models to each other by comparing selected neighbourhoods: domain-specific and general-language models showed the highest diversities. however, diversity decreases with a larger number of returned neighbors. concerning relational coverage, ssoar.de models performed best in all settings, except for the broader relation with top-@xmath0. in summary, the word embeddings produced in this work showed much better results than the general language models when compared to established knowledge resources such as a thesaurus. domain-specific word embeddings can improve the semantic relatedness metric and applications build upon. this is in line with related works (e.g., @xcite @xcite showing for other domains that domain-specific models can better capture semantic relations -even with a small corpus size (e.g., @xcite . the experiments showed that the underlying texts and their language have a significant impact on the resulting word embeddings. this is even more true for the applications that are based on them. (1) coverage of social science concepts is very different depending on the word embedding model. for example, applications that want to define and extend a working variable depend directly on the concepts contained in the model. (2) the models can be very diverse in terms of their semantic neighbors. applications based on them, for example, query expansion or reference words and their neighborhoods, lead to different results depending on the model. (3) for relational coverage, the domain-specific model contains more relations in the sense of a domain thesaurus. this may be important, for example, to keep the precision of search results high in query term expansion or to keep working variables precise in expansion. in summary, the performance of applications is directly dependent on the alignment of word embeddings, their underlying language and their domain. in future work, we want to compare the effects of specialized language with other embedding models such as word2vec, glove, or contextual embeddings such as bert.