Domain-specific Evaluation of Word Embeddings for Philosophical Text using Direct Intrinsic Evaluation (2022)

Abstract: We perform a direct intrinsic evaluation of word embeddings trained on the works of a single philosopher. Six models are compared to human judgements elicited using two tasks: a synonym detection task and a coherence task. We apply a method that elicits judgements based on explicit knowledge from experts, as the linguistic intuition of non-expert participants might differ from that of the philosopher. We find that an in-domain SVD model has the best 1-nearest neighbours for target terms, while transfer learning-based Nonce2Vec performs better for low frequency target terms.

## introduction many applications of artificial intelligence methods to textual data rely on language models pretrained on large amounts of web text. however, this does not necessarily yield models suited to the analysis of texts in the humanities, as such texts may deviate in style, vocabulary, register and other regards. on the other hand, models trained on texts from a specific humanities domain have far less data available to learn from. as a trade-off, transfer learning can be applied, where a model is first trained on a large, general-domain dataset and then tuned on a smaller, domain-specific dataset. we compare several tuning approaches based on word2vec @xcite to a baseline of a svd model trained only on domain-specific data, for the purpose of learning meaning representations of the terminology of a specific philosopher. in philosophy, there is interest in supporting the close reading of texts with the use of information retrieval methods @xcite based on distributional semantic (ds) models @xcite @xcite to provide a different perspective on texts @xcite . philosophical terms have domainspecific meanings: for example an accident is a non-essential property of an entity, rather than an unfortunate incident. thus, domain-specific models and methods of evaluation are necessary. ds models are often evaluated by comparing their performance to a gold standard, such as the simlex-999 dataset @xcite . however, these similarity rankings concern general language terms and their typical senses, rather than domainspecific philosophical terms. meanings of terms may differ between philosophers or even within the works of a single philosopher. rather than modeling a standard jargon that a group of people uses, we aim to model the semantics of some particular philosopher, with no 'native speaker' besides that philosopher. any evaluation of the quality of the semantic representations in this kind of model would require expert knowledge. for this reason we apply the direct intrinsic evaluation methods proposed by @xcite with expert participants. we evaluate six models: we use wikipedia data as a general-domain text corpus for training, and we use a domain-specific corpus of the works of willard v. o. quine for tuning. quine was a 20th century american philosopher, whose works are still of great interest to philosophers, logicians and linguists. this evaluation will show us which tuning approach, if any, performs best for creating meaning representations of philosophical terms, and for digital humanities applications more broadly. various digital humanities studies using word embeddings have been published, but they rarely include in-domain evaluations of those embeddings. @xcite apply word embeddings to study relationships between persons in 6th century latin text, but do not evaluate their model. @xcite train domain-specific word embeddings on an 18m word corpus of narratives on slavery in the american south, but do not evaluate them. @xcite use domain-specific vectors for aligning medieval text versions, extrinsically evaluating by having an expert manually inspect the resulting alignments, but without intrinsically evaluating the embeddings. @xcite use word embeddings to study vocabulary shifts and have human annotators associate words to topics and time periods for evaluation. they do not use any pre-trained models. @xcite do perform in-domain evaluation, evaluating word embedding models trained on two fantasy novel book series of about 1m tokens each, manually constructing test datasets with domain experts. they compare domain-specific models including word2vec to a transfer learning setup where a pretrained word2vec model is tuned on the fantasy novel corpus. they find that an in-domain word2vec model outperforms the other approaches in an analogy task and a word intrusion task. @xcite find that fine-tuning pre-trained bert embeddings does not help for named entity recognition in historical corpora, though this is in addition to the use of in-domain fasttext embeddings. in the philosophical domain, several domainspecific evaluation methods have been proposed, but none have directly evaluated model output. evaluated models include the widely used w ord2v ec (w2v; @xcite ) predictive model, n once2v ec (n2v; @xcite which is an adaptation of the skipgram w2v model designed for learning from few training examples in tiny text corpora, and countbased sv d models in the @xcite implementation. avoiding the issue of obtaining expert knowledge, @xcite evaluate these models using a metric of model consistency, which rewards models that yield similar vectors when trained on different samples of the same target term within the same domain. they found that n2v outperforms a svd baseline by this metric. here, the options are the k-nearest neighbours of target word 'information' of the various evaluated models. what word does not belong to the group? oortwijn et al. ( 2021 ) evaluated these models based on a conceptual network they constructed, comparing the similarity of learned embeddings for specific philosophical target terms to their position in this network. they found that domain-specific n2v and the count-based baseline models outperformed a domain-general w2v model. as the network was pre-defined, only a limited set of terms could be considered. @xcite propose the use of a more elaborate ground truth in evaluation that includes many relevant as well as irrelevant terms, centered around a specific concept as defined by a specific philosopher. this still would not account for creative model output. lastly, van boven and bloem (2021) proposed the use of direct intrinsic evaluation to provide more comprehensive coverage of anything the models might output. methods from @xcite are adapted to the scenario of eliciting expert knowledge where experts respond to nearest neighbour words that the model generates. van boven and bloem (2021) report competitive interrater agreement scores between experts for this method. we adopt this approach for philosophical domain model evaluation. ## methods and data the direct intrinsic evaluation method consists of a synonym detection task and a coherence task, adapted from @xcite . synonym detection entails selecting the most related word to target word t from a set of words, which are the knearest neighbours of t in each included model. in this task, participants thus indicate their preference between the outputs of all evaluated models. following @xcite , we analyse the results of both tasks through a random permutation test, with the number of permutations @xmath0, 000. we use the difference in mean scores (i.e. the percentage of votes) between models as our test statistic for the synonym detection task, and for the coherence task we use the precision scores (i.e. the proportion of correctly identified outliers per model). as each model has its own qualities and the tasks evaluate different aspects, it is possible for the two tasks to yield different 'winners'. ## discussion the models that perform best are sv d q (the nontransfer learning baseline) and n 2v con . sv d q received the most votes in the synonym detection task, and performed especially well at producing rank 1 nearest neighbours related to the target term. n 2v con scored higher in the coherence task, producing better clusters of related top neighbours, and producing better rank 10 neighbours in experiment 1. this suggests sv d q would be more suitable for applications where top 1 precision is important, while n 2v con would do better in exploratory applications where a larger range of related terms is examined. the poor performance of the popular w 2v model for this domain and corpus size is in line with the findings of @xcite . both the count-based approach and the predictive approach produced a well-performing model. the hyperparameters and the transfer learning setup seemed to affect the scores more than the chosen approach. this matches @xcite claim that design choices and parameter settings influence embedding quality more than the model. for philosophical inquiry it is important that representations of low frequency words are good, as few resources are available and low frequency terms can be crucial to understanding a concept. in the synonym detection task, n 2v con does better on low frequency words while sv d q does better on high frequency words. conversely, in the coherence task, n 2v con does better on high frequency words, though its low frequency performance is still equal to sv d q . as n 2v con scored well overall on our low frequency target terms and in @xcite consistency evaluation, it appears the most promising for modeling philosophical terms. however, it is remarkable that a simple svd baseline performs so well compared to w2v-based transfer learning approaches. together with results from other work @xcite , this suggests that in digital humanities applications, in-domain data should be favoured over transfer learning approaches at least when 1m tokens of training data (or more) foot_3 are available.