Eliciting Explicit Knowledge From Domain Experts in Direct Intrinsic Evaluation of Word Embeddings for Specialized Domains (2021)

Abstract: We evaluate the use of direct intrinsic word embedding evaluation tasks for specialized language. Our case study is philosophical text: human expert judgements on the relatedness of philosophical terms are elicited using a synonym detection task and a coherence task. Uniquely for our task, experts must rely on explicit knowledge and cannot use their linguistic intuition, which may differ from that of the philosopher. We find that inter-rater agreement rates are similar to those of more conventional semantic annotation tasks, suggesting that these tasks can be used to evaluate word embeddings of text types for which implicit knowledge may not suffice.

## introduction philosophical research often relies on the close reading of texts, which is a slow and precise process, allowing for the analysis of a few texts only. supporting philosophical research with distributional semantic (ds) models @xcite @xcite has been proposed as a way to speed up the process @xcite @xcite , and could increase the number of analysed texts, decreasing reliance on a canon of popular texts (cf. addressing the great unread, @xcite . however, we cannot evaluate semantic models of philosophical text using a general english gold standard, as philosophical concepts often have a very specific meaning. for example, the term abduction, usually meaning a kidnapping, denotes a specific type of inference in philosophy @xcite . therefore, models must be evaluated in a domain-specific way. the critical difference between the general case and the philosophy case is the following. it is easy to find native speakers of e.g. british english who have a good intuition of the meaning of its terms in general use, and the relations between them. this yields e.g. the simlex-999 word similarity dataset @xcite , covering frequent words and their typical senses. more difficult is finding 'native speakers' who have an intuition of the meaning of the terms used by a particular philosopher. the only candidate would be that philosopher themselves, and even then, the meaning of some of the terms used is the result of explicit analysis and definition rather than implicit language knowledge of the philosopher. uncommon terms with highly specific meanings are explicitly defined and debated, leading them to differ between philosophers or even within the works of a single philosopher. any accurate evaluation or annotation would require expert knowledge, and methods that can incorporate explicit knowledge, rather than judgements based on implicit knowledge of a standard language or jargon by one of its speakers. we test two direct evaluation methods for ds models described by @xcite on our case study, the works of willard v. o. quine, a 20th century american philosopher. instead of native english speaking crowdworkers, we selected expert participants who have studied this philosopher extensively. we aim to test whether these methods produce reliable results when participants need to use explicit rather than implicit knowledge, and consider the methods to be successful if inter-rater agreement matches that of other semantic evaluations. more broadly, our methodological findings apply to evaluation of ds models for specialized domains, language for specific purposes (lsp), historical language varieties or other language (varieties) for which no native annotators are available. ## related work most intrinsic evaluations compare word embedding similarities (e.g. in terms of cosine distance) to premade datasets of human similarity or relatedness judgements. sets of words are created and evaluated on semantic relations by participants, and the similarity between the assessments and an embedding space is used as a measure of performance. in specific domains, examples of such datasets of term ratings can be found for identifier names in source code @xcite , in the medical domain @xcite @xcite and in geosciences both for english @xcite and portuguese @xcite . the last two studies compare domain-specific embeddings to general domain embeddings and both find that the former perform better. a problem of these indirect datasets is that only naturally occurring, often high-frequency terms without any spelling variations, are evaluated, while ds models include many more variations @xcite . direct intrinsic evaluation methods, where participants respond directly to the output of models, can be categorized as absolute and comparative intrinsic evaluation @xcite . the former method evaluates embeddings individually and compares their final scores, while in the latter participants directly express their preferences between models. to our best knowledge, the only example of a domain-specific direct human evaluation is @xcite who evaluate french embeddings of health care terms by a human evaluation in which two medical doctors rate the relevance of the first five nearest neighbours of target terms from models trained on in-domain text. in the philosophical domain some evaluations have been conducted with other methods, sometimes incorporating expert explicit knowledge, but none are direct. in each of these studies the work of quine is utilized as data. firstly, @xcite propose a method of evaluating word embedding quality by measuring model consistency, not making use of expert knowledge. secondly, @xcite construct a conceptual network which serves as a ground truth of expert knowledge. they compare the similarity of embeddings for target philosophical terms to their position in the manually created network. here, the conceptual relatedness between terms is restricted to the property of sharing hypernyms, and only terms that were predefined in the ground truth can be considered for evaluation. @xcite introduce a more elaborate ground truth that is concept-focused, including more types of conceptual relations and including irrelevant as well as relevant terms for better evaluation of model precision. still, evaluation remains restricted to terms in the ground truth. only using direct evaluation methods we can attempt to evaluate all model output. ## task description we perform a synonym detection task and a coherence task. in these tasks, participants are asked to judge model-generated candidate terms that semantic models deem closest to a target term. in the synonym detection task, participants select the most similar word to target term t out of a set of options: the k-nearest neighbours of the target term in each model that is being compared. in the coherence task, the participant selects a semantical outlier in a set of words, where one of the words is not close to t in the model. we refer to @xcite for details and a comparison to other tasks for general semantic evaluation. our participant instructions are based on @xcite , who use the instructions of the wordsim-353 dataset @xcite . but as this study focuses on explicit knowledge, several adjustments are needed. although explicit knowledge is easier to verbalize than implicit knowledge, it involves controlled rather than automatic processing @xcite @xcite , so our version of the task might take longer. yet in order to retain the required focus, the test should not take too long. we therefore conduct a pilot study in which response times are measured to estimate task durations, and we adapt the size of the main study accordingly. the original task instructions do not define similarity, while other studies define it as co-hyponymy (turney and pantel, 2010) or synonymy @xcite . according to @xcite defining similarity is difficult as it depends on the context and downstream application in which the terms are used. we keep a consistent context, both training and evaluating in the domain of a particular philosopher, although the concern of capturing the multidimensional concept of similarity in a single number is valid also in this context. @xcite claim participants are likely to prefer synonyms when asked to select the most similar word. in this study we are looking to find any relationship present, rather than a specific one, and expect the experts to explicitly consider this, so we ask for relatedness. gladkova and drozd further argue that when asked for relatedness participants must choose between various relations present, a choice that can be subjective or random, and might reflect other factors such as "frequency, speed of association, and possibly the order of presentation of words". the first two factors are alleviated in this study as the participants must take quine's definitions of words into account rather than their own. this forces participants to think their answers through, which should reduce the association effects typical of fast-paced online studies. to account for effects of order of presentation, we randomize the order of the options. in our instructions, we define relatedness as synonymy, meronymy, hyponymy, and co-hyponymy, and provide examples. participants are also allowed to base their judgements on other types of relations. after the experiment we present a post-test survey, querying the types of relation participants based their judgements on, and task difficulty. furthermore, we change the option i don't know the meaning of one (or several) of the words in the synonym detection task to be none of these words is even remotely related, and also include a similar option in the coherence task, namely no coherent group can be formed from these words. this is done to avoid any random selection of words when there are no meaningful relations, making the responses more accurate. as we aim to gather explicit knowledge, participants are allowed to look up relevant information on presented words. for reproducibility, our instructions (and results) are included in the supplementary materials. 1 as the tasks require participants to be experts on the work of quine, the number of possible participants is limited. although participants are philosophers trained to work precisely and make consistent judgements, subjectivity can be a risk as participant must choose the relation they deem most important, while lacking context. we use inter-rater agreement to evaluate this. we report joint probability of agreement (percentage of agreement) as we have added the none options to avoid chance agreement. as joint probabilities cannot be compared across studies, we also report cohen's κ. all experiments 2 are conducted on the survey platform qualtrics 3 . participants are asked to exe1 to be found at https://github.com/ gvanboven/direct-intrinsic-evaluation 2 experiments were approved by the ethics committee of the faculty of humanities of the university of amsterdam. ## case study for philosophy we make use of the quine corpus (v0.5, betti et al., 2020), which includes 228 philosophical books, articles and bundles quine authored, consisting of 2.15 million tokens in total. as target terms for evaluation, we use @xcite test set for the the book word & object (quine, 1960), one of quine's most influential works. it consists of 55 terms that were selected from the book's index. we used 10 of these terms in the pilot study, 25 in the synonym detection task, 14 in the coherence task and 6 in both experiments. 4 one quine expert participated in the pilot study. the pilot study consists of short versions of the two tasks, both testing five target terms. in the synonym detection task, each target term has six candidate related terms from the models, that the participant should choose between. each term is tested three times with candidates of differing similarity from the model (nearest neighbour ranks @xmath0, 5, 50}). the pilot coherence (outlier) task has ten questions. the average response time for the synonym detection task was 109.5s and 42.1s for the coherence task. because for the first task this was higher than anticipated, we reduced the number of ranks to two and divided the task across two separate surveys. ## analysis to assess whether the method was successful we discuss some reliability metrics and examine disagreement examples. first of all, the fact that the data from one participant had to be excluded confirms the high standard of expertise required for participating in our version of the tasks. the results might have differed had there been more or different participants. however, other studies on expert explicit knowledge also execute tasks with two @xcite or three @xcite participants. inter-rater agreement scores for the two tasks were 58.1% (κ = 0.492) and 56.7% (κ = 0.345), indicating moderate or fair agreement. @xcite found the average inter-rater agreement of two raters of the wordsim-353 @xcite and men @xcite dataset to lie between κ = 0.21 and κ = 0.62. thus, agreement scores in this study are not lower than that of commonly used similarity datasets, despite participants having to agree on another person's semantics and including a none option. both experiments yield lower inter-rater agree-ment for the none option than for the other choices, shown in figure 1 (a) and (b). response times were also higher for the none option in both tasks ( according to the post-test survey, participants mostly based their judgements on sharing the same super term. relationships that were used without being listed in the instructions were antonymy, forming a technical bigram term together, having the same stem and being used in the same context by quine. we see this reflected when examining some examples of disagreement. in ## discussion in the post-test survey, participants commented that it was sometimes difficult to select the most related adjectives information application