Expert Concept-Modeling Ground Truth Construction for Word Embeddings Evaluation in Concept-Focused Domains (2020)

Abstract: We present a novel, domain expert-controlled, replicable procedure for the construction of concept-modeling ground truths with the aim of evaluating the application of word embeddings. In particular, our method is designed to evaluate the application of word and paragraph embeddings in concept-focused textual domains, where a generic ontology does not provide enough information. We illustrate the procedure, and validate it by describing the construction of an expert ground truth, QuiNE-GT. QuiNE-GT is built to answer research questions concerning the concept of naturalized epistemology in QUINE, a 2-million-token, single-author, 20th-century English philosophy corpus of outstanding quality, cleaned up and enriched for the purpose. To the best of our ken, expert concept-modeling ground truths are extremely rare in current literature, nor has the theoretical methodology behind their construction ever been explicitly conceptualised and properly systematised. Expert-controlled concept-modeling ground truths are however essential to allow proper evaluation of word embeddings techniques, and increase their trustworthiness in specialised domains in which the detection of concepts through their expression in texts is important. We highlight challenges, requirements, and prospects for future work.

## introduction a distinctive aspect of language is its capacity to express ideas and encode concepts, enabling us to treat certain natural language features as proxies for concepts and conceptual structures. mining concepts in text corpora is a valuable enterprise in many domains, but it is no trivial feat, as the link between concepts and the words expressing them cannot be (fully) automatised. in generic domains, concept mining can sometimes rely on resources that fix word-concept links, such as wordnet @xcite and thesauri @xcite . in expert domains, however, harder-to-come-by, specialised resources are needed. the issue is exacerbated in domains focusing on concepts and small corpora with high lexical variation and fine-grained meaning distinctions, like the history of the sciences and the humanities, and more broadly the history of ideas. importantly, the word-concept link problem is not limited to the humanities, or fields studying shifting meanings or changing concepts. it fundamentally applies to any corpora containing specialised terminology requiring substantial background knowledge for its decoding. retrieval of relevant information in text depends on correctly identifying which terms express the concepts of interest. any researcher or investigator struggling to sift through appreciably vast, lexically varied text collections to obtain specific, highly relevant information faces the problem, as do librarians and publishers wishing to give recommendations based on the ideas expressed in fulltext documents, rather than based on metadata or user profile @xcite . the word-concept link problem is just more obvious for corpora that are too big to be processed manually by otherwise very skilled humans, and harder the more specialised the information is. a widely used family of techniques in concept mining are word embeddings. word embeddings are mathematical representations of words exploited in distributional semantics models (dsms) @xcite @xcite . dsms are statistical language models based on the so-called distributional hypothesis @xcite , affirming that words with similar meanings tend to occur in similar contexts. dsms can help to cluster terms into (functional) synsets, and can be integrated in downstream applications such as information retrieval systems @xcite @xcite , which rank units of texts, such as paragraphs and sentences, by relevance. in certain specialised textual domains in which fine-grained study of concepts is traditionally performed exclusively by humans, the use of word embeddings on corpora of a few million tokens -too big a size for manual processing in a reasonable time -is, potentially, a game-changer @xcite . technology needs validation before it can be game-changing, however @xcite : word embedding techniques cannot gain a foothold in traditional domains such as history or philosophy if they cannot reasonably approximate or reliably support expert work, or if they introduce unknown biases @xcite . unfortunately, the performance of dsms is notoriously difficult to evaluate; it is even controversial how one should carry out evaluations in generic, big corpora domains @xcite . data sparsity makes for an even bigger difficulty: a few million tokens is very little data to create high quality dsms, due to their statistical nature. what would be a proper way to evaluate the quality of distributional models for concept mining in a specialised, concept-focused domain employing small corpora? answering this question would mean important progress for the prospect of concept mining in these domains, as we would know which models are best to choose for reliable downstream applications. we propose a novel method to construct replicable, expert-controlled, concept-modeling ground truths (henceforth: 'ground truths xc ') for the evaluation of word and paragraph vectors built from specialised corpora in concept-focused domains. ground truths xc should include or specify (a) a corpus; (b) a conceptual model and multiple research questions, distilled from the model by experts, that can be answered on the basis of the corpus; (c) lists of terms (preferably hierarchically ordered) that are manually selected and linked by experts to the concepts appearing in the model; (d) segmented units of text, ideally paragraphs, in which terms linked to the concepts appear and that are judged by experts relevant to the research questions, preferably with a varying degree of relevance. we also release a tool, hitparank, which speeds up the manual process of linking paragraphs (d) to concepts (b) via term lists (c). ground truths xc can be used to evaluate the performance of dsms trained on their source corpus as well as other corpora, as long as those corpora have characteristics similar to to the source corpus. particularly valuable will be ground truths xc that are extensive and varied enough for studying commonly used dsms. the method is generally applicable, language-independent, and extensible to multilingual data. it has its theoretical basis in a specific methodology in the history of ideas known as the 'model approach' @xcite , and is an operationalisation of that approach. according to the model approach, concepts cannot be properly studied unless they are represented as relational networks of terms, where the latter are the result of making textual interpretation explicit. foot_0 increase of replicability, objectivity and the explicitation of biases in humanities research are advantages of the method. although the model approach has been developed to the aim of studying concept change, it is in fact applicable to any concept, irrespective of their variability. to validate our proposal, we constructed a ground truth xc , quine-gt, concerning the concept of naturalized epistemology as represented in quine, a large corpus (c. 2 million tokens) in english by an eminent 20th century philosopher. in keeping with (a)-(d) above, quine-gt consists of clusters of terms and paragraphs annotated by experts as relevant, mildly relevant or non-relevant to answer three related research questions on ideas contained in the corpus. ## related work extensive work is done to apply natural language processing and machine learning techniques to problems of phrase extraction, e.g. @xcite , and concept mining, e.g. @xcite . however, whilst such fully-automated approaches are impressive in their generality and scale of application, concept-focused domains relying on fine-grained analysis of concepts require ground truths that represent 100% accurate expert knowledge. this means that there will always be humans in the loop, as the interest in specialised fields goes to computational techniques and tools that can assist researchers in mining concepts for specialised corpora. ground truths xc are extremely rare. the theoretical methodology behind their actual and concrete construction has, to the best of our ken, never been explicitly conceptualised and properly systematised. it is on the contrary rather customary to state that the notion of ground truth is inapplicable to the humanities and the social sciences ( nguyen et al. (2020) ), which are the domains in which theoretical reflection on textual representation of ideas and interpretation mostly tends to take place. most work on historical concept change does not involve ground truths xc in evaluation (cf. e.g. @xcite al. (2015), tahmasebi and risse (2017) ). humanities and social sciences @xcite and @xcite propose to base ground truths for concept-focused domains such as the history of ideas and intellectual history on betti and van den berg (2014)'s 'model approach'. these works focus however on turning conceptual models into computer science ontologies to detect concept change (as further developed in @xcite al. (2019) ), and lack details as how to actually build ground truths xc for use in language technology. mencarini (2018) signals the need for the model approach's operationalisation for language technology in the social sciences. @xcite combine the model approach with the use of ds modeling on a german 19th-century philosophy corpus. though they themselves provide no evaluation based on ground truths xc , and explicitly cherry-pick results of the system's output, @xcite do signal the need for proper evaluation of dsms relying on ground truths. building upon @xcite 's mixed method, which couples the model approach with manual annotations, @xcite offers an operationalisation of the model approach. they set up a model articulating the concepts of science, pseudo-science and epistemology as found in a lower quality, unlemmatised and unsegmented subset of our quine corpus, and associate their conceptual model with explicitly documented lists of terms. the terms are then queried via simple string searches afforded by voyant tools. foot_1 by these means, a number of textual fragments are retrieved and retained if relevant to certain research questions, and then annotated. our main theoretical innovations with respect to @xcite are two. first, the conceptualisation of a ground truth xc as the (a)-(d) package mentioned above. second, an increase in automatisation, generalisation and replicability of the method for ground truth xc construction. as for practical improvements: we improve the corpus' quality and usability, and, in releasing quine-gt and hitparank, we enlarge and refine the conceptual model, reformulate the research questions and the annotation model, expand the term lists, provide new and much more extensive lists of paragraphs, and perform more fine-grained annotations by multiple experts. bloem et al. (2019) apply nonce2vec (herbelot and baroni, 2017 ) -a modification of word2vec for tiny data -to provide consistent sentence embeddings for a very small corpus of two books by quine. they select terms from the books' subject index for comparing vectors, and document the difficulties of evaluation within that domain. in expanding the nonce2vec line of work to a corpus of neo-latin philosophical text, @xcite discuss dsms evaluation based on ground truths xc as a plan for the future. ## the quine corpus and hitparank we propose a general method applicable in the construction of any expert ground truth for any concept in concept-focused domains ('ground truths xc '). we take a ground truth xc to have (or have to specify) the following elements: (a) a corpus; (b) a conceptual model for concepts in that corpus from which research questions and annotation model are distilled; (c) term lists associated with (b); (d) selected text units annotated with degrees of relevance to research questions/concepts. ground truths xc are thus specific to the concept(s) modelled to answer specific questions, and intrinsically linked to specific corpora. although the corpora can be expanded, this can only happen under certain strictly controlled conditions. in the next section, we illustrate our general method by referring to an example by way of validation, a ground truth xc (quine-gt) that we built for a specific concept in a corpus (quine) comprising the virtually complete oeuvre in english of willard v. o. quine, a 20th century american philosopher. we start by describing quine, the corpus on which quine-gt is based, followed by the description of hitparank, a script developed by reynaert that we found to be helpful in the execution of the general method. in the following section we expound the general method using quine-gt as illustration. note that quine-gt is only one of the many possible ground truths xc that can be based on the same corpus constructed following the same general method. the quine corpus (version 0.5). the quine corpus consists of virtually all of quine's 228 books and articles, containing in total 819 documents (books are split into parts), 2,150,356 word tokens, 38,791 word types and 27,837 lemmatized word types. it includes texts in various genres and from different phases of quine's thought on various topics, including technical, and formula-heavy writings on logic and the foundations of mathematics. the corpus exhibits a high degree of lexical variation and many instances of fine-grained meaning distinctions. 3 the files originate from printed works, so substantial effort was required to produce a corpus of sufficiently high quality, as is often the case in computational humanities endeavors. in particular, it is key that the corpus accurately transcribes the text of the printed page, e.g. so that terms can be found by simple string matching. also, paragraphs must be properly segmented and distinguished from headings and footnotes, and given unique identifiers for easy reference by researchers and tools alike. the use of paragraphs for ground truth xc construction is not mandatory, but paragraphs are especially interesting segments in philosophical works as they tend to correspond to minimal units of thought. 4 what is mandatory is the use of units that can be segmented in a replicable and automatic way (such as sections, paragraphs, or sentences). information about word lemmas is also vital for our approach to concept mining both for query expansion (from lemmas to forms) and for mitigating sparsity when developing dsms of small corpora. here we briefly describe the preparation of the corpus through ocr and ocr correction, paragraph segmentation and the marking up of paragraphs and lemma information. the books and articles were converted from images in pdf files to plaintext format using abbyy finereader and ocr errors were corrected manually. books were split into separate files per chapter or section. data irrelevant to the content of the chapters, sections, and articles was removed, including e.g. repeated headers and page numbers, and the metadata was stored separately. some of the texts are rich in formulae and symbols which were replaced with codes as place holders, so as not to interfere unduly with linguistic analysis. we carried out extensive semi-automated work to restore the paragraphs in all the texts which involved dealing with issues including paragraphs split by page boundaries, footnotes and headings. some kinds of errors could be corrected via normalization scripts geared towards batches of texts that displayed similar issues, e.g. to deal with ad-hoc newlines that were erroneously present. this was complemented with manual editing and inspection of the original book images. at the same time issues relating to dashes and various forms of quotation marks were also resolved. we included metadata information (year, segment number for split books, etc) in the filenames of the texts in the corpus, for easy sorting and viewing in chronological order. the texts were converted to folia xml (van gompel and reynaert, 2013) using the english language module of ucto 5 , which also provided sentence segmentation and tokenization. folia markup can represent text as, and maintain unique identifiers for, sections, paragraphs, sentences and words. folia can also represent the lemma of each word. a dedicated folia xml module 6 called on spacy 7 using its core model for english 8 to provide lemmatization. hitparank -hitparank 9 is a perl script we developed that extracts the paragraphs possibly relevant to the research question from the corpus. the script has three modes of working: (i) extraction of word ngrams (currently @xmath0, from the lemmatized text layer). 10 (ii) checking the wildcard versions of the query terms selected by researchers against the actual ngrams extracted from the corpus by (i), and returning an expanded list of query terms and their actual corpus frequencies. this mode also shrinks the wildcard version of the lists by not returning non-occurring terms. (iii) extraction of the paragraphs that give hits on any of the query term lists and gathering statistics on all the extracted paragraphs, on the individual paragraph level and over the entire corpus. step 2. make research question(s) term-based and create initial lists of terms in order to be used for constructing ground truths xc for dsms evaluation, any concept-based research questions must be turned into research questions that can be answered by successfully querying for terms or paragraphs. this is in contrast with step 1 and typical research questions within philosophy, which are usually concept-based @xcite . to turn concept-based questions into term-based ones, all (sub)concepts at step 1 need to be manually associated by experts to clusters of terms in a certain corpus. through this association, a conceptual model can become an annotation model for textual corpora. at step 2 initial term lists are created, each relating to different aspects of the research questions. for example, if the question is about how a philosopher describes the relation between two concepts, then there will be at least one term list relating to each concept. the lists should ultimately consist of all terms that would indicate that the concept is present in a paragraph, e.g, for a question about plants, the term list would include different words for plants and types of plants. the initial term lists contain both full terms and wildcarded terms to capture multiple terms that share a stem, either by expanding the term forwards, backwards or both. each term is given a ranking, which usually indicates its weight in determining relevant paragraphs (say 1 to 3, 1 being the least relevant and 3 the most). to help with compiling term lists, researchers might read text passages identified as relevant to the research questions in previous studies. since frequently occurring ambiguous terms will result in many irrelevant paragraphs, they should be given a lower ranking or excluded. during this step and the next, researchers should expect to make extensive use of corpus analysis toolkits in order to check word and lemma usage in the corpus and their frequencies, e.g. the autosearch system foot_4 developed by the institute for the dutch language, based on their corpus indexing system blacklab foot_5 and whitelab corpus interface @xcite . ## discussion we have demonstrated how methodology from natural language processing research can be combined with methodology of the history of ideas to yield a concept-modeling ground truth controlled by domain experts. while ours is a resource-intensive approach, it can be broadly applied to tasks where language models make use of meaning representations that need to be grounded in an accurate conceptual model going beyond what is available in general resources such as wordnet. specifically, in distributional semantic modeling, our approach can address methodological issues in evaluating the quality of representations trained on specialized, domainspecific text by serving as a gold standard. in future work, our approach can be used to evaluate the quality of meaning representations created by various dsms from texts that are central to the history of specific scientific ideas. our methodology for creating ground truths xc can also be further developed into one that can be used to compare differences among expressions of concepts, particularly when applied to diachronic data to study semantic shifts across time. ground truth xc construction as mentioned, the units to annotate are, ideally, paragraphs, as these tend to correspond to one thought. a single, isolated sentence is e.g. hard to annotate because such a short unit rarely gives enough evidence to answer a research question, while a section might contain a lot of irrelevant, and thereby distracting, data. segmentation at paragraph level is however a challenging requirement: constructing or finding a sizeable corpus with acceptable quality for text analysis is already a major obstacle; getting a high-quality properly segmented corpus adds substantially to the difficulty. moreover, although units segmented in a replicable and automatic way are necessary to construct and properly evaluate dsms, philosophers and humanities experts are hardly used to build evidence up out of such segments: philosophers usually refer to the vague concept of a 'passage' of variable length as textual unit of evidence. in step 4 of our method, double filtering takes place on the basis of term list hits: the experts select paragraphs among those already selected by hitparank. this move has pragmatic reasons and its results depend on contingent factors (how many paragraphs are retrieved with different filters, how the terms in the lists relate to each other, etc). it also remains methodologically quite risky business: this type of filtering has immediate bearing on the final ground truth, ending up favoring precision over recall. in corpora that are too big for fully manual annotation, however, it is unworkable to annotate every paragraph. in such cases, the use of a tool like hitparank and further expert filtering are mandatory, but the type of filtering needs special care. when e.g. all terms are equally important to the research question, using a specific term as filter might not be sensible. in this case, purposive sampling (e.g. annotating a random sample out of the first hitparank selection) or a mix of methods might be the way to go. 15 in other cases, a trial and error approach with different filters might be fitting (by e.g. first checking whether at least some relevant units are retrieved with certain filters). the amount of annotated units should also match the purpose: training of certain dsms would require a rather large sample, therefore, the terms used for filtering should be such to give the most relevant results without giving too many to annotate but enough for training purposes. if this is not attainable, then a different research question should be considered. use of the ground truth the last point brings us to stress that a ground truth xc is only suitable for the specific question it was designed to answer. the ground truth xc created by our method can be used in computational settings to evaluate what model or family of models is better suited to model the target semantic domain (i.e. epistemology in quine). one way to do this would be to design an information retrieval task where the task is to retrieve the paragraphs that are relevant to the research question, out of the full corpus of texts from the ground truth. such a task would serve as an extrinsic evaluation of an underlying dsm, or as direct evaluation of ir systems. in an ir task, a research question from the ground truth will have to be translated by experts to one or multiple queries fitting the ir system in question. we could e.g. use @xcite or @xcite to create vectors for each paragraph in the ground truth, and then turn the query into a vector in the same semantic space, e.g. by summing the vectors of the words in the query. in such a system, the paragraph vectors that are closest to the query vector are the query results, and they should be relevant. when performing this task with multiple models, we can say that the better system or model for this text domain is the one that retrieves the most paragraphs that are relevant in the ground truth on the basis of that query, while retrieving the fewest paragraphs that are irrelevant in the ground truth. we can also say that a model that performs this task well, is likely to have good semantic representations of the target concepts and is a good conceptual model for that corpus. while this may seem like a highly specific result, models or hyperparameter settings that perform well in the domain of epistemology in quine may also perform well on other smaller corpora of concept-dense texts in specific fields. furthermore, if our method is used to construct multiple ground truths covering different domains and different kinds of corpora, we may be able to draw more general conclusions about the performance of various dsms used in nlp. future work when terms in the term lists are checked against the corpus in step 4 of the ground truth construction, the terms are lemmatised. in many cases lemmatisation is good since in concept-focused domains we are not interested in specific syntactical versions of words but in the concepts they express. however, there are cases in which one ideally only uses the full exact match of terms. for instance, the bigram classical studies is a part of list 3 as it is a scientific discipline. treated as a lemma, this also gives us paragraphs containing the bigram classical study, which does not refer to a scientific discipline. future work will try to improve on this by making it possible to choose between exact string matches or lemmas. further, we plan to experiment with additional clustering of the term lists into classes of various levels of conceptual granularity. this would allow us to make a pattern analysis of highly relevant paragraphs and to investigate where this type of conceptual patterns correlates with relevance.