Challenging distributional models with a conceptual network of philosophical terms (2021)

Abstract: Computational linguistic research on language change through distributional semantic (DS) models has inspired researchers from fields such as philosophy and literary studies, who use these methods for the exploration and comparison of comparatively small datasets traditionally analyzed by close reading. Research on methods for small data is still in early stages and it is not clear which methods achieve the best results. We investigate the possibilities and limitations of using distributional semantic models for analyzing philosophical data by means of a realistic use-case. We provide a ground truth for evaluation created by philosophy experts and a blueprint for using DS models in a sound methodological setup. We compare three methods for creating specialized models from small datasets. Though the models do not perform well enough to directly support philosophers yet, we find that models designed for small data yield promising directions for future work.

## introduction philosophers apply text analysis to understand and delineate the precise meaning of concepts and the relations between them in a given text. this includes comparative research that investigates differences in how concepts are viewed in different philosophical schools or by individual philosophers. @xcite point out that comparative research on concepts should follow a conceptual model approach. this approach states that we should not look at shifts of individual concepts in isolation, but rather address changes of a conceptual model as a whole. in such a system, relations between concepts are made explicit and comparative studies should identify how such relations change. previous studies have shown that distributional methods can be used to support philosophical research by retrieving passages relevant to concepts in an author's work (e.g., the concept of grounding within the work of bernard bolzano, @xcite , but can we also generate distributional semantic (ds) models that are precise enough to identify differences in concepts? this paper takes a first stab at addressing this question. in particular, we address the challenges involved in dealing with highly technical domainspecific terms that are defined in small corpora. as such, our use case has properties difficult for ds modeling, but typical for disciplines working with comparatively limited data. we compare domain-specific embeddings created using word2vec @xcite and a countbased svd model @xcite to those created by nonce2vec @xcite , specifically designed for dealing with tiny data. taking into account previous work criticizing the use of ds models for detecting sense-shift, we construct a data-specific ground truth, apply multiple evaluation metrics and verify whether results are stable across various random initializations. our results confirm that svd representations are superior to word2vec for small data and show that nonce2vec outperforms word2vec and, in most cases, svd. however, results are currently not accurate enough for providing evidence or new insights to philosophers. nevertheless, we are hopeful that better results can be obtained in the future by optimizing nonce2vec to deal with small rather than tiny data and by creating a bigger, more balanced ground truth. the main contributions of this paper are (1) a new ground truth of philosophical concepts linked to a clean philosophical corpus that is particularly challenging to model; (2) a blueprint for investigating ds models for domain specific research; (3) a comparative study of different approaches of creating embeddings for highly domain-specific terms. 1 after presenting related work, we describe the philosophical context: requirements, corpus and our ground truth. in section 4, we outline how the ds models we use are created. we then present our evaluation and results in section 5 which is followed by our conclusions and discussion. ## related work in this section we cover (1) other work related to distributional semantics (ds) for specific concepts and conceptual change (2) critical reflection on evaluation and the methodology involved and (3) work on small datasets and identification of domain specific meaning. ## philosophical background and data the goal of this section is to provide some insights into the process of interpreting philosophical texts and the use case for our experiments. we briefly describe the process and challenges of close-reading and how it could be supported by ds models. then we present a corpus of philosophical texts and a ground truth for philosophy. ## results the tuned models were evaluated against the ground truth. this section presents multiple evaluation tasks and results to (1) explore different aspects of model quality and (2) stress-test our findings. in these tasks, we use the 74 terms from the conceptual network that were clustered by the experts. cluster similarity our similarity task is defined as follows: given a target term t t , a term from the same cluster t sc and a term from a different cluster t dc , we test whether the target term t t is closer to t sc than to t dc . if the cosine similarity between t t and t sc is higher than the cosine similarity between t t and t dc , it is counted as correct, else as incorrect. we carry out this comparison for all possible term combinations and report the percentage of correct outcomes. we report the proportion of target-terms that are classified in the correct cluster. we also show the proportion of target terms that are clustered incorrectly but plausibly given their relation (via reference or regimentation) to other clusters. we exclude all three terms that are out of vocabulary in any of the models, as a difference in target terms distorts the comparison. this way, we ensure that all models are evaluated on the same terms. ## conclusion and discussion the results show that, in general, n2v and svd represent the ground truth clusters better than w2v on this type of data. furthermore, we see that using n2v or svd for smaller, domain-specific data outperforms a larger domain-general w2v model trained on a large corpus. n2v is able to learn higher-quality embeddings than w2v from small texts, as it was designed to, and we confirm previous work showing that the same holds for countbased models to a limited extent. clustering methods (centroid and k-means) do not detect anything close to the clusters defined in the ground truth, whereas more fine-grained methods (cluster similarity and knn) do yield results that are clearly above chance. the evaluation in terms of the dunn index is also promising. despite the overall low performance, we take this as an indication that the models group the terms with some systematicity. furthermore, the rankings of the different models remain consistent across evaluations. arriving at the same results through various methods can be seen as a fulfillment of @xcite stress-test requirement. manual inspection of the clusters indicates that the imbalance in the (already very small) dataset is problematic for a k-nearest neighbors classifier, which assigned almost all words to the two biggest clusters. we expect that the same may hold for centroids and k-means. in hindsight, we could have controlled for this by extending the dataset beyond the terms in the index of word and object. while this might have provided more accurate insights, we expect that most use-cases that work with small (or even tiny) data are most likely also working with similarly unbalanced data. standard machinelearning techniques aiming to abstract over examples are most likely not able to pick up (potentially weak) signals based on just a few examples. we therefore consider fine-grained and example-based methods a more promising direction. overall, research on small data is still in an early phase. we see that models designed to work with tiny data outperform others on low-frequency terms, but yield only slightly better or comparative results when compared on mid-or high-frequency terms. it has to be considered that these models are overall very similar to the standardly used models. future research should explore more balanced approaches which combine the strengths of both versions. for instance, by adjusting the settings of n2v based on the frequency of a target term. from the perspective of a philosopher who may want to make use of ds models to support their work, the results we obtained in this study are not good enough yet. the minimum for exploratory work would be that the vast majority of the terms is correctly clustered and all categories are exemplified. currently most terms are placed in the two largest categories, which might even give high accuracy but still does not represent the data well. thereby, exploration of the data with these models could give a wrong impression about how the terms relate to each other. for hypothesis testing, the required accuracy depends on the hypothesis being tested, but in principle it is possible when the model no longer makes clear mistakes (but it may not be able to always distinguish between conceptual and relational connections and still make errors on clear borderline cases). unfortunately, this level of accuracy was not yet reached either. more broadly, as studying language change through diachronic word embeddings adds a layer of complexity beyond the synchronic word embeddings we investigated, we expect that diachronic word embeddings trained on small data sets will not be able to reflect actual conceptual change and thus directly support philosophical research at this stage. we do, however, think that our results have laid the groundwork for using ds models for exploratory purposes. we should keep in mind that the conceptual network based on word and object calls for very fine-grained distinctions. while this may still be too challenging, we expect that exploring differences in terms used by different authors could be more realistic.