Text Document Clustering: W ordnet vs. TF - IDF vs. Word Embeddings (2021)

Abstract: In the paper, we deal with the problem of unsupervised text document clustering for the Polish language. Our goal is to compare the modern approaches based on language modeling (doc2vec and BERT) with the classical ones, i.e., TF-IDF and wordnet-based. The experiments are conducted on three datasets containing qualification descriptions. The experiments’ results showed that wordnet-based similarity measures could compete and even outperform modern embedding-based approaches.

## introduction the aim of the paper is to evaluate different semantic distance calculation methods using clusterization by the agglomerative clustering method regarding qualifications collected in the integrated qualifications register (iqr). it is a polish public register supporting the integrated qualifications system (iqs) and regulated by the act of 22 december 2015 on the integrated qualifications system. the iqr enables broad access to qualifications functioning in the national education system and enhances its transparency, as well as encourages the development of lifelong learning @xcite . as a repository of information about qualifications, the iqr does not meet the definition of big data -at least not yet -but still, it can benefit from the use of natural language processing methods allowing the calculation of similarity of documents and their clustering. the project entitled "operating and developing the integrated qualifications register" financed by the european social fund aims at developing several applications supporting citizens in their career decisions and policy-makers in their strategic choices. the main problem was how to compare and find similar qualifications from different sources, e.g., higher education (he) diplomas and vocational education and training (vet) certificates, and group them in meaningful and interpretable clusters. at the beginning of our work, we aimed at exploring content-based semantic similarity of qualifications, so we relied mostly on unsupervised clustering methods. eventually, we covered both unsupervised and supervised techniques. we evaluated traditional methods and modern ones, as we wanted to test several approaches regarding their efficiency, interpretability, and feasibility. here, we will present part of our work dealing with unsupervised methods. ## datasets the dataset covers several thousand documents containing descriptions of qualifications (out of a total number of about 10000 qualifications included in the iqs and iqr). these descriptions mainly consist of so-called learning outcomes statements (los), which characterize the knowledge, skills, and attitudes required to obtain a given qualification. los can be broken down into three main components: an action verb, a skill object, and a context of the performance demonstration, e.g., "(person) creates documents using word processing software". learning-outcomes-based qualifications framework is intended to "provide a common language allowing different stakeholders in education and training, as well as the labor market and society at large, to clarify skills needs and to respond to these in a relevant way" @xcite . it is assumed that los allow for comparison of qualifications across the sector, institutional, and national borders, which was why we started with a content-based semantic similarity of qualifications and clustering techniques. the iqr is a source of information about qualifications functioning in the iqs. however, it does not contain descriptions and learning outcomes for some qualifications, especially he diplomas. this information is available on university and government websites, usually in pdf files. to obtain the data, we used web-scraping and ocr techniques. as a result, the iqr data has been complemented by about 2000 descriptions. in the experiment, we used four manually labeled datasets (see ## clustering method in our work, we decided to use the agglomerative clustering algorithm @xcite . the method iteratively joins samples into subgroups basing on a linkage criterion (in this case, an average distance). the obtained dendrograms allowed us to determine the set of flat clusters for each different threshold defined by the joining points. a sample dendrogram with a fixed threshold is shown in figure 1 . ## quality metrics to evaluate the results, we decided to use adjusted mutual information @xcite score that allows comparison between two different clusterings. we may have used another measure (such as the adjusted rand index), but according to @xcite , ami is the better choice because it performs well for unbalanced datasets. the score was calculated between the ground truth labels and all sets of labels obtained from the clustering algorithm. adjusted mutual information score is one of the information-theoretically based measures. it is based on mutual information (mi), which comes naturally from entropy. symbol description x, y set of classes/clusters h entropy m i mutual information n m i normalized mutual information am i adjusted mutual information xi, yi i-th element of x/y (class or cluster) p (xi), p (yi) probability of the document being in ith class or cluster p (xi ŷj) intersection of p (xi) and p (yj) e(m i) expected value of mi the problem with mutual information is that the maximum is reached not only when labels from one set (clusters) match perfectly those from the other (classes), but also when they are further subdivided. the simple solution for that is to normalize mi by mean of entropy of x and y : normalized mutual information can be further improved ("corrected for a chance") by subtracting the expected value of mi from nominator and denominator: for word2vec, tf-idf, and wu-palmer methods, we used four variants with a different subset of words: ## conclusion the obtained results confirm the importance of developing dictionaries, knowledge bases, and domain ontologies. wordnet-based measures of similarity may compete with embedding-based approaches in the task of text document clustering. our research shows that the wu-palmer similarity metric can obtain comparable or even better (for the ppkz dataset) results than the classical tf-idf method and the modern doc2vec approach. as far as the similarity of qualifications based on learning outcomes is concerned, one of the challenges discovered during our work was that the domain similarity and groups of qualifications were distorted by their source. qualifications from the same source, e.g., from the same university or curriculum, tend to contain common, formulaic phrases. this problem will be addressed in further work.