TFW 2 V : An Enhanced Document Similarity Method for the Morphologically Rich F innish Language (2021)

Abstract: Measuring the semantic similarity of different texts has many important applications in Digital Humanities research such as information retrieval, document clustering and text summarization. The performance of different methods depends on the length of the text, the domain and the language. This study focuses on experimenting with some of the current approaches to Finnish, which is a morphologically rich language. At the same time, we propose a simple method, TFW2V, which shows high efficiency in handling both long text documents and limited amounts of data. Furthermore, we design an objective evaluation method which can be used as a framework for benchmarking text similarity approaches.

## introduction identifying documents that describe similar topics is a challenging yet important task. detecting similar documents automatically has a wide range of digital humanities applications such as ocr post-correction @xcite , automatic clustering and linking of documents @xcite and clustering of semantic fields within a document @xcite . assessing document similarity automatically becomes an important task especially due to the often unstructured nature of digital humanities research data (see @xcite ). this makes it possible to handle large text corpora in a more organized fashion by clustering similar texts together. in this paper, we explore different approaches to textual similarity detection, namely tf-idf, use, doc2vec and our own proposed approach named tfw2v foot_0 . our approach combines the traditional tf-idf method with word embeddings to improve the overall performance of the text similarity method. unlike the recent neural approaches such as bert @xcite , roberta @xcite or xlnet @xcite , our method does not rely on a large external corpus, but can be fully trained on the texts the similarity of which one is to assess. this is useful since our model can then work on corpora that represent a different era than what modern nlp models are trained on, or even for languages that do not have massive text collections readily available or are spoken by communities that do not have access to the computational resources needed to train large neural language models. ## related work a survey conducted by @xcite showed that 83% of text-based recommendation systems in digital libraries use tf-idf. there is also a recent survey paper on the current state of finnish nlp @xcite . there is a number of papers studying automatic detection of genres @xcite @xcite , which, as a task, is not too far from ours. however, in this section, we focus mainly on approaches on document similarity. @xcite have combined multiple document representation approaches, which are tf-idf, lda and doc2vec, to classify documents in a semi-supervised fashion. their results indicate that combining the features of the aforementioned models enhanced the performance of the classification task. truşcȃ (2019) has compared how different text representation models perform when training a support vector machine (svm) classifier. the results show that doc2vec was the superior model for the task addressed by the author, which is text categorization. @xcite also showed that clustering finnish text is more effective by doc2vec compared to lda. a recent study by @xcite compared wordnet -a manually constructed network of concepts-, tf-idf and word embeddings extracted from doc2vec and bert for unsupervised classification of polish text documents. their study showed that manually constructed knowledge bases, i.e. wordnet in this case, is a valuable resource for the task. @xcite merged tf-idf and word embeddings similarity scores to build the recommendation system for similar bug reports. @xcite have used text representation models to extract keywords from short texts collected from social media by employing a tex-trank @xcite algorithm which constructs a network and traverses it using random walk to discover the most important concepts. text representation models have also been utilized with deep neural networks to classify text by @xcite . tf-idf and word embeddings have also been used to assess the similarity of entities @xcite . in particular, the authors used the aforementioned methods to extract and predict properties for pokémon. ## experiments in this section, we apply four of the existing approaches to predict similarity of documents: doc2vec, universal sentence encoder (use), term frequency-inverse document frequency (tf-idf) and average weighted word vector (avgwv). later on, we propose a new method to optimize tf-idf by using a word embeddings model (tfw2v). all the experiments use the same datasets, the sampling process of which will be presented in the following section. ## evaluation turning now to the evaluation, as previously stated, we have 10 independent datasets for benchmarking. all the experimented parameters from the models are applied consistently for those datasets. we assess the performance of the 5 methods tf-idf, avgwv, doc2vec, use and tfw2v by three criteria: top-n precision, top-n bleu score and top-n ranking loss. we will explain those metrics in the following sections of the paper. ## conclusion in summary, we have presented a simple method to improve the tf-idf algorithm by using a word embeddings model. the proposed method outperforms the more complex models like doc2vec and use. we also compared it to a popular method avgwv which use the same combination of tf-idf and word2vec but in different way. it is very obvious that our proposed approach is surpassing the avgwv model. the weakness of avgwv is that it's hard to control the averaged vector representation of a document when all words and their tf-idf weights are taken into account. additionally, the impact from word vectors could come to too strong in some cases, which create the bias in the final decision. in our method tfw2v, we can control the effect of word embedding model on the similarity score. at the same time, not all the words are considered into the enhancing process but the important ones. thus, it is more stable, flexible and controllable to apply in various purposes. for example, in the document retriever system, the parameters can be set to get the relevant result as priority. on the other hand, in a recommender system, the parameters can be adjusted to get more creative result, thus it can look up for the under-discovered articles. it is clearly observable that with a morphologically rich language like finnish, tf-idf still works very effectively. however, by combining it with a word2vec model and our algorithm 1, the result is significantly enhanced. the method is entirely unsupervised and works well with a small dataset like in our experiment with only 2000 samples. in the future work, we will experiment this method for more languages and different lengths of document. the source code of this project will be provided as a python library foot_5 which is easy to install and apply for any dh research. the lack of dependency on neural language models trained on massive amounts of data makes our approach applicable in scenarios where such amounts of text are unfeasible to obtain.