Terminology-based Text Embedding for Computing Document Similarities on Technical Content (2019)

Abstract: We propose in this paper a new, hybrid document embedding approach in order to address the problem of document similarities with respect to the technical content. To do so, we employ a state-of-the-art graph techniques to first extract the keyphrases (composite keywords) of documents and, then, use them to score the sentences. Using the ranked sentences, we propose two approaches to embed documents and show their performances with respect to two baselines. With domain expert annotations, we illustrate that the proposed methods can find more relevant documents and outperform the baselines up to 27% in terms of NDCG.

## introduction many machine learning (ml) applications require the calculation of similarities between instances of the task under consideration. those instances could be of any nature such as numerical and/or categorical time series @xcite , gene expression @xcite or textual documents @xcite . perhaps the application of similarity assessment between objects that is vastly used on an everyday basis is the one used in search engines. as the most evident example, when a user is looking, via a given query and a search engine such as google, for relevant documents, there is basically a matching process happening behind. in such application, the user provides a query, as a set of keywords, and looks for the documents which best correspond to those keywords. this mainly leads to a ranking problem where the goal is to rank all the available documents (webpages in case of a web search engine) and provide the user with the most relevant documents. session search @xcite is an interesting challenge of such applications where the user reformulates the query based on the results returned by the system until the desired documents are found. as one can notice, the principal notion of the above-mentioned task is the similarity between instances. for example, a very common similarity measure between textual documents is the tf-idf which is based on two notions : the frequency of the terms appearing in the document, tf, and the importance of those terms through the entire set of documents, i.e. inverse document frequency, idf. many studies still use tf-idf to perform text-related task as it can, for many tasks, properly project the textual data into the numerical space such that the content is reflected accordingly. one can then use the tf-idf vectors of documents to measure the similarity between them, via cosine for instance. the tf-idf representation is widely used in different applications such as document clustering @xcite and topic modeling @xcite . as one can see, tf-idf embeds the document by operating at the word-level, i.e. it takes the tf-idf score of each word and represent the document as a (sparse) vector of size of the entire vocabulary. there is, however, another recent word-level operated technique, namely word2vec @xcite , which also aims at representing words. nevertheless, there are two main differences between these two approaches. firstly, tf-idf assigns a scalar to each word while word2vec represents each word by a vector. secondly, and more importantly, word2vec embeds each word using its context, i.e. its surrounding words. that being said, it is able to capture the semantical context of words. the latter has shown very good results in many applications such as text clustering and text classification @xcite . as mentioned previously, calculating document similarity and, consequently, finding similar documents is at the core of many ml tasks. sometimes, however, focusing on the entire document may not lead to capturing desired similar documents as not all parts of a document have the same importance level. for instance, if a document describes a novel device for people suffering from diabetes, then taking the entire document may not necessarily result in the similar documents talking about the same particular issue, but rather about the medical domain in general. note that this issue is different from data cleaning, and should be rather considered as data selection/weighting for the task of document embedding. in this paper, we investigate the above-mentioned problem and propose a technique for document embedding w.r.t. technical content of the documents. we show that the proposed method is able to find better similar documents once the technical content is concerned which, to the best of our knowledge, is the first research targeting this objective. to do so, we propose to first capture the keyphrases, i.e. composite keywords, of the document and, then, rank the sentences based on the keyphrases they contain. to detect keyphrases, we use the state-of-the-art technique explained in @xcite which is based on the k-core concept of graph-of-words. we then use the two previously-mentioned embedding techniques to represent the text and show that the proposed, hybrid method can efficiently capture relevant documents based on the technical content. the reminder of this paper is organized as follows : section 2 overviews the most related studies. section 3 first details the graph representation of documents and the k-core concept, before providing the framework of embedding documents using the sentence ranking obtained via the k-core approach. then, in section 4, we describe the experimental settings and the baselines as well as the collected dataset. we then report our results in the same section. finally, section 5 concludes the paper. terminology-based text embedding for computing document similarities on technical content tia@pfia 2019 2 related work over the past few decades, a large body of studies has been considering the relatedness of documents for a wide range of tasks such as text categorization or document classification. assessing the similarities between documents is at the core of many machine learning applications such as information retrieval, recommendation systems and text generation. in this section, we consider the most relevant studies with respect to the topic under investigation and position our work in regard to them. classically, the similarity between two documents has been measured by the cosine similarity between their tf-idf vectors, see @xcite and @xcite for example. alternatively, studies like @xcite considered other techniques such as lsi and multi-word methods and investigated their performance in different tasks. although tf-idf can still capture many characteristics of a document and boost the performance of many tasks such as topic modeling @xcite , they still fail to detect the entire context of a document such as the semantic relation between words @xcite . the paper of mikolov et al. @xcite introduced an entire new idea of textual representation where a document is analyzed at the word level and each word is represented based on its context, i.e. it carries semantic features of the document. the context of the word is defined by the co-occurring words. the model is simply a neural network where the vectorial embedding of a word is determined by those of its surrounding words. many variations of this approach have been later studied to, for instance, operate directly at the document level @xcite . in addition, the combination of tf-idf and word2vec has been widely used and shown to bring significant improvement in many applications @xcite . as it will be illustrated in section 4, we will make use of the combination of these two techniques to conduct a part of our experiments, and will show that they work very efficiently in capturing the technical side of documents. measuring the similarity between documents has been widely studied in the literature through different approaches. for instance, @xcite used a phrase recognition approach to detect similar documents, and @xcite employed radix tree to calculate similarities between web documents. in a different approach, @xcite introduces a graph-based method to exploit the hierarchical relations in order to efficiently calculate the similarity between documents. similarly, @xcite proposes to represent documents as typed heterogeneous information networks and, following the notion of graphs, it computes the distance between documents for the task of document clustering. perhaps the most related studies to ours are @xcite and @xcite . in @xcite , the authors propose to use a word-based method to capture similar documents via plsa. the document matching in this particular work is done based on the words appearing in the document. more precisely, the similarity between two documents is defined as linear combination of the cosine between the tf-idf vectors and the plsa-based representations of the words. although the objective of this paper is similar to our work in that they also rely on words-basis scoring, two main differences distinguish that research and ours. firstly, we focus on keyphrases and pay more attention on the parts of the text where those keyphrases are used, while @xcite relies on the entire content of the document. secondly, the present study uses the context-based techniques, such as word2vec to particularly target the semantic aspects of a document. in other words, if two documents describe the similar subjects with different vocabulary, the purely tf-idf based or topic modeling based approaches fail to see their similarities, while context-based techniques are able to capture that. in a similar manner, @xcite defines two documents to be similar if they have the same pieces of text such as sentences or paragraph, which has similar limitations as mentioned before. this current study, to the best of our knowledge, is the first one to propose technical content based similarity between documents using their semantic terminology. ## framework graph-based techniques have been widely used in representing textual documents, where, in general, entities of the text construct the nodes and the relation between them defines the edges. depending on the task in mind, the nodes could be words, phrases, sentences, etc. @xcite @xcite are only a few examples of such approach. following this line of thought, @xcite investigated the task of keyword extraction by representing a document as a graph-of-words and retrieving the main core of the graph. in the following, we briefly explain how this task is accomplished and, then, use the results of this technique to rank the sentences and eventually embed a document. ## conclusion in this paper, we have studied the problem of finding similar documents with respect to technical content. to do that, we proposed a hybrid approach which employs the graph representation techniques and sentence embedding. using the graph-of-words representation of a document and computing the cores of the graph, we first extract the keyphrases of the document of all cores of the graph. we then use this information to assign a score to each sentence ; once the scores are calculated, we use the embedding of each sentence and its score to calculate the embedding of the document. we proposed two approaches to embed sentence : (i) sent2vec, a state-of-the-art technique for sentence embedding, and (ii) a tf-idf weighted average of the words appearing in the sentence. we then compared those to two baselines, doc2vec, an existing method in the state-of-the-art to embed documents, and a tf-idf weighted average of the words appearing in the document. as dataset, we used 2.8m webpages of 43k startups that we crawled from the web, where we considered the combination of all webpages of a startup as a document. to evaluate, we asked human experts to score the output of all techniques. using the ndcg metric, we illustrated that the proposed metric can outperform the baselines up to 27% and, as a result, can provide better similar documents when technical content is concerned.