Multifaceted Domain-Specific Document Embeddings (2021)

Abstract: Current document embeddings require large training corpora but fail to learn high-quality representations when confronted with a small number of domain-specific documents and rare terms. Further, they transform each document into a single embedding vector, making it hard to capture different notions of document similarity or explain why two documents are considered similar. In this work, we propose our Faceted Domain Encoder, a novel approach to learn multifaceted embeddings for domain-specific documents. It is based on a Siamese neural network architecture and leverages knowledge graphs to further enhance the embeddings even if only a few training samples are available. The model identifies different types of domain knowledge and encodes them into separate dimensions of the embedding, thereby enabling multiple ways of finding and comparing related documents in the vector space. We evaluate our approach on two benchmark datasets and find that it achieves the same embedding quality as state-of-the-art models while requiring only a tiny fraction of their training data. An interactive demo, our source code, and the evaluation datasets are available online: https://hpi.de/naumann/s/multifaceted-embeddings and a screencast is available on YouTube: https://youtu.be/HHcsX2clEwg

## introduction many documents have an inherently multifaceted nature, a characteristic that domain experts could exploit when searching through large document collections. for example, doctors could search through medical archives for documents containing similar disease descriptions or related uses of a specific drug. however, one of the major challenges of information retrieval in such document collections is domain-specific language use: 1. training datasets to learn document representations are limited in size, 2. documents might express the same information by using completely different terms (vocabulary mismatch) or different levels of granularity (granularity mismatch), 3. and the lack of context knowledge prevents drawing even simple logical conclusions. domain-specific embeddings are available for a variety of domains, including scientific literature @xcite , patents @xcite , and the biomedical domain (kalyan and sangeetha, 2020). however, these approaches require large amounts of training data and computing resources. in this paper, we introduce and demonstrate our faceted domain encoder, a document embedding approach that produces comparative results on considerably smaller document collections and requires fewer computing resources. further, it provides a multifaceted view of texts while also addressing the challenges of domain-specific language use. to this end, we introduce external domain knowledge to the embedding process, tackling the problem of vocabulary and granularity mismatches. a screenshot of the demo is shown in figure 1 . the interactive demo, our source code, and the evaluation datasets are available online: https: //hpi.de/naumann/s/multifaceted-embeddings and a screencast is available on youtube: https://youtu. be/hhcsx2clewg . ## related work a popular approach for introducing external domain knowledge to the embedding process uses retrofitting of word vectors based on a graph of semantic relationships as a post-processing step @xcite . similarly, @xcite train fasttext embeddings on biomedical journal articles and additionally on sequences of medical terms sampled from a knowledge graph. dis2vec uses a lexicon of medical terms to bring word2vec vectors of domain terms closer together and to push out-of-domain vectors further away @xcite . unlike dis2vec, which concerns only whether a word is in the domain vocabulary or not, our approach handles diverse types of relationships between domain terms. nguyen et al. (2017) propose an extension of doc2vec, adding vectors for domain concepts as input for learning medical document embeddings. roy et al. @xcite annotate words in the input text with a list of matching entities and relationships from a knowledge graph and extend word2vec to jointly learn embeddings for words and annotations. their abstraction of the graph structure as text annotations enables the inclusion of different node types and edge connections into word embeddings. another work @xcite proposed k-bert, which extends bert @xcite by expanding input sentences with entities from a knowledge graph. multifaceted embeddings capture more than one view of a document. @xcite propose a multifaceted network embedding and apply community detection algorithms to learn separate embeddings for each community. @xcite suggest an extension to the deepwalk graph embedding, which learns separate node embeddings for different facets of nodes in a knowledge graph. similar to our approach, they propose to concatenate the obtained facet embeddings into a single representation. we learn separate embeddings for types of domain knowledge and concatenate them into an overall document representation. ## faceted domain encoder our faceted domain encoder is a supervised learning approach using a siamese neural network to encode documents and a knowledge graph as a source for additional domain information. the architecture is visualized in figure 2 . ## experiments for our experiments, we use two semantic textual similarity (sts) benchmarks from the biomedical domain, biosses @xcite and med-sts @xcite . the benchmarks comprise sentence pairs with relatedness scores assigned by domain experts. they measure embed-ding quality by comparing the annotator score with the embedding similarity of both sentences based on pearson correlation. to this end, biosses contains 100 sentence pairs collected from medical articles and judged by five domain experts at a scale of 0 to 4. we perform stratified 10-fold cross-validation as proposed by the benchmark authors. we divide the dataset into ten equally-sized subsets using the annotator scores for stratification. stratification ensures that each split has a similar distribution of related and unrelated sentence pairs. we train ten separate models on the subsets, always using one subset for testing and the remaining nine for training. note that we still use 30 percent of the training dataset for validation and early stopping: we stop the training process after the first epoch in which the loss on the validation set stops decreasing. med-sts contains 1,068 sentence pairs from medical records collected internally in the u.s. mayo clinics. two domain experts judged each sentence pair on a scale from 0 to 5. the dataset authors proposed a train-test split of 750 to 350 sentence pairs. additionally, we use 30 percent, or 225 pairs, of our training set for validation and early stopping. the experiment results listed in ## interactive user interface the user interface comprises three main parts: top center, bottom center, and sidebar. in the top center, the user can select a source document and one or all of the categories ("facets"). further, either a preprocessed (stop word removal, lemmatization) or a raw document version can be selected for the viewed documents and word highlighting can be ## conclusion current document embeddings require large amounts of training data and provide only a single view of document similarity, which prevents searches with different notions of similarity. in this paper, we introduced and demonstrated an approach for multifaceted domain-specific document embeddings. it is tailored to small document collections of only a few hundred training samples and leverages knowledge graphs to enhance the learned embeddings. experiments on two benchmark datasets show that our model outperforms state-of-the-art domain-agnostic embeddings and is on par with specialized biomedical document embeddings trained on extensive document collections while only using a tiny fraction of their training data. our demo provides a faceted view into documents by learning to identify different types of domain knowledge and encoding them into specific dimensions of the embeddings. thereby, it enables novel ways to compare documents and provides a comparatively high level of interpretability of neural-network-based document similarity measures. a promising path for future work is to remove our neural networks' reliance on ground truth data by designing a semi-supervised approach in which the model learns to update its training goal while discovering new domain terms by itself.