t BERT : Topic Models and BERT Joining Forces for Semantic Similarity Detection (2020)

Abstract: Semantic similarity detection is a fundamental task in natural language understanding. Adding topic information has been useful for previous feature-engineered semantic similarity models as well as neural models for other tasks. There is currently no standard way of combining topics with pretrained contextual representations such as BERT. We propose a novel topic-informed BERT-based architecture for pairwise semantic similarity detection and show that our model improves performance over strong neural baselines across a variety of English language datasets. We find that the addition of topics to BERT helps particularly with resolving domain-specific cases.

## introduction modelling the semantic similarity between a pair of texts is a crucial nlp task with applications ranging from question answering to plagiarism detection. a variety of models have been proposed for this problem, including traditional featureengineered techniques @xcite , hybrid approaches @xcite @xcite and purely neural architectures @xcite @xcite . recent pretrained contextualised representations such as elmo @xcite and bert @xcite have led to impressive performance gains across a variety of nlp tasks, including semantic similarity detection. these models leverage large amounts of data to pretrain text encoders (in contrast to just individual word embeddings as in previous work) and have established a new pretrain-finetune paradigm. while large improvements have been achieved on paraphrase detection @xcite , semantic similarity detection in community question answering (cqa) remains a challenging problem. cqa leverages user-generated content from question answering websites (e.g. stackexchange) to answer complex real-world questions @xcite . the task requires modelling the relatedness between question-answer pairs which can be challenging due to the highly domain-specific language of certain online forums and low levels of direct text overlap between questions and answers. topic models may provide additional signals for semantic similarity, as earlier feature-engineered models for semantic similarity detection successfully incorporated topics @xcite @xcite . they could be especially useful for dealing with domain-specific language since topic models have been exploited for domain adaptation @xcite . moreover, recent work on neural architectures has shown that the integration of topics can yield improvements in other tasks such as language modelling @xcite , machine translation @xcite , and summarisation @xcite . we therefore introduce a novel architecture for semantic similarity detection which incorporates topic models and bert. more specifically, we make the following contributions: ## datasets and tasks we select popular benchmark datasets featuring different sizes (small vs. large), tasks (qa vs. paraphrase detection) and sentence lengths (short vs. long) as summarised in ## baselines topic baselines as a simple baseline, we train a topic model (lda or gsdmm) on the training portion of each dataset (combining training sets for semeval subtasks) and calculate the jensen-shannon divergence @xcite (jsd) between the topic distributions of the two sentences. the model predicts a negative label if jsd is larger than a threshold and a positive label otherwise. we tune threshold, number of topics and alpha value based on development set f1. 5 previous systems for semeval, we compare against the highest performing system of earlier work based on f1 score. as these models rely on hand-crafted dataset-specific features (providing an advantage on the small datasets), we also include the only neural system without manual features @xcite . for msrp, we show a neural matching architecture @xcite . for quora, we compare against the interactive inference network @xcite using accuracy, as no f1 has been reported. siamese bilstm siamese networks are a common neural baseline for sentence pair classification tasks @xcite . we embed both sentences with pretrained glove embeddings (concatenated with elmo for bilstm + elmo) and encode them with two weight-sharing bilstms, followed by max pooling and hidden layers. bert we encode the sentence pair with bert's c vector (as in tbert) followed by a softmax layer and finetune all layers for 3 epochs with early stopping. following @xcite , we tune learning rates on the development set of each dataset. 4 evaluation we evaluate systems based on f1 scores ( overall trends the bert-based models outperform the other neural systems, while closely competing with the feature-engineered system on the relatively small semeval a dataset. the simple topic baselines perform surprisingly well in comparison to much more sophisticated models, indicating the usefulness of topics for the tasks. ## conclusion in this work, we proposed a flexible framework for combining topic models with bert. we demonstrated that adding lda topics to bert consistently improved performance across a range of semantic similarity prediction datasets. in our qualitative analysis, we showed that these improvements were mainly achieved on examples involving domain-specific words. future work may focus on how to directly induce topic information into bert without corrupting pretrained information and whether combining topics with other pretrained contextual models can lead to similar gains. another research direction is to investigate if introducing more sophisticated topic models, such as named entity promoting topic models (krasnashchok and jouili, 2018) into the proposed framework can further improve results.