Incorporating the Rhetoric of Scientific Language into Sentence Embeddings using Phrase-guided Distant Supervision and Metric Learning (2022)

Abstract: Communicative functions are an important rhetorical feature of scientific writing. Sentence embeddings that contain such features are highly valuable for the argumentative analysis of scientific documents, with applications in document alignment, recommendation, and academic writing assistance. Moreover, embeddings can provide a possible solution to the open-set problem, where models need to generalize to new communicative functions unseen at training time. However, existing sentence representation models are not suited for detecting functional similarity since they only consider lexical or semantic similarities. To remedy this, we propose a combined approach of distant supervision and metric learning to make a representation model more aware of the functional part of a sentence. We first leverage an existing academic phrase database to label sentences automatically with their functions. Then, we train an embedding model to capture similarities and dissimilarities from a rhetorical perspective. The experimental results demonstrate that the embeddings obtained from our model are more advantageous than existing models when retrieving functionally similar sentences. We also provide an extensive analysis of the performance differences between five metric learning objectives, revealing that traditional methods (e.g., softmax cross-entropy loss and triplet loss) outperform state-of-the-art techniques.

## introduction scientific articles explain new ideas or discoveries and attempt to convince readers of their validity and importance. a key characteristic that distinguishes these articles from other texts is their specific rhetorical structures. the most well-known example is the main section of a paper, organized ## methods our approach can be roughly divided into two parts: phrase-guided distant supervision (sections 3.1 and 3.2) and metric learning (section 3.3), as illustrated in figure 2 . ## conclusions and future work this paper presents scitoricsbert, a sentence representation model that recognizes the rhetorical aspects of scientific writing. the proposed model achieves more successful results than existing representation models in retrieving functionally similar sentences. we also provide empirical evidence that softmax cross-entropy loss is a strong baseline for learning task-specific sentence embeddings, which has practical implications for other studies on representation learning. future work should focus on improving our training methods using hard negatives (e.g., functionally dissimilar but lexically similar samples) and inves-tigating our model in downstream applications.