Structure-aware Sentence Encoder in Bert-Based S iamese Network (2021)

Abstract: Recently, impressive performance on various natural language understanding tasks has been achieved by explicitly incorporating syntax and semantic information into pre-trained models, such as BERT and RoBERTa. However, this approach depends on problem-specific fine-tuning, and as widely noted, BERT-like models exhibit weak performance, and are inefficient, when applied to unsupervised similarity comparison tasks. Sentence-BERT (SBERT) has been proposed as a general-purpose sentence embedding method, suited to both similarity comparison and downstream tasks. In this work, we show that by incorporating structural information into SBERT, the resulting model outperforms SBERT and previous general sentence encoders on unsupervised semantic textual similarity (STS) datasets and transfer classification tasks.

## introduction pre-trained models like bert @xcite and roberta @xcite have demonstrated promising results across a variety of downstream nlp tasks. though bert-like models have been shown to capture hidden syntax structures @xcite @xcite , recent works have achieved performance improvements on various natural language understanding (nlu) tasks through the use of a graph network that captures syntax and semantics information. @xcite demonstrate the value of syntax information for pronoun resolution tasks, using relational graph convolutional networks (rgcns) @xcite to incorporate syntactic dependency graphs. @xcite argue that semantics has not been brought to the surface of pre-trained models and propose to introduce semantic label information into roberta via rgcns. similar ideas have been applied to information extraction @xcite , sentence-pair classification @xcite and sentiment analysis @xcite tasks. though problem-specific fine-tuning is required, these improvements suggest that structural supervision is useful, and that rgcns serve as an effective structure encoder. bert can also be used as a general sentence encoder, either by using the cls token (the first token of bert output) or applying pooling over its outputs. however, this fails to produce sentence embeddings that can be used effectively for similarity comparison. furthermore, this method of using bert for similarity comparison is extremely inefficient, requiring sentence pairs to be concatenated and passed to bert for every possible comparison. in response, sentence-bert (sbert) has been proposed to alleviate this by fine-tuning bert on natural language inference (nli) datasets using a siamese structure @xcite . general-purpose sentence embeddings are generated which outperform previous sentence encoders on both similarity comparison and transfer tasks. in this paper, we show that it is possible to improve the sbert sentence encoder through the use of explicit syntactic or semantic structure. inspired by sbert's success in producing general sentence representations and previous efforts on introducing structural information into pre-trained models, we propose a model that combines the two by training a bert-rgcn model in a siamese structure. under specific structural supervision, the proposed model is able to produce structureaware, general-purpose sentence embeddings. our empirical results show that it outperforms sbert and previous sentence encoders on unsupervised similarity comparison and transfer classification tasks. furthermore, we find that the produced sentence representation generalises better especially on fine-grained classification tasks. hidden syntax structures in pre-trained models have been well explored. various probing methods have been used to investigate hidden structures @xcite @xcite . the impact of external structures on pre-trained models has also been questioned. @xcite examined the benefits of incorporating universal dependencies into pre-trained models. @xcite showed that the tree induced from pre-trained models could produce competitive results compared with external trees. however, recent improvements have still been observed on various nlu tasks by incorporating structural information into pre-trained models. @xcite proposed sentibert to incorporate constituency tree into bert for sentiment analysis. @xcite modelled each sentence as a directed dependency graph by using rgcns, and achieved large improvements on pronoun resolution. @xcite proposed a semanticsaware bert model by further encoding semantic information with bert using a gru @xcite . rgcns have also been used by @xcite to introduce semantic information into roberta, and achieved consistent improvements when fine-tuned on problem-specific datasets. similar efforts can be seen where researchers try to provide syntax information via self-attention mechanism @xcite . ## model inspired by @xcite , we train our model in a siamese network to update weights so as to produce similarity-comparable sentence representations. the model we propose consists of two components, as shown in figure 1 . bert: each sentence is first fed into the pretrained bert-base model to produce both a sentence representation, by applying mean-pooling, and an original contextualised sequence-length token representation, which is used to initialise a rgcn. ## experiments we compare our model with sbert foot_1 , infersent foot_2 , use foot_3 , average glove vectors, and also two strategies using pre-trained bert to produce sentence representations (bert-cls and bert-avg). for all experiments on these models, we use released pre-trained models and scripts to produce sentence embeddings. ## conclusion in this work, we show that sbert can be improved by explicitly incorporating structural information. by using rgcns to incorporate syntactic structure into supervision, our model is able to produce structure-aware, general-purpose sentence embeddings that achieve improved results on both unsupervised similarity comparison and transfer classification tasks, when compared against previous sentence encoders. by extending probing classifiers, we further show that our syntax-informed supervision method is particularly beneficial for fine-grained tasks such as sst-5 and trec.