B e LLM : Backward Dependency Enhanced Large Language Model for Sentence Embeddings (2024)

Abstract: Sentence embeddings are crucial in measuring semantic similarity. Most recent studies employed large language models (LLMs) to learn sentence embeddings. Existing LLMs mainly adopted autoregressive architecture without explicit backward dependency modeling. Therefore, we examined the effects of backward dependencies in LLMs for semantic similarity measurements. Concretely, we propose a novel model: backward dependency enhanced large language model (BeLLM). It learns sentence embeddings via transforming specific attention layers from uni- to bi-directional. We extensively experiment across various semantic textual similarity (STS) tasks and downstream applications. BeLLM achieves state-of-the-art performance in varying scenarios. It shows that autoregressive LLMs benefit from backward dependencies for sentence embeddings.

## introduction sentence embedding is fundamental in natural language processing (nlp). it captures essential semantics in text, benefiting various semantic similarity measurement scenarios @xcite , such as semantic matching @xcite and clustering @xcite . in previous work, the primary efforts employed smaller-scale bi-directional models @xcite @xcite to extensively explore the context to learn sentence embeddings. however, in the paradigm revolution of llms and the increasingly large model scales, most advanced nlp models adopted autoregressive (decoder-only) architectures with forward dependency modeling only @xcite . while some recent efforts used llms for sentence embeddings @xcite the representative word for sentence "two dogs play by the shore." is:" the representative word for sentence "two dogs play in the snow." is:" ... two dogs play by the shore ... ## quantitative pilot analysis before introducing bellm, we conduct a pilot analysis to explore how advanced llms capture dependencies in contexts. it is driven by the observations that mainstream llms @xcite employ an autoregressive architecture in an uni-directional manner. intuitively, it lacks the ability to learn backward dependencies @xcite , resulting in potential inferiority to capture dependencies compared to its bi-directional alternatives. meanwhile, llms possess remarkable emergent abilities @xcite , implicitly benefit dependency capturing. we consequently conduct a pilot analysis to quantify whether explicit backward dependency modeling would benefit advanced autoregressive llms. our analysis is inspired by a phenomenon discovered by @xcite . the study qualitatively shows that bi-directional networks (e.g., elmo (peters et al., 2018) and bert @xcite )) demonstrate higher intra-sentence similarity (indicated by word similarities) than uni-directional networks (e.g., gpt-2). here, we further explore this finding from a quantitative perspective. concretely, we experiment on the sts-b test set @xcite and explore how models in uniand bi-directional architectures capture dependencies. for the bi-directional model, we adopt bert (base) and select the embedding of the "cls" token as the pivot token to represent the sentence embedding. for the uni-directional, autoregressive architecture, we employ two representative llms: llama @xcite ) (7b) and chatglm @xcite ) (6b). they offer two different implementations of autoregressive architectures. for them, we choose the last token as the pivot token following @xcite . based on pivot tokens, we compute their spearman correlation with the remaining tokens in a sentence to reflect the dependency-capturing capabilities. the results are shown in a box plot in figure 2 . the results indicate that bert shows a higher spearman correlation, implying its better capability to capture dependencies compared to llama and chatglm. interestingly, bert achieves an average score that is about twice as high as that of llama and chatglm. it is possibly attributed to bert's bi-directional architecture allowing both forward and backward dependency modeling. in contrast, autoregressive models focus on forward dependencies only. the results imply the potential benefits of adding backward dependency modeling to autoregressive models for sentence embeddings. ## bellm the above analysis has revealed the potential benefits of adding backward dependencies to autoregressive llms. consequently, this section will describe our proposed model, bellm, with bidirectional attention layers. figure 3 shows the overall framework of bellm. subsequent sections are organized as follows: firstly, we present a degradation experiment to examine how to add backward layers. then, we introduce the architectures of bellm, followed by the training methods. ## experimental results the following will first present the main results with intrinsic evaluation (section 5.1), followed by the extrinsic transfer task results in section 5.2. to provide more insight, we will then analyze the ablation study and case study results in section 5.3 and 5.4, respectively. at last, we further examine the intra-sentence dependency of bellm to supplement the pilot analysis in section 2. ## related work bellm is in line with sentence embeddings. in contrast to early attempts focusing on embeddings on the word level @xcite , sentence embeddings capture sentence-level semantics to understand context better. many previous studies adopted unsupervised approaches to utilize largescale unlabeled text. here, a popular method is to use bert-alike transformers @xcite to encode sentence embeddings. based on that, contrastive learning @xcite @xcite was further exploited to explore semantic similarities for sentence embeddings with self-supervision. to better align sentence embeddings to human senses, other prior studies @xcite employed labeled data for supervised sentence embedding learning. here, pretrained language models typically worked as the backbone architectures @xcite . following that, there is growing attention to the use of pre-trained llms for sentence embeddings @xcite . however, existing works employed commonly used autoregressive llms, neglecting the potential benefits of backward dependencies to sentence embeddings. viewing this gap, we extensively explore the effects of backward dependencies in llms for sentence embeddings. ## conclusion our work has pointed out the benefits of coupling forward and backward dependencies in llms for sentence embeddings. we have introduced bellm, a novel llm with backward dependency-enhanced sentence embedding learning. in extensive experiments, bellm has achieved state-of-the-art performance across varying sts and downstream tasks.