How Well Do Text Embedding Models Understand Syntax? (2023)

Abstract: Text embedding models have significantly contributed to advancements in natural language processing by adeptly capturing semantic properties of textual data. However, the ability of these models to generalize across a wide range of syntactic contexts remains under-explored. In this paper, we first develop an evaluation set, named SR, to scrutinize the capability for syntax understanding of text embedding models from two crucial syntactic aspects: Structural heuristics, and Relational understanding among concepts, as revealed by the performance gaps in previous studies. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges, and such ineffectiveness becomes even more apparent when evaluated against existing benchmark datasets. Furthermore, we conduct rigorous analysis to unearth factors that lead to such limitations and examine why previous evaluations fail to detect such ineffectiveness. Lastly, we propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios. This study serves to highlight the hurdles associated with syntactic generalization and provides pragmatic guidance for boosting model performance across varied syntactic contexts.

## introduction text embedding plays a significant role in a variety of natural language processing applications including language understanding @xcite , information retrieval @xcite and question answering @xcite . recently, many methods @xcite @xcite @xcite , targeting at converting textual data into vector representations, have demonstrated a remarkable performance across massive text embedding benchmarks @xcite . however, as we delve deeper into different hierarchies of language comprehension, the question that naturally arises is: how well do these text embedding models understand syntax? for example, could the state-of-the-art text embedding models understand and distinguish the difference between "cats chase mice" and "mice chase cats"? syntax, constituted by a set of rules that define sentence structures, forms a pivotal aspect of natural language. it integrates both heuristics and compositional elements, establishing the bedrock for the expansive and intricate nature of human language @xcite . a thorough comprehension of syntax is essential for a text embedding model to effectively ascertain the relationships among words, thereby facilitating a level of language understanding that mirrors human cognitive processes. moreover, as text embedding models are increasingly deployed in llm-based agents and real-world applications, ensuring that they maintain a solid understanding of syntax is critical to guarantee their reliability and efficacy. in this study, we address this question by introducing a new evaluation set called sr designated to probe the ability of text embedding models from three syntactic aspects @xcite @xcite @xcite ): 1) structural heuristics: the rules and patterns that govern sentence structures, and 2) relational understanding among concepts: the models' capability to infer relationships between different concepts in text. these dimensions represent the multifaceted nature of syntax. we take this inquiry a step further by conducting an incisive analysis to uncover the underpinnings of these limitations. by examining various models and their responses to syntactic challenges, we delineate the factors that contribute to the manifestation of these limitations and the reasons why they have eluded detection in conventional benchmarks. recognizing these challenges is a precursor to addressing them. finally, we show that simply augmenting the training data with sr-like examples, which can be generated through chatgpt with designed prompts, can significantly enhance the generalization capabilities of text embedding models in syntactically diverse settings. this research is aimed at shedding light on the challenges of syntactic generalization in text embedding models. by establishing a more rigorous evaluation set and proposing strategies for enhancement, we contribute to the advancement of text embedding models that are capable of nuanced understanding and high performance across varied syntactic contexts. our work provides valuable guidance and sets the stage for future research aimed at achieving more syntactically aware and robust text embedding models. sr benchmark and code are released foot_0 . ## sr benchmark typical evaluation sets have often been inadequate in rigorously assessing a model's understanding of syntax. they tend to focus on high-level performance metrics, while overlooking the finer aspects of syntactic understanding. sr was designed to address the limitations and specifically evaluate text embedding models from two important syntactic aspects: structural heuristics, and relational understanding. these dimensions were chosen due to their fundamental roles in natural language understanding. in this section, we introduce the construction process of the sr benchmark. ## evaluating test embedding models on sr in this section, we employ the sr benchmark to evaluate the syntactic understanding capabilities of five state-of-the-art text embedding models, including sentencebert @xcite , simcse @xcite , sentence-t5 @xcite , one embedder @xcite and openai embedding @xcite . detailed information on these models can be found in their original papers. following previous works, we employ spearman's correlation as the evaluation metric to assess how well the relationship between the cosine similarities of the sentence pairs and the annotated scores. we present the evaluation results on the sr benchmark in interestingly, we observe that, despite being trained solely on natural language inference datasets, simcse outperforms sbert, instructor and openai embedding models. these competing models often have more diverse supervised pairwise training datasets or a greater number of parameters, or both. for instance, simcse achieves performance with an average spearman's correlation of 44.16% on the benchmark for assessing relational understanding, while sbert, instructor and openai embedding models achieve an average spearman's correlation of 29.67%, 38.74% and 31.45%, respectively. this could indicate that training on natural language inference tasks may provide valuable syntactic understanding that isn't captured through mere parameter scale or breadth of training data. sentence-t5, which is trained on a large scale of web-based question-answering datasets, demonstrates most robust performance. this might suggest that the diversity and complexity found in web-based question-answering data could be beneficial for embedding models in capturing syntactic aspects. we hypothesize that the web-based question-answering data, often encompassing a wide range of topics, styles, and structures, are likely to present a richer and more varied set of syntactic compositions compared to other data types. we use tree kernel @xcite as a measure of sentence structure diversity. the central idea of tree kernel is to count the number of common subtrees between two constituency pars. we compare the syntactic diversity of two corpora: webqa (233k sentences selected) and nli (275k sentences selected). we use stan-fordcorenlp to obtain constituency parse trees of sentences. then, we randomly select 1,000 parse trees and use the tree kernel to calculate their similarity. the results are shown in the score, the more diverse the sentence structures are. as we can see, sentences in webqa have lower tree kernel similarity than those in nli, indicating that webqa has more diverse sentence patterns and structures. this diversity could expose the model to a more comprehensive spectrum of syntactic elements, instilling a deeper sense of syntax into the embedding models. since prevailing text embedding models like sbert, simcse, etc. are mainly trained on nli, we argue that sentence-t5, which is trained on webqa, can capture more syntactic nuances of sentences. additionally, we found that these models generally struggle in the cqadupstack domain but show relatively better performance in the askubuntu domain. this discrepancy may be due to the specific challenges and complexities associated with the cqadupstack domain compared to the askubuntu domain. moreover, a noteworthy trend across all the text embedding models we assessed was the consistent pattern of comparatively better performance on the benchmark for relational understanding, as opposed to benchmarks for structural heuristics. this suggests that the current text embedding models may be more adept at capturing relations among concepts in sentences than they are at grappling with the finer points of syntax, such as sentence structures. this superior performance in capturing relations could be a result of the training data, which often tends to focus on semantic relationships. however, it might also indicate that the models have an innate aptitude for discerning semantic relations as compared to understanding complex syntactic structures. ## related work text embeddings text representation learning is a fundamental task in natural language processing. in recent literature, contrastive learning @xcite @xcite has emerged as the dominant paradigm for training text embedding @xcite @xcite @xcite @xcite . these approachese typically involves the use of large pairwise pretraining datasets with rich compositional structures, wherein the model is optimized to distinguish between similar and dissimilar text samples. contrastive pretraining is geared towards optimizing text embedding matching. as a result, most text embedding models @xcite @xcite are evaluated on the basis of their performance in semantic similarity matching tasks @xcite . however, these tasks not sufficiently encompass the complexity of syntax in natural language, which involve not only the arrangement of words but also their relationships and compositional semantics. syntax probing in nlp syntactic probing seeks to understand to what extent the learned representations in nlp models capture syntactic information and how such information can be effectively extracted and analyzed. there have been numerous studies focused on this topic, each offering unique insights and methods for syntactic generalization. @xcite @xcite @xcite @xcite . for example, @xcite assess the syntactic generalization of modern sentence encoders through a series of selected downstream tasks. their approach is more geared towards understanding what properties are encoded in the sentence embeddings and how these embeddings can be used for various nlp tasks. on the other hand, @xcite developed the hans dataset to specifically examine the performance of natural language inference (nli) models, particularly focusing on whether these models are adopting superficial syntactic heuristics over a deeper semantic understanding. their work is especially relevant in the context of nli tasks and aims to discern the methods that models use to arrive at decisions. in contrast, our work aims to construct a benchmark for directly evaluating the syntactic capabilities of text embedding models, without being confined to a specific task like nli. the primary goal is to facilitate the selection of robust and effective text embedding models for training in various applications by providing a direct measure of their syntactic understanding. this benchmark thus serves as an essential tool for researchers and practitioners who are looking to employ text embedding models that are both syntactically sound and effective in real-world applications. ## conclusion this paper highlights the shortcomings of current text embedding models in understanding syntax. we introduced the sr benchmark to analyze models across two syntactic dimensions: structural heuristics and relational understanding. our findings reveal that while these models are adept at semantic tasks, they struggle with syntax. we proposed a data augmentation technique using examples tailored for syntactic understanding, leading to notable performance gains on the sr benchmark. however, a slight performance dip was observed on the original test set. future work could refine augmentation strategies to balance syntactic and semantic learning and create more holistic evaluation benchmarks.