On the Relationship between Sentence Analogy Identification and Sentence Structure Encoding in Large Language Models (2024)

Abstract: The ability of Large Language Models (LLMs) to encode syntactic and semantic structures of language is well examined in NLP. Additionally, analogy identification, in the form of word analogies are extensively studied in the last decade of language modeling literature. In this work we specifically look at how LLMs’ abilities to capture sentence analogies (sentences that convey analogous meaning to each other) vary with LLMs’ abilities to encode syntactic and semantic structures of sentences. Through our analysis, we find that LLMs’ ability to identify sentence analogies is positively correlated with their ability to encode syntactic and semantic structures of sentences. Specifically, we find that the LLMs which capture syntactic structures better, also have higher abilities in identifying sentence analogies.

## introduction analogies facilitate the transfer of meaning and knowledge from one domain to another. making and identifying analogies is a central tenet in human cognition @xcite and is aided by humans' ability to process the structure of language. in the domain of nlp, several types of textual analogies are identified, such as word analogies @xcite @xcite , proportional word analogies @xcite @xcite , sentenceanalogies @xcite @xcite and more recently analogies of procedural/long text @xcite . this work explicitly looks at sentencelevel analogies which are sentence pairs that are analogues in meaning to each other 1 . in this work, we apply this process to a dataset of 100k sentences. the dataset is divided into 0.8 for training the structure probe and 0.1 for testing. despite the existence of several established benchmarks (e.g., superglue @xcite and glue @xcite ) which evaluate the abilities of llms extrinsically, @xcite propose a more challenging intrinsic benchmark that focuses on llms' ability to identify analogies across a range of complexities. identification of analogies relies on the utilization of implicit relational knowledge embedded within the relational structure of language @xcite . in this work we aim to explore the relationship between sentence analogy identification abilities and syntactic and semantic structure encoding abilities of llms foot_0 . specifically, our main contribution is an analysis of the relationship between the analogy identification ability and sentence structure encoding abilities of llms. additionally, we extend the sentence structure probing techniques introduced by @xcite (which only supports bert and elmo) to further work with encoder-decoderbased llms and llms that use two transformer architectures. finally, we extend the structure probing technique originally used for syntactic structure probing in the novel context of semantic structure probing. ## related work assessing the ability of neural networks (nn) to encode syntactic and semantic structures of language is well examined in nlp @xcite @xcite . @xcite emphasize that the meaning of sentences is inferred by the hierarchical structures provided by syntactic and semantic properties of language. syntactic parsing aims to derive the syntactic dependencies in a sentence, such as subjects, objects, quantifiers, determiners and other similar elements. early probing tasks @xcite tried to identify nns' abilities to capture syntactic structures by classifying sentences with single and plural subjects. later, @xcite showed that nns could capture the maximal parse tree depth. the structure probing technique used and extended in this work @xcite is related but distinct due to its ability to implicitly capture the parse tree structures through simple distance measures between the vector representations of the words. compared to syntactic parsing, the nlp communities' interest in semantic parsing is growing. semantic parsing maps natural language sentences to a complete, formal meaning representation. semantic parsing is achieved via combining the semantic role labelling (srl) approaches with syntactic dependency parsing @xcite and more recently via semantic dependency parsing @xcite . this work uses the semantic dependency parsing approach based on mean field variational inference (mfvi) augmented with character and lemma level embeddings introduced by @xcite . ## approach our approach to exploring the relationship between analogy identification and sentence structure encoding in llms is detailed in the following three subsections. we explain the dataset used, in section 3.1, the analogy identification abilities of llms in section 3.2 and the sentence structure encoding abilities of llms in section 3.3. ## experimental setup exploring the relationship between analogy identification and sentence structure encoding abilities of llms requires a representative score to quantify (i) analogy identification ability (analogyscore), (ii) semantic structure identification ability (sem-score), and (iii) syntactic structure identification ability (syntscore) of each llm. we obtain analogyscore by calculating the means of reported md measures obtained for each sentence-level dataset in @xcite . to obtain the semscore (see figure 1 ), we first parse all the sentences in our dataset using the mfvi approach @xcite . the resulting semantically parsed sentences (in conll-u format) 4 and the llm embeddings of the original sentences are then sent to the structure probing technique @xcite . the structure probe is trained on 80k sentences from the dataset and the dspr and uuas values representing parse distance and root accuracy (rootacc) value representing parse depth are reported on the test split with 10k sentences. finally, the semscore is computed as a combined score by taking the mean of the z-score normalizations of these three measures z dspr , z u u as , z rootacc (see to obtain the syntscore (see figure 1 ), we follow the same steps but parse the sentences syntactically. finally, we calculate the spearman's rank correlation (src) and kendall's rank correlation (krc) between analogyscore and syntscore, as well as analogyscore and semscore. ## results in this section, we look at the findings of this work with regard to semantic and syntactic structure encoding abilities and analogy identification abilities of llms. ## limitations several contemporary probing techniques, such as those outlined in @xcite and pimentel et al. ( 2020 ), have emerged subsequent to the methodology employed in the present investigation @xcite . nevertheless, in the context of our current study, we have only chosen to employ @xcite owing to its adaptable nature, which facilitates extension to various llms that are of particular interest to our current research. even though abstract meaning representation (amr) @xcite is a popular and widely used technique to parse sentences semantically, in current work, we use mfvi, a semantic parsing approach introduced by @xcite because of the limitations posed by the structure probing technique used @xcite . this technique requires the mapped llm embeddings and semantic dependency parsed sentences to be of the same length. however, as it is known, amrs abstract away from the syntactic idiosyncrasies of the language and overlook certain auxiliary words from the parse results, limiting its use in this work. the present study employs a semantic parsing technique reported to exhibit a high accuracy level of 94% @xcite . however, it is important to note that for the purposes of our investigation, we make the assumption that the semantically parsed sentences generated by this particular method are entirely accurate, thereby employing them as the gold standard data. it is worth mentioning that this choice may introduce some degree of bias into our examination of the semantic structure probing. ## conclusion and future directions this work explores the relationship between llms' ability to identify sentence analogies and encode sentence structures in their embeddings. through detailed experiments, we show that the sentence analogy identification ability of llms is positively correlated with their ability to encode syntactic and semantic structures of sentences. particularly, llms that better capture syntactic structures have a higher correlation to analogy identification. in summary this work explores how llms utilize the knowledge of semantic and syntactic structures of sentences to identify analogies. moving forward, we aim to explore the potential of extending the current approach to enhance explainability of llms within the broader domain of nlp.