Probing Internal Representations of Multi-Word Verbs in Large Language Models (2025)

Abstract: This study investigates the internal representations of verb-particle combinations, called multi-word verbs, within transformer-based large language models (LLMs), specifically examining how these models capture lexical and syntactic properties at different neural network layers. Using the BERT architecture, we analyze the representations of its layers for two different verb-particle constructions: phrasal verbs like “give up” and prepositional verbs like “look at”. Our methodology includes training probing classifiers on the model output to classify these categories at both word and sentence levels. The results indicate that the model’s middle layers achieve the highest classification accuracies. To further analyze the nature of these distinctions, we conduct a data separability test using the Generalized Discrimination Value (GDV). While GDV results show weak linear separability between the two verb types, probing classifiers still achieve high accuracy, suggesting that representations of these linguistic categories may be “non-linearly separable”. This aligns with previous research indicating that linguistic distinctions in neural networks are not always encoded in a linearly separable manner. These findings computationally support usage-based claims on the representation of verb-particle constructions and highlight the complex interaction between neural network architectures and linguistic structures.

## introduction 1.the linguistic problem multi-word verbs or verb-particle combinations are a linguistic category presented in the english language in which the lexical verb is combined with a particle to form an independent unit. it is called a phrasal verb when the lexical verb is combined with an adverbial particle like work out. it is a prepositional verb when the verb is combined * correspondence: patrick.krauss@fau.de with a prepositional particle like rely on @xcite . usually, the prepositional verbs are followed by a noun phrase. rather than the nature of the following particle, there are several differences between phrasal verbs and prepositional verbs. one main difference between the two categories is the particle placement in phrasal verbs and the foxed order in prepositional verbs. where in phrasal verbs, the particle can sometimes be separated from the verb and placed after the object. in contrast, the only grammatical form in prepositional verbs is the v+prepostion+object. ## discussion our findings indicate that while probing classifiers and gdv provide some investigations into how bert encodes differences between linguistic categories, they may not fully capture the complexities of linguistic representations. particularly, when distinguishing between phrasal and prepositional verbs based on token-level embeddings. as proposed by @xcite , lexico-semantic elements convey a portion of linguistic information, but they do not embody all structural and functional aspects present in a text. this limitation is particularly relevant in the case of the investigated cases in the study, where distinctions often emerge from interactions between lexical, syntactic, and semantic factors rather than being determined by individual token representations. our study focuses on bert-base-uncased as a widely used transformer model, but we acknowl-edge that different llm architectures may encode linguistic categories differently. future research could extend this analysis to other models, such as roberta-base or bert-large, to assess whether the observed patterns generalize across architectures. this perspective aligns with the constructionist approach to language processing (madabushi et al., 2020), which challenges the traditional separation of lexical and grammatical elements. instead, it proposes a continuum of constructions-where linguistic representations arise from learned pairings of form and meaning rather than being strictly lexical or grammatical only. from this point, phrasal and prepositional verbs might be better understood as integrated constructions, rather than purely lexical or syntactic units. consequently, probing classifiers, which primarily capture lexical or semantic properties in token-based classification tasks, may fail to fully account for the grammatical and contextual information that shapes the representation of these constructions. this is evident in the mismatch between classification accuracies and gdv values, suggesting that different methods may capture other dimensions of representation. several studies have discussed the limitations of probing classifiers @xcite . one major limitation is the disconnect between probing accuracy and the original model's internal processing. while probing classifiers can detect correlations between model embeddings and linguistic features, they do not necessarily indicate whether the model actively uses these properties for linguistic processing. this limitation is apparent in our findings: while probing classifiers achieved high accuracies, gdv analysis showed weak linear separability between phrasal and prepositional verbs, as indicated by the lack of significant correlation between classification accuracies and gdv values. this observed disagreement between classifier accuracies and gdv values aligns with previous research suggesting that internal representations in neural networks and large language models are not necessarily linearly separable @xcite @xcite @xcite @xcite . since gdv measures linear separability, it does not capture non-linearly structured representations. in contrast, probing classifiers can still detect nonlinearly separable distinctions, allowing them to identify linguistic categories that may be encoded in high-dimensional space. therefore, the low gdv scores do not suggest that bert fails to encode multi-word verb distinctions, but rather that these representations may require non-linear transformations to be fully distinguished. this point up the need for comprehensive analytical methods when investigating how llms structure linguistic knowledge and suggests that linear separability should not be the only one criterion for assessing learned representations.