XCOMPS : A Multilingual Benchmark of Conceptual Minimal Pairs (2025)

Abstract: In this work, we introduce XCOMPS, a multilingual conceptual minimal pair dataset that covers 17 languages.Using this dataset, we evaluate LLMs’ multilingual conceptual understanding through metalinguistic prompting, direct probability measurement, and neurolinguistic probing. We find that: 1) LLMs exhibit weaker conceptual understanding for low-resource languages, and accuracy varies across languages despite being tested on the same concept sets. 2) LLMs excel at distinguishing concept-property pairs that are visibly different but exhibit a marked performance drop when negative pairs share subtle semantic similarities. 3) More morphologically complex languages yield lower concept understanding scores and require deeper layers for conceptual reasoning.

## introduction large language models (llms) have demonstrated remarkable capabilities across various natural language understanding (nlu) tasks. recent advances, such as gpt-4 @xcite and llama 3 @xcite , have shown that llms can produce human-like outputs and handle complex linguistic phenomena. however, whether llms genuinely understand semantics or merely rely on shallow statistical correlations is disputable @xcite @xcite . one fundamental aspect of human conceptual understanding is that it is not dependent on specific linguistic forms or modalities @xcite . when humans learn and reason about concepts, they do not require the knowledge to be tied to a particular medium, such as text, images, or video, nor do they rely on a specific language. this raises an important question: ## conclusion in this work, we introduce the xcomps benchmark, which provides a multilingual conceptual minimal pair dataset for evaluating the language model's semantic understanding across 17 languages. this work reveals that while llms demonstrate surface-level multilingual capabilities, they lack a universal semantic reasoning mechanism that transcends language boundaries.