Do LLM s Recognize me, When I is not me: Assessment of LLM s Understanding of T urkish Indexical Pronouns in Indexical Shift Contexts (2024)

Abstract: Large language models (LLMs) have shown impressive capabilities in tasks such as machine translation, text summarization, question answering, and solving complex mathematical problems. However, their primary training on data-rich languages like English limits their performance in low-resource languages. This study addresses this gap by focusing on the Indexical Shift problem in Turkish. The Indexical Shift problem involves resolving pronouns in indexical shift contexts, a grammatical challenge not present in high-resource languages like English. We present the first study examining indexical shift in any language, releasing a Turkish dataset specifically designed for this purpose. Our Indexical Shift Dataset consists of 156 multiple-choice questions, each annotated with necessary linguistic details, to evaluate LLMs in a few-shot setting. We evaluate recent multilingual LLMs, including GPT-4, GPT-3.5, Cohere-AYA, Trendyol-LLM, and Turkcell-LLM, using this dataset. Our analysis reveals that even advanced models like GPT-4 struggle with the grammatical nuances of indexical shift in Turkish, achieving only moderate performance. These findings underscore the need for focused research on the grammatical challenges posed by low-resource languages. We released the dataset and code here.

## introduction large language models demonstrate remarkable capabilities in zero-shot and few-shot learning, excelling across a diverse range of tasks such as machine translation, text summarization, question answering, and solving complex mathematical problems @xcite @xcite . however, most large language models (llms) are primarily trained on data-rich languages like english, and their performance evaluations are also conducted in these languages (üstün * equal contribution et al., 2024). consequently, this focus on data-rich languages may lead to the under-exploration of challenges unique to low-resource languages. recent studies have evaluated the performance of large language models on linguistic tasks such as coreference resolution to examine their ability to match expressions referring to the same entity @xcite @xcite @xcite . in this study, we investigate llms' performance on interpreting indexical pronouns, with a focus on the indexical shift problem, a unique linguistic challenge related to but distinct from pronoun resolution, primarily encountered in lowresource languages like turkish @xcite , amharic @xcite , zazaki @xcite , uyghur @xcite , nez perce @xcite and japanese @xcite . indexical elements like i and here refer to the referents of the speech context such as the speaker or location of utterance. in most languages, these elements must be interpreted within the actual speech context, referring to the actual speaker or location of utterance. however, indexical shift occurs in some languages, like turkish, where an indexical element can refer to the referents of the reported context, rather than the actual speech context (see section 2 for details). indexical elements are substantially different from pronouns regarding what antecedents they can refer to and what factors restrict their interpretations. for example, while pronouns are almost always ambiguous and can refer to a wide range of entities, first person indexical unambiguously refers to the speaker of the utterance. even in languages that allow indexical shift, the first person indexical is ambiguous between only two possible referents (the speaker vs the attitude holder), being interpreted based on contex/world knowledge. moreover, turkish allows indexical shift only in some syntactic structures (finite embedded clauses) but not in others (e.g. nominalized embedded clauses), which makes indexical shift a unique challenge, requiring attention to specific syntactic rules and context (see section 3 for a detailed comparison of indexical elements and pronouns regarding coreference resolution). we investigate the capability of multilingual large language models to handle pronoun resolution within the context of indexical shift in turkish. to the best of our knowledge, this is the first study examining indexical shift in any language. therefore, we have released a turkish dataset specifically designed to evaluate llms on the indexical shift problem. our contributions in this work are as follows: in this regard, sentences like (2) are ambiguous between readings where first person indexical ben is shifted (referring to the attitude holder burak) or non-shifted (referring to actual speaker), and turkish speakers interpret such sentences based on previous context or upcoming sentences @xcite . for example, in a context where the actual speaker is the conversation topic, (2) would naturally be interpreted in the non-shifted reading (@xmath0, but if the conversation is about burak, the sentence would be naturally interpreted in the 1 one exception for this generalization is direct quotation (e.g. peter said/thought, 'i went to atlanta'.), where quoted material is interpreted as verbatim utterance/thought produced by its owner. direct quotation is out of the scope of this paper. burak burak yine again [ (ben-im) 1sg-gen mezun graduate ol-dug-um-u be-nmlz-1sg-acc ] san-ıyor. think-prog 'burak thinks again that {*he/speaker} graduated.' different from (2), the first person indexical ben in (3) can only refer to the actual speaker of the sentence (similar to english), regardless of the context it is uttered in (e.g. it cannot undergo indexical shift and refer to the attitude holder burak). this contrast between (2) and ( 3 ) is due to the syntactic/grammatical properties of the finite and nominalized embedded clauses in turkish, which are acquired by the native speakers of the language (see @xcite and @xcite for syntactic details). in this study, we aim to test whether llms are able to capture this grammatical contrast between two embedded clause types and successfully interpret indexical elements in syntactic environments that allow (e.g. finite embedded clauses) or block indexical shift (e.g. nominalized embedded clauses). ## related work there are substantial differences between pronouns and indexical elements. to begin with, even though syntactic/semantic factors can influence pronoun resolution by making some nouns more likely antecedents of pronouns (e.g. subject bias), they do not totally rule out other nouns as possible referents. 4 for example, previous work suggests that speakers mostly interpret the third person pronoun he in (4) as referring to the subject john (for syntactic or contextual reasons), but the object bill is still a possible antecedent, meaning that the sentence is ambiguous (e.g. @xcite @xcite . moreover, pronouns can refer to nouns that are contextually salient, but not present in the sentence. for example, the third person pronoun he can be interpreted as referring to a contextually salient person named peter. as a result, context plays a crucial role in how speakers interpret pronouns. previous work in the field (cited above) show that llms are able to use contextual information during coreference resolution and show good performance. indexical elements, on the other hand, must unambiguously refer to the discourse coordinates (e.g. speaker), except for indexical shifting environments. in syntactic contexts where indexical shift is allowed (e.g. turkish finite embedded clauses), indexical elements are similar to pronouns in that their referent can be ambiguous. however, indexical elements are still different from pronouns in that they are ambiguous between only two referents (speaker vs attitude holder), while pronouns are technically free to refer to an unlimited amount of antecedents (that can be salient in context). in summary, indexical elements are restricted by different syntactic factors than pronouns (e.g. clause type) and are usually unambiguous. moreover, even in contexts where indexical shift is possible, indexical elements are restricted to two possible referents, depending on whether indexical shift takes place or not, while pronouns are free to refer to a wide range of entities. thus, indexical elements and indexical shift create a unique challenge for llms, requiring to take into account the syntactic constraints regarding indexical shift while also 4 turkish indexical shift dataset to test llms' ability to understand indexical shift in turkish, we created a dataset containing 156 entries with sentences containing the turkish first person indexical (silent/dropped) that could potentially refer to the speaker (non-shifted) or to the attitude holder of the clause (shifted). since the interpretation of indexical elements in turkish (shifted vs non-shifted) depend on the context they appear in, we created two contexts for each experimental sentence: one priming the shifted reading, the other priming the non-shifted reading of the first-person indexical. moreover, for each context, we created a version of the sentence with a nominalized embedded clause like (3) (rather than finite embedded clause like (2)), where indexical shift is not allowed by the grammar (e.g. the indexical must refer to the speaker even if the reported context priming otherwise). together, this four context-sentence pairs lead to four different classes as summarized in we used sentences with three different verbs that allow indexical shift in turkish: iste 'to want', san 'to think/believe', düşün 'to think'. these verbs trigger specific morphosyntactic requirements in turkish. for example, iste 'to want' requires a subjunctive marker on the embedded verb, while san 'to think/believe' and düşün 'to think' require regular tense morphology. also, düşün 'to think' requires the complementizer diye while the other two does not require/allow diye. each entry in the dataset have the following information: in order to augment our dataset with different name pairs for testing, speaker and namenull placeholders are used instead of the speaker name and the reported third party with name-acc, name-gen, name-dat, name-loc, name-com were used for the accusative, genitive, dative, locative, and comitative forms of the third person subject's name. the turkish language indexical shift test dataset is open sourced along with the associated source code, here. ## experiments in this section, we explain our experimental setup, results, and discussion of the results. ## conclusion in this study, we assess large language models (llms) on pronoun resolution tasks within indexical shift contexts, focusing specifically on a lowresource language, turkish. to facilitate this evaluation, we release a turkish indexical shift dataset comprising 156 samples. we test recent multilingual models on this dataset and find their performance lacking. additionally, we observe that none of the llms' decisions are influenced by grammatical nuances, such as finite versus nominalized clauses, which contrasts with the behavior of native speakers. our findings highlight the need for greater attention to the grammatical challenges of low-resource languages in the development and evaluation of llms.