Un-considering Contextual Information: Assessing LLM s’ Understanding of Indexical Elements (2025)

Abstract: Large Language Models (LLMs) have demonstrated impressive performances in tasks related to coreference resolution. However, previous studies mostly assessed LLM performance on coreference resolution with nouns and third person pronouns. This study evaluates LLM performance on coreference resolution with indexical like I, you, here and tomorrow which come with unique challenges due to their linguistic properties. We present the first study examining how LLMs interpret indexicals in English, releasing the English Indexical Dataset with 1600 multiple-choice questions. We evaluate pioneering LLMs, including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and DeepSeek V3. Our results reveal that LLMs exhibit an impressive performance with some indexicals (I), while struggling with others (you, here, tomorrow), and that syntactic cues (e.g. quotation) contribute to LLM performance with some indexicals, while they reduce performance with others. Code and data are available at: https://github.com/metehanoguzz/LLMs-Indexicals-English

## introduction large language models (llms) have demonstrated remarkable capabilities in zero-shot and few-shot learning, excelling across a wide array of tasks such as machine translation, text summarization, and question answering @xcite @xcite . their versatility has led to widespread applications in diverse domains, including education, law, and medicine. as the use of llms continues to expand, understanding their underlying behaviors has become increasingly important. recent studies have evaluated the performance of large language models on linguistic tasks such as coreference resolution @xcite @xcite @xcite . previous work on coreference resolution mostly focused on how coreference is established between two third person entities such as proper names (e.g. andy, the mechanic) and third person pronouns (e.g. he, him, himself ) in english and other languages (e.g. @xcite . in this study, we investigate how llms establish coreference with indexical elements (e.g. i, you, here), which differ from third person nouns/pronouns in substantial ways and bring unique challenges for llms (see figure 1 as an example). we investigate how state-of-the-art llms interpret the indexical elements i, you, here and tomorrow in english sentences, and whether context or grammatical constraints influence their decisions. to the best of our knowledge, this is the first study examining llms' handling of indexical elements in english. our key contributions are as follows: indexicals are interpreted inside the context of utterance, referring to the actual speech-act coordinates like the author or the location of the actual utterance. as a result, if (1-b) is uttered by john, the indexical i can only be interpreted as referring to john as the speaker, leading to a reading like 'andrew said that john went to buckhead'. crucially, even though andrew's speech/claim is reported in (1-b), the indexical i cannot refer to andrew. direct quotation is an exception to this generalization, where reported material is interpreted as verbatim utterance/thoughts of its owner. thus, when indexicals appear inside direct quotation, they are interpreted inside the reported context, rather than the actual context of utterance. in other words, direct quotation 'shifts' the interpretations of indexicals into the reported context. for example, i and here in (2) appear inside direct quotation, where andrew's speech is reported. regardless of who utters (2), the sentence means that andrew was born in atlanta, so both i and here are 'shifted' into the reported context, where andrew is the speaker and atlanta is the location foot_0 indexicals differ from other pronominals in substantial ways. first, though syntactic and semantic factors can affect how a pronoun is interpreted (e.g. subject bias), pronouns are typically ambiguous regarding what/who they refer to. for example, the third person pronoun he in (3) is most naturally interpreted as referring to the subject john (for syntactic or contextual reasons), but the object bill is still a possible antecedent, causing ambiguity between two different readings (e.g. @xcite @xcite . in addition, he can refer to any contextually salient person that is not mentioned in the sentence (e.g. peter), which makes pronouns even more ambiguous and context-dependent. as a result, semantic/contextual information plays a crucial role in how pronouns are interpreted, and speakers use those cues to establish coreference with pronouns. for instance, if (4) is uttered in a context where john is a supportive and humble coworker, the most natural interpretation is that john suggests that bill should get promoted (@xmath0. however, if john is arrogant and jealous, the most natural interpretation is that john suggests that john should get promoted (@xmath1. (4) john told bill that he should get promoted. indexicals, on the other hand, unambiguously refer to the referents of the speech-act coordinates. for example, i in (5) refers to the speaker regardless of what we know about john or bill. i refers to the speaker even if john and/or bill are arrogant and jealous, so contextual information like this should be disregarded while interpreteting indexicals. (5) john told bill that i should get a promotion. in summary, indexicals are restricted by different syntactic factors than pronouns (e.g. quotation vs non-quotation) and are typically unambiguous, while pronouns are free to refer to a wide range of entities. thus, indexical elements create a unique challenge for llms, requiring to 'disregard' semantic/contextual cues that might prime interpretations through other antecedents (unlike pronouns). 3 experimental design 3.1 dataset curation we curated a dataset specifically designed to test how llms interpret indexical elements like i, you, here, and tomorrow in different contexts of utterance. specifically, we assess how these mod-els interpret indexicals in 'shifted' context prime, where context more naturally requires the indexical should be interpreted inside reported context (e.g. peter is one of the most arrogant students in my classroom. ... peter says that i am smart.) vs 'nonshifted' context prime, where context more naturally requires the indexical should be interpreted inside actual speech context (e.g. peter is very kind and supportive. ... peter says that i am smart.). we also included direct quotations in both contexts (e.g., peter says, "i am smart") to examine if llms can successfully consider syntactic factors (quotation vs regular sentences) while ignoring misleading information from the context during coreference resolution with indexical elements. for each type of indexical, we design 100 sentences and for each sentence we apply the four different transformations explained above. overall, we have 400 samples per indexical, compromising a total dataset of size 1600. we utilize gpt-4o to curate the dataset, by giving a detailed description of the task along with some in-context examples. then it is asked to generate scenarios with a stimulus sentence in two different contexts (see appendix a.1 for the prompts used), along with specific questions addressing the referent of the indexical in each sentence. to ensure the quality of the dataset, 25% of the dataset (400 @xmath2, and the evaluation process consisted of three steps. in the first step, we confirmed that all sentences were grammatically correct. in the second step, we confirmed that all quotation condition sentences had quoted embedded clauses, and all nonquotation condition sentences had regular (nonquoted) embedded clauses. we also made sure that the two sentence conditions were maximally similar, except for the quotation vs non-quotation status (i.e. the only difference between two sentence conditions was the quotation). in the third step, we checked the context prime texts for each condition in each sentence, making sure that the correct readings (shifted vs non-shifted) were primed by the context description. for example, for an item condition where here was supposed to be shifted, we confirmed that the context description would be most naturally followed by a sentence where here would be shifted. to eliminate potential gender bias, each dataset sample exclusively uses either male or female names, alternating to ensure a balanced distribution with 50% of the samples containing female names and 50% male names. this method promotes gender neutrality across the dataset. sample details are in appendix a.2, and the complete dataset is available in the supplementary materials. ## experimental results we present the experimental results in figure 2 and we discuss them in detail for each indexical type: indexical i. the results show that all four models perform near optimum in sentences without quotation with an average accuracy of 99% (always correctly selecting the non-shifted option). for the quoted sentences gpt-4o and gemini 1.5 pro again perform almost optimum with a mean accuracy larger than 94%, followed by claude 3.5 sonnet with 89% mean accuracy (correctly selecting the especially, the model performance drops significantly to 17% when context primes non-shifted readings, suggesting that inclusion of quotation makes the model more sensitive to the linguistically irrelevant effects of context prime. moreover, considering that the model performance only reaches 78% accuracy in quotation conditions seems to imply that the model might have a bias towards the non-shifted reading overall, reducing model performance in quotation conditions. ## related work coreference resolution has been extensively studied in prior research @xcite @xcite @xcite . however, indexical elements exhibit distinct syntactic properties compared to other (non-indexical) pronominals, as discussed in section 2. previous work by @xcite explored how llms interpret indexicals in turkish, where indexicals possess different grammatical properties than in english and can shift without quotation. to the best of our knowledge, this study is the first to evaluate the performance of llms in interpreting indexicals in english. ## discussion and conclusion our results show that llm performances are not uniform across different types of indexical elements and sentence types. while most llms perform at impressive levels interpreting the indexical i, their performances drop significantly in other indexicals, particularly in here and tomorrow. moreover, though sentence type (quotation vs non-quotation) affects llms performances in general, the effects are not similar across different indexical types and models. while quotation increases llms' performance with here, it decreases their performance with you and tomorrow. in addition, tomorrow seems to be affected by quotation in a greater magnitude than you. in conclusion, we find that different types of indexicals show unique patterns regarding how they are interpreted by the llms. our results diverge from those reported in @xcite , who tested how first person indexical in turkish ben 'i' was interpreted by different llms, including gpt-4o and show that llms exhibit very poor performance interpreting ben 'i'. here, we report that llms perform at an almost human-like level with interpreting the english indexical i. this might be due to lower amounts of available resources to train the models in turkish, compared to english. another reason for low performance in turkish could be due to the fact that turkish is a pro-drop language, meaning that the subject pronouns can be dropped (silent). @xcite used sentences where the first person indexical ben 'i' was dropped, which might have made the task more challenging for the llms considering that dropped indexicals can show different properties than overt ones @xcite . this linguistic difference between turkish and english might have caused different results between the turkish and english tests.