Unveiling Divergent Inductive Biases of LLM s on Temporal Data (2024)

Abstract: Unraveling the intricate details of events in natural language necessitates a subtle understanding of temporal dynamics. Despite the adeptness of Large Language Models (LLMs) in discerning patterns and relationships from data, their inherent comprehension of temporal dynamics remains a formidable challenge. This research meticulously explores these intrinsic challenges within LLMs, with a specific emphasis on evaluating the performance of GPT-3.5 and GPT-4 models in the analysis of temporal data. Employing two distinct prompt types, namely Question Answering (QA) format and Textual Entailment (TE) format, our analysis probes into both implicit and explicit events. The findings underscore noteworthy trends, revealing disparities in the performance of GPT-3.5 and GPT-4. Notably, biases toward specific temporal relationships come to light, with GPT-3.5 demonstrating a preference for “AFTER” in the QA format for both implicit and explicit events, while GPT-4 leans towards “BEFORE”. Furthermore, a consistent pattern surfaces wherein GPT-3.5 tends towards “TRUE”, and GPT-4 exhibits a preference for “FALSE” in the TE format for both implicit and explicit events. This persistent discrepancy between GPT-3.5 and GPT-4 in handling temporal data highlights the intricate nature of inductive bias in LLMs, suggesting that the evolution of these models may not merely mitigate bias but may introduce new layers of complexity.

## introduction temporal relations play a crucial role across diverse applications, including event summarization @xcite , predicting future events @xcite , and medical information processing @xcite . despite their importance, llms, especially those with limited context windows, face challenges in accurately sequencing events due to intricate temporal dependencies. efforts have been devoted to developing methodologies for effective temporal relation extraction @xcite @xcite @xcite , along with initiatives to create benchmark datasets with a temporal focus @xcite @xcite @xcite . however, discerning causal relationships between events adds complexity and can lead to misunderstandings. this complexity is strengthened by the absence of explicit temporal reasoning mechanisms, introducing biases in models' predictions and preferences for specific temporal relations. surprisingly, a notable gap exists in research exploring inductive bias in llms when discerning temporal relations. our study investigates the temporal comprehension abilities of gpt-3.5 and gpt-4 (openai, 2023), aiming to understand their grasp of temporal relationships. despite frequent model updates, significant biases were unveiled. using question answering (qa) and textual entailment (te) prompts, we queried both models to determine temporal relations. illustrated in figure 1 , the results expose variations in gpt-3.5 and gpt-4 performance, revealing biases towards specific temporal relationships. gpt-3.5 favors "after" in qa for implicit and explicit events, while gpt-4 leans towards "before." in te, gpt-3.5 tends towards "true," and gpt-4 prefers "false" for both implicit and explicit events. ## methodology our analysis involved two types of temporal data: one focusing on implicit events, actions, or situations not directly articulated in the text but inferred from context, while the other centered on explicit events explicitly mentioned in the context. furthermore, we delved into two prompt formats to gauge their influence on aiding llm in generating responses. these formats comprised the qa format, where questions prompt the model, and the te format, tailored to assess the logical relationships within sentences specifying temporal relations. question answering format. we initially conducted experiments using the qa format, focusing on explicit events. in this configuration, we tasked both models with determining the temporal relation ("before" or "after") between two provided events within the given context. the same approach was applied to implicit events. figure 2 provides the template and examples illustrating this format. textual entailment format. subsequently, we employed the textual entailment format as the next prompt type. in this format, we presented the model with a context along with a sentence declaring the temporal relation between two events, and then tasked the model with assessing its truthfulness. for every pair, there exists one true and one false label, as one corresponds to the gold label, and the other represents an incorrect label.examples illustrating this format are provided in figure 2 . inductive bias measurement: in our evaluation, we focused on probing the model's inductive biases related to temporal relations. to quantify the inductive bias, we examined the model's preference for "before" and "after" relations in the qa format and assessed its tendencies toward "true" and "false" in the textual entailment format. ## experimental settings dataset for our experimentation, we employed datasets such as timebank @xcite , tempeval @xcite , aquaint, and tracie @xcite . the analysis of implicit and explicit events was conducted separately. we extracted "be-fore"/"ibefore" as "before" and "after," /"iafter" as "after" from timebank, events from task c in tempeval featuring "before" or "after" relations. this yielded 1576 explicit events from timebank, tempeval, and aquaint datasets, comprising 815 "after" and 761 "be-fore" events. the dataset was duplicated for the entailment format, creating inverse relations with gold as "true" and inverse as "false," expanding the dataset to 3150 events with 1575 "true" and "false" values. for implicit events, the tracie dataset @xcite was used. transforming "starts after/ends after" into "after" and "starts before/ends before" into "before",the dataset included a total of 22,050 events evenly distributed between "true" and "false" labels, representing gold and inverse relations. among the 11,025 gold relations, 4,659 were identified as "after," while 6,366 were classified as "before". large language models. the gpt series, renowned as the leading range of large language models (llms), holds widespread popularity. our analysis began with these models due to their extensive usage, leaving the investigation of biases in other llms for potential future research. we conducted our analysis using openai's gpt-3.5 and gpt-4 models, specifically employing the latest stable versions: gpt-3.5-turbo-1106 and gpt-4-1106-preview. the gpt-3.5-turbo-1106 model has a default context window of 16k tokens, while gpt-4 features a context window of 128k tokens. ## conclusion and future work our study identified performance disparities between gpt-3.5 and gpt-4, with the latter showing more consistency. notably, biases were observed, as gpt-3.5 favored "after", while gpt-4 favored "before", and gpt-3.5 tended towards "true", while gpt-4 favored "false". this consistent yet contradictory pattern raises questions about whether new model releases might unintentionally introduce new biases. the observed biases across multiple datasets and prompt formats warrant a deeper exploration of the models' understanding of temporal data. future investigations should prioritize tasks involving temporal reasoning to address biases in gpt-3.5 and gpt-4, considering diverse datasets and prompt structures.