On the Transferability of Causal Knowledge for Language Models (2025)

Abstract: Language understanding includes identifying logical connections between events in a discourse, such as news and instructional text. We study the transferability of causal knowledge across these two domains by analyzing the extent to which understanding preconditions in narratives such as news articles can help models reason about cooking recipes, and vice-versa. Our experiments show that using instructions to pretrain small models on one domain before similarly finetuning it on the other shows a slight improvement over just finetuning it. We also find that finetuning the models on a mix of both types of data is better (~3-7%) for understanding causal relations in instructional text. While we find that the improvements do not translate to larger or already instruction tuned models, our analysis highlights the aspects of a plan that are better captured through the interoperability of causal knowledge.

## introduction understanding underlying causal relationships is an important component of understanding narratives such as news articles. these causal relationships often show up as implicit preconditions and effects of the described events or actions. preconditions provide a form of logical connection between events that explains why they occur together. they include background information and provide the glue to reason about chains of events common in narratives. preconditions also form the base for reasoning about other forms of text. instructional texts such as how-to procedures often contain prerequisites and details about world states. recognizing the causal elements in a story aids in identifying prerequisites in instructional text, while grasping procedural preconditions can enhance one's ability to track news events. humans use a shared framework to comprehend preconditions and other causal relations regardless of the type of text they are reading. in this paper, we aim to study whether understanding aspects of causal knowledge about narratives can help models better understand instructional text and vice versa. we use peko @xcite , a dataset of annotated preconditions between event pairs in news articles, and cat-bench @xcite , a benchmark testing step order prediction in cooking recipes. first, we establish the performance of different t5 @xcite and flant5 @xcite models by finetuning them on each dataset individually. next, we study how much understanding causal relations within one domain helps understand those in the other. this is done through causal pretraining, i.e., pretraining models on the first domain, finetuning on the second as well as evaluating on it. finally, we study whether models are able to capture different types of causal knowledge when trained on a data mix from both domains. our experiments show that learning various types of causal relations impacts models differently. base models benefit from training over such knowledge in different domains while larger models already contain it through their pretraining. our analysis finds that causal pretraining and multi-task finetuning help understand long range relations in plans and cases where two steps in the plan are not dependendent on each other, and highlights areas to better use different types of causal knowledge together. ## related work there is a vast body of research on extracting different types of relations between events including temporal @xcite , causal @xcite , paraphrasal @xcite , and precondition relationships @xcite . atomic @xcite is a crowd-sourced dataset of event-event relations, where given a simple target event (verb phrase and its arguments), crowd workers provided various types of commonsense knowledge. the rich event description (red) dataset (o'gorman et al., 2016) was created to model a broad set of event-event relations in news. caters @xcite contains data similar to preconditions captured through just one causal relation but focuses on 5 sentence short stories and only contains ∼400 data points. eventstoryline @xcite is also small in size and further does not explicitly capture preconditions. the precondition knowledge (peko) dataset @xcite ) contains largescale crowdsourced annotations about precondition relations between event pairs in news stories. understanding instructional text involves multiple aspects such as tracking entity states @xcite , linking actions @xcite @xcite , next event prediction @xcite @xcite and more. @xcite formalize several multiplechoice tasks related to step-and goal-relations in procedures. @xcite explore predicting dependencies in cooking recipes and related tasks. similar work has been done on identifying dependencies in multimodal instructions with images and text @xcite . @xcite clearly studies the prediction and explanation of temporal ordering constraints on the steps of an instructional plan. humans have the ability to utilize knowledge from previous experiences when learning a new task. prior work has explored techniques of transfer learning and domain adaptation to learn skills in various contexts. @xcite ; @xcite explored using parallel data from high resource languages to improve translation in low resource languages. @xcite ; @xcite use domain adaptation techniques for models to learn new tasks. similar to these, we investigate whether understanding causal knowledge in one domain helps with another. ## data to study the transferability of causal knowledge, we use cat-bench and peko, which contain information about dependencies between a plan's steps and preconditions about events respectively. @xcite is a dataset of causal dependency questions defined on cooking recipes to evaluate the causal and temporal reasoning abilities of models over instructional plans. specifically, it focuses on the ability to recognize temporal dependencies between steps i.e., deciding if one step must happen before or after another. for a recipe in the dataset, containing an ordered number of steps, the dataset contains either of two binary questions: (1) must step i happen before step j ? and (2) must step j happen after step i ? we pool questions from dependent pairs of steps into dep, and the rest into nondep foot_0 . peko is a dataset consisting of crowdsourced annotations of preconditions between event pairs in news articles. @xcite first subsample events and their temporal relations using caevo @xcite from the new york times annotated corpus @xcite . the resultant set was then filtered to retain only pairs of events that have a "before" or "after" temporal relation between them. these were further sampled and given to annotators who evaluated whether or not the candidate precondition event was an actual precondition for the target event resulting in 30k annotations. ## experiment details we provide critical information about the models and training regimes we use for our experiments. ## results particularly, comparing models of the same size, t5 and flant5 are better on this binary classification task than the reported bert model even though they are generative models. ft models for cat-bench show improved performance over any of the reported baselines, which is expected as the baselines are only zero-and few-shot settings. we observe mixed results when using the causal pretraining regime (cp). it is clear that first learning about preconditions from news events helps t5-base understand cause and effect relations implicitly encoded within the steps of a plan. we hypothesize that larger models already encode such knowledge in their parameters and such pretraining does not affect downstream performance. these findings also hold when first learning about plans followed by news events. clearly, transferring causal knowledge between generic news events and highly specific actions in a plan lead to improved reasoning across both. multi-task training (mft) over both datasets together improves t5-base performance over finetuning (ft) regardless of the target task. in fact, there is a large improvement (∼7%) on cat-bench in this regime, and a small improvement in understanding news events in peko. however, while the opposite is true for t5-large, the drop is negligible. this training paradigm does not impact flant5 performance on peko but mixing causal information in news with that in plans leads to slight decrease in understanding the latter. overall, we find that the training regime heavily impacts a model's performance on a causal understanding task. simply following one regime will not lead to improvements across all tasks, and it is important to explore the different options. ## analysis having established the differences in training regimes across different settings, we investigate the abilities t5-base on cat-bench to better understand our results.