Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events (2021)

Abstract: Tracking entities throughout a procedure described in a text is challenging due to the dynamic nature of the world described in the process. Firstly, we propose to formulate this task as a question answering problem. This enables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text understanding. Secondly, since the transformer-based language models cannot encode the flow of events by themselves, we propose a Time-Stamped Language Model (TSLM) to encode event information in LMs architecture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state-of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better results on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general.

## introduction a procedural text such as a recipe or an instruction usually describes the interaction between multiple entities and their attribute changes at each step of a process. for example, the photosynthesis procedure can contain steps such as 1. roots absorb water from soil; 2. the water flows to the leaf ; 3. light from the sun and co2 enter the leaf ; 4. the water, light, and co2 combine into a mixture; 5. mixture forms sugar. procedural text understanding is a machine reading comprehension task defined on procedural texts. answering questions such as "what is the location of the mixture at step 4", in the above example, requires tracking entities' interactions to predict their attributes at each step @xcite . this is quite challenging due to the dynamic nature of the entities' attributes in the context. transformer-based language models have shown promising results on multi-hop or single-hop question answering benchmarks such as hot-potqa @xcite , squad @xcite , and drop @xcite . however, it is hard to expect lms to understand the flow of events and pay attention to the time in the procedure (e.g., step 4) without extra modeling efforts. in recent research, different approaches are taken to address procedural reasoning based on language models using qa formulations. following the intuition that attributes of entities can be retrieved based on the current and previous steps, dynapro @xcite modifies the input to only contain those sentences in the input at each time. this will provide a different input to the model based on each question to help it detect changes after adding each step. kg-mrc @xcite also generates a dynamic knowledge graph at each step to answer the questions. however, this intuition is contradicted in some scenarios such as detecting inputs of the process. for instance, the answer to the question "where is light as step 0?" is "sun", even if it is not mentioned in the first sentence of the process. inputs are entities that are not created in the process. the architecture of the qa transformer-based lms is very similar to the traditional attention mechanism. other methods such as prolocal @xcite and proglobal @xcite have structured this task by finding the attention of each entity to the text at each step. to be sensitive to the changes at each step, prolocal manually changes the model's input by removing all steps except the one related to the question. proglobal computes attention to the whole context while adding a distance value. distance value is computed for each token based on its distance to the direct mention of the entity at each step. the current language models convey rich linguistic knowledge and can serve as a strong basis for solving various nlp tasks @xcite @xcite . that is why most of the state-of-the-art models on procedural reasoning are also built based on current language models @xcite . following the same idea, we investigate the challenges that current models are facing for dealing with procedural text and propose a new approach for feeding the procedural information into lms in a way that the lm-based qa models are aware of the taken steps and can answer the questions related to each specific step in the procedure. we propose the time-stamped language model (tslm model), which uses timestamp embedding to encode past, current, and future time of events as a part of the input to the model. tslm utilizes timestamp embedding to answer differently to the same question and context based on different steps of the process. as we do not change the portion of the input manually, our approach enables us to benefit from the pre-trained lms on other qa benchmarks by using their parameters to initialize our model and adapt their architecture by introducing a new embedding type. here, we use roberta @xcite as our baseline language model. we evaluate our model on two benchmarks, propara @xcite and npn-cooking @xcite . propara contains procedural paragraphs describing a series of events with detailed annotations of the entities along with their status and location. npn-cooking contains cooking recipes annotated with their ingredients and their changes after each step in criteria such as location, cleanliness, and temperature. tslm differs from previous research as its primary focus is on using pre-trained qa models and integrating the flow of events in the global representation of the text rather than manually changing the part of the input fed to the model at each step. tslm outperforms the state-of-the-art models in nearly all metrics of two different evaluations defined on the propara dataset. results show a 3.1% f1 score improvement and a 10.4% improvement in recall. tslm also achieves the stateof-the-art result on the location accuracy on the npn-cooking location change prediction task by a margin of 1.55%. in summary, our contribution is as follows: ## problem definition an example of a procedural text is shown in the procedural text understanding task is defined as follows. given a procedure p containing a list of n sentences @xmath0, ...s n }, an entity e and a time step t i , we find l, the location of that entity and specify the status s of that entity. status s is one value in the predefined set of {non-existence, unknown-location, known-location}. location l is a span of text in the procedure that is specified with its beginning and end token. we formulate the task as finding function f that maps each triplet of entity, procedure and time step to a pair of entity location and status: (s, l) = f (e, p, t i ) to predict the status and the location of entities at each step, we model f with a question answering setting. for each entity e, we form the input q e as participants paragraph state number water light co2 mixture sugar (before the process starts) state 0 soil sun ? --roots absorb water from soil state 1 root sun ? --the water flows to the leaf state 2 leaf sun ? --light from the sun and co2 enter the leaf state 3 leaf leaf leaf --the water, light, and co2 combine into a mixture state 4 ---leaf -mixture forms sugar state 5 ----leaf where is e? [sep] although q e is not a step-dependent representation and does not incorporate any different information for each step, our mapping function needs to generate different answers for the question "where is entity e?" based on each step of the procedure. for instance, consider the example in to model this, we create pairs of (q e , t i ) for each @xmath1, 1, ..., n}. for each pair, q e is timestamped according to t i using t imestamp(.) function described in sec. 3.2 and mapped to an updated step-dependent representation, q t i @xmath2, t i ). the updated input representation is fed to a language model (here roberta) to obtain the stepdependent entity representation, r t i e , as shown in equation 2. we discuss the special case of @xmath3. 3.2. we use the step-dependent entity representation, r t i e , and forward it to another mapping function g(.) to obtain the location and status of the entity e in the output. in particular the output includes the following three vectors, a vector representing the predictions of entity status s, another vector for each token's probability of being the start of the location span l, and a third vector carrying the probability of each word being the last token of the location span. the outputs of the model are computed according to the equation 3. ## discussion we provide more samples to support our hypothesis in solving the procedural reasoning task and answer some of the main questions about the ideas presented in tslm model. why is the whole context important? the main intuition behind tslm is that the whole context, not just previous information, matters in reasoning over a process. here, we provide some samples from propara to show why this intuition is correct. consider this partial paragraph, "step i: with enough time the pressure builds up greatly. step i + 1: the resulting volcano may explode.". looking at the annotated status and location, the "volcano" is being created at step i without even being mentioned in that step. this is only detectable if we look at the next step saying "the resulting volcano...". as another example, consider this partial paragraph: "step i: dead plants form layers called peat. ... step i + 3: pressure squeezes water out of the peat.". the annotation indicates that the location of "water" is being changed to "peat" at step i, which is only possible to detect if the model is aware of the following steps indicating that the water comes out of the peat. positional embedding vs time-stamp encoding: as mentioned before the whole context (future and past events) is essential for procedural reasoning at a specific step. however, the reasoning should focus on one step at a time, given the whole context. while positional encoding encodes the order of information at the token-level for reasoning over the entire text, we need another level of encoding to specify the steps' positions (bound-aries) and, more importantly, to indicate the step that the model should focus on when answering a question. advantages/disadvantages of tslm model: tslm integrates higher-level information into the token representations. this higher-level information can come from event-sequence (time of events), sentence-level, or any other higher source than the token-level information. the first advantage of tslm is that it enables designing a model which is aware of the whole context, while previous methods had to customize the input at each step to only contain the information of earlier steps. furthermore, using tslm enables us to use pretrained qa models on other datasets without requiring us to retrain them with the added time-stamped encoding. one main disadvantage of tslm model, which is natural due to the larger context setting in this model, is not being sensitive to local changes, which is consistent with the observation in the comparison between proglobal and prolocal models. ## related works scone @xcite , npn-cooking @xcite , babi @xcite , process-bank @xcite , and propara @xcite are benchmarks proposed to evaluate models on procedural text understanding. processbank @xcite ) contains procedural paragraphs mainly concentrated on extracting arguments and relations for the events rather than tracking the states of entities. scone @xcite aims to handle co-reference in a procedural text expressed about a simulated environment. babi @xcite is a simpler machine-generated textual dataset containing multiple procedural tasks such as motion tracking, which has encouraged the community to develop neural network models supporting explicit modeling of memories @xcite and gated recurrent models @xcite . npn-cooking @xcite contains recipes annotated with the state changes of ingredients on criteria such as location, temperature, and composition. propara @xcite provides procedural paragraphs and detailed annotations of entity locations and the status of their existence at each step of a process. inspired by propara and npn-cooking benchmarks, recent research has focused on tracking entities in a procedural text. query reduction net-works (qrn) @xcite performs gated propagation of a hidden state vector at each step. neural process network (npn) @xcite computes the state changes at each step by looking at the predicted actions and involved entities. prolocal @xcite predicts locations and status changes locally based on each sentence and then globally propagates the predictions using a persistence rule. proglobal @xcite predicts the status changes and locations over the whole paragraph using distance values at each step and predicts current status based on current representation and the predictions of the previous step. prostruct @xcite aims to integrate manually extracted rules or knowledgebase information on verbnet @xcite as constraints to inject common-sense into the model. kg-mrc @xcite uses a dynamic knowledge graph of entities over time and predicts locations with spans of the text by utilizing reading comprehension models. ncet @xcite updates entities representation based on each sentence and connects sentences together with an lstm. to ensure the consistency of predictions, ncet uses a neural crf over the changing entity representations. xpad @xcite is also proposed to make dependency graphs on the propara dataset to explain the dependencies of events over time. most recently, dynapro @xcite feeds an incremental input to pretrained lms' question answering architecture to predict entity status and transitions jointly. tslm differs from recent research, as we propose a simple, straightforward, and effective technique to make our model benefit from pre-trained lms on general mrc tasks and yet enhance their ability to operate on procedural text understanding. we explicitly inject past, current, and future timestamps into the language models input and implicitly train the model to understand the events' flow rather than manually feeding different portions of the context at each step. procedural reasoning has also been pursued within the multi-modality domain @xcite @xcite which has additional challenges of aligning the representation spaces of different modalities. ## conclusion we proposed the time-stamped language model (tslm model), a novel approach based on a simple and effective idea, which enables pre-trained qa models to process procedural texts and produce different outputs based on each step to track entities and their changes. tslm utilizes a timestamp function that causes the attention modules in the transformer-based lm architecture to incorporate past, current, and future information by computing a timestamp embedding for each input token. our experiments show a 3.1% improvement on the f1 score and a 10.4% improvement over the recall metric on propara dataset. our model further outperforms the state-of-the-art models with a 1.55% margin in the npn-cooking dataset accuracy for the location prediction task. as a future direction, it is worth investigating how common-sense knowledge can be integrated with the tslm setting by augmenting the process context using external sources of related domain knowledge. we also intend to investigate the effectiveness of our approach on similar tasks on other domains and benchmarks. as another future direction, it can be effective to apply an inference algorithm to impose the global consistency constraints over joint predictions in procedural reasoning instead of using naive post-processing rules.