Entity Tracking via Effective Use of Multi-Task Learning Model and Mention-guided Decoding (2023)

Abstract: Cross-task knowledge transfer via multi-task learning has recently made remarkable progress in general NLP tasks. However, entity tracking on the procedural text has not benefited from such knowledge transfer because of its distinct formulation, i.e., tracking the event flow while following structural constraints. State-of-the-art entity tracking approaches either design complicated model architectures or rely on task-specific pre-training to achieve good results. To this end, we propose MeeT, a Multi-task learning-enabled entity Tracking approach, which utilizes knowledge gained from general domain tasks to improve entity tracking. Specifically, MeeT first fine-tunes T5, a pre-trained multi-task learning model, with entity tracking-specialized QA formats, and then employs our customized decoding strategy to satisfy the structural constraints. MeeT achieves state-of-the-art performances on two popular entity tracking datasets, even though it does not require any task-specific architecture design or pre-training.

## introduction pre-trained language models have revolutionized the nlp field in recent years @xcite @xcite and also become more versatile with the novel encoderdecoder architecture @xcite , which allows them to handle different types of nlp tasks without further architectural changes. this versatility inherently facilitates cross-task knowledge transfer via multi-task learning @xcite , and thus helps push the boundary of many popular nlp tasks such as question answering @xcite and semantic parsing @xcite . however, entity tracking, which tracks the states and locations of an entity throughout the procedural procedural text: how is hydroelectricity generated? 1. water flows downwards thanks to gravity. 2. enters the dam at high pressure. 3. moving water spins the turbines in power plant. 4. … meet utilizes the multi-task learning in t5 to boost entity tracking performance, with a customized decoding strategy addressing the structural constraints in state prediction (e.g., "move" cannot happen after "destroy"). ## related work tracking the progression of an entity within procedural text, such as cooking recipes @xcite or scientific protocols @xcite @xcite , is challenging as it calls for a model to understand both superficial and intrinsic dynamics of the process. recent work on entity tracking can be divided into two lines. one focuses on designing task-specific fine-tuning architectures to ensure that the model makes stepgrounded predictions while following the structural constraints. for instance, rajaby faghihi and kordjamshidi (2021) introduce time-stamp embeddings into roberta @xcite to encode the index of the query step. @xcite frame entity tracking as a structured prediction problem and use a crf layer to promote global consistency under those structural constraints. in our case, we show that, with qa formulation, simply appending the index of the query step to the question and indexing the procedure produces step-specific predictions. moreover, we propose a customized offline crf-decoding strategy for structural constraints to compensate for the fact that it is hard to jointly train t5, our backbone lm, with a crf layer, like in previous methods. the other line of work focuses on domainspecific knowledge transfer @xcite @xcite . concretely, lemon @xcite achieves great performance by performing in-domain pretraining on 1 million procedural paragraphs. cgli @xcite shows that adding high-quality pseudo-labeled data (generated via self-training) during fine-tuning can also boost the model performance. in contrast, our work explores how entity tracking can benefit from out-of-domain knowledge via using off-the-shelf pre-trained multi-task learning models. ## method in this section, we present meet, a multi-task learning-enabled entity tracking approach. here, we first review the problem definition, and then lay out the details of meet. ## conclusion we presented meet, a t5-based entity tracking approach. this approach includes our newly proposed qa fine-tuning formats and a customized decoding strategy so that it can effectively encode the flow of events in the procedural text while following structural constraints. the state-of-the-art performances on two benchmark datasets demonstrate the effectiveness of meet, and further analyses verify that multi-task learning on out-of-domain tasks can be beneficial for entity tracking.