O pen PI - C : A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking (2023)

Abstract: Open-vocabulary state tracking is a more practical version of state tracking that aims to track state changes of entities throughout a process without restricting the state space and entity space. OpenPI (Tandon et al., 2020) is to date the only dataset annotated for open-vocabulary state tracking. However, we identify issues with the dataset quality and evaluation metric. For the dataset, we categorize 3 types of problems on the procedure level, step level and state change level respectively, and build a clean dataset OpenPI-C using multiple rounds of human judgment. For the evaluation metric, we propose a cluster-based metric to fix the original metric’s preference for repetition. Model-wise, we enhance the seq2seq generation baseline by reinstating two key properties for state tracking: temporal dependency and entity awareness. The state of the world after an action is inherently dependent on the previous state. We model this dependency through a dynamic memory bank and allow the model to attend to the memory slots during decoding. On the other hand, the state of the world is naturally a union of the states of involved entities. Since the entities are unknown in the open-vocabulary setting, we propose a two-stage model that refines the state change prediction conditioned on entities predicted from the first stage. Empirical results show the effectiveness of our proposed model, especially on the cleaned dataset and the cluster-based metric. The code and data are released at https://github.com/shirley-wu/openpi-c

## introduction state tracking is the task of predicting the states of the world after an action is performed. most existing work operate under a simplified closevocabulary setting, assuming the state space and involved entities are known (dalvi et al., 2018; * equal contribution @xcite , which limits their applicability. the more practical open-vocabulary setting assumes both the entities and the state space are unknown. the openpi dataset @xcite is, to our knowledge, the first and only dataset for this task. however, we find a series of issues concerning data quality and evaluation, which may hinder progress in this line of research. we identify three types of issues with the dataset: non-procedural documents, out-of-order steps, and ambiguous state changes. in particular, ∼32% of the state changes cannot be reliably inferred from the input, which we find encourages model hallucination. we filter out problematic data points and build a cleaner dataset via crowdsourcing. for evaluation, the greedy matching strategy employed by @xcite allows matching multiple predicted state changes to a single gold state change, inadvertently inflating the score when the model produces repetitive outputs. we propose a cluster-based metric that automatically merges repetitive stage changes and enforces 1-to-1 assignment between clusters. we propose two enhancements to the seq2seq generation model proposed for this task in @xcite . to capture the dependency between world states of consecutive time steps, we introduce an entity memory to preserve information about the world state for all previous steps. when predicting the state changes for subsequent actions, the model can access the state information of previous time steps. additionally, while close-vocabulary setting usually provides a list of involved entities to track, such a list is inaccessible in open-vocabulary setting. this requires the model to jointly identify involved entities and predict their state changes. to make the problem more tractable and help model learning, we propose an entity-conditioned prediction step where predictions are conditioned on each single entity extracted from the predictions of the first stage. our contributions can be summarized as follows: (1) we present a clean dataset openpi-c for openvocabulary state tracking which fixes the data quality issues in the original openpi dataset; (2) we design a clustering-based metric for state tracking evaluation that mitigates the original metric's preference for repetition; (3) we model temporal dependency and entity awareness by enhancing the generation model for open-vocabulary state tracking with a dynamic memory module and two-stage prediction. ## related work most existing work on entity state tracking @xcite @xcite is closed-vocabulary, assuming that the number of possible states and involved entities is limited and known. under this setting, state tracking can be modeled as a tagging problem @xcite @xcite which is not applicable for the open-vocabulary case. tandon and chatterjee (2022) proposed openpi dataset for the more practical open-vocabulary setting. they formulate the task as a generation problem to handle the open vocabulary challenge. the design of an external memory component has already been applied to close-vocabulary state tracking @xcite @xcite . however, they rely on known entities and only track a limited set of attributes. in this work, we use a dynamic memory that can handle emerging entities with open-vocabulary attributes. ## task and dataset the openpi dataset @xcite ) is, to our knowledge, the first and only dataset for openvocabulary state tracking. the texts are collected from wikihow and the state changes are manually annotated. ## method generation baseline as shown in figure 1 , the input to the model is the concatenation of the goal, steps, and a prompt "now, what happens?". in tandon et al. ( 2020 ), each state change will be represented as a templated sequence for generation. for example, (potato, shape, whole, cut in half) will be converted to "shape of potato was whole before and cut in half afterwards". ## experiments our experiments are based on pre-trained bart @xcite ). 1 we add another baseline that that concatenates all previous state changes to the input (denoted as "bart + concat states"). the main results are in to analyze the effect of dataset cleaning, we compare the outputs of models trained on the original dataset and cleaned dataset. as in figure 4 , the cleaned dataset encourages the model to stick to the input text and produce less hallucination. to quantify this effect, we manually examined 50 processes randomly sampled from the test set. of the 50 processes we examined, each process consists of multiple steps, and each step has multiple output state changes. we did a binary classification on each output state change to classify whether it contains hallucinations or not. overall, the model trained on openpi produced 749 hallucinated state changes while the model trained on openpi-c produced 393 (47.53% less). ## conclusion and future work in this paper we study the open vocabulary state tracking problem. we build upon the generation formulation introduced by @xcite and propose two techniques: (1) entity memory that models the temporal dependency by storing world states from previous steps, and (2) entityconditioned prediction that simplifies the task by predicting state changes conditioned on each single entity. we conduct human annotation to address data quality issues in the existing openpi dataset and thus propose a cleaned version of openpi dataset. we propose an improved cluster-based metric to overcome the original metric's preference towards repetition. for future work, we consider using external resources such as conceptnet @xcite to assist entity prediction.