C O C O :
C O NTROLLABLE
C O UNTERFACTUALS FOR E VALUATING D IALOGUE S TATE T
RACKERS
Anonymous authors
Paper under double-blind review
A BSTRACT
Dialogue state trackers have made signiﬁcant progress on benchmark datasets, but their generalization capability to novel and realistic scenarios beyond the heldout conversations is less understood.
We propose controllable counterfactuals (C O C O ) to bridge this gap and evaluate dialogue state tracking (DST) models on novel scenarios, i.e., would the system successfully tackle the request if the user responded differently but still consistently with the dialogue ﬂow?
C O C O leverages turn-level belief states as counterfactual conditionals to produce novel conversation scenarios in two steps: (i) counterfactual goal generation at turnlevel by dropping and adding slots followed by replacing slot values, (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue ﬂow.
Evaluating state-of-the-art DST models on MultiWOZ dataset with C O C O -generated counterfactuals results in a signiﬁcant performance drop of up to 30.8% (from 49.4% to 18.6%) in absolute joint goal accuracy. In comparison, widely used techniques like paraphrasing only affect the accuracy by at most 2%.
Human evaluations show that COCO-generated conversations perfectly reﬂect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations, further strengthening its reliability and promise to be adopted as part of the robustness evaluation of DST models.
I NTRODUCTION
Task-oriented dialogue (TOD) systems have recently attracted growing attention and achieved substantial progress (Zhang et al., 2019b; Neelakantan et al., 2019; Peng et al., 2020; Wang et al., 2020b;a), partly made possible by the construction of large-scale datasets (Budzianowski et al., 2018; Byrne et al., 2019; Rastogi et al., 2019).
Dialogue state tracking (DST) is a backbone of TOD systems, where it is responsible for extracting the user’s goal represented as a set of slot-value pairs (e.g., ( area , center ), ( food , British )), as illustrated in the upper part of Figure 1.
The DST module’s output is treated as the summary of the user’s goal so far in the dialogue and directly consumed by the subsequent dialogue policy component to determine the system’s next action and response.
Hence, the accuracy of the DST module is critical to prevent downstream error propagation (Liu and Lane, 2018), affecting the success of the whole system.
With the advent of representation learning in NLP (Pennington et al., 2014; Devlin et al., 2019; Radford et al., 2019), the accuracy of DST models has increased from 15.8% (in 2018) to 55.7% (in 2020).
While measuring the held-out accuracy is often useful, practitioners consistently overestimate their model’s generalization (Ribeiro et al., 2020; Patel et al., 2008) since test data is usually collected in the same way as training data. In line with this hypothesis, Table 1 demonstrates that there is a substantial overlap of the slot values between training and evaluation sets of the MultiWOZ DST benchmark (Budzianowski et al., 2018).
In Table 2, we observe that the slot co-occurrence distributions for evaluation sets tightly align with that of train split, hinting towards the potential limitation of the held-out accuracy in reﬂecting the actual generalization capability of DST models.
Inspired by this phenomenon, we aim to address and provide insights into the following question:
how well do state-of-the-art DST models generalize to the novel but realistic scenarios that are not captured well enough by the held-out evaluation set?
Most prior work (Iyyer et al., 2018; Jin et al., 2019) focus on adversarial example generation for robustness evaluation.
They often rely on perturbations made directly on test examples in the held- [Table] [Table] out set and assume direct access to evaluated models’ gradients or outputs.
Adversarial examples generated by these methods are often unnatural or obtained to hurt target models deliberately.
It is imperative to emphasize here that both our primary goal and approach signiﬁcantly differ from the previous line of work: (i)
Our goal is to evaluate DST models beyond held-out accuracy, (ii) We leverage turn-level structured meaning representation (belief state) along with its dialogue history as conditions to generate user response without relying on the original user utterance, (iii) Our approach is entirely model-agnostic, assuming no access to evaluated DST models, (iv) Perhaps most importantly, we aim to produce novel but realistic and meaningful conversation scenarios rather than intentionally adversarial ones.
We propose controllable counterfactuals (C O C O ) as a principled, model-agnostic approach to generate novel scenarios beyond the held-out conversations.
Our approach is inspired by the combination of two natural questions: how would DST systems react to (1) unseen slot values and (2) rare but realistic slot combinations?
C O C
O ﬁrst encapsulates these two aspects under a uniﬁed concept called a counterfactual goal obtained by a stochastic policy of dropping and adding slots to the original turn-level belief state followed by replacing slot values.
In the second step, C O C O conditions on the dialogue history and the counterfactual goal to generate a counterfactual conversation .
We cast the actual utterance generation as a conditional language modeling objective.
This formulation allows us to plug-in a pretrained encoder-decoder architecture (Raffel et al., 2020) as the backbone that powers the counterfactual conversation generation.
We also propose a strategy to ﬁlter utterances that fail to reﬂect the counterfactual goal exactly.
We consider value substitution (VS), as presented in Figure 1, as a special C O C O case that only replaces the slot values in the original utterance without adding or dropping slots.
When we use VS as a fall-back strategy for C O C O (i.e., apply VS when C O C O fails to generate valid user responses after ﬁltering), we call it
C O C
O
+.
Evaluating three strong DST models (Wu et al., 2019;
Heck et al., 2020; Hosseini-Asl et al., 2020) with our proposed controllable counterfactuals generated by C O C O and C O C
O
+ shows that the performance of each signiﬁcantly drops (up to 30.8%) compared to their joint goal accuracy on the original MultiWOZ held-out evaluation set.
On the other hand, we ﬁnd that these models are, in fact, quite robust to paraphrasing with back-translation, where their performance drops only 2%.
Analyzing the effect of training data augmentation with C O C O + shows that it consistently improves the robustness of the investigated DST models on counterfactual conversations generated by each of VS, C O C
O and C O C
O
+.
More interestingly, the same data augmentation strategy improves the joint goal accuracy of the best of these strong DST models by 1.3% on the original MultiWOZ evaluation set.
Human evaluations show that C O C O -generated counterfactual conversations perfectly reﬂect the underlying user goal with more than 95% accuracy and are found to be quite close to original conversations in terms of their human-like scoring.
This further proves our proposed approach’s reliability and potential to be adopted as part of DST models’ robustness evaluation.
We plan to publicly release C O C O -generated and human-veriﬁed set of examples (MultiWOZ-C O C O ) as additional evaluation set to be used by future research.
R ELATED W ORK
Dialogue State Tracking. DST has been a core component in current state-of-the-art TOD systems.
Traditional approaches usually rely on hand-crafted features or domain-speciﬁc lexicon (Henderson [Table] [Figure] et al., 2014; Wen et al., 2017) and require a predeﬁned ontology, making them hard to extend to unseen value.
To tackle this issue, various methods have been proposed.
Gao et al. (2019) treats DST as a reading comprehension problem and predicts slot values with start and end positions in the dialogue context.
Zhang et al. (2019a) proposes DS-DST, a dual-strategy model that predicts values in domains with a few possible candidates from classiﬁers and others from span extractors.
Furthermore, Heck et al. (2020) proposes TripPy, a triple copy strategy model, which allows it to copy values from the context, previous turns’ predictions and system informs.
An alternative to classiﬁcation and span prediction is value generation.
Wu et al. (2019) generates slot values with a pointer generator network See et al. (2017) without relying on ﬁxed vocabularies and spans.
(Hosseini-Asl et al., 2020) models DST as a conditional generation problem and directly ﬁnetunes GPT2 (Radford et al., 2019) on DST task and achieves state-of-the-art on the MultiWOZ.
Adversarial Example Generation. Adversarial example generation has been commonly studied in computer vision (Szegedy et al., 2014; Goodfellow et al., 2015).
Recently, it has received growing attention in NLP domain as well.
Papernot et al.
(2016) ﬁnds adversarial examples in the embedding space, and then remapped them to the discrete space.
Alzantot et al.
(2018) proposes a populationbased word replacing method and aims to generate ﬂuent adversarial sentences.
These methods often edit the original data greedily assuming access to the model’s gradients or outputs besides querying the underlying model many times (Jin et al., 2019).
Alternative line of work investigates generating adversarial examples in a model-agnostic way.
Iyyer et al. (2018) proposes to generate adversarial paraphrases of original data with different syntactic structure.
Jia and Liang (2017) automatically generates sentences with key word overlappings of questions in SQuAD (Rajpurkar et al., 2016) to distract computer systems without changing the correct answer or misleading humans.
Although different methods have been proposed to evaluate the robustness of NLP models, majority of the prior work in this line focus either on text classiﬁcation, neural machine translation or reading comprehension problems.
Perhaps the most similar existing works with ours are (Einolghozati et al., 2019; Cheng et al., 2019).
Einolghozati et al.
(2019) focuses on intent classiﬁcation and slot tagging in TOD while Cheng et al.
(2019) targets at synthetic competitive negotiation dialogues (Lewis et al., 2017) without DST component.
In this work, however, we focus on evaluating a core component of state-of-the-art TOD, DST, on the widely used benchmark, MultiWOZ. To the best of our knowledge, ours is the ﬁrst work to systematically evaluate the robustness of DST models.
B ACKGROUND
Multi-domain DST task deﬁnition. Let X t
= { ( U sys 1 , U usr 1 ) , ..., ( U sys t , U usr t ) } denote a sequence of turns of a dialogue until the t -th turn, where U sys i and U usr i ( 1 ≤ i ≤ t ) denote system and user utterance at the i -th turn, respectively.
In multi-domain DST, each turn ( U sys i , U usr i ) talks about a speciﬁc domain (e.g., hotel ), and a certain number of slots (e.g., price range ) in that domain.
We denote all N possible domain-slot pairs as S =
{ S 1 , ...S N } .
The task is to track the value for each S j ( 1 ≤ j ≤ N ) over X t (e.g., hotel-price range , cheap ).
Belief states can be considered at two granularities: turn-level ( L t ) and dialog-level ( B t ). L t tracks the information introduced in the last turn while B t tracks the accumulated state from the ﬁrst turn to the last.
As illustrated in the upper part of Figure 1, when the dialogue ﬂow arrives at the second turn, B 2 becomes { ( restaurant-area , center ), ( restaurant-food , British ), ( restaurant-book time , 18:00 ) } , while L 2 is { ( restaurant-food , British ), ( restaurant-book time , 18:00 ) } , essentially tracking the update to B t by the last turn.
Problem deﬁnition. Given a tuple < X t , L t , B t > , our goal is to generate a new user utterance ˆ U usr t to form a novel conversation scenario ˆ X t
= { ( U sys 1 , U usr 1 ) , ..., ( U sys t , ˆ U usr t ) } by replacing the original user utterance U usr t with ˆ U usr t . To preserve the coherence of dialogue ﬂow, we cast the problem as generating an alternative user utterance ˆ U usr t conditioned on a modiﬁed ˆ L t derived from original turn-level belief state L t in a way that is consistent with the global belief state B t .
This formulation naturally allows for producing a new tuple < ˆ X t , ˆ L t , ˆ B t > controllable by ˆ L t , where ˆ B t is induced by B t based on the difference between L t and ˆ L t .
As illustrated in the lower part of Figure 1, U usr 2 is replaced with the two alternative utterances that are natural and coherent with the dialogue history.
We propose to use the resulting set of < ˆ X t , ˆ L t , ˆ B t > to probe the DST models.
Paraphrase baseline with back-translation. Paraphrasing the original utterance U usr t is a natural way to generate ˆ U usr t .
With the availability of advanced neural machine translation (NMT) models, round-trip translation between two languages (i.e., back-translation (BT)) has become a widely used method to obtain paraphrases for downstream applications (Yu et al., 2018).
We use publicly available pretrained English → German ( log( g | e ) ) and German → English ( log( e | g ) ) NMT models.
We translate U usr t from English to German with a beam size K , and then translate each of the K hypotheses back to English with the beam size K .
Consequently, we generate K 2 paraphrase candidates of ˆ U usr t and then rank them according to their round-trip conﬁdence score log( g | e ) + log( e | g ) .
As paraphrases are expected to preserve the meaning of U usr t , we set ˆ L t =
L t and ˆ B t = B t .
C O C O
As illustrated in Figure 2, C O C O consists of three main pillars.
We ﬁrst train a conditional user utterance generation model p θ ( U usr t | U sys t , L t ) using original dialogues. Secondly, we modify L t into a possibly arbitrary ˆ L t by our counterfactual goal generator.
Given ˆ L t and U sys t , we sample ˆ U usr t ∼ p θ ( ˆ U usr t | U sys t , ˆ L t ) with beam search followed by two orthogonal ﬁltering mechanisms to further eliminate user utterances that fail to reﬂect the counterfactual goal ˆ L t .
4. V ALUE S UBSTITUTION
A robust DST model should correctly reﬂect value changes in user utterances when tracking user’s goal.
However, slot-value combinations, e.g. ( restaurant-book time , 18:00 ), in evaluation sets are limited and even have signiﬁcant overlaps with training data as shown in Table 1. To evaluate DST models with more diverse patterns, we propose a Value Substitution (VS) method to generate ˆ U usr t .
Speciﬁcally, for each value of S j in L t , if the value only appears in U usr t rather than U sys t , we allow it to be substituted. Otherwise, we keep it as is. This heuristic is based on the following three observations: (1) if the value comes from U sys t , e.g. TOD system’s recommendation of restaurant food, changing it may make the dialogue ﬂow less natural and coherent (2) if it never appears in the dialogue ﬂow, e.g. yes of hotel-parking , changing it may cause belief state label errors (3) if it only appears in U usr t , it is expected that changing the value won’t cause issues in (1) and (2).
For values that can be substituted, new values are sampled from a Slot-Value Dictionary , a predeﬁned value set for each domain-slot. These new values are then used to update their counterparts in U usr t , L t and B t . We defer the details of slot-value dictionary to section 4.2.
After the update, we get ˆ U usr t , ˆ L t and ˆ B t , and can use < ˆ X t , ˆ L t , ˆ B t > to evaluate the performance of DST models. An example of how VS works is illustrated in the lower part of Figure 1. At the second turn, as British and 18: are in L 2 and only appear in U usr 2 rather than U sys 2 , we can replace them with Chinese and 17:00 that are sampled from a slot-value dictionary, respectively, to get ˆ U usr 2 , ˆ L 2 and ˆ X 2 without interrupting the naturalness of the dialogue ﬂow. [Table]
(cid:55)(cid:86)(cid:86)(cid:71)(cid:84)(cid:67)(cid:80)(cid:69)(cid:71)(cid:3)(cid:3)(cid:42)(cid:71)(cid:80)(cid:71)(cid:84)(cid:67)(cid:86)(cid:75)(cid:81)(cid:80)(cid:3)(cid:47)(cid:81)(cid:70)(cid:71)(cid:78)(cid:3)(cid:39)(cid:87)(cid:84)(cid:75)(cid:80)(cid:73)(cid:3)(cid:54)(cid:84)(cid:67)(cid:75)(cid:80)(cid:75)(cid:80)(cid:73)(cid:3)(cid:50)(cid:74)(cid:67)(cid:85)(cid:71)
[Table]
4. C ONTROLLABLE C OUNTERFACTUAL G ENERATION
Back-translation (BT) and value-substitution (VS) provide controllability at different granularities.
BT only provides syntactic variety while preserving the meaning, hence the belief state. VS can only replace the values of the existing slots in an utterance while still having to exactly retain all the slots. However, neither of them are able to explore conversations with even slightly modiﬁed set of slots.
We propose a principled approach to unlock the capability of conversation generation that generalizes beyond just transformation of existing utterances.
We cast it as a task of generating novel user utterances ( U usr t ) from a given conversation history ( U sys t ) and a turn-level user goal ( L t ).
We propose to tackle this problem with a conditional generation model that utilizes a pretrained encoder-decoder architecture (Raffel et al., 2020; Lewis et al., 2020) to approximate p ( U usr t | U sys t , L t ) , where the concatenation of U sys t and L t is used as input to the encoder and U usr t is set to be the target sequence to be generated by the decoder, as illustrated in the lower-left of Fig. To learn this distribution, we factorize it by chain rule (Bengio et al., 2003) and train a neural network with parameters θ to minimize the aggregated negative log-likelihood J gen over each dialogue turn tuple ( U sys t , L t , U usr t ) where U usr t = ( U usr t, 1 , U usr t, 2 , . . . , U usr t,n t ) and U usr t,k is its k -th token:
[Equation] Once the parameters θ of the goal-conditioned utterance generation model p θ are learned from these tuples, it gives us the unique ability to generate novel conversation turns by plugging in an arbitrary but consistent counterfactual goal ˆ L t derived from L t .
An example of how the counterfactual goal generator operates is shown in the middle part of Figure 2.
The counterfactual goal generator has three components, namely operation, slot-value dictionary and slot-combination dictionary.
Operation decides to apply which combination of the following three meta-operations, namely drop , change and add on L t .
Drop is used to remove values from a non-empty slot in L t .
Change borrows the same operation in VS, to substitute existing values.
Add allows us to add new domainslot values into L t , giving us the power of generating valid but more complicated ˆ U usr t .
Slot-Value Dictionary has a pre-deﬁned value set S val j for each S j .
Once change and/or add metaoperation is activated for S j , counterfactual goal generator will randomly sample a value from S val j .
Slot-Combination Dictionary has a predeﬁned domain-slot set S add j for each S j .
When add metaoperation is activated, counterfactual goal generator will sample a domain-slot from the intersection among all S addj , where S j has non-empty values within L t .
Once a new domains-lot is sampled, its value will then be sampled from its corresponding value set as deﬁned in slot-value dictionary.
Given L t , the counterfactual goal generator ﬁrst takes L t as its input, and sequentially applies drop , change and add to output ˆ L t .
Given ˆ L t and U sys t , we can sample ˆ U usr t ∼ p θ ( ˆ U usr t | U sys t , ˆ L t ) with beam search.
We use a rule-based method to get ˆ B t of ˆ X t following (Chao and Lane, 2019).
Given B t − 1 and ˆ L t , we update the domain-slot in B t − 1 if its value in ˆ L t is not none .
Otherwise, we keep its value as it is in B t − 1 .
After the update, we use ˆ B t as the dialogue-level label of ˆ X t .
4. F ILTERING
We have presented methods to generate ˆ U usr t , but how do we make sure that the generated utterance correctly reﬂects the user goal represented by ˆ L t ?
To motivate our methods, we take an example generated by beam search located at the lower right of Figure 2 for illustration.
In this example, the ﬁrst hypothesis doesn’t include value 2 for restaurant-book people that is within ˆ L t .
On the contrary, the second hypothesis includes a value 18:00 for restaurant-book time that is not part of ˆ L t .
We call these two phenomenons de-generation and over-generation , respectively.
Filtering candidates with these issues is thus an important step to make sure ( U sys t , ˆ U usr t ) perfectly expresses the user goals in ˆ L t .
We propose two ﬁltering methods, namely slot-value match ﬁlter and classiﬁer ﬁlter , to alleviate de-generation and over-generation issues, respectively.
Slot-Value Match Filter.
To tackle with de-generation issue, we choose a subset of values in ˆ L t (values that should only appear in ˆ U usr t rather than U sys t ) to eliminate candidates that fail to contain all the values in the subset.
In Fig.
2, the ﬁrst hypothesis from the beam search output will be eliminated by this ﬁlter because it does not include the value 2 for restaurant-book people in ˆ L t .
Classiﬁer Filter. As shown in Table 2, the slot restaurant-book people frequently appears together with restaurant-book time in the data we use to train our generation model p θ ( ˆ U usr t | U sys t , ˆ L t ) . The imbalanced training data may cause that during inference, although ˆ L t may not include restaurantbook time , the model may still generate a user utterance
ˆ U usr t containing information about this slot.
To deal with this over-generation problem, we propose to use a N-way multi-label classiﬁer to eliminate such candidates. We employ BERT-base (Devlin et al., 2019) as its backbone:
[Equation] where H CLS t ∈ R d emb is the representations of CLS token of BERT with dimension d emb .
We then feed H CLS t into a linear projection layer followed by Sigmoid function: [Equation]
where W ∈ R N × d emb is the trainable weight of the linear projection layer and P j is probability that slot S j appears at t -th turn of X t with Y j as its label. The classiﬁer is trained with J cls , i.e. the mean binary cross entropy loss of every slot S j and achieves a precision of 92.3% and a recall of 93.5% on the development set 4 . During inference, the classiﬁer takes ˆ X t as input and predicts whether a slot
S i appears at t -th turn or not with threshold 0.5. We use this ﬁlter to eliminate generated candidates for which the classiﬁer predicts at least one slot S j mentioned in ( U sys t , ˆ U usr t ) while S j / ∈ ˆ L t . In Fig.
2, our classiﬁer ﬁlter eliminates the second hypothesis from the output of beam search because ˆ L t does not contain the slot restaurant-book time while it is mentioned in the generated utterance.
E XPERIMENTS 5. E XPERIMENTAL S ETUP
We consider three strong multi-domain DST models to evaluate the effect of C O C O -generated counterfactual conversations in several scenarios.
T RADE (Wu et al., 2019) builds upon pointer gener-
[Figure] [Figure]
TRADE
TripPy ator network and contains a slot classiﬁcation gate and a state generator module to generate states. T RIP P Y (Heck et al., 2020) introduces a classiﬁcation gate and a triple copy module, allowing the model to copy values either from the conversation context or previous turns’ predictions or system informs.
S IMPLE T OD (Hosseini-Asl et al., 2020) models DST as a conditional generation problem with conversation history as its condition and belief state as its target and ﬁnetunes on GPT2.
Evaluation.
We train each of these three models following their publicly released implementations on the standard train/dev/test split of MultiWOZ 2.1 (Eric et al., 2019) from scratch.
We use the joint goal accuracy to evaluate the performance of DST models.
It measures the model accuracy by comparing the predicted belief state with the ground-truth, where the prediction is marked correct if and only if the set of ( domain-slot , value) pairs in the model output exactly matches the oracle one.
Slot-Value Dictionary.
We carefully design two sets of slot-value dictionaries to capture the effect of unseen slot values from two perspectives, namely in-domain ( I ) and out-of-domain ( O ).
I is a dictionary that maps each slot to a set of values that appear in MultiWOZ test set, but not in the training set.
On the other hand, we construct O using external values (e.g., hotel names from Wikipedia) that fall completely outside of the MultiWOZ data for the slots (e.g., hotel-name , restaurant-name , etc.).
Otherwise, we follow a similar fall-back strategy for slots (e.g., hotel-internet ) with no possible external values beyond the ones (e.g., yes and no ) in the original data.
Slot-Combination Dictionary.
As illustrated in Table 2, held-out evaluation set follows almost the same slot co-occurrence distribution with training data.
This makes it difﬁcult to estimate how well
DST models would generalize on the valid conversation scenarios that just do not obey the same distribution.
C O C
O ’s ﬂexibility at generating a conversation for an arbitrary turn-level belief state naturally allows us to seek an answer to this question.
To this end, we design three slot combination dictionaries, namely freq , neu and rare .
A slot combination dictionary directly controls how different slots can be combined while generating counterfactual goals.
As suggested by their names, freq contains frequently co-occurring slot combinations (e.g., book people is combined only with book day and book time slots), while rare is the opposite of freq grouping rarely co-occurring slots together, and neu is more neutral allowing any meaningful combination within the same domain.
5. M AIN RESULTS
Before we begin reporting our results, it is important to note that several different post-processing strategies (e.g., output cleaning, employing semantic dictionary mapping) are used by different DST models.
To make a fair comparison across different models, we follow the same post-processing strategy employed by S IMPLE T OD evaluation script for T RADE and T RIP P Y as well.
We summarize our main results in Figure 3.
While all three DST models are quite robust to back-translation (BT), their performance signiﬁcantly drop on counterfactual conversations generated by each of VS,
C O C O and C O C O + compared to MultiWOZ held-out set accuracy (original).
Unseen Slot-Value Generalization.
We analyze the effect of unseen slot values for the two dictionaries ( I and O ) introduced in the previous section compared to the original set of slot values that have large overlap with the training data.
Results presented on the left part of Figure 3 show that the performance of DST models signiﬁcantly drops up to 11.8% compared to original accuracy even on the simple counterfactuals generated by VS strategy using in-domain unseen slot-value dictionary (I).
Furthermore, using out-of-domain slot-value dictionary (O) results in about 10% additional drop in accuracy consistently across the three models.
Consistent and similar drop in accuracy suggests that T RADE , S IMPLE T OD , and T RIP P Y are almost equally susceptible to unseen slot values.
Generalization to Novel Scenarios.
The right section of Figure 3 presents the main results in our effort to answer the central question we posed at the beginning of this paper.
Based on these results, we see that state-of-the-art DST models are having a serious difﬁculty generalizing to novel scenarios generated by both C O C O and C O C O + using three different slot combination strategies.
The generalization difﬁculty become even more serious on counterfactuals generated by C O C O +.
As expected, the performance drop consistently increases as we start combining less and less frequently co-occurring slots (ranging from freq to rare ) while generating our counterfactual goals.
In particular, C O C O +(rare) counterfactuals drops the accuracy of T RADE from 49.4% to 18.6%, pushing its performance very close to its lower bound of 13.8%.
Even the performance of the most robust model (T RIP P Y ) among the three drops by up to 25.8%, concluding that held-out accuracy for state-of-the-art DST models may not sufﬁciently reﬂect their generalization capabilities.
Transferability Across Models.
As highlighted before, a signiﬁcant difference and advantage of our proposed approach lies in its model-agnostic nature, making it immediately applicable for evaluation of any DST model.
As can be inferred from Figure 3, the effect of C OCO -generated counterfactuals on the joint goal accuracy is quite consistent across all three DST models.
This result empirically proves the transferability of C O C O , strengthening its reliability and applicability to be generally employed as a robustness evaluation of DST models by the future research.
5. H UMAN E VALUATION
[Table]
We next examine the quality of our generated data from two perspectives: “human likeliness” and “turn-level belief state correctness”.
The human likeliness evaluates whether a user utterance is ﬂuent and consistent with its dialog context.
The turn-level belief state correctness evaluates whether ( U sys t , ˆ U usr t ) exactly expresses goals in ˆ L t .
Both metrics are based on binary evaluation.
We randomly sample 100 turns in the original test data and their corresponding CoCo-generated ones.
For the C O C O -generated data, we have two different settings to examine its quality.
The ﬁrst is to use its original turn-level belief state to generate user utterance.
We ask three individuals with proﬁcient English and advanced NLP background to conduct the evaluation for original human response both in MultiWoZ and CoCo-generated counterpart.
We use majority voting to determine the ﬁnal human likeness and turn-level belief state correctness.
Table 3 shows the results.
We can see that the human evaluators could not distinguish between the MultiWOZ’s human utterance and our generated utterance.
Furthermore, C O C O (ori) generated slightly more “correct” responses than the original utterances in MultiWoZ 2.1.
A presumable reason is that annotation errors exist in MultiWoZ 2.1, while our C O C O are trained on recently released cleaner MultiWoZ 2.2, making generated data have higher quality.
The second setting is to evaluate
C O C O (freq)-, C O C O (neu)- and C O C O (rare)-generated data, as they hurt the DST models’ accuracy signiﬁcantly as show in Figure 3, and we need to verify the quality of the generated utterances.
All three variants of the C O C O -generated conversations consistently outperform human response in terms of the turn-level belief state correctness. Although C O C O (neu) and C O C O (rare) are slightly less human-like than the original human response, C O C O (freq)-generated utterances are found to
Original
Original ◇
VS
VS ◇
CoCo+(freq)
[Figure]
[Figure]
be difﬁcult to distinguish from the original human utterances.
These results demonstrate the effectiveness of our proposed approach in generating not only high-ﬁdelity but also human-like user utterances, proving its potential to be adopted as part of robustness evaluation of DST models.
5. A NALYSIS OF C O C O + AS D ATA A UGMENTATION D EFENSE
So far, we have focused on the generalization capability of DST models on C O C O -generated conversations using different slot-value and slot-combination dictionaries.
We have observed that all three
DST models are consistently most susceptible to conversations generated by C O C O +(rare) strategy.
Instead, we now seek to answer the following question: Would using conversations generated by
C O C O + (rare) to augment the training data help these DST models in better generalizing to unseen slot values and/or novel scenarios?
Towards exploring this direction in a principled way, we design a new slot value dictionary ( train-O ) similar to out-of-domain unseen slot-value dictionary ( O ).
For a fair comparison, we make sure that the slot values in train-O (please refer to Appendix H for the complete dictionary) do not overlap with the one ( O ) used for generating test conversations.
We ﬁrst retrain each DST model on the MultiWOZ training split augmented with C O C O +(rare)- generated conversations using train-O slot-value dictionary.
Retrained DST models are then evaluated on original test set as well as on the counterfactual test sets generated by VS and various versions of C O C O +.
Results presented in Figure 4 shows that retraining on the C O C O +(rare)- augmented training data improves the robustness of all three DST models across the board.
Most notably, this data augmentation strategy rebounds the performance of T RIP P Y on C O C O +(rare)- generated test set from 35.5% to 56.2%, signiﬁcantly closing the gap with its performance (61.3%) on the original held-out test set.
We also observe that retrained DST models obtains an improved joint goal accuracy on the original MultiWOZ test set compared to their counterparts trained only on the original MultiWOZ train split, further validating the quality of C O C O -generated conversations.
Finally, we would like to highlight that retrained T RIP P Y model achieves 62.6% joint goal accuracy, improving the previous state-of-the-art by 1.3%.
We leave the exploration of how to fully harness
C O C O as a data augmentation approach as future work.
C ONCLUSION
We propose a principled, model-agnostic approach (C O C O ) to evaluate dialogue state trackers beyond the held-out evaluation set.
We show that state-of-the-art DST models’ performances significantly drop when evaluated on the C O C O -generated conversations. Human evaluations validate that they have high-ﬁdelity and are human-like. Hence, we conclude that these strong DST models have difﬁculty in generalizing to novel scenarios with unseen slot values and rare slot combinations, conﬁrming the limitations of relying only on the held-out accuracy.
When explored as a data augmentation method, C O C O consistently improves state-of-the-art DST models not only on the C O C O -generated evaluation set but also on the original test set.
This further proves the beneﬁt and potential of our approach to be adopted as part of a more comprehensive evaluation of DST models.
A PPENDIX A S LOT - LEVEL A NALYSIS
Closer Look at the Effect of C O C O +(rare) on T RIP P Y .
In Figure 5, we take a closer look at the robustness of T RIP P Y through slot-level analysis across three major scenarios.
Comparison of blue and orange lines reveals that counterfactuals generated by C O C O +(rare) consistently drops the performance of T RIP P Y model (trained on the original MultiWOZ train split) across all the slots, signiﬁcantly hurting the accuracy of all the slots in train domain along with book day slot for hotel domains.
On the other hand, comparing green and orange lines clearly demonstrates the effectiveness of C O C O
+(rare) as a data augmentation defense (see Section 5.4 for further details), assisting T RIP P Y in recovering from most of the errors it made on C O C O +(rare) evaluation set.
In fact, it rebounds the joint goal accuracy of T RIP P Y from 35.5% to 56.2% as presented more quantitatively in Figure
[Figure]
B A BLATION S TUDY ON O PERATIONS
In Table 4, we present ablation results on three meta operations (i.e., drop , change , add ) that are used to generate counterfactual goals.
The result in the ﬁrst row corresponds to the performance of three DST models on evaluation set generated by C O C O including all three meta operations along with the classiﬁer ﬁlter.
Each row analyzes the effects of the corresponding meta operation or classiﬁer by removing it from full models.
From Table 4, we observe that add operation hurts the performance of the three models the most.
This may indicate that the investigated DST models are more vulnerable against user utterances including more rare slot combinations.
[Table]
C
F ULL FIGURE FOR M AIN R ESULT
D T RAINING DETAILS D. T RAINING DETAILS OF CONTROLLABLE GENERATION MODEL
We instantiate p θ ( U usrt | U syst , L t ) with T5-small (Raffel et al., 2020) and build it upon Huggingface transformers library.
During training, we use Adam optimizer (Kingma and Ba, 2015) with an initial learning rate 5 e − 5 and set linear warmup to be 200 steps.
The batch size is set toand training epoch is set to be 10.
The maximum sequence length of both encoder and decoder is set to be 100.
We select the best checkpoint according to lowest perplexity on development set.
D. T RAINING DETAILS OF CLASSIFIER FILTER
We employ BERT-base-uncased as the backbone of our classiﬁer ﬁlter.
During training, we use
Adam optimizer (Kingma and Ba, 2015) with an initial learning rate 2 e − 5 and set linear warmup proportion to 0.1.
We set maximum input length as 512, batch size as 108 and training epoch as 5.
We select the best checkpoint based on the highest recall on development set.
E D IVERSITY E VALUATION
[Table]
[Table]
[Table]
[Table]
[Figure]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
G S LOT -C OMBINATION D ICTIONARY
Please ﬁnd the different slot-combination dictionaries introduced in the main paper below
[Table] [Table]
[Table]
H S LOT -V ALUE D ICTIONARY
Please ﬁnd the different slot-value dictionaries introduced in the main paper below.
[Table] [Table]
[Table]