This paper presents a methodology and toolkit for creating a rule-based multi-domain conversational agent for transactions from (1) language annotations of the domains' database schemas and APIs and (2) a couple of hundreds of annotated human dialogues. There is no need for a large annotated training set , which is expensive to acquire. The toolkit uses a pre-defined abstract dialogue state machine to synthesize millions of dialogues based on the domains' information. The annotated and synthesized data are used to train a contextual semantic parser that interprets the user's latest utterance in the context of a formal representation of the conversation up to that point. Developers can refine the state machine to achieve higher accuracy. On the MultiWOZ benchmark, we achieve over 71 \% turn-by-turn slot accuracy on a cleaned, reannotated test set, without using any of the original training data. Our state machine can model 96 \% of the human agent turns. Our training strategy improves by 9\% over a baseline that uses the same amount of hand-labeled data , showing the benefit of synthesizing data using the state machine .
The traditional dialogue state tracking (DST) task tracks the dialogue state given the past history of user and agent utterances. This paper proposes to replace the utterances before the current turn with a formal representation , which is used as the context in a semantic parser mapping the current user utterance to its formal meaning. In addition, we propose TOC (Task-Oriented Context), a formal dialogue state representation. This approach eliminates the need to parse a long history of natural language utterances; however, it adds complexity to the dialogue annotations. We propose Skim, a contextual semantic parser , trained with a sample-efficient training strategy: (1) a novel abstract dialogue state machine to synthesize training sets with TOC annotations; (2) data augmentation with automatic paraphrasing, (3) few-shot training, and (4) self-training. This paper also presents MultiWOZ 2.4, which consists of the full test set and a partial validation set of MultiWOZ 2.1, reannotated with the TOC representation. Skim achieves 78 \% turn-by-turn exact match accuracy and 85\% slot accuracy, while our annotation effort amounts to only 2 \% of the training data used in MultiWOZ 2.1. The MultiWOZ 2.4 dataset will be released upon publication .
R
This paper presents a methodology and toolkit for creating a rule-based multi-domain conversational agent for transactions from (1) language annotations of the domains' database schemas and APIs and (2) a couple of hundreds of annotated human dialogues. There is no need for a large annotated training set
The traditional dialogue state tracking (DST) task tracks the dialogue state given the past history of user and agent utterances. This paper proposes to replace the utterances before the current turn with a formal representation
R
expensive to acquire. The toolkit uses a pre-defined abstract dialogue state machine to synthesize millions of dialogues based on the domains' information. The annotated and synthesized data are used to train
used as the context in a semantic parser mapping the current user utterance to its formal meaning. In addition, we propose TOC (Task-Oriented Context), a formal dialogue state representation. This approach eliminates the need to parse a long history of natural language utterances; however, it adds complexity to the dialogue annotations. We propose Skim,
R
that interprets the user's latest utterance in the context of a formal representation of the conversation up to that point. Developers can refine the
, trained with a sample-efficient training strategy: (1) a novel abstract dialogue
R
achieve higher accuracy. On the MultiWOZ benchmark, we achieve over 71
synthesize training sets with TOC annotations; (2) data augmentation with automatic paraphrasing, (3) few-shot training, and (4) self-training. This paper also presents MultiWOZ 2.4, which consists of the full test set and a partial validation set of MultiWOZ 2.1, reannotated with the TOC representation. Skim achieves 78
R
slot accuracy on a cleaned, reannotated test set, without using any of the original training data. Our state machine can model 96
exact match accuracy and 85\% slot accuracy, while our annotation effort amounts to only 2
R
human agent turns. Our training strategy improves by 9\% over a baseline that uses the same amount of hand-labeled data , showing the benefit of synthesizing data using the state machine
training data used in MultiWOZ 2.1. The MultiWOZ 2.4 dataset will be released upon publication