Generative models for dialog system have gained a lot of interest because of the success of recent RNN and Transformer based models in complex natural language tasks like question answering and summarization. Although , the task of dialog response generation is generally seen as a sequence to sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn important utterance and conversation level features  Sordoni et al. ( 2015 ); Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer based models dominating the seq2seq problems lately, a natural question is to understand the applicability of the notion of hierarchy in transformer based dialog systems. In this paper, we  show how a standard transformer can be morphed into a hierarchical one by using specially designed attention masks and positional embeddings. Our experiments show strong improvements in context-to-response generation performance for task-oriented dialog system over the current state-of-the-art approaches .
Generative models for dialog systems have gained much interest because of the recent success of RNN and Transformer based models in tasks like question answering and summarization. Although the task of dialog response generation is generally seen as a sequence-to-sequence (Seq2Seq) problem, researchers in the past have found it challenging to train dialog systems using the standard Seq2Seq models. Therefore, to help the model learn meaningful utterance and conversation level features , Sordoni et al. ( 2015b ); Serban et al. (2016) proposed Hierarchical RNN architecture, which was later adopted by several other RNN based dialog systems. With the transformer-based models dominating the seq2seq problems lately, the natural question to ask is the applicability of the notion of hierarchy in transformer based dialog systems. In this paper, we propose a generalized framework for Hierarchical Transformer Encoders and show how a standard transformer can be morphed into any hierarchical encoder, including HRED and HIBERT like models, by using specially designed attention masks and positional encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models for task-oriented dialog systems through a wide range of experiments .
R
system have gained a lot of
systems have gained much
R
success of recent
recent success of
D
complex natural language
D
,
R
sequence to sequence
sequence-to-sequence
R
important
meaningful
A
,
R
2015
2015b
R
transformer based
transformer-based
R
a natural question is to understand
the natural question to ask is
A
propose a generalized framework for Hierarchical Transformer Encoders and
R
a hierarchical one
any hierarchical encoder, including HRED and HIBERT like models,
R
embeddings. Our experiments show strong improvements in context-to-response generation performance
encodings. We demonstrate that Hierarchical Encoding helps achieve better natural language understanding of the contexts in transformer-based models
R
system over the current state-of-the-art approaches
systems through a wide range of experiments