Transformers to Learn Hierarchical Contexts in Multiparty Dialogue for Span-based Question Answering (2020)

Abstract: We introduce a novel approach to transformers that learns hierarchical representations in multiparty dialogue. First, three language modeling tasks are used to pre-train the transformers, token- and utterance-level language modeling and utterance order prediction, that learn both token and utterance embeddings for better understanding in dialogue contexts. Then, multi-task learning between the utterance prediction and the token span prediction is applied to fine-tune for span-based question answering (QA). Our approach is evaluated on the FriendsQA dataset and shows improvements of 3.8% and 1.4% over the two state-of-the-art transformer models, BERT and RoBERTa, respectively.

## introduction transformer-based contextualized embedding approaches such as bert @xcite , xlm (conneau and lample, 2019), xlnet @xcite , roberta @xcite , and albert @xcite have re-established the state-of-the-art for practically all question answering (qa) tasks on not only general domain datasets such as squad @xcite , ms marco @xcite , triviaqa @xcite , newsqa @xcite , or narrativeqa @xcite , but also multiturn question datasets such as sqa @xcite , quac @xcite , coqa @xcite , or cqa @xcite . however, for span-based qa where the evidence documents are in the form of multiparty dialogue, the performance is still poor even with the latest transformer models @xcite due to the challenges in representing utterances composed by heterogeneous speakers. several limitations can be expected for language models trained on general domains to process dialogue. first, most of these models are pre-trained on formal writing, which is notably different from colloquial writing in dialogue; thus, fine-tuning for the end tasks is often not sufficient enough to build robust dialogue models. second, unlike sentences in a wiki or news article written by one author with a coherent topic, utterances in a dialogue are from multiple speakers who may talk about different topics in distinct manners such that they should not be represented by simply concatenating, but rather as sub-documents interconnected to one another. this paper presents a novel approach to the latest transformers that learns hierarchical embeddings for tokens and utterances for a better understanding in dialogue contexts. while fine-tuning for span-based qa, every utterance as well as the question are separated encoded and multi-head attentions and additional transformers are built on the token and utterance embeddings respectively to provide a more comprehensive view of the dialogue to the qa model. as a result, our model achieves a new state-of-the-art result on a span-based qa task where the evidence documents are multiparty dialogue. the contributions of this paper are: foot_0 transformer encoder (te) softmax e w 11 e s 1 ⋯ e μ ij e w 1n ⋯ e c o μ ij ⋯ ⋯ [cls] s 1 w 11 w 1n μ ij ⋯ ⋯ s m w m1 w mn ⋯ e s m e w m1 e w mn ⋯ transformer encoder (te) softmax ⋯ ⋯ [clsi] s i w i1 μ ij w in ⋯ e w in e w i1 e s i e μ ik ⋯ e c i o μ ij (a) token-level mlm ( §2.1.1) transformer encoder (te) softmax e w 11 e s 1 ⋯ e μ ij e w 1n ⋯ e c o μ ij ⋯ ⋯ [cls] s 1 w 11 w 1n μ ij ⋯ ⋯ s m w m1 w mn ⋯ e s m e w m1 e w mn ⋯ transformer encoder (te) softmax ⋯ ⋯ [clsi] s i w i1 μ ij w in ⋯ e w in e w i1 e s i e μ ik ⋯ e c i o μ ij (b) utterance-level mlm ( §2.1.2) transformer encoder (te) softmax e w 11 e s 1 ⋯ e μ ij e w 1n ⋯ e c o μ ij ⋯ ⋯ [cls] s 1 w 11 w 1n μ ij ⋯ ⋯ s m w m1 w mn ⋯ e s m e w m1 e w mn ⋯ transformer encoder (te) softmax ⋯ ⋯ [clsi] s i w i1 μ ij w in ⋯ e w in e w i1 e s i e μ ik ⋯ e c i o μ ij transformer encoder (te) ⋯ [cls1] s 1 w 11 w 1n [clsm] s′ m w′ m1 w′ mn ⋯ ⋯ ⋯ ⋯ [clsi] ⋯ s′ i w′ i1 ⋯ w′ in ⋯ ⋯ tl2 tl1 softmax o ν t c 1 t c m ⋯ t c i ⋯ e c 1 e c m ⋯ ⋯ e c i e w 11 e s 1 e s m e w m1 e w 1n ⋯ e w mn ⋯ e s i e w i1 ⋯ e w in ⋯ ⋯ ⋯ ⋯ (c) utterance order prediction ( §2.1.3) figure 1 : the overview of our models for the three pre-training tasks (section 2.1). ## transformers for learning dialogue this section introduces a novel approach for pretraining (section 2.1) and fine-tuning (section 2.2) transformers to effectively learn dialogue contexts. our approach has been evaluated with two kinds of transformers, bert @xcite and roberta @xcite , and shown significant improvement to a question answering task (qa) on multiparty dialogue (section 3). ## conclusion this paper introduces a novel transformer approach that effectively interprets hierarchical contexts in multiparty dialogue by learning utterance embeddings. two language modeling approaches are proposed, utterance-level masked lm and utterance order prediction. coupled with the joint inference between token span prediction and utterance id prediction, these two language models significantly outperform two of the state-of-the-art transformer approaches, bert and roberta, on a span-based qa task called friendsqa . we will evaluate our approach on other machine comprehension tasks using dialogues as evidence documents to further verify the generalizability of this work.