Local and Global Contexts for Conversation (2024)

Abstract: The context in conversation is the dialog history crucial for multi-turn dialogue. Learning from the relevant contexts in dialog history for grounded conversation is a challenging problem. Local context is the most neighbor and more sensitive to the subsequent response, and global context is relevant to a whole conversation far beyond neighboring utterances. Currently, pretrained transformer models for conversation challenge capturing the correlation and connection between local and global contexts. We introduce a local and global conversation model (LGCM) for general-purpose conversation in open domain. It is a local-global hierarchical transformer model that excels at accurately discerning and assimilating the relevant contexts necessary for generating responses. It employs a local encoder to grasp the local context at the level of individual utterances and a global encoder to understand the broader context at the dialogue level. The seamless fusion of these locally and globally contextualized encodings ensures a comprehensive comprehension of the conversation. Experiments on popular datasets show that LGCM outperforms the existing conversation models on the performance of automatic metrics with significant margins.

## introduction the role of context is significant in the similarity of words in a language. the contexts of a word are the neighboring tokens or grammatical structures. contextualized embeddings encode both words and their contexts and generate contextualized representations. language modeling captures distributed semantics embedded within these contextualized representations. the transformer-based pretrained language models (lms) have become a foundation for nlp-like tasks @xcite . a well-established best practice in the field has consistently demonstrated that the utilization of large language models (llms) tends to yield superior performance in a wide range of nlp tasks, including conversational applications (say @xcite @xcite @xcite , among others). conversation models (cms) are generative sequence-sequence models for general-purpose conversations and learn the multi-agent distribution of utterances simultaneously. most existing cms are based on lms, in which the lms are used for accomplishing conversation by collaboration between agents that own their lms or share a single lm in the spirit of parameter sharing (ps), where multiple models share the parameters in part or whole. in this paper, we consider the cms with a single lm for two-agent conversation, such as human-machine dyadic dialogue. more specifically, cms use either vanilla transformer @xcite as single-turn dialogue, such as question answering, where only the current utterance is considered as the history at any given turn, or for multi-turn dialogue adapt the transformer architecture by concatenating multiple turns sequentially to capture the evolving context @xcite @xcite . prominent examples of such cms include transfertransfo @xcite , meena @xcite , blender @xcite , athena @xcite and lamda @xcite , among others. the context in conversation is the dialog history crucial for multi-turn dialogue. cms require an understanding of the dialog history, in the context of previous pairwise utterances and the current query at any turn. for example, as humans in everyday dialogue, the speaker's intent often cannot be detected by looking at the utterance level. in contrast, the speaker's acts are specific to each utterance and change throughout a whole dialog history at the dialogue level. one of the key challenges faced by cms lies in striking the right balance between staying current which involves giving preference to recent utterances, and drawing from the past effectively accumulating a prior understanding of the dialogue. the process of learning the relevant historical contexts necessary for fostering grounded and meaningful conversations remains a challenging problem in this domain. a criticism of the existing cms is their inability to effectively utilize the available dialog history and gain a comprehensive view of a conversation @xcite . a common problem of those cms is their failure to establish meaningful correlations and connections between individual utterances. they often treat all the words as a single sequence and concatenate multiple turns in history into a single sequence, which neglects the distinct contexts of individual utterances within the broader dialogue history. to address the inherent problem of current cms, we propose a more nuanced approach. in our model, we define each utterance as local context for tokens at the utterance level and whole a dialogue as global context for inter-utterances at the dialogue level. moreover, we find it valuable to position the relationships among inter-utterances within a dialog history relative to one another. in our model, the conversation at different turns tells on each other, and all together, they tell what we talk about. namely, we introduce a local and global cm (lgcm) for multi-turn dialogue in open domain. it is a local-global hierarchical transformer model, illustrated in figure 1 . it is an encoder-decoder architecture in which the decoder is the same as transformer @xcite with the crossattention between the encoder and the decoder, but the encoder is a hierarchical attention structure. the encoder of lgcm consists of local encoders and global encoder. the local encoders are implemented by a standard transformer module (self-attention) for each utterance in the local context using absolute position encoding (ape). the global encoder consists of inter-attention and gate for contextualized representations in the global context, which are sent to the cross-attention in the decoder. the inter-attention is the attention between the current and all the utterances using relative positional encoding (rpe) @xcite . the gate fuses the representations of the local encoders and the inter-attention by a nonlinear transformation for local-global contextualized representation, see explanation in the subsection 3.2. in summary, the main contributions of this paper are the following: experiments on popular datasets (dailydialog, multiwoz, personachat) show that lgcm takes advantage of the distinction between local and global contexts and outperforms the existing cms on the performance of automatic metrics (ppl, bleu, meteor, nist, rougel) with significant margins (the best ratios range from 35.49% to 71.61%). in the next section, we discuss the related works. in section 3, we present lgcm in detail. in section 4, we experiment on comparing lgcm with strong baseline cms. finally, we make some concluding remarks. ## related works we concentrate on the cms that use transformerbased lms (see surveys @xcite for transformers and @xcite for lms). most cms use lms for multi-turn dialogue in open-domain @xcite @xcite @xcite . sota cms were large lms (llms) trained specifically for conversation, such as chatgpt foot_1 , among other similar models. although llms can achieve the best practice from time to time, they scale up the transformer, especially involving concatenating the dialog history into a single sequence. small models are suitable for the study of cms first, as the saying goes, it is difficult for a big ship to turn around. representative cms are strong baselines based on small lms such as gpt @xcite and bert @xcite . among them @xcite @xcite @xcite , transfertransfo @xcite trained especially on the basis of gpt, dialogpt @xcite on gpt2 @xcite , and dialogbert @xcite on bert for dialog response generation. hierarchical encoders are a common framework for conversation. hred was first introduced as two-level rnns for multi-turn dialogue with a fuse between utterance and context dependencies @xcite @xcite . most the attention and fuse mechanism. in essence, the concept of a hierarchical localglobal architecture is not a novel one. however, what sets our model apart is our innovative approach to establishing meaningful correlations and connections between local and global contexts. we achieve this by introducing the inter-attention and gate mechanisms, which work in tandem to facilitate more coherent and contextually relevant conversations. ## conclusions pretrained transformer models are adjusted by concatenating contexts into a single lengthy sequence. it is imperative to explore a variety of methods to encode the context effectively. we have introduced a local and global conversation model for multi-turn dialogues in open domain. this model harnesses a hierarchical transformer encoder architecture, seamlessly integrating local and global contexts to enhance the efficacy of conversation. we have underscored the significance of distinguishing between the local context for tokens within an utterance at the utterance level and the global context for inter-utterances within a dialogue at the dialogue level. we hope that this study contributes to the comprehension of language models and conversational ai.