Time-Considerable Dialogue Models via Reranking by Time Dependency (2023)

Abstract: In the last few years, generative dialogue models have shown excellent performance and have been used for various applications. As chatbots become more prevalent in our daily lives, more and more people expect them to behave more like humans, but existing dialogue models do not consider the time information that people are constantly aware of. In this paper, we aim to construct a time-considerable dialogue model that actively utilizes time information. First, we categorize responses by their naturalness at different times and introduce a new metric to classify responses into our categories. Then, we propose a new reranking method to make the existing dialogue model time-considerable using the proposed metric and subjectively evaluate the performances of the obtained time-considerable dialogue models by humans.

## introduction in the last few years, generative dialogue models have achieved outstanding performance @xcite @xcite and have been used in various applications, including search engines, recommendations, healthcare, finance, and more @xcite . as chatbots permeate our daily lives, more and more people expect chatbots to behave in a human-like manner. examples of research to make chatbots more human-like include the introduction of common sense @xcite , empathy @xcite , personas @xcite , and so forth. the common point among these studies is that they have achieved richer dialogue by actively utilizing not only internal information obtained through the conversation but also external information that does not appear in the current dialogue. on the other hand, time information, didn't you have breakfast? i'm hungry. what do you want to eat? (1) an you will get fat if you eat at this time. i don't like doing sports. the most basic and important external information, still does not seem to be considered important in dialogue models. humans are basically always aware of time in conversation, whether explicitly or implicitly, because the naturalness of utterances and responses may change based on their spoken times (e.g., time of the day, day of the week, and season). given an utterance and its spoken time, a response can be categorized into the following four types by focusing on the time variations of its naturalness: for a dialogue model to achieve natural conversation, it is expected to generate natural responses n and to avoid generating unnatural responses u. if a dialogue model correctly evaluates the naturalness of responses considering the spoken time, we call it time-aware; otherwise, we call it time-unaware. many existing dialogue models are time-unaware because they are trained on datasets without time information. as a result, a time-unaware dialogue model may consider tu responses as natural as n responses because they are natural at some time. in other words, it may generate a tu response, which is inappropriate at the spoken time. the simplest way to construct a time-aware dialogue model is to train a general dialogue model with time information @xcite . recently, the output of a large language model (llm) was adjusted by giving an appropriate prompt @xcite @xcite , and prompting the time information to an llm is another promising way to achieve a time-aware dialogue model. the difference between time-unaware and timeaware dialogue models is that the former considers a response as natural if it is natural at some time, while the latter does if it is natural at its spoken time; namely, time-aware dialogue models avoid generating tu responses. prior research has shown that users' impressions of dialogue models are improved by actively utilizing external information @xcite @xcite . referring to this fact, we assume that users' impressions will be similarly improved by actively utilizing time information; namely, users prefer tn to an (we empirically verify this assumption in section 3). under this assumption, we aim to realize a timeconsiderable dialogue model that actively outputs more tn than an. the difference between timeaware and time-considerable models is that the former only considers time information to evaluate the naturalness of responses at a given spoken time (time-aware naturalness), while the latter actively generates responses whose naturalness varies with time. to realize time-considerable models, we need a new criterion to distinguish between tn and an and a new mechanism to generate tn responses. in this paper, we propose a new reranking method that is a post-processing method to make existing dialogue models more time-considerable. as a preliminary analysis, we first verify our assumption users tend to prefer tn to an by human evaluation in section 3. we next formally define the nuta categories and propose an automatic metric for the nuta categories in section 4. in section 5, we propose a new reranking method using the proposed automatic metric to make existing dialogue models time-considerable and subjectively evaluate obtained time-considerable models to verify whether our reranking method improves response qualities. ## related work previous studies have pointed out that general dialogue models trained on large-scale datasets tend to generate neutral (bland, generic, and hackneyed) responses @xcite . to tackle this issue, some dialogue models utilizing external information to generate interesting and informative responses have been proposed, where examples of external information include system/user persona @xcite @xcite , knowledge graphs @xcite , knowledge sources @xcite , interpersonal relationships (utami and bickmore, 2019), and situated environments @xcite . some studies have empirically shown that generating responses specific to external information improved users' impression of the dialogue models @xcite @xcite . in this paper, we introduce the time information as external information and consider that t responses, whose naturalness varies with time, are specific to time information. furthermore, to verify whether the time information improves model performance as well as other external information, we evaluate the qualities of t responses by human evaluations. in this paper, we define a dialogue model as timeaware if the model evaluates the naturalness of responses considering their spoken times. @xcite proposed a time-aware dialogue model that is an encoder-decoder model based on long-short term memory (lstm) @xcite inspired by @xcite and trained on utterance-response pairs with their timestamps ex-tracted from twitter. another recent technique to achieve a time-aware dialogue model is prompting llms, where prompting is a technique to guide llms in generating high-quality and relevant responses by providing detailed descriptions and/or input-output examples of the target task as input @xcite . various promptingbased methods for utilizing external information have been proposed, and their examples include system/user persona information @xcite , knowledge sources @xcite , and fictional character's style @xcite . in this paper, we aim to construct a time-considerable dialogue model, which actively utilizes the time information, and propose a new reranking method to make the existing dialogue models more time-considerable using time-aware naturalness represented by a time-aware dialogue model. while time-aware dialogue models only consider time to evaluate the naturalness of responses, time-considerable dialogue models actively generate responses whose naturalness varies with time. ## preliminary analysis through this paper, we assume users prefer tn to an, and in this section, we verify this assumption by human evaluations. we first constructed a nuta dataset consisting of the tuples of utterances, their spoken times, their responses, and their nuta categories: an, tn, tu, and au. then, we conducted a subjective evaluation to determine which category of responses users found most interesting and informative. ## automatic metric for nuta categories we propose a new automatic metric for classifying the nuta categories of given responses and experimentally show that our metric can correctly categorize the responses of the nuta dataset. ## time-considerable dialogue models we propose a new reranking method to make existing dialogue models time-considerable. we applied our method to various existing models, including gpt-4, which is a state-of-the-art llm, and evaluated the time-considerable dialogue models to verify whether they improved response qualities. ## conclusion we proposed a new reranking method to construct time-considerable dialogue models that distinguish between always natural responses an and temporally natural responses tn and actively output tn. we verified the assumption users prefer tn to an by human evaluations and introduced a new metric to classify the nuta categories of responses. we proposed a new reranking method to make existing dialogue models time-considerable using the metric and empirically showed that our method improved the qualities of existing models. a promising future study is to control the degree to which a time-considerable dialogue model actively uses time information in different situations. not only time information but all external information is valuable when used appropriately, but excessive use may harm the users' impression. therefore, we plan to develop a mechanism to estimate an appropriate td@t value that a response should have in the current situation and to output or generate a response with the estimated value.