Don’t Shoot The Breeze: Topic Continuity Model Using Nonlinear Naive B ayes With Attention (2024)

Abstract: Utilizing Large Language Models (LLM) as chatbots in diverse business scenarios often presents the challenge of maintaining topic continuity. Abrupt shifts in topics can lead to poor user experiences and inefficient utilization of computational resources. In this paper, we present a topic continuity model aimed at assessing whether a response aligns with the initial conversation topic. Our model is built upon the expansion of the corresponding natural language understanding (NLU) model into quantifiable terms using a Naive Bayes approach. Subsequently, we have introduced an attention mechanism and logarithmic nonlinearity to enhance its capability to capture topic continuity. This approach allows us to convert the NLU model into an interpretable analytical formula. In contrast to many NLU models constrained by token limits, our proposed model can seamlessly handle conversations of any length with linear time complexity. Furthermore, the attention mechanism significantly improves the model’s ability to identify topic continuity in complex conversations. According to our experiments, our model consistently outperforms traditional methods, particularly in handling lengthy and intricate conversations. This unique capability offers us an opportunity to ensure the responsible and interpretable use of LLMs.

## introduction the rise of large-scale language models (llms) @xcite has empowered chatbots to handle various business tasks, such as serving as office assistants @xcite , coding companions @xcite , and data explorers @xcite . however, leveraging llms for these roles often presents challenges like hallucination @xcite , offensive language @xcite , prompt injection @xcite , and adversarial attacks @xcite . in addition to these common issues, specific business applications may introduce unique problems, such as maintaining topic continuity. for example, when using llms as a customer service chatbot, llms are employed to address inquiries about specific products or services. however, because llm responses are inherently random, there's no guarantee that they will consistently remain focused on the intended topics, potentially resulting in a subpar user experience. on the other hand, if users veer off into unrelated topics, it could also lead to the waste of valuable computational resources. therefore, ensuring topic coherence between the customer and the chatbot is crucial. in customer service, users initially describe their concerns. when these concerns pertain to the business's operations, the customer and chatbot collaborate on solutions @xcite . ensuring a smooth conversation involves assessing if the current sentence logically follows the prior ones. for example, if a user discussing refunds suddenly asks, "can you help me order a pizza?" -it's off-topic. this concept is formalized as a natural language understanding model (nlu) @xcite , denoted as p (y|s 1 , s 2 , . . . ; s n ). here, s i (for @xmath0, and s n is the current one. the binary variable y indicates whether s n aligns with preceding sentences, keeping the conversation on-topic. in practical use, when users interact with llm, we assess if each new sentence, whether from the user or the llm, keeps the conversation on-topic. if it goes offtopic, we guide it back to business-related subjects or may end the conversation. so, we assume the previous n-1 sentences are on-topic, and we calculate whether the newly added n th sentence still aligns with the ongoing conversation. this simplifies the problem to determining whether the n th sentence has a reasonable contextual relationship with the previous n-1 sentences. the most commonly used approach to address this issue is a bert-based language model @xcite . these models are inherently equipped with the capability to evaluate the contextual relationship between two sentences. however, employing this approach consistently gives rise to two inevitable challenges: 1) token size limit and 2) lack of sentence attention. regarding the first challenge, imagine using a language model to assess the connection between (s 1 + s 2 + . . . + s n -1 ) and the current sentence s n in a conversation. as the conversation grows, the text often exceeds most language models' token limits, typically set at 512 tokens for many bert-based models. regarding the second challenge, most language models are trained on sentence pairs from articles where semantic relationships are consistently close. however, real conversations often involve looser semantic connections. for example, a customer might say, "earlier, you asked about the missing product serial number, but now i've found it." this response references a part of the conversation from several rounds back. concatenating s 1 ∼ s n -1 as context can lead to the model struggling to judge the appropriateness of s n as a follow-up. in summary, an effective conversational topic continuity model must address two key challenges: 1) handling lengthy conversations, and 2) accommodating semantic leaps. to address these challenges, we introduce an innovative topic continuity model that integrates logarithmic nonlinearity and sentence attention into the naive bayes framework @xcite . our method provides a fully analytical formulation of the problem, effectively addressing the aforementioned issues and delivering significantly superior performance compared to conventional methods. ## conclusion with the rapid development of large language models (llms), the effective utilization of llms in various business scenarios has become an important issue. in this paper, we propose a method that ensures user conversations with llms remain focused on fixed topics. this method is based on the introduction of non-linear transformations and attention mechanisms through an extension of naive bayes. experimental results across various scenarios consistently demonstrate that our approach outperforms traditional methods. we believe this method will be highly beneficial for using llms in topic-constrained scenarios.