Findings of the First Workshop on Simulating Conversational Intelligence in Chat (2024)

Abstract: The aim of this workshop is to bring together experts working on open-domain dialogue research. In this speedily advancing research area many challenges still exist, such as learning information from conversations, engaging in realistic and convincing simulation of human intelligence and reasoning. SCI-CHAT follows previous workshops on open domain dialogue but with a focus on the simulation of intelligent conversation as judged in a live human evaluation. Models aim to include the ability to follow a challenging topic over a multi-turn conversation, while positing, refuting and reasoning over arguments. The workshop included both a research track and shared task. The main goal of this paper is to provide an overview of the shared task and a link to an additional paper that will include an in depth analysis of the shared task results following presentation at the workshop.

## introduction despite substantial progress in conversational ai over the past number of years and heightened attention amongst the general public, effective evaluation of such systems remains a challenge. the ideal evaluation of dialogue models consists of measurement of performance via a large group of human users who partake in conversations with models in a live evaluation and report the successes or failures 1 https://arxiv.org/abs/2402.06420 that take place. past attempts at live human evaluation of dialogue systems have yet to be successful, as results have either relied fully on automatic metrics known to correlate poorly with human evaluation (if at all), or discarded human evaluation as they were unfortunately deemed unreliable @xcite , and past challenges are likely due to the nature of the problem. there often exists an excessively large number of potential good responses (rendering reference-based evaluation as vastly under-rewarding systems), in addition to the challenges of evaluating the many facets of human conversation that enable simulation of intelligence. open domain dialogue susequently provides what we consider to be one of the most challenging evaluation tasks in nlp. in this shared task, we revisit live human evaluation of models, and apply methods proven successful in distinct nlp tasks to the open domain dialogue. ## shared task the shared task has the focus of simulating any kind of intelligent conversation and participants were asked to submit an automated dialogue agent api with the aim of carrying out nuanced conversations over multiple dialogue turns, and the ability to posit, refute and reason over arguments. participating systems were then interactively evaluated in a live human evaluation following the procedure described in @xcite . 1 ## results and analysis results and analysis of the competition are provided at the following url: https://arxiv.org/ abs/2402.06420 . ## conclusion this paper describes an outline of the shared task currently underway to assess the ability of stateof-the-art dialogue models to simulate intelligence in conversation. in-depth results and analysis will be provided here on completion of the live human evaluation of models: https://arxiv.org/abs/ 2402.06420 all data acquired within the context of the shared task will be made public, providing an important resource for improving human evaluation and automatic metrics in this research area.