The D ial P ort tools (2022)

Abstract: The DialPort project ( http://dialport.org/ ), funded by the National Science Foundation (NSF), covers a group of tools and services that aim at fulfilling the needs of the dialog research community. Over the course of six years, several offerings have been created, including the DialPort Portal and DialCrowd. This paper describes these contributions, which will be demoed at SIGDIAL, including implementation, prior studies, corresponding discoveries, and the locations at which the tools will remain freely available to the community going forward.

## introduction the dialport project 1 has created tools and services that respond to needs voiced by many in the dialog research community during several workshops organized by the principle investigators (pis). its offerings are available at no cost to the community with the goal of helping researchers gather high quality data, and easily assess and compare their dialog systems. this paper and its corresponding demos showcase the dialport portal 2 and dial-crowd 3 . there is an increasing need for large amounts of natural dialog data that can be obtained at reasonable cost and in an interactive manner. static datasets are ineffective for both evaluation and optimization. this has led to the creation of the di-alport portal, which facilitates the collection of flexible and evolving data as well as interactive assessment with real users. notably, the portal was used to connect systems and collect data for the interactive evaluation of dialog track @xcite at dstc9 @xcite . another community need centers around how to gather high quality data when using crowdsourcing platforms. dialcrowd has been constructed to facilitate crowdsourcing by guiding researchers to give clear, understandable explanations of the task to the workers who produce or annotate data. it also aids in calculating the correct level of worker payment. finally, it includes several methods of data quality assessment. the university of southern california (usc) is a partner in dialport. the team at usc works on a tools repository foot_0 and the real challenge. this paper gives background and describes in detail the parts of both the portal and dialcrowd. it also provides information on how to access and use them. as the dialport project draws to an end, the paper indicates the permanent sites where these tools will reside. ## dialport portal the dialport portal was initially conceived with the objective of listing many dialog systems from a variety of sites. this type of platform, with demonstrations, links, and references to various systems, is valuable to both researchers and real users. the concept of the portal evolved, and the different systems were linked such that a user could interact with all of the connected systems, transitioning seamlessly between systems, with the dialog state (consisting of slots such as city or date) shared across systems @xcite . as dialog systems continued to improve, especially with the advent of engaging response generation models @xcite , the portal recruited real users through facebook advertising with the objective of providing researchers with a platform to collect interactive dialogs with real users @xcite . ## dialcrowd to address the many issues that present themselves when using crowdsourcing to collect high quality data, dialcrowd was created. dialcrowd (lee et al., 2018) is a dialog assessment toolkit which aids researchers with human intelligence task (hit) creation. requesters follow templates on the di-alcrowd site, which generate a hit that can be linked for a worker on any crowdsourcing site. the second version of this tool (huynh et al., 2022) focuses on collecting high-quality data with tools such as: • links to create better instructions • prompts to provide examples and counterexamples with explanations seen in figure 2 • functionality for adding golden data and duplicate data in each hit • payment suggestions • a feedback area • overall statistics from the hit (time, patterns in the responses, inter-annotator agreement) this allows for requesters to create a wellstructured hit which allows workers to provide better quality annotations. consequently, it makes it easier to filter responses from potential bots. additional tools include the capability to include a mandatory consent form at the start of the hit, and detailed style changes for the hit. further description of the system along with corresponding images can be found in @xcite . one dialcrowd template, intent classification, has been merged into the new home for dialcrowd, parlai foot_1 , and is now available for use. ## the dialport demo the demos of the dialport portal and dashboard and of dialcrowd at sigdial will include: ## conclusion and future directions the tools presented in this demo help dialog researchers in data gathering and assessment. as the community uses them, more types of applications will arise. the tools have been created in a way that enable additions as the field and the needs evolve. figure 3 : the home page for a system on the dialport dashboard. general information about the conversations collected from the system are displayed. sections such as "words and phrases" and "graphs" can be expanded or collapsed to view additional information about the system. figure 4 : using the dialport dashboard to find all conversations in a system with more than 3 utterances