Simultaneous Job Interview System Using Multiple Semi-autonomous Agents (2022)

Abstract: In recent years, spoken dialogue systems have been applied to job interviews where an applicant talks to a system that asks pre-defined questions, called on-demand and self-paced job interviews. We propose a simultaneous job interview system, where one interviewer can conduct one-on-one interviews with multiple applicants simultaneously by cooperating with the multiple autonomous job interview dialogue systems. However, it is challenging for interviewers to monitor and understand all the parallel interviews done by the autonomous system at the same time. As a solution to this issue, we implemented two automatic dialogue understanding functions: (1) response evaluation of each applicant’s responses and (2) keyword extraction as a summary of the responses. It is expected that interviewers, as needed, can intervene in one dialogue and smoothly ask a proper question that elaborates the interview. We report a pilot experiment where an interviewer conducted simultaneous job interviews with three candidates.

## introduction owing to the widespread use of online job interviews during the covid-19 situation, spoken dialogue systems supporting job interviews to make them more efficient are being investigated. in conventional face-to-face job interviews, interviewers conducted interviews with many applicants one by one, which was time-consuming. therefore, ondemand interviews have been widely adopted as an alternative to face-to-face interviews, such as hirevue 1 and modern hire 2 . in this style, job applicants answer predefined typical questions and then submit video recordings of interviews. however, there is a lack of the much needed interaction between interviewers and applicants since applicants only respond to predefined questions. therefore, to elicit sufficient information from applicants for their selection becomes difficult. in this study, we propose a new framework for a spoken dialogue system that makes job interviews more interactive and efficient than that of on-demand interviews. the proposed framework is a cooperation between system and humans, namely semi-autonomous agents. with this framework, job interviewers can conduct multiple job interviews simultaneously. specifically, a human job interviewer (operator) cooperates with multiple autonomous job interview agents to conduct one-onone interviews with multiple applicants simultaneously (figure 1 ). for most of the session, an autonomous agent conducts a job interview with each job applicant, and the human interviewer (operator) monitors them. the interviewer can intervene in any of the dialogues when necessary and then asks specific follow-up questions that cannot be generated by the autonomous agent. these followup questions are necessary to make job interviews more interactive and substantial. in this paper, we describe the framework of the proposed system and report a pilot experiment. next, we describe the proposed simultaneous job interview system. in this system, each applicant is interviewed by the above autonomous agent, and the human interviewer observes these multiple interview sessions. if the human interviewer wants to directly interact with any applicant, the interviewer can switch from the autonomous agent and then interact with the applicant. for example, the interviewer can ask specific follow-up questions that cannot be generated by the autonomous agent. then, after the interviewer ends the intervention, the autonomous agent continues the session. in this system, the interviewer is required to comprehend each applicant's answer and then ask proper follow-up questions and also decide on the timing of intervention. however, due to the cognitive ability of humans, it is not possible to understand the contents of multiple dialogues simultaneously. even if each log of automatic speech recognition is generated and shown to the interviewer, it is difficult to follow all of them. it is necessary to summarize the information of each session. therefore, we introduce response evaluation and keyword extraction that enable the interviewer to follow the dialogues done by multiple agents, as follows. ## system implementation figure 3 depicts the configuration of the proposed system. the autonomous job interview system runs for each applicant. the input speech is segmented by a pause and fed into an automatic speech recognition with the sub-word-based attention mecha-nism. the recognition results are concatenated within the same turn and then used for the response evaluation and keyword extraction. the interface of the job interviewer agent is realized by mmdagent @xcite . the system utterances are played with a text-to-speech engine. figure 2 shows the gui for an interviewer where they can monitor multiple dialogues. this interface consists of mainly three items: (1) the dialogue history of each applicant, (2) the results of response evaluation, and (3) the results of keyword extraction. the human interviewer can select any applicant they want to intervene by clicking a button in the gui. once the interviewer selects an applicant, they can interact with each other directly, meanwhile, autonomous agents talk with the other applicants simultaneously. ## pilot experiment we conducted a pilot experiment to confirm if the proposed system can handle multiple job interviews. in this experiment, a within-subject comparison was made between the fully autonomous system without human intervention (baseline) and the proposed system with three applicants. the subjects were 30 undergraduate and graduate students as applicants in the setting of "a student who participates in a first-round interview of some company." note that the company was selected by each participant freely and independently. they were divided into groups of three persons in the condition 2 where each was rated on a 5-point scale from 1 to 5. this experiment was conducted in japanese. we present some comments given by the subjects after the experiment. following were the comments regarding the proposed system. "i thought it was efficient to let the machine ask the typical questions that have to be asked during the interview and let a human engage in interaction more advanced." "i get very nervous when the questions are asked back. it is good to have a realistic sense." the baseline fully autonomous system received the following comments. "i did not feel like i was being listened to." "i did not really feel like i was being interviewed because i was always told "i see" after each answer. i did not feel like i was being interviewed very much." ## conclusions we propose a simultaneous job interview system that allows human interviewers to interact with multiple applicants in real-time based on response evaluation and keyword extraction. for the interface of interviewers, the response evaluation and keyword extraction were implemented for making efficient intervention. in the pilot experiment, we showed the effectiveness of the proposed system and confirmed the proposed architecture would potentially be accepted as a new framework for future job interviews.