HR - M ulti WOZ : A Task Oriented Dialogue ( TOD ) Dataset for HR LLM Agent (2024)

Abstract: Recent advancements in Large Language Models (LLMs) have been reshaping Natural Language Processing (NLP) task in several domains. Their use in the field of Human Resources (HR) has still room for expansions and could be beneficial for several time consuming tasks. Examples such as time-off submissions, medical claims filing, and access requests are noteworthy, but they are by no means the sole instances. However the aforementioned developments must grapple with the pivotal challenge of constructing a high-quality training dataset. On one hand, most conversation datasets are solving problems for customers not employees. On the other hand, gathering conversations with HR could raise privacy concerns. To solve it, we introduce HR-Multiwoz, a fully-labeled dataset of 550 conversations spanning 10 HR domains. Our work has the following contributions:(1) It is the first labeled open-sourced conversation dataset in the HR domain for NLP research. (2) It provides a detailed recipe for the data generation procedure along with data analysis and human evaluations. The data generation pipeline is transferrable and can be easily adapted for labeled conversation data generation in other domains. (3) The proposed data-collection pipeline is mostly based on LLMs with minimal human involvement for annotation, which is time and cost-efficient.

## introduction recent advances in natural language processing (nlp) have been applied in a variety of tasks in the human resources (hr) domain ranging from skill extraction @xcite , job understanding @xcite to candidate sourcing @xcite . unlike nlp research in other domains @xcite , numerous hr processes remain highly inefficient, such as requesting time off, scheduling meetings, submitting tickets for it issues, or filing medical claims. in fact, the asana work index report shows that knowledge workers spend 60 percent of their time on repetitive work. llm agent @xcite uses llms as its central computational engine, allowing it to carry on conversations, do tasks, reason, and display a degree of autonomy. similar to other domains @xcite , creating an llm agent to help with these tasks could save a significant amount of time for employees and improve job satisfaction. a good llm agent should be able to understand the requirements of users @xcite . the ideal dataset to evaluate or train an hr llm agent should contain conversations between a virtual assistant and employees, annotated with dialogue states. dialogue states contain representations of a conversation's current context such as intentions and relevant information. for a dataset to be useful in building/evaluating an hr llm agent, it must satisfy the following four requirements: (1) the information in the dialogue state must be extractive. when using an llm agent to file a medical claim, employees must be able to trust that the system will accurately retrieve the right number. thus, the extracted information must be from the conversation. (2) the information in the dialogue state should contain long entity. when using a llm agent to solve a code bug, employees have to provide more detail about the code issue. this means that the extracted information should be long enough to give the llm agent correct information. (3) the dataset must be hr specific and discuss about hr-relevant tasks. (4) the conversation must be empathetic. in real conversations with hr, it is important to communicate with employees respectfully. this could enhance inclusive culture within the organization. the llm agent built on this dataset could also be empathetic. there are many open-source conversation datasets. schema-guided dialogue (sgd) (rastogi et al., 2020) is a dialogue dataset with evolving ontologies, introducing new test set slots and services, emphasizing dst performance and zeroshot generalization. sgd-x (lee et al., 2022) expands on sgd, presenting five additional schema styles. m2m @xcite connects a developer, who provides the task-specific information, and a framework, which provides the task-independent information, for generating dialogues centered around completing the task. mul-tiwoz @xcite features humanhuman dialogues using a stable ontology. however, all these datasets are customer-facing instead of employee-facing. also, none of them is fully extractive or related to hr. the extracted information is also short in size. hr llm agent trained on these models may not be empathetic, extract complete information from employees and misunderstand employees' intent. thus, it is essential to create a new dataset for hr application. on the other hand, collecting real datasets is difficult because the company cannot share these conversations with the public as it could leak employee confidential information. in response, we create an hr domain-specific dataset for llm hr agent. it is extractive, contains a long entity, is hr-specific, and contains empathetic conversations. we summarize our contributions as follows: ## methods our proposed data generation method is inspired by multiwoz @xcite . in multiwoz, two annotators played the roles of a user and a wizard. the user was given a specific goal in a certain domain (like booking a hotel), and the wizard, having access to a database, responded to the user's requests. however, it requires a lot of human labeling which is expensive. with recent advancements of llms (brown et al., 2020), we can use llms to replace humans in generating more diverse scenarios and rephrasing conversations. at a high level, our generation process includes developing expert-validated hr schemas, generating diverse user profiles, creating realistic scenarios via claude, randomizing and merging, rephrasing dialogues using claude, and applying extractive modeling using deberta model @xcite and human labeling to get high-quality labels. we chose claude because of cost and ethical reasons as explained in appendix g. for each step, we provide detailed instructions and human labeling guidance in the appendix. this makes our data genera-tion method easily transferable, reproducible and transparent. it took 2 days and costed 38.32 dollars for llms inference and 49.82 dollars for human labeling. this makes our method time and cost effective. the detailed data generation pipeline is in fig 1 and a provided example is in fig 2 .