PAED : Zero-Shot Persona Attribute Extraction in Dialogues (2023)

Abstract: Persona attribute extraction is critical for personalized human-computer interaction. Dialogue is an important medium that communicates and delivers persona information. Although there is a public dataset for triplet-based persona attribute extraction from conversations, its automatically generated labels present many issues, including unspecific relations and inconsistent annotations. We fix such issues by leveraging more reliable text-label matching criteria to generate high-quality data for persona attribute extraction. We also propose a contrastive learning- and generation-based model with a novel hard negative sampling strategy for generalized zero-shot persona attribute extraction. We benchmark our model with state-of-the-art baselines on our dataset and a public dataset, showing outstanding accuracy gains. Our sampling strategy also exceeds others by a large margin in persona attribute extraction.

## introduction persona attribute extraction in dialogues (paed) is a crucial task for persona-based dialogue systems @xcite . it can extract persona attribute information from conversations. then, a dialogue system can use extracted persona attributes to generate personalized, userpreference-aware responses to user queries. previous works define the task as a sentence-level @xcite or utterance-level @xcite classification task. a model learns to classify whether a text contains persona information or not. however, the identified persona-informed texts are still unstructured, resulting in the lower utility of the extracted information in downstream dialogue systems, e.g., irrelevant contexts and different representations towards the same persona attribute. thus, we define persona attribute extraction as a triplet extraction task. a model should extract a subject, an object, and the persona-relevant relation linking the subject and object from utterances. the extracted attributes should be in the form of triplets (s, r, o), where the relation (r) indicates the persona attribute type of the subject (s) towards the object (o). although the existing relation triplet extraction (rte) task aims to extract triplets from documents @xcite , its framework cannot be directly transferred to paed task because the sentences in documents describe the facts or knowledge in the real world and each pair of entities in triplets can be connected by very limited relations. for example, entities 'eiffel tower' and 'france' are very likely connected by relation located_in in traditional rte datasets @xcite . but the subject and object in dialogues can be linked by many relations, causing hard sample problems, e.g., the relation between 'i' and 'my father' may be live_with, raised_by, get_along, etc. hence it is essential to formulate a framework for paed, which is capable of processing hard samples. to the best of our knowledge, @xcite proposed the largest persona attribute triplet extraction dataset in dialogues based on the triplet annotation of dialogue natural language inference (nli) dataset @xcite . however, we observe that the relation labels were not welldefined in the dialogue nli dataset and the dataset of @xcite that inherits the same label set, containing unspecific relation types. for example, negative expressions such as never, and don't have, are collectively categorized as the relation type of other in both datasets. dialogue nli missed many triplet labels, resulting in inconsistent annotations, while the dataset of @xcite introduced considerably less reliable labels because utterances and triplet labels were automatically paired by greedy rules. motivated by addressing the unspecific and inconsistent annotation issues of dialogue nli and avoiding the unreliable label-utterance pairing of @xcite , we aim to deliver a rigorous dataset for paed. we source data from dialogue nli and per-sonachat @xcite datasets, forming a new dataset, termed personaext. we manually correct 1896 triplet labels of the original dialogue nli dataset to improve specificity. we use a more conservative strategy to assign triplet labels to utterances to improve label reliability and consistency: only the triplet selected by both trained classifiers @xcite and term frequency-inverse document frequency (tf-idf) is assigned to the utterances. we conduct a human evaluation on personaext and the dataset of @xcite . it shows improvements of person-aext in label specificity and annotation accuracy. we formulate paed as a generalized zero-shot learning (gzsl) task because it is common that the training utterances for a model cannot cover all the relation types. the hard sample issue becomes more severe in gzsl setting. thus, we propose a generation-based framework with a novel hard negative sampling (hns) strategy for zero-shot paed. our hns strategy consists of a meta-vae sampler and a contrastive structured constraint (csc). meta-vae sampler uses |r| latent variables of variational autoencoder (vae) @xcite to represent |r| kinds of utterance distributions under |r| different relations. it pairs an utterance under a certain relation with one under another relation as positive and hard negative samples, if the distance between their distributions is the shortest. csc is designed to disperse the paired samples in semantic vector space. on average, our framework surpasses the strongest baseline on our personaext by 1.06% and on the public fewrel dataset @xcite by 0.8% (single triplet) & 3.18% (multiple triplets); our meta-vae sampler exceeds others @xcite @xcite in paed by 2.66%. the main contributions of this work are: (1) we develop a paed dataset, personaext, with 1,896 re-annotated triplets and 6,357 corrected utterancetriplet pairs. (2) we present a generation-based framework for zero-shot paed. a novel hns strategy, meta-vae sampler with csc, is presented to enhance the performance of our model. (3) our model achieves better results than strong baselines in zero-shot paed and negative sampling. our code and data are publicly available foot_0 . ## personaext construction our personaext dataset is developed for paed, constructed from multi-turn chitchat datasets, e.g., personachat @xcite and dialogue nli @xcite . we use personachat as the data source because it is a dialogue corpus with many personal profiles. personachat was built by two crowd-workers to chat with each other, conditioned on predefined personas, e.g., food preference, job status, and education. each persona includes 4 to 6 persona sentences. the dataset contains 1,155 personas with over 10,907 dialogues. dialogue nli annotated triplet (s, r, o) for dialogue utterances u and persona sentences p in per-sonachat. they generated entailment, neutral, or contradiction labels for pairs (p, u) and (p, p) based on annotated triplets. for instance, i adopted a cat and i have a cat named alfred are both labeled with a triplet (i, have_pet, cat). then, they are considered as entailment; sentences with different triplets are regarded as neutral or contradiction. however, many utterances in dialogue nli do not have triplet labels, although the utterances contain persona information. @xcite employed a greedy method to improve the label coverage of dialogue nli. a triplet label of persona sentence p or utterance u i is assigned to another utterance u j if an entailment relationship of (p, u j ) or (u i , u j ) is predicted by either bert (kenton and toutanova, 2019) or tf-idf classifiers. the triplets of dialogue nli and the dataset of @xcite are somewhat unreliable, having issues in the unspecific label set definition and inconsistent utterance-triplet pairing. thus, in our personaext construction, an automatic intersection assignment strategy and manual attribute triplet label correction are used to improve label qualities. ## generalized zero-shot paed we propose a framework for paed in generalized zero-shot learning (gzsl) setting @xcite . the framework consists of two parts: a persona attribute generator (pag) and a persona attribute extractor (pae). pag is trained to generate synthetic dialogue utterances containing persona descriptions. pae is trained on the synthetic data and extracts attribute triplets of unseen target data. pae is a pretrained language model (plm) based extractor enhanced by our proposed meta-vae sampler with csc loss. ## conclusion in this work, we studied generalized zero-shot learning for persona attribute extraction in dialogues (paed). we first built personaext based on personachat and dialogue nli through a semiautomatic annotation framework, yielding consistent and specific triplet labels. then we proposed an effective and interpretable meta-vae sampler with csc loss to process hard negative samples and incorporated it into pae for generalized zeroshot paed task. empirical results demonstrate that our framework surpasses the strongest baseline by a large margin. a visualized quantitative analysis provides a thorough explanation for the mechanism of our meta-vae sampler and csc in sample representations.