Dialogue Term Extraction using Transfer Learning and Topological Data Analysis (2022)

Abstract: Goal oriented dialogue systems were originally designed as a natural language interface to a fixed data-set of entities that users might inquire about, further described by domain, slots and values. As we move towards adaptable dialogue systems where knowledge about domains, slots and values may change, there is an increasing need to automatically extract these terms from raw dialogues or related non-dialogue data on a large scale. In this paper, we take an important step in this direction by exploring different features that can enable systems to discover realisations of domains, slots and values in dialogues in a purely data-driven fashion. The features that we examine stem from word embeddings, language modelling features, as well as topological features of the word embedding space. To examine the utility of each feature set, we train a seed model based on the widely used MultiWOZ data-set. Then, we apply this model to a different corpus, the Schema-guided dialogue data-set. Our method outperforms the previously proposed approach that relies solely on word embeddings. We also demonstrate that each of the features is responsible for discovering different kinds of content. We believe our results warrant further research towards ontology induction, and continued harnessing of topological data analysis for dialogue and natural language processing research.

## introduction dialogue systems are becoming increasingly popular as natural language interfaces to complex services. goal-oriented dialogue systems, which we see as the main area of application of the results presented here, are intended to be capable of conversing with a user to solve one or more tasks. they need to provide factual information and plan ahead over the course of multiple turns of dialogue. thus, they differ fundamentally from chat-based dialogue systems, which aim to engage the user in interesting conversation by offering entertainment. chatbased systems have been successfully trained using fully end-to-end approaches founded on large pretrained models @xcite @xcite . in contrast, state-of-the-art goal-oriented dialogue systems continue to rely on a pre-defined ontology: a database comprising domains (i.e., general topics for interaction), slots (constructs belonging to a particular topic), and values (concrete instantiations of such constructs) @xcite @xcite @xcite . consequently, state-of-the-art goal-oriented dialogue systems still have a high reliance on manual labour. firstly, the underlying ontology needs to be manually designed for each domain of conversation @xcite . secondly, the dialogue system needs to learn from a certain amount of dialogue data labelled with concepts from that ontology in order to recognize and understand these concepts in context @xcite . this manual annotation is again challenging, time-consuming and expensive @xcite . there is thus a strong need for methods that can automate ontology construction from raw data. moreover, ontology construction from raw dialogue data would have two-fold benefits: the dialogue data would be labelled automatically as the ontology is constructed, thus rendering any human involvement unnecessary. in this work, we concentrate exclusively on the first step of ontology construction: term extraction. the terms relate to regions of importance in the raw text. the subsequent steps of ontology construction, which we do not consider here, usually involve some form of clustering to boil down the extracted terms to a smaller number of concepts before they are finally organized into a full ontology. traditionally, term extraction begins by extracting terms based on frequency, in a way that aims to maximize recall @xcite . as frequency alone is a fairly primitive feature, this first step has close to zero precision and typically results in far too many terms. this makes further substantial filtering necessary within the term extraction step @xcite . filtering typically relies on heuristics or pre-existing natural language processing (nlp) models that have been trained on unrelated data, e.g., semantic parsers @xcite . heuristics as well as nlp models require substantial amounts of linguistic expertise to be created. in this work, we take a purely data-driven approach toward dialogue term detection to circumvent these limitations. the high dimensional data spaces arising from word embeddings are hard to understand and visualize. topological data analysis (tda) is a collection of mathematical tools which provides measurements of the geometry of highdimensional point clouds at various scales. the major advantage of topological features is their invariance under small deformations and rotations, as opposed to the coordinates of the embedding vectors. this leads to characteristics that are very generalizable and not dependent on the exact data set used for training. the utility of tda for nlp and dialogue modelling in particular are still underexplored. we believe that information that can be gathered using topological methods has considerable predictive power concerning term extraction, which to the best of our knowledge we exploit with this work for the very first time. starting from the approach of @xcite , we train a bio-tagging @xcite model on the widely used multi-woz @xcite data-set as the seed set by fine-tuning general purpose large pretrained language models. our bio-tagger accepts various features as input, all of which uniquely contribute to solving the task. we measure the zero-shot transfer ability of our proposed models on the schema-guided dialogue @xcite data-set, another well-established large-scale corpus for dialogue modelling. our contributions are as follows: ## related work it is normally assumed that the ontology is provided and built independently of the dialogue system. for instance, in information seeking dialogue systems, this would be a structured representation of the database. approaches to ontology learning from texts generally involve enriching a small ontology with new concepts and new relationships using text mining methods such as linguistic techniques and lexico-syntactic patterns @xcite , clustering techniques @xcite , statistical techniques @xcite and association rules @xcite . the majority of these methods require some form of human intervention. the potential of machine learning in this area has been demonstrated in the never-ending language learning (nell) project @xcite . nell learns factual knowledge from years of self-supervised experience in harvesting the web, using previously learned knowledge to improve subsequent learning. in the pipeline of knowledge base construction, term extraction is typically the first step. one example of a term extractor is presented in @xcite . it uses a part-of-speech (pos) tagger to select nouns, verbs and adjectives to which a number of heuristic frequency-based probabilistic models are applied to select term candidates. wordnet @xcite is employed to handle misspellings. a number of more recent methods for knowledge base construction start with a similar approach as @xcite . in @xcite we can also see heavy reliance on frequency, the use of dependency parsers in @xcite , as well as rules based on lexical and numerical features and the use of wordnet as in @xcite . a notable example of dialogue ontology induction is presented in @xcite , where a rule-based semantic parser is used as a starting point to propose an initial set of concepts. a more data-driven approach is presented by @xcite who proposed training a bio-tagger on finetuned contextual embeddings to induce slots. the approach is validated on multiwoz via leave-oneout domain experiments. we take this work as a starting point. in very recent work, @xcite propose ontology induction using language modelling attention maps and regularized probabilistic context free grammar to detect regions of interest in text, followed by clustering. this work is complementary to ours, and it would be interesting to explore its combination with our proposal. the 'beyond domain apis' track of the 9th dialog system technology challenge (dstc9) @xcite aimed to remove friction in task-oriented dialogue systems where users might issue a request that is out of a system's scope. while dstc9 aimed to integrate non-dialogue data into dialogue, none of the challenge submissions attempted ontology construction or expansion. topological data analysis remains largely underutilized in natural language processing. one notable exception is the work presented by @xcite . it shows that the wasserstein norm of degree zero persistence of punctured neighbourhoods in a static word embedding correlates with the polysemy of a word. tymochko et al. (2021) apply persistent homology to word embedding point clouds with the goal of distinguishing fraudulent from genuine scientific publications. their best performing model utilizes persistence features derived from time-delay embeddings of term frequency data. @xcite compute persistent homology of a filtered graph constructed from the attention maps of a pretrained language model and harness the features for an artificial text detection task. consider a discrete point cloud p ⊂ r m equipped with a distance such as the euclidean metric or the cosine distance. to apply topological tools to p , we need to turn p into a geometric space. one such 'geometrization' is the vietoris-rips complex vr ε , which produces, for each non- we investigate two fundamentally different feature sets to increase the generalization capability of models fine-tuned for bio-tagging. for each feature, we use a specific input projection and train a transformer followed by a token-level classification head. this architecture is illustrated in figure 2 . as the models extract different terms depending on the feature type they are trained on, we use the union of the predictions of all the tda models, respectively, of all the models, to obtain the final set of terms. one may also build a combined model using all features as joint input, however due to the nature of the training this would maximize accuracy and not recall. ## experiments we conducted experiments to answer the following questions: (1) is it possible to train a model on the seed data-set that achieves a high recall rate on the unseen ontology? (2) which of the proposed features is most valuable for that purpose? (3) what kind of concepts is the model able to find? note that we are mainly focusing on recall as evaluation measure, while retaining the f1-score of the baseline model. improvements in precision can be achieved with further post-processing, such as clustering @xcite . ## conclusion to the best of our knowledge, we present the first application of topological features in dialogue term extraction. our results show that these features distinguish content from non-content words, in a way that can be generalized from a training domain to unseen domains. we believe that these findings are only the tip of the iceberg, and warrant further investigation of topological features in nlp in general. in addition, we have shown that masked language modelling scores are useful for term extraction as well. in combination, the features we investigate allow us to make a significant step towards automatic ontology construction from raw data.