Local Topology Measures of Contextual Language Model Latent Spaces with Applications to Dialogue Term Extraction (2024)

Abstract: A common approach for sequence tagging tasks based on contextual word representations is to train a machine learning classifier directly on these embedding vectors. This approach has two shortcomings. First, such methods consider single input sequences in isolation and are unable to put an individual embedding vector in relation to vectors outside the current local context of use. Second, the high performance of these models relies on fine-tuning the embedding model in conjunction with the classifier, which may not always be feasible due to the size or inaccessibility of the underlying feature-generation model. It is thus desirable, given a collection of embedding vectors of a corpus, i.e. a datastore, to find features of each vector that describe its relation to other, similar vectors in the datastore. With this in mind, we introduce complexity measures of the local topology of the latent space of a contextual language model with respect to a given datastore. The effectiveness of our features is demonstrated through their application to dialogue term extraction. Our work continues a line of research that explores the manifold hypothesis for word embeddings, demonstrating that local structure in the space carved out by word embeddings can be exploited to infer semantic properties.

## introduction the prevailing approach to sequence tagging tasks such as named entity recognition or dialogue term extraction involves a two-step process: start with a general contextual vector representation for text sequences, for instance the embedding vectors created by a pretrained language model, then train a separate tagging model on top of the vector representations @xcite . optionally, assuming differentiability of the model and target function, one can fine-tune the representation model such that its embeddings are more suitable for the tagging task @xcite . while highly effective, the representations may be expensive to compute, and fine-tuning a language model is not always feasible, for instance if the underlying model is hidden behind an application programming interface (api). thus, it is desirable to develop tagging methods which achieve the best performance on the given representations. in fact, the performance of prompting-based approaches with large language models (llms) on named entity recognition tasks has lagged behind that of supervised sequence tagging approaches @xcite . additionally, this leads to problems such as hallucinations and potential dataset contamination, which prevent a fair evaluation. a more fundamental limitation of the prevailing paradigm is that the relation of a single input sequence to other sequences in the dataset cannot be taken into account. both the representation module and the tagging module commonly have a limited maximum context length. they cannot process the entire dataset at once, but rather need to be provided with single sentences or paragraphs at a time. the limited context can lead to suboptimal performance @xcite . for example, consider named entity recognition for an isolated sentence such as prince was prominently featured at the event. the word prince is ambiguous. in a corpus containing news articles, prince or prince harry likely appear in many articles related to the british royal family. in a different corpus, the term curry prince might appear frequently in the context of restaurant reviews. so only with regard to the entire corpus under consideration, an informed choice on how to tag prince in the example sentence could be made. in this work, we show that the relation between the representation of a single token and its containing corpus can be captured by studying the latent space -the collection of the language model's hidden states -surrounding the corresponding embedding vector. the geometry of these hidden states is known to capture both syntactic and semantic properties of the underlying text. for instance, @xcite find that distances between the contextual vectors of bidirectional encoder representations from transformers (bert) @xcite correspond to parse tree embeddings based on the grammatical structure of the input phrases. here, we study neighborhoods of embedding vectors from a topological viewpoint, and introduce descriptors of the shapes of these neighborhoods that are stable under symmetries such as permutations, translations, and rotations. in particular, we define descriptors based on persistent homology, a well-established tool of topological data analysis @xcite . ## application of local topology we perform a correlation analysis of local features and conduct a case study to explore the efficacy of local topology measures. specifically, we apply our proposed topological feature augmentations to the task of dialogue term extraction. ## conclusion in this work, we introduce a topological deep learning approach to enrich feature learning-based sequence tagging methods with contextual topological data. our methods do not depend on access to the underlying feature creation method, nor on external knowledge bases. once these local topological descriptors are computed, they offer the potential for reuse across different tasks, thereby mitigating the initial computational investment. one limitation lies in our method still requiring labels on the seed dataset. though our results in the case study hint at a correlation between dialogue terms and higher wasserstein norms, we have yet to establish a clear-cut purely feature based criterion for distinguishing terms from non-terms in dialogue data. looking ahead, we conjecture that the utility of our approach extends beyond the term extraction task investigated in this study. given its generic design and challenge, it is plausible that it is applicable to other language models and modalities. although our empirical evaluations have been confined to masked language models, the difficulty of the term extraction task provides optimism that our method could be advantageous in other scenarios where understanding the relation between individual data points and a datastore is critical. reproducibility statement the multiwoz2.1 and sgd datasets are publicly available through the convlab-3 unified data format @xcite , and we release our preprocessing, local topological feature computation and tagging model training code. 2