Triple-Encoders: Representations That Fire Together, Wire Together (2024)

Abstract: Search-based dialog models typically re-encode the dialog history at every turn, incurring high cost.Curved Contrastive Learning, a representation learning method that encodes relative distances between utterances into the embedding space via a bi-encoder, has recently shown promising results for dialog modeling at far superior efficiency.While high efficiency is achieved through independently encoding utterances, this ignores the importance of contextualization. To overcome this issue, this study introduces triple-encoders, which efficiently compute distributed utterance mixtures from these independently encoded utterances through a novel hebbian inspired co-occurrence learning objective in a self-organizing manner, without using any weights, i.e., merely through local interactions. Empirically, we find that triple-encoders lead to a substantial improvement over bi-encoders, and even to better zero-shot generalization than single-vector representation models without requiring re-encoding. Our code (https://github.com/UKPLab/acl2024-triple-encoders) and model (https://huggingface.co/UKPLab/triple-encoders-dailydialog) are publicly available.

## introduction traditional search-based approaches in conversational sequence modeling like convert @xcite represent the entire context (query) in one context vector (see figure 1 ). this has two major drawbacks: (a) recomputing the entire vector at each turn is computationally expensive, and (b) it is difficult to compress the context's relevant information for any possible candidate response into a single vector. furthermore, the encoder models are limited to a maximum number of tokens, usually 512. curved contrastive learning (ccl) @xcite demonstrated that it is possible to encode 2020 ) and @xcite . similar to ccl we only need to encode and compute similarity scores of the latest utterance. at the same time, we achieve contextualization through pairwise mean-pooling with previous encoded utterances combining the advantages of both previous works. our analysis shows that the co-occurrence training pushes representations that occur (fire) together closer together, leading to stronger additive properties (wiring) when being superimposed (compared to @xcite ) and thus to a better next utterance selection. for the first time, in this paper we propose a method that contextualizes utterance embeddings in dialog sequences in a self-organizing manner, without the use of additional weights, i.e, merely through local interactions (in form of efficient vector algebra) between separately encoded utterances after appropriate pre-training. while previous work has shown that mean pooling is a strong method for sentence composition from tokens @xcite , we demonstrate that this can be generalized to a higher abstraction level: distributed pairwise sequential composition (illustrated in figure 1 ). to realize this, we present triple-encoders, which segment the context space of ccl into two distinct latent spaces denoting the relative order of utterances in the context. by linearly combining (averaging) representations from these sub-spaces through a co-occurrence learning objective, we create new contextualized embeddings that we can incorporate into ccl, resulting in contextualized curved contrastive learning (c3l). at inference time, our method efficiently contextualizes independently encoded utterances based on solely local interactions (without any additional weights): our method applies only (1) mean pooling, a (2) matrix multiplication for computing the similarity and one (3) summation (across the sequential dimension) operation to aggregate similarity scores. while we focus on modeling dialog in this paper, the sequential modularity of our method can in principle be used for any text sequence. pilot experiments on next sentence selection of children stories are reported in appendix e, while we leave thorough exploration to future work. our experiments are aimed at the following research questions: rq1: what is the effect of triple-encoder training (c3l) + triple encoder at inference compared to ccl? rq2: what is the effect of triple-encoder training (c3l) while encoding utterances at inference time without contextualization (like ccl)? our experimental results suggest that our approach improves substantially over standard ccl. notably, our method outperforms convert @xcite in a zero-shot setting while our method requires no additional learnable parameters for contextualization. while triple-encoder train-figure 2 : difficult example for next utterance selection based on solely independent utterances. here the model must know that both utterances occur together as it requires considering them jointly to derive the third utterance (in red). this is reflected by the significant gap in the normalized rank between our contextualized approach and the uncontextualized approach of @xcite . ## related work we will start with related work on embedding compositionality, conversational sequence modeling and self-organizing maps. next, we will describe retrieval methods that have been a motivation to our distributed representations. lastly, we will discuss ccl as the core foundation of our work. ## contextualized ccl via triple-encoders we design an extension to the ccl framework that addresses the aforementioned issue while retaining the same order of encoding complexity at inference as the bi-encoder model (o(n)), resulting in contextualized curved contrastive learning (c3l). to enhance the ccl embeddings with contextualization, two additional special tokens are added to the before space, [b1] and [b2]. these tokens denote the relative order of the utterances in the dialog, i.e. [b1]u i ∧ [b2]u j ⇔ @xmath0. to model the interaction between these different context utterances, we choose a simple mean operation as shown on the right of figure 4 . given that the [b1] and [b2] tokens create distinct representations, they're effectively projecting the utterance into two different embedding spaces. by combining utterance states of [b1] and [b2] through a mean operation, a new combined state that carries information from both original states is created. @xcite have shown that the curved scores (a linear decrease of similarity) outperform classic hard positives/negatives in sequence modeling and enhance sequential information that enables better planning capability. to keep the phenomenon of moving in a relativistic fashion through a continuous temporal dimension with respect to the after space, we construct the pairwise utterance mixture distance to a following utterance as the average of the two individual distances, as defined in ccl. formally, for window size w and 0 < @xmath1, if the distance between [b1]u i and [a]u k is 1 -k-i w , and the distance between [b2]u j and [a]u k is 1 -k-j w , then the joint representation of utterances in the before space should have distance norm(2 -2k-(i+j) w ). hence, we enforce for positive examples: ## application of curved contrastive learning this section describes how the bi-encoder and triple-encoder are used after training to solve the two previously introduced tasks @xcite sequence modeling (section 4.1) and short-term planning (section 4.2). note, when using the tripleencoder as bi-encoder, we use the same setup as ccl bi-encoders. ## experiments our experiments are conducted on the newly introduced gte (general-purpose text embedding) model @xcite as well as the robertabase models @xcite from the ccl paper @xcite ) which we will use as baselines. furthermore, we add a non-relativistic approach for sequence modeling evaluation, convert @xcite . apart from that, we investigate ablations to our introduced triple-encoders using the same special tokens but without the curved property of the temporal dimension akin to curved contrastive learning as well as every component separately. the sequence modeling models are trained and evaluated on two datasets, dailydialog @xcite and mdc @xcite , a task-oriented dialogue dataset. the models are also evaluated on zero-shot performance on per-sonachat @xcite . we will furthermore evaluate the triple-encoder on short-term planning proposed by @xcite where we evaluate our approach also with hits@k. to compare to previous work we use the same setup, with candidates generated with dialogpt @xcite (@xmath2.8, @xmath3.8) on history lengths of 2, 5, 10 with goal distances of 1, 2, 3, 4. our code including all hyperparameters can be found in our github repository. ## evaluation & discussion we start the evaluation with the sequence modeling performance of triple-encoder in section 6.1, followed by the short-term planning performance in section 6.2. we will address the research question rq1 by comparing triple-encoders trained with c3l to ccl bi-encoders. to answer rq2 we will furthermore compare to c3l bi-encoders (triple-encoder as bi-encoder), e.g training with c3l (section 3) but using the model at inference as bi-encoder by only using the [b2] token (section 4.1.1). we provide a comprehensive analysis on different training setups in the appendix b and an analysis of the contribution of every component in the triple-encoder setup during inference in sequence modelling (appendix c). in summary, we find that pre-training with ccl (which includes directional negatives) and then continuing training with triples yields the best performance. furthermore, all our model components bring a benefit. ## conclusion in this paper, we presented a novel approach for conversational sequence modeling, addressing the limitations of traditional methods such as convert. our triple-encoder leverages the concept of curved contrastive learning and enhances it by incorporating contextualization through a hebbian-inspired co-occurrence learning where representations that fire in a sequence together, wire together. this enables a more efficient and effective representation of dialog sequences without the need for additional weights, merely through local interactions, a first-of-its-kind approach that exhibits these selforganizing properties. as a result, our method outperforms single vector representation models on long sequences in zero-shot settings. our work demonstrates the distributed modularity of sequential representations by only mapping sequential properties within latent sub-spaces, i.e. all information is stored in the geometry of the latent space. for future work, we envision the exploration of triple-encoders for sequence modeling tasks other than dialog and story modeling. to encourage the community to contribute in this direction we release our model and open-source our code. 5325 building on @xcite work we face similar limitations. in particular we address in this section the random splitting of our dataset in shortterm planning, the use of synthetic data from llms to generate candidates replies, the generalizability to other datasets/tasks and response selection in the era of llms. splitting data for the short-term planning experiments: like in @xcite , our shortterm planning results are limited by the fact that we split at fixed positions in the dialog, which might not necessarily be planable. while this suggests that the models perform slightly better if planning were always possible, it offers an unbiased comparison between the different models. usage of synthetic data in short-term planning experiments: additionally, the candidates for this task are generated by a large language model (llm) where two issues can arise: (1) an utterance might lead to a goal that is not very likely given the context (see @xcite ) or (2) where the true utterance is out of distribution of the llm candidates and this true utterance can only reach the goal. datasets: one further limitation of our work is that our models are only tested on three dialog datasets and only one story generation dataset. response selection in era of llms: while large language models are becoming more and more popular in response generation, they still suffer from hallucinations @xcite , which is why retrieval is still popular, especially in legal and medical domains @xcite .