Paper: Meaning Representations from Trajectories in Autoregressive Models (ICLR-2024)

Reviewer comment (Writing):
The authors define a 'syntactic meaning representation' as the function $M_s$. I'm unclear as to why it is named 'syntactic meaning representation' (I'm also unsure why 'syntactic' is included). It is simply the conditional probability of the prompt. I don't find it to be a useful 'representation' for training or other applications like vector-space representation. In fact, they merely use the divergence between the conditional probabilities to measure text similarity. If this approach hasn't been taken by others, then this divergence could be considered novel. Thus, I cautiously suggest they use 'inverse perplexity mapping' rather than 'meaning representation'. The paper's title could then be 'A New Measurement for Sentence Similarity via Sampling Trajectories in Autoregressive Models'. Furthermore, 'semantic representations' appear suddenly following Algorithm 1. I recommend they use 'inverse perplexity mapping for substrings' instead of 'semantic representations'.

Author response:
>**The authors define a 'syntactic meaning representation' as the function $M_s$. I'm unclear as to why it is named 'syntactic meaning representation' (I'm also unsure why 'syntactic' is included).**

We included “syntactic” to stress that our definition of meaning representation is completely determined by syntactic/formal properties of natural language (_i.e._, what strings can extend other strings), as opposed to being defined in terms of external referents. It is also consistent with standard definitions in the theory of formal languages (https://en.wikipedia.org/wiki/Syntactic_monoid) which capture the spirit of our perspective. 

>**It is simply the conditional probability of the prompt**

Since we define this more generally to be the score distribution over all possible continuations of the given prompt, the score attribution mechanism is not required to be the conditional probability distribution over these trajectories. In our experiments, we use inverse perplexity, which is also not an actual probability (since does not sum to 1). 

The reason we choose this more general definition is to allow composition operations in this space of meanings that might not be obtainable from any individual prompt. One example, which we use when measuring asymmetric relations, is $M_u \land M_v$. The resulting distribution will in general not be the “conditional probability” of any particular prompt, yet resides in the same geometric space which allows computation of distances such as $d(M_u \land M_v, M_v)$.

>**I don't find it to be a useful 'representation' for training or other applications like vector-space representation.**

Our representations attain strong results on downstream tasks. In contrast, previous methods for extracting representations from autoregressive models are less effective, as shown in our experiments.

In general, there is no formal difference between the expressive power of our representations and vector-space representations obtained from traditional embedding models like CLIP: any task that can be performed using CLIP embeddings can also be accomplished with ours (albeit with greater computational cost). Indeed, any method that measures semantic similarity between arbitrary pairs of input texts actually provides an implicit semantic representation (which could potentially be made explicit via multidimensional scaling methods). It can also be used for classification tasks directly using kernel methods. Furthermore, as argued above, our representation is also naturally interpretable, since coordinates correspond to trajectories.

Lastly, our experiments on the SICK-R dataset in the revised draft also show that our representations handle compositional knowledge even better than vector-space representations from contrastive trained models. 

>**In fact, they merely use the divergence between the conditional probabilities to measure text similarity. If this approach hasn't been taken by others, then this divergence could be considered novel.**

Divergence between conditional probabilities of next-token predictions is a natural baseline that is commonly used in existing fine-tuning or prompt-based methods. In contrast, we compute the distance between score distributions over trajectories as a similarity measure, which has never been adopted by any prior work to the best of our knowledge.

>**The paper's title could then be 'A New Measurement for Sentence Similarity via Sampling Trajectories in Autoregressive Models'.**

We appreciate the reviewer’s suggestion, but we believe our current title is a more faithful representation of our work for the abovementioned, and the following reasons:

1. We propose a working definition of meaning representation for autoregressive models. Quantifying semantic similarity is just one out of the applications enabled by our proposed definition. We demonstrate its versatility by applying it to tasks involving asymmetric relations, such as quantifying entailment between sentences and hypernym/hyponym relations between words.

2. The applicability of our method extends beyond sentence comparison, as it was successfully used with a broader range of autoregressive models, including vision-language models. 

3. We do not compare sampled trajectories, but rather compare distributions over trajectories. This distinction is crucial, particularly when comparing meaning representations of words, which we define as the distribution of scores over contexts in which the words appear. Hence even though such contexts cannot be sampled directly, our proposed meaning representation remains well-defined and applicable, allowing for alternative methods of obtaining trajectories such as context retrieval from a corpus.

Reviewer comment (Reproducibility):
For the final version, sharing the code on GitHub would be beneficial for readers.

Author response:
>**For the final version, sharing the code on GitHub would be beneficial for readers.**

We agree and indeed plan to release our code alongside the final version.

Reviewer comment (Experiments):
The similarity distance depends on the performance of the autoregressive model. If the model is fine-tuned, e.g., for a chatbot, then how would the similarity change?

Author response:
>**The similarity distance depends on the performance of the autoregressive model. If the model is fine-tuned, e.g., for a chatbot, then how would the similarity change?**

Great question! We believe that fine-tuning primarily focuses on syntactic alignment with human instructions rather than semantic understanding. For instance, a model trained to output "2" more frequently than "3-1" in response to "1+1=" might not necessarily indicate an improved understanding of the equation but rather an enhanced ability to format its output to match human expectations for a chatbot. While fine-tuning does alter meaning representations, it does not guarantee better alignment with human annotations on semantic similarity tasks. This is demonstrated in Appendix B, Table 6, where we compare LLaMA-13B with its instruction-tuned and RLHF variants – Vicuna and StableVicuna, respectively – revealing that all three models exhibit comparable performance on semantic similarity tasks.

Reviewer comment (Writing):
In the first sentence of the second paragraph under **Meaning representation for prompts**, I think $M_u$ should be changed to $M_s$ and $t\in \mathcal{A}^1$ should be changed to $t\in \mathcal{A}^*$.

Author response:
>**In the first sentence of the second paragraph under Meaning representation for prompts, I think $M_u$ should be changed to $M_s$ and $t \in A^1$ should be changed to $t \in A^\ast$**

$M_u → M_s$: Thank you for catching this, we have updated our draft.

$t \in A^1$: This is intentional, since we were referring to a special case of Equation (1) where the maximum length of the sampled trajectories is restricted to 1 token.

Reviewer comment (Theory):
In the second sentence of the paragraph **Containments of semantic representations**, the definition of 'partial order' should be revised. Since $\sum_{\mathrm{len}(t) = m} M_u(t)^{m} = \sum_{\mathrm{len}(t) = m} M_v (t)^{m} = 1$, we cannot have $M_u(t) < M_v(t)$ for all $t\in \mathcal{A}^*$. Please update the definition of 'partial order.'

Author response:
>**In the second sentence of the paragraph Containments of semantic representations, the definition of 'partial order' should be revised. Since $\sum_{\mathrm{len}(t) = m} M_u(t)^{m} = \sum_{\mathrm{len}(t) = m} M_v (t)^{m} = 1$, we cannot have $M_u(t) < M_v(t)$ for all $t \in A^\ast$. Please update the definition of 'partial order.'**

Thank you for pointing this out, we clarified our definition in the revised draft. Indeed, this is precisely the reason for leveraging the soft notion of containment $d(M_u \land M_v, M_u)$ to quantify this relation. Note that the composition $M_u \land M_v$ also lives in the same space of meanings, and the partial ordering $M_u \land M_v \leq M_u$ and  $M_u \land M_v \leq M_v$ always holds by definition.

Reviewer comment (Reproducibility):
What is the temperature $\lambda$ in the experiment? Is it the temperature parameter used during the inference process in the autoregressive model? Please clarify this.

Author response:
>**What is the temperature $\lambda$ in the experiment? Is it the temperature parameter used during the inference process in the autoregressive model? Please clarify this.**

Yes, we have updated our draft to clarify this, thank you.

Reviewer comment (Experiments):
I believe we should use the training set for the model to compute $\overline{M}_u$. Does the text corpus WikiText reflect the distribution of the training set?

Author response:
>**I believe we should use the training set for the model to compute $\overline{M}_u$. Does the text corpus WikiText reflect the distribution of the training set?**

Due to limitations in accessing the full data distributions used to train many of the large language models employed, we utilized WikiText as our corpus. We empirically found that it is sufficiently large, and contains enough contexts to evaluate the meaning representation of the words we tested. Nevertheless, we concur with the reviewer's suggestion that larger and more relevant corpora would further improve evaluation.

Reviewer comment (Reproducibility):
It has a relatively rigid and narrow use case, since this method can only be used for pairwise comparison and since it's not obvious how to fine-tune it.

Author response:
>**It has a relatively rigid and narrow use case, since this method can only be used for pairwise comparison** 

Our representations can actually be used in much the same way as traditional embedding models such as CLIP or BERT (albeit with a greater computational cost), for example for tasks like retrieval or classification. Moreover, our representation has the additional benefit of interpretability, since unlike most embedding models, the “coordinates” used have meaningful interpretations as text continuations (trajectories).

>**it's not obvious how to fine-tune it.**

We note that our measure of similarity can be directly used with existing contrastive learning-based fine-tuning methods. Typical contrastive learning methods are limited to aligning next/mean-token predictions, which we show in our experiments are actually an inadequate representation of meaning. 

Based on this, we believe that contrastive fine-tuning with our measure of pairwise similarity could be used to better align the meaning spaces of autoregressive models with humans. Indeed, while contrastive fine-tuning methods typically sacrifice the language modeling capabilities of the LLM by converting it into an embedding model, we hypothesize that fine-tuning using our method can preserve generation abilities given appropriate regularization.

While fine-tuning models for better alignment with humans are beyond the scope of our paper, we appreciate the reviewer’s suggestion and agree that it represents a valuable avenue for future research.

Reviewer comment (Evaluation):
The work frames it as producing "interpretable" vectors, but the work was somewhat lacking in an actual exploration of that interpretability.

Author response:
>**The work frames it as producing "interpretable" vectors, but the work was somewhat lacking in an actual exploration of that interpretability.**

Unlike traditional vector-based embeddings, where the interpretation of individual vector components is elusive, each component of our method's representations corresponds to a specific text continuation, or trajectory. This is illustrated in Figure 1 of our paper, in which each bar corresponds to the score attributed to a distinct trajectory. This provides a way to interpret the difference in meaning between sentences based on the trajectories that differentiate them. For example, in Figure 1, we observe that the sentences "A man is playing a flute" and "A dog is barking at a fly" differ because "The dog's owner is telling him to stop" is a relevant continuation for the latter but not for the former.

Reviewer comment (Experiments):
I liked the idea of the entailment and hypernymy work, but it felt a bit convoluted: the way they approached both tasks seems to have lead to them comparing to weak baselines, despite NLI and wordnet link detection being well-explored areas.

Author response:
>**I liked the idea of the entailment and hypernymy work, but it felt a bit convoluted**

Our proposed evaluation methods serve as a proxy to assess the effectiveness of our entailment and hypernym/hyponym definitions. The original definitions are more general, providing a quantitative measure of the strength of these relationships between any two inputs. However, the absence of human-labeled datasets quantifying the degree of entailment between sentences or the hyponym relation between words necessitates an alternative approach, which our evaluation methods aim to address.

>**the way they approached both tasks seems to have lead to them comparing to weak baselines, despite NLI and wordnet link detection being well-explored areas.**

Despite the well-established nature of these tasks, they are rarely evaluated on pre-trained autoregressive models in a zero-shot setting due to the poor performance of existing methods, as rightly pointed out by the reviewer. Our findings corroborate this observation, demonstrating that existing baselines fail even for binary prediction of these relations under such settings. Our method effectively tackles this challenge.

Reviewer comment (Experiments):
As mentioned above: do the authors feel that this would be viable to fine-tune, or is it limited entirely to zero-shot STS settings (plus entailment/hypernymy)?

Author response:
>**As mentioned above: do the authors feel that this would be viable to fine-tune, or is it limited entirely to zero-shot STS settings (plus entailment/hypernymy)?**

Fine-tuning is definitely viable for the goal of aligning the meaning space of models with that of humans, and an idea for doing so using our method was described above. We would love to see future work exploring this exciting direction.

It is, however, beyond the scope of this paper, since fine-tuning disrupts the meaning representation attributed by the original model, which was the subject of investigation of our work.

We also present additional experiments on SICK-R in the revised draft, showing that even in the zero-shot setting, our representations effectively handle compositional knowledge, achieving even better performance than contrastive trained models.

Reviewer comment (Theory):
Wouldn't the general hypernymy assumptions often be violated under negation?

Author response:
>**Wouldn't the general hypernymy assumptions often be violated under negation?**

In general, it is indeed not true that there exists an implication/hypernym relation between every pair of sentences/words. For instance, as the reviewer rightfully pointed out, negation can alter the existing relations between a pair of words or sentences. However, our proposed representation quantifies how strongly the particular relation holds via the distance $d(M_u \land M_v, M_v)$. Hence, although we used this assumption to test whether our representations can capture asymmetric semantic relations, our definitions remain applicable more broadly.

Reviewer comment (Experiments):
For the baseline encoder-only models, it is better to include larger models like BERT-large and RoBERTa-large using their CLS tokens and token averages.

Author response:
Thank you for the suggestion, we added the suggested comparisons to Table 1.

Reviewer comment (Theory):
The discussion about partial ordering between sentences is a bit puzzling. Since Tu and Tv are samples from u and v respectively, the former set of trajectories usually gives high Mu values for u and vise versa, so it hardly happens that Mu < Mv or Mu > Mv for all t in Tu U Tv. Besides, the discussion of entailment suddenly shifts from the comparison between Mu and Mv to the comparison between d(Mu cup Mv, Mu) and d(Mu cup Mv, Mv). This part is also quite puzzling.

Author response:
Indeed $M_u \leq M_v$ or $M_v \leq M_u$ rarely happens in practice. This is precisely the reason we employ the alternate formulation pointed out by the reviewer. Note that $M_u \leq M_v$ can also be expressed as $d(M_u \land M_v, M_u) = 0$, where $M_u \land M_v := \min(M_u, M_v)$. Intuitively, this means that $M_u \leq M_v$ if and only if $M_u$ is fully captured by their intersection. The difference now is that this expression provides a continuous measure of how strongly the containment relation holds. We added some discussion in the main body of the paper and in Appendix F which we hope can help better clarify this. This also relates to the next comment by the reviewer:

Reviewer comment (Experiments):
For the experiments of entailment and hypernym/hyponym evaluation, there is an assumption that either of those relations exists between the given input pairs, which is not realistic.

Author response:
We chose to assume that a entailment/hypernym relation exists and to formulate the task as prediction of the direction of that relation in order to present a simple quantitative evaluation of our definitions. Indeed, we are not aware of datasets that provide human-labeled annotations quantifying how strongly these relations hold between two arbitrary inputs. In general, our method actually produces a single number that quantifies how strongly each relation holds (even if neither strictly holds), measured by the distance described above.

Reviewer comment (Experiments):
As is pointed out in the limitation paragraph, the proposed model is computationally higher than other baseline models. In the Appendix, it is tested using a fixed set of trajectories, which seems to cause a big performance degradation.

Author response:
While computational cost is indeed a limitation of our method, it can be effectively mitigated by employing fewer and shorter trajectories. As evident in Figure 3 of Appendix A.1, performance starts to saturate with just 10 trajectories of maximum 10 tokens each.

Reviewer comment (Reproducibility):
For the Hyponym test, the contexts for a given word u are not sampled but retrieved from. It is not clear how the value Mu(s u t) is obtained.

Author response:
We elaborated on this in Appendix C.2, Algorithm 2 of the Appendix. In particular compared to sentence-level meaning, there are two main differences: (1) as correctly pointed out, contexts are retrieved rather than sampled, since “reverse-sampling” the prefixes for any given word is not trivial, and (2) the scores for each context are computed over the entire context, rather than only over the continuations following the word.

Reviewer comment (Reproducibility):
Is it guaranteed that the set of trajectories for an input u always the same? Or, does it vary each time sampling for a sentence u is conducted?

Author response:
This depends on the sampling method. Beam search generates a fixed set of trajectories, but multinomial sampling, which we use in our experiments, will generate different trajectories based on the random seed. However, we experimentally found that this variance in trajectories sampled across different seeds barely affects results.

Reviewer comment (Experiments):
The Autoregressive Model Baselines look very weak. I am not sure they are worth being included.

Author response:
It is not immediately obvious that those baselines do not perform well in a zero-shot setting. Most of the baseline methods are also used as components by other works such as SGPT and Sentence-T5, in both the zero-shot and fine-tuned settings. As such, even though they perform poorly, we believe that a comparison with those baselines is helpful for completeness and to be consistent with relevant prior work.