Paper:
As distributed approaches to natural language semantics have developed and diversified, embedders for linguistic units larger than words (e.g., sentences) have come to play an increasingly important role. To date, such embedders have been evaluated using benchmark tasks (e.g., GLUE) and linguistic probes. We propose a comparative approach, nearest neighbor overlap (N2O), that quantifies similarity between embedders in a task-agnostic manner. N2O requires only a collection of examples and is simple to understand: two embedders are more similar if, for the same set of inputs, there is greater overlap between the inputs' nearest neighbors. We use N2O to compare 21 sentence embedders and show the effects of different design choices and architectures.

Review:
The paper proposes N2O, a tool for probing the similarity among sentence embedders. Given two sentence embedders, N2O measures the amount of overlap of the k-nearest neighbor sets reported by the two embedders, averaged over a sample of probing queries. Cosine similarity is used as the similarity metric. The paper computes all-pair N2O scores for common sentence embedders and analyzes the results.
Overall, while the idea of probing similarity between embedders is interesting, the paper has the following weaknesses:
- The use of neighbor overlap to compare embedders has precedence in the context of word embeddings [https://www.aclweb.org/anthology/C16-1262/ | https://hal.archives-ouvertes.fr/hal-01806468/ | https://www.aclweb.org/anthology/Q18-1008/]. The methods in these works are mostly identical to N2O with some minor variations (e.g., using Jaccard distance).
- Using an existing method is justified if it provides new insights for the new setting. However, the results do not offer a lot of new insights. The fact that static embeddings, ELMo, and BERT give different neighbors is unsurprising. (The paper even admitted this.) Moreover, since sentence embedders are usually used as features when fine-tuned on downstream tasks, the importance of observations based on the pre-fine-tuned models is unclear. Contrast this with the work on word embedding similarity. Since some research areas (e.g., social science) directly use the similarity of word embeddings to conclude findings, the insights of how different word embeddings behave are more directly applicable.
- The paper does address some design choices, such as the number of neighbors and the number of probing queries, showing that different choices have little effects on the scores. However, the use of cosine similarity is problematic. As noted in the paper, some embedders such as ELMo were not trained on similarity objectives. BERT does have the next-sentence task, but the sparse attention-based architecture does not necessarily push similar sentences to have low cosine similarity.
Additional comments and questions:
- Instead of using the amount of overlap for a fixed k, it would be nice to have a metric that captures the whole distribution. For instance, maybe two embedders disagree on the first 20 neighbors, but end up retrieving the same set when considering 50 neighbors. The current overlap-based metric cannot capture such a phenomenon.
- Page 2: The method is technically not task-agnostic --- the task is sentence similarity with respect to a specific corpus.
- The paper tests variants of the same embedders by training on different corpora, with the conclusion that mismatched corpora give lower N2O scores. What is the N2O between two embedders of the same type trained on the same corpora but with different seeds?