Paper: Learning Multiplex Embeddings on Text-rich Networks with One Text Encoder (ICLR-2024)

Reviewer comment (Evaluation):
When conducting experiments, we usually encourage authors to repeat the same experiments multiple times and report both mean and standard deviation. However, I see mean results only, but not standard deviation, which is hard for readers to judge the significance of the outperformance over baselines.

Author response:
Thank you for the suggestion! We conduct a two-sample independent t-test to compare our method with the best baseline (in each column). Our method outperforms the best baseline method significantly with the p-value of that t-test less than 0.05. We will clarify this in our revision.

Reviewer comment (Experiments):
Insufficient baselines. There are models that work on text-rich networks with pretrained language models, such as GraphFormer [1] and LinkBERT [2]. Though they are designed not for multiplex networks, it is worth comparing against them to highlight the advantage of the multiplex network proposed by this submitted work.

Author response:
Thanks for pointing out the two baselines. We agree with the reviewer that it would be interesting to compare these methods, so we add the results on the Mathematics network and Sports network below (also added to Appendix A.12): 

Mathematics

| Model                      | cb | sa | sv | cr | ccb | Avg.|
| -------------------- | ----- | ----- | ---- | ----- | ----- | ----- |
| LinkBERT                   | 74.18 | 62.52 | 8.84  | 44.53 | 59.76 | 49.96 |
| GraphFormers               | 64.54 | 47.96 | 5.19  | 36.70 | 52.13 | 41.30 |
| Ours                       | 79.40 | 72.51 | 14.03 | 47.81 | 62.24 | 55.20 |

Sports

| Model                      | cop | cov | bt | cob | Avg.|
| ---------------- | ----- | ----- | --- | ----- | ----- |
| LinkBERT                   | 67.65 | 76.34 | 80.76 | 77.73 | 75.62 |
| GraphFormers               | 66.92 | 78.10 | 79.14 | 78.29 | 75.61 |
| Ours                       | 67.92 | 79.85 | 81.52 | 81.54 | 77.71 |


From the results, we can find that our method outperforms LinkBERT and GraphFormers consistently. The main reason is that these two baseline methods only generate one embedding for each text unit, expecting that all types of relations between texts can be captured by these single-view embeddings, which do not always hold in real-world scenarios. GraphFormers adopts the same graph propagation and aggregation module for edges of different semantics, which results in degraded performance. That being said, we agree with the reviewer that both GraphFormers and LinkBERT are related studies, we have cited these papers in our revision.

Reviewer comment (Writing):
Some math notations are not well defined, such as superscript $m$ at second line in page 4 and superscript $s$ at second last line in page 4. Authors are suggested to provide a table to summarize main mathematical notations used.

Author response:
Thanks for pointing it out. $m$ refers to the number of relation prior tokens used in relation-conditioned node embedding learning, and $s$ refers to the number of prior tokens used in the downstream task with learning to select. In our experiments, we find the model quite robust to $m$ and $s$. Both $m$ and $s$ are set as 5.

Reviewer comment (Experiments):
How to handle a previously unseen relation during testing phase?

Author response:
We have two ways to conduct inference for an unseen relation during the testing phase: 1) If we do not have any labeled data for the unseen relation, we can use direct inference with an evident source relation (proposed in Sec 4.2). A straightforward way is to select the source relation which has the closest semantics to the unseen relation. 2) If we have some labeled data, we can let METERN learn to select the source relations for the unseen relation inference (proposed in Sec 4.2).

Reviewer comment (Writing):
What does superscript $m$ mean and how to set a value for it? Same question for superscript $s$.

Author response:
Thanks for pointing it out. $m$ refers to the number of relation prior tokens used in relation-conditioned node embedding learning, and $s$ refers to the number of prior tokens used in the downstream task with learning to select. In our experiments, we find the model quite robust to $m$ and $s$. Both $m$ and $s$ are set as 5.

Reviewer comment (Presentation):
The paper could benefit from a more detailed and structured discussion of related works. Currently the line spacing is too small and more detailed discussions can provide more context for readers.

Author response:
We thank the reviewer for the comment. We have added the most relevant paper to our current related work section, but we are happy to add more related works to provide more context for readers. The newly added related works on “multi-task learning” and “learning on graphs for real-world applications” can be found in Appendix A.9. We also appreciate further suggestions for related work.

Reviewer comment (Theory):
The paper introduces "relation prior tokens" as a novel concept, but lacks thorough discussions of the computational complexity and scalability implications of this approach.

Author response:
We conduct the theoretical complexity analysis in Section 4.3 and the empirical complexity analysis in Section 5.6 (Table 6). From both the theoretical analysis and empirical results, we can find that: the time complexity and memory complexity of training METERN are on par with those of training baseline text encoders.

Reviewer comment (Experiments):
While the paper emphasizes the problem of multiplex representation learning in text-rich networks, it could benefit from real-world use cases or examples to illustrate the practical applications and impact of the proposed method.

Author response:
The METERN model can be deployed to real-world academic paper systems (e.g., Google Scholar) and e-commerce systems (e.g., Amazon). For the academic paper systems, we can use only one model (METERN) to conduct paper analysis from multiple perspectives (semantics, topic, authorship, etc) and achieve paper recommendation, venue recommendation, collaborator recommendation, and paper analysis. For the e-commerce systems, we can deploy METERN to analyze items from different aspects (category, brand, price, etc) and conduct item recommendations, brand recommendations, and item retrieval. Potential real-world scenarios also include the literature domain and legal domain where texts are associated with rich structure information from different aspects. A more detailed discussion of real-world use cases can be found in Section A.9 in the Appendix.

Reviewer comment (Experiments):
still there are few questions about claims on scalability (please see question section)

Author response:
**Scalability of multiplex representation learning.** 1) Training multiplex embeddings: Our method shares similar training time and memory complexity with text encoder baselines (Table 6). 2) Inference for downstream tasks: In practice, we do not need to save the multiplex embeddings for all documents offline. Instead, we only need to save the embeddings of the source relation which are useful for the downstream tasks. In addition, we can save the METERN parameters and conduct online inference (with our proposed two inference mechanisms) for downstream tasks.

Reviewer comment (Experiments):
the scalability of "learning-to-select-source-relations" for downstream tasks for novel relation types is unclear?

Author response:
**How does “learn-to-select” work for novel relations?** 1) If we have labeled data for this novel source relation, we can retrain the model for multiplex representation learning. Then we will obtain the relation prior embeddings for it for “learn-to-select” inference.  2) If we do not have labeled data for the novel relation but the relation is similar to one or several of our source relations, we can use the prior embeddings corresponding to the similar source relations for this novel relation for “learn-to-select” inference. 3) If we do not have labeled data for the novel relation and the novel relation is fully out of distribution compared with the source relations, there is essentially no signal in the available data to capture the semantics of this new novel relation and it should not be used in “learn-to-select” inference to guarantee performance.

Reviewer comment (Experiments):
how does text encoder handle the concept drift induced due to different relation or data distribution or domains?

Author response:
**Concept drift.** The concept drift does exist across different relations. As shown in Figure 6 in the Appendix, the raw data distributions of text pairs in different relations are quite different. The raw data distribution shift will result in the performance changes shown in Figure 7. In our methods, we use relation embeddings to capture the specific knowledge for different relation distributions and learn the shared knowledge across relations with the text encoder.

Reviewer comment (Experiments):
how to predetermine the number of relation types and corresponding relation prior tokens?

Author response:
**Number of relation types and prior tokens.** The number of *relation types* is predetermined by the network of interest (e.g., for e-commerce item networks, we can have four relation types: co-purchase, co-view, bought-together, and co-brand, according to user behavior and item features). The number of *relation prior tokens* is a hyperparameter in our model. In our experiments, we find the model quite robust to the number of prior tokens and we set it as 5.

Reviewer comment (Reproducibility):
In multiplex text-rich network, what is the span (context) of associated texts to compute node representation?

Author response:
**The span of associated texts.** The span of node-associated texts is a flexible choice in multiplex text-rich networks. Our method proposes a generic framework to learn multiplex embeddings and is not limited by the span of text. In our experiments, we use text spans that can be encoded by pretrained language models (less than PLM max length). For papers in academic networks, we use their title/abstract; For items in e-commerce networks, we use their title/descriptions. We can also choose other types of texts (e.g., content for paper) and we leave it for future study.

Reviewer comment (Novelty):
Transformers in essence are a special case of GNN - they semantically compute a different contextualized representation for each token (or multiple text tokens) in the text sequence and encode different relationship in different contexts. How does it differ with the proposed method of computing multiplex representation for texts for nodes?

Author response:
**Relationship between Transformers and METERN?** This is a great point. We would like to claim the difference between Transformers and METERN from three perspectives: 1) *Node (token v. document).* If we treat Transformers as a special GNN, each node in the graph will be a token and the framework will finally output representations for tokens. However, in METERN, each node corresponds to a document where the model outputs are document representations.  2) *Graph (token complete graph v. real-world semantic graph).* If we treat Transformers as a special GNN, the graph for Transformers will be a complete graph between token nodes, while in our case, the networks are real-world graphs where the edges contain semantic information. 3) *Condition (context v. relation).* If we treat Transformers as a special GNN, the framework will calculate the context-conditioned token (node) representations, while in our framework, we calculate the relation-conditioned document (node) representations.

Reviewer comment (Writing):
Here, the example are contrary to proposed method - multiplex representation for a token or text sequence for example for the word "llama", however, in the proposed method, node is representing a document, which is longer text and may consists of several thematic structures like topics. Please clarify.

Author response:
**The example of “llama”.** Thanks for the comments. The example of “llama” in the Introduction is supposed to claim the weakness of multiplex GNN methods, which represent the texts associated with each node as bag-of-words or context-free embeddings and are not enough to characterize the contextualized text semantics.

Reviewer comment (Experiments):
how does the multiplex representation learning approach scale as the number of relation scales between entities? Similarly, if many relations exists among entities, there are many representations of the same texts due to different relations. How does the approach scale in terms of handling multiplex embedding representations for the sake of leveraging overlapping semantics across relations?

Author response:
**How to handle overlapping semantics across relations.** Thanks for this great comment. 1) During training, the relation prior embeddings will capture the semantics of different relations. If two relations have overlapping semantics, it will be reflected in their relation prior embeddings (similar prior embeddings). 2) During inference, the “learn-to-select” mechanism will select related source relations for the downstream tasks. If two source relations have overlapping semantics, the “learn-to-select” mechanism will automatically learn appropriate weights by accounting for their semantic similarity.

Reviewer comment (Experiments):
On one hand, the method proposes to use only one text encoder to learn different relation embeddings however on other side, generating as many embeddings of the same text in different relations. Scalability is still challenging with this approach in real-world application?

Author response:
**Scalability of multiplex representation learning.** 1) Training multiplex embeddings: Our method shares similar training time and memory complexity with text encoder baselines (Table 6). 2) Inference for downstream tasks: In practice, we do not need to save the multiplex embeddings for all documents offline. Instead, we only need to save the embeddings of the source relation which are useful for the downstream tasks. In addition, we can save the METERN parameters and conduct online inference (with our proposed two inference mechanisms) for downstream tasks.

Reviewer comment (Theory):
How does the "learn-to-select-source-relations" inference work for novel relations on the downstream task? the approach seems to works with relations of known type or the downstream task has to select a relation from one of the source relations?

Author response:
**How does “learn-to-select” work for novel relations?** 1) If we have labeled data for this novel source relation, we can retrain the model for multiplex representation learning. Then we will obtain the relation prior embeddings for it for “learn-to-select” inference.  2) If we do not have labeled data for the novel relation but the relation is similar to one or several of our source relations, we can use the prior embeddings corresponding to the similar source relations for this novel relation for “learn-to-select” inference. 3) If we do not have labeled data for the novel relation and the novel relation is fully out of distribution compared with the source relations, there is essentially no signal in the available data to capture the semantics of this new novel relation and it should not be used in “learn-to-select” inference to guarantee performance.

Reviewer comment (Novelty):
The approach is very much prefix tuning applied to this problem of knowledge graph inference, so as such there is little novelty in the approach.

Author response:
Thank you so much for your thoughtful review!

Regarding your questions:
1. **Novelty**. *Focus*. We would like to argue that existing works [1] mainly focus on efficient model training, which is different from the focus of our work, which is to explore a new task of learning relation-conditioned document representations (i.e., multiplex representations on text-rich networks). *Techniques.* We propose a simple and effective multi-task learning method tailored for the multiplex nature of text-rich networks. The learned relation prior tokens capture the semantics of relations. These tokens can be flexibly used in the inference process either through manual selection, facilitating a plug-and-play functionality, or serving as prototype representations upon which mixup weights are automatically learned to quickly adapt to potentially intricate and unseen relations. *Applications.* The learned multiplex representations not only benefit relation prediction (link prediction corresponding to a specific relation) but also help various downstream tasks (shown in Tables 3,4, and 5) on the text-rich networks. We propose METERN to unify all task-solving with one model on multiplex text-rich networks.

Reviewer comment (Experiments):
Some of the baselines like multiplex GNN use weaker bag of word encoder so they are not directly comparable.

Author response:
2. **GNN baselines.** Thanks for pointing out the two baselines. We agree that it would be interesting to compare with a multiplex GNN equipped with pretrained text embeddings (MPNet-v2 is used here. We also tried BERT, and found that MPNet-v2 has better performance). We show the results on the Mathematics network and Sports network below (also added to Appendix A.10):

Mathematics

| Model | cb | sa | sv | cr | ccb| Avg.  |
|--------|-------|-------|-------|-------|-------|-------|
| DMGI (BOW)              | 46.55 | 42.62 | 6.11  | 27.80 | 38.87 | 28.85 |
| HDMI (BOW)              | 52.65 | 52.71 | 5.54  | 31.80 | 42.54 | 37.05 |
| DMGI (MPNet)            | 54.13 | 53.06 | 7.40  | 31.39 | 43.98 | 37.99 |
| HDMI (MPNet)            | 57.34 | 54.45 | 6.59  | 33.45 | 44.24 | 39.21 |
| Ours                    | 79.40 | 72.51 | 14.03 | 47.81 | 62.24 | 55.20 |

Sports

| Model| cop | cov | bt | cob | Avg.|
|---------|-----|-----|----|-----|-----|
| DMGI (BOW)             | 41.37 | 46.27 | 41.24 | 31.92 | 40.20 |
| HDMI (BOW)             | 45.43 | 61.22 | 55.56 | 52.66 | 53.72 |
| DMGI (MPNet)           | 43.37 | 63.69 | 58.16 | 51.72 | 54.24 |
| HDMI (MPNet)           | 43.12 | 62.65 | 57.88 | 51.75 | 53.85 |
| Ours                   | 67.92 | 79.85 | 81.52 | 81.54 | 77.71 |

From the results, we can find that the performance of multiplex GNNs with pretrained text embeddings is consistently better than the performance of multiplex GNNs with bag-of-words embeddings. However, our method still outperforms both baselines by large margins.

Reviewer comment (Experiments):
For other baselines, it's hard to see which baselines actually use a comparable encoder to the proposed model; ideally all baselines should have the same underlying encoder.

Author response:
3. **Text encoder.** We compare METERN with three types of baselines: 1) large-scale corpora finetuned text encoders, 2) multiplex graph neural networks, and 3) multi-relation learning language models. They have different philosophies to capture the multiplex relation semantics. The first category of methods is finetuned on very large text corpora, generating single embeddings for documents, and presuming one single embedding can capture the multiplex relation and have good generalization ability. The second category of methods learns multiplex node representations from a graph structure perspective. We show the result of this category with pretrained text embeddings in our previous answer. The third category of methods tries to capture different relation semantics with one text encoder and all these methods (including ours) adopt BERT-base as the initial text encoder for a fair comparison.

Reviewer comment (Experiments):
The approach is missing comparison with the simpler, no training required, baseline of adding a natural language description of the relation before encoding (instead of learned prefix embeddings). This approach is also directly applicable at inference time without requiring any additional training for unseen relations, and should potentially be applied with larger LM embeddings as well since there is no additional training needed.

Author response:
4. **Natural language description of relation + LM baseline.** Thanks for raising this baseline. We implement this baseline by adding a natural language description of the relation before encoding and using the large LM OpenAI-ada-002 (the best OpenAI embedding method) as the backbone text encoder. The results are shown below (added to Appendix A.11):

Mathematics
| Model  | cb | sa | sv | cr | ccb | Avg.|
| --------- | ----- | ----- | ---- | ----- | ----- | ----- |
| OpenAI-ada-002 (relation description) | 38.87 | 26.40 | 2.43  | 30.72 | 18.74 | 23.43 |
| Ours                    | 79.40 | 72.51 | 14.03 | 47.81 | 62.24 | 55.20 |

Sports
| Model | cop | cov | bt | cob | Avg.|
| ----------- | ----- | ----- | ---- | ----- | ----- |
| OpenAI-ada-002 (relation description)| 49.56 | 71.75 | 58.36 | 54.63 | 58.58 |
| Ours                   | 67.92 | 79.85 | 81.52 | 81.54 | 77.71 |

From the results, we can find that our method outperforms this baseline by a large margin, which demonstrates that the relation embeddings learned by our method to represent relations are better than directly prompting large LMs via natural language descriptions.