Paper: Topic Modeling as Multi-Objective Contrastive Optimization (ICLR-2024)

Reviewer comment (Evaluation):
The purpose of multi-objective optimization is to find a (possibly diversified) set of Pareto optimal solutions so that it represents the Pareto frontier. Therefore, I would like to know the average and standard deviation of the $\alpha$ obtained at the end of each run to see how it is diversified. If it is not diversified, then I would like to know the reason.

Author response:
**Q3: The average and standard deviation of $\alpha$.**

We observe that the values of $\alpha$ vary from the starting epoch to the final epoch. In particular, in the beginning, the mean and standard deviation of alpha are about 0.001. This means that in the beginning, it is better to focus on optimizing the ELBO objective of the neural topic model. 

Gradually the value of $\alpha$ increases and converges to approximately 0.3 in the last epoch. This validates the usefulness of our setwise contrastive learning that gives beneficial mutual information signal to topic modeling. 

We specify the values of $\alpha$ after epochs for 20NG and IMDb datasets in the following table. **We have added $\alpha$ values for all other datasets to Appendix J in our rebuttal revision.**

| |              |         |            |          |          |          |
|---------------|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|:-------------:|
| **Dataset/Epoch** |       **0**      |      **100**      |      **200**      |      **300**      |      **400**     |      **500**      |
| 20NG          | 0.002 ± 0.001 | 0.003 ± 0.001 | 0.081 ± 0.005 | 0.126 ± 0.008 | 0.159 ± 0.002 | 0.292 ± 0.003 |
| IMDb          | 0.001 ± 0.001 | 0.008 ± 0.002 | 0.096 ± 0.008 | 0.143 ± 0.002 | 0.178 ± 0.009 | 0.297 ± 0.003 |

Reviewer comment (Writing):
Judging from the definition of $\mathcal{L}_{set}$ in Algorithm 1, both the positive and the negative pairs are generated within the same set and the loss is then accumulated for all the sets. If this is the case, then the first sentence of the Hard Negative Sampling paragraph sounds confusing. Differentiating among sets may be more difficult, but the contrastive pair for differentiating is generated within a set not among sets. In Algorithm 1, $\exp(s_i^{min}, s_i^{+})$ should be $\exp(f(s_i^{min}, s_i^{+}))$.

Author response:
**Q4: The first sentence of the Hard Negative Sampling paragraph**

Sincerely thank you for your note. We have provided two modifications for Algorithm 1 in our rebuttal revision to more accurately describe our method:

- Include the iteration $p$ that loops over the permutation matrix: We diversify our document sets by permuting the document index within a batch. **We have added the index $p$ to the formula using MaxPool and MinPool operations to reflect this (line 10-13 in Algorithm 1)**.

- Modify the contrastive loss: In our implementation, we consider a negative pair as a document set $i$ with all other document sets $j$ which are generated based on the permutation matrix $M$ (except for the set which includes corresponding augmented documents of set $i$). **We have corrected the denominator of our contrastive loss to reflect this (line 16 in Algorithm 1).**

Therefore, as in the first sentences of our hard negative sampling paragraph, we put a document into multiple sets and compare among them. This would create hard negative samples and benefit contrastive learning. 

**Q5: In Algorithm 1, $\text{exp}(s_{i}^{min}, s_{i}^{+})$ should be $\text{exp}(f(s_{i}^{min}, s_{i}^{+}))$**

Thank you so much for your note. **We have edited this detail in Algorithm 1 (line 16) of our rebuttal revision.**

Reviewer comment (Novelty):
Since [1] also uses contrastive learning to capture useful semantics of topic vectors which is similar to the proposed method, this paper does not clearly compare with [1] and explain its novelty.

Author response:
We clarify that the main difference between our work and TSCTM (Wu et al., 2022 [1]) lies in that their work is restricted to short texts, *e.g.* TagMyNews dataset (mean length: 8 words, max length: 17 words), and GoogleNews (mean length: 5.38 words, max length: 12 words). In contrast, we work with texts of multiple lengths, *e.g.* AGNews (mean length: 20.19 words, max length: 76 words), 20NG (mean length: 152.21 words, max length: 12419 words), and Wiki (mean length: 130.14 words, max length: 640 words).

We also compare our model with TSCTM in the settings of $T = 50$ topics:

|            |   20NG   |        |   IMDb   |        |   Wiki   |        |  AGNews  |        |
|------------|:--------:|:------:|:--------:|:------:|:--------:|:------:|:--------:|:------:|
| **Method** | **NPMI** | **TD** | **NPMI** | **TD** | **NPMI** | **TD** | **NPMI** | **TD** |
| TSCTM      |   0.271  |  0.668 |   0.149  |  0.741 |   0.327  |  0.753 |   0.252  |  0.783 |
| Our Model  |   0.340  |  0.913 |   0.291  |  0.916 |   0.496  |  0.959 |   0.351  |  0.934 |

We see that our model outperforms TSCTM on both topic diversity and topic coherence. We believe the reason is that our setwise contrastive learning captures more mutual information than TSCTM in documents of various lengths. Thank you for your comment. **We’ve added these to Table 4 and Table 5 of our rebuttal revision.**

Reviewer comment (Experiments):
This paper omits important baselines. For example, [1] also presents great performance in this task but this paper does not compare with it in the experiments. Which contrastive learning method performs better?

Author response:
We clarify that the main difference between our work and TSCTM (Wu et al., 2022 [1]) lies in that their work is restricted to short texts, *e.g.* TagMyNews dataset (mean length: 8 words, max length: 17 words), and GoogleNews (mean length: 5.38 words, max length: 12 words). In contrast, we work with texts of multiple lengths, *e.g.* AGNews (mean length: 20.19 words, max length: 76 words), 20NG (mean length: 152.21 words, max length: 12419 words), and Wiki (mean length: 130.14 words, max length: 640 words).

We also compare our model with TSCTM in the settings of $T = 50$ topics:

|            |   20NG   |        |   IMDb   |        |   Wiki   |        |  AGNews  |        |
|------------|:--------:|:------:|:--------:|:------:|:--------:|:------:|:--------:|:------:|
| **Method** | **NPMI** | **TD** | **NPMI** | **TD** | **NPMI** | **TD** | **NPMI** | **TD** |
| TSCTM      |   0.271  |  0.668 |   0.149  |  0.741 |   0.327  |  0.753 |   0.252  |  0.783 |
| Our Model  |   0.340  |  0.913 |   0.291  |  0.916 |   0.496  |  0.959 |   0.351  |  0.934 |

We see that our model outperforms TSCTM on both topic diversity and topic coherence. We believe the reason is that our setwise contrastive learning captures more mutual information than TSCTM in documents of various lengths. Thank you for your comment. **We’ve added these to Table 4 and Table 5 of our rebuttal revision.**

Reviewer comment (Reproducibility):
I have a question about the baselines. Table 5 shows topic diversity (TD) score of WeTe is 0.878±0.012 when T = 50 in AG News dataset. However, in [1], this score is 0.966 without data augmentation and 0.991 with data augmentation. There is a large gap between these scores. I wonder how authors implement these baselines. Could you please provide more details or any codes?

Author response:
We clarify that to measure the topic diversity, TSCTM [1] and WeTe [2] extract 25 top words in each topic, while we only use 10 top words, which would make the evaluation harder. If we follow strictly follow their protocol, most of the topic diversity values will be ~0.999, which cannot discriminatively evaluate the topic models.

Reviewer comment (Evaluation):
Statistical significance tests are missing. It is unclear whether the gaps between the proposed model and baselines/ablation versions are statistically significant or not. In particular, some gaps in Tables 3 and 6 are quite subtle, and the variances of classification scores in Table 2 are unknown, therefore p-values should be reported.

Author response:
**Q1: p-values should be reported.**

Thank you for your suggestion. We have conducted a t-test between our method and the second-best method in Table 2-6, i.e. NTM+CL, Linear-$\alpha=0.5$, NTM+CL, NTM+CL, and GAP, respectively. We obtain $p$-values as follows:

- **t-test for Table 2:**

| Metric  | 20NG  | IMDb  | AGNews |
|---------|-------|-------|--------|
| $p$-value | 0.048 | 0.023 | 0.019  |

- **t-test for Table 3:**

| |  Wiki  | |
|---------|:------:|:-------:|
|    **Metric**       | $T = 50$ | $T = 200$ |
| $p$-value |  0.010 |  0.039  |

- **t-test for Table 4:**

|         |  20NG  |       |         |       |  IMDb  |       |         |       |
|---------|:------:|:-----:|:-------:|:-----:|:------:|:-----:|:-------:|:-----:|
|         | $T = 50$ |       | $T = 200$ |       | $T = 50$ |       | $T = 200$ |       |
| **Metric**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD** |
| $p$-value |  0.034 | 0.032 |  0.047  | 0.047 |  0.024 | 0.029 |  0.015  | 0.024 |

- **t-test for Table 5:**

|         |  Wiki  |       |         |       | AGNews |       |         |       |
|---------|:------:|:-----:|:-------:|:-----:|:------:|:-----:|:-------:|:-----:|
|         | $T = 50$ |       | $T = 200$ |       | $T = 50$ |       | $T = 200$ |       |
| **Metric**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD** |
| $p$-value |  0.037 | 0.044 |  0.005  | 0.018 |  0.033 | 0.047 |  0.029  | 0.026 |

- **t-test for Table 6:**

|            |   20NG   |           |   IMDb   |           |   Wiki   |           |  AGNews  |           |
|:----------:|:--------:|:---------:|:--------:|:---------:|:--------:|:---------:|:--------:|:---------:|
| **Metric** | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ |
| $p$-value  |   0.044  |   0.013   |   0.013  |   0.017   |   0.001  |   0.005   |   0.008  |   0.019   |

We see that all of the results are statistically significant ($p$-value < 0.05). We believe the reason is that our method better refines the topic representation through setwise contrastive signal and our effective Pareto solution for the NTM. Thank you again for your advice. **We have added these to Appendix I in our rebuttal revision.**

Reviewer comment (Evaluation):
Only automatic metrics (e.g., NPMI and TD) are used to evaluate topic quality. Although the authors also examine document classification as a downstream task, the classification performance is just an indirect measurement of topic quality. Recent work [1] has shown that automatic metrics may deviate from humans' judgment. Therefore, I feel human evaluation (e.g., the intrusion test [2]) is still needed.

Author response:
**Q2: Human evaluation is needed.**

Thank you for your comment. We have followed previous works [1, 2, 3] to utilize two automatic metrics, *i.e.* topic diversity and NPMI, to measure topic quality. However, we acknowledge that human evaluation will provide a more rigorous evaluation. We have conducted human rating and word intrusion tasks for 20NG and IMDb on Amazon Mechanical Turk (AMT) between our topics and those of NTM+CL, and **obtain the result in Figure 5 in Appendix K of our rebuttal revision.**

We see that human judgement also prefers our generated topics than the baseline ones. Our inter-annotator agreement is 0.72, which indicates high agreement. Thank you for your interesting suggestion, **we have added these to Appendix K in our rebuttal revision.**

**References:**

[1] Thong Nguyen and Anh Tuan Luu, Contrastive learning for neural topic model, NeurIPS 2021.

[2] Xu et al., Hyperminer: Topic taxonomy mining with hyperbolic embedding, NeurIPS 2022.

[3] Wang et al., Representing mixtures of word embeddings with mixtures of topic embeddings, arXiv 2022.

Reviewer comment (Evaluation):
Could you conduct statistical significance tests to compare your method with the baselines in the experiment tables?

Author response:
**Q1: p-values should be reported.**

Thank you for your suggestion. We have conducted a t-test between our method and the second-best method in Table 2-6, i.e. NTM+CL, Linear-$\alpha=0.5$, NTM+CL, NTM+CL, and GAP, respectively. We obtain $p$-values as follows:

- **t-test for Table 2:**

| Metric  | 20NG  | IMDb  | AGNews |
|---------|-------|-------|--------|
| $p$-value | 0.048 | 0.023 | 0.019  |

- **t-test for Table 3:**

| |  Wiki  | |
|---------|:------:|:-------:|
|    **Metric**       | $T = 50$ | $T = 200$ |
| $p$-value |  0.010 |  0.039  |

- **t-test for Table 4:**

|         |  20NG  |       |         |       |  IMDb  |       |         |       |
|---------|:------:|:-----:|:-------:|:-----:|:------:|:-----:|:-------:|:-----:|
|         | $T = 50$ |       | $T = 200$ |       | $T = 50$ |       | $T = 200$ |       |
| **Metric**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD** |
| $p$-value |  0.034 | 0.032 |  0.047  | 0.047 |  0.024 | 0.029 |  0.015  | 0.024 |

- **t-test for Table 5:**

|         |  Wiki  |       |         |       | AGNews |       |         |       |
|---------|:------:|:-----:|:-------:|:-----:|:------:|:-----:|:-------:|:-----:|
|         | $T = 50$ |       | $T = 200$ |       | $T = 50$ |       | $T = 200$ |       |
| **Metric**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD**  |  **NPMI**  |   **TD**  |   **NPMI**  |   **TD** |
| $p$-value |  0.037 | 0.044 |  0.005  | 0.018 |  0.033 | 0.047 |  0.029  | 0.026 |

- **t-test for Table 6:**

|            |   20NG   |           |   IMDb   |           |   Wiki   |           |  AGNews  |           |
|:----------:|:--------:|:---------:|:--------:|:---------:|:--------:|:---------:|:--------:|:---------:|
| **Metric** | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ | $T = 50$ | $T = 200$ |
| $p$-value  |   0.044  |   0.013   |   0.013  |   0.017   |   0.001  |   0.005   |   0.008  |   0.019   |

We see that all of the results are statistically significant ($p$-value < 0.05). We believe the reason is that our method better refines the topic representation through setwise contrastive signal and our effective Pareto solution for the NTM. Thank you again for your advice. **We have added these to Appendix I in our rebuttal revision.**

Reviewer comment (Evaluation):
Could you perform human evaluation (e.g., the intrusion test) to directly examine the quality of extracted topics?

Author response:
**Q2: Human evaluation is needed.**

Thank you for your comment. We have followed previous works [1, 2, 3] to utilize two automatic metrics, *i.e.* topic diversity and NPMI, to measure topic quality. However, we acknowledge that human evaluation will provide a more rigorous evaluation. We have conducted human rating and word intrusion tasks for 20NG and IMDb on Amazon Mechanical Turk (AMT) between our topics and those of NTM+CL, and **obtain the result in Figure 5 in Appendix K of our rebuttal revision.**

We see that human judgement also prefers our generated topics than the baseline ones. Our inter-annotator agreement is 0.72, which indicates high agreement. Thank you for your interesting suggestion, **we have added these to Appendix K in our rebuttal revision.**

**References:**

[1] Thong Nguyen and Anh Tuan Luu, Contrastive learning for neural topic model, NeurIPS 2021.

[2] Xu et al., Hyperminer: Topic taxonomy mining with hyperbolic embedding, NeurIPS 2022.

[3] Wang et al., Representing mixtures of word embeddings with mixtures of topic embeddings, arXiv 2022.