Paper: Hexa: Self-Improving for Knowledge Augmented Dialogue System (ICLR-2024)

Reviewer comment (Evaluation):
The authors have stated `2) a novel bootstrapping scheme with a guided prompt and a modified loss function for diverse and appropriate generation of intermediate and final responses to be self-trained;' as a major contribution. Nonetheless, there is no evaluation of the diversity in the main paper.

Author response:
Following your suggestion, we move the evaluation of the diversity from Appendix to main paper (see Section 5.5 in the revised paper).

Reviewer comment (Experiments):
Although Hexa has notable improvement in the evaluation of the generated dialogue responses, this work lacks enough experiments to verify *Does Hexa really improve the final performance by improving the performance of intermediate modules?*. I think this is important.

Author response:
In parts of the paper, we present evidence to the claim that the performance of final response if enhanced by improving the intermediate modules’ performances. 
    
    1. In Table 4, we show an example where a bad search query leads to retrieval of irrelevant documents, causing a generation of low quality response. Improving the intermediate modules help avoiding such cases.
    2. In Table 16 of appendix, we show results comparing the performance of the responses with and without the intermediate modules. The results reveal a significant degradation in performance when intermediate modules are absent for both BB3 and Hexa.
    3. In Table 8, we show results of module-wise evaluation of the intermediate modules. The result indicates that the final decision is correlated with the performance of the search and entity knowledge modules. The search decision and search query scores does not portray the relationship to the final response since the ground truth used to measure these score for the search query and decision may not be the optimal in our experiment setting due to the discrepancy in the search engine used. 
    
    We therefore conclude that enhancing the intermediate modules can lead to improvement in the final response for knowledge-augmented dialogue generation.

Reviewer comment (Experiments):
It seems the relevant baselines haven’t been used (such as LLaMA 2-chat, GPT3/4) where the field has evolved a lot in recent years.

Author response:
**Comparison for recent LLMs.**
    
    → We do not consider models like LLama2-chat as a baseline for the following reasons:
    
    1. Excessively large difference in the number of training tokens of the base model: The number of training tokens of LLama2 is approximately twenty times larger than the number of training tokens of BB3-3B (2T tokens for LLama2 vs 100B tokens for BB3).
    2. Possibility of data contamination: Referencing scores from [1], we found that LLama2-7B-chat achieves unusually high score on the benchmarks used in this paper, e.g., TriviaQA, that is higher than the score of BLOOM-175B despite the size difference and the increase in the number of documents used by the BLOOM-175B model. This naturally causes a suspect that training data of LLama2 have high chance of being contaminated with the evaluation datasets used in this paper. 
    
    Please consider that with the growing concern of data contamination in LLM evaluations [2, 3], experimental design becomes more challenging when evaluating on popular benchmarks, as used in this paper. These benchmarks have not been explicitly defined as held-out datasets by the recent LLMs, such as LLama2. Additionally, since the training data is not publicly available, we cannot split the clean and contaminated evaluation sets, as done in [4].
    
 `[1] Large Language Models Struggle to Learn Long-Tail Knowledge`

`[2] Llama 2: Open Foundation and Fine-Tuned Chat Models`

`[3] Rethinking Benchmark and Contamination for Language Models with Rephrased Samples`

`[4] PaLM: Scaling Language Modeling with Pathways`

Reviewer comment (Reproducibility):
The availability of code is not discussed. Implementation details related to the resources, framework, model cards, training days, etc. would help in reproducibility.

Author response:
**Reproducibility.**
    
    → Hexa is implemented in huggingface environment and the model is initialized with BB3-3B, which is publicly available. Hexa is trained using A100-80G 8GPUs for approximately 48 hours over 11 iterations. We include these statements in Appendix.J in the revised paper.

Reviewer comment (Evaluation):
Could the authors provide more detail about the experiments involving SentenceBERT for the similarity measure and also an intuition why it performed better/worse than BLEU/ROUGE?

Author response:
**Detail about the experiments involving SentenceBERT.**
    
    → Shown in Table 15 in the appendix, in the seen dataset, Hexa with ROUGE-L achieves highest ROUGE score while Hexa with sentenceBERT score as reward function achieves highest sentenceBERT score, as expected. The similarity measured by sentenceBERT behaves differently from the BLEU/ROUGE on the basis that BLEU/ROUGE is more strict while sentenceBERT may pick up non-overlapping similarity such as paraphrases and synonyms. Therefore, answers generated by Hexa w/ SentenceBERT may not necessarily have high overlapping scores like BLEU/ROUGE scores while responses generated by Hexa w/ ROUGE-L, based on word overlapping, will likely have high SentenceBERT score, as overlapping words would naturally lead to high semantic similarity.

Reviewer comment (Experiments):
Could the authors explain how this approach would work in real-life scenarios?

Author response:
**Real-world scenarios.**
    
    → We provide descriptions of real world application of Hexa in two different cases.
    
    1. Cases that does not require training.
        
        These are cases where the intermediate pipeline is identical or is a subset of the pipeline described in this paper. Further more, as long as the search engine operates similar to the ones used in our paper and the there is not much of a domain shift in the input queries, our trained model may be used straight out the the box, as seen in the performance of unseen-tasks in Table 12.
        
    2. Cases that require re-training.
        
        When there is a significant domain-shift of the use cases or when the pipeline differs from the one used in this paper, Hexa requires retraining of the model. In this case, the users are advised to collect input-output pairs and retrain Hexa. Note that since Hexa is a end-to-end training method, the users are not required to collect any of any training data of the intermediate modules beforehand. We leave further investigation of adapting Hexa to totally unseen pipeline as a future work.

Reviewer comment (Evaluation):
Metric Relevance: In the age of sophisticated dialogue systems like ChatGPT, conventional metrics like F1 and ROUGE-L might not be the most reliable measures of performance.

Author response:
We agree with the reviewer that according to the recent standards of characteristics that chat-models should have, such as helpfulness and insightfulness, measures like F1 and ROUGE-L may bot be the best option. However, we would like to also emphasize that in order to measure how well the response is grounded on the given knowledge, such as in the QA, KGD, and ToD tasks, metrics like F1 and ROUGE that measures the word overlap are reliable options and are also used in recent works like [1].
    
Also, there has been reports of biases that exists in the evaluation by LLMs such as GPT-4 [2]. On behalf of the matter, we provide results from more reliable metric of human evaluation in the paper, in which Hexa has shown a significant performance increase over the baseline.

Reviewer comment (Experiments):
Lack of SOTA Comparison: The paper doesn't compare its method with state-of-the-art chat-based language models like LLama2-chat, which would be crucial for understanding its relative performance.

Author response:
We do not consider models like LLama2-chat as a baseline for the following reasons:
    
1. Excessively large difference in the number of training tokens of the base model: The number of training tokens of LLama2 is approximately twenty times larger than the number of training tokens of BB3-3B (2T tokens for LLama2 vs 100B tokens for BB3).
2. Possibility of data contamination: Referencing scores from [3], we found that LLama2-7B-chat achieves unusually high score on the benchmarks used in this paper, e.g., TriviaQA, that is higher than the score of BLOOM-175B despite the size difference and the increase in the number of documents used by the BLOOM-175B model. This naturally causes a suspect that training data of LLama2 have high chance of being contaminated with the evaluation datasets used in this paper.

Please consider that with the growing concern of data contamination in LLM evaluations [4, 5], experimental design becomes more challenging when evaluating on popular benchmarks, as used in this paper. These benchmarks have not been explicitly defined as held-out datasets by the recent LLMs, such as LLama2. Additionally, since the training data is not publicly available, we cannot split the clean and contaminated evaluation sets, as done in [6].

Reviewer comment (Experiments):
Outdated Baseline Model: The use of the STAR model as a baseline for human evaluation is noted as a potential weakness due to its age, which might not accurately reflect the current advancements in the field.

Author response:
Of the recent works that bases on the scheme of bootstrap then fine-tune, [7, 8] may be a possible candidate baselines. However, the methodologies in these works are too specific to their applications, complicating the adaptation of their methods to knowledge-augmented dialogue generation. On the other hand, the method suggested in STaR is general enough that it can easily be adopted in knowledge-augmented dialogue generation task. Thus, we choose STaR as the baseline.