Paper: Instructive Decoding: Instruction-Tuned Large Language Models are Self-Refiner from Noisy Instructions (ICLR-2024)

Reviewer comment (Evaluation):
The empirical gain in Table 1 seems incremental (e.g. only 1-2 points of overall improvement)

Author response:
In addressing the concerns about the incremental improvements highlighted in `Table 1`, it's important to recognize the broader impact of our method. While the overall average improvement might appear modest, a closer examination reveals significant gains in specific tasks and categories. This suggests a nuanced and substantial impact that goes beyond mere numbers.

Moreover, although improvements in metrics like exact-match and Rouge-L might seem marginal, a qualitative analysis paints a different picture. As illustrated in `Figure 5` and `Figure 7(a)`, our method encourages the generation of a wider variety of responses, indicating an enhanced understanding and a more contextually rich output. These qualitative improvements, though not fully captured by quantitative metrics, represent a significant stride in model performance. This multifaceted impact of our method, encompassing both quantitative and qualitative enhancements, underscores its efficacy and the substantial advancement it brings to the field.

Lastly, it is also important to emphasize, as seen in `Figure 2` and `Table 2`, that the improvement in T0-3B and Alpaca-7B models, which were not instruction-tuned on SNI, is far from marginal but rather considerable. This observation underscores the potency of our method across various models, independent of the training dataset. Our approach demonstrates robust effectiveness in enhancing model performance, regardless of the model type or the specifics of the training dataset. This widespread applicability and significant improvement across different models reinforce the strength and versatility of our methodology.

We additionally include a comprehensive analysis of the responses from ID in various perspectives ( `Appendix F` and `Appendix G` in our revised manuscript). 
- Relationship between Token Probability and Consistency
- Distributions on the Number of Generated Tokens
- Qualitative Analysis
    - Wrong Response Patterns From ID
    - How ID Refines Response when Model size Increases
    - Limitations of Using Fixed References in Rouge-L Evaluation

This is to help readers to get more insights from our generic framework.

Reviewer comment (Novelty):
The idea is not particularly groundbreaking

Author response:
While we acknowledge that the core concept of manipulating model outputs via noisy instructions may not initially seem groundbreaking, the novelty and impact of our work lie in its practical application and the subtle yet significant innovations we've introduced. Our method ID represents a significant step forward in the realm of instruction-tuned language models. By ingeniously leveraging the anchoring effect within a contrastive framework, we've demonstrated how subtle shifts in instruction presentation can lead to marked improvements in model performance, especially in terms of generalization to unseen tasks.

Moreover, the simplicity and effectiveness of ID offer substantial benefits in terms of computational efficiency. Unlike approaches that require extensive model scaling or additional training, ID achieves comparable, if not superior, improvements without the need for expanding model size or incurring additional computational costs. This aspect of our work is particularly valuable in the current landscape of NLP, where computational resources are a major bottleneck.

Furthermore, our extensive experiments and analyses underscore the method's robustness and versatility across different tasks and model sizes. The qualitative improvements in model outputs, as evidenced in our various case studies and examples, attest to the method's capacity to enhance the richness and contextuality of language model responses.

In conclusion, while the fundamental idea of manipulating model outputs may not be entirely new, the specific way we've applied this concept, combined with our comprehensive analytical approach, makes our contribution both innovative and significant for the field.

Reviewer comment (Evaluation):
it’s still unclear (for a practitioner or researcher) to what extent the noisy instruction would be helpful for a new task. Maybe some qualitative analysis in those no-improvement tasks would provide some intuition on why the noisy instruction is harmful in the proposed instructive decoding.

Author response:
## W1. Intuition on why the noisy instruction is harmful in the proposed instructive decoding.

We concur that conducting a qualitative analysis of instances where there is no improvement can offer valuable insights to potential practitioners about how ID functions. In `Appendix G: Qualitative Analysis on ID Responses` of our revised manuscript, we have carried out a comprehensive analysis on the generated responses. We highlight our analysis as follows:

1) When the responses are wrong, **ID tends to include the words in the given instructions in its response**, while Baseline response tends to include the words from the instance sentences. For example, in the degrading category (CR in OpenSNI-7B), Baseline responses include the words from instance but ID response include the words in the given instruction. This suggests that while ID consistently improves performance in most cases, **it may reduce effectiveness in tasks that require the use of exact words from the provided instance.** This aligns with our observations in the main experiment (`Table 1`), where a drop in performance is primarily seen in the OE, CR, and KT categories, which often demand such specific word usage.

2) As the model size increases, we notice that the Baseline responses become more sophisticated, and ID refines these responses more effectively. In our analysis of how ID enhances performance in certain categories with the larger Tk-XXL model, yet initially performs less well in the smaller Tk-XL model (specifically in KT and QR categories), we find that the Baseline responses from the smaller model are quite basic, and so are the ID responses. However, as the model size grows, it starts generating responses that are more aligned with the instructions, and ID improves these responses further. This suggests that ID is more beneficial when the model, even if not fully complying, can understand the given instructions to some degree.

3) We also emphasize that when we analyze ID responses from OpenSNI-7B on WR category, which shows performance decline, it rather generating semantically valid responses but the Rouge-L score cannot capture the enhanced ID responses. For example, in a WR instance `climb : ladder. stir: ?` case, the reference response is “soup”. ID response it with `\u2615`, which corresponds to “hot beverage” in the Uni-code emoji, which also semantically valid in this context. This implies that Rouge-L evaluation on the ID responses has limitation under fixed references, emphasizing the difficulties in accurately assessing open-ended generation tasks.

We invite you to review our extensive qualitative analysis in `Appendix G`, where we've added a wide range of examples and insights to further elucidate our findings.

Reviewer comment (Experiments):
is it possible to further fine-tune the model via the new logits (i.e.,  $z_t - \tilde z_t$ in Algorithm1)

Author response:
## Q1. Further fine-tuning via the new logits

Thank you for your insightful suggestion regarding the potential use of $z_t - \epsilon * \tilde{z}_t$ as a training objective for instruction-tuned models. We appreciate the depth of your proposal and its implications for advancing the field.

In considering this approach, our initial perspective is that it may offer limited advantages for models that have already undergone comprehensive instruction tuning. The primary utility of noisy instructions, as leveraged in our method, seems most pronounced when models exhibit suboptimal performance in understanding the held-out task instructions. For models that have already achieved a high level of proficiency on seen tasks (i.e., training dataset) through extensive training, the gains from integrating noisy instruction effects might be marginal.

That being said, we do see potential value in exploring this method during the early stages of instruction tuning. It could serve as an innovative approach to enhance the model's foundational understanding of varied instructions, potentially setting a more robust stage for subsequent fine-tuning processes. Furthermore, we acknowledge the intriguing possibility of interpreting this method as a form of sequential knowledge distillation (or self-distillation) using tokens from ID, which might offer a novel pathway to optimize model performance without necessarily relying on larger model architectures.

However, it's important to note that our current research has been primarily focused on exploring avenues to boost model performance without engaging in additional training cycles. As such, we have not yet delved into experimental investigations of this particular training approach. Nevertheless, we recognize its potential as a fruitful direction for future research and intend to consider it in our ongoing efforts to refine and enhance instruction-tuned models.

In conclusion, your suggestion has certainly opened an exciting area for further exploration, and we are eager to incorporate these ideas into our future work. We believe that such innovative approaches hold great promise for the continued evolution of language model training methodologies.

Reviewer comment (Evaluation):
I wonder if the improvement from ID correlates with the response length required for a task. E.g., is ID more useful in tasks that require long response generation?

Author response:
## Q2. Correlation btw the Improvement and the response length

In response to the question about the correlation between ID's improvement and task response length, we conducted experiments with Tk-Instruct XL (3B), XXL (11B), and OpenSNI (7B) models on the SuperNatInst dataset. Please see `Figure 13` in `Appendix F: Quantitative Analysis of Tokens Generated by ID` of our revised manuscript. We analyzed the number of tokens per response, using kernel density estimation for distribution analysis. Our findings show that while ID with 'opposite' instruction slightly reduces response length, there is no significant difference in response lengths compared to baseline models.

This indicates that ID's effectiveness is not tied to response length but rather to its ability to refocus the model's processing in line with given instructions. Our results demonstrate ID's broad applicability and consistent benefits across various tasks, highlighting its versatility in enhancing model performance irrespective of response length.

Reviewer comment (Experiments):
While I appreciated the analysis of sensitivity to the \epsilon parameter, given that there is only a relatively small region of \epsilon where the approach outperforms the baseline, I'd appreciate more discussion of how \epsilon was chosen and whether it was the same across all tasks in each experiment.

Author response:
Thank you for your insightful comments regarding the selection of ϵ and its effectiveness across different tasks and models. We agree that a detailed exploration of ϵ is essential for the robustness of our method.

To clarify, ϵ in our method plays a crucial role in determining the extent of influence that the noisy instruction exerts on the prediction process. Mathematically, a positive value for ϵ is indeed justified, as it effectively divides the probability of the token predictions based on the noisy instruction from those based on the original instruction [1]. This division, rather than a simple additive approach, allows for a more nuanced adjustment in the model's output, reflecting the contrast between the two sets of instructions. In our experiments, we observed that while the choice of ϵ indeed has an impact on different tasks, this effect was relatively marginal. More prominent was the influence of ϵ on different model architectures.

Determining the optimal ϵ value is indeed a complex issue. In our experiments, we approached this selection empirically, considering the balance between the influence of the noisy instruction and the base instruction. Our choice of ϵ at 0.3 or 0.2 was informed by extensive trial and error across different tasks and models, aiming to find a value that consistently yielded improved performance without overly skewing the model's responses.

We observed that the impact of ϵ can vary depending on the model size and task complexity. For instance, smaller models like Tk-3B showed better improvements with ϵ at 0.3, while larger models like Tk-11B optimized at 0.2. This suggests a nuanced relationship between model architecture, task, and the effectiveness of the noisy instruction.

Although we reported a consistent ϵ value of 0.3 in our paper for simplicity and to maintain a model-agnostic approach, **our future work** includes developing more sophisticated methods for selecting ϵ. This could involve task-specific optimizations or leveraging metrics like the **sine distance between embeddings of responses guided by the base and noisy instructions** (`Table A`).

Table A. Zero-shot Rouge-L score on unseen tasks in the held-out set of SUPNATINST with Tk-Instruct XL.

| Method | Base | Opposite (eps = 0.3) | Sine epsilon |
| --- | --- | --- | --- |
| Overall RougeL | 45.36 | 46.69 | 46.26 |

We plan to explore these ideas further in the appendix. However, it's important to emphasize that our paper is the first to propose such a generic framework for instruction following. Challenges like determining the optimal ϵ are not unique to our approach; they are akin to unresolved issues in other deep learning methods such as label smoothing or weight decay. Our work lays the foundation for addressing these challenges in the context of instruction following, and we look forward to further research in this area.

[1] ACL 2023, Li, Xiang Lisa, et al. "Contrastive decoding: Open-ended text generation as optimization"

Reviewer comment (Theory):
This is a minor weakness, but I didn't really understand the connection to "anchoring". Anchoring seems to be about sensitivity to the first thing appearing in the text, while this method doesn't privilege any particular positions in the text but contrasts against the negative prompt at all positions.

Author response:
In response to the query regarding the connection to 'anchoring' in our methodology, we appreciate the opportunity to clarify this aspect. Our method indeed diverges from the traditional interpretation of the anchoring effect, which emphasizes sensitivity to initial text elements. However, the essence of our approach aligns with the cognitive bias of anchoring in a broader sense.

Our strategy employs noisy instructions, not to anchor the model's attention to a specific position within the text, but to anchor the model's overall prediction tendency. By introducing a consistently contrasted negative prompt across the input, we leverage the anchoring effect to guide the model’s prediction process. This ensures that, regardless of text position, the model's predictions are uniformly influenced by this contrast, leading to outputs that are more aligned with the intended instruction.

In contrast to traditional contrastive methods that often rely on amateur models or less proficient predictions, our approach is more task-specific and efficient. By generating noisy predictions that are divergent yet relevant to the task, we provide a specific set of alternatives for the model to contrast with, thereby refining its adherence to the base instructions. This not only enhances the accuracy and relevance of the model's responses but also broadens its applicability.

To summarize, our method employs the anchoring effect within a contrastive framework in a novel manner. It focuses on influencing the model's prediction tendency rather than its attention to specific text positions. This innovative approach represents a significant advancement in enhancing the instruction-following capabilities of language models and reflects our commitment to pushing the boundaries of traditional methodologies.

Reviewer comment (Novelty):
While the related work was generally good, there is room to mention a few other related papers on contrastive methods for generation. But I also don't think this is a major weakness, as the application to instruction following is new.

Author response:
Thank you for suggesting additional works regarding contrastive methods for our literature survey. We appreciate your suggestions and have incorporated them to enhance our discussion.

Reviewer comment (Experiments):
How was the \epsilon value chosen for the important experiments in the paper (e.g. Table 1 - Table 3)?

Author response:
Thank you for your insightful comments regarding the selection of ϵ and its effectiveness across different tasks and models. We agree that a detailed exploration of ϵ is essential for the robustness of our method.

To clarify, ϵ in our method plays a crucial role in determining the extent of influence that the noisy instruction exerts on the prediction process. Mathematically, a positive value for ϵ is indeed justified, as it effectively divides the probability of the token predictions based on the noisy instruction from those based on the original instruction [1]. This division, rather than a simple additive approach, allows for a more nuanced adjustment in the model's output, reflecting the contrast between the two sets of instructions. In our experiments, we observed that while the choice of ϵ indeed has an impact on different tasks, this effect was relatively marginal. More prominent was the influence of ϵ on different model architectures.

Determining the optimal ϵ value is indeed a complex issue. In our experiments, we approached this selection empirically, considering the balance between the influence of the noisy instruction and the base instruction. Our choice of ϵ at 0.3 or 0.2 was informed by extensive trial and error across different tasks and models, aiming to find a value that consistently yielded improved performance without overly skewing the model's responses.

We observed that the impact of ϵ can vary depending on the model size and task complexity. For instance, smaller models like Tk-3B showed better improvements with ϵ at 0.3, while larger models like Tk-11B optimized at 0.2. This suggests a nuanced relationship between model architecture, task, and the effectiveness of the noisy instruction.

Although we reported a consistent ϵ value of 0.3 in our paper for simplicity and to maintain a model-agnostic approach, **our future work** includes developing more sophisticated methods for selecting ϵ. This could involve task-specific optimizations or leveraging metrics like the **sine distance between embeddings of responses guided by the base and noisy instructions** (`Table A`).

Table A. Zero-shot Rouge-L score on unseen tasks in the held-out set of SUPNATINST with Tk-Instruct XL.

| Method | Base | Opposite (eps = 0.3) | Sine epsilon |
| --- | --- | --- | --- |
| Overall RougeL | 45.36 | 46.69 | 46.26 |

We plan to explore these ideas further in the appendix. However, it's important to emphasize that our paper is the first to propose such a generic framework for instruction following. Challenges like determining the optimal ϵ are not unique to our approach; they are akin to unresolved issues in other deep learning methods such as label smoothing or weight decay. Our work lays the foundation for addressing these challenges in the context of instruction following, and we look forward to further research in this area.

[1] ACL 2023, Li, Xiang Lisa, et al. "Contrastive decoding: Open-ended text generation as optimization"

Reviewer comment (Evaluation):
Can any intuition be given for why the approach seems to work best in the generalization setting (3.3 "Generalization Capabiltiies")?

Author response:
We appreciate the reviewer's question regarding the pronounced performance improvements of T0 and Alpaca models in generalization settings under our Instructive Decoding (ID) method **(`Table 2`)**. The key to understanding this lies in the initial performance baseline of these models and the unique impact of ID on them.

The T0 and Alpaca, while robust models, start at a lower performance level on the SupNatInst and UnNatInst datasets due to their training on diverse data sources. This initial training often instills certain biases or memorization patterns that are not directly aligned with the specific challenges and formats of the SupNatInst and UnNatInst tasks.

ID significantly enhances the performance of these models. It achieves this by disrupting their pre-existing biases and learned patterns through the introduction of noisy instructions. This disruption shifts the models' reliance away from their trained patterns and biases, prompting them to leverage their intrinsic language processing abilities more effectively.

Such a reorientation is particularly beneficial for handling the varied tasks in SupNatInst and UnNatInst, which may be different from their original training scenarios. While T0 and Alpaca models' performance after ID application doesn't surpass that of Tk-instruct or OpenSNI models (trained specifically on SupNatInst), the relative improvement is substantial. This is because ID's intervention compensates for their initial lower baseline and effectively counters the limitations imposed by their pre-training.

To conclude, the remarkable improvements in generalization capabilities for T0 and Alpaca models using ID highlight the method's efficacy in augmenting model adaptability, especially for models initially trained on datasets different from those they are tested on.

Reviewer comment (Reproducibility):
What method is used to choose the random words from NLTK (i.e. what word list is used? something involving antonyms?)

Author response:
In "Random Words", we create a noisy instruction by concatenating N random words from entire NLTK words list (total 236,736 words) without any specific preconditions, which aims to distract the model's response generation.

Reviewer comment (Experiments):
The discussion in the "ID with Opposite" section hints that some of the baseline models might be mode-splitting, and that (at least on these classification tasks), approaches might benefit from a decoding approach that uses min-Bayes risk or consensus decoding: group together words based on their similarities and choose to generate a word from a large group.

Author response:
Thank you for your insightful suggestion regarding the connection between our ID method and the Min-Bayes Risk (MBR) decoding approach. The idea of leveraging MBR principles in the context of language model decoding is indeed an interesting perspective that aligns well with the underlying mechanism of ID.

To elaborate, in our ID method, the suppression of output space influenced by noisy instructions can be viewed as an effort to refine the hypothesis set, akin to MBR. This suppression may aid in mitigating mode-splitting issues observed in baseline models, particularly for classification tasks. Through ID, we effectively narrow down the hypothesis space by discounting less probable or contradictory predictions induced by noisy instructions. This approach resonates with the MBR strategy of focusing on a more coherent and consistent set of hypotheses.

We also note that while MBR typically involves grouping similar translation hypotheses and selecting the most probable one, our method uniquely contributes by actively shaping the hypothesis space through the use of noisy instructions. This not only aids in generating more contextually appropriate responses but also opens up new avenues for applying MBR principles in diverse NLP tasks beyond translation.

We appreciate this opportunity to explore the parallels and distinctions between ID and MBR. Your feedback has inspired us to delve deeper into this comparison and consider its implications for future research. We plan to expand upon these insights and incorporate a more detailed discussion in the Appendix of our revised manuscript, drawing from relevant literature [2-5] to contextualize our approach within the broader framework of modern generation techniques.

References:

[2] "It's MBR All the Way Down: Modern Generation Techniques Through the Lens of Minimum Bayes Risk," Arxiv 23.10.

[3] "An Empirical Study of Translation Hypothesis Ensembling with Large Language Models," EMNLP 23.

[4] "Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation," EMNLP 22.

[5] "DC-MBR: Distributional Cooling for Minimum Bayesian Risk Decoding," Arxiv 22.10.

Reviewer comment (Evaluation):
I wasn't really clear on the support for the conclusions in "Visualization of Embeddings", that as models get better at understanding instructions (e.g. are larger), performance of ID increases.

Author response:
Thank you for your valuable feedback regarding the clarity of our conclusions in the "Visualization of Embeddings" section. We acknowledge the importance of substantiating our findings and appreciate the opportunity to provide further clarification.

In our study, we utilized embedding visualizations as a tool to illustrate the model's ability to differentiate between noisy and baseline instructions. Specifically, **`Figure 7(b)`** of our paper presents a comprehensive analysis of this aspect. Through these visualizations, we observed a distinct correlation: the degree of separation in the embeddings appears to directly influence the performance gains achieved through our ID.

To exemplify, we analyzed two distinct task scenarios: Keyword Tagging (KT) and Word Analogy (WA). In WA tasks, the results were particularly striking, with the Tk-XXL model consistently outperforming the baseline across all variants of noisy instructions. This suggests a robust ability of the model to discern and adapt to the variations introduced by ID. Conversely, in KT tasks, we noticed a performance decline with certain noisy instructions, indicating a more complex interplay between the model’s understanding and the specific nature of the task.

This pattern of performance, as reflected in both the embedding visualization and the quantitative results (**presented in `Table 1` and `Figure 4(b)`**), points towards an interesting observation: larger models tend to benefit more significantly from the nuances introduced by noisy instructions. We interpret this as an indication of the larger models' enhanced capacity to process and differentiate between the subtle variations in instruction context, which is crucial for the effectiveness of ID.

We hope this explanation provides a clearer picture of how embedding visualizations support our conclusions and the underlying mechanisms of ID. We are committed to continuously refining our methods and presentation to ensure the utmost clarity and utility of our findings.

Reviewer comment (Writing):
"aims to steer the models toward particular" in Section 1. Should "toward" be "away from"?

Author response:
We recognize the ambiguity in our original phrasing and appreciate the opportunity to clarify. The goal of our ID method is to utilize the anchoring effect in a way that deliberately guides models towards specific, possibly sub-optimal predictions. This is achieved through the strategic use of noisy instruction variants. To more accurately convey this concept, we propose the following revision:

*'...The essence of ID lies in the introduction of noisy instruction variants. These are designed to anchor the model's output in a specific direction, potentially away from the most optimal predictions. This deliberate steering enables a clear anchoring effect within the language models, facilitating a contrastive approach in our decoding process...’*

Reviewer comment (Writing):
"pragmatics" in the motivation for Label Adherence and Label Coherence is a bit misleading; this is much closer to "semantics".

Author response:
We appreciate the reviewer's insight regarding our usage of the term `pragmatics`. We recognize that 'pragmatics' might not accurately encapsulate the essence of our analysis, particularly in the context of label coherence and label adherence for classification tasks. Our primary concern is with the semantic correctness of responses in relation to the given instructions.

Accordingly, we agree that `semantics` is a more appropriate term for what we aim to evaluate. This term better reflects our focus on the alignment of responses with the semantic content of instructions. (but, we will think about it more and fix it until the camera-ready deadline. We are also currently considering various alternative expressions such as `instructional alignment'.) We will incorporate this modification into our revised manuscript to ensure clarity and precision in our communication. Thank you for highlighting this aspect, and we believe this change will make our manuscript more comprehensible to our readers.