Paper: JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuning (ICLR-2024)

Reviewer comment (Experiments):
Context window consumption: JsonTuning demands much more tokens than TextTuning and it poses a challenge both in the training and inference time. One key aspect of current LLMs is their context window, and how to use it efficiently. The authors should provide evidence relating to the extra overhead on training/inference of JsonTuning.

Author response:
> Analysis of the tokens used for JsonTuning and TextTuning

We have calculated the input and output tokens used for each training and evaluation example in both TextTuning and JsonTuning. The average tokens of all training and evaluation examples for these methods are presented in the following table:

|  | Training | Evaluation | 
| -------- | -------- | -------- | 
| Text w/ label space and control information | 317.53 | 236.24 |
| Json w/ label space and control information | 394.61 | 296.55 |
| Text w/o label space and control information | 278.17 | 183.91 |
| Json w/o label space and control information | 349.23 | 235.16 |

The table shows that JsonTuning uses approximately 25% more tokens for training and evaluation compared to TextTuning. The additional tokens are primarily used to encode the JSON structured format.

There are potential ways to reduce the tokens used for JsonTuning:

* JsonTuned models are robust to different prompts, so using lengthy prompts may not be necessary. We can replace them with concise prompts.
* JsonTuning offers the flexibility to adjust the structure. Simplifying the input and output structures could be helpful, as JsonTuning requires more tokens for more complex structures.

Reviewer comment (Experiments):
It is not also obvious to transform the user's request into a structured format. Despite the authors' arguments on the generalization capability of JsonTuning, structured object like Json still lags far behind natural language in terms of expressivity. The tasks concerned in the work are relatively easy and can be transformed into json format with less effort. However, I am not convinced this generalizes to real industrial use cases where the user's request could be far more complex.

Author response:
> Regarding the expressivity of natural language texts and JSON structures

The expressivity of the constructed JSON structures is at least equivalent to that of their corresponding natural language texts, as JSON structures can easily be converted into natural language texts. For instance, consider the following JSON input and text input. By replacing placeholders, such as {question} and {options}, in the JSON structures with their respective values, the text input can be obtained.

**JSON Input**: {"input": {"question": "Who is the CEO of Google?", "options": "(A) Sundar Pichai (B) Bill Gates (C) Tim Cook (D) Satya Nadella", "instruction": "Answering the following question: {question} {options}. Answer: {answer}"}}

**Text Input**: "Answering the following question: Who is the CEO of Google? (A) Sundar Pichai (B) Bill Gates (C) Tim Cook (D) Satya Nadella. Answer:" 

However, this transformation may lose some information, such as what are the key elements of the task. Our proposed JsonTuning method aims to address this issue. Consequently, JSON structures offer more valuable information than natural language texts, resulting in higher expressivity. 

> Regarding transforming the user's request into a structured format, especially for complex use cases

Transforming user requests into a structured JSON format can be efficiently achieved by developing a user interface that enables users to input key task elements, task prompts, and control information. This approach ensures that users can effectively communicate their requirements without needing to understand JSON structures. As the input and output JSON structures are consistent across all tasks, we can automatically construct them based on the user inputs. For more complex use cases, the user interface can guide the model in identifying the essential information for the target task and the desired outputs.

JsonTuning offers several advantages for applications:

* Robustness: The method's resilience to varying prompts significantly reduces the need for users to test multiple prompts for optimal performance. If users do not wish to provide a task prompt, a default prompt can be generated based on the key task elements they supply, thereby minimizing user effort. Furthermore, as backend models are updated or changed, the optimal prompt for natural language texts may vary, necessitating adjustments for each model version. In contrast, Json-tuned models may not require prompt modifications even when the backbone model is updated, considerably reducing user effort.

* Output Control: The output control component allows users to accurately specify and easily parse the output, which can be challenging when using natural language texts. Users only need to comprehend basic types such as string, object, and array. The descriptions and examples of these data types can be provided through the user interface. This approach necessitates minimal human effort, and the underlying JSON structures can be automatically constructed once users indicate the output elements they require.

By implementing JsonTuning in real-world applications, a more professional and efficient user experience can be achieved.

Reviewer comment (Reproducibility):
A simple question: assume a user requests the model to produce output following json format at the first place. Then how the output control be defined on the input side? I suppose it would be a nested json and could be fragile for the model to take as input.

Author response:
> Assume a user requests the model to produce output following json format at the first place. Then how the output control be defined on the input side? I suppose it would be a nested json and could be fragile for the model to take as input.

Output control in JsonTuning is specifically designed to accommodate users' unique requirements, enabling a more accurate representation of their intentions compared to TextTuning. Consequently, defining the output control relies on the users' specific objectives. For instance, in a medical diagnosis task that aims to identify diseases based on certain symptoms, the output could be an array of potential diseases, the most probable disease, or diseases associated with particular organs such as the heart or lungs.

Regarding your concern about the complexity of the output control, it is crucial to acknowledge that the complexity of the output control depends on the particular use case and user requirements. A well-trained LLM should effectively manage various output structures with diverse complexities. To achieve this, It is essential to train the LLM on diverse examples, including various JSON structures and output control patterns, to ensure its adaptability to different patterns and generate the desired output. This is why we introduce InstrucUIE for training, as it allows the model to handle complex structures effectively. Our evaluation results on the EE task demonstrate that the Json-tuned model can generalize to more complex structures even when trained only on simpler structures. Furthermore, our case studies on controllability illustrate that the model can generate the output in compliance with the output control effectively.

Reviewer comment (Experiments):
Introduction of Additional Knowledge in Input: The utilization of Json formatting can aid models in generating outputs that conform to the constraints specified in the Json input. However, it also introduces supplementary knowledge, such as information pertaining to input and output types, which is originally absent in the unstructured textual instructions. To facilitate a fair comparison, it becomes essential to incorporate the information present in the Json input but absent in the textual input into the original text instructions. This allows for a comprehensive assessment of whether the Json format indeed outperforms the text format.

Author response:
> Introduction of Additional Knowledge in Input

In Section 4: Analysis of our paper, we conduct an ablation study of JsonTuning without label space and control information, where JsonTuning still outperforms TextTuning in terms of generalization. To address your question comprehensively, we perform additional experiments with TextTuning, incorporating label space and control information. The generalization and robustness results on LLaMA-7b are as follows:

Generalization Results:

| Model | MMLU | BBH | NER | RE | EE | TQA | NL2SQL | Average |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Text w/ label space and control information | 44.56 | 36.77 | 35.70 | 15.77 | 1.45 / 0.00 | 16.40 | 14.60 | 23.50 |
| Json w/ label space and control information | 44.69 | 37.08 | 41.80 | 15.56  | 3.09 / 8.24 | 16.40 | 16.40 | **25.37**
| Text w/o label space and control information | 43.11 | 32.48 | 36.47 | 13.60 | 1.08 / 0.00 | 18.20 | 8.60 | 21.86
| Json w/o label space and control information | 42.66 | 36.23 | 47.65 | 16.23 | 0.99 / 0.29 | 10.40 | 12.80 | **23.80**

Robustness results on MMLU with 10 different prompts (we calculate the mean and standard deviation (Std) of the performance using these 10 prompts.):

| Model | Mean | Std | 
| -------- | -------- | -------- | 
| Text w/ label space and control information | 42.02 | 3.31 | 
| Json w/ label space and control information | **44.63** | **0.11** |
| Text w/o label space and control information | 38.82 | 6.28 |
| Json w/o label space and control information | **42.19** | **0.21** |

We can make the following observations: (1) JsonTuning consistently surpasses TextTuning in both generalization and robustness, regardless of the inclusion of label space and control information. Importantly, JsonTuning significantly reduces the model's sensitivity to different prompts, as evidenced by the lower variance in performance across prompts. (2) TextTuning with control information still struggles to generalize well to more complex structured tasks, as illustrated by the poor performance on the EE task. JsonTuning demonstrates a marked improvement in this regard. These findings emphasize the importance of structured formats in instruction tuning. Furthermore, our proposed components – label space and control information – consistently enhance the model's generalization capabilities and robustness, highlighting their value in instruction tuning.

Reviewer comment (Experiments):
Potential Incompleteness of Generalization Experiments: While the training dataset encompasses a diverse range of tasks, including Flan and P3++, there appears to be some overlap between the types of tasks assessed during testing and those encountered in training. It is pertinent to explore whether the model can adeptly generalize to entirely unseen task types, which may not have been covered by the training data. Furthermore, the evaluation tasks, primarily centered on structure prediction, demand output format constraints. However, for the field of instruction tuning, there exists a research interest in user-oriented instruction following evaluation, exemplified by AlpacaEval. In cases involving more open-ended instruction data, the Json format may not bring additional benefits since there are no specific constraints imposed on the output format. It also remains uncertain how the model performs on the Super-NI test set, a widely employed dataset assessing a model's ability to follow instructions.

Author response:
> Potential Incompleteness of Generalization Experiments

Our evaluation encompasses various unseen tasks and datasets. While some task types, such as multiple-choice QA, are encountered during training, the specific evaluation tasks and datasets remain unseen. Furthermore, tasks like EE and TQA represent entirely novel task types. Our JsonTuning method allows models to generalize to more complex EE structures, even if they were not encountered during training, demonstrating the robust generalization capabilities of our Json-tuned models.

We conduct an evaluation of LLaMA-7B-Json and LLaMA-7B-Text on AlpacaEval. Both models were trained with label space and control information. To assess LLaMA-7B-Json, we construct an input JSON structure for a natural language question Q, represented as {"input": {"question": Q, "instruction": "{question}\n{answer}"}, "output control": {"answer": {"type": "string"}}}. The win rates for LLaMA-7B-Json and LLaMA-7B-Text are 4.78% and 4.64%, respectively. The slightly better performance of LLaMA-7B-Json suggests that JsonTuning may offer additional advantages in open-ended scenarios.

Since the Super-NI test set is already incorporated into the Flan2022 collection, it would not be appropriate to evaluate models on this test set, as many instances in it are seen during training.

Reviewer comment (Novelty):
Question on Missing Relevant References: There might be several missing relevant references. The first one has studied Json-format instruction data. The second one is a resource that has a similar formulation with Super-NI and can be represented in well-structured Json format as well:  
- Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning. Yin et al., 2023.  
- Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation. Yin et al., 2023.

Author response:
> Regarding the missing references

Thank you for your suggestion. The first paper, titled "Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learning," focuses on understanding the importance of different parts of a task definition and identifying key elements in task definitions. The second paper, "Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curation," proposes a paradigm for automatically curating instruction-tuning data, which can be represented in a well-structured JSON format.

We acknowledge the valuable contributions of both papers in the instruction tuning field and their relevance to our work. We will cite these papers in the updated version of our paper.

Reviewer comment (Evaluation):
In my opinion this paper needs some sort of analysis of the number of additional tokens introduced by the JSON format for training and inference. The additional training cost is probably negligible and unimportant, but the additional FLOPS and encoded/decoded tokens for the JSON format will add up for inference. Note: I am not saying that the fact that JSON-formatted examples have extra tokens is a weakness, but this extra cost should at least be quantified in my opinion.

Author response:
> Analysis of the tokens used for JsonTuning and TextTuning

We have calculated the input and output tokens used for each training and evaluation example in both TextTuning and JsonTuning. The average tokens of all training and evaluation examples for these methods are presented in the following table:

|  | Training | Evaluation | 
| -------- | -------- | -------- | 
| Text w/ label space and control information | 317.53 | 236.24 |
| Json w/ label space and control information | 394.61 | 296.55 |
| Text w/o label space and control information | 278.17 | 183.91 |
| Json w/o label space and control information | 349.23 | 235.16 |

The table shows that JsonTuning uses approximately 25% more tokens for training and evaluation compared to TextTuning. The additional tokens are primarily used to encode the JSON structured format.

There are potential ways to reduce the tokens used for JsonTuning:

* JsonTuned models are robust to different prompts, so using lengthy prompts may not be necessary. We can replace them with concise prompts.
* JsonTuning offers the flexibility to adjust the structure. Simplifying the input and output structures could be helpful, as JsonTuning requires more tokens for more complex structures.

Reviewer comment (Experiments):
How do text-tuned models behave as a base for JSON-formatted examples? For example, I think some of the following questions should be answered or at least discussed: What if we text-tune the model and then JSONTune it on a small dataset? What if we text-tune the model and then use JSON-formatted examples in a few shot prompt?

Author response:
> What if we text-tune the model and then Json-tune it on a small dataset?

To explore this, we continue instruction-tuning the text-tuned model with JsonTuning using varying numbers of examples (120, 600, 1200, 6000, and 12000) while maintaining the relative data ratio of the Flan 2022 collection and structured tasks in our primary experiments. The LLaMA-7b results on MMLU and NL2SQL are as follows:

|  | 120 | 600 | 1200 | 6000 | 12000 |
| -------- | -------- | -------- |  -------- | -------- | -------- | 
| MMLU | 0.00 | 40.83 | 39.83 | 42.45 | 41.85 |
| NL2SQL | 0.00 | 4.40 | 3.80 | 5.20 | 3.40 |

The results of the original text-tuned model on MMLU and NL2SQL are 43.11 and 8.60, respectively. 

We observe that: (1) A very limited number of JSON-formatted examples (e.g., 120) is insufficient for the text-tuned model to handle JSON-formatted tasks. (2) Tuning the text-tuned model with a small set of JSON-formatted examples results in degraded performance. These observations suggest that using the text-tuned model as a base for JsonTuning is not appropriate.

> What if we text-tune the model and then use JSON-formatted examples in a few shot prompt?

This approach is ineffective. We sampled 100 MMLU examples and used 3 JSON-formatted examples as few-shot prompts. The text-tuned LLaMA-7b model still generates natural language texts instead of JSON structures and these texts seem non-sense. While a significantly larger text-tuned model may generate valid JSON structures, its performance could be substantially lower than the JSON-tuned model.

Reviewer comment (Experiments):
How often was invalid JSON generated? What do the authors think about constrained decoding schemes like [Scholak et al](https://arxiv.org/abs/2109.05093)? Is there any point of using constrained decoding to improve JSONTuned models?

Author response:
> How often was invalid JSON generated? What do the authors think about constrained decoding schemes like Scholak et al, is there any point of using constrained decoding to improve JSONTuned models?

The proportion of invalid JSON structures generated by LLaMA-7B on evaluation tasks is as follows:

| MMLU | BBH | NER | RE | EE | TQA | NL2SQL | 
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | 
0.00% | 1.57% | 5.74% | 0.27% | 4.19% | 0.00% | 3.60%

As anticipated, tasks with intricate output structures, such as NER and EE, exhibit a higher ratio of invalid JSON structures compared to tasks with simpler output structures, such as MMLU and BBH.

PICARD, a constrained decoding method proposed by Scholak et al., is designed to produce valid SQL queries by rejecting inadmissible tokens at each decoding step. In the context of our JsonTuning paradigm, employing a constrained decoding scheme that ensures the generation of valid output structures adhering to the output control could be advantageous. However, implementing such a method may be challenging, and it could potentially increase decoding time. As an alternative, we may utilize post-processing techniques to obtain valid JSON structures, as they are typically efficient and cost-effective. Previously, we treated invalid output structures as NULL, which might have been overly strict for evaluation purposes. Moving forward, we plan to investigate suitable post-processing techniques that can be broadly applied to various output structures, aiming to enhance the performance and validity of Json-tuned models.

Reviewer comment (Novelty):
Significance: the format of instruction tuning data is only one of the many system choices of the overall instruction tuning. Others include tasks, base model, domains, and languages. Since the paper only focuses on the data format on a selection of tasks, the significance is limited.

Author response:
> Regarding the significance of our work

The performance of instruction tuning is influenced by several key factors, including the base model, tuning data, and data format. While the base model and tuning data have received considerable attention from researchers, the significance of data format has often been overlooked. In this paper, we aim to emphasize the importance of data format in instruction tuning, focusing on its role in generalization, robustness, and output control. 

Our experiments and analysis demonstrate that a structured data format, such as JSON, is crucial for effective instruction tuning. First, our method, JsonTuning, consistently outperforms TextTuning across various base models and tuning data quantities, highlighting its generalizability and complementarity with other factors. Second, JsonTuning substantially enhances the robustness of instruction-tuned models, an improvement that is difficult to achieve solely through the base model and tuning data optimization. For instance, our robustness results indicate that upgrading the base model from LLaMA to LLaMA2 does not significantly enhance robustness; however, JsonTuning considerably improves it. Finally, JSON structures offer well-defined control information for output generation, a feature that is challenging to achieve using natural language text alone.

Reviewer comment (Experiments):
Soundness: It is not clear if a TextTuning model with candidate answers and output control in plain text would also perform as good as JsonTuning. In other words, how much gain was from the structured format of JSON itself?

Author response:
> It is not clear if a TextTuning model with candidate answers and output control in plain text would also perform as well as JsonTuning. In other words, how much gain was from the structured format of JSON itself?

In Section 4: Analysis of our paper, we conduct an ablation study of JsonTuning without label space and control information, where JsonTuning still outperforms TextTuning in terms of generalization. To address your question comprehensively, we perform additional experiments with TextTuning, incorporating label space and control information. The generalization and robustness results on LLaMA-7b are as follows:

Generalization Results:

| Model | MMLU | BBH | NER | RE | EE | TQA | NL2SQL | Average |
| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Text w/ label space and control information | 44.56 | 36.77 | 35.70 | 15.77 | 1.45 / 0.00 | 16.40 | 14.60 | 23.50 |
| Json w/ label space and control information | 44.69 | 37.08 | 41.80 | 15.56  | 3.09 / 8.24 | 16.40 | 16.40 | **25.37**
| Text w/o label space and control information | 43.11 | 32.48 | 36.47 | 13.60 | 1.08 / 0.00 | 18.20 | 8.60 | 21.86
| Json w/o label space and control information | 42.66 | 36.23 | 47.65 | 16.23 | 0.99 / 0.29 | 10.40 | 12.80 | **23.80**

Robustness results on MMLU with 10 different prompts (we calculate the mean and standard deviation (Std) of the performance using these 10 prompts.):

| Model | Mean | Std | 
| -------- | -------- | -------- | 
| Text w/ label space and control information | 42.02 | 3.31 | 
| Json w/ label space and control information | **44.63** | **0.11** |
| Text w/o label space and control information | 38.82 | 6.28 |
| Json w/o label space and control information | **42.19** | **0.21** |

We can make the following observations: (1) JsonTuning consistently surpasses TextTuning in both generalization and robustness, regardless of the inclusion of label space and control information. Importantly, JsonTuning significantly reduces the model's sensitivity to different prompts, as evidenced by the lower variance in performance across prompts. (2) TextTuning with control information still struggles to generalize well to more complex structured tasks, as illustrated by the poor performance on the EE task. JsonTuning demonstrates a marked improvement in this regard. These findings emphasize the importance of structured formats in instruction tuning. Furthermore, our proposed components – label space and control information – consistently enhance the model's generalization capabilities and robustness, highlighting their value in instruction tuning.

Reviewer comment (Reproducibility):
How did you construct training data for JsonTuning from plain text instruction tunning data? Did you use human annotator to do the conversion or use another language model?

Author response:
> How did you construct training data for JsonTuning from plain text instruction tunning data? Did you use human annotator to do the conversion or use another language model?

We generate the training data automatically without relying on human annotators or language models for construction. Specifically, we perform the following tasks: (1) We modify the data generation code for the Flan2022 collection (https://github.com/google-research/FLAN/tree/main/flan/v2) to include the task input and output elements, prompt, and the label space if available and for each example. (2) For tasks with limited label space in subcollections of Flan2022 such as CoT and Super Natural Instructions v2, we construct the label space by gathering their possible task outputs, as their task sources do not provide explicit label spaces. (3) For the Flan2022 collection, we create JSON input and output structures based on the prompts used for each example, as they clearly indicate input and output elements, with all outputs being string types. For InstructUIE, we manually construct 10 prompts for each task and design the control information to build the input and output structures. We will make our processed data and data processing code publicly available.

Reviewer comment (Experiments):
Why did you choose JSON format instead of other popular structured formats, such as XML and YAML?

Author response:
> Why did you choose JSON format instead of other popular structured formats, such as XML and YAML?

Our proposed structure-to-structure paradigm can be implemented using various structured formats, including JSON, XML, and YAML. We selected JSON because (1) it has native support for various data types such as string, array, and object;  (2) its syntax is simpler and consistent, facilitating easy parsing and generation; (3) it is natively supported in most programming languages and is highly compatible with Python.
Nevertheless, the choice ultimately depends on specific use cases, requirements, and preferences. XML may be preferable for support of metadata, namespaces, or more complex data such as charts or images, while YAML could be suitable for prioritizing human readability and simplicity.