Paper: COLLIE: Systematic Construction of Constrained Text Generation Tasks (ICLR-2024)

Reviewer comment (Novelty):
Although I understand that the authors are focusing on more "basic" units, such as tokens, sentence, etc, so this paper can be more practical and useful for downstream applications, such as pretraining and evaluating LLMs, most of current work on constrained text generation seem to focus on text summarization, including controllable text summarization (e.g., MACSum Zhang et al, 2023). However this paper does not mention any of this, and does not discuss any potential that the proposed method can be applied to those constrained text generation (summarization) tasks. I personally suggest that the authors provide some discussion or explain on this issue.

Author response:
Thanks for your great suggestion! We have incorporated the related work and discussion in the draft!

Reviewer comment (Novelty):
The idea is not very novel, and similar ideas have already been explored. For example, [1] also constructed a dataset using similar constraints for instruction fine-tuning.

Author response:
* Thanks for bringing up this concurrent work, which we have cited and discussed in the revised paper! 
    * It focuses on turning constraints into NL instructions and fine-tuning smaller language models such as T5-11B, and the constraints themselves are fairly simple and limited (only 5 types). In contrast, Collie focuses on making constrained text generation more flexible, customizable, and harder, challenging even SoTA LLMs like GPT-4. 
    * We believe these two efforts are orthogonal and can be complementary --- we can use these harder and more flexible constraints from Collie-v1 and other Collie constructed datasets to diagnose and analyze LMs (as shown in the paper), and improve the instruction tuning of LMs.

Reviewer comment (Novelty):
The dataset may not be very useful. Specifically, because the rules are too vague/arbitrary, the extracted ground truth is not useful for the evaluation process: the authors only use them for comparing fluency. In addition, since the rules can be arbitrarily designed, this compiled dataset does not hold much value, because many similar datasets can be compiled with different engineering details.

Author response:
* The extracted ground truth text is not only useful for comparing fluency, but also to ensure the extracted constraint values are solvable using natural text. We believe it is actually  important to evaluate constraint satisfaction using the constraints instead of the extracted text, as there could be very diverse generations that all satisfy the same constraints. 
    * As stated in the paper, our main contribution is twofold: the Collie framework for constructing constrained text generation tasks, and the Collie-v1 dataset using a concrete and preliminary set of constraint types. The facts that “rules can be arbitrarily designed” and “many similar datasets can be compiled with different engineering details” are exactly the merit of our proposed Collie framework --- while previous datasets feature a fixed set of simple constraints and ad-hoc pipelines for data collection, the Collie framework allows automatic and scalable construction of arbitrary grammar-expressible constraints.

Reviewer comment (Reproducibility):
In the limitations, the authors mentioned potential problems with filtering and processing functions. Could the authors elaborate on what issues might exist for the extraction process?

Author response:
* Every text corpus has artifacts (e.g., tables, footnotes, references, headings that might not be natural sentences or paragraphs). We design filtering and processing functions to patch or eliminate passages with these artifacts. However, it is impossible to design a filtering system that is perfect, so there could be downstream reference texts or constraints that look unnatural. This is more of a fundamental limitation with any approach that leverages text in the wild, and not a limitation specific to Collie or CTG construction. We have revised the limitation part to better explain this, thanks!

Reviewer comment (Presentation):
Some important details for instruction rendering should be moved to the main paper.

Author response:
As per your suggestion, we have moved more details for instruction rendering to the main paper (section 3, in blue).

Reviewer comment (Evaluation):
The paper mentions that the technique can be used for constraining words, word blacklisting, however a qualitative analysis is missing for the same in the current version.

Author response:
By word blacklisting we mean something similar to constraint para02: "Generate a paragraph with at least 4 sentences, but do not use the words “the”, “and” or “of”." We have complete model outputs in our supplementary material, but we can also add some of these examples to the appendix. Thank you for the suggestion.

Reviewer comment (Reproducibility):
Section 5: Performance enhancement through feedback - It might help to list down the details of how the feedback is being used.

Author response:
Thanks! We have moved some details for use of feedback to the main paper, as per your suggestion (section 5.1, in blue).