Paper: KITAB: Evaluating LLMs on Constraint Satisfaction for Information Retrieval (ICLR-2024)

Reviewer comment (Reproducibility):
For the open datasets (Open Library and WikiData), is there some precautions taken to "snapshot" these to a specific point in time for reproducibility purposes?

Author response:
**For the open datasets (Open Library and WikiData), is there some precautions taken to "snapshot" these to a specific point in time for reproducibility purposes?** - Indeed, for the sake of reproducibility, we follow two key precautions:
- *Inclusion of Raw Data and Metadata*: In our published dataset, in addition to our clean processed book data associated with each query, we have included the raw snapshot pulled from OpenLibrary. We have also included the metadata (birth year and popularity metric) associated with each author. This is integrated alongside the main KITAB dataset.
- *Provision of Code Scripts*: We provide code scripts not only for the evaluation of KITAB, but also for every step of the construction of the dataset itself, allowing users to refresh and rebuild the dataset as needed and also generate a different version with a different set of authors if necessary.

Reviewer comment (Reproducibility):
The procedures use a few proprietary tools to develop the dataset: Azure Cognitive Services NER and Language API, Geonames in the dataset preparation. If the framework code release is open-sourced, future developers could substitute open or more reproducible components in these places, but this means that it would be difficult to create a new version of the dataset "out of the box".

Author response:
**The procedures uses a few proprietary tools to develop the dataset: Azure Cognitive Services NER and Language API, Geonames in the dataset preparation.** - We experimented with a few options to identify an effective and inclusive entity recognition tool and found that Azure Cognitive Services provided the most reliable and comprehensive results for our dataset. However, if users would like to replicate or update KITAB, and would like to adhere to non-proprietary tools, they can easily adapt our published code scripts to replace Azure's API with any open-source NER software they see fit. Additionally, we used Geonames, which is already an open-source tool, making it readily accessible for those who wish to use or adapt our methodology. For reference, you can check out Geonames - All Cities with a population > 1000 — Opendatasoft.

Reviewer comment (Reproducibility):
Likewise for GPT-3.5 and -4: are all the queries and responses captured somehow for reproducibility of the experiments and analysis? Some of these reproducibility concerns potentially undercut the usefulness and extensibility of the framework.

Author response:
**Likewise for GPT-3.5 and -4: are all the queries and responses captured somehow for reproducibility of the experiments and analysis? Some of these reproducibility concerns potentially undercut the usefulness and extensibility of the framework.** -
This feedback is very insightful; we have opted not to release our model responses along with KITAB, primarily due to the dynamic nature of LLMs, especially GPT models and the inconsistencies that can arise from different endpoints, leading to potential confusion. However, we have thoroughly documented all of our experiments, capturing both the raw and processed outputs from the models. To facilitate research and extend the utility of our work, we plan to include a note in our published dataset, encouraging researchers to contact the authors directly if they wish to access these detailed records. This approach is intended to maintain the integrity and reproducibility of our research while acknowledging the evolving landscape of LLMs.

Reviewer comment (Experiments):
Can fine-tuned LLMs do well on KITAB? The results and dataset would be more compelling with another FINE-TUNED setting, even if it had to be done with an OSS model.

Author response:
**Can fine-tuned LLMs do well on KITAB? The results and dataset would be more compelling with another FINE-TUNED setting, even if it had to be done with an OSS model** - This is indeed a promising direction. While our experiments have primarily been with general-purpose models and haven't extensively explored models fine-tuned for specific tasks like constraint satisfaction, KITAB offers a valuable opportunity for such experimentation, in particular when seen as a dynamic data collection approach which we facilitate with the release of the data collection workflow code. It would be interesting to see how instruction-tuned models perform in this setting and how KITAB performance on these models aligns with general IR performance. To this end however, we believe that fine-tuning or adaptation for constraint satisfaction is a basic skill that all pretrained models need to have to enable controlled generation and that it is not a task for which every user or customer should have to fine-tune for.

Reviewer comment (Writing):
What does KITAB stand for? I couldn't find a definition.

Author response:
**What does KITAB stand for? I couldn't find a definition.** - The dataset is named after the word kitab (https://en.wikipedia.org/wiki/Kitab), which is the word for "book" in Arabic, Swahili, Urdu, Hindi and various Indian and Turkic languages.

Reviewer comment (Experiments):
The author(s) have indeed conducted a thorough consideration of multiple facets in constructing the dataset, such as the number of constraints, the variety of constraints, and unsatisfiability. Nevertheless, the scope of the proposed dataset is fundamentally narrow, being confined to the domain of books. This domain-specific focus could limit its applicability to a broader range of IR research. It could be better if data from a variety of domains can be included.

Author response:
**Limited Scope. … The scope of the proposed dataset is fundamentally narrow, being confined to the domain of books.** - As mentioned in the general response, we chose the book domain as a domain that has sufficient public data to work with, but we believe that the problem of LLMs having difficulties in answering such queries is more ubiquitous. Based on research and usage evidence, similar issues have been observed when models are required to answer queries of the same nature in the scientific research domain or encyclopedia-like questions by models fabricating titles of papers that do not exist on a given topic (e.g. https://twitter.com/IttefaqM/status/1715392338535563594?s=20). Most prominently, even the earlier hallucination observed in Bard (see https://www.theverge.com/2023/2/8/23590864/google-ai-chatbot-bard-mistake-error-exoplanet-demo) was related to a similar query that also contained constraints which were not satisfied by the model (i.e., “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” with the model including a discovery that did *not* actually involve the James Webb Space Telescope). While we hope that more datasets of a similar nature evolve in the community, we hope that KITAB paves the path for such work and shows the value of being able to collect and leverage clean and complete data for similar evaluations. Note that one of the main challenges with constructing datasets in a similar vein is related to data cleaning and completeness. Even for domains like movies, celebrities etc., well-known repositories are far from clean and require the integration of several data sources and robust cleaning approaches as we further discuss in the paper and supplementary material. In addition, we also wanted to provide future users with a way to create more evaluation data dynamically by using public APIs, given rising concerns around data contamination.

Reviewer comment (Novelty):
The targeted problem is very special. The authors consider it as an IR problem, but it resembles more a database querying problem. Once the book references are transformed into a structured database, the problem may be much simpler.

Author response:
Thanks for engaging in the discussion!

Even though KITAB only focuses on the literature domain, it is a suitable benchmark for studying the more general and widespread issue by offering a structured way of verifying information. We hope that future efforts will extend this to further domains.

While many of our queries can be casted to SQL and tested on structured information, we'd like to note that information is not always structured or even available as such. That is why users are indeed using systems like BingChat, Bard, and ChatGPT to ask such questions.

Similar issues to what we present in KITAB have been observed when models are required to answer queries of the same nature in the scientific research domain or encyclopedia-like questions by models fabricating titles of papers that do not exist on a given topic (e.g. https://twitter.com/IttefaqM/status/1715392338535563594?s=20). Most prominently, even the earlier hallucination observed in Bard (see https://www.theverge.com/2023/2/8/23590864/google-ai-chatbot-bard-mistake-error-exoplanet-demo) was related to a similar query that also contained constraints which were not satisfied by the model (i.e., “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” with the model including a discovery that did not actually involve the James Webb Space Telescope).

Reviewer comment (Experiments):
The intent of the authors is not to examine how GPT models can handle such a search problem, but to use this to test the ability of LLMs to handle queries containing constraints. However, the appropriateness of the task is questionable.

Author response:
Thanks for engaging in the discussion!

Even though KITAB only focuses on the literature domain, it is a suitable benchmark for studying the more general and widespread issue by offering a structured way of verifying information. We hope that future efforts will extend this to further domains.

While many of our queries can be casted to SQL and tested on structured information, we'd like to note that information is not always structured or even available as such. That is why users are indeed using systems like BingChat, Bard, and ChatGPT to ask such questions.

Similar issues to what we present in KITAB have been observed when models are required to answer queries of the same nature in the scientific research domain or encyclopedia-like questions by models fabricating titles of papers that do not exist on a given topic (e.g. https://twitter.com/IttefaqM/status/1715392338535563594?s=20). Most prominently, even the earlier hallucination observed in Bard (see https://www.theverge.com/2023/2/8/23590864/google-ai-chatbot-bard-mistake-error-exoplanet-demo) was related to a similar query that also contained constraints which were not satisfied by the model (i.e., “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” with the model including a discovery that did not actually involve the James Webb Space Telescope).

Reviewer comment (Evaluation):
It is expected that a general LLM may not perform the task well because they have not been trained for the task. The main concern is whether the types of constraints used can really test the ability of LLMs to satisfy constraints, or if the tests are on LLMs’ ability to understand the constraints and to execute filtering of books accordingly in this very special case.

Author response:
**Generality of the dataset. Can the presented constraint types really test the ability of LLMs to satisfy constraints?** - We consider 6 different constraint types to test for *lexical*, *entity-based*, and *temporal* constraints. Indeed, the pretrained models are not directly trained to satisfy these constraints, but that is the case for many emergent abilities of interest. Yet, we’d like to understand and measure model performance on these abilities given that current models do show non-trivial (albeit non satisfactory) performance in such tasks. 

Even amongst the constraint types we consider, models may have received direct or indirect training signals. For example, entity recognition (human/city name) is a task that has been largely studied in NLP and may therefore have more explicit training data used for pretraining, while the model’s ability to apply string operations is less studied and perhaps requires different skills. This is why we disaggregate our analysis and show how performance varies across different constraints. While it is true that some of these constraints can be easily satisfied through SQL, our goal here is to test the model’s direct linguistic abilities as at the application-level not all user queries are or can be rewritten as SQL queries. Some of the constraint types we consider, can actually be mapped to concrete *librarian tasks* such as retrieving an author’s work (List all books by an author), indexing or searching by a given constraint (List all books by an author whose title starts with a letter), finding characters mentioned in the title (or content in more advanced versions) etc. While our goal isn’t to completely solve the library domain, we use this domain as an illustration of a much larger problem in factual inaccuracy when models are not able to follow constraints.

Reviewer comment (Evaluation):
Even if LLMs are able to do the job well, it is questionable whether one would be able to draw interesting or general conclusions from the experiments shown in the paper about LLMs’ ability to satisfy constraints.

Author response:
Thanks for engaging in the discussion!

Even though KITAB only focuses on the literature domain, it is a suitable benchmark for studying the more general and widespread issue by offering a structured way of verifying information. We hope that future efforts will extend this to further domains.

While many of our queries can be casted to SQL and tested on structured information, we'd like to note that information is not always structured or even available as such. That is why users are indeed using systems like BingChat, Bard, and ChatGPT to ask such questions.

Similar issues to what we present in KITAB have been observed when models are required to answer queries of the same nature in the scientific research domain or encyclopedia-like questions by models fabricating titles of papers that do not exist on a given topic (e.g. https://twitter.com/IttefaqM/status/1715392338535563594?s=20). Most prominently, even the earlier hallucination observed in Bard (see https://www.theverge.com/2023/2/8/23590864-google-ai-chatbot-bard-mistake-error-exoplanet-demo) was related to a similar query that also contained constraints which were not satisfied by the model (i.e., “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” with the model including a discovery that did not actually involve the James Webb Space Telescope).

Reviewer comment (Experiments):
Given the very special characteristics of the dataset, the value of the KITAB dataset for the purpose of examining LLMs’ capability of constraint satisfaction in search is questionable.

Author response:
Thanks for engaging in the discussion!

Even though KITAB only focuses on the literature domain, it is a suitable benchmark for studying the more general and widespread issue by offering a structured way of verifying information. We hope that future efforts will extend this to further domains.

While many of our queries can be casted to SQL and tested on structured information, we'd like to note that information is not always structured or even available as such. That is why users are indeed using systems like BingChat, Bard, and ChatGPT to ask such questions.

Similar issues to what we present in KITAB have been observed when models are required to answer queries of the same nature in the scientific research domain or encyclopedia-like questions by models fabricating titles of papers that do not exist on a given topic (e.g. https://twitter.com/IttefaqM/status/1715392338535563594?s=20). Most prominently, even the earlier hallucination observed in Bard (see https://www.theverge.com/2023/2/8/23590864-google-ai-chatbot-bard-mistake-error-exoplanet-demo) was related to a similar query that also contained constraints which were not satisfied by the model (i.e., “What new discoveries from the James Webb Space Telescope can I tell my 9 year old about?” with the model including a discovery that did not actually involve the James Webb Space Telescope).

Reviewer comment (Evaluation):
While the ground truth is determined correctly, there may be some problem in the measure of irrelevance and completeness, as some fuzzy matches are allowed. As the authors admit, there may be some overestimation of the performance, so the experimental results are only indicative and may contain a possible rate of errors.

Author response:
**Evaluation procedure. While the ground truth is determined correctly, there may be some problem in the measure of irrelevance and completeness, as some fuzzy matches are allowed.** - Thanks for raising this concern, it is indeed important for the measurement to be realistic. We view performance on KITAB as an “upper bound” on the LLM’s actual ability for these types of simple information retrieval tasks. Our main qualitative result is that existing LLMs already struggle to obtain good performance on KITAB, and therefore, this casts doubt on their ability to perform these sorts of constraint satisfaction tasks in general settings. This is why we are willing to overestimate model performance; indeed, our main result demonstrates that even with all of these relaxations, state-of-the art LLMs still struggle immensely with this elementary (but important) task.

In addition, relaxing the evaluation process is crucial for testing constraint satisfaction abilities in the language domain in a realistic manner without undermining model capabilities. For example, the evaluation should not over penalize the model if the model is asked to output a title that starts with the letter “c” and includes the title “The Citadel” in its results. Indeed, librarian practices do include such titles as this categorization is most useful for users. Similarly, if the model outputs the following title “Gödel, Escher, Bach” as a book by Douglas R. Hofstadter but does not mention its full title (“Gödel, Escher, Bach: An Eternal Golden Braid”) this should be considered a correct answer as the book is known with both titles. Similar reasoning goes to accounting for potential typos in the model’s answer or in our data by using fuzzy matching. In fact, we consider the evaluation process together with the appropriate implemented relaxation process as a practical contribution of the paper and will open source the artifacts to support reproducibility to this end.

As mentioned in the paper (Section 3.1 - Book collection) and supplementary material (C.1), we also run a manual annotation to separately evaluate how irrelevance and completeness is impacted by perhaps books that we were not able to retrieve with the sources used (OpenLibrary and WikiData) and find that only 5% of the GPT4 answers contain at least one book that exists on the internet from the same author but is not captured from KITAB. The impact of this on actual irrelevance and completeness is a lot lower because the number of non-found books per query is usually less than 2 while the satisfactory set is larger.

Reviewer comment (Evaluation):
For the estimation of irrelevance or partial satisfaction, one case of match is when one is a string subset of another: for each ki, it is checked if there exists a book in the ground truth set by that author which is either a string subset match in both directions. It is unclear whether a threshold on the percentage of the substring is applied, or if a substring of any length is accepted.

Author response:
**Do you also apply a threshold on the percentage of the substring, or a substring of any length is accepted?** - Let’s take the example of 2 possible titles for the same book t1= “Gödel, Escher, Bach: An Eternal Golden Braid” and t2= “Gödel, Escher, Bach”. We would consider this a match if t2 \in t1 or if t1 \in t2. Therefore, we only consider full substrings as a match.