Paper: Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generation (ICLR-2024)

Reviewer comment (Novelty):
It is not well-explained which part of DSG is designed to address the **Atomic Question** and **Full Semantic Coverage** properties. Why the QG methods of TIFA and VQ^A cannot achieve these two properties?

Author response:
Thanks for the great suggestion — we made a revision to explain the point more clearly (Sec. 3, the paragraph above the subsection on DSG-1k dataset).

In gist, to the question “why the QG methods of TIFA and VQ^2 cannot achieve atomicity and semantic coverage?”, the short answer is: because they do not have a structured intermediary representation in the question generation process (like DSG’s semantic tuples) to a) define the semantic coverage for a prompt, and b) thereby allowing for the verification of atomicity.

To expand the answer in detail, DSG uses semantic tuples (e.g., attribute(bike, blue), entity(bike), relation(bike, wall, next to)) as an intermediary representation in question generation: prompt -> semantic tuples -> questions. As such, leveraging the structuredness of the semantic tuple intermediary representation, we can then clearly define "atomicity" and "semantic coverage" and thereby modulate DSG's behavior to generate questions accordingly:

- Atomicity: an atomic question corresponds to only a single semantic tuple. For example, "is there a bike?" maps to one semantic tuple "entity(bike)", and is atomic; "is there a blue bike?" maps to two semantic tuples "entity(bike)" and "attribute(bike, blue)", and is thus NOT atomic.
- Semantic coverage: the full semantics of a prompt is represented by a set of unique semantic tuples (no duplicates), such that no entities, attributes, relations, or global descriptions can be added. 

TIFA and VQ^2, on the other hand, generate questions directly, thus there is no means available to encourage atomicity and semantic coverage (i.e., with a direct prompt-to-questions configuration w/o semantic tuples intermediary). In our human evaluation of TIFA and VQ^2 questions (Table 2), we indeed find large numbers of non-atomic questions and violations of semantic coverage (with duplication and missing entities/attributes/relations).

Reviewer comment (Evaluation):
Some categories are under-represented in the DSG-1k dataset. According to Table 9, six categories have less than 30 examples and VQA-Human correlation is not computed for these categories.

Author response:
We apologize for the confusion introduced here. Please allow us to clarify (also added clarification in the caption the table referred to – Table 13 (Table 9 in the original PDF)):

Specifically, Table 13 of the revised PDF (Table 9 in the original PDF) examines questions generated from **TIFA160 prompts** (160 in total), where we used DSG-1k’s ‘question types’ to present specific scores,  rather than from the whole DSG-1k prompts. TIFA160 is a relatively small sample, so they do not reflect some types of questions such as text rendering. In Table 14 of the revised PDF (Table 10 in the original PDF), we illustrated three T2I models’ VQA accuracy in the full range of DSG-1k’s semantic categories.

Reviewer comment (Evaluation):
It is unclear whether DSG correlates better with humans, compared with TIFA and VQ^A, in each fine-grained category.

Author response:
We agree that a by-category human correlation study is a valuable addition. Following the suggestion, we conducted an additional experiment using TIFA160 prompts, by calculating correlation Likert 1-5 scores with each of their 4 data splits:
- COCO: general purpose;
- PaintSkills: focusing on spatial relations;
- PartiPrompts: focusing on attributional failure cases (e.g., color, counting, etc.);
- DrawBench: focusing on simple compositionality.

As shown in the following table, DSG produces higher human correlation scores on 3 out of 4 splits, as well as the best overall performance. Even in the only data split it does not come on top (i.e., DrawBench), the DSG’s correlation is very close to the top-performing TIFA’s. We incorporated the new experiment result into Appendix D.2 in our revised PDF, thanks for the suggestion!

|        | COCO     | PaintSkills | PartiPrompts | DrawBench | Overall  |
|---|---|---|---|---|---|
| TIFA   | 0.30/0.22| 0.56/0.40  | 0.49/0.37    | **0.59/0.46** | 0.43/0.32|
| VQ^2A  | 0.17/0.13| 0.25/0.17  | 0.28/0.21    | 0.45/0.37 | 0.21/0.16|
| DSG    | **0.53/0.42** | **0.64/0.53**  | **0.61/0.49**    | 0.56/0.44 | **0.57/0.46** |

Reviewer comment (Evaluation):
The relationship between the claimed desired question properties and the final VQA-Human correlation is not well demonstrated. In other words, the relationship between Precision/Recall/Atomicity/Uniqueness in Table 2 and Spearman/Kendall correlation in Table 3 is unclear.

Author response:
For context, we would like to clarify that the Precision/Recall/Atomicity/Uniqueness validation (Table 2) is intended to check the quality of the generated questions on the extent to which they match human judgment, i.e., “are these the right validation questions to ask to check T2I alignment?”, and importantly, achieving strong human correlation in the answers while maintaining the high quality in question generation.

A natural doubt that we expect may arise is: without having the atomicity, semantic coverage, etc. quality we use to evaluate generated questions, TIFA and VQ^2 can still achieve decent human correlation, then what’s the point in having these quality metrics?

We believe the answer lies with **interpretability** and **reliable diagnostics**, especially on the instance/question-level — for any individual question, we strive to achieve an unambiguous understanding of human judgments. For example, a response to a non-atomic question “Is there a blue bike parked next to a door?” would be hard to interpret — if the answer to this question is ‘no’, then is it that the ‘bike’ is not generated properly? or is it that the ‘location of the bike’ is wrong?

Therefore, to summarize:
- With the Precision/Recall/Atomicity/Uniqueness validation, we show that DSG produces high-quality questions per human intuition. This ensures interpretability and reliable diagnostics;
- With the human correlation experiments, we show that DSG results at the same time correlate well (and better than previous work) with human judgments.

Reviewer comment (Evaluation):
There is no comparison with TIFA and VQ^2A in terms of precision and recall in Table 2.

Author response:
We agree it is possible to evaluate TIFA and VQ^2 by eliciting human judgments on precision and recall, however in practice, there exists a failure mode that renders the evaluation not implementable (indicated by their low atomicity in Table 2) — the issue is rooted in how TIFA/VQ^2 and DSG are designed – we clarify as follows:

Note that, due to the absence of the concept of atomicity, we frequently observe TIFA and VQ^2 questions that cover two or more semantic details (e.g. “is there a blue bike next to the door?” covers the entity “bike”, the color of it, and its relative position to another entity “door”). As we argue in the Sec. 1 introduction (the A-B-C-D bullet points on reliability), “loaded questions” like these do not lend themselves well to acquiring unambiguous answers (either by a VQA model or a human rater). 

If we disregard the reliability concern to look at precision/recall alone, loaded questions can score high precision/recall, by simply adding a question mark to the original prompt to form a question (e.g., “A blue motorcycle parked by paint chipped doors.” $\rightarrow$ “A blue motorcycle parked by paint chipped doors?”). This would render the precision/recall validation uninformative.

Our DSG, on the other hand, uses semantic tuples as a structured intermediary semantic representation to handle atomicity – this wards it off the “loaded question” failure mode mentioned above in the precision/recall validation (e.g., if a “loaded question” that maps to more than 1 semantic tuple is produced, it is considered a wrong question).

Reviewer comment (Presentation):
It is stated on page 4 that DSG only involve binary yes/no questions. Why are there multi-choice questions in the human annotation UI in Figure 10?

Author response:
It is true that DSG only involves binary yes/no questions. The questions inside Figure 10 are generated by TIFA, where multiple-choice questions are permitted. We use the same human annotation UI for questions generated by DSG and VQ2A, where we provide yes/no answer choices.

Reviewer comment (Experiments):
Can a better VQA model lead to better alignment in Tables 3, 4, and 6?

Author response:
Yes. This can be found in the results we already report in the paper. Please allow us to clarify:

Empirically, based on the result in Table 5 (per-question VQA-Human matching accuracy), we see a common trend in Tables 3, 4, and 6 — if a VQA model’s result correlates better with human Likert scores, the model also tends to have a higher matching accuracy on the per-question level.

Reviewer comment (Experiments):
Apart from TIFA and VQ2A, more methods could be compared, including CLIPScore and caption based approaches.

Author response:
Please note that we mentioned “For human correlation, Hu et al. (2023) for example achieves 47.2 Kendall’s τ in a sizable correlation analysis with human 1-5 Likert judgments, with CLIPScore τ = 23.1.’’,
For completeness, we also calculated and included correlation results on the 5 single-summary metrics considered in Hu et al. (2023), i.e., BLEU-4, ROUGE-L, METEOR, SPICE, (with BLIP-2 Captioning) and CLIPScore (with CLIP ViT-B/32) in Appendix E in the revised PDF.


|  | Spearman's ρ | Kendall's τ |
|---|---|---|
| **Captioning  (w/ BLIP-2)** | 
| BLEU-4 | 26.1 | 18.3 |
| ROUGE-L | 34.2 | 23.9 |
| METEOR | 37.9 | 26.9 |
| SPICE | 32.9 | 23.6 |
| **Cosine similarity (CLIP ViT-B/32)** | 
| CLIPScore | 27.6 | 19.1 |
| **QG/A frameworks  (w/ PaLI)** | 
| VQ$^2$A | 20.7 | 15.7 |
| TIFA | 43.1 | 32.3 |
| DSG |  **57.0** | **45.8** |

(NOTE: To ensure consistency, we did not directly report the result from Hu et al. (2023) Table 2. Our numbers are calculated with the same human 1-5 Likerts **we** collected – the same Likerts are used in calculating the correlation results in our Table 3. Further, also note that Hu et al. (2023) calculated human Likerts based (by rules, see their Appendix C.1) on raters’ answers to the GPT-3 generated questions, whereas we collected them independently with a single 1-5 consistency question, please see our Figure 12.)

Further, we set focus on fine-grained QG/A as an overall more informative / diagnostic framework than single-summary metrics (represented by CLIPScore), and intend to show that, not only does QG/A work well on the coarse-grained model-level in evaluating performance (as TIFA/VQ^2 show in their comparison to a range of single-summary methods), we can improve it further (with DSG) to provide reliable fine-grained diagnostics.

Reviewer comment (Evaluation):
The comparison of runtime among different methods should be added.

Author response:
We compare the runtime of TIFA and our DSG. For apple-to-apple comparison, we use the same LLM (gpt-3.5-turbo-16k) and VQA model (mPLUG-large). On a single A6000 GPU, we measure the time for obtaining the final T2I alignment score for 20 image-text pairs. We use 20 prompts sampled from TIFA-160 prompts and images generated by Stable Diffusion v2.1.

As in the following result, we find DSG takes a shorter time than TIFA, primarily because TIFA tends to generate more questions than DSG, increasing the inference time of VQA models.

* TIFA: 481s for 20 image-text pairs (24s per image-text pair)
* DSG: 332s for 20 image-text pairs (16s per image-text pair)

Please note that the cost for many image-text pairs could be saved in many ways: e.g., batch processing (separate QG and QA stages, processing multiple images), caching (re-using image features when a VQA model answers to questions paired with the same image), using smaller models (through quantization, distillation, compression), etc.

Reviewer comment (Evaluation):
In table 3, for Instruct-BLIP, the Spearman’s ρ of DSG is lower than that of TIFA, authors should explain this phenomenon briefly.

Author response:
While Spearman's rho shows "Instruct-BLIP & TIFA" > "Instruct-BLIP & DSG", Kendall's Tau shows "Instruct-BLIP & TIFA" < "Instruct-BLIP & DSG". There are some cases with tied ranks, where Kendall’s Tau correlation is more reliable than Spearman’s rho correlation when handling data with tied ranks [1,2].

[1] Taylor, Wilson L. “Correcting the Average Rank Correlation Coefficient for Ties in Rankings.” Journal of the American Statistical Association, vol. 59, no. 307, 1964, pp. 872–76. JSTOR, https://doi.org/10.2307/2283105. Accessed 19 Nov. 2023.

[2] Marie-Therese Puth, Markus Neuhäuser, Graeme D. Ruxton, “Effective use of Spearman's and Kendall's correlation coefficients for association between two measured traits”, Animal Behaviour, Volume 102, 2015, Pages 77-84, ISSN 0003-3472, https://doi.org/10.1016/j.anbehav.2015.01.010.