Paper: Alt-Text with Context: Improving Accessibility for Images on Twitter (ICLR-2024)

Reviewer comment (Evaluation):
There is a lack of clarity on how the BLEU@4 score is calculated in the evaluation. The reviewer notes that the BLEU score generally ranges from 0 to 1 and measures the precision of n-grams in generated text compared to the ground truth. However, the paper reports a BLEU@4 score of around 1.8, which is unusual. The reviewer suggests the authors clarify the calculation method (e.g., whether a modified version was used) and provide better guidance on how to interpret these BLEU@4 values.

Author response:
Regarding BLEU score, we are using a publicly available implementation of all automatic metrics from pycocoevalcap, the same one used by ClipCap and several other papers. As indicated in the Table 1 caption, all numbers are x100 (i.e. BLEU ranges 0-100), which is standard practice for reporting in much of the prior literature.

Reviewer comment (Reproducibility):
The paper lacks important and adequate methodological details. It refers readers to several appendices for more information, but no appendices are included in the manuscript. This diminishes readability and prevents assessment of the quality and robustness of the results. The reviewer recommends including the appendices to explain how the user study was conducted, how participants were recruited, and their backgrounds and expertise regarding best practices for generating accessibility descriptions. These details are essential for understanding the research quality.

Author response:
We tried to follow ICLR’s instructions for supplementary material by uploading it separately from the main submission PDF. The appendix is in its own PDF wrapped in a zip file, which you should see listed under “Supplementary Material”. At least from our end it can be downloaded and viewed from the OpenReview submission page. Please let us and the chairs know if you still cannot access it.

Reviewer comment (Reproducibility):
There are concerns about the paper’s approach to releasing the dataset. Given recent changes to Twitter’s API, it has become extremely difficult to rehydrate tweets from their IDs. As a result, simply releasing Twitter IDs and media URLs will not enable researchers to reproduce results or further study the problem. The reviewer suggests the authors release more dataset attributes—specifically, the tweet’s text—to allow interested researchers to reproduce results without relying on the closed and expensive new Twitter APIs.

Author response:
Regarding rehydration, we understand your concerns and wish that there was a more convenient way to release our dataset as we do believe in the importance of our research being replicable and easy to follow up on. However we have to balance this with users’ right to be forgotten and the website’s terms of use. If a user no longer wishes for their image and associated tweet to be either online or in an open training set and deletes them, we have a responsibility to comply with that. We’re in a tough situation here as the API was changed while this paper was in progress, but we believe that this approach has the best tradeoff between facilitating reproducibility and respecting user rights. If you have other suggestions for how we could do this, we’re certainly open to hearing them.

Reviewer comment (Evaluation):
The paper does not properly explain how the qualitative assessment described in Section 6.4 is performed. This omission makes it unclear whether the assessment was systematic and whether its insights are representative or generalizable. The reviewer recommends including more details on how samples for the quantitative analysis are selected and how the qualitative assessment is conducted, including whether the assessors are domain experts in accessibility description generation and aware of best practices.

Author response:
The human evaluation was conducted in a systematic way – there’s some information in 5.2 and more in Appendix E (see above). To summarize, we randomly selected 1k examples from our test set and hired annotators from amazon mechanical turk to assess model outputs. Turkers were given detailed instructions on the task, shown contrasting examples of both well and poorly written alt-text, and asked to read a document with further details on best practices according to Twitter’s accessibility guidelines. Before allowing them to participate in the actual study, we also conducted a dummy round on a small number of tweets with two candidate alt-text descriptions each. Only turkers who correctly selected the best alt-text for all of these were allowed into the next round which we report numbers on in our paper.

Reviewer comment (Novelty):
The novelty of the proposed method is really limited excpet the tweet-text-based reranking.

Author response:
Thank you for your time and feedback! We’d argue that the tweet text reranking is far from the most novel thing in our model (as well as not being the primary source of our quantitative gains), rather the way we approach alt-text generation (which is in general understudied) by way of *contextual text conditional* image captioning is in our opinion more new and interesting, and also accounts for more of our numerical performance improvement over prior work. While work exists that has projected image features into text space, there’s much less precedent for our way of creating a prefix from multiple modalities, which we empirically demonstrate is what’s responsible for our gains.

Reviewer comment (Experiments):
The experiment is somewhat not extensive. For example, from Table 1, it seems that the tweet-based reranking is the most important component. But the authors did not tried to incorporate the reranking with the baselines, which is not fair.

Author response:
On the topic of reranking though, we’d like to clarify our experiments – and critically emphasize that as stated in our paper, we *did* try reranking on ClipCap and found that it yielded similar results. We’ll try to make that more clear in the next revision. Specifically we found that it improved performance there as well by a similar amount, for example boosting BLEU from 0.681 to 0.793 (we’ll include those numbers in the next revision as well). You’ll notice by comparing the two bolded rows in the table that improvement from reranking itself is similarly small but noticeable for our model also. But we want to strongly emphasize that outside of the ablation at the bottom section of the table, the comparisons between our model and the baselines are apples-to-apples as far as the decoding strategy and therefore fair to directly compare. Column 2 indicates the decoding strategy used for each row, and you can see rows corresponding to our model and ClipCap decoded with the exact same strategy (i.e. no reranking for either). In fact the primary gain comes not from reranking but from the specific combination of beam search with no repeats *which is exactly what we do for ClipCap in all experiments* even though they themselves in their original paper do not.