Paper: Deep-Learning Approaches for Optimized Web Accessibility: Correcting Violations and Enhancing User Experience (ICLR-2024)

Reviewer comment (Novelty):
Although the introduction effectively underscores the importance of enhancing and automating accessibility, it could benefit from a more comprehensive integration within the landscape of the really closely related work. While the authors touch on recent automatic and semi-automatic approaches, they just mention that there have also been studies that considered improving accuracy through semi-automatic techniques of manual correction and even a few automating this correction using prompt engineering (most related), without providing specific examples or literature. If the deployment of prompt engineering is something completely new in this context, the authors must either point this out more clearly or highlight the differences and advantages of their approach compared to the existing state of the art.

Author response:
* introduction has been improved to include better context among relevant work. Notably, we emphasize the novelty of our approach by framing it as a tool for end users vs. companies or web developers, hence the difficulty in comparing our results to existing research.
    * (Section 1) “Currently, no accessibility correction tool exists with the intent of serving users with a tool to correct web pages on the go. While the scope of previous studies was limited to addressing alt-text generation with computer vision, our tool can be used more ubiquitously and purposefully by web users to handle web accessibility violations when desired.”

Reviewer comment (Presentation):
Regarding the cited related work, it should also be noted that the mentioned automatic strategies mainly cover the generation of alternative text for images, and in this context the insufficient quality of the automatic alt-text approach of Facebook is pointed out. This aligns with the statement on page 1, that the most common issue in accessibility is the lack of text equivalents or alternate text for images, but is also incidental after it becomes apparent (page 7) that images cannot be processed at all by the system presented by the authors.

Author response:
> *Since missing alternate text for images is in general a key issue, how would approaches addressing alt-text, such as the one of Facebook, perform on your data set? This could help assess the overall performance (assuming the URLs contain a realistic number of images)*
>
For missing alt-text errors, our approach would likely be able to correct some but not all errors by suggesting possible alt-text using text-based context surrounding the presence of an image. Unlike research that might target individual issues like alt-text, our method employs a holistic strategy to comprehensively correct a wider range of violations present in web content. Our goal is to provide a more versatile and automated solution that can handle a diverse range of accessibility issues. 

Approaches specifically addressing missing alt-text may be able to rectify alt-text issues, but without a comprehensive view of other errors, the improvement in accessibility would still be limited. According to our dataset, there were still at least 6 types of errors that occurred more frequently than alt-text errors in the dataset, and employing an alt-text correcting approach on our dataset would still leave more than 95% of the accessibility errors unresolved.

Reviewer comment (Experiments):
There is no quantitative or qualitative comparison with existing methods. There is unfortunately no substantial evidence presented that the authors' method is in any way at least equivalent to existing approaches, or even superior to them.

Author response:
> *no quantitative or qualitative comparison to existing methods*
>
    * throughout the paper, we have updated the introduction, results, discussion, and conclusion to better incorporate references to existing work and to support the novelty of our work as a tool that can enable users to make websites more accessible as and when needed. To our knowledge, existing methods like ours (employing prompt engineering for automated text-based corrections accessible to users) are not in existence or feasible to compare to

Reviewer comment (Theory):
I don't see the necessity of the mathematical formulas in section 3.4. The expressions appear to artificially introduce complexity, making it more unnecessarily challenging for the reader to comprehend. Rather than introducing variables with no subsequent significance, it would be more effective to use an illustration of the pipeline or at least pseudo code. Especially since the definitions lack clarity and precision, for instance, the term "sub" was used without prior explanation, and both the definition and the necessity of F_{HTML\_fix}'s definition remain incomprehensible, raising doubts that it is mathematically well-defined. Furthermore, most of the variables are unnecessary for defining the terms of R and I. A clear verbal introduction would have sufficed and enhanced comprehension.

Author response:
* math: the purpose of introducing mathematical expressions in section 3.4 is to provide a precise and formal representation of our approach. However, we understand the concern about the potential complexity for the reader. Before introducing the term “sub,” we clarify that “we substitute the incorrect HTML tag with the correction generated.” This explanation logically leads the reader to understand that “sub” serves to substitute an incorrect HTML for the correction. Our goal is to enable readers to grasp the intricacies of our approach without unnecessary complexity. Given the presence of Figure 5 and 6, which illustrate the correction pipeline with visuals, we believe that the combination of figures and mathematical expressions strike a balance between formality and accessibility in understanding.

Reviewer comment (Presentation):
Section 5 is not really a discussion, but a summary. There is no critical examination present.

Author response:
* > *Section 5 is not really a discussion, but a summary.* 
>
The discussion has been updated to include more critical examination of the impact of our research and its significance within a real-world context. (Section 5) “To our knowledge, existing approaches for correcting accessibility violations do not automate the correction process using prompt engineering techniques in a way that can be easily adapted as a tool for users.”

Reviewer comment (Writing):
Typo in Figure 4: "messasges".

Author response:
* missing spaces in Table 2 have been corrected, typo in Figure 4 corrected, citations have been corrected

Reviewer comment (Presentation):
Page 2 and 3 contain faulty links to literature.

Author response:
* missing spaces in Table 2 have been corrected, typo in Figure 4 corrected, citations have been corrected

Reviewer comment (Experiments):
Why is there no comparison to other existing methods?

Author response:
> *Why is there no comparison to other existing methods?*
>
The absence of comparison to existing methods stems from the fact that there are currently no known benchmarks or papers that attempt to do exactly what we are doing, namely employing prompt engineering to correct accessibility violations in an efficient manner. As a result, there is no means to determine whether we are beating the state of the art. Instead of seeing our research as a tool to help developers quickly build accessible tools, we have reframed the practical use case of our approach to be something that an end user can use (e.g. via a Chrome extension) that automatically makes webpages accessible. This therefore avoids a comparison with humans in terms of accuracy when fixing the DOM. 
    * highlight: (Section 6) “Such automated violation correction methods could be adapted into a web-based tool or extension to enable users to correct accessibility violations of websites when needed.”

Reviewer comment (Experiments):
Since missing alternate text for images is in general a key issue, how would approaches addressing alt-text, such as the one of Facebook, perform on your dataset? Even if the authors' method ignores images, this could help to assess the overall performance (assuming the URLs contain a realistic number of images).

Author response:
> *Since missing alternate text for images is in general a key issue, how would approaches addressing alt-text, such as the one of Facebook, perform on your data set? This could help assess the overall performance (assuming the URLs contain a realistic number of images)*
>
For missing alt-text errors, our approach would likely be able to correct some but not all errors by suggesting possible alt-text using text-based context surrounding the presence of an image. Unlike research that might target individual issues like alt-text, our method employs a holistic strategy to comprehensively correct a wider range of violations present in web content. Our goal is to provide a more versatile and automated solution that can handle a diverse range of accessibility issues. 

Approaches specifically addressing missing alt-text may be able to rectify alt-text issues, but without a comprehensive view of other errors, the improvement in accessibility would still be limited. According to our dataset, there were still at least 6 types of errors that occurred more frequently than alt-text errors in the dataset, and employing an alt-text correcting approach on our dataset would still leave more than 95% of the accessibility errors unresolved.

Reviewer comment (Experiments):
How were the 25 URLs in the dataset selected? Which URLs were previously used by related work? Why weren't the same ones used, or is there no information available?

Author response:
> *How were the 25 URLS in the dataset selected? Which URLs were previously used by related work? Why weren’t the same ones used, or is there no information available?*
>
Currently, there is no available information on URLs employed in related work. In fact, to our knowledge, no research of similar approach exists in which the researchers test a collection of URLs and make the selected URLs publicly available. Playwright can only perform tests on URLs beginning with “http.” Hence, we hand-selected 25 URLs in an effort to diversify represented categories (see Table 3). In our new PDF, we have also added a line in Section 2.1 detailing some example URLs: “Section 2.1 “Some example websites included were Google Calendar, Slack, Audible, BBC, Quora, New York Times, and ResearchGate.” Future research would involve more efforts to incorporate a larger range of URLs to examine our approach as a tool in multiple contexts.

Reviewer comment (Writing):
Typo in line "Section 2 presents our the creation of our dataset benchmark system for evaluating error correction."

Author response:
We thank the reviewer for their feedback and for acknowledging our innovative approach and its impact on digital inclusivity. 

In our new pdf, we have addressed the following weaknesses to provide a stronger and clearer interpretation of our results:
* typo in Section 2 resolved

Reviewer comment (Evaluation):
Results from Table 2 to 5 are not discussed. Since the paper can be organised in a compact way, authors can discuss the result to give more clarity.

Author response:
We thank the reviewer for their feedback and for acknowledging our innovative approach and its impact on digital inclusivity. 

In our new pdf, we have addressed the following weaknesses to provide a stronger and clearer interpretation of our results:
* more in-depth discussion of results from Tables 2 through 5 and their significance has been included in the paper
    * Discussed the connection between the most frequent errors (visual/layout-based errors) and the model’s capability to correct text-based errors more easily than more semantic or visual errors 
    * (section 4) “As detailed in Table 4, the model demonstrated strong proficiency in addressing text-based web accessibility issues, achieving 100% success rates in correcting violations of landmark-no-duplicate-content, label, skip-link, and ARIA-required-attribute”
    * (section 4) “Future work, as indicated by Table 3, could also focus on datasets primarily consisting of Government/Important, Research, and News/Information websites, as these categories typically have higher numbers of text-based violations and to test what specific text-based violations the model performs better in.”

Reviewer comment (Writing):
FEW-SHOT GUIDED PROMPTING could be written more elaborately. Some of the questions: what does "by increasing the examples provided in the system message" mean?

Author response:
We thank the reviewer for their feedback and for acknowledging our innovative approach and its impact on digital inclusivity. 

In our new pdf, we have addressed the following weaknesses to provide a stronger and clearer interpretation of our results:
* Few-Shot Guided Prompting (Section 3.2) has been described in more detail. “the Few-Shot prompt included only the incorrect and correct lines of HTML and did not include a thought. The examples included a missing header, a missing image description, a missing href URL, and a missing landmark as violation errors.”

>*What does “by increasing the examples provided in the system message” mean?*
>
 “Increasing the examples provided in the system message” refers to including more than one example error in the system message. For example, Figure 4 illustrates a system message with one example of a missing heading text error. To improve the performance of our model, we could experiment with including additional errors to provide more context. For example, we could add a color contrast error with an example correction, as well as a button name error with an example correction. 

To clarify this, we updated Section 3.2 with the explanation “We implemented Few-Shot guided prompting by providing more violation-correction examples in our system message.”

Reviewer comment (Evaluation):
As above in weaknesses - major concern on discussion of results.

Author response:
We thank the reviewer for their feedback and for acknowledging our innovative approach and its impact on digital inclusivity. 

In our new pdf, we have addressed the following weaknesses to provide a stronger and clearer interpretation of our results:
* discussion of results has been improved to add more clarity

Reviewer comment (Reproducibility):
Sec 3.3 was unclear: What system was created for generating the prompts?

Author response:
Section 3.3 has been updated to include more detail on the process of generating the prompts. “We generated the system and user messages by accessing specific columns of the dataset for each error and formatting the information through prompt engineering (see Figure 4 for an example).”

Reviewer comment (Experiments):
The types of websites in the study were also unclear, e.g., were these static/dynamic papers?

Author response:
The types of websites used for our benchmark were all dynamic. For example, we included websites such as New York Times, Audible, and Google Calendar. However, the websites were fed through Playwright based on their state at the time of dataset creation. 

In the new PDF, we listed some example websites in our dataset in Section 2.1 (“Some example websites included were Google Calendar, Slack, Audible, BBC, Quora, New York Times, and ResearchGate.”)

Reviewer comment (Experiments):
There needs to be a baseline to compare the performance of the approach. The score allows us to compare the different prompting techniques but not judge whether the approach overall is better than humans with accessibility testing tools.

Author response:
Overall, we reframed our approach to be more of a tool for individual end users, instead of for web developers or companies. As a result, a comparison to existing methods is unfeasible, but the ability to decrease accessibility violations is notable for serving users with impairments.

Reviewer comment (Novelty):
A few references were missing — pages 2 and 3.

Author response:
References have been corrected and updated

Reviewer comment (Presentation):
"table II below" — should this be "Table 2"? (page 7)

Author response:
Table 2 reference in Section 4 has been updated as “table 2”

Reviewer comment (Writing):
What is ARIA; define?

Author response:
Section 4 includes a brief definition of ARIA: “ARIA, or Accessible Rich Internet Applications, is a suite of attributes and roles designed to enhance the semantic value of web elements and complex interfaces, particularly for assistive technologies, by conveying states, properties, and roles beyond what's inherently available in HTML.”

Reviewer comment (Experiments):
What types of websites were used: dynamic or static? Were there React apps or similar apps? OR Would this approach work for React or similar apps?

Author response:
The types of websites used for our benchmark were all dynamic. For example, we included websites such as New York Times, Audible, and Google Calendar. However, the websites were fed through Playwright based on their state at the time of dataset creation. 

In the new PDF, we listed some example websites in our dataset in Section 2.1 (“Some example websites included were Google Calendar, Slack, Audible, BBC, Quora, New York Times, and ResearchGate.”)

There were some websites that were built using React, including Collegeboard, SoundCloud, Vimeo, BBC, Eventbrite, GrubHub, New York Times, Patreon, and ResearchGate. This approach worked for these websites, and they were able to correct several of their violations.

Reviewer comment (Experiments):
Were there any perfect web pages? If yes, what did the proposed model do in that case?

Author response:
There were no web pages within our dataset that had zero accessibility errors. This is due to the fact that when run through Playwright, a web page without any violation errors would not return any results and therefore could not fill up dataset columns. However, based on some tests conducted during our research, if a web page has zero violations, the LLM may return a “correction” that results in the same initial HTML tag (and thus if the replacement process were applied using the LLM’s response, there would be no change in the DOM).

Reviewer comment (Evaluation):
What was the maximum number of violations observed?

Author response:
In our dataset, the maximum number of violations initially observed was 13. It is likely that this number could be much larger if we expanded our dataset to include more URLs.

Reviewer comment (Writing):
Were the typos in the "Suggested change" part of Fig 4 intentional? Was this an actual example?

Author response:
Typos in Fig 4 have been updated. This is an example of a system message and user message that would be fed into the models.

Reviewer comment (Experiments):
How many system and user messages were generated?

Author response:
Each violation generated one system message and one user message. So since we had 171 total rows in our dataset, 171 system messages and 171 user messages were generated.