Original: VUT:
Revised: VUT:

Original: V ERSATILE
Revised: V ERSATILE

Original: UI
Revised: UI

Original: T RANSFORMER
Revised: T RANSFORMER

Original: FOR
Revised: FOR

Original: M ULTIM ODAL M ULTI -T ASK U SER I
Revised: M ULTIM ODAL M ULTI -T ASK U SER I

Original: NTERFACE M ODELING
Revised: NTERFACE M ODELING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language.
Revised: User interface modeling is inherently multimodal, which involves several distinct types of data: images, structures and language.

Original: The tasks are also diverse, including object detection, language generation and grounding.
Revised: The tasks are also diverse, including object detection, language generation and grounding.

Original: In this paper, we present
Revised: In this paper, we present

Original: VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model.
Revised: VUT, a Versatile UI Transformer that takes multimodal input and simultaneously accomplishes 5 distinct tasks with the same model.

Original: Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input.
Revised: Our model consists of a multimodal Transformer encoder that jointly encodes UI images and structures, and performs UI object detection when the UI structures are absent in the input.

Original: Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both question-answering and command grounding with respect to the UI.
Revised: Our model also consists of an auto-regressive Transformer model that encodes the language input and decodes output, for both question-answering and command grounding with respect to the UI.

Original: Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT has achieved accuracy either on par with or exceeding the accuracy when the model is trained for  individual tasks separately.
Revised: Our experiments show that for most of the tasks, when trained jointly for multi-tasks, VUT substantially reduces the number of models and footprints needed for performing multiple tasks, while achieving accuracy exceeding or on par with baseline models trained for each individual task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Modern graphical user interfaces speciﬁcally touchscreen mobile UIs enable a rich problem space for modeling where the input is inherently multimodal, which consists of several distinct types of data.
Revised: Modern graphical user interfaces speciﬁcally touchscreen mobile UIs enable a rich problem space for modeling where the input is inherently multimodal, which consists of several distinct types of data.

Original: A user interface screen exists in both a visual form, i.e., a screenshot, and a structural representation, i.e., a tree-like view hierarchy.
Revised: A user interface screen exists in both a visual form, i.e., a screenshot, and a structural representation, i.e., a tree-like view hierarchy.

Original: Based on graphical user interfaces, there is a wide spectrum of modeling tasks that either directly enhance user experiences or advance the development of intelligent user interfaces.
Revised: Based on graphical user interfaces, there is a wide spectrum of modeling tasks that either directly enhance user experiences or advance the development of intelligent user interfaces.

Original: For example, previous work developed models and datasets for grounding a language command into an executable UI action (Li et al., 2020a), generating language description for accessibility on mobile devices (Li et al., 2020b; Wang et al., 2021), and understanding the usability of user interfaces (Swearngin & Li, 2019) or identifying the objects on the screen (Zhang et al., 2021).
Revised: For example, previous work developed models and datasets for grounding a language command into an executable UI action (Li et al., 2020a), generating language description for accessibility on mobile devices (Li et al., 2020b; Wang et al., 2021), and understanding the usability of user interfaces (Swearngin & Li, 2019) or identifying the objects on the screen (Zhang et al., 2021).

Original: Previous work has also started learning effective representation of user interface screens (He et al., 2020; Li et al., 2021a), which can potentially beneﬁt downstream tasks.
Revised: Previous work has also started learning effective representation of user interface screens (He et al., 2020; Li et al., 2021a), which can potentially beneﬁt downstream tasks.

Original: Although these previous works have made progress in addressing individual problems, it is important to investigate the feasibility of learning all these tasks with a single model.
Revised: Although these previous works have made progress in addressing individual problems, it is important to investigate the feasibility of learning all these tasks with a single model.

Original: In addition to achieving a scientiﬁc understanding of how these UI tasks are related, it is extremely valuable to obtain such a multi-task model, which can potentially reduce the number of models that need to be developed and deployed.
Revised: In addition to achieving a scientiﬁc understanding of how these UI tasks are related, it is extremely valuable to obtain such a multi-task model, which can potentially reduce the number of models that need to be developed and deployed.

Original: This is crucial for mobile devices that have limited computing resources.
Revised: This is crucial for mobile devices that have limited computing resources.

Original: In this work, we propose VUT—Versatile UI Transformer, which handles three types of data: images, structures (view hierarchies) and language, and simultaneously performs ﬁve unique tasks that are representative in the UI modeling literature, including UI object detection, natural language command grounding, widget captioning, screen summarization and UI tappability prediction.
Revised: In this work, we propose VUT—Versatile UI Transformer, which handles three types of data: images, structures (view hierarchies) and language, and simultaneously performs ﬁve unique tasks that are representative in the UI modeling literature, including UI object detection, natural language command grounding, widget captioning, screen summarization and UI tappability prediction.

Original: A major challenge we need to address is how to unify these distinct tasks as well as their heterogeneous datasets such that they can be learned by a single model.
Revised: A major challenge we need to address is how to unify these distinct tasks as well as their heterogeneous datasets such that they can be learned by a single model.

Original: To this end, we devise a general formulation for UI modeling tasks based on ﬁve inherent types of information that deﬁne a task.
Revised: To this end, we devise a general formulation for UI modeling tasks based on ﬁve inherent types of information that deﬁne a task.

Original: We also aim to design a compact model architecture such that it remains stable for addressing a diverse and potentially growing set of tasks, for which we make each model component multi-purpose.
Revised: We also aim to design a compact model architecture such that it remains stable for addressing a diverse and potentially growing set of tasks, for which we make each model component multi-purpose.

Original: Speciﬁcally, VUT comprises two Transformer architectures (Figure 1): the Image-Structure model, and the Question-Answer model.
Revised: Speciﬁcally, VUT comprises two Transformer architectures (Figure 1): the Image-Structure model, and the Question-Answer model.

Original: The Image-Structure model encodes the entire screenshot of a UI along its view hierarchy tree, with early fusion of the two modalities, which is guided by a focus map when a given object is inquired.
Revised: The Image-Structure model encodes the entire screenshot of a UI along its view hierarchy tree, with early fusion of the two modalities, which is guided by a focus map when a given object is inquired.

Original: In addition to being the UI encoder, the Image-Structure modelpredicts UI objects when the view hierarchy is absent on the input,which achieves the UI object
Revised: In addition to being the UI encoder, the Image-Structure model

Original: [Figure]
Revised: [Figure]

Original:  detection task.
Revised: predicts UI objects when the view hierarchy is absent on the input,which achieves the UI object detection task.

Original: The Question-Answer model encodes a question while attending to the UI encodings from the Image-Structure model.
Revised: The Question-Answer model encodes a question while attending to the UI encodings from the Image-Structure model.

Original: It decodes a text answer when the task response is language, e.g., widget captioning (Li et al., 2020c) or screen summarization (Wang et al., 2021).
Revised: It decodes a text answer when the task response is language, e.g., widget captioning (Li et al., 2020c) or screen summarization (Wang et al., 2021).

Original: For the grounding task whose output is an object reference, the Question-Answer model serves as the question encoder and its hidden state is used to locate UI elements to be acted upon.
Revised: For the grounding task whose output is an object reference, the Question-Answer model serves as the question encoder and its hidden state is used to locate UI elements to be acted upon.

Original: We highlight the relation of VUT with previous works in Table 1, and discuss their differences further in the following section.
Revised: We highlight the relation of VUT with previous works in Table 1, and discuss their differences further in the following section.

Original: We experiment with our model on 5 datasets, and compare the accuracy of VUT when it is trained alone for each task and jointly with multiple tasks.
Revised: We experiment with our model on 5 datasets, and compare the accuracy of VUT when it is trained alone for each task and jointly with multiple tasks.

Original: Our experiments show that VUT is able to perform all the ﬁve tasks simultaneously and achieve the performance on par with or surpass that when each task is learned alone.
Revised: Our experiments show that VUT is able to perform all the ﬁve tasks simultaneously and achieve the performance on par with or surpass that when each task is learned alone.

Original: The main contributions of our work are as follows.
Revised: The main contributions of our work are as follows.

Original: • We formulate multi-modal multi-task learning for a new domain—graphical user interfaces— with one model to accomplish a wide range of tasks for enhancing mobile user experiences.
Revised: • We formulate multi-modal multi-task learning for a new domain—graphical user interfaces— with one model to accomplish a wide range of tasks for enhancing mobile user experiences.

Original: • We design VUT based on a two-tower Transformer architecture, one for handling imagestructure and the other for language data, where each Transformer is multi-purpose by both encoding and decoding its own modality, with cross-tower attention.
Revised: • We design VUT based on a two-tower Transformer architecture, one for handling imagestructure and the other for language data, where each Transformer is multi-purpose by both encoding and decoding its own modality, with cross-tower attention.

Original: • We experiment with VUT on 5 distinct UI task, and thoroughly investigated the effect of these tasks when learned alone or jointly with ablations and analysis, which show the feasibility for achieving diverse UI tasks using a single model. [Table]
Revised: • We experiment with VUT on 5 distinct UI tasks, and thoroughly investigated the effect of these tasks when learned alone or jointly with ablations and analysis, which show the feasibility for achieving diverse UI tasks using a single model, which offers the value for reducing the number of models and storage footprints needed for deployment (Appendix E).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Extensive work has been conducted in multi-modal modeling with vision and languages (Li et al., 2019;
Revised: Extensive work has been conducted in multi-modal modeling with vision and languages (Li et al., 2019;

Original: Lu et al., 2019; Hu & Singh, 2021; Lu et al., 2020; Tan & Bansal, 2019;
Revised: Lu et al., 2019; Hu & Singh, 2021; Lu et al., 2020; Tan & Bansal, 2019;

Original: Kim et al., 2021;
Revised: Kim et al., 2021;

Original: Zhou et al., 2020; Gupta et al., 2021).
Revised: Zhou et al., 2020; Gupta et al., 2021).

Original: Existing works differ in the form of input they consume and the objectives of modeling.
Revised: Existing works differ in the form of input they consume and the objectives of modeling.

Original: One category of work focuses on pretraining to learn an effective cross-modality representation for downstream tasks and the other directly learns multiple tasks end-to-end (Table 1).
Revised: One category of work focuses on pretraining to learn an effective cross-modality representation for downstream tasks and the other directly learns multiple tasks end-to-end (Table 1).

Original: VUT belongs to the latter.
Revised: VUT belongs to the latter.

Original: In terms of the forms of multimodal data these models consume, most models handle image and text input.
Revised: In terms of the forms of multimodal data these models consume, most models handle image and text input.

Original: However, a unique form of data in UI modeling is the structure input of view hierarchies, which only VUT, UIBert and ActionBert use. 
Revised: However, a unique form of data in UI modeling is the structure input of view hierarchies, which only VUT, UIBert and ActionBert use. [Table]

Original: Many existing works feed object regions, instead of the entire image to the model, which requires a pretrained object detection model (Lu et al., 2019; Bai et al., 2021;
Revised: Many existing works feed object regions, instead of the entire image to the model, which requires a pretrained object detection model (Lu et al., 2019; Bai et al., 2021;

Original: He et al., 2020; Zhou et al., 2020;
Revised: He et al., 2020; Zhou et al., 2020;

Original: Tan & Bansal, 2019; Lu et al., 2020) or address tasks only regarding the entire image (Kim et al., 2021; Hu & Singh, 2021).
Revised: Tan & Bansal, 2019; Lu et al., 2020) or address tasks only regarding the entire image (Kim et al., 2021; Hu & Singh, 2021).

Original: Although ActionBert (He et al., 2020) and UIBert (Bai et al., 2021) also address the UI domain, they are targeted for representation learning, and do not support multiple tasks simultaneously.
Revised: Although ActionBert (He et al., 2020) and UIBert (Bai et al., 2021) also address the UI domain, they are targeted for representation learning, and do not support multiple tasks simultaneously.

Original: As a result, they do not deal with language input of task descriptions.
Revised: As a result [Table] they do not deal with language input of task descriptions.

Original: Their text input is those scraped from the 
Revised: Their text input is those scraped from the UI screen, e

Original: UI screen, e.g., using OCR.
Revised: [Table] g., using OCR.

Original: In addition, these models require predetermined object regions similar to many BERT-based multi-modal models.
Revised: In addition, these models require predetermined object regions similar to many BERT-based multi-modal models.

Original: In contrast, object detection is one of the tasks that VUT addresses.
Revised: In contrast, object detection is one of the tasks that VUT addresses.

Original: In terms of modeling techniques, the work that is closely related to ours is GPV-I (Gupta et al., 2021), which uses DETR (Carion et al., 2020) for object detection, and ViBERT (Lu et al., 2019) for multimodal modeling.
Revised: In terms of modeling techniques [Table] we designed a novel Transformer architecture for multi-task modeling of the UI domain, based on building blocks previously proposed for natural images and language, e.g., (Hu et al., 2020; Lu et al., 2020). the work that is closely related to ours is GPVI (Gupta et al., 2021), which uses DETR (Carion et al., 2020) for object detection, and ViBERT (Lu et al., 2019) for multimodal modeling.

Original: In addition to the obvious deviation our work, e.g., VUT uses structure input but GPV-I does not, there are several important architecture differences.
Revised: In addition to the obvious deviation our work, e.g., VUT uses structure input but GPV-I does not, there are several important architecture differences.

Original: While GPV-I directly embeds DETR, an encoder-decoder model, into its architecture, VUT uses a single tower design where both the image and object queries are fed to the same Transformer encoder.
Revised: While GPV-I directly embeds DETR, an encoder-decoder model, into its architecture, VUT uses a single tower design where both the image and object queries are fed to the same Transformer encoder.

Original: This design choice is motivated by our goal to achieve a compact architecture, which the Image-Structure model serves both image-structure encoding and object detection (when the structure input is absent in the input).
Revised: This design choice is motivated by our goal to achieve a compact architecture, which the Image-Structure model serves both image-structure encoding and object detection (when the structure input is absent in the input).

Original: As shown in our experiment, the single tower architecture of VUT’s Image-Structure model showed clear advantage over the encoder-decoder architecture in DETR for the UI object detection task.
Revised: As shown in our experiment, the single tower architecture of VUT’s Image-Structure model showed clear advantage over the encoder-decoder architecture in DETR for the UI object detection task.

Original: To address the unique domin of UI tasks, we also introduce focus map to guide the model towards the object being inquired.
Revised: To address the unique domin of UI tasks, we also introduce focus map to guide the model towards the object being inquired.

Original: VUT’s question-answer Transformer is designed based on existing auto-regressive multi-task language models (Raffel et al., 2019; Brown et al., 2020) where a question or a command is fed to the model as a preﬁx, and the responses are decoded token by token.
Revised: VUT’s question-answer Transformer is designed based on existing auto-regressive multi-task language models (Raffel et al., 2019; Brown et al., 2020) where a question or a command is fed to the model as a preﬁx, and the responses are decoded token by token.

Original: One difference is that for the language command grounding task, instead of generating a language response, the last hidden state of the language model is used, as a question encoding, to retrieve a UI object on the screen.
Revised: One difference is that for the language command grounding task, instead of generating a language response, the last hidden state of the language model is used, as a question encoding, to retrieve a UI object on the screen.

Original:  P ROBLEM F ORMULATION
Revised:  P ROBLEM F ORMULATION

Original: A graphical user interface contains a collection of UI elements for fulﬁlling a coherent set of tasks.
Revised: A graphical user interface contains a collection of UI elements for fulﬁlling a coherent set of tasks.

Original: There are often ﬁve types of data involved to formulate a UI task: < S, V, T, Q, A > (Figure 1).
Revised: There are often ﬁve types of data involved to formulate a UI task: < S, V, T, Q, A > (Figure 1).

Original: S is the screenshot image that captures the visual appearance of the UI screen.
Revised: S is the screenshot image that captures the visual appearance of the UI screen.

Original: V is the view hierarchy tree that represents the underlying structure of the screen.
Revised: V is the view hierarchy tree that represents the underlying structure of the screen.

Original: T is the target object on the screen to be inquired.
Revised: T is the target object on the screen to be inquired.

Original: Q is the natural language description of the task, which can be an open-ended question such as " What is the caption of the element? ", a yes-or-no question such as " Does the object look clickable? " or a command such as " Click on the Next button. ".
Revised: Q is the natural language description of the task, which can be an open-ended question such as " What is the caption of the element? ", a yes-or-no question such as " Does the object look clickable? " or a command such as " Click on the Next button. ".

Original: See the full list of Q used in our experiments in Appendix A.
Revised: See the full list of Q used in our experiments in Appendix A.

Original: Finally, A is the natural language answer to the question Q when the form of the response for the task is supposed to be natural language.
Revised: Finally, A is the natural language answer to the question Q when the form of the response for the task is supposed to be natural language.

Original: Depending on each task setup, these data types appear as either input or output.
Revised: Depending on each task setup, these data types appear as either input or output.

Original: We elaborate on the formation of each task here, and use F to denote the function for achieving each task.
Revised: We elaborate on the formation of each task here, and use F to denote the function for achieving each task.

Original: 3. UI O BJECT D ETECTION
Revised: 3. UI O BJECT D ETECTION

Original: Given the screenshot image, S , the task is to detect each UI element on the screen, such as Text Field,
Revised: Given the screenshot image, S , the task is to detect each UI element on the screen, such as Text Field,

Original: Toggle Button, or Image View.
Revised: Toggle Button, or Image View.

Original: This task is similar to the typical object detection task in natural images (Carion et al., 2020) or recent UI object detection work (Zhang et al., 2021).
Revised: This task is similar to the typical object detection task in natural images (Carion et al., 2020) or recent UI object detection work (Zhang et al., 2021).

Original: However, our task is more challenging in that it needs to detect different types of container objects, which determine how UI objects are visually structured of the screen, such as Linear Layout, Frame Layout or List.
Revised: However, our task is more challenging in that it needs to detect different types of container objects, which determine how UI objects are visually structured of the screen, such as Linear Layout, Frame Layout or List.

Original: In total there are 21 types of leaf or non-leaf objects in a view hierarchy.
Revised: In total there are 21 types of leaf or non-leaf objects in a view hierarchy.

Original: See Appendix D for the full list of objects we detect.
Revised: See Appendix D for the full list of objects we detect.

Original: UI object detection is important for improving accessibility and enabling other intelligent features such as UI adaptation when view hierarchy is not available.
Revised: UI object detection is important for improving accessibility and enabling other intelligent features such as UI adaptation when view hierarchy is not available.

Original: As a screen understanding task, UI object detection is beneﬁcial to other UI modeling tasks as we will show in our experiments.
Revised: As a screen understanding task, UI object detection is beneﬁcial to other UI modeling tasks as we will show in our experiments.

Original: The task is formulated as the follow (Equation 1).
Revised: The task is formulated as the follow (Equation 1).

Original: [Equation]
Revised: [Equation]

Original: Note that this task is achieved solely based on the single-tower Image-Structure Transformer (Figure 1) and does not rely on the question-answer model.
Revised: Note that this task is achieved solely based on the single-tower Image-Structure Transformer (Figure 1) and does not rely on the question-answer model.

Original: V ∅ , T ∅ and Q ∅ represent each type of data masked out or missing in the input.
Revised: V ∅ , T ∅ and Q ∅ represent each type of data masked out or missing in the input.

Original: 3. W IDGET C APTIONING
Revised: 3. W IDGET C APTIONING

Original: Generating natural language description for user interface elements is important for accessibilityand language-based interaction in general.
Revised: Generating natural language description for user interface elements is important for accessibilityand language-based interaction in general.

Original: The widget captioning task was initially proposed by Li et al.
Revised: The widget captioning task was initially proposed by Li et al.

Original: (2020b)
Revised: (2020b)

Original: and it extends the classic image captioning tasks (Xu et al., 2015) to the UI domain.
Revised: and it extends the classic image captioning tasks (Xu et al., 2015) to the UI domain.

Original: In this task, given the UI view hierarchy, V , the screenshot image, S , and the target element to be captioned, T , the model predicts a natural language phrase, A , that best describes the functionality of the object (Equation 2).
Revised: In this task, given the UI view hierarchy, V , the screenshot image, S , and the target element to be captioned, T , the model predicts a natural language phrase, A , that best describes the functionality of the object (Equation 2).

Original: [Equation]
Revised: [Equation]

Original: The model uses the information of S , V and T via the Image-Structure model.
Revised: The model uses the information of S , V and T via the Image-Structure model.

Original: The examples of Q are "What is the caption of the element?"
Revised: The examples of Q are "What is the caption of the element?"

Original: and "What best describes the object?", and the examples of
Revised: and "What best describes the object?", and the examples of

Original: A are "Forward", and "Shopping Cart".
Revised: A are "Forward", and "Shopping Cart".

Original: 3. S CREEN S UMMARIZATION
Revised: 3. S CREEN S UMMARIZATION

Original: Instead of focusing on an individual element as the widget captioning task.
Revised: Instead of focusing on an individual element as the widget captioning task.

Original: screen summarization that is recently proposed by Wang et al.
Revised: screen summarization that is recently proposed by Wang et al.

Original: (2021) is the task that describes the entire UI screen (Equation 3) by producing a summary phrase.
Revised: (2021) is the task that describes the entire UI screen (Equation 3) by producing a summary phrase.

Original: [Equation]
Revised: [Equation]

Original: The examples of task prompts, Q , for screen summarization are "What is the description of the screen?" and "What best summarizes the UI?"
Revised: The examples of task prompts, Q , for screen summarization are "What is the description of the screen?" and "What best summarizes the UI?"

Original: Q signals the model the type of responses it should generate when it is jointly trained with other tasks such as widget captioning.
Revised: Q signals the model the type of responses it should generate when it is jointly trained with other tasks such as widget captioning.

Original: The screen summarization task is broadly related to multimodal summarization tasks in the literature, but is speciﬁc to the user interface domain.
Revised: The screen summarization task is broadly related to multimodal summarization tasks in the literature, but is speciﬁc to the user interface domain.

Original: 3. L ANGUAGE C OMMAND G ROUNDING
Revised: 3. L ANGUAGE C OMMAND G ROUNDING

Original: An important feature of modern smartphone interfaces is to interpret the natural language command of users as executable actions, e.g., Voice Control 2 .
Revised: An important feature of modern smartphone interfaces is to interpret the natural language command of users as executable actions, e.g., Voice Control 2 .

Original: Previous work has investigated language grounding on user interfaces (Pasupat et al., 2018;
Revised: Previous work has investigated language grounding on user interfaces (Pasupat et al., 2018;

Original: Li et al., 2020a).
Revised: Li et al., 2020a).

Original: In this task, given the UI, S and V , and the language command, Q , the model needs to predict which object on the screen can fulﬁll the command, which is the opposite to the two preceding tasks of language generation.
Revised: In this task, given the UI, S and V , and the language command, Q , the model needs to predict which object on the screen can fulﬁll the command, which is the opposite to the two preceding tasks of language generation.

Original: [Equation]
Revised: [Equation]

Original: Note that instead of generating a natural language response like widget caption and screen summarization, this task locates the target object, T , on the screen.
Revised: Note that instead of generating a natural language response like widget caption and screen summarization, this task locates the target object, T , on the screen.

Original: In our dataset, we asked labelers to refer to an object on the screen in an unconstrained manner.
Revised: In our dataset, we asked labelers to refer to an object on the screen in an unconstrained manner.

Original: The possibility of Q is unbounded, which can be any phrase the user feels like using for manipulating the UI, e.g., "Go to the next screen", or "Tap on the checkout button".
Revised: The possibility of Q is unbounded, which can be any phrase the user feels like using for manipulating the UI, e.g., "Go to the next screen", or "Tap on the checkout button".

Original: A command can also refer to an object indirectly using the relation to others, such as "Click the icon to the right of the search box."
Revised: A command can also refer to an object indirectly using the relation to others, such as "Click the icon to the right of the search box."

Original: 3. T APPABILITY P REDICTION
Revised: 3. T APPABILITY P REDICTION

Original: Lastly, we include a task related to automatic usability assessment.
Revised: Lastly, we include a task related to automatic usability assessment.

Original: Whether a user perceives a UI object as tappable is an important usability issue.
Revised: Whether a user perceives a UI object as tappable is an important usability issue.

Original: The mismatch between the tappability of an object as perceived by the user and its actual clickability has constantly plagued mobile user experiences.
Revised: The mismatch between the tappability of an object as perceived by the user and its actual clickability has constantly plagued mobile user experiences.

Original: Swearngin & Li (2019) proposed tappability prediction as a binary classiﬁcation task based on the view hierarchy and the screen appearance of the element.
Revised: Swearngin & Li (2019) proposed tappability prediction as a binary classiﬁcation task based on the view hierarchy and the screen appearance of the element.

Original: In this work, to avoid introducing additional task heads, we formulate tappability prediction as a yes-or-no QA task (Equation 5).
Revised: In this work, to avoid introducing additional task heads, we formulate tappability prediction as a yes-or-no QA task (Equation 5).

Original: [Equation]
Revised: [Equation]

Original: which outputs "yes" when the perception of object is predicted as tappable, and "no" otherwise
Revised: which outputs "yes" when the perception of object is predicted as tappable, and "no" otherwise

Original: Note that all the tasks share the Image-Structure model.
Revised: Note that all the tasks share the Image-Structure model.

Original: Except the UI Object Detection task, they also share the Question-Answer model.
Revised: Except the UI Object Detection task, they also share the Question-Answer model.

Original: We do not introduce a special task token.
Revised: We do not introduce a special task token.

Original: Instead, Q is served as the natural language task indicator for signaling the model to produce different answers for each task.
Revised: Instead, Q is served as the natural language task indicator for signaling the model to produce different answers for each task.

Original: Q also carries the actual task speciﬁcs for the grounding task to ﬁnd the object on the UI.
Revised: Q also carries the actual task speciﬁcs for the grounding task to ﬁnd the object on the UI.

Revised: These ﬁve tasks are very different in nature.

Revised: UI object detection only takes image input and generates

Revised: UI objects.

Original: M ODEL A RCHITECTURE
Revised: Command grounding leverages all the input modalities and outputs an object reference. Tappability, UI summarization and widget captioning share the same input and output modalities but are fundamentally different. Tappability is concerned with the usability of a speciﬁc UI object. In contrast, UI summarization and widget captioning generate functional descriptions about UIs, with the former focusing on the entire screen and the latter concentrating on a speciﬁc object.

Revised: These heterogeneous tasks pose challenges for multi-task learning.

Original: As shown in Figure 1, the architecture of VUT includes two Transformer models, i.e., the ImageStructure model and the Question-Answer model, and three output heads for three types of responses that accomplish 5 tasks.
Revised: As shown in Figure 1, the architecture of VUT includes two Transformer models, i.e., the ImageStructure model and the Question-Answer model, and three output heads for three types of responses that accomplish 5 tasks.

Original: 4. I MAGE -S TRUCTURE T RANSFORMER
Revised: 4. I MAGE -S TRUCTURE T RANSFORMER

Original: The Image-Structure Transformer is a two-modal model, which takes an image and the corresponding view hierarchy, and outputs the hidden representation of the image and each node in the view hierarchy.
Revised: The Image-Structure Transformer is a two-modal model, which takes an image and the corresponding view hierarchy, and outputs the hidden representation of the image and each node in the view hierarchy.

Original: For the image modality, we compute the content embedding, C s , and its positional encoding, P s , according to DETR (Carion et al., 2020):
Revised: For the image modality, we compute the content embedding, C s , and its positional encoding, P s , according to DETR (Carion et al., 2020):

Original: C s =
Revised: C s =

Original: Dense ( ResNet ( S )) and P s = PE ( S mask ) .
Revised: Dense ( ResNet ( S )) and P s = PE ( S mask ) .

Original: S mask is the binary non-padding mask of the image S .
Revised: S mask is the binary non-padding mask of the image S .

Original: We omit details such as tensor reshaping and broadcasting in our discussions here.
Revised: We omit details such as tensor reshaping and broadcasting in our discussions here.

Original: C s ∈ R M × D and P s ∈ R M × D where M is the ﬂattened size of the feature map after ResNet and D is the dimension of the representation.
Revised: C s ∈ R M × D and P s ∈ R M × D where M is the ﬂattened size of the feature map after ResNet and D is the dimension of the representation.

Original: For the view hierarchy modality, when V is absent in the object detection task (Section 3.1), the content embedding for the modality, C v , is all zeros, and the positional encoding, P v , is a learned embedding vector for each query position.
Revised: For the view hierarchy modality, when V is absent in the object detection task (Section 3.1), the content embedding for the modality, C v , is all zeros, and the positional encoding, P v , is a learned embedding vector for each query position.

Original: When the view hierarchy is present in the input, each object in the view hierarchy tree, V , is embedded based on the set of attributes it possesses.
Revised: When the view hierarchy is present in the input, each object in the view hierarchy tree, V , is embedded based on the set of attributes it possesses.

Original: The discrete attributes such as type, clickability, and text content are embedded separately to the same dimension and then combined via addition to form the content embedding of each element, C v .
Revised: The discrete attributes such as type, clickability, and text content are embedded separately to the same dimension and then combined via addition to form the content embedding of each element, C v .

Original: When there are multiple tokens in text content, max pooling is used to acquire a single ﬁxed-size representation of text content.
Revised: When there are multiple tokens in text content, max pooling is used to acquire a single ﬁxed-size representation of text content.

Original: The positional encoding of each object, P v , is computed based on its bounding box [top, left, right, bottom] and DOM positions [pre-order idx, post-order idx, depth].

Original: Each is treated as a continuous vector position that is encoded via learnable Fourier representation (Li et al., 2021b) and then summed together to form the positional encoding of the object.
Revised: The positional encoding of each object, P v , is computed based on its bounding vector position that is encoded via learnable Fourier representation (Li et al., 2021b) and then summed together to form the positional encoding of the object.

Original: When V is not present in the input, the position encoding is simply a learned embedding for each index position in the input.
Revised: When V is not present in the input, the position encoding is simply a learned embedding for each index position in the input.

Original: The ﬁnal embedding of the view hierarchy modality includes two tensors: C v ∈ R N × D and P v ∈ R N × D .
Revised: The ﬁnal embedding of the view hierarchy modality includes two tensors: C v ∈ R N × D and P v ∈ R N × D .

Original: P s and P v are positional encoding within each modality.
Revised: P s and P v are positional encoding within each modality.

Original: Because the embeddings from the two modalities will jointly participate in the self attention of the Transformer encoder, it is important to make their positional encoding global instead of local to each modality.
Revised: Because the embeddings from the two modalities will jointly participate in the self attention of the Transformer encoder, it is important to make their positional encoding global instead of local to each modality.

Original: To this end, we add a learnable modal embedding to each of these modality-speciﬁc positional encoding: P (cid:48) s =
Revised: To this end, we add a learnable modal embedding to each of these modality-speciﬁc positional encoding: P (cid:48) s =

Original: P s + E s and P (cid:48) v =
Revised: P s + E s and P (cid:48) v =

Original: P v + E v , where E s ∈ R 1 × D and E v ∈ R 1 × D are the learnable embeddings for the image and view hierarchy modality respectively.
Revised: P v + E v , where E s ∈ R 1 × D and E v ∈ R 1 × D are the learnable embeddings for the image and view hierarchy modality respectively.

Original: When the task is with respect to a speciﬁc object on the screen (Section 3.2 and 3.5), T is passed to the model so that it can pay more attention to the object.
Revised: When the task is with respect to a speciﬁc object on the screen (Section 3.2 and 3.5), T is passed to the model so that it can pay more attention to the object.

Original: To achieve this unique need of UI tasks, we modulate the content embedding of both modalities, C s and C v using a focus map α s produced from
Revised: To achieve this unique need of UI tasks, we modulate the content embedding of both modalities, C s and C v using a focus map α s produced from

Original: T as follows.
Revised: T as follows.

Original: [Equation]
Revised: [Equation]

Original: RegionMask ( · ) creates a 2D binary mask from the bounding box of the target object T bbx where it is all ones inside the the box and all zeros outside.
Revised: RegionMask ( · ) creates a 2D binary mask from the bounding box of the target object T bbx where it is all ones inside the the box and all zeros outside.

Original: Flatten ( · ) ﬂattens the 2D mask to 1D. β and τ are the hyperparameters that regulate how much the model should focus on the target object versus the rest of the image.
Revised: Flatten ( · ) ﬂattens the 2D mask to 1D. β and τ are the hyperparameters that regulate how much the model should focus on the target object versus the rest of the image.

Original: ⊗ denotes element-wise multiplication.
Revised: ⊗ denotes element-wise multiplication.

Original: We perform a similar modulation for the target object in V .
Revised: We perform a similar modulation for the target object in V .

Original: [Equation]
Revised: [Equation]

Original: OneHot ( · ) produces a one-hot vector from the index of the target object, T idx , in the view hierarchy.
Revised: OneHot ( · ) produces a one-hot vector from the index of the target object, T idx , in the view hierarchy.

Original: We then concatenate the embeddings of the two modalities along the ﬁrst dimension to form the input the Transformer encoder:
Revised: We then concatenate the embeddings of the two modalities along the ﬁrst dimension to form the input the Transformer encoder:

Original: C = Concat [ C (cid:48) s ; C (cid:48) v ] and P = Concat [ P (cid:48) s ; P (cid:48) v ] , where C ∈ R ( M + N ) × D and P ∈ R ( M + N )
Revised: C = Concat [ C (cid:48) s ; C (cid:48) v ] and P = Concat [ P (cid:48) s ; P (cid:48) v ] , where C ∈ R ( M + N ) × D and P ∈ R ( M + N )

Original: × D are the ﬁnal content embedding and positional encoding respectively, which are fed to a multi-layer Transformer encoder:
Revised: × D are the ﬁnal content embedding and positional encoding respectively, which are fed to a multi-layer Transformer encoder:

Original: H =
Revised: H =

Original: Transformer_Encoder ( C, P ) , where the hidden representation H ∈ R ( M + N ) × D .
Revised: Transformer_Encoder ( C, P ) , where the hidden representation H ∈ R ( M + N ) × D .

Original: We then split H for the hidden representation for each modality:
Revised: We then split H for the hidden representation for each modality:

Original: H s = H
Revised: H s = H

Original: [: M ] and H v
Revised: [: M ] and H v

Original: = H
Revised: = H

Original: [ M :] while result in the hidden representations for each modality:
Revised: [ M :] while result in the hidden representations for each modality:

Original: H s ∈ R M × D and H v ∈ R N × D .
Revised: H s ∈ R M × D and H v ∈ R N × D .

Original: 4. Q UESTION -A NSWER T RANSFORMER
Revised: 4. Q UESTION -A NSWER T RANSFORMER

Original: The Question-Answer Transformer is a language model that encodes the question Q and decodes the answer A and produces the hidden representation for the grounding task.
Revised: The Question-Answer Transformer is a language model that encodes the question Q and decodes the answer A and produces the hidden representation for the grounding task.

Original: The input of the model,
Revised: The input of the model,

Original: X = x 1: t , where t is the length of the sequence, is either just the token sequence of Q for the grounding task (Section 3.4), or the concatenation of Q and the decoded answer A (cid:48) when a language answer is to be generated.
Revised: X = x 1: t , where t is the length of the sequence, is either just the token sequence of Q for the grounding task (Section 3.4), or the concatenation of Q and the decoded answer A (cid:48) when a language answer is to be generated.

Original: During training with teaching forcing, A (cid:48)
Revised: During training with teaching forcing, A (cid:48)

Original: =
Revised: =

Original: A .
Revised: A .

Original: During auto-regressive inference, A (cid:48) is the predicted token sequence up to a step.
Revised: During auto-regressive inference, A (cid:48) is the predicted token sequence up to a step.

Original: [Equation]
Revised: [Equation]

Original: where x i is the i th token in the sequence ( 1 ≤ i ≤ t ), and E ( · ) and PE ( · ) compute the content embedding and the positional encoding of each token in the sequence.
Revised: where x i is the i th token in the sequence ( 1 ≤ i ≤ t ), and E ( · ) and PE ( · ) compute the content embedding and the positional encoding of each token in the sequence.

Original: H s and H v are accessed via
Revised: H s and H v are accessed via

Original: Transformer encoder-decoder attention.
Revised: Transformer encoder-decoder attention.

Original: The sequence of hidden states, g 1: t , g i ∈ R D , are used for predicting the next token for generating an answer or for retrieving a target UI object in the view hierarchy for the grounding task.
Revised: The sequence of hidden states, g 1: t , g i ∈ R D , are used for predicting the next token for generating an answer or for retrieving a target UI object in the view hierarchy for the grounding task.

Original: 4. O UTPUT H EADS
Revised: 4. O UTPUT H EADS

Original: Object Detection Head : For the UI Object Detection task (Section 3.1), we borrow the object output layer from DETR (Carion et al., 2020), using H v as the input the layer: Y type =
Revised: Object Detection Head : For the UI Object Detection task (Section 3.1), we borrow the object output layer from DETR (Carion et al., 2020), using H v as the input the layer: Y type =

Original: H v W type and
Revised: H v W type and

Original: Y bbx =
Revised: Y bbx =

Original: φ ( H v , θ bbx ) W bbx where W type ∈ R N × 22 is the linear projection to output the object type logits.
Revised: φ ( H v , θ bbx ) W bbx where W type ∈ R N × 22 is the linear projection to output the object type logits.

Original: An additional PADDING type is included on top of the original 21 UI object classes.
Revised: An additional PADDING type is included on top of the original 21 UI object classes.

Original: φ ( · ) is the multi-layer perceptron parametized by θ bbx and W bbx ∈
Revised: φ ( · ) is the multi-layer perceptron parametized by θ bbx and W bbx ∈

Original: R N × 4 is the linear projection for generating the coordinates.
Revised: R N × 4 is the linear projection for generating the coordinates.

Original: The logits
Revised: The logits

Original: Y type ∈
Revised: Y type ∈

Original: R N × 22 and the coordinates Y bbx ∈ R N × 4 are for both generating object predictions and computing optimal compound loss using Hungarian Matching during training Carion et al.
Revised: R N × 22 and the coordinates Y bbx ∈ R N × 4 are for both generating object predictions and computing optimal compound loss using Hungarian Matching during training Carion et al.

Original: Text Head : For the three tasks that require a text response, A , (Section 3.2,3.3 and 3.5), we apply a softmax layer on top of the decoder hidden state, g 1: t (Equation 8), to generate each answer token. 
Revised: Text Head : For the three tasks that require a text response, A , (Section 3.2,3.3 and 3.5), we apply a softmax layer on top of the decoder hidden state, g 1: t (Equation 8), to generate each answer token. [Equation]

Original: [Equation]
Revised: [Equation]

Original: where a i is the i th token in the answer sequence A , and | Q | is the length of the question.
Revised: where a i is the i th token in the answer sequence A , and | Q | is the length of the question.

Original: W txt ∈
Revised: W txt ∈

Original: R D ×| vocab | is the learnable weights and | vocab | is the vocabulary size.
Revised: R D ×| vocab | is the learnable weights and | vocab | is the vocabulary size.

Original: For the three tasks, we optimize the model for the cross-entropy loss over the predicted and ground-truth answer token sequences.
Revised: For the three tasks, we optimize the model for the cross-entropy loss over the predicted and ground-truth answer token sequences.

Original: Pointer Head :
Revised: Pointer Head :

Original: For the grounding task (Section 3.4), we use the last hidden state from the Transformer decoder as a "pointer" Vinyals et al.
Revised: For the grounding task (Section 3.4), we use the last hidden state from the Transformer decoder as a "pointer" Vinyals et al.

Original: (2015) to match against all the objects in the UI based on their hidden representations, using dot product similarity (Equation 10). 
Revised: (2015) to match against all the objects in the UI based on their hidden representations, using dot product similarity (Equation 10). [Equation]

Original: [Equation]
Revised: [Equation]

Original: where h j is the j th row in H v that is the hidden representation of the j th object in the view hierarchy.
Revised: where h j is the j th row in H v that is the hidden representation of the j th object in the view hierarchy.

Original: W ptr ∈
Revised: W ptr ∈

Original: R D
Revised: R D

Original: × D is the learnable projection and g | Q | is the last hidden state from the decoder (Equation 8), which is able to access the entire question (command) sequence, Q , via the decoder self attention.
Revised: × D is the learnable projection and g | Q | is the last hidden state from the decoder (Equation 8), which is able to access the entire question (command) sequence, Q , via the decoder self attention.

Original: Compared to previous UI ground work (Li et al., 2020a), we used the last hidden state as the "pointer" instead of embedding pooling of a bag of word in a span.
Revised: Compared to previous UI ground work (Li et al., 2020a), we used the last hidden state as the "pointer" instead of embedding pooling of a bag of word in a span.

Original: We optimize the model by minimizing the cross-entropy loss between the predicted and the ground-truth object index.
Revised: We optimize the model by minimizing the cross-entropy loss between the predicted and the ground-truth object index.

Original:  E XPERIMENTS 5. D ATASETS
Revised:  E XPERIMENTS 5. D ATASETS

Original: For the UI Object Detection task, we use RICO, a public corpus of mobile user interfaces (Deka et al., 2017) that contains 64,462 unique Android screens from 9,362 different apps.
Revised: For the UI Object Detection task, we use RICO, a public corpus of mobile user interfaces (Deka et al., 2017) that contains 64,462 unique Android screens from 9,362 different apps.

Original: Each screen comes with an RGB screenshot and a corresponding view hierarchy.
Revised: Each screen comes with an RGB screenshot and a corresponding view hierarchy.

Original: A view hierarchy is a tree structure of nodes with 21 unique types, which we consolidated from the Android View class attributes in the original dataset (see Appendix D).
Revised: A view hierarchy is a tree structure of nodes with 21 unique types, which we consolidated from the Android View class attributes in the original dataset (see Appendix D).

Original: Plus the special PADDING type, there are 22 types to be predicted.
Revised: Plus the special PADDING type, there are 22 types to be predicted.

Original: A view hierarchy has a maximum of 128 nodes in our dataset.
Revised: A view hierarchy has a maximum of 128 nodes in our dataset.

Original: For the Widget Captioning task, we used a public dataset (Li et al., 2020b), which includes more than 200k human annotations for over 46k unique UI objects from 17k RICO screens.
Revised: For the Widget Captioning task, we used a public dataset (Li et al., 2020b), which includes more than 200k human annotations for over 46k unique UI objects from 17k RICO screens.

Original: For Screen
Revised: For Screen

Original: Summarization, we used a public dataset (Wang et al., 2021) that consists of 112k human created summaries for 22,301 unique Android screens from RICO.
Revised: Summarization, we used a public dataset (Wang et al., 2021) that consists of 112k human created summaries for 22,301 unique Android screens from RICO.

Original: We created the Language Grounding dataset, which has 10k human annotations for operating UI objects of 1432 unique screens from 26 Android build-in apps like Settings.
Revised: We created the Language Grounding dataset, which has 10k human annotations for operating UI objects of 1432 unique screens from 26 Android build-in apps like Settings.

Original: A human rater generates commands such as "Click the button below battery info", and the maximum length of a command phrase is 20 words.
Revised: A human rater generates commands such as "Click the button below battery info", and the maximum length of a command phrase is 20 words.

Original: The Tappability Prediction dataset includes tappability annotations for more than 18,669 UI elements from 3,218 Android screens.
Revised: The Tappability Prediction dataset includes tappability annotations for more than 18,669 UI elements from 3,218 Android screens.

Original: In the data collection, given a target UI element highlighted on a screen, 5 different human raters are asked to answer yes or no for whether the target object looks tappable to them.
Revised: In the data collection, given a target UI element highlighted on a screen, 5 different human raters are asked to answer yes or no for whether the target object looks tappable to them.

Original: We use the majority voting to determine the label of each element.
Revised: We use the majority voting to determine the label of each element.

Original: [Table] We split each dataset into training, validation and test sets (Table 2), and ensure there is no overlap of apps (or screens) between a training set and any of the test sets of different tasks.
Revised: We split each dataset into training, validation and test sets (Table 2), and ensure there is no overlap of apps (or screens) between a training set and any of the test sets of different tasks.

Original: This is important  because in the multi-task learning condition, VUT learns from all the training sets.
Revised: This is important [Table] because in the multi-task learning condition, VUT learns from all the training sets.

Original: Thus the union of apps and screens across all the training sets should not overlap any of the test set.
Revised: Thus the union of apps and screens across all the training sets should not overlap any of the test set.

Original: We also ensure our test datasets are the same with the released benchmarks for widget captioning and screen summariztion so that the results are comparable with reported SOTAs.
Revised: We also ensure our test datasets are the same with the released benchmarks for widget captioning and screen summariztion so that the results are comparable with reported SOTAs.

Original: We unify these heterogeneous datasets to follow the same feature taxonomies based on our problem formulation so that they can be consumed by the same model.
Revised: We unify these heterogeneous datasets to follow the same feature taxonomies based on our problem formulation so that they can be consumed by the same model.

Original: 5. R ESULTS 5.2. C OMPARING VUT WITH DETR & C ENTER N ET FOR O BJECT D ETECTION
Revised: 5. R ESULTS 5.2. C OMPARING VUT WITH DETR & C ENTER N ET FOR O BJECT D ETECTION

Original: In this experiment, we compare VUT with two benchmark models for UI Object Detection.
Revised: In this experiment, we compare VUT with two benchmark models for UI Object Detection.

Original: The standard DETR architecture (Carion et al., 2020) uses a 6-layer Transformer encoder and a 6-layer
Revised: The standard DETR architecture (Carion et al., 2020) uses a 6-layer Transformer encoder and a 6-layer

Original: Transformer decoder.
Revised: Transformer decoder.

Original: To have a similar number of parameters in the model, we let VUT ImageStructure transformer to use a 12-layer Transformer encoder in this experiment.
Revised: To have a similar number of parameters in the model, we let VUT ImageStructure transformer to use a 12-layer Transformer encoder in this experiment.

Original: CenterNet (Duan et al., 2019) has been a popular choice for object detection, which achieved SOTA results on natural images.
Revised: CenterNet (Duan et al., 2019) has been a popular choice for object detection, which achieved SOTA results on natural images.

Original: In this experiment, we let CenterNet use ResNet-101 as its backbone to reach a similar parameter size.
Revised: In this experiment, we let CenterNet use ResNet-101 as its backbone to reach a similar parameter size.

Original: Table 3 shows that VUT’s Image-Structure model clearly outperforms DETR and
Revised: Table 3 shows that VUT’s Image-Structure model clearly outperforms DETR and

Original: CenterNet on the UI Object Detection dataset.
Revised: CenterNet on the UI Object Detection dataset.

Original: Previously, Carion et al.
Revised: Previously, Carion et al.

Original: (2020) found more encoding layers can lead to better accuracy.
Revised: (2020) found more encoding layers can lead to better accuracy.

Original: It is worth noting that VUT’s Image-Structure model uses an encoder-only architecture that achieves better accuracy.
Revised: It is worth noting that VUT’s Image-Structure model uses an encoder-only architecture that achieves better accuracy.

Original: [Table]
Revised: [Table]

Original: Note that these results can not be compared directly with those previously reported Zhang et al.
Revised: Note that these results can not be compared directly with those previously reported Zhang et al.

Original: Our task is more challenging for detecting 21 different UI object types including several container elements, instead of 12 objects in previous work.
Revised: Our task is more challenging for detecting 21 different UI object types including several container elements, instead of 12 objects in previous work.

Original: In addition, previous work used a dataset that is manually labeled by human and also employed heavy post-process to improve the accuracy.
Revised: In addition, previous work used a dataset that is manually labeled by human and also employed heavy post-process to improve the accuracy.

Original:  C OMPARING M ULTI -T ASK WITH S INGLE T ASK L EARNING
Revised: 5.2. C OMPARING M ULTI -T ASK WITH S INGLE T ASK L EARNING

Original: For each task, we report its accuracy when it is learned alone and with other tasks jointly.
Revised: For each task, we report its accuracy when it is learned alone and with other tasks jointly.

Original: We also show the benchmark results on these datasets reported by previous works for comparison, when they are available.
Revised: We also show the benchmark results on these datasets reported by previous works for comparison, when they are available.

Original: In this experiment, we use a 6-layer Image-Structure model and a 6-layer QuestionAnswer model.
Revised: In this experiment, we use a 6-layer Image-Structure model and a 6-layer QuestionAnswer model.

Original: Each task head and model parts are used only for batches that are speciﬁc for each task during training.
Revised: Each task head and model parts are used only for batches that are speciﬁc for each task during training.

Original: For Widget Captioning, Screen Summarization and Tappability, text head (Equation 9) is used for generating answers.
Revised: For Widget Captioning, Screen Summarization and Tappability, text head (Equation 9) is used for generating answers.

Original: For Language Command Grounding, the grounding head (Equation 10) is used instead.
Revised: For Language Command Grounding, the grounding head (Equation 10) is used instead.

Original: Because the UI Object Detection task requires many more iterations to converge than other tasks, we start our multi-task learning by training VUT forthe UI Object Detection task, and then training VUT jointly for multiple tasks by alternating batches from each dataset. This learning strategy is sensible because by learning from the UI Object task, the model can learn useful information about how to encode the screen pixels. As it is consistently shown in our experiments, joint learning that involves Object Detection can often boost the learning of the other four tasks.
Revised: See Appendix B.1 for training schedule details.

Original: [Table]

Original: As shown in Table 4, 5, 6 and 7, multi-task learning, though more challenging than single-task learning, can often perform on par with single-task learning.
Revised: As shown in Table 4, 5, 6 and 7, multi-task learning, though more challenging than single-task learning, can often perform on par with single-task learning.

Original: It consistently outperforms single-task learning for most conﬁgurations and metrics.
Revised: It consistently outperforms single-task learning for most conﬁgurations and metrics.

Original: There is a decrease of accuracy for the Grounding task when text-generation related tasks are involved.
Revised: There is a decrease of accuracy for the Grounding task when text-generation related tasks are involved.

Original: We suspect that the grounding task, which 
Revised: We suspect that the grounding task, which relies on the last hidden state of the Question-Answer model, probably competes with the three text-generation tasks by “pulling” the hidden representations of the Question-Answer model towards different

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: UI Object Detection task when different tasks are jointly learned. relies on the last hidden state of the Question-Answer model, probably competes with the three text-generation tasks by “pulling” the hidden representations of the Question-Answer model towards different directions.
Revised: directions.

Original: One phenomenon that we consistently observed is that having the Object Detection task in multi-task learning often outperforms the conﬁguration without involving Object Detection.
Revised: One phenomenon that we consistently observed is that having the Object Detection task in multi-task learning often outperforms the conﬁguration without involving Object Detection.

Original: For theObject
Revised: For the

Original:  Detection task itself, we found there is a drop of accuracy when batch-alteration for multi-task learning starts.
Revised: Object Detection task itself, we found there is a drop of accuracy when batch-alteration for multi-task learning starts.

Original: Yet, it mostly recovers its accuracy (see Table 8).
Revised: Yet, it mostly recovers its accuracy (see Table 8).

Original: These experiments show that instead of treating Object Detection as a standalone pretraining task, it is feasible for it to be part of the multi-task learning where VUT achieves all the tasks through a single model.
Revised: These experiments show that instead of treating Object Detection as a standalone pretraining task, it is feasible for it to be part of the multi-task learning where VUT achieves all the tasks through a single model.

Original: We start the training by learning the Object Detection task that takes 300k steps, using DETR’s default training schedule (Carion et al., 2020).

Original: For joint-learning of all the 5 tasks, we sample batches from the Object Detection, Widget Captioning, Scren Summarization, Command Grounding and

Original: Tappability datasets with the weights [15 , 10 , 10 , 20 , 1] and the model is trained with 100k steps.

Original: The learning rate is 1e-4 and decayed to 1e-5 after 50k steps.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We present VUT, a multi-modal Transformer for multi-task modeling of user interfaces.
Revised: We present VUT, a multi-modal Transformer for multi-task modeling of user interfaces.

Original: Our model takes in three types of data, i.e., UI screenshot images, view hierarchy structures, and natural language questions.
Revised: Our model takes in three types of data, i [Table] e., UI screenshot images, view hierarchy structures

Revised: [Table]

Original:  Our experiments based on 5 datasets show that VUT achieves ﬁve types of UI tasks simultaneously, and show the promise of providing uniﬁed modeling for the user interface domain.
Revised: and natural language questions. Our experiments based on 5 datasets show that VUT achieves ﬁve types of UI tasks simultaneously [Table] and show the promise of providing uniﬁed modeling for the user interface domain.

Original: A Q UESTIONS
Revised: A Q UESTIONS

Original: As discussed in the paper there are three types of question input, Q , fed into the Question-Answer model: commands, questions for yes-or-no responses and questions for open-ended answers.
Revised: As discussed in the paper, there are three types of question input, Q , fed into the Question-Answer model: commands, questions for yes-or-no responses, and questions for open-ended answers.

Original: When
Revised: When

Original: VUT is joint-learning all the tasks, it is required to recognize all the questions for different types of tasks.
Revised: VUT is joint-learning all the tasks, it is required to recognize all the questions for different types of tasks.

Original: For questions that expect a textual response, e.g., Widget Captioning, Screen Summarization, and Tappability Prediction, VUT also needs to generate an answer corresponding to each type of question.
Revised: For questions that expect a textual response, e.g., Widget Captioning, Screen Summarization, and Tappability Prediction, VUT also needs to generate an answer corresponding to each type of question.

Original: A. C OMMANDS
Revised: A. C OMMANDS

Original: For the Language Command Grounding task, commands that refer to a speciﬁc object in the screen are fed to the model by which the model is trained to locate the referred object.
Revised: For the Language Command Grounding task, commands that refer to a speciﬁc object in the screen are fed to the model by which the model is trained to locate the referred object.

Original: Example commands are shown below.
Revised: Example commands are shown below.

Original: These commands are created by human annotators for a target UI object shown on a screen.
Revised: These commands are created by human annotators for a target UI object shown on a screen.

Original: A human annotator is asked to come up with different commands referring to each highlighted target object.
Revised: A human annotator is asked to come up with different commands referring to each highlighted target object.

Original: • Click on the notiﬁcation bar above the status option.
Revised: • Click on the notiﬁcation bar above the status option.

Original: • Press on the back arrow button.
Revised: • Press on the back arrow button.

Original: • Select the icon above the clock option.
Revised: • Select the icon above the clock option.

Original: • Swipe down the notiﬁcation bar.
Revised: • Swipe down the notiﬁcation bar.

Original: A. Q UESTIONS FOR Y ES - OR -N O A NSWERS
Revised: A. Q UESTIONS FOR Y ES - OR -N O A NSWERS

Original: For the Tappability Prediction task, we generate synthetic Yes-or-No questions based on the following regular expression pattern.
Revised: For the Tappability Prediction task, we generate synthetic Yes-or-No questions based on the following regular expression pattern.

Original: The model is trained to decode yes or no as the answer to the question.
Revised: The model is trained to decode yes or no as the answer to the question.

Original: Is the [object|element|widget|
Revised: Is the [object|

Original: control] [clickable | tappable]
Revised: element|widget|control] [clickable|tappable]

Original: The question examples that are generated based on the regular expression are the following.
Revised: The question examples that are generated based on the regular expression are the following.

Original: • Is the object tappable?
Revised: • Is the object tappable?

Original: • Is the widget clickable?
Revised: • Is the widget clickable?

Original: • Is the element tappable?
Revised: • Is the element tappable?

Original: A. Q UESTIONS FOR O PEN -E NDED A NSWERS
Revised: A. Q UESTIONS FOR O PEN -E NDED A NSWERS

Original: For Screen Summarization summary and captioning tasks, the model will need to generate an openended answer.
Revised: For Screen Summarization summary and captioning tasks, the model will need to generate an openended answer.

Original: We use the following regular expressions to generate questions for these tasks.
Revised: We use the following regular expressions to generate questions for these tasks.

Original: VUT is trained to decode a screen summary or a widget caption following the question.
Revised: VUT is trained to decode a screen summary or a widget caption following the question.

Original: What is the [summary|description] of the [screen|UI]?
Revised: What is the [summary|description] of the [screen|UI]?

Original: What is the [caption|description] of the [object|element|widget|control]?
Revised: What is the [caption|description] of the [object|element|widget|control]?

Original: Below are the question examples generated from the above regular expressions.
Revised: Below are the question examples generated from the above regular expressions.

Original: • What is the summary of the screen?
Revised: • What is the summary of the screen?

Original: • What is the description of the UI?
Revised: • What is the description of the UI?

Original: What is the caption of the widget?
Revised: What is the caption of the widget?

Original: • What is the description of the object?
Revised: • What is the description of the object?

Original: B M ODEL & T RAINING D ETAILS
Revised: B M ODEL & T RAINING D ETAILS

Original: For the comparison with DETR and CenterNet on the UI Object Detection task, VUT uses a 12-layer
Revised: For the comparison with DETR and CenterNet on the UI Object Detection task, VUT uses a 12-layer

Original: Transformer encoder as the Image-Structure model that amounts to 48M trainable parameters, which is fewer than 50M trainable parameters of DETR with a 6-layer encoder and a 6-layer decoder, and 49M parameters of CenterNet.
Revised: Transformer encoder as the Image-Structure model that amounts to 48M trainable parameters, which is fewer than 50M trainable parameters of DETR with a 6-layer encoder and a 6-layer decoder, and 49M parameters of CenterNet.

Original: Image Embedder (Figure 1) is the ResNet backbone.
Revised: Image Embedder (Figure 1) is the ResNet backbone.

Original: For the remaining experiments, VUT uses a 6-layer Transformer encoder for the Image-Structure model, and a 6-layer Transformer decoder for the Question-Answer model.
Revised: For the remaining experiments, VUT uses a 6-layer Transformer encoder for the Image-Structure model, and a 6-layer Transformer decoder for the Question-Answer model.

Original: When all the tasks are jointly trained, there are 64M parameters.
Revised: When all the tasks are jointly trained, there are 64M parameters.

Original: Task-speciﬁc heads and word piece embeddings and projections are the main contributors to the growth of the parameter size.
Revised: Task-speciﬁc heads and word piece embeddings and projections are the main contributors to the growth of the parameter size.

Original: When only a subset of these tasks is involved in the training, e.g., Widget Captioning + Object Detection, there will be fewer trainable parameters involved because only part of the full model is in use.
Revised: When only a subset of these tasks is involved in the training, e.g., Widget Captioning + Object Detection, there will be fewer trainable parameters involved because only part of the full model is in use.

Original: All the VUT variants use the following conﬁgurations: #Attention_Heads=8, Hidden_Dimension=256, Transformer_MLP_Dimension=2048,
Revised: All the VUT variants use the following conﬁgurations: #Attention_Heads=8, Hidden_Dimension=256, Transformer_MLP_Dimension=2048,

Original: Transformer_QKV_Dimension=256.
Revised: Transformer_QKV_Dimension=256.

Original: Note that in our Image-Structure model, positional encoding is added to the input of each layer of the Transformer.
Revised: Note that in our Image-Structure model, positional encoding is added to the input of each layer of the Transformer.

Original: This is in contrast to our Question-Answer model where the positional encoding of each token is only added to the input of the ﬁrst layer.
Revised: This is in contrast to our Question-Answer model where the positional encoding of each token is only added to the input of the ﬁrst layer.

Original: We use the learned embedding for positional encoding in the Question-Answer model.
Revised: We use the learned embedding for positional encoding in the Question-Answer model.

Original: During training, we use 10% for both the attention and the
Revised: During training, we use 10% for both the attention and the

Original: MLP dropout in the Questions-Answer Transformer, and we also apply 10% dropout on the encodings from the Image-Structure model before cross attention.
Revised: MLP dropout in the Questions-Answer Transformer, and we also apply 10% dropout on the encodings from the Image-Structure model before cross attention.

Original: During the 5-task joint learning, the attention and the MLP dropout rates are 10% for the Image-Structure Transformer and the encoder-decoder cross-attention dropout rate is 20%.
Revised: During the 5-task joint learning, the attention and the MLP dropout rates are 10% for the Image-Structure Transformer and the encoder-decoder cross-attention dropout rate is 20%.

Original: During auto-regressive decoding for interference, the maximum decoding length is 30 that covers the total length of a question and an answer.
Revised: During auto-regressive decoding for interference, the maximum decoding length is 30 that covers the total length of a question and an answer.

Original: We use the same method as BERT for tokenizing phrases into a sequence of word pieces 3 , which results in a vocabulary size of 28,536.
Revised: We use the same method as BERT for tokenizing phrases into a sequence of word pieces 3 , which results in a vocabulary size of 28,536.

Original: The maximum size for a screenshot image is 1080 × 1080 .
Revised: The maximum size for a screenshot image is 1080 × 1080 .

Original: We randomly resize each image for image augmentation.
Revised: We randomly resize each image for image augmentation.

Original: The maximum number of UI objects and containers on each screen is capped to 128.
Revised: The maximum number of UI objects and containers on each screen is capped to 128.

Original: We implement VUT based in JAX 4 , a library for machine learning.
Revised: We implement VUT based in JAX 4 , a library for machine learning.

Original: We train each VUT model with a batch size of 64 screens/examples, which the training is parallelized across 64 TPU v3 cores.
Revised: We train each VUT model with a batch size of 64 screens/examples, which the training is parallelized across 64 TPU v3 cores.

Original: C A BLATION S TUDY ON F OCUS O BJECT
Revised: B. T RAINING S CHEDULE D ETAILS

Revised: Because the UI Object Detection task requires many more iterations to converge than other tasks, we start our multi-task learning by training VUT for the UI Object Detection task, and then training

Revised: VUT jointly for multiple tasks by alternating batches from each dataset.

Revised: This learning strategy is sensible because by learning from the UI Object task, the model can learn useful information about tokenizer#

Revised: 4 https:/ how to encode the screen pixels.

Revised: As it is consistently shown in our experiments, joint learning that involves Object Detection can often boost the learning of the other four tasks.

Revised: We start the training by learning the Object Detection task that takes 300k steps, using DETR’s default training schedule (Carion et al., 2020).

Revised: For joint-learning of all the 5 tasks, we sample batches from the Object Detection, Widget Captioning, Scren Summarization, Command Grounding and

Revised: Tappability datasets with the weights [15 , 10 , 10 , 20 , 1] and the model is trained with 100k steps.

Revised: The learning rate is 1e-4 and decayed to 1e-5 after 50k steps.

Revised: For individual task training, we train each model until it converges with a batch size of 64.

Revised: For the UI

Revised: Object Detection task, we train the model with the most default setup of DETR with 6-layer encoder and 6-layer decoder and 8-head attention and 256 hidden size, using 300k steps.

Revised: The learning rate schedule includes one learning rate decay from 1e-4 to 1e-5 at the 200k steps.

Revised: The Widget Captioning task uses a 6-layer Image-Structure model and a 6-layer Question-Answer model plus the Text head.

Revised: Similarly, the model was trained 45k steps to converge.

Revised: For the Screen Summarization task, we used the same model conﬁguration as Widget Captioning, and the model converges at 50k steps.

Revised: The

Revised: Language Command Grounding task uses a similar model setup as the Widget Captioning and the

Revised: Screen Summarization tasks, except that it uses the Grounding Head instead of the Text Head.

Revised: It took the model 27k steps to converge.

Revised: For training the model for each of these tasks, we decay the learning rate once from 1e-4 to 1e-5 at 15k steps.

Revised: For learning the Tappability Prediction task alone, we used the same model setup as the two text related tasks (Summarization and Captioning).

Revised: We found the model is very prone to overﬁtting in spite of using a large dropout rate.

Revised: So we train the model with early stopping.

Revised: For Object Detection + another individual task, we sampled batches from the two datasets with weights similar to the ones used for 5 task joint training.

Revised: For Widget Captioning, Screen Summary and Grounding, the models are trained to converge with 35k, 50k and 50k steps, with 1e-4 initial learning rate which decays to 1e-5 at 25k, 15k and 25k steps, respectively.

Revised: For the Tappability task, we use a initial learning rate of 1e-5 and train the model with early stopping.

Revised: For 4 task joint training (without Object Detection), we sampled batches from the 4 tasks (Widget

Revised: Captioning, Screen Summary, Grounding, Tappability) with the weights [5, 5, 10, 1].

Revised: We trained the model with 150k steps and batch size 64.

Revised: The learning rate schedule includes one learning rate decay from 1e-4 to 1e-5 at the 50k steps.

Original: For Widget Captioning and Tappability task, VUT predicts the caption or the tappability answer of a given object on the screen—the focus object .
Revised: For Widget Captioning and Tappability task, VUT predicts the caption or the tappability answer of a given object on the screen—the focus object .

Original: To inform the Image-Structure Transformer the location of the given object, we apply a focus map to the Image Embeder outputs and the embeddings of the view hierarchy objects so that the focus object has more weight on its pixel area and object embedding than the others (see Equation 6 and 7).
Revised: To inform the Image-Structure Transformer the location of the given object, we apply a focus map to the Image Embeder outputs and the embeddings of the view hierarchy objects so that the focus object has more weight on its pixel area and object embedding than the others (see Equation 6 and 7).

Original: Speciﬁcally, we used β = 2 . 0 and τ = − 1 . 0 in our experiments.
Revised: Speciﬁcally, we used β = 2 . 0 and τ = − 1 . 0 in our experiments.

Original: To show the effectiveness of the focus modulation, we conduct an ablation study using the Widget Captioning model with three settings: 1) applying the focus map to both the image and structure modalities, 2) applying the focus map to the structure modality only, and 3) concatenating an embedding of { 0 , 1 } to the object embedding for the structure modality with 1 indicating the focus object.
Revised: To show the effectiveness of the focus modulation, we conduct an ablation study using the Widget Captioning model with three settings: 1) applying the focus map to both the image and structure modalities, 2) applying the focus map to the structure modality only, and 3) concatenating an embedding of { 0 , 1 } to the object embedding for the structure modality with 1 indicating the focus object.

Original: As shown in Table 9, the model performs the best when focus map is used on both the image and the structure modalities, indicating the necessity and effectiveness to inform the model the focus object location in the pixels and the object list.
Revised: As shown in Table 9, the model performs the best when focus map is used on both the image and the structure modalities, indicating the necessity and effectiveness to inform the model the focus object location in the pixels and the object list.

Original: Using the focus map only on the object embedding is not sufﬁcient because the Image-Structure model needs to learn from the data to ﬁnd the corresponding pixel areas of the focus object.
Revised: Using the focus map only on the object embedding is not sufﬁcient because the Image-Structure model needs to learn from the data to ﬁnd the corresponding pixel areas of the focus object.

Original: Yet, using { 0 , 1 } embedding is the least effective.
Revised: Yet, using { 0 , 1 } embedding is the least effective.

Original: [Table]
Revised: [Table]

Original: D UI O BJECT T YPES
Revised: D UI O BJECT T YPES

Original: We process the RICO dataset for the UI Object Detection task.
Revised: We process the RICO dataset for the UI Object Detection task.

Original: As discussed in the paper, among the 64,462 screens of the original dataset, we particularly use those veriﬁed by human raters for validation and test datasets.
Revised: As discussed in the paper, among the 64,462 screens of the original dataset, we particularly use those veriﬁed by human raters for validation and test datasets.

Original: For each element on the screen, we extract its attributes such as its UI object type, its bounding box position on the screen, whether it is clickable or enabled.
Revised: For each element on the screen, we extract its attributes such as its UI object type, its bounding box position on the screen, whether it is clickable or enabled.

Original: We exclude all the elements that are marked as invisible as they have no correspondence with pixels on the screen.
Revised: We exclude all the elements that are marked as invisible as they have no correspondence with pixels on the screen.

Original: There are many custom object types in the dataset and many of them inherit from common Android widget types 5 .
Revised: There are many custom object types in the dataset and many of them inherit from common Android widget types 5 .

Original: We consolidate rare object types (such as TABWIDGET or VIDEOVIEW) to their closest ancestor type that are in the common widget library.
Revised: We consolidate rare object types (such as TABWIDGET or VIDEOVIEW) to their closest ancestor type that are in the common widget library.

Original: With the consolidation, there are
Revised: With the consolidation, there are

Original: UI object types in the dataset (Figure 2), which has a long tail distribution.
Revised: UI object types in the dataset (Figure 2), which has a long tail distribution.

Original: As we can see from the distribution, some elements are dedicated as leaves in a view hierarchy, e.g., ImageButton or
Revised: As we can see from the distribution, some elements are dedicated as leaves in a view hierarchy, e.g., ImageButton or

Original: RadioButtion, and some elements are primarily for determining the layout or as non-terminal nodes, e.g., LinearLayout.
Revised: RadioButtion, and some elements are primarily for determining the layout or as non-terminal nodes, e.g., LinearLayout.

Original: There are cases that a type is used for both leaves and non-leaves, e.g., View.
Revised: There are cases that a type is used for both leaves and non-leaves, e.g., View.

Original: The
Revised: The

Original: View type often catches UI elements that cannot be easily classiﬁed into a speciﬁc type.
Revised: View type often catches UI elements that cannot be easily classiﬁed into a speciﬁc type.

Original: Together, these make UI Object Detection a challenging task.
Revised: Together, these make UI Object Detection a challenging task.

Original: E E XAMPLES OF P REDICTION R ESULTS
Revised: E M ODEL S IZE AND I NFERENCE T IME C OMPARISON

Revised: In this section, we compare model size and inference time of the 5 tasks when they are learned jointly and individually.

Revised: We can see the joint model has the largest parameter size, as it includes all the task heads.

Revised: However, the size of the 5-task joint model is substantially smaller than the total size of each individual single-task model (Table 10).

Revised: The inference time of the joint model for each task is similar that of the task-speciﬁc model (Table 11), as only the part that is required by the task is activated in the joint model during inference.

Revised: Altogether, VUT can achieve the ﬁve tasks with little inference time overhead and a substantially small model footprint.

Revised: F E XAMPLES OF P REDICTION R ESULTS

Original: We here show examples of predictions versus ground-truth for each task, on the test data, as achieved by a single model of VUT, when it learns all the tasks jointly.
Revised: We here show examples of predictions versus ground-truth for each task, on the test data, as achieved by a single model of VUT, when it learns all the tasks jointly.

Original: [Figure]
Revised: [Figure]

Original: Command : "tap on the voice icon"
Revised: Command : "tap on the voice icon"

Original: Command : "press on the icon below calculator icon"
Revised: Command : "press on the icon below calculator icon"

Original: Command : "select the weather text which is below the notiﬁcation bar"
Revised: Command : "select the weather text which is below the notiﬁcation bar"

Original: Prediction : "search bar to search for the location"
Revised: Prediction : "search bar to search for the location"

Original: Reference : "page displaying a search box in the app"
Revised: Reference : "page displaying a search box in the app"

Original: Prediction : "page displaying music track in music app"
Revised: Prediction : "page displaying music track in music app"

Original: Reference : "screen shows music playing on an app"
Revised: Reference : "screen shows music playing on an app"

Original: Prediction : "pop-up showing to create an account"
Revised: Prediction : "pop-up showing to create an account"

Original: Reference : "pop-up displaying to setup the account details"
Revised: Reference : "pop-up displaying to setup the account details"

Original: Prediction : "copy text"
Revised: Prediction : "copy text"

Original: Reference : "copy to clipboard option"
Revised: Reference : "copy to clipboard option"

Original: Prediction : "select the emoji"
Revised: Prediction : "select the emoji"

Original: Reference : "select guitar lizard"
Revised: Reference : "select guitar lizard"

Original: Prediction : "enter password"
Revised: Prediction : "enter password"

Original: Reference : "input conﬁrm password"
Revised: Reference : "input conﬁrm password"

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Revised: The 5-task joint model performs the ﬁve tasks simultaneously, which otherwise require ﬁve models that amount to a much larger model

Revised: [Table]