M UG : I NTERACTIVE M ULTIMODAL G ROUNDING ON U SER I NTERFACES
Anonymous authors
Paper under double-blind review
A BSTRACT
We present M UG , a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen.
Prior works modeled multimodal UI grounding in one round: the user gives a command and the agent responds to the command.
Yet, in a realistic scenario, a user command can be ambiguous when the target action is inherently difﬁcult to articulate in natural language.
M UG allows multiple rounds of interactions such that upon seeing the agent responses, the user can give further commands for the agent to reﬁne or even correct its actions.
Such interaction is critical for improving grounding performances in real-world use cases.
To investigate the problem, we create a new dataset that consists of 77 , 820 sequences of human user-agent interaction on mobile interfaces in which 20% involves multiple rounds of interactions.
To establish our benchmark, we experiment with a range of modeling variants and evaluation strategies, including both ofﬂine and online evaluation—the online strategy consists of both human evaluation and automatic with simulators.
Our experiments show that allowing iterative interaction signiﬁcantly improves the absolute task completion by 18% over the entire test set and 31% over the challenging subset.
Our results lay the foundation for further investigation of the problem.
I NTRODUCTION
Natural language understanding on graphical user interfaces (GUIs) is crucial for realizing humancomputer interaction and assisting scenarios that have accessibility difﬁculties (Sarsenbayeva, 2018).
Speciﬁcally, interpreting user commands into executable actions has drawn increasing interests as it manifests rich research problems including multimodal modeling and natural language grounding (e.g., Li et al., 2017;
Gur et al., 2019;
He et al., 2020; Li et al., 2020a; 2021).
Prior works often consider UI grounding in a single-pass fashion where the model predicts actions with a given instruction without looking backward to reﬁne prediction.
However, in a realistic scenario, user instructions can be ambiguous or inaccurate especially when the target action is difﬁcult or inconvenient to articulate.
Reasoning in such cases is inherently iterative.
Therefore, it is important and beneﬁcial to incorporate interaction for resilient grounding (Suhr et al., 2019; Chandu et al., 2021).
In this paper, we investigate interactive grounding on GUIs, which aligns multimodal input to actionable objects of a screen.
We focus on single-screen interaction which is the building block of UI reasoning.
Speciﬁcally, we introduce the M UG ( M ulti-turn U
I G rounding) task in which the user iteratively guides the agent to select a desired UI object (see Fig. 1).
With a given UI and a target object, the user instructs the agent via natural language, ranging from casual intent to more descriptive commands.
The agent infers which UI object is intended by the user and and highlights it.
If the agent is correct, the user can conﬁrm the selection and the grounding is completed.
Otherwise, the user issues further guidance, e.g., "Click the one below" , to the agent to reﬁne its selection.
We collecte the M UG dataset from live interaction sessions between pairs of human annotators—one acts as the user and the other as the agent.
Our dataset has 77 , 820 examples, each records the transaction history in a session.
Specially, 20% of the dataset are challenging ones as their human commands need multiple rounds to ground, even for human agents.
To establish the benchmark, we experiment with a range of variants to model the dynamics between the two roles.
While the main goal of the task is to develop agent models for grounding, we also develop the user models for online instruction simulation.
We build our models upon a Transformer-
[Figure]
based encoder-decoder architecture (Li et al., 2021), and experiment with various learning methods, including traditional sequence modeling and reinforcement learning.
To fully examine the model performances, we evaluate the agent model with a spectrum of evaluation strategies, including both ofﬂine and online evaluations.
For the online evaluation, we employ both automatic and human evaluations, which include interactions between the agent and the user (either a human or the user model) and offer a comprehensive probe into model understanding.
Our experiments show that incorporating interaction substantially improves UI grounding task completion by 18% on the entire dataset and 31% on the challenging set, both in absolute scales.
Furthermore, our robustness measurements suggest M UG , while being a seemingly easy single-screen task, is actually difﬁcult since neural agents sometimes struggle to correct themselves, resulting in repeated wrong selections across multiple turns.
This suggests large rooms for future improvement in grounding agents.
In summary, our key contributions 1 are:
We introduce M UG , a novel interactive vision-language task that focuses on multi-turn language grounding on a graphical UI screen, which is a challenging task that is meant to improve language grounding in realistic UIs.
We create a rich dataset that includes 77,820 examples recorded from live sessions between pairs of human users and agents.
And 20% of the data are challenging for both human annotators and neural agents.
We experiment with a range of model variants and evaluation strategies, showing that iterative interaction signiﬁcantly improves grounding accuracy by 18% and 31% on the entire and challenging test sets respectively, with automatic assistance from our user models.
Our work lays a solid foundation for future investigations on collaborative grounding.
B ACKGROUND
Multi-modal modeling has a long history of research (e.g., Winograd, 1972; Barnard & Forsyth, 2001;
Lavrenko et al., 2003; Plummer et al., 2015; Yu et al., 2016).
One important area focuses on grounding objects in images where the natural language is used as an additional input (Chen et al., 2017;
Yu et al., 2016; 2018; Fukui et al., 2016;
Deng et al., 2021).
Interactive Multimodal Grounding Prior works have formulated grounding as a multi-step reasoning task, e.g., navigation via multiple steps of grounding (e.g., Ku et al., 2020;
Gur et al., 2019).
Our work differs by focusing on agent’s ability to self-correct in synchronized turns of interaction on a UI screen.
It is also conceptually linked to repeated reference game (Hawkins et al., 2020), except we use a different form of communication (language-action) instead of dialogue (languagelanguage).
Our task leverages iteratively reﬁned instructions on atomic action instead of the increased instruction utility over multi-step actions (Effenberger et al., 2021).
Our work models both the user and the agent, and let them communicate online.
This is different from single-sided modelings (Suhr et al., 2019; Kojima et al., 2021).
Our observation that interaction improves grounding is also in line with dialogue-based works (e.g., Haber et al., 2019;
Takmaz et al., 2020).
UI Grounding Grounding UI objects involves automatic completion of actions on web or mobile interfaces (e.g. Pasupat et al., 2018;
Li et al., 2020a; He et al., 2020).
It is also an important accessibility task for users who are situationally impaired when they are occupied by real-world tasks at hand (Sarsenbayeva, 2018).
Compared to grounding on natural images, these tasks usually take well-speciﬁed user commands and aim to select the object that best matches the command.
The UI image is often encoded via ResNet (He et al., 2016) or ViT (Dosovitskiy et al., 2020).
The structure and text features of UI are often encoded by Transformer model (Vaswani et al., 2017).
Fusing multimodal information is widely handled by cross-attention (e.g. He et al., 2020; Li et al., 2021; Bai et al., 2021).
We adopt these neural components in our benchmark.
Mobile UI Datasets Many grounding tasks, while covering multiple screens, remain one-pass reasoning, such as P IXEL H ELP (Li et al., 2020a) and M O TIF
(Burns et al., 2022).
Prior works (e.g.,
Todi et al., 2021) model sequence of (action, state) pairs via reinforcement learning (RL).
In contrast,
M UG focuses on correcting a single action on one screen.
Tab.
1 summarizes key differences among other Mobile UI datasets.
Importantly, M UG is a challenging task as it enables corrective interaction in synchronized turn between user and agent.
[Table]
T ASK F ORMULATIONS
As a grounding task, M UG involves two participants: a user and an agent.
Our formulation includes both roles to provide a holistic view of interactive grounding.
The user’s goal is to instruct, via natural language, the agent to select the desired object g on the UI screen S .
The unique aspect of
M UG is that it allows the user to guide the agent iteratively to identify the target action by issuing a series of commands, each in response to the agent’s prior inference.
We separate such user-agent interaction into turns.
At turn t , the interaction consists of:
[Equation]
where the user ﬁrst instructs the agent with command c t , and the agent responds with a suggestion of action a t .
Here a t is essentially the index of object.
The task is completed when a t = g .
3. A GENT T ASK
In M UG , the action space for the agent consists of a set of UI objects to click on the interface, e.g., in Fig 1.
Intuitively, we would want the agent to take the desired action g as early as possible, i.e., as few turns as possible.
Thus, at turn t , the agent models
[Equation]
where θ denotes the agent parameters.
This iterative grounding early stops once a t = g or t reaches a maximum number of turns allowed.
3. U SER T ASK
The user’s role is to provide guidance to the agent through iteratively reﬁned instructions.
In contrast to one-pass prediction tasks (e.g. Pasupat et al., 2018;
He et al., 2020) where the agent makes a oneshot guess, a M UG user issues follow-up commands that are dependent of prior instructions c [0 ,t − 1] and agent actions a [0 ,t − 1] , which is formalized as the following:
[Equation]
where φ denotes the user.
Here, the user model is aware of the target object g .
Interplay between User and Agent The agent task (Eq. 1) is the pivot of M UG .
The user task (Eq. 2) aims to guide agent towards task completion, which potentially includes online training.
In our benchmark, we let the user and agent play together.
Although automatic evaluation is not as realistic as human evaluation, it offers a fast, low-cost, and reproducible environment.
This setting also allows us to study various questions surrounding the interplay between the two, e.g., whether an automatic user can assist an agent?
and whether agent errors would confuse the user?
D ATASET C REATION
As there is no available dataset for model training and evaluation, we developed an interactive labeling interface to collect data for M UG .
Our data collection involves two human annotaters to play the roles of the user and the agent respectively in a live session.
The user and the agent have two separate views, running on different machines (Appx.
A).
Both views share the same UI screen and a message box showing instruction history.
Our task embodies the eyes-on, hands-free situation for mobile interaction where the user is required to only use language for the task, and the machine responds its prediction by highlighting.
The user can commit the action if the prediction is conﬁrmed.
During a session, only the user can see the target; and the the message box is read-only to the agent thus no language-base dialogue would take place.
4. A NNOTATION W ORKFLOW
We use the UI corpus, mobile UI screenshots and view hierarchies, from R ICO (Deka et al., 2017) and auxiliary object features from the C LAY dataset (Li et al., 2022).
Each session starts with a randomly sampled UI object (e.g., a button ), from the visible view hierarchy, as the target object g .
User annotators are encouraged to articulate their initial command ( c 0 ) casually or goal-oriented.
We consider such design to cover the realistic scenarios discussed in Sec. 1, and potentially free users from composing long and precise instructions.
In the agent view, all clickable objects on the UI screen are revealed with their bounding boxes highlighted, which show what objects the agent can select, without indicating which one is the target g .
The current agent selection is reﬂected on both the user and the agent’s view.
The session continues to the user’s turn if the agent selection does not match g .
In follow-up turns, the user is not allowed to repeat a command issued in previous turns, and likewise the agent is not allowed to select an previously chosen object.
Upon the agent selection matching the target object in the user view the task is completed.
Each session allows up to 5 turns and we ﬁlter out those unﬁnished.
4. R ESULTS & A NALYSIS
We collected 77,820 examples based on 31,265 unique screens from 7,132 apps (see details in
Table 2).
We split the dataset into the training, development, and test sets.
We use app-wise split (Li et al., 2020b) to avoid potential leaking across sets.
As shown in Table 2, the three splits have a similar distribution regarding the number of turns in each example.
In general, users tend to provide short instructions ( ∼ 4 words).
Across the dataset, annotators completed ∼ 80% examples in one turn ( i.e., one instruction and one selection ).
The rest 20%
[Table]
dataset is considered to be more challenging, with the majority taking 2 − 3 turns.
We call this 20% as the Challenging subset.
In Sec. 6, we will show that examples requiring more turns for human agents are also more difﬁcult for grounding models.
We refer readers to Appx.
F for examples.
We should note that while the 20% multi-turn ratio seems a low percentage but it leads to large impact in practice. Firstly, it lines up an upper bound for agent ground accuracy (i [Table] e., 80% ).
Moreover, real-world grounding sessions can also span multiple screens. If we apply the same 20% rate here, the probability for a multi-screen session to require correction is much more signiﬁcant. For instance, 67% with 5 screens. Lastly, we will see in Sec. 6 that neural agents could also struggle at instructions that are trivial for human to execute.
In Table
[Table]
we categorize 200 Challenging examples from the development split.
We found followup commands are mainly to make spatial adjustments ( 50% + 10% ) or to add extra information.
[Table]
G ROUNDING M ODELS
We aim to have a general architecture for the UI domain and explore its variants to model multi-turn interaction.
Our agent model is based on a transformer encoder-decoder network, inspired by (Li et al., 2021).
The encoder comprehends the objects on the UI screen (Sec. 5.1) and the decoder predicts a target action based on the UI and interaction history (Sec. 5.2).
5. UI E NCODER
Our encoder processes the interface S .
Each S consists of two modalities of information, i.e., a screenshot I S and view hierarchy features ψ (Deka et al., 2017; Li et al., 2022).
The concrete list of ψ is in Appx.
D.
The output is an encoding v k for each object indexed by k , similar to (e.g., Li et al., 2020a;
He et al., 2020; Li et al., 2020b):
[Equation]
[Equation] [Equation] For the image, we use a pre-trained ResNet-50 (He et al., 2016) which is ﬁne-tuned with other modules.
The resulted Φ S (grid size of h × w ) is then mapped to object level by region-of-interest (ROI) pooling (Ren et al., 2015).
Finally, we fuse multimodal features for each object by a transformer encoder T enc .
The output v stands for a sequence of object encodings which are interaction-agnostic.
5. G ROUNDING D ECODER
We use a causal transformer T dec to predict from interaction history.
The architecture is similar to (Li et al., 2021), except we use multi-turn interaction as input, e.g., c [ is a vector z t that summarizes prior interaction up to c t :
[Equation]
where a t denotes object index, either from model prediction or human selection.
The speciﬁc input to Eq.
5 will be subject to modeling variants in Sec 6.1.
For classiﬁcation, we use a linear layer f to score the k -th object:
[Equation]
E XPERIMENTS
The goal of our experiments is to explore training and evaluation methods for M UG and establish a benchmark.
Simply matching the initial instruction and object text is insufﬁcient as text features are often incomplete.
For instance, the validation split has 46% objects missing text, and a deterministic classiﬁer using METEOR (Banerjee & Lavie, 2005) has only 21% F1.
In this section, we present setups for the neural architecture discussed in Sec. 5.
In Sec. 6.1, we explore multiple variants for the agent.
For automatic evaluation, we present a simple and effective heuristics-based user model and a neural version in Sec.
Finally, we show extensive F1 results in Sec.
6.4 and 6.5, robustness in 6.6, ablations in 6.7 and 6.8, and error analysis in 6.9.
We refer readers to Appx.
E for hyperparameters, Appx. H for sample predictions, and Appx.
G for extended discussion. To avoid confusion, we thereafter use a (cid:48) t to refer to the selection predicted by the agent model at turn t , while a t to the human agent’s selection.
Similarly, we refer c (cid:48) t to instruction generated by the user model while c t to the one from the human user.
6. A GENT M ODELS
Our agent models use the T enc and T dec (in Sec. 5) as a backbone, denoted as θ .
Recall that T enc processes S while T dec processes interaction.
Here, we discuss different handlings of T dec .
Single or Multi-turn Model
The ﬁrst factor we investigate is how allowing multiple turns helps grounding.
For each example, we can feed the entire interaction history as input to the agent model and supervise agent selection on the very last turn T :
[Equation]
We can further reduce the input to be ( S, c 0 ) only, making a single-turn model.
To evaluate singleturn model with multi-turn examples, we simply concatenate all c t into one instruction.
Instruction-only Model To understand how it helps grounding by taking into account of previous actions of the agent in the multi-turn model (Eq. 7), we introduce the command-only baseline, which ignores agent actions (selections) in the interaction history:
[Equation]
Imitation Model Instead of supervising the agent only at the last turn, we can model the entire action sequence as an imitation model:
[Equation]
This variant investigates whether the supervision of the intermediate actions helps.
Ofﬂine RL Lastly, because each turn the agent action affects how the user responds, M UG can be formulated as a RL problem where the user and the UI constitute the environment.
We use the
Decision Transformer (Chen et al., 2021) for ofﬂine RL, by inserting extra learnable return tokens w t to the T dec before each action: T dec ( v, c 0 , w 0 , v a 0 , ..., c t , w t ) .
The model is:
[Equation]
[Equation] Possible discrete return tokens are { 1 , 2 , 3 , 4 } where 1 on the last turn.
During testing, we follow Chen et al.
(2021) to force the current turn to have return 1 and adjust prior returns. [Table]
U SER MODELS
Here, we design a simple and effective heuristics-based user model [Table] and then develop a neural version.
To show automatic online evaluation is a promising direction for M UG , we also conducted human evaluation on a shared set of 500 examples from the test split (Sec. 6.7)
[Table]
Heuristics-based Model We observe that, when the selection a (cid:48) is incorrect, we can deterministically devise a follow-up instruction by using a template as below:
Not the a (cid:48) t , click the g to/on the dir .
This template is to be instantiated on view hierarchy features (in Appx.
D).
Compared to human follow-ups, heuristic ones are more speciﬁc and longer, such as:
[Table]
Neural Instruction Model We extend the Multi agent architecture to model follow-up commands:
[Equation]
which has decoder T dec ( v, v g , c 0 , v a 0 , c 1 , ..., v a t − 1 ) at turn t .
For training, we teacher-force at each turn ( t > 0 ).
We found that using heuristics as prompt greatly boosts development CIDEr (Vedantam et al., 2015) to from 70 to 78 .
For inference, we use greedy decoding with a maximum length of 12 .
6. M ETRICS
We focus on evaluating the agent model as it is the pivot task of M UG .
Intuitively, we would want the agent to take the desired action g with as few turns as possible.
That is,
[Equation]
where, in practice, we compute F1 t with early stop over turns to avoid double counting.
Clearly, an agent with high F1 and a lower value of t is better than an agent that requires more turns for the same accuracy.
With t limited to 0 , the task is reduced to a one-pass grounding task.
In an extreme case, we consider an agent with high F1
but ﬂat changes in F1 t > 0 to be problematic, since it questions the agent’s understanding about the interface.
For more comprehensive testing, we also use a simple robustness metric for prediction changes across turns:
[Equation]
(13) which is the percentage of examples that have duplicate actions within T valid turns.
We observe that user models, unlike human annotators, sometimes give the same instruction across turns, especially when the agent repeats the same error.
Therefore, we only count agent selections that are associated with unique instructions in each example, and ignore other turns.
6. O FFLINE R ESULTS
Tab. 4 presents ofﬂine results on the test set, over the Challenging (see Sec. 4) and the All sets.
During inference, we use instructions from the human user and actions from the human agent for turns in between and ask an agent model to predict at each turn.
Doing so requires agent models to correct human agent actions, instead of the model’s own.
Clearly, the models that take into account interaction history outperform those use none or partially.
While the Ins-only and the Imitation models perform closely on the All set, they bear larger margins on the Challenging and online tests.
[Table]
6. O NLINE R ESULTS
Tab. 5 presents online test scores.
In general, models that are supervised by action sequences (i.e.,
Imitation and Ofﬂine RL ) perform better.
Both heuristics-based and neural user models are able to guide agents towards task completion.
Comparing Single ’s F1 0 and Imitation ’s F1 4 , we see that properly using interaction boosts task completion by 18 and 31 on the Challenging and All test sets.
The average F1 4 ‘s show that heuristics-based user works better, except that the Imitation collaborates better with the neural user.
This might be attributed to the neural user is trained to mimic human command patterns which can be ambiguous and short, while heuristics are more precise while being artiﬁcial.
This also implies that a large room for further improvement to the user modeling.
Overall, we can see interactive grounding is a challenging task, even on a single screen.
The agent modeling involves robust multimodal understanding to self-correct.
The user modeling requires controlled language generation, which is still an open problem.
The best task completion rate on the
Challenging subset is only ∼ 55% , suggesting a large room for future improvements.
6. A GENT R OBUSTNESS
We take a deeper look at agent behavior in Tab. 6.
We observe that agents with higher F1 tend to be more robust (lower Γ ).
The best agent model ( Imitation ) repeats the same mistake for only 16 .
8% on the All test set.
However, if we ignore those examples ﬁnished in 1 turn i.e., T > 1 columns, the repeating rate rises to ∼ 40% .
The Heuristics user, while generally improves agent F1 more than the Neural user, has a mixed robustness impact on the Imitation and Ofﬂine RL agents.
On weaker agents (the ﬁrst 3 rows), the Heuristics user leads to more salient robustness.
These observations suggest improving agent F1 has a more direct and positive impact on robustness.
[Table]
A UTOMATIC E VALUATION V . S . H UMAN E VALUATION
To show automatic online test is a promising surrogate for human-in-the-loop evaluation [Table] we compare Single with Multi 2 with a group of human annotators (acting as the user )
(Tab. 7).
We ask the user annotators to follow the same annotation interface and guideline in Sec.
[Table]
and let them to use the trained agent model to ground their commands. That is, human plays the user role and a trained agent model plays the agent role. This setting maximally mimics a realistic situation where a human user guides the agent to locate a target solely using language commands. The results (Tab. 7) are generally consistent with those from the automatic evaluation (Tab. 5).
We should also note that such human study is not meant to reﬂect every minor differences in automatic evaluations.
[Table]
6. A BLATION ON H EURISTICS
To show agent improves from follow-up instructions effectively, instead of overﬁtting potential artifacts in the dataset [Table] we report our ablation studies in Tab. 8.
Speciﬁcally [Table] we focus on the heuristicsbased user since it offers well-controlled instruction generation.
We can see that random heuristics underperform by ∼ 14% and repeating the initial instruction is even worse.
The Γ scores also suggest that randomly instantiated instructions are less effective in guiding the agent.
[Table]
6. E RROR A NALYSIS
We manually analyze errors from the best agent ( Imitation ).
In Tab. 9, we inspect 30 failed development examples (i.e., unﬁnished after 5 turns) that are subject to the Neural user.
Due to the role interplay, we also count problematic commands.
We observe that the user model sometimes issues repetitive or uninformative instructions starting from the 3 rd turn, leading the agent to the same wrong selection.
This might be caused by the data sparsity for examples with ≥ 3 turns.
[Table]
C ONCLUSIONS
In this paper, we presented M UG , a novel and challenging task for multimodal grounding on UI.
M UG requires a grounding agent being able to correct its own prediction, and allows a user to guide the agent via natural language instructions.
We contribute a new dataset for the task, investigate various modeling options for the agent and developed multiple evaluation strategies including two user models so as to enable automatic online testing.
We found that interaction greatly improves grounding accuracy in the UI domain.
Our experiments and analyses also suggest large room for grounding performances, even on a seemingly easy single screen task, which calls for future investigation.
Our work contributes to the general effort of multimodal language understanding and its robustness by enabling synchronized multi-turn interactivity.
A L ABELING I NTERFACE
Fig.
2 presents the user and agent views in our data collection interface.
In the user view, the user can send commands in the message box, to instruct the agent to select the target object as highlighted by a red bounding box on the UI screen.
On the agent’s view, the agent annotator can respond the user request by performing object selection on the UI screen which has all the clickble objects highlighted.
But there is no indication of the target object so the agent annotator has to guess from the user instruction.
The agent is not allowed to text back to the user.
The agent’s current selection is reﬂected on the UI screen so the user understands how to further instruct the agent.
The annotation task is designed based on the eyes-on hands-free situation of mobile interaction.
B D ETAILS OF THE L ABELING T ASK
The labelers of the task were native English speakers and had experience using mobile phones.
They were trained with a few pilot tasks to get familiar with the task, during which we also improved the labeling interface and the guidelines based on labelers’ feedback.
The dataset was completed bylabelers in 10 batches.
The labeling quality was monitored by sampling examples from each batch for manual examination. [Figure]
C L ANGUGAE D IVERSITY
The word-level vocabulary in the training set consists of 13 , 794 unique words.
Fig.
3 shows the distribution of the 50 most frequent words in the training split with certain non-content words (e.g., is , of , comma ) ﬁltered out.
D V IEW H IERARCHY F EATURES
Tab.
10 lists the complete view hierarchy features we used.
We unify each feature into a real-valued vector.
These view hierarchy features are ﬁrst represented with trainable embeddings, and then encoded by the transformer model (Sec. 6.1).
For text attributes (e.g., text ), we max-pool their noncontextualized token embeddings, which are randomly initialized and trained.
For discrete-valued attributes (e.g., type ), we use a trainable vector for each possible value.
The ordering of objects in transformer input follows the pre-order traversal in the view hierarchy (which is a tree structure).
We then combine the vision representations of individual UI objects via ROI pooling over ResNet featuremap of the encoded screenshot image, and view hierarchy encoding to form a multimodal representation of each UI object for the downstream computation of the model.
We consider these view hierarchy features to be auxiliary.
There is often a huge gap between what command the user would issue based on what they see on the UI, and what the underlying information is for the UI.
As we discussed in Sec. 6, about 46 % of UI objects do not have a text label, and the user would need to come up with their own language description about the object, which is why the text matching baseline fails. Even when there are text descriptions, they are not necessarily what the user would articulate since a user command can be abstract. Fundamentally, the internal representation of the UI is often inaccessible or uninterpretable to the user, thus calling for the help of multimodal modeling and interaction modeling. [Table]
E H YPERPARAMETERS & T RAINING
For all our agent models, we use the same conﬁgurations, which are grid-searched based on models’ ofﬂine validation performances.
Our hyperparameters are chosen from the best ofﬂine development F1 scores.
For the number of self-attention modules, we grid-searched in { 1 , 2 , 4 , 6 } , which resulted in 2 hidden layers for the user interface Transformer encoder and 6 hidden layers for the grounding decoder.
Each self-attention module uses 8-head self and encoder-decoder attention with a 256 hidden size.
The dropout rate for attention and MLP layers is 0 . 1 , which is grid-searched in { 0 . 1 , 0 . 2 , 0 . 5 } .
For learning rate, we grid-searched from { 1 e - 3 , 3 e - 4 , 1 e - 4 , 3 e - 5 , 1 e - 5 } , and use 3 e - 4 with linear warmup with cosine annealing for the ﬁrst 10 k steps.
All the models are trained to 100 k steps with a batch size of 128 on a 32 -core Google Cloud TPUv3.
Models are evaluated every 1 k steps and the version with the best development ofﬂine F1 4 is saved.
The training time for our agent model is around 8 hours.
Our neural user model has the same grid-searched conﬁguration as the agent, i.e., 2 encoder layers, 6 decoder layers, 0 . 1 for dropout, and the same warmup scheduling.
The best learning rate is 1 e - 4 .
Different from the agent model, we found the neural user model’s development CIDEr score quickly drops after 6 k steps, possibly due to overﬁtting and data sparsity, thus its training early-stops there.
F E XAMPLES IN THE M UG D ATASET
We present some examples from the M UG dataset in Fig.
4 and 5.
Each example contains instructions and selections from human user and agent annotators. G L IMITATIONS
English-only Dataset While non-English examples exists, we acknowledge that M UG mostly consists of English UI.
Other languages do exist in the dataset, but consists of a small portion.
Specifically, our instructions are English-only.
Future extensions to our work should address or alleviate this issue. Platform-speciﬁc Interfaces Our interfaces, since coming from RICO, only consist of Android screens. In practice, it is also difﬁcult to obtain non-Android interfaces. We acknowledge this is an application limitation.
And the bias from the top and bottom banner of Android could make trained model brittle in other domains.
Going beyond Single Screen We aim to establish the task and report baseline performances for future work.
The interaction in M UG happens within the same user interface.
A natural extension would be extending the task to span over sequence of interfaces.
Indeed, the task would become more challenging, and potentially require large ofﬂine training data and reliable online simulation.
Better User Model
{+The current best neural instruction generation we use has a CIDEr 78 . 
+}
on the validation set.
We acknowledge there is space for further improvement.
Note that our neural instructions are trained on multi-turn examples in M UG , which amounts to ∼ 20% of the training data.
It suggests external resources could be useful for improving user model performances.
Interaction Dynamics between User and Agent It would be helpful to study how/why the agent sometime repeatedly makes incorrect actions in Tab. 6, such as whether repeated mistakes are due to the lack of language utility/diversity in user instruction or the lack of understanding in the agent.
H P REDICTION E XAMPLES
Here, we demonstrate predictions from the Imitation model.
Fig.
6 demonstrates successfully solved examples following the instructions generated by the Heuristic user model, while failed ones are in
Fig.
Similarly, Fig. 8 demonstrates solved ones following the instructions generated by the Neural user model, and failed ones are in Fig.