Original: XT2: T RAINING AN X- TO -T EXT T YPING I NTERFACE WITH O NLINE L EARNING FROM I MPLICIT F EEDBACK
Revised: X2T: T RAINING AN X- TO -T EXT T YPING I NTERFACE WITH O NLINE L EARNING FROM U SER F EEDBACK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We aim to help users communicate their intent to machines using ﬂexible, adaptive interfaces that translate arbitrary user input into desired actions.
Revised: We aim to help users communicate their intent to machines using ﬂexible, adaptive interfaces that translate arbitrary user input into desired actions.

Original: In this work, we focus on assistive typing applications in which a user cannot operate a keyboard, but can instead supply other inputs, such as webcam images that capture eye gaze.
Revised: In this work, we focus on assistive typing applications in which a user cannot operate a keyboard, but can instead supply other inputs, such as webcam images that capture eye gaze.

Original: Standard methods train a model on a ﬁxed dataset of user inputs, then deploy a static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging.
Revised: Standard methods train a model on a ﬁxed dataset of user inputs, then deploy a static interface that does not learn from its mistakes; in part, because extracting an error signal from user behavior can be challenging.

Original: We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learning fromimplicit user feedback on the accuracy of the interface’s actions.
Revised: We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user: online learning from user feedback on the accuracy of the interface’s actions.

Original: In the typing domain, we leverage backspaces asimplicit feedback that the interface did not perform the desired action.
Revised: In the typing domain, we leverage backspaces as feedback that the interface did not perform the desired action.

Original: We propose an algorithm called x-to-text (XT2) that trains a predictive model of thisimplicit feedback signal, and uses this model to ﬁne-tune any existing, default interface for translating user input into actions that select words or characters.
Revised: We propose an algorithm called x-to-text (X2T) that trains a predictive model of this feedback signal, and uses this model to ﬁne-tune any existing, default interface for translating user input into actions that select words or characters.

Original: We evaluate XT2 through a small-scale online user study with 12 participants who type sentences by gazing at their desired words, and a large-scale observational study on handwriting samples from 60 users.
Revised: We evaluate X2T through a small-scale online user study with 12 participants who type sentences by gazing at their desired words, and a large-scale observational study on handwriting samples from 60 users.

Original: The results show that XT2 learns to outperform a non-adaptive default interface, stimulates user co-adaptation to the interface, personalizes the interface to individual users, and can leverage ofﬂine data collected from the default interface to improve its initial performance and accelerate online learning.
Revised: The results show that X2T learns to outperform a non-adaptive default interface, stimulates user co-adaptation to the interface, personalizes the interface to individual users, and can leverage ofﬂine data collected from the default interface to improve its initial performance and accelerate online learning.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recent advances in user interfaces have enabled people with sensorimotor impairments to more effectively communicate their intent to machines.
Revised: Recent advances in user interfaces have enabled people with sensorimotor impairments to more effectively communicate their intent to machines.

Original: For example, Ward et al.
Revised: For example, Ward et al.

Original: (2000) enable users to type characters using an eye gaze tracker instead of a keyboard, and Willett et al.
Revised: (2000) enable users to type characters using an eye gaze tracker instead of a keyboard, and Willett et al.

Original: (2020) enable a paralyzed human patient to type using a brain implant that records neural activity.
Revised: (2020) enable a paralyzed human patient to type using a brain implant that records neural activity.

Original: The main challenge in building such interfaces is translating high-dimensional, continuous user input into desired actions.
Revised: The main challenge in building such interfaces is translating high-dimensional, continuous user input into desired actions.

Original: Standard methods typically calibrate the interface on predeﬁned training tasks for which expert demonstrations are available, then deploy the trained interface.
Revised: Standard methods typically calibrate the interface on predeﬁned training tasks for which expert demonstrations are available, then deploy the trained interface.

Original: Unfortunately, this does not enable the interface to improve with use or adapt to distributional shift in the user inputs.
Revised: Unfortunately, this does not enable the interface to improve with use or adapt to distributional shift in the user inputs.

Original: In this paper, we focus on the problem of assistive typing: helping a user select words or characters without access to a keyboard, using either eye gaze inputs (Ward et al., 2000) or handwriting inputs  (Willett et al., 2020).
Revised: In this paper, we focus on the problem of assistive typing: helping a user select words or characters without access to a keyboard, using either eye gaze inputs (Ward et al., 2000), or handwriting inputs (see Figure 6 in the appendix), which can be easier to provide than direct keystrokes (Willett et al., 2020).

Original: To enable any existing, default interface to continually adapt to the user, we train a model using online learning from user feedback.
Revised: To enable any existing, default interface to continually adapt to the user, we train a model using online learning from user feedback.

Original: The key insight is that the user providesimplicit feedback on the interface’s actions via backspaces , which indicate that the interface did not perform the desired action in response to a given input.
Revised: The key insight is that the user provides feedback on the interface’s actions via backspaces , which indicate that the interface did not perform the desired action in response to a given input.

Original: By learning from this implicit signal instead of an explicit label, we do not require any additional effort from the user to improve the interface.
Revised: By learning from this naturally-occurring feedback signal instead of an explicit label, we do not require any additional effort from the user to improve the interface.

Original: Furthermore, because our method is applied on top of the user’s default interface, our approach is complementary to other work that develops state-of-the-art, domain-speciﬁc methods for problems like gaze tracking and handwriting recognition.
Revised: Furthermore, because our method is applied on top of the user’s default interface, our approach is complementary to other work that develops state-of-the-art, domain-speciﬁc methods for problems like gaze tracking and handwriting recognition.

Original: Figure 1 describes our algorithm: we initialize our model using ofﬂine data generated by the default interface, deploy our interface as an augmentation to the default interface, collect online feedback, and update our model.
Revised: Figure 1 describes our algorithm: we initialize our model using ofﬂine data generated by the default interface, deploy our interface as an augmentation to the default interface, collect online feedback, and update our model.

Original: [Figure]
Revised: [Figure]

Original: We formulate assistive typing as an online decision-making problem, in which the interface receives observations of user inputs, performs actions that select words or characters, and receives a reward signal that is automatically constructed from the user’s backspaces.
Revised: We formulate assistive typing as an online decision-making problem, in which the interface receives observations of user inputs, performs actions that select words or characters, and receives a reward signal that is automatically constructed from the user’s backspaces.

Original: To improve the default interface’s actions, we ﬁt a neural network reward model that predicts the reward signal given the user’s input and the interface’s action.
Revised: To improve the default interface’s actions, we ﬁt a neural network reward model that predicts the reward signal given the user’s input and the interface’s action.

Original: Upon observing a user input, our interface uses the trained reward model to update the prior policy given by the default interface to a posterior policy conditioned on optimality, then samples an action from this posterior (see Figure 1).
Revised: Upon observing a user input, our interface uses the trained reward model to update the prior policy given by the default interface to a posterior policy conditioned on optimality, then samples an action from this posterior (see Figure 1).

Original: We call this method x-to-text (XT2), where x refers to the arbitrary type of user input; e.g., eye gaze or handwriting (see Figurein the appendix).
Revised: We call this method x-to-text (X2T), where x refers to the arbitrary type of user input; e.g., eye gaze or handwriting (see Figurein the appendix).

Original: Our primary contribution is the XT2 algorithm for continual learning of a communication interface fromimplicit user feedback.
Revised: Our primary contribution is the X2T algorithm for continual learning of a communication interface from user feedback.

Original: We primarily evaluate XT2 through an online user study withparticipants who use a webcam-based gaze tracking system to select words from a display.
Revised: We primarily evaluate X2T through an online user study with 12 participants who use a webcam-based gaze tracking system to select words from a display.

Original: To run ablation experiments that would be impractical in the online study, we also conduct an observational study with 60 users who use a tablet and stylus to draw pictures of individual characters.
Revised: To run ablation experiments that would be impractical in the online study, we also conduct an observational study with 60 users who use a tablet and stylus to draw pictures of individual characters.

Original: The results show that XT2 quickly learns to map input images of eye gaze or character drawings to discrete word or character selections.
Revised: The results show that X2T quickly learns to map input images of eye gaze or character drawings to discrete word or character selections.

Original: By learning from online feedback, XT2 improves upon a default interface that is only trained once using supervised learning and, as a result, suffers from distribution shift (e.g., induced by changes in the user’s head position, lighting, and other visual conditions over time in the eye gaze experiment).
Revised: By learning from online feedback, X2T improves upon a default interface that is only trained once using supervised learning and, as a result, suffers from distribution shift; e.g., induced by changes in the user’s head position, lighting, and other visual conditions over time in the eye gaze experiment.

Revised: X2T automatically overcomes calibration problems with the gaze tracker by adapting to the mis-calibrations over time without the need for re-calibration.

Original: Furthermore, XT2 leverages ofﬂine data generated by the default interface to accelerate online learning, stimulates co-adaptation from the user in the online study, and personalizes the interface to the handwriting style of each user in the observational study.
Revised: Furthermore, X2T leverages ofﬂine data generated by the default interface to accelerate online learning, stimulates coadaptation from the user in the online study, and personalizes the interface to the handwriting style of each user in the observational study.

Original:  L EARNING TO I NFER I NTENT FROM U SER I NPUT
Revised:  L EARNING TO I NFER I NTENT FROM U SER I NPUT

Original: In our problem setting, the user cannot directly perform actions; e.g., due to a sensorimotor impairment.
Revised: In our problem setting, the user cannot directly perform actions; e.g., due to a sensorimotor impairment.

Original: Instead, the user relies on an assistive typing interface, where the user’s intended action is inferred from available inputs such as webcam images of eye gaze or handwritten character drawings.
Revised: Instead, the user relies on an assistive typing interface, where the user’s intended action is inferred from available inputs such as webcam images of eye gaze or handwritten character drawings.

Original: As such, we formulate assistive typing as a contextual bandit problem (Langford & Zhang, 2008; Yue & Joachims, 2009; Li et al., 2010; Lan & Baraniuk, 2016; Gordon et al., 2019).
Revised: As such, we formulate assistive typing as a contextual bandit problem (Langford & Zhang, 2008; Yue & Joachims, 2009; Li et al., 2010; Lan & Baraniuk, 2016; Gordon et al., 2019).

Original: At each timestep, the user provides the interface with a context x ∈ X , where X is the set of possible user inputs (e.g., webcam images).
Revised: At each timestep, the user provides the interface with a context x ∈ X , where X is the set of possible user inputs (e.g., webcam images).

Original: The interface then performs an action u ∈ U , where U is the set of possible actions (e.g., word selections).
Revised: The interface then performs an action u ∈ U , where U is the set of possible actions (e.g., word selections).

Original: We assume the true reward function is unknown, since the user cannot directly specify their desired task (e.g., writing an email or ﬁlling out a form).
Revised: We assume the true reward function is unknown, since the user cannot directly specify their desired task (e.g., writing an email or ﬁlling out a form).

Original: Instead of eliciting a reward function or explicit reward signal from the user, we automatically construct a reward signal from the user’s backspaces .
Revised: Instead of eliciting a reward function or explicit reward signal from the user, we automatically construct a reward signal from the user’s backspaces .

Original: The key idea is to treat backspaces asimplicit feedback on the accuracy of the interface’s actions.
Revised: The key idea is to treat backspaces as feedback on the accuracy of the interface’s actions.

Original: Our approach to this problem is outlined in Figure 1.
Revised: Our approach to this problem is outlined in Figure 1.

Original: We aim to minimize expected regret, which, in our setting, is characterized by the total number of backspaces throughout the lifetime of the interface.
Revised: We aim to minimize expected regret, which, in our setting, is characterized by the total number of backspaces throughout the lifetime of the interface.

Original: While a number of contextual bandit algorithms with lower regret bounds have been proposed in prior work (Lattimore & Szepesv´ari, 2020), we use a simple strategy that works well in our experiments: train a neural network reward model to predict the reward given the user’s input and the interface’s action, and select actions with probability proportional to their predicted optimality.
Revised: While a number of contextual bandit algorithms with lower regret bounds have been proposed in prior work (Lattimore & Szepesv´ari, 2020), we use a simple strategy that works well in our experiments: train a neural network reward model to predict the reward given the user’s input and the interface’s action, and select actions with probability proportional to their predicted optimality.

Original: Our approach is similar to prior work on deep contextual multi-armed bandits (Collier & Llorens, 2018) and NeuralUCB (Collier & Llorens, 2018), except that instead of using Thompson sampling or UCB to balance exploration and exploitation, we use a simple, stochastic policy.
Revised: Our approach is similar to prior work on deep contextual multi-armed bandits (Collier & Llorens, 2018) and NeuralUCB (Collier & Llorens, 2018), except that instead of using Thompson sampling or UCB to balance exploration and exploitation, we use a simple, stochastic policy.

Original: 2. M ODELING U SER B EHAVIOR ANDI MPLICIT F EEDBACK
Revised: 2. M ODELING U SER B EHAVIOR AND F EEDBACK

Original: Unlike in the standard multi-armed bandit framework, we do not get to observe an extrinsic reward signal that captures the underlying task that the user aims to perform.
Revised: Unlike in the standard multi-armed bandit framework, we do not get to observe an extrinsic reward signal that captures the underlying task that the user aims to perform.

Original: To address this issue, we infer rewards from naturally-occurring user behavior.
Revised: To address this issue, we infer rewards from naturally-occurring user behavior.

Original: In particular, in the assistive typing setting, we take advantage of the fact that we can observe when the user backspaces ; i.e., when they delete the most recent word or character typed by the interface.
Revised: In particular, in the assistive typing setting, we take advantage of the fact that we can observe when the user backspaces ; i.e., when they delete the most recent word or character typed by the interface.

Original: To infer rewards from backspaces, we make two assumptions about user behavior: (1) the user can perform a backspace action independently of our interface (e.g., by pressing a button); (2) the user tends to backspace incorrect actions; and (3) the user does not tend to backspace correct actions.
Revised: To infer rewards from backspaces, we make two assumptions about user behavior: (1) the user can perform a backspace action independently of our interface (e.g., by pressing a button); (2) the user tends to backspace incorrect actions; and (3) the user does not tend to backspace correct actions.

Original: Hence, we assign a positive reward to actions that were not backspaced, and assign zero reward to backspaced actions.
Revised: Hence, we assign a positive reward to actions that were not backspaced, and assign zero reward to backspaced actions.

Original: Formally, let r ∈ { 0 , 1 } denote this reward signal, where r = 0 indicates an incorrect action and r = 1 indicates a correct action.
Revised: Formally, let r ∈ { 0 , 1 } denote this reward signal, where r = 0 indicates an incorrect action and r = 1 indicates a correct action.

Original: For example, if the user wants to type “banana” and provides the interface with eye gaze input x , but the interface incorrectly types “apple”, then the user may choose to backspace “apple”, in which case the system automatically logs the input-action-reward triple ( x , u = apple , r = 0) .
Revised: For example, if the user wants to type “banana” and provides the interface with eye gaze input x , but the interface incorrectly types “apple”, then the user may choose to backspace “apple”, in which case the system automatically logs the input-action-reward triple ( x , u = apple , r = 0) .

Original: 2. T RAINING THE R EWARD M ODEL TO P REDICT F EEDBACK
Revised: 2. T RAINING THE R EWARD M ODEL TO P REDICT F EEDBACK

Original: In order to perform actions that minimize expected regret – i.e., the total number of backspaces over time – we need to learn a model that predicts whether or not the user will backspace a given action in a given context.
Revised: In order to perform actions that minimize expected regret – i.e., the total number of backspaces over time – we need to learn a model that predicts whether or not the user will backspace a given action in a given context.

Original: To do so, we learn a reward model p θ ( r | x , u ) , where p θ is a neural network and θ are the weights.
Revised: To do so, we learn a reward model p θ ( r | x , u ) , where p θ is a neural network and θ are the weights.

Original: Since the reward r ∈ { 0 , 1 } can only take on one of two values, p θ is a binary classiﬁer.
Revised: Since the reward r ∈ { 0 , 1 } can only take on one of two values, p θ is a binary classiﬁer.

Original: We train this binary classiﬁer on a dataset D of input-action-reward triples ( x , u , r ) .
Revised: We train this binary classiﬁer on a dataset D of input-action-reward triples ( x , u , r ) .

Original: In particular, we ﬁt the model p θ by optimizing the maximum-likelihood objective, i.e., the binary cross-entropy loss,
Revised: In particular, we ﬁt the model p θ by optimizing the maximum-likelihood objective, i.e., the binary cross-entropy loss,

Original: [Equation]
Revised: [Equation]

Original: Since XT2 learns from human-in-the-loop feedback, the amount of training data is limited by how frequently the user operates the interface.
Revised: Since X2T learns from human-in-the-loop feedback, the amount of training data is limited by how frequently the user operates the interface.

Original: To reduce the amount of online interaction data needed to train the reward model, we use ofﬂine pretraining.
Revised: To reduce the amount of online interaction data needed to train the reward model, we use ofﬂine pretraining.

Original: We assume that the user already has access to some default interface for typing.
Revised: We assume that the user already has access to some default interface for typing.

Original: We also assume access to an ofﬂine dataset of input-action pairs generated by the user and this default interface.
Revised: We also assume access to an ofﬂine dataset of input-action pairs generated by the user and this default interface.

Original: We assign zero rewards to the backspaced actions and positive rewards to the non-backspaced actions in this ofﬂine dataset, and initially train our reward model to predict these rewards given the user’s inputs and the default interface’s actions.
Revised: We assign zero rewards to the backspaced actions and positive rewards to the non-backspaced actions in this ofﬂine dataset, and initially train our reward model to predict these rewards given the user’s inputs and the default interface’s actions.

Original: Thus, when XT2 is initially deployed, the reward model has already been trained on the ofﬂine data, and requires less online interaction data to reach peak accuracy.
Revised: Thus, when X2T is initially deployed, the reward model has already been trained on the ofﬂine data, and requires less online interaction data to reach peak accuracy.

Original: 2. U SING THE R EWARD M ODEL TO S ELECT A CTIONS
Revised: 2. U SING THE R EWARD M ODEL TO S ELECT A CTIONS

Original: Even with ofﬂine pretraining, the initial reward model may not be accurate enough for practical use.
Revised: Even with ofﬂine pretraining, the initial reward model may not be accurate enough for practical use.

Original: To further improve the initial performance of our interface at the onset of online training, we combine our reward model p θ ( r | x , u ) with the default interface ¯ π ( u | x ) .
Revised: To further improve the initial performance of our interface at the onset of online training, we combine our reward model p θ ( r | x , u ) with the default interface ¯ π ( u | x ) .

Original: We assume that ¯ π is a stochastic policy and that we can evaluate it on speciﬁc inputs, but do not require access to its implementation.
Revised: We assume that ¯ π is a stochastic policy and that we can evaluate it on speciﬁc inputs, but do not require access to its implementation.

Original: We set our policy π ( u | x ) = p ( u | x , r = 1) to be the probability of an action conditional on optimality, following the control-as-inference framework (Levine, 2018).
Revised: We set our policy π ( u | x ) = p ( u | x , r = 1) to be the probability of an action conditional on optimality, following the control-as-inference framework (Levine, 2018).

Original: Applying
Revised: Applying

Original: Bayes’ theorem, we get p ( u | x , r = 1) ∝ p ( r = 1 | x , u ) p ( u | x ) .
Revised: Bayes’ theorem, we get p ( u | x , r = 1) ∝ p ( r = 1 | x , u ) p ( u | x ) .

Original: The ﬁrst term is given by our reward model p θ , and the second term is given by the default interface.
Revised: The ﬁrst term is given by our reward model p θ , and the second term is given by the default interface.

Original: Combining these, we get the policy
Revised: Combining these, we get the policy

Original: [Equation]
Revised: [Equation]

Original: This decomposition of the policy improves the initial performance of our interface at the onset of online training, and guides exploration for training the reward model.
Revised: This decomposition of the policy improves the initial performance of our interface at the onset of online training, and guides exploration for training the reward model.

Original: It also provides a framework for incorporating a language model into our interface, as described in Section 4.3.
Revised: It also provides a framework for incorporating a language model into our interface, as described in Section 4.3.

Original: Algorithm
Revised: Algorithm

Original: Require ¯ π, θ init (cid:46) default interface, pretrained reward model parameters while true do x ∼ p user ( x ) (cid:46) user gives input u ∼ π ( u | x ) ∝ p θ ( r = 1 | x , u )¯ π ( u | x ) (cid:46) interface performs action r ← 0 if user backspaces else(cid:46) infer reward from implicit feedback
Revised: Require ¯ π, θ init (cid:46) default interface, pretrained reward model parameters while true do x ∼ p user ( x ) (cid:46) user gives input u ∼ π ( u | x ) ∝ p θ ( r = 1 | x , u )¯ π ( u | x ) (cid:46) interface performs action r ← 0 if user backspaces else(cid:46) infer reward from user feedback

Original: D ← D ∪ { ( x , u , r ) } (cid:46) store online input-action-reward data θ ← θ init + ∇ θ (cid:80) ( x , u ,r ) ∈D log ( p θ ( r | x , u ))
Revised: D ← D ∪ { ( x , u , r ) } (cid:46) store online input-action-reward data θ ← θ init + ∇ θ (cid:80) ( x , u ,r ) ∈D log ( p θ ( r | x , u ))

Original: (cid:46) train reward model to convergence w/SGD
Revised: (cid:46) train reward model to convergence w/SGD

Original: Our x-to-text (XT2) method is summarized in Algorithm 1.
Revised: Our x-to-text (X2T) method is summarized in Algorithm 1.

Original: In the beginning, we assume the user has already been operating the default interface ¯ π for some time.
Revised: In the beginning, we assume the user has already been operating the default interface ¯ π for some time.

Original: In doing so, they generate an ‘ofﬂine’ dataset that we use to train the initial reward model parameters θ init .
Revised: In doing so, they generate an ‘ofﬂine’ dataset that we use to train the initial reward model parameters θ init .

Original: When the user starts using XT2, our interface π already improves upon the default interface ¯ π by combining the default interface with the initial reward model via Equation 2.
Revised: When the user starts using X2T, our interface π already improves upon the default interface ¯ π by combining the default interface with the initial reward model via Equation 2.

Original: As the user continues operating our interface, the resulting online data is used to maintain or improve the accuracy of the reward model.
Revised: As the user continues operating our interface, the resulting online data is used to maintain or improve the accuracy of the reward model.

Original: At each timestep, the user provides the interface with input x  .
Revised: At each timestep, the user provides the interface with input x ∼ p user ( x ) .

Revised: Although standard contextual bandit methods assume that the inputs x are i.i.d.

Revised: , we ﬁnd that X2T performs well even when the inputs are correlated due to user adaptation (see Section 4.2) or the constraints of natural language (see Section 4.4).

Original: The interface then uses the policy in Equation 2 to select an action u .
Revised: The interface then uses the policy in Equation 2 to select an action u .

Original: We then re-train the reward model p θ , using stochastic gradient descent to optimize the maximum-likelihood objective in Equation 1.
Revised: We then re-train the reward model p θ , using stochastic gradient descent to optimize the maximum-likelihood objective in Equation 1.

Original: Appendix A.1 discusses the implementation details.
Revised: Appendix A.1 discusses the implementation details.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Prior methods for training interfaces with supervised learning typically collect a dataset of inputaction pairs, then ﬁt a model that predicts actions given inputs.
Revised: Prior methods for training interfaces with supervised learning typically collect a dataset of inputaction pairs, then ﬁt a model that predicts actions given inputs.

Original: These methods tend to either assume access to ground-truth action labels from the user (Anumanchipalli et al., 2019; Wang et al., 2016;
Revised: These methods tend to either assume access to ground-truth action labels from the user (Anumanchipalli et al., 2019; Wang et al., 2016;

Original: Karamcheti et al., 2020), or assume the user always intends to take the optimal action (Gilja et al., 2012; Dangi et al., 2013; 2014; Merel et al., 2015).
Revised: Karamcheti et al., 2020), or assume the user always intends to take the optimal action (Gilja et al., 2012; Dangi et al., 2013; 2014; Merel et al., 2015).

Original: Unfortunately, the user may not always be able to provide ground-truth action labels.
Revised: Unfortunately, the user may not always be able to provide ground-truth action labels.

Original: Furthermore, in order for the system to compute optimal actions, the user must be instructed to perform speciﬁc calibration tasks for which the optimal policy is already known.
Revised: Furthermore, in order for the system to compute optimal actions, the user must be instructed to perform speciﬁc calibration tasks for which the optimal policy is already known.

Original: These calibration tasks may not be representative of the tasks that the user intends to perform.
Revised: These calibration tasks may not be representative of the tasks that the user intends to perform.

Original: This can lead to a distribution mismatch between the inputs that the model is trained on during the calibration phase, and the inputs that the model is evaluated on at test time when the user performs their desired tasks.
Revised: This can lead to a distribution mismatch between the inputs that the model is trained on during the calibration phase, and the inputs that the model is evaluated on at test time when the user performs their desired tasks.

Original: Standard methods address this problem by periodically repeating the calibration process (Willett et al., 2020; Prorokovi ´ c et al., 2020), which can be time-consuming, disruptive, and requires assumptions about when and how frequently to re-calibrate.
Revised: Standard methods address this problem by periodically repeating the calibration process (Willett et al., 2020; Prorokovi´c et al., 2020), which can be time-consuming, disruptive, and requires assumptions about when and how frequently to re-calibrate.

Original: XT2 overcomes these issues by continually learning from user feedback on tasks that naturally arise as the user types, rather than imposing separate training and test phases.
Revised: X2T overcomes these issues by continually learning from user feedback on tasks that naturally arise as the user types, rather than imposing separate training and test phases.

Revised: X2T does, however, require access to groundtruth backspace actions.

Original: Extensive prior work on text entry systems (MacKenzie & Tanaka-Ishii, 2010) enables users to type using eye gaze (Ward et al., 2000),
Revised: Extensive prior work on text entry systems (MacKenzie & Tanaka-Ishii, 2010) enables users to type using eye gaze (Ward et al., 2000),

Original: Braille (Oliveira et al., 2011), gestures (Jones et al., 2010), and palm keyboards for wearable displays (Wang et al., 2015).
Revised: Braille (Oliveira et al., 2011), gestures (Jones et al., 2010), and palm keyboards for wearable displays (Wang et al., 2015).

Original: XT2 differs in that it enables the user to type using arbitrary inputs like eye gaze or handwriting, rather than a ﬁxed type of input that restricts the ﬂexibility of the system and must be anticipated in advance by the system designer.
Revised: X2T differs in that it enables the user to type using arbitrary inputs like eye gaze or handwriting, rather than a ﬁxed type of input that restricts the ﬂexibility of the system and must be anticipated in advance by the system designer.

Original: XT2 trains a typing interface through reinforcement learning (RL) with human-in-the-loop feedback instead of an explicit reward function.
Revised: X2T trains a typing interface through reinforcement learning (RL) with human-in-the-loop feedback instead of an explicit reward function.

Original: COACH (MacGlashan et al., 2017; Arumugam et al., 2019),
Revised: COACH (MacGlashan et al., 2017; Arumugam et al., 2019),

Original: TAMER (Knox & Stone, 2009; Warnell et al., 2017), and preference learning (Sadigh et al., 2017;
Revised: TAMER (Knox & Stone, 2009; Warnell et al., 2017), and preference learning (Sadigh et al., 2017;

Original: Christiano et al., 2017) also train an RL agent without access to extrinsic rewards, but require explicit user feedback on the agent’s behavior.
Revised: Christiano et al., 2017) also train an RL agent without access to extrinsic rewards, but require explicit user feedback on the agent’s behavior.

Original: XT2 differs in that it learns from implicit feedback, which requires no additional effort from the user to train the agent.
Revised: X2T differs in that it learns from naturally-occurring feedback, which requires no additional effort from the user to train the agent.

Original: Other prior work trains RL agents from implicit signals, such as electroencephalography (Xu et al., 2020), peripheral pulse measurements (McDuff & Kapoor, 2019), facial expressions (Jaques et al., 2017; Cui et al., 2020), and clicks in web search (Radlinski & Joachims, 2006).
Revised: Other prior work trains RL agents from implicit signals, such as electroencephalography (Xu et al., 2020), peripheral pulse measurements (McDuff & Kapoor, 2019), facial expressions (Jaques et al., 2017; Cui et al., 2020), and clicks in web search (Radlinski & Joachims, 2006).

Original: XT2 differs in that it trains an interface that always conditions on the user’s input when selecting an action, rather than an autonomous agent that ignores user input after the training phase.
Revised: X2T differs in that it trains an interface that always conditions on the user’s input when selecting an action, rather than an autonomous agent that ignores user input after the training phase.

Original: Furthermore, XT2 focuses on the assistive typing domain, where, to the best of our knowledge, backspaces have not yet been used to train an interface through RL.
Revised: Furthermore, X2T focuses on the assistive typing domain, where, to the best of our knowledge, backspaces have not yet been used to train an interface through RL.

Original: Related work on assistive robotic teleoperation interfaces proposes human-in-the-loop RL methods that minimally modify the user’s actions (Pilarski et al., 2011; Broad et al., 2017; Reddy et al., 2018; Schaff & Walter, 2020; Du et al., 2020; Jeon et al., 2020).
Revised: Related work on assistive robotic teleoperation interfaces proposes human-in-the-loop RL methods that minimally modify the user’s actions (Pilarski et al., 2011; Broad et al., 2017; Reddy et al., 2018; Schaff & Walter, 2020; Du et al., 2020; Jeon et al., 2020).

Original: XT2 differs in that it learns to operate on arbitrary types of user inputs, instead of assuming the user provides suboptimal action inputs.
Revised: X2T differs in that it learns to operate on arbitrary types of user inputs, instead of assuming the user provides suboptimal action inputs.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: We seek to answer the following questions: Q1 : Does XT2 improve with use and learn to outperform a non-adaptive interface?
Revised: We seek to answer the following questions: Q1 (Sec. 4.1): Does X2T improve with use and learn to outperform a non-adaptive interface?

Original: Q2 : Does the user adapt to the interface while the interface adapts to the user?
Revised: Q2 (Sec. 4.2): Does the user adapt to the interface while the interface adapts to the user?

Original: Q3 :{+Q
+}
Revised: {+Q
+}

Original:  Does XT2 personalize the interface to different input styles?
Revised: (Sec. 4.3): Does X2T personalize the interface to different input styles?

Original: Q4 : Do ofﬂine pretraining and an informative prior policy accelerate online learning?
Revised: Q4 (Sec. 4.4): Do ofﬂine pretraining and an informative prior policy accelerate online learning?

Original: Q5 : Does online learning improve the interface beyond the initialization provided by ofﬂine pretraining?
Revised: Q5 (Sec. 4.4): Does online learning improve the interface beyond the initialization provided by ofﬂine pretraining?

Original: To answer Q1-2 , we run a user study with 12 participants who use webcam images of their eyes to type via gaze (see Figure 1).
Revised: To answer Q1-2 , we run a user study with 12 participants who use webcam images of their eyes to type via gaze (see Figure 1).

Original: To answer Q3-5 , we conduct an observational study with prerecorded images of handwritten characters drawn by 60 users with a tablet and stylus (see Figure 6 in the appendix).
Revised: To answer Q3-5 , we conduct an observational study with prerecorded images of handwritten characters drawn by 60 users with a tablet and stylus (see Figure 6 in the appendix).

Original: In our experiments, we use default interfaces that are not necessarily the state of the art in gaze tracking or handwriting recognition, but are instead chosen to test the hypotheses in Q1-5 .
Revised: In our experiments, we use default interfaces that are not necessarily the state of the art in gaze tracking or handwriting recognition, but are instead chosen to test the hypotheses in Q1-5 .

Original: Appendix A.1 describes the experiment design and implementation of XT2 in detail.
Revised: Appendix A.1 describes the experiment design and implementation of X2T in detail.

Original: 4. A DAPTING THE I NTERFACE TO THE U SER
Revised: 4. A DAPTING THE I NTERFACE TO THE U SER

Original: In this experiment, we aim to test XT2’s ability to improve over time, relative to a non-adaptive, default interface.
Revised: In this experiment, we aim to test X2T’s ability to improve over time, relative to a non-adaptive, default interface.

Original: To that end, we formulate a gaze-based word selection task in which we display a list of words to the user (see Figure 1), ask them to look at their desired word, record an image from their webcam that captures their eye gaze, and predict their desired word.
Revised: To that end, we formulate a gaze-based word selection task in which we display a list of words to the user (see Figure 1), ask them to look at their desired word, record an image from their webcam that captures their eye gaze, and predict their desired word.

Original: To measure objective performance, we ask the user to type speciﬁc goal sentences.
Revised: To measure objective performance, we ask the user to type speciﬁc goal sentences.

Original: To simplify the experiment, we restrict the action space U
Revised: To simplify the experiment, we restrict the action space U

Original: = { 1 , 2 , 3 , ..., 8 } to the eight buttons on the screen, and always assign the next word in the sentence to one of those eight buttons (see Figure 1).
Revised: = { 1 , 2 , 3 , ..., 8 } to the eight buttons on the screen, and always assign the next word in the sentence to one of those eight buttons (see Figure 1).

Original: We evaluate (1) a default interface that uses iTracker (Krafka et al., 2016) to estimate the user’s 2D gaze position on the screen and select the nearest button, and (2) XT2.
Revised: We evaluate (1) a default interface that uses iTracker (Krafka et al., 2016) to estimate the user’s 2D gaze position on the screen and select the nearest button, and (2) X2T.

Original: We calibrate the default interface once at the beginning of each experimental condition for each user, by asking the user to look at each of the eight buttons one by one, recording 20 eye image samples for each button incycles, and training a 2D gaze position estimator using the iTracker method.
Revised: We calibrate the default interface once at the beginning of each experimental condition for each user, by asking the user to look at each of the eight buttons one by one, recording 20 eye image samples for each button incycles, and training a 2D gaze position estimator using the iTracker method.

Original: After calibration, the default interface stays ﬁxed, and does not adapt to the user within the session; in part, because we do not know in advance which word the user intends to select, so we cannot automatically harvest the necessary paired data of gaze images and targets to re-calibrate the interface.
Revised: After calibration, the default interface stays ﬁxed, and does not adapt to the user within the session; in part, because we do not know in advance which word the user intends to select, so we cannot automatically harvest the necessary paired data of gaze images and targets to re-calibrate the interface.

Original: Instead of periodically interrupting the user to gather new paired data and re-calibrate, XT2 continually adapts to the user throughout the session, following Algorithm 1.
Revised: Instead of periodically interrupting the user to gather new paired data and re-calibrate, X2T continually adapts to the user throughout the session, following Algorithm 1.

Original: Appendix A.1 describes the implementation of the default interface and XT2 in further detail.
Revised: Appendix A.1 describes the implementation of the default interface and X2T in further detail.

Original: We measure the performance of each method using the ground-truth accuracy of action selections.
Revised: We measure the performance of each method using the ground-truth accuracy of action selections.

Original: To access ground-truth actions for calculating accuracy, we instruct the user to try to select a speciﬁed word from the list displayed on their screen.
Revised: To access ground-truth actions for calculating accuracy, we instruct the user to try to select a speciﬁed word from the list displayed on their screen.

Original: Note that this instruction is not an essential component of XT2, and is used purely to evaluate objective performance in this experiment.
Revised: Note that this instruction is not an essential component of X2T, and is used purely to evaluate objective performance in this experiment.

Original: The results in panel (a) of Figure 2 show that at the onset of online learning, both XT2 and the default interface predict the user’s intended action with the same accuracy, but quickly diverge.
Revised: The results in panel (a) of Figure 2 show that at the onset of online learning, both X2T and the default interface predict the user’s intended action with the same accuracy, but quickly diverge.

Original: The default interface’s performance degrades over time, potentially due to distribution shift in the user’s inputs caused by changes in head position, background lighting, and other visual conditions.
Revised: The default interface’s performance degrades over time, potentially due to distribution shift in the user’s inputs caused by changes in head position, background lighting, and other visual conditions.

Original: In contrast, XT2 maintains the interface’s strong initial performance throughout the experiment, by continually updating the reward model to reﬂect changes in the user’s inputs.
Revised: In contrast, X2T maintains the interface’s strong initial performance throughout the experiment, by continually updating the reward model to reﬂect changes in the user’s inputs.

Original: Panel (b) shows that
Revised: Panel (b) shows that

Original: XT2 signiﬁcantly improves the performance of 10 out of 12 participants, and that there are diminishing returns to XT2 when the default interface already performs well.
Revised: X2T signiﬁcantly improves the performance of 10 out of 12 participants, and that there are diminishing returns to X2T when the default interface already performs well.

Original: We ran a one-way repeated measures ANOVA on the action prediction accuracy dependent measure from the default and XT conditions, with the presence of XT2 as a factor, and found that f (1 , 11) = 17 .
Revised: We ran a one-way repeated measures ANOVA on the action prediction accuracy dependent measure from the default and X2T conditions, with the presence of X2T as a factor, and found that f (1 , 11) = 17 .

Original: 23 , p < 0 .
Revised: 23 , p < 0 .

Original: The  subjective evaluations in Table 2 in the appendix corroborate these results: users reported feeling that XT2 selected the words they wanted and improved over time more than the default interface.
Revised: The [Figure] subjective evaluations in Table 2 in the appendix corroborate these results: users reported feeling that X2T selected the words they wanted and improved over time more than the default interface.

Original: Panel (c) qualitatively illustrates how XT2 helps the interface recover from incorrect 2D gaze position estimates: each green ‘x’ shows that even when the default interface estimates a gaze position far from the intended button, which would normally cause an incorrect action prediction, the reward model can adjust the overall action prediction back to the correct button via Equation 2.We also ﬁnd [Figure]
Revised: Panel (c) qualitatively illustrates how X2T helps the interface recover from incorrect 2D gaze position estimates: each green ‘x’ shows that even when the default interface estimates a gaze position far from the intended button, which would normally cause an incorrect action prediction, the reward model can adjust the overall action prediction back to the correct button via Equation 2.

Original:  that XT2 performs well even when the user’simplicit feedback is slightly noisy: the user backspaces mistakes, and does not backspace correct actions, in 98 .
Revised: We also ﬁnd that X2T performs well even when the user’s feedback is slightly noisy: the user backspaces mistakes, and does not backspace correct actions, in 98 .

Original: 6% of their interactions.
Revised: 6% of their interactions.

Original: 4. U SER A DAPTATION TO THE I NTERFACE
Revised: 4. U SER A DAPTATION TO THE I NTERFACE

Original: In the previous experiment, we tested XT2’s ability to adapt the interface to the user.
Revised: In the previous experiment, we tested X2T’s ability to adapt the interface to the user.

Original: Prior work on human-machine co-adaptation (Nikolaidis et al., 2017) and the evolution of communication protocols between humans (Hawkins et al., 2020) suggests that an adaptive interface may not only learn to assist a static user, but even stimulate the user to adapt their input style to the interface.
Revised: Prior work on human-machine co-adaptation (Nikolaidis et al., 2017) and the evolution of communication protocols between humans (Hawkins et al., 2020) suggests that an adaptive interface may not only learn to assist a static user, but even stimulate the user to adapt their input style to the interface.

Original: In this experiment, we investigate whether the user adapts to the gaze-based interface described in Section 4.1.
Revised: In this experiment, we investigate whether the user adapts to the gaze-based interface described in Section 4.1.

Original: To do so, we perform a counterfactual experiment: instead of training and evaluating XT2 on inputs x generated by the user while they were typing with XT2, we train and evaluate XT2 on inputs x generated by the user while they were typing with the default interface.
Revised: To do so, we perform a counterfactual experiment: instead of training and evaluating X2T on inputs x generated by the user while they were typing with X2T, we train and evaluate X2T on inputs x generated by the user while they were typing with the default interface.

Original: During the counterfactual experiment, instead of asking the user for new inputs, we simply replay the old inputs that were intended for the default interface, and automate backspaces.
Revised: During the counterfactual experiment, instead of asking the user for new inputs, we simply replay the old inputs that were intended for the default interface, and automate backspaces.

Original: This enables us to test if the user adapted their inputs to XT2, or if the user provided the same distribution of inputs to both the default interface and XT2.
Revised: This enables us to test if the user adapted their inputs to X2T, or if the user provided the same distribution of inputs to both the default interface and X2T.

Original: The results in Figure 3 suggest that the user does indeed adapt their input style to the interface, and that this user adaptation improves performance.
Revised: The results in Figure 3 suggest that the user does indeed adapt their input style to the interface, and that this user adaptation improves performance.

Original: Comparing XT2’s actual performance (orange curve) to XT2’s counterfactual performance on replayed user inputs that the user originally intended for the default interface (teal curve), we see that XT2 is able to perform much better with inputs intended for XT2 compared to inputs intended for the default interface.
Revised: Comparing X2T’s actual performance (orange curve) to X2T’s counterfactual performance on replayed user inputs that the user originally intended for the default interface (teal curve), we see that X2T is able to perform much better with inputs intended for X2T compared to inputs intended for the default interface.

Original: From this result alone, one might infer that the user provided XT2 with inputs that were more generically predictable than the inputs they provided to the default interface.
Revised: From this result alone, one might infer that the user provided X2T with inputs that were more generically predictable than the inputs they provided to the default interface.

Original: However, by comparing the default interface’s performance on replayed user inputs that the user originally intended for XT2 (gray curve) to XT2’s performance on the same inputs (orange curve), we see that XT2 performs better than the default interface on the same inputs.
Revised: However, by comparing the default interface’s performance on replayed user inputs that the user originally intended for X2T (gray curve) to X2T’s performance on the same inputs (orange curve), we see that X2T performs better than the default interface on the same inputs.

Original: This suggests that the user’s inputs to XT2 are not merely easier to predict, but in fact adapted speciﬁcally to XT2.
Revised: This suggests that the user’s inputs to X2T are not merely easier to predict, but in fact adapted speciﬁcally to X2T.

Original: In other words, XT2 stimulates user co-adaptation to the interface.
Revised: In other words, X2T stimulates user co-adaptation to the interface.

Original: 4. P ERSONALIZING THE I NTERFACE FOR D IFFERENT U SERS
Revised: 4. P ERSONALIZING THE I NTERFACE FOR D IFFERENT U SERS

Original: In this experiment, we demonstrate XT2’s ability to operate on a different type of user input: drawings of characters (see Figure 6 in the appendix); which, for some users, can be easier to provide than direct keystrokes (Willett et al., 2020).
Revised: In this experiment, we demonstrate X2T’s ability to operate on a different type of user input: drawings of characters (see Figure 6 in the appendix); which, for some users, can be easier to provide than direct keystrokes (Willett et al., 2020).

Original: We also investigate to what extent XT2 learns a personalized interface that is uniquely adapted to the individual user.
Revised: We also investigate to what extent X2T learns a personalized interface that is uniquely adapted to the individual user.

Original: To that end, we analyze handwriting samples from 60 users, collected through a tablet and stylus, from the UJI Pen Characters Database (Llorens et al., 2008).
Revised: To that end, we analyze handwriting samples from 60 users, collected through a tablet and stylus, from the UJI Pen Characters Database (Llorens et al., 2008).

Original: Each sample consists of a sequence of 2D positions that traces the trajectory of the user’s pen tip as they draw a known character.
Revised: Each sample consists of a sequence of 2D positions that traces the trajectory of the user’s pen tip as they draw a known character.

Original: We conduct an observational study with this data by sampling goal sentences, replaying a randomly-selected handwriting sample of the user’s desired next character from the goal sentence, and treating each drawing as the user input x ; akin to the replay experiment in Section 4.2.
Revised: We conduct an observational study with this data by sampling goal sentences, replaying a randomly-selected handwriting sample of the user’s desired next character from the goal sentence, and treating each drawing as the user input x ; akin to the replay experiment in Section 4.2.

Original: This observational study is equivalent to an online study, except that it does not permit user co-adaptation to the interface since it replays logged user inputs, and assumes that theimplicit feedback signal is not noisy when automating backspaces.
Revised: This observational study is equivalent to an online study, except that it does not permit user co-adaptation to the interface since it replays logged user inputs, and assumes that the feedback signal is not noisy when automating backspaces.

Original: To test XT2’s robustness to noise and distributional shift in the user’s input, we perturb the replayed pen tip positions by adding Brownian noise (see Figure 6 in the appendix).
Revised: To test X2T’s robustness to noise and distributional shift in the user’s input, we perturb the replayed pen tip positions by adding Brownian noise (see Figure 6 in the appendix).

Original: We evaluate (1) a default interface trained to classify handwritten EMNIST characters (Cohen et al., 2017), and (2) XT2.
Revised: We evaluate (1) a default interface trained to classify handwritten EMNIST characters (Cohen et al., 2017), and (2) X2T.

Original: We intentionally train the default interface on EMNIST images instead of UJI Pen Characters drawings, to model real-world applications with a distribution mismatch between the training data and test data, as discussed in Section3.
Revised: We intentionally train the default interface on EMNIST images instead of UJI Pen Characters drawings, to model real-world applications with a distribution mismatch between the training data and test data, as discussed in Section

Original:  To address the challenge of selecting from 27 possible character actions, we use a language model (Dai et al., 2019) to predict the prior likelihood of an action p LM ( u t | u 0: t − 1 ) given the preceding characters u 0: t − 1 in the user’s text ﬁeld.
Revised: [Table] To address the challenge of selecting frompossible character actions, we use a language model (Dai et al., 2019) to predict the prior likelihood of an action p LM ( u t | u 0: t − 1 ) given the preceding characters u 0: t − 1 in the user’s text ﬁeld.

Original: We use this language model in both the default interface and XT2.
Revised: We use this language model in both the default interface and X2T.

Original: In particular, we set ¯ π ( u | x ) ∝ p φ ( u | x ) p LM ( u | u 0: t − 1 ) , where p φ is the EMNIST image classiﬁer.
Revised: In particular, we set ¯ π ( u | x ) ∝ p φ ( u | x ) p LM ( u | u 0: t − 1 ) , where p φ is the EMNIST image classiﬁer.

Original: As in Section 4.1, the default interface stays ﬁxed throughout the experiment and does not adapt to the user, since re-training the default interface would require interrupting the user to collect paired data [Table] to each individual user.
Revised: As in Section 4.1, the default interface stays ﬁxed throughout the experiment and does not adapt to the user, since re-training the default interface would require interrupting the user to collect paired data.

Original: The results in Figure 4 show that XT2 signiﬁcantly outperforms the default interface (orange vs. gray).
Revised: The results in Figure 4 show that X2T signiﬁcantly outperforms the default interface (orange vs. gray).

Original: We ran a one-way repeated measures
Revised: We ran a one-way repeated measures

Original: ANOVA on the action prediction accuracy dependent measure from the default and XT2 conditions, with the presence of XT2 as a factor, and found that f (1 , 59) = 309 . 95 , p < 0 .
Revised: ANOVA on the action prediction accuracy dependent measure from the default and X2T conditions, with the presence of X2T as a factor, and found that f (1 , 59) = 309 . 95 , p < 0 .

Original: Furthermore, Table 1 shows that XT2 learns an interface that is particularly suited to the user whose data it was trained on: when an interface trained on user i ’s data is evaluated on data from user j (cid:54)
Revised: Furthermore, Table 1 shows that X2T learns an interface that is particularly suited to the user whose data it was trained on: when an interface trained on user i ’s data is evaluated on data from user j (cid:54)

Original: = i instead of user i , performance degrades substantially.
Revised: = i instead of user i , performance degrades substantially.

Original: 4. A BLATION E XPERIMENTS
Revised: A BLATION E XPERIMENTS

Original: In this experiment, we aim to test the importance of three components of XT2: ofﬂine pretraining, an informative prior policy, and online learning.
Revised: In this experiment, we aim to test the importance of three components of X2T: ofﬂine pretraining, an informative prior policy, and online learning.

Original: As discussed in Sections 2.2 and 2.3, to improve the initial performance of XT2 and accelerate online learning, we pretrain on ofﬂine data collected using the default interface, and incorporate prior knowledge from the default interface into the prior policy ¯ π in Equation 2.
Revised: As discussed in Sections 2.2 and 2.3, to improve the initial performance of X2T and accelerate online learning, we pretrain on ofﬂine data collected using the default interface, and incorporate prior knowledge from the default interface into the prior policy ¯ π in Equation 2.

Original: Using the handwriting recognition task from Section 4.3, we conduct ablation experiments in which we drop out each of the three components, one by one, and measure any resulting changes in performance.
Revised: Using the handwriting recognition task from Section 4.3, we conduct ablation experiments in which we drop out each of the three components, one by one, and measure any resulting changes in performance.

Original: In the ﬁrst condition, we test the effect of not performing ofﬂine pretraining, by initializing
Revised: In the ﬁrst condition, we test the effect of not performing ofﬂine pretraining, by initializing

Original: XT2 with random weights θ init .
Revised: X2T with random weights θ init .

Original: In the second condition, we test the effect of not incorporating prior knowledge into our policy, by setting the prior policy ¯ π to be a uniform distribution over actions.
Revised: In the second condition, we test the effect of not incorporating prior knowledge into our policy, by setting the prior policy ¯ π to be a uniform distribution over actions.

Original: In the third condition, we test the effect of not learning online and instead relying solely on ofﬂine pretraining, by freezing the reward model parameters after ofﬂine pretraining and not storing any online data.
Revised: In the third condition, we test the effect of not learning online and instead relying solely on ofﬂine pretraining, by freezing the reward model parameters after ofﬂine pretraining and not storing any online data.

Original: The results in Figure 4 show that ofﬂine pretraining is helpful at the onset of online learning (orange vs. teal), but does not have a substantial effect on performance after some online data has been collected.
Revised: The results in Figure 4 show that ofﬂine pretraining is helpful at the onset of online learning (orange vs. teal), but does not have a substantial effect on performance after some online data has been collected.

Original: This is unsurprising, since leveraging ofﬂine data is only necessary when insufﬁcient online data has been collected.
Revised: This is unsurprising, since leveraging ofﬂine data is only necessary when insufﬁcient online data has been collected.

Original: Using the default interface as a prior policy in Equation 2 is critical for XT2’s performance throughout the experiment (orange vs. red).
Revised: Using the default interface as a prior policy in Equation 2 is critical for X2T’s performance throughout the experiment (orange vs. red).

Original: This is also unsurprising, given that the default interface contains useful prior knowledge about how to interpret user inputs.
Revised: This is also unsurprising, given that the default interface contains useful prior knowledge about how to interpret user inputs.

Original: Online learning is not particularly helpful at the onset of the experiment (orange vs. purple), but has a substantial effect on performance over time.
Revised: Online learning is not particularly helpful at the onset of the experiment (orange vs. purple), but has a substantial effect on performance over time.

Original: This result shows that learning from on-policy data is critical for XT2.
Revised: This result shows that learning from on-policy data is critical for X2T.

Original: In other words, XT learns best when it learns from its own mistakes, rather than the mistakes of the default interface.
Revised: In other words, X2T learns best when it learns from its own mistakes, rather than the mistakes of the default interface.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We propose the x-to-text (XT2) algorithm for continual learning of a communication interface fromonline, implicit user feedback.
Revised: We propose the x-to-text (X2T) algorithm for continual learning of a communication interface from user feedback.

Original: The key idea is to infer a reward signal from user behavior, and train the interface through reinforcement learning.
Revised: The key idea is to infer a reward signal from user behavior, and train the interface through reinforcement learning.

Original: We instantiate this idea in the typing domain by treating backspaces asimplicit signals that the interface performed incorrect actions.
Revised: We instantiate this idea in the typing domain by treating backspaces as signals that the interface performed incorrect actions.

Original: In our online user study on gaze-based word selection, we show that XT2 learns to outperform a non-adaptive interface in under 30 minutes (including ofﬂine data collection), and that the user simultaneously adapts their input style to the interface.
Revised: In our online user study on gaze-based word selection, we show that X2T learns to outperform a non-adaptive interface in under 30 minutes (including ofﬂine data collection), and that the user simultaneously adapts their input style to the interface.

Original: In our observational study on handwritten character recognition, we show that XT2 can personalize the interface to individual users, and that ofﬂine pretraining and incorporating prior knowledge from the default interface are helpful for improving the initial performance of XT2 and accelerating online learning.
Revised: In our observational study on handwritten character recognition, we show that X2T can personalize the interface to individual users, and that ofﬂine pretraining and incorporating prior knowledge from the default interface are helpful for improving the initial performance of X2T and accelerating online learning.

Original: These experiments broadly illustrate how online learning fromimplicit user feedback can be used to train a human-machine interface.
Revised: These experiments broadly illustrate how online learning from user feedback can be used to train a human-machine interface.

Revised: One limitation of this work is that we assume backspaces can be generated independently of X2T.

Revised: This assumption restricts X2T to settings where the user can communicate a binary signal; e.g., through a button press or a sip-and-puff interface.

Revised: One direction for future work is to relax this assumption by training a binary classiﬁer to generate feedback signals from, e.g., facial expressions or brain activity.

Original: Our formulation of the assistance problem as a contextual bandit is limited in that it does not incorporate temporal context into decision-making.
Revised: Our formulation of the assistance problem as a contextual bandit is limited in that it does not incorporate temporal context into decision-making.

Original: One direction for further algorithm development is to extend XT2 to use more general reinforcement learning algorithms (Sutton & Barto, 2018) that can learn sequential assistance policies.
Revised: One direction for further algorithm development is to extend X2T to use more general reinforcement learning algorithms (Sutton & Barto, 2018) that can learn sequential assistance policies.

Original: XT2 is also limited in that the beneﬁt from using XT2 to ﬁne-tune the default interface may decrease as the default interface is improved, as suggested by the diminishing returns in Figure 2.
Revised: X2T is also limited in that the beneﬁt from using X2T to ﬁne-tune the default interface may decrease as the default interface is improved, as suggested by the diminishing returns in Figure 2.

Original: One direction for future empirical work is to test XT2 with state-of-the-art default interfaces.
Revised: One direction for future empirical work is to test X2T with state-of-the-art default interfaces.

Original: Methods like XT2 that learn from their mistakes and stimulate user co-adaptation provide a general mechanism for improving user interfaces; not only for assistive 
Revised: Methods like X2T that learn from their mistakes and stimulate user co-adaptation provide a general mechanism for improving user interfaces; not only for assistive typing, but also for other domains, such as brain-computer interfaces for prosthetic limb control and augmented reality displays for visually-impaired users.

Original: A A PPENDIX A. I MPLEMENTATION D ETAILS
Revised: A A PPENDIX A. I MPLEMENTATION D ETAILS

Original: Stochastic gradient descent.
Revised: Stochastic gradient descent.

Original: We use Adam (Kingma & Ba, 2014) to optimize the maximumlikelihood objective in Equation 1.
Revised: We use Adam (Kingma & Ba, 2014) to optimize the maximumlikelihood objective in Equation 1.

Original: Ofﬂine pretraining.
Revised: Ofﬂine pretraining.

Original: To pretrain XT2 on ofﬂine data, we ﬁrst train the reward model p θ to convergence on the ofﬂine data, and store the learned network weights θ init .
Revised: To pretrain X2T on ofﬂine data, we ﬁrst train the reward model p θ to convergence on the ofﬂine data, and store the learned network weights θ init .

Original: Once a minimum of four online input-action-reward triples have been collected, after every interaction, we re-train the reward model p θ to convergence on the data collected online, using the values θ init to initialize the network weights (e.g., instead of a random initialization).
Revised: Once a minimum of four online input-action-reward triples have been collected, after every interaction, we re-train the reward model p θ to convergence on the data collected online, using the values θ init to initialize the network weights (e.g., instead of a random initialization).

Original: A.1. O NLINE U SER S TUDY : T YPING WITH E YE G AZE
Revised: A.1. O NLINE U SER S TUDY : T YPING WITH E YE G AZE

Original: Eye image features.
Revised: Eye image features.

Original: Instead of operating directly on 224x224 images of the user’s eyes, we use the activations of the last fully-connected layer in the iTracker model (Krafka et al., 2016) as the input x to our reward model p θ ( r | x , u ) .
Revised: Instead of operating directly on 224x224 images of the user’s eyes, we use the activations of the last fully-connected layer in the iTracker model (Krafka et al., 2016) as the input x to our reward model p θ ( r | x , u ) .

Original: When training the reward model p θ , we freeze the iTracker network weights used to generate this 128-dimensional input x .
Revised: When training the reward model p θ , we freeze the iTracker network weights used to generate this 128-dimensional input x .

Original: Reward model architecture and learning.
Revised: Reward model architecture and learning.

Original: We set the learning rate for Adam to 10 − 3 , and batch size to 128.
Revised: We set the learning rate for Adam to 10 − 3 , and batch size to 128.

Original: In our experiments, for every input x , there is exactly one desired action u ∗ that will result in a positive reward r = 1 , while all other actions u (cid:54) =
Revised: In our experiments, for every input x , there is exactly one desired action u ∗ that will result in a positive reward r = 1 , while all other actions u (cid:54) =

Original: u ∗ result in a zero reward r = 0 .
Revised: u ∗ result in a zero reward r = 0 .

Original: In other words, p ( r = 1 | x , u ) = p ( u = u ∗ | x ) .
Revised: In other words, p ( r = 1 | x , u ) = p ( u = u ∗ | x ) .

Original: Hence, we structure the reward model for
Revised: Hence, we structure the reward model for

Original: XT2 as p θ ( r = 1 | x , u ) =
Revised: X2T as p θ ( r = 1 | x , u ) =

Original: f θ ( u | x ) , where f θ is an action classiﬁer.
Revised: f θ ( u | x ) , where f θ is an action classiﬁer.

Original: Furthermore, since we expect the reward model to learn to implicitly estimate the user’s gaze position in order to predict actions, we directly incorporate this inductive bias into the model: we structure the action classiﬁer as f θ ( u | x ) ∝ exp ( −(cid:107) g θ ( x ) − pos( u ) (cid:107) 2 ) , where g θ ( x ) outputs a 2D position of the user’s estimated gaze, and pos( u ) is the known 2D position of the button for action u .
Revised: Furthermore, since we expect the reward model to learn to implicitly estimate the user’s gaze position in order to predict actions, we directly incorporate this inductive bias into the model: we structure the action classiﬁer as f θ ( u | x ) ∝ exp ( −(cid:107) g θ ( x ) − pos( u ) (cid:107) 2 ) , where g θ ( x ) outputs a 2D position of the user’s estimated gaze, and pos( u ) is the known 2D position of the button for action u .

Original: Note that even though g θ ( x ) outputs a 2D position, the parameters θ are still trained on the reward prediction objective in Equation 1 (e.g., instead of a 2D gaze position prediction objective).
Revised: Note that even though g θ ( x ) outputs a 2D position, the parameters θ are still trained on the reward prediction objective in Equation 1 (e.g., instead of a 2D gaze position prediction objective).

Original: We represent g θ using a feedforward neural network with one hidden layer containing 64 units, a dropout layer with a dropout rate of 0.
Revised: We represent g θ using a feedforward neural network with one hidden layer containing 64 units, a dropout layer with a dropout rate of 0.

Original: between the hidden layer and the output layer, and ReLU activations.
Revised: between the hidden layer and the output layer, and ReLU activations.

Original: At each timestep t , we record 10 eye images { x it } 10 i =1 at a sampling rate of 10 Hz, and average our predictions over these 10 inputs.
Revised: At each timestep t , we record 10 eye images { x it } 10 i =1 at a sampling rate of 10 Hz, and average our predictions over these 10 inputs.

Original: Speciﬁcally, for XT2, we set g θ ( x t ) =
Revised: Speciﬁcally, for X2T, we set g θ ( x t ) =

Original: 110 (cid:80) 10 i =1 g θ ( x it ) .
Revised: 110 (cid:80) 10 i =1 g θ ( x it ) .

Original: For the default interface, we average the 2D gaze position estimates across the 10 samples before predicting the action whose button position is nearest to the average gaze position estimate.
Revised: For the default interface, we average the 2D gaze position estimates across the 10 samples before predicting the action whose button position is nearest to the average gaze position estimate.

Original: We initialize θ init with the ofﬂine pretraining scheme described in Section 2.2, using 250 input-action-reward triples collected with the default interface.
Revised: We initialize θ init with the ofﬂine pretraining scheme described in Section 2.2, using 250 input-action-reward triples collected with the default interface.

Original: Experiment design.
Revised: Experiment design.

Original: We recruited 11 male and 1 female participants, with an average age of 21.
Revised: We recruited 11 male and 1 female participants, with an average age of 21.

Original: Each participant was provided with the rules of the task and a short practice period of 20 interactions to familiarize themselves with the system.
Revised: Each participant was provided with the rules of the task and a short practice period of 20 interactions to familiarize themselves with the system.

Original: Each interaction – which consisted of providing an input, observing the interface’s action, and deciding whether or not to backspace – took an average ofseconds.
Revised: Each interaction – which consisted of providing an input, observing the interface’s action, and deciding whether or not to backspace – took an average ofseconds.

Original: Each participant completed three phases of experiments: A, B, and C.
Revised: Each participant completed three phases of experiments: A, B, and C.

Original: In phase A, they operate the default interface for 250 steps, generating an ofﬂine dataset of input-action-reward triples that we use to initialize XT2.
Revised: In phase A, they operate the default interface for 250 steps, generating an ofﬂine dataset of input-action-reward triples that we use to initialize X2T.

Original: In phase B, they operate XT2 for 250 steps.
Revised: In phase B, they operate X2T for 250 steps.

Original: In phase C, they operate the default interface for 250 steps.
Revised: In phase C, they operate the default interface for 250 steps.

Original: To avoid the confounding effects of user learning or fatigue over time, we counterbalance the order of phase B and C: six randomly-selected participants completed phase B before C, and the other six participants completed phase C before B. Phase A is used solely to generate ofﬂine data to initialize XT2 in phase B. One participant’s room lighting changed substantially during phase B.
Revised: To avoid the confounding effects of user learning or fatigue over time, we counterbalance the order of phase B and C: six randomly-selected participants completed phase B before C, and the other six participants completed phase C before B. Phase A is used solely to generate ofﬂine data to initialize X2T in phase B. One participant’s room lighting changed substantially during phase B.

Original: Since their performance during phase A was substantially better than during phase B, we use their phase A data (instead of phase B data) to measure the default interface’s performance on this one participant.
Revised: Since their performance during phase A was substantially better than during phase B, we use their phase A data (instead of phase B data) to measure the default interface’s performance on this one participant.

Original: We sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), following prior work on speech interfaces (Makin et al., 2020).
Revised: We sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), following prior work on speech interfaces (Makin et al., 2020).

Original: We set the same goal sentences for each user in each condition.
Revised: We set the same goal sentences for each user in each condition.

Original: Deterministic policy.
Revised: Deterministic policy.

Original: In our experiments, we ﬁnd that sampling actions u ∼ π ( u | x ) from the stochastic policy π does not substantially improve exploration, and can in fact degrade performance by not always choosing the optimal action.
Revised: In our experiments, we ﬁnd that sampling actions u ∼ π ( u | x ) from the stochastic policy π does not substantially improve exploration, and can in fact degrade performance by not always choosing the optimal action.

Original: This is most likely due to the use of the default interface ¯ π in Equation 2, which already provides an effective exploration mechanism.
Revised: This is most likely due to the use of the default interface ¯ π in Equation 2, which already provides an effective exploration mechanism.

Original: Hence, instead of randomly sampling actions, we deterministically select the highest-likelihood action: u ← arg max u π ( u | x ) .
Revised: Hence, instead of randomly sampling actions, we deterministically select the highest-likelihood action: u ← arg max u π ( u | x ) .

Original: A.1. O BSERVATIONAL S TUDY : T YPING BY D RAWING C HARACTERS
Revised: A.1. O BSERVATIONAL S TUDY : T YPING BY D RAWING C HARACTERS

Original: Perturbing user inputs.
Revised: Perturbing user inputs.

Original: We perturb the character drawings in the UJI Pen Characters Database by decomposing each drawing into a sequence of pen tip velocity vectors, adding Brownian noise to each velocity, then integrating over the perturbed velocities to yield a complete, perturbed drawing (see Figure 6 in the appendix).
Revised: We perturb the character drawings in the UJI Pen Characters Database by decomposing each drawing into a sequence of pen tip velocity vectors, adding Brownian noise to each velocity, then integrating over the perturbed velocities to yield a complete, perturbed drawing (see Figure 6 in the appendix).

Original: We compute the Brownian noise by sampling an independent
Revised: We compute the Brownian noise by sampling an independent

Original: Gaussian noise vector with zero mean and variance of 2 · 10 − 4 at each timestep, and summing over these noise vectors from time 0 to time t to compute the Brownian noise vector for time t .
Revised: Gaussian noise vector with zero mean and variance of 2 · 10 − 4 at each timestep, and summing over these noise vectors from time 0 to time t to compute the Brownian noise vector for time t .

Original: The same Brownian noise vector is applied to all the velocity segments in a given drawing.
Revised: The same Brownian noise vector is applied to all the velocity segments in a given drawing.

Original: Each user input x is a 28x28 image of the user’s complete, perturbed drawing of a given character.
Revised: Each user input x is a 28x28 image of the user’s complete, perturbed drawing of a given character.

Original: There are 27 possible characters in the action space U : 26 lower-case letters, and space (which we represent using the digit 7).
Revised: There are 27 possible characters in the action space U : 26 lower-case letters, and space (which we represent using the digit 7).

Original: Reward model architecture and learning.
Revised: Reward model architecture and learning.

Original: We set the learning rate for Adam to 5 · 10 − 4 , batch size to 128, pretrain on the ofﬂine data for 20 epochs, and sample actions from the stochastic policy described in Equation 2 (instead of the deterministic policy described in Appendix A.1.1).
Revised: We set the learning rate for Adam to 5 · 10 − 4 , batch size to 128, pretrain on the ofﬂine data for 20 epochs, and sample actions from the stochastic policy described in Equation 2 (instead of the deterministic policy described in Appendix A.1.1).

Original: We also limit the size of the replay buffer D in Algorithm 1 to the latest 500 input-action-reward triples.
Revised: We also limit the size of the replay buffer D in Algorithm 1 to the latest 500 input-action-reward triples.

Original: We represent the reward model p θ as a neural network with the following architecture: 28x28 input layer, 32x5x5 convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, 64x5x convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, and a fully-connected output layer.
Revised: We represent the reward model p θ as a neural network with the following architecture: 28x28 input layer, 32x5x5 convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, 64x5x convolutional layer, 2x2 max pool layer, dropout layer with dropout rate of 0.3, and a fully-connected output layer.

Original: We initialize θ init with the ofﬂine pretraining scheme described in Section 2.2, using 1000 input-action-reward triples collected with the default interface.
Revised: We initialize θ init with the ofﬂine pretraining scheme described in Section 2.2, using 1000 input-action-reward triples collected with the default interface.

Original: Experiment design.
Revised: Experiment design.

Original: The UJI Pen Characters Database (Llorens et al., 2008) contains handwriting samples from 60 users.
Revised: The UJI Pen Characters Database (Llorens et al., 2008) contains handwriting samples from 60 users.

Original: For each user, it includes two repetitions of lowercase letters, uppercase letters, and 10 digits, for a total of 1364 samples per user.
Revised: For each user, it includes two repetitions of lowercase letters, uppercase letters, and 10 digits, for a total of 1364 samples per user.

Original: In our observational study, we randomly sample target sentences, and replay user inputs that attempt to type the characters in those target sentences.
Revised: In our observational study, we randomly sample target sentences, and replay user inputs that attempt to type the characters in those target sentences.

Original: In the replays, we automatically backspace incorrect actions; a realistic modeling choice, given that in the online user study in Section 4.1, the user did indeed backspace mistakes, and did not backspace correct actions, in 98 .
Revised: In the replays, we automatically backspace incorrect actions; a realistic modeling choice, given that in the online user study in Section 4.1, the user did indeed backspace mistakes, and did not backspace correct actions, in 98 .

Original: 6% of their interactions.
Revised: 6% of their interactions.

Original: As in the online user study, we run each user’s data through two conditions: default and XT2. Since this is an observational study, we do not need to counterbalance the order of the two conditions.
Revised: As in the online user study, we run each user’s data through two conditions: default and X2T. Since this is an observational study, we do not need to counterbalance the order of the two conditions.

Original: For the personalization experiment in
Revised: For the personalization experiment in

Original: Table 1, we assign a randomly-selected, constant value to the random seed of the Brownian noise for each user in each condition.
Revised: Table 1, we assign a randomly-selected, constant value to the random seed of the Brownian noise for each user in each condition.

Original: This ensures that comparisons between entry ( i, i ) and entries ( i, j (cid:54) = i ) in the 4x4 table are not confounded by differences in the input noise, and are only inﬂuenced by systematic differences in the users’ individual handwriting styles.
Revised: This ensures that comparisons between entry ( i, i ) and entries ( i, j (cid:54) = i ) in the 4x4 table are not confounded by differences in the input noise, and are only inﬂuenced by systematic differences in the users’ individual handwriting styles.

Original: As in the online user study, we sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), and set the same goal sentences for each user in each condition.
Revised: As in the online user study, we sample goal sentences from the MOCHA-TIMIT database (Wrench, 1999), and set the same goal sentences for each user in each condition.

Original: Language model.
Revised: Language model.

Original: We use the Transformer-XL language model (Dai et al., 2019) – speciﬁcally, the word-level version that is pretrained on the One Billion Word dataset (Chelba et al., 2013) – to compute the prior likelihood p LM ( u t | u 0: t − 1 ) .
Revised: We use the Transformer-XL language model (Dai et al., 2019) – speciﬁcally, the word-level version that is pretrained on the One Billion Word dataset (Chelba et al., 2013) – to compute the prior likelihood p LM ( u t | u 0: t − 1 ) .

Original: To compute character-level likelihoods, we marginalize over the 60000 words in the language model’s vocabulary that occur the most frequently in the
Revised: To compute character-level likelihoods, we marginalize over the 60000 words in the language model’s vocabulary that occur the most frequently in the

Original: One Billion Word training corpus, not including words with punctuation and converting upper-case to lower-case characters.
Revised: One Billion Word training corpus, not including words with punctuation and converting upper-case to lower-case characters.

Original: We feed the previously-typed characters in the current sentence (i.e., not including the previous sentences) as context to the language model.
Revised: We feed the previously-typed characters in the current sentence (i.e., not including the previous sentences) as context to the language model.

Original: [Table]
Revised: [Table]

Original: The system selectedthe words I wanted
Revised: system selected

Original: [Table]
Revised: [Table]

Original: improved

Original: [Table] using the system [Table] The system did not select the
Revised: words

Original: [Table]
Revised: [Table]

Original: wanted
Revised: wanted

Original: [Table]
Revised: [Table]

Original: mistake
Revised: mistake

Original: [Table]
Revised: [Table]

Revised: [Table]