First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization
Anonymous Author(s)
AfﬁliationAddress
Abstract
How can we train an assistive human-machine interface (e.g., an electromyographybased limb prosthesis) to translate a user’s raw command signals into the actionsof a robot or computer when there is no prior mapping, we cannot ask the userfor supervision in the form of action labels or reward feedback, and we do nothave prior knowledge of the tasks the user is trying to accomplish?
The key ideain this paper is that, regardless of the task, when an interface is more intuitive,the user’s commands are less noisy.
We formalize this idea as a completelyunsupervised objective for optimizing interfaces: the mutual information betweenthe user’s command signals and the induced state transitions in the environment.
Toevaluate whether this mutual information score can distinguish between effectiveand ineffective interfaces, we conduct a large-scale observational study on 540Kexamples of users operating various keyboard and eye gaze interfaces for typing,controlling simulated robots, and playing video games.
The results show thatour mutual information scores are predictive of the ground-truth task completionmetrics in a variety of domains, with an average Spearman’s rank correlation of⇢ = 0 .
In addition to ofﬂine evaluation of existing interfaces, we use ourunsupervised objective to learn an interface from scratch: we randomly initializethe interface, have the user attempt to perform their desired tasks using the interface,measure the mutual information score, and update the interface to maximize mutualinformation through reinforcement learning.
We evaluate our method througha small-scale user study with 12 participants who perform a 2D cursor controltask using a perturbed mouse, and an experiment with one expert user playing the
Lunar Lander game using hand gestures captured by a webcam.
The results showthat we can learn an interface from scratch, without any user supervision or priorknowledge of tasks, with less than 30 minutes of human-in-the-loop training.
Introduction
Imagine communicating with an intelligent extraterrestrial for the ﬁrst time.
They are trying to get usto do something (e.g., synthesize a novel molecule and build a space elevator out of it), but we do notknow what that task is, and we cannot understand their language.
They might speak, write, wavetheir limbs at us in a video, or send us messages through some other modality.
They might even bewatching how we act on their messages, and adapting the content of their messages to our behavior.
How do we translate their messages
into
the actions
they want us to take
To make progress on this problem in the absence of (known) alien signals, we study the relatedproblem of helping humans communicate their intent to machines via arbitrary command signals;for example, controlling a robotic arm through a brain-computer interface [1].
When designinghuman-machine interfaces, we typically either (a) use human ingenuity to design an interface that is a s d f
[Figure]
is unintuitive intuitive for a human to operate; (b) get direct supervision on how commands should map to actions,so we can train the interface through supervised learning; or (c) get reward feedback on the system’sperformance, so that we can improve the interface through reinforcement learning (see Sec.
4 for priorwork).
Unfortunately, none of this is possible with alien users: we do not know what is intuitive forthem; we cannot ask them to perform speciﬁc tasks so that we can collect labeled data for supervisedlearning; we do not know what tasks they are trying to accomplish; and we cannot understand anyfeedback they might give on the quality of our actions.
Existing approaches are also inconvenient forhuman users who would like to perform highly-personalized tasks using non-traditional interfaces,such as controlling a vehicle through a microphone 1 , but do not want to be interrupted and asked forsupervision or feedback.
Without any source of supervision or prior knowledge about the user’s desired tasks, it seemsimpossible to design a good interface.
Recent work proposes optimizing for the user’s ability tocontrol their environment [3].
This is useful, but there tend to be many different interfaces that enablethe user to inﬂuence the environment, and most of them are unintuitive to operate (e.g., comparethe QWERTY keyboard to a keyboard with randomly-permuted keybindings).
How do we capturewhat is intuitive, not only when the user is a human, but also when the user is an alien whose priorover ‘natural’ interfaces may not overlap with those of a human?
The key idea (and assumption) inthis paper: the more intuitive the interface, the less noisy the user’s commands (Fig. 1b).
Thus, agood interface should not only maximize the user’s ability to inﬂuence the environment, but alsominimize the entropy of the user’s commands.
We show that this corresponds to maximizing themutual information between the user’s command and the induced state transition (Sec. 2.1).
Of course, we cannot optimize this mutual information objective without interacting with the user.
What we thus propose is a co-adaptation process, wherein we start with a random interface, theuser learns about it as they use it, we measure the mutual information, and update the interface tomaximize the mutual information through reinforcement learning.
As the interface improves, theuser’s ability to complete their desired tasks using the interface also increases.
We call this methodthe mutual information-maximizing interface (MIMI).
more convenient.
This is why I will sometimes make unfamiliar sounds with my voice – I am controlling the vehicle’s systems...
Ng makes a yapping sound, and the van pulls out onto the frontage road...
Ng laughs sharply, like distant ack-ack, and the van almost swerves off the road... Ng sings a little song. A robot arm unfolds itself from the ceiling of the van, crisply yanks the vial from her hand, swings it around, and holds it in front of a video camera set into the dashboard.” Our primary contribution is an unsupervised mutual information objective for evaluating interfacesthat can be used even when the ground-truth reward function for the user
’s desired task is unknown.
To evaluate whether this unsupervised objective can be used to distinguish between effective andineffective interfaces, we analyze ofﬂine datasets of users operating various interfaces in ﬁve differentdomains from prior work, and measure the correlation between the ground-truth rewards for each taskand our unsupervised mutual information scores (Sec. 5.1).
The results show that, in each domain, theinterface with the largest mutual information score also has the largest ground-truth reward (Fig.
5 inthe appendix).
We also contribute the MIMI algorithm for training an interface to understand humancommands from scratch, without prior knowledge of the user’s desired tasks or explicit feedback fromthe user.
We evaluate MIMI through an online user study with 12 participants who use a perturbedmouse to perform a 2D cursor control task (Sec. 5.2).
The results show that, in under 30 minutes ofhuman-in-the-loop training, MIMI learns an interface that is intuitive for users to operate and enablesusers to reach their goals more quickly than a random interface.
We also showcase MIMI’s ability toscale to more complex tasks and command modalities through a study with one expert user (the ﬁrstauthor) who uses hand gestures to play the Lunar Lander game (Sec. 5.3).
An Unsupervised Mutual Information Objective for Evaluating Interfaces
In our problem setting, users cannot directly act in the environment, and must rely on an interface totake actions for them (e.g., due to a motor impairment, or because they are remotely operating spacerobots, or because they are aliens communicating with humans over long distances).
The interfacedoes not have direct access to the user’s desired task, but can observe command signals that the userprovides to the interface (e.g., through a webcam or microphone).
We formulate the command-toaction translation problem as a contextual Markov decision process (CMDP) [4].
Each observationis decomposed into two variables: the environment state s t (e.g., the position and orientation of arobot arm) and the user’s command signal x t (e.g., a webcam image of their eye gaze).
The interface⇡ ( a t | s t , x t ) takes an action a t given the state s t and command x t .
We do not assume access to theuser’s desired task (which represents the ‘context’ in the CMDP), the space of desired tasks, theground-truth reward function R ( s t , a t ) , or the state transition dynamics.
We approach this problemby deﬁning a surrogate reward function that correlates positively with the ground-truth reward.
Wedeﬁne this surrogate to be the information rate of the interface, which we formalize as the mutualinformation between the user’s command x t and the induced state transition ( s t , s t +1 ) .
Fig.
1aoutlines our method.
2.1 Characterizing the Information Rate of the Interface
Viewing our system through the lens of information theory [5, 6], the user interacts with the environment through a noisy communication channel that mediates perception (blue in Fig.
1) andcontrol (orange in Fig. 1).
When the environment sends a message s t to the user, the user’s internaldecision-making process noisily converts the message s t into a command x t .
When the user sends amessage x t to the interface, the interface converts the message x t into an action a t ⇠ ⇡ ( a t | s t , x t ) ,and the environment converts the action a t into a next state s t +1 ⇠ p ( s t +1 | s t , a t ) following unknowndynamics.
In this section, we derive the information rate of this channel, and show how maximizingit enables the user to perform their desired tasks.
The amount of information that is transmitted from the user to the environment can be characterizedas the conditional mutual information I ( x t , s t +1 | s t ) (orange in Fig. 1).
Maximizing this term enablesthe user to inﬂuence the next state s t +1 by modulating their command x t .
However, there can bemany different interfaces that achieve an equally-high value of I ( x t , s t +1 | s t ) .
For example, considertyping on a QWERTY keyboard.
Randomly permuting the keybindings would preserve the valueof I ( x t , s t +1 | s t ) , since there would still be a one-to-one mapping between physical keys and typedcharacters, but the resulting layout would probably not be as intuitive to use as QWERTY.
In order tomaximize the intuitiveness of the interface, we need to consider the amount of information that getstransmitted in the other direction: from the environment to the user.
The amount of information that gets transmitted from the environment to the user can be characterizedas the mutual information
I ( s t , x t ) (blue in Fig. 1).
The main assumption in our method is thatmaximizing I ( s t , x t ) , which can be rewritten as H ( x t )  H ( x t
| s t ) , leads to an intuitive interface.
The idea behind penalizing the conditional entropy
H ( x t | s t ) is that, when the user operates an unintuitive interface, they provide noisier commands x t given the current state s t (similar to themaximum causal entropy model of noisily-rational user behavior [7]).
This is because an unintuitiveinterface can confuse users, cause them to provide random commands when they are under timepressure to perform tasks, or cause them to keep trying different commands to see which ones willtrigger their desired actions.
On the other hand, an intuitive interface makes it easy for the user todetermine which command will induce their desired next state, leading to more deterministic userbehavior (Fig. 1b).
Penalizing the conditional entropy H ( x t | s t ) alone is not enough, however.
Forexample, consider an interface that minimizes the conditional entropy to zero by frustrating the userand causing them to always provide the same command, no matter the state or desired task (e.g., aconstant ‘no op’ signal).
Hence, we must also maximize the marginal entropy H ( x t ) , which leads toan interface that encourages the user to provide a wide distribution of commands overall (e.g., byenabling the user to visit a variety of states).
Summing the two mutual information terms, we get the total amount of information that the usersends and receives, which simpliﬁes to the mutual information between the user’s command x t andthe induced state transition ( s t , s t [Equation]
[Equation]
2.2 Estimating Mutual Information Rewards
Computing the mutual information reward R ( ⇡ ) exactly would require taking an expectation withrespect to the state transition dynamics of the environment as well as the user’s policy for givingcommands, both of which are unknown.
Hence, we estimate the mutual information by collectingsamples ( s t , x t , s t +1 ) of the user operating the interface ⇡ .
Alg.
1 outlines this sample-basedevaluation of R ( ⇡ ) .
To estimate I ( x t , ( s t , s t +1 )) from the samples in a dataset D in our experiments, we use the TUBAestimator [8, 9].
In particular, we ﬁt a “statistics network” T  and variational parameters a tomaximize the mutual information lower bound,
[Equation]
where ¯ x is sampled uniformly at random from the dataset D .
Intuitively, T  is a discriminator thatlearns to distinguish between realistic pairs ( x t , ( s t , s t +1 )) and unrealistic pairs (¯ x , ( s t , s t +1 )) .
Inour experiments, we represent T  and a each as a feedforward neural network that outputs a scalar.
Two practical issues emerge when applying the standard TUBA estimator to our problem setting.
First, we can only collect a small amount of data D to evaluate a given interface during our userstudies, which can lead to high-variance estimates of the mutual information.
To address this issue,we adopt the following approach from prior work on data-efﬁcient mutual information estimation[10]: instead of using the ﬁnal training loss I TUBA as our mutual information estimate, we use thevalidation loss (line 9 in Alg. 1).
Second, we must avoid substantial wall-clock delays to the userwhile we optimize the mutual information lower bound in line 8 of Alg. 1.
Typically, in order toaccurately estimate the mutual information, one must optimize the lower bound to convergence.
This can take many steps of stochastic gradient descent, and blocks the user from proceeding to the nextepisode.
In our setting, however, we ﬁnd that optimizing to convergence is unnecessary.
The key ideais that, since we ultimately use the mutual information estimates as rewards to compare interfaces,only the relative values of the estimates matter to us.
Hence, we only take 1K gradient steps (with asmall batch size of 64) to ﬁt the estimator in all our experiments.
We ﬁnd that, even after only 1Ksteps, the mutual information estimates already tend to correlate positively with the ground-truth taskrewards (Fig. 2).
Learning an Interface through Mutual Information Maximization
In principle, one could represent the interface ⇡ in any manner (e.g., as a deep neural network) anduse any reinforcement learning algorithm to optimize MIMI-E VALUATE ( ⇡ ) .
In our experiments, weparameterize the interface as a linear model ⇡ ✓ , where ✓ are the weights and biases, and maximize
MIMI-E VALUATE ( ⇡ ) using an off-the-shelf Bayesian optimization algorithm ( scikit-optimize )[11] based on Gaussian process regression [12].
Initially, we randomly sample the interface parameters ✓ 0 , have the user attempt to perform their desired tasks using the interface ⇡ ✓ 0 for 10 episodes, andcompute the mutual information reward MIMI-E VALUATE ( ⇡ ✓ 0 ).
Given all pairs of interface parameters and corresponding mutual information rewards { ( ✓ i , MIMI-E VALUATE ( ⇡ ✓ i )) } h i =0 evaluated sofar, we ﬁt a Gaussian process regression model that predicts the mutual information reward giventhe interface parameters, and select the next interface ✓ h +1 by optimizing one of three acquisitionfunctions—expected improvement, lower conﬁdence bound, or probability of improvement—chosenuniformly at random.
Details in the appendix.
Related Work
Developing an assistive human-machine interface from scratch requires translating the unknownlanguage of commands into actions.
Translating unknown languages is a broad area of research,encompassing linguistics [13], cryptanalysis [14], and deep neural network interpretability [15, 16,17].
Prior work on multi-agent reinforcement learning proposes inductive biases that use mutualinformation maximization to promote positive signalling and listening in cooperative tasks [18, 19].
This prior work studies the emergence of communication in autonomous agents, whereas MIMIis intended for the unsupervised learning of co-adaptive user interfaces.
Our problem setting alsodiffers in that we cannot train the speaker agent (i.e., the human), so we must craft an objective forthe listener agent (i.e., the interface) that indirectly promotes informative signaling from the speaker.
The literature on adaptive interfaces spans multiple ﬁelds, including brain-computer interfaces[20, 21, 22, 23, 24, 25], natural language interfaces [26, 27], and robotic teleoperation interfaces[28, 29, 30, 31, 3, 32].
In contrast to this prior work, MIMI does not require knowledge of the user’sdesired tasks or direct supervision.
Prior work has trained reinforcement learning agents to optimize implicit user feedback contained in
EEG signals [33], peripheral pulse measurements [34], facial expressions [35, 36], and clicks [37].
These approaches typically assume access to a dataset of user observations (e.g., facial expressions)and explicit rewards (e.g., labels that indicate positive or negative emotion) in order to train a rewardmodel that infers rewards from user observations.
Other prior work on human-in-the-loop RL assumesdirect access to explicit user feedback during the RL phase [38, 28, 39, 40, 30].
MIMI, in contrast,does not require access to explicit rewards at any time.
Mutual information maximization has previously been used to acquire diverse, complex behaviorsthrough reinforcement learning in the absence of extrinsic rewards [41, 42, 43, 44, 45, 46].
Theseprior methods aim to train an autonomous agent, whereas MIMI aims to train a user interface.
Our mutual information objective in Eqn.
1 is related to empowerment [47], which measures thechannel capacity between an agent’s actions and the environment state.
Empowerment has previouslybeen used as an intrinsic motivation for autonomous reinforcement learning agents [48], and anauxiliary objective for assistive agents that preserve a user’s ability to reach various states [3].
Ourobjective differs in that it measures not only the user’s ability to inﬂuence state transitions, but alsothe intuitiveness of the interface.
Experimental Evaluation
Our experiments focus on measuring MIMI’s effectiveness at evaluating interfaces, learning interfacesfrom scratch, and scaling interface optimization to complex tasks and interface modalities wherethere is no clear, intuitive solution.
We aim to answer the following questions.
Q1 (Sec. 5.1): Can
MIMI’s mutual information score distinguish between effective and ineffective interfaces for a varietyof users, interface modalities, environments, and tasks?
Q2 (Sec. 5.2):
Can we learn an interfacefrom scratch in a completely unsupervised manner by maximizing mutual information rewards with
MIMI?
Q
(Sec. 5.3): Does unsupervised human-in-the-loop reinforcement learning scale to morecomplex interface modalities and tasks?
To answer Q1 , we conduct an observational study on ﬁvedatasets from prior work.
To answer Q2 , we conduct a user study with 12 participants who use aperturbed mouse to perform a 2D cursor control task.
To answer Q3 , we study how MIMI can beused to enable an expert user (the ﬁrst author) to play the Lunar Lander game through hand gesturescaptured by a webcam.
5.1 Ofﬂine Evaluation of Existing Interfaces
In this experiment, we aim to evaluate whether the mutual information score in MIMI can distinguishbetween effective and ineffective interfaces for a wide variety of users, interface modalities, environments, and tasks ( Q1 ).
We take data from prior work on adaptive interfaces in which the ground-truthrewards were measured, and check whether MIMI’s unsupervised evaluation of those interfacescorrelates with the true reward that users received when performing tasks via those interfaces.
Weexamine data from four prior works: X2T [49], ASHA [50], shared autonomy via deep reinforcementlearning (SAvDRL) [51], and internal-to-real dynamics transfer (I2R) [52].
In each of these priorworks, users were asked to operate at least two different interfaces: a non-adaptive baseline interface,and an adaptive interface that learns to assist users over time.
In the X2T experiments, participantsused their eye gaze (a 128-dimensional command signal x t from their webcam) to select 1 of 8buttons on a screen in order to type a phrase.
In the ASHA experiments, participants used their eyegaze to control a 7-DoF simulated robotic arm to either ﬂip a light switch or reach for a bottle.
In the SAvDRL and I2R experiments, participants used the directional keys on a keyboard to play the Lunar
Lander game [53].
The appendix describes each domain in detail.
The data from these prior userstudies totals over 540K timesteps.
For each domain, each user, and each experimental condition (e.g., non-adaptive vs. adaptiveinterface), we measure our mutual information score by averaging over 10 different random seedsused to ﬁt the estimator in Eqn. 2.
The ground-truth reward function for each domain is deﬁned in theappendix.
The results in Fig.
2 show that, in 4 out of 5 domains, MIMI’s mutual information score foran interface is predictive of the ground-truth task reward a user is able to obtain using that interface,with an average Spearman’s rank correlation coefﬁcient of ⇢ = 0 .
Furthermore, in each domain,the interface with the largest mutual information score also has the largest ground-truth reward (Fig.5 in the appendix).
In analyzing the data from the SAvDRL experiments, we initially encountered an unexpected result:a strong negative correlation between our mutual information scores and the ground-truth rewards(Fig. 2d).
The reason is that, in the SAvDRL experiments, the assistant tends to help the user play the
Lunar Lander game by preventing them from crashing, which necessarily involves ignoring a largefraction of the user’s commands.
This leads to a low mutual information between the user’s commandx t and the one-step state transition ( s t , s t +1 ) .
However, even though the assistant appears to reducethe user’s inﬂuence over the next state, it actually increases the user’s inﬂuence over later states in theepisode by preventing the user from crashing immediately.
This leads us to propose a generalizedmutual information objective, I ( x t , ( s t , s t +  )) , in which the time offset  is a hyperparameter thancan be set to a value other than  = 1 .
Fig.
2d shows how modifying the time offset  in the generalized mutual information objective
I ( x t , ( s t , s t +  )) affects the correlation between the mutual information scores and the ground-truthrewards in the SAvDRL experiments: for low values of  (including the default value of  = 1 ),there is a negative correlation, whereas for sufﬁciently high values of  (i.e., the maximum episodelength of 10 3 steps), there is a positive correlation.
In Figures 2e and 2f, we set  to the maximumepisode length, so that the MIMI objective I ( x t , ( s t , s T )) always measures mutual information withrespect to the ﬁnal state s T of each episode.
In general, choosing the value of  requires prior [Figure] knowledge of the timescale of the user’s desired inﬂuence over the system (i.e., does the user’scommand indicate what should happen at the next timestep, or what should happen by the end of theepisode?).
This is the primary limitation on the generality of our method.
5.2 User Study: 2D Cursor Control with Perturbed Mouse
In the previous experiment, we showed that MIMI’s mutual information score correlates positivelywith the ground-truth task rewards in a variety of ofﬂine datasets ( Q1 ).
However, it is not clear fromthis result alone that maximizing mutual information rewards will yield an interface that also performswell in terms of the ground-truth task rewards ( Q2 ).
To answer Q2 , we conduct a small-scale userstudy with 12 participants who use a perturbed mouse to perform a simple 2D cursor control task.
The goal of the task is to drive a cursor from the center of the screen to a randomly-sampled targetposition (Fig. 3e).
The ground-truth reward is the average negative distance to target throughout theepisode.
The user’s 2D mouse position command is used to control the 2D velocity of the cursor(details in the appendix).
The interface ⇡ ✓ applies a rotation to the 2D mouse position to get the2D velocity (i.e., ✓ consists of a single parameter: the rotation angle).
The initial interface ⇡ ✓ 0 is arandom rotation, and is usually difﬁcult for the user to operate (see scattered trajectories in Fig. 3f).
Users struggle to anticipate and adjust for the rotation, which leads to them initially moving awayfrom the target, oscillating around the target as they approach it because they keep steering in thewrong direction, or getting stuck on the boundaries of the environment because they forget how tosteer away from the boundary.
Each of the 12 participants completed two phases of experiments: A and B.
In phase A, they operate5 interfaces with randomly-sampled parameters ✓ for 10 episodes each.
In phase B, they operate anadaptive interface optimized by MIMI (Sec. 3).
Each interface proposed by MIMI is evaluated for 10episodes, and we halt the experiment after at most 30 interfaces have been evaluated, yielding a totalof at most 300 episodes per participant.
To avoid the confounding effect of overall user improvementor fatigue over time, we counterbalance the order of phases A and B. Details in the appendix.
The results in Fig.
3 show that, even though MIMI starts from scratch with a random interface andonly aims to maximize mutual information rewards (blue), it enables users to achieve substantially
[Figure]
higher ground-truth rewards (orange) than the random baseline interfaces (gray).
Fig.
3d shows that
MIMI tends to converge to one of two interfaces: moving the cursor in the same direction as themouse, or rotating the mouse direction 180 degrees (which tends to be easy for users to understandand invert; see the appendix).
5.3 Lunar Lander with Hand Gestures
In the previous experiment, we showed that MIMI is capable of learning an interface from scratchthrough human-in-the-loop RL with the mutual information objective, albeit on a simple task andinterface modality (2D cursor control via 2D mouse commands).
In this section, we demonstratethat this result extends to a more complex task and interface modality ( Q3 ): an expert user (the ﬁrstauthor) playing the Lunar Lander game [53] using hand gestures.
The ground-truth reward penalizescrashing, and gives a bonus for landing between the ﬂags.
The user’s hand pose command is a 4Dsignal that consists of the distances from the thumb tip to each of the four other ﬁngertips.
The Lunar Lander game has a 2D continuous action space that controls the main engine and lateral thrusters(details in the appendix).
The interface ⇡ ✓ is a linear transformation from the 4D hand pose commandto the 2D thruster action space (i.e., ✓ consists of 8 parameters).
The initial interface ⇡ ✓ 0 is a randomlinear function, and is usually difﬁcult for the user to operate (e.g., because it requires making difﬁculthand gestures, or never ﬁres the lateral thrusters).
Lunar Lander is a challenging game, even whenplayed with a joystick or keyboard.
What makes it more challenging in this setting is that, unlike 2Dcursor control, where there is an obvious prior mapping that moves the cursor in the direction of themouse without any rotation, there is no obvious mapping from hand gestures to thruster actions thatthe user would have in mind.
The results in Fig.
4 show that, after 200 episodes of online training, MIMI learns an interfacethat enables the expert user to maneuver the lander, usually avoid crashing, and occasionally landsuccessfully.
The interface enables the user to ﬁre the right thruster by touching their index ﬁngerto their thumb, ﬁre the left thruster and main engine by touching their ring ﬁnger to their thumb,and ﬁre the main engine by slightly curling all ﬁngers.
The 8 interface parameter values ✓ learnedthrough MIMI would have been difﬁcult to determine in advance, given uncertainty about the range
[Figure]
of comfortable hand pose commands, and the non-obvious intuitiveness of using individual ﬁngersfor lateral thrusters vs. all ﬁngers for the main engine.
Interestingly, the ﬁnal interface does not enablethe user to ﬁre the left thruster in isolation (i.e., all gestures that ﬁre the left thruster also ﬁre the mainengine simultaneously).
This turns out to be okay, since the user wants to descend, and alternatingbetween ﬁring the left thruster and main engine together then ﬁring the right thruster without themain engine is a reasonable strategy for descending without tilting.
This would have been difﬁcult toanticipate in advance, but nevertheless emerged through the co-adaptation process between the userand the interface.
Discussion
We presented a proof of concept that, through unsupervised human-in-the-loop reinforcement learning,we can learn an assistive interface from scratch in less than 30 minutes without any explicit userfeedback or prior knowledge of the user’s desired tasks.
Our experiments show that, for a varietyof users (12 participants in our user study, and 12 in each of the ofﬂine evaluations), commandmodalities (high-dimensional eye gaze and hand gestures, and low-dimensional keyboard commands),environments (typing, simulated robotic control, playing a video game, and cursor control), and tasks(ﬂipping light switches and reaching for bottles with the same simulated robotic arm), MIMI’s mutualinformation objective can be used to rank interfaces without observing ground-truth rewards for theuser’s desired task.
The main limitation of our experiments is that the largest interface only has 8 parameters, due toconstraints on the duration of a user study and the data efﬁciency of the Bayesian optimizationalgorithm we use for RL.
One direction for future work is to instantiate MIMI with a data-efﬁcientdeep RL algorithm that scales to learning high-dimensional policies, such as REDQ [55].
Another direction for future work is evaluating MIMI on real-world systems in which explicit userfeedback is sparse or unavailable, such as brain-computer interfaces for users with total locked-insyndrome.
MIMI could also be helpful for designing creative tools for artists and musicians where the‘task’ or ‘ground-truth reward function’ is inherently difﬁcult to specify, such as a tool that helps usersnavigate the high-dimensional latent space of a generative model of images using physical gesturesand virtual reality [56], or an instrument that provides users with a playable, low-dimensional latentspace of music audio [57].