C OUNTERING L ANGUAGE D RIFT VIA G ROUNDING
Anonymous authors
Paper under double-blind review
A BSTRACT
While reinforcement learning (RL) shows a lot of promise for natural language processing—e.g. when ﬁne-tuning natural language systems for optimizing a certain objective—there has been little investigation into potential language drift : when an external reward is used to train a system, the agents’ communication protocol may easily and radically diverge from natural language.
By re-casting translation as a communication game, we show that language drift indeed happens when pre-trained agents are ﬁne-tuned with policy gradient methods.
We contend that simply adding a “naturalness” constraint to the reward, e.g. by using language model log likelihood, does not fully address the issue, and argue that (perceptual) grounding is required.
That is, while language model constraints impose syntactic conformity, they do not lead to semantic correspondence.
Our experiments show that grounded models give the best communication performance, while retaining
English syntax along with the ability to convey the intended semantics.
I NTRODUCTION
In the summer of 2017, the internet was brieﬂy abuzz with the mistaken viral message that a leading
AI research lab had to “unplug its AI” because it “had gone rogue”.
What had in fact happened was that two chatbots, under certain conditions, had, rather unsurprisingly, started diverging from their
English training data and had instead reverted to their own ungrammatical communication protocol for solving a negotiation task (Lewis et al., 2017).
Instead of saying something like “hats have no value for me” the system would starting saying things like “hat have zero to me to me to me to me”.
As was soon made clear by the parties involved, this sort of language drift is to be expected if we are optimizing for an external reward, for example one based on whether or not two agents successfully accomplish a negotiation.
While language drift is to be expected under external reward, it is natural to ask what we can do to avoid it.
Consider policy gradient methods, for example, and suppose we sample an output sequence from an English language decoder for a given task: sampling a non-grammatical sequence might still be rewarded if we manage to solve the task (e.g., due to some correct words; or because an interlocutor understood us anyway, or guessed correctly), which would quickly move the decoder away (drifting) from English.
If we were able to keep drift in check, we could maximize reward while retaining the “Englishness” of the decoder, with obvious beneﬁts for interpretability and interaction with humans.
That is, while search space size prohibits the direct usage of policy gradient methods for training natural language decoders from scratch, we could prevent pre-trained models from drifting while we optimize for the desired reward using policy gradients.
Thus, the ability to stop policy gradient methods from diverging from natural language enables interesting long-term possibilities for exploration: imagine e.g. ﬁne-tuning a pre-trained language model trained on large amounts of data, call it a “language module”, for a given generation task with limited data.
When training chit-chat dialogue agents, for example, we often want to optimize for some very high-level reward, such as engagingness or consistency, with hardly enough data to learn simple English grammar.
Or consider what might happen when we train agents using self-play to actively use natural language to change the other agent’s (mental) state, rather than having a model passively observe language usage in some corpus or dataset, as usually happens.
In this work, we study the question of language drift.
Drawing inspiration from Lee et al.
(2018), we re-cast translation as a communication game.
Two machine translation (MT) agents—i.e., encoderdecoder models with attention—are tasked with successfully translating source language sequences
[Figure]
to the target language using a third pivot language as an intermediary.
The communication channel (the output of the ﬁrst agent’s decoder, which is fed to the second agent’s encoder as input) is updated via policy gradient methods to optimize for translating into the target language, effectively ﬁne-tuning two separate pre-trained MT models via a pivot language.
While the subject of communication need not be language (e.g., for Lee et al.
(2018), agents learn to translate by communicating about images), three-way translation via pivot is an excellent way for studying the current problem: we can check exactly to what extent the communicated sequence corresponds to both the intended meaning, as well as to the gold standard sequence.
We can think of this setup as an agent aiming to communicate its state to another agent via some protocol—yet in this case, the mental states and intermediary communication protocol are completely interpretable.
In what follows, we show that language drift happens, and quite dramatically so, when ﬁne-tuning using policy gradients.
We then show that the most intuitive way of solving this problem—adding an “Englishness” constraint, such as the log-probability assigned by a language model, to the reward function—does not in fact lead to the desired consequences.
Indeed, there is nothing preventing such models from learning to translate “Two giraffes standing next to a white truck in the savanna” from French to German via “Democracy is a political system” as the English intermediary.
Hence, we contend that what is missing is grounding : while language model constraints impose syntactic conformity, they do not lead to semantic correspondence.
Humans don’t invent unique idiolects for every individual interlocutor, exactly because they are grounded: we share strong priors, social and behavioral norms, and a common sensorimotor experience of our physical environment.
Thus, “not going rogue” means not only sticking to the prescribed language, but more importantly preserving meaning, which means staying grounded.
Our experiments show that ﬁne-tuning the communication channel with visual grounding leads to the highest communication performance (Fr → En → De) as well as the best retention of original syntax and intended semantics.
Our token frequency analysis corroborates our hypothesis, and shows that grounding is key for preserving the token frequency distribution of the pivot language (English).
P RIOR W ORK
Our work is inspired by recent work in protocols or languages that emerge from multi-agent interaction (Lazaridou et al., 2017; Lee et al., 2018;
Andreas et al., 2017; Evtimova et al., 2018; Kottur et al., 2017; Havrylov & Titov, 2017; Mordatch & Abbeel, 2017).
Work on the emergence of language in multi-agent settings goes back a long way (Steels, 1997; Nowak & Krakauer, 1999; Kirby, 2001; Briscoe, 2002; Skyrms, 2010).
In our case, we are speciﬁcally interested in tabula inscripta agents that are already pre-trained to generate natural language, and we are primarily concerned with keeping their language natural during further training.
Reinforcement Learning (RL) has been applied to ﬁne-tuning models for various natural language generation tasks, including summarization (Ranzato et al., 2015; Paulus et al., 2017), information retrieval (Nogueira & Cho, 2017), MT (Gu et al., 2017; Bahdanau et al., 2016) and dialogue (Li et al., 2017).
Our work can be viewed as ﬁne-tuning MT systems using an intermediary pivot language.
In
MT, there is a long line of work of pivot-based approaches, most notably Muraki (1986) and more recently with neural approaches (Wang et al., 2017; Cheng et al., 2017; Chen et al., 2018).
There has also been work on using visual pivots directly (Hitschler et al., 2016; Nakayama & Nishida, 2017; Lee et al., 2018).
Grounded language learning in general has been shown to give signiﬁcant practical improvements in various natural language understanding tasks (Gella et al., 2017; Elliott & K´ad´ar, 2017; Chrupała et al., 2015; Kiela et al., 2017; K´ad´ar et al., 2018). Meanwhile, Bowman et al. (2016) found a powerful decoder to ignore the latent representation in VAEs for language.
T ASK AND M ODELS
We recast translation as a communication game involving two MT agents:
Fr → En and En → De (see Figure 1).
Our dataset consists of N triples of aligned sentences { Fr i , En i , De i } Ni =1 , where
En i is only used for evaluation.
We ﬁrst feed the French sentence Fr i to Agent A, which generates an English message
En i as output.
Agent B is then trained to maximize the log likelihood of the ground truth German sentence given the English message, i.e. log p ( De i | En i ) .
Agent A is trained using REINFORCE (Williams, 1992) with reward R = log p B ( De i | En i ) .
This encourages Agent
A to develop helpful communication policies for Agent B, and allows Agent B to adapt to Agent A’s new policies.
In other words: communication via the pivot language (English) is a success if we are able to translate the intended source sequence (French) into the desired target sequence (German).
Both agents are pre-trained individually before communication, meaning that we start off with English as an intermediate language in the early stages of the game.
This work examines what happens to the intermediate language as we ﬁne-tune the system jointly: will the agents keep communicating in English, or diverge?
And if so, what can we do to prevent that from happening?
3. A UXILIARY T ASKS
To help reduce the search space of intermediate languages, we use two auxiliary tasks: language modelling (LM) and image-caption retrieval (henceforth called the grounding model).
Language Model Given a language model pre-trained on a standard English corpus, the log likelihood of the English message informs its general “Englishness”.
We incorporate this into the reward for Agent A, so that it learns to send messages that are plausible English.
2 Reward for Agent A is:
[Equation]
Grounding Model Let us assume we have access to a set of images { Img i } associated with each triple { Fr i , En i , De i } .
Given a pre-trained image-caption retrieval model, such as VSE++ (Faghri et al., 2018), the log likelihood of the image given the English message (and vice versa) informs how much the English message is grounded in the original semantic content (Kiela et al., 2017).
We incorporate the ranking loss into Agent A’s reward.
[Equation]
Note that β LM , β G are hyperparameters.
3. T RAINING O BJECTIVE
For brevity the t -th token in the i -th English sentence
En i ; t is abbreviated to En t , and En i to En.
Policy Gradient Training At decoding timestep t , Agent A takes an action (outputs token En t ) given an environment (previous hidden states and previous token
En t − 1 ).
It receives reward R at the end of the sequence, from which we subtract a state-dependent baseline R t to reduce variance.
Therefore, we maximize ( R − R t ) log p ( En t | En <t , Fr ) .
In addition, we employ entropy regularization on Agent A’s decoder to encourage exploration.
Hence, Agent A’s overall objective function is given as:
[Equation]
where H and MSE denote entropy and mean squared error losses.
Cross Entropy Training Agent B is trained using standard cross entropy loss, i.e.
[Equation]
We jointly train both agents by maximizing L =
L A + L B .
E XPERIMENTAL S ETTINGS
In this section we provide the details of our experimental setup: a Fr → X → De translation task where the intermediate language X is initialized as English, and subsequently ﬁne-tuned with policy gradient.
On a trilingual corpus consisting of three languages (Fr, En and De), we can measure communication success with Fr → De BLEU (Papineni et al., 2002), while Fr → En BLEU informs how closely the intermediate language resembles English, at any given point during ﬁne-tuning.
Datasets Agents are initially pre-trained on IWSLT Fr
→ En and En → De.
Fine-tuning is performed on
Multi30k Task 1(Elliott et al., 2016).
That is, importantly, there is no overlap in the pre-training data and the ﬁne-tuning data.
Multi30k Task 1 consists of 30k images and one caption per image in English, French, German and Czech (of which we only use the ﬁrst three).
For the English language model, we compare four different datasets: WikiText103, MS COCO and Flickr30k. The image-caption retrieval model is trained on Flickr30k: the same set of 30k images as Multi30k but containing 5 English captions per image. Following Faghri et al.
(2018), We randomly crop training images at every epoch.
We use 2048-dimensional ﬁnal-layer features from a pretrained and ﬁxed ResNet-152 (He et al., 2016).
Preprocessing The same tokenization and vocabulary are used across different tasks and datasets.
We lowercase and tokenize our corpora with Moses (Koehn et al., 2007) and use subword tokenization with Byte Pair Encoding (BPE) (Sennrich et al., 2016) with 10k merge operations.
This allows us to use the same vocabulary across different models seamlessly (translation, language model, image-caption ranker model).
Controlling the English message length When ﬁne-tuning the agents, we observe that the length of English messages becomes excessively long.
As Agent A has no explicit incentive to output the (cid:104) EOS (cid:105) symbol, it tends to keep transmitting the same token repeatedly.
Excessively long messages obscure evaluation of the communication protocol.
For instance, BLEU score quickly deteriorates as the message length becomes longer, as it is a precision metric.
When the message length is ﬁxed, a drop in BLEU score will by necessity mean that the intermediate language has drifted away more.
For this reason, we constrain the length of English messages to be no longer than the length of their
French source sentence, or shorter if the model outputs the (cid:104) EOS (cid:105) symbol early.
Recall that Agent
B is supervised to predict the (cid:104) EOS (cid:105) symbol, so does not suffer from this issue.
Model Architecture and Pretraining Our MT agents are standard sequence-to-sequence models with attention (Bahdanau et al., 2015) with unidirectional, 1-layer GRU with 256 hidden units and 256-dimensional embeddings.
During initial pre-training on IWSLT, we early-stop based on BLEU score on the development set (tst2013).
The best checkpoints give 34.05 BLEU and 21.94 BLEU on IWSLT Fr
→ En and En → De development sets with greedy decoding.
For our value function, we use a 2-layer MLP with a ReLU nonlinearity.
The language model is a 1-layer recurrent language model with 512 LSTM hidden units.
The imagecaption retrieval model is a recently proposed VSE++ model (Faghri et al., 2018), with unidirectional 1-layer GRU with 512 hidden units and a single fully connected layer from 2048-dimensional ResNet features to 512-dimensional GRU hidden states.
We report the performance of the pretrained models used in our experiments in Tables 1, 2 and 3.
[Table] [Table]
[Table]
Training Details When ﬁne-tuning our agents, we perform learning rate annealing and early stopping based on Fr → De BLEU (communication performance) on the Multi30k development set.
We use Adam (Kingma & Ba, 2014) with initial learning rate of 0.001 and dropout (Srivastava et al., 2014) rate of 0.1.
We grid search over learning rate schedule and reward coefﬁcients ( α pg , α entr , α b , β LM , β G ).
For our joint systems with policy gradient ﬁne-tuning [Table] we run every model three times with different random seeds and report averaged results (see Table 4) [Table]
Baseline and Upper Bound
Our main quantitative experiment has three baselines:
• Pretrained checkpoints (on IWSLT).
• Ensembling :
Given Fr, we let Agent A generate K English hypotheses with beam search, { En j } Kj =1 .
Then, we let Agent B generate the German translation De using an ensemble of K source sentences (Firat et al., 2016; Zoph & Knight, 2016).
Fr → En ﬁxed : We ﬁx Agent A and only ﬁne-tune Agent B using L B .
Meanwhile [Table] we also train an NMT model of the same architecture and size directly on the Fr → De task in Multi30k Task 1.
This serves as an upper bound on the Fr → De performance achievable with available data.
Q UANTITATIVE R ESULTS
[Table]
In Table [Table] the top three rows are our baselines.
The pretrained model performs relatively poorly on
Fr → De, conceivably because it was pretrained on a different corpus
[Table] and Agent B was was given
Agent A’s output as source.
Ensembling multiple English hypotheses for Agent B (row 2) gives negligible increase in Fr → De performance.
When only Agent B is ﬁne-tuned, we observe 6 BLEU score increase in Fr → De.
When the joint system is ﬁne-tuned on German log likelihood with policy gradients (PG), we observe a large, 8 BLEU increase increase in Fr → De at the cost of a substanstial, 15 BLEU score drop in
Fr → En.
This clearly shows that optimizing on some external reward causes a drastic language drift.
When the agent is trained with the “Englishness” constraint (PG+LM), we notice a signiﬁcant improvement in Fr → En BLEU.
When the LM is trained on WikiText103, a widely used language modelling dataset, we observe improvement of 9 BLEU scores.
When the training corpus is closer to the target domain, such as MS COCO or Flickr30k, we see more than 10 BLEU score increase.
Fr → De translation also improves by 2–3 BLEU scores.
However, we see the biggest improvements in performance when agents are trained using visual grounding feedback.
This is particularly pronounced with the LM trained on WikiText103: introducing visual grounding leads to more than 2 BLEU score improvement in Fr → En, and 1 BLEU score improvement in Fr → De.
We hypothesize that the “Englishness” constraint forces agents to communicate with correct syntax and ﬂuency, while the image-caption retrieval model restricts the search space of languages to ones that are grounded by visual semantics.
To see if grounding is really necessary, we train a stronger LM on all three datasets combined, but ﬁnd this still leads to more language drift than using visual grounding: the PG+LM+G model with the LM trained on MS COCO outperforms this by 3 BLEU scores on Fr → En.
In Figure 2, we observe that vanilla PG ﬁne-tuning quickly leads to highly “un-English” communication, as can be seen from a distinct increase in LM NLL.
It is also worth noting that while
PG+LM achieves better LM NLL than PG+LM+G, it gives much lower Fr → En BLEU score than the grounded model (PG+LM+G).
This is another indication that simply encouraging naturalness is not enough; grounding is key.
A close investigation into the token statistics of each communication strategy reveals that PG ﬁnetuning causes the word frequency distribution to be ﬂatter.
The PG model has negative frequency difference values for the most frequent tokens, indicating that PG downweighs frequent words severely.
On the other hand, PG+LM gives highly positive frequency differences, meaning that language modelling alone disproportionately emphasizes frequent tokens.
Visual grounding keeps the token frequency distribution close to the original pretrained regimes.
Analyzing the top-k most frequent words shows that PG+LM disproportionately favors quotation marks, which are very common tokens in many language modelling datasets but occur rarely in Multi30k (see also Appendix A). [Table]
Table 5 reinforces the ﬁnding that vanilla PG ﬁne-tuning leads to ﬂatter token frequency distributions, as the number of unique tokens used by PG is greater than that of the pretrained model.
Meanwhile, PG+LM uses fewer tokens overall, signifying that it uses a relatively small set of tokens frequently.
Also note that PG, despite using a more diverse set of tokens, uses the smallest number of unique symbols per sentence (/sent) and overall (/all).
This implies that PG communication is often repetitive.
Introducing extra tasks seems to mitigate this, and the grounded model (PG+LM+G) learns a frequency distribution that most closely resembles the original distribution.
To gain further insight into the agents’ communication protocols, we compare the degree of drift by part-of-speech.
Table 6 shows that PG tends to ignore many function words.
Note, for example, that it almost completely ignores periods.
Models trained with LM and grounding losses retain function words with much higher accuracy.
With content words, on the other hand, PG fares relatively better.
Conceivably, when optimizing Agent A’s policy on the communication task alone, it is more crucial to relay content information to Agent B, and this might cause agents to ignore syntactic conformity in the original intermediate language.
We argue that LM and grounding reduces the space of intermediate languages to a much reasonable language space, facilitating learning [Table]
Q UALITATIVE R ESULTS
In the ﬁrst example of Table [Table] it is clear that PG’s English message has signiﬁcantly diverged from
English: it is highly repetitive (“table table table table table”) and is missing some key content words such as “man” and “jacket”.
However, Agent B still generates the German word for ‘man’.
The grounded model’s message (PG+LM+G) is distinctly the most ﬂuent and semantically correct.
In the second example, observe that the PG Agent B misinterprets “talking talking a coach a coach” into “spricht mit einem spieler” (talking to a player).
The PG+LM+G model again generates a ﬂawless English sentence.
Also note that it communicates both colors (red and white) successfully from French to German, while the other two models fail to do so. [Table] [Table]
mit [Table] [Table] a punk sitting sitting on on a broken De ref ein kind sitzt auf einem felsen .
De hyp ein kind sitzt auf einem felsen . Fr src un petit enfant est assis `a une table, en train de manger un goˆuter. En ref a toddler is sitting at a table eating a snack . En hyp a punk sits sitting sitting next next a airline De ref ein kleines kind sitzt an einem tisch und isst einen snack .
De hyp ein kind sitzt an einem tisch und liest ein buch . We observe some instances of token ﬂipping with the PG model. For example, one particular PG model uses “punk” to describe “child” (see Table 8).
As no occurrence of “punk” in any training data is associated with “child”, the agents must have acquired this new meaning assignment during ﬁne-tuning.
Among 35 examples in Multi30k development set where the English reference contains “child”, the model uses “punk” 15 times, indicating this is no random phenomenon. We show similar examples from the PG+LM model in Appendix B.
We did not observe such examples with the PG+LM+G model.
C ONCLUSION
In this paper, we show that language drift happens when ﬁne-tuning natural language agents with some external (non-linguistic) reward using policy gradients, and propose a few approaches to avoid this.
Most importantly, we ﬁnd that simply encouraging “naturalness”, e.g. via adding a language model log likelihood to the reward, does not lead to the desired consequences.
Instead, we contend that grounding is what we need to avoid language drift.
Our empirical results show that grounding leads to best communication performance (highest Fr → De BLEU), while also showing least signs of language drift (highest Fr → En BLEU).
Analyzing token frequencies in exchanged messages reveals that pure PG ﬁnetuning tends to learn ﬂatter token distributions, and encouraging naturalness disproportionately emphasizes frequent tokens, while the grounded model best retains the original token frequencies. B E VIDENCE OF TOKEN FLIPPING IN THE PG+LM MODEL
[Table]
Similar to Table 8, we ﬁnd evidence of token ﬂipping for the PG+LM model, where the agents use “can@@” (@@ is a subword BPE token marker) to mean “poodle”.
This shows that language drift still happens even when a language model is used.