Paper:
DRL-Sense: Deep Reinforcement Learning for Multi-Sense Word Representations
This paper proposes DRL-Sense—a multisense word representation learning model, to address the word sense ambiguity issue, where a sense selection module and a sense representation module are jointly learned in a reinforcement learning fashion. A novel reward passing procedure is proposed to enable joint training on the selection and representation modules. The modular design implements pure senselevel representation learning with linear time sense selection (decoding). We further develop a non-parametric learning algorithm and a sense exploration mechanism for better flexibility and robustness. The experiments on benchmark data show that the proposed approach achieves the state-of-the-art performance on contextual word similarities and comparable performance with Google’s word2vec while using much less training data.
1 Introduction
Recently, deep learning methodologies have dominated several research areas in natural language processing (NLP), such as machine translation, language understanding, and dialogue systems. However, most of applications usually utilize word-level embeddings to obtain semantics. Considering that natural language is highly ambiguous, the standard word embeddings may suffer from polysemy issues. Neelakantan et al. (2014) pointed out that, due to triangle inequality in vector space, if one word embedding has two different senses, the sum of the distance between the word embedding and its synonymous embedding in each sense would be larger or equal to the distance between these synonymous embed-
dings1. Because this inaccurate distance measurement may degrade downstream NLP tasks, multisense word representations are proposed to address the ambiguity issue in a single word embedding scenario (Reisinger and Mooney, 2010; Huang et al., 2012)
This paper proposes DRL-Sense—a novel reinforcement learning based framework for learning multi-sense representations, which is composed by two main modules: a sense selection module for inferring the most probable sense for a word given its context, and a sense representation module for representing word senses in a continuous space. Our modular design implements pure sense-level representation learning while maintaining linear time sense selection. The proposed model is optimized by reinforcement learning and incorporates a non-parametric algorithms and a sense exploration mechanism without changing the network architecture2. Our contributions are four-fold: • We are among the first to study reinforcement
learning for sense embedding learning, taking account of the Markov property in sense selection given local contexts. • DRL-Sense is the first system that achieves
pure sense-level representation learning with linear time complexity on sense selection. • We develop non-parametric learning and
sense exploration mechanisms for a general neural sense selection module to achieve better flexibility and robustness. • Our experimental results show the state-of-
the-art performance on contextual word similarities and comparable performance with word2vec (Mikolov et al., 2013b) using only 1/100 size of training data.
1d(bank, finance) + d(bank, coast) ≥ d(finance, coast) 2The code will be released after the paper gets accepted.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
2 Related Work
There are three dominant types of approaches for learning multi-sense word representations in the literature: 1) clustering methods, 2) probabilistic modeling methods, and 3) lexical ontology based methods. Our reinforcement learning based approach can be loosely connected to clustering methods and probabilistic modeling methods.
Reisinger and Mooney (2010) is the first work to propose multi-sense word representations on vector space based on clustering techniques. With the power of deep learning, some work exploit neural networks to learn embeddings with senseannotated corpus based on clustering (Huang et al., 2012; Neelakantan et al., 2014). Chen et al. (2014) replaced the clustering procedure with a word sense disambiguation model based on WordNet (Miller, 1995). Kågebäck et al. (2015) further exploited a weighting mechanism on context in the clustering procedure and evaluated their system on word sense induction. Vu and Parker (2016) proposed an iterative process on the twostage clustering-embedding learning framework. Moreover, Guo et al. (2014) leveraged bilingual resources to perform clustering. However, most of the above approaches separates the clustering procedure and the representation learning procedure, which may suffer from the error propagation issue. Instead, our reinforcement learning model utilizes a reward signal to propagate the statistical information from sense representations to optimize sense selection.
In order to make the model more flexible, probabilistic modeling methods are usually utilized for learning multi-sense emnbeddings, where Tian et al. (2014) and Jauhar et al. (2015) conducted probabilistic modeling with EM training. Li and Jurafsky (2015) exploited the Chinese Restaurant Process and demonstrates efficacy of multi-sense word representations on several downstream NLP tasks. Furthermore, Bartunov et al. (2016) developed a non-parametric Bayesian extension on the skip-gram model (Mikolov et al., 2013b). However, all above methods fail to maintain pure sense-level representation learning due to the complicated computation in their EM algorithms (Bartunov et al., 2016), where sense-level embeddings are trained with word-level embeddings.
Recently, Qiu et al. (2016) proposed an EM algorithm to learn representations without wordlevel embeddings, where the computational cost
is high when selecting the sense identity sequence because the time for searching all sense combination within a context window is exponential. In contrast, by exploiting embeddings in the sense selection module, our model performs linear time sense selection, while maintaining a pure senselevel representation learning module. Another difference between the prior work and ours is that, Qiu et al. (2016) used WordNet (Miller, 1995) to obtain the number of senses for words, and our solution supports non-parametric learning for automatically deciding the sense number of each word.
Furthermore, most related work either directly exploited sense representations (Neelakantan et al., 2014; Qiu et al., 2016) or combined with online expectation sense counts (Li and Jurafsky, 2015; Jauhar et al., 2015) to select senses. The proposed approach incorporates a sense selection module, which can be learned together with the representation learning module through reinforcement learning.
Unlike a lot of relevant work that requires additional resources such as the lexical ontology (Rothe and Schütze, 2015; Jauhar et al., 2015; Chen et al., 2015; Iacobacci et al., 2015) or bilingual data (Guo et al., 2014; Ettinger et al., 2016; Šuster et al., 2016), which may not be available in the target task, our model can be trained on the corpus only. Also, some prior work proposed to learn topical embeddings and word embeddings jointly in order to consider the contexts (Liu et al., 2015a,b), whereas this paper focuses on learning multi-sense word embeddings.
3 Proposed Approach: DRL-Sense
This work proposes a framework to learn two key modules for multi-sense word representations: a sense selection module and a sense representation module. The sense selection module decides which sense to use given a text context, whereas the sense representation module learns meaningful representation for sense by their statistical characteristics.
Considering that the sense representation module requires the sense identity of each word from the sense selection module, and the sense selection module may also benefit from the semantics carried by sense representations, these two modules should be tangled. Hence, a naive two-stage algorithm or two separate learning algorithms proposed by prior work are not optimal. We propose
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
𝐶𝑡−𝑚 𝐶𝑡+𝑚𝐶𝑡 = 𝑤𝑖… …
𝑞( ഥ𝐶𝑡 , 𝑧𝑖1) 𝑞( ഥ𝐶𝑡, 𝑧𝑖2) 𝑞( ഥ𝐶𝑡, 𝑧𝑖3)
𝐶𝑡′−𝑚 𝐶𝑡′+𝑚𝐶𝑡′ = 𝑤𝑗… …
𝑞(𝐶𝑡′, 𝑧𝑗1) 𝑞(𝐶𝑡′, 𝑧𝑗2) 𝑞(𝐶𝑡′, 𝑧𝑗3)
𝑧𝑖1
Sense Representation Module
…𝑃(𝑧𝑗2|𝑧𝑖1) 𝑃(𝑧𝑢𝑣|𝑧𝑖1)
→ Q-learning reward signal
← greedy sense selection
Sense Selection Module
← Q -learn in g rew ard sign al → greed y sen se selectio n
Sense selection for target word 𝐶𝑡 Sense selection for collocated word 𝐶𝑡′
negative sampling
matrix 𝑄𝑖
matrix 𝑉
matrix 𝑈 matrix 𝑃
matrix 𝑄𝑗
matrix 𝑃
sample collocation Sense Selection Module
Figure 1: The RRL-Sense architecture illustration. Two shared-weight sense selection modules surround a sense representation module. The sense selection module infers the target word sense given its context. The selected senses are passed to the sense representation module for learning sense embeddings. Then, the sense embedding collocation is passed back to the sense selection modules for reinforcement learning.
to learn two modules in a reinforcement learning manner by introducing a novel reward passing procedure to enable joint training. In addition, a nonparametric learning algorithm is incorporated to enable automatic sense induction, where the number of senses for each word is not predefined but automatically learned for practical usage. Also, simple mechanisms are applied to allow the sense exploration while incorporating the sense selection prior.
3.1 Model Architecture
Our model architecture is illustrated in Figure 1 and detailed below.
3.1.1 Sense Selection Module
The sense selection module decides the sense for a word given its context. Assuming that a word sense is determined by the local context, we use the Markov property to formulate the sense selection module as a Markov Decision Process (MDP) to infer the most probable sense based on its local context. Given a corpus C and a context window m, we extract a local context C̄t = {Ct−m, ..., Ct−1, Ct+1, ..., Ct+m} as a state St, and the selection of a sense zik ∈ Zi for a word wi = Ct given context C̄t as an action Aik given the state St, where Zi is the set of senses of word wi. However, this formulation lacks a central element of MDP: a reward signal for measuring the fitness of the selected sense zik. Therefore, if we have meaningful sense representations containing statistical estimation, we can use the estimation as a surrogate reward to simulate the reward signal in the sense selection module.
3.1.2 Sense Representation Module
A successful sense selection module can be applied to each word in a corpus for obtaining its sense identity. Various techniques about word embeddings can be directly employed after mapping all words in a corpus to its sense identity. The typical method is to formulate the sense selection problem as a maximum likelihood estimation (MLE) problem for the collocation likelihood. In this paper, we use the skip-gram formulation (Mikolov et al., 2013b), because it is more light-weight and thus requires less training time: only two sense selections are required for stochastic training, while GloVe (Pennington et al., 2014) and continuous bag-of-words (CBOW) (Mikolov et al., 2013a) require the whole corpus and the whole context window for sense selection respectively.
Specifically, we first create input sense embedding matrixU and collocation estimation matrix V as the representations to be learned. Given a word wi and collocated word wj with corresponding local contexts, we map them to their sense identities as zik and zjl by the sense selection module, and maximize the sense collocation log likelihood logCo(·) using a negative sampling approximation (Mikolov et al., 2013b),
logCo(zik, zjl | U, V ) = log σ(UTzikVzjl) (1)
+ M∑ v=1 Ezuv∼pneg(z)[log σ(−U T zik Vzuv)],
where pneg(z) is the distribution over all senses for negative samples.
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
3.2 Reinforcement Learning
With the above two modules, we treat the sense collocation likelihood as a surrogate reward signal for the sense selection module, and then cast the whole formulation as a reinforcement learning task: An optimal policy πθ with parameters θ should select the sense that leads to an optimal collocation likelihood in (1). In order to estimate the sense collocation likelihood, two senses, zik and zjl, should be determined, which leads a two-step MDP: one for the target word wi and the other for the collocated word wj within a context window. A policy gradient algorithm (Sutton et al., 1999) can formulate the policies as decision probability distributions πiθ(zik|C̄t) and π j θ(zjl|C̄t′) for words wi, wj given their contexts C̄t, C̄t′ . Hence, a joint formulation can be written as follows,
max U,V,θ Ek∼πiθ(·)[El∼πjθ(·)[logCo(zik, zjl | U, V )]]. (2) The neat formulation is differentiable and supports stochastic optimization using doubly stochastic gradient (Lei et al., 2016). However, there are two major drawbacks of this formulation. First, it requires a policy πθ to fit a valid probability distribution, and this constraint underestimates the probability of senses that have not been selected yet during optimization. Second, if using stochastic gradient ascent (detailed in Appendix A) to optimize equation (2), it would actually lower the probability estimation for the selected sense because logCo(·) ≤ 0, which is not desirable if the model accurately select the right sense.
3.2.1 Q-Learning Formulation
To address the above issue, a Q-learning algorithm (Mnih et al., 2013) with a novel reward passing procedure is applied to this task, because it optimizes the selected action (sense) independently to the other actions. Also, considering that the Qlearning algorithm is not constrained by a jointprobabilistic formulation, we reduce the range of the reward function to facilitate the training process by replacing the collocation log likelihood logCo ∈ (− inf, 0] with (its monotonous) likelihood Co ∈ [0, 1] as the reward function (Mnih et al., 2013).
Our model utilizes a CBOW architecture to model the fitness for a sense zik of a wordwi under a local context C̄t. Specifically, given a word embedding matrix P , the local context can be modeled as the summation of word embeddings from
Algorithm 1: DRL-Sense Training Algorithm for wi = Ct ∈ C do
sample wj = Ct′(0 < |t′ − t| ≤ m); k = arg maxk′ q(C̄t, zik′); l = arg maxl′ q(C̄t′ , zjl′); optimize U, V by (1) for the sense representation module; optimize P,Q by (4) for the sense selection module;
its context C̄t. The fitness representation of the selected sense can be formulated with a 3-mode tensorQ, whose dimensions denote words, senses, and latent variables. Then we model the Q-value, q(C̄t, zik), for selecting the sense zik under the context C̄t as
q(C̄t, zik) = σ(Q T ik ∑ j∈C̄t Pj), (3)
where σ is a sigmoid function to form a valid probabilistic estimation.
3.2.2 DRL-Sense Training
To jointly train sense selection and sense representation modules, we first sample a collocated word pair wi, wj with respective contexts C̄t, C̄t′ , and use the estimated Q-value to select the most probable senses zik, zjl. The selected senses are passed to the sense representation module to optimize the sense collocation likelihood. Afterwards, the estimated collocation likelihood is passed back as a reward signal to optimize the sense selection module. The loss function is defined as cross-entropy H(·) due to the probability distribution.
min P,Q
H(Co(zik, zjl | U, V ), q(C̄t, zik))
+H(Co(zik, zjl | U, V ), q(C̄t′ , zjl)). (4)
Algorithm 1 shows the proposed DRL-Sense model training procedure. There are two major contributions in our modular design. First, efficient sense selection with word embeddings and pure sense representation learning are simultaneously achieved. Second, reinforcement learning allows both modules to be jointly trained.
3.3 Non-Parametric Learning
In order to automatically discover the sense number for each word, we incorporate word sense induction into our model (Song et al., 2016). Instead
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
𝐶𝑡−𝑚 𝐶𝑡+𝑚
𝑞( ഥ𝐶𝑡 , 𝑧𝑖1) 0.5 0.5 𝑞( ഥ𝐶𝑡 , 𝑧𝑖2)
If 𝑞( ഥ𝐶𝑡, 𝑧𝑖1) < 0.5 If … < 0.5
weight vector zero vector selected vector
𝐶𝑡 = 𝑤𝑖… … 𝐶𝑡−𝑚 𝐶𝑡+𝑚
𝑞( ഥ𝐶𝑡 , 𝑧𝑖1) 0.5
𝐶𝑡 = 𝑤𝑖… …
Sense Selection Module
Figure 2: The proposed non-parametric algorithm with initialization. The red block indicates the newly discovered sense zi2 for the target word wi based on the algorithm.
of non-parametric clustering (Neelakantan et al., 2014) or the stochastic process (Li and Jurafsky, 2015) applied by prior work, this paper proposes a novel mechanism in a general neural model to conduct non-parametric sense discovery.
Because the formulated Q-value is a probabilistic estimation, the model allows the selected word to create a new sense when all Q-values of its existing senses are lower than 0.5. Therefore, the prediction layer is initialized to 0 so that all Qvalues remain σ(0) = 0.5 until having been selected due to independent updates in Q-learning. Figure 2 illustrates the sense discovery procedure, where the model expands a word’s sense set when its context has all Q-values of existing senses less than 0.5.
3.4 Word Sense Exploration
Due to high ambiguity in natural language, a greedy sense selection strategy may not work well in the early training stage, because the sense selection module does not learn well. Such a drawback also exists in the literature using clustering algorithms (Neelakantan et al., 2014; Kågebäck et al., 2015) and hard-EM algorithms (Qiu et al., 2016; Jauhar et al., 2015). This issue is known as exploration-exploitation trade-off (Sutton and Barto, 1998). Li and Jurafsky (2015) proposed to sample on a Chinese Restaurant Process to introduce uncertainty. In our neural network architecture for computing Q-values, we perform dropout (Srivastava et al., 2014) in the hidden layer ∑ j∈C̄t Pj for our sense selection module to introduce uncertainty.
3.5 Sense Selection Prior
Incorporating priors in sense selection is a key element for probabilistic models (Li and Juraf-
sky, 2015; Jauhar et al., 2015), most previous work used online selection counts of each sense as its prior. As the key utility of the prior is to incorporate the sense selection preference for a word wi (disregarding its context {Ct−m, ..., Ct−1, Ct+1, ..., Ct+m}) in its sense selection process, we achieve similar prior utility by adding the target word wi into the input of the sense selection module as C̄t = {Ct−m, ..., Ct+m} such that the Q-values indicate the sense preference of the target word.
4 Experiments
We evaluate our proposed DRL-Sense model and compare with other multi-sense word representation models for both quantitative and qualitative experiments.
4.1 Experimental Setup and Configuration
Our model is trained on the April 2010 Wikipedia dump (Shaoul and Westbury, 2010), which contains approximately 1 billion tokens. For fair comparison, we adopt the same vocabulary set as Huang et al. (2012) and Neelakantan et al. (2014). For preprocessing, we convert all words to lower case, apply the Stanford tokenizer and the Stanford sentence tokenizer (Manning et al., 2014), and remove all sentences with less than 10 tokens.
In the experiments, the context window size is set to 6 (|C̄t| = 13). Subsampling technique introduced by word2vec (Mikolov et al., 2013b) is applied to accelerate the training process. The (upper bound of) number of senses per word in Q is set to 3 for fair comparison with prior work (Neelakantan et al., 2014). The learning rate is set to 0.025. Dropout is annealed from 0.5 to 0.0 within the first epoch. The embedding dimension is 300. We initialize Q and V as zeros, and P and U as the uniform distribution [− √ 1/100, √ 1/100] such that each embedding has unit length in expectation (Lei et al., 2015). For the negative sampling distribution in (1), we use 1/3 unigram of a word for each of its sense to compute the negative sampling probability using the 3/4rd power trick in Mikolov et al. (2013b). Our model uses 5 negative senses for negative sampling in (1).
In optimization, we conduct mini-batch training with 2048 batch size using the following procedure: 1) select senses in the batch; 2) optimize U, V using stochastic training within the batch for efficiency; 3) optimize P,Q using mini-
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Method MaxSimC AvgSimC Huang et al. (2012) 26.1 65.7 Neelakantan et al. (2014) 60.1 68.6 Tian et al. (2014) 63.6 65.4 Li and Jurafsky (2015) 66.4 - Bartunov et al. (2016) 53.8 61.2 Qiu et al. (2016) 64.9 66.1 Proposed: DRL-Sense 66.6 65.2 - Non-Parametric 65.1 67.0 - Sense Exploration 66.2 64.5 - Sense Selection Prior 66.3 66.0
Table 1: Spearman’s rank correlation ρ x100 on the SCWS dataset.
batch training for robustness. To further stabilize the reward signal for the sense selection module, we only use the collocated sense σ(UTzik , Vzjl) in (1) to approximate the collocation likelihood Co(zik, zjl | U, V ) for Q-learning in (4).
4.2 Experiment 1: Contextual Word Similarity
To evaluate the quality of sense embeddings, we compute the similarity score between each word pair given their respective local contexts and compare with the human-judged score using Stanford’s Contextual Word Similarities (SCWS) data (Huang et al., 2012). Specifically, given a list of word pairs with corresponding contexts, S = {(wi, C̄t, wj , C̄t′)}, we calculate the Spearman’s rank correlation ρ between human-judged similarity and model similarity estimations. Two major contextual similarity estimations are introduced by Reisinger and Mooney (2010): AvgSimC and MaxSimC (also referred to LocalSim in Neelakantan et al. (2014)). AvgSimC is a soft measurement that addresses the contextual information with a probability estimation:
AvgSimC(wi, C̄t, wj , C̄t′) = Zi∑ k=1 Zj∑ l=1 p(zik|C̄t)p(zjl|C̄t′)d(zik, zjl),
which weights the similarity measurement of each sense zik and zjl by their probability estimations. On the other hand, MaxSimC is a hard measurement that only evaluates on the most probable senses:
MaxSimC(wi, C̄t, wj , C̄t′) = d(zik, zjl),
zik = arg max k′ p(zik′ |C̄t), zjl = arg max l′ p(zjl′ |C̄t′).
The baselines for comparison include classic clustering methods (Huang et al., 2012; Neelakantan et al., 2014), EM algorithms (Tian et al., 2014; Qiu et al., 2016; Bartunov et al., 2016), and Chinese Restaurant Process (Li and Jurafsky, 2015), where all approaches are trained on the same corpus except Li and Jurafsky (2015)3 and Qiu et al. (2016) used more recent Wikipedia dumps. The embedding sizes of all baselines are 300, except 50 in Huang et al. (2012). In our method, the probability assignment on each sense p(zik|C̄t) is calculated by replacing the sigmoid layer with a softmax layer in our sense selection module as:
∑
4.2.1 Embedding Quality Analysis
Our DRL-Sense model achieves the state-of-theart performance on MaxSimC, demonstrating superior quality on independent sense embeddings. In real world applications, using multiple sense vectors for a word simultaneously may bring additionally computational overhead over conventional single word embedding scenarios, and also change the existing neural network architecture for word embeddings. In contrast, by simply replacing the most probable sense identity with each word identity (as in MaxSimC), the computation framework and cost for downstream NLP tasks remains the same as conventional word embedding approaches. Rothe and Schütze (2015) reported that the traditional single word representation model word2vec (Mikolov et al., 2013b), released by Google and trained on 100 billion tokens Google’s internal dataset, can achieved 66.6% similarity correlation on the SCWS dataset. Table 1 shows that our proposed DRL-Sense achieves comparable performance as word2vec using only 1/100 size of Google’s internal data, while using multi-sense representations in the single embedding scenario (MaxSimC).
3We use Li and Jurafsky (2015)’s result on 1 billion tokens for fair comparison.
4More detail is available in Appendix B.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Method ESL-50 RD-300 TOEFL-80 (1) Conventional Word Embedding GC 47.73 45.07 60.87 SG 52.08 55.66 66.67 (2) Word Sense Disambiguation IMS+SG 41.67 53.77 66.67 (3) Direct Learning from Corpus EM 27.08 33.96 40.00 NP-MSSG 45.24 50.00 81.16 DRL-Sense 54.76 60.71 72.46 (4) Retrofitting with WordNet Retro-GC 63.64 66.20 71.01 Retro-SG 56.25 65.09 73.33
Table 2: Accuracy on synonym selection
4.2.2 Probability Assignment Analysis
To interpret the performance between MaxSimC and AvgSimC, we claim that an ideal model should have MaxSimC≥AvgSimC in Table 1. The reason is that, any sense selection model seeks a function f(zik | C̄t) to evaluate the fitness of each sense zik under a context C̄t, such as the cluster similarity in clustering methods, the likelihood or vanilla probability estimation in probabilistic models, and the Q-value in our model. Hence, to achieve the optimal expected fitness E[f(zik | C̄t)], the optimal probability assignment on senses is only selecting one and the optimal sense:
popt(zik | C̄t) = { 1 if k = arg maxk′ f(zik′ |C̄t), 0 otherwise.
Note that the above probability assignment makes AvgSimC=MaxSimC. Because the expected fitness E[f(zik | C̄t)] over any probability distribution would be no greater than the fitness value over the above distribution, the similar phenomenon (MaxSimC≥AvgSimC) can be observed in the robust fitness estimation (cluster similarity/likelihood/Q-value). Therefore, from Table 1, our DRL-Sense model demonstrates not only superior performance under a single sense embedding scenario but also reasonable and robust sense selection.
4.3 Experiment 2: Synonym Selection
We further evaluate our model on synonym selection using multi-sense word representations (Jauhar et al., 2015). Three standard synonym selection datasets, ESL-50 (Turney, 2001), RD-300 (Jarmasz and Szpakowicz, 2004), and
TOEFL-80 (Landauer and Dumais, 1997), are performed. In the datasets, each question consists of a question word wQ and four answer candidates {wA, wB, wC , wD}, and the goal is to select the most semantically synonymous choice among the four candidates. For example, in the TOEFL-80 dataset, a question shows {(Q) enormously, (a) appropriately, (b) uniquely, (c) tremendously, (d) decidedly}, and the answer is (c).
In the experiments, our model selects the synonym of the question word wQ by the collocation likelihood as a proxy of their semantic similarity. That is, for pairs of words (wQ, wT ), T ∈ {A,B,C,D}, we select the pair with the maximum bi-directional collocation likelihood,
Sim(wQ, wT )
= max j∈ZQ,l∈ZT max(Co(zQj , zT l), Co(zT l, zQj))
= max j∈ZQ,l∈ZT
max(σ(UTQjVT l), σ(U T T lVQj)).
Our model is compared with following baselines: (1) conventional word embedding: global context vectors (GC) (Huang et al., 2012) and skip-gram (SG) (Mikolov et al., 2013b); (2) applying supervised word sense disambiguation (using the IMS system (Zhong and Ng, 2010)) and then applying SG on disambiguated corpus: IMS+SG; (3) direct training multi-sense word representations on corpus: EM algorithm (Jauhar et al., 2015), non-parametric multi-sense skipgram (NP-MSSG) (Neelakantan et al., 2014) and our DRL-Sense; (4) retrofitting existing word embeddings to sense embeddings using WordNet: retrofitting GC (Retro-GC) and retrofitting SG (Retro-SG) (Jauhar et al., 2015). Note that we do not compare with the methods in (4) on Table 1, because sense representations are generated without a sense selector, and thus cannot be evaluated using MaxSimC and AvgSimC. Also, the methods in (4) can be treated as an upperbound of the methods in (3) due to the usage of additional supervised information from WordNet.
The experimental results are shown on Table 2. Except for retrofitting methods and NP-MSSG in TOEFL-80, our DRL-Sense model significantly outperforms all baselines. In addition, our model also performs better than Retro-GC for TOEFL80, although Retro-GC uses the supervised knowledge from WordNet, showing that our model obtains good sense embeddings by learning with the robust sense selector. Moreover, our method out-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
Sense k-NN Senses mercury-1 mercury comet apollo comets wnba mercury-2 mercury sulfide beryllium zirconium apple-1 apple blackberry macintosh ibm novell apple-2 avocado apple apricot blackberry prunus lucky-1 lucky gotti co-starring gravano starred lucky-2 beyonce singin beenie baby choo lil lucky-3 hilfiger starbucks t-bone puff armani
Table 3: Sense representations with neighboring words using the collocation likelihood.
performs the EM method and NP-MSSG (Neelakantan et al., 2014), despite the highest score of AvgSimC on Table 1, echoing the superior quality of our sense vectors as suggested in the MaxSimC measurement.
4.4 Qualitative Analysis
We qualitatively evaluate non-parametric learning and sense representation learning performance.
4.4.1 Non-Parametric Learning Effectiveness
The largest challenge of multi-sense representation learning is that most words can be represented by a single embedding (non-polysemous or a single embedding can model multiple senses well). Hence, the feasible solution is to retain most words a single sense embedding, while leaving polysemous words multiple sense embeddings. Huang et al. (2012) and Neelakantan et al. (2014) only selected approximately 6,000 and 30,000 words from a vocabulary with size ≈ 100,000 for multisense representation learning. Qiu et al. (2016) utilized a coarse version (Navigli et al., 2007) of WordNet inventory (Miller, 1995) to determine the sense number for each word before training. The sense selection module in our learned DRL-Sense produces 71,194 words with a selected sense, 22,271 words with 2 senses, and 5,683 words with 3 senses based on the training data. Without any multi-sense determination procedure prior to training, our method automatically determines only about 30,000 polysemous words, where the number is close to Neelakantan et al. (2014)’s setting, demonstrating the effectiveness of our nonparametric learning algorithm.
4.4.2 Representation Learning Effectiveness
To evaluate the quality of sense embeddings, we show the k-nearest neighbors (k-NN) for each sense. For example, given the first sense of the word “apple”, denoted as “apple-1”, we select
Figure 3: The 2-D visualization using t-SNE (Maaten and Hinton, 2008) for the words “mercury” and “apple” on multi-sense word representations with L2 distance neighbors.
k senses with the highest collocation likelihood from the sense representation module and list in
Table 35. The learned sense embeddings of the words “mercury” and “apple” clearly correspond to different senses: “planet” (mercury-1), “metal element” (mercury-2), “computer” (apple-1), and “fruit” (apple-2) as visualized in Figure 3. Instead of actual senses, Table 3 also shows that our model is able to automatically detect a finegrained aspect division within a word “lucky” in an unsupervised manner. Specifically, it divides the semantics of “lucky” into three aspects: “starring” (lucky-1), “musician” (lucky-2), and “brands” (lucky-3), expanding the potential usage of the learned embeddings6.
5 Conclusion
We propose DRL-Sense—a novel deep reinforcement learning framework to jointly learn a word sense selection module and a sense representation module. Our model implements non-parametric learning for word sense induction and exploration for word sense selection. The experiments show that our DRL-Sense model achieves the state-ofthe-art performance for the benchmark contextual word similarity task and most of synonym selection datasets under the same setting. In the future, we plan to investigate reinforcement learning methods to incorporate multi-sense word representations for downstream NLP tasks.
5For clarity purpose, we omit the sense identity in the kNN senses.
6Note that the sense induction procedure highly depends on the training corpus, so it may fail to discover all senses in an existing sense inventory.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
A Doubly Stochastic Gradient
For clarity purpose, we abbreviate the collocation log likelihood logCo(ik, jl|U, V ) as L(·) in this section. To derive doubly stochastic gradient for equation (2), we first resolve the expectation form as:
J(U, V, θ)
= Ek∼πiθ(·)[El∼πjθ(·)[L(·)]] = ∑ k ∑ l πiθ(zik|C̄t)π j θ(zjl|C̄t′)L(·).
The gradient of J with respect to Θ should be:
∂J(U, V, θ)
∂Θ
= ∂
∂Θ ∑ k ∑ l πiθ(zik|C̄t)π j θ(zjl|C̄t′)L(·)
= ∑ k ∑ l πiθ(zik|C̄t)L(·) ∂πjθ(zjl|C̄t′) ∂Θ
+ ∑ k ∑ l πjθ(zjl|C̄t′)L(·) ∂πiθ(zik|C̄t) ∂Θ
= Ek∼πiθ(·)[ ∑ l L(·) ∂ log πjθ(zjl|C̄t′) ∂Θ πjθ(zjl|C̄t′)]
+ E l∼πjθ(·) [ ∑ k L(·) ∂ log πiθ(zik|C̄t) ∂Θ πiθ(zik|C̄t)]
= Ek∼πiθ(·)[El∼πjθ(·)[L(·) ∂
∂Θ (log πiθ(zik|C̄t) + log π j θ(zjl|C̄t′))]]
Accordingly, if we conduct typical stochastic gradient ascent training on J(U, V,Θ) with respect to Θ from samples k and l with a learning rate η, the update formula will be:
∆Θ = ηL(·) ∂ ∂Θ (log πiθ(zik|C̄t) + log π j θ(zjl|C̄t′)).
However, because the collocation likelihood is a valid probability distribution, the collocation log likelihood should always be non-positive: L(·) = logCo(zik, zjl|U, V ) ≤ 0. Therefore, as long as the collocation log likelihood L is negative, the update formula is to minimize the likelihood of choosing l and k, despite the fact that l and k may be good choices. On the other hand, if the log likelihood reaches 0, it also indicates that the likelihood reaches infinity and computational overflow on U and V .
B Supplementary Experiments
In this section, we conduct ablation experiments by removing each components separately in our system to test the efficacy of proposed mechanisms. Specifically, we test the following components:
• Non-parametric: To remove the nonparametric learning mechanism in our model, we initialize the prediction layer Q in the sense selection module from uniform distribution [− √ 1/100, √ 1/100] as the
word embedding P .
• Sense exploration: To remove the word sense exploration mechanism in our model, we turn off the dropout mechanism during training.
• Sense selection prior: To remove the sense selection prior mechanism in our model, we remove the target word wi in the input of the sense selection module as C̄t = {Ct−m, ..., Ct−1, Ct+1, ..., Ct+m}.
In addition, we also test the efficacy of specific module according to their complexity. We test the setting when the embedding dimension in the sense selection module is set to 50 (50D/300D), and the embedding dimensions in the both modules are set to 50 (50D/50D).
The experiment results are shown in Table 4. Consistent improvement in terms of MaxSimC from each component to the proposed DRL-Sense model can be observed. In addition, the improvement from increasing the complexity (embedding dimension) demonstrate the efficacy of both modules (50D-50D→50D-300D and 50D300D→DRL-Sense).

Review:
TMP Strength: The paper propose DRL-Sense model that shows a marginal improvement on SCWS dataset and a significant improvement on ESL-50 and RD-300 datasets.  Weakness: The technical aspects of the paper raise several concerns: Could the authors clarify two drawbacks in 3.2? The first drawback states that optimizing equation (2) leads to the underestimation of the probability of sense. As I understand, eq(2) is the expected reward of sense selection, z_{ik} and z_{jl} are independent actions and there are only two actions to optimize. This should be relatively easy. In NLP setting, optimizing the expected rewards over a sequence of actions for episodic-task has been proven doable (Sequence Level Training with Recurrent Neural Networks, Ranzato 2015) even in a more challenging setting of machine translation where the number of actions ~30,000 and the average sequence length ~30 words. The DRL-Sense model has maximum 3 actions and it does not have sequential nature of RL. This makes it hard to accept the claim about the first drawback.  The second drawback, accompanied with the detail math in Appendix A, states that the update formula is to minimize the likelihood due to the log-likelihood is negative. Note that most out-of-box optimizers (Adam, SGD, Adadelta,  u2026) minimize a function f, however, a common practice when we want to maximize f we just minimize -f. Since the reward defined in the paper is negative, any standard optimizer can be use on the expected of the negative reward, which is always greater than 0. This is often done in many modeling tasks such as language model, we minimize negative log-likelihood instead of maximizing the likelihood. The authors also claim that when  the log-likelihood reaches 0, it also indicates that the likelihood reaches infinity and computational flow on U and V d (line 1046-1049). Why likelihood u2192infinity? Should it be likelihood u21921?  Could the authors also explain how DRL -Sense is based on Q-learning? The horizon in the model is length of 1. There is no transition between state-actions and there is not Markov-property as I see it (k, and l are draw independently). I am having trouble to see the relation between Q-learning and DRL-Sense.  In (Mnih et al., 2013), the reward is given from the environment whereas in the paper, the rewards is computed by the model. What  the reward in DRL-Sense? Is it 0, for all the (state, action) pairs or the cross-entropy in eq(4)?    Cross entropy is defined as H(p, q) = -  sum_{x} q(x)  log q(x), which variable do the authors sum over in (4)? I see that q(C_t, z_{i, k}) is a scalar (computed in eq(3)), while Co(z_{ik}, z_{jl}) is a distribution over total number of senses eq(1). These two categorial variables do not have the same dimension, how is cross-entropy H in eq(4) is computed then?  Could the authors justify the dropout exploration? Why not epsilon-greedy exploration? Dropout is often used for model regularization, preventing overfitting. How do the authors know the gain in using dropout is because of exploration but regularization?  The authors states that Q-value is a probabilistic estimation (line 419), can you elaborate what is the set of variables the distribution is defined? When you sum over that set of variable, do you get 1? I interpret that Q is a distribution over senses per word, however  definition of q in eq(3) does not contain a normalizing constant, so I do not see q is a valid distribution. This also related to the value 0.5 in section 3.4 as a threshold for exploration. Why 0.5 is chosen here where q is just an arbitrary number between (0, 1) and the constrain   sum_z q(z) = 1 does not held? Does the authors allow the creation of a new sense in the very beginning or after a few training epochs? I would image that at the beginning of training, the model is unstable and creating new senses might introduce noises to the model.  Could the authors comment on that?  General discussion What  the justification for omitting negative samples in line 517? Negative sampling has been use successfully in word2vec due to the nature of the task: learning representation. Negative sampling, however does not work well when the main interest is modeling a distribution p() over senses words. Noise contrastive estimation is often preferred when it comes to modeling a distribution. The DRL-Sense, uses collocation likelihood to compute the reward, I wonder how the approximation presented in the paper affects the learning of the embeddings.  Would the authors consider task-specific evaluation for sense embeddings as suggested in recent research [1,2]  [1] Evaluation methods for unsupervised word embeddings. Tobias Schnabel, Igor Labutov, David Mimno and Thorsten Joachims.  [2] Problems With Evaluation of Word Embeddings Using Word Similarity Tasks . Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, Chris Dyer  --- I have read the response.