Paper:
Tandem Anchoring: a Multiword Anchor Approach for Interactive Topic Modeling
Interactive topic models are powerful tools for those seeking to understand large collections of text. However, existing sampling-based interactive topic modeling approaches scale poorly to large data sets. Anchor methods, which use a single word to uniquely identify a topic, offer the speed needed for interactive work but lack both a mechanism to inject prior knowledge and lack the intuitive semantics needed for user-facing applications. We propose combinations of words as anchors, going beyond existing single word anchor algorithms—an approach we call “Tandem Anchors”. We begin with a synthetic investigation of this approach then apply the approach to interactive topic modeling in a user study and compare it to interactive and non-interactive approaches. Tandem anchors are faster and more intuitive than existing interactive approaches. Topic models distill large collections of text into topics, giving a high-level summary of the thematic structure of the data without manual annotation. In addition to facilitating discovery of topical trends (Gardner et al., 2010), topic modeling is used for a wide variety of problems including document classification (Rubin et al., 2012), information retrieval (Wei and Croft, 2006), author identification (Rosen-Zvi et al., 2004), and sentiment analysis (Titov and McDonald, 2008). However, the most compelling use of topic models is to help users understand large datasets (Chuang et al., 2012). Interactive topic modeling (Hu et al., 2014) allows non-experts to refine automatically generated topics, making topic models less of a “take it or leave it” proposition. Including humans input during training improves the quality of the model and allows users to guide topics in a specific way, custom tailoring the model for a specific downstream task or analysis. The downside is that interactive topic modeling is slow—algorithms typically scale with the size of the corpus—and requires non-intuitive information from the user in the form of must-link and cannot-link constraints (Andrzejewski et al., 2009). We address these shortcomings of interactive topic modeling by using an interactive version of the anchor words algorithm for topic models. The anchor algorithm (Arora et al., 2013) is an alternative topic modeling algorithm which scales with the number of unique word types in the data rather than the number of documents or tokens (Section 1). This makes the anchor algorithm fast enough for interactive use, even in web-scale document collections. A drawback of the anchor method is that anchor words—words that have high probability of being in a single topic—are not intuitive. We extend the anchor algorithm to use multiple anchor words in tandem (Section 2). Tandem anchors not only improve interactive refinement, but also make the underlying anchor-based method more intuitive. For interactive topic modeling, tandem anchors produce higher quality topics than single word anchors (Section 3). Tandem anchors provide a framework for fast interactive topic modeling: users improve and refine an existing model through multiword anchors (Section 4). Compared to existing methods such as Interactive Topic Models (Hu et al., 2014), our method is much faster.
Topic models distill large collections of text into topics, giving a high-level summary of the thematic structure of the data without manual annotation. In addition to facilitating discovery of topical trends (Gardner et al., 2010), topic modeling is used for a wide variety of problems including document classification (Rubin et al., 2012), information retrieval (Wei and Croft, 2006), author identification (Rosen-Zvi et al., 2004), and sentiment analysis (Titov and McDonald, 2008). However, the most compelling use of topic models is to help users understand large datasets (Chuang et al., 2012).
Interactive topic modeling (Hu et al., 2014) allows non-experts to refine automatically generated
topics, making topic models less of a “take it or leave it” proposition. Including humans input during training improves the quality of the model and allows users to guide topics in a specific way, custom tailoring the model for a specific downstream task or analysis.
The downside is that interactive topic modeling is slow—algorithms typically scale with the size of the corpus—and requires non-intuitive information from the user in the form of must-link and cannot-link constraints (Andrzejewski et al., 2009). We address these shortcomings of interactive topic modeling by using an interactive version of the anchor words algorithm for topic models.
The anchor algorithm (Arora et al., 2013) is an alternative topic modeling algorithm which scales with the number of unique word types in the data rather than the number of documents or tokens (Section 1). This makes the anchor algorithm fast enough for interactive use, even in web-scale document collections.
A drawback of the anchor method is that anchor words—words that have high probability of being in a single topic—are not intuitive. We extend the anchor algorithm to use multiple anchor words in tandem (Section 2). Tandem anchors not only improve interactive refinement, but also make the underlying anchor-based method more intuitive.
For interactive topic modeling, tandem anchors produce higher quality topics than single word anchors (Section 3). Tandem anchors provide a framework for fast interactive topic modeling: users improve and refine an existing model through multiword anchors (Section 4). Compared to existing methods such as Interactive Topic Models (Hu et al., 2014), our method is much faster.
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
1 Vanilla Anchor Algorithm
The anchor algorithm computes the topic matrix A, where Av,k is the conditional probability of observing word v given topic k, e.g., the probability of seeing the word “lens” given the camera topic in a corpus of Amazon product reviews. Arora et al. (2012) finds these probabilities by assuming that every topic contains at least one ‘anchor’ word which has a non-zero probability only in that topic. Anchor words make computing the topic matrix A tractable because the occurrence pattern of the anchor word mirrors the occurrence pattern of the topic itself.
To recover the topic matrix A using anchor words, we first compute a V × V cooccurrence matrix Q, where Qi,j is the conditional probability p(wj |wi) of seeing word type wj after having seen wi in the same document. A form of the Gram-Schmidt process on Q finds anchor words {g1 . . . gk} (Arora et al., 2013).
Once we have the set of anchor words, we can compute the probability of a topic given a word (the inverse of the conditioning in A). This coefficient matrix C is defined row-wise for each word i
C∗i,· = argmin Ci,· DKL
( Qi,· ∥∥∥∥ K∑ k=1 Ci,kQgk,· ) ,
(1) which gives the best reconstruction (based on Kullback-Leibler divergence DKL) of non-anchor words given anchor words’ conditional probabilities. For example, in our product review data, a word such as “battery” is a convex combination of the anchor words’ contexts (Qgk,·) such as “camera”, “phone”, and “car”. Solving each row of C is fast and is embarrassingly parallel. Finally, we apply Bayes’ rule to recover the topic matrix A from the coefficient matrix C.
The anchor algorithm can be orders of magnitude faster than probabilistic inference. The construction of Q requires only a single pass over the data and can be pre-computed for interactive use-cases. Once Q is constructed, topic inference scales with the size of Q which, in turn, depends on the square of the vocabulary size V . In contrast, traditional topic model inference typically requires multiple passes over the entire data. Techniques such as Online LDA (Hoffman et al., 2010) or Stochastic Variation Inference (Hoffman et al., 2013) could improve this to a single pass over the
Anchor Top Words in Topics backpack backpack camera lens bag room carry fit cameras equipment comfortable camera camera lens pictures canon digital lenses batteries filter mm photos bag bag camera diaper lens bags genie smell
room diapers odor
Table 1: Three separate attempts to construct a topic concerning camera bags in Amazon product reviews with single word anchors. The term “backpack” is a good anchor because it uniquely identifies the topic. However, both “camera” and “bag” are poor anchors for this topic.
entire data. However, even if such techniques were to be adapted to incorporate human guidance, a single pass is not tractable for interactive use.
2 Tandem Anchor Extension
Single word anchors can be opaque to users. For an example of bewildering anchor words, consider a camera bag topic from a collection of Amazon product reviews (Table 1). The anchor word “backpack” may seem strange. However, this dataset contains nothing about regular backpacks; thus, “backpack” is unique to camera bags. Bizarre, low-to-mid frequency words are often anchors because anchor words must be unique to a topic; intuitive or high-frequency words cannot be anchors if they have probability in any other topic.
If we instead ask users to give us representative words for this topic, we would expect combinations of words like “camera” and “bag.” However, with single word anchors we must choose a single word to anchor each topic. Unfortunately, because these words might appear in multiple topics, individually they are not suitable as anchor words. The anchor word “camera” generates a general camera topic instead of camera bags, and the topic anchored by “bag” includes bags for diaper pails (Table 1).
Instead, we need to use sets of representative terms as an interpretable, parsimonious description of a topic. This section discusses strategies to build anchors from multiple words and the implications of using multiword anchors to recover topics. This extension not only makes anchors more interpretable but also enables users to manually construct effective anchors in interactive topic modeling settings.
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
2.1 Anchor Facets
We first need to turn words into an anchor. If we interpret the anchor algorithm geometrically, each row of Q represents a word as a point in V -dimensional space. We then model each point as a convex combination of anchor words to reconstruct the topic matrix A (Equation 1). Instead of individual anchor words (one anchor word per topic), we use anchor facets, or sets of words that describe a topic. The facets for each anchor form a new pseudoword, or an invented point in V -dimensional space (described in more detail in Section 2.2).
While these new points do not correspond to words in the vocabulary, we can express nonanchor words as convex combinations of pseudowords. To construct these pseudowords from their facets, we combine the co-occurrence profiles of the facets. These pseudowords then augment the original cooccurrence matrix Q with K additional rows corresponding to synthetic pseudowords forming each of K multiword anchors. We refer to this augmented matrix as S. The rest of the anchor algorithm proceeds unmodified.
Our augmented matrix S is therefore a (V + K) × V matrix. As before, V is the number of token types in the data and K is the number of topics. The first V rows of S correspond to the V token types observed in the data, while the additionalK rows correspond to the pseudowords constructed from anchor facets. Each entry of S encodes conditional probabilities so that Si,j is equal to p(wi |wj). For the additionalK rows, we invent a cooccurrence pattern that can effectively explain the other words’ conditional probabilities.
2.2 Combining Facets into Pseudowords
We now describe more concretely how to combine an anchor facets to describe the cooccurrence pattern of our new pseudoword anchor. In tandem anchors, we create vector representations that combine the information from anchor facets. Our anchor facets are G1 . . .GK , where Gk is a set of anchor facets which will form the kth pseudoword anchor. The pseudowords are g1 . . . gK , where gk is the pseudoword from Gk. These pseudowords form the new rows of S. We give several candidates for combining anchors facets into a single multiword anchor; we compare their performance in Section 3.
Vector Average An obvious function for com-
puting the central tendency is the vector average. For each anchor facet,
Sgk,j = ∑ i∈Gk Si,j |Gk| , (2)
where |Gk| is the cardinality of Gk. Vector average makes the pseudoword Sgk,j more central, which is intuitive but inconsistent with the interpretation from Arora et al. (2013) that anchors should be extreme points whose linear combinations explain more central words.
Or-operator An alternative approach is to consider a cooccurrence with any anchor facet in Gk. For word j, we use De Morgan’s laws to set
Sgk,j = 1− ∏ i∈Gk (1− Si,j). (3)
Unlike the average, which pulls the pseudoword inward, this or-operator pushes the word outward, increasing each of the dimensions. Increasing the volume of the simplex spanned by the anchors explains more words.
Element-wise Min Vector average and oroperator are both sensitive to outliers and cannot account for polysemous anchor facets. Returning to our previous example, both “camera” and “bag” are bad anchors for camera bags because they appear in documents discussing other products. However, if both “camera” and “bag” are anchor facets, we can look at an intersection of their contexts: words that appear with both. Using the intersection, the cooccurrence pattern of our anchor facet will only include terms relevant to camera bags.
Mathematically, this is an element-wise min operator,
Sgk,j = min i∈Gk Si,j . (4)
This construction, while perhaps not as simple as the previous two, is robust to words which have cooccurrences which are not unique to a single topic.
Harmonic Mean Leveraging the intuition that we should use a combination function which is both centralizing (like vector average) and ignores large outliers (like element-wise min), the final combination function is the element-wise harmonic mean. Thus, for each anchor facet
Sgk,j = ∑ i∈Gk ( S−1i,j |Gk| )−1 . (5)
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Since the harmonic mean tends towards the lowest values in the set, it is not sensitive to large outliers, giving us robustness to polysemous words.
2.3 Finding Topics
After constructing the pseudowords of S we then need to find the coefficients Ci,k which describe each word in our vocabulary as a convex combination of the multiword anchors. Like standard anchor methods, we solve the following for each token type:
C∗i,· = argmin Ci,· DKL
( Si,· ∥∥∥∥ K∑ k=1 Ci,kSgk,· ) .
(6) Finally, we appeal to Bayes’ rule, we recover the topic-word matrix A from the coefficients of C.
3 High Water Mark for Tandem Anchors
Before addressing interactivity, we apply tandem anchors to real world data, but with anchors gleaned from metadata. Our purpose is twofold. First, we determine which combiner from Section 2.2 to use in our interactive experiments in Section 4 and second, we confirm that well-chosen tandem anchors can improve topics. In addition, we examine running time of tandem anchors and compare to traditional model-based interactive topic modeling techniques. We cannot assume that we will have metadata available to build tandem anchors, but we use them here because they provide a high water mark without the variance introduced by study participants.
3.1 Experimental Setup
We use the well-known 20 Newsgroups dataset (20NEWS) used in previous interactive topic modeling work: 18,846 usenet postings from 20 different newgroups in the early 1990s. We remove stopwords and words which appear in fewer than 100 documents or more than 1,500 documents.
To seed the tandem anchors, we use the titles of newsgroups. To build each multiword anchor facet, we split the title on word boundaries and expand any abbreviations or acronyms. For example, the newsgroup title ‘comp.os.mswindows.misc’ becomes {“computer”, “operating”, “system”, “microsoft”, “windows”, “miscellaneous”}. We do not fully specify the topic; the title gives some intuition, but the topic modeling algorithm must still recover the complete
topic-word distributions. This is akin to knowing the names of the categories used but nothing else. Critically, the topic modeling algorithm has no knowledge of document-label relationships.
3.2 Experimental Results
Our first evaluation is a classification task to predict documents’ newsgroup membership. We do not aim for state-of-the-art accuracy, but the experiment shows title-based tandem anchors yield topics closer to the underlying classes than GramSchmidt anchors. After randomly splitting the data into test and training sets we learn topics from the test data using both the title-based tandem anchors and the Gram-Schmidt single word anchors.1 For multiword anchors, we use each of the proposed combiner functions from Section 2.2. The anchor algorithm only gives the topic-word distributions and not word-level topic assignments, we infer token-level topic assignments using Latent Dirichlet Allocation (Blei et al., 2003) with fixed topics discovered by the anchor method. We use our own implementation of Gibbs sampling with fixed topics and a symmetric document-topic Dirichlet prior with concentration α = .01. Since the topics are fixed, this inference is very fast and can be parallelized on a perdocument basis. We then train a hinge-loss linear classifier on the newsgroup labels using Vowpal Wabbit2 with topic-word pairs as features. Finally, we infer topic assignments in the test data and evaluate the classification using those topicword features. For both training and test, we exclude words outside the LDA vocabulary.
The topics created from multiword anchor facets are more accurate than Gram-Schmidt topics (Figure 1). This is true regardless of the combiner function. However, harmonic mean is more accurate than the other functions.3
Since 20NEWS has twenty classes, accuracy alone does not capture confusion between closely related newsgroups. For example, accuracy penalizes a classifier just as much for label-
1With fixed anchors and data the anchor algorithm is deterministic, so we use random splits instead of the standard train/test splits so that we can compute variance.
2http://hunch.net/˜vw/ 3Significant at p < 0.01/4 when using two-tailed t-tests with a Bonferroni correction. For each of our evaluations, we verify the normality of our data (D’Agostino and Pearson, 1973) and use two-tailed t-tests with Bonferroni correction to determine whether the differences between the different methods are significant.
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
●●
● ●● ●
● ●
●
● ● ●
●●
● ●
●
● ●● ●
● ●●
Accuracy ARI F−Measure VI Coherence
Gram−Schmidt
Title+Average
Title+Or
Title+Min
Title+HMean
0. 55
0. 60
0. 65
0. 70
0. 30 0. 35 0. 40 0. 45 0. 50
0. 56 0. 60 0. 64 0. 68 0. 72 2. 4 2. 7 3. 0 3. 3 3. 6 − 23 0 − 22 5 − 22 0 − 21 5 − 21 0
Figure 1: Using metadata can improve anchor-based topic models. For all metrics, the unsupervised Gram-Schmidt anchors do worse than creating anchors based on Newsgroup titles (for all metrics except VI, higher is better). For coherence, Gram-Schmidt does better than two functions for combining anchor words, but not the element-wise min or harmonic mean.
ing a document from ‘rec.sport.baseball’ with ‘rec.sport.hockey’ as with ‘alt.atheism’ despite the similarity between sports newsgroups. Consequently, after building a confusion matrix between the predicted and true classes, external clustering metrics reveal confusion between classes.
The first clustering metric is the adjusted Rand index (Yeung and Ruzzo, 2001), which is akin to accuracy for clustering, as it gives the percentage of correct pairing decisions from a reference clustering. Adjusted Rand index also accounts for chance groupings of documents. Next we use Fmeasure, which also considers pairwise groups, balancing the contribution of false negatives, but without the true negatives. Finally, we use variation of information. This metric measures the amount of information lost by switching from the gold standard labels to the predicted labels (Meilă, 2003). Since we are measuring the amount of information lost, lower variation of information is better.
Based on these clustering metrics, tandem anchors can yield superior topics to those created using single word anchors (Figure 1). As with accuracy, this is true regardless of which combination function we use. Furthermore, harmonic mean produces the least confusion between classes.4
The final evaluation is topic coherence by Newman et al. (2010), which measures whether the topics make sense, and correlates with human judgments of topic quality. Given V , the set of
4Significant at p < 0.01/4 when using two-tailed t-tests with a Bonferroni correction. For each of our evaluations, we verify the normality of our data (D’Agostino and Pearson, 1973) and use two-tailed t-tests with Bonferroni correction to determine whether the differences between the different methods are significant.
the n most probable words of a topic, coherence is
∑ v1,v2∈V log D(v1, v2) + D(v2) (7)
where D(v1, v2) is the co-document frequency of word types v1 and v2, and D(v2) is the document frequency of word type v2. A smoothing parameter prevents zero logarithms.
Figure 1 also shows topic coherence. Although title-based anchor facets produce better classification features, topics from Gram-Schmidt anchors have better coherence than title-based anchors with the vector average or the or-operator. However, when using the harmonic mean combiner, title-based anchors produce the most human interpretable topics.4
Harmonic mean beats other combiner functions because it is robust to ambiguous or irrelevant term cooccurrences an anchor facet. Both the vector average and the or-operator are swayed by large outliers, making them sensitive to ambiguous terms in an anchor facet. Element-wise min also has this robustness, but harmonic mean is also able to better characterize anchor facets as it has more centralizing tendency than the min.
3.3 Runtime Considerations
Tandem anchors will enable users to direct topic inference to improve topic quality. However, for the algorithm to be interactive we must also consider runtime. Cook and Thomas (2005) argue that for interactive applications with user-initiated actions like ours the response time should be less than ten seconds. Longer waits can increase the cognitive load on the user and harm the user interaction.
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Algorithm #Doc Runtime (seconds) Tandem Anchors 18828 2.13
Fast ITM 13284 24.8 Utopian 515 48.0
Table 2: Reported run times for various interactive topic modeling algorithms. Only tandem anchors is fast enough to be interactive. For Tandem Anchors, we ran on a single core of an AMD Phemon II X6 1090T processor. For Fast ITM, we use the time reported by Hu and Boyd-Graber (2012). For Utopian, we use the time published by Choo et al. (2013).
Fortunately, the runtime of tandem anchors is amenable to interactive topic modeling. Our run time results are summarized in Table 2. On 20NEWS, interactive updates take roughly two seconds. Furthermore, larger datasets typically have a sublinear increase in distinct word types, so we can expect to see similar run times, even on much larger datasets.
Compared to other interactive topic modeling algorithms, tandem anchors has a very attractive run time. For example, using an optimized version of the sampler for the Interactive Topic Model described by Hu and Boyd-Graber (2012), and the recommended 30 iterations of sampling, the Interactive Topic Model is well beyond our desired update time for interactive use and an order of magnitude slower than tandem anchors.
Another promising interactive topic modeling approach is Utopian (Choo et al., 2013), which uses non-negative factorization, albeit without the benefit of anchor words. Utopian is much slower than tandem anchors. Even on the small InfoVisVAST dataset, Utopian takes nearly a minute to converge. Furthermore, since Utopian scales linearly with the size of the data, for moderately sized datasets such as 20NEWS, Utopian is infeasible for interactive topic modeling due to run time.
While each of these interactive topic modeling algorithms do achieve reasonable topics, only our algorithm fits the run time requirements for interactivity. Furthermore, since tandem anchors scales with the size of the vocabulary rather than the size of the data, this trend will only become more pronounced as we increase the amount of data.
4 Interactive Anchor Words
Given high quality anchor facets, the tandem anchor algorithm can produce high quality topic
models (particularly when the harmonic mean combiner is used). Furthermore, the tandem anchor algorithm is fast enough to be interactive (as opposed to model-based approaches such as the Interactive Topic Model). We now turn our attention to our main experiment: tandem anchors applied to the problem of interactive topic modeling. We compare both single word and tandem anchors in our study. We do not include the Interactive Topic Model or Utopian, as their run times are too slow for our users.
4.1 Interface and User Study
To show that interactive tandem anchor words are fast, effective, and intuitive, we ask users to understand a dataset using the anchor word algorithm. For this user study, we recruit twenty participants drawn from a university student body. The student median age is twenty-two. Seven are female, and thirteen are male. None of the students had any prior familiarity with topic modeling or the 20NEWS dataset.
Each participant sees a simple user interface (Figure 2) with topic given as a row with two columns. The left column allows users to view and edit topics’ anchor words; the right column lists the most probable words in each topic.5 The user can remove an anchor word or drag words from the topic word lists (right column) to become an anchor word. Users can also add additional topics by clicking the “Add Anchor” to create additional anchors. If the user wants to add a word to a tandem anchor set that does not appear in the interface, they manually type the word (restricted to the model’s vocabulary). When the user wants to see the updated topics for their newly refined anchors,
5While we use topics generated using harmonic mean for our final analysis, users were shown topics generated using the min combiner. However, this does not change our result.
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
they click “Update Topics”. We give each a participant a high level overview of topic modeling. We also describe common problems with topic models including intruding topic words, duplicate topics, and ambiguous topics. Users are instructed to use their best judgement to determine if topics are useful. The task is to edit the anchor words to improve the topics. We asked that users spend at least twenty minutes, but no more than thirty minutes. We repeat the task twice: once with tandem anchors, and once with single word anchors.6
4.2 Quantitative Results
We now validate our main result that for interactive topic modeling, tandem anchors yields better topics than single word anchors. Like our titlebased experiments in Section 3, topics generated from users become features to train and test a classifier for the 20NEWS dataset. Based on our results with title-based anchors, we use the harmonic mean combiner in our analysis. As before, we report not only accuracy, but also multiple clustering metrics using the confusion matrix from the classification task. Finally, we report topic coherence.
Figure 3 summarizes the results of our quantitative evaluation. While we only compare user generated anchors in our analysis, we include the unsupervised Gram-Schmidt anchors as a baseline. Some of the data violate assumptions of normality. Therefore, we use Wilcoxon’s signed-rank test (Wilcoxon, 1945) to determine if the differences between multiword anchors and single word anchors are significant.
Topics from user generated multiword anchors yield higher classification accuracy (Figure 3). Not only is our approach more scalable than the Interactive Topic Model, but we also achieve higher classification accuracy than Hu et al. (2014).7 Tandem anchors also improve clustering metrics.8
While user selected tandem anchors produce better classification features than single word anchors, users selected single word anchors produce topics with similar topic coherence scores.9
6The order in which users complete these tasks is counterbalanced.
7However, the values are not strictly comparable, as Hu et al. (2014) use the standard chronological test/train fold, and we use random splits.
8Significant at p < 0.01 when using Wilcoxon’s signedrank test.
9The difference between coherence scores was not statis-
To understand this phenomenon, we use quality metrics (AlSumait et al., 2009) for ranking topics by their correspondence to genuine themes in the data. Significant topics are likely skewed towards a few related words, so we measure the distance of each topic-word distribution from the uniform distribution over words. Topics which are close to the underlying word distribution of the entire data are likely to be vacuous, so we also measure the distance of each topic-word distribution from the underlying word distribution. Finally, background topics are likely to appear in a wide range of documents, while meaningful topics will appear in a smaller subset of the data.
Figure 4 reports our topic significance findings. For all three significance metrics, multiword anchors produce more significant topics than single word anchors.8Topic coherence is based solely on the top n words of a topic, while both accuracy and topic significance depend on the entire topicword distributions. With single word anchors, topics with good coherence may still be too general. Tandem anchors enables users to produce topics with more specific word distributions which are better features for classification.
4.3 Qualitative Results
We examine the qualitative differences between how users select multiword anchor facets versus single word anchors. Table 3 gives examples of topics generated using different anchor strategies. In a follow-up survey with our users, 75% find it easier to affect individual changes in the topics using tandem anchors compared to single word anchors. Users who prefer editing multiword anchors over single word anchors often report that multiword anchors make it easier to merge similar topics into a single focused topic by combining anchors. For example, by combining multiple words related to Christianity, users were able to create a topic which is highly specific, and differentiated from general religion themes which included terms about Atheism and Judaism.
While users find that use tandem anchors is easier, only 55% of our users say that they prefer the final topics produced by tandem anchors compared to single word anchors. This is in harmony with our quantitative measurements of topic coherence, and may be the result of our stopping criteria: when users judged the topics to be useful.
tically significant using Wilcoxon’s signed-rank test.
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
● ●● ● ●●
●
Accuracy ARI F−Measure VI Coherence
Tandem
Singleword
Gram−Schmidt
0. 55
0. 60
0. 65
0. 70 0. 30 0. 35 0. 40 0. 45 0. 50 0. 55
0. 60
0. 65
0. 70 2. 7 3. 0 3. 3 3. 6 − 24 0 − 23 0 − 22 0 − 21 0 − 20 0 − 19 0
Figure 3: Classification accuracy and coherence using topic features gleaned from user provided multiword and single word anchors. Grahm-Schmidt anchors are provided as a baseline. For all metrics except VI, higher is better. Except for coherence, multiword anchors are best.
●
●●●
●
●●●
●
●●●
uniform vacuous background
Tandem
Singleword
Gram−Schmidt
1 2 3
0. 5
1. 0
1. 5
2. 0
0. 5
1. 0
1. 5
2. 0
Figure 4: Topic significance for both single word and multiword anchors. In all cases higher is better. Multiword anchors produce topics which are more significant than single word anchors.
Anchor Top Words in Topic Automatic Gram Schmidt love love god evolution romans heard car game game games team hockey baseball heard Interactive Single-word evolution evolution theory science faith quote facts religion religion god government state jesus israel baseball baseball games players word teams car hockey hockey team play games season players Interactive Tandem atheism god exists prove god science evidence reason faith objective christian jesus jesus christian christ church bible christians jew israel israel jews jewish israeli state religion baseball bat ball hit baseball ball player games call hockey nhl team hockey player nhl win play
Table 3: Comparison of topics generated for 20NEWS using various types of anchor words. Users are able to combine words to create more specific topics with tandem anchors.
However, 100% of our users feel that the topics created through interaction were better than those generated from Gram-Schmidt anchors. This was true regardless of whether we used tandem anchors or single word anchors.
Our participants also produce fewer topics when using multiword anchors. The mean difference between topics under single word anchors and multiple word anchors is 9.35. In follow up interviews, participants indicate that the easiest way to resolve an ambiguous topic with single word anchors was to create a new anchor for each of the ambiguous terms, thus explaining the proliferation of topics for single word anchors. In contrast, fixing an ambiguous tandem anchor is simple: users just add more terms to the anchor facet.
5 Conclusion
Tandem anchors extend the anchor words algorithm to allow multiple words to be combined into anchor facets. For interactive topic modeling, using anchor facets in place of single word anchors produces higher quality topic models and are more intuitive to use. Furthermore, our approach scales much better than existing interactive topic modeling techniques, allowing interactivity on large datasets for which interactivity was previous impossible.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
- Strengths: Clear description of methods and evaluation Successfully employs and interprets a variety of evaluations Solid demonstration of practicality of technique in real-world interactive topic modeling  - Weaknesses: Missing related work on anchor words Evaluation on 20 Newsgroups is not ideal Theoretical contribution itself is small   - General Discussion: The authors propose a new method of interactive user specification of topics called Tandem Anchors. The approach leverages the anchor words algorithm, a matrix-factorization approach to learning topic models, by replacing the individual anchors inferred from the Gram-Schmidt algorithm with constructed anchor pseudowords created by combining the sparse vector representations of multiple words that for a topic facet. The authors determine that the use of a harmonic mean function to construct pseudowords is optimal by demonstrating that classification accuracy of document-topic distribution vectors using these anchors produces the most improvement over Gram-Schmidt. They also demonstrate that their work is faster than existing interactive methods, allowing interactive iteration, and show in a user study that the multiword anchors are easier and more effective for users.  Generally, I like this contribution a lot: it is a straightforward modification of an existing algorithm that actually produces a sizable benefit in an interactive setting. I appreciated the authors  efforts to evaluate their method on a variety of scales. While I think the technical contribution in itself is relatively small (a strategy to assemble pseudowords based on topic facets) the thoroughness of the evaluation merited having it be a full paper instead of a short paper. It would have been nice to see more ideas as to how to build these facets in the absence of convenient sources like category titles in 20 Newsgroups or when initializing a topic model for interactive learning.  One frustration I had with this paper is that I find evaluation on 20 Newsgroups to not be great for topic modeling: the documents are widely different lengths, preprocessing matters a lot, users have trouble making sense of many of the messages, and naive bag-of-words models beat topic models by a substantial margin. Classification tasks are useful shorthand for how well a topic model corresponds to meaningful distinctions in the text by topic; a task like classifying news articles by section or reviews by the class of the subject of the review might be more appropriate. It would also have been nice to see a use case that better appealed to a common expressed application of topic models, which is the exploration of a corpus.  There were a number of comparisons I think were missing, as the paper contains little reference to work since the original proposal of the anchor word model. In addition to comparing against standard Gram-Schmidt, it would have been good to see the method from Lee et. al. (2014),  Low-dimensional Embeddings for Interpretable Anchor-based Topic Inference d. I also would have liked to have seen references to Nguyen et. al. (2013),  Evaluating Regularized Anchor Words d and Nguyen et. al. (2015)  Is Your Anchor Going Up or Down? Fast and Accurate Supervised Topic Models d, both of which provide useful insights into the anchor selection process.  I had some smaller notes: - 164:  u2026entire dataset - 164-166: I m not quite sure what you mean here. I think you are claiming that it takes too long to do one pass? My assumption would have been you would use only a subset of the data to retrain the model instead of a full sweep, so it would be good to clarify what you mean. - 261&272: any reason you did not consider the and operator or element-wise max? They seem to correspond to the ideas of union and intersection from the or operator and element-wise min, and it wasn t clear to me why the ones you chose were better options. - 337: Usenet should be capitalized - 338-340: Why fewer than 100 (as that is a pretty aggressive boundary)? Also, did you remove headers, footers, and or quotes from the messages? - 436-440: I would have liked to see a bit more explanation of what this tells us about confusion. - 692: using tandem anchors  Overall, I think this paper is a meaningful contribution to interactive topic modeling that I would like to see available for people outside the machine learning community to investigate, classify, and test hypotheses about their corpora.  POST-RESPONSE: I appreciate the thoughtful responses of the authors to my questions. I would maintain that for some of the complimentary related work that it's useful to compare to non-interactive work, even if it does something different.