Paper:
Here’s My Point: Argumentation Mining with Pointer Networks
Argumentation mining seeks to uncover the argument structure present in argumentative text. In order to determine this structure, one must understand how different individual components of the overall argument are linked. General consensus in this field dictates that the argument components form a hierarchy of persuasion, which manifests itself in a tree structure. This work provides the first neural network-based approach to argumentation mining, focusing on the dual tasks of extracting links between argument components, and classifying types of argument components. We propose to use a joint model based on a Pointer Network architecture to simultaneously solve these tasks. In doing so, we construct a joint model that simultaneously attempts to learn the type of argument component, as well as continuing to predict links between argument components. The proposed joint model achieves state-of-the-art results on two separate evaluation corpora, achieving far superior performance than a regular Pointer Network model. Our results show that optimizing for both tasks is crucial for high performance.
1 Introduction
Computational approaches to argument mining/understanding have become very popular (Persing and Ng, 2016; Cano-Basave and He, 2016; Wei et al., 2016; Ghosh et al., 2016; Palau and Moens, 2009; Habernal and Gurevych, 2016). An important avenue in this work is to understand the structure in argumentative text (Persing and Ng, 2016; Peldszus and Stede, 2015; Stab
and Gurevych, 2016; Nguyen and Litman, 2016). One fundamental assumption when working with argumentative text is the presence of Arguments Components (ACs). The types of ACs are generally characterized as a claim or a premise (Govier, 2013), with premises acting as support (or possibly attack) units for claims. To model more complex structures of arguments, some annotation schemes also include a major claim AC type (Stab and Gurevych, 2016, 2014b).
Generally, the task of processing argument structure encapsulates four distinct subtasks: 1) Given a sequence of tokens that represents an entire argumentative text, determine the token subsequences that constitute non-intersecting ACs; 2) Given an AC, determine the type of AC (claim, premise, etc.); 3) Given a set/list of ACs, determine which ACs have directed links that encapsulate overall argument structure; 4) Given two linked ACs, determine whether the link is a supporting or attacking relation. In this work, we focus on subtasks 2 and 3.
There are two key assumptions our work makes going forward. First, we assume subtask 1 has been completed, i.e. ACs have already been identified. Second, we follow previous work that assumes a tree structure for the linking of ACs (Palau and Moens, 2009; Cohen, 1987; Peldszus and Stede, 2015; Stab and Gurevych, 2016). Specifically, a given AC can only have a single outgoing link, but can have numerous incoming links. Furthermore, there is a ‘head’ component that has no outgoing link (the top of the tree). However, we note that in a broader sense, depending on the corpus (see Section 4), an argument structure can be either a single tree or a forest, consisting of multiple trees. Figure 1 shows an example that we will use throughout the paper to concretely explain how our approach works. First, the left side of the figure presents the raw text of a paragraph in a per-
2
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
First, [ ::::::: cloning ::::: will ::: be :::::::::: beneficial :::: for ::::: many ::::::: people :::: who :::: are ::: in ::::: need ::: of ::::: organ ::::::::: transplants]AC1. In addition, [it shortens the healing process]AC2. Usually, [it is very rare to find an appropriate organ donor]AC3 and [by using cloning in order to raise required organs the waiting time can be shortened tremendously]AC4.
AC1 Claim
AC2 Premise
AC4 Premise AC3 Premise
Figure 1: An example of argument structure with four ACs. The left side shows raw text that has been annotated for the presence of ACs. Squiggly and straight underlining means an AC is a claim or premise, respectively. The ACs in the text have also been annotated for links to other ACs, which is show in the right figure. ACs 3 and 4 are premises that link to another premise, AC2. Finally, AC2 links to a claim, AC1. AC1 therefore acts as the central argumentative component.
suasive essay (Stab and Gurevych, 2016), with the ACs contained in square brackets. Squiggly verse straight underlining differentiates between claims and premises, respectively. The ACs have been annotated as to how they are linked, and the right side of the figure reflects this structure. The argument structure with four ACs forms a tree, where AC2 has two incoming links, and AC1 acts as the head, with no outgoing links. We also specify the type of AC, with the head AC marked as a claim and the remaining ACs marked as premises. Lastly, we note that the order of arguments components can be a strong indicator of how components should related. Linking to the first argument component can provide a competitive baseline heuristic (Peldszus and Stede, 2015; Stab and Gurevych, 2016).
Given the task at hand, we propose a modification of a Pointer Network (PN) (Vinyals et al., 2015b). A PN is a sequence-to-sequence model that outputs a distribution over the encoding indices at each decoding timestep. The PN is a promising model for link extraction in argumentative text because it inherently possesses three important characteristics: 1) it is able to model the sequential nature of ACs; 2) it constrains ACs to have a single outgoing link, thus partly enforcing the tree structure; 3) the hidden representations learned by the model can be used for jointly predicting multiple subtasks. We also note that since a PN is a type of sequence-to-sequence model (Sutskever et al., 2014), it allows the entire sequence to be seen before making prediction. This is important because if the problem were to be approached as standard sequence modeling (Graves and Schmidhuber, 2009; Robinson, 1994), mak-
ing predictions at each forward timestep, it would only allow links to ACs that have already been seen. This is equivalent to only allowing backward links. We note that we do test a simplified model that only uses hidden states from an encoding network to make predictions, as opposed to the sequence-to-sequence architecture present in the PN (see Section 5).
PNs were originally proposed to allow a variable length decoding sequence (Vinyals et al., 2015b). Alternatively, the PN we implement differs from the original model in that we decode for the same number of timesteps as there are input components. We also propose a joint PN for both extracting links between ACs and predicting the type of AC. The model uses the hidden representation of ACs produced during the encoding step (see Section 3.4). Aside from the partial assumption of tree structure in the argumentative text, our models do not make any additional assumptions about the AC types or connectivity, unlike the work of (Peldszus, 2014). Lastly, in respect to the broad task of parsing, our model is flexible because it can easily handle non-projective, multiroot dependencies. We evaluate our models on the corpora of (Stab and Gurevych, 2016) and (Peldszus, 2014), and compare our results with the results of the aformentioned authors.
2 Related Work
In this section, we discuss previous argumentation mining work that deals with the subtask pipeline we discussed in the previous section. This can be labeled as a ‘micro’ approach to argumentation mining (Stab and Gurevych, 2016) (in con-
3
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
trast, there has been a number of efforts to identify argument structure at a higher level (Boltuzic and Šnajder, 2014; Ghosh et al., 2014; Cabrio and Villata, 2012)). Palau and Moens (2009) is an early work in this area, using a hand-crafted ContextFree Grammar to determine the structure of ACs in a corpus of legal texts. Lawrence et al. (2014) leverage a topic modeling-based AC similarity to uncover tree-structured arguments in philosophical texts. Unlike these methodologies, recent work offers data-driven approaches to the task of predicting links between ACs. Stab and Gurevych (2014b) approach the task as a binary classification problem. The authors train an SVM with various semantic and structural features. Peldszus and Stede have also used classification models for predicting the presence of links (2015).
Various authors have also proposed to jointly model link extraction with other subtasks from the argumentation mining pipeline, using either a Integer Linear Programming (ILP) framework (Persing and Ng, 2016; Stab and Gurevych, 2016) or directly feeding previous subtask predictions into a tree-based parser. The former joint approaches are evaluated on annotated corpora of persuasive essays (Stab and Gurevych, 2014a, 2016), and the latter on a corpus of microtexts (Peldszus, 2014). The ILP framework is effective in enforcing a tree structure between ACs when predictions are made from otherwise naive base classifiers.
Unrelated to argumentation mining specifically, recurrent neural networks have previously been proposed to model tree/graph structures in a linear manner. (Vinyals et al., 2015c) use a sequenceto-sequence model for the task of syntactic parsing. The authors linearize input parse graphs using a depth-first search, allowing it to be consumed as a sequence, achieving state-of-the-art results on several syntactic parsing datasets. (Bowman et al., 2015) experiment on an artificial entailment dataset that is specifically engineered to capture recursive logic (Bowman et al., 2014). The text is annotated with brackets, in an original attempt to provide easy input into a recursive neural network. However, standard recurrent neural networks can take in complete sentence sequences, brackets included, and perform competitively with a recursive neural network.
3 Pointer Network for Link Extraction
In this section we will describe how we use a PN for the problem of extracting links between ACs. We begin by giving a general description of the PN model.
3.1 Pointer Network
A PN is a sequence-to-sequence model (Sutskever et al., 2014) with attention (Bahdanau et al., 2014) that was proposed to handle decoding sequences over the encoding inputs, and can be extended to arbitrary sets (Vinyals et al., 2015a). The original motivation for a pointer network was to allow networks to learn solutions to algorithmic problems, such as the traveling salesperson and convex hull, where the solution is a sequence over candidate points. The PN model is trained on input/output sequence pairs (E,D), where E is the source and D is the target (our choice of E,D is meant to represent the encoding, decoding steps of the sequence-to-sequence model). Given model parameters Θ, we apply the chain rule to determine the probability of a single training example:
p(D|E; Θ) = m(E)∏ i=1 p(Di|D1, ..., Di−1, E; Θ)
(1) where the function m signifies that the number of decoding timesteps is a function of each individual training example. We will discuss shortly why we need to modify the original definition of m for our application. By taking the log-likelihood of Equation 1, we arrive at the optimization objective:
Θ∗ = arg max Θ ∑ E,D log p(D|E; Θ) (2)
which is the sum over all training example pairs. The PN uses Long Short-Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) for sequential modeling, which produces a hidden layer h at each encoding/decoding timestep. In practice, the PN has two separate LSTMs, one for encoding and one for decoding. Thus, we refer to encoding hidden layers as e, and decoding hidden layers as d.
The PN uses a form of content-based attention (Bahdanau et al., 2014) to allow the model to produce a distribution over input elements. This can also be thought of as a distribution over input indices, wherein a decoding
4
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
Component 1 Component 2 Component 3 Component 4
LSTM D1
LSTM D2
LSTM D3
LSTM D4 LSTM E1 LSTM E2 LSTM E3 LSTM E4
Figure 2: Applying a Pointer Network to the example paragraph in Figure 1 with LSTMs unrolled over time.
step ‘points’ to the input. Formally, given encoding hidden states (e1, ..., en), the model calculates p(Di|D1, ..., Di−1, E) as follows:
uij = v T tanh(W1ej +W2di) (3)
p(Di|D1, ..., Dj−1, E) = softmax(ui) (4)
where matrices W1, W2 and vector v are parameters of the model (along with the LSTM parameters used for encoding and decoding). In Equation 3, prior to taking the dot product with v, the resulting transformation can be thought of as creating a joint, hidden representation of inputs i and j. Vector ui in equation 4 is of length n, and index j corresponds to input element j. Therefore, by taking the softmax of ui, we are able to create a distribution over the input.
3.2 Link Extraction as Sequence Modeling
A given piece of text has a set of ACs, which occur in a specific order in the text, (C1, ..., Cn). Therefore, at encoding timestep i, the model is fed a representation of Ci. Since the representation is large and sparse (see Section 3.3 for details on how we represent ACs), we add a fully-connected layer before the LSTM input. Given a representation Ri for AC Ci the LSTM input Ai becomes:
Ai = σ(WrepRi + brep) (5)
where Wrep, brep in turn become model parameters, and σ is the sigmoid function1. Similarly, the decoding network applies a fully-connected layer with sigmoid activation to its inputs, see Figure 3. At encoding step i, the encoding LSTM produces hidden layer ei, which can be thought of as a hidden representation of AC Ci.
1We also experimented with relu and elu activations, but found sigmoid to yield the best performance.
In order to make the PN applicable to the problem of link extraction, we explicitly set the number of decoding timesteps to be equal to the number of input components. Using notation from Equation 1, the decoding sequence length for an encoding sequence E is simply m(E) = |{C1, ..., Cn}|, which is trivially equal to n. By constructing the decoding sequence in this manner, we can associate decoding timestep i with AC Ci.
From Equation 4, decoding timestep Di will output a distribution over input indices. The result of this distribution will indicate to which AC component Ci links. Recall there is a possibility that an AC has no outgoing link, such as if it’s the root of the tree. In this case, we state that if ACCi does not have an outgoing link, decoding step Di will output index i. Conversely, if Di outputs index j, such that j is not equal to i, this implies thatCi has an outgoing link to Cj . For the argument structure in Figure 1, the corresponding decoding sequence is (1, 1, 2, 2). The topology of this decoding sequence is illustrated in Figure 2. Observe how C1 points to itself since it has no outgoing link.
Finally, we note that we modify the PN structure to have a Bidirectional LSTM as the encoder. Thus, ei is the concatenation of forward and backward hidden states −→e i and←−e n−i+1, produced by two separate LSTMs. The decoder remains a standard forward LSTM.
3.3 Representing Argument Components
At each timestep of the decoder, the network takes in the representation of an AC. Each AC is itself a sequence of tokens, similar to the recently proposed Question-Answering dataset (Weston et al., 2015). We follow the work of (Stab and Gurevych, 2016) and focus on three different types of features to represent our ACs: 1) Bag-of-Words of the AC; 2) Embedding representation based on GloVe embeddings (Pennington et al., 2014); 3) Structural features: Whether or not the AC is the first AC in
5
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
Bi-LSTM E1
FC1 FC1 FC1 FC1
Bidirectional LSTM Encoder
Component 1 Component 2 Component 3 Component 4
FC2 FC2 FC2 FC2
Claim Premise Premise Premise
LSTM D1
LSTM D2
LSTM D3
Bi-LSTM E2 Bi-LSTM E3 Bi-LSTM E4
FC3 FC3 FC3
a paragraph, and Whether the AC is in an opening, body, or closing paragraph. See Section 6 for an ablation study of the proposed features.
3.4 Joint Neural Model
Up to this point, we focused on the task of extracting links between ACs. However, recent work has shown that joint models that simultaneously try to complete multiple aspects of the subtask pipeline outperform models that focus on a single subtask (Persing and Ng, 2016; Stab and Gurevych, 2014b; Peldszus and Stede, 2015). Therefore, we will modify the architecture we proposed in Section 3 so that it would allow us to perform AC classification (Kwon et al., 2007; Rooney et al., 2012) together with link prediction. Knowledge of an individual subtask’s predictions can aid in other subtasks. For example, claims do not have an outgoing link, so knowing the type of AC can aid in the link prediction task. This can be seen as a way of regularizing the hidden representations from the encoding component (Che et al., 2015).
Predicting AC type is a straightforward classification task: given AC Ci, we need to predict whether it is a claim or premise. Some annotation schemes also include the class major claim (Stab and Gurevych, 2014a), which means this can be a multi-class classification task. For encoding timestep i, the model creates hidden representation ei. This can be thought of as a representation of AC Ci. Therefore, our joint model will simply pass this representation through a fully connected layer as follows:
zi = Wclsei + bcls (6)
where Wcls, bcls become elements of the model
parameters, Θ. The dimensionality of Wcls, bcls is determined by the number of classes. Lastly, we use softmax to form a distribution over the possible classes.
Consequently, the probability of predicting component type at timestep i is defined as:
p(Ci) = p(Ei| −→ E i, ←− E i; Θ) (7)
p(Ei| −→ E i, ←− E i; Θ) = softmax(zi) (8)
Finally, combining this new prediction task with Equation 2, we arrive at the new training objective:
Θ∗ = arg max Θ α ∑ E,D log p(D|E; Θ)
+(1− α) ∑ E log p(E|Θ) (9)
which simply sums the costs of the individual prediction tasks, and the second summation is the cost for the new task of predicting AC type. α ∈ [0, 1] is a hyperparameter that specifies how we weight the two prediction tasks in our cost function. The architecture of the joint model, applied to our ongoing example, is illustrated in Figure 3.
4 Experimental Design
As we have previously mentioned, our work assumes that ACs have already been identified. That is, the token sequence that comprises a given AC is already known. The order of ACs corresponds directly to the order in which the ACs appear in the text. Since ACs are non-overlapping, there is no
6
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
Type prediction Link prediction Model Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 Base Classifier .794 .891 .611 .879 .717 .508 .917 ILP Joint Model .826 .891 .682 .903 .751 .585 .918 BLSTM .810 .830 .688 .912 .754 .589 .919 PN - - - - .709 .511 .906 Joint Model No FC Input .791 .826 .642 .906 .708 .514 .901 Joint Model .849 .894 .732 .921 .767 .608 .925
Table 1: Results on the Persuasive Essay corpus.
Type prediction Link prediction Model Macro f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 Simple .817 - - .663 .478 .848 Best EG .869 - - .693 .502 .884 MP+p .831 - - .720 .546 .894 Base Classifier .830 .712 .937 .650 .446 .841 ILP Joint Model .857 .770 .943 .683 .486 .881 PN .813 .692 .934 .740 .577 .903
Table 2: Results on the Microtext corpus.
ambiguity in this ordering. We test the effectiveness of our proposed model on a dataset of persuasive essays (Stab and Gurevych, 2016), as well as a dataset of microtexts (Peldszus, 2014). The feature space for the persuasive essay corpus has roughly 3,000 dimensions, and the microtext corpus feature space has between 2,500 and 3,000 dimensions, depending on the data split (see below).
The persuasive essay corpus contains a total of 402 essays, with a frozen set of 80 essays held out for testing. There are three AC types in this corpus: major claim, claim, and premise. Also, in this corpus individual structures can be either trees or forests. We follow the creators of the corpus and only evaluate ACs within a given paragraph. That is, each training/test example is a sequence of ACs from a paragraph. This results in a 1,405/144 training/test split.
The microtext corpus contains 112 short texts. Unlike the persuasive essay corpus, each text in this corpus is itself a complete example, as well as a single tree. Since the dataset is small, the authors have created 10 sets of 5-fold cross-validation, reporting the the average across all splits for final model evaluation. This corpus contains only two types of ACs: claim and premise. The annotation of argument structure of the microtext corpus varies from the persuasive essay corpus; ACs can be linked to other links, as opposed to ACs. Therefore, if AC Ci is annotated to be linked to link l,
we create a link to the source AC of l. On average, this corpus has 5.14 ACs per text. Lastly, we note that predicting the presence of links is directional (ordered): predicting a link between the pair Ci, Cj(i 6= j) is different than Cj , Ci.
We implement our models in TensorFlow (Abadi et al., 2015). Our model has the following parameters: hidden input dimension size 512, hidden layer size 256 for the bidirectional LSTMs, hidden layer size 512 for the LSTM decoder, α equal to 0.5, and dropout (Srivastava et al., 2014) of 0.9. We believe the need for such high dropout is due to the small amounts of training data (Zarrella and Marsh, 2016), particularly in the Microtext corpus. All models are trained with Adam optimizer (Kingma and Ba, 2014) with a batch size of 16. For a given training set, we randomly select 10% to become the validation set. Training occurs for 4,000 epochs. Once training is completed, we select the model with the highest validation accuracy (on the link prediction task) and evaluate it on the held-out test set. At test time, we take a greedy approach and select the index of the probability distribution (whether link or type prediction) with the highest value.
5 Results
The results of our experiments are presented in Tables 1 and 2. For each corpus, we present f1 scores
7
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
Type prediction Link prediction Model Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1 No structural .808 .824 .694 .907 .760 .598 .922 No BOW .796 .833 .652 .902 .728 .543 .912 No Embeddings .827 .874 .695 .911 .750 .581 .918 Only Avg Emb* .832 .873 .717 .917 .751 .583 .918 Only Max Emb* .843 .874 .732 .923 .766 .608 .924 Only Min Emb* .838 .878 .719 .918 .763 .602 .924 All features .849 .894 .732 .921 .767 .608 .925
Table 3: Feature ablation study. * indicates that both BOW and Structural are present, as well as the stated embedding type.
Type prediction Link prediction Bin Macro f1 MC f1 Cl f1 Pr f1 Macro f1 Link f1 No Link f1
1 ≤ len < 4 .863 .902 .798 .889 .918 .866 .969 4 ≤ len < 8 .680 .444 .675 .920 .749 .586 .912 8 ≤ len < 12 .862* .000* .762 .961 .742 .542 .941
Table 4: Results of binning test data by length of AC sequence. * indicates that this bin does not contain any major claim labels, and this average only applies to claim and premise classes. However, we do not disable the model from predicting this class: the model was able to avoid predicting this class on its own.
for the AC type classification experiment, with a macro-averaged score of the individual class f1 scores. We also present the f1 scores for predicting the presence/absence of links between ACs, as well as the associated macro-average between these two values.
We implement and compare four types of neural models: 1) The previously described PN-based model depicted in Figure 3 (called Joint Model in the tables); 2) The same as 1), but without the fully-connected input layers (called Joint Model No FC Input in the table); 3) The same as 1), but the model only predicts the link task, and is therefore not optimized for type prediction (called PN in the tables); 4) A non-sequence-to-sequence model that uses the hidden layers produced by the BLSTM encoder with the same type of attention as the PN (called BLSTM in the table). That is, di in Equation 3 is replaced by ei.
In both corpora we compare against the following previously proposed models: Base Classifier (Stab and Gurevych, 2016) is a feature-rich, taskspecific (AC type or link extraction) SVM classifier. Neither of these classifiers enforce structural or global constraints. Conversely, the ILP Joint Model (Stab and Gurevych, 2016) provides constraints by sharing prediction information between the base classifiers. For example, the model attempts to enforce a tree structure among ACs
within a given paragraph, as well as using incoming link predictions to better predict the type class claim. For the microtext corpus only, we have the following comparative models: Simple (Peldszus and Stede, 2015) is a feature-rich logistic regression classifier. Best EG (Peldszus and Stede, 2015) creates an Evidence Graph (EG) from the predictions of a set of base classifiers. The EG models the potential argument structure, and offers a global optimization objective that the base classifiers attempt to optimize by adjusting their individual weights. Lastly, MP+p (Peldszus and Stede, 2015) combines predictions from base classifiers with a Minimum Spanning Tree Parser (MSTParser).
6 Discussion
First, we point out that the Joint Model achieves state-of-the-art on 10 of the 13 metrics in Tables 1 and 2, including the highest results in all metrics on the Persuasive Essay corpus, as well as link prediction on the Microtext corpus. The performance on the Microtext corpus is very encouraging for several reasons. First, the fact that the model can perform so well with only a hundred training examples is rather remarkable. Second, although we motivate the use of a PN due to the fact that it partially enforces the tree structure in argumentation, other models we compare against explic-
8
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
itly contain further constraints. For example, only premises can have outgoing links, and there can be only one claim in an AC. Moreover, the MP+p model directly enforces the tree constraint unique to the microtext corpus. Even though the Joint Model does not have the tree constraint directly encoded, it able to learn the structure effectively from the training examples so that it can outperform the Mp+p model for link prediction. As for the other neural models, the BLSTM model performs competitively with the ILP Joint Model on the persuasive essay corpus, but trails the performance of the Joint Model. We believe this is because the Joint Model is able to create two different representations for each AC, one each in the encoding/decoding state, which benefits performance in the dual tasks. Conversely, the BLSTM model must encode information relating to type as well as link prediction in a single hidden representation. On one hand, the BLSTM model outperforms the ILP model on link prediction, yet it is not able to match the ILP Joint Model’s performance on type prediction, primarily due to the BLSTM’s poor performance on predicting the major claim class. Another interesting outcome is the importance of the fully-connected layer before the LSTM input. The results show that this extra layer of depth is crucial for good performance on this task. Without it, the Joint Model is only able to perform competitively with the Base Classifier. The results dictate that even a simple fullyconnected layer with sigmoid activation can provide a useful dimensionality reduction for feature representation. Finally, the PN model, only optimized for link prediction, suffers a large drop in performance, conveying that the dual optimization of the Joint Model is crucial for high performance in the link prediction task.
Table 3 shows the results of an ablation study for AC feature representation. Regarding link prediction, BOW features are clearly the most important, as their absence results in the highest drop in performance. Conversely, the presence of structural features provides the smallest boost in performance, as the model is still able to record stateof-the-art results compared to the ILP Joint Model. This shows that the Joint Model is able to capture structural cues through sequence modeling and semantics. When considering type prediction, both BOW and structural features are important, and it is the embedding features that provide the least
benefit. The Ablation results also provide an interesting insight into the effectiveness of different ‘pooling’ strategies for using individual token embeddings to create a multi-word embedding. The popular method of averaging embeddings (which is used by (Stab and Gurevych, 2016) in their system) is in fact the worst method, although its performance is still competitive with the previous state-of-the-art. Conversely, max pooling results are on par with the Joint Model results from Table 1.
Table 4 shows the results on the Persuasive Essay test set with the test examples binned by sequence length. First, it is not surprising to see that the model performs best when the sequences are the shortest. As the sequence length increases, the accuracy on link prediction drops. This is possibly due to the fact that as the length increases, a given AC has more possibilities as to which other AC it can link to, making the task more difficult. Conversely, there is actually a rise in no link prediction accuracy from the second to third row. This is likely due to the fact that since the model predicts at most one outgoing link, it indirectly predicts no link for the remaining ACs in the sequence. Since the chance probability is low for having a link between a given AC in a long sequence, the no link performance is actually better in longer sequences.
7 Conclusion
In this paper we have proposed how to use a modified PN (Vinyals et al., 2015b) to extract links between ACs in argumentative text. We evaluate our models on two corpora: a corpus of persuasive essays (Stab and Gurevych, 2016), and a corpus of microtexts (Peldszus, 2014). The Joint Model records state-of-the-art results on the persuasive essay corpus, as well as achieving state-of-the-art results for link prediction on the microtext corpusThe results show that jointly modeling the two prediction tasks is crucial for high performance. Future work can attempt to learn the AC representations themselves, such as in (Kumar et al., 2015). Lastly, future work can integrate subtasks 1 and 4 into the model. The representations produced by Equation 3 could potentially be used to predict the type of link connecting ACs, i.e. supporting or attacking; this is the fourth subtask in the pipeline. In addition, a segmenting technique, such as the one proposed by (Weston et al., 2014), can accomplish subtask 1.
9
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899

Review:
The paper presents an application of Pointer Networks, a recurrent neural network model original used for solving algorithmic tasks, to two subtasks of Argumentation Mining: determining the types of Argument Components, and finding the links between them. The model achieves state-of-the-art results.  Strengths:  - Thorough review of prior art in the specific formulation of argument mining handled in this paper. - Simple and effective modification of an existing model to make it suitable for the task. The model is mostly explained clearly. - Strong results as compared to prior art in this task.  Weaknesses:  - 071: This formulation of argumentation mining is just one of several proposed subtask divisions, and this should be mentioned. For example, in [1], claims are detected and classified before any supporting evidence is detected. Furthermore, [2] applied neural networks to this task, so it is inaccurate to say (as is claimed in the abstract of this paper) that this work is the first NN-based approach to argumentation mining. - Two things must be improved in the presentation of the model: (1) What is the pooling method used for embedding features (line 397)? and (2) Equation (7) in line 472 is not clear enough: is E_i the random variable representing the *type* of AC i, or its *identity*? Both are supposedly modeled (the latter by feature representation), and need to be defined. Furthermore, it seems like the LHS of equation (7) should be a conditional probability. - There are several unclear things about Table 2: first, why are the three first baselines evaluated only by macro f1 and the individual f1 scores are missing? This is not explained in the text. Second, why is only the  PN  model presented? Is this the same PN as in Table 1, or actually the Joint Model? What about the other three? - It is not mentioned which dataset the experiment described in Table 4 was performed on.  General Discussion:  - 132: There has to be a lengthier introduction to pointer networks, mentioning recurrent neural networks in general, for the benefit of readers unfamiliar with  sequence-to-sequence models . Also, the citation of Sutskever et al. (2014) in line 145 should be at the first mention of the term, and the difference with respect to recursive neural networks should be explained before the paragraph starting in line 233 (tree structure etc.). - 348: The elu activation requires an explanation and citation (still not enough well-known). - 501:  MC ,  Cl  and  Pr  should be explained in the label. - 577: A sentence about how these hyperparameters were obtained would be appropriate. - 590: The decision to do early stopping only by link prediction accuracy should be explained (i.e. why not average with type accuracy, for example?). - 594: Inference at test time is briefly explained, but would benefit from more details. - 617: Specify what the length of an AC is measured in (words?). - 644: The referent of  these  in  Neither of these  is unclear. - 684:  Minimum  should be  Maximum . - 694: The performance w.r.t. the amount of training data is indeed surprising, but other models have also achieved almost the same results - this is especially surprising because NNs usually need more data. It would be good to say this. - 745: This could alternatively show that structural cues are less important for this task. - Some minor typos should be corrected (e.g.  which is show , line 161).  [1] Rinott, Ruty, et al.  Show Me Your Evidence-an Automatic Method for Context Dependent Evidence Detection.  EMNLP. 2015.  [2] Laha, Anirban, and Vikas Raykar.  An Empirical Evaluation of various Deep Learning Architectures for Bi-Sequence Classification Tasks.  COLING. 2016.