A Structured Span Selector
Abstract
Several natural language processing tasks such as coreference resolution and semantic role labeling require selecting text spans and making decisions upon them.
A typical approach for these tasks is to naively score all spans and greedily select spans for downstream processing.
This approach ignores any useful linguistic biases such as spans that need to be considered for these problems are typically syntactic constituents.
In this paper, we propose a novel linguistically-motivated “structured” span-selection model which learns to make use of the partial span-level annotation provided for such problems.
Compared to previous approaches, our approach gets rid of the heuristic greedy span selection scheme, allowing us to model the downstream task on an optimal set of spans.
We evaluate our model on two popular span prediction tasks: coreference resolution and semantic role labeling; and show improvements on both tasks in terms of performance as well as efﬁciency.
Introduction
Span selection , i.e., the problem of selecting a continuous segment (a span ) of the input text, is a common design pattern 1 in many span prediction problems in NLP.
Common span prediction problems include coreference resolution (Stede, 2011) where the selected spans are mentions, semantic role labeling (Palmer et al., 2010) where the selected spans are arguments, question answering (Jurafsky and Martin, 2000) where the selected spans are answers, and named entity recognition (Smith, 2011) where the selected spans are entities.
The span selection step is usually followed by a task speciﬁc model.
For instance, in coreference resolution, the selected spans (mentions) are clustered according to the entity they refer to; whereas, in semantic role labeling the spans (arguments) are
Coref: [The President] 1 has said (cid:2)
[he] 1 and [his] 1 wife, now a [New York] 3 senator (cid:3) 2 will spend weekends at [their] 2 house in Chappaqua.
SRL:
[The most important thing about Disney]
1- ARG 1 is[that [it]
2- ARG 1 is 2 [a global brand]
2- ARG 2 ]
1- ARG 2 .
classiﬁed into a set of roles.
An example of two span selection tasks is shown in Fig.
In most of the tasks mentioned above, span selection is the ﬁrst step in a pipeline.
2 After a set of candidate spans is determined, a classiﬁer (often a neural network) is typically used to score the candidate spans.
As the number of spans to consider in the input text can be quadratic in the length of input, these spans are greedily selected as potential antecedents, roles or answers.
While greedy span selection has become the de-facto approach in span prediction problems, it has several issues.
First, such approaches typically ignore the inherent structure in the problem.
For example, spans of interest in problems such as coreference and SRL are typically syntactic constituents, an assumption supported by quantitative results 3 .
Lack of linguistic constraint on the spans of interest leads to a waste of computational resource, as all O ( n 2 ) possible spans are enumerated by the model.
Thus, in this paper, we propose a structured span selector that solves the aforementioned problems.
Our span selector is modeled as a chart parser
White house secretary is a fan of
Bulls σ σ σ σ
S
Spans of interest
White house
White house secretary
Bullsa fan of bulls which learns to score partial, possibly nested, span annotations given in the training corpus.
Instead of optimizing the full parse, we directly learn to estimate the anchored probability of constituents using the inside scores in the parse.
The anchored probability can be easily computed by backpropagating through the inside score (Eisner, 2016).
Fig.
2 illustrates an example parse of our WCFG and the difference between the traditional greedy approach and our approach.
We apply our span selector to one coreference (Joshi et al., 2020) and one SRL model (He et al., 2018).
In both cases, we optimize the loglikelihood of the joint distribution deﬁned by the span selector and the conditional for the downstream task as deﬁned in Lee et al.
(2017) and
He et al.
In contrast to previous approaches which use a set of heuristics 4 to prune the set of spans to be considered, our span selector directly generates the optimal set of spans.
The number of resulting spans are signiﬁcantly lower than the number of spans considered in previous works, resulting in reduction in memory required to train these models.
The selected spans are also more accurate.
The proposed approach leads to signiﬁcant gains in both downstream tasks considered in this work: coreference and SRL.
We ﬁnd that our approach improves the performance of the end-to-end coreference model on the OntoNotes English dataset (by 0.4 CoNLL F1 points) and the LitBank dataset (by 0.7 CoNLL F1 points).
On SRL, our model also achieves consistent gains over previous work.
Background and Related Work 2. Span Selection as a Design Pattern
Many NLP tasks involve reasoning with textual spans.
Such tasks include coreference resolution, semantic role labelling, named entity recognition and question answering.
Models for these span prediction tasks often follow a common design pattern.
The architectures for these models often decompose into two components: (i) a span selection component where the model ﬁrst selects a set of spans of interest, and (ii) a span prediction component where a prediction (e.g. entity or role assignment) is made for the chosen set of spans.
As shown in previous works (Zhang et al., 2018;
Wu et al., 2020), the quality of the span selector can have a large impact on the overall model performance.
The span selector typically selects spans by selecting the start token and the end token in the span.
Thus, there are inherently (cid:0) N 2 (cid:1) textual spans (the complexity is O ( n 2 ) ) within a document with
N tokens to choose from.
Previous span selection models (Lee et al., 2017; He et al., 2018) enumerate all possible spans in a brute force manner and feed greedily selected top-k span candidates for the downstream prediction step.
However, several span selection tasks require spans that are syntactic constituents, which is a useful but often neglected inductive bias in these works.
For instance, in coreference resolution, span selection models are supposed to extract noun phrases, pronouns, and sometimes, verbs.
While in semantic role labeling, span selection models are required to extract syntactic constituents that could form semantic arguments of a predicate, such as noun phrases, prepositional phrases, and adverbs.
Our work uses a context-free parsing formulation to enumerate spans.
In context-free grammars, the number of valid syntactic constituents in a sequence of length n is bounded by O ( n ) , as the constituents can be viewed as the internal nodes of a binary parse tree in Chomsky normal form.
Compared with brute-force enumeration and greedy pruning, this inductive bias serves us as a natural pruning strategy to reduce the complexity from O ( n 2 ) to O ( n ) and provides us a more natural and linguistically informed way to model span-based tasks in NLP, through which we can employ parsing techniques and retrieve the optimal span selection for downstream tasks.
To further motivate our work, we provide a background on two popular span prediction tasks considered in our work: coreference resolution and semantic role labelling (SRL).
We also describe some previous works on these tasks and contrast them to our work.
Then, we also describe how our model can work with partial span-level annotations provided by datasets for these tasks.
2. Coreference Resolution
Most coreference resolution models involve two stages: mention detection and mention clustering.
Traditional pipeline systems relied on parse trees and hand-engineered rules to extract mentions (Raghunathan et al., 2010).
However, this was reversed by Lee et al.
(2017) who showed that we can directly detect mentions as well as assign antecedents for them in an end-to-end manner.
Alongside our work, some other works (Zhang et al., 2018; Swayamdipta et al., 2018; Miculicich and Henderson, 2019; Yu et al., 2019) have also explored better mention proposers for coreference.
Zhang et al.
(2018) use multi-task loss to directly optimize the mention detector.
Swayamdipta et al.
(2018) also leverage syntactic span classiﬁcation as an auxiliary task to assist coreference.
Miculicich and Henderson (2019) and Yu et al.
both improve the mention detector with better neural network structures.
Yet they still need to manually set a threshold to control the number of selected candidate mentions and non of them could produce an optimal span selection for downstream task.
Finkel and Manning (2009) situate NER in a parsing framework by explicitly incorporating named entity types into parse tree labels.
In contrast, our work requires neither syntax annotations nor hyperparameter tuning for mention selection.
Zhang et al.
(2018) proposed a multi-task style loss to provide direct supervision to mention detection.
Yet the seemingly simplistic and heuristic mention detection scheme by greedily taking a ﬁxed top portion of all candidate spans still remains unchanged in the aforementioned work.
Pretrained contextual word representations (Joshi et al., 2019, 2020; Lee et al., 2018) and higher-order inference techniques (Lee et al., 2018;
Kantor and Globerson, 2019; Xu and Choi, 2020) have brought major improvements to the original model by Lee et al..
Aside from this line of work,
Wu et al.
(2020) formulate coreference resolution as a question answering task, and achieve state-ofthe-art results on multiple benchmarks.
2. Semantic Role labeling
Semantic role labeling (SRL) extracts relations between predicates and their arguments.
Two major lines of work in SRL are sequence tagging models (
He et al., 2017; Marcheggiani et al., 2017) and span-based models (He et al., 2018; Ouchi et al., 2018; Li et al., 2019).
Sequence tagging models for SRL convert semantic role annotations to BIO sequences.
The tagger generates a label sequence for one single predicate at a time.
However, span-based models generate the set of all candidate arguments in one forward pass, and classify their semantic roles with regard to each predicate.
As discussed in (He et al., 2018), span-based models empirically perform better than sequence tagging models as they incorporate span-level features.
Span-based models also do better at long-range dependencies as well as agreement with syntactic boundaries.
Very recently, Zhang et al.
(2021) explore a novel direction by reducing semantic role labeling to a dependency parsing task and ﬁnd promising results.
2. Nested and Partial Span Annotations
Nested mentions and partially annotated mentions are the two major concerns of this work.
Most datasets for span prediction problems contain partial annotations of mentions.
For example, single- tons are not annotated in Ontonotes (Pradhan et al., 2012).
In the coreference resolution example given in Fig.
1, the bracketed nested spans are annotated, while the underlined spans are valid mentions that are missed out, since they do not co-refer with any other mention in the same document.
The same issues also exist in SRL.
In the SRL example in Fig.
1, there are two predicates (boxed words in the example) in one sentence.
Their arguments are nested (i.e. ARG1 and ARG2 of the second predicate locate within ARG2 of the ﬁrst predicate).
Besides, many syntactic constituents that could form valid arguments are unannotated.
Our span selector can be trained with partial span annotations provided in the training data.
See § for further details.
A Structured Span Selection Model
We ﬁrst start by giving a formal deﬁnition of a span selection task.
We deﬁne a document D as a sequence of sentences w 1 , . . .
w | D | .
Each sentence w in the document is a sequence of words [ w 1 , . .
. , w | w | ] .
A span is a contiguous subsequence of words in a sentence.
For instance, we denote the span from position i to position k as w i · · · w k , which we abbreviate as σ ik .
3. A Weighted Context-Free Grammar
In this section, we formalize the linguistic inductive bias that a majority of spans considered in span selection tasks are constituents .
Under this assumption, the set of selected spans in a sentence should be compatible with a phrase structure tree 5 , i.e., there should not be any spans crossing each other in the selected set of spans.
To model this tree structure, we propose to learn a weighted context-free grammar (WCFG) (cid:104) Σ , N, S, R (cid:105) , where Σ is a ﬁnite set of terminal symbols, N is a ﬁnite set of nonterminal symbols, S ∈ N is the start symbol, and R is the set of production rules.
A learnable weight function s p : R → R ≥ 0 assigns a non-negative weight to each rule in R .
This function naturally extends to assign a weight to each CNF parse t by
[Equation]
where all internal nodes of t are enumerated.
Most importantly, our WCFG deﬁnes a distribution over the set of all parses of a sentence T given by
[Equation]
where Z = (cid:80) t ∈T s p ( t ) is the sum of the scores of all parses.
Despite its formulation, we design our WCFG to be succinct — we only care about two types of nonterminals other than the start symbol S , that is, is a span of interest in the downstream task or not .
Due to this, among all the production rules, only those with spans of interest on the right-hand side can affect the weight of a derivation 6 .
In a
Chomsky Normal Form derivation, a span of interest appears once and only once on the right-hand side.
In this respect, a parse t can be uniquely mapped to a set of spans of interest by gathering all nonterminals of that type.
We denote the resulting set after mapping as M = span ( t ) , and the set of all sets of spans of interest as M .
Thus, the score of a parse can be re-expressed as a product of the scores of all spans of interest in it:
[Equation]
The problem of mention detection is hence converted from subset selection to ﬁnding an optimal parse tree that maximizes the score:
[Equation]
where the classical algorithm for parsing, i.e., the
CKY algorithm, can be employed to obtain an exact solution.
The recursive function when i < k used is:
[Equation]
where γ ([ i, k ]) is the score of the optimal derivation over span [ i, k ] .
When i =
k , γ ([ i, k ]) =
s p ( σ ik ) .
We now brieﬂy introduce the inside-outside algorithm to pave the way for the next section.
For a span [ i, k ] , its inside value can be expressed as
[Equation]
When i = k , β ([ i, k ]) =
exp s p ( σ ik ) .
The inside value of the entire sentence β ([0 , n ]) is exactly Z , the sum of scores of all parses.
Spans from a WCFG
We have been using the notation of anchored nonterminal σ ik , when a parse contains a span of interest that ranges over [ i, k ] , without formally introducing it.
Further, an anchored probability , i.e., how likely span [ i, k ] would be of interest in any parse, can be computed.
Let T ik be the set of parses that contains σ ik , the anchored probability of σ ik can be expressed as:
[Equation]
As described by Eisner (2016), we can calculate p ( σ ik | w ) by backpropagating through the logtransformed inside value log Z = log β ([0 , n ]) :
[Equation]
which can be efﬁciently computed with the automatic differentiation in modern deep learning tools such as PyTorch.
For details of deduction of Eq.
(8), we refer the readers to (Eisner, 2016, Sec. 4.3).
We use p ( σ ik ) as shorthand for p ( σ ik | w ) without causing ambiguities from now.
Adaptations to Downstream Tasks
In this section, we introduce how our structured span selector can be applied in end-to-end frameworks for coreference resolution and SRL.
4. Coreference Resolution
The goal of coreference resolution is to link every mention σ ij to its antecedent , which is itself another mention in the same document.
Note that in the formulation of coreference discussed in this task, a mention may have a dummy antecedent, denoted as (cid:15) , if it is not a mention or does not co-refer with any previous span.
One core technical contribution of this paper is a joint model for mention detection and downstream tasks.
We write σ ij (cid:32) σ k(cid:96) to denote that σ ij corefers with σ k(cid:96) ; in other words, that σ k(cid:96) is σ ij ’s antecedent.
Our paper introduces the following framework for joint modeling mention detection and coreference resolution:
[Equation]
From now, we use σ ij as a shorthand for σ ij , and p ( σ ij (cid:32) σ k(cid:96) ) for p ( σ ij (cid:32) σ k(cid:96) | σ ij ) .
The joint distribution is then p ( σ ij (cid:32) σ k(cid:96) , σ ij ) =
p ( σ ij (cid:32) σ k(cid:96) ) p ( σ ij ) .
In principle, modelers can choose p ( σ ij ) and p ( σ ij (cid:32) σ k(cid:96) )
according to their taste.
In this work, we implement p ( σ ij ) as the anchored probability in a WCFG, as described in §3.
and p ( σ ij (cid:32) σ k(cid:96) ) as Lee et al.
(2017)’s popular coreference model, described in §4.1.
Lee et al.
(2017) as Mention Linker.
We choose p ( σ ij (cid:32) σ k(cid:96) ) to be the one from (Lee et al., 2017).
In principle, we can use any mention-ranking model that computes a distribution over Y ij , the set of preceding spans of [ i, j ] .
But we choose (Lee et al., 2017) because it is well-known and effective:
[Equation]
s m ( σ ij ) is a score for span [ i, j ] being a mention and s a ( σ ij , σ k(cid:96) ) for [ k, (cid:96) ] to be a coreferent antecedent of [ i, j ] .
Both s a and s m are computed by neural networks that take span representations as inputs.
The score s ( σ ij , (cid:15) ) is ﬁxed to 0 .
Training, Decoding and Other Concerns.
As for mention detection, Lee et al.
’s
model adopts a naïve greedy algorithm by taking the top λ | D | spans with the highest mention scores s m , where λ is a hyperparameter that has to be manually deﬁned for different datasets.
However, ﬁnding a proper ratio λ can be very tricky.
Since in many coreference datasets such as OntoNotes, only the mentions that are referred to more than once are annotated (as discussed in §3.1), learning a mention detector from such data requires the ability to handle partially annotated examples.
The case that [ i, j ] is assigned to an antecedent assignment (cid:15) (i.e. y ij = (cid:15) ) happens when (a) [ i, j ] is a singleton mention, or (b)
[ i, j ] is the ﬁrst appearance in a mention cluster, or (c)
[ i, j ] is not a valid mention.
To handle the partial annotation and retrieve valid mentions that are not annotated in the dataset, we split the probability into two separate terms corresponding to the above scenarios:
[Equation]
We expect singleton mentions to have a larger ﬁrst term and smaller second term, and the opposite for non-mentions.
The ﬁnal objective function is then the negative log-likelihood of the joint distribution deﬁned at the beginning of §4.1:
[Equation]
where G ij is the ground truth mention cluster that span [ i, j ] belongs to.
For mention detection, the time complexity of training and inference for one sentence is O ( n ) with parallel computation.
Sentence-level parallelism can also be applied to all the sentences.
Thus, the overall time complexity is O ( n max ) , the length of the longest sentence in the document.
4. Semantic Role Labeling
The goal of semantic role labeling is to classify the semantic role of every argument σ ij with respect to a given predicate.
Following the notation style of §4.1, we use σ ij l (cid:32) v to denote that σ ij plays the semantic role l in the frame of predicate v .
The joint probability can be written as p (cid:16) σ
[Equation]
v, σ
[Equation]
(cid:17) =
p
[Equation]
(cid:96) (cid:32) v (cid:17) p ( σ ij ) .
Our joint framework for argument detection and semantic role labeling is then:
[Equation]
He et al.
as Role Classiﬁer.
As this work focuses on span selection, we choose the role classiﬁer to be the popular and effective one from (He et al., 2018), and use gold predicates v during training and evaluation.
The semantic role label l takes value from a discrete label space L .
The classiﬁer then models the following probability distribution:
[Equation]
Similar to the coreference model of Lee et al.
(2017), s m ( σ ij ) is the score for span [ i, j ] to be an argument and s r ( σ ij , v, (cid:96) ) for [ i, j ] to play the role l for predicate v .
Note that the score for (cid:15) label (i.e. no relation) s ( σ ij , v, (cid:15) ) is set to constant 0 , similar to the null antecedent case in coreference.
Training Objective.
He
et al.
’s model also suffers from the challenge of tuning the hyperparameter λ and the issue of partial annotation for the span selection model.
To deal with these, we use a similar strategy as in §4.1.
Again, the probability of null relation is split into two terms:
[Equation]
Thus, the training objective for SRL with our structured span selection model is then:
[Equation]
where l is the correct semantic label of σ ij with regard to predicate v .
Experiments 5. The Greedy Baseline
Previous work has mostly considered a greedy procedure for span selection as opposed to Eq.
The approach produces a score s g ( σ ik ) independently for each span σ ik .
As the number of spans σ ik is potentially very large, the set of spans are greedily pruned to a set of size K .
For instance, in SRL, where spans are selected for each sentence:
[Equation]
However, in coreference resolution, a set of spans are selected for the entire document:
[Equation]
where atop K is shorthand for argtop K .
We will see in our experiments that tuning K can be quite challenging.
Moreover, as the greedy approach scores each span independently, it cannot judiciously make use of the provided span annotation.
5. Datasets
Coreference: We employ the CoNLL-2012 English shared task dataset (OntoNotes) (Pradhan et al., 2012) and LitBank (Bamman et al., 2020) in our experiments.
For OntoNotes, we apply the speaker encoding in (Wu et al., 2020), that is using special tokens ( <speaker> , </speaker> ) to denote the speaker’s name, as opposed to the original binary features used by Lee et al.
This simple change brings a consistent boost to the performance by 0 .
2 CoNLL F1.
A major difference between these two datasets is that LitBank has singleton mention annotations while OntoNotes does not.
For LitBank, we use the standard 10-fold cross-validation setup, as is the standard practice.
SRL:
We use the CoNLL-2012 SRL dataset.
Gold predicates are provided to the model.
5. Coreference Resolution
We report the average precision, recall, and F scores of the standard MUC, B 3 , and CEAF φ 4 and the average CoNLL F1 score on OntoNotes test set in Tab. 1.
The average CoNLL F1 scores on LitBank are shown in Tab. 2.
For OntoNotes, we run the experiments with 5 random initializations and the improvements reported are signiﬁcant under the two-tailed t -test.
We compare our models with several representative previous work.
In order to focus on comparing the impact of mention detection, we apply no higher-order inference techniques in our models and report the non-higher order result from (Xu and Choi, 2020).
Joshi et al.
(S) is the major baseline that uses SpanBERT (Joshi et al., 2020) and is trained with the speaker encoding discussed in §5.2.
This encoding gives 0 .
2 F1 score improvement over the result reported in (Xu and Choi, 2020).
Our model with the structured mention detector achieves 80 .
5 F1 score, a 0 .
4 F1 improvement over the baseline.
While on LitBank, our model achieves 76 .
3 F1, which is a 0 .
7 F1 improvement over Joshi et al..
It can also be observed that this gain mainly comes from improved recall, which is because we have a superior mention detector that can retrieve mentions with better accuracy.
We further analyze this result in the following section.
5.3. Analysis of Mention Detector
Next, we examine the performance of our proposed mention detection scheme.
As shown in Fig.
3, compared with Joshi et al.
(S), our model predicts a far more accurate set of mentions with a higher recall.
In contrast to Joshi et al.
(S) who select 0 .
| D | mention spans, our method only selects 0 .
| D | mention spans with a slightly higher mention recall.
The smaller span set would make the overall coreference model more efﬁcient as well.
5.3. Analysis of Structured Modeling
To see how our structured modeling beneﬁts the overall coreference model, we further compare our
[Figure]
approach with a baseline Sigmoid which replaces the p ( σ ij ) in Eq.
(8) with a simple non-structured estimator:
[Equation]
where σ is the sigmoid function σ ( x ) = 1 1+exp( − x ) .
The loss function used for Sigmoid is identical with our structured model as given in Eq.
Through this comparison, we aim to show the effectiveness of structured modeling.
We also build a multi-task learning baseline MTL similar to (Swayamdipta et al., 2018).
The model performs an auxiliary task which classiﬁes spans into noun phrase, other syntactic constituent, or non-constituent.
The coefﬁcient of the multi-task loss is set to 0 .
1 as in (Swayamdipta et al., 2018).
This comparison is shown in Tab.
We ﬁnd that replacing P ( i ) with unstructured P σ ( i ) degrades the performance by 0 .
3 F1 score.
Thus, we conclude that the structured probability function is more expressive than P σ ( i ) as it models the global annotation for each sentence.
In contrast, the mention detector of Joshi et al.
(S) and Sigmoid models each span independently.
5.3. Qualitative Analysis
To shed some light on where the gains of our proposed model come from, we also exhibit some qualitative results in this section.
Nested Mentions.
We ﬁrst investigate the capability of our structured model in handling nested
[Table]
speaker
The improvements
[Table]
Bamman
[Table] [Table]
mentions.
Tab. 4 shows the recall rate of mentions of different nested depth.
Here, nested depth refers to the level of nesting in the mentions.
E.g. in the ﬁrst example given in §2.4, The president is of depth 1, while he and his wife, now a New York senator is of depth 2.
As shown in Tab. 4, the gains of our method are larger for deeply nested mentions, which highlights the capability of our structured span detector to handle harder nested mentions that cannot be handled by the greedy selector.
[Table]
Widths of Mentions.
We also compare the recall rate for mentions of different widths in Tab. 5.
We show that our model can detect longer spans better, which are usually more difﬁcult to detect.
For spans with 5-12 words, our structured model still maintains a recall rate of 96.5%, compared to a sharp drop of the greedy unstructured model.
[Table]
5. Semantic Role Labeling
For semantic role labeling, we report the precision, recall, and F1 score on the CoNLL-2012 SRL dataset.
The gold predicates are provided during both training and evaluation.
Therefore, the model have to focus on extracting the correct arguments and classifying their roles for each predicate.
The [Table] [Table] results are shown in Tab. 6 in comparison with previous span-based models.
He et al.
SpanBERT refers to the model of He et al.
(2018) with SpanBERT large (Joshi et al., 2020) as sentence encoder.
5.4. Analysis of Argument Detection
Next, we report the performance of our span selector on the SRL task.
Following the same trend as coreference resolution, we ﬁnd that our structured model is able to extract much more accurate arguments and thus, signiﬁcantly reduce the memory consumption for the downstream task.
While keeping a comparable recall rate of gold arguments (96.5% for the greedy selector and 96.2% for ours ), our span selector reduces the number of enumerated spans by 21.2%.
We compare the accuracy of retrieving unlabeled argument spans in Tab. 7.
BIO refers to the taggerstyle SRL model using the same text encoder.
Our model outperforms the other two baselines.
5. Qualitative Examples
In this section, we show a qualitative example to illustrate the grammar learned by our span selector for coreference resolution and SRL.
Two sets of extracted spans for the same input sentence are shown in Tab. 8.
For coreference resolution, our model selects maximal NPs (containing all modiﬁers) and verbs.
While in SRL, the parse tree consists of much denser and syntactically heterogeneous spans of NPs, PPs, modal verbs, and adverbs, etc.. This comparison empirically shows
Input
The world’ s ﬁfth Disney park will soon open to the public here .
Coref [[The world’ s] ﬁfth [Disney] park] will soon [open] to [the public] here .
SRL [The world’ s [ﬁfth] [Disney] [park]] [will] [soon] [ [open] [to the public]] [here] .
[Table]
that our model is capable and robust enough to learn much more complex underlying grammars from partial annotation.
5. Memory Efﬁciency
We further analyze the memory efﬁciency of our model.
We evaluate the peak GPU memory usage on the development set of OntoNotes.
For both tasks, we see a signiﬁcant reduction in memory usage of 27% for coreference and 24% for SRL.
Conclusion and Future Work
In this paper, we proposed a novel structured model for span selection.
In contrast to prior span selection methods, the model is structured, which allows it to model the spans better.
Instead of a greedy span selection procedure, the span selector uses partial span annotations provided in data to directly obtain the set of optimal spans.
We evaluated our span selector on two typical span prediction tasks, namely coreference resolution and semantic role labeling, and achieved consistent gains in terms of accuracy as well as efﬁciency.
In the future, we plan to generalize the proposed span selection model to other NLP tasks, e.g. extractive question answering (Lee et al., 2016).
It would also be interesting to experiment with various pretraining techniques for span selection.
Ethical Considerations
To the best of our knowledge, the datasets used in our work do not contain sensitive information.
Although our model has been evaluated under academic settings, it could also be applied in sensitive contexts, e g. healthcare or legal scenarios.
It is essential that necessary anonymization and robustness evaluation is undertaken before using our model in these settings.
A A Weighted Context-Free Grammar
In our WCFG (cid:104) Σ , N, S, R (cid:105) , there are only three types of nonterminals, i.e. N = { S, σ, ¯ σ } , where
S is the start symbol, σ is the span of interest, and ¯ σ is the spans that are not of interest.
Consequently, there are very few production rules in R .
And we only assign nontrivial weights to rules A → BC where at least one of B, C is σ .
That is to say, in Eq.
(1), s p ( r )
=
for all r =
A → BC where neither B nor C is σ .
This grammar is described in (Gaddy et al., 2018, Sec.
2.5) to eliminate the need for binarization for n -ary trees.
B Experimental Settings
The systems are implemented with PyTorch.
We use SpanBERT large as text encoder.
We train the model for 20 epochs and select the best performing model on the development set for testing.
The documents are split into 512 word segments to ﬁt in SpanBERT large .
Models used for coreference resolution have 402 million learnable parameters, and models for SRL have 382 million learnable parameters.
We closely follow the hyperparameter settings of Joshi et al.
and build our models upon the codebase of Xu and Choi (2020) 7 under Apache
License 2.0.
The learning rate of SpanBERT large parameters is set to 1 × 10 − 5 with 0 .
01 decay rate, and the learning rate of task parameters is set to 3 × 10 − 4 .
The dropout rate of feedforward neural network scorers is set to 0 .
When training our model, we randomly sample 0 .
1 | D | negative spans that are not mentions and add their negative loglikelihood − log p ( σ ij ) to the training objective.
This is to prevent p ( σ ij ) from converging to 1 .
For SRL task, we use a batch size of 32 forepochs and the same learning rate with coreference resolution.
The same negative sampling technique is applied.
Our models are trained on Nvidia Tesla V
GPUs with 32GB memory.
The average training time is around 8 hours for Joshi et al.
(S) baseline, and around 9 hours for our model.
For SRL models, training takes 25 hours.
C Results on the Development Set
In this section
[Table]
we report the results that our models get on the development sets of OntoNotes and
[Table]
[Table] [Table]