Original: S CI C O : Hierarchical Cross-Document Coreference for Scientiﬁc Concepts
Revised: S CI C O : Hierarchical Cross-Document Coreference for Scientiﬁc Concepts

Original: Anonymous authors
Revised: Anonymous authors

Original: Abstract
Revised: Abstract

Original: Determining coreference of concept mentions across multiple documents is a fundamental task in natural language understanding.
Revised: Determining coreference of concept mentions across multiple documents is a fundamental task in natural language understanding.

Original: Work on cross-document coreference resolution (CDCR) typically considers mentions of events in the news, which seldom involve abstract technical concepts that are prevalent in science and technology.
Revised: Work on cross-document coreference resolution (CDCR) typically considers mentions of events in the news, which seldom involve abstract technical concepts that are prevalent in science and technology.

Original: These complex concepts take diverse or ambiguous forms and have many hierarchical levels of granularity (e g., tasks and subtasks), posing challenges for CDCR.
Revised: These complex concepts take diverse or ambiguous forms and have many hierarchical levels of granularity (e g., tasks and subtasks), posing challenges for CDCR.

Original: We present a new task of Hierarchical CDCR (H-CDCR) with the goal of jointly inferring coreference clusters and hierarchy between them.
Revised: We present a new task of Hierarchical CDCR (H-CDCR) with the goal of jointly inferring coreference clusters and hierarchy between them.

Original: We create S CI C O , an expert-annotated dataset for H-CDCR in scientiﬁc papers, 3X larger than the prominent ECB+ resource.
Revised: We create S CI C O , an expert-annotated dataset for H-CDCR in scientiﬁc papers, 3X larger than the prominent ECB+ resource.

Original: We study strong baseline models that we customize for H-CDCR and highlight challenges for future work.
Revised: We study strong baseline models that we customize for H-CDCR and highlight challenges for future work.

Original: 1. Introduction
Revised: 1. Introduction

Original: Cross-document coreference resolution (CDCR) identiﬁes and links textual mentions that refer to the same entity or event across multiple documents.
Revised: Cross-document coreference resolution (CDCR) identiﬁes and links textual mentions that refer to the same entity or event across multiple documents.

Original: This fundamental task has seen much work recently [Choubey and Huang, 2017, Kenyon-Dean et al., 2018, Barhom et al., 2019, Cattan et al., 2021a,b, Caciularu et al., 2021] and can beneﬁt various downstream applications such as multi-hop question answering [Dhingra et al., 2018, Wang et al., 2019], multi-document summarization [Falke et al., 2017], and discovery of cross-document relations [Hope et al., 2017, 2020, 2021].
Revised: This fundamental task has seen much work recently [Choubey and Huang, 2017, Kenyon-Dean et al., 2018, Barhom et al., 2019, Cattan et al., 2021a,b, Caciularu et al., 2021] and can beneﬁt various downstream applications such as multi-hop question answering [Dhingra et al., 2018, Wang et al., 2019], multi-document summarization [Falke et al., 2017], and discovery of cross-document relations [Hope et al., 2017, 2020, 2021].

Original: Existing datasets for CDCR, such as ECB+ [Cybulska and Vossen, 2014], focus on mentions of news events involving concrete entities such as people or places.
Revised: Existing datasets for CDCR, such as ECB+ [Cybulska and Vossen, 2014], focus on mentions of news events involving concrete entities such as people or places.

Original: Abstract technical concepts are largely unexplored despite their prevalence in domains such as science and technology, and can pose signiﬁcant challenges for CDCR: they often take diverse forms (e.g., class-conditional image synthesis and categorical image generation ) or are ambiguous (e.g., network architecture in AI vs. systems research).
Revised: Abstract technical concepts are largely unexplored despite their prevalence in domains such as science and technology, and can pose signiﬁcant challenges for CDCR: they often take diverse forms (e.g., class-conditional image synthesis and categorical image generation ) or are ambiguous (e.g., network architecture in AI vs. systems research).

Original: These complex concepts also have many hierarchical levels of granularity, such as tasks that can be divided into ﬁner-grained subtasks, where reference to a speciﬁc concept entails also a reference to the higher-level concept (e.g., CRF entails the sequence tagging task), unlike events and entities in ECB+ that are treated as non-hierarchical.
Revised: These complex concepts also have many hierarchical levels of granularity, such as tasks that can be divided into ﬁner-grained subtasks, where reference to a speciﬁc concept entails also a reference to the higher-level concept (e.g., CRF entails the sequence tagging task), unlike events and entities in ECB+ that are treated as non-hierarchical.

Original: [Figure]
Revised: [Figure]

Original: In this paper, we formulate a novel task of hierarchical CDCR .
Revised: In this paper, we formulate a novel task of hierarchical CDCR .

Original: The task is to infer (1) cross-document entity coreference clusters of concept mentions in scientiﬁc papers, and (2) referential hierarchy between clusters , where referring to a child cluster entails reference to the parent.
Revised: The task is to infer (1) cross-document entity coreference clusters of concept mentions in scientiﬁc papers, and (2) referential hierarchy between clusters , where referring to a child cluster entails reference to the parent.

Original: Figure 1 shows the structure we aim to induce given mentions in context.
Revised: Figure 1 shows the structure we aim to induce given mentions in context.

Original: Our task is the ﬁrst to consider uniﬁed CDCR and hierarchy between clusters of mentions in context, unlike most work focusing on lexiconlevel taxonomies over uncontextualized words [Shwartz et al., 2016, Zhang et al., 2018].
Revised: Our task is the ﬁrst to consider uniﬁed CDCR and hierarchy between clusters of mentions in context, unlike most work focusing on lexiconlevel taxonomies over uncontextualized words [Shwartz et al., 2016, Zhang et al., 2018].

Original: CDCR in science can support many applications.
Revised: CDCR in science can support many applications.

Original: One example illustrating our setting is faceted query by example [Mysore et al., 2021], where given a paper, a user may highlight a speciﬁc span (facet)
Revised: One example illustrating our setting is faceted query by example [Mysore et al., 2021], where given a paper, a user may highlight a speciﬁc span (facet)

Original: (e.g., network architecture ), and explore other papers that refer to the same concept while resolving ambiguity and clustering diverse surface forms.
Revised: (e.g., network architecture ), and explore other papers that refer to the same concept while resolving ambiguity and clustering diverse surface forms.

Original: Selecting a facet may also show concept hierarchies enabling users to browse and navigate collections [Hope et al., 2020]
Revised: Selecting a facet may also show concept hierarchies enabling users to browse and navigate collections [Hope et al., 2020]

Original: (e.g., by viewing a list of many types of network architectures, automatically identiﬁed).
Revised: (e.g., by viewing a list of many types of network architectures, automatically identiﬁed).

Original: See §6 for more discussion of related areas and problems that could beneﬁt from CDCR in the scientiﬁc domain.
Revised: See §6 for more discussion of related areas and problems that could beneﬁt from CDCR in the scientiﬁc domain.

Original: To advance research in this area and enable supervised model training and evaluation, we create a new large-scale dataset named S CI C O ( S cientiﬁc C oncept I nduction C orpus), which is annotated by domain experts.
Revised: To advance research in this area and enable supervised model training and evaluation, we create a new large-scale dataset named S CI C O ( S cientiﬁc C oncept I nduction C orpus), which is annotated by domain experts.

Original: S CI C O consists of clusters of mentions in context and a hierarchy over them, as shown in Figure 1.
Revised: S CI C O consists of clusters of mentions in context and a hierarchy over them, as shown in Figure 1.

Original: The corpus is drawn from computer science papers, and the concept mentions are methods and tasks from across CS.
Revised: The corpus is drawn from computer science papers, and the concept mentions are methods and tasks from across CS.

Original: To build S CI C O , we develop a new candidate generation approach built on three resources: a low-coverage KB, a noisy hypernym extractor, and curated candidates.
Revised: To build S CI C O , we develop a new candidate generation approach built on three resources: a low-coverage KB, a noisy hypernym extractor, and curated candidates.

Original: We evaluate strong baseline models, ﬁnding that a cross-encoder model addressing coreference and hierarchy jointly outperforms others.
Revised: We evaluate strong baseline models, ﬁnding that a cross-encoder model addressing coreference and hierarchy jointly outperforms others.

Original: Our main contributions include:
Revised: Our main contributions include:

Original: • We formulate the novel task of hierarchical cross-document coreference (H-CDCR), and explore it within scientiﬁc papers.
Revised: • We formulate the novel task of hierarchical cross-document coreference (H-CDCR), and explore it within scientiﬁc papers.

Original: • We release S CI C O , an expert-annotated dataset 3X larger than the prominent ECB+ CDCR dataset.
Revised: • We release S CI C O , an expert-annotated dataset 3X larger than the prominent ECB+ CDCR dataset.

Original: • We build a model for H-CDCR that outperforms multiple baselines while leaving much room for future improvement.
Revised: • We build a model for H-CDCR that outperforms multiple baselines while leaving much room for future improvement.

Original: 2. The H-CDCR Task 2.1 Problem Formulation
Revised: 2. The H-CDCR Task 2.1 Problem Formulation

Original: Our goal is to induce clusters of contextualized mentions that corefer to the same concept, and to infer a hierarchy over these concept clusters.
Revised: Our goal is to induce clusters of contextualized mentions that corefer to the same concept, and to infer a hierarchy over these concept clusters.

Original: Formally, we are given a set of documents D from a diverse corpus.
Revised: Formally, we are given a set of documents D from a diverse corpus.

Original: We assume each d ∈ D comes annotated with mentions (spans of text, see Table 1)
Revised: We assume each d ∈ D comes annotated with mentions (spans of text, see Table 1)

Original: [Table]
Revised: [Table]

Original: denoting concepts.
Revised: denoting concepts.

Original: Denote by M d = { m 1 , m 2 , ..., m n } the set of mentions in document d and by
Revised: Denote by M d = { m 1 , m 2 , ..., m n } the set of mentions in document d and by

Original: M the set of mentions across all d ∈ D .
Revised: M the set of mentions across all d ∈ D .

Original: The ﬁrst goal, similar to cross-document coreference resolution, is to cluster the mentions in M into disjoint clusters
Revised: The ﬁrst goal, similar to cross-document coreference resolution, is to cluster the mentions in M into disjoint clusters

Original: C = { C 1 , ..., C t } , with each cluster consisting of mentions { m | m ∈ C i } that all refer to the same underlying concept (see Figure 1).
Revised: C = { C 1 , ..., C t } , with each cluster consisting of mentions { m | m ∈ C i } that all refer to the same underlying concept (see Figure 1).

Original: To account for the difﬁculty in precisely delineating the “borders” of extremely ﬁne-grained concepts in scientiﬁc literature, clusters are allowed to include subtle variations around a core concept (e.g., CRF model , CRF tagger ).
Revised: To account for the difﬁculty in precisely delineating the “borders” of extremely ﬁne-grained concepts in scientiﬁc literature, clusters are allowed to include subtle variations around a core concept (e.g., CRF model , CRF tagger ).

Original: The second goal is to infer a hierarchy over clusters.
Revised: The second goal is to infer a hierarchy over clusters.

Original: Deﬁne a graph G C
Revised: Deﬁne a graph G C

Original: =
Revised: =

Original: ( C , E ) , with vertices representing C (mention clusters), and directed edges (cid:15) ij ∈ E , each edge representing a hierarchical relation between clusters C i and C j which reﬂects referential hierarchy .
Revised: ( C , E ) , with vertices representing C (mention clusters), and directed edges (cid:15) ij ∈ E , each edge representing a hierarchical relation between clusters C i and C j which reﬂects referential hierarchy .

Original: A relation C 1 −→ C 2 exists when the concept underlying C 2 entails a reference to C 1 (e.g., language model −→ BERT ).
Revised: A relation C 1 −→ C 2 exists when the concept underlying C 2 entails a reference to C 1 (e.g., language model −→ BERT ).

Original: Put together, the goal in our Hierarchical Cross Document Coreference (H-CDCR) task is:
Revised: Put together, the goal in our Hierarchical Cross Document Coreference (H-CDCR) task is:

Original: Given documents D and mentions M , construct clusters C and a hierarchy graph G C
Revised: Given documents D and mentions M , construct clusters C and a hierarchy graph G C

Original: = ( C , E ) by learning from a set of N examples { ( D k , M k , G k C ) } Nk =1 .
Revised: = ( C , E ) by learning from a set of N examples { ( D k , M k , G k C ) } Nk =1 .

Original: In our experiments, we focus on mentions referring to tasks and methods in computer science papers (see examples in Table 1 and §3).
Revised: In our experiments, we focus on mentions referring to tasks and methods in computer science papers (see examples in Table 1 and §3).

Original: 2.2 Evaluation Metrics
Revised: 2.2 Evaluation Metrics

Original: Comparing an extracted set of mention clusters and hierarchies to a gold standard for evaluation is non-trivial.
Revised: Comparing an extracted set of mention clusters and hierarchies to a gold standard for evaluation is non-trivial.

Original: Evaluation metrics for coreference resolution (e.g., MUC, B 3 , CEAFe, and LEA) do not involve relations between clusters, and models for inferring hierarchical relations such as hypernymy do so over pairs of individual terms.
Revised: Evaluation metrics for coreference resolution (e.g., MUC, B 3 , CEAFe, and LEA) do not involve relations between clusters, and models for inferring hierarchical relations such as hypernymy do so over pairs of individual terms.

Original: Therefore, in addition to reporting established metrics for coreference resolution, we devise two speciﬁc metrics for our novel uniﬁed task.
Revised: Therefore, in addition to reporting established metrics for coreference resolution, we devise two speciﬁc metrics for our novel uniﬁed task.

Original: Cluster-level Hierarchy Score
Revised: Cluster-level Hierarchy Score

Original: In H-CDCR, hierarchical relations are deﬁned over clusters of mentions.
Revised: In H-CDCR, hierarchical relations are deﬁned over clusters of mentions.

Original: This complicates the evaluation of the hierarchy, since a system may output a different set of clusters from the gold due to coreference errors.
Revised: This complicates the evaluation of the hierarchy, since a system may output a different set of clusters from the gold due to coreference errors.

Original: Our cluster-level score is intended to evaluate hierarchical links without penalizing coreference errors a second time.
Revised: Our cluster-level score is intended to evaluate hierarchical links without penalizing coreference errors a second time.

Original: Let C SP −→ C SC be a hierar-
Revised: Let C SP −→ C SC be a hierar-

Original: [Figure]
Revised: [Figure]

Original: chical link output by the system between a parent cluster C SP and a child cluster C SC .
Revised: chical link output by the system between a parent cluster C SP and a child cluster C SC .

Original: We deﬁne this link to be a true positive iff there exists some pair of mentions in these two clusters that are also in a parent-child relationship in the gold hierarchy.
Revised: We deﬁne this link to be a true positive iff there exists some pair of mentions in these two clusters that are also in a parent-child relationship in the gold hierarchy.

Original: That is, it is positive iff there exist mentions p ∈ C SP and c ∈ C SC which are also in two gold clusters, i.e. p ∈ C GP and c ∈ C GC , such that C GP −→ C GC in the gold.
Revised: That is, it is positive iff there exist mentions p ∈ C SP and c ∈ C SC which are also in two gold clusters, i.e. p ∈ C GP and c ∈ C GC , such that C GP −→ C GC in the gold.

Original: 2 If not, the output link counts as a precision error.
Revised: 2 If not, the output link counts as a precision error.

Original: We deﬁne recall errors analogously, swapping system and gold in the above formalism.
Revised: We deﬁne recall errors analogously, swapping system and gold in the above formalism.

Original: As an example, see Figure 2, where p = IE and c = Pattern-based deﬁnition extraction .
Revised: As an example, see Figure 2, where p = IE and c = Pattern-based deﬁnition extraction .

Original: The metric treats this link as a true positive, since there is at least one pair of mentions in the gold over which the same hierarchy relation holds.
Revised: The metric treats this link as a true positive, since there is at least one pair of mentions in the gold over which the same hierarchy relation holds.

Original: Requiring at least one pair, rather than greater overlap, is intended to maximally decouple the coreference and hierarchy penalization.
Revised: Requiring at least one pair, rather than greater overlap, is intended to maximally decouple the coreference and hierarchy penalization.

Original: We also report results when requiring a more conservative 50% overlap, to examine robustness to this choice.
Revised: We also report results when requiring a more conservative 50% overlap, to examine robustness to this choice.

Original: Path-Distance Score
Revised: Path-Distance Score

Original: The cluster-level hierarchy score does not take into account the degree of separation between concepts in the graph.
Revised: The cluster-level hierarchy score does not take into account the degree of separation between concepts in the graph.

Original: We devise a score that does, following previous work on graph-based semantic distances [Lee et al., 1993].
Revised: We devise a score that does, following previous work on graph-based semantic distances [Lee et al., 1993].

Original: Here, we consider for each pair of mentions ( m 1 , m 2 ) the number of cluster-level directed edges needed to traverse from m 1 to m 2 , plus one (mentions in the same cluster have a distance of 1).
Revised: Here, we consider for each pair of mentions ( m 1 , m 2 ) the number of cluster-level directed edges needed to traverse from m 1 to m 2 , plus one (mentions in the same cluster have a distance of 1).

Original: For each mention pair, we compute this distance for both the ground-truth and the model (see Figure 2), and the ratio between the smaller of the two distances and the larger.
Revised: For each mention pair, we compute this distance for both the ground-truth and the model (see Figure 2), and the ratio between the smaller of the two distances and the larger.

Original: For pairs that are disconnected in the gold or the system but not both, we treat the ratio as zero, and pairs disconnected in both are ignored.
Revised: For pairs that are disconnected in the gold or the system but not both, we treat the ratio as zero, and pairs disconnected in both are ignored.

Original: The ratios are averaged to obtain the ﬁnal score.
Revised: The ratios are averaged to obtain the ﬁnal score.

Revised: [Figure]

Revised: More formally, let W be the union of mention pairs that have a path in the gold or system cluster-graph.

Revised: The path-distance score is deﬁned as follows:

Revised: [Equation]

Revised: where p gold ( i, j ) and p sys ( i, j ) are the distances between the mentions i and j in the gold and the system, respectively.

Original: This metric gives partial credit for similar but not exactly matching graphs, e.g., a predicted parent-child relation is not considered a complete error if the gold speciﬁes a coreference relation (the ratio is 1/2).
Revised: This metric gives partial credit for similar but not exactly matching graphs, e.g., a predicted parent-child relation is not considered a complete error if the gold speciﬁes a coreference relation (the ratio is 1/2).

Original: This can help in ambiguous cases; e.g., whether CRF model is a parent of CRF tagger or belongs in the same cluster.[Figure]
Revised: This can help in ambiguous cases; e.g., whether CRF model is a parent of CRF tagger or belongs in the same cluster.

Original: 3. S CI C O Data Construction
Revised: 3. S CI C O Data Construction

Original: Obtaining annotations for H-CDCR is non-trivial; showing annotators all possible pairs of documents is not feasible, and presenting randomly drawn mentions is ineffective as they will rarely be related.
Revised: Obtaining annotations for H-CDCR is non-trivial; showing annotators all possible pairs of documents is not feasible, and presenting randomly drawn mentions is ineffective as they will rarely be related.

Original: We thus follow work on data collection for coreference tasks [Cybulska and Vossen, 2014,
Revised: We thus follow work on data collection for coreference tasks [Cybulska and Vossen, 2014,

Original: Jain et al., 2020, Ravenscroft et al., 2021, Eirew et al., 2021] and bootstrap with existing resources.
Revised: Jain et al., 2020, Ravenscroft et al., 2021, Eirew et al., 2021] and bootstrap with existing resources.

Original: Our process relies on two primary ingredients: a large corpus of mentions in papers, and a set of seed taxonomies that we leverage to ﬁnd mentions that are likely to be coreferring or have hierarchical relations.
Revised: Our process relies on two primary ingredients: a large corpus of mentions in papers, and a set of seed taxonomies that we leverage to ﬁnd mentions that are likely to be coreferring or have hierarchical relations.

Original: Annotators are then asked to build clusters of mentions and hierarchical relations between them (as in Figure 1).
Revised: Annotators are then asked to build clusters of mentions and hierarchical relations between them (as in Figure 1).

Original: Below, we describe these two ingredients and how we use them for candidate selection.
Revised: Below, we describe these two ingredients and how we use them for candidate selection.

Original: Our overall data construction approach is illustrated in Figure 3.
Revised: Our overall data construction approach is illustrated in Figure 3.

Original: 3.1 Documents, Mentions and Seed taxonomies
Revised: 3.1 Documents, Mentions and Seed taxonomies

Original: We populate our dataset of papers and mentions from two data sources: (i) a large corpus of 10M
Revised: We populate our dataset of papers and mentions from two data sources: (i) a large corpus of 10M

Original: CS abstracts from [Lo et al., 2020] and (ii) 438 full-text AI papers from SciREX [Jain et al., 2020].
Revised: CS abstracts from [Lo et al., 2020] and (ii) 438 full-text AI papers from SciREX [Jain et al., 2020].

Original: We extract mentions referring to methods and tasks, using the DyGIE++ IE model [Wadden et al., 2019] trained on SciERC [Luan et al., 2018].
Revised: For the 10M abstracts, we extract mentions referring to methods and tasks, using the DyGIE++ IE model [Wadden et al., 2019] trained on SciERC [Luan et al., 2018].

Original: SciREX has the advantage of introducing mentions from full paper texts, vetted by a human annotator for quality.
Revised: SciREX has the advantage of introducing mentions from full paper texts, vetted by a human annotator for quality.

Original: We select mention sets from our corpus that are suitable for annotators to label.
Revised: We select mention sets from our corpus that are suitable for annotators to label.

Original: This requires identifying subsets that are likely to contain coreferent or hierarchically-related mentions.
Revised: This requires identifying subsets that are likely to contain coreferent or hierarchically-related mentions.

Original: We bootstrap such subsets with existing resources described brieﬂy below (see more details in Appendix A).
Revised: We bootstrap such subsets with existing resources described brieﬂy below (see more details in Appendix A).

Original: High-precision, low-coverage AI KB.
Revised: High-precision, low-coverage AI KB.

Original: Papers With Code 3 (PwC) is a public resource that maintains a hierarchical KB of ML related tasks, methods, datasets and results.
Revised: Papers With Code 3 (PwC) is a public resource that maintains a hierarchical KB of ML related tasks, methods, datasets and results.

Original: For example, subtasks of
Revised: For example, subtasks of

Original: Image Classiﬁcation include displaced people recognition and satellite image classiﬁcation .
Revised: Image Classiﬁcation include displaced people recognition and satellite image classiﬁcation .

Original: Corpus-level hypernym extraction.
Revised: Corpus-level hypernym extraction.

Original: To form a higher-coverage resource, we extract a broad (lower-precision) hierarchy of tasks and methods.
Revised: To form a higher-coverage resource, we extract a broad (lower-precision) hierarchy of tasks and methods.

Original: Speciﬁcally, we extract all hypernym relations from the 10M CS abstracts using the DyGIE++ model [Wadden et al., 2019] trained on SciERC.
Revised: Speciﬁcally, we extract all hypernym relations from the 10M CS abstracts using the DyGIE++ model [Wadden et al., 2019] trained on SciERC.

Original: The model extracts hypernym relations between uncanonicalized mentions that appear in the same sentence.
Revised: The model extracts hypernym relations between uncanonicalized mentions that appear in the same sentence.

Original: We form a hierarchy across the entire corpus by aggregating hyponyms of the same concept string across all papers (see Appendix A.1 for technical details).
Revised: We form a hierarchy across the entire corpus by aggregating hyponyms of the same concept string across all papers (see Appendix A.1 for technical details).

Original: Curated list of lexically diverse candidates.
Revised: Curated list of lexically diverse candidates.

Original: Automatically collecting lexically diverse mentions is challenging; while our two automated approaches capture such examples, they also capture many lexically similar mentions, a common issue in coreference datasets (§5.2).
Revised: Automatically collecting lexically diverse mentions is challenging; while our two automated approaches capture such examples, they also capture many lexically similar mentions, a common issue in coreference datasets (§5.2).

Original: To increase the diversity of S CI C O and enable more transparency in evaluation (§5.2), we enrich S CI C O with a manuallycurated collection of 60 groups of closely related but lexically diverse concepts (400 in total).
Revised: To increase the diversity of S CI C O and enable more transparency in evaluation (§5.2), we enrich S CI C O with a manuallycurated collection of 60 groups of closely related but lexically diverse concepts (400 in total).

Original: For example, one group contains { deep learning , neural models , DNN algorithms } , another includes { class imbalance , skewed label distribution , imbalanced data problem } .
Revised: For example, one group contains { deep learning , neural models , DNN algorithms } , another includes { class imbalance , skewed label distribution , imbalanced data problem } .

Original: 3.2 Candidate Retrieval
Revised: 3.2 Candidate Retrieval

Original: We now turn to how we use the resources described in the previous section in order to select candidate mentions for annotation.
Revised: We now turn to how we use the resources described in the previous section in order to select candidate mentions for annotation.

Original: Careful candidate generation is often necessary when collecting data for cross-document coreference [Cybulska and Vossen, 2014, Ravenscroft et al., 2021], since the vast majority of random mention pairs are easy negatives.
Revised: Careful candidate generation is often necessary when collecting data for cross-document coreference [Cybulska and Vossen, 2014, Ravenscroft et al., 2021], since the vast majority of random mention pairs are easy negatives.

Original: Following standard CDCR terminology, we refer to a batch of candidate mentions to be annotated together as a “topic”.
Revised: Following standard CDCR terminology, we refer to a batch of candidate mentions to be annotated together as a “topic”.

Original: Given a topic, annotators are asked to form coreference clusters of mentions and label hierarchical relationships between clusters.
Revised: Given a topic, annotators are asked to form coreference clusters of mentions and label hierarchical relationships between clusters.

Original: Restricting the number of mentions in a topic is necessary in order to make the quadratic complexity of the task tractable.
Revised: Restricting the number of mentions in a topic is necessary in order to make the quadratic complexity of the task tractable.

Original: We form topics of mentions (Figure 3 (3)) as follows.
Revised: We form topics of mentions (Figure 3 (3)) as follows.

Original: First, we merge the PwC and hypernyms graphs (de-duplicating nodes) and form groups of candidate concepts where each group consists of a single parent concept and its children.
Revised: First, we merge the PwC and hypernyms graphs (de-duplicating nodes) and form groups of candidate concepts where each group consists of a single parent concept and its children.

Original: We add our manually-curated groups to this set.
Revised: We add our manually-curated groups to this set.

Original: We then form a topic from each group by matching each of its concepts against our large-scale corpus (§3.1).
Revised: We then form a topic from each group by matching each of its concepts against our large-scale corpus (§3.1).

Original: Speciﬁcally, we retrieve the most similar mentions to each concept, in terms of cosine similarity of embeddings output by a neural model (details in Appendix A.2).
Revised: Speciﬁcally, we retrieve the most similar mentions to each concept, in terms of cosine similarity of embeddings output by a neural model (details in Appendix A.2).

Original: The union of the retrieved mentions (and their contexts) for a given group form a single topic in our data set (Figure 3 (4)).
Revised: The union of the retrieved mentions (and their contexts) for a given group form a single topic in our data set (Figure 3 (4)).

Original: The mention  retrieval step enriches S CI C O with ambiguous cases (e.g., references to information extraction with very different meanings), as well as ﬁne-grained variants of concepts.
Revised: The mention S retrieval step enriches S CI C O with ambiguous cases (e.g., references to information extraction with very different meanings), as well as ﬁne-grained variants of concepts.

Original: 3.3 Data Annotation
Revised: 3.3 Data Annotation

Original: Dataset annotation for our task is challenging and requires solid knowledge in computer science research.
Revised: Dataset annotation for our task is challenging and requires solid knowledge in computer science research.

Original: We hired 4 PhDs and graduate students  through UpWork 4 , all authors of at least two scientiﬁc publications.
Revised: We hired 4 PhDs and graduate students in CS through UpWork 4 , all authors of at least two scientiﬁc publications.

Revised: Annotators were paid competitively with a hourly reward between $ and $30 (2-3 topics per hour depending on their size).

Original: In addition to guidelines and tutorials, we also guide annotators to consider the faceted query by example [Mysore et al., 2021] application discussed in the Introduction, to help resolve ambiguity.
Revised: In addition to guidelines and tutorials, we also guide annotators to consider the faceted query by example [Mysore et al., 2021] application discussed in the Introduction, to help resolve ambiguity.

Original: As a concrete example consider two mentions, m 1 = ELMo model and m 2 = ELMo embedding .
Revised: As a concrete example consider two mentions, m 1 = ELMo model and m 2 = ELMo embedding .

Original: Searching for one and retrieving the other should usually be acceptable (indicating coreference), while searching for POS Tagging should showPenn TreeBank POS tagging in a hierarchy of results, but not the other direction, indicating referential hierarchy (see Appendix A.3).
Revised: Searching for one and retrieving the other should usually be acceptable (indicating coreference). By contrast, Penn TreeBank POS tagging should be annotated as a child of POS Tagging , since when issuing a faceted query for the PTB variant of POS tagging, in most cases we would not wish to see a list inundated with variants such as POS tagging in tweets or different languages; conversely, searching for POS Tagging should show a hierarchy of subsumed concepts, including speciﬁc variants. While this framing still leaves some room for subjectivity, we embrace it rather than attempt to formulate many complex rules with inevitably limited coverage. To ensure quality we also provided extensive feedback after the ﬁrst round of annotation.

Revised: We provide annotators with a sample of candidate mentions together with their surrounding context and some metadata of the paper such as title, venue, year and the link to the paper itself, as shown in Figure 3 (4).

Original:  3.4 S CI C O Properties
Revised: To annotate S CI C O , we extend CoReﬁ [Bornstein et al., 2020], a recent tool for coreference annotation, by enabling annotation of cluster hierarchies (see Appendix A.3 for more details). Annotators are asked to annotate both the clusters and relations between the clusters. We repeat this process for each topic in our pool. 3.4 S CI C O Properties

Original: [Table]
Revised: [Table]

Original: Table 2 shows summary statistics of S CI C O .
Revised: Table 2 shows summary statistics of S CI C O .

Original: Notably,
Revised: Notably,

Original: S CI C O includes over 26K mentions across about 10K clusters and 6K hierarchical relations.
Revised: S CI C O includes over 26K mentions across about 10K clusters and 6K hierarchical relations.

Original: S CI C O is 3 times larger than prominent CDCR benchmark ECB+ [Cybulska and
Revised: S CI C O is 3 times larger than prominent CDCR benchmark ECB+ [Cybulska and

Original: Vossen, 2014].
Revised: Vossen, 2014].

Original: Mentions in S CI C O are taken from over 20K scientiﬁc documents covering diverse CS concepts, larger than ECB+ by an order of magnitude.
Revised: Mentions in S CI C O are taken from over 20K scientiﬁc documents covering diverse CS concepts, larger than ECB+ by an order of magnitude.

Original: The average number of connected components across topics is 6 . 8 , and the average depth of the maximal component (tree) is 3 .
Revised: The average number of connected components across topics is 6 . 8 , and the average depth of the maximal component (tree) is 3 .

Original: To measure agreement, all annotators labeled the same 40 randomly chosen topics (groups of candidates, totalling about 2200 mentions).
Revised: To measure agreement, all annotators labeled the same 40 randomly chosen topics (groups of candidates, totalling about 2200 mentions).

Original: Following common evaluation practices in coreference resolution, we measure micro-averaged pairwise agreement (denoted AVG), considering one annotation as gold, and the other as predicted (metrics are symmetric).
Revised: Following common evaluation practices in coreference resolution, we measure micro-averaged pairwise agreement (denoted AVG), considering one annotation as gold, and the other as predicted (metrics are symmetric).

Original: We also measure the average maximal pairwise IAA, as an “upper bound” measure of human performance (MAX-micro).
Revised: We also measure the average maximal pairwise IAA, as an “upper bound” measure of human performance (MAX-micro).

Original: Finally we compute the maximal IAA scores for each topic, and average those 40 scores (MAX-macro).
Revised: Finally we compute the maximal IAA scores for each topic, and average those 40 scores (MAX-macro).

Original: {+Models for coreference resolution are traditionally evaluated using different metrics, MUC [Vilain et al., 1995], B 
+}
Revised: {+Models for coreference resolution are traditionally evaluated using different metrics, MUC [Vilain et al., 1995], B 
+}

Revised: [Bagga and Baldwin, 1998], CEAFe

Revised: [Luo, 2005] and LEA [Moosavi and

Revised: Strube, 2016], while the main evaluation is CoNLL F1, which is the average F1 of MUC, B 3 and

Revised: CEAFe.

Original: For CoNLL F1, AVG is 82 .
Revised: For CoNLL F1, AVG is 82 .

Original: 7( ± 2 . 5) , MAX-micro is 85 .
Revised: 7( ± 2 . 5) , MAX-micro is 85 .

Original: 8 and MAX-macro 90 .
Revised: 8 and MAX-macro 90 .

Original:  For our cluster-hierarchy F1 metric we get AVG of 68 .
Revised: 5 See more coreference metrics in Table 7 in Appendix C. For our cluster-hierarchy F1 metric we get AVG of 68 .

Original: 9( ± 2 . 2) , MAX-micro of 72 . 1 and MAX-macro of 82 .
Revised: 9( ± 2 . 2) , MAX-micro of 72 . 1 and MAX-macro of 82 .

Original: The path-distance agreement scores are, respectively, 64 . 5( ± 3 . 5 ), 70 . 0 , and 78 .
Revised: The path-distance agreement scores are, respectively, 64 . 5( ± 3 . 5 ), 70 . 0 , and 78 .

Original: Importantly, as we will see in §5 these IAA rates are substantially higher than our best model’s performance, leaving much room for future modelling.
Revised: Importantly, as we will see in §5 these IAA rates are substantially higher than our best model’s performance, leaving much room for future modelling.

Original: 4. Models
Revised: 4. Models

Original: We now present our models for H-CDCR.
Revised: We now present our models for H-CDCR.

Original: We start by presenting baselines that separately predict coreference clusters and the hierarchy between them (§4.1).
Revised: We start by presenting baselines that separately predict coreference clusters and the hierarchy between them (§4.1).

Original: Then, we describe a model that optimizes both subtasks simultaneously (§4.2).
Revised: Then, we describe a model that optimizes both subtasks simultaneously (§4.2).

Original: 4.1 Baseline Models Coreference Model
Revised: 4.1 Baseline Models Coreference Model

Original: We use a recent state-of-the-art cross-document coreference model [Cattan et al., 2020] for predicting clusters of mentions (denoted by CA ; see details in Appendix B.1).
Revised: We use a recent state-of-the-art cross-document coreference model [Cattan et al., 2020] for predicting clusters of mentions (denoted by CA ; see details in Appendix B.1).

Original: We experiment with several variants of this model.
Revised: We experiment with several variants of this model.

Original: First, to evaluate how S CI C O differs from existing CDCR datasets, we explore two versions trained only on external CDCR resources: CA News trained on ECB+ [Cybulska and Vossen, 2014] and CA Sci-News , trained on CD 2 CR [Ravenscroft et al., 2021] that includes coreference annotation between a single news article and linked scientiﬁc paper with mentions extracted using NER.
Revised: First, to evaluate how S CI C O differs from existing CDCR datasets, we explore two versions trained only on external CDCR resources: CA News trained on ECB+ [Cybulska and Vossen, 2014] and CA Sci-News , trained on CD 2 CR [Ravenscroft et al., 2021] that includes coreference annotation between a single news article and linked scientiﬁc paper with mentions extracted using NER.

Original: Next, we train the model on S CI C O .
Revised: Next, we train the model on S CI C O .

Original: In addition to the
Revised: In addition to the

Original: RoBERTa ( CA RoBERTa ) pretrained language model used in [Cattan et al., 2020], we explore models specialized to our scientiﬁc domain: CS-RoBERTa
Revised: RoBERTa ( CA RoBERTa ) pretrained language model used in [Cattan et al., 2020], we explore models specialized to our scientiﬁc domain: CS-RoBERTa

Original: ( CA CS-RoBERTa )
Revised: ( CA CS-RoBERTa )

Original: [Gururangan et al., 2020] and
Revised: [Gururangan et al., 2020] and

Original: SciBERT ( CA SciBERT ) [Beltagy et al., 2019].
Revised: SciBERT ( CA SciBERT ) [Beltagy et al., 2019].

Original: Hierarchy Model
Revised: Hierarchy Model

Original: We consider a referential hierarchical relation between concepts x and y as an entailment relation between y and x [Glockner et al., 2018].
Revised: We consider a referential hierarchical relation between concepts x and y as an entailment relation between y and x [Glockner et al., 2018].

Original: 6 We use a model for textual entailment (RoBERTa-large-MNLI)
Revised: 6 We use a model for textual entailment (RoBERTa-large-MNLI)

Original: [Liu et al., 2019], representing each cluster C by the concatenation of its mentions (without context)
Revised: [Liu et al., 2019], representing each cluster C by the concatenation of its mentions (without context)

Original: [CLS] C k 

 C j 
 , and running RoBERTa-large-MNLI on all n ( n − 1) cluster pairs ( C k , C j ) .
Revised: [CLS] C k 

 C j 
 , and running RoBERTa-large-MNLI on all n ( n − 1) cluster pairs ( C k , C j ) .

Original: The [CLS] embedding is fed into an output layer for entailment classiﬁcation.
Revised: The [CLS] embedding is fed into an output layer for entailment classiﬁcation.

Original: To avoid creation of cyclical graphs, we adopt a greedy approach adding relations iteratively, starting from the highest entailment scores, discarding relations creating cycles.
Revised: To avoid creation of cyclical graphs, we adopt a greedy approach adding relations iteratively, starting from the highest entailment scores, discarding relations creating cycles.

Original: We apply this model to clusters obtained by each baseline.
Revised: We apply this model to clusters obtained by each baseline.

Original: 4.2 Uniﬁed Model for H-CDCR
Revised: 4.2 Uniﬁed Model for H-CDCR

Original: Deciding whether two mentions refer to the same concept or have a hierarchical relation is sometimes non-trivial (e.g POS Tagging −→ PTB POS tagging but deep neural networks and neural networks are equivalent).
Revised: Deciding whether two mentions refer to the same concept or have a hierarchical relation is sometimes non-trivial (e.g POS Tagging −→ PTB POS tagging but deep neural networks and neural networks are equivalent).

Original: Therefore, we develop a uniﬁed model by formulating our task as multiclass classiﬁcation, where each mention pair ( m 1 , m 2 ) can be assigned into four classes (0) m 1 , m 2 corefer (1) m 1 −→ m 2 (2) m 2 −→ m 1 or (3) not related.
Revised: Therefore, we develop a uniﬁed model by formulating our task as multiclass classiﬁcation, where each mention pair ( m 1 , m 2 ) can be assigned into four classes (0) m 1 , m 2 corefer (1) m 1 −→ m 2 (2) m 2 −→ m 1 or (3) not related.

Original: Consider a topic (pool of candidates, §3.2), with a set of mentions M .
Revised: Consider a topic (pool of candidates, §3.2), with a set of mentions M .

Original: During training, we learn a pairwise scorer f ( · , · ) by minimizing:
Revised: During training, we learn a pairwise scorer f ( · , · ) by minimizing:

Original: [Equation]
Revised: [Equation]

Original: where 1 -hot y is one of the four classes and N is the number of training pairs.
Revised: where 1 -hot y is one of the four classes and N is the number of training pairs.

Original: For f ( · , · ) we use Longformer [Beltagy et al., 2020], a transformer-based language model for processing long sequences so that we can encode pairs of full paragraphs.
Revised: For f ( · , · ) we use Longformer [Beltagy et al., 2020], a transformer-based language model for processing long sequences so that we can encode pairs of full paragraphs.

Original: We also use CDLM, a recent variant of
Revised: We also use CDLM, a recent variant of

Original: Longformer pre-trained for cross-document tasks [Caciularu et al., 2021].
Revised: Longformer pre-trained for cross-document tasks [Caciularu et al., 2021].

Revised: During pretraining, Longformer and CDLM apply local attention — attention only to tokens in a ﬁxed-sized window around each token.

Revised: When ﬁne-tuning on a speciﬁc task, global attention — attention to all tokens in the sequence — can be assigned to a few target tokens to encode global information.

Revised: Following Caciularu et al.

Original: See Appendix B.2 for full details on how we employ each model.
Revised: [2021], we take each mention and its corresponding paragraph, inserting mention markers <m> and </m> surrounding the mention to obtain a mention representation. For CDLM, we add the document markers <doc-s> and </doc-s> surrounding each document. Then, we concatenate the representations of m 1 and m 2 separated by a separator token 
 , and add a [CLS] token at the beginning. We assign global attention to the [CLS] and the mention markers of the two mentions.

Revised: The [CLS] vector is ﬁnally fed into a linear layer

Revised: W d × 4 , for ﬁne-tuning the model.

Original: [Table]
Revised: [Table]

Original: We evaluate strong

Original: CDCR baselines, and a uniﬁed multiclass model for H-CDCR that outperforms the baselines.

Original: At inference time, we build clusters using agglomerative clustering over predicted coreference scores, in the same way as in the baseline described above (§4.1).
Revised: At inference time, we build clusters using agglomerative clustering over predicted coreference scores, in the same way as in the baseline described above (§4.1).

Original: Then, for each pair of clusters ( C 1 , C 2 ) , we aggregate crosscluster mention-pair predictions for hierarchical relations as follows.
Revised: Then, for each pair of clusters ( C 1 , C 2 ) , we aggregate cross-cluster mentionpair predictions for hierarchical relations as follows.

Original: Given a pair ( m 1 , m 2 ) where m 1 ∈ C 1 and m 2 ∈ C 2 , we compute the probability score for m 1 being a child of m 2 , and deﬁne the score of C 1 being the child of C 2 as the average of all pairwise scores for all { ( m i , m j ) | m i∈ C 1 , m j ∈ C 2 } : s ( C 1 , C 2 ) = [Equation]
Revised: Given a pair ( m 1 , m 2 ) where m∈ C 1 and m 2 ∈ C 2 , we compute the probability score for m 1 being a child of m 2 , and deﬁne the score of Cbeing the child of C 2 as the average of all pairwise scores for all { ( m i , m j ) | m i

Original: [Equation]
Revised: [Equation]

Original: To avoid cycles we apply the same greedy approach as in the baseline models (§4.1) and stop when the hierarchy score is below a tuned threshold.
Revised: To avoid cycles we apply the same greedy approach as in the baseline models (§4.1) and stop when the hierarchy score is below a tuned threshold.

Original: Full implementation details and hyperparameters are described in Appendix B.3.
Revised: Full implementation details and hyperparameters are described in Appendix B.2.

Original: 5. Experimental Results
Revised: 5. Experimental Results

Original: Table 3 presents the results of the baseline models as well as our uniﬁed solution
Revised: Table 3 presents the results of the baseline models as well as our uniﬁed solution.

Revised: Results of coreference are reported using the CoNLL F1 metric.

Original: We also report performance with our two metrics introduced earlier (§2.2).See Appendix C for all coreference metrics, as well as recall and precision for the hierarchy score.
Revised: We also report performance with our two metrics introduced earlier (§2.2).See Appendix C for all coreference metrics, as well as recall and precision for the hierarchy score.

Original: The results show that training a model to predict both coreference and hierarchy in a multiclass setup outperforms baselines by a large margin across all metrics.
Revised: The results show that training a model to predict both coreference and hierarchy in a multiclass setup outperforms baselines by a large margin across all metrics.

Original: In terms of coreference, using the CA baselines (§4.1) trained on external datasests leads to the lowest results.
Revised: In terms of coreference, using the CA baselines (§4.1) trained on external datasests leads to the lowest results.

Original: Training the same model on S CI C O boosts results, with a signiﬁcant boost from SciBERT [Beltagy et al., 2019].
Revised: Training the same model on S CI C O boosts results, with a signiﬁcant boost from SciBERT [Beltagy et al., 2019].

Original: Focusing on hierarchy, we examine results in terms of our cluster-level hierarchy metric (§2.2) which is designed to not doubly-penalize coreference errors, and the path-based metric.
Revised: Focusing on hierarchy, we examine results in terms of our cluster-level hierarchy metric (§2.2) which is designed to not doubly-penalize coreference errors, and the path-based metric.

Original: For all baselines, we use a state-of-art pre-trained entailment model [Liu et al., 2019] (see §4) for predicting relations between clusters, which leads to poor results in comparison to the uniﬁed model.
Revised: For all baselines, we use a state-of-art pre-trained entailment model [Liu et al., 2019] (see §4) for predicting relations between clusters, which leads to poor results in comparison to the uniﬁed model.

Original: 5.1 Ablation Study
Revised: 5.1 Ablation Study

Original: [Table]
Revised: [Table]

Original: We conduct an ablation study examining our multiclass formulation and utility of wider contexts.
Revised: We conduct an ablation study examining our uniﬁed formulation and utility of wider contexts.

Original: To ablate the multiclass model, we train two models using the same architecture (§4.2): one for coreference only, the other for hierarchy only.
Revised: To ablate the uniﬁed model (multiclass), we train two models using the same architecture (§4.2): one for coreference only, the other for hierarchy only.

Original: In the former, we consider hierarchical relations as negative pairs, and train a binary classiﬁcation model with classes (0) unrelated and (1) coreference.
Revised: In the former, we consider hierarchical relations as negative pairs, and train a binary classiﬁcation model with classes (0) unrelated and (1) coreference.

Original: The hierarchy-only model has classes (0) unrelated, (1) parent-child and (2) child-parent.
Revised: The hierarchy-only model has classes (0) unrelated, (1) parent-child and (2) child-parent.

Original: During inference, we apply the same approach in (§4.2) for creating clusters and inferring cluster hierarchy.
Revised: During inference, we apply the same approach in (§4.2) for creating clusters and inferring cluster hierarchy.

Original: As shown in Table 4, the uniﬁed model outperforms the disjoint approach in the cluster-hierarchy and path-based scores, indicating the utility of optimizing for both tasks simultaneously.
Revised: As shown in Table 4, the uniﬁed model outperforms the disjoint approach in the cluster-hierarchy and path-based scores, indicating the utility of optimizing for both tasks simultaneously.

Original: Learning a single model also results in half the number of parameters than in the disjoint approach, while achieving substantially better results.
Revised: Learning a single model also results in half the number of parameters than in the disjoint approach, while achieving substantially better results.

Original: Finally, using only a mention’s sentence as context (instead of full paragraph) also leads to a performance drop.
Revised: Finally, using only a mention’s sentence as context (instead of full paragraph) also leads to a performance drop.

Original: 5.2 Lexical Diversity Impact on Coreference
Revised: 5.2 Lexical Diversity Impact on Coreference

Original: [Table]
Revised: [Table]

Original: We explore how coreference performance correlates with lexical diversity (and ambiguity).
Revised: explore how coreference performance correlates with lexical diversity (and ambiguity).

Original: We examine a simple baseline that uses the Levenshtein edit distancein agglomerative clustering.
Revised: We examine a simple baseline that uses the Levenshtein edit distance

Original: Surface-form matching baselines are known to have comparatively fair performance in CDCR datasets [Barhom et al., 2019, Eirew et al., 2021].
Revised: Surface-form matching baselines are known to have comparatively fair performance in CDCR datasets [Barhom et al., 2019, Eirew et al., 2021].

Original: We take the bottom 10% and bottom 20% of topics ranked by the baseline’s CoNLL F (20 and 40 topics, respectively).
Revised: We take the bottom 10% and bottom 20% of topics ranked by the baseline’s CoNLL F (20 and 40 topics, respectively).

Original: We also examine the set of manually curated lexically diverse topics (§3.1).
Revised: We also examine the set of manually curated lexically diverse topics (§3.1).

Original: In Table 5, we report micro-averaged results for each subset.
Revised: In Table 5, we report micro-averaged results for each subset.

Original: Our uniﬁed model does substantially worse than its overall performance (e.g., the model does not identify coreference between scientiﬁc paper analysis and scholarly document analysis , manual annotation and human labeling ).
Revised: Our uniﬁed model does substantially worse than its overall performance (e.g., the model does not identify coreference between scientiﬁc paper analysis and scholarly document analysis , manual annotation and human labeling ).

Original: We also check that the edit-distance baseline does not correlate with inter-annotator agreement, to ensure IAA rates reported earlier can serve as a measure of human performance across levels of diversity.
Revised: We also check that the edit-distance baseline does not correlate with inter-annotator agreement, to ensure IAA rates reported earlier can serve as a measure of human performance across levels of diversity.

Original: Using Pearson and Spearman correlation tests, we conﬁrm this hypothesis (p-values of 0.10, 0.21).
Revised: Using Pearson and Spearman correlation tests, we conﬁrm this hypothesis (p-values of 0.10, 0.21).

Original: These results indicate S CI C O contains subsets with varying levels of lexical diversity that correlate with coreference difﬁculty, requiring richer information and models to resolve.
Revised: These results indicate S CI C O contains subsets with varying levels of lexical diversity that correlate with coreference difﬁculty, requiring richer information and models to resolve.

Original: 6. Related Work Entity coreference resolution
Revised: 6. Related Work Entity coreference resolution

Original: Entity coreference work focuses on mentions within a single document (WD), while cross-document (CD) work focuses on coreference between events in the news [Cybulska and Vossen, 2014, Vossen et al., 2018].
Revised: Entity coreference work focuses on mentions within a single document (WD), while cross-document (CD) work focuses on coreference between events in the news [Cybulska and Vossen, 2014, Vossen et al., 2018].

Original: In the science domain, some work has been done on entity coreference in the WD setting [Luan et al., 2018, Jain et al., 2020].
Revised: In the science domain, some work has been done on entity coreference in the WD setting [Luan et al., 2018, Jain et al., 2020].

Original: Unlike previous work, in this paper we considered CDCR jointly with inferring hierarchy, for a domain with abstract technical concepts that are nested in many levels of granularity that can be hard to tell apart.
Revised: Unlike previous work, in this paper we considered CDCR jointly with inferring hierarchy, for a domain with abstract technical concepts that are nested in many levels of granularity that can be hard to tell apart.

Original: Entity linking
Revised: Entity linking

Original: Entity linking (EL) involves linking mentions of entities to knowledge base (KB) entries.
Revised: Entity linking (EL) involves linking mentions of entities to knowledge base (KB) entries.

Original: In science, KBs are often scarce and highly incomplete [Hope et al., 2021].
Revised: In science, KBs are often scarce and highly incomplete [Hope et al., 2021].

Original: In our work, we used a low-coverage KB to bootstrap data collection and did not assume to have a KB during training or inference.
Revised: In our work, we used a low-coverage KB to bootstrap data collection and did not assume to have a KB during training or inference.

Original: Recent work [Angell et al., 2021] has shown that clustering WD mentions can boost EL in biomedical papers; our work on CD mentions could help by pooling information from diverse contexts, and potentially detect entities missing from KBs [Lin et al., 2012, Wu et al., 2016].
Revised: Recent work [Angell et al., 2021] has shown that clustering WD mentions can boost EL in biomedical papers; our work on CD mentions could help by pooling information from diverse contexts, and potentially detect entities missing from KBs [Lin et al., 2012, Wu et al., 2016].

Original: Taxonomy construction
Revised: Taxonomy construction

Original: Most work in this area generates a graph where nodes are single terms representing a concept (e.g., hypernym-hyponym pairs extracted from single sentences)
Revised: Most work in this area generates a graph where nodes are single terms representing a concept (e.g., hypernym-hyponym pairs extracted from single sentences)

Original: [Hearst, 1992, Roller et al., 2018].
Revised: [Hearst, 1992, Roller et al., 2018].

Original: This approach does not resolve ambiguity of identical surface forms (e.g., information extraction ) and lexical diversity of concepts.
Revised: This approach does not resolve ambiguity of identical surface forms (e.g., information extraction ) and lexical diversity of concepts.

Original: Recent work in the data mining community [Shang et al., 2020] focused on unsupervised construction of taxonomies, with clusters of uncontextualized terms given as input.
Revised: Recent work in the data mining community [Shang et al., 2020] focused on unsupervised construction of taxonomies, with clusters of uncontextualized terms given as input.

Original: Our work can potentially help these and related applications [Poon and Domingos, 2010] by introducing context and supervision.
Revised: Our work can potentially help these and related applications [Poon and Domingos, 2010] by introducing context and supervision.

Original: Word sense induction
Revised: Word sense induction

Original: A related line of work focuses on learning to induce multiple senses of words from text, to capture polysemy and resolve ambiguity.
Revised: A related line of work focuses on learning to induce multiple senses of words from text, to capture polysemy and resolve ambiguity.

Original: Such work typically employs uncontextualized word embeddings
Revised: Such work typically employs uncontextualized word embeddings

Original: [Athiwaratkun and Wilson, 2017, Arora et al., 2018] or, recently, phrase-level embeddings [Chang et al., 2021].
Revised: [Athiwaratkun and Wilson, 2017, Arora et al., 2018] or, recently, phrase-level embeddings [Chang et al., 2021].

Original: Relatedly, Shwartz and Dagan [2016] label pairs of words for equivalent senses depending on context, in general language rather than abstract technical concepts.
Revised: Relatedly, Shwartz and Dagan [2016] label pairs of words for equivalent senses depending on context, in general language rather than abstract technical concepts.

Original: 7. Conclusion
Revised: 7. Conclusion

Original: We present S CI C O , a dataset for a novel task of hierarchical cross-document coreference resolution (H-CDCR) in the challenging setting of scientiﬁc texts.
Revised: We present S CI C O , a dataset for a novel task of hierarchical cross-document coreference resolution (H-CDCR) in the challenging setting of scientiﬁc texts.

Original: S CI C O is annotated by domain experts and is three times larger than comparable datasets from the news domain.
Revised: S CI C O is annotated by domain experts and is three times larger than comparable datasets from the news domain.

Original: We evaluate strong baseline models on our data.
Revised: We evaluate strong baseline models on our data.

Original: A joint approach that infers both coreference and hierarchical relationships in the same model outperforms multiple baselines, but leaves substantial room for improvement.
Revised: A joint approach that infers both coreference and hierarchical relationships in the same model outperforms multiple baselines, but leaves substantial room for improvement.

Original: Appendix A. Data Collection A.1 Hypernym extraction details
Revised: Appendix A. Data Collection A.1 Hypernym extraction details

Original: In our corpus-level hypernym extraction procedure, we use standard surface-level string normalization to unify mentions across the corpus (removing punctuation, lemmatizing, lowercasing and the
Revised: In our corpus-level hypernym extraction procedure, we use standard surface-level string normalization to unify mentions across the corpus (removing punctuation, lemmatizing, lowercasing and the

Original: Levenshtein edit-distance with threshold 0.8), resulting in distinct 250K tasks and 1.2M methods with 340K and 1.4M hierarchical edges between them respectively.
Revised: Levenshtein edit-distance with threshold 0.8), resulting in distinct 250K tasks and 1.2M methods with 340K and 1.4M hierarchical edges between them respectively.

Original: We then sample parent-child mentions and siblings to form our candidates (see Figure 3 (1-2)).
Revised: We then sample parent-child mentions and siblings to form our candidates (see Figure 3 (1-2)).

Original: In addition to surfacing candidates for the referential hierarchy, another signiﬁcant advantage of this resource is indeed one of its main weaknesses (discussed in §6): it produces coreferring mentions of the same concept in the form of hyponyms of a shared parent, which we use to enrich S CI C O for the CDCR task (e.g., two different papers mentioning image synthesis or image generation as children of computer vision tasks ).
Revised: In addition to surfacing candidates for the referential hierarchy, another signiﬁcant advantage of this resource is indeed one of its main weaknesses (discussed in §6): it produces coreferring mentions of the same concept in the form of hyponyms of a shared parent, which we use to enrich S CI C O for the CDCR task (e.g., two different papers mentioning image synthesis or image generation as children of computer vision tasks ).

Original: While this automatically extracted hierarchy has broad coverage, it suffers from noise.
Revised: While this automatically extracted hierarchy has broad coverage, it suffers from noise.

Original: 7 To reduce noise, an annotator who did not take part in the ﬁnal data annotation (§3.3) was given a set of generated candidates and post-processed to ﬁlter overly generic or noisy spans.
Revised: 7 To reduce noise, an annotator who did not take part in the ﬁnal data annotation (§3.3) was given a set of generated candidates and post-processed to ﬁlter overly generic or noisy spans.

Original: A.2 Mention Retrieval Details
Revised: A.2 Mention Retrieval Details

Original: Formally, denote by S
Revised: Formally, denote by S

Original: =
Revised: =

Original: ( V, E ) the union of the PwC and hypernym graphs.
Revised: ( V, E ) the union of the PwC and hypernym graphs.

Original: For a parent concept p ∈ V , let { c } p be its set of direct descendants (children).
Revised: For a parent concept p ∈ V , let { c } p be its set of direct descendants (children).

Original: We take the parent vertex p and its child vertices { c } p , and add the 60 curated groups to this set, to form a complete set of candidates we denote by C p .
Revised: We take the parent vertex p and its child vertices { c } p , and add the 60 curated groups to this set, to form a complete set of candidates we denote by C p .

Original: For the sources making up S , the concept names in PwC and the curated lists are detached from any speciﬁc paper context, which is required for building S CI C O .
Revised: For the sources making up S , the concept names in PwC and the curated lists are detached from any speciﬁc paper context, which is required for building S CI C O .

Original: While the extracted hypernym relations do come from speciﬁc sentences, using only those contexts as candidates would bias models to focus only on within-document relations.
Revised: While the extracted hypernym relations do come from speciﬁc sentences, using only those contexts as candidates would bias models to focus only on within-document relations.

Original: In addition, we wish to diversify our the surface forms of mentions by retrieving subtle variants (e.g., retrieving for BERT mentions such as BERT model ,
Revised: In addition, we wish to diversify our the surface forms of mentions by retrieving subtle variants (e.g., retrieving for BERT mentions such as BERT model ,

Original: BERT architecture , BERT-based representation , etc.).
Revised: BERT architecture , BERT-based representation , etc.).

Original: Thus, we augment each group by retrieving similar mentions from our corpus.
Revised: Thus, we augment each group by retrieving similar mentions from our corpus.

Original: Speciﬁcally, for each C p
Revised: Speciﬁcally, for each C p

Original: we ﬁnd similar mentions for each c ∈ C p .
Revised: we ﬁnd similar mentions for each c ∈ C p .

Original: We use an encoding function f : c (cid:55)→ R d that maps the surface form of each selected c to f ( c ) , a d dimensional vector.
Revised: We use an encoding function f : c (cid:55)→ R d that maps the surface form of each selected c to f ( c ) , a d dimensional vector.

Original: Following the approach of [Reimers and Gurevych, 2019], the encoding function is obtained by ﬁne-tuning a language model pre-trained on computer science papers [Gururangan et al., 2020] on two semantic similarity tasks,
Revised: Following the approach of [Reimers and Gurevych, 2019], the encoding function is obtained by ﬁne-tuning a language model pre-trained on computer science papers [Gururangan et al., 2020] on two semantic similarity tasks,

Original: STS [Cer et al., 2017] and SNLI [Bowman et al., 2015].
Revised: STS [Cer et al., 2017] and SNLI [Bowman et al., 2015].

Original: We then apply f ( · ) to all distinct mentions in our large-scale corpus with over 30M mentions.
Revised: We then apply f ( · ) to all distinct mentions in our large-scale corpus with over 30M mentions.

Original: Finally, we augment each group with the top K highest-scoring mentions by cosine similarity to each c ∈ C p in the initial group and take the union of retrieved results.
Revised: Finally, we augment each group with the top K highest-scoring mentions by cosine similarity to each c ∈ C p in the initial group and take the union of retrieved results.

Original: To manage the scale, we employ a system designed for fast similarity-based search [Johnson et al., 2017].
Revised: To manage the scale, we employ a system designed for fast similarity-based search [Johnson et al., 2017].

Original: 9 To make sure we sample enough mentions from SciREX despite its comparatively small size, we sample from it with the same proportion as from the 10M abstracts.
Revised: 9 To make sure we sample enough mentions from SciREX despite its comparatively small size, we sample from it with the same proportion as from the 10M abstracts.

Original: A.3 Annotation Process
Revised: A.3 Annotation Interface

Original: To ensure quality we provided extensive feedback after the ﬁrst round of annotation.

Original: Annotators were paid competitively with a hourly reward between $20 and $30.

Original: We provide annotators with a sample of candidate mentions together with their surrounding context and some metadata of the paper such as title, venue, year and the link to the paper itself, as shown in Figure 3 (4).

Original: Annotators are asked to annotate both the clusters and relations between the clusters.

Original: We repeat this process for each topic in our pool.

Original: Faceted query-by-example framing

Original: We provide an expanded version of the example discussed in §3.3.

Original: Consider two mentions m 1 = ELMo model and m 2 =

Original: ELMo embedding shown to an annotator.

Original: We ask annotators to think of m 1 as a selected facet (with its context) used as a query.

Original: We retrieve a list of more mentions in context, one of them being m 2 .

Original: Considering the query m 1 , papers with the mention ELMo embedding should in most cases be acceptable as a retrieved result, and vice versa — hence they belong in the same cluster.

Original: By contrast, Penn TreeBank POS tagging should be annotated as a child of POS Tagging , since when issuing a faceted query for the PTB variant of POS tagging, in most cases we would not wish to see a list inundated with variants such as POS tagging in tweets or different languages; conversely, searching for POS Tagging should show a hierarchy of subsumed concepts, including speciﬁc variants.

Original: While this framing still leaves some room for subjectivity, we embrace it rather than attempt to formulate many complex rules with inevitably limited coverage.

Original: Annotation Interface

Original: To annotate S CI C O , we extend CoReﬁ [Bornstein et al., 2020] with the ability to annotate hierarchy of clusters.
Revised: To annotate S CI C O , we extend CoReﬁ [Bornstein et al., 2020] with the ability to annotate hierarchy of clusters.

Original: The hierarchy is kept in sync with the clusters – any modiﬁcation of the clusters (e.g., merging two clusters) affects automatically the hierarchy (e.g., unifying their children).
Revised: The hierarchy is kept in sync with the clusters – any modiﬁcation of the clusters (e.g., merging two clusters) affects automatically the hierarchy (e.g., unifying their children).

Original: Therefore, annotators can annotate both the clusters and the hierarchy at the same time or alternate between them.
Revised: Therefore, annotators can annotate both the clusters and the hierarchy at the same time or alternate between them.

Original: In addition, annotators can (and are encouraged to) add some notes in the hierarchy to help them distinguish between ambiguous concepts.
Revised: In addition, annotators can (and are encouraged to) add some notes in the hierarchy to help them distinguish between ambiguous concepts.

Original: As shown in Figure 3, we extend CoReﬁ by displaying metadata for each paper, including the link to the Semantic Scholar URL of the paper.
Revised: As shown in Figure 3, we extend CoReﬁ by displaying metadata for each paper, including the link to the Semantic Scholar URL of the paper.

Original: This additional context is often helpful in order to annotate complex cases.
Revised: This additional context is often helpful in order to annotate complex cases.

Original: For example, as shown in Table 1, for resolving the ambiguity between two mentions of cross-view training , it may be useful to see that one paper was published a few years before the other one coined its own cross-view training as a name for their semi-supervised model.
Revised: For example, as shown in Table 1, for resolving the ambiguity between two mentions of cross-view training , it may be useful to see that one paper was published a few years before the other one coined its own cross-view training as a name for their semi-supervised model.

Original: Appendix B. Models B.1 Cattan et al. [2020]’s CDCR model
Revised: Appendix B. Models B.1 Cattan et al. [2020]’s CDCR model

Original: Here, we describe the cross-document resolver of [Cattan et al., 2020] that we use as baseline for S CI C O (§4.1).
Revised: Here, we describe the cross-document resolver of [Cattan et al., 2020] that we use as baseline for S CI C O (§4.1).

Original: This model is based on contextualized token and span representations.
Revised: This model is based on contextualized token and span representations.

Original: For a document d , we are given a sequence of tokens x = { x 1 , x 2 , . . .
Revised: For a document d , we are given a sequence of tokens x = { x 1 , x 2 , . . .

Original: , x T } where T is the length of the document.
Revised: , x T } where T is the length of the document.

Original: We ﬁrst obtain a contextualized embedding of each x t ∈ x using the RoBERTa-large pre-trained language model [Liu et al., 2019].
Revised: We ﬁrst obtain a contextualized embedding of each x t ∈ x using the RoBERTa-large pre-trained language model [Liu et al., 2019].

Original: Each mention span i is a contiguous subsequence of x , denoted x i
Revised: Each mention span i is a contiguous subsequence of x , denoted x i

Original: = { x ( i ) 1 , x
Revised: = { x ( i ) 1 , x

Original: ( i ) 2 , . . .
Revised: ( i ) 2 , . . .

Original: , x ( i ) S } .
Revised: , x ( i ) S } .

Original: Let ˆx i
Revised: Let ˆx i

Original: = { ˆ x ( i ) 1 , ˆ x ( i ) 2 , . . .
Revised: = { ˆ x ( i ) 1 , ˆ x ( i ) 2 , . . .

Original: , ˆ x ( i ) S } be the sequence of embedded tokens in span i , where ˆ x ( i ) t is the embedding of token t in span i .
Revised: , ˆ x ( i ) S } be the sequence of embedded tokens in span i , where ˆ x ( i ) t is the embedding of token t in span i .

Original: Then, similarly to Lee et al.
Revised: Then, similarly to Lee et al.

Original: [2017], each span i is represented (2) by the concatenation of embeddings of the boundary tokens in the span ( ˆ x ( i ) 1 and ˆ x ( i ) S ), an attention-weighted sum of the token embeddings in span i
Revised: [2017], each span i is represented (3) by the concatenation of embeddings of the boundary tokens in the span ( ˆ x ( i ) 1 and ˆ x ( i ) S ), an attention-weighted sum of the token embeddings in span i

Original: (Attn ( ˆx i ) ), and a feature vector denoting the span width (number of tokens).
Revised: (Attn ( ˆx i ) ), and a feature vector denoting the span width (number of tokens).

Original: [Equation]
Revised: [Equation]

Original: Given a pair of contextualized span embeddings g i and g j from two different documents, we feed them to a pairwise scorer (3) in the form of a simple feed-forward network that receives as input the concatenation of two span representations and their element-wise product, and outputs the likelihood that the two mentions corefer.
Revised: Given a pair of contextualized span embeddings g i and g j from two different documents, we feed them to a pairwise scorer (4) in the form of a simple feed-forward network that receives as input the concatenation of two span representations and their element-wise product, and outputs the likelihood that the two mentions corefer.

Original: [Equation]
Revised: [Equation]

Original: At inference time, the coreference clusters are formed by applying agglomerative clustering [Ward Jr, 1963] over a pairwise distance matrix populated with scores s ( i, j ) , using the averagelinkage cluster merging criterion.
Revised: At inference time, the coreference clusters are formed by applying agglomerative clustering [Ward Jr, 1963] over a pairwise distance matrix populated with scores s ( i, j ) , using the averagelinkage cluster merging criterion.

Original: Following [Cattan et al., 2020], this model does not involve ﬁne-tuning of the underlying language model due to prohibitive memory costs — it begins with encoding all documents using an LM, then computes the pairwise scores over all pairs of mention representations.
Revised: Following [Cattan et al., 2020], this model does not involve ﬁne-tuning of the underlying language model due to prohibitive memory costs — it begins with encoding all documents using an LM, then computes the pairwise scores over all pairs of mention representations.

Original: We note that our uniﬁed approach is the only model we evaluate that ﬁne-tunes the
Revised: We note that our uniﬁed approach is the only model we evaluate that ﬁne-tunes the

Original: Longformer on our data.
Revised: Longformer on our data.

Original: B.2 Longformer and CDLM

Original: During pretraining, Longformer and CDLM apply local attention — attention only to tokens in a ﬁxed-sized window around each token.

Original: When ﬁne-tuning on a speciﬁc task, global attention — attention to all tokens in the sequence — can be assigned to a few target tokens to encode global information.

Original: Following Caciularu et al. [2021], we take each mention and its corresponding sentence, inserting mention markers <m> and </m> surrounding the mention to obtain a mention representation. For CDLM, we add the document markers <doc-s> and </doc-s> surrounding each document. Then, we concatenate the representations of m 1 and m 2 separated by a separator token 
 , and add a [CLS] token at the beginning.
Revised: [Table]

Original: We assign global attention to the [CLS] and the mention markers of the two mentions.

Original: The [CLS] vector is ﬁnally fed into a linear layer

Original: W d × 4 , for ﬁne-tuning the model.

Original: B.3 Hyper parameters for the Uniﬁed Model
Revised: B.2 Hyper parameters for the Uniﬁed Model

Original: We develop our model in Pytorch [Paszke et al., 2019] and PytorchLightning [Falcon et al., 2019] using the Transformers library [Wolf et al., 2020] and the AdamW optimizer.
Revised: We develop our model in Pytorch [Paszke et al., 2019] and PytorchLightning [Falcon et al., 2019] using the Transformers library [Wolf et al., 2020] and the AdamW optimizer.

Original: We train our model on 1 epoch using a batch size of 4 and gradient accumulation of 4 and a learning rate of 1E-5.
Revised: We train our model on 1 epoch using a batch size of 4 and gradient accumulation of 4 and a learning rate of 1E-5.

Original: We conduct our experiments on 8 Tesla V100 32GB GPUS using distributed data parallelism.
Revised: We conduct our experiments on 8 Tesla V100 32GB GPUS using distributed data parallelism.

Original: The training time takes about 2.5 hour for the uniﬁed model and inference 25 minutes.
Revised: The training time takes about 2.5 hour for the uniﬁed model and inference 25 minutes.

Original: Our model includes 148M parameters.
Revised: Our model includes 148M parameters.

Original: We ﬁne-tune the threshold for the agglomerative clustering and the stopping criterion for the hierarchy on the validation set, in a range of { 0.4, 0.6 } for both, set to 0.
Revised: We ﬁne-tune the threshold for the agglomerative clustering and the stopping criterion for the hierarchy on the validation set, in a range of { 0.4, 0.6 } for both, set to 0.

Original: and 0.4 respectively.
Revised: and 0.4 respectively.

Original: We take the values that achieve the best path ratio metric on the validation set.
Revised: We take the values that achieve the best path ratio metric on the validation set

Revised: [Table]

Original: Appendix C. Coreference and Hierarchy Results
Revised: Appendix C. Coreference and Hierarchy Results

Original: Table 6 presents the results of all models according to the cluster-level hierarchy score (§2.2) in terms of recall, precision and F1.
Revised: Table 6 presents the results of all models according to the cluster-level hierarchy score (§2.2) in terms of recall, precision and F1.

Original: [Table] Table 7 presents the results of the inter-annotator agreement (IAA), all baseline models as well as the pipeline and uniﬁed, according to all common coreference metrics (MUC; Vilain et al., 1995, B 3 ; Bagga and Baldwin, 1998, CEAFe; Luo, 2005 and LEA; Moosavi and Strube, 2016).
Revised: Table 7 presents the results of the inter-annotator agreement (IAA), all baseline models as well as the pipeline and uniﬁed, according to all common coreference metrics (MUC, B 3 , CEAFe, LEA and CoNLL F1).

Original: We obtain coreference metrics using the python implementation of the standard coreference metrics [Moosavi and Strube, 2016]
Revised: We obtain coreference metrics using the python implementation of the standard coreference metrics [Moosavi and Strube, 2016].

Original: [Table]
Revised: [Table]

Original: [Table] https://github.com/ns-moosavi/coval/
Revised: , LEA and CoNLL F1. Since IAA is computed by considering one annotator as gold and the other as system, the IAA is reported only according to the symetric F1.