Original: KnowFi: Knowledge Extraction from Long Fictional Texts
Revised: KnowFi: Knowledge Extraction from Long Fictional Texts

Original: Anonymous authors

Original: Abstract
Revised: Abstract

Original: Knowledge base construction has recently been extended to ﬁctional domains like multivolume novels and TV/movie series, aiming to support explorative queries for fans and sub-culture studies by humanities researchers.
Revised: Knowledge base construction has recently been extended to ﬁctional domains like multivolume novels and TV/movie series, aiming to support explorative queries for fans and sub-culture studies by humanities researchers.

Original: This task involves the extraction of relations between entities.
Revised: This task involves the extraction of relations between entities.

Original: State-of-the-art methods are geared for short input texts and basic relations, but ﬁctional domains require tapping very long texts and need to cope with nonstandard relations where distant supervision becomes sparse.
Revised: State-of-the-art methods are geared for short input texts and basic relations, but ﬁctional domains require tapping very long texts and need to cope with nonstandard relations where distant supervision becomes sparse.

Original: This work addresses these challenges by a novel method, called KnowFi, that combines BERT-enhanced neural learning with judicious selection and aggregation of text passages.
Revised: This work addresses these challenges by a novel method, called KnowFi, that combines BERT-enhanced neural learning with judicious selection and aggregation of text passages.

Original: Experiments with several ﬁctional domains demonstrate the gains that KnowFi achieves over the best prior methods for neural relation extraction.
Revised: Experiments with several ﬁctional domains demonstrate the gains that KnowFi achieves over the best prior methods for neural relation extraction.

Original: 1. Introduction
Revised: 1. Introduction

Original: Motivation and Problem: Relation extraction (RE) from web contents is a key task for the automatic construction of knowledge bases (KB).
Revised: Motivation and Problem: Relation extraction (RE) from web contents is a key task for the automatic construction of knowledge bases (KB).

Original: It involves detecting a pair of entities in a text document and inferring if a certain relation (predicate) holds between them.
Revised: It involves detecting a pair of entities in a text document and inferring if a certain relation (predicate) holds between them.

Original: Extracted triples of the form (subject, predicate, object) are used for populating and growing the KB.
Revised: Extracted triples of the form (subject, predicate, object) are used for populating and growing the KB.

Original: Besides this major use case, RE also serves other applications like text annotation and summarization, semantic search, and more.
Revised: Besides this major use case, RE also serves other applications like text annotation and summarization, semantic search, and more.

Original: Work on KB construction has mostly focused on general-purpose encyclopedic knowledge, about prominent people, places, products etc.
Revised: Work on KB construction has mostly focused on general-purpose encyclopedic knowledge, about prominent people, places, products etc.

Original: and basic relations of wide interest such as birthplaces, spouses, writing of books, acting in movies etc.
Revised: and basic relations of wide interest such as birthplaces, spouses, writing of books, acting in movies etc.

Original: Vertical domains have received some attention, including health, food, and consumer products.
Revised: Vertical domains have received some attention, including health, food, and consumer products.

Original: Yet another case are KBs about ﬁctional works [Hertling and Paulheim, 2020, Labatut and Bost, 2019], such as Game of Thrones (GoT), the Marvel Comics (MC) universe, Greek Mythology or epic books such as War and Peace by Leo Tolstoy or the Cartel novels by Don Winslow.
Revised: Yet another case are KBs about ﬁctional works [Hertling and Paulheim, 2020, Labatut and Bost, 2019], such as Game of Thrones (GoT), the Marvel Comics (MC) universe, Greek Mythology or epic books such as War and Peace by Leo Tolstoy or the Cartel novels by Don Winslow.

Original: For
Revised: For

Original: KBs about ﬁctional domains, the focus is less on basic relations like birthplaces or spouses, but more on relations that capture traits of characters and key elements of the narration.
Revised: KBs about ﬁctional domains, the focus is less on basic relations like birthplaces or spouses, but more on relations that capture traits of characters and key elements of the narration.

Original: Relations of interest are allies, enemies, membership in clans, betrayed, killed etc.
Revised: Relations of interest are allies, enemies, membership in clans, betrayed, killed etc.

Original: Applications of ﬁction KBs foremost including supporting fans in entity-centric search.
Revised: Applications of ﬁction KBs foremost including supporting fans in entity-centric search.

Original: Some of the ﬁctional domains have huge fan communities, and search engines frequently receive queries such as “Who killed Catelyn Stark?”
Revised: Some of the ﬁctional domains have huge fan communities, and search engines frequently receive queries such as “Who killed Catelyn Stark?”

Original: (in GoT).
Revised: (in GoT).

Original: Entity summarization is a related task, for example, a user asking for the most salient traits of Ygritte (in GoT).
Revised: Entity summarization is a related task, for example, a user asking for the most salient traits of Ygritte (in GoT).

Original: Although ﬁction serves to entertain, some of the more complex domains reﬂect sub-cultural trends and the zeitgeist of certain epochs.
Revised: Although ﬁction serves to entertain, some of the more complex domains reﬂect sub-cultural trends and the zeitgeist of certain epochs.

Original: Analyzing their narrative structures and networks of entities is of interest to humanities scholars.
Revised: Analyzing their narrative structures and networks of entities is of interest to humanities scholars.

Original: For example, superhero comics originated in the 1940s and boomed in post-war years, reﬂecting that era’s zeitgeist (revived now).
Revised: For example, superhero comics originated in the 1940s and boomed in post-war years, reﬂecting that era’s zeitgeist (revived now).

Original: War and Peace has the backdrop of the Napoleonic wars in Russia, and the Cartel trilogy blends facts and ﬁction about drug traﬃcking.
Revised: War and Peace has the backdrop of the Napoleonic wars in Russia, and the Cartel trilogy blends facts and ﬁction about drug traﬃcking.

Original: KBs enable deeper analyses of such complex texts for historians, social scientists, media psychologists and cultural-studies scholars.
Revised: KBs enable deeper analyses of such complex texts for historians, social scientists, media psychologists and cultural-studies scholars.

Original: State of the Art and its Limitations: RE with pre-speciﬁed relations for canonicalized entities is based on distant supervision via pre-compiled seed triples [Mintz et al., 2009,
Revised: State of the Art and its Limitations: RE with pre-speciﬁed relations for canonicalized entities is based on distant supervision via pre-compiled seed triples [Mintz et al., 2009,

Original: Suchanek et al., 2009].
Revised: Suchanek et al., 2009].

Original: Typically, these training seeds come from initial KBs, which in turn draw on Wikipedia infoboxes.
Revised: Typically, these training seeds come from initial KBs, which in turn draw on Wikipedia infoboxes.

Original: The best RE methods are based on this paradigm of distant supervision, leveraging it for neural learning (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]).
Revised: The best RE methods are based on this paradigm of distant supervision, leveraging it for neural learning (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]).

Original: They work well for basic relations, as there is no shortage of training samples (e.g., for birthplace or spouse).
Revised: They work well for basic relations, as there is no shortage of training samples (e.g., for birthplace or spouse).

Original: One of their key limitations is the bounded size of input text passages, typically a few hundred tokens only.
Revised: One of their key limitations is the bounded size of input text passages, typically a few hundred tokens only.

Original: This is not a bottleneck for basic relations where single sentences (or short paragraphs) with all three SPO components are frequent enough (e.g., in the full text of
Revised: This is not a bottleneck for basic relations where single sentences (or short paragraphs) with all three SPO components are frequent enough (e.g., in the full text of

Original: Wikipedia articles).
Revised: Wikipedia articles).

Original: However, for RE with non-standard relations over long ﬁctional texts such as entire books, these limitations are major bottlenecks, if not showstoppers.
Revised: However, for RE with non-standard relations over long ﬁctional texts such as entire books, these limitations are major bottlenecks, if not showstoppers.

Original: This paper addresses the resulting challenges (also included among the open challenges in the overview by [Han et al., 2020]):
Revised: This paper addresses the resulting challenges (also included among the open challenges in the overview by [Han et al., 2020]):

Original: • How to go about distant supervision for RE targeting non-standard relations that have only few seed triples?
Revised: • How to go about distant supervision for RE targeting non-standard relations that have only few seed triples?

Original: • How to cope with very long input texts, such as entire books, where relevant cues for RE are spread across passages?
Revised: • How to cope with very long input texts, such as entire books, where relevant cues for RE are spread across passages?

Original: Approach and Contributions: This paper presents a complete methodology and system for relation extraction from long ﬁctional texts, called KnowFi (Knowledge extraction from
Revised: Approach and Contributions: This paper presents a complete methodology and system for relation extraction from long ﬁctional texts, called KnowFi (Knowledge extraction from

Original: Fictional texts).
Revised: Fictional texts).

Original: Our method leverages semi-structured content in wikis of fan communities on fandom.com (aka wikia.com ).
Revised: Our method leverages semi-structured content in wikis of fan communities on fandom.com (aka wikia.com ).

Original: We extract an initial KB of background knowledge forpopular domains (TV series, movies, games).
Revised: We extract an initial KB of background knowledge forpopular domains (TV series, movies, games).

Original: This serves to identify interesting relations and to collect distant supervision samples.
Revised: This serves to identify interesting relations and to collect distant supervision samples.

Original: Yet for many relations this results in very few seeds.
Revised: Yet for many relations this results in very few seeds.

Original: To overcome this sparseness challenge and to generalize the training across the wide variety of relations, we devise a similarity-based ranking technique for matching seeds in text passages.
Revised: To overcome this sparseness challenge and to generalize the training across the wide variety of relations, we devise a similarity-based ranking technique for matching seeds in text passages.

Original: Given a long input text, KnowFi judiciously selects a number of context passages containing seed pairs of entities.
Revised: Given a long input text, KnowFi judiciously selects a number of context passages containing seed pairs of entities.

Original: To infer if a certain relation holds between two entities, KnowFi’s neural network is trained jointly for all relations as a multi-label classiﬁer.
Revised: To infer if a certain relation holds between two entities, KnowFi’s neural network is trained jointly for all relations as a multi-label classiﬁer.

Original: Extensive experiments with long books on ﬁve diﬀerent ﬁctional domains show that
Revised: Extensive experiments with long books on ﬁve diﬀerent ﬁctional domains show that

Original: KnowFi clearly outperforms state-of-the-art RE methods.
Revised: KnowFi clearly outperforms state-of-the-art RE methods.

Original: Even on conventional short-text benchmarks with standard relations, KnowFi is competitive with the best baselines.
Revised: Even on conventional short-text benchmarks with standard relations, KnowFi is competitive with the best baselines.

Original: As an extrinsic use case, we demonstrate the value of KnowFi’s
Revised: As an extrinsic use case, we demonstrate the value of KnowFi’s

Original: KB for the task of entity summarization.
Revised: KB for the task of entity summarization.

Original: The paper’s novel contributions are:
Revised: The paper’s novel contributions are:

Original: • a system architecture for the new problem of relation extraction from long ﬁctional texts, like entire novels and text contents by fan communities (Section 3); • a method to overcome the challenge of sparse samples for distant supervision for nonstandard relations (Section 4).
Revised: • a system architecture for the new problem of relation extraction from long ﬁctional texts, like entire novels and text contents by fan communities (Section 3); • a method to overcome the challenge of sparse samples for distant supervision for nonstandard relations (Section 4).

Original: • a method to overcome the challenge of limited input size for neural learners, by judiciously selecting relevant contexts and aggregating results (Section 5); • a comprehensive experimental evaluation with a novel benchmark for relation extraction from very long documents (Section 6), with code and data release upon publication.
Revised: • a method to overcome the challenge of limited input size for neural learners, by judiciously selecting relevant contexts and aggregating results (Section 5); • a comprehensive experimental evaluation with a novel benchmark for relation extraction from very long documents (Section 6), with code and data release upon publication

Original: 2. Related Work
Revised: 2. Related Work

Original: Relation Extraction (RE): Early work on RE from text sources has used rules and patterns, (e.g., [Agichtein and Gravano, 2000, Craven et al., 1998, Etzioni et al., 2004, Reiss et al., 2008]), with pattern learning based on the principle of relation-pattern duality [Brin, 1998].
Revised: Relation Extraction (RE): Early work on RE from text sources has used rules and patterns, (e.g., [Agichtein and Gravano, 2000, Craven et al., 1998, Etzioni et al., 2004, Reiss et al., 2008]), with pattern learning based on the principle of relation-pattern duality [Brin, 1998].

Original: Open IE [Banko et al., 2007, Mausam, 2016, Stanovsky et al., 2018] uses linguistic cues to jointly infer patterns and triples, but lacks proper normalization of SPO arguments.
Revised: Open IE [Banko et al., 2007, Mausam, 2016, Stanovsky et al., 2018] uses linguistic cues to jointly infer patterns and triples, but lacks proper normalization of SPO arguments.

Original: RE with pre-speciﬁed relations, on the other hand, is usually based on distant supervision via pre-compiled seed triples [Mintz et al., 2009, Suchanek et al., 2009].
Revised: RE with pre-speciﬁed relations, on the other hand, is usually based on distant supervision via pre-compiled seed triples [Mintz et al., 2009, Suchanek et al., 2009].

Original: A variety of methods have been developed on this paradigm, from probabilistic graphical models (e.g., [Pujara et al., 2015, Sa et al., 2017]) to deep neural networks (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]).
Revised: A variety of methods have been developed on this paradigm, from probabilistic graphical models (e.g., [Pujara et al., 2015, Sa et al., 2017]) to deep neural networks (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]).

Original: Distantly supervised neural learning has become the method of choice, with diﬀerent granularities.
Revised: Distantly supervised neural learning has become the method of choice, with diﬀerent granularities.

Original: Sentence-level RE: Most neural methods operate on a per-sentence level.
Revised: Sentence-level RE: Most neural methods operate on a per-sentence level.

Original: Distantsupervision samples of SPO triples serve to identify sentences that contain an entity pair (S and O) which stand in a certain relation.
Revised: Distantsupervision samples of SPO triples serve to identify sentences that contain an entity pair (S and O) which stand in a certain relation.

Original: The sentence is then treated as a positive training sample for the neural learner.
Revised: The sentence is then treated as a positive training sample for the neural learner.

Original: At test-time, the trained model can tag entity mentions and predict if the sentence expresses a given relation or not.
Revised: At test-time, the trained model can tag entity mentions and predict if the sentence expresses a given relation or not.

Original: This basic architecture has been advanced with bi-LSTMs, attention mechanisms and other techniques (e.g., [Zhang et al., 2017, Cui et al., 2018, Trisedya et al., 2019]).
Revised: This basic architecture has been advanced with bi-LSTMs, attention mechanisms and other techniques (e.g., [Zhang et al., 2017, Cui et al., 2018, Trisedya et al., 2019]).

Original: A widely used benchmark for sentence-level
Revised: A widely used benchmark for sentence-level

Original: RE is TacRed [Zhang et al., 2017].
Revised: RE is TacRed [Zhang et al., 2017].

Original: With recent advances on pre-trained language models like BERT [Devlin et al., 2019a] (or ElMo, GPT-3, T-5 and the like), the currently best RE methods leverage this asset for representation learning [Shi and Lin, 2019, Soares et al., 2019, Wadden et al., 2019, Yu et al., 2020].
Revised: With recent advances on pre-trained language models like BERT [Devlin et al., 2019a] (or ElMo, GPT-3, T-5 and the like), the currently best RE methods leverage this asset for representation learning [Shi and Lin, 2019, Soares et al., 2019, Wadden et al., 2019, Yu et al., 2020].

Original: Document-level RE: To expand the scope of inputs, Wang et al.
Revised: Document-level RE: To expand the scope of inputs, Wang et al.

Original: [2019] proposed RE from documents, introducing the DocRed benchmark.
Revised: [2019] proposed RE from documents, introducing the DocRed benchmark.

Original: However, the notion of documents is still very limited in size, given the restrictions in neural network inputs, typically around 10 sentences (e.g., excerpts from Wikipedia articles).
Revised: However, the notion of documents is still very limited in size, given the restrictions in neural network inputs, typically around 10 sentences (e.g., excerpts from Wikipedia articles).

Original: Wang et al.
Revised: Wang et al.

Original: [2020] is a state-of-theart method for this document-level RE task, utilizing BERT and graph convolutions for representation learning.
Revised: [2020] is a state-of-theart method for this document-level RE task, utilizing BERT and graph convolutions for representation learning.

Original: Zhou et al.
Revised: Zhou et al.

Original: [2021] further enhanced this approach.
Revised: [2021] further enhanced this approach.

Original: None of these methods can handle input documents that are larger than a few tens of sentences.
Revised: None of these methods can handle input documents that are larger than a few tens of sentences.

Original: KnowFi is the ﬁrst method that is geared for book-length input.
Revised: KnowFi is the ﬁrst method that is geared for book-length input.

Original: Fiction Knowledge Bases: Unterstanding characters in literary texts and constructing networks of their relationships and interactions has become a small topic in NLP (e.g., [Chaturvedi et al., 2016, Labatut and Bost, 2019, Srivastava et al., 2016]).
Revised: Fiction Knowledge Bases: Unterstanding characters in literary texts and constructing networks of their relationships and interactions has become a small topic in NLP (e.g., [Chaturvedi et al., 2016, Labatut and Bost, 2019, Srivastava et al., 2016]).

Original: The work of 
Revised: The work of [Chu et al., 2019, 2020] has advanced this theme for entity typing and type taxonomies for WikiaInfoboxes [Figure] Relations

Original: [Figure]
Revised: [Figure]

Original: [Chu et al., 2019, 2020] has advanced this theme for entity typing and type taxonomies for ﬁctional domains.
Revised: BERT MultiLabelClassifier ContextEmbed‘s ﬁctional domains.

Original: However, this work does not address learning relations between entities for KB population.
Revised: However, this work does not address learning relations between entities for KB population.

Original: The DBkWik project [Hertling and Paulheim, 2020] has leveraged structured infoboxes of fan communities at wikia (now renamed to fandom.com ), to construct a large KB of ﬁctional characters and their salient properties.
Revised: The DBkWik project [Hertling and Paulheim, 2020] has leveraged structured infoboxes of fan communities at wikia (now renamed to fandom.com ), to construct a large KB of ﬁctional characters and their salient properties.

Original: However, this is strictly limited to relations and respective instances that are present in infoboxes.
Revised: However, this is strictly limited to relations and respective instances that are present in infoboxes.

Original: Our work leverages wikia infoboxes for distant supervision, but our method can extract more knowledge from a variety of text sources, including storylines and synopses by fans and, most demandingly, the full text of entire books.
Revised: Our work leverages wikia infoboxes for distant supervision, but our method can extract more knowledge from a variety of text sources, including storylines and synopses by fans and, most demandingly, the full text of entire books.

Original: 3. System Overview
Revised: 3. System Overview

Original: The architecture of the KnowFi system is illustrated in Figure 1.
Revised: The architecture of the KnowFi system is illustrated in Figure 1.

Original: There are two major components:
Revised: There are two major components:

Original: • Distant supervision involves pre-processing infoboxes from Wikia-hosted fan communities, to obtain seed pairs of entities.
Revised: • Distant supervision involves pre-processing infoboxes from Wikia-hosted fan communities, to obtain seed pairs of entities.

Original: These are used to retrieve relevant passages from the underlying text corpora: either synopses of storylines in Wikia or full-ﬂedged content of original books.
Revised: These are used to retrieve relevant passages from the underlying text corpora: either synopses of storylines in Wikia or full-ﬂedged content of original books.

Original: As the number of passages per entity pair can be very large in books, we devise a judicious ranking of passages and feed only the top-k passages into the next stage of training the neural network.
Revised: As the number of passages per entity pair can be very large in books, we devise a judicious ranking of passages and feed only the top-k passages into the next stage of training the neural network.

Original: Details are in Section 4.
Revised: Details are in Section 4.

Original: • Multi-context neural learning feeds the top-k passages, with entity markup, jointly into a BERT-based encoder [Devlin et al., 2019b].
Revised: • Multi-context neural learning feeds the top-k passages, with entity markup, jointly into a BERT-based encoder [Devlin et al., 2019b].

Original: On top of this representation learning, a multi-label classiﬁer predicts the relations that hold for the input entity pair.
Revised: On top of this representation learning, a multi-label classiﬁer predicts the relations that hold for the input entity pair.

Original: Details are in Section 5.
Revised: Details are in Section 5.

Original: Note that a passage can vary from a single sentence to a long paragraph.
Revised: Note that a passage can vary from a single sentence to a long paragraph.

Original: The two seed entities would ideally occur in the same sentence, but there are many cases where they are one or two sentences apart.
Revised: The two seed entities would ideally occur in the same sentence, but there are many cases where they are one or two sentences apart.

Original: Figure 2 shows example texts from a GoT synopses in Wikia and from one of the original books.
Revised: Figure 2 shows example texts from a GoT synopses in Wikia and from one of the original books.

Original: The pre-processing of Wikia infoboxes resulted in 2.37M SPO triples for ca.
Revised: The pre-processing of Wikia infoboxes resulted in 2.37M SPO triples for ca.

Original: 8,000 diﬀerent relation names between a total of 461.4k entities, obtained from 142 domains (movie/TV series, games etc.).
Revised: 8,000 diﬀerent relation names between a total of 461.4k entities, obtained from 142 domains (movie/TV series, games etc.).

Original: This forms our background knowledge for distant supervision.
Revised: This forms our background knowledge for distant supervision.

Original: For obtaining matching passages, we focused on the 64 most frequent relations, including friend, 
Revised: For obtaining matching passages, we focused on the 64 most frequent relations, including friend, ally, enemy and family relationships.

Revised: Note that this stage is not domain-speciﬁc.

Revised: Later we apply the learned model to speciﬁc domains such as GoT or Marvel Comics.

Original: Excerpt from Game of Thrones synopses at Wikia:
Revised: Excerpt from Game of Thrones synopses at Wikia:

Original: Eighteen years before the War of the Five Kings, Rhaegar Targaryen allegedly abducted Lyanna Stark in a scandal that led to the outbreak of Robert‘s Rebellion.
Revised: Eighteen years before the War of the Five Kings, Rhaegar Targaryen allegedly abducted Lyanna Stark in a scandal that led to the outbreak of Robert‘s Rebellion.

Original: Rhaegar eventually returned to fight n the war, but not before leaving Lysanna behind at the Tower of Joy, guarded by Lord Commander Gerold Hightower and
Revised: Rhaegar eventually returned to fight n the war, but not before leaving Lysanna behind at the Tower of Joy, guarded by Lord Commander Gerold Hightower and

Original: Ser Arthur Dayne of the Kingsguard.
Revised: Ser Arthur Dayne of the Kingsguard.

Original: Eddard Stark rode to war along her betrothed, Robert Baratheon, to rescue his sister and avenge the deaths of their father and brother at the orders of Aerys II, the Mad King.
Revised: Eddard Stark rode to war along her betrothed, Robert Baratheon, to rescue his sister and avenge the deaths of their father and brother at the orders of Aerys II, the Mad King.

Original: Excerpt from Harry Potter book:
Revised: Excerpt from Harry Potter book:

Original: Harry had been a year old the night that Voldemort - the most powerful Dark wizard for a century, a wizard who had been gaining power steadily for eleven years, arrived at his house and killed his father and mother .
Revised: Harry had been a year old the night that Voldemort - the most powerful Dark wizard for a century, a wizard who had been gaining power steadily for eleven years, arrived at his house and killed his father and mother .

Original: Voldemort had then turned his wand on Harry ; he had performed the curse that had disposed of many fullgrown witches and wizards in his steady rise to power and, incredibly, it had not worked.
Revised: Voldemort had then turned his wand on Harry ; he had performed the curse that had disposed of many fullgrown witches and wizards in his steady rise to power and, incredibly, it had not worked.

Original: Instead of killing the small boy, the curse had rebounded upon Voldemort .
Revised: Instead of killing the small boy, the curse had rebounded upon Voldemort .

Original: Harry had survived with nothing but a lightning-shaped cut on his forehead, and Voldemort had been reduced to something barely alive .
Revised: Harry had survived with nothing but a lightning-shaped cut on his forehead, and Voldemort had been reduced to something barely alive .

Original: ally, enemy and family relationships.

Original: Note that this stage is not domain-speciﬁc.

Original: Later we apply the learned model to speciﬁc domains such as GoT or Marvel Comics.

Original: 4. Distant Supervision with Passage Ranking
Revised: 4. Distant Supervision with Passage Ranking

Original: The KnowFi approach to distant supervision diﬀers from prior works in two ways:
Revised: The KnowFi approach to distant supervision diﬀers from prior works in two ways:

Original: • Passage ranking: Identifying the best passages that contain seed triples, by judicious ranking, and using only the top-k passages as positive training samples.
Revised: • Passage ranking: Identifying the best passages that contain seed triples, by judicious ranking, and using only the top-k passages as positive training samples.

Original: • Passages with gaps: Including passages where the entities of a seed triple merely occur in separate sentences with other sentences in between.
Revised: • Passages with gaps: Including passages where the entities of a seed triple merely occur in separate sentences with other sentences in between.

Original: Passage ranking: Seed pairs of entities are matched by many sentences or passages in the input corpora.
Revised: Passage ranking: Seed pairs of entities are matched by many sentences or passages in the input corpora.

Original: For example, the pair (Herminone, Harry appears in 1539 sentences in the the seven volumes of the Harry Potter series together.
Revised: For example, the pair (Herminone, Harry) appears in 1539 sentences in the the seven volumes of the Harry Potter series together.

Original: Many of these contain cues that they stand in the friends relation, but there are also many sentences where the co-occurrence is merely accidental.
Revised: Many of these contain cues that they stand in the friends relation, but there are also many sentences where the co-occurrence is merely accidental.

Original: This is a standard dilemma in distant supervision for multi-instance learning [Riedel et al., 2010, Li et al., 2020].
Revised: This is a standard dilemma in distant supervision for multi-instance learning [Riedel et al., 2010, Li et al., 2020].

Original: Our approach is to identify the best passages among the numerous matches, by judicious ranking on a per-relation basis.
Revised: Our approach is to identify the best passages among the numerous matches, by judicious ranking on a per-relation basis.

Original: For each relation, we build a prototype representation by selecting sentences that contain lexical matches of all three SPO arguments, where the predicate is matched by its label in the background knowledge or a short list (average length of three) of synonyms and close hyponyms or hypernyms (e.g., “allegiance” or “loyalty” matching ally ), manually collected from WordNet.
Revised: For each relation, we build a prototype representation by selecting sentences that contain lexical matches of all three SPO arguments, where the predicate is matched by its label in the background knowledge or a short list (average length of three) of synonyms and close hyponyms or hypernyms (e.g., “allegiance” or “loyalty” matching ally ), manually collected from WordNet.

Original: Newly seen passages for entity pairs can then be scored against the per-relation prototypes by casting both into tf-idf-weighted bag-of-word models (or alternatively, word2vec-style embeddings) and computing their cosine distance.
Revised: Newly seen passages for entity pairs can then be scored against the per-relation prototypes by casting both into tf-idf-weighted bag-of-word models (or alternatively, word2vec-style embeddings) and computing their cosine distance.

Original: This way, we rank candidate passage for each seed pair and target relation.
Revised: This way, we rank candidate passage for each seed pair and target relation.

Original: Passages with gaps: Unlike encyclopedic articles, long texts on ﬁctional domains have a narrative style where single sentences are unlikely to give the full information in the most compact way.
Revised: Passages with gaps: Unlike encyclopedic articles, long texts on ﬁctional domains have a narrative style where single sentences are unlikely to give the full information in the most compact way.

Original: Therefore, we consider multi-sentence contexts where entity mentions[Figure] across diﬀerent sentences.
Revised: Therefore, we consider multi-sentence contexts where entity mentions across diﬀerent sentences.

Original: In addition to simple paragraphs, we consider passages with gaps where we include sentences that are not necessarily contiguous but leave out uninformative sentences.
Revised: In addition to simple paragraphs, we consider passages with gaps where we include sentences that are not necessarily contiguous but leave out uninformative sentences.

Original: This way, we maximize the value of limited-size text spans fed into the neural  learner.
Revised: This way, we maximize the value of limited-size text spans fed into the neural [Figure] learner.

Original: This is in contrast to earlier techniques that consume whole paragraphs and rely on attention mechanism for giving higher weight to informative parts.
Revised: This is in contrast to earlier techniques that consume whole paragraphs and rely on attention mechanism for giving higher weight to informative parts.

Original: KnowFi has two conﬁguration parameters: the maximum number of sentences allowed between sentences that contain seed entities, and the number of sentences directly preceding or following the occurrence of a seed entity.
Revised: KnowFi has two conﬁguration parameters: the maximum number of sentences allowed between sentences that contain seed entities, and the number of sentences directly preceding or following the occurrence of a seed entity.

Original: In our experiments, we include text where the two entities appear at most 2 sentences apart and 1 preceding and 1 following sentence for each of the entity mentions, up to 512 tokens which is the current limit of BERT-based networks.
Revised: In our experiments, we include text where the two entities appear at most 2 sentences apart and 1 preceding and 1 following sentence for each of the entity mentions, up to 512 tokens which is the current limit of BERT-based networks.

Original: Negative training samples:
Revised: Negative training samples:

Original: In addition to the positive training samples by the above procedure, we generate negative samples by the following random process.
Revised: In addition to the positive training samples by the above procedure, we generate negative samples by the following random process.

Original: For each relation r , we pick random entities e 1 and e 2 for each of the S and O roles such that there are other entities x and y for which the background knowledge asserts (e1, r, x) and (y, r, e2) with x (cid:54) =
Revised: For each relation r , we pick random entities e 1 and e 2 for each of the S and O roles such that there are other entities x and y for which the background knowledge asserts (e1, r, x) and (y, r, e2) with x (cid:54) =

Original: e 2 and y (cid:54) =
Revised: e 2 and y (cid:54) =

Original: e 1.
Revised: e 1.

Original: This improves on the standard technique of simply choosing any pair e 1 , e 2 for which ( e 1 , r, e 2) does not hold, by selecting more diﬃcult cases and thus strengthening the learner.
Revised: This improves on the standard technique of simply choosing any pair e 1 , e 2 for which ( e 1 , r, e 2) does not hold, by selecting more diﬃcult cases and thus strengthening the learner.

Original: For example, both Herminone and Malfoy have some friends, but they are not friends of each other.
Revised: For example, both Herminone and Malfoy have some friends, but they are not friends of each other.

Original: The training of KnowFi uses a 1:1 ratio of positive to negative samples.
Revised: The training of KnowFi uses a 1:1 ratio of positive to negative samples.

Original: 5. Multi-Context Neural Extraction
Revised: 5. Multi-Context Neural Extraction

Original: KnowFi is trained with and applicable to multiple passages as input to an end-to-end
Revised: KnowFi is trained with and applicable to multiple passages as input to an end-to-end

Original: Transformer-based network with full backpropagation of cross-entropy loss.
Revised: Transformer-based network with full backpropagation of cross-entropy loss.

Original: Our neural architecture has two speciﬁc components: a per-passage layer to learn BERT-based representations for each passage, and an aggregation layer that combines the signals from all input passages.
Revised: Our neural architecture has two speciﬁc components: a per-passage layer to learn BERT-based representations for each passage, and an aggregation layer that combines the signals from all input passages.

Original: In the experiments in this paper, the aggregation layer is conﬁgured to concatenate the representations of all passages, but other options are feasible, too.
Revised: In the experiments in this paper, the aggregation layer is conﬁgured to concatenate the representations of all passages, but other options are feasible, too.

Original: Each input passage is encoded with markup of entity mentions.
Revised: Each input passage is encoded with markup of entity mentions.

Original: In addition, we determine semantic types for the entities, using the SpaCy tool ( https://spacy.io/ ) that provides one type for each mention, chosen from a set of 18 coarse-grained types (person,  nationality/religion, event, etc.).
Revised: In addition, we determine semantic types for the entities, using the SpaCy tool ( https://spacy.io/ ) that provides one type for each mention, chosen from a set of 18 coarse-grained types (person, [Table] [Table] nationality/religion, event, etc.).

Original: The type of each entity mention in a passage is appended to the input vector.
Revised: The type of each entity mention in a passage is appended to the input vector.

Original: Figure 3 illustrates the neural network for multi-context RE.
Revised: Figure 3 illustrates the neural network for multi-context RE.

Original: 6. LoFiDo Benchmark
Revised: 6. LoFiDo Benchmark

Original: To evaluate RE from long documents, we introduce the LoFiDo corpus (Long Fiction
Revised: To evaluate RE from long documents, we introduce the LoFiDo corpus (Long Fiction

Original: Documents).
Revised: Documents).

Original: We compile SPO triples from infoboxes of 142 Wikia fan communities.
Revised: We compile SPO triples from infoboxes of 142 Wikia fan communities.

Original: After cleaning extractions and clustering synonyms, we obtain a total of 64 relations such as enemy, friend, ally, religion, weapon, ruler-of , etc.
Revised: After cleaning extractions and clustering synonyms, we obtain a total of 64 relations such as enemy, friend, ally, religion, weapon, ruler-of , etc.

Original: For evaluating KnowFi and various baselines, we focus on 5 especially rich and diverse domains (i) Lord of the Rings (a series of three epic novels by J.R.R Tolkien), (ii) A Song of Ice and Fire (a series of ﬁve fantasy novels by George R.R. Martin, well-known for the
Revised: For evaluating KnowFi and various baselines, we focus on 5 especially rich and diverse domains (i) Lord of the Rings (a series of three epic novels by J.R.R Tolkien), (ii) A Song of Ice and Fire (a series of ﬁve fantasy novels by George R.R. Martin, well-known for the

Original: Game of Thrones TV series based on it), (iii) Harry Potter (a series of seven books, written by J.K Rowling), (iv) Dune (a science-ﬁction novel by Frank Herbert), and (v) War and
Revised: Game of Thrones TV series based on it), (iii) Harry Potter (a series of seven books, written by J.K Rowling), (iv) Dune (a science-ﬁction novel by Frank Herbert), and (v) War and

Original: Peace (a classic novel by Leo Tolstoy).
Revised: Peace (a classic novel by Leo Tolstoy).

Original: For the ﬁrst four, Wikia infoboxes provide ground truth; for War and Peace, we manually crafted a small ground-truth KB.
Revised: For the ﬁrst four, Wikia infoboxes provide ground truth; for War and Peace, we manually crafted a small ground-truth KB.

Original: 20% of the triples from each of these universes are withheld for testing.
Revised: 20% of the triples from each of these universes are withheld for testing.

Original: For the ﬁrst four domains, we consider both original novels as well as narrative synopses from Wikia as input sources.
Revised: For the ﬁrst four domains, we consider both original novels as well as narrative synopses from Wikia as input sources.

Original: War and Peace is not covered by Wikia.
Revised: War and Peace is not covered by Wikia.

Revised: LoFiDo Statistics.

Revised: Our LoFiDo corpus contains 81,025 instances for training and 20, instances for validation.

Revised: For testing, we use ﬁve speciﬁc universes, which take input from both books and Wikia texts.

Original:  Further details on this dataset are in Appendix A, B and C. We will publicly release data and code upon publication.
Revised: The total number of instances in the test data from Wikia texts is 14,610, while in the case of books, it is 64,120. Ground-truth data for ﬁve test universes are provided for evaluation. Table 1 shows statistics on the training and validation data, while Table 2 shows statistics on the ground-truth of ﬁve domains in the test data. Further details on this dataset are in Appendix A and B. Code and data of KnowFi are available at https://www.mpi-inf.mpg.de/yago-naga/knowfi .

Original: 7. Experiments 7.1 Setup
Revised: 7. Experiments 7.1 Setup

Original: Baselines.
Revised: Baselines.

Original: We compare KnowFi to three state-of-the-art baselines on RE:
Revised: We compare KnowFi to three state-of-the-art baselines on RE:

Original: • BERT-Type [Shi and Lin, 2019] which uses BERT-based encodings augmented with entity type information, also based on SpaCy output in our experiments for fair comparison.
Revised: • BERT-Type [Shi and Lin, 2019] which uses BERT-based encodings augmented with entity type information, also based on SpaCy output in our experiments for fair comparison.

Original: • BERT-EM [Soares et al., 2019] which include entity markers in input sequences; • GLRE [ Wang et al., 2020] which additionally computes global entity representations and uses them to augment the text sequence encodings.
Revised: • BERT-EM [Soares et al., 2019] which include entity markers in input sequences; • GLRE [Wang et al., 2020] which additionally computes global entity representations and uses them to augment the text sequence encodings.

Original: The ﬁrst two baselines run on a per-sentence basis, whereas GLRE is a state-of-the art method for extractions from short documents, which we train on paragraph-level inputs.
Revised: The ﬁrst two baselines run on a per-sentence basis, whereas GLRE is a state-of-the art method for extractions from short documents, which we train on paragraph-level inputs.

Original: The inputs for these models (i.e. sentences or paragraphs) are randomly selected.
Revised: The inputs for these models (i.e. sentences or paragraphs) are randomly selected.

Original: KnowFi Parameters.
Revised: KnowFi Parameters.

Original: For passage selection, we rely on TF-IDF-based bag-of-words similarity between the passages and relation contexts, where each relation context contains the top-100 tokens, based on their tﬁdf scores.
Revised: For passage selection, we rely on TF-IDF-based bag-of-words similarity between the passages and relation contexts, where each relation context contains the top-100 tokens, based on their tﬁdf scores.

Original: For selecting passages as multi-context input, we compute the cosine between tf-idf-based vectors of each passage against the relation-speciﬁc prototype vector; we select all passages with cosine above 0.5 as positive training samples.
Revised: For selecting passages as multi-context input, we compute the cosine between tf-idf-based vectors of each passage against the relation-speciﬁc prototype vector; we select all passages with cosine above 0.5 as positive training samples.

Original: For the neural network, we use BERT LARGE ( https://huggingface.co/transformers/model_ doc/bert.html ) with 24 layers, 1024 hidden size and 16 heads.
Revised: For the neural network, we use BERT LARGE ( https://huggingface.co/transformers/model_ doc/bert.html ) with 24 layers, 1024 hidden size and 16 heads.

Original: The learning rate is 5 e −with Adam, the batch size is 8, and the number of training epochs is 10.
Revised: The learning rate is 5 e −with Adam, the batch size is 8, and the number of training epochs is 10.

Original: [Table]

Original: [Table]

Original: Evaluation Metrics.
Revised: Evaluation Metrics.

Original: The evaluation uses standard metrics like precision, recall and
Revised: The evaluation uses standard metrics like precision, recall and

Original: F1, averaged over all extracted triples.
Revised: F1, averaged over all extracted triples.

Original: We report micro-averaged numbers for all relations together [Table] and drill down on selected relations of interest.
Revised: We report micro-averaged numbers for all relations together, and drill down on selected relations of interest.

Original: In addition, we report numbers for
Revised: In addition, we report numbers for

Original: HITS@k and MRR.
Revised: HITS@k and MRR.

Original: As ground-truth, we perform two diﬀerent modes of evaluation:
Revised: As ground-truth, we perform two diﬀerent modes of evaluation:

Original: • Automated evaluation is based on ground-truth from Wikia infoboxes.
Revised: • Automated evaluation is based on ground-truth from Wikia infoboxes.

Original: This is demanding on precision, but penalizes recall because of its limited coverage.
Revised: This is demanding on precision, but penalizes recall because of its limited coverage.

Original: • Manual evaluation is based on obtaining assessments of extracted triples via crowdsourcing.
Revised: • Manual evaluation is based on obtaining assessments of extracted triples via crowdsourcing

Original: This way, we include correct triples that are not in Wikia infoboxes, and thus achieve higher recall.
Revised: This way, we include correct triples that are not in Wikia infoboxes, and thus achieve higher recall.

Original: 7.2 ResultsAutomated Evaluation.
Revised: 7.2 Results

Revised: Automated Evaluation.

Original: Table 1 shows average precision, recall and F1 score.
Revised: Table 3 shows average precision, recall and F1 score.

Original: We can see that sentence-level baselines achieve comparatively high coverage, due to considering every sentence.
Revised: We can see that sentence-level baselines achieve comparatively high coverage, due to considering every sentence.

Original: Yet their precision is extremely low.
Revised: Yet their precision is extremely low.

Original: GLRE and KnowFi achieve much higher precision, though GLRE fails to achieve competitive recall, presumably because its training on all paragraphs lowers its predictive power.
Revised: GLRE and KnowFi achieve much higher precision, though GLRE fails to achieve competitive recall, presumably because its training on all paragraphs lowers its predictive power.

Original: As an illustration, GLRE produces only 173 assertions from all Harry Potter books, while KnowFi produces 600.
Revised: As an illustration, GLRE produces only 173 assertions from all Harry Potter books, while KnowFi produces 600.

Original: We also observe that for all methods, extraction from books is considerably harder than from the more concise synopses in Wikia.
Revised: We also observe that for all methods, extraction from books is considerably harder than from the more concise synopses in Wikia.

Original: In addition to the P/R/F1 scores, in Table 2 we also take an entity-centric view and evaluate how well correct extractions rank.
Revised: In addition to the P/R/F1 scores, in Table 4 we also take an entity-centric view and evaluate how well correct extractions rank.

Original: The HITs@k metric reports how often a correct result appears among the top extractions per entity-relation pair (e.g., among topextracted enemies of Harry Potter), while MRR reports the mean reciprocal rank of the ﬁrst extraction.
Revised: The HITs@k metric reports how often a correct result appears among the top extractions per entity-relation pair (e.g., among top- [Table] [Table] extracted enemies of Harry Potter), while MRR reports the mean reciprocal rank of the ﬁrst extraction.

Original: We can observe that KnowFi outperforms all baselines on both metrics.
Revised: We can observe that KnowFi outperforms all baselines on both metrics.

Original: Manual Evaluation.
Revised: Manual Evaluation

Original: The low absolute scores in the above evaluation largely stem from incomplete automated ground truth.
Revised: The low absolute scores in the above evaluation largely stem from incomplete automated ground truth.

Original: We therefore conducted an additional manual evaluation.
Revised: We therefore conducted an additional manual evaluation.

Original: For each domain, we select top 100 extractions from the results and used crowdsourcing to manually label their correctness.
Revised: For each domain, we select top 100 extractions from the results and used crowdsourcing to manually label their correctness.

Original: The annotators were Amazon master workers with all [Table]
Revised: The annotators were Amazon master workers with all time approval rate > 90%

Original: [Table]
Revised: [Table]

Original: time approval rate > 90%, and additional test questions were used to ﬁlter responses.
Revised: and additional test questions were used to ﬁlter responses.

Original: We observed high inter-annotator agreement, on average of 0.81.
Revised: We observed high inter-annotator agreement, on average of 0.81.

Original: Table 3 shows results of our manual evaluation on four domains (Dune was left out due to complexity).
Revised: Table 5 shows results of our manual evaluation on four domains (Dune was left out due to complexity).

Original: As one can see, KnowFi outperforms the baselines on most input texts [Table] and achieves a remarkable precision on both books and wikia texts (average of 0.57 on books and 0.73 on wikia texts).
Revised: As one can see, KnowFi outperforms the baselines on most input texts, and achieves a remarkable precision on both books and wikia texts (average of 0.57 on books and 0.73 on wikia texts).

Original: We repeat the entity-centric evaluation with manual labels for three relations of special interest in ﬁction, friend , enemy and ally .
Revised: We repeat the entity-centric evaluation with manual labels for three relations of special interest in ﬁction, friend , enemy and ally .

Original: We select 10 popular entities each from LoTR,
Revised: We select 10 popular entities each from LoTR,

Original: GoT and Harry Potter.
Revised: GoT and Harry Potter.

Original: The resulting precision scores are shown in Table 4.
Revised: The resulting precision scores are shown in Table 6.

Original: As one can see,
Revised: As one can see,

Original: KnowFi is achieves high precision among its top extractions, e.g., 78% and 73% precision at rank 1 for friend assertions from books/Wikia texts.
Revised: KnowFi is achieves high precision among its top extractions, e.g., 78% and 73% precision at rank 1 for friend assertions from books/Wikia texts.

Original: Evaluation on Short-Text Datasets.
Revised: Evaluation on Short-Text Datasets.

Original: To evaluate the robustness of KnowFi, we also evaluate its performance on the existing sentence-level RE dataset TACRED, and the short document-level RE dataset DocRED.
Revised: To evaluate the robustness of KnowFi, we also evaluate its performance on the existing sentence-level RE dataset TACRED, and the short document-level RE dataset DocRED.

Original: The results are shown in Tbl.
Revised: The results are shown in Tbl.

Revised: We ﬁnd KnowFi’s

Revised: [Table]

Original: We ﬁnd KnowFi’s performance on TACRED is on par with BERT-Type and BERT-EM (0.66 test-F1, versus 0.63 and 0.62 for the baselines), the modest gain indicating that the combination of entity types and markers is beneﬁcial.
Revised: Peace). [Table] [Table] performance on TACRED is on par with BERT-Type and BERT-EM (0.66 test-F1, versus 0.63 and 0.62 for the baselines), the modest gain indicating that the combination of entity types and markers is beneﬁcial.

Original: On DocRED, KnowFi achieved 0.51 F1-score, slightly below the GLRE model at 0.57 F1-score.
Revised: On DocRED, KnowFi achieved 0.51 F1-score, slightly below the GLRE model at 0.57 F1-score.

Original: We hypothesize that the modest losses stem from the fact that GLRE is speciﬁcally tailored for the short documents of TACRED, where multicontext aggregation is not relevant.
Revised: We hypothesize that the modest losses stem from the fact that GLRE is speciﬁcally tailored for the short documents of TACRED, where multicontext aggregation is not relevant.

Original: At the same time, the single contexts GLRE considers have no inherent size limitation, unlike the 2-sentence distance threshold used in KnowFi.[Table]
Revised: At the same time, the single contexts GLRE considers have no inherent size limitation, unlike the 2-sentence distance threshold used in KnowFi.

Original: In Lord of the Rings , which summary is more informative for Frodo Baggins :

Original: Summary 1: < Frodo, has parent, Drogo > , < Frodo, has culture, Shire > , < Frodo, has enemy, Sauron > , < Frodo, has friend, Sam > , < Frodo, has weapon, Sting >

Original: Summary 2: < Frodo, has owner, Gandalf > , < Frodo, has weapon, Ring > , < Frodo, has parent, Drogo > , < Frodo, has aﬃliation, Sam > , < Frodo, has culture, Marish >

Original:  Study.
Revised: Ablation Study.

Original: To evaluate the impact of passage ranking, we ran KnowFi without passage ranking for both training and prediction.
Revised: To evaluate the impact of passage ranking, we ran KnowFi without passage ranking for both training and prediction.

Original: Instead, passages were randomly selected.
Revised: Instead, passages were randomly selected.

Original: In automated evaluation, without passage ranking, KnowFi achieves comparable recall but lower precision: 0.07 vs. 0.14 on books and 0.12 vs. 0.17 on Wikia texts.
Revised: In automated evaluation, without passage ranking, KnowFi achieves comparable recall but lower precision: 0.07 vs. 0.14 on books and 0.12 vs. 0.17 on Wikia texts.

Original: This pattern is also observed in manual evaluation, where KnowFi, without passage ranking, achieves a precision of 0.43 vs. 0.57 on books and 0.55 vs. 0.73 on Wikia texts.
Revised: This pattern is also observed in manual evaluation, where KnowFi, without passage ranking, achieves a precision of 0.43 vs. 0.57 on books and 0.55 vs. 0.73 on Wikia texts.

Original: Further experiments and ablation studies are in Appendix D and E.
Revised: Further experiments and ablation studies are in Appendix C and D.

Original: Error Analysis.
Revised: Error Analysis.

Original: The precision gain from automated to manual evaluation (Table 1 vs.
Revised: The precision gain from automated to manual evaluation (Table 3 vs.

Original: Table 3) indicates that ground-truth incompleteness is a confounding factor.
Revised: Table 5) indicates that ground-truth incompleteness is a confounding factor.

Original: By inspecting a sample of 50 false positives, we found that 20% originated from incomplete ground truth, while 54% were indeed not inferrable from the given contexts.
Revised: By inspecting a sample of 50 false positives, we found that 20% originated from incomplete ground truth, while 54% were indeed not inferrable from the given contexts, pointing to limitations of the distant-supervision-based training (e.g., extracting friendship from the sentence “Thorin came to Bilbo’s door” ).

Original: Another 15% were errors in determining the subject or object in complex sentences with many entity mentions.
Revised: Another 15% were errors in determining the subject or object in complex sentences with many entity mentions, e.g., extracting friend(Boromir, Gimli) from the sentence “Boromir went ahead, Legolas and Gimli, who by now had become friends, followed.”

Original: Finally, 7% of the false positives captured semantically related relations but missed the correct ones.
Revised: Finally, 7% of the false positives captured semantically related relations but missed the correct ones.

Original: By sampling false negatives, we found that in 52% of the cases the retrieved contexts did not allow the proper inference, indicating limitations in the context retrieval and ranking.
Revised: By sampling false negatives, we found that in 52% of the cases the retrieved contexts did not allow the proper inference, indicating limitations in the context retrieval and ranking.

Original: In 33% of the cases, a human reader could spot the relation in the top-ranked contexts (e.g., hasCulture (Legolas, Elf) in “He saw Legolas seated with three other Elves” ).
Revised: In 33% of the cases, a human reader could spot the relation in the top-ranked contexts (e.g., hasCulture (Legolas, Elf) in “He saw Legolas seated with three other Elves” ).

Original: Appendix
Revised: Appendix

Original: F shows some anecdotal examples for the output of KnowFi and baselines.
Revised: E shows some anecdotal examples for the output of KnowFi and baselines.

Revised: In Lord of the Rings , which summary is more informative for Frodo Baggins :

Revised: Summary 1: < Frodo, has parent, Drogo > , < Frodo, has culture, Shire > , < Frodo, has enemy, Sauron > , < Frodo, has friend, Sam > , < Frodo, has weapon, Sting >

Revised: Summary 2: < Frodo, has owner, Gandalf > , < Frodo, has weapon, Ring > , < Frodo, has parent, Drogo > , < Frodo, has aﬃliation, Sam > , < Frodo, has culture, Marish >

Original: 8. Extrinsic Use Case: Entity Summarization
Revised: 8. Extrinsic Use Case: Entity Summarization

Original: To assess the salience in the extractions produced by KnowFi, we pursued a user study to compare entity summaries, one from KnowFi and one by randomly drawing from one of the three baselines.
Revised: To assess the salience in the extractions produced by KnowFi, we pursued a user study to compare entity summaries, one from KnowFi and one by randomly drawing from one of the three baselines.

Original: Each entity summary includes at most 5 best extractions (distinct relations) from the book series Lord of the Rings, Game of Thrones and Harry Potter.
Revised: Each entity summary includes at most 5 best extractions (distinct relations) from the book series Lord of the Rings, Game of Thrones and Harry Potter.

Original: For each domain, we generate summaries for 5 popular entities.
Revised: For each domain, we generate summaries for 5 popular entities.

Original: We give pairs of summaries, with randomized order, to Amazon master workers for selecting the more informative one.
Revised: We give pairs of summaries, with randomized order, to Amazon master workers for selecting the more informative one.

Original: Table 6 shows an example of this crowdsourcing task.
Revised: Table 8 shows an example of this crowdsourcing task.

Original: The annotators preferred KnowFibased summaries over BERT-Type, BERT-EM and GLRE in 93%, 64% and 81% of the cases, respectively.
Revised: The annotators preferred KnowFibased summaries over BERT-Type, BERT-EM and GLRE in 93%, 64% and 81% of the cases, respectively.

Original: 9. Conclusion
Revised: 9. Conclusion

Original: To the best of our knowledge, this paper is the ﬁrst attempt at relation extraction (RE) from long ﬁctional texts, such as entire books.
Revised: To the best of our knowledge, this paper is the ﬁrst attempt at relation extraction (RE) from long ﬁctional texts, such as entire books.

Original: The presented method, KnowFi, is speciﬁcally geared for this task by its judicious selection and ranking of passages.
Revised: The presented method, KnowFi, is speciﬁcally geared for this task by its judicious selection and ranking of passages.

Original: KnowFi outperforms strong baselines on RE by a substantial margin, and it performs competitively even on the short-text benchmarks TacRed and DocRed.
Revised: KnowFi outperforms strong baselines on RE by a substantial margin, and it performs competitively even on the short-text benchmarks TacRed and DocRed.

Original: The absolute numbers for precision and recall show that there is still a lot of room for improvement.
Revised: The absolute numbers for precision and recall show that there is still a lot of room for improvement.

Original: This underlines our hypothesis that long ﬁctional texts are a great challenge for RE.
Revised: This underlines our hypothesis that long ﬁctional texts are a great challenge for RE.

Original: Our LoFiDo corpus of Wikia texts, book contents, and ground-truth labels will be made available to foster further research.
Revised: Our LoFiDo corpus of Wikia texts, book contents, and ground-truth labels will be made available to foster further research.

Revised: All information can be found at https://www.mpi-inf.mpg.de/yago-naga/knowfi .

Revised: Acknowledgments

Revised: We thank Matteo Cannaviccio and Filipe Mesquita from Diﬀbot Inc. for their advice and support on labeling data, and Diﬀbot Inc. for free access to their RE API.

Revised: We also thank our anonymous reviewers for their valuable comments.

Original: Appendix A. Training Data Extraction
Revised: Appendix A. Training Data Extraction

Original: Many current KBs like Yago, DBpedia or Freebase have been built by extracting the information from infoboxes, category network and leveraging the markup language of Wikipedia.
Revised: Many current KBs like Yago, DBpedia or Freebase have been built by extracting the information from infoboxes, category network and leveraging the markup language of Wikipedia.

Original: The relations of these KBs are then used as schema for many later supervised relation extractors.
Revised: The relations of these KBs are then used as schema for many later supervised relation extractors.

Original: However, for ﬁction, Wikipedia has too low coverage of entities and relevant relations.
Revised: However, for ﬁction, Wikipedia has too low coverage of entities and relevant relations.

Original: Wikia.
Revised: Wikia.

Original: Wikia (or Fandom) is the largest web platform for ﬁction and fantasy.
Revised: Wikia (or Fandom) is the largest web platform for ﬁction and fantasy.

Original: It contains over 385k communities with total of over 50M pages.
Revised: It contains over 385k communities with total of over 50M pages.

Original: Each community (usually discusses about one ﬁctional universe) is organized as a single Wiki.
Revised: Each community (usually discusses about one ﬁctional universe) is organized as a single Wiki.

Original: With a wide range of coverage on ﬁction and fantasy, Wikia is one of top 10 most visited websites in the US (as of 2020) 1 .
Revised: With a wide range of coverage on ﬁction and fantasy, Wikia is one of the 10 most visited websites in the US (as of 2020) 1 .

Original: Crawling.
Revised: Crawling.

Original: We download all universes which contain over 1000 content pages and have available dump ﬁles from Wikia, and get total of 142 universes in the end.
Revised: We download all universes which contain over 1000 content pages and have available dump ﬁles from Wikia, and get total of 142 universes in the end.

Original: From these universes, we extract all information from their category networks and infoboxes, and build a background knowledge base for each universe.
Revised: From these universes, we extract all information from their category networks and infoboxes, and build a background knowledge base for each universe.

Original: Deﬁnition A.1.
Revised: Deﬁnition A.1.

Original: Background KB of an universe is a collection of entities, entity mentions, simple facts that describe relations between entities and a type system of the universe.
Revised: Background KB of an universe is a collection of entities, entity mentions, simple facts that describe relations between entities and a type system of the universe.

Original: Background knowledge extraction.
Revised: Background knowledge extraction.

Original: To extract the background KBs, we follow a simple procedure:
Revised: To extract the background KBs, we follow a simple procedure:

Original: • Type system construction: The type system is extracted from Wikia category network.
Revised: • Type system construction: The type system is extracted from Wikia category network.

Original: We adapt the technique from the TiFi system [Chu et al., 2019] to structure and clean the type system.
Revised: We adapt the technique from the TiFi system [Chu et al., 2019] to structure and clean the type system.

Original: • Entity detection: Entities and entity mentions can be easily extracted from the dump ﬁle.
Revised: • Entity detection: Entities and entity mentions can be easily extracted from the dump ﬁle.

Original: We consider page titles as the entities in the universe (except administration and category pages).
Revised: We consider page titles as the entities in the universe (except administration and category pages).

Original: On the other hand, entity mentions only appear in texts.
Revised: On the other hand, entity mentions only appear in texts.

Original: By using Wiki markup, each mention can be extracted and linked to the entity with a conﬁdent score which is computed based on its frequency.
Revised: By using Wiki markup, each mention can be extracted and linked to the entity with a conﬁdent score which is computed based on its frequency.

Original: • Infobox extraction: Facts about each entity are extracted from its infobox.
Revised: • Infobox extraction: Facts about each entity are extracted from its infobox.

Original: Infobox is presented in table format with the entity’s attributes and their values.
Revised: Infobox is presented in table format with the entity’s attributes and their values.

Original: Each extracted fact is presented in a triple with subject, predicate (relations) and object.
Revised: Each extracted fact is presented in a triple with subject, predicate (relations) and object.

Original: In particular, we consider the main entity as 
Revised: In particular, we consider the main entity as subject

Original: subject, the attributes as predicates, and the values as objects.
Revised: , the attributes as predicates, and the values as objects.

Original: We manually check if there is any misspelling in the relations and merge them if necessary.
Revised: We manually check if there is any misspelling in the relations and merge them if necessary.

Original: This results an average of 158k entities and 13.5k facts in each universe.
Revised: This results an average of 158k entities and 13.5k facts in each universe.

Original: The information from these background KBs is then used for all three later steps.
Revised: The information from these background KBs is then used for all three later steps.

Original: Relation Filtering.
Revised: Relation Filtering.

Original: After extracting the background KBs, we get all relations from the facts of all universes and consider them as relation candidates that can be extracted in ﬁctional domains.
Revised: After extracting the background KBs, we get all relations from the facts of all universes and consider them as relation candidates that can be extracted in ﬁctional domains.

Original: However, beside meta relations which are not really related to the[Figure] [Figure] content of universes, such as season, page, episode,... , there is much noise in the relations since they are manually created by fans.
Revised: However, beside meta relations which are not really related to the content of universes, such as season, page, episode,... , there is much noise in the relations since they are manually created by fans.

Original: To remove noise and keep popular relations, we do relation ﬁltering as follows:
Revised: To remove noise and keep popular relations, we do relation ﬁltering as follows:

Original:  • Pre-processing: a combination of stemming and keeping relations with length at least 3 (except for some relations like job, age, son, etc.).
Revised: [Figure] [Figure] • Pre-processing: a combination of stemming and keeping relations with length at least 3 (except for some relations like job, age, son, etc.).

Original: • Infrequent-relation removing: we only keep relations which are in at least 5 universes and appear in over 20 facts.
Revised: • Infrequent-relation removing: we only keep relations which are in at least 5 universes and appear in over 20 facts.

Original: • Meta-relation removing: we manually check if the relation is a meta-relation.
Revised: • Meta-relation removing: we manually check if the relation is a meta-relation.

Original: In total, there are 247 relations considered as meta-relations.
Revised: In total, there are 247 relations considered as meta-relations.

Original: • Misspelling detection: Misspelling relations are manually detected and grouped with the correct relations, for example, aﬃlation and aﬃliation .
Revised: • Misspelling detection: Misspelling relations are manually detected and grouped with the correct relations, for example, aﬃlation and aﬃliation .

Original: Grouping:
Revised: Grouping:

Original: Synonym relations are manually grouped together, for example, leader and commander .
Revised: Synonym relations are manually grouped together, for example, leader and commander .

Original: After relation ﬁltering, we reduce the number of relations from over 8,000 to 64 relations.
Revised: After relation ﬁltering, we reduce the number of relations from over 8,000 to 64 relations.

Original: These relations are considered as popular relations in ﬁctional domains and used as targets for the relation extraction step.
Revised: These relations are considered as popular relations in ﬁctional domains and used as targets for the relation extraction step.

Original: We realize that in ﬁctional domains, the relations expressing the friendly or hostile relationship between two entities are interesting, hence, we keep friend and enemy as two relations which are always extracted.
Revised: We realize that in ﬁctional domains, the relations expressing the friendly or hostile relationship between two entities are interesting, hence, we keep friend and enemy as two relations which are always extracted.

Original: Figure 4 shows statistics on training data.
Revised: Figure 4 shows statistics on training data.

Original: We publish the training data as supplementary material.
Revised: We publish the training data as supplementary material.

Original: Appendix B. LoFiDo Statistics

Original: Our LoFiDo corpus contains 81,025 instances for training and 20,257 instances for validation.

Original: For testing, we use ﬁve speciﬁc universes, which take input from both books and Wikia texts.

Original: The total number of instances in the test data from Wikia texts is 14,610, while in the case of books, it is 64,120.

Original: Ground-truth data for ﬁve test universes are provided for evaluation.

Original: Table 7 shows statistics on the training and validation data, while Table 8 shows statistics on the ground-truth of ﬁve domains in the test data.

Original: [Table]

Original: [Table]

Original: Appendix C. Background KB Statistics
Revised: Appendix B. Background KB Statistics

Original: One of our contribution is the background KB dataset on popular universes in ﬁctional domains.
Revised: One of our contribution is the background KB dataset on popular universes in ﬁctional domains.

Original: To have an overview about the dataset, Table 9 shows some statistics on our background KBs database, which include information about universes, entities, type systems, relations and facts.
Revised: To have an overview about the dataset, Table 9 shows some statistics on our background KBs database, which include information about universes, entities, type systems, relations and facts.

Original: From the 5 domains used for testing
Revised: From the 5 domains used for testing, the number of relations varies from 10 to 20, and the number of ground-truth triples varies from 1,100 to 4,700 for the ﬁrst three domains, and was between 100 and 200 the last two. Statistics Top

Original: [Table]
Revised: [Table]

Original: the number of relations varies from 10 to

Original: [Table]
Revised: [Table]

Original: and the number of ground-truth triples varies from 1,100 to 4,700 for the ﬁrst three domains, and was between 100 and 200 the last two.

Original: Appendix D. Additional Experiments 
Revised: Appendix C. Additional Experiments Similarity

Original: Similarity Threshold.
Revised: Threshold.

Original: In our experiments, we consider all passages with cosine above 0.
Revised: In our experiments, we consider all passages with cosine above 0.

Original: as positive training samples (section 7.1).
Revised: as positive training samples (section 7.1).

Original: To assess the eﬀect of the similarity threshold, we conduct an ablation study on it.
Revised: To assess the eﬀect of the similarity threshold, we conduct an ablation study on it.

Original: Table 10 reports the automated results of KnowFi on both books and Wikia texts, where the threshold varies.
Revised: Table 10 reports the automated results of KnowFi on both books and Wikia texts [Table] where the threshold varies.

Original: For the author response we completed two other runs (threshold 0.4 and 0.6) that indicate modest inﬂuence, for the ﬁnal version we would provide insights for all threshold values from 0 to 1 (in 0.1 step size).
Revised: For the author response we completed two other runs (threshold 0.4 and 0.6) that indicate modest inﬂuence, for the ﬁnal version we would provide insights for all threshold values from 0 to 1 (in 0.1 step size)

Original: The results show that, with a similarity threshold around 0.5, the model achieves the best

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: The results show that, with a similarity threshold around 0.5, the model achieves the best

Original: F1.
Revised: F1.

Original: By increasing the similarity threshold, the model is able to achieve higher precision but lower recall and vice versa.
Revised: By increasing the similarity threshold, the model is able to achieve higher precision but lower recall and vice versa.

Original: Embedding-based Passage Ranking.
Revised: Embedding-based Passage Ranking.

Original: KnowFi uses a simple TF-IDF-based schema for passage ranking.
Revised: KnowFi uses a simple TF-IDF-based schema for passage ranking.

Original: To assess the eﬀectiveness of this method, we conduct an ablation study on the ranking step.
Revised: To assess the eﬀectiveness of this method, we conduct an ablation study on the ranking step.

Original: Instead of using TF-IDF, we compute the embeddings of the passages and the relation contexts using Sentence-BERT [Reimers and Gurevych, 2019].
Revised: Instead of using TF-IDF, we compute the embeddings of the passages and the relation contexts using Sentence-BERT [Reimers and Gurevych, 2019].

Original: The cosine similarity between the passage embedding and the relation context embedding is then computed using the sklearn library.
Revised: The cosine similarity between the passage embedding and the relation context embedding is then computed using the sklearn library.

Original: We select all passages with cosine above 0.
Revised: We select all passages with cosine above 0.

Original: (range [-1,1]), as positive training samples, with maximum of 5 passages per each training instance.
Revised: (range [-1,1]), as positive training samples, with maximum of 5 passages per each training instance.

Original: Table 11 shows the automated results of KnowFi on both books and Wikia texts.
Revised: Table 11 shows the automated results of KnowFi on both books and Wikia texts.

Original: The higher scores on recall shows that the embeddings can help the model capture the semantic relationships between passages and relation contexts better, especially when handling the cases of synonymy, while TF-IDF only handles the cases of lexical matching.
Revised: The higher scores on recall shows that the embeddings can help the model capture the semantic relationships between passages and relation contexts better, especially when handling the cases of synonymy, while TF-IDF only handles the cases of lexical matching.

Original: However, in general, both techniques are on par, and embeddings do not improve the results, in terms of F1-score.
Revised: However, in general, both techniques are on par, and embeddings do not improve the results, in terms of F1-score.

Original: GLRE with Passage Ranking.
Revised: GLRE with Passage Ranking.

Original: In our experimental setup, the inputs of GLRE [Wang et al., 2020] (for both train and test) are randomly selected.
Revised: In our experimental setup, the inputs of GLRE [Wang et al., 2020] (for both train and test) are randomly selected.

Original: To assess the eﬀect of passage ranking on GLRE, we conduct a study where the inputs of GLRE are selected by using our method for passage ranking.
Revised: To assess the eﬀect of passage ranking on GLRE, we conduct a study where the inputs of GLRE are selected by using our method for passage ranking.

Original: Table 12 shows that, by using passage ranking to ﬁlter the inputs, GLRE is able to achieve higher precision and recall, compared to GLRE without passage ranking.
Revised: Table 12 shows that, by using passage ranking to ﬁlter the [Table] [Table]

Original: However, this enhanced variant is still inferior to KnowFi by a substantial margin
Revised: Wang et al

Original: [Table]
Revised: [Table]

Revised: + Passage

Original: [Table]
Revised: [Table]

Revised: inputs, GLRE is able to achieve higher precision and recall, compared to GLRE without passage ranking.

Revised: However, this enhanced variant is still inferior to KnowFi by a substantial margin.

Original: Appendix E. Impact of Training Data Quality
Revised: Appendix D. Impact of Training Data Quality

Original: Training data is one of the most important factors that impact the quality of the supervised models, therefore, it is essential to maintain the quality of the training data, especially when working on speciﬁc domains where the training data is usually not available.
Revised: Training data is one of the most important factors that impact the quality of the supervised models, therefore, it is essential to maintain the quality of the training data, especially when working on speciﬁc domains where the training data is usually not available.

Original: To evaluate the quality of our training data, we compare KnowFi and it variant (e.g. without using passage ranking on training data collection) with two other methods which are trained using manual training datasets:
Revised: To evaluate the quality of our training data, we compare KnowFi and a variant (i.e., without using passage ranking on training data collection) with two other methods which are trained using manual training datasets:

Original: • TACRED
Revised: • TACRED

Original: [Zhang et al., 2017], a popular dataset for relation extraction on the sentence level.
Revised: [Zhang et al., 2017], a popular dataset for relation extraction on the sentence level.

Original: We train our relation extraction model using TACRED and use the model to extract the relations from the test data.
Revised: We train our relation extraction model using TACRED and use the model to extract the relations from the test data.

Original: • Diﬀbot [Mesquita et al., 2019], a commercial api for relation extractions.
Revised: • Diﬀbot [Mesquita et al., 2019], a commercial API for relation extractions.

Original: We run  Diﬀbot API on our test data to extract the relations.
Revised: We run the Diﬀbot API on our test data to extract the relations.

Revised: Note that these comparisons are not systematic, as there are confounding other diﬀerences between these two methods and KnowFi.

Revised: In particular, the comparison with TACRED also comes with a reduction to sentence-level training examples only, while the Diﬀbot API is a pre-trained general-purpose extractor with necessary limitations in more speciﬁc use cases.

Original: We automatically evaluate the extractions on three popular relations, spouse, sibling, child , since they are contained in all datasets
Revised: We automatically evaluate the extractions on three popular relations, spouse, sibling, child , since they are contained in all datasets.

Original: Results.
Revised: Results.

Original: Table 13 reports the results on two universes, Lord of the Rings and Game of
Revised: Table 13 reports the results on two universes, Lord of the Rings and Game of

Original: Thrones .
Revised: Thrones .

Original: The results show that, our training data achieves comparable results with other datasets and even higher F1-scores, in both books and Wikia texts. 
Revised: The results show that, our training data achieves comparable results with other datasets and even higher F1-scores, in both books and Wikia texts. Universes Models [Table]

Revised: [Table]

Original: Appendix F. Anecdotal Examples
Revised: Appendix E. Anecdotal Examples

Original: Table 14 gives examples for the output of the various methods on sample contexts.
Revised: Table 14 gives examples for the output of the various methods on sample contexts.

Original: The red color texts denote subjects and the blue color texts denote objects.
Revised: The red color texts denote subjects and the blue color texts denote objects.

Original: [Table]