Combining Analogy with Language Models for Knowledge Extraction
Anonymous authors
Abstract
Learning structured knowledge from natural language text has been a long-standing challenge.
Previous work has focused on speciﬁc domains, mostly extracting knowledge about named entities (e.g. countries, companies, or persons) instead of general-purpose world knowledge (e.g. information about science or everyday objects).
In this paper we combine the Companion Cognitive Architecture with the BERT Language Model to extract structured knowledge from text, with the goal of automatically inferring missing commonsense facts from an existing knowledge base.
Using the principles of distant supervision, the system learns functions called query cases that map statements expressed in natural language into knowledge base relations.
Afterwards, the system uses such query cases to extract structured knowledge using analogical reasoning.
We run experiments on 2,
Simple English Wikipedia articles, where the system is able to learn high precision facts about a variety of subjects from a few training examples, outperforming strong baselines.
1. Introduction
Knowledge bases (KBs) have proven to be a very useful resource in many artiﬁcial intelligence applications including Dialogue, Question Answering, and Semantic Search [Bradeˇsko et al., 2010, Lukovnikov et al., 2017, Ribeiro et al., 2019].
While KBs serve as a good source of explicit knowledge, they often suﬀer from missing relations and facts.
For this reason, extracting knowledge from text with the intent of improving existing KBs has been a longstanding goal in the ﬁeld of Natural Language Processing.
This task is usually referred to as knowledge extraction and is considered challenging because of the inherent diversity and ambiguity of natural language text.
In recent years, there has been substantial progress in automated knowledge extraction, but the resources built by these methods are typically limited by at least one of these four constraints: (1) They lack general world knowledge about common nouns, targeting named entities instead.
Examples include NELL [Carlson et al., 2010] and FreeBase [Bollacker et al., 2008] (2) They work with a small set of semantic relations.
For instance, WordNet [Miller, 1998] has less than a dozen.
(3) They are supervised in nature, requiring vast amounts of training data.
Examples include KBP37 [Zhang and Wang, 2015] and TACRED
[Zhang et al., 2017].
(4) They represent entities and relations as text strings, which are often ambiguous and harder to manipulate.
Examples include ConceptNet [Speer et al., 2017],
Aristo Tuple KB [Mishra et al., 2017], ATOMIC [Sap et al., 2019], Dice [Chalier et al., 2020], and Ascent [Nguyen et al., 2021].
By contrast, our goal is to populate an existing ontology, seeded by the facts and schemas already deﬁned in the KB.
We use the Companion Cognitive Architecture [Forbus and
Hinrich, 2017] which has a general-purpose semantic parser capable of generating semantic representations grounded by relations and entities present in the architecture’s KB.
In general, KBs containing common-sense world knowledge are incomplete.
Consequently, a system has to rely on a small amount of training samples to learn new facts (in our KB around 94.8% of relations are associated with less than 100 facts).
To solve this problem, we propose a novel technique based on analogical Q/A training [Crouse et al., 2018].
Our approach eﬃciently creates query cases which are later analogically retrieved and combined to interpret new texts.
Our system uses distant supervision to automatically generate training data for analogical training which extracts general knowledge from
Simple English Wikipedia 1 articles.
In this work we combine structural similarity with
BERT-guided word sense disambiguation and fact classiﬁcation for few-shot learning of common-sense facts.
To the best of our knowledge, we are the ﬁrst to combine these two techniques.
2. Related Work
Our work relates to learning structured knowledge from natural language text with the goal of extending a KB without relying on labeled corpora.
Such tasks are often classiﬁed as information extraction (IE) or relation extraction (RE).
Learning under distant supervision for RE without labeled data was initially introduced by Mintz et al.
They showed how text can be heuristically aligned with a knowledge graph (KG) to generate instances used to train a classiﬁer.
Subsequent systems incorporated not only syntactic textual features but also jointly learned KG information to perform extraction, either by syntactic-semantic rules from random walks [Lao et al., 2012], scoring functions of low-dimensional embeddings [Weston et al., 2013], mutual attention between
KG and text [Han et al., 2018a], or neural encoder-decoder models [Distiawan et al., 2019].
Other recent methods have focused on using neural networks in the context on few-shot relation extraction [Han et al., 2018b, Gao et al., 2019, Qu et al., 2020, Wang et al., 2021].
Such methods have achieved impressive results on RE but focused on resources that either require many hundreds of training examples or mostly extract facts about named entities.
The work of Sharma et al.
[2019] is more related to ours as they used distant supervision to learn Cyc style facts [Lenat and Guha, 1993] and were able to leverage the structure of the KB’s ontology to improve accuracy.
Their experiments however were limited to a small set of relations and they used more shallow textual patterns instead of semantic-parsed interpretation of sentences.
To avoid dependency on pre-deﬁned schemas, Open IE [Yates et al., 2007, Fader et al., 2011, Angeli et al., 2015, Romero et al., 2019, Romero and Razniewski, 2020] extracts knowledge from text using an open set of relations and entities.
Commonly, the relation names are represented as text linking two arguments.
This representation leads systems to commonly generate multiple instances of the same relation.
Riedel et al.
[2013] tackled the problem of identifying equivalence between extracted relations by learning canonical schemas.
Mishra et al.
[2017] used this approach to create a large KB with subject-predicateobject triples of generic statements about the world.
The advantage of our approach over
Open IE is that we output symbolic, unambiguous representation of knowledge.
Textual representations make reasoning harder, and even though some of the predicates are typed, they often suﬀer from word sense ambiguity.
Analogy uses the relational structure from semantic representations of texts as a way of learning abstract concepts without vast amounts of labeled data.
This approach has been previously applied to solve many natural language problems such as word sense disambiguation [Barbella and Forbus, 2013], question answering [Crouse et al., 2018], multimodal dialog systems [Wilson et al., 2019] and process understanding [Ribeiro et al., 2019].
In our work, analogical similarity is used to retrieve rule-like constructs called query cases.
Retrieval of stored cases has been extensively used by many case-based reasoning systems [Daniels and
Rissland, 1995, Burke et al., 1997].
However, these case-based reasoning approaches diﬀer from analogical learning since they use shallow pattern matching or standard information retrieval methods instead of structural similarity applied to parsed semantics.
3. Problem Deﬁnition
The task is to populate an existing KB with facts extracted from natural language text, constrained by the entities and relations already deﬁned in such KB.
The standard relation extraction task represents structured knowledge as graphs [Weston et al., 2013, Zhang et al., 2020].
Relations are edges in the graph and are stored as triples ( h, r, t ), as in (Spain,borders,France) .
Here, the knowledge is represented using the CycL higher-order logic and OpenCyc ontology [Lenat and Guha, 1993], which is more expressive.
Further details on CycL can be found in Appendix A.
Formally, we denote the knowledge base as KB =
( C , R , F ), where C is the set of concepts (in CycL they can be either collections, entities, or logical functions), R is the set of relations, and F is the set of facts.
Each fact f ∈ F is a nested tuple, represented as a tree data structure, where the root node f 1 ∈ R is the relation and each internal node f 2 , · · · , f n are the nested arguments.
A text corpus is used as input, where each sentence is a sequence of tokens s = t 1 , · · · , t m .
The ﬁnal task is to ﬁnd all arguments for relations r ∈ R ; that express common-sense beliefs extracted from sentences in the corpus.
Note that our proposed task is harder than common RE tasks.
They often only attempt to infer either r given ( h, ? , t ) or t given ( h, r, ?), instead of predicting all arguments given a relation.
4. Background
We use the Companion Cognitive Architecture as a foundation for our system, as it contains key components that allow for eﬃcient knowledge extraction.
Namely, we use the NextKB knowledge base, the CNLU natural language processing system [Tomai and Forbus, 2009], and analogical capabilities provided by SME, the Structure Mapping Engine [Forbus and
Hinrich, 2017].
We brieﬂy explain each of these components below.
Knowledge Base:
The NextKB 2 knowledge base integrates material from OpenCyc and
FrameNet [Baker et al., 1998], as well as a large-scale lexicon [McFate and Forbus, 2011].
All facts in NextKB are represented using the CycL ontology language and they are used as seeds for further inference of missing facts.
Natural Language Understanding:
The CNLU language system combines Allen’s (1995) bottom-up chart parser with ideas from Discourse Representation
Theory [Kamp and Reyle, 2013] to produce semantic representations from the input sentences.
Specifically, CNLU produces predicate calculus interpretations represented in CycL language.
CNLU is task agnostic and the syntactic parse and word sense ambiguities are explicitly represented as distinct and internally consistent choice sets.
Choice sets are often derived from FrameNet semantic frames.
Logical constraints among the statements are designed such that by choosing one semantic interpretation, other conﬂicting choices are ruled out.
Analogy:
We use the Structure-Mapping Engine (SME) by Forbus and Hinrich [2017] to match stored query cases.
This component compares two structured, relational representations and creates one or more mappings between them.
These mappings consist of (1) a set of correspondences, specifying what entities and predicates in one description go with entities and predicates in the other, (2) a score indicating the similarity between the representations, and (3) candidate inferences that project information from one description to the other.
For more computationally eﬃcient retrieval, our approach uses MAC/FAC [Forbus et al., 1995] to apply query cases learned during training for extraction of structured knowledge.
BERT Language Model:
The Bidirectional Encoder Representations from Transformers (BERT)
[Devlin et al., 2018] is a language model trained on a large unlabeled corpus (around 3.3B words from English Wikipedia and BooksCorpus) that jointly conditions on both left and right context for all layers.
We use BERT for word sense disambiguation and classiﬁcation of extracted facts, as described in following sections.
The ﬁrst input token is always a special token [CLS] which is used for sentence classiﬁcation while sentences boundaries are marked with the special token [SEP].
The ﬁnal hidden vector of the special [CLS] token is deﬁned as T CLS ∈ R H , and the ﬁnal hidden vector for the i th input token t i as T i ∈
R H , where H is the hidden state size.
5. Knowledge Base Completion System
Here we describe the knowledge base completion system, which is sketched in Figure 1.
The pipeline consists of three steps as described below.
5.1 Retrieval of Candidate Relations
During the training phase of the pipeline, the system learns what patterns in a sentence could entail the discovery of new facts.
The system uses CNLU to process each sentence from the input corpus, producing predicate calculus semantic representations.
Afterwards, the KB is used to retrieve facts that mention more than one concept (collections or entities) which are part of the semantic representation.
For instance, the sentence “Cats use many diﬀerent sounds for communication, including meowing and purring.”
could potentially be paired with (animalTypeMakesSoundType Cat PurringSound) while the sentence “Herbivores only eat plants.” could be paired with (relationAllExists eatsWillingly
Herbivore Plant) .
These facts are referred to as candidate relations, represented as the set Ψ where Ψ i ∈ F .
Their collections and entities are called anchor concepts Ω where Ω i ∈ C .
5.2 Word Sense Disambiguation
The CNLU semantic representation may contain conﬂicting word senses (e.g. “Mouse” can be either Mouse-Rodent or ComputerMouse ) and these senses are often derived from
FrameNet semantic frames.
To disambiguate the word sense choices we use a BERT-based frame classiﬁer that assigns a score to each semantic form given its context.
We ﬁne-tune the BERT language model using FrameNet annotated data set 3 .
Given a sentence with tokens t 1 , . . .
, t N , we want to compute the probability of a FrameNet frame
F k being associated with a subsequence of such tokens t i , . .
t j .
The model assigns probabilities to each frame resembling the method described in Tan and Na [2019].
During ﬁne-tuning, a new network layer with weights W ∈ R K × 4 H is introduced, where K is the number of diﬀerent FrameNet frames.
Finally, the scoring function for a frame F k is given by:
[Equation]
We assume that the subsequence of tokens is of size four (i.e. j − i = 4).
Otherwise, the concatenation hidden states [ T i : T j ] are padded with zeroes until it reaches the size 4 H .
All choice sets that are not associated with FrameNet frames were assigned a probability drawn from a discrete uniform distribution.
This disambiguation module helps the system choose the correct interpretations for both the antecedents and the learned facts.
5.3 Construction of Query Cases
[Figure]
Anchor concepts are marked by the logical function GeneralizationFn.
Antecedents can be interpreted as input features while consequent as output predictions
Given the set of sentence fact pairs, the system performs a modiﬁed version of Analogical Q/A Training [Crouse et al., 2018] to create what we call query cases.
These query cases can be thought of as discrete functions that map from a subset of sentence semantic forms (antecedents) to new structured knowledge facts (consequent).
Query cases can also be interpreted as discrete functions mapping from input features to output predictions.
The main challenge is selecting the relevant subset of the semantics that entails the chosen fact.
We use the procedure for semantic selection (Algorithm 1) to automatically choose such subset.
This algorithm takes into consideration properties of the semantic representation and anchor concepts to rank the semantic forms that should be added to the query case, namely: semantic connectivity, fact correlation and type generality.
First, the semantic connectivity is used to judge which syntactic and semantic features connect the anchor concepts together, which could potentially identify the relation between such concepts.
Second, when there are multiple possible interpretations for the anchor concepts, the fact correlation helps with disambiguation by prioritizing concepts that are more closely connected to the candidate relation.
Third, type generality gives lower priority to concepts which are too speciﬁc since the system needs to generalize the candidate relation to new concepts.
The type generality takes as input a concept X , and outputs the number of facts in the KB that contain such concept.
The formula is as follows:
[Equation]
The subroutine Ont-Con was introduced by Ribeiro et al.
[2019] and computes the ontological similarity of two expressions.
Given two collections, entities, or relations ϕ 1 , ϕ 2 ∈ {C ∪ R} in the KB, and the set of structural relations of the ontology (examples in OpenCyc are isa , argIsa , genls ), form a graph between ϕ 1 and ϕ 2 .
Let P be the set of paths between ϕ 1 and ϕ 2 , e u,w be the vertices in each of these paths and deg ( x ) be the out-degree of a vertex x .
Then traversing the nodes in this graph can give the relatedness between two concepts.
Therefore, we deﬁne the ontological connection to be:
[Equation]
| P | The semantics selection algorithm ﬁlters out many of the false positive candidate relations that have low scores.
Once the subset of semantic forms is selected, the query case is created.
One example of such query case generated by the sentence “Male deer have
Input: Sentence semantics Υ, retrieved candidate relation Ψ, anchor concepts Ω
Output: Subset of semantics Υ ∗ ⊆
Υ
Parameters: λ 1 , λ 2 , λ 3 , λ 4 , ε First select connecting forms υ u,v ⊆ Υ from paths between any two anchors Ω u and Ω v through the semantic representation graph with depth up to 3; foreach Υ u ∈ Υ do if Ω v ∈ H u then con ← | υ v,x | ; fac ← (cid:80) v Ont-Con (Ω v , Ψ); gen ← Gen (Ω v ); scr [Υ u ] ← λ 1 ∗ con
+ λ 2 ∗ log 10 (1 + fac ) + λ 3 ∗ log 10 (1 + gen ) ; else Ω (cid:48) ← all collections from H u ; scr [Υ u ] ← λ 2 ∗ (cid:80) v Ont-Con (Ω (cid:48) v , Ψ); end scr [Υ u ] ← scr [Υ u ] + λ 4 ∗
Fra-Pro (Υ u ); end Υ (cid:48) ← sort Υ by scr
[Υ u ]; Υ ∗ ← ∅ ; foreach Υ (cid:48)
u ∈ Υ (cid:48) do if not Conflicts (Υ (cid:48) u , Υ ∗ v ) and scr
[Υ u ] > ε 1 then Υ ∗ ← Υ ∗ ∪ Υ (cid:48)
u ; end end antlers.”
is depicted in Figure 2.
All query cases created during the training phase are stored in the KB for subsequent retrieval.
5.4 Learning New Facts
When faced with natural language text, the system must extract structured knowledge that could be expressed by the known relations from the training data.
The ﬁrst step is to parse the text using CNLU, which generates the semantic representation of the sentences.
Such semantic representations are used as probes for analogical retrieval using MAC/FAC and further processed using SME, which returns a small number of query cases stored during training that best matches the semantic representation of the sentence (up to 20 cases per probe).
Ontological Scoring:
In order to decide if the retrieved cases contain suﬃcient evidence that the semantics of the sentence entails a new fact, the system uses a heuristic to compute scores for each instantiated case.
This ontological scoring ensures that the learned facts are relatively similar to the original training facts, that the word senses of the semantic interpretation have high probability and that the concepts are not too generic.
Given the sentence semantics Υ and the instantiated query case with antecedents A and consequent C , anchors of the instantiated consequent Ω c and the anchors of the original query case consequent Ω o .
Then the score is given by:
[Equation]
The system uses this score to ﬁlter out new facts that are below a hyper parameter threshold ε 2 .
The new facts, the provenance and the scores are all stored in the KB.
BERT Fact Classiﬁcation
The second classiﬁcation function takes advantage of BERT’s contextual representation and word embeddings to capture common sense knowledge which were implicitly stated in its large pre-training corpus.
Inspired by the approach used by Yao et al.
[2019], we ﬁne-tuned BERT to predict the plausibility of extracted facts.
For BERT input, we represent concepts and relations by an automatically constructed textual format.
For instance, the relation (pathogenCausesConditionType Borrelia LymeDisease) is converted into “[CLS] pathogen causes condition type [SEP] lyme disease [SEP] borrelia
[SEP]” which is used as input to the model.
To create negative training examples, we simply switch a concept in a positive training example from F with some other randomly chosen concept from C .
We also create negative examples by shuﬄing the order of concepts inside the fact (excluding facts in the set of symmetrical relations R ◦ ).
Considering a simpliﬁed representation of a fact as a triple ( h, r, t ) where r ∈ R and h, t ∈ C , the set of negative training examples is deﬁned by:
[Equation]
When ﬁne-tuning BERT we use both F and F − as training data.
A new network layer with weights W (cid:48) ∈ R 2 × H is introduced.
The scoring function s τ for a fact τ ∈ F ∪ F − is given by the formula: s τ = sigmoid ( T CLS W (cid:48) T ) where s τ ∈ R 2 is a two dimensional vector representing the model output.
The pre-trained parameter weights and new weights are updated using gradient descent that minimizes the cross-entropy loss:
[Equation]
Where y τ is the output label deﬁned by y τ =
if τ ∈ F and y τ
= 0 if τ ∈ F − .
After training, this model achieves a validation accuracy of 89.0% on the task of fact classiﬁcation.
In following experiments, the BERT fact classiﬁcation is applied after the ontological score for further ﬁltering of learned facts.
6. Experiments 6.1 Dataset and Corpus
In order to evaluate these ideas, we experimented with extracting knowledge from Simple
English Wikipedia articles.
We select a subset of the articles by compiling a list of common nouns, and ﬁltering out articles with titles that are not part of this list.
The ﬁnal corpus contains 2,679 articles.
We also select a subset of the NextKB knowledge base which we believe expresses general facts about the world.
Relations which we considered not proper for extraction (e.g. quotedIsa and givenNames ) were removed.
The ﬁnal data set contains 66,649 facts with 3,745 distinct relations, covering 21,462 concepts.
Knowledge bases representing generic facts are often incomplete, where the frequency of relations follows a long tail distribution [Xiong et al., 2018].
Our data set also follows this pattern as shown in Figure 3, which further justiﬁes the use of data eﬃcient learning methods such as analogical learning for this type of task.
6.2 Baselines
[Figure]
We compare our system against three baselines.
The ﬁrst is based on the Textto-Text Transfer Transformer (T5)
[Raffel et al., 2019], which is ﬁne-tuned to take as input a sentence and output a sequence of structured knowledge facts.
For instance, the sentence “Books are made of paper.”
and the structured fact “typicalMainConstituent-TypeType [S]
BookCopy [S] Paper” were used as input and output pairs during training.
The other two baselines are Relation Extraction models from OpenNRE [Han et al., 2019], namely the CCN [Zeng et al., 2014] and BERT [Devlin et al., 2018] based encoders.
We adapt the task by excluding any facts with more than two arguments.
The surface form of the concepts in the KB (taken from CNLU lexical mappings) are used to identify possible head and tail arguments. The Relation Extraction models use these head and tail concepts as inputs and predict the relation among them.
Appendix B contains further details about training and hyper parameters.
6.3 Evaluation and Results
After running each system on the sentences from Simple English Wikipedia we evaluate the learned facts.
Notice that all extracted facts that were already present the KB were discarded.
Often the evaluation of relation extraction system is done by creating a held-out test set
[Mintz et al., 2009, Han et al., 2018a].
This would not be appropriate for this data set since the KB is very sparse and has many missing facts.
For this reason, we opted for manual evaluation of the results.
We randomly sampled 8% of the extracted facts which were given to a graduate level annotator for manual evaluation.
We report the estimated precision for each compared method in Table 1.
The Text-to-Text (T5) was the best baseline, with estimated accuracy of 26.0% and able to extract 6772 new facts.
The adapted Relation Extraction models with CNN and BERT encoders extracted 2,448 and 2,804 new facts, respectively.
Interestingly, these models
[Table] performed fairly well on the distant supervision development set (achieving over 80% F scores), but they did not perform as well when it comes to discovering facts (not already in the knowledge base), possibly because such facts are outside of the distribution of the training data.
Potential improvements to these baseline models include using better word senses for aligning sentences with facts or learning KG embeddings [Distiawan et al., 2019, Wang et al., 2021].
The purely analogical system has an estimated accuracy of 45.7% and is able to extract 4,458 facts covering 58 distinct relations.
The accuracy improves considerably after applying the BERT fact classiﬁcation, jumping to 71.4%.
Further break-down of precision for a subset of the relations is shown in Appendix C. After inspecting the results more closely, we noticed how the system is capable of performing well even in the case of sparse training data.
For instance, the relation (relationAllExists eatsWillingly ?
x ?y) only has four instances in the NextKB set of facts, however the system is able to expand that to 156 new facts.
We investigated the inaccuracies of the Analogy system and found that roughly 42% of the failures were related to semantic parsing errors.
The remaining errors were mostly due to errors in the BERT-based disambiguation component (which were often caused by the lack of training data from FrameNet) or the incorrect semantic selection during the creation of query cases.
7. Conclusion
With the intention of populating large common-sense KBs, our work introduced a pipeline for extracting structured knowledge from a text corpus.
While learning instance level knowledge about named entities (e.g. companies or persons) is important, we believe that learning type-leveled information is more relevant for building general purpose AI systems.
We have presented a hybrid approach built on the Companion Cognitive Architecture that combines analogical learning and BERT-based word sense disambiguation and fact classiﬁcation.
We show how this approach is eﬀective even when applied to sparse KBs, where the frequency of relations follows a long-tail distribution.
The system was able to outperform strong baselines and learn new facts with reasonable accuracy from a small set of distantly supervised training examples, expanding the number of facts for certain relations by two orders of magnitude.
Some directions for future work include (1) Processing more data and extracting more facts from the web.
Simple English Wikipedia has served as a starting point for showing the eﬀectiveness of our approach, but to signiﬁcantly increase the coverage of an existing KB a larger scale text corpus would be required.
(2) Extracting richer representations of knowledge.
We hypothesize that our system is suitable for learning and identifying knowledge represented as frames, since it can learn facts with arbitrary arity and depth.
For instance, this could be useful for extracting knowledge about events or processes.
Exploring more statistical properties of query cases.
We believe that identifying which subset of antecedents contributes most for a given relation could improve the system’s accuracy.
Acknowledgments
Omitted for blind review.
Appendix A. CycL Representation
The key diﬀerences between CycL representation and other commonly used representations are (1) CycL represents concepts as disambiguated constants.
For example, the concept “Mouse” can be represented by either be Mouse-Rodent or ComputerMouse , depending on the word sense.
Concepts are represented by collections, e g. ComputerMouse is a collection, while Mouse32 might denote an instance of that collection, a speciﬁc computer mouse; (2) relations can be of arbitrary arity, instead of only using binary relations, which allows relationships like between to be represented; (3) relations can be higher-order, e.g. (relationAllExists eatsWillingly Omnivore Meat) takes predicates and concepts (aka Collections in OpenCyc) as arguments; (4) logical functions can be used to produce new terms, e.g. (FruitFn AppleTree) represents the collection of the fruits of apple trees.
Appendix B. Training and Hyperparameters
Follows the training details, including choice of Hyperparameters for the system sub-components and neural models which were described previously. [Figure]
Word Sense Disambiguation
The word sense disambiguation trained on the FrameNet data.
The BERT base and uncased model was ﬁne-tuned using the Adam optimizer, with a learning rate of 1 · 10 − 4 , trained for 10 epochs.
Semantic Selection and Ontological Scoring:
These modules used a linear combination of scores to select the best semantic representation, as well as scoring learned facts.
The values for the parameter as as follows: λ 1 = 1 , λ 2 = 0 .
2 , λ 3 = 5 , λ 4 = 5 , ε 1 = 0 .
3 , λ 5 = 3 , λ 6 = 1 , λ 7 = 3 , λ 8 = 3 , ε 2 = 2 . 5 , .
These values were chosen depending on the importance of each score, taking into consideration the results on a sample of the training data.
Fact Classiﬁer:
The fact classiﬁcation model ﬁne-tuned a BERT (base uncased) using a learning rate 5 · 10 − 5 , AdamW optimizer and trained for 3 epochs.
The generated dataset was split into 63,805 training and 7,090 validation data-points.
Text-to-Text:
When training the text-to-text model we ﬁne tune T5-small on 3, sentence-fact pairs using learning rate 3 · 10 − 4 , AdamW optimizer and trained for 2 epochs.
Relation Extraction:
The relation extraction models are trained on 28,800 training and 3,000 validation sentence-fact pairs, %50 of which were negative examples. Negative examples (labeled as “not related”) were created using sentences that reference two entities that are not connected by a fact in the KB.
All facts with more than two examples were discarded.
The model with CNN encoder was trained with a learning rate of 1 · 10 − 1 , and 160 batch size for 500 epochs.
The BERT based model was trained with a learning rate of 2 · 10 − 5 , and 64 batch size for 10 epochs.
The remaining parameter were kept as the OpenNRE defaults.
Appendix C. Result Breakdown
Figure 4 shows the precision break-down for a subset of the relations (the manual evaluation sampled up to 30 extraction per relation).