Multilingual Fact Linking
Abstract
Knowledge-intensive NLP tasks can beneﬁt from linking natural language text with facts from a Knowledge Graph (KG).
Although facts themselves are language-agnostic, the fact labels (i.e., language-speciﬁc representation of the fact) in the KG are often present only in a few languages.
This makes it challenging to link KG facts to sentences in languages other than the limited set of languages.
To address this problem, we introduce the task of Multilingual Fact Linking (MFL) where the goal is to link fact expressed in a sentence to corresponding fact in the KG, even when the fact label in the KG is not available in the language of the sentence.
To facilitate research in this area, we present a new evaluation dataset, I NDIC L INK .
This dataset contains 11,293 linked
WikiData facts and 6,429 sentences spanning English and six Indian languages.
We propose a
Retrieval+Generation model, R E
FC O G, that can scale to millions of KG facts by combining Dual
Encoder based retrieval with a Seq2Seq based generation model which is constrained to output only valid KG facts.
R E FC
O G outperforms standard Retrieval+Re-ranking models by 10.7 pts in
Precision@1.
In spite of this gain, the model achieves an overall score of 52.1, showing ample scope for improvement in the task.
Code and data are available at github.com/SaiKeshav/mﬂ
1. Introduction
Knowledge Graphs (KG) are a large and useful source of information about the world and are composed of curated facts regarding entities.
Each fact asserts a relation that exists between two entities, which can be expressed as (subject; relation; object) .
KGs have proven to be helpful for NLP tasks like Question Answering [Saxena et al., 2020] and are used in a variety of industry applications
[Dong, 2017, Fensel et al., 2020].
KG facts represent knowledge explicitly in a structured format that allows for greater interpretability.
Linking KG facts with text has several beneﬁts.
Fact linking can potentially be used by systems to verify claims and detect misinformation.
Moreover, KG facts allow for rich search experiences by supporting knowledge panels which require triggering the correct panels by connecting search queries to KG facts.
Recent works [Verga et al., 2021] have shown that augmenting pretrained transformers with KG facts allow for modular management of knowledge that can be tuned according to the end application, such as adding new facts or removing stale ones.
Improved fact-linked text can beneﬁt such models even further.
Knowledge graphs such as Wikidata 1 often contain entity and property labels in multiple languages (although with severe skew [Kaffee et al., 2017]).
For example, Argovia and Argau are the
English and Spanish entity labels for entity Q11972 in Wikidata.
Similarly, country and pa´ıs are the English and Spanish property labels for property P17.
Combining these, we deﬁne the notion of fact labels which are language-speciﬁc representations of an otherwise language-agnostic fact.
For example, (Aargau; country; Switzerland) and (Argovia; pa´ıs; Suiza) are the English and Spanish fact labels for the fact (Q11972; P17; Q39) in Wikidata.
While linking KG facts to text has received some attention in literature [Elsahar et al., 2018], most of that work has been restricted to English fact labels and text.
There is a growing need to connect fact and their mentions in text, especially when languages of the fact label and text are different.
For example, we would like to link a fact with an English fact label with a text in Hindi or Telugu.
We refer to this problem as Multilingual Fact Linking (MFL) .
We deﬁne the task as predicting all the KG facts that are implied by a sentence even when the the sentence and KG fact labels are present in different languages.
To the best of our knowledge there has been no prior work on this important problem.
This challenging problem is made further difﬁcult due to language skew among the fact labels in a KG.
In Figure 1, we present the language distribution of fact labels (across ten languages) obtained from a subset of Wikidata.
We ﬁnd that the fact labels are heavily-skewed towards higherresources languages (ﬁrst ﬁve), compared to the remaining ﬁve Indian languages, viz.,
Hindi (HI), Telugu (TE), Tamil (TA), Urdu (UR), and Gujarati (GU).
These languages are spoken by hundreds of millions of speakers, although they are severely understudied in the NLP community.
We focus on the problem of MFL, with particular emphasis on Indian languages, in this paper.
To facilitate research on the problem of multilingual fact linking, we curate a new evaluation dataset, I NDIC L INK , containing parallel sentences in English and six Indian languages tagged with the corresponding Wikidata facts expressed in them.
The English sentences and facts are taken from the relation extraction dataset, WebRED [Orm´andi et al., 2021].
The test examples are manually translated into different languages and automatic translation of sentences are used for training.
We use KG facts from Wikidata and explore different strategies to use their fact labels in English and other languages, wherever available.
Since multilingual fact linking involves selecting a subset of facts from the entire set of KG facts, the label space may be in the scale of millions.
To handle this, we make use of Dual Encoder-Cross
Encoder paradigm, where a dual encoder [Reimers and Gurevych, 2019b] is used to quickly retrieve
[Table] [Table]
et al., 2020]
[Table]
[Orm´andi et al.,
[Table] [Table]
[Table] [Table]
et al.
[Table]
2018] { Table
[Table] [Table]
(ThisPaper)
[Table]
( tebal jura baasel-laindshaaft aur aaragau ke svis kaintan mein phaila hua hai )
[Table]
In summary the main contributions of this are as follows:
We introduce the task of Multilingual Fact Linking to link KG facts with their mentions in text, especially when there is a mismatch between the languages of the fact label and text.
We present I NDIC L INK , an evaluation dataset for MFL in English and six Indian languages, a widely used but less studied set of languages in the NLP community.
To the best of our knowledge, this is the ﬁrst dataset of its kind for these languages.
We propose R E
FC
O G, a novel Retrieval+Generation model for the task of MFL.
The proposed method signiﬁcantly outperforms standard Retrieval+Re-Ranking models.
2. Related Work
External sources of knowledge are helpful for NLP tasks such as question answering and fact veriﬁcation.
KILT [Petroni et al., 2020] uses a retrieval+seq2seq model with Wikipedia documents as the knowledge source to solve many knowledge-intensive tasks.
FAE [Verga et al., 2021] uses WikiData
KG to inject knowledge into pretrained language models in a modular fashion.
Colon-Hernandez et al.
[2021] surveys various methods of combining structured knowledge such as KGs and pretrained language models.
However, they don’t handle multilingual examples in both the text or KG.
Linking text to KGs has been traditionally explored in various settings, such as entity linking, relation classiﬁcation and extraction, fact extraction and fact linking.
We list various KG-related tasks and their corresponding inputs and outputs in Table 1.
Entity Linking :
Entity linking involves ﬁnding the KG entity referred by a mention marked in the input sentence.
Multilingual pretrained models have advanced Entity Linking across languages.
MEL [Botha et al., 2020] uses a Dual Encoder, Cross Encoder pipeline trained on hard negatives to achieve strong performance on 104 languages.
mGENRE [Cao et al., 2021] is an entity linking model that autoregressively generates the linked entity name using a Seq2Seq model.
Moreover, it augments the decoder with a preﬁx trie in order to generate only valid entity names.
We ﬁnd that combining both of these models by using a Dual Encoder, Constrained Generation pipeline leads to strong multilingual fact linking performance.
Relation Classiﬁcation : The task involves identifying the relation between a pair of entity mentions.
It is also referred to as relation extraction.
The recently released WebRED [Orm´andi et al., 2021] provides a dataset of English sentences annotated with the corresponding Wikidata relation/predicate and the linked entities.
We create I NDIC L INK using these examples.
Other multilingual relation classiﬁcation datasets such as RELX [Koksal and Ozgur, 2020] are unusable for fact alignment as they don’t provide the linked entities.
Fact Extraction : The task involves joint entity and relation extraction [Zhong and Chen, 2021, Sui et al., 2020] that focus on discovering new facts that are not present in the KG.
Whereas, fact linking deals with connecting existing KG facts with text.
Therefore, fact linking models make use of KG facts (which maybe in the scale of millions) whereas fact extraction systems do not.
Fact Linking :
Existing fact linking/alignment systems such as T-REx [Elsahar et al., 2018] align
English DBPedia abstracts with Wikidata triples and provide a corpus of 11 million high quality alignments.
They use adhoc pipelines of entity linking, coreference resolution and string matching based predicate linkers.
In our experiments, we ﬁnd that our end-to-end linker, R E
FC O G outperforms such pipeline systems.
In another line of work, multilingual fact retrieval [Jiang et al., 2020] is used to judge the factual knowledge captured within LM parameters by predicting masked entities in facts.
However, we are only concerned with retrieving the facts present in input text.
3. Multilingual Fact Linking: Problem Overview
A knowledge graph (KG) contains a list of entities E , relations R and facts F , where the i th fact links two entities ( e x , e y ∈ E ) with a relation ( r y ∈ R ) and is deﬁned as, F i =
( e x ; r y ; e z ) .
Multilingual KGs like Wikidata also contain textual labels of entities and relations in multiple languages.
Given the set L of languages in the KG, for each language l , we construct the fact label by concatenating the labels of the entities and relation, if available.
The label of F i in language l ∈ L ,
F li , exists if the entity and relation labels are available in language l , i.e., if e lx , r ly and e lz exist in
KG, then, F li =( e lx ; r ly ; e lz ).
For example, in Table 1, (Q12146; P17; Q39) and (Landschaft; country;
Switzerland) represent the fact and it’s English label.
Multilingual Fact Linking (MFL) aims to discover the set of linked KG facts, LinkedF acts ( T m ) ⊆
F that are explicitly expressed in text T of language m .
The output set can be formally deﬁned as ( ⇒ is used to represent the fact expressed in text):
[Equation]
In order to predict this set, the fact labels available in different languages are used.
In Section 4, we describe the I NDIC L INK dataset curated for the MFL task and in Section 5, we discuss the baselines and proposed models for the task.
[Table]
4. IndicLink: A New Dataset for Fact Linking in Indian Languages
For curating a fact linking dataset, we need a collection of sentences and the KG facts expressed in them.
We use subset of Wikidata facts as the oracle KG facts by considering all facts that exist between top 1 million Wikipedia entities.
Existing entity linking datasets only provide the mentioned entities, but we also need the relations between entities.
So we re-purpose a relation classiﬁcation dataset for fact linking by collecting the entity pair that express the particular relation.
We use WebRED [Orm´andi et al., 2021] for this purpose as it is the largest relation classiﬁcation dataset covering over 100 relations.
Multiple relations associated with a sentence are kept as separate examples in WebRED.
However, not all examples have associated Wikidata entities.
In some cases, the relation maybe expressed between entity mentions that are literals such as dates/numbers or refer to entities that don’t exist in Wikidata.
Therefore, for each sentence we collect facts from examples that mention a relation between valid Wikidata entities in the sentence.
We associate a special N ULL fact for sentences which have no expressed relations.
WebRED contains sentences only in English.
We extend it to multiple languages by translating the test sentences using professional translators.
To ensure high quality of translations, we uselayers of quality checks - initial translation, review and proof-reading.
To encourage research in Indian languages, which have historically lacked knowledge-linked resources, we consider six Indian languages – Hindi, Telugu, Tamil, Urdu, Gujarati and Assamese – for our multilingual fact linking dataset, I NDIC L INK .
Table 2 contains the number of test examples and KG facts considered.
For 6,429 sentences we end up with 11,293 facts implying an average of 1.7 linked facts per sentence.
We show some examples in Appendix A.
We explore automated techniques for getting training examples as it is expected that high quality language-speciﬁc training data will be unavailable for the vast majority of languages.
We translate 31K WebRED training sentences into the respective Indian languages using Google Translate.
The one exception is Assamese which does not have a translation system available.
5. R E FC O G: Proposed Method for MFL 5.1 Dual Encoder - Cross Encoder architecture
The Dual Encoder - Cross Encoder architecture is commonly used for tasks such as semantic search [Reimers and Gurevych, 2019a], and more recently for tasks like Entity Linking [Botha et al., 2020] that require classiﬁcation over a large target space.
It typically involves a pipeline of dual encoder based retrieval model followed by a cross encoder re-ranking model to get the most relevant targets for the given input text.
The retrieval model returns the top– k targets from the entire set.
Then the re-ranking model scores the top– k targets using a slower model that would have been intractable to apply on the complete set.
Apart from re-ranking cross encoders, we also experiment with a
Seq2Seq cross encoder.
We explain the dual encoder and cross encoders used below.
5.2 Fact-Text Dual Encoder
Dual Encoders are generally used for neural retrieval.
They encode the input and target text independently and use the cosine similarity between the embeddings to assess their relevance.
This strategy can be scaled to millions of KG facts as all the facts can be encoded beforehand, independent of the input text.
The input text and facts are encoded using the text encoder and fact encoder, respectively.
We build an approximate nearest neighbours index (using FAISS [Johnson et al., 2019]) of the fact embeddings which can be used to retrieve the top– k facts efﬁciently, for a given text embedding.
We initialize both the encoder parameters with LaBSE
[Feng et al., 2020] weights as it is pre-trained for cross-lingual text retrieval over 109 languages.
In section 7.3, we explore various choices for choosing the language of fact label, F Reti , that is used for retrieval.
5.3 Constrained Generation Cross Encoder
Seq2Seq models can be used as cross encoders that take as input the concatenated text and retrieved facts and uses a decoder to generate the most conﬁdent fact.
This allows the model to produce facts that are not returned by DE as well, overcoming the issue of error propagation in re-ranking systems whose performance is limited by the facts returned by the dual encoder.
To ensure that only valid
KG facts are generated, following mGENRE [Cao et al., 2021], we constrain the beam search of the decoder to follow a trie constructed from the English labels of KG facts (all facts considered have an
English label available).
At every step of decoding, the vocabulary is restricted to the words allowed by the trie to follow the preﬁx generated so far.
We note that the current formulation is optimized for generating a single best fact and we leave it to future work to extend it to produce a set of facts.
For now, we consider the top– b facts resulting from the beam search as the ﬁnal system output.
[Equation]
We initialize the Seq2Seq weights from the trained mGENRE model.
For fair comparison, we use the same initialization for the cross-encoders in the classiﬁcation models as well.
KG Lookup : We maintain a dictionary of the fact id and corresponding fact label (constructed from entity and relation labels).
We take the predicted fact label and look up this dictionary to get the corresponding fact id.
In case constraints are not applied, then decoder may generate invalid fact labels not present in the dictionary.
We refer to DE+CoGen model as Retrieval based Fact-Constrained Generation (R E FC O G).
A schematic of the model is shown in Figure 2.
5.4 Alternative Cross Encoders: Re-ranking
Since dual encoders encode the source and target independently, they fail to capture ﬁnegrained interactions between them.
In prior work [Botha et al., 2020], classiﬁcationbased cross encoders are often used to rescore the retrieved results.
The re-scoring is done by concatenating input text with each of the retrieved results, allowing for interattention between the input text and fact label.
We explore two classiﬁcation-based based cross encoder architectures (shown in Figure 3) for re-ranking the top– k facts returned by a retrieval model ( F T 1 , F T 2 ..F Tk ).
Independent Classiﬁcation : The input text T is concatenated with the textual descriptions of the retrieved facts, [ F Ti ] Rnk (the language chosen for ranking may be different from that used for retrieval, [ F Ti ] Ret ) and passed through the cross encoder.
An MLP operates on the pooled embedding to produce a score between 0 and 1 indicating it’s conﬁdence on whether the fact is expressed in the text or not.
The scoring function can be expressed as,
[Equation]
In this case, each fact is considered independently and does not make use of the dependencies that may exist among the facts.
Joint Classiﬁcation :
In order to score the facts jointly [Kolluru et al., 2021], the score of the i th fact, F Ti is computed by concatenating the text T along with all the fact labels.
The i th fact is specially marked with a beginning { and ending } , so that the model is aware of the fact that has to be scored.
This concatenated string is embedded using the cross encoder.
Similar to Independent
Classiﬁcation case, the MLP then scores the pooled embeddings.
Since the embeddings are pooled from all the facts, the model can capture the dependencies among facts as well.
[Equation]
6. Experimental Setting
Dataset : As described in Section 5, we use the auto-translated examples of WebRED (total 186K examples with 31K in EN, HI, TE, TA, UR, GU and 0 for AS) to train the model and the manually translated test examples of I NDIC L INK for evaluation.
We randomly choose 5% of the training examples to be used as validation set for early stopping, during model training.
Evaluation Metrics : We compare the facts predicted by the model with the gold set of facts and report the value of Precision@1 (P@1) and Recall@5 (R@5).
P@1 is the fraction of examples where the most conﬁdent fact is contained in the gold set.
3 R@5 is the fraction of gold facts that are present in the top-5 predicted facts.
We also compute macroP@1 , the macro-average counterpart to P@1, where the gold facts are divided into relation-speciﬁc classes, performance computed independently in each class and then averaged across all classes.
Implementation : We implement all models in Pytorch framework.
We use Sentence Transformers [Reimers and Gurevych, 2019b] library for training dual encoder models and use GENRE codebaseand fairseq [Ott et al., 2019] library for implementing the various cross encoders.
We train the dual encoder and generation models for 5 epochs each.
Total training time is 6 hrs on A100 GPU.
7. Experiments
We conduct experiments to address the following three questions:
• How well does R E FC O G, a retrieve+generation architecture, work for Multilingual Fact Linking, especially when compared to retrieve+reranking models for the task?
(Section 7.1) •
What is the effect of different components in generative decoding?
(Section 7.2) • How to effectively utilize multilingual fact labels during retrieval as well as generative decoding stages of R E FC O G?
(Section 7.3)
We
note that translation systems may
not
be available at
inference time for
certain languages.
For example, Assamese, one of the languages we consider in our experiments, is currently not supported by Google Translate.
Hence, in our experiments, we aim to evaluate multilingual fact linking by relying on the cross-lingual ability of the model, rather than test-time translation.
7.1 Effectiveness of R E FC O G
In Table 3, we compare the results of various models trained on I NDIC L INK .
All the cross encoder models use English fact labels and DE A LL -Sum for retrieval (which is explained in detail in
Section 7.3).
We notice that the DE models particularly struggle with retrieving N ULL fact, as its label (“None”) does not have any word overlap with the sentence.
Therefore, in I ND C LS -N and
J NT C LS -N, we deterministically add N ULL to the input, along with the remaining DE outputs.
This is not required for the R E FC O G model as it is free to generate facts even if they are not returned by the retrieval module.
R E FC O G model outperforms classiﬁcation based re-ranking by 10.7, 15.2 pts in P@1, R@5, respectively.
Within the re-ranking models, I ND C LS achieves better performance compared to J NT C LS , indicating that providing the other facts tend to confuse the model.
[Table]
J NT C LS ) classiﬁcation based re-ranking on top of DE A LL -Sum .
Ablations indicate the importance of DE and joint prediction of S, R and O for the R E FC
O G model.
Constraints reduce the P@1, R@5 metrics but ensure production of only valid facts.
Please see Section 7.1 and Section 7.2 for further details.
7.2 R E FC O G ablations
In Table 3, we consider four variants of the R E FC
O G model: (1) R E
FC
O G w/o Constraints: after removing the constraints on the decoder beam search, (2) R E
FC
O G w/o SRO links: predicting the subject (S), relation (R) and object (O) of the fact independent of one another, (3) R E FC
O G w/o DE: removing DE facts from Cross Encoder input, and (4) R E
FC
O G w/o DE, Constraints: removing DE facts and constraints.
Removing constraints leads to generation of incorrect facts in 865/6253 examples and reduces performance by 0.2 pts in P@1 and 2 pts R@5, respectively.
Instead of predicting the entire fact jointly, we perform an ablation in which we predict each of the components independently.
This leads to reduction in performance of 18.1 pts in P@1, showing the importance of jointly predicting all the components.
Unlike the re-ranking models, R E FC O G can generate facts even in the absence of Dual Encoder retrieved facts.
This allows us to evaluate the model performance without the Dual
Encoder.
The results indicate that DE facts are responsible for (4.9, 5.9) pts improvement in P@1,
R@5.
7.3 Effect of Multilingual Fact Labels
We consider fact labels in English (E L ), the language of input text T - Text Language (T L ), English and TL combined (ET L ), and all the languages in IndicLink (A LL ).
For the dual encoder, we either form a single embedding for the fact by concatenating the labels in various languages or embed the different language labels separately.
In either case, we only consider languages for which the fact has a label available.
The score for a fact D is computed as the cosine similarity between the text embedding and fact embedding.
In case multiple embeddings are associated with a fact, we aggregate their individual scores either through a sum/max operation.
The various types of fact labels and scoring operations can be summarized as follows:
• E L (or T L ): Using the English (or Text language) label of the fact.
[Table]
• ET L -Concat (or A LL -Concat): Using the concatenated English and Text language label of the fact (or concatenation of all language labels available).
• ET L -Max (or A LL -Max):
Embed the labels in each language in ET L (or A LL ) separately and consider the max of their cosine similarity as the score for the fact.
• ET L -Sum (or A LL -Sum):
Use the sum of scores of labels in each language in ET L -Sum (or A LL -Sum).
We ﬁnd that embedding the fact labels separately and adding their individual scores leads to the best performance across all languages in P@1, R@5.
[Table]
don’t
[Table]
similar improvements on concatenating the various language labels.
To study this further, we also compute the macroP@1 in Table 5.
The results on the Complete test set indicates that using language fact labels other than English does not help in P@1 but result in a modest increase of 3.7 pts in macroP@1 for ET LConcat.
This indicates that other language fact labels can help in facts that involve less-frequently occurring relations.
Also, language fact labels are not consistently available in all languages.
So we construct a modiﬁed test set that uses only KG facts where fact labels are available in all languages and the test examples that can be answered with these KG facts.
On this subset,
ET L -Concat shows an increase of 19.6 pts in macroP@1, compared to using only English fact labels.
This shows that robustly handling sparse availability of fact labels can improve performance.
A LL -Concat does not seem to improve performance over ET L -Concat which may be due to larger sequence lengths when all language labels are concatenated.
7.4 R E FC O G Error Analysis
We analyze the errors made by R E FC
O G
and classify it into the following three types:
• Rare relations : The model struggles with facts that contain rare relations which can be traced back to WebRED data where the top-10 relations are responsible for 80% of the examples.
• N ULL fact : The model wrongly predicts N ULL in 33% of the examples, demonstrating the difﬁculty in detecting absence of any facts.
• Issues with Gold : Few examples contain relations that are not explicitly implied by the sentence but require background world knowledge.
We show sample model predictions in Appendix A.
8. Conclusion
In this work, we introduce the task of multilingual fact linking and present a new evaluation dataset
I NDIC L INK containing examples in English and six Indian languages.
We explore various dual encoder, cross encoder architectures and ﬁnd that the proposed Retrieval+Generation model, R EFC O G, outperforms classiﬁcation-based reranking systems by 10.2 pts in P@1.
We also release the marked entity mentions for each linked fact, which also makes it usable for evaluating entity linking and relation classiﬁcation, forming the ﬁrst such resource for Indian languages.
We leave it to future work to further explore these directions.
Acknowledgements
We thank Vidhi Kapoor, Sumit Chabbra and the Google Localization Team for their timely help with manual translations.
We also thank Jan Botha for insightful discussions on the MEL system and Tom Kwiatkowski for helpful comments on an earlier version of the paper.
Appendix A. R E FC O G Qualitative Examples
For three examples from I NDIC L INK , we list the sentence in English, the corresponding translations in Indian languages and the gold facts (along with their English labels) annotated for the example.
For the sentence in each language, we also list the facts output by DE and the R E FC
O G beam search in the order of decreasing conﬁdence.
Example 1 :
Language : English
Sentence :
However, with both Hedda and Ingrid competing at the same Olympic Games, but in different sports, the sisters were historic in a Norwegian context.
Gold Facts : (Q262950: Hedda Berntsen; P27: country of citizenship; Q20: Norway) (Q262950:
Ingrid Berntsen; P27: country of citizenship; Q20: Norway)
DE Facts : (Anita Hegerland; country of citizenship; Norway) (Astrid S; country of citizenship; Norway)
(Helge Ingstad; country of citizenship; Norway) (Ane Stangeland Horpestad; member of sports team; Norway women’s national football team) (Icemaiden; country of citizenship; Norway)
R
[Table]
FC O G Beam Search :
N ULL
[Table]
country
[Table]
(Q20: Norway; P17: Country; Q20: Norway) (Q236632: Anna Carin Zidek; P1344: participant of; Q9672: 2006 Winter Olympics) (Q43247:
Ingrid Bergman; P27: country of citizenship; Q34: Norway)
Language : Hindi
Sentence : Please see Figure 4.
Gold Facts : (Q262950: Hedda Berntsen; P27: country of citizenship; Q20: Norway) (Q262950:
Ingrid Berntsen; P27: country of citizenship; Q20: Norway)
DE Facts : (Anita Hegerland; country of citizenship; Norway) (Ann Kristin Aarønes; member of sports team; Norway women’s national football team)
(Andrine Hegerberg; member of sports team; Norway women’s national football team) (2012 European Women’s Handball Championship; participating team)
(Norway women’s national handball team; sport; handball)
R E FC O G Beam Search : (Q270197: Ann Kristin Aarønes; P27: country of citizenship; Q20: Norway)
N ULL (Q264681:
Camilla Herrem; P27: country of citizenship; Q20: Norway) (Q236632: Anna Carin
Zidek; P1344: participant of; Q9672: 2006 Winter Olympics) (Q270197: Ann Kristin Aarønes;
P1344: participant of; Q8531: 1996 Summer Olympics)
Example 2 :
Language : English
Sentence : Microsoft Conﬁrms MKV File Support in Windows 10.
Gold Facts : (Q18168774: Windows 10; P3931: developer; Q2283: Microsoft)
DE Facts : (Microsoft Windows; has edition; Windows 10) (Windows 10; copyright holder; Microsoft) (Final Fantasy VIII; platform; Microsoft Windows)
(Windows 10; developer; Microsoft)
R E FC O G Beam Search : (Q18168774: Windows 10; P3931: copyright holder; Q2283: Microsoft)
N ULL (Q18168774:
Windows 10; P178: developer; Q2283: Microsoft) (Q1406: Microsoft Windows; P178: developer; Q2283: Microsoft)
(Q245006: Final Fantasy VIII ;
P400: platform ; Q1406: Microsoft Windows)
Language : Bangla
Sentence : Please see Figure 5.
Gold Facts : (Q18168774: Windows 10; P3931: developer; Q2283: Microsoft)
DE Facts : (Microsoft Windows; has edition; Windows 10) (Windows 10; developer; Microsoft) (Windows 10; copyright holder; Microsoft)
R E FC O G Beam Search : (Q18168774: Windows 10; P3931: copyright holder; Q2283: Microsoft) (Q18168774: Windows 10; P178: developer; Q2283: Microsoft)
N ULL (Q1406: Microsoft Windows; P178: developer; Q2283: Microsoft)
(Q2283: Microsoft; P1830: owner of; Q192527: Windows Media Player)
Example 3 :
Language : English
Sentence :
Wind Cave National Park, which is another area administered by the National Park Service, borders portions of the forest in the southeast.
Gold Facts : (Q1334313: Wind Cave National Park; P137: operator; Q308439: National Park Service)
DE Facts : (Wind Cave National Park; operator; National Park Service)
(Virgin Islands National Park; operator; National Park Service) (Valley Forge National Historical Park; operator; National Park Service) (Weir Farm National Historic Site; operator; National Park Service) (Hovenweep National Monument; operator; National Park Service)
R E FC O G Beam Search : (Q1334313: Wind Cave National Park; P137: operator; Q308439: National Park Service)
N ULL (Q308439: National Park Service; P355: subsidiary; Q3719:
National Register of Historic Places) (Q308439: National Park Service; P749: parent organization; Q608427: United States Department of the Interior) (Q1334313: Wind Cave National Park; P131: located in the administrative territorial entity; Q490716:
Custer County)
Language : Urdu
Sentence : Please see Figure 6.
Gold Facts : (Q1334313: Wind Cave National Park; P137: operator; Q308439: National Park Service)
DE Facts : (Joshua Tree National Park; operator; National Park Service)
(Cuyahoga Valley National Park; operator; National Park Service)
(Kicking Horse Pass; located in protected area; Yoho National Park) (Jewel Cave National Monument; operator; National Park Service)
(Gateway National Recreation Area; operator; National Park Service)
R E FC O G Beam Search :
N ULL (Q152820: Cuyahoga Valley National Park; P137: operator; Q308439: National Park Service)
(Q735202: Joshua Tree National Park; P137: operator; Q308439: National Park Service) (Q36600:
The Hague; P206: located in or next to body of water;Q1693: North Sea) (Q36600:
The Hague; P131: located in the administrative territorial entity; Q694: South Holland)