One-shot to Weakly-Supervised Relation Classiﬁcation using Language Models
Abstract
Relation classiﬁcation aims at detecting a particular relation type between two entities in text, whose methods mostly requires annotated data.
Data annotation is either a manual process for supervised learning, or automated, using knowledge bases for distant learning.
Unfortunately, both annotation methodologies are costly and time-consuming since they depend on intensive human labour for annotation or for knowledge base creation.
With recent evidence that language models capture some sort of relational facts as knowledge bases, one-shot relation classiﬁcation using language models has been proposed via matching a given instance against examples.
The only requirement is that each relation type is associated with an exemplar.
However, the matching approach often yields incorrect predictions.
In this work, we propose NoelA, an auto-encoder using a noisy channel, to improve the accuracy by learning from the matching predictions.
NoelA outperforms BERT matching and a bootstrapping baseline on TACRED and reWiki80.
1. Introduction
Relation classiﬁcation (RC) 1 detects the connection type between two entities in text such as place of birth between “Murat Kurnaz” and “Germany”, Figure 1.
It is crucial for downstream applications such as knowledge base construction [Ji and Grishman, 2011] and question answering
[Xu et al., 2016].
Most work in RC relies on either manually- or automatically-annotated data using knowledge bases (KBs)
[Zhang et al., 2017, Mintz et al., 2009, Riedel et al., 2010].
This dependency leads to diﬃculty in generalising such methods to novel domains where labelled data or knowledge bases are not available.
Recent work attempts to address the low resource scenario, namely few-shot relation extraction [Han et al., 2018, Baldini Soares et al., 2019], which requires few examples per relation type during testing.
However, they rely on a large available training and validation sets to learn the classiﬁcation task.
Perez et al.
[2021] suggests to name this setting, multi-distribution few-shot learning .
per:country_of_birth Obama was born in US .
org:founded_by
William Penn founded Pennsylvania.
per:spouse
Marie Curie is married to Pierre Curie. ...
P r e - t r a i ned
Language
M ode l
Relation examplars 2.
per:country_of_birth org:founded org:
city_of_headquarters 0.
per:country_of_birth Sentence
Sentence
Murat Kurnaz , a Turkish national who was born and grew up in Germany .
W ea k
S upe r
v
i s i on D a t a
R e l a t i on C l a ss i f i e r
In contrast, our work evaluates the extreme setting when no separate training and development sets are given.
Our setting considers only one example from each relation type, namely relation exemplar .
We employ the matching model in Baldini Soares et al.
[2019] to compute the similarities between an input sentence and those exemplars.
After gathering all the scores, we assign the relation with the highest score to the input sentence, such as country of birth in Figure 1.
The above LM-based one-shot relation predictions are often incorrect.
To improve the performance, we propose to learn from these predictions, which we name noisy annotations .
We propose NoelA (short for Noisy Channel Auto-encoder ), which employs two mechanisms to alleviate the negative impact of noisy labels.
Firstly, as entity types have been shown to be helpful for RC
[Hancock et al., 2018, Ma et al., 2019, Tran et al., 2020], NoelA reconstructs entity types of the two input entities so that the entity type bias is used when predicting relations.
Secondly, we use a noisy channel [Sukhbaatar et al., 2014, Goldberger and Ben-Reuven, 2016, Wang et al., 2019] to explicitly model the prediction noise.
We conducted experiments on two relation classiﬁcation datasets: TACRED [Zhang et al., 2017] and reWiki80, a modiﬁcation of Wiki80 [Han et al., 2019].
The two datasets have signiﬁcantly diﬀerent characteristics:
the relation type distribution (skewed and uniform), the number of relation types (41 and 80), and the domain (news and Wikipedia).
We show the performance of diﬀerent LMs as baselines in order to verify our LM choice.
We also demonstrate that bootstrapping [Reed et al., 2014], a traditional approach when supervision is scarce, is not eﬀective.
Meanwhile, NoelA outperforms the BERT matching by 9% accuracy on TACRED, and 6% on reWiki80.
We conduct an extensive analysis to understand the biases captured in pre-trained LMs and the contributions of each component in NoelA.
2. Background
Relation classiﬁcation (RC) is the task of assigning a (predeﬁned) relation type to a pair of entities in a sentence.
We denote R = { r 1 , r 2 , ..., r m } the set of m relation types.
Given a sentence s of n words s = ( w 1 , ..., w n ), two entities h, t (called head and tail entities respectively), and their corresponding semantic types e h , e t , the task is to identify the relation r ∈ R between the two entities.
For instance, in Figure 1, h and t are “Murat
Kurnaz” and “Germany”.
Their entity types are e h
= PERSON, e t = LOCATION.
The relation between them is r = country of birth .
Our work focuses on predeﬁned relation types excluding NA, i.e. “no relation”.
[Figure]
meanpooling
[Figure]
LM
BERT
Murat Kurnaz , ... was born ... in Germany ...
PER
LOC Linear & ReLU
Linear & Softmax
Linear & Softmax (PER, LOC)
Noisy Channel country_of_death
E n c o d e r
D e c o d e r country_of_birth
Linear& ReLU
[Figure]
A parametric probabilistic relation classiﬁer is given by p r ( r | s, h, t ; θ ) (with parameters θ ) which assigns a probability to a relation type r given a sentence s , head and tail entities h, t .
Recent RC approaches adopt the diagram s, h, t → encoder → classiﬁer → r , converting (cid:104) s, h, t (cid:105) to a ﬁxed-size vector representation before applying a classiﬁcation layer (often a pair of linear-softmax over the relation type set R )
True One-shot Setting
In this work, we assume the extreme scenario where only oneshot training set is available, i.e., a list of desired relation types and a single exemplar for each relation is given.
For example, for the relation country of birth , its exemplar is “[Obama] was born in [the USA]”.
Those exemplars are short and simple, often less than 10 words.
Following Perez et al.
[2021], we refer this scenario as true one-shot setting to diﬀerentiate with previous work [Han et al., 2018], in which a model has access to either separate training or development sets for tuning.
Since our work does not aim at creating a new dataset, we use the predeﬁned relation set of the evaluation datasets (TACRED and reWiki80) and manually create an exemplar for each relation type.
By creating an exemplar for each relation, we can simulate the process of imagining artiﬁcial examples when we are given a set of relation types at hand.
The relation types and corresponding exemplars used in our experiments are presented in Appendix C.
We note that our one-shot setting is very challenging even without detecting NA ( no relation ).
Temporally leaving the problem of predicting NA is in line with the few-shot relation classiﬁcation research.
In particular, the few-shot setting initialised by Han et al.
[2018] and further research following it [Baldini Soares et al., 2019] also ignore NA.
3. Matching Using Language Models
As shown in recent work, an LM trained on massive raw data (e.g., BERT) can capture some level of semantic similarity.
For instance, “[A] is the mother of [B]” is more similar to “[A] gave birth to [B]” than to “[A] works for [B]”.
Therefore, we can use matching similarity computed by the LM to assign similar scores between an unseen sentence and the exemplars.
Finally, the relation type assigned to the sentence is the one that has the highest score.
Let f be a function mapping sentence s and two entities h, t to a vector in R d .
We compute the matching score sim ( f ( s 1 , h 1 , t 1 ) , f ( s 2 , h 2 , t 2 )) , where sim is any function that computes the similarity between two vectors, such as dot product.
f produces the vector of (cid:104) s, h, t (cid:105) in two steps, depicted in Figure 2a.
We ﬁrst compute the entity representations of h, t (i.e., x h , x t ) by taking average of the words ( w ) that the mention contains, e.g., x h = avg w i ∈ h ( w i ).
Then, we concatenate the two entity representations to form the relation candidate representation x s,h,t =
[ x h ; x t ].
We note that our BERT matching, i.e. when
BERT is used, is similar to Baldini Soares et al.
[2019]’s BERT with mention pooling.
4. Noisy Channel Auto-encoder (NoelA)
Because the prediction by LMs above is inevitably noisy, we propose NoelA (depicted in
Figure 2b) to learn from noisy data.
4.1 Encoder
The encoder maps (cid:104) s, h, t (cid:105) to x s,h,t ∈ R d .
To compute a vector representation (e.g., x h ) for each entity mention (e.g., h ), we take mean pooling over the entity span from BERT.
We then concatenate the two vectors of h and t with their entity types’ embeddings ( x e h , x e t ∈ R d e ), and apply a linear and a ReLU layers:
[Equation]
4.2 Decoder
Diﬀerently from traditional decoders, our decoder does not completely reconstruct the input (cid:104) s, h, t (cid:105) .
It reconstructs the entity types e h , e t of h, t only, and predicts the relation r hidden in the input by computing p r ( r | s, h, t ).
Relation Classiﬁer
After having a vector representation of (cid:104) s, h, t (cid:105) , we apply a linear and softmax (over the relation type set R ) layers to compute p r ( r | s, h, t ) [Equation]
Noisy Channel
Although a large-scale pre-trained LM is assumed to passively memorise some relational facts, the annotation using such model results in relatively noisy labels.
We thus explicitly model the annotation noise by q ( r (cid:48) | r, s, h, t ), the probability of assigning s to r (cid:48) given the correct relation type r .
This probabilistic function is called “noisy channel”
[Goldberger and Ben-Reuven, 2016].
Not knowing the correct r , we marginalise over the relation type set R :
[Equation]
It is often assumed that r (cid:48) is independent from (cid:104) s, h, t (cid:105) given r , thus we base q ( r (cid:48) | r, s, h, t ) =
q ( r (cid:48) | r ) on a matrix C ∈ R | R | 2 :
[Equation]
where c ij is the entry of C at row i , column j .
Initialising q ( r (cid:48) | r ) has been shown crucial for learning p ( r | s, h, t )
[Goldberger and BenReuven, 2016].
We initialise C with a matrix computed by the confusion of choosing relation types by the used LM matching.
Formally, let count ( r (cid:48) , r ) be the number of times when r (cid:48) (cid:54) =
r appear together in the top- k candidate lists for all sentences, then
[Equation]
This initialisation provides to the learning an idea of how much the LM is confused r with r (cid:48) .
For instance, country of birth and country of death are likely confusing (because in the past an average person often died and was born in the same city/town).
On the other hand, country of birth and spouse are easy to distinguish one from the other.
In our experiments, we did not ﬁne-tune q ( r (cid:48) | r ) and chose k =
(cid:98)| R | / 4 (cid:99) .
Entity Type Reconstruction
Another way to tolerate the annotation noise is to inject into the model useful biases.
Our encoder uses the entity types of h, t to compute the vector representation x s,h,t , because entity types have been shown to be helpful for RC
[Hancock et al., 2018, Ma et al., 2019, Tran et al., 2020].
Intuitively, relation types are constrained by entity types, e.g., country of birth is constituted by an object such as person and a location .
However, if trained on noisy labels only, the model may not be able to make use of entity types to tolerate the noise.
Therefore, we force the model to capture the entity type bias by reconstructing the entity types of h, t .
Formally speaking, denoting E the entity type set (e.g., E = { PER , LOC , ORG , MISC } ), we compute the reconstruction probability using a linear and a softmax (over E × E ) layers:
[Equation]
where x ee = ReLU(Linear( x s,h,t ))
∈ R d ee .
4.3 Learning
Given D a noisy dataset, we train NoelA by minimising the following loss:
[Equation]
where L NC is the negative log-likelihood of predicting the noisy labels
[Equation]
L ETR is entity type reconstruction loss, which is the negative log-likelihood of predicting the entity types
[Equation]
L DR is the dispersion regularisation term proposed by Simon et al. [2019] and λ ∈ R is its coeﬃcient.
A motivation behind the use of this regulariser is that relation distributions of the noisy data are peaky (Figure 3). Learning from such peaky distributions may lead the model biased towards frequent predicted relations and result in predicting a subset of the relation types. To prevent this issue, we employ the dispersion regulariser that encourages the model to predict a diverse set of relations.
Following Simon et al. [2019], we set λ to 0 .
01 in all of our experiments.
5. Experiments
Our implementation was developed using the Transformers library [Wolf et al., 2019] and
PyTorch [Adam et al., 2017].
We use accuracy as evaluation metric.
5.1 Settings Datasets
We conducted experiments on two English datasets TACRED
[Zhang et al., 2017] and reWiki80 whose statistics are shown in Appendix A. TACRED is a widely used dataset for supervised relation extraction, in which we removed the no relation instances.
reWiki80 is a rearranged variant of the Wiki80 dataset used in Han et al.
[2019], originated from FewRel [Han et al., 2018].
Since the test set of Wiki80 is not provided, we used the development set for testing.
We took 20% of the training data as the development set for analysis.
4 We tagged the data sets using the Stanford NER tagger [Manning et al., 2014].
The two datasets are signiﬁcantly diﬀerent in several aspects.
reWiki80 has almost double the relation types than TACRED (80 vs 41).
TACRED’s relation distribution is skewed while reWiki80’s is uniform.
TACRED’s sentences are from news, and reWiki80’s are from Wikipedia.
For each dataset, we manually created a data-agnostic exemplar for each relation in which head and tail entities were randomly selected and mostly unseen in the two datasets.
Considering Wiki80, 51.72% entities in our examplars are seen in the dataset but onlypair of entities occur in the training dataset.
Regarding TACRED, 28.81% of entities are seen in the dataset but no pairs of entities in the exemplars occur in the training set.
We use BERT to generate weak labels on the two training sets, namely Noisy Data .
The labels originally given in the datasets are Gold Data .
Pre-trained LMs
We examined the base versions of three LMs: BERT [Devlin et al., 2019], GPT2 [Radford et al., 2019], and SpanBERT [Joshi et al., 2020].
The used BERT version is uncased while the used GPT2 and SpanBERT are cased.
Note that the BERT matching is similar to the BERT mention pooling model for one-shot RC proposed by
Baldini Soares et al.
Relation Classiﬁcation Settings
The hyper-parameters of our relation classiﬁers are given in Appendix A.
We used the Adam optimiser [Kingma and Ba, 2014] with a widely used learning rate of 3 .
10 − 4 , and early stopping based on the accuracy over the exemplars with a patience of 5.
We note that most of the hyperparameters were taken from previous work as the setting does not allow us to use the gold annotations of the development sets for ﬁne-tuning.
Besides, the performance on the exemplar set is not able to reﬂect the generalisability of a model.
Models in Comparison
We compare NoelA against several baselines.
The ﬁrst two are
Random (randomly assigning relation types) and Frequency (choosing the most frequent relation types).
The others are three LMs (including GPT2-small [Radford et al., 2019], SpanBERT-base [Joshi et al., 2020], BERT-base [Devlin et al., 2019]) and bootstrap-hard [Reed et al., 2014, a bootstrapping approach;]. The bootstrapping model used in our experiments is proposed by Reed et al.
[2014], namely Bootstrap-hard .
The idea is to consider the relation type predicted by the current classiﬁer for an instance at each training step as the noisy label.
In particular, we employ the encoder and relation classiﬁer from Section 4 (main text) with the loss L bootstrap − hard ( θ ).
The loss is a combination of L NC ( θ ) and L model ( θ ) (Eq. (8) and Eq. (11), respectively), the negative log-likelihood loss of the label predicted by the model with the current θ .
[Equation]
where β is set to 0 .
8 following Reed et al.
[2014] and L model is computed as follows.
[Equation]
We also tried their “soft” bootstrapping that minimises the entropy of the predicted label probability distribution H ( p ( .
| s, h, t ; θ )).
However, the entropy regulariser caused the model collapsed.
We thus did not include in our comparison.
5.2 Results
The results are shown in Table 1 (further details in Appendix B).
In general, the BERT matching yields substantially higher accuracy than the baselines and other two LM matching methods on both datasets.
It is worth noting that, the results of BERT in Table 1 are not comparable to those reported in Baldini Soares et al.
In particular, we take into account all relation types (41 in TACRED and 80 in reWiki), while they consider only 5 or 10 relation types for each testing example ( N way K shot setting, § 7).
NoelA substantially outperforms bootstrap-hard about 3-6%.
Bootstrap-hard only performs on par with BERTwET, our NoelA without the two learning-from-noisy-labels mechanisms and the dispersion regularisation.
This suggests that bootstrapping may not work on such noisy data where the seed set is too small, i.e., one example per category.
Removing the components one-by-one reduces the performance of our model.
The entity type reconstruction (–ETR) has less impact on the reWiki80 test set, since it only reduces a [Table] [Figure] marginal score (0.69%).
However, the contribution of the dispersion regulariser (–DR) is inconsistent on the two datasets.
6. Analysis 6.1 Relation Distribution
As the two datasets have diﬀerent relation distributions, we ﬁrstly look at them and those yielded by BERT matching (Figure 3).
In TACRED, the gold distribution is skewed towards a few relation types such as per:title , org:top members/employees .
BERT matching however is in favour of infrequent ones such as org:shareholders , per:charges .
In reWiki80, although the gold distribution is uniform, BERT matching’s distribution is multi-modal.
This observation shows inappropriate biases of BERT matching, suggesting that one can improve the annotation by injecting inductive bias to BERT matching to make the predicted relation [Figure] [Figure] [Figure]
[Figure] distribution close to the gold. Furthermore, the diﬀerence between the gold distribution and the one yielded by BERT matching by some means explains the under-performance of the bootstrapping baseline as bootstrapping often suﬀers from semantic drift.
6.2 Accuracy of BERT Matching
We show the accuracy of BERT matching according to relation types in Figure 4.
On
TACRED, BERT matching performs exceptionally well for per:charges , per:age , but poorly for org:dissolved , org:subsidiaries .
Although Figure 4a gives an intuition that the overall accuracy should be substantially higher than 15.46%, it is not the case because most frequent types have low accuracy.
This observation again suggests the need for biasing BERT matching towards frequent relation types.
On reWiki80, the highest accuracy is for mountain range , and low is for part of , subsidiary , operating system .
Because the gold relation distribution is uniform rather than skewed, the overall accuracy of BERT matching on reWiki80 (27.48%) is substantially higher than that on TACRED (15.46%). BERT matching’s confusion matrices are illustrated in Figure 5. The diagonal line is clearly shown for reWiki80 while it is lighter for TACRED. This explains the matching
[Figure]
per
[Figure] of_ [Figure] per:
[Figure]
contains_ administrative_ territorial_ entity licensed_to_broadcast_to successful _candidate league main_subj religion
[Figure]
reWiki performance on TACRED is low, as BERT gets confused by other relations.
Generally on both datasets, we observe that BERT matching performs poorly for human-human relations such as parent, mother, sibling, and spouse.
The reason is that these relations usually occur in the context of each other.
BERT also performs worse on human-location relations such as residence are citizenship that are often hold between the same pair of entities.
Meanwhile, place of birth/death are typically the same place in the past, causing the confusion between the two relations.
6.3 Accuracy of NoelA versus BERT Matching
Next, we show the accuracy diﬀerence of NoelA in comparison with BERT matching in Figure 6.
The improvement of NoelA over BERT matching is mediocre on TACRED but visible on reWiki80.
Nevertheless, the overall accuracy gain of NoelA on TACRED (9.33%) is substantial.
This is due to the skewness of the gold relation distribution of
TACRED (Figure 3a): a slight improvement for highly frequent relation types would lead to a substantial overall accuracy improvement.
The observation here indicates an interesting behaviour of NoelA: it seems to adjust its attention according to the hidden gold relation distribution.
On TACRED, NoelA trades oﬀ the accuracy loss for some infrequent relation types against the accuracy gain for some frequent ones.
On reWiki80, NoelA, on the other hand, pays attention to most relation types since all of them are equally frequent.
6.4 Impact of Entity Type Reconstruction
We examine to what extend entity types can help to predict gold relations.
To do so, we measure the mutual information between entity type pairs (ET) and gold relations (R), and between gold relations (R) and gold relations (R).
In information theory, given two random variables X, Y , the mutual information I ( X, Y ) measures the amount of information in X that tells us about Y and vice versa.
Therefore, the more helpful to predict relations (R) entity types (ET) are, the larger I ( ET, R ) is.
Because the maximum value of I ( ET, R ) is
I ( R, R ), we propose the below normalisation
[Equation]
If entity types are not related to gold relations, I ( ET ; R ) = 0; thus ˆ I ( ET ; R ) = 0.
Otherwise, if gold relations are determined by entity types, I ( ET ; R )
=
I ( R ; R ), leading to ˆ
I
ET ; R )
= 1.
For TACRED ˆ I ( ET ; R ) = 0 .
81 whereas ˆ I ( ET ; R )
= 0 .
33 for reWiki80.
This explains why the impact of the entity type construction on reWiki80 is substantially smaller than on
TACRED.
7. Related Work Probing pre-trained LMs
Prior work suggests that pretrained LMs capture factual information that they have seen during pretraining.
A few studies have been introduced to extract such factual knowledge from LMs including relations between entities [Petroni et al., 2019, Jiang et al., 2020].
These studies deﬁne templates for each relation type, which are used in combination with a given subject to predict the corresponding object.
While Petroni et al.
[2019] manually deﬁne such templates, following studies propose few techniques for automating prompt generation and selection
[Jiang et al., 2020, Shin et al., 2020, Gao et al., 2021].
A subsequent line of research is to exploit pre-trained LMs for weak supervision, such as Schick and Sch¨utze [2020] 7 and our two baselines (bootstrap-hard and BERTwET, see
Table 1).
Our full work goes beyond that by employing learning-from-noisy-label techniques, resulting in substantial improvements.
Few-shot relation extraction
Few-shot relation extraction have been introduced ﬁrstly by Han et al.
[2018], following the N way K shot setting.
In particular, N unseen relation types and their corresponding K examples are provided during evaluation.
This setting assumes access to data from other distributions during training, which allows a model to learn the target task.
Moreover, N does not cover the full set of relation types but a subset of it, e.g., N usually equals to 5 or 10.
Another setting, requiring less supervision, is the BERT-based one-shot RC, proposed by Baldini Soares et al.
They evaluate
BERT-based relation matching model in N way one-shot setting without using any training data.
Their model, however, uses the development set for tuning the hyperparameters.
Diﬀerent from the both settings, our few-shot is true in the sense proposed by Perez et al.
[2021]: neither training sets or development sets exist.
8. Conclusion and Future Work
We demonstrated one-shot relation classiﬁcation using LMs by generating noisy relation data.
To reduce the impact of noisy labels, we proposed NoelA (Noisy Channel Auto-encoder) which can learn the latent correct labels by explicitly modeling noise and using entity type bias.
NoelA gains a promising 6 and 9% accuracy over BERT on reWiki80 and TACRED, respectively, demonstrating the potential of using weak supervision from LMs.
Interestingly, we observed from the analysis of NoelA’s accuracy that our model can adjust towards the latent gold relation distribution.
We note that no relation is important for relation classiﬁcation, hence leave it for future work.
We also believe that our NoelA is applicable to few-shot because the noisy channel idea is independent on number of examples per relation.
In theory, the more examples we have, the better we can denoise the annotation.
Appendix A. Experimental Settings
[Table]
Table 2 shows the statistics of TACRED and reWiki80 datasets.
We used exemplars as the development set and stopped the training process based on the accuracy on it.
For every model, we conducted ﬁve runs with diﬀerent initialised parameters and computed the average performance.
We list the hyper-parameters of NoelA and bootstrap-hard in Table 3.
We note that a small number of instances were eliminated when training the models due to max length constraint.
The numbers of removed instances in TACRED [Zhang et al., 2017] train/dev/test sets are as follows: 148, 47 and 20 instances, respectively.
There is no instance beyond the restricted length in reWiki80 [Han et al., 2019].
Additionally, regarding entity type embeddings, we distinguish the entity types of subject and object, e.g., PERSION-SUBJ and PERSION-OBJ.
All experiments were performed on a compute node which has an Intel Skylake CPU and an NVIDIA V100 GPU (16GB GPU RAM).
Appendix B. Detailed Results
We also report average accuracy over ﬁve runs on TACRED and reWiki80 development and test sets in Table 4 with the average training time (Avg.
Runtime; minutes).
We note that we do not use the development sets during training or for early stopping, we only use them for analysis.
[Table]
[Table]
reWiki80 development and test sets.
[-0 2 4 6 8 10121416182022242628303234363840BERT Annotated Relation 04 10141822263034 
-]
[-G o l d R e l a t i on 0. [Figure] 03691215182124273033363942454851545760636669727578 BERT Annotated Relation 
 G o l d R e l a t i on 0.-]We also report the average training time (Avg.
Runtime; minutes).
Appendix C. Exemplars
We present all the exemplars used for TACRED and reWiki80 in Table 5 and Table 6, respectively.
All exemplars are manually created by one author and partially revised by another author.
[Table]
The World Health Organization ( WHO ) is a specialized agency of the United Nations responsible for international public health .
[Table]
:city
[Table]
Facebook ’s headquarter is located in Menlo Park, California,
United States .
[Table]
org:country
[Table]
Facebook ’s headquarter is located in Menlo Park, California,
United States .
[Table]
President Truman dissolved the O.S.S. in 1945 .
[Table]
Facebook was founded in 2004 .
[Table]
Facebook was founded by Mark Zuckerberg .
[Table]
Germany is a founding member of the European Union .
[Table]
Germany is a founding member of the European Union .
[Table]
IBM total number of employees in 2019 was 383800 .
[Table]
Alphabet is the parent of Google .
[Table]
Tearfund is an international Christian relief and development agency .
[Table]
The largest shareholder of Google is Larry Page .
[Table]
of headquarters
Facebook ’s
headquarter
is located in Menlo Park, California ,
United States .
[Table]
Cafe Nero is a child organization of Rome Bidco .
[Table]
Tedros Adhanom is the WHO current director .
[Table]
gov.uk is a United Kingdom public sector information website .
[Table]
Peter Higgs is now at
the age of
[Table]
Mary I of England was also known as bloody Mary .
[Table]
Richard Feynman died of abdominal cancer .
[Table]
Jeﬀrey Dahmer was convicted of 15 murders .
[Table]
Michael Douglas is a child of Kirk Douglas .
[Table]
Richard Feynman lived in New York .
[Table]
Obama was born in Honolulu , Hawaii .
[Table]
Richard Feynman died in Los Angeles , California , US .
[Table]
countries
[Table]
Richard Feynman
[Table]
Obama
[Table]
Richard Feynman died in Los Angeles , California
[Table]
Obama was
[Table]
Richard Feynman died in
[Table]
Kayleigh McEnany is
[Table]
White House press secretary
[Table]
Barack Obama
[Table]
politician
[Table]
Craig Robinson
[Table]
Fred Trump is Donald Trump
[Table]
father
[Table]
Maximilian Kolbe
[Table]
Peter Higgs was awarded a PhD degree from King ’s College
London .
[Table]
Alexander Watson is the brother of Emma Watson
[Table]
Marie Curie is married to Pierre Curie .
[Table]
Obama was born in Honolulu, Hawaii
[Table]
Richard Feynman died in Los Angeles , California , U.
[Table]
Barack Obama lives in Washington
[Table]
Barack Obama was the 44th president of the United States .
[Table]