DP-KB: Data Programming with Knowledge Bases Improves Transformer Fine Tuning for Answer Sentence Selection
Abstract
While transformers demonstrate impressive performance on many knowledge intensive (KI) tasks, their ability to serve as implicit knowledge bases (KBs) remains limited, as shown on several slot-ﬁlling, question-answering (QA), fact veriﬁcation, and entity-linking tasks.
In this paper, we implement an efﬁcient, data-programming technique that enriches training data with KB-derived context and improves transformer utilization of encoded knowledge when ﬁne-tuning for a particular QA task, namely answer sentence selection (AS2).
Our method outperforms state of the art transformer approach on WikiQA and TrecQA, two widely studied AS2 benchmarks, increasing by 2.0% p@1, 1.3% MAP, 1.1%
MRR, and 4.4% p@1, 0.9% MAP, 2.4% MRR, respectively.
To demonstrate our improvements in an industry setting, we additionally evaluate our approach on a proprietary dataset of Alexa QA pairs, and show increase of 2.3% F1 and 2.0%
MAP.
We additionally ﬁnd that these improvements remain even when KB context is omitted at inference time, allowing for the use of our models within existing transformer workﬂows without additional latency or deployment costs.
Introduction
Transformers are powerful sequence-to-sequence models that leverage the mechanism of self-attention [28] to capture long-range dependencies, such as relationships between words in a natural language text.
Compared to sequential models such as recurrent or convolutional networks, they make efﬁcient use of available processing power.
Transformer based models such as BERT, XLNet and BART [4, 33, 18, 17] hold state of the art on many natural language processing (NLP) tasks, including next sentence prediction, natural language generation, and natural language inference [23, 17, 10].
In addition to efﬁciently encoding linguistic information from unlabelled text, their top performance on knowledge-intensive (KI) NLP tasks [21], such as question-answering [24] have led to the hypothesis that transformers also encode relational knowledge, and as such serve as parameterized, implicit knowledge bases (KBs) [22].
However, it has also been shown that transformer knowledge acquisition [22, 25] and subsequent utilization [27, 12] can be uncontrollable, highly context dependent, and tightly coupled to language acquisition.
These limitations may impact performance on downstream tasks, including KI tasks
[Table]
Q: How old is Elton
[Table]
husband
[Table]
Correct
[Table] [Table]
[Table]
Q: How many humps on a Camel?
Correct:
The two surviving species of camel are the dromedary, or one-humped camel, which is native to the Middle East and the Horn of Africa; and the Bactrian, or two-humped camel, which inhabits Central Asia.
Selected:
A camel is an even-toed ungulate within the genus Camelus, bearing distinctive fatty deposits known as "humps" on its back.
[Table]
What some legal uses of meth?
Correct: Although rarely prescribed, methamphetamine hydrochloride is approved by the U.S. Food and Drug Administration (FDA) for the treatment of attention deﬁcit hyperactivity disorder and obesity under the trade name Desoxyn.
Selected: Methamphetamine, also known as metamfetamine, meth, ice, crystal, glass, tik, N-methylamphetamine, methylamphetamine, and desoxyephedrine, is a psychostimulant of the phenethylamine and amphetamine class of psychoactive drugs.
like answer sentence selection (AS2) [31].
Table 1 is illustrative of limitations of some of these deﬁciencies of transformers in precisely leveraging encoded information.
Transformer models that show top performance [7, 15] on widely studied benchmarks [32, 31] still fail to classify many QA pairs correctly.
In Example 1 , the model is unable to leverage knowledge of the identity between Elton
John’s husband and David Furnish .
In Example 2 , one-humped or two-humped are not recognizable as quantities pertaining to the uncommonly quantity humps .
Example 3 shows the difﬁculty in reasoning for the a rare prescriptive use of the illicit drug methamphetamine .
These examples also illustrate relevance of this task as a means to assess impact of deﬁciencies in transformer knowledge utilization.
In this paper study, we propose an efﬁcient, data-programming approach utilizing a KB that improves performance on answer selection tasks and demonstrate that some of these limitations can be mitigated during ﬁne-tuning with simple data augmentation technique.
A number of recent studies have also studied approaches that aim to improve transformer performance on KI tasks, proposing the use of differentiable knowledge retrievers [8, 13, 16], retrieval-augmented generation (RAG) [13], KB embeddings such as KnowBERT [20] and ERNIE [34], and pre-training on verbalized KBs such as KELM [1].
While these approaches offer promising beneﬁts for transformer knowledge encoding and retrieval, to our knowledge, none of them have been shown to outperform existing state of the art for answer selection, a task that is essential to several question answering services provided by commercial voice assistants.
Additionally, each of these approaches is signiﬁcantly complex and require signiﬁcant work to leverage in production applications.
Our approach, on the other hand, leverages ElasticSearch to tag KB entries in input QA pairs, derives weaksupervision signals from tagged KB entries, and incorporates this context only during ﬁne-tuning.
We show that our simple, efﬁcient and data-programming method confers signiﬁcant performance beneﬁts over the state of the art for answer sentence selection, even when KB context is omitted at inference time.
The main contributions of our work are:
• We show that several limitations in the use of transformers implicit KBs can be overcome using a simple data-programming approach by outperforming state-of-the-art models on several QA tasks: 1.
increasing by 2.0% p@1, 1.3% MAP, 1.1% MRR and 4.4% p@1, 0.9% MAP, 2.4% MRR on WikiQA and TrecQA respectively, two widely used AS2 benchmarks.
increasing by 2.3% F1 and 2.0% MAP on AlexaQA pairs, a proprietary commercial answer classiﬁcation benchmark.
• We show that KB is not needed at inference time, allowing our
trained models to be used as drop-in replacements for existing transformer-based AS2 systems.
Background 2.1 Transformers
The transformer [28] is an architecture for efﬁciently transforming one sequence into another via self-attention , a mechanism that differentially weighs the signiﬁcance of discrete tokens in an input sequence.
Compared to sequentially aligned or convolutional networks such as RNNs and
CNNs, transformer models have proven to be extremely effective at efﬁciently capturing long-range dependencies between words in natural language [4, 18, 17], including some structured knowledge such as the husband of relation between named entities [22].
Pre-training transformer models on large, unstructured corpora of unlabelled text [3, 11] allows them to capture linguistic and factual knowledge prior to subsequent ﬁne-tuning on downstream tasks.
The suitability of transformer based models such as BERT [4] for this type of transfer learning dramatically increases their reusability, driving state of the art results on many tasks [7, 15, 8, 33, 9] in addition to widespread adoption in industry.
2.2 Transformer Limitations as Knowledge Bases
While transformers have demonstrated strong performance on question-answering [25] and ﬁll-inthe-blank cloze tasks [22, 12] without access to external information, the modulation of transformer knowledge acquisition and utilization is limited.
Cloze task [22] and question answering [25] probes demonstrate transformer knowledge acquisition is largely uncontrollable and often only results in the acquisition of frequently observed information.
Further, transformer recall of factual knowledge on cloze tasks remains tightly bound to learned linguistic representation [12].
In a systematic study [27] on multiple tasks, it was shown that transformers lack robust multi-hop reasoning faculties, are insensitive to adverbial modiﬁes like "always", "some", and "never", and are unable to robustly compare quantities.
For example, while RoBERTa [18] appears able to effectively compare numbers, it is unable to compare when values are given in ages .
Other studies additionally have shown insensitivity to negation [5], difﬁculty with misspellings and short, simple sequences [26], and sensitivity to sequence length, punctuation, and subject-verb agreement [2].
2.3 Answer Sentence Selection (AS2) and Answer Classiﬁcation
Answer sentence selection (AS2) consists of ranking answer candidates given a question and one or more answer candidates, while binary answer classiﬁcation consists of classifying answer candidates as correct or incorrect given the same input.
Both tasks encourage models that leverage encoded knowledge to select the most correct answer and thus may be used to probe model knowledge and reasoning capabilities.
Let q be a question, C q = { c 1 , . .
c n } be a set of answer sentence candidates for q , we deﬁne R as a ranking function, which orders the candidates in C q according to a score, p ( q, c i ) , indicating the probability of c i to be a correct answer for q .
Answer sentence selection is performed by taking the highest scoring candidate in C q , while binary answer classiﬁcation is performed by assigning the label of the highest probability class as determined by the ranking function
R .
Widely used metrics for AS2 performance are mean average precision (MAP) and mean reciprocal rank (MRR), while mean average precision (MAP) and F-score (F1) are commonly used for binary answer classiﬁcation.
To our knowledge, transformer models [7, 15] demonstrate a strong state of the art on the AS2 task.
Modeling 3.1 Datasets
We study popular Answer Sentence Selection datasets to evaluate the beneﬁts of our approach for this task:
Answer Sentence Natural Questions (ASNQ)
[7], WikiQA [32], and TrecQA [30].
ASNQ is a large scale QA dataset derived from Google’s Natural Questions [14] dataset, with more than ∼ 84K unique questions.
The train split of this dataset is used to transfer a pre-trained transformer model to the AS2 task.
WikiQA [32] and TrecQA [30] are widely studied benchmark datasets for Answer
Sentence Selection with over ∼ 1.1K and ∼ 1.2K unique questions respectively.
We utilize the clean versions of both WikiQA and TrecQA, as well as the TRAIN-ALL split of TrecQA for ﬁne-tuning.
All of these datasets are available under dataset speciﬁc licenses that permit their use and distribution for academic purposes.
We additionally evaluate the beneﬁts of our approach in an industry setting using the binary answer classiﬁcation task using AlexaQA.
AlexaQA is a proprietary benchmark dataset that contains ∼ 107K unique questions obtained from de-identiﬁed samples of Amazon Alexa QA trafﬁc with correct/incorrect labels assigned by expert annotators.
Detailed statistics of each dataset by split are shown in
Table 6 in Appendix A.
3.2 Dataset Preprocessing
We implement a novel data enrichment pipeline that use an ElasticSearch index of 20.7M item and relation labels obtained by popularity-based ﬁltering of Alexa’s KB.
Our pipeline tags KB entries in input text by aggregating the results of three queries on our index.
For each word w in the set of words W = { w 1 , . . . , w n } in the input text, we tag w
i
as a KB entry
if:
• w i is an exact match for a label in the index • w i is contained by a label in the index • w i and w i + 1 is a quorum match for a label in the index
Consecutive labels matching the same entry are assumed joined together, matches are sorted for relevance, and the top result is selected as the KB entry.
KB meta-data for entries is derived from selected KB properties, such as the collection property that indicates classiﬁcations such as celebrity
, book ,
or album .
3.3 Incorporating KB-derived Context for Transformer Training
Metadata for each entry tagged by the preprocessing pipeline (Section 3.2) is resolved to a textual representation using corresponding KB labels.
An example of the JSON produced from this resolution is shown below:
{ "text": "David Furnish is 57 years old.", "kb_tags": [{ "kb_id": "e-478772", "popularity": 0.981, "candidate_title": "David Furnish", "candidate_aliases": "
[Table] [Table]
[Equation]
married
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Inspired by other studies [19, 1] that verbalize structured data for use in language models, we insert the textual representation of KB context directly into model input.
This approach may distract the model from attending to the QA pair itself if too much context is added and we thus employ two strategies to prevent this.
First, we limit metadata to the collection property, whose values include common categories such as "celebrity", "quantity", and "generic drug form".
The collection property in our KB has many analogous properties in other KBs, for example, the instance of relation in Wikidata [6].
Second, we employ a ﬁlter that constrains the number of entries from which KB context is added.
The intersection ﬁlter exploits the intuitive hypothesis that correct QA pairs will contain the same
KB entries, adding context only if the same entry is tagged in both the question and the answer.
For example, this ﬁlter adds context for entry David Furnish from the QA pair: Q: how old is Elton John’s husband;
A: David Furnish is 57 years old because the question contains Elton John’s husband , an alias for "David Furnish" in our KB, and the answer contains David Furnish .
The intersection ﬁlter excluded context for entries like 57 and husband , even though entries for both exist in our
KB.
We additionally study the 1-best ﬁlter, which selects the KB entry from the answer with the highest popularity in our KB as a more lenient alternative.
Two strategies of concatenating context to question/answer text are also explored: append and prepend ; in both cases, the model’s special separator token 2 is used to separate the context from question/answer text.
An example of the resulting sequences are shown below:
• Append : how old is elton john’s husband <\s> john furnish is 57 years old.
he was born on october 25, 1962 <\s> celebrity <\s> celebrity • Prepend : <\s> celebrity <\s> celebrity <\s>
how old is elton john’s husband <\s> john furnish is 57 years old.
he was born on october 25,
3.4 Model Architecture
Our approach builds upon the Transfer-and-Adapt (TANDA)
[7] architecture, the state of the art approach for answer sentence selection, by leveraging KB-derived context to address deﬁciencies observed in transformer knowledge utilization for this task.
As illustrated in ﬁgure Figure 1, we transfer a pre-trained RoBERTa-base model [18] to the answer sentence selection task by ﬁne-tuning on ASNQ and adapt the transferred model via further ﬁne-tuning on our target dataset, either WikiQA,
TrecQA, or AlexaQA.
Training incorporates KB-derived context in both transfer and adapt steps, as discussed below.
During inference, we optionally remove KB context so as to evaluate our approach as a drop-in replacement for existing transformer-based AS2 systems.
In order to isolate the beneﬁts of our approach, we reuse the same optimizer, hyper-parameters, and early stopping strategy described in [7] and only alter the sequence length, increasing from 128 to 256 to accomodate additional context.
Experiments on ASNQ, WikiQA, and TrecQA use AWS
EC2 p3dn.24xlarge hosts, and those on AlexaQA use AWS SageMaker ml.p3.16xlarge notebook instances.
Results
Performance of KB augmented transformer models for standard ﬁne-tuning (FT) on ASNQ is shown in Table 2.
Transfer-and-Adapt performance with KB augmentation is reported for WikiQA, TrecQA, and AlexaQA in Tables 3, 4 and 5 respectively.
We indicate the datasets used in Transfer-and-Adapt setting using two arguments, transfer dataset → adapt dataset with numerics in parentheses indicate training epochs.
Baseline models - i.e. the RoBERTa base TANDA state-of-the-art set by [7] - are indicated by * and lack the -KB sufﬁx.
We additionally evaluate a setting in which KB context is omitted at inference time to explore the ability of our approach to modulate transformer knowledge utilization.
Results for this setting are reported for each dataset and are indicated by the value of the Incl.
KB at Inference column.
The results in the tables below demonstrate that:
• KB context improves ﬁne-tuning performance on ASNQ, increasing the p@1, MRR and MAP by 2.9%, 3.0%, and 2.9% after 9 epochs.
• Training with KB context improves on the strong performance set by the state of the art TANDA approach on widely studied benchmarks, increasing the p@1, MRR and MAP by 2%, 1.3%, and 1.1% and 4.4%, 0.9%, and 2.4% on WikiQA and TrecQA respectively.
• The beneﬁts of KB context generalize to our industry setting, increasing the F1 and MAP by 2.3% and 2.0% over the TANDA state of the art, RoBERTa ASNQ(9)
→ AlexaQA(1), and by .4% and 2.3% over the more challenging baseline, RoBERTa ASNQ(1)
→ AlexaQA(1).
• Models trained with our approach continue to outperform the TANDA state of the art even when KB context is omitted at inference time; in other words, the beneﬁts of KB context are primarily realized during model training.
[Table]
RoBERTa
[Table]
RoBERTa FT
[Table]
KB
[Table] [Table]
RoBERTa FT ASNQ-KB(9)
[Table]
Intersection
[Table]
RoBERTa FT ASNQ-KB(9)
[Table]
RoBERTa FT ASNQ-KB(9)
[Table] [Table]
KB
[Table]
) →
[Table] [Table]
RoBERTa ASNQ-KB(9)
→ WikiQA-KB(
[Table] [Table]
RoBERTa ASNQ-KB(9)
→ WikiQA-KB(9)
[Table]
RoBERTa ASNQ-KB(9)
→ WikiQA-KB
[Table] [Table]
-KB
[Table]
) →
[Table]
KB
[Table] [Table]
RoBERTa ASNQ-KB(9)
→ WikiQA-KB(9)
[Table] [Table]
→
[Table]
RoBERTa
[Table]
KB
[Table]
) →
[Table]
KB
[Table] [Table]
RoBERTa ASNQ-KB(9)
→ TrecQA-KB
[Table] [Table]
RoBERTa ASNQ-KB(9)
→ TrecQA-KB(9)
[Table]
RoBERTa ASNQ-KB(9)
→ TrecQA-KB(9)
[Table]
RoBERTa ASNQ-KB(9)
→ TrecQA-KB(9)
[Table]
RoBERTa ASNQ-KB(9)
→ TrecQA-KB
[Table]
[Table] [Table]
F
[Table]
Discussion 5.1 Comparing Context Generation Strategies
Results reported in Tables 2, 3, 4 and 5 all demonstrate that our approach outperforms the state of the art approach, even in the more challenging setting where KB context is omitted at inference time.
We explain the robustness of our models to the omission of KB context in light of the proportion of each dataset that our approach impacts.
The intersection ﬁlter adds KB context to only 3.38% of the ASNQ dataset, 5.27% of TrecQA, and 8.33% of WikiQA while the 1 best ﬁlter adds context for 31.17% for
ASNQ, 51.1% for TrecQA, 40.79% for WikiQA.
We hypothesize that the large number of training examples seen without context allows the model to leverage context as a for weak supervision that encourages knowledge utilization and elaborate further in subsection 5.2 below.
These results show that the more intuitive intersection ﬁlter performs better than the 1 best ﬁlter for both concatenation strategies, despite impacting between signiﬁcantly less of each dataset.
We conclude that the explicit conceptual alignment provided by the intersection conveys additional beneﬁts beyond the addition of conceptual keywords provided by the 1 best ﬁlter.
The prepend strategy outperforms the append strategy on all datasets other than TrecQA, a deviation that we attribute to the small size of the TrecQA test set.
We explicate these ﬁndings in light of the positional invariance the prepend strategy - that is, prepend always adds context in the same position in the sequence, whereas append does not.
As a result, prepend models appear better able to attend to context and outperform their append counterparts, even though prepend models suffer more when context is omitted at inference.
5.2 Impact of KB Context
We leverage the three illustrative examples presented in Table 1 to probe the impact of our KB context and its potential to address the previously studied [12, 27] deﬁciencies of transformers as implicit
KBs.
Models trained with our approach classify each of these examples correctly, even when KB is omitted at inference, indicating that they may be able to exploit our context to reﬁne their utilization of encoded knowledge.
In order to identify the mechanism behind these beneﬁts, we compare the attention of TANDA with that of our best model, prepend, intersection , using box plots of attention intensity and bar plots of activate head counts per layer in Appendix B.
Example 1 requires the model to leverage encoded knowledge in order to make the connection between "husband" and "David Furnish" necessary to recognize that the phrase "is 57 years old" answers the question phrase "how old" .
Figure 2 presents model attention weights between tokens "how" and "57" and between "husband" and "David" , where it can be seen that our approach signiﬁcantly improves both the quantity of heads attending to these keywords and the intensity of this attention.
It is likely that model pre-training has encoded this knowledge, given that the second sentence on David Furnish’s Wikipedia page reads: "He is married to English musician Sir Elton
John" .
Unsurprisingly, changing the question or the answer text to remove this relation - to either "How old is David Furnish" or "Elton John’s husband David Furnish is 57 years old" - produces the correct answer from the TANDA model.
Example 2 probes transformer ability to robustly recognize that "one-humped" and "two-humped" are values for the quantity sought by "how many" and are related to the subject "Camel" .
We hypothesize that the KB context "animal" added for similar entities during training increases attention on "camel" tokens and their modiﬁers, "one-humped" and "two-humped" in this case.
Figurecompares model attention weights of tokens "many" and "Camel" with the values "one" and "two" and again demonstrate that our approach signiﬁcantly increases the intensity of model attention between these terms.
Changing the answer to use common numeric values "the Dromedary Camel has 1 hump...and the Bactrian Camel has 2 humps" is sufﬁcient for the TANDA model to select the correct answer.
Example 3 illustrates whether the model is able to connect the adverbial phrase "some legal uses" in the question with the phrase approved...for the treatment of... in the correct answer.
Interestingly, the KB context added for "meth" and entities like it is "generic drug" , which we hypothesize may encourage attention to relevant terms like "treatment" that are not commonly used in context of the subject "meth" .
Figure 4 shows the weights connecting "treatment" with "uses" and "meth" and further demonstrates the impact of our approach on model attention.
We conclude that in some cases, the context itself may provide relevant information that helps the model more effectively utilize uncommon knowledge, like that meth may be used as a medical treatment.
Conclusion
In this paper, we presented a data-programming approach that enriches transformer training data with
KB-derived context, and demonstrate that it beats state of the art approach on several challenging knowledge-intensive question-answering benchmarks such as ASNQ, WikiQA, TrecQA, and Alexa
QA.
Our ﬁndings indicate that our approach addresses some deﬁciencies of transformer knowledge utilization that negatively impact AS2 performance.
We probed the mechanism of our approach with challenging examples that highlight the potential ways in which our KB context may allow transformers to better utilize encoded knowledge.
Our method is simple, efﬁcient and task-agnostic, and training beneﬁts remain even when KB context is omitted at inference time.
We believe that our approach provides a way to rapidly integrate the beneﬁts of KBs within the deployed inference pipelines utilized in many virtual-assistant workﬂows.
While we improve on the state of the art approach in AS we do acknowledge that our approach may face limitations of its own.
While our approach is efﬁcient in that it not require signiﬁcant pre-training, unlike KB based approaches like KELM, KnowBERT and ERNIE as well as retrieval oriented approaches like REALM and RAG, it is inefﬁcient in that it likely does not leverage the full richness of our KB.
This has the negative consequence that our approach still requires signiﬁcant task-speciﬁc training and thus consumes signiﬁcant GPU hours and the natural resources used to power them.
Further work beyond the data-programming approach that we propose in the direction of more effective transformer architectures that enhance knowledge utilization can lessen this impact and provide models capable of more completely disentangling knowledge and language acquisition.
A Dataset statistics
The table below shows the distribution of the datasets studied in this by each split, such as train , dev , and test .
These demonstrate that our pipeline is able to tag at least one KB entry in each input QA pair, indicating that our simple tagging method is effective at producing KB context.
[Table]
B Attention Weight Comparison
In the graphs below
[Table]
we illustrate the impact of our approach on model attention for the challenging
AS2 examples presented in Table 1.
We do not add KB context at inference for any of these examples, opting to visualize the impact of our approach in the more challenging "omit KB" setting.
We leverage
BertViz [29] to extract model attention weights and quantify model attention between meaningful