F I D-L IGHT : E FFICIENT AND E FFECTIVE R ETRIEVAL -A UGMENTED T EXT G ENERATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Retrieval-augmented generation models offer many beneﬁts over standalone language models: besides a textual answer to a given query they provide provenance items retrieved from an updateable knowledge base.
However, they are also more complex systems and need to handle long inputs.
In this work, we introduce FiDLight to strongly increase the efﬁciency of the state-of-the-art retrieval-augmented
FiD model, while maintaining the same level of effectiveness.
Our FiD-Light model constrains the information ﬂow from the encoder (which encodes passages separately) to the decoder (using concatenated encoded representations).
Furthermore, we adapt FiD-Light with re-ranking capabilities through textual source pointers, to improve the top-ranked provenance precision.
Our experiments on a diverse set of seven knowledge intensive tasks (KILT) show FiD-Light consistently improves the Pareto frontier between query latency and effectiveness.
FiD-Light with source pointing sets substantial new state-of-the-art results on six KILT tasks for combined text generation and provenance retrieval evaluation, while maintaining reasonable efﬁciency.
I NTRODUCTION
Enabling machine learning models to access information contained in parametric or non-parametric storage (i.e., retrieval-enhanced machine learning) can lead to efﬁciency and/or effectiveness improvements in a wide range of learning tasks (Zamani et al., 2022).
For example, retrievalaugmented generation (Lewis et al., 2020), which is the focus of this paper, has a manifold of beneﬁts over closed-loop language modelling in knowledge intensive tasks: Answers can be grounded in (multiple) speciﬁc pieces of information which enables clear attribution (Dehghani et al., 2019;
Rashkin et al., 2021; Lamm et al., 2021); the knowledge base can easily be managed, updated, and swapped (Izacard et al., 2022); the decomposition of retrieval and generation module offers clear efﬁciency-effectiveness tradeoff controls; and the data structure of combined retrieval and text generation enables many insightful failure analyses.
However, with these beneﬁts also come downsides, such as a higher system complexity with higher training and inference cost.
Therefore, our goal is to reduce costs as much as possible, while retaining effectiveness, to make these beneﬁts more widely available.
The most effective approach for knowledge intensive tasks, such as those contained in the KILT benchmark (Petroni et al., 2021), is the Fusion-in-Decoder (FiD) model proposed by Izacard & Grave (2020).
The FiD model uses an external retriever, such as a dense retrieval model, to gather candidate passages, which are encoded with the query by a T5-encoder (Raffel et al., 2020); the encoded vectors are concatenated and fed through a T5-decoder to produce a single output string.
FiD can synthesize answers from multiple different sources, which leads to state-of-the-art results in many tasks from open domain QA to fact veriﬁcation (Hofst ¨ atter et al., 2022; Izacard et al., 2022).
While undoubtedly the leading architecture – in terms of effectiveness for knowledge intensive generation tasks – the FiD model is resource intensive.
In state-of-the-art conﬁgurations concatenating all encoded tokens before the decoding leads often to sequences longer than 10 thousand vectors, coupled with auto-regressive decoding, this leads to a high inference latency.
In Figure 1 we plot the average latency of a single query measured on a single TPUv4 of the encoder and decoder modules
[Figure]
of FiD.
The ﬁrst observation is the overpowering 93% of time spent on decoding in FiD. A common and straightforward approach to reduce the latency of FiD is to reduce the number of input passages, e.g., to only 10 passages.
While this approach naturally reduces the overall latency, the decoding latency still requires 10 times as long as the encoding (see Figure 1).
Crucially, this approach will also reduce the model’s effectiveness substantially, as we show later in this work (see § 4.3).
To overcome the inefﬁciencies of the decoding, we propose FiD-Light , a simple yet effective adaptation of the FiD model.
The connection between the encoder and decoder has a large capacity for information in FiD. In contrast, the retrieval community, showed that in applications, such as dense retrieval with dot-product scoring, encoded information may be compressed to a fraction of the original input length, including representing passages in a single (Hofst¨atter et al., 2021) or multiple vectors (Chen et al., 2020).
Following these footsteps, we propose to compress the number of vectors per encoded passage, to a fraction of the input vectors, before they are accessed by the decoder.
Using this approach FiD-Light is able to ingest a large number of passages with strongly reduced latency, as illustrated in Figure 1.
Here we still use 40 passages, showing the same encoding time as FiD, but a substantially faster decoding (now on par with the encoding time), for a total latency lower than FiD with 10 passages.
The knowledge intensive tasks we aim to solve ideally require a system to produce both a generated output text, as well as a ranked list of provenance items from the knowledge base.
However, FiD is limited to only produce output text.
Falling back to return the original candidate ranking is usually sub-optimal with low-precision.
To incorporate re-ranking capabilities into FiD-Light we adapt a passage marker workﬂow proposed by Lakhotia et al.
(2021) as part of FiD-Ex.
They marked the input passages with textual indices, and trained the model to output the relevant indices in the output text.
We ﬁnd that using these textual indices or source pointers directly as output, as Lakhotia et al.
(2021) proposed, is brittle and prone to distribution shifts in the number of expected relevant passages between training and evaluation (see § 4.2).
Therefore, our FiD-Light SP approach re-ranks the selected passages to the top of the ranked list, without discarding the rest of the retrieved list, for higher robustness and improved results.
We conduct experiments on seven tasks of the KILT benchmark composed by Petroni et al.
(2021) spanning open domain QA, slot ﬁlling, fact veriﬁcation, and dialogue tasks.
We study the following research questions to demonstrate the efﬁcacy of our proposed FiD-Light SP model:
RQ1 What impact does training the retrieval module have on FiD-Light SP downstream results?
The quality of the ﬁnal result is strongly bound by the recall quality of the retriever module.
While many complex end-to-end training procedures have been proposed (Singh et al., 2021; Izacard et al., 2022), we focus on simple, yet effective directly supervised dense retrieval training.
We show that a simple retrieval training comfortably outperforms a zero-shot retrieval baseline from Hofst¨atter et al.
(2022) and the resulting FiD-Light SP downstream results take a major step towards a realistic oracle retriever ceiling.
RQ2 How robust is our source pointing and re-ranking workﬂow applied to FiD and FiD-Light?
We use available passage relevance information for each task in the KILT benchmark to train our source pointer output via text markers.
We train the FiD(-Light) generator to output the indices for all relevantly retrieved passages during training, before generating the textual answer.
We observe that FiD(-Light)
SP is learning an expected distribution for the number of selected passages, which might not match relevance distributions during evaluation.
To mitigate this problem we propose to use the source pointer to re-rank the initial list.
We show this improves the results over FiD-Ex.
Comparing the effectiveness of the source pointers between different FiD-Light settings and the FiD baseline we ﬁnd FiD SP to rapidly lose effectiveness when the number of input passages is reduced, while FiD-Light SP is able to hold the passage precision at much lower latency.
RQ3 How does FiD-Light SP compare to the FiD SP baseline in efﬁciency-effectiveness tradeoffs?
The common approach to speed up FiD is to reduce the number of input passages.
To this we compare our FiD-Light SP model using a static number of passages, but varying the number of vectors fed into the decoder as well as changing the T5 backbone size.
We show that while FiD SP with fewer passages strongly degrades, FiD-Light SP is able to hold most of the initial maximum effectiveness of FiD SP , while being 3 × faster.
This Pareto optimal result between latency and effectiveness is complemented when we increase the T5-backbone sizes in FiD-Light SP to receive the beneﬁts of larger models, while still outperforming the initial FiD SP baseline in terms of efﬁciency.
Overall
FiD-Light SP is Pareto optimal on six out of the seven tested tasks.
RQ4 How does FiD-Light SP compare to related methods on the KILT benchmark?
We submitted three representative conﬁgurations of FiD-Light SP to the blind-evaluated KILT leaderboard test set to compare them to other methods for knowledge intensive tasks.
We evaluate FiDLight SP on the main metric of the KILT benchmark: combined KILT-scores (which only counts a text generation score if the R-Precision for the query is 1).
We show FiD-Light SP outperforms previous SOTA models by considerable margins on the KILT-scores on six tasks.
We set new SOTA results compared to the previous best methods on
: - QA HotpotQA +11.1 K-EM (+61.3%), NQ +7.5 K-EM (+17.2%),
TriviaQA +5.8 K-EM (+10.0%)
- Slot Filling zsRE +10.8 K-AC (+14.8%), T-REx +0.5 K-AC
(+0.7%) -
Fact Veriﬁcation FEVER +6.0 K-AC (+7.6%)
We hope these results demonstrate to the community that SOTA results are achievable with reasonable efﬁciency and that efﬁcient retrieval-augmented generation has a promising future ahead.
B ACKGROUND AND R ELATED W ORK
In this section, we ﬁrst review the FiD model and FiD-Ex workﬂow, which adds textual explanation markers to FiD. We further discuss other related work in this area.
2. F I D (F USION IN D ECODER ) WITH E XPLANATIONS
A critical capability for retrieval-augmented models is to be able to synthesize and utilize information from multiple distinct retrieved items (Zamani et al., 2022).
To effectively implement this paradigm Izacard & Grave (2020) proposed the FiD model, which re-wires the computational graph between an of-the-shelf pre-trained Transformer Encoder and Decoder (Vaswani et al., 2017).
Usually FiD is initialized with the pre-trained T5 model (Raffel et al., 2020).
Given a query q , we retrieve a set of n candidate passages using a separate retrieval module.
The retriever is independently trained, and can take any traditional, neural or hybrid architecture.
As in Izacard & Grave (2020), we use a single dense retriever, as it has been shown to outperform traditional retrieval methods (Hofst¨atter et al., 2022).
To encode the information, FiD concatenates the query q with each retrieved passage p and independently feeds (one per index i ) the sequences through a Transformer encoder ( T E ):
[Equation]
The resulting encoded representations – using one vector per token – are concatenated into a single long sequence, which is fed through the Transformer decoder ( T D ), autoregressively during inference, to produce a single output sequence o :
[Equation]
FiD has two main limitations: (1) the text-only output does not provide any information about the exact passage(s) which were used to synthesize the output; and (2) the long input sequence leads to highly inefﬁcient autoregressive decoding (as shown in Figure 1).
While the expected output is relatively short (in the magnitude of dozens of tokens), the input to the decoder is large with
O ( n ∗ ( | q | + | p | ))
tokens (in the magnitude of thousands of tokens).
To alleviate limitation (1) Lakhotia et al.
(2021) adapt the FiD workﬂow with textual explanations (FiD-Ex) inspired by the WT5 (Why?, T5) concept proposed by Narang et al.
For FiD-Ex, the FiD architecture is left untouched; Lakhotia et al.
(2021) only adapt the textual input and target output.
The input to the encoder is augmented with indices (from 1 to n ) to identiﬁy individual passages:
[Equation]
And the target output t during training is augmented with the indices (using the regular tokens for the numbers and spaces as separators for multiple indices) of all the known relevant passages R + in the retrieved set:
[Equation]
On one hand, this textual formulation packs more capabilities in the same text based architecture, on the other hand we note that this discrete selection of the top- | R + | passages from the candidate set is a strong departure from the prevalent pairwise re-ranking models.
It opens a new range of induced biases about expected distributions of | R + | not studied before.
During inference the output is parsed to extract the indices as numbers and remove the additional textual markers to evaluate the output text.
2. R ELATED W ORK
Efﬁcient Generation Models.
To enable their ubiquitous use, a key component besides their safety, is the efﬁciency of text generators to run at scale.
Naturally, many studies work to achieve this goal from various angles.
Schuster et al.
(2022) propose an adaptive early exiting language model, which exits the decoder stack of Transformer layers early for easy to predict tokens.
The LongT model focuses on improving the efﬁciency of the encoder for long input sequences (Guo et al., 2021), in contrast we focus on the decoder efﬁciency, as FiD’s encoder input is usually short.
We believe our FiD-Light adaptations are orthogonal to many other algorithmic and engineering-based generation efﬁciency improvements and can be combined in future work.
For a comprehensive overview over efﬁcient transformer architectures, we refer the reader to Tay et al.
Retrieval-Enhanced Machine Learning.
The foundational retrieval-augmented models, e.g.,
FiD (Izacard & Grave, 2020), RAG (Lewis et al., 2020), and REALM, (Guu et al., 2020) are trained to solve individual tasks.
Many of their recent improvements optimized end-to-end processes (e.g., EMDR2 (Singh et al., 2021)), ensembling multiple modules (e.g., R2-D2 (Fajcik et al., 2021)), or creating multiple training loops to update the indexed documents multiple times (e.g., Hindsight (Paranjape et al., 2021)).
In contrast, we focus on architectural efﬁciency improvements with a simple training paradigm.
Recently, more task-independent retrieval-enhanced language models emerged, such as retrieving from a text-snippet database (Borgeaud et al., 2021) or learning to retrieve from the web with reinforcement learning (Nakano et al., 2021).
For more information on retrieval-enhanced machine learning models, we refer the reader to Zamani et al.
Improving and Adapting the FiD Model.
To integrate passage relevance prediction into FiD,
Asai et al.
(2021) add a second decoding module, which is called for every query-passage sequence to indicate its relevance.
They also use this setup to generate silver-relevance scores for unjudged passages.
Yu et al.
(2022) replace the retrieval module with a large language model to generate supporting documents, which are then fused to generate the answer by a default FiD implementation.
The current top-systems on the KILT leaderboard (Hofst¨atter et al., 2022; Izacard et al., 2022) use strong retrievers in combination with large T5-backbones for FiD. They also improve the supervised training by using better data sampling or pre-training procedures for more data efﬁcient ﬁne-tuning.
We continue in the spirit of these related works with additional efﬁciency and capability improvements of FiD.
[Figure]
F I D-L IGHT WITH S OURCE P OINTERS
With FiD-Light SP we overcome the two main limitations of the FiD-Ex model and workﬂow: We drastically increase the efﬁciency of the decoder, by reducing its computational requirement, and we improve the robustness of the passage selection with a source pointing workﬂow, by shifting our view from an explanation to a second, parallel-solved task: re-ranking passages.
We provide an overview of our FiD-Light SP model and source pointer workﬂow in Figure 2.
Decoder Efﬁciency.
Following our initial observation, that FiD spends most time in the decoding phase (Figure 1), we adapt the original FiD decoding step (Eq. 2) to reduce the length of each encoded query-passage pair to k vectors via a function f :
[Equation]
This reduces the input length from the previous O ( n ∗ ( | q | + | p | )) to O ( n ∗ k ) , where k (cid:28)
| q | + | p | .
The exact compression ratio depends on the required tokens for the used tasks; we experiment with conﬁgurations from a 6x to 384x fold reduction.
In our experiments, for simplicity, we instantiate f k as the ﬁrst k vectors of each sequence.
While this architecture change is simple, it strongly disrupts previous assumptions that every encoded token is accessible for decoding in the T5 architecture.
Its simplicity also means that the community can easily adapt existing codebases with this change to beneﬁt from the efﬁciency improvements.
Source Pointing Robustness To enable source pointing in FiD-Light, we train the model with the source pointing concept proposed by Lakhotia et al.
(2021) in FiD-Ex.
Our novel contribution is how we handle the output of the source pointers at inference time.
If we use them directly as result, as in FiD-Ex, we are prone to instability in the number of returned passages.
The question of processing the output further almost becomes philosophical: if we treat the source pointers as explanations we can not process them any further without corrupting the explanation.
While, there might be a correlation between the textual output and the source pointed passages, we are treating ﬁnding the source passages as a concurrent task to generating the text output.
Because we are not claiming them to be explanations we can now process them further.
We propose to merge the initial ranked candidate list of passages C with the source pointing selected passage by re-ranking the selected passages (found in the decoded output ˆ o ) to the top of the list:
[Equation]
To compute all selected passages r ∈
ˆ o we ﬁrst parse the output ˆ o with a simple parser for the trained format given in Eq. 4, including a conversion from the text-tokens representing the indices to integers.
In case the model selects multiple passages we keep the selection order of the model output.
If a task contains graded relevance annotations for training passages, we can train the model to follow the grades, if only binary relevance is available (as in the case with KILT), we keep the rankordering of the multiple selected passages from the initial candidate list.
This change leads to higher robustness in our provenance results, as distribution differences between training and evaluation
[Table]
R ESULTS
We empirically address the research questions laid out in the introduction.
We study the importance of the retriever module, the efﬁcacy of the source pointer workﬂow, the tradeoff between efﬁciency and effectiveness using a controlled baseline, and ﬁnally we compare our FiD-Light SP to related methods on the blind-evaluated KILT leaderboard.
We detail our experiment design in Appendix A.
4. I NFLUENCE OF THE R ETRIEVER
The retrieval module is the backbone for all retrieval-augmented generation.
The generation quality is to a large extent bound by the retrieval quality, especially if the retrieved information is not memorized by the generator.
To answer RQ1 What impact does training the retrieval module have on FiD-Light SP downstream results?
we have to be careful to acknowledge the uncertainty of sparse ranking annotations (Hofst ¨ atter et al., 2022).
To accurately quantify the retriever’s contribution, we compare the downstream effect of a zero-shot, a ﬁne-tuned (methodology described in detail in Appendix B), and two oracle retrievers in Table 1.
In the ﬁrst section (rows 1-3) retrievers are evaluated without access to relevance judgements (a real-world environment), whereas in the second section (rows 4 & 5) we infuse relevance information during the evaluation (oracle environment).
We ﬁnd that training the retriever with in-domain training data (row 2) consistently improves results over a zero-shot retriever (row 1) as used by (Hofst ¨ atter et al., 2022).
While always ingesting all known relevant passages during training (row 3) does not signiﬁcantly change the downstream performance.
To account for annotation uncertainty in our retriever as oracle experiments, we study two scenarios: 1) infusing all known relevant passages into the retrieved candidate list (row 4) and 2) setting the candidates to be only the known relevant passages (row 5).
Commonly, the community compares their results only against the second oracle scenario, showing a large headroom for future improvements for the retriever (Glass et al., 2021; Shuster et al., 2021).
However, we argue, due to the sparsity of the annotations, we should compare the results to our more realistic ﬁrst oracle scenario (row 4).
It still shows a signiﬁcant opportunity for improvement, albeit the total headroom is roughly halfed across the board.
Future work may explore more ﬁne-tuning aspects, but we decide to select the simple ﬁne-tuned retriever (row 2).
4. S OURCE P OINTER R OBUSTNESS
While the initial source pointer concept has been proposed by FiD-Ex as sentence markers for explainability, we are the ﬁrst to study their application in the more complex passage ranking context combined with our compressed FiD-Light architecture.
Therefore, we study RQ2 How robust is our source pointing and re-ranking workﬂow applied to FiD and FiD-Light?
As introduced earlier, we train the source pointing capabilities into FiD(-Light) by ﬂagging all known relevant passages retrieved in the candidate passage set.
By directly using the size of the known relevant item set during training we instill a strong expectation prior into the model of how many passages ought to be relevant for a given task.
Note, if a known relevant passage is not retrieved we cannot use it for training the generator.
In Figure 3, we observe these effects for four representative tasks of the KILT benchmark.
Each of these tasks shows a different expected distribution target.
We note that the training distribution differs from the target, as it skips non-recalled relevant items.
We ﬁnd the model output distribution on the validation set to closely match the training distribution (albeit here we make no claims about the correctness of the selected passages).
[Figure]
e l a t i v e
O cc
u rr e n c e
[Table]
However, focusing on higher passage counts in Figure 3 (a) TriviaQA and (c) FEVER shows that the model struggles to output 3 passages as often as it is expected to do.
This weakness becomes visible, when we evaluate the standard R-Precision of the selection, which needs at least R returned items to reach the full score, given R known relevant items.
To overcome this limitation, we propose instead of directly outputting the selection (FiD-Ex) to move the selected passages to the top of the ranked list.
This essentially transforms FiD(-Light) into a re-ranking model.
In Table 2, we show the ablation study to conﬁrm the usefulness of the proposed re-ranking on ﬁnal downstream results.
Our approach is strictly positive or neutral for the results, as we are ﬁlling up holes, that would result in penalties.
Conﬁrming our hypothesis originating in
Figure 3, we see stat.
signiﬁcant improvements across all conﬁgurations on the two task, where the model struggled to ﬁll up the full distribution: TriviaQA and FEVER.
While in this work we do not change the KILT evaluation methodology and optimize our models towards the current standard evaluation, we note that these ﬁndings represent interesting avenues for future work requiring evaluation setup changes: We may choose to train the model to only select a single passage or even re-rank the whole list with our textual source pointers as re-rankers.
[Figure]
We might be tempted to directly compare the intersetting results in Table 2, for example FiD SP in row 2 with FiD-Light SP in row 4 (T5-Base).
Here we observe, especially on HotpotQA and TriviaQA, a quality reduction, which would lead us to the conclusion that source pointing in FiD-Light is less robust than FiD.
To put these results into perspective, we exemplary selected HotpotQA and plot the query latency as well as the R-Precision of the models in Figure 4.
For FiD SP , we modulate the number of input passages; for FiDLight we modulate the number of vectors k fed to the decoder and the backbone size.
We clearly observe a stark reduction in quality for the FiD SP model, when the number of input passages is reduced.
On the other hand our FiD-Light SP variants are able to almost keep the same level of effectivness, and larger backbones, while still faster than the FiD SP baseline also produce a higher quality.
Therefore, an equal-efﬁciency comparison in Table 2 involves row 2 and row 8 (using
T5-XL).
We are diving deeper in these tradeoffs in the next section.
[Figure]
4. E FFICIENCY - E FFECTIVENESS T RADEOFF
Ultimately, we as a community want our research be applied to real world use, to beneﬁt society.
A major component, besides concerns about safety and social biases as summarized by Bender et al.
(2021), is the efﬁciency of the deployed system.
To understand the impact of our proposed
FiD-Light architecture we study RQ3 How does FiD-Light SP compare to the FiD SP baseline in efﬁciency-effectiveness tradeoffs?
The KILT benchmark gives us the opportunity to study our changes in a large variety of tasks, with different properties, so that we can make conﬁdent claims about the efﬁcacy of our changes.
In
Figure 5 we show our ablation results per task.
For each task we report the average query latency (y-axes) and the main KILT-score effectiveness metric (x-axes).
The gray line indicates our FiD baseline by modulating input passage counts – from 40 down to 1.
Our FiD-Light models all have access to the full 40 passages, and here we are modulating T5 sizes as well as the number of vectors (1, 8, 32, 64) fed into the decoder.
We start our discussion with the open domain QA tasks in Figure 5 (a, b, & c) as they provide a similar picture:
Comparing our FiD-Light SP model with the baseline we do observe a drop in effectiveness from the strongest baseline (gray dotted vertical line) when using the same T5-Base model.
However, due to the more efﬁcient architecture we are able to swap backbones and earn the beneﬁts of those larger models in terms of effectiveness.
At the same time we outperform the latency of the baseline as well, shifting the Pareto optimum.
Interestingly, the FiD-Light SP model with T5-XL and only a single encoded vector per passage shows a larger drop in effectiveness than the counterparts for smaller T5’s.
The only 2-label classiﬁcation task, FEVER, shown in Figure(d), exhibits the lowest reduction in effectiveness, when constraining the number of encoded vectors in FiD-Light SP .
This is likely due to the fact, that only little generation is necessary to solve the task.
Therefore, our FiD-Light SP conﬁgurations improve the Pareto optimum again.
The slot-ﬁlling tasks in Figure 5 (e & f) show less impact of the T5 size, with little improvement for Large and XL over the Base conﬁgurations.
Fortunately, we also observe a similarly small reduction in effectiveness for reducing the number of encoded FiD-Light SP vectors, leading to our ﬁnal Pareto gains.
In conclusion we observe clear and statistically signiﬁcant improvements between FiD SP and FiDLight SP – both in terms of effectiveness and efﬁciency – across a variety of KILT tasks.
FiD-Light SP
[Table] [Table]
can lower the query latency by more than 2x and still deliver higher effectiveness by upgrading the language model backbone size.
4. C OMPARISON TO R ELATED W ORK
In addition to showing improvements over our own baselines, we now demonstrate the effectiveness of FiD-Light SP in a broader context and answer RQ4 How does FiD-Light SP compare to related methods on the KILT benchmark?
The community is fortunate to have a blind-evaluation leaderboard for all KILT tasks 3 at our disposal to compare our approaches on a level playing ﬁeld, where everyone may submit their highly-tuned systems.
While the top spots of a leaderboard are typically not populated by efﬁcient methods, we nevertheless submitted three different conﬁgurations of
FiD-Light SP – all more efﬁcient than our FiD baseline with 40 input passages.
We selected a single checkpoint to submit for all tasks, so as to demonstrate our multi-task capabilities and not overﬁt a single submission to a single task.
We show the leaderboard results for the main KILT-score metrics in Table 3.
For the independent breakdown of text generation and retrieval leaderboard scores we direct the reader to Appendix C.
Even our T5-Base conﬁguration in row 8 already outperforms previous SOTA results on ﬁve out of the seven tasks.
With T5-Large and T5-XL (both continuously reducing the number of encoded vectors, to increase efﬁciency) set new SOTA results on six out of the seven tasks.
Only WoW remains a weak spot, albeit not dramatically different to previous results.
The fusion capabilities of FiD paired with our robust source pointing set especially impressive results on the challenging
HotpotQA task, where exactly two distinct passages containing parts of the answer have to be placed on top of the ranked list.
Here, we outperform previous methods by 61% or 11.1 KILT-EM points.
On the other two QA task we reach +7.5 K-EM (+17.2%) for NQ and +5.8 K-EM (+10.0%) for
TriviaQA.
The zsRE task with +10.8 K-AC (+14.8%) and FEVER with +6.0 K-AC (+7.6%) round off our strong new SOTA results across a variety of tasks.
C ONCLUSION
We proposed the FiD-Light model with a robust source pointing workﬂow to overcome efﬁciency and versatility limitations in the previous state-of-the-art retrieval-augmented generation model FiD.
We adapted the FiD model architecture to compress the amount of information fed to the decoder, for drastically reduced inference latency.
We demonstrated at the same time only a modest reduction in effectiveness, which can be alleviated with larger T5-backbones leading to Pareto optimal results on six KILT tasks.
Our multi-task system achieved substantial new state-of-the-art results for combined retrieval and generation metrics on six KILT tasks compared to previous methods on the public leaderboard.
These results demonstrate that we do not need to always scale up to achieve the highest effectiveness, enabling more researchers to work on this problem in the future.
A E XPERIMENT D ESIGN
Implementation.
Our experiment setup follows the state-of-the-art multi-task relevance sampled training sets of Hofst ¨ atter et al.
All our experiments are based on the T5X framework (Roberts et al., 2022).
We start with a GTR-Base dense retrieval model (Ni et al., 2021), which is pre-trained on the MSMARCO passage retrieval task (Bajaj et al., 2016) and has been shown to generalize well on the BEIR benchmark (Thakur et al., 2021).
We train our FiD(-Light) models using T5 v1.1 as language model backbone (Raffel et al., 2020) on TPUs.
We attach task-speciﬁc markers to the queries for the multi-task training.
We cap the input at 384 tokens (combined query and passage) and a maximum of 64 output tokens.
For training, we use a batch size of 128 with up to 40 retrieved passages, and a learning rate of 10 − 3 with the Adafactor optimizer (Shazeer &
Stern, 2018).
We do not tune our models to a speciﬁc checkpoint, rather train them all for 50K steps.
The only special case is T5-XL, which uses a learning rate of 5 ∗ 10 − 4 and is trained for 30K steps.
During decoding we use beam search with a beam size of 4.
Datasets.
We conduct experiments on 7 KILT tasks:
HotpotQA (Yang et al., 2018), TriviaQA (Joshi et al., 2017), Natural Questions (NQ) (Kwiatkowski et al., 2019), T-REx (Elsahar et al., 2018),
Zero Shot RE (zsRE) (Levy et al., 2017), FEVER (Thorne et al., 2018), and Wizard of Wikipedia (WoW) (Dinan et al., 2018).
We give an overview over the dataset in Table 4.
We used the ﬁltered training & passage sets from Hofst ¨ atter et al.
and the original evaluation sets from Petroni et al.
Evaluation.
We follow the KILT evaluation setup proposed by Petroni et al.
(2021), in particular we focus on the main KILT-score metrics, which combines both a text output metric M (such as
EM, Accuracy, or F1) with R-Precision ( RP ) per query, before aggregating the individual query results over the query result set Q :
[Equation]
In essence, KILT-scores only count the text score M if the R-Precision of the query is 1, meaning all R relevant passages or documents are returned on the top- R positions of the ranked list.
This metric makes the assumption that only a few (1 to 2) items are marked as relevant, as is the case in the KILT dataset.
To reduce the noise in our dev results, we present the mean and a 95% conﬁdence interval measured with a t-statistic of the last 10 checkpoints (every thousand steps from 40K to 50K training steps).
For our leaderboard submission, we selected a single checkpoint for all tasks.
Unfortunately, we can not compute statistical signiﬁcance tests compared to other methods, as the submission ﬁles and gold-labels are not publicly available.
[Table]
B D ENSE R ETRIEVAL T UNING R ESULTS
In our experiments we use a ”double-ﬁnetuned” GTR dense retriever retriever: First it was trained on the MSMARCO retrieval task (Bajaj et al., 2016) by Ni et al.
and then we ﬁne-tuned their checkpoint further on our combined KILT training set to create a single generalized KILT retrieval module, akin to Maillard et al.
We created passage retrieval training triples containing a query, a known relevant passage, and a sampled negative passage (randomly sampled from the top100 GTR zero-shot rankings for the query).
We then ﬁne-tuned the retriever for 100K steps using the
GTR default parameters in the t5x Retrieval framework.
We did not employ knowledge distillation (Hofst ¨ atter et al., 2020) or complex end-to-end losses (Izacard et al., 2022), to demonstrate the effectiveness of our approach in a simple setting which likely is orthogonal to more complex training setups.
This approach means, that while we expect to learn retrieve better results, we may overshoot our target and overﬁt on the training data, leading to a growing divide in the the train vs. test performance.
This matters strongly in our retrieval-augmented generation setup, because we use the fully trained retrieval model as the source for our generation training data.
We aim to detect and avoid unnecessary distribution shifts to actually train the generator on the expected retrieval performance and not an overﬁtted training set.
We choose to modulate the learning rate to control for and study the train vs. test distribution shift.
We focus on the recall at the highest cutoff we use in our experiments (the top-40) and provide our results in Table 5.
First, we show the zero-shot results, as used by the previous state-of-the-art
FiD models from Hofst ¨ atter et al.
(2022), followed by our novel ﬁne-tuned GTR models.
Our ﬁrst observation is that in all tasks we are able to achieve signiﬁcant R@40 gains on the dev set compared to the zero-shot baseline – ranging from 0.13 to 0.20 absolute changes.
Concerning our learning rate study, we ﬁnd too high learning rates (especially 0.1 and 0.05) show a high ∆ T, which indicates a strong distribution shift between train and test.
If we were to only train one of the high learning rate checkpoints and compare the dev results to the zero-shot baseline we could be tempted to use them, as their dev results look strong.
However, due to our ﬁne-grained analysis we see that it would introduce a strong distribution shift.
Another interesting observation we make is how different task categories seem to converge at different velocities – the open domain QA tasks reach their optimal dev results with higher learning rates, while the other tasks fare better with lower rates.
Curiously, we would have guessed a reverse trend, as the initial MSMARCO retrieval task is more closely aligned to QA, suggesting less needed movement.
We did not continue to tune the composition of our retrieval training as it is only a secondary contribution to this work and the differences are quite small compared to the margin we achieve to the zero shot baseline.
Therefore, we decided to go forward with the 0.005 learning rate, as it overall gives the best results with low distribution shifts.
[Table]
C D ETAILED R ELATED W ORK C OMPARISONS
In Table 3 we focused on the combined retrieval and text generation KILT scores.
Now, we investigate our results further, by analyzing the two components independently in Table 6.
For each task we report the leaderboard text generation test score (EM, AC, or F1) and the retrieval quality via
R-Precision.
As previously noted, (Izacard & Grave, 2020; Hofst ¨ atter et al., 2022), there is a strong correlation between model size and text generation quality on KILT.
For better comparability, and to not ”poison” the task with only very large models, that are not trainable for many of our fellow researchers, we report small and large model numbers for FiD-Light.
Looking at the existing leaderboard entries we observe the top systems mostly rely on the FiD architecture.
The most recent and highest performing approaches are FiD generators with relevance sampling and Atlas training regimes (row 8,9).
It is important to note, that these two systems are very inefﬁcient: They run 50 and 100 passages through FiD per query and use T5-XL and T5-XXL respectively.
They also only focus on the text generation part of the KILT challenge, and chose not to submit any supporting passages for the generation.
This is in large part due to the fact, that FiD on its own does not provided a ranking component to the passages, which leads to under-performing results.
Our FiD-Light SP entries cover multiple T5 and k encoded vector sizes.
While there is our expected spread of the text generation quality based on the the T5 size, we observe that this spread is substantially smaller for the R-Precision metric.
To be able to compare methods, the KILT leaderboard computes the R-Precision on a document level.
We transformed our passage ranking to document ranks, by taking the highest ranked passage per document as the document rank, and removing subsequent passages from that document from the ranked list.
Overall all our models beginning with T5-Base set new SOTA results across the board for the ranking sub-task, even considering we only re-rank 40 passages.
Analysing the text generation quality, we see no new SOTA results for
FiD-Light SP , but we remain competitive with the largest and slowest entries in the leaderboard.
To conclude, we showed the reason for our overall strong SOTA results on the KILT scores in Table 3 is the combination of competitive text generation quality with strong SOTA ranking results shown in Table 6.
[Figure]
D F AILURE A NALYSIS
The setup of the knowledge intensive text generation with supporting passages, not only enables positive evaluation via the KILT scores, but also a rich quantitative failure analysis.
As BoydGraber & B ¨ orschinger (2019); Hofst ¨ atter et al.
(2022) argued, we should spend more time and energy looking beyond our aggregated metrics.
Therefore, in Figure 6 we look at the composition of the raw output results of FiD-Light SP (without re-ranking) in 4 potential outcomes: 1) both passage and text results are wrong; 2) correct passage, but wrong text; 3) correct text, but wrong passages; and 4) both result parts are correct.
We analyze the results of two T5-backbones across our KILT tasks.
Interestingly, we do not observe converging trends in their failures between the Base and XL backbones across tasks.
But we do see strong differences in the distribution of failure types between tasks.
The open domain QA tasks are more likely to fail, especially both parts.
For the FEVER fact veriﬁcation, if we scored the relevant passage on top we are very likely to also get the right boolean answer.
The large part of wrong passage selection, but right answer in TriviaQA is likely attributed to its high degree of noise as observed by Hofst ¨ atter et al.
HotpotQA remains the most challenging task with the highest double failure rate.
We note that the KILT tasks are highly noisy: we only have 1-2 relevant marked passages in most cases and few if any textual variations of the text answers.
This is also the reason we did not run this analysis on WoW, which has no exact text matches.
We hypothesize, that if both result parts fail, we are more likely to have a true failure of the model compared to only failing one aspect, which could indicate a noise issue in the datasets.
However, to conﬁdently claim this we would need to conduct a thorough annotation campaign of the existing results.
We created an interactive website for inspecting all model outputs of FiD-Light split by our failure analysis modes from Figure 6.
The website displays the user 10 random results, per category and task, so as not to enable cherry picking by us.
Every refresh of the website creates a new random sample allowing the users to playfully, yet targeted explore the datasets and results in a qualitative way.
The website is available at: anonymized