A Sequence Modelling Approach to Question Answering in Text-Based Games
Jan Buys
Abstract
Interactive Question Answering (IQA) requires an intelligent agent to interact with a dynamic environment in order to gather information necessary to answer a question.
IQA tasks have been proposed as means of training systems to develop language or visual comprehension abilities.
To this end, the Question Answering with Interactive Text (QAit) task was created to produce and benchmark interactive agents capable of seeking information and answering questions in unseen environments.
While prior work has exclusively focused on IQA as a reinforcement learning problem, such methods suffer from low sample efficiency and poor accuracy in zero-shot evaluation.
In this paper, we propose the use of the recently proposed
Decision Transformer architecture to provide improvements upon prior baselines.
By utilising a causally masked GPT-2 Transformer for command generation and a BERT model for question answer prediction, we show that the
Decision Transformer achieves performance greater than or equal to current state-of-the-art
RL baselines on the QAit task in a sample efficient manner.
In addition, these results are achievable by training on sub-optimal random trajectories, therefore not requiring the use of online agents to gather data.
Introduction
Traditional methods for question answering (QA) and machine reading comprehension (MRC) are primarily concerned with the retrieval of declarative knowledge , that is, explicitly stated or static descriptions of entities in text documents or within a knowledge base (KB)
(Trischler et al., 2017).
These models tend to answer questions primarily through basic pattern matching skills, further differentiating their abilities from those of humans.
Conversely, procedural knowledge is the sequence of actions required to perform a task (Georgeff and
Lansky, 1986).
To this end, interactive question answering (IQA) has been proposed as a framework for teaching MRC systems to gather the information necessary for question answering (Yuan et al., 2019).
IQA requires an agent to interact with some dynamic environment in order to gather the required knowledge to answer a question (Gordon et al., 2018).
As such, the task is well-suited to be approached as a reinforcement learning (RL) problem.
Yuan et al.
(2019) proposed Question Answering using interactive text (QAit) as a means of testing the knowledge gathering capabilities of an agent required to answer a question about its environment.
Here an agent interacts with a partially observable text-based environment, created using
Microsoft TextWorld (Côté et al., 2018), in order to gather information and answer questions about the attributes, location, and existence of objects.
The QAit task thus aims to benchmark generalisation and provides an environment to train agents capable of gathering information and answering questions.
Yuan et al.
’s proposed baselines (using DQN (Mnih et al., 2015), DDQN (Van Hasselt et al., 2016), and Rainbow (Hessel et al., 2018)) all suffered from low sample efficiency and relatively poor performance on all three question types (location, attribute, and existence).
These shortcomings suggest that alternative architectures and methodologies are required to improve performance within the QAit setting.
Transformers (Vaswani et al., 2017) have shown success in modelling a diverse range of highdimensional problems (Brown et al., 2020;
Ramesh et al., 2021; Devlin et al., 2019).
Additionally, existing language models such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pre-Trained)
(Radford et al., 2019) have been utilised to reduce the size of datasets required for training downstream language tasks (Lee and Hsiang, 2019; Mager et al., 2020).
These benefits coupled with the demon- strated ability of Transformers to model long sequences by utilising the self-attention mechanism makes this architecture ideal for IQA.
Recent work (Chen et al., 2021; Janner et al., 2021) have shown the applicability of Transformers to sequential decision making problems as an alternative solution to
RL problems.
These approaches frame RL trajectories as sequences of states, actions, and rewards modelled autoregressively by a Transformer.
This sequence modelling approach is referred to as the
Decision Transformer (DT) (Chen et al., 2021).
In this paper we apply the Decision Transformer to QAit, replacing the online interaction and training methodology of RL approaches with a Decision
Transformer that utilises the GPT-2 (Radford et al., 2019) architecture, closely following the methodology outlined by (Chen et al., 2021).
We propose an additional QA module that is a fine-tuned BERT model, with the aim of leveraging pre-trained language models to provide more accurate answers to questions.
We show that by framing the QAit task as a sequence modelling problem, a Decision
Transformer matches or exceeds the performance of previous RL-based benchmarks when trained on random episodic rollouts, while using significantly less data.
Our main contributions are as follows:
We show that an offline reinforcement learning method is able to match the performance of online value-based reinforcement learning baselines in the QAit environment.
We show that by framing IQA as a sequence modelling problem, the performance of the QAit baselines can be matched using significantly less training data.
We show that the Decision Transformer architecture is able to learn policies comparable to those of online reinforcement learning methods from purely random data, illustrating the architecture’s ability to find structure in inherently noisy data.
Background 2. QAit
QAit is implemented in TextWorld 1 (Côté et al., 2018), an open-source simulator for training reinforcement learning (RL) agents for decision making and language comprehension.
QAit text-based environments are generated procedurally via sampling from a distribution of world settings.
There are two environment map types: A fixed map contains six rooms, whereas random maps sample their number of rooms from a uniform distribution
U (2 , 12) .
QAit requires an agent to answer questions about the location, existence and attributes of objects in an environment.
An agent interacts with a QAit environment using text commands that consist of an action, modifier, and object triplet, e.g., "open black oven".
A generated environment consists of rooms each containing randomly assigned objects and location names.
The agent moves around in the environment for a pre-defined number of time steps or until the predicted command action is "wait".
TextWorld responds to agent commands with a state string containing information about the room the agent is in and the objects present.
2.1. Question types
An agent is required to answer one of three question types:
• Location questions assess an agent’s ability to navigate the environment to find the location of an object.
For example, "Where is copper key?" could be answered with "garden" or "toolbox".
• Existence questions requires the agent to navigate and interact with the environment to gather knowledge and determine whether an object exists.
Questions are phrased as "is there any X in the world?", where X is an entity in the vocabulary, and answers are either yes ("1") or no ("0").
• Attribute questions require that the agent interacts with an object to determine whether it has a particular characteristic or quality.
For such question types, the level of interaction and movement required in observing a sufficient amount of information to answer a question greatly exceeds location and existence questions.
Answers are also either yes or no.
For example, "Is stove hot?" requires an agent to find and interact with "stove" to answer the question correctly.
Comprehension of both the question and the environment are required.
Entities often have arbitrary names and attributes, making memorisation impossible.
2.1. Rewards
Yuan et al.
(2019) proposed two reward types:
Sufficient Information: Sufficient information is a metric used to evaluate the amount of information gathered by the agent and whether or not the information was sufficient to answer the question (Yuan et al., 2019).
It is also used as part of the reward function.
The sufficient information score is calculated when the agent decides to stop the interaction and answer the question.
For each question type, the sufficient information score is calculated as follows:
• Location:
A score of 1 is given if, when the agent decides to stop the interaction, the entity mentioned in the question is present in the final observation.
This indicates the agent has witnessed the information it needs to answer the question successfully.
If the mentioned entity is not present in the final observation then a score of 0 is given.
• Existence: If the true answer to the question is yes then a score of 1 is given if the entity mentioned in the question is present in the final observation.
If the true answer to the question is no , then a score between 0 andis given proportional to the amount of exploration coverage of the environment the agent has performed.
Intuitively this can be seen as a confidence score - if the agent witnesses the entity, it is 100% confident of its existence; otherwise, until it explores the entire environment, it cannot be completely confident.
• Attribute: Attribute questions have a set of heuristics defined to
verify each attribute and assign a score of sufficient information.
Each attribute has specific commands that need to be executed for sufficient information to be gathered.
This also depends on the agent being in certain states for these outcomes to be observed correctly, e.g. an agent needs to be holding an object to try to eat the object.
Exploration Reward: The agent is also given an exploration reward (Yuan et al., 2018) whenever entering a previously unseen state in order to promote exploration of the environment.
2.1. Evaluation
(Yuan et al., 2019) trained agents on multiple Number of Games settings, i.e., number of unique environments that an agent interacts with during train- ing.
In this paper, we restrict our experiments to the 500 games setting when generating offline training data for the Decision Transformer.
We measure an agent’s performance through both sufficient information score and question answering accuracy.
Models are evaluated in a zeroshot evaluation on the QAit test set in order to assess agents’ generalisation abilities.
Each question type and map type have their own unique set of 500 never-before-seen games, each containing a single question.
2. Decision Transformer
The Decision Transformer (Chen et al., 2021) architecture approaches reinforcement learning problems by autoregressively modelling a trajectory of actions/commands, states, and rewards.
Command triples (action, modifier, noun) are conditioned upon the total reward that can still be gathered from interacting with the environment.
This is referred to as the returns-to-go (RTG) R t = [Equation]
Tt ′ =
t r t ′ where T is the trajectory length and r t is the reward at time step t .
Thus the initial return-to-go R 1 represents the total reward to be gained from a given episode.
After every episodic play-through, the trajectory is represented as ( R 1 , s 1 , a 1 , R 2 , s 2 , a 2 ...R T , s T , a T ) , where R t is the RTG, s is a state, and a an action/command.
An example QAit trajectory is shown in Figure 2.
The trajectory representation enables training a sequence model such as GPT-2 (Radford et al., 2019), as command prediction is based on gaining some future reward, rather than on how much reward has already been obtained.
During testing the model is conditioned on total desired reward by setting R 1 and the starting state to generate command sequences autoregressively.
If an agent obtains some reward while interacting with the world, this is deducted from its RTG in subsequent time steps.
Approach
Training a Decision Transformer requires offline training data for supervised learning.
Online reinforcement learning, in contrast, sees an agent continually interacting with the environment to gather experience and update its policy based on observed rewards.
3. Training data generation with random rollouts
We generate offline training data using random rollouts for each map type and question type in [Table] the 500 games setting.
The rollouts are generated using a random agent which uniformly samples commands from all admissible commands for a particular time step.
This restriction stems from the sparsity of the action space (approximately 1654possible commands compared to approximatelyadmissible commands): sampling commands from the complete vocabulary results in mostly invalid commands.
Thus, by only using admissible commands in data generation, we intend for the Decision Transformer to learn which command triplets are admissible (as this is unknown during testing).
The sequence of commands and observed states are recorded along with the reward for each command.
Training dataset statistics are given in Table 1.
3. Decision Transformer
The maximum trajectory input length of the Decision Transformer is set to K = 50 time steps, which is the maximum length of a QAit episode.
This allows the DT to access the entire trajectory for command generation and question answering.
Token embeddings for states and command sequences are obtained using a single embedding layer.
At each time step the sequence of tokens representing the current state of the environment is encoded with a GRU (Cho et al., 2014) with the final hidden state h n representing the entire encoded environment state.
We concatenate the question to the end of each state sequence, separated by a “< | >” delimiter token.
Commands, which can consist of up to 3 tokens, are similarly encoded with another
GRU.
Embeddings for returns-to-go R t are also learnt and projected to the embedding dimension.
Finally, a positional embedding representing the environment time step t is concatenated to each input (returns, states & commands) after the embedding and GRU layers.
The embedded and positionally encoded command, state, and return-to-go inputs are fed into the GPT model.
Figure 1 shows the
Decision Transformer architecture with example input.
At each time step t , the Decision Transformer encoding x t is fed into four linear decoders.
Three decoders predict the next command’s action, modifier, and object components, while the fourth decoder predicts the answer to the question (this is the same at each time step).
Although we also use a separate QA module to predict the final answer, the answer decoder allows the Decision Transformer to learn some primitive level of question answering, thereby allowing the QA loss to help guide command generation.
Chen et al. (2021) found that predicting states and returns-to-go at each time step did not improve performance.
This motivates exclusively predicting command triples, along with answers to questions.
The model is trained to optimize the sum of the cross-entropy losses of action, modifier, object, and answer prediction.
For each question and map type configuration, a set of unique validation games were generated wherein an agent must interact with an environment to answer a question.
During training, the Decision Transformer is evaluated on the set of hold-out games every 250 iterations to monitor sufficient information scores and to avoid overfitting.
3. QA module
The QA module consists of a pretrained BERT encoder with a linear classification layer.
The per-time step state sequences are joined into a single long sequence of tokens with the question appended at the end.
A [CLS] token is added to the beginning of the sequence, and a [SEP] token before and after the question.
This concatenated sequence is tokenised by a Bert-Base-Uncased tokeniser (Devlin et al., 2019) and padded or fronttruncated (keeping the most recent part of the state sequence) to return a 512 token sequence which is then fed to the BERT encoder.
We subsequently pass BERT’s output vector corresponding to the
CLS token to a linear layer.
For attribute and existence questions this model performs binary classification (to predict “yes” or “no”), while for location questions it produces a softmax over the vocabulary.
We use cross-entropy to calculate the loss between predicted and correct answers.
Training and validation sets that consist entirely of valid trajectories are created for the QA module.
We use the validation set (20% of the generated trajectories) to save the model with the highest validation QA accuracy (after 30 training epochs).
In order to simulate more realistic QA training data, we feed the QA training examples back into the trained DT and use it to predict where to cut off the generated sequence, as during testing there is no guarantee that the DT would have explored up until the correct answer has been found.
The sequence is cut off when the DT predicts the stop action ( wait ) or the time step limit is exceeded.
3. Decoding
To generate the command sequence from the DT during testing, instead of greedy decoding we sample each next command from the probability distri- [Table] butions over the action, modifier, and object.
This motivation is similar to that of stochastic decoding algorithms in natural language generation: The stochasticity minimises the risk of the Decision
Transformer entering a loop in which the same command is generated repeatedly, and more closely mirrors natural language which avoids utterances with too high probability (Zarrieß et al., 2021).
In the case of answer prediction, we deterministically take the argmax of the output.
3. Returns Tuning
A shortcoming in the DT’s methodology is the expectation for the environment’s maximum achievable return to be known a priori .
Using an inappropriate value can greatly hamper performance, resulting in premature halting or needlessly excessive exploration.
Thus, we tune the value of the initial returns-to-go R 1 as a hyperparameter.
We create a validation set of 50 games to evaluate the question answering performance of both the DT’s answer prediction head and BERT model.
For each question and map type we consider R 1 either set as a fixed value or sampled from an exponential distribution.
Following the methodology used by Yuan et al.
for selecting the best model during training, we tune R 1 to maximize the sum of the sufficient information score and question answering accuracy.
See Appendix A for details.
Results
We evaluate the Decision Transformer both where its own answer prediction head is used for questions answering and where this is done with the
BERT QA model.
Test set results on the 500 games setting are given in Table 2, together with with RL model results as reported by Yuan et al.
We also report results of training the DT on reduced datasets with only 10,000 episodes, in order to further evaluate the sample efficiency of our approach (see section 4.5).
Training results are available in the Appendix in Tables 7 and 8.
Table 3 gives the BERT QA model’s validation accuracy.
We discuss the results for each question type.
Overall, DT-BERT outperforms the Decision
Transformer’s answer prediction head in location and attribute type questions, while the DT gives a higher accuracy on existence questions.
At a high level, these performance differences depend on the state and action space that the model was required to learn and navigate.
Existence type and attribute type questions may depend on long-range dependencies.
For example, existence type questions require the ability to know whether an object has been witnessed or not.
When the answer is that an object doesn’t exist, the problem is more than just word matching within the last few states.
We believe that this is why the Decision Transformer
QA head outperforms the BERT model on existence and attribute questions since it has access to the entire state trajectory.
On questions whose answers were more likely to be found within the last 512 tokens, DT-BERT achieves higher question answering accuracy than the Decision Transformer.
4. Location Questions
The DT’s answer prediction head has a lower QA accuracy than previous RL approaches on both fixed and random maps.
However its sufficient information scores, reflecting the DT’s information gathering capacities, are higher.
For location type questions, QA accuracy normally matches suffi-
[Table] [Table]
[Table]
cient information results due to QA modules effectively performing word matching once an agent arrives in the correct state.
We see this in the
RL methods’ results as their sufficient information scores are very close to their QA accuracies.
This indicates that the DT’s question answering prediction head is underfitting the training data.
DT-BERT outperforms the QA accuracy of the
RL models on both fixed and random maps.
On random maps, QA accuracy is slightly higher than sufficient information, suggesting that in a small number of cases the BERT model may be able to deduce the answer from the context even when it does not explicitly appears in the trajectory.
The performance gap between the BERT QA model and the DT means that a question can still be correctly answered even if the DT stops in an incorrect state.
The high BERT QA accuracy for location questions can also be seen in Table 3.
We suggest two reasons for the BERT QA model answering location type questions more accurately than the Decision Transformer’s prediction head.
First, it is easier for BERT model to learns skills basic pattern matching skills such as identifying entity and location names from state strings.
Second, exploration is not as highly encouraged with location questions as with existence and attribute types.
Less exploration means fewer states visited, allowing the state context window to contain less noisy state strings than other question types.
4. Existence Questions
The DT outperforms RL baselines on sufficient information and QA accuracy in the random maps setting for existence questions.
However on fixed maps it performs worse than the DQN.
The BERT
QA model underperforms the DT answer prediction head here in both map types, suggesting that jointly optimising answer and command prediction leads to improved performance on existence type questions.
Reasoning about the existence of an object within a TextWorld environment requires knowledge about the entirety of the world.
Therefore, existential questions require an agent to fully explore an environment to answer whether or not an entity exists within it.
The Decision Transformer’s selfattention mechanism makes performing long-term credit assignments possible.
The answer prediction head of the DT can thus draw upon information gathered in all previous states to inform question answering.
As a result, the ability to model dependencies that stretch throughout all states encountered allows the DT to outperform the BERT model, whose context window is constrained to 512 tokens.
4. Attribute Questions
None of the models achieve results that are substantially above 50% on attribute questions, confirming the challenge of this question type.
The
Decision Transformer did obtain higher sufficient information than all RL baselines.
DT-BERT obtains higher QA accuracies than the DT answer prediction head; it obtains the highest QA accuracy among all the models on random maps, and performs slightly worse than the DQN on fixed maps.
Despite the Decision Transformer’s ability to learn long-term dependencies via its attention mechanism, we posit that the contextualised embeddings of BERT are able to model a richer semantic representation of TextWorld’s state-strings than the embeddings learnt by the DT.
This better capturing of the semantic space enables BERT to more fully utilise the context with which it was provided by using pre-existing understanding to help answer questions posed in natural language.
4. Rewards and Performance
Based on validation set performance, the optimal initial return-to-go for location type questions was determined to be 2.0 for both fixed and random map settings.
This is lower than for existence and attribute types, indicating that exploration is not as highly encouraged.
In location type questions, the entity definitively exists somewhere within the environment.
This means that the action space required to answer questions of locality is reduced to traversals and basic interactions with containers.
Therefore, less exploration is needed as the information to answer a question is more easily ac- quired.
Too high an initial reward would promote unnecessary actions with a high likelihood of leading the agent astray from stopping in the correct state.
Existence questions require far more exploration of an environment than location type questions.
Higher starting rewards reflect this need for greater exploration and are associated with better QA and sufficient information scores, as seen in Tablein the Appendix.
These higher values promote a more complete traversal of the world, allowing for gathering information required to answer the question.
However, too high an initial reward means that entering a correct state and receiving a reward of 1.0 may not affect the model’s decision making.
If the DT has a current RTG of 5.0 and enters the correct state that rewards 1.0, the RTG from then onwards is 4.0.
The return-to-go of 4.0 does not suggest to the model that it has entered the correct state, meaning it carries on exploring and gathering information.
Likewise, too small a reward could prematurely cause an agent to stop exploring due to gaining rewards for entering new states via the exploration bonus.
Therefore, we observe that the best RTG values err on the larger side, which encourages greater world exploration.
Attribute type questions are considered the most sparsely rewarded of all three types (Yuan et al., 2019).
We therefore expected higher rewards to be associated with better accuracies.
The results, however, paint a different picture.
In a fixed map, where the state space is, on average, smaller than that of random maps, we see that a smaller reward yields the best score.
This reduction is likely a result of the reduced state and action space making too much exploration and interaction with the environment degrade performance.
On the other hand, in a random map setting higher rewards yields better QA and sufficient information scores, allowing us to conclude that higher rewards promote more exploration and thus allows the model to better answer the question.
4. Sample Efficiency
The RL agents in QAit were trained for more than 200K episodes.
In comparison, most of our Decision Transformers were trained on around 40K episodes (Table 1).
The test set results therefore show that DT is able to match or outperform the previous RL methods when trained on approximately 25% of the number of episodes.
Moreover, all training data used for the DT was generated via random rollouts - indicating that the Decision Transformer has the ability to learn optimal policies from suboptimal data.
We also found that fine-tuning a BERT model for QA on the random rollout data works well, as long as the DT is used to determine where to cut off the trajectory.
In order to further elucidate the DT’s sample efficient learning capabilities, we generated new datasets for all question and map types that only contained 10 thousand episodes.
The validation results can be seen Table 6 in the Appendix.
These experiments indicate that the DT trained on even fewer offline trajectories can achieve results on par with or better than both previous baselines as well as identical models trained on more data.
Here we see fixed map sufficient information scores being improved for all question types and QA accuracy increasing for attribute and existence questions.
However, QA accuracy for location type questions is worse than previous baselines in both random and fixed maps (see Table 2).
While the results are not consistently better, they do further indicate the sample efficiency of the Decision Transformer.
Conclusion
We showed that interactive question answering can be framed as a sequence modelling problem by training Transformers for action generation and answer prediction using random roll-outs.
Results show that the Decision Transformer approach matches or outperforms current reinforcement learning approaches for QAit on most question types and maps type configurations in thegame setting.
Additionally, the approach is more sample efficient than reinforcement learning approaches, reducing the amount of training data required even though the data generated via random rollouts is suboptimal.
Fine-tuning a BERT model for question answering on the same generated dataset improves performance over using the
Decision Transformer directly for question answering in two of the three question types.
Acknowledgements
This work is based on research supported in part by the National Research Foundation of South Africa (Grant Number: 129850).
Some computations were performed using facilities provided by the University of Cape Town’s ICTS High Performance
Computing.
A Hyperparameter Tuning
Hyperparameters of the DT and BERT are given in
Table 4.
DT
BERT
Hyperparameters
Value
Number of layers
Number of attention heads
Embedding dimension
Batch size
State context window tokens
Context length
[Table]
K) -
Max Epochs
Dropout
[Table]
[Table]
[Table]
Learning rate
[Table]
Grad
[Table]
Weight decay
[Table]
A. Decision Transformer A.1. Location
As can be seen in Table 5, the sufficient information score peaking at R 1 = 2 indicates optimal statespace exploration for location questions when the potential for future reward is moderate for random and fixed map types.
While the QA accuracy was highest for both settings when the initial reward was the maximum of the training set, we opted to test the DT
’s question answering and information gathering capabilities at R 1 = 2 as this yielded the highest combined sufficient information and QA accuracy score.
A.1. Existence
Using both sufficient information and QA accuracy, the optimal initial reward for fixed map existence questions was determined to be 4.0, with the DT achieving a QA accuracy of 0.660 and a sufficient information score of 0.263 on the validation set.
In random map settings, the DT scored a validation accuracy of 0.720 with a corresponding sufficient information score of 0.298, where the initial reward was determined to be the maximum of the training set 3 . 94 .
A.1. Attribute
The best sufficient information and QA accuracy combinations for the Decision Transformer were achieved at an initial reward of 2.0 for fixed and 5.0 for random map types.
On the validation set, the fixed map DT achieved a QA accuracy of 0.
and a SI score of 0.056.
Random map saw a similar
SI of 0.057 but worse QA accuracy of 0.460.
A. BERT Model A.2. Location
Based on data gathered using the online-evaluation dataset, the optimal initial return-to-go for location type questions was 2.0.
Using the BERT model for
QA yielded an accuracy of 0.227 for fixed maps and 0.393 for random maps.
The BERT model achieved a higher QA accuracy than sufficient information score during evaluation, indicating that the context window spanning multiple states was a boon to QA accuracy.
During training, the BERT model achieved almost perfect scores for questionanswering on the held-out set of offline trajectories, seen in Table 3.
A.2. Existence
Using the online-validation set, we determined optimal starting reward values of 3.0 for fixed map and 3.94 for random.
These values were associated with a QA accuracy of 0.64 for fixed and 0.647 for random map types.
However, scores were significantly lower than the offline validation set used during training, where QA accuracy of 0.778 and 0.779 was achieved for fixed and random maps, respectively.
A.2. Attribute
In the offline validation set, the BERT model scored a QA accuracy of 0.616 for random and 0.780 for fixed map settings.
On the online validation set, we observed the maximum combination of QA and sufficient information for the BERT model at an
R 1 of 3.0 for fixed and 2.0 for random where the
BERT QA model had an accuracy of 0.507 and 0.660 for random and fixed map types, respectively.
However, we opted to use the maximum of the train set 4 . 03 when evaluating on the test set for random map types.
This is due to the BERT QA model having a high standard deviation of 0.156 and an average QA accuracy of 0.640, indicating greater potential for high QA accuracy.
Moreover, the sufficient information score associated with this accuracy is 0.056 - higher than the random map with an initial reward of 2.0.
[Table]
as well as the Decision Transformer’s average sufficient information (SI) score on validation set at different initial return-to-go (RTG) values.
Bold values indicate the combined highest QA and sufficient information score with the associated initial RTG value also bolded.
Sampling indicates R 1 was randomly sampled from an exponential distribution.
Max represents the maximum of the training set for that experiment configuration (see Table 1).
Summary statistics were calculated over 4 seeds - see code implementation for details.
[Table]
[Table] [Table]