Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning
Abstract
Recurrent neural networks have a strong inductive bias towards learning temporally compressed representations, as the entire history of a sequence is representedby a single vector.
By contrast, Transformers have little inductive bias towardslearning temporally compressed representations, as they allow for attention overall previously computed elements in a sequence.
Having a more compressed representation of a sequence may be beneﬁcial for generalization, as a high-levelrepresentation may be more easily re-used and re-purposed and will contain fewerirrelevant details.
At the same time, excessive compression of representationscomes at the cost of expressiveness.
We propose a solution which divides computation into two streams.
A slow stream that is recurrent in nature aims to learn aspecialized and compressed representation, by forcing chunks of K time steps intoa single representation which is divided into multiple vectors.
At the same time,a fast stream is parameterized as a Transformer to process chunks consisting of
K time-steps conditioned on the information in the slow-stream.
In the proposedapproach we hope to gain the expressiveness of the Transformer, while encouraging better compression and structuring of representations in the slow stream.
Weshow the beneﬁts of the proposed method in terms of improved sample efﬁciencyand generalization performance as compared to various competitive baselines forvisual perception and sequential decision making tasks.
Introduction
The interplay between fast and slow mechanisms for information processing and perception has beenstudied in both cognitive science and machine learning [5, 33].
In the brain, short-term and long-termmemory have developed in a specialized way.
Short-term memory is allowed to change very quicklyto react to immediate sensory inputs and perception.
It also tends towards high capacity storage ofall pieces of information which may be relevant for future reasoning [40, 3, 4].
By contrast, longterm memory changes slowly [41, 39], is highly selective and involves repeated consolidation.
Itcontains a set of memories that summarize the entire past, only storing details about observationswhich are most relevant [25, 6].
Deep Learning has seen a variety of architectures for processing sequential data [34, 54, 16].
Forexample.
recurrent neural networks compress information about a sequence into a single hiddenstate. Transformers get rid of the recurrent state by dynamically capturing information betweenpositions using multi-head dot product attention[56].
Transformers [56] have become the dominantarchitecture across a wide range of domains including vision [22], natural language [21, 51, 10,62, 17, 52], and reinforcement learning [12, 38]. They have eclipsed recurrent neural networks[34, 54, 16] in almost all sequence processing domains due to their high representational capacityand scalability.
Despite their wide applicability, it is well known that Transformers are very datahungry and work well mainly at scale. This can be attributed to their inductive bias towards modeling [Figure] r
[Figure]
tt e n t i o
[Figure]
N
[Figure] all possible pairwise interactions in the sequence which results in no consolidation of information. This lack of selectivity in the attention mechanism also leads to a high computational complexitywhich scales quadratically with input size.
Additionally, modeling all possible pairwise interactionsmaybe extremely wasteful and may result in capturing unnecessary information not useful for thedownstream task [26, 37].
The goal of this work is to design an architecture for autoregressivemodeling that has an inductive bias towards learning temporally compressed representation thatretains the beneﬁts of Transformers while preserving long-range interactions. For learning temporally compressed representations, we start by dividing the computation of the
Transformer into two streams of processing - a fast stream and a slow stream.
Inspired by the ideaof long-term and short-term memory, we want the fast stream to have a short-term memory witha high capacity that reacts quickly to sensory input. We refer to this fast stream as the perceptualmodule and implement it using a Transformer since they are known to have high representationalcapacity.
On the other hand, we want the slow stream to have a long-term memory which updatesat a slower rate and summarizes the most important information in the input sequence. We refer tothis slow stream as the Temporal Latent Bottleneck. Implementation-wise, we divide the input into ﬁxed size chunks (Figure 1).
The fast stream operateswithin each chunk while the slow stream consolidates and aggregates information across chunks updating itself once per chunk. This leads to information asymmetry between fast and slow stream asthe fast stream contains ﬁne-grained local information while the slow stream contains coarse-graineddistant information.
Such kind of information asymmetry has shown to improve generalization andadaptation performance of learned policies in the context of RL [27, 24]. The fast and slow streamsinteract with each other though bottleneck of attention. The division of computation into a fast andslow stream eliminates the need for capturing all possible pairwise interactions and thus introducingselectivity in the attention mechanism resulting in a much lower computational complexity which isnot quadratic in the input size. We show that the limited capacity of the slow stream and consolidation of information by a recurrent neural network prevents the model from capturing unnecessaryinformation not useful for the downstream task.
We evaluate the proposed model in a number ofdomains showing that it consistently outperforms competent baselines showing improved generalization to scenarios not seen during training.
Methodology
We now present the proposed approach in detail.
Our model jointly leverages the strengths of Transformers [56] and recurrent neural networks [16, 34].
2.1 Desiderata for Fast and Slow Streams of Processing
We give the detailed description of the proposed model in the next section.
Here, we give anoverview of our architecture and discuss some of its key properties.
Given an input sequence, itis ﬁrst divided into chunks of size K .
Each chunk is processed by perceptual module represented bya Transformer (denoted as F ).
While processing each chunk, F is also conditioned on information from the Temporal Latent Bottleneck module G .
The slow stream is a recurrent stream which hasits own state consisting of a set of N vectors (or slots) also called temporal latent bottleneck statedenoted as I in Figure 1.
In the following sections, we use the term temporal latent bottleneck to refer to the temporal latent bottleneck state I .
This state is updated once per chunk using informationfrom the perceptual module through a cross attention mechanism.
The perceptual module operates within each chunk while the temporal latent bottleneck operatesacross chunks slowly updating itself after each chunk has been processed by the perceptual module.
Thus, the only way the perceptual module gets information about inputs beyond its own chunkis through the temporal latent bottleneck.
An added advantage of this is that the computationalcomplexity of the attention mechanism in the proposed model is O ( TK ( K 2 + KN )) while that ofa Transformer is O ( T 2 ) , where T is the length of the sequence, K is the chunk size, and N isthe number of temporal latent bottleneck state vectors.
Since K << T and N << T , we cansee that TK ( K 2 + KN ) < T 2 .
Therefore the proposed model has a much lower computationalcomplexity compared to a Transformer.
Furthermore, the capacity of the temporal latent bottleneckis limited and much smaller than that of the perceptual module.
This encourages the temporal latentbottleneck to represent the most salient information about the past while the perceptual modulerepresents only local information.
This creates an information asymmetry between the two streams.
This information asymmetry leads to the perceptual module having a ﬁne grained view of the nearbyinputs but a very coarse grained view of the distant past.
This is very different from the usual selfattention which attends to all tokens in the sequence at the same level of granularity.
An advantage of having a compressed representation of the past is that it allows the model to forgetirrelevant information.
For example, if an agent is navigating in a large maze, it does not need to haveﬁne grained knowledge of its actions from the distant past.
In the case of a Transformer, it wouldattend to every step from the past (including steps from the distant past) which may be irrelevantin the present context thus wasting its capacity in modeling irrelevant details.
Another importantcomponent of the proposed model is top-down attention which conveys contextual information fromthe high-level Temporal Latent Bottleneck module to the processing of low-level perceptual module.
Past works [49, 23, 32, 20] have shown that top-down attention improves generalization and adaptation performance of the learned model. One difference between these works and the proposedmodel is that in their case the multiple streams operate at the same temporal granularity while inour case the streams operate at a different time scales (because of information asymmetry.
Throughour experiments, we show the advantage of the proposed architecture over these works.
Next, wedescribe the detailed implementation of the proposed model.
2.2 Computational Steps
We denote the input X as a sequence of T tokens - X = [ x 0 , x 1 , x 2 , . .
, x t ] .
We chunk this inputinto chunks of size K resulting in (cid:98) T/K (cid:99) chunks.
We refer to l th chunk as X l .
We represent thestate of the temporal latent bottleneck I (i.e. the slow stream) as a set of M d -dimensional vectors.
As mentioned previously, we denote the temporal latent bottleneck module as G and the perceptualmodule as F .
G updates the temporal latent bottleneck state while F processes chunks X l to formthe latent representation ¯ X lPerceptual Module
[Equation]
Temporal Latent Bottleneck Module I
[Equation]
Preliminaries .
The central components of our model are the key value attention mechanism [7, 56]and the FFN module [56].
We use two forms of the attention mechanism -(1) Self Attention [56]: Inthis the query and key vectors refer to the same vector; (2) Cross Attention [26, 37, 28]:
In this thequery and key vectors refer to seperate vectors.
Perceptual Module F .
As mentioned previously, the perceptual module refers to the fast stream thatacts directly on the input.
The perceptual module operates on each chunk separately.
Therefore, atany time the input to the perceptual module are the tokens corresponding to a particular chunk X l =[ x l × K , x l × K +1 , . . .
, x l × K + K ] .
The perceptual module is a Transformer with self attention layers,cross attention layers, and FFNs.
It has 2 kinds of layers - (1) SELF ATTENTION + FFN; (2) CROSS ATTENTION + FFN.
The SELF ATTENTION + FFN layers process the input tokens and the CROSS ATTENTION + FFN layers integrate top-down information from the temporal latent bottleneck state
I as follows -
[Equation]
We include one CROSS ATTENTION + FFN layer per R SELF ATTENTION + FFN layers.
Thediagramatic representation of the perceptual module is presented in Figure 1 (in the processing ofchunk X l ).
In the ﬁgure, we set R = 1 .
Temporal Latent Bottleneck Module G .
The temporal latent bottleneck (TLB) module representsthe slow stream that operates on the temporal latent bottleneck state I .
I is updated using information from a particular chunk processed by the perceptual module.
This update happens once foreach chunk of the perceptual module resulting in (cid:98) T/K (cid:99) updates for I .
Since the temporal latentbottleneck state I updates at a lower frequency than the perceptual module, it is expected to capture more stable and slowly changing features while the perceptual module captures faster changingfeatures resulting in multiple scales of information representation.
An update to the temporal latent bottleneck state I consists of a cross attention operation where the queries come from I andthe keys and values come from the output of the perceptual module.
This cross attention operationis followed by an FFN update to I .
Consider the perceptual module outputs for a chunk l to be¯ X l = [¯ x l × K , . .
, ¯ x l × K + K ] .
The update operation is implemented as follows:
[Equation]
141 The temporal latent bottleneck module introduces the notion of recurrence in our model.
We showthe details of this module in Figure 1 (inside the circle).
Perceptual Module + Temporal Latent Bottleneck Model.
We now present our complete architecture integrating both the perceptual module and the temporal latent bottleneck together.
Given asequence of tokens X
=
[ x 0 , x 1 , x 2 , . .
, x t ] .
We chunk this input into chunks of size K resultingin (cid:98) T/K (cid:99) chunks.
The chunks are processed sequentially one after the other.
For a chunk k , itis ﬁrst processed using the perceptual module conditioned on information from the temporal latentbottleneck state.
The outputs of the chunk are used to update the temporal latent bottleneck state
I .
The resultant temporal latent bottleneck state is then used to process the next chunk.
The fullmodel is presented in Figure 1.
We use a Transformer as the perceptual module in our experiments.
Thus our main contribution is introducing a temporal latent bottleneck into Transformers and showing its advantages through a variety of experiments.
We also present the detailed algorithm for theproposed approach in the Appendix (Algorithm 1).
The proposed model is similar to a parallel work called Block Recurrent Transformers [35].
Thereare few differences between our work and theirs.
First, they use a sliding window attention, whilewe divide the input into chunks.
In their paper, they perform cross attention and self attention inparallel while we ﬁnd that doing them sequentially and performing cross attention once per R selfattention steps yields better results.
They also use special tricks to deal with some instabilities intheir case, while we ﬁnd no such instabilities in our model.
Also, while their focus is mainly onnatural language tasks, we focus on a broader variety of tasks.
{+We defer the rest of the discussionon related works to Appendix Section 
+}
Experiments
Our goal is to show the wide applicability and beneﬁts offered by the temporal latent bottleneck ,which we refer to as TLB.
We demonstrate that the proposed model outperforms competitive baselines across many domains including vision, reinforcement learning, and natural language.
Ourmain goal is to show that the proposed approach has high expressive power like Transformers whilealso being sample efﬁcient unlike Transformers.
Thus our main baselines are based on the original Transformer architecture.
For example, we compare against ViT [22] in image classiﬁcation, Decision Transformer [12] in Reinforcement Learning, and Vanilla Transformer in rest of the tasks.
We also compare against some representative baseline that offer some of the key properties that ourmodel offers.
For example, we compare against state-of-the art Swin Transformer [46] which is astrong baseline for image classiﬁcation and is also hierarchical similar to the proposed model.
We also compare against Transformer LS [63] which also processes long-term and short-term information using different attention streams.
Furthermore, we also compare against Transformer-XL,which also divides the input into segments similar to the proposed model.
Another key point ofthe proposed model is that any position cannot attend to any information from the future beyond itschunk since the temporal latent bottleneck only summarizes the past, not the future .
Meanwhile, allthe baselines we consider have bidirectional context i.e. they can attend to all of the past and thefuture.
We observe that despite this limitation, the proposed model outperforms all the consideredbaselines.
3.1 Temporal Latent Bottleneck For Perception
[Table]
Image Classiﬁcation.
Recently, Transformershave been widely applied for visual perceptionand have shown strong performance improvements over CNNs in tasks such as image classiﬁcation, semantic segmentation, instance segmentation, etc.
In this work we focus on imageclassiﬁcation using Transformers.
For a modelto do well on image classiﬁcation, it shouldlearn to only focus on the relevant informationand ignore other details (eg. background information).
Self attention does not inherently havethis inductive bias of ignoring irrelevant information since it models all pairwise interactionsbetween the inputs.
We posit that adding a limited bandwidth temporal latent bottleneck into the Transformer will allow the model to focus only on the most important information in the imagewhich should enable the model to perform well. [Table] Results .
We test our hypothesis on the CIFAR10 and CIFAR100 [43] image classiﬁcation datasets.
We also test the generalizationabilities of the models by comparing their performance on images of higher resolution ( 128 ×128 ) than seen during training ( 64 × 64 ).
We use ViT [22] and Swin Transformer V2 (denoted as Swin V2)
[46] as our baselines.
Swin Transformer V2 has a key strength of generalizing tohigher resolution images than those seen during training, making it a strong baseline.
Theinput image is split into patches of size 4 × 4and fed in rastor order to all the models.
Forthe proposed model we use ViT as the perceptual module and add a temporal latent bottleneck module to it.
We call this model V I T + TLB.
To predict the classiﬁcation scores, we take themean across the ﬁnal temporal latent bottleneck state vectors and pass the resulting representationthrough an MLP.
We present the results for this experiment in table 1.
We can see that V I T + TLBoutperforms V I T for all cases and performs competitively to Swin Transformer V2 .
For furtherhyperparameter details, we refer the reader to Appendix section 7.1.
Quantitative Analysis .
One essential component of our model is top-down conditioning.
Top downinformation helps in integrating information from the past as well as high-level information into theperceptual module.
We hypothesize that both these kinds of information are important for the modelto perform well. To test this, we design two baselines - (1) V I T + TLB
(N O P AST I NFO ): Inthis baseline, we do not allow the TLB to communicate to the perceptual module, therefore theperceptual module has no information about the past; (2) V I T + TLB
(N O T OP -D OWN C ONDN ): In this baseline, we have a separate temporal latent bottleneck module at every layer, thereforethe perceptual module has access to past information but does not have access to any high-levelinformation through top-down feedback.
We show the results for this ablation in Table 2.
We cansee that the performance of both the baselines perform worse than the proposed V I T + TLB model.
This shows that both high-level information through top-down feedback and information from thepast is important for the model to perform well. tleneck pays most attention to. To do the attention scores Qualitative Analysis .
To get a better understanding ofwhat the temporal latent bottleneck is doing, we visualize the parts of the image where the temporal latent bottleneck pays most attention while it is being updated bythe perceptual module.
We present this visualization in
Figure 2.
We can see that the temporal latent bottlenecklearns to pay the most attention to the foreground in eachcase.
This further conﬁrms our hypothesis that the limitedcapacity bottleneck focuses on the most important information required to solve the downstream task.
Self Supervised Learning.
Many recent works haveused vision Transformers for self-supervised learning[8, 2, 30, 11, 45, 44].
Here we show a proof-of-conceptthat introducing a temporal latent bottleneck in Vision Transformers results in better self-supervised representations.
We consider the SiT model from [2] for this experiment.
They use 3 objectives to pretrain their model -(1) The Reconstruction Objective - Reconstructs the inputimage, (2) The Rotation Prediction Objective - Predictsthe rotation angle from [ 0 ◦ , 90 ◦ , 180 ◦ , 270 ◦ ], and (3)
The
Constrastive Objective (similar to SimCLR [13]).
For the proposed approach, we introduce a temporal latent bottleneck into SiT resulting in the SiT + TLB model.
SiT also uses additional trainablecontrastive and rotation tokens as input for calculating the contrastive and rotation objectives respectively.
For SiT + TLB, we take the mean across the temporal latent bottleneck state vectorsand use the resulting representation for computing the rotation and contrastive objectives.
We use achunk length of 20 for the SiT + TLB model.
We pretrain the model for 400 epochs and evaluatethe pretrained model at different epochs using linear probing.
Results . To evaluate the model, we pretrain themodel on the STL10 dataset [19] and evaluatethe learned representation using linear probingon the CIFAR10 dataset [43].
We present theresults for this experiment in Figure 3.
We cansee that the proposed approach outperforms SiTthus showing the effectiveness of the proposedarchitecture for self-supervised learning.
Foradditional experimental results and details, werefer the reader to Appendix section 7.2.
3.2 Temporal LatentBottleneck for Sequential Decision Making
Transformers have recently been used forsequential decision making in reinforcementlearning tasks such as Atari and BabyAI [12,36].
These works deploy Transformers in theofﬂine RL setting where a large number of trajectories are available either through another trained agent or an expert agent.
The Transformer istrained as an autoregressive generative model that predicts actions conditioned on the past context.
We incorporate the temporal latent bottleneck module into the Transformer and explore its beneﬁtsin the RL setting.
We test the proposed model in the BabyAI [14] and Atari [9] benchmarks.
Wedescribe our setups in detail below.
Instruction Based Decision Making: BabyAI.
BabyAI [14] provides a suite of environmentswhere the agent has to carry out a given instruction in a partially-observable maze.
These instructions include competencies such as going to an object in the maze, placing an object beside anotherobject in the maze, opening a door with a key, etc.
Some environments in the benchmark containinstructions that combine multiple competencies sequentially.
For example, pick up a red ball and in Baby benchmark has a different type of instruction that tests a differentcompetency.
the most complicated environment that contains instructions from all [Figure] competencies.
For more details regarding the various environments from the BabyAI benchmark, we refer the reader to Appendix section 7.4.
We train our models with behavior cloning using expert trajectories from an oracle.
For evaluation,we test the model by directly deploying it in the environment.
We report the success rate which measures whether the agent successfully carried out the given instruction or not.
We use a Transformer[56] as the baseline in these experiments.
For the proposed model, we introduce a temporal latentbottleneck into the Transformer-based perceptual module.
For the baseline Transformer model, weappend the language instruction to the sequence of states allowing the model to attend to the language instruction at each layer.
For the proposed model, the language instruction is appended toeach chunk, allowing each chunk to attend to it. Results .
We consider two settings - Single task and Multi task .
In the single task setting, weevaluate the proposed approach on individual environments from the BabyAI benchmark while inthe multi-task setting we train a single model on 8 different environments. improves performance across various atari games. Results are averaged across 10 seeds. [Table]
Single Task.
We present the results for BossLevel in Figure 4 (left) and present the results for the other tasks in Appendix Figure 8.
We can see that while Transformer and Transformer + TLB achieve almost similar performance at convergence.
However, Transformer+ TLB is much more sample efﬁcient, converging much faster .
We posit that the temporallatent bottleneck module prohibits the modelfrom paying attention to unnecessary information which allows it to converge faster.
Multi Task.
We present the results for the multi task setting in Figure 4 (right).
We train the modelon 8 environment - PutNext, Unlock, Synth, GoToSeq, SynthLoc, GoToImpUnlock, BossLevel.
We evaluate the model on the same 8 environments.
We report the average success rate across 8games.
We can see that the Transformer + TLB model converges faster and also outperforms the
Transformer .
We refer the reader to the appendix for more details regarding the model and training.
Atari.
[12] recently introduced the Decision Transformer (DT) which learns to play various gamesin the Atari benchmark from suboptimal trajectories of a learned agent.
Decision Transformer models the ofﬂine RL problem as a conditional sequence modelling task.
The model uses a causal maskand supervised training to match the actions in the ofﬂine dataset conditioned on the future expectedreturns and the past history.
This is done by feeding into the model the states, actions, and thereturn-to-go ˆ R c = (cid:80) Cc (cid:48)
=
c r c , where c denotes the timesteps.
This results in the following trajectory representation: τ = (cid:0)
ˆ R 1 , s 1 , a 1 , ˆ R 2 , s 2 , a 2 , ˆ R 3 , s 3 , a 3 , . . .
(cid:1) , where a c denotes the actions ands c denotes the states.
At test time, the start state s 1 and desired return ˆ R 1 is fed into the model and it autoregressively generates the rest of the trajectory.
Experimental results show that DT canleverage the strong generalization capabilities of Transformers and achieve the desired returns in awide variety of tasks in Atari and OpenAI Gym, outperforming previous approaches in ofﬂine RL.
[Table] We use the same setup as used in [12] forour experiments.
We set the context lengthto a ﬁxed number C .
During training, Ctimesteps from an episode are sampled and fedinto the model resulting in a trajectory of length3 C (considering 3 modalities - returns-to-go,states, and actions).
Each modality is processedinto an embedding of size d .
The state is processed using a convolutional encoder into anembedding of size d .
The resulting trajectoryis fed into the decision Transformer.
The outputs corresponding to the states s c are fed intoa linear layer to predict the action a c to be takenat timestep c .
For the proposed model, we incorporate a temporal latent bottleneck moduleinto the decision Transformer.
Results .
We present our results in Table 3.
The model is trained on 1% of the Atari DQN-replaydataset [1] (500K transitions for each game).
We use the same 4 games used in [12]: Pong, Seaquest, Qbert, and Breakout.
We can see that the proposed model outperforms decision Transformer in allthe considered games thus showing the effectiveness of the proposed model .
More details regardingthe model and training can be found in the appendix section 7.5.
3.3 Temporal Latent Bottleneck for Long Range Dependencies
Here, we test the effectiveness of the proposed model in modelling long range dependencies.
Weapply the proposed model on the ListOps and text classiﬁcation tasks from the Long Range Arena(LRA) benchmark [55].
Both these tasks have very long sequences ranging from 1K to 4K tokens.
Thus, for a model to do well, it has to learn to capture dependencies across very long time scales.
Additionally, all these tasks have an inherent hierarchical structure.
For example, Listops consistsof nested list operations which makes it hierarchical.
For text classiﬁcation, the inputs consist oftext in the form of bytes.
Therefore, the model has to learn to compose bytes into characters andcharacters into words.
We hypothesize that the multi-scale nature of the proposed model will beextremely useful in modelling such hierarchical information.
The temporal latent bottleneck whichoperates at a slow scale can behave as a composer that composes low-level information in relevantway to solve the desired task. Results .
For this experiment, we use the same setup as[63].
For the proposed model, we use a Transformer as the perceptual model and implement the temporal latent bottleneck as describedin Section 2.2.
We take the mean across the temporal latent bottleneck state vectors and use theresulting representation for classiﬁcation.
We compare the model against the long-short Transformer(LS) model [63], which is a recently proposed model for the long range arena benchmark, and thevanilla Transformer model [56].
We present the results in Table 4.
We can see that the proposedmodel outperforms both the baselines in both the tasks thus showing its usefulness in modeling longrange dependencies .
Furthermore, for text classiﬁcation, we outperform the state-of-the-art model
S4 model [29] which achieves a performance of 76.02%.
For further details, we refer the reader to Appendix section 7.3. Analysis .
Here we perform an ablation to show that the Temporal Latent Bottleneck does not onlycontain short-term information but also summarizes information from long term past.
To test thishypothesis we design a baseline in which the current chunk attends to the previous few chunksinstead of attending to the temporal latent bottleneck.
We ﬁnd that this baseline achieves a performance of 32.10 ± 0.019 % compared to the proposed models 38.2 ± 0.0001 % on the ListOps task.
Thisshows that the Temporal Latent Bottleneck contains information about the long-term past.
Additionally, here also we perform an experiment to probe the importance of top-down communication (i.e.high level to low level feedback).
To do this we use the same Transformer + TLB (No Top-Down
Condn) baseline used in Table 2.
We ﬁnd that this baseline achieves a performance of 37.57 ± 0.003 %
Sequence Feedback Transformer +
[Table]
[Table]
which is lower than the performance of the proposed Transformer + TLB model which achieves38.2 ± 0.0001 % which further shows that top-down information from high-level to low-level is important for the model to perform well.
We perform additional experiments to give us more insight into the behavior of the proposed model.
We present these experiments in Appendix Section 7.3.
In Tables 6 and 7 we analyse the effectof the chunk size and the number of state vectors on the performance of the model on the ListOpstask.
We ﬁnd that for both cases, the model has an optimal number above or below which theperformance drops.
We explain our ﬁndings in more detail in Appendix Section 7.3.
We also plotthe sample efﬁciency and convergence curves for ListOps and Text Classiﬁcation respectively in
Appendix Figure 7.
We ﬁnd that the proposed model is more sample efﬁcient than the baseline
Transformer LS for the ListOps task.
Temporal Latent Bottleneck for Copying Task.
Here, we study the copying task used in [34].
Inthe copying task, the model receives a sequence of 10 digits followed by blank inputs for a largenumber of steps, and then the model is asked to output the sequence of digits it received initially.
Therefore, the model has to remember the original sequence of digits across long time scales.
Wecan control the sequence length of this task by controlling the length of the blank input.
The main motive behind studying this task is comparing the model to the Feedback Transformermodel introduced in [23] which also has top-down attention similar to the proposed model but doesnot represent information at multiple scales.
We compare both the models on the copying taskfor sequence lengths 100, 200, 300, 400, 500, and 600.
We present the results for this task in
Figure 5.
We can see that while both Transformer + TLB and Feedback Transform perform well forlow sequence lengths, the performance of Feedback Transformer drops for longer sequence lengthsabove 400 while the proposed Transformer + TLB model still achieves perfect accuracy at longsequence lengths.
We also compare the sample efﬁciencies to achieve perfect accuracy for both themodels.
We present this result in Table 5.
We can see that the proposed Transformer + TLB is moresample effecient than the baseline Feedback Transformer achieving perfect accuracy in much lessernumber of samples in each case.
For further details we refer the reader to Appendix Section 7.6.
Conclusion
We have developed an approach aimed at introducing selectivity in the interactions across time-stepsin a transformer by splitting processing into two streams: (a) a slow stream that is updated in a recurrent manner and (b) a fast stream that processes the visual input.
The two streams are parameterizedindependently and interact with each other via attentional bottleneck.
The information processed bythe fast stream is used to change the state of the slow stream, and the information in the slow streamis used by the fast stream as contextual information to process the input.
Through our experimentswe show that the proposed approach works well across wide range of domains and problems.
Onelimitation of the proposed model is that the chunk size is ﬁxed and treated as a hyperparameter whichrequires some domain knowledge.
Future work should explore methods for dynamic chunking.