Original: Toward Semantic History Compression for Reinforcement Learning
Revised: Toward Semantic History Compression for Reinforcement Learning

Original: Anonymous Author(s)

Original: Abstract
Revised: Abstract

Original: Agents interacting under partial observability require access to past observationsvia a memory mechanism in order to approximate the true state of the environment.
Revised: Agents interacting under partial observability require access to past observations via a memory mechanism in order to approximate the true state of the environment.

Original: Recent work suggests that leveraging language as abstraction provides benefitsfor creating a representation of past events.
Revised: Recent work suggests that leveraging language as abstraction provides benefits for creating a representation of past events.

Original: History Compression via Language Models (HELM) leverages a pretrained Language Model (LM) for representingthe past.
Revised: History Compression via Language Models (HELM) leverages a pretrained Language Model (LM) for representing the past.

Original: It relies on a randomized attention mechanism to translate environmentobservations to token embeddings.
Revised: It relies on a randomized attention mechanism to translate environment observations to token embeddings.

Original: In this work, we show that the representationsresulting from this attention mechanism can collapse under certain conditions.
Revised: In this work, we show that the representations resulting from this attention mechanism can collapse under certain conditions.

Original: This causes blindness of the agent to certain subtleties in the environment.
Revised: This causes blindness of the agent to subtle changes in the environment that may be crucial for solving a certain task.

Original: Wepropose a solution to this problem consisting of two parts.
Revised: We propose a solution to this problem consisting of two parts.

Original: First, we improve upon HELM by substituting the attention mechanism with a feature-wise centering-andscaling operation.
Revised: First, we improve upon HELM by substituting the attention mechanism with a feature-wise centering-and-scaling operation.

Original: Second, we take a step toward semantic history compressionby encoding the observations with a pretrained multimodal model such as CLIP,which further improves performance.
Revised: Second, we take a step toward semantic history compression by leveraging foundation models, such as CLIP, to encode observations, which further improves performance.

Original: With these improvements our model is ableto solve the challenging MiniGrid-Memory environment.
Revised: By combining foundation models, our agent is able to solve the challenging MiniGrid-Memory environment.

Original: Surprisingly, however,our experiments suggest that this is not due to the semantic enrichment of therepresentation presented to the LM but only due to the discriminative powerprovided by CLIP. 
Revised: Surprisingly, however, our experiments suggest that this is not due to the semantic enrichment of the representation presented to the LM, but rather due to the discriminative power provided by CLIP. We make our code publicly available at https://github.com/ml-jku/helm .

Original:  Introduction
Revised:  Introduction

Original: In Reinforcement Learning (RL) an agent interacts with an environment and learns from feedbackprovided in the form of a reward function.
Revised: In Reinforcement Learning (RL) an agent interacts with an environment and learns from feedback provided in the form of a reward function.

Original: RL agents that are deployed in the real world often haveto cope with partial observability.
Revised: RL agents that are deployed in the real world often have to cope with partial observability.

Original: Therefore, the capability to approximate the true state of theirsurrounding environment by virtue of an agent’s perception is crucial (Åström, 1964; Kaelbling et al.,1998).
Revised: Therefore, the capability to approximate the true state of their surrounding environment by virtue of an agent’s perception is crucial (Åström, 1964; Kaelbling et al., 1998).

Original: To this end, many agents employ a memory mechanism to track events that occurred in thepast.
Revised: To this end, many agents employ a memory mechanism to track events that occurred in the past.

Original: In this memory, it is much more efficient to store abstract representations of the past ratherthan every detail the agent encountered.
Revised: In this memory, it is much more efficient to store abstract representations of the past rather than every detail the agent encountered.

Original: Thus, memory mechanisms such as LSTM (Hochreiter &
Revised: Thus, memory mechanisms such as LSTM (Hochreiter &

Original: Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) compress sequences of high-dimensionalobservations.
Revised: Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) compress sequences of high-dimensional observations.

Original: Similarly, also humans memorize abstract concepts rather than every detail of information theyencountered in the past.
Revised: Similarly, also humans memorize abstract concepts rather than every detail of information they encountered in the past (Richards & Frankland, 2017; Bowman & Zeithamova, 2018).

Original: In humans, the ability to abstract is heavily influenced by the exposureto language in early childhood (Waxman & Markow, 1995).
Revised: In humans, the ability to abstract is heavily influenced by the exposure to language in early childhood (Waxman & Markow, 1995).

Original: By design, language is very wellsuited to form abstractions and it is used on a daily basis to pass on information between humans.
Revised: By design, language is very well suited to form abstractions and it is used on a Language and Reinforcement Learning Workshop at Neural Information Processing Systems, 2022. [Figure] distance δ

Revised: [Figure]

Revised: daily basis to pass on information between humans.

Original: Therefore, it is a natural choice as a medium for compressing compounding information.
Revised: Therefore, it is a natural choice as a medium for compressing compounding information.

Original: Prior workhas illustrated that pretrained LMs can efficiently compress sequences of observations and facilitateagent learning in partially observable RL environments (HELM, Paischer et al., 2022).
Revised: Prior work has illustrated that pretrained LMs can efficiently compress sequences of observations and facilitate agent learning in partially observable RL environments (HELM, Paischer et al., 2022).

Original: The keychallenge hereby is to map image-based observations to language representations.
Revised: The key challenge hereby is to map image-based observations to language representations.

Original: HELM tacklesthis problem using a randomly chosen mapping called FrozenHopfield (FH) that is — due to its [Figure] distance δ
Revised: HELM tackles this problem using a randomly chosen mapping called FrozenHopfield (FH) that is — due to its randomness — inherently unable to form meaningful abstractions.

Original: [Figure] randomness — inherently unable to form meaningful abstractions. In fact, we show that under certainconditions, the representations are prone to collapse rendering the agent unable to distinguish betweendifferent inputs (seeFigure Fig. 1).
Revised: In fact, we show that under certain conditions, the representations are prone to collapse rendering the agent unable to distinguish between different inputs (see Fig. 1).

Original: We build upon HELM by (i) substituting FH with a feature-wisecentering-and-scaling operation and (ii) incorporating a CLIP image encoder (Radford et al. , 2021),which is pretrained in a multimodal fashion on web data consisting of images and text.
Revised: We build upon HELM by (i) substituting FH with a feature-wise centering-and-scaling operation and (ii) incorporating a CLIP image encoder (Radford et al., 2021), which is pretrained in a multimodal fashion on web data consisting of images and text.

Original: We term theresulting new method HELMv2.
Revised: We term the resulting new method HELMv2.

Original: Using HELMv2, we are able to distinguish even minute differencesin the input when necessary, which drastically enhances downstream performance.
Revised: Using HELMv2, we are able to distinguish even minute differences in the input when necessary, which drastically enhances downstream performance.

Original: We demonstrate the effectiveness of HELMv2 on a diverse set of partially observable environments.
Revised: We demonstrate the effectiveness of HELMv2 on a diverse set of partially observable environments.

Original: Concretely, we train on 2D MiniGrid (Chevalier-Boisvert et al., 2018), and 3D MiniWorld environments (Chevalier-Boisvert, 2018).
Revised: Concretely, we train on 2D MiniGrid (Chevalier-Boisvert et al., 2018), and 3D MiniWorld environments (Chevalier-Boisvert, 2018).

Original: HELMv2 yields significant improvements over HELM in allenvironments.
Revised: HELMv2 yields significant improvements over HELM in all environments.

Original: Further, we conduct ablation studies, which show that the improvements are not onlydue to the CLIP image encoder but also due to the replacement of the FH.
Revised: Further, we conduct ablation studies, which show that the improvements are not only due to the CLIP image encoder but also due to the replacement of the FH.

Original: Finally, we construct amapping that successfully conveys the semantics extracted by CLIP to the LM.
Revised: Finally, we construct a mapping that successfully conveys the semantics extracted by CLIP to the LM.

Original: Surprisingly, however,using this mapping does not further improve the results in the selected environments.
Revised: Surprisingly, however, using this mapping does not further improve the results in the selected environments.

Original:  Methods
Revised:  Methods

Revised: The HELM framework consists of a history compression component, a learned CNN encoder of the current timestep, and an actor-critic head.

Revised: The CNN and the actor-critic head are learned, while the history compression part is kept frozen.

Original: HELM has demonstrated that pretrained LMs are well suited for compressing past observations thatare randomly mapped to language tokens, which is realized by FH.
Revised: HELM has demonstrated that pretrained LMs are well suited for compressing past observations that are randomly mapped to language tokens. It performs two forms of compression: (i) spatial compression, and (ii) temporal compression.

Revised: The former is realized with the FH mechanism, while the latter is performed with a pretrained TransformerXL (TrXL, Dai et al., 2019).

Original: The FH mechanism consistsof a random matrix P and an attention mechanism over pretrained token embeddings (VocabAttn).
Revised: The FH mechanism consists of a random matrix P and an attention mechanism over pretrained token embeddings (VocabAttn).

Original: More formally, let E =
Revised: More formally, let E =

Original: ( e 1 , . . .
Revised: ( e 1 , . . .

Original: , e k ) ⊤ ∈ R k × m be the token embedding matrix of the pretrained LM consisting of k embeddings e i ∈ R m .
Revised: , e k ) ⊤ ∈ R k × m be the token embedding matrix of the pretrained LM consisting of k embeddings e i ∈ R m .

Original: At every timestep t , we obtain inputs x t ∈ R m for the LM from observations o t ∈ R n via the FH mechanism by
Revised: At every timestep t , we obtain inputs x t ∈ R m for the LM from observations o t ∈ R n via the FH mechanism by

Original: [Equation]
Revised: [Equation]

Original: where σ is the softmax function and P ∈ R m × n has entries sampled from N (0 , n/m ) .
Revised: where σ is the softmax function and P ∈ R m × n has entries sampled from N (0 , n/m ) .

Original: The resultingx t lies in the convex hull of the token embeddings of the LM.
Revised: The resulting x t lies in the convex hull of the token embeddings of the LM.

Original: The parameter β is a scaling factor thatcontrols the dispersion of x t within that convex hull.
Revised: The parameter β is a scaling factor that controls the dispersion of x t within that convex hull.

Original: HELMcs Our aim is to avoid representation collapse caused by the FH mechanism.
Revised: HELMcs Our aim is to avoid representation collapse caused by the FH mechanism.

Original: We illustrate in
Revised: We illustrate in

Original: Fig.
Revised: Fig.

Original: 1 that FH is prone to representation collapse even for higher values of β , especially if observationstend to be visually similar.
Revised: 1 that FH is prone to representation collapse even for higher values of β , especially if observations tend to be visually similar.

Original: To sidestep this issue, we substitute the attention mechanism in FH with afeature-wise centering-and-scaling operation.
Revised: To sidestep this issue, we substitute the attention mechanism in FH with a feature-wise centering-and-scaling operation.

Original: Let B be a batch of observations o t and let µ B ∈ R n beits mean feature vector and σ B ∈ R n the vector of standard deviations both taken over B .
Revised: Let B be a batch of observations o t and let µ B ∈ R n be its mean feature vector and σ B ∈ R n the vector of standard deviations.

Original: Likewiselet µ E ∈ R m and σ E ∈ R m be means and standard deviations of the token embeddings E .
Revised: Likewise let µ E ∈ R m and σ E ∈ R m be means and standard deviations of the token embeddings E .

Original: Then wecompute x t as
Revised: Then we compute x t as

Original: [Equation]
Revised: [Equation]

Original: where diag takes a vector and constructs a diagonal matrix from it.
Revised: where diag takes a vector and constructs a diagonal matrix from it.

Original: We refer to this setting as
Revised: We refer to this setting as

Original:  HELMv2 We obtain another setting by encoding the observations with the ResNet-50 CLIP imageencoder, which has the same output dimension as the embeddings e i used by TransformerXL (TrXL,
Revised: HELMcs. HELMv2 We obtain another setting by encoding the observations with the ResNet-50 CLIP image encoder, which has the same output dimension as the embeddings e i used by TrXL, thus eliminating the need for the random mapping P .

Original: Dai et al., 2019), which we use as LM, thus eliminating the need for the mapping P .

Original: That is, welet z t = CLIP( o t ) ∈
Revised: That is, we let z t = CLIP( o t ) ∈

Original: R m and construct B  from z t .
Revised: R m and construct B ϕ from z t .

Original: Consequently, µ B  ∈ R m and σ B  ∈ R m .
Revised: Consequently, µ B ϕ ∈ R m and σ B ϕ ∈ R m .

Original: Wecompute the LM inputs as
Revised: We compute the LM inputs as

Original: [Equation]
Revised: [Equation]

Revised: The complexity imposed by the CLIP encoder is negligible since it is kept frozen and is only utilized during inference.

Original:  Experimental Results
Revised:  Experimental Results

Original: We investigate the limitations of HELM on a memory environment (Memory, Fig. 1, left).
Revised: We investigate the limitations of HELM on the MiniGrid-Memory environment (Memory, Fig. 1, left).

Original: The taskfor the agent is to remember the object in the  roomit spawns in and navigate through a corridorending in a T-junction and containing two objects, one of which is equivalent to the object in thestarting room.
Revised: The task for the agent is to memorize the object in the starting room and match it to the object at the end of the T-junction.

Original: If the agent chooses the wrong directionat the junction the episode ends withouteliciting any reward.
Revised: If the agent chooses the wrong direction the episode ends without eliciting any reward.

Original: However, the objects only slightly differ in shape (green ball vs green key).
Revised: The objects only slightly differ in shape (green ball vs green key).

Original: Wedemonstrate that FH collapses to the same representation for both objects by measuring the distancesbetween the softmax distribution σ ( · ) over token embeddings (Fig. 1, right).
Revised: We demonstrate that FH collapses to the same representation for both objects by measuring the distances δ between the softmax distribution σ ( · ) over token embeddings (Fig. 1, right).

Original: Indeed, we observe thateven for high values of β  there is very little chance that HELM can discriminate between the twoobjects.
Revised: For very high values of β > 1 e 4 , the softmax converges to a one-hot encoding and δ directly corresponds to the probability of mapping to a single, but different token embedding. Therefore, with higher values of β , there is only a 10% chance of avoiding collapse to the same token embedding.

Original: We measure the cosine similarityand ℓ 2 -distances between the embeddings of both observations from HELM, HELMcs, and HELMv2.
Revised: Moreover, we measure the cosine similarity between the embeddings of both observations for HELM, HELMcs, and HELMv2.

Original: Compared toHELM and HELMcs, HELMv2 is able to separatethe two observations much better (see Table 1). Method
Revised: Compared to

Original: Cosine Similarity ℓ 2 -distances HELM, β =0.99 ± 1.680.005 ± 0.
Revised: HELM and HELMcs, HELMv2 is able to separate the two observations (see Table 1 in Appendix D).

Original: HELM, β =0.99 ± 5.65e0.048 ± 0. HELM, β =0.99 ± 8.2e0.57 ± 0. HELM, β =
Revised: Moreover, we observe the random function P consistently conflates the inputs, while VocabAttn with high values of β again separates them to some extent.

Original: [Table]
Revised: [Table]

Original: We compare HELMv2 to HELM on a set of partially observable environments (see Fig. 2). Particularly, we select the RedBlueDoors environment (MiniGrid-RedBlueDoors-6x6-v0) from ChevalierBoisvert et al.
Revised: We train all our methods with Proximal Policy Optimization (PPO, Schulman et al., 2017) on RGB observations. We conducted hyperparameter searches as mentioned in Appendix E and evaluate via the interquantile mean (IQM) and 95% bootstrapped confidence intervals (CIs)

Original: (2018) , and the TMaze environment (MiniWorld-TMaze-v0) from Chevalier-Boisvert(2018). Furthermore, we show results for  the Memory environment (MiniGrid-MemoryS11-v0)as shown in  Fig. 
Revised: (Agarwal et al., 2021). To test for statistical significance, we perform a Wilcoxon test (Wilcoxon, 1945) at the end of training. We compare all our methods to an LSTM baseline on the Memory environment, the same set of partially observable MiniGrid environments as in (Paischer et al., 2022), and a set of eight more 3D MiniWorld environments (see Fig. 2).

Original: HELMv2 significantly outperforms HELM on all environments ( p = 3 .[-
-]
Revised: HELMv2 significantly outperforms HELM on MiniGrid ( p = 2 .[-
-]

Original:  e- 4 ,p = 2 .[-
-]
Revised: 24 e- 10 ) and MiniWorld ( p = 0 .[-
-]

Original: e- 11 , p = 9 .
Revised: Also, HELMcs significantly outperforms HELM on the MiniGrid environemnts ( p = 8 .

Original: 19 e- 
Revised: 7 e- 4 ).

Original: 9 , for Memory, RedBlueDoors, TMaze, respectively). Due to its limitations, HELM does not achieve performance better than randomly choosing an object for the Memoryenvironment.
Revised: HELM, HELMcs and LSTM do not achieve performance better than randomly choosing a path at the end of the corridor for the Memory environment within 2M interaction steps.

Original: Contrary, HELMv2 is able to distinguish and memorize the objects and solve thistask.
Revised: In contrast, HELMv2 is able to distinguish and memorize the objects and solves the task.

Original: On the 3D TMaze environment HELMv2 shows drastically improved performance compared to

Original: HELM. Additionally, we conduct an ablation study to show how VocabAttn affects the performanceof HELM.
Revised: Additionally, we conduct an ablation study to isolate the different components of HELMv (see Appendix D).

Original: Indeed, we illustrate that the attention over token embeddings consistently results  inadrop in performance in Appendix C Fig. 4. All our methods are trained with Proximal Policy Optimization (PPO, Schulman et al., 2017). Weevaluate all methods via the interquantile mean (IQM) and 95% bootstrapped confidence intervals(CIs) as suggested by Agarwal et al. To test for statistical significance, we perform a Wilcoxontest (Wilcoxon, 1945) at the end of training.
Revised: We find that the centering-and-scaling operation leads to much better results if performed in the abstract CLIP space, rather than in pixel space.

Original: We follow the architectural design of Paischer et al.(2022) and perform the same hyperparameter search for all our environments.  Future Work and Discussion In this work, we want to pave the way toward a semantic history compression for RL, that is topreserve the semantic concepts of an observation when mapping it into the LM space. In this regard,
Revised: Moreover, the choice of vision backbone affects final performance on the Memory environment.

Original: [Figure]
Revised: [Figure]

Original: cc u m u l a t ed R e w a r d Memory HELMv2HELMHELMcs
Revised: In this work

Original: [Figure]
Revised: [Figure]

Original: w a r d TMaze HELMv2HELMHELMcs we provide compelling evidence thatit isindeed feasible to create such a semantic  mapping between CLIP and the LM (see Fig. 3).
Revised: we want to pave the way toward a semantic history compression for RL, that is to preserve the semantic concepts of an observation when mapping it to the LM space.

Original: When using this semantic mapping in the two selected minimalistic

Original: RL environments, however, we do not observe a significant improvement over HELMv2.

Original: We can compute such a mapping from the vocabularies of the CLIP language encoder V CLIP andthe LM encoder V LM alone.
Revised: We can compute such a mapping from the vocabularies of the CLIP language encoder V CLIP and the LM encoder V LM without access to additional data.

Original: First, we identify the overlap between the two vocabulariesV OV =
Revised: First, we identify the overlap between the two vocabularies

Original: V  CLIP ∩ V LM with size l = |V OV | .
Revised: V OV = V CLIP ∩ V LM with size l = |V OV | .

Original: Next, we embed V OV in the CLIP output space and the LM inputspace yielding embedding matrices F = ( f 1 , . ..
Revised: Next, we embed V OV in the CLIP output space and the LM embedding space yielding embedding matrices F = ( f 1 , . .

Original: , f l ) ∈ R n × l and E l ∈ R m × l , respectively.
Revised: , f l ) ∈ R n × l and E l ∈ R m × l , respectively.

Original: Wecompute an orthogonal linear mapping matrix W according to the Procrustes method, which iscommonly used for aligning monolingual embedding spaces (Artetxe et al., 2018; Hoshen & Wolf,2018; Smith et al., 2017; Lample et al., 2018; Zhang et al., 2016;
Revised: We compute an orthogonal linear mapping matrix W according to the Procrustes method (Schönemann, 1966), as commonly used for aligning monolingual embedding spaces (Artetxe et al., 2018; Hoshen & Wolf, 2018; Smith et al., 2017; Lample et al., 2018; Zhang et al., 2016;

Original: Xing et al., 2015; Minixhofer et al.,2022).
Revised: Xing et al., 2015; Minixhofer et al., 2022).

Original: We refer to this setting as SHELM (for semantic HELM) and elaborate in more detail on it in Appendix A.
Revised: We refer to this setting as SHELM (for semantic HELM) and elaborate in more detail on it in Appendix B.

Original: Due to the alignment of theCLIP imageencoder and the CLIP language encoder, this mapping canbe used to project images to the LM space while preserving its semantics.
Revised: Due to the alignment of the image and text modalities in CLIP space, W can be used to project images to the LM space while preserving its semantics.

Original: We visualize sampleimages and the closest tokens in the LM space in Fig.
Revised: We visualize sample images and the closest tokens in the LM space in Fig.

Original: Further, in Appendix B we conduct aquantitative analysis using publicly available image captioning datasets and show that our mappingcan identify tokens prevalent in captions provided by humans.Table 5 in Appendix C corroboratesthese impressions.
Revised: Further, in Appendix C we conduct a quantitative analysis using publicly available image captioning datasets and show that our mapping can identify tokens prevalent in captions provided by humans.

Original: [Figure]
Revised: [Figure]

Original: We show preliminary results of SHELM in Fig.
Revised: We show preliminary results of SHELM in Fig.

Original: Our findings are that SHELM improves upon
Revised: Surprisingly, we observe performance on-par with

Original: HELM, since it avoids the apparent representation collapse in the FH mechanism due to the orthogonal mapping W . However, we observe performance on-par with HELMv2 ( 0 . 979 ± 0 . 002for RedBlueDoors, 0 . 887 ± 0 .
Revised: HELMv2.

Original: 018 for TMaze). We believe this is due to the fact that our selectedenvironments are artificial 2D and 3D scenes for which the CLIP image encoder is unable to sufficiently extract semantically meaningful features. 
Revised: We believe this is due to the fact that our selected environments are simulated 2D and 3D scenes for which the CLIP image encoder is unable to sufficiently extract semantically meaningful features. This is in line with findings of Fan et al. (2022) who minimally finetune CLIP for the

Original: This might change in real world environments withobservations close to the CLIP pre-training domain.
Revised: Minedojo environment. We validate this finding by taking a closer look at the token rankings for the two observations of the Memory environment in Fig. 4 in the Appendix. However, we expect to see a difference in more realistic environments with observations closer to the CLIP pre-training domain (Tam et al.

Original: We aim to investigate this question in futurework.
Revised: We aim to investigate this question in future work Acknowledgements

Original: A Linear Mappings between Language Embedding Spaces
Revised: The ELLIS Unit Linz, the LIT AI Lab, the Institute for Machine Learning, are supported by the Federal State Upper Austria. IARAI is supported by Here Technologies. We thank the projects AIMOTION

Revised: (LIT-2018-6-YOU-212), AI-SNN (LIT-2018-6-YOU-214), DeepFlood (LIT-2019-8-YOU213), Medical Cognitive Computing Center (MC3), INCONTROL-RL (FFG-881064), PRIMAL (FFG-873979), S3AI (FFG-872172), DL for GranularFlow (FFG-871302), AIRI FG 9-N (FWF36284, FWF-36235), ELISE (H2020-ICT-2019-3 ID: 951847).

Revised: We thank Audi.

Revised: JKU Deep Learning

Revised: Center, TGW LOGISTICS GROUP GMBH, University SAL Labs initiative of Silicon Austria Labs (SAL), FILL Gesellschaft mbH, Anyline GmbH, Google, ZF Friedrichshafen AG, Robert Bosch

Revised: GmbH, UCB Biopharma SRL, Merck Healthcare KGaA, Verbund AG, Software Competence Center

Revised: Hagenberg GmbH, TÜV Austria, Frauscher Sensonic and the NVIDIA Corporation.

Revised: A Environments

Revised: We choose 8 diverse 3D environments of the MiniWorld benchmark suite:

Revised: • CollectHealth:

Revised: The agent spawns in a room filled with acid and must collect medikits in order to survive as long as possible.

Revised: • FourRooms:

Revised: The agent must reach a red box that is located in one of four interconnected rooms.

Revised: • MazeS3Fast:

Revised: A procedurally generated maze in which the agent needs to find a goal object.

Revised: • PickupObjs:

Revised: Several objects are placed in a large room and must be collected by the agent.

Revised: Since the agent receives a reward of 1 for each collected object, the reward is unbounded.

Revised: • PutNext: Several boxes of various colors and sizes are placed in a big room.

Revised: The agent must put a red box next to a yellow one.

Revised: • Sign: The agent spawns in a U-shaped maze containing various objects of different colors.

Revised: One side of the maze contains a sign which displays a color in written form.

Revised: The aim is to collect all objects in the corresponding color.

Revised: • TMaze: The agent must navigate towards an object that is randomly placed at either end of a T-junction.

Revised: YMaze:

Revised: Same as TMaze, but with a Y-junction.

Revised: We neglect the OneRoom and the Hallway environments, since those are easily solved by all our methods.

Revised: Further, we neglect the Sidewalk environment since it is essentially the same task as Hallway with a different background.

Revised: Since the reward of PickupObjs and CollectHealth are unbounded, we normalize them to be in the range of (0 , 1] , which is the reward received in all other environments.

Revised: For a more detailed description of the MiniGrid environments we refer the reader to Paischer et al.

Original: Our aim is to create a linear mapping W from the aligned multimodal space of CLIP to the embeddingspace of a pretrained LM encoder.
Revised: Our aim is to create a linear mapping W from the aligned multimodal space of CLIP to the embedding space of a pretrained LM encoder.

Original: We argue that a linear mapping is sufficient since the semanticrelation between the tokens is equivalent in both spaces and semantic relations between tokenscan be expressed additively (Mikolov et al., 2013; Tewel et al., 2021).
Revised: We argue that a linear mapping is sufficient since the semantic relation between the tokens is equivalent in both spaces and semantic relations between tokens can be expressed additively (Mikolov et al., 2013; Tewel et al., 2021).

Original: Therefore, given two CLIPtokens f 1 , f 2 , it does not matter whether we perform the addition before or after the mapping, i.e.,
Revised: Therefore, given two CLIP tokens f 1 , f 2 , it does not matter whether we perform the addition before or after the mapping, i.e.,

Original: W ( f 1 + f 2 ) =
Revised: W ( f 1 + f 2 ) =

Original: W f 1 + W f 2 .
Revised: W f 1 + W f 2 .

Original: Further, using linear models to map between monolingual wordembedding spaces is established in the literature (Artetxe et al., 2018; Hoshen & Wolf, 2018; Smithet al., 2017; Lample et al., 2018; Zhang et al., 2016; Xing et al., 2015; Minixhofer et al., 2022).
Revised: Algorithm

Original: Require: CLIP language encoder CLIP LM , Vocabulary of CLIP V CLIP , Language Model Encoder
Revised: Require: CLIP language encoder CLIP LM , Vocabulary of CLIP V CLIP , Language Model Embedding

Original:  LM ( v i ) , Language Model vocabulary V LM
Revised: Layer LM ( v i ) , Language Model vocabulary V LM

Original: V OV ← V CLIP ∩ V LM ▷ Search for overlapping vocabulary f i = CLIP LM ( v i ) for v
Revised: V OV ← V CLIP ∩ V LM ▷ Search for overlapping vocabulary f i = CLIP LM ( v i ) for v

Original: i ∈ V OV
Revised: i ∈ V OV

Original: ▷ Embed tokens in CLIP output space e [Equation] =
Revised: ▷ Embed tokens in CLIP output space e i =

Original: [Equation] for v i ∈ V OV
Revised: LM ( v i ) for v i ∈ V OV

Original: ▷ Embed tokens in LM input space
Revised: ▷ Embed tokens in LM input space

Original: W ← create_mapping ( F , E )
Revised: W ← create_mapping ( F , E )

Original: ▷ Compute mapping between embeddings
Revised: ▷ Compute mapping between embeddings

Original: We define the vocabulary of the CLIP language encoder as V CLIP  and  thevocabulary of the LM encoderas V LM .
Revised: We consider all publicly available CLIP backbone variants and align their output spaces with the

Original: First, we search for an overlap between the two vocabularies and retrieve the indices of thetokens in the overlap V OV , where l = |V OV | . Next, we embed the tokens in the CLIP output space toobtain a token embedding matrix F = ( f 1 , . . . , f l ) ∈ R n × l . Let E =
Revised: TrXL used in HELM.

Original: ( e 1 , . . , e l ) ∈ R m × l be thetoken embedding matrix from the pretrained LM consisting of l embeddings e i ∈ R m correspondingto tokens inV OV . We compute a linear mapping matrix W that minimizes the distance between thetwo token embedding matrices ∥ W F − E ∥ 2 F .
Revised: According to Algorithm 1 we determine the overlap in the CLIP and TrXL vocabularies.

Original: In principle any linear model can be applied to mapfrom the CLIP space to the LM space.
Revised: In principle any linear model can be applied to map from the CLIP space to the LM space.

Original: Algorithm 1 shows the algorithm for computing the mappingused for SHELM.

Original: The prevalent alignment of modalities in the CLIP output space allows mapping visual inputs tothe LM space.

Original: In turn we can substitute the FH mechanism in HELM with CLIP followed by ourmapping W to preserve the semantics of an observation in the LM space.

Original: We compare four different types of linear models:
Revised: We compare four different types of linear models:

Original: • Ordinary Least Squares (Linear), minimizing
Revised: • Ordinary Least Squares (Linear), minimizing

Original: [Equation]
Revised: [Equation]

Original: • Least Squares with Thikonov regularization (Ridge):
Revised: • Least Squares with Thikonov regularization (Ridge):

Original: [Equation]
Revised: [Equation]

Original: • Orthogonal Procrustes (Least Squares with orthogonality constraint, Schönemann (1966); Gower  Dijksterhuis 
Revised: • Orthogonal Procrustes (Least Squares with orthogonality constraint, Schönemann (1966); Gower & Dijksterhuis (2005)):

Original: [Equation]
Revised: [Equation]

Original: • Robust Procrustes (RobProc, (Groenen et al., 2005)) iteratively refines [Table] spaces. ViT-L/14* received images resized to 336 pixels as input during pretraining.
Revised: • Robust Procrustes (RobProc, (Groenen et al., 2005)) iteratively refines the Orthogonal Procrustes method based on the predicion error ∥ F W − E ∥ 2 F 1 .

Original: Linear heavily overfits to the training set.

Original: Larger backbones lead to a better test accuracy.

Original: The Procrustes method is commonly used for aligning embedding spaces across different languagessince it preserves monolingual invariance (Artetxe et al., 2016; Minixhofer et al., 2022).

Original: We considervarious choices of CLIP backbones and align their output spaces with the TrXL used in HELM.

Original: Wedetermine the intersection of the vocabulary of the CLIP language encoder and the TrXL languageencoder.

Original: In total, there are 5285 tokens that appear in both, V CLIP , and V LM .
Revised: In total, there are 5285 tokens that appear in both, V CLIP , and V LM .

Original: Prior work has foundthat as little as ten word correspondences are sufficient to train an orthogonal mapping betweenmonolingual embedding spaces of closely related languages (Zhang et al., 2016).
Revised: Prior work has found that as little as ten word correspondences are sufficient to train an orthogonal mapping between monolingual embedding spaces of closely related languages (Zhang et al., 2016).

Original: This assumes acertain degree of isomorphism between the embedding spaces.
Revised: This assumes a certain degree of isomorphism between the embedding spaces.

Original: Since we train a mapping betweenembedding spaces of the same language, we expect this assumption to hold in our setting as well.
Revised: Since we train a mapping between embedding spaces of the same language, we expect this assumption to hold in our setting as well.

Original: The CLIP output space and the TrXL input space differ greatly in their statistics, therefore we performcentering and scaling as preprocessing.
Revised: The CLIP output space and the TrXL input space differ greatly in their statistics, therefore we perform centering and scaling as preprocessing.

Revised: The Procrustes method is commonly used for aligning embedding spaces across different languages since it preserves monolingual invariance (Artetxe et al., 2016; Minixhofer et al., 2022).

Original: This way, during inference in  the RL experiments we cansimply re-center and re-scale by the statistics ofLM space to obtain a suitable input for the TrXL.
Revised: We consider various choices of CLIP visione encoders and align their output spaces with the TrXL used in HELM. We determine the intersection of the vocabulary of the CLIP language encoder and the TrXL language encoder.

Original: We perform a 5-fold cross validation and measure the accuracy, considering each token as its ownclass.
Revised: We perform a 5-fold cross validation and measure the accuracy, considering each token as its own class.

Original: Table 2 shows the average train and test accuracy for the different linear mapping methods forvarious CLIP backbones and TrXL.
Revised: Table 2 shows the average train and test accuracy for the different linear mapping methods for various CLIP backbones and TrXL.

Original: We observe a strong overfitting effect for Linear, while the two
Revised: We observe a strong overfitting effect for Linear, while the two

Original: Procrustes variants generalize best.
Revised: Procrustes variants generalize best.

Revised: We do not show variance estimates since those are negligibly small.

Original: B Mapping Visual Input to the Language Space
Revised: [Table] C Mapping Visual Input to the Language Space

Revised: The prevalent alignment of modalities in the CLIP output space allows mapping visual inputs to the LM space.

Revised: In turn we can substitute the FH mechanism in HELM with CLIP followed by our mapping W to preserve the semantics of an observation in the LM space.

Revised: This way, during inference in the RL experiments we can simply re-center and re-scale by the statistics of LM space to obtain a suitable input for the TrXL.

Original: We conduct a quantitative analysis of the different mapping methods.
Revised: We conduct a quantitative analysis of the different mapping methods.

Original: In this regard, we use apre-existing image/caption dataset to quantify how well the semantics of an image are preservedafter mapping to the LM space.
Revised: In this regard, we use a pre-existing image/caption dataset to quantify how well the semantics of an image are preserved after mapping to the LM space.

Original: We draw a random subset of 1000 image-caption pairs of thepopular MSCOCO dataset (Lin et al., 2014).
Revised: We draw a random subset of 1000 image-caption pairs of the popular MSCOCO dataset (Lin et al., 2014).

Original: The subset is filtered to contain only image-text pairswhere the captions contain at least 5 tokens of our computed vocabulary overlap in Appendix A. Forpreprocessing of the captions we remove stop words and apply stemming.
Revised: The subset is filtered to contain only image-text pairs where the captions contain at least 5 tokens of our computed vocabulary overlap in Appendix B. For preprocessing of the captions we remove stop words and apply stemming to tokenized captions and tokens in the overlap.

Original: Next, we propagate theimage through various CLIP backbones, to obtain an image embedding and map it to the LM spaceusing our pre-computed mappings.
Revised: Next, we propagate the images through various CLIP backbones, to obtain an image embedding and map it to the LM space using our pre-computed mappings.

Original: Finally, we rank tokens in the LM space based on their cosinesimilarity to the mapped image.
Revised: Finally, we rank tokens in the LM space based on their cosine similarity to the mapped image.

Original: Based on the obtained ranking we compute the Mean Reciprocal Rank (MRR, Craswell, 2009) and the Normalized Discounted Cumulative Gain (NDCG, Järvelin & Kekäläinen, 2002).
Revised: Based on the obtained ranking we compute the Mean Reciprocal Rank (MRR, Craswell, 2009) and the Normalized Discounted Cumulative Gain (NDCG, Järvelin & Kekäläinen, 2002).

Original: We also include a comparison to token ranking in the CLIP space as a baseline todetermine how much information is lost after applying our mapping. Table 3 shows the NDCG for different linear mappings trained to map between RN50 CLIP backboneand TrXL.
Revised: Table 3 shows the NDCG for various CLIP backbones mapping to the embedding space of TrXL relative to ranking in the CLIP space (rNDCG). We observe that the Procrustes mapping consistently outperforms the Linear and Ridge mapping. Furthermore, there is no improvement in iteratively refining the Procrustes method as in RobProc.

Original: The results for MRR are shown in Table 4.
Revised: The results for MRR are shown in Table 4.

Original: We observe that the Robust Procrustes mappingconsistently outperforms the Linear and Ridge mapping.

Original: Further, there is a slight improvement forlarger model architectures. Remarkably, on average the third ranked token is contained in a caption for the Procrustes mapping, while for CLIP it is every second ranked token. 
Revised: Remarkably, on average the third ranked token is contained in a caption for the Procrustes mapping, while for CLIP it is every second ranked token. The high variance stems from images that resulted in suboptimal rankings.

Original: From this analysis weconclude that most of the semantics are preserved with our linear mapping.
Revised: From this analysis we conclude that most of the semantics are preserved with our linear mapping.

Revised: To validate that the CLIP image encoder fails to sufficiently extract semantics of environment observations

Original: [Table]
Revised: [Table]

Revised: we take a closer look at token rankings for the observations in Fig.

Revised: Indeed, the tokens exhibiting the highest similarity in the CLIP space do not describe the semantics of the image, but rather similar concepts, i

Original: [Table]
Revised: [Table]

Original: C Additional Experimental Results
Revised: e., the token pong is ranked higher when the ball is present in the observation. [Table] [Table] [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: l

Revised: [Figure]

Revised: t ed R e w a r d

Revised: MiniGrid

Revised: HELMv2 HELMclip

Revised: HELMshift HELM

Revised: [Figure]

Revised: A cc u m u l a t ed R e w a r d

Revised: MiniWorld

Revised: HELMv2 HELMclip

Revised: HELM HELMshift

Revised: D Additional Experimental Results

Revised: We conduct ablation studies to isolate the effect of the different components.

Revised: In this regard, we add two additional settings: (i) HELMshift, and (ii) HELMclip.

Revised: HELMshift adds the centeringand-scaling operation from HELMcs to the original HELM setting.

Original: We compare HELMv2 to an additional setting, namely using CLIP in combination with the attentionmechanism over token embeddings (HELMclip) on the RedBlueDoors and the TMaze environments.
Revised: HELMclip uses the same ResNet-50 CLIP image encoder with the centering-and-scaling operation followed by VocabAttn. We compare the performance of HELMclip and HELMshift to HELMv2 and HELM on all MiniGrid, and MiniWorld environments.

Original: Fig.
Revised: Fig.

Original: 4 shows  the  performance of HELMclip compared to the same methods as mentioned in Section 3.
Revised: 5 shows that adding the centering-and-scaling operation to HELM (HELMshift) does not lead to improved performance. In fact, on MiniWorld environments it even leads to worse performance than HELM. We suspect this is due to the frequent pixel changes in the observation space that do not correspond to significant changes in the environment state. While we also use image observations in MiniGrid the observation space is much more abstract since entire tiles change at once.

Revised: Substituting P in HELMshift with the CLIP encoder (HELMclip) however, results in an immediate improvement on MiniGrid environments.

Original: Again, HELMv2 consistenly outperforms HELMclip across both environments.
Revised: Since HELMclip still uses the VocabAttn, it is prone to representation collapse if changes in the CLIP space are subtle (see Fig. 1). Finally, discarding VocabAttn (HELMv2) drastically improves performance on both, MiniGrid and MiniWorld environment suites.

Revised: While VocabAttn can help in some environments, its proneness to collapse overall hurts performance.

Original: Next, we showadditional results for the SHELM method which we introduced in Section 4.
Revised: To show the effect of different image encoders, we perform an ablation where we substitute the CLIP image encoder with a ResNet (He et al., 2016, HELMv2-RN34-IN), and a Vision Transformer (Dosovitskiy et al., 2021, HELMv2-ViT-L16-IN), pretrained in a supervised manner on the popular ImageNet dataset (Deng et al., 2009). Additionally, we compare to a Vision Transformer version of CLIP (HELMv2-ViT-B/16).

Revised: First, we measure cosine similarities to quantify the ability of the different vision encoders to separate the two observations (see Fig.

Original: Weinstantiate SHELM with  the RN50 backbone of CLIP and the corresponding RobProc mapping.
Revised: 6, right). The HELMv2-ViT-B/ variant exhibits the lowest cosine similarity between both observations. However, better separability does not correlate with improved downstream performance (see Fig. 6, left). In fact, the best performing agent uses the ViT-L16 pretrained on ImageNet.

Original: Further, we add one more setting which substitutes theRobProc mapping of SHELM with a randomorthogonal matrix sampled from the Haar distribution (Stewart, 1980) (SHELM+RandOrtho).
Revised: An explanation for this might be the difference in scale (Table 6) or the different pretraining paradigms (i.e. supervised vs. contrastive). We aim at answering this question in future work. Next, we show additional results for the SHELM method which we introduced in Section 4. We select the best backbone-mapping combination by ranking all combinations according to the average absolute NDCG (aNDCG), and select the top 5 settings.

Original: Weshow results for SHELM, SHELM+RandOrtho, and HELMv2 in Fig.
Revised: Furthermore, we perform a Wilcoxon test for statistical significance between the combinations after bonferroni correction (Bonferroni, 1936). The two combinations, ViT-B/16+Procrustes, and ViT-L/14*+Procrustes, significantly outperform all competitors in terms of absolute NDCG. Due to the imposed complexity of ViT-L/14* (see Table 6), we choose ViT-B/16+Procrustes to instantiate SHELM.

Original: Surprisingly, We find that
Revised: We show preliminary results of

Original: SHELM+RandOrtho performs on-par with SHELM, and HELMv2.
Revised: SHELM on the Memory environment, and our MiniGrid and MiniWorld environments in Fig. Additionally, we compare with HELMv2+RandOrtho, which samples random orthogonal matrices from the Haar distribution (Stewart, 1980).

Original: These results indicate that forsolving our selected tasks, no semantic information is required.
Revised: Surprisingly, we observe no statistically significant differences in performance between the different methods.

Original: A possible reason for this is that
Revised: [Figure]

Original: CLIP is unable to sufficiently extract semantically meaningful features.
Revised: [Figure]

Original: Finally, we conduct an additional analysis on how much the different methods preserve the semanticsof natural images.
Revised: Finally, we conduct an additional analysis on how much the different methods preserve the semantics of natural images.

Original: In this regard, we perform the same quantitative analysis as in Appendix B forthe different methods presented in this work.
Revised: In this regard, we perform the same quantitative analysis as in Appendix C for the different methods presented in this work.

Original: Again, we compute the MRR and  NDCG for our [Figure] [Figure]
Revised: Again, we compute the MRR and the absolute NDCG for our MSCOCO subset of image-caption pairs.

Original: SHELM+RandOrtho performs on- and

Original: MSCOCO subset of image-caption pairs.

Original: Results are shown in Table 5.
Revised: Results are shown in Table 5.

Original: We observe that methodssuch as HELMv2,HELMclip, or SHELM+RandOrtho yield on-par or worse results compared toa random ranking.
Revised: We observe that methods such as HELMv2, or HELMv2+RandOrtho yield on-par or worse results compared to a random ranking.

Original: This is due to the fact, that for lower values of β the VocabAttn tends to outputthe mean over token embeddings.
Revised: This is due to the fact, that for lower values of β the VocabAttn tends to output the mean over token embeddings.

Original: In this case, the resulting ranking is equal across image, and thus,results in lower MRR and NDCG.
Revised: In this case, the resulting ranking is equal across image, and thus, results in lower MRR and NDCG.

Original: Also substituting the RobProc mapping with a random mappingdrawn from the Haar distribution leads to random rankings.
Revised: Also substituting the RobProc mapping with a random mapping drawn from the Haar distribution leads to random rankings.

Original: Contrary, SHELM yieldsa ranking thatis closest the ranking performed by CLIP (see Table 3.
Revised: Contrary, SHELM yields the best token rankings (see Table 3.

Original: Thus, for SHELM, we observe that most ofthe semantics are preserved.
Revised: Thus, for SHELM, we observe that most of the semantics are preserved.

Original: Adding VocabAttn after the RobProc mapping again results in loss ofinformation and a suboptimal ranking.
Revised: Adding VocabAttn after the RobProc mapping again results in loss of information which is mirrored in both MRR and NDCG. E Hyperparameter Search

Revised: We search for hyperparameters for all methods that we trained and adapt the gridsearch conducten in

Revised: Paischer et al.

Revised: Particularly, we search for learning rate in { 5 e- 4 , 3 e- 4 , 1 e- 5 , 5 e- 5 } , entropy coefficient in { 0 . 05 , 0 . 01 , 0 . 005 , 0 . 001 } , rollout length in { 32 , 64 , 128 } for HELMv2, and HELMcs.

Original:  [Table]
Revised: Since our analysis in Section 3 showed that for β = 1 , 10 the spatial compression collapses to the mean over pretrained token embeddings, we alter the grid for β of HELM to { 100 , 500 , 1000 , 5000 } . To decrease wall-clock time of HELM variants, we vary the size of the memory register of TrXL such [Table] [Table]

Revised: that it can fit the maximum episode length (see Table 7).

Revised: We lower the number of interaction steps for the gridsearch if we observe convergence before the 500k interaction steps.

Revised: If no convergence is observed within the 500K interaction steps, we tune

Revised: [Table]