Self-Supervised Learning Through Efference Copies
Anonymous Author(s)
AffiliationAddress email
Abstract
State-of-the-art (SOTA) machine learning (ML) without labels is based on selfsupervised learning (SSL).
SSL advances are highly demanded given the scarcityof labelled data and the cost of human supervision.
Biology can offer inspirationfor advanced SSL models that could also bring insights into learning in the brain.
SSL commonly transforms each training datapoint into a pair of views, usesthe knowledge of this pairing as a positive self-supervisory sign, and potentiallyopposes it to unrelated, negative examples.
Here, first we show that this type ofself-supervision is a concrete – albeit constrained – implementation of a conceptfrom neuroscience: the Efference Copy (EC).
Specifically, like SSL, the brainalso transforms the environment through efference, i.e. outgoing motor commands,however it sends to itself an EC of the full commands, i.e. more than a meresign.
Second, we implement this insight as an SSL framework of Self-supervision
Through Efference Copies (S-TEC), and we show empirically that S-TEC improvesrecent strong SSL baselines on image classification.
Third, we hypothesize andprovide preliminary evidence that S-TEC’s improvement to the representation isby placing towards the border of each class heavily transformed data specifically,resulting in better separation between the classes’ supports and more structuredinter-class space.
Overall, S-TEC conceptually generalizes the SSL framework,and hypothesizes a testable positive influence from motor outputs onto sensoryrepresentations of the brain.
Introduction
Deep Learning (DL) has drawn inspiration from Neuroscience and also offers models for understanding aspects of the brain (Richards et al., 2019).
DL has been extremely successful, largely owing tolabelled big datasets.
However, such labelling is a costly procedure carried out by human supervisors.
As a result, supervised learning excludes the bulk of available data and limits the tasks accessibleto ML.
Fully unsupervised ML techniques do exist, however they rarely reach the performance ofsupervised learning.
On the other hand, recently, a category of algorithms that are self-supervisedhas emerged.
In self-supervised learning (SSL), the model itself generates the supervisory signal,so that human supervision is not needed, and then uses that signal for supervised learning.
Recent
SSL algorithms generate the supervisory signal by using pairs of inputs where it is known whetherthey are of the same or of a different instance, therefore self-generating positive or negative labels.
Examples can be associated as being positive, e.g. based on their temporal proximity, if the input is ina sequential domain (Oord et al., 2018).
Advanced SSL algorithms generate themselves the positivepairs of inputs, by augmenting the training dataset (He et al., 2020;
Chen et al., 2020a; Grill et al.
,2020; Caron et al., 2020; Mitrovic et al., 2021).
Further improving these algorithms can additionallyalleviate the need for costly human supervision of DL.
Moreover, it even has the potential to outperform supervised learning (Tomasev et al., 2022), because more information exists in the comparisonof complete input pairs as opposed to human-labelled individual examples.
Therefore, improving
SSL consists in devising representation learning methods that better capture that information.
Self-supervised learning
Motorcontrol
Environment
Object
Changed sensory input
Action
Efference copy (EC)
Raw image
Augmentations
A
B
ResNet
MLP MLP
EC
ResNet
Discrete classes
Sensoryprocessing
Inversemodel
Shared parameters
Gradients
Downstream tasks: classification, ...
Efference copies in the central nervous system
Inspiration for such improvements may be found in the operation of the biological central nervoussystem (CNS), which appears to learn mostly without external supervision.
In the present study wetake this abstract hope and formulate it as a concrete link from SSL to a specific mechanism in the
CNS.
We begin by observing first, that the CNS of vertebrate animals is believed to have evolvedwith the main purpose of performing sensory-motor control and learning, and second, that the data manipulations that augment the training examples in ML implementations of SSL can be viewedas motor actions.
The search for analogies then can focus on looking for possible self-supervisorysignals within biological motor control and learning.
A particularly well-suited and well-established signal in the sensory-motor system is that of the Efference Copy (EC) (von Helmholtz, 1867; McNamee and Wolpert, 2019) or Corollary Discharge (Sperry,1950).
Namely, it has been shown that when a component of the CNS addresses the body’s muscleswith an efferent, i.e. outgoing, motor command or action, often it also sends a copy to the CNSitself, see Figure 1A. ECs have multiple functions and abundant supporting evidence (Kennedy et al.
,2014; McNamee and Wolpert, 2019; Kilteni et al., 2020; Latash, 2021).
For example, certain motorcommands responsible for the locomotion of frogs are generated in the spinal cord, but are copiedto the brainstem, which is responsible for motor control of the eye (von Uckermann et al., 2013).
The body-movement-related disturbances to the visual field are then predicted and appropriatelycounteracted by eye movements that stabilize the frog’s gaze.
Therefore, one function of EC is tocoordinate different motor controllers of the body.
Another function of EC is to focus sensory processing on externally-generated and unpredictedstimuli by cancelling predictable sensations of self-generated actions.
E.g. humans cannot ticklethemselves effectively, because by using its ECs the CNS predicts the sensory consequence of itsown action, and cancels it before it is perceived (Blakemore et al., 1998).
The role of ECs in humansis actually broader and very central to motor control.
Specifically, the control of bodily movementsinvolves forward internal models that the brain maintains, i.e. models that predict the sensory inputsthat result from each motor command (Kawato, 1999; McNamee and Wolpert, 2019).
These forwardmodels rely on access to motor commands to generate their predictions, and that access is provided by
ECs.
Importantly, motor control also involves inverse models, which map representations of targetedmovement sensations to their possible actions (Rizzolatti et al., 1998; Kawato, 1999) (Figure 1A).
Inaddition to motor control, ECs also underlie motor learning (Witney et al., 1999; Troyer and Doupe,
[Figure]
Singleunknown
C
[Figure]
For example, when learningan inverse model, the structures that calculate errors must access the efference.
Given the pervasive role of ECs in sensory processing, motor control, and motor learning, wehypothesize that ECs could play a key role in the learning of sensory representations too, and that itdoes so through the learning of inverse models.
More specifically, we hypothesize that, if EC actsas a self-supervisory learning signal, then it improves the sensory learning process, e.g. improvingthe later classification of input examples.
Rather than physiological experiments, or biologicallydetailed simulations, we will test the hypothesis in an abstract ML setting.
Nevertheless, we willuse mechanisms that do have plausible biophysical implementations.
In addition, our model couldimprove ML methods by providing more of the information content of paired input datapoints to SSL.
That is because ECs can be rich and diverse signals, i.e. they can provide the full description of theactions that generate input pairs, and can do so for varied types of actions.
S-TEC: Self-supervision Through Efference Copies3.1 Definitions and key principles.
Our basic assumption is that (a) an EC is available , i.e. a copy of the efferent motor commands,or actions.
The essence of our strategy is to use this EC as a target label to (b) learn an inversemodel , mapping sensory inputs to the motor outputs that caused the changed inputs in the firstplace (Kawato, 1999).
We assume (c) a hierarchical model , e.g. a multilayer neural network.
Weconjecture that a model that improves on this motor-oriented task, will also improve its intermediatesensory representations as a direct consequence, which are then useful to a wider variety of sensorytasks.
In our experiments, we use the representation for classification.
In order to concretize themodel, let x ∈ D generally denote sensory inputs.
Furthermore, we define the motor commands asactions a ∈ A ego that result in transformed inputs x ′ =
T ( x, a ) , denoting with T the transformationfunction, see also Figure 1A.
In the following, we will simply write the EC as a probability distributionp EC ( a | x, x ′ ) to indicate the distribution of values it will assume given the sensory inputs are x beforean action was taken, and are x ′ thereafter.
We utilize this as a ground truth that the inverse modelneeds to predict.
More formally, we denote the to-be-learned inverse sensory-motor mapping byq θ ( a | x, x ′ ) with free parameters θ that surmise synaptic weights.
Learning then is the minimizationof the discrepancy between the ground truth p EC and our model q θ :
[Equation]
In the above formulation, we denote with D KL the Kullback-Leibler divergence, and introduce theloss function L that will be helpful later.
The broad concept given in equation (1) is so far agnostic tothe specific types of actions and sensory inputs.
To render the matter more concrete, and to align itwith the examples for sensory modalities in Section 2, we will focus on the visual sensory domain.
This also facilitates the validation of our approach by ML experiments on contemporary datasetsand architectures, see Section 4.
We assume that only one type of sensory object is observed witheach sensory input.
We denote the set of possible actions as A ego .
To account for the fact that themodel concerns sensory-motor control in the physical world, (d) actions must account for two typesof sensory transformations (see Figure 2A), namely:
• (d1) Object-identity-related actions a id .
This category of action switches between sensed
objects, e.g. by a saccade of the eyes, bringing entirely new objects into focus, or not.
In the
context of standard vision datasets that are comprised of static images, we simply exchange
the currently viewed image with a randomly sampled new one.
The two types of actions in
this category, i.e. switching or not, are a id = 1 or a id = 0 respectively.
• (d2) Same-object manipulations a manip .
This category is identity-preserving, i.e. maintain-
ing the sensed object but the observer actively manipulates it or its view, e.g.: turning the
object, or moving to a closer vantage point.
With static images, this kind of transformation
is naturally formed by commonly used image augmentation operations.
E.g. spatial trans-
formations that crop an image with random size and random aspect ratio can simulate the
movement to a different point at a closer distance whereas mirroring the image horizontally
corresponds well to rotating a symmetric 3D object, see Figure 1B.
We denote an action that
transforms one augmented view x into the other augmented view x ′ by a manip .
The object-identity-related actions are of two types a id ∈ { 0 , 1 } .
Therefore, this part of the actionrepresentation is categorical.
Based on this, (e) we model the entire action representation ascategorical , i.e. including a manip .
This is to follow the biological evidence that the brain maintainsuniform principles throughout its organization, e.g. throughout the cortex (Douglas et al.
, 1989),including motor areas (Bastos et al., 2012).
Moreover, there is significant evidence that this uniformorganization does specifically have a categorical structure, were different actions are representedby different clusters of neurons (Graziano, 2016).
Importantly, this allows learning the associatedinverse model by means of a classification task, as will be introduced later.
As the overall action a is composed by two parts, i.e. a = ( a id , a manip ) , we can summarize thiscategorical structure as a graphical model and decision tree, shown in Figure 2B and C. Hence,given the same object continues to be in focus, i.e. a id = 0 , then there exist several options for theobject-manipulating action a manip .
In the other case, where focus is switched to a different object, i.e.a id = 1 , there is no value of the object-manipulation action a manip that relates x and x ′ .
To formallyrepresent this in the decision tree, we assign all probability to some unknown a manip in that case.
So far we have not described how each class of action a manip , and therefore its copy EC, isparametrized by the motor controller.
Based on the fact that EC conveys to the observer the actionthat himself is taking, it is appropriate to (f) use an egocentric representation of actions instead ofaligning the actions with an allocentric reference point, i.e. with the environment.
To do so, noticethat the spatial transformations introduced in (d2) are affine, thus can be represented with theirassociated transformation matrices.
This logical parametrization allows us to conveniently computethe egocentric action that is needed to turn x into x ′ :
By multiplication of the transformation matrixthat gave rise to one view from the original with the inverted transformation matrix that gave rise tothe other.
This highlights a difference to the allocentric representation of actions that was chosen inother work (Lee et al., 2021), where transformations were aligned to the original, allocentric referenceframe (i.e. differences of scales rather than their quotient as it would emerge here).
3.2 Formalism.
Through the preceding dependency structure (Figure 2B, C), the inverse model naturally decomposesinto two more specific inverse models, where one is attributed to object-identity-related actions,and the other to the same-object manipulations, to which we simply refer to as “ identity-relatedinverse model ” q θ ( a id | x, x ′ ) and “ manipulation-related inverse model ” q θ ( a manip | a id , x, x ′ )respectively.
Therefore we have that q θ ( a | x, x ′ ) =
q θ ( a id | x, x ′ )
q θ ( a manip | a id , x, x ′ ) .
Applying the same also for the ground truth p EC enables us to split the loss function into separateparts L =
L id + L manip that reflect learning of the identity-related inverse model and learning of the manipulation-related inverse model correspondingly.
More precisely, the loss dedicated to the identity-related inverse model is given by L id = D KL ( p EC ( a id | x, x ′ ); q θ ( a id | x, x ′ )) ,while similarly, the loss dedicated to the manipulation-related inverse model is given by
L manip =
D KL ( p EC ( a manip | a id , x, x ′ ); q θ ( a manip | a id , x, x ′ )) , see also Appendix D for details.
In practice, we also include regularization losses L reg , see Appendix C, and weight the relativeimportance of the loss terms by hyperparameters λ .
Therefore, the loss that we consider is given by:
[Equation]
Instantiation of the inverse models.
The specific formulation of q θ ( a | x, x ′ ) determines the lossfunction that will be optimized.
We consider several options for the formulation of q θ ( a id | x, x ′ ) thatallow us to recover various contemporary approaches for SSL as we discuss in Section 4 (Results).
On the other hand, for the manipulation-related inverse model, we opted for a categorical actionrepresentation that clusters similar actions as advocated in key principle (e).
We implemented this bysubdividing the support of each single component a k, manip of a manip (consisting of 6 componentsfor the affine transformation) into a number of K discrete bins.
We indicate these discretized versionsof the real actions with a hat (cid:98) · and define the probability of a k, manip being in bin b as:
[Equation]
where we have introduced a feature extractor f (ResNets in our case, see Figure 1) and the functionsψ k,j (for which we used MLPs) to express the model’s belief that a k, manip assumes a value in discretebin j .
Note that the functions f and ψ both are learnable, but the dependence on θ is omitted forbrevity.
We refer to Figure 3 (Appendix E) for an ablation study on alternative instantiations of themanipulation-related inverse model.
Results
4.1 Recovering common SSL methods through the identity-related inverse model.
The specific instantiation of the identity-related inverse model allows us to recover several commonapproaches for SSL using the concept of ECs.
In particular, we show that with suitable choice ofq θ ( a id | x, x ′ ) one can recover contrastive losses (or equally instance discrimination) such as employedin SimCLR (Chen et al., 2020a) from the identity-related loss, see below and Appendix D.7.1.
Similarly, we show in Appendix D.7.2 that a different formulation of q θ ( a id | x, x ′ ) will give rise tothe loss used for BYOL (Grill et al., 2020) (a non-contrastive approach), while it is also possible toreduce the identity-related inverse model to ReLIC (Mitrovic et al., 2021), see Appendix D.7.3.
In the following, we consider the example where the identity-related inverse model gives rise to thecontrastive loss of SimCLR (Chen et al., 2020a).
Identity-related inverse model for obtaining instance discrimination SimCLR.
It turns out thatan optimization of the instance discrimination task, as often used in SSL, implements learning ofa suitably chosen identity-related inverse model.
For that, we define the probability of no switchin identity, i.e. a id = 0 , in the same way as used for the positive view in common contrastivelearning (Chen et al., 2020a), while adopting the notation provided by Mitrovic et al.
[Equation]
Here, ϕ is a function that computes a similarity between the features produced by f , i.e. it comparesthe intermediate sensory representations.
We defined it as a scalar product between projected features:ϕ ( h, h ′ ) :
= (cid:80) i g i
( h ) g
i ( h ′ ) , whereas g is a multi-layer perceptron (MLP), following typical choicesin the literature (see also Appendix).
The scalar τ is a temperature hyperparameter, and the set C iscomposed of additional candidate inputs to which x is compared to (through the denominator).
Notethat g is learnable also, thus depending on θ .
In order to obtain the commonly used instance discrimination loss for learning the identity-relatedinverse model, we will now assume that the EC is a perfect copy of a , hence p EC ( a | x, x ′ )
assignsall probability to the true action a that was applied.
In doing so, we obtain an upper bound of theobjective (2) that we use for our S-TEC experiments.
Its associated component dedicated to theidentity-related inverse model is the typical contrastive learning objective:
[Equation]
where we introduced x ′′ to always represent an input that is related to x through a id = 0 , see
Appendix D for the derivation.
We use x ′′ also for the second term, relating to the manipulationrelated inverse model, to reflect that the loss is only applied in that condition.
For additionalconvenience we write j ′′ to refer to the bin in which the value of a k, manip , relating x and x ′′ , falls.
4.2 Experimental evaluation
[Table]
In the preceding Sections we have derived our framework that connects the concept of ECs tocurrent methods for SSL.
Here, we present the results of our experimental evaluations that aim toassess the quality of the representations that can be learned with our approach.
For this purpose, weconsidered various image datasets, including CIFAR-10/100 (Krizhevsky, 2009), STL-10 (Coateset al., 2011) as well as the ImageNet ILSVRC-2012 dataset (Russakovsky et al., 2015), and compare
S-TEC against several other SSL algorithms, such as SimCLR (Chen et al., 2020a), MoCo v2 (Chenet al., 2020c), BYOL (Grill et al., 2020) and ReLIC (Mitrovic et al., 2021).
We follow the sameprocedure for all experiments, where we first performed SSL and subsequently determine the classprediction accuracy of a linear classifier that is trained on the emergent representations.
During SSLthe same data augmentation methods as in (Chen et al., 2020a) are applied throughout, i.e. also colouraugmentations, but these were not considered for training an inverse model with S-TEC, see also
Appendix A for details to the augmentations applied.
We adopted the ResNet v1 framework (Heet al., 2016), and used specifically ResNet-18 and ResNet-50 architectures as the feature extractor f ,while ϕ and ψ were generally implemented as multi-layer perceptrons (MLPs), see Figure 1B, and
Appendix B for architectural details.
These networks were optimized during SSL by gradient-descentusing the adaptive rate scaling of the LARS algorithm (You et al., 2017) with learning rate warmupand decay.
If not otherwise stated, SSL was performed for 1,000 epochs.
See Appendix C for detailsto the optimization with SSL and for details of training the linear classifier.
For all our comparisonsand ablation studies, we stress that the overlap in the implementation is maximal .
Especially forthe comparison between SimCLR and S-TEC (and between ReLIC and S-TEC ∗ )
, we emphasize thatonly loss functions are changed and manipulation-related inverse models are added.
As mentioned in Section 4.1, the identity-related inverse model can also be instantiated based onother methods for SSL, such as ReLIC (Mitrovic et al., 2021) through suitable choice of q θ ( a id | x, x ′ ) ,see Appendix D. We denoted this specific variation with S-TEC ∗ (i.e. target networks etc.).
[Figure]
CIFAR-10/100 and STL-10.
We report the accuracies that linear classifiers could attain after SSLon the respective datasets in Table 1.
Using 5 independent runs for each setting that we consideredrevealed that the manipulation-related inverse model in the case of S-TEC or S-TEC ∗ consistentlyincreased the accuracy of a linear classifier over the respective baseline.
[Table]
[Table] [Table]
We also compared our approach with the resultsof Lee et al.
(2021), who considered a similar augmentation-aware training setting, that wasmainly focused on the transferability of representations between domains.
We considered the case inwhich their method exhibited the strongest improvement on STL-10, see Table 6 of (Lee et al., 2021)(“ crop ”), and retrained their model using the samenumber of epochs (1,000).
We used their implementation and employed the same image augmentationsas we did.
Results are shown in Table 2.
Conversely,we also tested our methods in a 200 epoch trainingbudget, as originally done by Lee et al.
(2021) andincluded the best reported performance that theyobtained, see Table 7 of (Lee et al., 2021), notingthat our method did not outperform in this case.
To investigate the differences between our approachand that of (Lee et al., 2021), we conducted an ablation study exchanging the egocentric action representation that we used with an allocentric one.
Additionally, we examined the impact of replacing actionclassification with L2 regression.
We performed experiments on CIFAR-100 using ResNet-18s, findingthat Egocentric+Classification yielded 66.6 % accuracy over the next best setting Allocentric+L2 Regression with 66.1%, which was employed by Leeet al.
(2021), see Figure 3A (and Appendix E).
Wehypothesize that classification affords the modelmore flexibility in its output distribution, thus it canhandle uncertainty of its action prediction better.
ImageNet.
We experimented with ImageNet
ILSVRC-2012 (Russakovsky et al., 2015) to demonstrate that S-TEC and S-TEC ∗ also scale.
We performed training for either 100 or 300 epochs andreport the results in Table 3, confirming that S-TEC is not restricted to small datasets.
Semantic segmentation SSL aims to install generally useful representations into models.
Thus,we considered a further downstream task beyond classification: semantic segmentation on PASCAL
VOC (Everingham et al., 2010) using fully convolutional networks (Long et al., 2015) with a ResNet50 backbone.
We initialized the layers of this backbone with the parameters that resulted from SSLon ImageNet for 300 epochs, and then fine-tuned the entire network on the new task (4 runs withdifferent initializations of the decoder).
Evaluation was performed on the val2012 split and the meanintersection over union (mIoU) is reported in Table 4.
[Figure]
Hyperparameter dependence and further details on learning dynamics.
To assess the dependence of our results on hyperparameters, introduced through S-TEC, we carried out several studieson CIFAR-100 with ResNet-18s.
Specifically, we performed a sweep over λ manip that scales theimpact of the manipulation-related loss, as well as a sweep over the number of bins K used in theclassification for action components.
Performance depends significantly on the choice of λ manip , butis less affected by the number of bins as long as there are not too many (i.e. K < 10 ), see Figure 3Bfor results.
Lastly, we also exhibit the loss dynamics and learning progress in Appendix E.3.
Audio (LibriSpeech)
Finally, we also attempted to improve representation learning in the audiodomain.
Specifically, we considered the same data and model as introduced by Oord et al.
Inaddition to the time-sensitive identity-related inverse model, as it emerges with CPC, we also addeda time-insensitive identity-related inverse model.
This allowed us to achieve 65.4 % accuracy onphoneme classification (with frozen features) as opposed to 65.1% that we obtained with CPC.
Analysis and intuitions
Increased information content.
A consequence of learning a manipulation-related inverse modelis that additional information must be expressed by the feature extractor f .
Curiously, it had beenshown in several other works that strong methods for SSL only perform well on downstream tasks,such as classification, if intermediate representations are used.
E.g. Chen et al.
(2020a) showed thatinserting a nonlinear MLP between the feature extractor f and the loss for contrastive SSL resultedin significantly better performance on subsequent linear classification, as opposed to the control casewhere this MLP was missing.
This effect is explained by loss of information that is not necessarilyimportant for the contrastive SSL objective, but for downstream tasks.
In fact, recent work (Chenet al., 2020b; Mitrovic et al., 2021) observed better performance if the depth of MLP was furtherincreased.
This supports the viewpoint that additional information, albeit being potentially redundantto the contrastive SSL objective, is desired for downstream tasks of interest, see also Lee et al.
Better organization of class borders.
A different perspective potentially opens up new avenues toimprove SSL: We conjecture that learning the additional manipulation-related inverse model using
L manip for S-TEC, see equation (2), encourages the feature space to be better organized.
Firstly, notethat contrastive SSL promotes local clusters of representations that group single images and theiraugmentations (Wang and Liu, 2021), see Figure 4A.
In turn, the single images will be linked to eachother by means of overlapping augmentations (Mitrovic et al., 2021) and form extended clusters thatrepresent classes.
The representations of these links, which are likely given by strongly augmented(cropped) images, are placed at the convenience of the SSL objective, and can potentially also settlewithin the center region of specific classes.
We argue that it is semantically more meaningful if suchlink-forming, potentially strongly augmented, images are located near nascent class borders.
Thus, creating more sensible bridges between these boundaries, while at the same time, increasing thedistances between clusters of unaugmented ones , see Figure 4B.
Consider the example in Figure 4A, B, which visualizes images depicting a truck with its associated“tire” augmentation, and analogously, a plane with its “tire” augmentation.
In that example, S-TECimplicitly encourages representations of more strongly augmented images, the “truck tire”, to belocated closer to the border, since they share a similar magnitude of augmentation with stronglyaugmented images of other class, the “plane tire”, due to the manipulation-related inverse model.
Thus, the representations of images that are not distorted will be more concentrated for S-TEC (Panel
B), as compared to standard SSL that are not aware of the magnitude of augmentation (Panel A).
Qualitatively, this hypothesis is supported in Figure 4C, D, E and F, where we visualize the featurescomputed by a ResNet-50 on images of the testing set of CIFAR-10, including also their augmentations.
To do so, we computed lower-dimensional projections by the means of UMAP (McInnes et al.,2018), and color-coded the magnitude of augmentation (defined as 1 minus the relative area of thecropped image).
Comparing Figure 4C and D, the model trained with S-TEC (Panel D) is aware ofthe zoom level of the augmentation and places more augmented images in similar regions, i.e. theborders, while the model trained with a standard SSL objective (SimCLR in this case) is oblivious toit (Panel C).
If we compute the embeddings of the original, unaugmented, images using the precedingprojection, as displayed in Figure 4E and F, the apparent class center distance increases for S-TEC(Panel F), due to the now missing augmented images on the border.
Quantitatively, we computed for each class separately the distance between the centroid of augmentedimage representations and the centroid of unaugmented image representations.
Averaged over allclasses, we find that this distance is 10.5 for S-TEC and 0.4 for SimCLR, noting that both latentspaces exhibit similar scales.
This further confirms that SimCLR clusters these subsets (augmentedand unaugmented images) of one class around a single centroid, whereas S-TEC separates them.
Related work
SSL through auxiliary tasks.
The idea of SSL by solving high-level queries about input manipulations was considered previously: E.g. Doersch et al.
; Noroozi and Favaro (2016) proposed totransform input images into patches and attempted to solve context prediction and jigsaw-puzzlesrespectively, while others found it useful to predict a prior rotation transformation (Gidaris et al.,2018).
In contrast to such spatial prediction tasks, a different line of work by Zhang et al.
(2016)discovered that colourization of black-and-white images also creates useful features for downstreamtasks.
Since aforementioned auxiliary tasks are orthogonal at large, prior works studied combinationsand/or extensions of those (Doersch and Zisserman, 2017; Zhang et al., 2019).
Contrastive SSL.
Opposing to the preceding strategies of training on (handcrafted) auxiliary tasksare algorithms that originated from the idea of mutual information (MI) maximization between theinput and representations thereof.
In particular, the prevalent strategy of mini-batch training renderedit practical to compare – and contrast – representations of different and related inputs, which enablesthe maximization of MI (Gutmann and Hyvärinen, 2010).
Following this perspective, algorithms fordeep networks were introduced (Oord et al., 2018), and further progress ensued, where the commonblueprint for the algorithm is to transform input into pairs and to contrast those against other, unrelatedones (Henaff, 2020;
He et al., 2020; Chen et al., 2020a,b, 2021).
Tschannen et al.
(2020) opened adiscussion whether it works well due to maximization of MI, which resulted in the search for differentexplanations, e.g. through causal interventions (Mitrovic et al., 2021; Tomasev et al., 2022).
On theother hand, the profound utility of contrastive SSL has also inspired other research that attempts toconnect it to hypothesized learning mechanisms in the brain.
For instance, Illing et al.
(2021) showthat contrastive SSL can give rise to deep representations with local learning rules.
Non-contrastive SSL.
Meanwhile, several other studies explored non-contrastive avenues toachieve SSL, which have gained more traction due to their attractive properties, such as not needingnegative examples.
Some are based on a form of bootstrapping that requires the network to produceconsistent representations under content-preserving input transformations (Grill et al., 2020; Chenand He, 2021), hence dispensing with contrastive samples altogether.
Furthermore, Caron et al.(2020) demonstrate that enforcing consistency between cluster assignments can install potent featureextraction capabilities into a model, while also not requiring pair-wise contrasting.
Augmentation-aware self-supervision.
Algorithms based on contrastive SSL typically aim forrepresentations that are invariant to input transformations.
While this seems appropriate in principle,subsequent studies have shown that this is not always favourable, as this strategy can exclude certaininformation from representations that could otherwise make them more useful for varying downstreamtasks (Xiao et al., 2021).
Based on similar arguments, Lee et al.
(2021) proposed to predict differencesof certain transformation parameters in addition to training on a standard SSL objective (Chen et al.
,2020c,a; Chen and He, 2021) to improve the transferability of learned representations to otherdomains.
These are promising results; however, on the main performance tests of SSL, i.e. testingthe representations in the same domain as the training domain, these prior augmentation-awareapproaches have not achieved the same performance advantage as compared to S-TEC, see Table 2.
The relation of our approach to prior work.
Our approach generalizes methods that bringrepresentations of paired inputs close together into a framework that also introduces semanticstructure between paired inputs: by using their known relationship, i.e. transformations between them.
This generalized and unified framework emerges from the concept of ECs and the feedback that itprovides to create an inverse model.
As a result, our approach is also augmentation-aware, but itsfoundation on sensory-motor principles and neuroscience instructs important elements (Section 3)that are missing from the earlier augmentation-aware approaches (Xiao et al., 2021; Lee et al., 2021),but have been studied in part by research on local learning (Illing et al., 2021).
Currently, contrastive
SSL is one of the dominant approaches in the literature, and our approach improves it in our tests,see Table 1 and 3.
Moreover, we also confirmed that the benefits of our approach transfer to other,orthogonal improvements, such as provided by ReLIC, see same Tables.
Conclusion
S-TEC provides a theoretical framework that is derived formally from first principles of biologicalsensory-motor control, unifies several important SSL approaches, and extends them through thecombination of two inverse models.
We showed that designing S-TEC’s action representations and theinverse model in a biologically-principled way is crucial for improving performance.
Performance isconsistently better over several strong baselines in image classification tasks and image segmentation.
S-TEC as a framework provides a different angle from which SSL could be further improved in thefuture.
Moreover, S-TEC is based on and constrained by established biological principles.
As aresult, its conclusions may feed back to neuroscience, suggesting that the learning of inverse modelsand the availability of ECs positively impacts sensory skill.
This hypothesis that our work heregenerated, is testable.
It predicts that subjects exposed to a motor learning task in a novel sensoryenvironment through active movements, would perceive the new environment better than participantsthat only experience passive exploration of the environment.
S-TEC’s biological implications couldbe strengthened even within the computational setting, by using optimization algorithms that are morebiologically plausible than backpropagation.
Multiple such options have recently been described, andsome have already shown to be applicable to SSL (Illing et al., 2021).
Adding further biophysicaldetail to the neural networks, such as spiking neurons, could further enhance its biological relevance.
Limitations.
We have shown that S-TEC generalizes and extends contrastive SSL as proposedby Chen et al.
(2020a)
; Mitrovic et al. (2021).
In the future, we would like to address this limitation,and show similar improvements based on ReLICv2 (Tomasev et al., 2022).
In addition, we haveemployed a limited set of actions (given by basic augmentations used for SSL).
We believe furtherimprovements can be obtained by introduction of broader actions/augmentations.
Another potentiallimitation is that we have explored the improvements solely for ResNet architectures (He et al., 2016).
Hence, a target for future research is to incorporate architectural improvements, and demonstrate theadvantage of S-TEC in the presence of self-attention (Vaswani et al., 2017).
Potential negative societal impacts.
Our approach S-TEC provides an improvement to contemporary SSL, thus inheriting potential negative societal impacts of SSL.
Since S-TEC, as also SSL, canbe applied to unlabelled (uncurated) data, our concern is that a malicious application of S-TEC tohealth databases can help adversaries to exploit patients, by creating better profiles through the richerfeedback that we propose.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] Will be in thesupplemental material
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] Will be in the
supplemental material
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[No]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of
instructions given to participants and screenshots, ifapplicable?
N/A
(b) Did you describe any potential participant risks, with links to Institutional Review
Board
(IRB) approvals, if applicable?
[N/A
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A Experimental methods: Input transformations
In practice, mini-batch training was performed, hence we applied input transformations to eachdatapoint twice to ensure that for every x , there is always one x ′′ related through a id = 0 , i.e. havingthe same underlying object identity.
As a result, using a batch size of B different images will cause2 B images being processed at a time.
We exhibit below the complete list of data augmentation methods in the order that they were appliedduring SSL.
Note that only Random crop and Random horizontal mirroring were considered as partof the same-object manipulations a manip , see Section A.2.
Random crop.
For each transformation we randomly extracted a patch of the image with an area sampleduniformly between 8% and 100% of the original area with an aspect ratio sampled loguniformly between 34 and 43 .
This patch was resized to 32x32, 96x96 or 224x224 pixels for
CIFAR-10/100, STL-10 or ImageNet respectively (using bilinear interpolation).
Random horizontal mirroring.
For each transformation we mirrored the image separately with a probability of 50% horizontally.
Random color jittering.
With a probability of 80%, we randomly altered separately for each transformation thebrightness, contrast, saturation and hue in a random order.
More accurately, brightness wasadjusted by multiplication of the pixel values with a factor that was uniformly sampled in[1 − u, 1 + u ] (I.e. multiplicative change in brightness).
The contrast was adapted by scalingthe distance of the pixels from their mean, i.e. a ( x − µ ) + µ , where µ was the average pixelvalue of the image, and a was sampled uniformly in [1 − u, 1 + u ] .
The saturation wasadapted similarly, but in this case the mean was computed per pixel location and weighted(red: 0.2989, green: 0.587, blue: 0.114).
The change in hue was performed in HSV color
space by adding to the H channel a value sampled uniformly in [ − u, u ] modulo 1.
Random conversion to grayscale.
For each transformation the image was converted separately to grayscale with a probability
of 20%.
For this conversion the same weighting strategy as employed during saturationchange described above was used.5.
Random gaussian blur (Only for STL-10 and ImageNet).With probability of 50%, we applied a Gaussian blur filter separately for each transformation.
This filter had kernel edge dimensions of 10% of the image width and height (rounded touneven edge lengths)
and used a standard deviation that was sampled randomly for each
transformation uniformly in [0.1, 2.0] for size 224x224 and scaled proportionally in case ofother dimensions.
Random solarization (Only for STL-10 and ImageNet).
With a
probability
of
, we also applied solarization of the image for each transformation
separately.
This was performed by inverting pixels with a value above 0.5 (assuming a pixelvalue range of 0 to
Here,
inversion
refers to a mapping x
(cid:55)→
1 −
x
Note that we excluded the loss for the manipulation-related inverse model L manip if eitherx or x ′ had been solarized.
A.1 Object-identity-related actions a id
Since we applied training in mini-batches with B different images, the action a id = 1 is simply givenby x and x ′ corresponding to different image identities in the batch.
A.2 Same-object manipulations a manip
The same-object manipulations a manip , as introduced in Section 3.1 in the main manuscript, are onlyapplied if the object identity of x and x ′ remains the same, which is the case when a id = 0 .
The same-object manipulations a manip , as defined in our setting, took into account only spatialoperations: Random crop and Random horizontal mirroring .
To represent this action, we first computed for each transformation the affine matrix M x that generates the particular cropped view ofx from the original image.
More specifically, this matrix M x transforms points on the canvas of thenew view x to the points on the canvas of the original image, i.e. M x determines the source positionof the new pixels.
To compute M x , let w x
and h x denote the width and height of the crop in pixels as sampled fromthe Random crop operation.
In addition, let l x ( t x ) be the distance of the crop’s left (top) edge fromthe original image’s left (top) edge.
Additionally, let W and H denote the width and height of theoriginal image respectively.
Furthermore, let f x be -1 if the Random horizontal mirroring operationdictates a mirroring and 1 if not.
With these definitions M x is defined by:
[Equation]
Since we identify with a manip the action that turns x into x ′ , we are interested in the affine transformation matrix M x → x ′ that transforms x to x ′ (i.e. it computes the source location of pixels in x ′ onthe canvas of x ).
It is given by:
[Equation]
Finally, we identify the spatial action a manip with the two top rows of this matrix:
[Equation]
Categorical targets.
In order to classify the values of the matrix of a manip , we subdividedthe interval of values that can be assumed into K = 6 bins.
For that we first define limitsmanip min =
( − 2 , − 2 , − 0 .
5 , − 2 , − 2 , − 0 .
and manip max = (2 , 2 , 0 .
5 , 2 , 2 , 0 . 5) , which ultimatelyallows us to express the discretized (cid:98) a k, manip of the main manuscript in Section 3.2 as:
[Equation]
B Experimental methods: Architectures
The architectures for feature encoder f were residual convolutional networks as introduced by Heet al.
(i.e. ResNet v1).
More specifically, we used ResNet-18 or ResNet-50, depending on theexperiment, and used the activations after global average pooling as the output of f .
The functions ϕ , for the identity-related inverse model, and ψ , for the manipulation-related inversemodel, were based on multilayer perceptrons (MLPs) with batch normalization and rectified linearactivation (ReLU).
B.1 Identity-related inverse model ϕ .
The function ϕ was defined by a cosine similarity of the outputs of an MLP g :
[Equation] [Equation]
where the MLP g had 1 hidden layer with batch normalization and ReLU activations.
Batch normalization was also used for its output (except when target networks were used in S-TEC*).
The numberof hidden and output units of g differed among experiments, see Table 5 for concrete dimensions.
B.2 Manipulation-related inverse model ψ .
The manipulation-related inverse model ψ was defined in the main manuscript as a function of twofeature vectors.
It was implemented as an MLP, applied on the concatenation of both inputs, with onehidden layer that contained 512 units with batch normalization and ReLU activation.
The output ofψ was 36 dimensional in total, producing predictions for each component a k, manip , in which of the
K = 6 bins its value falls.
[Table]
B.3 S-TEC*
We also experimented with ReLIC Mitrovic et al.
(2021), which modifies the approach by introducingtarget networks, and by adding a regularization term, see Section C.3.
Due to the use of targetnetworks, a modified definition of q θ is necessary:
[Equation]
where we used ˜ f as a target network that follows the weights of f using an exponential movingaverage (Mitrovic et al., 2021) with same decay properties as in (Grill et al., 2020) using an initaldecay τ = 0 .
In addition, the exponential moving average was also applied to the MLP g suchthat ˜
[Equation]
( a
[Equation]
B.4 MoCo
For our implementation of MoCo v2 (Chen et al., 2020c), we used target networks ˜ f and ˜ g with thesame exponential moving average schedule as used by (Grill et al., 2020) with an initial decay ofτ = 0 . 99 .
All other architectural settings were kept equal to the SimCLR setting, see Table 5.
Thesize of the dictionary, i.e. the bank of contrastive embeddings, was set to 65k.
B.5 BYOL
For our implementation of BYOL, we followed the architectural principles as provided by Grill et al.(2020).
However, for CIFAR-10/100 we used a hidden dimension of 512 for projection and predictor,as well as an output dimensionality of 64.
For STL-10, we used a hidden dimension of 2048 with anoutput dimensionality of 128.
C Experimental methods: OptimizationC.1 SSL phase
We used stochastic gradient-descent with a momentum of 0.9 along with the LARS adaptive learningrate mechanism (You et al., 2017), but excluded batch normalization and bias parameters from it.
Weused a batch size B of 1024 for all our experiments, except for those on ImageNet, where we used a
B = 1680 .
Recall that B denotes the number of different images, each of which was subject to 2augmentations, resulting in 2 B images processed at a time.
We employed linear scaling of the learning rate with respect to the batch size, with cosine learningrate decay and with 10 epochs of linear warmup, see Table 6 for learning rates per 256 batch size.
Global weight decay was used as part of L reg with a coefficient of 10 − 6 .
Specifically for the ResNet, we note that the last batch normalization layer in each residual block wasinitialized with zero scale to stabilize training (Goyal et al., 2017).
C.2 Linear classification on frozen features
For classification we trained a linear classifier on top of the frozen features using stochastic gradientdescent.
We trained this linear classifier alongside SSL training in the cases of CIFAR-10/
[Table]
and ImageNet, but without propagating gradients into the ResNet feature extractor f (i.e. we usedstop_gradient for the classification loss), noting that similar results were achieved with a subsequent optimization protocol consistent with (Chen et al., 2020a).
Specifically for STL-10, we trained the linear classifier separately in a subsequent optimizationprocedure with stochastic gradient descent and Nesterov momentum of 0.9, using a learning rate of0.01 per 256 batch size (we found the value of 0.175 to work best in the case of BYOL and MoCo).
In this case, only random cropping and random horizontal mirroring were used as augmentationmethods.
This optimization program was carried out for 2,000 epochs using cosine learning ratedecay and 10 epochs of warmup.
For the weights of the linear classifier (excluding bias), a weightdecay regularization with a coefficient of 5 · 10 − 4 was used.
C.3 Regularization in S-TEC*
For S-TEC* we used target networks along with an additional regularization as proposed in (Mitrovicet al., 2021).
The goal of the added term is to ensure invariant instance discrimination with respectto data augmentation.
Recall the definition of q θ ( a id =
| x, x 1 , θ ) in equation (3) of the mainmanuscript.
It introduces a probability distribution over instances x 1 .
We denote the emergingdistribution by q c 1 ,θ ( x 1 ) .
Likewise, we also introduce a distribution over instances x 2 that emergesby q θ ( a id = 0 | x 2 , x ′ , θ ) which we call q c 2 ,θ ( x 2 ) , see also Section D.3 and D.7.3.
The additionalregularization then consists in penalizing differences in these two distributions in terms of a KullbackLeibler divergence:
[Equation]
which was added to L reg using a coefficient of 1.
D Derivations and additional theoryD.1 Sketch of the derivation
We begin from the definition that poses the optimization of the inverse model as a minimization ofthe Kullback-Leibler divergence as defined in equation (1) of the main manuscript.
Based on thisand on biologically-inspired assumptions (see Section 2 of the main manuscript), the inverse modelbecomes a classifier of pairs of sensory inputs into actions.
Since we introduced the action as being composed of two categories, namely Object-identity-relatedactions and Same-object manipulations (see Figure S2), we can decompose the loss into a sum oftwo losses, L id and L manip respectively (see Section D.9).
This, in turn, allows us to learn the twoassociated inverse models separately.
Subsequently, we elaborate on the assumption that this model represents an embodied natural setting.
From there, we show that different SSL methods emerge from S-TEC, depending on the specifics ofthe classifier’s mathematical definition and the corresponding EC-based learning.
The methods werecover include existing and proven ones, such as SimCLR (Chen et al., 2020a), BYOL (Grill et al.,2020), or ReLIC (Mitrovic et al., 2021).
D.2 Concretizing the inverse model as a classifier
The Kullback Leibler divergence loss of the inverse model (equation (1) of the main manuscript)is computed between (a) a probability distribution over true actions, which are copied through EC(given pairs of sensory inputs), and (b) the modelled probability distribution over actions, which is provided by the inverse model (given pairs of sensory inputs).
Grounded on biological evidence (see
Section 2 of the main manuscript), we assume that the probability distribution of the EC reflects aperfect copy, and that the probability distributions involved are discrete.
The inverse model estimates which actions were executed that caused the sensory inputs to changefrom x to x ′ .
Based on biological evidence (see Section 2 of the main manuscript), we chose torepresent this inverse model as a classifier q θ ( a | x, x ′ ) (parametrized by θ ) that assigns probability tospecific actions, given the inputs x before the action was executed, and the inputs x ′ after the action’sexecution.
We identified two categories of actions in Section 3 of the main manuscript, and therefore chose toview actions as being composed of two components a =
( a id , a manip ) .
As a result, two sub-inversemodels (sub-classifiers) can be defined:
• the identity-related inverse model q θ ( a id | x, x ′ ) , and
• the manipulation-related inverse model q θ ( a manip | a id , x, x ′ )by means
of q
θ ( a | x, x ′ ) =
q θ ( a id | x
x ′ )
q
θ ( a manip
| a
id
, x, x
, see also Figure S2.
In turn, in order toinstall the function into this classifier, we defined a loss: the KL divergence in equation (1), whichcan be decomposed into a sum of two divergences, according to the aforementioned factorization.
See Section D.9 for proof that the decomposition is equivalent to the original loss.
This allows us notonly to specify the inverse model as two separate classifiers, but also to learn them separately.
Identity-related
inverse model
The identity-related inverse model has to effectively solve a binaryclassification problem for a id to identify whether the main object of the sensory inputs x is the sameas the main object of the subsequent sensory input x ′ (in which case a id = 0 ).
The probability that isassigned to this event is denoted by:
[Equation]
The specific implementation of the identity-related inverse model (i.e. the classifier for a id ) can use avariety of criteria to determine whether two sensory inputs represent the same object identity.
It istypically based on the similarity of the sensory inputs (more accurately the representations thereof).
Examples of such (dis-)similarity measures include (a) dot product , (b) mean squared error , (c)
KL-divergence , and more could be envisioned.
Manipulation-related inverse model As commented in the main manuscript, the manipulationrelated inverse model has to infer which manipulations were performed, when the same underlyingobject remains in focus, i.e. a id = 0 .
Also this inverse model can be conceived in various forms,depending on which interactions and manipulations are possible for the same object.
For manipulationof static images, an example is to classify the components of affine transformation matrices as weproposed in Section A.2.
D.3 Concretizing the embodied natural setting
To account for a generic and natural setting, we assume that the inverse model’s parent entity, i.e. theobserver or agent that performs the actions, may have contextual information from the environment inaddition to the observed x and x ′ .
More specifically, we assume that in addition to the perception ofx as the main object, the agent also perceives some additional context C pre .
Likewise, after executionof the action a , we assume that the agent perceives x ′ and some additional context C .
In summary,the agent perceives the set { x } ∪ C pre before executing the action a , and { x ′ } ∪ C thereafter.
Wefurther specify that in the case where a id =
the identity of x and x ′ is the same.
On the other hand,in the case that the agent switches the focus to a different main object ( a id = 1 ), then the identity ofthe object x remains in the broader context of the agent and is still represented in the set C in someform.
We additionally assume that the agent has prior knowledge about the physical environment’s conservation laws, i.e. that one object cannot take more than one identity, and that objects do not vanishwithout cause.
Altogether, this prior knowledge imposes a constraint on the probabilities of x havingthe same identity as some other x n in the context.
More accurately, we say the identity of x must be conserved in the set { x ′ } ∪ C after taking the action:
[Equation]
which is imposed on the identity-related inverse model.
As we will show, using the entire context { x ′ } ∪ C during learning or only x ′ , determines if theemerging SSL approach with S-TEC belongs to the contrastive category of methods or not.
D.4 Learning the identity-related inverse model for a id
D.4.1 Using context
When the entire context { x ′ }∪ C is available during learning, utilizing the assumptions and emergingconstraints from Section D.3, we arrive at two implications:1.
Learning the identity-related inverse model q θ ( a id | x, x ′ ) consists in maximizing the probability in equation (13), if the EC dictates that x and x ′ share the same object identity.
Onthe other hand, if the EC dictates that the identity of x does not match the identity of x ′ , theprobability in equation (13) is to be minimized.2.
Consider specifically the case where x and x ′ do not share the same identity.
It was assumedthat the original x stays preserved in the context C in some, possibly altered, form.
Wedenote this preserved item sharing the same identity by x ′′ ∈ C .
Through conservation inequation (14) it follows that maximizing the probability q θ ( a id = 0 | x, x ′′ ) has as a resultthe minimization of the probability q θ ( a id =
| x, x n ) for all other x n ∈ { x ′ } ∪ C \ { x ′′ }“negative” objects in the context, therefore explicit separate minimization for the negativeexamples is not necessary.
Furthermore, this type of learning can use directly the conservation in equation (14) for mutualcomparison of x with the items inside the set { x ′ } ∪ C , typically through the use of normalization ofsimilarities (e.g. “softmax” of similarity scores), such as how it was defined in equation (3) of themain manuscript.
We refer to Section D.10 for a concrete proof of the second implication, following the same idea thatexplicit minimization for “negative” objects is not necessary, which then connects this to the upperbound objective in equation (5) in the main manuscript.
D.4.2 Not using context
If contextual objects (see Section D.3) are unavailable, then learning the identity-related inversemodel can still be implemented by a formulation of the probability in equation (13), solely on thebasis of the similarity between the representations of x and x ′ , without a comparison to the context.
That choice therefore implies a non-contrastive type of SSL.
By further specifying the options ofthe realization of this model, we show concretely in Section D.7.2 that existing non-contrastive SSLmethods emerge (Grill et al., 2020).
In this non-contrastive setting, where the task is to maximize the similarity between paired sensoryrepresentations, a trivial solution could be found, where all objects collapse to the same representation.(Grill et al., 2020)
As a result, a potentially trivial solution can occur, where all objects collapse tothe same representation, thus maximizing the similarity of all possible representations.
This has been recognized and it has been shown that such trivial solution can be mitigated byusing separate feature extractors for the two representations, and by optimizing them differently bylearning in different timescales (i.e. “online networks” and “target networks”)
(Grill et al., 2020).
Weconjecture that this complexity and its drawbacks are potentially not necessary when a complete ECis employed through S-TEC, since an additional classification task involving a manip must be solvedthat would naturally prevent such collapse, since it demands separation between representations ofdifferently manipulated views.
D.5 Learning the manipulation-related inverse model for a manip
On the other hand, we have established that in addition to the identity-related inverse model, therealso exists the manipulation-related inverse model that classifies which manipulations a manip wereapplied to an object, if the identity of x and x ′ remains the same (Figure S2C, a id = 0 ).
The learning procedure of the manipulation-related inverse model depends on the specific definitionof the model, as well as the type of manipulation actions a manip that it models.
In this work, weconsidered a manip as being composed of the components of affine transformation matrices (see mainmanuscript’s Section 2 for the motivation, and Sections A.2 and D.2 for details).
The correspondinginverse model for a manip was implemented as a classifier that predicts in which bin, i.e. class, thecomponents of the actions fell, see Section A.2.
As a result, learning the manipulation-related inverse model consisted of training multiple classifiersfor all the components of a manip , in other words minimization of the cross-entropy loss betweenthe target classes in which the components of the manipulation action a manip fell and the predictedclasses for these components.
D.6 Simultaneous training of two inverse models for a id and a manip
Ultimately we aim to train both of these inverse models simultaneously, which could be naivelycarried out by simply adding together the corresponding losses.
However, in practice, the tasksof the two inverse models can differ in their difficulty, and thus require a different weighting toenable learning of both simultaneously.
For this reason we have introduced a weighting factor λ manipthat scales the impact of the loss concerning the manipulation-related inverse in relation to the onecorresponding to the identity-related inverse model, see Section ??
for a sweep over this parameter.
Furthermore, since our main goal is to achieve the best possible sensory representations installedin one model, we want to share parts of the architecture for both inverse models regarding a manipand a id .
A direct consequence from doing so is that there may be an interaction between parts ofthe representation space relating to a manip and other parts of the representation space relating to a id ,which we briefly elaborated on in Section 6 of the main manuscript.
Another conjecture, as pointed out in Section D.4.2, is that the presence of the loss relating to a manipin addition to the loss relating to a id could help to prevent collapse of representations, although wehave not tested this hypothesis.
In a similar vein, the particular point at which the two inverse modelsextract the respective features for their further use, and their depth, may have significant impact onthe organization of representations.
D.7 Recovering prior SSL techniques from S-TEC
D.7.1 Recovering SimCLR
In case that the entire context { x ′ } ∪ C of objects is available and used during learning, we can obtainthe NT-Xent type of loss as used in SimCLR (Chen et al., 2020a).
This emerges from a specificchoice for modelling the identity-related inverse model’s inferred probabilities q θ ( a id = 0 | x, x n ) , asfollows.
By using the dot-product similarity between the representations of x and each example x n ∈ { x ′ }∪ Cin the context, and then applying a “softmax” operation to convert into probability, yields equation (3)of the main manuscript as the inverse model for a id .
In addition, statement (2) of Section D.4.1 isemployed, which poses the learning of q θ ( a id | x, x ′ ) as a maximization of q θ ( a id = 0 | x, x ′′ ) for ax ′′ ∈ { x ′ } ∪ C that shares the same identity as x .
This maximization of the softmax probability forthe “positive” pair is equivalent to minimizing its negative logarithm, which is, in fact, the NT-Xentloss.
In Section D.10 we also show formally that NT-Xent optimizes an upper bound to the original
Kullback Leibler divergence objective for learning the identity-related inverse model.
Thus, we haverecovered NT-Xent (Chen et al., 2020a) as a special case of our S-TEC framework, which has beenthe core mechanism in some of the best performing methods for SSL (Chen et al., 2020b, 2021).
D.7.2 Recovering BYOL
In the case where no context is available or employed during learning of the identity-related inversemodel, the “BYOL” approach (Grill et al., 2020) can be recovered if the identity-related inversesub-model of S-TEC is realized differently.
Specifically, in order to arrive at the approach of Grill et al.
(2020), we begin by defining our identityrelated inverse model q θ ( a id | x, x ′ ) in accordance to a normal distribution, such that its optimizationwill result in a mean squared error loss, which is what is used in BYOL.
Namely, we define:
[Equation]
15) where we have introduced ˜ f and ˜ g to indicate that these networks can be different from f and g butrelated.
In Grill et al.
(2020), they are related to the original network f and g via an exponentialmoving average (target networks).
The constant ϵ denotes a small number.
We then define that the probability assigned to a id = 1 (see Figure S2) is a small constant.
Since theprobability will generally not sum to 1 for these two cases, we formally introduce a id = 2 that doesnot occur in practice, i.e. p EC ( a id = 2 | x, x ′ )
= 0 :
[Equation] [Equation]
Inserting these definitions into the loss of S-TEC’S identity-related inversed model, L id =
D KL ( p EC ( a id | x, x ′ ); q θ ( a id | x, x ′ ))
(see Section D.9), recovers the approach of (Grill et al., 2020)
[Equation] [Equation]
Therefore, BYOL has emerged as another special case of S-TEC.
D.7.3 Recovering ReLIC
Finally, we hypothesize that the approach for ReLIC (Mitrovic et al., 2021) along with its assortedinvariance penalty can also be recovered from our framework if one postulates that both contexts{ x } ∪ C pre and { x ′ } ∪ C are available and used during learning.
In this case, the idea to arrive there is to base the classifier q θ ( a id = 0 | x, x ′ ) on two factors:
The probability of the context-aware model that was used to obtain NT-Xent, see Sec-
tion D.4.1, and D.7.1,
An overall confidence of the identity-related inverse model that is defined based on theconsistency between the contexts { x } ∪ C pre and { x ′ } ∪ C .
Consider specifically the second point that is added on top of what we considered already in the caseof SimCLR in D.7.1.
For the purposes of this derivation, we refer to the identity-related inverse modelthat emerges for obtaining SimCLR, see the definitions in D.7.1 and in D.10, as q (NT)
θ ( a id | x, x ′ ) .
Furthermore, we define probability distributions q c 1 ,θ = q (NT)
θ ( a id | x, x 1 ) with x 1 ∈ { x ′ } ∪ Cand q c 2 ,θ ( x 2 ) =
q (NT) θ ( a id | x ′′ , x 2 ) with x 2 ∈ { x } ∪ C pre in order to cross-compare probabilityassignments between the same objects in both contexts, see C.3 (recall that x ′′ ∈ C denotes the itemwith the same identity as x ′ ).
The consistency between these distributions is quantified by a KullbackLeibler divergence D KL ( q c 1 ,θ ( x 1 ); q c 1 ,θ ( x 2 ))
=:
D c 1 ,c 2 , and is used as an overall confidence for thepredictions of the inverse model in the following way:
[Equation]
while, similar to D.7.2, we add a a id = 2 that does not occur in practice such as to absorb theremaining probability: q θ ( a id = 2 | . . . )
= 1 − q θ ( a id = 0 | . . . )
− q θ ( a id = 1 | . . . ) .
Note that α is ahyperparameter.
It remains to substitute these definitions into L id , which yields:
[Equation]
Thus, we obtain the ReLIC method including its consistency loss (Mitrovic et al., 2021) by suitabledefinition of the inverse model in equation (19).
D.8 Summary: S-TEC as a generalization of SSL methods
From first principles of sensory-motor control in Neuroscience, and the assumption that learningoccurs in the physical world, we recovered prior SSL methods.
However the full S-TEC modelis broader, as it also includes a manip in its inverse model, which is not exploited by the methodswe recovered through a id .
In the main text’s Section 2, we showed that, from ECs a manip is partof the same framework, and, in our experiments and analyses in the other sections of the mainmanuscript, we showed that it is actually useful to combine the two, if implemented according to ECsand sensory-motor principles.
Moreover, from S-TEC’s framework, other powerful instantiations can be imagined.
For example,we have mentioned that possibly non-contrastive approaches without a target network could becomefunctional, by avoiding representation collapse, through a manip .
Further SSL concepts emerge byimplementing S-TEC’s elements differently, e.g. by using different technical implementations of theinverse-model’s classifier.
D.9 Decomposition of the loss
In the following we show how the decomposition of the loss function L into the two components
L id and L manip emerges.
Starting from the definition of the loss we can expand on the definitionof the Kullback-Leibler divergence using the graphical model introduced in Figure 2B of the mainmanuscript:
[Equation]
where we have introduced A manip to accommodate all possibilities that a manip can realize.
This expression can be grouped differently in order to simplify:
[Equation]
which eventually gives rise to two separate Kullback-Leibler divergences:
[Equation]
Since p EC ( a manip | a id = 1 , x, x ′ ) and q θ ( a manip | a id = 1 , x, x ′ ) are fixed and 1 for the same, formallyintroduced, unknown a manip , we obtain:
[Equation]
D.10 Instance discrimination as an upper bound to the identity-related inverse model loss
In the following, we provide proof that instance discrimination is an upper bound to L id as introducedin Section 3.2 of the main manuscript.
Overall, the idea to achieve this, is to recognize that trainingthe identity-related inverse model to always identify the correct positive example (i.e. related througha id = 0 ) will at the same time allow this model to predict the case a id = 1 .
We begin by the definition of the loss for learning the identity-related inverse model:
[Equation]
Since there are only two possibilities for a id ∈ { 0 , 1 } it follows that:
[Equation]
const
[Equation] [Equation]
From the definition of q θ ( a id = 0 | x, x ′ ) in equation (3) in the main manuscript, we have that 949 (cid:80) x n ∈ C
q
θ
( a id = 0 | x, x n ) = 1 − q θ ( a id =
| x, x ′ ) , which further implies that we can take anyx ′′ ∈ C and obtain the inequality q θ ( a id =
| x, x ′′ ) ≤ 1 − q θ ( a id =
| x, x ′ ) .
Inserting this inequality into equation (28), and due to the monotony of the logarithm, we obtain:
[Equation]
Finally, if on the other hand p EC represents a perfect copy of a id , then we can define x ′′ to alwaysrepresent the example that forms a positive pair with x ′ (i.e. through a id = 0 )and obtain equation (5)in the main manuscript, which is the instance discrimination task of (Chen et al., 2020a):
[Equation] [Equation]
where const = 0 is a result of 0 entropy in p EC ( a id | x, x ′ ) .
E Additional results
E.1 Visual depiction of the distribution of accuracies
In addition to the results presented in Table 1 to 3 in the main manuscript, Figure 5 depicts thedistribution of performance values for SimCLR and S-TEC visually
E.2 Ablation
We also performed an ablation study for ResNet-18s trained on CIFAR-100 to more specificallyassess which components helped to improve the representations, as measured by linear classificationaccuracy, the most.
Subject to this study were two key mechanisms that were previously introduced:categorical and egocentric action representation.
[Figure] [Figure]
ResNet
[Figure]
SimCLR
S
[Figure]
ResNet
[Figure]
18 CIFAR
[Figure] [Figure]
S
[Figure] [Figure]
[Figure] [Figure]
ResNet
[Figure]
18 STL
[Figure] [Figure]
S
[Figure]
TEC 93.
[Figure] [Figure]
[Figure] [Figure]
Categorical action representation.
The modelling of actions in a categorical manner induces asoftmax cross entropy loss for the optimization of the manipulation-related inverse model, which werefer to as “Classification”.
Alternatively, actions and the predictions thereof can be represented intheir continuous form, which gives rise to a standard L2 regression loss, which we denote as “L
Regression”, pointing out that this strategy was employed by Lee et al.
Egocentric action representation.
On the other hand, our manipulation-related inverse model wastrained to predict the actions that would be required to move from one view into the other based on itsown perspective.
This egocentric viewpoint is in contrast to the allocentric approach chosen in (Leeet al., 2021), where differences in the view are predicted based on the original image: i.e. for randomcropping, differences in the cropping scale and differences of the crop’s borders from the top and theleft of the original image are predicted.
We tested the possible combinations of the choices for action representation (optimizing separatelyλ manip ∈ { 0 . 1 , 0 . 2 , 0 .
and report the average performance of 5 independent runs each in
Figure 3A of the main text.
These results confirm that categorical and egocentric action representationsperform best as evaluated on linear classification accuracy.
E.3 Optimization progress
For insight in the optimization dynamics, we provide learning curves for runs of SimCLR and S-TECon the datasets of CIFAR-10 (see Figure 6 and 9), CIFAR-100 (see Figure 7 and 10) as well as losscurves in the case of STL-10 (see Figure 8 and 11).
All of the provided curves were obtained using5 independent runs for each scenario that was considered.
We report the averages of these as boldcurves, which were additionally processes using a moving average filter.
Unprocessed individualmetrics are shown as thin transparent lines.
We report in each scenarios the following metrics:
Loss of the manipulation-related objective (only for S-TEC),
accuracy of the manipulation-related inverse model (only for S-TEC), which is defined as
the average accuracy that this inverse models picks the correct action clusters (measured onthe training set),3.
loss of the identity-related inverse model, and
accuracy of the identity-related inverse model, which is reported as the fraction of positiveviews x and x ′′ being correctly identified, see also D.10.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
L o
[Figure]
m a n i p
[Figure]
1e 0.
A cc u r a c y m a n i p     5 Iteration 1e 5.
L o ss i d
[Figure]
1e 0.
A cc u r a c y i d
SimCLR S-
[Figure] [Figure]