Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space
Anonymous Author(s)
AffiliationAddress email
Abstract
Humans are remarkably flexible in understanding viewpoint changes due to visualcortex supporting the perception of 3D structure.
In contrast, most of the computervision models that learn visual representation from a pool of 2D images often failto generalize over novel camera viewpoints.
Recently, the vision architectureshave shifted towards convolution-free architectures, visual Transformers, whichoperate on tokens derived from image patches.
However, these Transformersdo not perform explicit operations to learn viewpoint-agnostic representation forvisual understanding, as in convolutions.
To this end, we propose a 3D Token
Representation Layer (3DTRL) that estimates the 3D positional information of thevisual tokens and leverages it for learning viewpoint-agnostic representations.
Thekey elements of 3DTRL include a pseudo-depth estimator and a learned cameramatrix to impose geometric transformations on the tokens.
These enable 3DTRL torecover the 3D positional information of the tokens from 2D patches.
In practice,3DTRL is easily plugged-in into a Transformer.
Our experiments demonstrate theeffectiveness of 3DTRL in many vision tasks including image classification, multiview video alignment, and action recognition.
The models with 3DTRL outperformtheir backbone Transformers in all the tasks with minimal added computation.
Introduction
Over the past few years, computer vision models have developed rapidly from CNNs [5, 25, 54] tonow Transformers [15, 23, 55].
With these models, we can now accurately classify objects in animage, align image frames among video pairs, classify actions in videos and more.
Despite theirsuccess, many of the models neglect that the world is 3D and do not extend beyond the XY imageplane [19].
While humans can readily estimate the 3D structure of a scene from 2D pixels of animage, most of the existing vision models with 2D images do not take the 3D structure of the worldinto consideration.
This is one of the reason why humans are able to recognize objects in images andactions in videos regardless of their viewpoint but the vision models often fail to generalize overnovel viewpoints [10, 19, 40].
Consequently, in this paper, we develop an approach to learn viewpoint-agnostic representations for arobust understanding of the visual data.
Naive solutions to obtain viewpoint-agnostic representationwould be either supervising the model with densely annotated 3D data, or learning representationfrom a large scale 2D datasets with samples encompassing different viewpoints.
Given the fact thatsuch high quality data are expensive to acquire and hard to scale, an approach with a higher sampleefficiency without 3D supervision is desired.
To this end, we propose a 3D Token Representation Layer (3DTRL), incorporating 3D cameratransformations into the recent successful visual Transformers [6, 15, 34, 55].
3DTRL first recoverscamera-centered 3D coordinates of each token by depth estimation.
Then 3DTRL estimates a camera matrix to transform these camera-centered coordinates to a 3D world space.
In this world space,3D locations of the tokens are absolute and view-invariant, which contain important information forlearning viewpoint-agnostic representations.
Therefore, 3DTRL incorporates such 3D positionalinformation in the Transformer backbone in the form of 3D positional embeddings, and generates output tokens with 3D information.
Unlike visual Transformers only relying on 2D positional embedding,models with 3DTRL are more compliant with learning viewpoint-agnostic representations.
We conduct extensive experiments on various vision tasks to confirm the effectiveness of 3DTRL.
Our 3DTRL outperforms the Transformer backbones on 3 image classification, 5 multi-view videoalignment, and 2 multi-view action recognition datasets in their respective tasks.
Moreover, 3DTRL isa light-weighted, plug-and-play module that achieves the above improvements with minimal (2%computation and 4% parameters) overhead.
In summary, we present a learnable, differentiable layer 3DTRL that efficiently and effectively learnsviewpoint-agnostic representations.
Background: Pinhole Camera Model
3DTRL is based on the standard pinhole camera model widely used in computer vision.
Thus, wefirst briefly review the pinhole camera model.
In homogeneous coordinate system, given a point withworld coordinate p world , a camera projects a pixel at p on an image by:
[Equation]
1) where K is the intrinsic matrix and [ R | t ] is the extrinsic matrix.
K is further represented by
[Equation]
where c is the focal length and ( u 0 , v 0 ) are offset.
In this work, we explore visual understanding inmulti-view setting, thus aiming at learning viewpoint-agnostic representations.
In this setting, a scenemay be captured with different cameras positioned at non-identical locations and viewing angles.
Here, the world coordinate p world is the same across all the cameras while pixel projection p is differentacross cameras.
We focus on how to estimate the world coordinates p world from their correspondingimage pixels at p .
Estimating p world from p involves two transformations that correspond to theinverse of K and [ R | t ] which might not be known beforehand.
Thus, we estimate them from imagepatches instead, which is a key procedure in 3DTRL.
3D Token Representation Layer (3DTRL)
In this section, we detail how 3DTRL estimates 3D positional information in a Transformer and isintegrated.
We will introduce 3DTRL for image analysis, then we adapt it to video models.
3.1 Overview
3DTRL is a simple yet effective plug-in module that can be inserted in between the layers of visual
Transformers (Figure 1 left).
Given a set of input tokens S , 3DTRL returns a set of tokens with
3D information.
The number of tokens and their dimensionality are kept unchanged.
Within themodule (Figure 1 right), 3DTRL first performs 3D estimation using two estimators: (1) pseudo-depthestimator and (2) camera parameter estimator.
Then the tokens are associated with their recoveredworld 3D coordinates, which are transformed from estimated depth and camera matrix.
Finally,3DTRL generate 3D positional embeddings from these world coordinates and combine them withinput S to generate the output of 3DTRL.
In the aspect that we insert 3DTRL in between the Transformer model, 3DTRL implicitly leveragesthe Transformer layers before it to be a part of the 3D estimators, and layers after that to be the actual3D feature encoder (Figure 1 left).
This avoids adding a large number of parameters while resultingin reasonable estimations.
We empirically find that placing 3DTRL at a shallow-medium layer ofthe network yields better results (See Section 4.3), which is a trade-off between model capacity forestimation and 3D feature encoding.
3.2 3D Estimation of the Input Tokens
3DTRL first estimates both the camera-centered 3D coordinates of each token using depth estimationand a camera matrix shared by all the tokens of an image.
Then, the camera-centered 3D coordinatesare transformed to the “world” coordinates by the camera matrix.
Pseudo-depth Estimation.
Given input tokens S
= { s 1 , . . .
, s N } ∈ R N × m , 3DTRL first performspseudo-depth estimation of each token s n .
The pseudo-depth estimator is a function f : R m → Rthat outputs the depth d n = f ( s n ) of each token individually.
We implement f using a 2-layer MLP.
We call this pseudo-depth estimation since it is similar to depth estimation from monocular imagesbut operates at a very coarse scale, given that each token corresponds to an image patch rather than asingle pixel in Transformer.
After pseudo-depth estimation, 3DTRL transforms the pseudo-depth map to camera-centered 3Dcoordinates.
Recall that in ViT [15], an image X is decomposed into N patches { X 1 , . . .
, X N } ∈ 92 R N × P × P × 3 , where P × P is the size of each image patch and the tokens S are obtained from alinear projection of these image patches.
Thus, each token is initially associated with a 2D locationon the image plane, denoted as ( u, v ) .
By depth estimation, 3DTRL associates each token with onemore value d .
Based on the pinhole camera model explained in Section 2, 3DTRL transforms u, v, dto a camera-centered 3D coordinate ( x, y, z ) by:
[Equation]
Since we purely perform the aforementioned estimation from monocular images and the the cameraintrinsic matrix is unknown, we simply set c to a constant hyperparameter.
To define coordinatesystem of 2D image plane ( u, v ) , we set the center of the original image is the origin (0 , 0) forconvenience, so that the image plane and camera coordinate system shares the same origin.
We usethe center of the image patch to represent its associate ( u, v ) coordinates.
We believe that this depth-based 3D coordinate estimation best leverages the known 2D geometry,which is beneficial for later representation learning.
We later confirm this in our ablation study (in
Section 4.3), in which we compare it against a variant directly estimating ( x, y, z ) .
Camera Parameter Estimation.
The camera parameters are required to transform the estimatedcamera-centered 3D coordinates p cam to the world coordinate system.
These camera parameters areestimated jointly from all input tokens S .
This involves estimation of two matrices, a 3 × 3 rotationmatrix R and a 3 × 1 and translation matrix t through an estimator g .
We implement g using a MLP.
Specifically, we use a shared MLP stem to aggregate all the tokens into an intermediate representation.
Then, we use two separated fully connected heads to estimate the parameters in R and t respectively.
We note the camera parameter estimator as a whole:
[ R | t ] = g ( S ) .
To ensure R is mathematicallyvalid, we first estimate the three values corresponding to yaw, pitch and roll angles of the camerapose, and then convert them into a 3 × 3 rotation matrix.
Transform to World Coordinates.
Now, with the estimated camera parameters, 3DTRL transformsestimated camera-centered coordinates p cam into the world space, a 3D space where 3D coordinatesof the tokens are absolute and viewpoint-invariant.
Following the pinhole camera model, we recoverp by
[Equation]
[Equation]
3.3 Incorporating 3D Positional Information in Transformers
The last step of 3DTRL is to leverage the estimated 3D positional information in Transformerbackbone.
For this, we choose to adopt a typical technique of incorporating positional embeddingthat is already used in Transformers [15, 56].
In contrast to 2D positional embedding in ViTs [15],3DTRL learns a 3D embedding function h : R [-
-]3 → R m to transform estimated world coordinatesp world to positional embeddings p 3D .
This 3D embedding function h is implemented using a two-layer
MLP.
Then, the obtained 3D positional embedding is incorporated in the Transformer backbone bycombining it with the token representations.
The outcome is the final token representations { s 3D } :
[Equation]
where p 3D =
h ( p world ) .
After 3D embedding, the resultant token representations are associated with a 3D space, thus enablingthe remaining Transformer layers to encode viewpoint-agnostic token representations.
We ablateother ways of incorporating the 3D positional information of the tokens in Section 4.3.
3.4 3DTRL in Video Models
Notably, 3DTRL can be also easily generalized to video models.
For video models, the input to3DTRL is a set of spatial-temporal tokens { S 1 , . . .
, S T } corresponding to a video clip containing
T frames, where S t = { s t 1 , . . .
, s tN } are N tokens from t -th frame.
We simply extend our moduleto operate on an additional time dimension, where depth estimation and 3D positional embeddingare done for each spatial-temporal tokens s tn individually:
d tn = f
( s tn ) , p 3D tn
= h ( p world tn ) .
Cameraparameters are estimated per input frame ( S t ) in a dissociated manner, namely [ R | t ] t = g ( S t ) ,resulting in a total of T camera matrices per video.
We investigate another strategy of cameraestimation in the supplementary, where only one camera matrix is learned for all frames.
However,our studies have substantiated the effectiveness of learning dissociated camera matrices per frame.
Experiments
We conduct extensive experiments to demonstrate the efficacy of viewpoint-agnostic representationslearned by 3DTRL in multiple vision tasks: (i) image classification, (ii) multi-view video alignment,and (iii) video action classification.
4.1 Image Classification
In order to validate the power of 3DTRL, we first evaluate ViT [15] with 3DTRL for image classification task using CIFAR-10 [32], CIFAR-100 [32] and ImageNet-1K [14] datasets.
Training We use the training recipe of DeiT [55] for training our baseline Vision Transformermodel on CIFAR and ImageNet datasets from scratch .
We performed ablations to find an optimallocation in ViTs where 3DTRL should be plugged-in (Section 4.3).
Thus, in all our experimentswe place 3DTRL after 4 Transformer layers, unless otherwise stated.
The configuration of our
DeiT-T, DeiT-S, and DeiT-B is identical to that mentioned in [55].
All our transformer models aretrained for 50 and 300 epochs for CIFAR and ImageNet respectively.
Further training details andhyper-parameters can be found in the supplementary material.
[Table]
6 (+3.3)
DeiT-S 22.1M 4.24G 77.
71.1 +3DTRL 23.0M 4.33G 80.7 (+3.5) 61.5 (+6.9) 79.7 (+0.3) 72.7 (+1.6)
DeiT-B 86.6M 16.7G 76.
70.6 +3DTRL 90.1M 17.2G 82.8 (+6.2) 61.8 (+9.9) 81.2 (+0.2) 74.7 (+4.1)
Results From Table 1, 3DTRL with all DeiT variants shows consistent performance improvementover the baseline on the CIFAR datasets, with only ∼ 2% computation overhead and ∼ 4% moreparameters.
Despite fewer training samples, 3DTRL significantly outperforms the baseline in CIFAR,showing the strong generalizability of 3DTRL when limited training data are available.
We arguethat multi-view data is not available in abundance, especially in domains with limited data (like inmedical domain), thus 3DTRL with its ability to learn viewpoint-agnostic representations will beinstrumental in such domains.
We also find the performance improvement on ImageNet is less thanthat on CIFAR.
This is because ImageNet has limited viewpoints in both training and validationsplits, thus reducing the significance of performing geometric aware transformations for learningview agnostic representations.
In contrast, CIFAR samples present more diverse camera viewpoints,so it is a more suitable dataset for testing the quality of learned viewpoint-agnostic representations.
Robustness on Viewpoint-perturbed Data In order to emphasize the need of learning viewpointagnostic representations, we further construct a viewpoint -perturbed ImageNet-1K validation set byapplying random perspective transformations to images.
Example images are shown in Figure 2.
Wenote that perspective transformation on these static images is not equivalent to real viewpoint changesbecause the zero-paddings and interpolations do not happen in real-world cameras.
Nonetheless, webelieve it is a meaningful experiment as real image datasets with multi-view settings are lacking inthe vision community.
Table 1 shows that the performance improvement with 3DTRL on perturbeddata is greater than that on original data.
This demonstrates the robustness of 3DTRL for perspectivetransformed data, thanks to the learned viewpoint-agnostic representations.
4.2 Multi-view Video Alignment
Video alignment [17, 22, 39] is a task to learn a frame-to-frame mapping between video pairs withclose semantic embeddings.
In particular, we consider a multi-view setting that aligning videoscaptured from the same event but different viewpoints.
Thus, video pairs from the same event aretemporally synchronized. has synchronized videos of at leastviewpoints.
All datasets except Pouring have one ego-centric view, highlighted in red boxes.
More details are available in supplementary material.
Datasets We use 5 multi-view datasets from a wide range of environments: Minecraft (MC) –video game, Pick , Can , and Lift from robot simulators (PyBullet [8] and Robomimic [37]), and
Pouring from real-world human actions [46].
Example video frames are provided in Figure 3.
Eachdataset contains synchronized videos from multiple cameras (viewpoints).
There is one ego-centriccamera per dataset except Pouring, which are continuously moving with the subject.
These ego-centricvideos make the alignment more challenging.
Detail dataset statistics are available in supplementary.
Training We follow common video alignment methods [46] to train an encoder that outputs framewise embeddings.
We still use DeiT [55] as a baseline model (DeiT+TCN) and apply 3DTRL to it(+3DTRL), similar to image classification.
During training, we use the time-contrastive loss [46] toencourage temporally closed embeddings to be similar while temporally far-away embeddings to beapart.
Then, we obtain alignments via nearest-neighbor such that an embedding u i from video 1 isbeing paired to its nearest neighbor v j in video 2 in the embedding space.
And similarly u j fromvideo 2 is paired with its nearest neighbor v k in video 1.
We use ImageNet-1K pre-trained weightsfor experiments on Pouring, but we train from scratch for other datasets considering that simulationenvironments are out of real-world distribution. [Figure]
Evaluation We evaluate the alignment by three metrics: Alignment Error [46], Cycle Error, and
Kendall’s Tau [31].
Let the alignment pairs between two videos be ( u i , v j ) and ( v j , u k ) .
In brief,
Alignment Error measures the temporal mismatching | i − j | of ( u i , v j ) .
Cycle Error is based oncycle-consistency [17, 58], where two pairs ( u i , v j ) and ( v j , u k ) are called consistent when i = k .
Thus, Cycle Error measures the inconsistency based on distance metric | i − k | .
Kendall’s Tau ( τ )measures ordering in pairs.
Given a pair of embeddings from video 1 ( u i , u j ) and their correspondingnearest neighbors from video 2 ( v p , v q ) , the indices tuple ( i, j, p, q ) is concordant when i < j andp < q
or i > j and p > q .
Otherwise the tuple is discordant .
Kendall’s Tau computes the ratio ofconcordant pairs and discordant pairs over all pairs of frames.
Let N be the number of frames in avideo, then the formal notations of the three metrics are:
[Equation]
We establish two evaluation protocols: (a) Seen and (b) Unseen .
In Seen , we train and test modelson videos from all cameras.
However, in Unseen , we hold out several cameras for test, which is arepresentative scenario for validating the effectiveness of 3DTRL.
Detail of experimental settings areprovided in supplementary.
Results Figure 4 illustrates the evaluation results of 2 viewpoint settings over 5 datasets, comparedto the DeiT baseline.
3DTRL outperforms the baseline consistently across all datasets.
In particular,3DTRL improves the baseline by a large margin in Pouring and MC, corroborating that 3DTRL adaptsto diverse unseen viewpoints.
The improvements of 3DTRL on Pick and MC also suggests the stronggeneralizability when learning from smaller datasets.
With enough data (Lift and Can), 3DTRL stilloutperforms but the gap is relatively small.
When evaluating in Unseen setting, both methods haveperformance drop.
However, 3DTRL still outperforms in Pick, MC, and Can, which suggests therepresentations learned by 3DTRL are able to generalize over novel viewpoints.
MC has the largestviewpoint diversity so it is hard to obtain reasonable align results in the unseen setting for both themodels.
In Table 2, we further compare 3DTRL with previous methods on certain datasets.
Note that
Disentanglement [48] uses extra losses whereas others use time-contrastive loss only.
We find that3DTRL with only single-frame input is able to surpass the strong baselines set by models using extralosses [48] or multiple input frames [16].
We also vary the number of training videos in Pouringdataset, and results from Figure 5 show that 3DTRL benefits from more data.
Meanwhile, 3DTRL canoutperform the baseline while only using 22% of data the baseline used.
4.3 Ablation Studies
We conduct our ablation studies on image models mostly using CIFAR for image classification and
Pick for multi-view video alignment.
MLP vs. 3DTRL.
3DTRL is implemented by several MLPs with required geometric transformsin between.
In this experiment, we replace 3DTRL with the similar number of fully-connected [Table] [Figure] [Table]
layers with residual connection, to have comparable parameters and computation as 3DTRL.
Resultsare provided in Table 3.
We find that MLP implementation is only comparable with the baselineperformance despite the increase in parameters and computation.
Thus, we confirm that the geometrictransformations imposed on the token representations is the key to make 3DTRL effective.
Token Coordinates Estimation.
In this ablation, we show how estimating only depth compared toestimating a set of 3 coordinates xyz differs in 3DTRL.
We find at Line 3 and 4 Table 3 that depthestimation is better because it uses precise 2D image coordinates when recovering tokens in 3Dspace, whereas estimating xyz regresses for the 3D coordinates without any geometric constraints.
Also, we find that estimating xyz hampers the performance in image classification task more thanvideo alignment.
This is because estimating 3D coordinates is harder when the training samples arein unconstrained scenarios.
In contrast, video pairs in an alignment dataset share the same scenecaptured from different view angles, which facilitates the recovery of token in 3D space.
How to incorporate 3D positional information in Transformers?
By default, we use Equation 5to incorporate the 3D positional information in Transformer backbone through a learned positionalembedding p 3 D .
In this experiment, we directly infuse the estimated 3D world coordinates p worldwithin the token representation by concatenating them across the channel axis.
The ( m + 3) -dfeature is then projected back to m -d by a MLP.
We keep parameters and computation comparable tothe default 3DTRL.
We test this 3DTRL variant (Embedding → Concat.) and results are presentedin Table 3.
We find that the concatenation variant outperforms the default variant of 3DTRL on
CIFAR-10, but comparable and worse results in CIFAR-100 and Pick.
This observation substantiatesthe instability of using raw 3D coordinates.
In comparison, the use of 3D positional embedding isgeneralizable to more challenging and diverse scenarios.
Where should we have 3DTRL?
We vary the location of 3DTRL to empirically study the optimallocation of 3DTRL in a 12-layer DeiT-T backbone.
In Figure 6 on the right, we find that inserting3DTRL at the earlier layers (after 1/3 of the network) yields the best performance consistently onboth datasets. 4.4 Qualitative Visualization of 3DTRL’s Pseudo-depth Map
Although previous experiments have shown that 3DTRL is beneficial for different vision tasks, in thissection, we visualize some qualitative results for a deeper understanding of 3DTRL’s effectiveness.
The qualitative results include visualization of estimated pseudo-depth maps which are from the keyintermediate step of 3DTRL.
For this analysis, we use an ImageNet-1K trained DeiT-T+3DTRL and its validation samples for visualization.
In Figure 7, we observe a fine separation of the object-ofinterest and the background in most of the pseudo-depth maps.
Thus, these predicted pseudo-depthmaps are sufficient to recover the tokens corresponding to object-of-interest in 3D space.
4.5 3DTRL for Video Representation Learning
[Table]
3DTRL[-w/o K 69.5 (+0.2) 39.
 (+1.7) 77.5 (+0.3) 58.9 (+1.2) 88.8 (+1.1) + 3DTRL w/-]
[Table] ) 88.6 (+0.9)
In this section, we illustrate how 3DTRL can be adapted for video models.
Since we aim at learningviewpoint-agnostic representations, as a natural choice we validate the effectiveness of 3DTRL onvideo datasets with multi-camera setups and cross-view evaluation.
Consequently, we conduct ourexperiments on two multi-view action recognition datasets: Toyota Smarthome [11] (Smarthome)and NTU-RGB+D [47] (NTU) for the task of action classification.
For evaluation on Smarthome, wefollow Cross-View 2 (CV2) and Cross-Subject (CS) protocols proposed in [11], whereas on NTU, wefollow Cross-View (CV) protocol proposed in [47].
In cross-view protocols, the model is trained ona set of cameras and tested on a different set of cameras.
Similarly for cross-subject protocol, themodel is trained and tested on different set of subjects.
Note that for Smarthome dataset, we report theaction classification accuracy on two metrics (i) classification accuracy ( Acc ) and (ii) mean per-classaccuracy ( mPA ).
More details on NTU and Smarthome datasets is provided in the supplementarymaterial.
Network architecture & Training/Testing TimeSformer [3] is a straightforward extension of
ViT [15] for videos which operates on spatio-temporal tokens from videos, so that 3DTRL can beeasily deployed to TimeSformer as well.
We use clips of size 8 × 224 × 224 × 3 , with framessampled at a rate of 1/32.
We use a ViT-B encoder with patch size 16 × 16 .
Similar to our previousexperimental settings, we place 3DTRL after 4 Transformer blocks in TimeSformer.
We adoptthe training strategies mentioned in [3] and thus, train all the video models for 15 epochs.
Duringinference, we sample one and 10 temporal clips from the entire video on NTU and Smarthomedatasets respectively.
We use 3 spatial crops (top-left, center, bottom-right) from each temporal clipand obtain the final prediction by averaging the scores for all the crops.
Results In Table 4, we present the action classification results on Smarthome and NTU datasetswith 3DTRL plugged in TimeSformer.
Firstly, we present the results for the models without anypre-training and then for the models with Kinetics-400 [30] (K400) pre-training.3DTRL can easily take advantage of pre-trained weights because it does not change the relyingbackbone Transformer – just being added in between blocks.
In Table 4, we present results for twofine-tuning scenarios: (a) 3DTRL w/o K400 and (b)
3DTRL w/ K400.
For the first scenario (a),
TimeSformer is initialized with K400 pre-training weights and leave the parameters in 3DTRL randomly initialized.
Then in the fine-tuning stage, all the model parameters including that of 3DTRL istrained.
In the second scenario (b), all parameters in TimeSformer and 3DTRL are pre-trained on
K400 from scratch and fine-tuned on the respective datasets.
We find that all the variants of 3DTRL outperforms the baseline TimeSformer results.
Our experimentsshow that although there is an improvement with 3DTRL compared to the baseline for differentfine-tuning strategy, it is more significant when 3DTRL is pre-trained with K400.
However, whenlarge-scale training samples are available like in NTU, 3DTRL does not require K400 pre-training.
To sum up, 3DTRL can be seen as a crucial ingredient for learning viewpoint-agnostic videorepresentations.
Related Work
There has been a remarkable progress in visual understanding with the shift from the use of
CNNs [5, 25, 33, 54] to visual Transformers [15].
Transformers have shown substantial improvementsover CNNs in image [6, 15, 23, 29, 34, 55, 60] analysis and video [1, 3, 35, 42, 45] understandingtasks due to its flexibility in learning global relations among visual tokens.
Studies also combine
CNNs with Transformer architectures to leverage the pros in both the structures [9, 21, 36, 49].
In addition, Transformer has been shown to be effective in learning 3D representation [63].
However, these advancements in architecture-types have not addressed the issue of learning viewpointagnostic representation.
Viewpoint-agnostic representation learning is drawing increasing attention in the vision community due to its wide range of downstream applications like 3D objectdetection [44],video alignment [7, 17, 18], action recognition [50, 51], pose estimation [24, 53], robotlearning [26, 28, 46, 48, 52], and other tasks.
There is a broad line of work towards directly utilizing 3D information like depth [24], pose [12, 13],and point clouds [41, 43], or in some cases deriving 3D structure from paired 2D inputs [57].
However,methods rely on the availability of multi-modal data which is hard to acquire are not scalable.
Consequently, other studies have focused on learning 3D perception of the input visual signal inorder to generalize the learned representation to novel viewpoints.
This is done by imposing explicitgeometric transform operations in CNNs [4, 27, 40, 44, 59], without the requirement of any 3Dsupervision.
In contrast to these existing works, our Transformer-based 3DTRL imposes geometrictransformations on visual tokens to recover their representation in a 3D space.
To the best of ourknowledge, 3DTRL is the first of its kind to learn a 3D positional embedding associated with thevisual tokens for viewpoint-agnostic representation learning in different image and video relatedtasks.
Conclusion
In this work, we have presented 3DTRL, a plug-and-play module for visual Transformer that leverages3D geometric information to learn viewpoint-agnostic representations.
Within 3DTRL, by pseudodepth estimation and learned camera parameters, it manages to recover positional information oftokens in a 3D space.
Through our extensive experiment, we confirm 3DTRL is generally effective ina variety of visual understanding tasks including image classification, multi-view video alignment,and cross-view action recognition, by adding minimum parameters and computation overhead.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyper-parameters,
how they
[Table]
were chosen)?
[Yes]
[Table]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes](b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes](d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiableinformation or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
[Table]