Orthogonal Transformer: An Efﬁcient Vision Transformer Backbone with Token Orthogonalization
Abstract
We present a general vision transformer backbone, called as Orthogonal Transformer, in pursuit of both efﬁciency and effectiveness.
A major challenge for vision transformer is that self-attention, as the key element in capturing long-range dependency, is very computationally expensive for dense prediction tasks (e.g., object detection).
Coarse global self-attention and local self-attention are then designed to reduce the cost, but they suffer from either neglecting local correlations or hurting global modeling.
We present an orthogonal self-attention mechanism to alleviate these issues.
Speciﬁcally, self-attention is computed in the orthogonal space that is reversible to the spatial domain but has much lower resolution.
The capabilities of learning global dependency and exploring local correlations are maintained because every orthogonal token in self-attention can attend to the entire visual tokens.
Remarkably, orthogonality is realized by constructing an endogenously orthogonal matrix that is friendly to neural networks and can be optimized as arbitrary orthogonal matrices.
We also introduce Positional MLP to incorporate position information for arbitrary input resolutions as well as enhance the capacity of MLP.
Finally, we develop a hierarchical architecture for Orthogonal Transformer.
Extensive experiments demonstrate its strong performance on a broad range of vision tasks, including image classiﬁcation, object detection, instance segmentation and semantic segmentation.
Introduction
Recently, transformer [46] has made a tremendous success in the ﬁeld of natural language processing (NLP).
Beneﬁting from the self-attention (SA) mechanism, it has a strong capacity in building long-range dependencies in sequential data.
Since long-range modeling is also essential for a wide range of vision tasks, transformer has been adapted into computer vision by converting an image into a sequence of patches [13] (called as tokens) and achieved competitive performance compared to
CNN [43, 44].
Moreover, self-attention has a quadratic computational complexity to the number of tokens, resulting in intolerably expensive computational cost for dense prediction tasks, e.g., object detection and segmentation.
Many efforts have been made to design efﬁcient self-attention mechanisms for vision transformer.
PVT [50] employs a pyramid architecture with downsampled key and value tokens to reduce the computation of global attention.
Swin [35] performs self-attention in a local region with shifted window to allow cross-region connection.
GG-Transformer [63] and CrossFormer [52] present dilated self-attention to learn large-scale features efﬁciently.
As shown in Fig.
1 (a-c), the aforementioned attention mechanisms can reduce the number of tokens conveniently.
However, the penalty is losing ﬁne-level details for coarse global self-attention (including downsampled and dialted ones) or hurting long-range modeling for local self-attention [41, 52, 42, 63].
This paper presents an orthogonal self-attention (OSA) mechanism to capture global dependency without losing ﬁne-level details.
As shown in Fig.
1 (d), we orthogonalize tokens within local regions, permute them into token groups and perform group-wise self-attention in the orthogonal space.
We apply Orthogonal Transformation (OT) for the tokens for the following reasons.
Firstly, tokens have a lower resolution in the orthogonal space than in the original visual space, thus reducing the computational cost (details see Sec. 3.2.2).
Secondly, the orthogonal space is reversible to the visual space without information loss (thus outperforming downsampled self-attention), and can be easily reversed back since the inverse matrix A (cid:48) is just the transpose of the orthogonal matrix A , i.e.,
A (cid:48)
=
A T .
Thirdly, OT can separate tokens into linearly independent groups.
Thereby computing self-attention in such groups helps to explore different properties in representation.
Lastly, OT builds connections among adjacent tokens explicitly and is more capable of modeling local correlations than dilated partition.
Despite of these beneﬁts, it often needs complex optimization algorithms [31, 32] or imposing extra penalties on the loss functions [58, 48] to enforce orthogonality.
In this work, we construct an endogenously orthogonal matrix that is friendly to neural networks with gradient optimizers.
We build an orthogonal matrix A ∈ R n × n by the product of n Householder transformations { H i =
I − 2 u i u T i | i = 0 , .., n − 1 } , where u i ∈ R n is a learnable unit vector.
In this way, A is naturally orthogonal and can be optimized as arbitrary orthogonal matrices.
In particular, the dilated self-attention can be viewed as a degeneration of OSA when A = I .
The strong capacity in exploring diverse transformations makes OSA more essential for the model to achieve competitive performance.
Based on the orthogonal self-attention mechanism, we propose a new Orthogonal Transformer in this work.
As shown in Fig.
2, it follows the hierarchical design [50, 35] and serves as a generalpurpose backbone for computer vision.
We stack window self-attention (WSA) and orthogonal self-attention (OSA) alternatively to capture both global and local dependencies.
Note that although our OSA is superior in preserving local details, combing it with WSA can further enhance Orthogonal
Transformer.
To improve the ﬂexibility for arbitrary resolutions, we adopt convolutional position embedding to incorporate position information.
Speciﬁcally, Positional MLP (PMLP) is introduced by equipping MLP with a depth-wise convolution (DConv) after the non-linear activation.
Such a simple design not only enables the network to generate position information ﬂexibly, but also enhance the capacity of MLP.
Besides, it also allows for downsampling within the transformer block with strided DConv.
We empirically show that downsampling within the block can achieve better performance than outside.
Extensive experiments demonstrate the strong performance of Orthogonal Transformer on a wide range of vision tasks.
For example, without extra training data or supervision (such as token labeling [26] and distillation [43]), our large model Ortho-L achieves 85.4 top-1 accuracy on ImageNet-1K image classiﬁcation, surpassing the previous state-of-the-art Swin Transformer [35] by +1.2 with similar model size and FLOPs.
Our base model Ortho-B achieves 53.0 box AP and 45.9 mask AP on the COCO detection task, 49.8 mIOU on the ADE20K semantic segmentation task, surpassing the Swin Transformer counterpart by +1.2 , +1.2 and +2.2 , respectively.
Under a smaller setting of
FLOPs, our Ortho-S even obtain larger performance gains, i.e., +2.1 on ImageNet classiﬁcation with 224 × 224 resolution, +1.8 box AP, +1.6 mask AP on COCO detection and +4.0 mIoU on ADE20K segmentation.
Related Work
Vision Transformers.
Transformer network is ﬁrstly proposed in the natural language processing (NLP) ﬁeld and achieves superior performance in many NLP tasks [46, 27].
Recently, there emerges a trend towards incorporating the transformer into computer vision tasks, which are previously dominated by CNNs [19, 40].
ViT [13] is the pioneering work of vision transformers.
After that, there are a large number of works focusing on designing a general vision Transformer backbone [9, 17, 35, 5, 50, 14, 66, 63, 61, 12, 52, 60, 26] for different vision tasks.
To adapt transformer to the image inputs, hierarchical architectures [35, 50], efﬁcient self-attentions [35, 63, 25] and diverse positional encodings [46, 38, 10, 12] are proposed.
Vision transformers are also applied to different downstream vision tasks, such as object detection [4, 69], semantic segmentation [39, 67, 53], image restoration [6, 47], and video processing [33, 1].
We propose a new vision transformer backbone to tackle various vision tasks efﬁciently and effectively.
Efﬁcient Self-Attentions.
Many efﬁcient self-attention mechanisms [30, 29, 8, 2, 49, 37] have been proposed in the ﬁeld of NLP to efﬁciently handle long sequences.
Since the image resolution is usually high in many vision tasks, the global self-attention can be applied only for a few times and in the low resolution feature space.
Early approaches, such as ViT [13] and DeiT [43], employ large image patches to restrict the number of tokens.
PVT [50] and Twins [9] use downsampled tokens to compute self-attention.
Many works adopt local self-attention to limit the token number by attending only subregions of the input, such as the window attention [35], axis attention [21], and criss-cross attention [25].
Inspired by dilated convolution [62], GG Transformer [63] and CrossFormer [52] introduce dilated self-attention to capture long-distance dependency.
Unlike existing methods, our orthogonal self-attention can attend to the entire tokens at a low computational cost.
Global dependency can be captured without losing local details.
Positional Encodings.
Recently, researchers propose to add different kinds of position information to the token feature or self-attention process.
Absolute positional encoding (APE) [46] is the ﬁrst work to add position encodings to the Transformer inputs.
Relative positional encoding (RPE) [35, 38] considers the relative distance between two tokens instead of the absolute position.
Conditional positional encoding (CPE) [10] takes feature as inputs and generates position information via a convolution layer.
We follow CPE but compensate position information inside the MLP module.
This design enables MLP to explore local correlations, thus enhancing the model performance.
Orthogonality.
Orthogonality has been widely used in neural networks to regularize neurons [58, 24, 48] or learning disentangled representations [54].
In contrast to existing works, we employ orthogonality to group visual tokens for efﬁcient self-attention.
Besides, most of previous works depend on complex optimization algorithms, such as singular value decomposition [31] and iterative approximation [32], or imposing extra regularization terms on the loss functions [58, 48].
Unlike them, we construct an endogenously orthogonal matrix that is more friendly to neural networks with gradient optimizers.
Method 3.1 Overall Architecture
An overview of the Orthogonal Transformer architecture is illustrated in Fig.
For an input image x ∈ R H × W × 3 , we follow [57] and adopt several stacked convolution layers to obtain H 4 × W
patch tokens.
Similar to Swin [35], the transformer blocks are divided into 4 stages to produce hierarchical representations.
We stack blocks of window self-attention (WSA) and orthogonal self-attention (OSA) alternatively for better capacity of global and local modeling.
For the Orthogonal Transformer blocks (OTBs), we introduce Positional MLP (PMLP) with an additional depth-wise convolution (DConv) after GELU in MLP to provide position information.
Based on PMLP, we design two kinds of OTBs, one with transition and one without.
OTBs with transition aim to reduce the spatial resolution by strided DConv and a residual strided convolution.
The ﬁrst three stages are composed of stacked OTBs without transition following by a single OTB with transition, while the last stage only consists of OTBs without transition.
3.2 Orthogonal Self-Attention
Orthogonal self-attention (OSA) is proposed to enable the transformer layers to encode high-resolution images efﬁciently.
Unlike previous works [35, 50, 9, 63], OSA performs self-attention in the orthogonal space to capture global dependency without neglecting local correlations.
It can cover the same global receptive ﬁeld with much less cost than standard self-attention.
3.2.1 Orthogonal Transformation
Orthogonal transformation (OT) corresponds to an orthogonal matrix
A satisfying A T A
=
I , where
I is the identity matrix.
As shown in Fig.
1, we employ orthogonal transformation
A to convert visual tokens into orthogonal tokens for self-attention computation.
Since it is hard to enforce orthogonality in neural networks [31, 32], we ﬁrst introduce a simple yet effective way to construct an endogenously orthogonal matrix that can be optimized as arbitrary orthogonal matrices.
It is based on the following linear algebra theorem [45] (Proof is provided in the appendix).
Theorem 1.
Every real orthogonal n × n matrix is the product of at most n real orthogonal
Householder transformations.
Given n tokens, inspired from Theorem 1, we construct an orthogonal matrix A ∈ R n × n as:
[Equation]
where H i = I − 2 u
i u T i is Householder transformation, u i is a unit column vector.
Providing a set of n random-initialized vectors { v i | i = 0 , ..., n − 1 } , the orthogonal matrix can be reformulated as:
[Equation]
We set { v i | i = 0 , ..., n − 1 } as learnable parameters, and hence A can be optimized as arbitrary orthogonal matrices in neural networks with gradient optimizers.
Remarkably, A can maintain orthogonality endogenously throughout the training process without extra regularizer.
3.2.2 OSA Block
To clarify, we ﬁrst deﬁne two terms:
• Orthogonal window size m o is the size of the sub-window on which OT is performed.
We perform OT separately for local windows to control its computation complexity.
• Orthogonal groups n o is the number of groups into which the tokens are separated by OT.
Since the orthogonal matrix is square, n o = m 2 o .
Now we elaborate how OSA works in the following three steps, token orthogonalization, self-attention for orthogonal tokens, token reverse.
Token Orthogonalization.
Given the input feature Z ∈ R ( h × w ) × c (viewed as h × w tokens with the dimension c ), it is ﬁrstly divided into a grid of non-overlapped windows of size m o × m o , i.e.,
Z ∈ R n w × n o × c , where n w = hm o
× wm o and n o = m 2 o .
Z can be viewed as n w sub-windows, of which each consists of n o tokens, i.e., Z = { Z i ∈ R n o × c
|
i
= 0 , . .
. , n w − 1 } .
Then we perform
OT separately for Z i by multiplying the orthogonal matrix A ∈ R n o × n o and get orthogonalized tokens ˆ Z i =
A Z i ∈ R n o × c .
Finally, we combine the n w sub-windows and permute the feature ˆ Z ∈ R n w × n o × c into n o groups of orthogonal tokens ˆ Z j ∈ R n w × c (where j = 0 , . . .
n o − 1 ).
Self-attention for orthogonal tokens.
We perform standard multi-head self-attention (MSA) groupwisely for ˆ Z j .
Since ˆ Z j has a smaller token number than the input Z (actually hm o × wm o against h × w ), with an appropriate m o , performing self-attention in the orthogonal space can signiﬁcantly reduce the computation cost.
Besides, for OSA, local correlations can be modeled because ˆ Z j is computed from the adjacent tokens in Z , and local details are preserved due to the fact that ˆ Z and Z are reversible to each other.
Token reverse.
After the self-attention computation, we employ A T , as the inverse of A , to recover the visual tokens from the enhanced orthogonal representations.
The token reverse process is not repeated herein because it is exactly the inverse process of token orthogonalization except that the orthogonal matrix is A T .
The complete process of OSA can be deﬁned as:
[Equation]
We simplify the process of token orthogonalization as A Z for description convenience.
Note that when m o = 1 , OSA equals to the global self-attention.
Complexity Analysis.
The computation complexity of the standard global self-attention is
[Equation]
For our OSA, the computational complexity is
[Equation]
The third term is produced by OT but can be ignored when n o (cid:28) √ hw (this is usually true for high-resolution vision tasks).
Compared to the standard global self-attention, OSA’s computation cost is reduced signiﬁcantly from O (( hw ) 2 ) to
[Table]
3.3 Window Self-Attention
Since Orthogonal Transformer is composed of stacked window self-attentions (WSAs) and OSAs
[Table]
we give a brief review of WSA here.
WSA
ﬁrst employ window partition (WP) to split the input feature Z ∈ R ( h × w ) × c into hm w × wm w non-overlapped windows of size m w × m w , then performs
MSA within each window
[Table]
and ﬁnally reconstruct the tokens from the enhanced representation.
The complete process of WSA is deﬁned as:
[Equation]
(6) where f WP and f − 1 WP denote window partition and its reverse, respectively.
WSA has a linear computational complexity to token number hw :
[Equation] [Equation]
3.4 Positional MLP
Earlier ViTs adopt absolute position embedding (APE) [46] or relative position embedding (RPE) [35, 38] to handle the position information but they cannot be well adapted for arbitrary input resolutions [10].
Recently, convolution position embedding (CPE) [10] is proposed to generate position information by convolutional layers.
In detail, CPE adds position encodings into the input tokens outside the transformer blocks.
We follow CPE [10] but apply it in a different way.
As shown in Fig.
2, we introduce Positional MLP (PMLP) to tackle position information in the MLP module.
We equip PMLP with a depth-wise convolution (DConv) after the non-linear GELU.
This design can not only provide position information, but also enhance the capacity of MLP in exploring local correlations.
Based on it, we further develop PMLP with transition by setting the stride of DConv as 2 and adding a residual strided convolution.
PMLP without transition is deﬁned as:
[Equation]
while PMLP with transition is deﬁned as
[Equation]
where f ( s 1) DConv and f ( s 2)
DConv denote depth-wise convolutions with stride 1 and stride 2, respectively, and f ( s 2)
Conv denote a convolution with stride 2.
Discussions.
Note that depth-wise convolution (DConv) has been applied in previous vision transformers.
The major difference between the proposed Positional MLP and others is that we use strided DConv to perform downsampling within the transformer block.
In this way, we do not need extra patch merging layer to perform feature downsampling.
In Table 6, we empirically show that it can achieve better performance than outside the transformer block.
We also conduct extensive experiments to investigate the best location of DConv in Appendix D.4.
There exist various manners to employ DConv in vision transformers (e.g., PVT v2 [51], CeiT [64], CMT [16] and ours adopt
DConv in different locations).
We believe our work is suggestive to ﬁnd appropriate manner of incorporating depth-wise convolution into transformers.
3.5 Architecture Variants
As shown in Table 1, we build four different Orthogonal Transformer backbones by changing the block number and channel number in each stage.
Speciﬁcally, Ortho-S, Ortho-B and Ortho-L are designed to have a similar setting of FLOPs and model size with their Swin Transformer counterparts.
[Table]
For all the models
[Table]
we set the orthogonal window sizes m o for OSA in the four stages as 8, 4, 2, 1, respectively
[Table]
and the window size for WSA as 7.
The expansion ratios in MLP are set as 3 for
Ortho-T and 4 for the other three variants.
For the convolutional patch embedding, we borrow early convolutions from [57] and apply 5 convolutions with the same setting of [57].
For the transformer blocks, we set the last one in each of the ﬁrst 3 stages as OTB with transition.
The second FC and residual convolution change the feature channels.
The convolutions reduce the spatial resolution with the stride of 2.
The kernel sizes of DConv and the residual convolution are set to be 5 × 5 and 3 × 3 , respectively.
More details are in the Appendix.
Experiments
We conduct experiments on a wide range of vision tasks: image classiﬁcation on ImageNet-1K [11], object detection and instance segmentation on COCO 2017 [34], and semantic segmentation on
ADE20K [68].
We also take ablation studies to validate the importance of each component.
More details about experimental settings and extra ablation studies are in the Appendix.
4.1 Image Classiﬁcation on ImageNet-1K
Experimental Settings.
We benchmark our Orthogonal Transformer on ImageNet-1K [11] image classiﬁcation.
We follow the same training strategy in DeiT [43] and adopt the strong data augmentation and regularization, except for repeated augmentation [22] that does not improve performance.
For a fair comparison, we do not use extra training data or extra supervision techniques, such as token labeling [26] and distillation [43], because most of previous works didn’t use them.
All our models are trained from scratch for 300 epochs with the input size of 224 × 224 .
An AdamW optimizer with a cosine decay learning rate scheduler and 5 epochs of linear warm-up is employed.
The initial learning rate, weight decay, and batch-size are 0.001, 0.05, and 1024, respectively.
For Ortho-T, we use a smaller weight decay of 0.01.
The maximum rates of increasing stochastic depth [23] are set as 0.1/0.2/0.4/0.5 for the models from tiny to large.
For the results of 384 × 384 , we ﬁne-tune the models for 30 epochs with learning rate of 1e-5, weight decay of 1e-8 and bach-size of 512.
[Table] [Table]
Results.
Table 2 reports the image classiﬁcation results on ImageNet-1K. It clearly shows that
our
Orthogonal Transformer has a stronger performance than previous models under similar settings of FLOPs and model size.
Speciﬁcally, the tiny model Ortho-T achieves a 74.0% Top-1 accuracy with only 0.7G FLOPs, surpassing QuadTree-B-b0, T2T-7 and PVTv2-b0 by 2%, 2.3% and 3.5%, respectively.
The small model Ortho-S achieves the same accuracy as base model CrossFormer-B with 51% fewer FLOPs.
As for 384 × 384 input size, the large model Ortho-L achieves an accuracy of 85.4%, surpassing Swin-B by 1.2% with similar model size, and outperforming CaiT-s48 with 26% fewer FLOPs.
These quantitative comparisons with SOTA methods demonstrate the efﬁciency and effectiveness of Orthogonal Transformer.
4.2 Object Detection on COCO
Experimental Settings.
Experiments on object detection and instance segmentation are conducted on COCO 2017 dataset [34].
Following [35], we use our transformer models as the backbone network, and use Mask-RCNN [18] and Cascaded Mask R-CNN [3] as the detection and segmentation heads.
For both tasks, the backbones are pretrained on ImageNet-1K, and ﬁne-tuned on the COCO training set with AdamW optimizer.
We take experiments on two common settings: "1 × " (12 training epochs) and "3 × +MS" (36 training epochs with multi-scale training).
The conﬁgurations follow the setting of [35] and are implemented with MMDetection [7].
Results.
Table 3 and Table 4 show the results with Mask R-CNN and Cascade Mask R-CNN, respectively.
The results show that our method achieves the best performance in all comparisons.
For vision transformers with Mask R-CNN framework, our Ortho-S outperforms Focal-T (having similar model size and computational cost) by +2.2 box AP, +1.5 mask AP with the 1 × schedule and +1.
box AP, +0.9 mask AP with the 3 × multi-scale learning schedule.
The results with Cascade Mask
R-CNN also show that our Orthogonal Transformer exceeds the counterparts by evident margins.
[Table] [Table]
shifted window sa 24.
dilated sa 24.
ortho.
sa 24.
window/ortho.
sa 24.
no pos.
abs.
pos.
rel.
pos.
[Table] [Table]
[Table] [Table]
[Table]
conv.
pos.
w/o early convs 23.
[Table]
[Table] [Table]
[Table] [Table]
early convs 24.
outside transition 24.
[Table]
inside transition 24.
4.3 Semantic Segmentation on ADE20K
Experimental Settings.
Experiments on semantic segmentation are conducted on the ADE20K dataset[68].
We employ Semantic FPN [28] and Upernet [56] as the segmentation heads and replace the backbones with our Orthogonal Transformer.
For a fair comparison, we follow the same setting of PVT [50] to train Semantic FPN 80k iterations and apply the setting of Swin [35] to train Upernet for 160k iterations.
More details are provided in the Appendix.
Results.
The results on semantic segmentation are listed in Table 5.
Our method obtains large performance gain over other vision transformers.
For methods that using Semantic FPN as the segmentation head, Orthogonal Transformer surpasses Twins by +3.9 , +2.6 on mIOU.
The results with Upernet indicate that Orthogonal Transformer achieves +2.9 , +1.2 absolutely higher MultiScale(MS) mIOU than Focal Transformer, and reports 51.2 MS mIOU with 81M parameters.
The comparisons on downstream tasks further validate the superiority of our method.
4.4 Ablation Study
In this section, we conduct ablation study to inspect the respective roles of each part in Orthogonal
Transformer, using ImageNet-1K image classiﬁcation, Mask R-CNN ( 1 × ) on COCO object detection, and Semantic FPN on ADE20K semantic segmentation.
Self-Attention Mechanism.
Ablations of the orthogonal self-attention (SA) are reported in Table 6.
Orthogonal SA outperforms window SA and dilated SA consistently on the three tasks.
Since window
SA and dilated SA belong to ﬁne-grained local SAs and coarse-grained global SAs respectively, the results may validate the superiority of orthogonal SA in capturing global dependency while preserving local details.
The shifted window SA achieves similar performance with window SA.
It is designed to allow for cross-window connection.
In the Ortho-S backbone, DConv in MLP allows for cross-window interaction.
That may be why shifted window SA is not superior to window SA herein.
Besides, replacing half of orthogonal SA with window SA can lead to slight performance gain with 0.1G decrease of FLOPs, justifying the design of Orthogonal Transformer.
Position Encoding.
Comparisons of different position embedding methods are reported in Table 6.
Convolution position embedding (CPE) in PMLP performs better than those without position encoding and those with APE or RPE.
The gains are speciﬁcally large on downstream high-resolution tasks, indicating that the design of our CPE is more suitable for arbitrary large input resolutions.
Patch Embedding.
We borrow the idea of early convolutions from [57] to perform overlapped patch embedding.
The results in Table 6 indicate that Ortho-S with early convolutions outperforms the counterpart with non-overlapped patch embedding.
Transition.
Most of previous works using hierarchical architectures [35, 50] perform the transition of token number and feature dimension between stages.
In this work, we adopt a different manner and employ the strided convolutions in PMLP to perform transition in the transformer block.
The results in Table 6 imply that Ortho-S with inside transition can yield performance gain as well as computation reduction compared to the counterpart with outside transition.
Conclusion
We present a new vision transformer, called as Orthogonal Transformer, to serve as a strong general backbone for computer vision.
The orthogonal self-attention mechanism is introduced to capture global information while exploring local correlations.
Positional MLP is developed to handle position information within the transformer block for arbitrary input resolutions.
Orthogonal Transformer achieves state-of-the-art performance in various vision tasks, including image classiﬁcation, object detection, instance segmentation and semantic segmentation.
We expect to apply Orthogonal
Transformer for more vision tasks, such as image processing and video prediction.
Acknowledgment
This work is partially funded by National Key Research and Development Program of China (Grant
No. 2020AAA0140000), National Natural Science Foundation of China (Grant No. 62006228,
U21B2045), and Youth Innovation Promotion Association CAS (Grant No. 2022132)