QueryPose: Sparse Multi-Person Pose Regression via Spatial-Aware Part-Level Query
Abstract
We propose a sparse end-to-end multi-person pose regression framework, termed
QueryPose, which can directly predict multi-person keypoint sequences from the input image.
The existing end-to-end methods rely on dense representations to preserve the spatial detail and structure for precise keypoint localization.
However, the dense paradigm introduces complex and redundant post-processes during inference.
In our framework, each human instance is encoded by several learnable spatial-aware part-level queries associated with an instance-level query.
First, we propose the Spatial Part Embedding Generation Module (SPEGM) that considers the local spatial attention mechanism to generate several spatial-sensitive part embeddings, which contain spatial details and structural information for enhancing the part-level queries.
Second, we introduce the Selective Iteration Module (SIM) to adaptively update the sparse part-level queries via the generated spatial-sensitive part embeddings stage-by-stage.
Based on the two proposed modules, the part-level queries are able to fully encode the spatial details and structural information for precise keypoint regression.
With the bipartite matching, QueryPose avoids the hand-designed post-processes and surpasses the existing dense end-to-end methods with 73.6 AP on MS COCO mini-val set and 72.7 AP on CrowdPose test set.
Code is available at https://github.com/buptxyb666/QueryPose.
Introduction
Multi-person pose estimation (MPPE) aims to locate all person keypoints from the input image, which is a fundamental yet challenging task in computer vision.
With the prevalence of deep learning techniques [1, 2, 3], MPPE has achieved remarkable progress and played an important role in many other vision tasks, such as activity recognition [4, 5, 6, 7] and pose tracking [8, 9].
[Figure]
In general, the existing multi-person pose estimation solutions can be summarized as top-down and bottom-up paradigms.
The top-down strategy [10, 11, 12, 13, 14, 15] uses the human detector to predict person boxes and then leverages the single-person pose estimation model to localize the keypoints on each cropped human image.
The two independent models lead to the non-end-to-end pipeline, or called two-stage pipeline.
Moreover, the human detector involves extra memory as well as computational cost.
The bottom-up strategy [16, 17, 18, 19] uses the keypoint heatmap to locate all person keypoints at first and then assigns them to individuals via heuristic grouping process, as shown in Figure 1(a).
Some researches attempt to leverage densely pixel-wise regression [20, 21, 22, 23, 24, 25], as shown in Figure 1(b), which predicts the center heatmap and pixel-wise keypoint offset in parallel.
It can be regarded as a special case of the bottom-up paradigm.
Both of them can achieve end-to-end optimization thus are more efficient than the two-stage pipeline.
However, in order to maintain the local details and spatial information, ones leverage the dense representations (e.g., keypoint or center heatmap).
The dense manner requires the hand-crafted post-processes to suppress duplicate predictions or perform keypoint grouping during inference.
The post-processes are non-differentiable and always involve many hand-designed parameters to tune.
In this paper, we aim to build a sparse end-to-end MPPE solution to eliminate the complex and redundant post-processes.
Recently, query-based paradigm [26, 27, 28, 29, 30] has attracted much attention due to its purely sparse and end-to-end differentiable training/inference pipeline.
The variants [30, 31, 29] are employed for several computer vision tasks and achieve promising performance.
The success inspires us to construct a sparse end-to-end MPPE method via the learnable query.
Nevertheless, how to effectively apply the query-based paradigm to the MPPE task is rarely explored.
Intuitively, keypoints are similar to the corner points of the bounding box, and the instance-level object query can be used to regress the keypoint coordinates.
However, we observe that it only achieves inferior performance.
We argue that the instance-level query loses the local details and destroys the spatial structure, which are both critical for pose estimation task.
Towards the aforementioned issue, we introduce the learnable part-level queries to learn spatialaware features and further construct a sparse end-to-end multi-person pose regression framework, termed QueryPose.
First, we divide the human region into several local parts and present the Spatial
Part Embedding Generation Module (SPEGM), which uses the local spatial attention mechanism to generate the spatial-sensitive part embeddings for enhancing part-level queries.
Second, we introduce the Selective Iteration Module (SIM) to adaptively update the part-level queries via the generated spatial-sensitive part embeddings stage-by-stage.
The local details and structural features in the learnable part-level queries will be enhanced, and the noise will be filtered.
Based on the bipartite matching, we eliminate all hand-crafted post-processes and directly output the multi-person keypoint coordinate sequences for the input image.
Without bells and whistles, our sparse framework outperforms all dense end-to-end methods on MS COCO [32] and CrowdPose [33].
The main contributions can be summarized into three aspects as follows:
• We propose to leverage the sparse learnable spatial-aware part queries to encode local spatial features.
Specifically, we present the Spatial Part Embedding Generation Module (SPEGM) to generate the spatial-sensitive part embeddings via the local spatial attention mechanism, which preserves the
local spatial features for enhancing the part-level queries.
• We further introduce the Selective Iteration Module (SIM) to adaptively update the learnable part-level queries by using the newly generated spatial-sensitive part embedding stage-bystage.
Accordingly, the encoded local details and spatial structure are strengthened, and the distractive information is ignored.
• With two proposed modules and one-to-one bipartite matching, we construct a purely sparse query-based MPPE framework, termed QueryPose, which removes the complex postprocesses and directly outputs the multi-person keypoint coordinates.
QueryPose achieves state-of-the-art performance and surpasses the most existing end-to-end MPPE methods on both MS COCO and CrowdPose.
Related Work
In this section, we review three relevant parts to our method including top-down paradigm, bottom-up paradigm and query-based paradigm.
Top-down paradigm.
The top-down paradigm conducts single-person pose estimation on each cropped human image.
Most top-down methods [13, 12, 10, 8] concentrate on learning the reliable high-quality feature representations for predicting keypoint heatmap.
A few of researches [34, 35] try to improve the post-process and some others [36, 37] attempt to exploit regression methods to bypass the keypoint heatmap.
However, top-down methods consist of the independent human detector and single-person pose estimation model, leading to the non-end-to-end optimization pipeline.
Bottom-up paradigm.
The bottom-up paradigm [38, 18, 39, 16, 40] formulates this task as per-pixel keypoint positioning and grouping process.
Moreover, densely pixel-wise regression [20, 21, 22, 23, 24, 41, 42] can be considered a special case of the bottom-up paradigm.
One decomposes the MPPE into pixel-wise center localization and keypoint offset regression.
However, both of them leverage the dense representations, thus introducing the non-differentiable post-processes during inference.
Query-based paradigm.
The query-based paradigm [27, 28, 26, 29, 31, 30, 14] uses the learnable object query and one-to-one label assignment to achieve the purely sparse pipeline, which is applied to several vision tasks (e.g., object detection [27, 28, 26] and segmentation [29, 31, 30]).
However, rare research employs the query-based paradigm for MPPE in end-to-end manner.
PRTR [14] trivially uses the keypoint query with multi-head self-attention to probe the image feature, while losing the spatial local structural information, resulting in inferior performance.
In our framework, we propose to leverage several spatial-aware part-level queries to learn local details and spatial structures.
Method 3.1 Overall Architecture
The overall framework is shown in Figure 2.
The network is divided into three parts, including backbone, box decoder and keypoint decoder.
The keypoint decoder consists of the proposed Spatial
Part Embedding Generation Module and Selective Information Module.
The box decoder and keypoint decoder are repeated S times to build the cascade framework.
S is set to 6 by default, following previous methods [27, 28].
We only take stage s as example for illustration.
Backbone.
Given an input image I , we extract the multiple level features P 2 ∼ P 5 via the feature pyramid network built upon the backbone, where P l indicates the feature with 1 / 2 l resolution of input and 256 channels.
We feed multi-level features P 2 ∼ P 5 to box decoder and only high-resolution Pto keypoint decoder.
Box decoder.
The design follows Sparse R-CNN [28], we adopt a small set of learnable proposal boxes B ∈ R N × 4 as dynamic anchors to provide the prior information of object position.
Each proposal box corresponds to a learnable instance-level query.
Concretely, we utilize the 7 × 7 RoIAlign operation to extract features ROI box according to the learnable boxes B .
The extract features are one-to-one correspondence with learnable instance-level queries Q I ∈ R N × d .
After exploring the self-attention across Q I by multi-head self-attention (MHSA) layer, we aggregate the extracted feature ROI box into instance-level query via the dynamic channel MLP ( DyMLP channel in Eq. 1), and feed the enhanced instance-level query into regression head ( Head box in Eq. 1) to predict the
[Figure]
PoseROIAlign
SpatialAttention ...
[Figure] [Figure]
offset for refining the input proposal box.
The above process is summarized as follows:
[Equation]
Keypoint decoder.
After the above process, the instance-level query Q I has encoded the information of the human instance, thus can be used to regress the keypoints.
However, it only obtains unsatisfactory performance.
We argue that the keypoint localization requires more inherent spatial structure and local details than box regression, while Q I loses both spatial structure and local details.
In the light of this, we present to utilize several learnable spatial-aware part-level queries Q P to encode the human pose with spatial structure and local details maintained.
Q P = { Q P n | Q P n ∈ R M × d p } Nn =1 , where n refers to n -th instance and M is the number of part-level query for each instance.
First, by using the refined boxes, we perform the 14 × 14 RoIAlign operation only on high-resolution feature Pto extract the more detailed features ROI pose , which are more suitable for pose estimation.
Second, the proposed Spatial Part Embedding Generation Module (SPEGM) is applied for producing the spatial-sensitive part embeddings E P
= { E P n | E P n ∈ R M × d p } N n =1 via the local spatial attention mechanism.
Each spatial-sensitive part embedding focuses on the specific local part.
Thus the local spatial details and structural information can be preserved.
Third, we utilize the Selective
Iteration Module (SIM) to adaptively update the learnable spatial-aware part-level queries via the spatial-sensitive part embeddings generated in the current stage.
Consequently, the local details and spatial structure information of part-level queries are enhanced, and the noise is filtered.
Then, the spatial-aware part queries are sent into the multi-head self-attention (denoted as MHSA) layer to explore the spatial relationships across the different local parts.
Accordingly, the global structure of each person instance is also encoded into the part-level queries.
Finally, we leverage the non-shared linear layers (denoted as Linear in Eq. 2) to regress the keypoint coordinates respectively.
The computing procedure of keypoint decoder can be formulated as follows:
[Equation]
As shown in Figure 3(b), the instance-level query Q I is iterated across the box decoder and keypoint decoder serially, and the part-level queries Q p are iterated across keypoint
[Figure]
of next stage adopts the predicted box of current stage as proposal box.
The keypoint coordinates are predicted independently for each stage.
3.2 Spatial Part Embedding Generation Module
Considering that the instance-level query Q I loses the local details and spatial structure information compared with dense representations, we propose the Spatial Part Embedding Generation Module to generate the spatial-sensitive part embeddings E p , which are able to maintain more spatial local features conducive to keypoint localization.
The spatial-sensitive part embeddings E p are used to enhance the learnable spatial-aware part-level queries.
Given the single ROI feature ROI pose ∈ R d × H × W as example
, We firstly use the stacked 3 ×convolutional layer and a deconvolutional layer to recover the higher resolution representation
ROI pose ∈ R d × 2 H × 2 W .
Afterward, we take the spatial attention into consideration on the extracted
ROI features ROI pose .
In particular, we utilize a 3 × 3 convolutional layer to squeeze the features along the channel dimension and produce the M -channels attention map A ∈ R M × 2 H × 2 W .
Each channel corresponds to a spatial attention map for specific local parts, M refers to the number of local parts.
We flatten the attention map A along the spatial dimension and normalize the sum of attention weight to 1.0 via the softmax function.
Then, by manipulating the spatial attention maps of the specific local parts, we assemble the spatial feature to generate M spatial-sensitive part embeddings.
The process is formulated as: E P =
Linear ( A ⊗
ROI Tpose ) ∈ R M × d p , where ⊗ indicates dot product.
The spatial-sensitive part embeddings focus on the different local parts, as shown in Figure 5.
Thus the local details and structural information are sufficiently preserved compared with only a single instance-level query.
3.3 Selective Iteration Module
For enabling the learnable part-level queries Q P to learn the spatial-aware features, we propose the Spatial Part Embedding Generation Module (SPEGM) to generate several spatial-sensitive part embeddings E p .
However, we consider that due to the inaccurate bounding boxes and inconsistent optimization goals in the early stage, the spatial-sensitive part embeddings contain disturbance information.
Motivated by this, we present the Selective Iteration Module to adaptively update and refine the part-level queries Q P via the spatial-sensitive part embeddings E p .
First, we directly sum the learnable part-level queries Q s − 1 P ∈ R M
×
d p output from previous stage and the spatial-sensitive part embeddings E sP ∈ R M × d p generated in current stage.
Second, we utilize a Multi-Layer Perceptron together with sigmoid operation to output two weight vectors W s E P and W s − 1 Q P , which serve as two gates to control the contributions of the E sP and Q s − 1 P .
Finally, We leverage the weighted summation to fuse the E sP and Q s − 1 P via W s E P and W s − 1 Q P .
The above process is formulated as:
[Equation]
By using the Selective Iteration Module, the informative local details and spatial structure in part-level queries Q P will be refined, and the noise will be ignored.
3.4 Training and Testing Details
Training.
In our framework, we leverage the instance-level query Q I to conduct the binary classification and bounding box regression, then use several spatial-aware part-level queries Q p to regress corresponding keypoints.
The ground truth of each human instance contains the class label, box corners and keypoints coordinates, which are denoted as C , {( x box 1 , y box 1 ), ( x box 2 , y box 2 )}, and { ( x kpk , y kpk ) }
Kk =1 respectively.
Following previous methods [27, 28], we adopt bipartite matching to perform one-to-one label assignment.
The set-based loss function for instance-wise query is formulated as: L Q I = λ cls ∗ L cls + λ L 1 ∗ L L 1 + λ giou ∗ L giou , where L cls refers to focal loss to perform binary classfication, L L 1 and L giou denote the L1 loss and generalized IoU loss between the predicted box and the corresponding label.
the loss weight λ cls , λ L 1 , λ giou are set to 2.0, 5.0 and 2. respectively.
[Table]
For keypoint regression, we leverage normalizing flow to capture the latent keypoint distribution P θ,ϕ ( x | ROI pose )
[Table]
and model the keypoint regression loss from the perspective of maximum likelihood estimation following RLE [37].
The density distribution P θ,ϕ ( x | ROI pose ) reflects the probability of the keypoint annotation at the position x in ROI pose , where θ and ϕ are the trainable parameters of our regression network and flow model respectively.
For easier optimization, the flow model F ϕ is used to map the simple distribution to deformed one
P ϕ ( ¯x ) .
Our regression network outputs two values ˆ µ and ˆ σ for shifting and rescaling the distribution P ϕ ( ¯x ) , which is formulated as x = ¯x ∗ ˆ σ + ˆ µ .
The process enables the network to learn how the output deviates from annotations.
The different spatial-aware part queries are sent into the non-shared fully-connected layers to regress ˆ µ ∈ R K × 2 and ˆ σ ∈ R K × 2 , where ˆ µ is the corresponding prediction of µ g = { ( x kpk , y kpk ) }
Kk =1 .
Based on the matched pairs, the set-based keypoint regression loss for the spatial-aware part-level queries is formulated as:
[Equation]
It is noteworthy that µ g is normalized by the height and width of the predicted bounding box and shifted to (-0.5, 0.5) for stabilizing the training process.
The keypoint out of the predicted bounding box will be masked.
The total loss of our framework can be written as: L total =
L Q I + L Q P .
Testing.
ˆ σ can be regarded as the uncertainty score of keypoint localization, thus we define the final pose score as 1 K (cid:80)
Kk
=1 (1 − ˆ σ k ) ∗ ¯ C , where ¯ C is the predicted instance score.
During inference, our method eliminates hand-crafted post-procedures (e.g., keypoint NMS, center NMS and grouping) and only need to select the pose candidates with high pose score.
Experiments and Analysis
In this section, we briefly illustrate the dataset, evaluation metric as well as implementation details in subsection 4.1.
Next, we delve into the proposed modules and conduct the comprehensive ablation experiments to verify the effectiveness in subsection 4.2.
Finally, we compare QueryPose with the previous methods on both MS COCO and CrowdPose in subsection 4.3.
4.1 Experimental Setup
Dataset.
MS COCO [32] is a large-scale 2D human pose benchmark, which is split into train2017, mini-val, test-dev2017 sets.
We train our model on COCO train2017 set with 57k images, conduct the evaluation on the mini-val set with 5k images and test-dev2017 set with 20K images respectively.
CrowdPose [33] contains 20,000 images with 80,000 annotated human instances.
The train, validation and test set are partitioned in proportional to 5:1:4.
[Table]
[Table] [Table]
[Table] [Table]
Evaluation Metric.
We leverage average precision and average recall based on different Object
Keypoint Similarity (OKS) [32] thresholds to evaluate performance.
For COCO dataset, AP M and
AP L indicate AP over medium and large-sized instances respectively.
For CrowdPose, AP E , AP M ,
AP H refer to AP scores over easy, medium and hard instances according to dataset annotations.
Implementation Details.
During training, we use random horizontal flip and random crop with probability 0.5 to augment the training samples.
The input images are randomly resized
[Table]
and the short side is randomly sampled from 480 to 800 pixels with the aspect ratio kept.
The number of the sparse instance-level query is set to 100 by default, it can even be reduced to 50 without significant performance degradation due to the foreground only containing the person.
The model is trained by using AdamW [43] optimizer with a mini-batch size of 16 (2 per GPU) on eight Tesla A100 GPUs.
The initial learning rate is linearly scaled to 2.5e-5 and dropped 10 × after 220k, 260k iterations, and terminated at 280k iterations (2 × training schedule).
The learning rate of the flow model is 10 × of the regression model.
All codes are implemented based on Detectron2 [44].
During inference, we keep the aspect ratio and resize the short side of the images to 800 pixels.
4.2 Ablation Study
We conduct the ablative studies on MS COCO mini-val set to explore the contributions of the proposed components and delve into the superiority of its design.
All experiments adopt ResNet-50-FPN and 1 × training scheme.
We first define the baseline as employing the stacked 3 × 3 conv-relu + global average pooling on ROI pose to directly regress keypoint coordinates.
Analysis of the part-level query.
As shown in Table 1, to verify the superiority of the part-level query over the instance-level query, we leverage four different interaction methods with image feature ROI pose : (1) MHSA refers to multi-head self-attention layer.
(2) DyMLP channel indicates the dynamic MLP acting on the channel dimension, where the weight of MLP is generated by the corresponding query.
(3) DyMLP spatial refers to the dynamic MLP acting on the spatial dimension.
[Table]
(4) Spatial Attn.
denotes using global spatial attention to generate the instance embedding to enhance the instancelevel query.
We observe that part-level query can consistently achieve the notable improvements than instance-level query in case of utilizing the different ways to interact with the image features ROI pose .
This phenomenon proves that part-level query is able to preserve more details of local area than instance-level query.
Analysis of Spatial Part Embedding Generation Module.
To study the superiority of the Spatial
Part Embedding Generation Module (SPEGM), we compare the performance with different interaction methods in Table 1.
Notably, using the SPEGM to generate spatial-sensitive part embeddings for boosting the learnable part-level queries achieves the superior performance (63.8 AP) than the other interaction ways.
Furthermore, SPEGM improves the baseline by 4.6 AP as reported in Table 2(b).
We suggest that the local spatial attention is able to provide the spatial prior information and focus on
[Figure]
the informative local regions, as shown in Figure 5.
Furthermore, compared with dynamic MLP with over 10M parameters, SPEGM is more effective and efficient with only 1.1M parameters.
Analysis of Selective Iteration Module.
Selective Iteration Module (SIM) is presented to
[Figure]
adaptively update the learnable spatialaware part queries output from the previous stage.
We conduct the controlled experiments to investigate the superiority of the proposed SIM compared with the other two iteration methods, including summation and concatenation.
As reported in Table 2(a), the Selective Iteration Module achieves 64.9 AP and improves the non-iteration structure by 1.
AP.
Moreover, SIM outperforms the summation and concatenation by 0.7 and 0.
AP, respectively.
We consider that the Selective Iteration Module can adaptively boost the informative spatial details and filter the noise.
Analysis of the dimension for the part-level query.
The part-level queries are used to encode the local spatial detail and structure.
Based on the overall network, we explore the dimension of the spatial-aware part query.
As reported in Table 2(c), we observed that setting the dimension to 128 is sufficient, and the larger dimension can not bring additional gains.
Analysis of the part division scheme.
As shown in Figure 4 and Table 2(d), we probe four different part division schemes to design an efficient and effective partition strategy.
The scheme (c) achieves the optimal performance.
Compared with the scheme (a) and (b), the part-level query in scheme (c) is capable of obtaining local structural information of adjacent keypoints.
Moreover, each part in scheme (c) is rigid without articulated structure, while the part in scheme (d) is deformable, thus obtaining a slightly worse result.
Iteration pipeline of the instance-level query.
As shown in Figure 3 and Table 3, we explore the two different iteration pipeline of the instance-level query Q I .
We observe that the instance-level query iterated across box decoder and keypoint decoder serially can improve 1.6 AP box and 0.8 AP kps than only iterated across box decoder.
4.3 Results
COCO Mini-val set.
Table 4 reports the results of end-to-end methods on COCO mini-val set.
Compared with dense end-to-end methods, QueryPose with ResNet-50 outperforms Mask R-CNN
[Table] [Table]
PointSetNet
[Table] [Table]
HrHRNet † [16] 70.
DEKR † [24] 71.
CenterGroup † [46] 71.
LQR † [41] 71.
AdaptivePose † [20] 71.
QueryPose-W 72.
QueryPose-SwinL 72.
[45] with ResNet-101 by 2.6 AP.
By only using HRNet-W32, we achieve 72.4 AP and markedly outperform all dense counterparts with larger backbone.
QueryPose with HRNet-W48 obtains 73.6 AP and significantly surpasses the previous most competitive HrHRNet-W48 ( Flip ) [16], SWAHR-W ( Flip ) [40], CenterGroup-W48[46] by 3.8 AP, 2.8 AP, 4.5 AP respectively.
Furthermore, QueryPose outperforms the regression-based AdaptivePose-W48 [20] and DEKR-W48( Flip ) [24] by 3.6 AP and 2.6 AP.
QueryPose obtains 7.4 AP improvements over the existing sparse end-to-end method PRTR [14].
Finally, we use the transformer-based backbone Swin-Large [47] and also achieve 73.3 AP, which proves that our method is compatible for various network architecture.
QueryPose eliminates the time-consuming post-processes and achieves the faster inference speed compared with the above state-of-the-art dense competitors.
[Table]
COCO test-dev2017 set.
As shown in Table 5, we achieve 72.3 AP with HRNet-W48 and 72.2 AP with Swin-Large without flip and multi-scale testing.
QueryPose exceeds all end-to-end methods and narrows the gap with two-stage methods.
The results verify the superior keypoint positioning ability of QueryPose in single-forward pass.
CrowdPose test set.
We further evaluate the QueryPose on CrowdPose.
The model is trained on the train and val set, then evaluated on test set as previous methods [16, 40, 24].
We list the comparisons in Table 6.
QueryPose achieves 72.1 AP with HRNet-W48 and 72.7 AP with Swin-large on CrowdPose test set.
The results outperform the most existing MPPE methods.
Discussion.
We leverage the sparse spatial-aware part-level queries to encode the local detail and spatial structure instead of dense feature representation.
Our sparse method achieves the better performance, proving that dense heatmap is not all you need for pose estimation.
In crowded scenes, the sparse paradigm probably avoids the issues caused by the NMS process (e.g., the heavily overlapped instances or keypoints may be removed after center or keypoint NMS).
Accordingly, the sparse paradigm may be more compatible for crowded scenes.
Furthermore, we observe that the typical single-stage regression-based methods (without ROI) require the online auxiliary keypoint heatmap learning during training stage, which can bring 1.5-2.5 AP improvements for different approaches (e.g., AdaptivePose[20], PointSetNet[42]).
In contrast, The online auxiliary keypoint heatmap learning is useless for QueryPose, while the heatmap pretrain is more suitable for QueryPose.
We only leverage HRNet to verify the phenomenons and bring the 1.
AP improvements.
Conclusion
In this paper, we present a sparse end-to-end multi-person pose regression framework termed as
QueryPose.
With two proposed modules, i.e., Spatial Part Embedding Generation Module and
Selective Iteration Module, QueryPose is capable of utilizing the sparse spatial-aware part-level queries to capture the local spatial features instead of previous dense representations and achieving the better performance than most dense MPPE methods.
Our method proves the potential of the sparse regression method for the multi-person pose estimation task.
We believe that our core insight is able to benefit some other methods and inspire future research.
Acknowledgements
This work is supported by the National Natural Science Foundation of China
No