Original: A TTEND TO WHO YOU ARE : S UPERVISING SELFATTENTION FOR K EYPOINT D ETECTION AND I NSTANCE -A WARE A SSOCIATION
Revised: A TTEND TO WHO YOU ARE : S UPERVISING SELFATTENTION FOR K EYPOINT D ETECTION AND I NSTANCE -A WARE A SSOCIATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Revised: This paper presents a new method to solve keypoint detection and instance association by using Transformer.

Original: Bottom-up multi-person pose estimation models need to detect keypoints and learn associative information between keypoints.
Revised: For bottom-up multi-person pose estimation models, they need to detect keypoints and learn associative information between keypoints.

Original: We argue that these problems can be entirely solved by the Transformer model.
Revised: We argue that these problems can be entirely solved by Transformer.

Original: Specifically, the self-attention in Transformer measuresthe pairwise dependencies between  locations, which can play a role in providing association information for keypoints grouping.
Revised: Specifically, the self-attention in Transformer measures dependencies between any pair of locations, which can provide association information for keypoints grouping.

Original: However, the naive attention patterns are still not subjectively controlled, so there is no guarantee that the keypoints will always attend to the instances to which they belong.
Revised: However, the naive attention patterns are still not subjectively controlled, so there is no guarantee that the keypoints will always attend to the instances to which they belong.

Original: To address it we propose a novel approach of  multi-person keypoint detection and instance association using instance masks to supervise self-attention.
Revised: To address it we propose a novel approach of supervising self-attention for multi-person keypoint detection and instance association.

Original: By supervising self-attention to be instance-aware, we can assign the detected keypoints to the correct human instances based on the pairwise attention scores, without using pre-defined offset vector fields or embedding like CNN-based bottom-up models.
Revised: By using instance masks to supervise self-attention to be instance-aware, we can assign the detected keypoints to their instances based on the pairwise attention scores, without using pre-defined offset vector fields or embedding like CNN-based bottom-up models.

Original: An additional benefit of our method is that the instance segmentation results of any number of people can be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline.
Revised: An additional benefit of our method is that the instance segmentation results of any number of people can be directly obtained from the supervised attention matrix, thereby simplifying the pixel assignment pipeline.

Original: The experiments on the COCO multi-person keypoint detection challenge and person instance segmentation task demonstrate the effectiveness and simplicity of the proposed method.
Revised: The experiments on the COCO multi-person keypoint detection challenge and person instance segmentation task demonstrate the effectiveness and simplicity of the proposed method, and show a promising way to control self-attention behavior for specific purposes.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Multi-person pose estimation approaches usually can be classified into two schemes: top-down or bottom-up.
Revised: Multi-person pose estimation approaches usually can be classified into two schemes: top-down or bottom-up.

Original: Unlike the top-down scheme that converts the pipeline into two independent tasks – detection and single-pose estimation, the bottom-up scheme is confronted with more challenging problems.
Revised: Unlike the top-down scheme that converts the pipeline into two independent tasks – detection and single-pose estimation, the bottom-up scheme is confronted with more challenging problems.

Original: An unknown number of persons with any scale, posture, or occlusion condition may appear at any location of the input image.
Revised: An unknown number of persons with any scale, posture, or occlusion condition may appear at any location of the input image.

Original: The bottom-up approaches need to detect all body joints first and group them into instances second.
Revised: The bottom-up approaches need to detect all body joints first and group them into instances second.

Original: In the typical systems such as DeeperCut (Insafutdinov et al., 2016), OpenPose (Cao et al., 2017),
Revised: In the typical systems such as DeeperCut (Insafutdinov et al., 2016), OpenPose (Cao et al., 2017),

Original: Associative Embedding (Newell et al., 2017), PersonLab (Papandreou et al., 2018), PifPaf (Kreiss et al., 2019) and CenterNet (Zhou et al., 2019), keypoint detection and grouping are usually regarded as two heterogeneous learning targets.
Revised: Associative Embedding (Newell et al., 2017), PersonLab (Papandreou et al., 2018), PifPaf (Kreiss et al., 2019) and CenterNet (Zhou et al., 2019), keypoint detection and grouping are usually regarded as two heterogeneous learning targets.

Original: This requires the model to learn the keypoint heatmaps encoding position information and the human knowledge guided signals encoding association information such as part hypotheses, part affinity fields, associative embeddings or offset vector fields.
Revised: This requires the model to learn the keypoint heatmaps encoding position information and the human knowledge guided signals encoding association information such as part hypotheses, part affinity fields, associative embeddings or offset vector fields.

Original: In this paper we explore whether we can exploit the instance semantic clues implicitly used by the model to group the detected keypoints into individual instances.
Revised: In this paper we explore whether we can exploit the instance semantic clues implicitly used by the model to group the detected keypoints into individual instances.

Original: Our key intuition is that, when the model predicts a location of a specific keypoint, it may know the human instance region this keypoint belongs to, which means that the model has implicitly associated related joints together.
Revised: Our key intuition is that, when the model predicts a location of a specific keypoint, it may know the human instance region this keypoint belongs to, which means that the model has implicitly associated related joints together.

Original: For example, when an elbow is recognized, the model may learn its strong spatial dependencies in its adjacent wrist or shoulder but weak dependencies in the joints of other persons.
Revised: For example, when an elbow is recognized, the model may learn its strong spatial dependencies in its adjacent wrist or shoulder but weak dependencies in the joints of other persons.

Original: Therefore, if we can read out such information learned and encoded in the model, the detected keypoints can be correctly grouped into instances, without the help of the human pre-defined associative signals.
Revised: Therefore, if we can read out such information learned and encoded in the model, the detected keypoints can be correctly grouped into instances, without the help of the human pre-defined associative signals.

Original:  We argue that self-attention model (Vaswani et al., 2017) can meet this requirement because it can provide image-specific pairwise similarities between  image locations, and the resulting attention patterns show object-related semantics.
Revised: [Figure] We argue that self-attention model (Vaswani et al., 2017) can meet this requirement because it can provide image-specific pairwise similarities between any pair of image positions without distance limitation, and the resulting attention patterns show object-related semantics.

Original: Hence, we attempt to use self-attention mechanism to per- [Figure] form multi-person pose estimation.
Revised: Hence, we attempt to use self-attention mechanism to perform multi-person pose estimation.

Original: But instead of following the top-down strategy with the single person region as the input, we feed the Transformer model the high-resolution input images with the presence of multiple persons, and expect it to output the heatmaps encoding multi-person keypoint locations.
Revised: But instead of following the top-down strategy with the single person region as the input, we feed the Transformer model the high-resolution input images with the presence of multiple persons, and expect it to output the heatmaps encoding multi-person keypoint locations.

Original: Our initial results show that 1) the heatmaps outputted by Transformer can also accurately respond to multiple persons’ keypoints at multiple candidate locations; 2) the attention scores between the detected keypoint locations tend to be higher within the same person, but lower across different persons.
Revised: Our initial results show that 1) the heatmaps outputted by Transformer can also accurately respond to multiple persons’ keypoints at multiple candidate locations; 2) the attention scores between the detected keypoint locations tend to be higher within the same person but lower across different persons.

Original: Based on these findings, we introduce an attention-based parsing algorithm to group the detected keypoints into different human instances.
Revised: Based on these findings, we introduce an attention-based parsing algorithm to group the detected keypoints into different human instances.

Original: Unfortunately, the naive self-attention does not always show desirable properties.
Revised: Unfortunately, the naive self-attention does not always show desirable properties.

Original: In many cases, a detected keypoint also probably have relatively higher attention scores with those belonging to different person instances.
Revised: In many cases, a detected keypoint also probably have relatively higher attention scores with those belonging to different person instances.

Original: This will definitely lead to wrong associations and implausible human poses.
Revised: This will definitely lead to wrong associations and implausible human poses.

Original: To address this issue, we propose a novel method that leverages a loss function to explicitly supervise the attention area of each person instance by the mask of the instance.
Revised: To address this issue, we propose a novel method that leverages a loss function to explicitly supervise the attention area of each person instance by the mask of the instance.

Original: The results show that supervising self-attentions in such a way can achieve the expected instance-discriminative characteristics without affecting the standard forward propagation of Transformer.
Revised: The results show that supervising self-attentions in such a way can achieve the expected instance-discriminative characteristics without affecting the standard forward propagation of Transformer.

Original: Such characteristics guarantee the effectiveness and accuracy of the attention-based grouping algorithm.
Revised: Such characteristics guarantee the effectiveness and accuracy of the attention-based grouping algorithm.

Original: The results on the COCO keypoint detection challenge show that our models with limited refinement can achieve comparable performances compared with the highly optimized bottom-up pose estimation systems (Cao et al., 2017; Newell et al., 2017; Papandreou et al., 2018).
Revised: The results on the COCO keypoint detection challenge show that our models with limited refinement can achieve comparable performances compared with the highly optimized bottom-up pose estimation systems (Cao et al., 2017; Newell et al., 2017; Papandreou et al., 2018).

Original: Meanwhile, we also can easily obtain the person instance masks by sampling the corresponding attention areas, thereby avoiding an extra pixel assignment or grouping algorithm.
Revised: Meanwhile, we also can easily obtain the person instance masks by sampling the corresponding attention areas, thereby avoiding an extra pixel assignment or grouping algorithm.

Original: 1. K EY C ONTRIBUTIONS
Revised: 1. K EY C ONTRIBUTIONS

Original: Using self-attention to unify keypoint detection, grouping and human mask prediction .
Revised: Using self-attention to unify keypoint detection, grouping and human mask prediction .

Original: We use
Revised: We use

Original: Transformer to solve the challenging multi-person keypoint detection, grouping and mask prediction in a unified way.
Revised: Transformer to solve the challenging multi-person keypoint detection, grouping and mask prediction in a unified way.

Original: We realize that the self-attention shows instance-related semantics, which can be served as the association information in a bottom-up fashion.
Revised: We realize that the self-attention shows instance-related semantics, which can be served as the association information in a bottom-up fashion.

Original: We further use instance masks to supervise the self-attention.
Revised: We further use instance masks to supervise the self-attention.

Original: It ensures that each keypoint is assigned to the correct human instance according to the attention scores, making it easy to obtain the instance masks as well.
Revised: It ensures that each keypoint is assigned to the correct human instance according to the attention scores, making it easy to obtain the instance masks as well.

Original: Supervising self-attention “for your need” .
Revised: Supervising self-attention “for your need” .

Original: A common practice of using Transformer models is to use task-specific signals to supervise the final output of transformer-based models, such as class labels, object box coordinates, keypoint positions or semantic masks.
Revised: A common practice of using Transformer models is to use task-specific signals to supervise the final output of transformer-based models, such as class labels, object box coordinates, keypoint positions or semantic masks.

Original: In this method, a key novelty is to use some type of constraint terms to control the behaviors the self-attention.
Revised: In this method, a key novelty is to use some type of constraint terms to control the behaviors of self-attention.

Original: The results show that under supervision the self-attention can achieve instance-aware characteristics for multi-person pose estimation and mask prediction, without destroying the standard forward of Transformer.
Revised: The results show that under supervision the self-attention can achieve instance-aware characteristics for multi-person pose estimation and mask prediction, without destroying the standard forward of Transformer.

Original: This demonstrates that using appropriate guidance signals makes self-attention controllable and help the model learning, which is also applicable to other vision tasks such asfor instance segmentation (Wang et al., 2021) and object detection (Carion et al., 2020).
Revised: This demonstrates that using appropriate guidance signals makes self-attention controllable and help the model learning, which is also applicable to other vision tasks such as instance segmentation (Wang et al., 2021) and object detection (Carion et al., 2020).

Original:  M ETHOD 2. P ROBLEM S ETTING
Revised:  M ETHOD 2. P ROBLEM S ETTING

Original: Given a RGB image I of size 3 × H × W , the goal of 2D multi-person pose estimation is to estimate all persons’ keypoints locations: S = (cid:8) ( x ki , y ki ) | i = 1 , 2 , ..., N ; k = 1 , 2 , ..., K (cid:9) , where N is the number of persons in this image and K is the number of defined keypoint types.
Revised: Given a RGB image I of size 3 × H × W , the goal of 2D multi-person pose estimation is to estimate all persons’ keypoints locations: S = (cid:8) ( x ki , y ki ) | i = 1 , 2 , ..., N ; k = 1 , 2 , ..., K (cid:9) , where N is the number of persons in this image and K is the number of defined keypoint types.

Original: We follow the bottom-up strategy.
Revised: We follow the bottom-up strategy.

Original: First, the model detects all the candidate locations for each type of keypoints in an image: C = C 1 (cid:83) C 2 (cid:83) ...
Revised: First, the model detects all the candidate locations for each type of keypoints in an image: C = C 1 (cid:83) C 2 (cid:83) ...

Original: (cid:83) C K , where C k
Revised: (cid:83) C K , where C k

Original: = { (ˆ x i , ˆ y i )
Revised: = { (ˆ x i , ˆ y i )

Original: | i = 1 , 2 , ..., N k } represents the k -th type of keypoint set with N k detected candidates.
Revised: | i = 1 , 2 , ..., N k } represents the k -th type of keypoint set with N k detected candidates.

Original: Second, a heuristic decoding algorithm g groups all candidates into M skeletons based on the association information A , which determines a unique person ID for each keypoint location.
Revised: Second, a heuristic decoding algorithm g groups all candidates into M skeletons based on the association information A , which determines a unique person ID for each keypoint location.

Original: We formulate this process as: g ((ˆ x i , ˆ y i ) , C , A ) → m ∈ { 1 , 2 , ..., M } .
Revised: We formulate this process as: g ((ˆ x i , ˆ y i ) , C , A ) → m ∈ { 1 , 2 , ..., M } .

Original: Next, we present the model architecture and show how to use self-attention as the association information A .
Revised: Next, we present the model architecture and show how to use self-attention as the association information A .

Original: We analyze the problems when using the naive self-attention as the grouping reference.
Revised: We analyze the problems when using the naive self-attention as the grouping reference.

Original: We propose to supervise self-attention via instance masks for keypoints grouping.
Revised: We propose to supervise self-attention via instance masks for keypoints grouping.

Original: We present two types of grouping algorithm from the body-first and part-first views.
Revised: We present two types of grouping algorithm from the body-first and part-first views.

Original: Finally, we describe how we obtain the person instance masks and how we use the obtained masks to refine the results.
Revised: Finally, we describe how we obtain the person instance masks and how we use the obtained masks to refine the results.

Original: 2. N ETWORK A RCHITECTURE AND N AIVE S ELF -A TTENTION 
Revised: 2. N ETWORK A RCHITECTURE AND N AIVE S ELF -A TTENTION Architecture

Original: Architecture.

Original: We use a simple architecture combination that includes ResNet (He et al., 2016) and
Revised: We use a simple architecture combination that includes ResNet (He et al., 2016) and

Original: Transformer encoder (Vaswani et al., 2017), like the design of TransPose (Yang et al., 2021).
Revised: Transformer encoder (Vaswani et al., 2017), like the design of TransPose (Yang et al., 2021).

Original: The downsampled feature maps of ResNet with r stride are flattened to a sequence of L × d size and sent to Transformer where L =
Revised: The downsampled feature maps of ResNet with r stride are flattened to a sequence of L × d size and sent to Transformer where L =

Original: Hr × Wr .
Revised: Hr × Wr .

Original: Several transposed convolutions and a 1 × 1 convolution are used to upsample the Transformer output into the target keypoint heatmap size
Revised: Several transposed convolutions and a 1 × 1 convolution are used to upsample the Transformer output into the target keypoint heatmap size

Original: K × H 4 × W 4 .
Revised: K × H 4 × W 4 .

Original: Heatmap loss.
Revised: Heatmap loss.

Original: To observe what patterns the self-attentions layers spontaneously learn, we first only leverage the mean square error (MSE) loss between the predicted heatmap ˆH k and the groundtruth heatmap H k to train the model:
Revised: To observe what patterns the self-attentions layers spontaneously learn, we first only leverage the mean square error (MSE) loss between the predicted heatmap ˆH k and the groundtruth heatmap H k to train the model:

Original: [Equation]
Revised: [Equation]

Original: where M is a mask that masks out the crowd areas and small size person segments in the whole image.
Revised: where M is a mask that masks out the crowd areas and small size person segments in the whole image.

Original: After the model is trained only by heatmap loss, the keypoint detection results show the trained model can accurately localize keypoints of multiple persons.
Revised: After the model is trained only by heatmap loss, the keypoint detection results show the trained model can accurately localize keypoints of multiple persons.

Original: Issues for naive self-attention.
Revised: Issues in naive self-attention.

Original: We obtain the keypoint locations from heatmaps and further visualize the attention areas of these locations.
Revised: We obtain the keypoint locations from heatmaps and further visualize the attention areas of these locations.

Original: As revealed by the examples shown in Figure 1, using the naive self-attention matrices as the association reference poses several challenges: 1) There are multiple attention layers in Transformer, each of which shows distinct characteristics.
Revised: As revealed by the examples shown in Figure 1, using the naive self-attention matrices as the association reference poses several challenges: 1) There are multiple attention layers in Transformer, each of which shows distinct characteristics.

Original: Selecting which attention layers as the association reference and how to process the raw attention require a very thoughtful fusion and post-processing strategy.
Revised: Selecting which attention layers as the association reference and how to process the raw attention require a very thoughtful fusion and post-processing strategy.

Original: Although most of the sampled keypoint locations show local attention areas, especially for the people they belong to, some keypoints may still produce relatively high attention scores for the parts of other people at a longer distance.
Revised: Although most of the sampled keypoint locations show local attention areas, especially for the people they belong to, some keypoints may still produce relatively high attention scores for the parts of other people at a longer distance.

Original: It is almost impossible to determine a perfect attention threshold for all situations, which makes keypoint grouping highly dependent on specific experimental observations.
Revised: It is almost impossible to determine a perfect attention threshold for all situations, which makes keypoint grouping highly dependent on specific experimental observations.

Original: As a consequence, the attention-based grouping cannot ensure the correctness of the keypoint assignment, leading to inferior performance. 
Revised: As a consequence, the attention-based grouping cannot ensure the correctness of the keypoint assignment, leading to inferior performance. [Figure]

Original: 2. S UPERVISING S ELF -A TTENTION BY INSTANCE MASKS
Revised: 2. S UPERVISING S ELF -A TTENTION BY INSTANCE MASKS

Original: To address the aforementioned challenges of using the naive self-attention for keypoints grouping, we S upervise S
Revised: To address the aforementioned challenges of using the naive self-attention for keypoints grouping, we S upervise S

Original: elf-
Revised: elf-

Original: A ttention ( SSA ) to be what we expect.
Revised: A ttention ( SSA ) to be what we expect.

Original: Ideally, the expected attention pattern should be that each keypoint location only attends to the person instance it belongs to.
Revised: Ideally, the expected attention pattern should be that each keypoint location only attends to the person instance it belongs to.

Original: The value distribution (0 or 1) in a person instance mask provides an ideal guidance signal to supervise the pairwise keypoints’s locations to have lower or higher attention scores.
Revised: The value distribution (0 or 1) in a person instance mask provides an ideal guidance signal to supervise the pairwise keypoints’s locations to have lower or higher attention scores.

Original: Then we propose a sparse sampling method based on the instance keypoint locations to supervise the specific attention matrix generated by the self-attention computation in Transformer, as illustrated in Figure 2.[Figure]
Revised: Then we propose a sparse sampling method based on the instance keypoint locations to supervise the specific attention matrix generated by the self-attention computation in Transformer, as illustrated in Figure 2.

Original: Instance mask loss.
Revised: Instance mask loss.

Original: We suppose that the p -th person’s keypoints groudtruth locations are (cid:8) ( x kp , y kp , v kp ) (cid:9) K k
Revised: We suppose that the p -th person’s keypoints groudtruth locations are (cid:8) ( x kp , y kp , v kp ) (cid:9) K k

Original: =1 , where v kp ∈ { 0 , 1 } is a visibility flag, i.e., v kp = 0 : not labeled, v kp = 1 : labeled.
Revised: =1 , where v kp ∈ { 0 , 1 } is a visibility flag, i.e., v kp = 0 : not labeled, v kp = 1 : labeled.

Original: We take out the immediate attention matrix A =
Revised: We take out the immediate attention matrix A =

Original: Softmax( QK ⊤ √ d ) ∈ R L × L of the specific layer in Transformer 1 to leverage the supervision.
Revised: Softmax( QK ⊤ √ d ) ∈ R L × L of the specific layer in Transformer 1 to leverage the supervision.

Original: We first reshape the attention matrix A into a tensor A of ( h × w ) ×
Revised: We first reshape the attention matrix A into a tensor A of ( h × w ) ×

Original: ( h × w ) size, where h = H/r, w = W/r .
Revised: ( h × w ) size, where h = H/r, w = W/r .

Original: Then we transform the keypoint coordinates into the coordinate system of the downsampled feature maps.
Revised: Then we transform the keypoint coordinates into the coordinate system of the downsampled feature maps.

Original: And then we take out the corresponding rows of the attention matrix specified by these locations.
Revised: And then we take out the corresponding rows of the attention matrix specified by these locations.

Original: So we can obtain the reshaped attention map at each keypoint location: A [ int ( y kp /r ) , int ( x kp /r ) , : , :] .
Revised: So we can obtain the reshaped attention map at each keypoint location: A [ int ( y kp /r ) , int ( x kp /r ) , : , :] .

Original: For a person instance, we sample and average the attention maps based on its visible keypoint locations to estimate the mean attention map.
Revised: For a person instance, we sample and average the attention maps based on its visible keypoint locations to estimate the mean attention map.

Original: We name it as person attention map A p :
Revised: We name it as person attention map A p :

Original: [Equation]
Revised: [Equation]

Original: Assuming the groundtruth mask of the p -th person in the image is M p , we also use the MSE loss function to supervise the attention matrix sparsely.
Revised: Assuming the groundtruth mask of the p -th person in the image is M p , we also use the MSE loss function to supervise the attention matrix sparsely.

Original: Since the self-attention scores have been normalized by the softmax function, we need to rescale the A p by dividing its maximum value so that the rescaled A p is closer to the value range of the annotated mask.
Revised: Since the self-attention scores have been normalized by the softmax function, we need to rescale the A p by dividing its maximum value so that the rescaled A p is closer to the value range of the annotated mask.

Original: Note that the size of A p is Hr ×
Revised: Note that the size of A p is Hr ×

Original: Wr while the grountruth instance mask is constructed to be H 4 × W 4 size.
Revised: Wr while the grountruth instance mask is constructed to be H 4 × W 4 size.

Original: So we use r/ 4 times bilinear interpolation to resize the A p to have the same size as the instance mask.
Revised: So we use r/ 4 times bilinear interpolation to resize the A p to have the same size as the instance mask.

Original: We formulate the instance mask loss as:
Revised: We formulate the instance mask loss as:

Original: [Equation]
Revised: [Equation]

Original: Objective.
Revised: Objective.

Original: So the overall objective for training the model is:
Revised: So the overall objective for training the model is:

Original: [Equation]
Revised: [Equation]

Original: where α and β are two coefficients to balance two types of learning.
Revised: where α and β are two coefficients to balance two types of learning.

Original: In the standard self-attention computation of Transformer, the attention matrix is computed by the inner products of queries and keys .
Revised: In the standard self-attention computation of Transformer, the attention matrix is computed by the inner products of queries and keys .

Original: Its gradient back-propagation information is entirely derived from the subsequent attention weighted sum of values .
Revised: Its gradient back-propagation information is entirely derived from the subsequent attention weighted sum of values .

Original: By introducing the instance mask loss to supervise the self-attention, the gradient learning direction for the supervised attention matrix has two sources: the implicit gradient signal from keypoint heatmaps learning and the explicit similarity constraint from instance mask learning.
Revised: By introducing the instance mask loss to supervise the self-attention, the gradient learning direction for the supervised attention matrix has two sources: the implicit gradient signal from keypoint heatmaps learning and the explicit similarity constraint from instance mask learning.

Original: Choosing approximate values of α and β is critical for training the model well.
Revised: Choosing approximate values of α and β is critical for training the model well.

Original: We set α = 1 , β = 0 .
Revised: We set α = 1 , β = 0 .

Original: 01 to balance both heatmap learning and mask learning
Revised: 01 to balance both heatmap learning and mask learning.

Original: [Figure]
Revised: [Figure]

Original: When the well-trained model makes a single forward pass for a given image, we can decode the multi-person human poses and masks from the outputted keypoint heatmaps and the supervised attention matrix in the immediate attention layer.
Revised: When the well-trained model makes a single forward pass for a given image, we can decode the multi-person human poses and masks from the outputted keypoint heatmaps and the supervised attention matrix in the immediate attention layer.

Original: We first conduct non-maximum suppression in a 7 × 7 local window on the keypoint heatmaps and obtain all local maximum locations whose scores exceed the threshold t .
Revised: We first conduct non-maximum suppression in a 7 × 7 local window on the keypoint heatmaps and obtain all local maximum locations whose scores exceed the threshold t .

Original: We put all these candidates into a queue and decode them into skeletons using the attention-based algorithm.
Revised: We put all these candidates into a queue and decode them into skeletons using the attention-based algorithm.

Original: Using the self-attention similarity matrix with quadratic complexity inevitably brings redundant computation.
Revised: Using the self-attention similarity matrix with quadratic complexity inevitably brings redundant computation.

Original: However, in part, this also makes minimal assumptions about where the keypoints of the instances may appear and the number of persons in the image.
Revised: However, in part, this also makes minimal assumptions about where the keypoints of the instances may appear and the number of persons in the image.

Original: Next we present the self-attention based algorithms from the body-first and part-first views.
Revised: Next we present the self-attention based algorithms from the body-first and part-first views.

Original: Body-first view .
Revised: Body-first view .

Original: This view aims to decode each person skeleton one-by-one from the queue.
Revised: This view aims to decode each person skeleton one-by-one from the queue.

Original: Assuming we have the sorted all types of candidate keypoints by descending order of score in a single queue, we pop out the first keypoint (maybe any keypoint type) to seed a new skeleton S , and then greedily find the best matched adjacent candidate keypoint from the queue.
Revised: Assuming we have the sorted all types of candidate keypoints by descending order of score in a single queue, we pop out the first keypoint (maybe any keypoint type) to seed a new skeleton S , and then greedily find the best matched adjacent candidate keypoint from the queue.

Original: For the seeded S with the initial keypoint, we find the other keypoints along the search path according to a defined human skeleton kinematic tree.
Revised: For the seeded S with the initial keypoint, we find the other keypoints along the search path according to a defined human skeleton kinematic tree.

Original: When looking for a certain type of joint, the founded joints (denoted as the set S f ) of this skeleton S induce a basin of attraction to “attract” the joint that most likely belongs to it, as illustrated in Figure 3.
Revised: When looking for a certain type of joint, the founded joints (denoted as the set S f ) of this skeleton S induce a basin of attraction to “attract” the joint that most likely belongs to it, as illustrated in Figure 3.

Original: For a certain unmatched point p c = ( x, y, s ) in the candidate set C k of the keypoint type k , we use the mean attention scores between the current found keypoints and p c as the metric to measure the attraction from this skeleton 2 : Attraction( p c , S f )
Revised: For a certain unmatched point p c = ( x, y, s ) in the candidate set C k of the keypoint type k , we use the mean attention scores between the current found keypoints and p c as the metric to measure the attraction from this skeleton 2 : Attraction( p c , S f )

Original: = 

Revised: =

Original: |S f | (cid:80) ( x ′ ,y ′ ,s ′ )
Revised: |S f | (cid:80) ( x ′ ,y ′ ,s ′ )

Original: ∈S f s ′ ·
Revised: ∈S f s ′ ·

Original: A [ y, x, y ′ , x ′ ] .
Revised: A [ y, x, y ′ , x ′ ] .

Original: Thus the candidate point with the highest score ×
Revised: Thus the candidate point with the highest score ×

Original: Attraction is considered to belong to the current skeleton S :
Revised: Attraction is considered to belong to the current skeleton S :

Original: [Equation]
Revised: [Equation]

Original: We repeat the process above and record all the matched keypoints until all keypoints of this skeleton have been found.
Revised: We repeat the process above and record all the matched keypoints until all keypoints of this skeleton have been found.

Original: Then we need to decode the next skeleton.
Revised: Then we need to decode the next skeleton.

Original: We pop the first unmatched keypoint to seed a new skeleton S ′
Revised: We pop the first unmatched keypoint to seed a new skeleton S ′

Original: again.
Revised: again.

Original: We follow the previous steps to find keypoints belonging to this instance.
Revised: We follow the previous steps to find keypoints belonging to this instance.

Original: Note if the Attraction( p ∗ c , S f ) is smaller than a threshold λ (empirically set to 0.0025), this type of keypoint in this skeleton to be empty (zero-filling).
Revised: Note if the Attraction( p ∗ c , S f ) is smaller than a threshold λ (empirically set to 0.0025), this type of keypoint in this skeleton to be empty (zero-filling).

Original: It is also worth noting that we also consider the keypoints that have already been claimed by a previous skeleton S , but only when
Revised: It is also worth noting that we also consider the keypoints that have already been claimed by a previous skeleton S , but only when

Original: Attraction( p c , S ′ f ) >
Revised: Attraction( p c , S ′ f ) >

Original: Attraction( p c , S ) , we assign the matched p c to the current skeleton S ′ . 
Revised: Attraction( p c , S ) , we assign the matched p c to the current skeleton S ′ . ( int ( x/r , int ( y/r ) we operation [Table]

Original: Part-first view .
Revised: Part-first view .

Original: This view aims to decode all human skeletons part-by-part.
Revised: This view aims to decode all human skeletons part-by-part.

Original: Given all candidates for each keypoint type, we initialize multiple skeleton seeds (cid:8) S 1 , S 2 , ..., S m (cid:9) with the most easily detected keypoints such as nose.
Revised: Given all candidates for each keypoint type, we initialize multiple skeleton seeds (cid:8) S 1 , S 2 , ..., S m (cid:9) with the most easily detected keypoints such as nose.

Original: Then we follow a fixed order to connect the candidate parts to the current skeletons.
Revised: Then we follow a fixed order to connect the candidate parts to the current skeletons.

Original: These skeletons can be seen as multiple clusters consisting of found keypoints.
Revised: These skeletons can be seen as multiple clusters consisting of found keypoints.

Original: Like the body-first view, we also use the mean attention attraction Attraction( p c , S tf ) from the found keypoints in the skeletons as the metric to assign the candidate parts (Figure 3).
Revised: Like the body-first view, we also use the mean attention attraction Attraction( p c , S tf ) from the found keypoints in the skeletons as the metric to assign the candidate parts (Figure 3).

Original: But in the part-first view, we compute the pairwise distance matrix between the candidate parts and existing skeletons, and then we use the Hungarian algorithm (Kuhn, 1955) to solve this bipartite graph matching problem.
Revised: But in the part-first view, we compute the pairwise distance matrix between the candidate parts and existing skeletons, and then we use the Hungarian algorithm (Kuhn, 1955) to solve this bipartite graph matching problem.

Original: Note, if an Attraction( p c , S t ) that represents a matching in the solution is lower than a threshold λ , we use this corresponding candidate part to start a new skeleton seed.
Revised: Note, if an Attraction( p c , S t ) that represents a matching in the solution is lower than a threshold λ , we use this corresponding candidate part to start a new skeleton seed.

Original: We repeat the process above until all types of candidate parts have been assigned.
Revised: We repeat the process above until all types of candidate parts have been assigned.

Original: This part-first grouping algorithm can achieve the optimal solution for assigning local parts to the skeletons although it cannot[Table] guarantee the global optimal assignment.
Revised: This part-first grouping algorithm can achieve the optimal solution for assigning local parts to the skeletons although it cannot guarantee the global optimal assignment.

Original: We choose the part-first grouping as the default.
Revised: We choose the part-first grouping as the default.

Original: And we compare both algorithms on the performance, complexity and runtime in Appendix A.5.
Revised: And we compare both algorithms on the performance, complexity and runtime in Appendix A.5.

Original: 2. M ASK P REDICTION
Revised: 2. M ASK P REDICTION

Original: The instance masks are easy to obtain after the detected keypoints have been grouped into skeletons.
Revised: The instance masks are easy to obtain after the detected keypoints have been grouped into skeletons.

Original: To produce the instance segmentation results, we sample the visible keypoint locations (cid:8) (ˆ x km , ˆ y km , ˆ v km ) (cid:9) K k
Revised: To produce the instance segmentation results, we sample the visible keypoint locations (cid:8) (ˆ x km , ˆ y km , ˆ v km ) (cid:9) K k

Original: =1 of the m -th instance from the supervised self-attention matrix: 
Revised: =1 of the m -th instance from the supervised self-attention matrix: ˆ

Original: [Equation]
Revised: A m = (cid:80) k δ (ˆ v km > 0) · A [ˆ y km , ˆ x km , : , :] (cid:80) k δ (ˆ v k m > 0) .

Original: Then we achieve the estimated instance mask:
Revised: Then we achieve the estimated instance mask:

Original: ˆ M m =
Revised: ˆ M m =

Original: [Equation]
Revised: [Equation]

Original:  where σ is a threshold (0.4 by default) to determine the mask region.
Revised: > σ , where σ is a threshold (0.4 by default) to determine the mask region.

Original: When we obtain the initial skeletons and masks for all person instances, the joints of a person may fall in multiple incomplete skeletons, but their corresponding segments (sampled attention areas) may overlap.
Revised: When we obtain the initial skeletons and masks for all person instances, the joints of a person may fall in multiple incomplete skeletons, but their corresponding segments (sampled attention areas) may overlap.

Original: Thus we further perform non-maximum suppression to merge instances if the Intersection-over-Max (IoM) of two masks exceeds 0.3, where Max denotes the maximum area between two masks.
Revised: Thus we further perform non-maximum suppression to merge instances if the Intersection-over-Max (IoM) of two masks exceeds 0.3, where Max denotes the maximum area between two masks.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Dataset .
Revised: Dataset .

Original: We evaluate our method on the COCO keypoint dectection challenge (Lin et al., 2014) and on the instance segmentation of the COCO person category.
Revised: We evaluate our method on the COCO keypoint dectection challenge (Lin et al., 2014) and on the instance segmentation of the COCO person category.

Original: Model setup .
Revised: Model setup .

Original: We follow the model architecture design of TransPose (Yang et al., 2021) to predict the keypoint heatmaps.
Revised: We follow the model architecture design of TransPose (Yang et al., 2021) to predict the keypoint heatmaps.

Original: The setup is built on top of pre-existing ResNet and Transformer Encoder.
Revised: The setup is built on top of pre-existing ResNet and Transformer Encoder.

Original: We use the Imagenet pre-trained ResNet-101 or ResNet-151 as the backbone whose final classification layer is replaced by a 1 × 1 convolution to reduce the channels from 2048 to d (192).
Revised: We use the Imagenet pre-trained ResNet-101 or ResNet-151 as the backbone whose final classification layer is replaced by a 1 × 1 convolution to reduce the channels from 2048 to d (192).

Original: The normal output stride of ResNet backbone is 32 but we increase the feature map resolution of its final stage (C5 stage) by adding the dilation and removing the stride, i.e., the downsampling ratio r of ResNet is 16.
Revised: The normal output stride of ResNet backbone is 32 but we increase the feature map resolution of its final stage (C5 stage) by adding the dilation and removing the stride, i.e., the downsampling ratio r of ResNet is 16.

Original: We use a regular Transformer with 6 encoder layers with a single attention head for each layer.
Revised: We use a regular Transformer with 6 encoder layers with a single attention head for each layer.

Original: The hidden dimension of FFN is 384.
Revised: The hidden dimension of FFN is 384.

Original: See more training and inference details in Appendix A.1.
Revised: See more training and inference details in Appendix A.1.

Original: 3. R ESULTS ON COCO KEYPOINT DETECTION AND PERSON INSTANCE SEGMENTATION
Revised: 3. R ESULTS ON COCO KEYPOINT DETECTION AND PERSON INSTANCE SEGMENTATION

Original: The standard evaluation metric for COCO keypoint localization is the object keypoint similarity (OKS) and the mean average precision (AP) over 10 thresholds (0.5,0.55,...,0.95) is regarded as the performance metric.
Revised: The standard evaluation metric for COCO keypoint localization is the object keypoint similarity (OKS) and the mean average precision (AP) over 10 thresholds (0.5,0.55,...,0.95) is regarded as the performance metric.

Original: We train our models on COCO train2017 set, and evaluate the model on the val2017 and test-dev2017 sets, as shown in Table 1, Table 2 and Figure 4.
Revised: We train our models on COCO train2017 set, and evaluate the model on the val2017 and test-dev2017 sets, as shown in Table 1 and Table 2.

Original: We mainly compare with the typical bottom-up models that have similar pipelines to our method: OpenPose (Cao et al., 2017),
Revised: We mainly compare with the typical bottom-up models that have similar pipelines to our method: OpenPose (Cao et al., 2017),

Original: PersonLab (Papandreou et al., 2018), and AE (Newell et al., 2017).
Revised: PersonLab (Papandreou et al., 2018), and AE (Newell et al., 2017).

Original: Following the works (Cao et al., 2017; Newell et al., 2017), we also refine the grouped skeletons using a single pose estimator.
Revised: Following the works (Cao et al., 2017; Newell et al., 2017), we also refine the grouped skeletons using a single pose estimator.

Original: We adopt the COCO pretrained TransPose-R-A4 (Yang et al., 2021) that has a very similar architecture to our model and has only 6M parameters.
Revised: We adopt the COCO pretrained TransPose-R-A4 (Yang et al., 2021) that has a very similar architecture to our model and has only 6M parameters.

Original: We apply the single pose estimator to each single scaled person region achieved by the box containing the person mask.
Revised: We apply the single pose estimator to each single scaled person region achieved by the box containing the person mask.

Original: Note that the refinement results are highly dependent on the effect of the grouping and mask prediction, and we only update the keypoint estimates where the predictions of the two models are almost the same.
Revised: Note that the refinement results are highly dependent on the effect of the grouping and mask prediction, and we only update the keypoint estimates where the predictions of the two models are almost the same.

Original: The concrete update rule is whether the keypoint similarity (KS) metric 3 computing between two keypoints exceeds 0.75, indicating that the distance between two predicted locations is already very small.
Revised: The concrete update rule is whether the keypoint similarity (KS) metric 3 computing between two keypoints exceeds 0.75, indicating that the distance between two predicted locations is already very small.

Original: [Figure]
Revised: [Figure]

Original: Analysis .
Revised: Analysis .

Original: We further analyze the differences between pure bottom-up results and the refined ones through the benchmarking and error diagnosis tool (Ronchi & Perona, 2017).
Revised: We further analyze the differences between pure bottom-up results and the refined ones through the benchmarking and error diagnosis tool (Ronchi & Perona, 2017).

Original: We compare our methods with the typical OpenPose model and the Transformer-based model.
Revised: We compare our methods with the typical OpenPose model and the Transformer-based model.

Original: The yielded localization error bars (Figure 5) reveal the weaknesses and strengths of our model: (1) Jitter error :
Revised: The yielded localization error bars (Figure 4) reveal the weaknesses and strengths of our model: (1) Jitter error :

Original: The heatmap localization precision under the existence of multi-person is still not as accurate as the localization precision of single pose estimate; (2) Missing error : Since our algorithm does not ensure that the coordinate of every keypoint in a detected pose has been predicted, if the GT coordinate of a keypoint is annotated, zero-filling coordinates will seriously pull down the calculated OKS value.
Revised: The heatmap localization precision under the existence of multi-person is still not as accurate as the localization precision of single pose estimate; (2) Missing error : Since our algorithm does not ensure that the coordinate of every keypoint in a detected pose has been predicted, if the GT coordinate of a keypoint is annotated, zero-filling coordinates will seriously pull down the calculated OKS value.

Original: Thus, for the evaluation, it is necessary to produce complete predictions.
Revised: Thus, for the evaluation, it is necessary to produce complete predictions.

Original: When we further use the single pose estimator to fill the missing joints with zero scores in the initially grouped skeletons, it achieves about 7 AP gains (Table 1) and reduces the missing error (shown in Figure 5(d)); (3)
Revised: When we further use the single pose estimator to fill the missing joints with zero scores in the initially grouped skeletons, it achieves about 7 AP gains (Table 1) and reduces the missing error (shown in Figure 4(d)); (3)

Original: Inversion error :
Revised: Inversion error :

Original: Forcing diverse keypoint types in an individual instance to have higher query-key similarity may make it difficult for the model to distinguish different keypoint types, especially the left and right inversion; (4) Swap error :
Revised: Forcing diverse keypoint types in an individual instance to have higher query-key similarity may make it difficult for the model to distinguish different keypoint types, especially the left and right inversion; (4) Swap error :

Original: We notice that our pure bottom-up model has fewer swap errors (1.2 % , shown in Figure 5(c)), which represents less confusion between semantically similar parts of different instances.
Revised: We notice that our pure bottom-up model has fewer swap errors (1.2 % , shown in Figure 4(c)), which represents less confusion between semantically similar parts of different instances.

Original: It indicates that compared with OpenPose model, our attention-based grouping strategy performs relatively better in assigning parts to their corresponding instances.
Revised: It indicates that compared with OpenPose model, our attention-based grouping strategy performs relatively better in assigning parts to their corresponding instances.

Original: [Table] [Table]
Revised: We show the qualitative human poses and instance segmentation results in Appendix A.6.

Original: Person instance segmentation .
Revised: Person instance segmentation .

Original: We evaluate the instance segmentation results on COCO val split (person category only).
Revised: We evaluate the instance segmentation results on COCO val split (person category only).

Original: We compare our method with PersonLab (Papandreou et al., 2018).
Revised: We compare our method with PersonLab (Papandreou et al., 2018).

Original: In
Revised: In

Original: Table [Table] we report the results with a maximum of 20 person proposals due to the convention of the
Revised: Table 3, we report the results with a maximum of 20 person proposals due to the convention of the

Original: COCO person keypoint evaluation protocol.
Revised: COCO person keypoint evaluation protocol.

Original: The results on the mean average precision (AP) show that our model still has a gap in the segmentation performance compared with PersonLab.
Revised: The results on the mean average precision (AP) show that our model still has a gap in the segmentation performance compared with PersonLab.

Original: We argue that this is mainly because we conduct the mask learning on low-resolution attention maps that have  been downsampled 16 times w.r.t.
Revised: We argue that this is mainly because we conduct the mask learning on low-resolution attention maps that have [Table] [Table] been downsampled 16 times w.r

Revised: [Table]

Original:  the 640 2 or 800 2 input resolution while the reported PersonLab result is based on 8 times downsampling w.r.t the 1401 2 input resolution.
Revised: t. the 640 2 or 800 2 input resolution while the reported PersonLab result is based on 8 times downsampling w.r.t the 1401 2 input resolution.

Original: As shown in Table 3, our model performs worse on small and medium scales but achieves comparable or even superior performance on large scale persons even if PersonLab uses a larger resolution.
Revised: As shown in Table 3, our model performs worse on small and medium scales but achieves comparable or even superior performance on large scale persons even if PersonLab uses a larger resolution.

Original: 3. C OMPARISON BETWEEN NAIVE SELF - ATTENTION AND SUPERVISED SELF - ATTENTION
Revised: 3. C OMPARISON BETWEEN NAIVE SELF - ATTENTION AND SUPERVISED SELF - ATTENTION

Original: To study the differences in model learning when trained with and without supervising self-attention, we compare their convergences in the heatmap loss and instance mask loss, since the overfitting on
Revised: To study the differences in model learning when trained with and without supervising self-attention, we compare their convergences in the heatmap loss and instance mask loss, since the overfitting on

Original: COCO train data is usually not an issue.
Revised: COCO train data is usually not an issue.

Original: As illustrated in Figure 6, compared with training the naive self-attention model, supervising self-attention achieves a better fitting effect in the mask learning, while achieving an acceptable sacrifice on the fitting of heatmap learning.
Revised: As illustrated in Figure 5, compared with training the naive self-attention model, supervising self-attention achieves a better fitting effect in the mask learning, while achieving an acceptable sacrifice on the fitting of heatmap learning.

Original: It is worth noting that the instance mask training loss curve of the naive self-attention model drops slightly, which suggests that the spontaneously formed attention pattern has a tendency to instance-awareness.
Revised: It is worth noting that the instance mask training loss curve of the naive self-attention model drops slightly, which suggests that the spontaneously formed attention pattern has a tendency to instance-awareness.

Original: To quantitatively evaluate the performance of using naive self-attention patterns for keypoint grouping, we average the attentions from all transformer layers as the association reference (shown in Figure 1).
Revised: To quantitatively evaluate the performance of using naive self-attention patterns for keypoint grouping, we average the attentions from all transformer layers as the association reference (shown in Figure 1).

Original: When we use the totally same conditions (including model configuration, training testing settings and grouping algorithm) of the supervised self-attention model based on (res152, s16, i640), we achieve 29.0AP on COCO validation set, which is far from the 50.7AP result achieved by supervising self-attention.
Revised: When we use the totally same conditions (including model configuration, training testing settings and grouping algorithm) of the supervised self-attention model based on (res152, s16, i640), we achieve 29.0AP on COCO validation set, which is far from the 50.7AP result achieved by supervising self-attention.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Transformer.
Revised: Transformer.

Original: Transformer (Vaswani et al., 2017) has shown very powerful visual relation modeling capability in various computer vision tasks, such as image classification (Dosovitskiy et al., 2020; Touvron et al., 2020), object detection (Carion et al., 2020; Zhu et al., 2020), semantic seg- [Figure] mentation (Zheng et al., 2021), tracking (Sun et al., 2020; Meinhardt et al., 2021), human pose estimation (Lin et al., 2021; Li et al., 2021a; Yang et al., 2021;
Revised: Transformer (Vaswani et al., 2017) has shown very powerful visual relation modeling capability in various computer vision tasks, such as image classification (Dosovitskiy et al., 2020; Touvron et al., 2020), object detection (Carion et al., 2020; Zhu et al., 2020), semantic segmentation (Zheng et al., 2021), tracking (Sun et al., 2020; Meinhardt et al., 2021), human pose estimation (Lin et al., 2021; Li et al., 2021a; Yang et al., 2021;

Original: Li et al., 2021b; Stoffl et al., 2021) and etc.
Revised: Li et al., 2021b; Stoffl et al., 2021) and etc.

Original: The common practice of these methods is to use the task-specific supervision signals such as class labels, object box coordinates, keypoint positions or semantic masks to supervise the final output of transformer-based models.
Revised: The common practice of these methods is to use the task-specific supervision signals such as class labels, object box coordinates, keypoint positions or semantic masks to supervise the final output of transformer-based models.

Original: They may visualize the attention maps to understand the  model but few works directly use it as an explicit function in the inference process.
Revised: They may visualize the attention maps to understand the [Figure] model but few works directly use it as an explicit function in the inference process.

Original: Our work gives a successful example of explicitly using and supervising attention for a specific purpose.
Revised: Our work gives a successful example of explicitly using and supervising attention for a specific purpose.

Original: Human Pose Estimation & Instance segmentation.
Revised: Human Pose Estimation & Instance segmentation.

Original: Multi-person pose estimation methods are usually classified into two categories: top-down (TD) or bottom-up (BU).
Revised: Multi-person pose estimation methods are usually classified into two categories: top-down (TD) or bottom-up (BU).

Original: TD models first detect persons, and then estimate single pose for each person, such as G-RMI (Papandreou et al., 2017),
Revised: TD models first detect persons, and then estimate single pose for each person, such as G-RMI (Papandreou et al., 2017),

Original: Mask-RCNN (He et al., 2017), CPN (Chen et al., 2018), SimpleBaseline (Xiao et al., 2018), and
Revised: Mask-RCNN (He et al., 2017), CPN (Chen et al., 2018), SimpleBaseline (Xiao et al., 2018), and

Original: HRNet (Sun et al., 2019).
Revised: HRNet (Sun et al., 2019).

Original: BU models need to detect the existence of various types of keypoints at any position and scale.
Revised: BU models need to detect the existence of various types of keypoints at any position and scale.

Original: And matching keypoints into instances requires the model to learn dense association signals pre-defined by human knowledge.
Revised: And matching keypoints into instances requires the model to learn dense association signals pre-defined by human knowledge.

Original: OpenPose (Cao et al., 2017) proposes part affinity field (PAF) to measure the association between keypoints by computing the integral along the connecting line.
Revised: OpenPose (Cao et al., 2017) proposes part affinity field (PAF) to measure the association between keypoints by computing the integral along the connecting line.

Original: Associative Embedding (Newell et al., 2017) abstracts an embedding as the human ‘tag’ ID to measure the association.
Revised: Associative Embedding (Newell et al., 2017) abstracts an embedding as the human ‘tag’ ID to measure the association.

Original: PersonLab (Papandreou et al., 2018) constructs midrange offset as the geometric embedding to group keypoints into instances.
Revised: PersonLab (Papandreou et al., 2018) constructs midrange offset as the geometric embedding to group keypoints into instances.

Original: In addition, single-stage methods (Zhou et al., 2019; Nie et al., 2019) also use offset field to assign keypoints to their centers.
Revised: In addition, single-stage methods (Zhou et al., 2019; Nie et al., 2019) also regress offset field to assign keypoints to their centers.

Original: Compared with them, we use Transformer to capture the intra-dependencies within a person and inter-dependencies across different persons.
Revised: Compared with them, we use Transformer to capture the intra-dependencies within a person and inter-dependencies across different persons.

Original: And we explicitly exploit the intrinsic property of selfattention mechanism to solve the association problem, rather than using pre-defined offset fields or embeddings.
Revised: And we explicitly exploit the intrinsic property of self-attention mechanism to solve the association problem, rather than regressing highly abstracted offset fields or embeddings.

Original: The generic instance segmentation methods also can be categorized into top-down and bottom-up schemes.
Revised: The generic instance segmentation methods also can be categorized into top-down and bottom-up schemes.

Original: Top-down approaches predict the instance masks based on the object proposals, such as FCIS (Li et al., 2017) and Mask-RCNN (He et al., 2017).
Revised: Top-down approaches predict the instance masks based on the object proposals, such as FCIS (Li et al., 2017) and Mask-RCNN (He et al., 2017).

Original: Bottom-up approaches mainly cluster the semantic segmentation results to obtain instance segmentation using an embedding space or a discriminative loss to measure the pixel association like (Newell et al., 2017; De Brabandere et al., 2017; Fathi et al., 2017).
Revised: Bottom-up approaches mainly cluster the semantic segmentation results to obtain instance segmentation using an embedding space or a discriminative loss to measure the pixel association like (Newell et al., 2017; De Brabandere et al., 2017; Fathi et al., 2017).

Original: Compared with them, our method uses self-attention to measure the association and estimates instance masks based on instance keypoints.
Revised: Compared with them, our method uses selfattention to measure the association and estimates instance masks based on instance keypoints.

Original:  D ISCUSSION 
Revised:  D ISCUSSION AND F UTURE W ORKS

Original: This paper proposes to exploit Transformer to solve keypoint detection and instance association.
Revised: This paper presents a new method to solve keypoint detection and instance association by using Transformer.

Original: We supervise the inherent characteristics of self-attention – the feature similarity between any pair of positions – to solve the grouping problem of the keypoints or pixels.
Revised: We supervise the inherent characteristics of self-attention – the feature similarity between any pair of positions – to solve the grouping problem of the keypoints or pixels.

Original: Unlike a typical CNN-based bottom-up model, it no longer requires a pre-defined vector field or embedding as the associative reference, thus reducing the model redundancy and simplifying the pipeline.
Revised: Unlike a typical CNN-based bottom-up model, it no longer requires a pre-defined vector field or embedding as the associative reference, thus reducing the model redundancy and simplifying the pipeline.

Revised: We demonstrate the effectiveness and simplicity of the proposed method on the challenging COCO keypoint detection and person instance segmentation tasks.

Original: There remain limitations in the current approach.
Revised: The current approach also brings limitations and challenges.

Original: Due to the quadratic complexity of the standard Transformer, the modelperformance still struggles in  scaling  the model size and the  input resolution.
Revised: Due to the quadratic complexity of the standard Transformer, the model still struggles in simultaneously scaling up the Transformer capacity and the resolution of the input image.

Original: The selection of loss criteria,  architecture, and training procedures can be further optimized.
Revised: The selection of loss criteria, model architecture, and training procedures can be further optimized.

Original: In addition, the reliance on the instance mask annotations also can be removed, such as by imposing high and low attention constraint losses only on the pairs of keypoint locations.
Revised: In addition, the reliance on the instance mask annotations also can be removed in future works, such as by imposing high and low attention constraints only on the pairs of keypoint locations.

Revised: While, the current approach still has not yet beaten the CNN-based bottom-up state-of-the-art counterparts developed with many sophisticated designs, we believe it is promising to exploit or supervise self-attention to solve the detection and association problems in multi-person pose estimation, and other tasks or applications.

Original: A A PPENDIX A. T RAINING DETAILS
Revised: A A PPENDIX A. T RAINING DETAILS

Original: In the training phase, we use data augmentation with random scale factor between 0.75 and 1.5, random flip with probability 0.5, random rotation with ± 30 degrees, and random translate with ±pixels along the horizontal and vertical directions.
Revised: In the training phase, we use data augmentation with random scale factor between 0.75 and 1.5, random flip with probability 0.5, random rotation with ± 30 degrees, and random translate with ±pixels along the horizontal and vertical directions.

Original: The input size is 640 2 or 800 2 and thus the input sequence length of Transformer is 1600 or 2500.
Revised: The input size is 640 2 or 800 2 and thus the input sequence length of Transformer is 1600 or 2500.

Original: We use the Post-Norm Transformer architecture and ReLU activation function in FFN.
Revised: We use the Post-Norm Transformer architecture and ReLU activation function in FFN.

Original: We supervise the 4-th self-attention layer by default (the total layer depth is 6).
Revised: We supervise the 4-th self-attention layer by default (the total layer depth is 6).

Original: By convention, we use 2 transposed convolution layers to upsample the
Revised: By convention, we use 2 transposed convolution layers to upsample the

Original: Transformer output size to 160 × 160 or 200 × 200.
Revised: Transformer output size to 160 × 160 or 200 × 200.

Original: The standard deviation for the Gaussian kernel in the generated heatmaps is set to 2.
Revised: The standard deviation for the Gaussian kernel in the generated heatmaps is set to 2.

Original: We use Adam optimizer to train the model.
Revised: We use Adam optimizer to train the model.

Original: The model is distributed across 8 Tesla V100 GPUs with a total batchsize of 16 or 8.
Revised: The model is distributed across 8 Tesla V100 GPUs with a total batchsize of 16 or 8.

Original: The initial learning rate is set to batchsize8 × 0 . 0001 , and decays 10 times at the 150-th and 200-th epochs respectively, with a total of 240 training epochs.
Revised: The initial learning rate is set to batchsize8 × 0 . 0001 , and decays 10 times at the 150-th and 200-th epochs respectively, with a total of 240 training epochs.

Original: A. I NFERENCE DETAILS
Revised: A. I NFERENCE DETAILS

Original: The threshold score t for obtaining candidate keypoint from heatmaps is set to 0.0025.
Revised: The threshold score t for obtaining candidate keypoint from heatmaps is set to 0.0025.

Original: The final person masks are achieved by bilinear interploating the estimated ˆ M m to the original image size.
Revised: The final person masks are achieved by bilinear interploating the estimated ˆ M m to the original image size.

Original: The skeleton kinematic tree used in the body-first grouping is defined as a graph structure: the vertices are all types of keypoints that are denoted as the numbers from 0 to 16 by the order defined by COCO dataset; the edges are defined as [(0, 1), (0, 2), (0, 3), (0, 4), (3, 5), (4, 6), (5, 7), (5, 11), (6, 8), (6, 12), (7, 9), (8, 10), (11, 13), (13, 15), (12, 14), (14, 16), (5, 6), (15, 16), (13, 14), (11, 12)].
Revised: The skeleton kinematic tree used in the body-first grouping is defined as a graph structure: the vertices are all types of keypoints that are denoted as the numbers from 0 to 16 by the order defined by COCO dataset; the edges are defined as [(0, 1), (0, 2), (0, 3), (0, 4), (3, 5), (4, 6), (5, 7), (5, 11), (6, 8), (6, 12), (7, 9), (8, 10), (11, 13), (13, 15), (12, 14), (14, 16), (5, 6), (15, 16), (13, 14), (11, 12)].

Original: A. A BLATION ON WHICH ATTENTION LAYERS SHOULD BE SUPERVISED
Revised: A. A BLATION ON WHICH ATTENTION LAYERS SHOULD BE SUPERVISED

Original: Supervising the self-attention matrix in different Transformer layer depths may have different effects on the heatmap and mask learning.
Revised: Supervising the self-attention matrix in different Transformer layer depths may have different effects on the heatmap and mask learning.

Original: To study such effects, we train a smaller proxy model to compare their differences in the fitting of heatmap and instance mask loss on a small subset (1/5) split of the COCO train set.
Revised: To study such effects, we train a smaller proxy model to compare their differences in the fitting of heatmap and instance mask loss on a small subset (1/5) split of the COCO train set.

Original: The model configurations are: ResNet-50 based, 576 2 input resolution andtransformer layers with d = 160 and 320 hidden dimensions in FFN.
Revised: The model configurations are: ResNet-50 based, 576 2 input resolution andtransformer layers with d = 160 and 320 hidden dimensions in FFN.

Original: Note that using a smaller model and small-scale training data inevitably reduces the overall performances of the model
Revised: Note that using a smaller model and small-scale training data inevitably reduces the overall performances of the model

Original: [Figure]
Revised: [Figure]

Original: but we only aim to find the relative differences in supervising at different Transformer layers.
Revised: but we only aim to find the relative differences in supervising at different Transformer layers.

Original: As illustrated in
Revised: As illustrated in

Original: Figure 7, we do not observe significant differences in both heatmap loss and instance mask loss when leveraging the mask supervision in different Transformer layers.
Revised: Figure 6, we do not observe significant differences in both heatmap loss and instance mask loss when leveraging the mask supervision in different Transformer layers.

Original: We further evaluate all these models on the COCO validation set.
Revised: We further evaluate all these models on the COCO validation set.

Original: As shown in Table 4, supervising one of the last three attention layers achieves better performance compared with supervising the first two layers.
Revised: As shown in Table 4, supervising one of the last three attention layers achieves better performance compared with supervising the first two layers.

Original: Especially, supervising the penultimate or third-to-last layer shows a better performance.
Revised: Especially, supervising the penultimate or third-to-last layer shows a better performance.

Original: This suggests that leveraging the instance mask loss in this layer depth is a better trade-off between heatmap learning and mask learning.
Revised: This suggests that leveraging the instance mask loss in this layer depth is a better trade-off between heatmap learning and mask learning.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: A. W ILL AN INDEPENDENT SELF - ATTENTION HEAD BE BETTER THAN A SHARED ONE TO LEVERAGE THE INSTANCE MASK LOSS ?
Revised: A. W ILL AN INDEPENDENT SELF - ATTENTION HEAD BE BETTER THAN A SHARED ONE TO LEVERAGE THE INSTANCE MASK LOSS ?

Original: Intuitively, using an independent self-attention head may be helpful to reduce the effect of introducing an intermediate instance mask loss on the standard Transformer forward.
Revised: Intuitively, using an independent self-attention head may be helpful to reduce the effect of introducing an intermediate instance mask loss on the standard Transformer forward.

Original: Thus we try to mitigate the negative effect on the heatmap localization by using an independent self-attention head to leverage the mask supervision.
Revised: Thus we try to mitigate the negative effect on the heatmap localization by using an independent self-attention head to leverage the mask supervision.

Original: This design will need to insert an extra self-attention layer to the transformer intermediate output, as shown in Figure 8.
Revised: This design will need to insert an extra self-attention layer to the transformer intermediate output, as shown in Figure 7.

Original: However, by comparing the convergence of the training losses, we find no obvious difference in the heatmap loss fitting between using shared self-attention attention and independent self-attention, while, the independent self-attention performs relatively better in fitting the instance mask loss.
Revised: However, by comparing the convergence of the training losses, we find no obvious difference in the heatmap loss fitting between using shared self-attention attention and independent self-attention, while, the independent self-attention performs relatively better in fitting the instance mask loss.

Original: When we test their performances on COCO validation set, we find both designs achieve similar performances, as shown in Table 5.
Revised: When we test their performances on COCO validation set, we find both designs achieve similar performances, as shown in Table 5.

Original: Such results indicate that using an independent layer to the intermediate loss bring little gain, and introducing an intermediate instance mask loss may generate a weak effect on the prediction of keypoint heatmaps.
Revised: Such results indicate that using an independent layer to the intermediate loss bring little gain, and introducing an intermediate instance mask loss may generate a weak effect on the prediction of keypoint heatmaps.

Original: We conjecture that the existence of the residual path parallel to the supervised self-attention layer may also adaptively reduce the effect of the instance mask loss on the subsequent transformer layers, since we only leverage the sparse constraints to the self-attention matrix in a certain transformer layer
Revised: We conjecture that the existence of the residual path parallel to the supervised self-attention layer may also adaptively reduce the effect of the instance mask loss on the subsequent transformer layers, since we only leverage the sparse constraints to the self-attention matrix in a certain transformer layer

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: A. R UNTIME AND COMPLEXITY ANALYSIS
Revised: A. R UNTIME AND COMPLEXITY ANALYSIS

Original: We take the ResNet-101 based model as the exemplar to test two types of grouping algorithm.
Revised: We take the ResNet-101 based model as the exemplar to test two types of grouping algorithm.

Original: We use the total 5000 images from COCO validation set.
Revised: We use the total 5000 images from COCO validation set.

Original: For each image, we run the model forward on a single GPU and the grouping algorithm on the CPU.
Revised: For each image, we run the model forward on a single GPU and the grouping algorithm on the CPU 4 , where the grouping runtime is far less than the model forward.

Original: In Table 6, we provide a controlled study to compare their differences in the model performance, theoretical complexity for per part assignment, runtime for the whole inference pipeline (keypoint detection, grouping and instance segmentation).
Revised: In Table 6, we provide a controlled study to compare their differences in the model performance, theoretical complexity for per part assignment, runtime for the whole inference pipeline (keypoint detection, grouping and instance segmentation).

Original: Note that the complexity is a theoretical analysis based on the assumption that there are N existing skeletons and N candidates for a certain part type.
Revised: Note that the complexity is a theoretical analysis based on the assumption that there are N existing skeletons and N candidates for a certain part type.

Original: We report the performances and runtime for the pure bottom-up result and the ones with refinement.
Revised: We report the performances and runtime for the pure bottom-up result and the ones with refinement.

Original: [Table]
Revised: [Table]

Original: In Table 7 we compare our models with the mainstream bottom-up models, in terms of the number of model parameters and computational complexity of the model forward pass.
Revised: In Table 7 we compare our models with the mainstream bottom-up models, in terms of the number of model parameters and computational complexity of the model forward pass.

Original: The results of Hourglass (Newell et al., 2017), PersonLab (Papandreou et al., 2018), and HigherHRNet (Cheng et al., 2020) are taken from the HigherNet paper (Cheng et al., 2020).
Revised: The results of Hourglass (Newell et al., 2017), PersonLab (Papandreou et al., 2018), and HigherHRNet (Cheng et al., 2020) are taken from the HigherNet paper (Cheng et al., 2020).

Original: We can see that compared with them, our models have fewer parameters and less computational complexity in the model forward pass.
Revised: We can see that compared with them, our models have fewer parameters and less computational complexity in the model forward pass.

Original: [Table]
Revised: [Table]

Revised: A. V ISUALIZATION FOR HUMAN SKELETONS , INSTANCE MASKS AND KEYPOINT ATTENTION AREAS .

Revised: In Figure

Revised: [Table]

Revised: we visualize the qualitative results predicted by our pure bottom model based on ResNet152 and 512 2 input resolution.

Revised: Our model still can perform relatively well even in some hard cases, such as occluded persons and crowded scene with the existence of a large number of people ( > 30) (shown in the 4-th row in Figure 9).

Revised: We also can see that the attention areas of the sampled keypoints belonging to a specific person can accurately and reasonably attend to the target person and not attend to the areas excluding the person.