Paper: SYRAC: Synthesize, Rank, and Count (ICLR-2024)

Reviewer comment (Novelty):
For the fully supervised part, the authors only discuss the density-based crowd counting methods. In other words, many localization-based methods should be discussed, making the related work more comprehensive.

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
1. **Counting vs. Detection:** We appreciate this insight. We would be happy to expand our discussion to emphasize how crowd counting is a different task than detection and thus can rely on different features, for two reasons: 1) Counting is a more relaxed task than detection in that the former does not require the localization of objects like the latter, 2) counting may leverage features that respond to multiple quantities of objects (not only to one object) in different spatial constellations, e.g. clusters of pedestrians, whereas in object detection, the features delineate a single object. Additionally, typical object detection datasets will have around 1 to 20 objects in an image. Crowd counting datasets tend to have an average of more than 100 pedestrians per image, and density based methods tend to outperform detection based methods in these scenarios.

Reviewer comment (Experiments):
The authors have pointed out that the prompt count is not reliable but using it as the GT count directly during the training phase. It makes me confused. I think it would be better to rank the generated 60 images using the pre-trained backbone first. Secondly, fine-tune the GT count according to the ranking results. Specifically, image A and image B are generated using the same prompt count 20. However, ranking results present that image A contains fewer persons than image B, so the GT count of image A could be fine-tuned to be smaller than the GT count of image B.

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
2. **Ranking Noisy Synthetic Counts:** The goal of our synthetic ranking pre-training strategy is to mitigate label noise by narrowing down the feature set to those specifically linked to crowd quantity before initiating fine-tuning with the noisy synthetic counting data. While we could pseudo-label the noisy synthetic data using the ranking network, the ranking network may itself be unreliable due to a domain shift between the synthetic ranking dataset and the noisy synthetic counting dataset.

Reviewer comment (Experiments):
I understand that the input of the generation process is complete images without cropping, but the inference process uses image patches as input. There may be resolution gaps. How about cropping the original images into patches in the generation process instead?

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
3. **Cropping before Synthesis:** We did run experiments where we cropped a portion of a source image and generated a synthetic ranking pair that way, similar to your idea. However, this performed poorly in practice. We suspect that was due to the way that pedestrians are distributed in crowd counting images, resulting in many crops that had no crowds in them. As an example, the perspective of an image might cause the top half of the image to contain only sky or buildings or other unrelated features while the bottom half contains the crowd. This setup would result in comparing a real image with no crowds to a synthetic image with no crowds, this potentially impacts feature learning.

Reviewer comment (Evaluation):
There is a lack of quantitative analysis about the reliability of the generation process. Specifically, the authors can sample n source images for generation and statistics on the percentage of images where the objects were successfully removed.

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
4. **Reliability of Ranking Labels:** We agree that it would be good to provide a more concrete analysis of the reliability of the synthetic ranking data. We manually inspected 50 ranking pairs from each dataset to estimate the accuracy. For ShanghaiTechB, there were no incorrectly ranked examples. For ShanghaiTechA there was 1 incorrectly ranked example. For QNRF there were 2 incorrectly ranked examples. And, for JHU++, there were 2 incorrectly ranked examples. Additionally, we use a pre-trained crowd counting model (DM-COUNT [1]) to estimate the crowd count for 500 real and synthetic ranking pairs to provide an additional estimate of the reliability of the synthetic ranking data. The estimated accuracy is 99.6% for ShanghaiTechB, 99.2% for ShanghaiTechA, 97.2% for QNRF, and 90.2% for JHU++. These additional experiments corroborate the idea that the synthetic ranking data is highly reliable.

Reviewer comment (Experiments):
The authors think the ranking information is reliable, and the prompt count is relatively unreliable. Thus, the authors pre-train the backbone using the ranking information and freeze the backbone during the training phase to resist the prompt count noise. I agree that the ranking information is more reliable. However, I am not sure it is necessary to fix the backbone as only fine-tuning the linear layer may limit the learning potential on the prompt count, which is considered ground truth. There could be an ablation study on fine-tuning the backbone during the training phase.

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
5. **Fine-tuning Backbone:** We agree that it would be useful to have an ablation study examining different strategies for tuning the pre-trained network with the noisy counting data. We did run experiments where we included the noisy synthetic counting data during pre-training with the synthetic ranking data. However, that did not achieve good results.

Reviewer comment (Novelty):
The motivation to synthesize the ranking crowd image is still unclear since one can utilize the existing datasets to generate the ranking image pairs, like CrowdCLIP.

Author response:
We thank the reviewer for their valued insights. We would like to address some of their concerns:
7. **Motivation for Synthetic Ranking:** We do utilize existing datasets to generate the ranking pairs, as those images are the source of the synthetic ranking data. Methods like CrowdCLIP generate ranking pairs by performing “intra-image ranking”. In Table 3 we demonstrate that intra-image ranking underperforms compared to our synthetic ranking strategy.

Reviewer comment (Experiments):
What is the rationale behind the setting of N, which is the crowd count to generate synthetic images? What is the quality of the generated images? Is it possible to provide a measure of variance to assess the feasibility of this method?

Author response:
1. **Quality & Settings for Noisy Synth Images:** To assess the quality + variance of our synthetic crowd counting image dataset, we've included a table below. The table reports the quality by using a fully-supervised model to estimate, for each synthetic image, the count and compares it to the desired count dictated by the prompt. The table reveals noise across count predictions for each count category. Despite this noise, the average predicted count closely aligns with the given count in the prompt.  
With regards to the count category selections in N, we used the following rationales for setting N =  {1, 5, 10, 20, 50, 100, 200}
    - **Exponential:** We select values from small to large in an approximately exponential fashion. We chose 200 as a maximum because we use patching to split the image into regions with smaller counts (<200) during inference.
    - **Computational Cost:** We avoid generating images from a more granular count range (i.e. 1, 2, 3, 4, …, 200) due to the increased computational cost this would incur. 
   - **Simplicity:** Our aim wasn't to optimize for specific prompts. While there might exist combinations of N that yield better performance, determining these without introducing validation data, thereby adding supervision, is challenging. We see this as a strength of our method, as the selection of N is based on simple heuristics, and yet it still works.

Reviewer comment (Experiments):
There are only six categories for N. Why not train the model by a classification task? In situations where the labels are not stable, the classification task seems to be able to maintain a relatively high level of accuracy.

Author response:
2. **Regression vs. Classification:** The count categories chosen for N = {1, 5, 10, 20, 50, 100, 200} represent a subset of counts among positive unbounded integers. Given the nature of our task, predicting counts intuitively aligns with regression rather than classification. We did consider different methods for fitting the pre-trained features to the noisy crowd counting data, including classification. We found that nothing performed better than linear regression. However, it’s possible that we did not explore this sufficiently and there may be a formulation where classification outperforms regression.

Reviewer comment (Experiments):
The synthetic images do not include images with 0 crowd count. Does this method have the capability to handle datasets that consist of a large portion of (background) images with no people, such as NWPU?

Author response:
3. **Zero Count Images:** The synthetic images do include images with 0 crowd count. We didn’t explicitly show this in Figure 1. However, we do explain our strategy for generating images with 0 crowd count in the last two sentences of section 3.2.

Reviewer comment (Evaluation):
How does the computational cost of generating synthetic images using the diffusion model compare to that of other unsupervised counting models?

Author response:
4. **Computational Cost:** The question of computational cost is an important one, as generating synthetic data does incur an additional computational cost beyond the burden of training the network. We generated all synthetic data on a single GPU, and we also trained our models on a single GPU.

Reviewer comment (Experiments):
Figure 6 illustrates that the features exhibit an underlying crowd-count-based ordering. However, it would be more convincing if features from supervised counting models could be provided for comparison.

Author response:
6. **Comparison to Full-Supervised Features:** This is a great point, thank you! We included Figure 6 to corroborate the idea that the pre-training features capture a quantity based distribution. However, it would be useful to further explore that distribution and compare it to features learned by different methods with different supervision budgets. We expect that more supervision would lead to similar but better organization of the latent space with regards to object count.

Reviewer comment (Evaluation):
Does the ranking loss merely train the model to distinguish between real and synthetic images?

Author response:
8. **Real vs. Synth Spurious Correlations:** This is a critical question. Deep learning performance normally suffers due to spurious correlations, and so we expect that our method may be prone to these spurious correlations, as well. However, we rely on the following evidence to suggest that our method produces a network which learns something coherent beyond the separation of real from synthetic:
    - Our method outperforms other unsupervised methods, and our pre-training strategy outperforms other pre-training strategies.
    - Figure 6 demonstrates a quantity based distribution.
    - There is significant domain overlap between the synthetic and real distributions, as the synthetic images are generated using real images as a source. We provide a measure of this domain overlap in the supp. materials.
    - In sec 5, we mention the use of a non-negativity constraint in the final layer of the network (+ ReLU activation), which prevents the weights and output from becoming negative. Thus, we expect that it will be harder for the network to exploit spurious correlations, as it cannot set the output of the network to be negative when synthetic features are present.

That said, we could explore ways to further ensure that we mitigate learning features that distinguish real from synthetic images by producing ranked pairs of images where both images in each pair are synthetic (e.g. remove objects then remove some more; or by using SD to synthesize images then removing objects from those synthetic image), or by using domain adversarial learning.

Reviewer comment (Presentation):
Could the author provide a more comprehensive explanation for Figure 6? Both small and large counts are distributed throughout the entire space in the QNRF dataset. It is difficult to interpret the UMAP results without any explanation.

Author response:
**Explanation for Figure 6.:** Figure 6 illustrates the UMAP representation of the training images based on the pre-trained synthetic ranking features, with points color-coded according to the logarithm of the ground truth count. While interpreting dimensionality reduction results can be complex, we included this figure to demonstrate that there is a coherent quantity-based structure to the latent space learned during the pre-training step. We can interpret this qualitatively by noticing clustering of similarly colored points, which suggests that images with similar counts tend to have similar features.

Reviewer comment (Novelty):
What is the advantage of generating synthetic data via stable diffusion, especially when compared with the large synthetic dataset GCC[1]? Both approaches are label-free, but GCC contains more detailed count and localization information. Additionally, [1] achieves better counting performance than the proposed method when there are no human-labeled annotations. It would be helpful to clarify the specific advantages of this paper.

Author response:
**GCC vs. SYRAC:** There are several advantages to using synthetic data generated by Stable Diffusion even when GCC already exists.
    - **Labor Burden:** producing 3D synthetic counting data is not labor free, despite not requiring manual annotations. It requires acquiring new 3D assets for each new object category whereas our method can extend to any novel category that is recognized by Stable Diffusion. And, it requires designing complex 3D scenes as well as environmental conditions such as rain or limited light. Our approach does not carry this burden.
     - **Domain Shift:** GCC exhibits a significant domain shift when compared to real crowd counting datasets, given that it is produced from 3D assets. Our method maintains a smaller domain shift by using real images as a seed for the synthetic ranking images and also due to the fact that Stable Diffusion is trained using natural images. We have added details to the supplementary material where we provide a measure of domain overlap between GCC and the synthetic ranking datasets, which demonstrates that our synthetic ranking data is closer to the real distribution than the GCC dataset.
     - **Different Requirements:** While we currently underperform when compared to GCC based methods, we are also working on a fundamentally different problem. GCC requires human labor to produce, whereas our method does not. GCC provides density map annotations, whereas ours does not. Finally, methods that utilize GCC typically focus on domain adaptation due to the large domain gap between GCC and real datasets, whereas our method produces data that is already close to the true distribution.    
Given these reasons, we believe our paper serves as a valuable contribution to the research community

Reviewer comment (Evaluation):
The impact of patch size is only presented in the table. Could the authors provide a deeper analysis and discussion on the reasons for the observation that different patch sizes lead to different performance?

Author response:
**Patch Size:** To understand the impact of patch size, we consider two points:
     - **Label Noise:** There is significant noise in the true underlying count for synthetic crowd counting data produced by Stable Diffusion. This noise increases as the expected count embedded within the prompt increases (see table below). Thus, we expect that prediction noise will be concentrated around images with larger underlying counts. To counter this, we adopt patching during inference, aiming to reduce the object count within each patch. Our findings (Table 2) indicate that increased patching benefits higher count datasets, improving performance by mitigating noise. However, this strategy may compromise images with smaller crowd counts, as larger objects get fragmented across patches. This was apparent in a performance drop among those images.
     - **Limited Representation for Dense Crowds:** The synthetic ranking generation process often removes many objects in dense crowds, resulting in limited ranking examples for densely crowded images. Consequently, our model might struggle to learn features specific to highly dense crowds. Splitting these images into smaller patches during inference may align them with a distribution where the model performs more effectively.