Paper: VideoClusterNet: Self-Supervised and Adaptive Face Clustering for Videos (ICLR-2024)

Reviewer comment (Experiments):
Compared to related works that use joint face ID model adaptation and clustering, the uniqueness of this work is not stated clearly. Is the performance increase comes from better face ID model adpatation or just the following clustering method? If better face ID model, is it because of the teacher-student branch method used in this work or other perspectives?

Author response:
1. **Clarification on method novelty:** We kindly refer the reviewer to the “Joint Representation and Clustering” paragraph in Section 2 regarding the comparison of our proposed method with closely related methods. We believe that our method’s performance increase is a result of both the self-supervised (SSL) model finetuning and the final clustering modules. Empirical evidence for this fact can be observed in Section 5 (Ablation Analysis). Specifically, Table 3 and Table 4 prove the individual effectiveness of our model finetuning mechanism and clustering algorithm, respectively. Unlike closely related prior work, the model adaptation stage is more effective majorly due to the self-distillation procedure and its sole dependence on positive match pairs. Our final clustering algorithm directly benefits from the model finetuning stage by incorporating the SSL training objective as a distance metric. Such a learned metric helps boost the algorithm’s performance as it efficiently evaluates inter-track distances directly in an embedding space, which is optimized for reduction in the same identity distances. Also, any sub-optimal performance that may be induced due to any user-defined matching thresholds is avoided through the adaptive custom threshold computed for each track. Such a thresholding mechanism helps the clustering algorithm automatically adjust to how effectively the model can match a given track’s identity across a true positive match pair. We will update our paper with this detailed explanation for the reader’s enhanced understanding of our method’s salient novelty.

Reviewer comment (Novelty):
Why we need a new benchmark? The existing ones not challenging enough for practical use? Why? Any quantitative/qualitative comparison among the porposed one and existing ones? The statement should be put clearly in the paper.

Author response:
2. **Need for new benchmark dataset:** We kindly refer the reviewer to the second last paragraph in Section 1 (Introduction) where we detail a few primary reasons for the need for a comprehensive movie benchmark dataset for video face clustering. To justify it further, the problem we address in the paper, and its application for TV series/movies has seen comparatively lesser interest in the last few decades as opposed to other Computer Vision related tasks. Also, complicated usage and copyrights for TV series/movies meant that providing open source datasets based on them wasn’t feasible or otherwise costly and time-consuming w.r.t. providing annotations. As a result, the majority of related approaches in this domain end up evaluating their method on a very specific TV series/movie dataset while not being able to provide means to access it publicly. A direct consequence is thus not being able to have fair comparisons with prior works, forcing authors to evaluate across a large set of niche datasets. We present our dataset with this view in mind and sincerely hope that it will largely benefit the research community in this niche domain.  
Finally, we refer the reviewer to our response to the first reviewer (**RNxJ**)’s concern regarding dataset analysis and comparison. We also refer the reviewer to Appendix J, which summarizes the uniqueness and advantages of our proposed dataset over existing ones.

Reviewer comment (Experiments):
The paper does not provide enough information about the proposed dataset, particularly in terms of presenting its uniqueness and advantages. There is no visualizations or statistical analyses to demonstrate distinctions from existing datasets. Furthermore, the description of dataset annotations is unclear. It remains uncertain whether the term "Varying Parameter" mentioned in Figure 1 is associated with dataset annotations.

Author response:
In addition to the details on our proposed dataset presented in Section 4.3 and Appendix E, we have updated our paper to provide a comprehensive analysis and comparison of our dataset with existing literature in Appendix F. This includes a comparison of some key dataset attributes like unique character and track count, cast racial diversity, and average track face quality score (computed using our face quality estimation module presented in Section 3.6). We also present a detailed histogram comparison of the dataset face quality distribution to give more insights into each dataset’s average quality score. In addition, we provide dataset histogram comparisons across critical attributes, specifically scene lighting and facial blur. We are confident that this revision further strengthens our argument of providing a challenging, more diverse movie face clustering dataset. We sincerely thank the reviewer for his comments on this specific aspect of our paper. Also, in Figure 1 the term “varying parameter” denotes the dominant image attributes that are significant outliers for a given face crop. It isn’t part of the available dataset annotations and is used simply for enhanced reader understanding. We have added this clarification in the updated paper version in Appendix J.

Reviewer comment (Novelty):
The authors did not compare their dataset with existing movie person identification (PI) datasets, such as MovieNet [1], which includes annotations suitable for PI tasks.

Author response:
Even though MovieNet dataset [1] comprises a wide variety of annotations on a large number of mainstream movies, we would like to kindly point out that its annotations aren’t directly useful for our task of video face clustering. It unfortunately consists of person bounding boxes (covering the entire character's body) and not face bounding boxes, which are essential for our task. Furthermore, complex post processing could be used to detect face bounding boxes in each dataset frame and match them against the ground truth person boxes, in order to associate the identity of each face. For this, the movie frame data would be essential to predict face boxes. To the best of our knowledge, neither the source movie dataset nor the links to the respective movies are available for download from https://movienet.github.io/. Due to these reasons, we strongly believe that a dataset comparison with MovieNet in its current state wouldn’t be fair for our intended task as it doesn’t contain relevant annotations for the same. We would however like to thank the reviewer for mentioning this specific dataset as it could potentially be relevant in a future extension of our work, wherein the video clustering is applied for person bounding boxes rather than just face boxes. We have updated our paper to acknowledge the same in the “Limitations and Future Works” paragraph in Section 5.

Reviewer comment (Experiments):
While the paper claims not to require pre-defined parameters, fixed values are still set in the quality assessment and coarse track matching modules to generate adaptive thresholds. However, these fixed values lack empirical or theoretical support.

Author response:
We would like to highlight here that the track quality estimation module (see Section 3.6) can be viewed as a pre-processing module for our final clustering algorithm (see Section 3.7). It is highly effective in filtering out non-identifiable face tracks that are often present in mainstream media content. Also, the coarse track matching module (see Section 3.5) is used in conjunction with the model finetuning module (see Section 3.4). We do acknowledge that both of the referenced modules contain minor empirically evaluated parameter values. However, we claim in our listed contributions (fourth contribution in the Introduction) that we specifically have a parameter-free video face clustering algorithm. Both of the aforementioned modules are essentially external to our core clustering algorithm. We would like to confidently reiterate that the core clustering algorithm is fully automated in the true sense. We would also like to provide clarification on specific values set for certain parameters in both referenced modules. For Equation 4 in the main paper, backed by empirical evidence, the value of 2.7 was first loosely set by fitting a Gaussian distribution onto the given set of track face quality scores. To select/filter the lower ~1.5% outliers, which are often less than the threshold of (mean - 2.8*std) in a Gaussian distribution, we sampled values in the range of 2.6 to 3.0 and empirically found that 2.7 worked optimally for our test set consisting of a wide range of movies. Regarding the coarse matching module, the motivation to select crop pdf values in the lower 25% range was again based on empirical evidence. To effectively select the true outliers, which would result in a more accurate computation of a given track’s custom match threshold, we experimented with values ranging from 5% to 40%. We found out that setting it to 25% provides effective true positive track matches while avoiding any false positive matches (which is a critical performance parameter) over a wide range of movie track sets. We will update this in the future revised version accordingly.

Reviewer comment (Evaluation):
Regarding the t-SNE embedding visual comparison on the MovieFaceCluster dataset in Figure 5, the visualizations generated by the proposed method seem to closely approximate the ground truth, which exhibits some anomalous clusters. Authors may explore the limitations of proposed method and present failure cases so as to provide more in-depth insights into video face clustering.

Author response:
We believe that the majority of the anomalous clusters visible in Figure 5 for the ground truth visualization are a combination of two main factors. First, there exist certain sub-optimalities in the t-sne embedding method [2] used to find a two-dimensional visualization sub-space. That partly results in the appearance of those outlier clusters due to failure in finding the optimal subspace to have ground truth clusters in a more compact representation. Second, a few of the outlier clusters seen as part of the cluster sets: cluster_1, cluster_17, cluster_5 consist of tracks possessing higher than average difficulty for face clustering. As a result, these tracks appear distant from their respective cluster centers. We would like to highlight here that our method still accurately manages to assign such outlier clusters to their correct ground truth cluster even though they provide a harder face clustering challenge. We kindly refer the reviewer to the “Limitations and Future Works” paragraph in Section 5 of our paper, where we highlight in detail the core limitations of our work and specify future directions to improve on them.

Reviewer comment (Experiments):
Most of the methods used for comparison are outdated, with only one introduced within the last three years.

Author response:
There are two major selection criteria we used to choose the methods to benchmark our proposed method’s performance in Table 1. First, methods having results reported on The Big Bang Theory (BBT) and Buffy the Vampire Slayer (BVS) datasets were selected as we noticed a major overlap in the use of these two datasets for evaluation in the literature. Certain methods were omitted based on the fact that they only reported results on certain rare TV series datasets that weren’t publicly available for evaluation and/or didn’t have necessary usage rights. Secondly, methods that reported results on publicly available versions of BBT and BVS datasets were selected. For a fair comparison, all methods need to have a constant face track pre-processing module (details in Section 3.2) to ensure that clustering performance is evaluated on the same set of face tracks. Methods that did not satisfy these criterias also, unfortunately, didn’t provide publicly available implementations of their work, which prevented us from re-evaluating these methods on the available input data. For the same reason, we decided to re-implement certain closely related works ourselves for a fair comparison in Table 2. We remark that to the best of our knowledge, Aggarwal et al. (2022)[3] is the only paper published within the last 3 years that we haven’t reported in Table 1. It unfortunately failed to meet the second aforementioned criterion. We sincerely hope that this clarification helps the reviewer provide more insights into our evaluation methods selection criteria shown in Table 1. We would be open to any paper suggestions from the reviewer that may be closely related to our work that we might have missed.

Reviewer comment (Presentation):
Please ensure the consistency of citation formats. For instance, there is an inconsistency in the citations of Table 1 in Section 4.1, where both 'Table 1' and 'Tab. 1' are used.

Author response:
We would like to thank the reviewer for mentioning this minor issue. We have resolved it in the updated paper.