Paper: Out of Sight: A Framework for Egocentric Active Speaker Detection (ICLR-2024)

Reviewer comment (Evaluation):
In Table 1, the authors report vASD mAP scores on the Ego4D’s validation set. The problem is that they report mAP@0.5 (which is different from mAP) for previous methods when they report mAP only for their method. When we compute the mAP score, we only use the ground-truth face bounding boxes. However, computing mAP@0.5 involves comparing IoU between the face bounding-box detections and the ground-truth, therefore mAP@0.5 is estimated much lower than mAP.

Author response:
Since the review bases his/her entire rejection argument in the use of the mAP metric and the comparison against SPELL and  STHG, we refer him/her to the global replies titled 'Comparison against ArXiv Submisions' and 'mAP Metric'

Reviewer comment (Evaluation):
The authors didn’t report the vASD mAP of Min et al. Min (2022) although they report eASD mAP of it. This kind of partial reporting might make their method look more powerful, but doesn’t seem appropriate.

Author response:
Since the review bases his/her entire rejection argument in the use of the mAP metric and the comparison against SPELL and  STHG, we refer him/her to the global replies titled 'Comparison against ArXiv Submisions' and 'mAP Metric'

Reviewer comment (Experiments):
SPELL (2022) and STHG (2023) achieve 71.3% vASD mAP and 75.7% vASD mAP on Ego4D’s validation set, respectively, which significantly outperform the proposed method. Furthermore, STHG (2023) achieves 85.6% eASD mAP, which also outperforms the proposed method in this paper. The authors should refer to the challenge reports and recognize them, as recommended by the Ego4D organizers.

Author response:
Since the review bases his/her entire rejection argument in the use of the mAP metric and the comparison against SPELL and  STHG, we refer him/her to the global replies titled 'Comparison against ArXiv Submisions' and 'mAP Metric'

Reviewer comment (Novelty):
The performance exceeds existing works, but the gap between the proposed method and LoCoNet is not too significant, given that this paper is specifically tailored to this dataset.

Author response:
While LoCoNet has a close performance in the vASD task we highlight that Out-of-Sight has a much simpler architecture (we dont model inter-speaker or long-term single speaker relationships), despite the simple architecture, direct and accurate estimates for the egoASD and vASD emerge from the direct multi-modal modeling without using the spatial and temporal attention schemes proposed in LoCoNet. In a single end-to-end trainable model Out-of-Sight can model both the egoASD and vASD. 
For further comparison, LoCoNet requires an input tensor of up to 200 frames to achieve optimal performance, meanwhile O2S achieves improved results covering only 35 frames.

Reviewer comment (Experiments):
The methods used such as "face position" and "weighted loss paper" are the sources of most of the improvement compared to LoCoNet, but these tricks might be overfitting to the biases that exist specifically in this dataset. Does the method still generalise to existing non-egocentric ASD datasets, such as AVA-ASD or ASW?

Author response:
Certainly O2S learns from  global spatial patterns found in Egocentric video (as shown by the improved performance using the face location). As outlined in the paper this correlates to the camera wearer’s  gaze. However,  using the face positions is not exclusive to O2S or Egocentric video. Zhang et al. have already demonstrated the usefulness of this information source in commercial movies (where being at the center of the scenes correlates with directing the attention of the audience to an important character). 
We see this additional information as common prior in the ASD task, regardless of the setup, egocentric or commercial movies. In the egocentric domain, we are the first to show the usefulness of this spatial prior.

Reviewer comment (Writing):
I am not sure if "active speaker detection" is an appropriate term for the overall task. vASD in this paper is usually called ASD in other literature, and the ASD usually does not encompass what is called eASD in this paper. A similar work (Jiang et al., 2022) does not refer to this task as ASD.

Author response:
We diverted slightly from the standard terminology trying to emphasize that both tasks have a strong correlation even when part of the cross-modal modeling does not have a direct visual support. Our transformer-encoder decoder represents a straightforward way to model both. In the final version we will follow a more standard naming of  the task, but still emphasize on the connection between them.

Reviewer comment (Novelty):
Regardless of the term used, I am not sure if the proposed combined problem (eASD+vASD) is useful, in between vASD and AV speaker diarisation. The camera wearer is a specific identity, whereas we do not consider the identity information of the visible speakers.

Author response:
We must note that ASD and speaker diarization can be two independent tasks, for the case of audio only diarization. Certainly the AudioVIsual diarization and improved ASD pipeline task benefits the diarization taks. We see O2S as a method that can bridge the gap in both domains. It can directly estimate if the speech event must be attributed to visible speakers (thus an AV diarization method could be used for improved performance) or if the event must be attributed to unseen speech (thus audio only diarization must be used).

Reviewer comment (Writing):
Is "at last 4.41%" at the end of abstract is the intended expression?

Author response:
We thank the reviewer for the careful reading of our paper, we will fix the typos and reference errors in the final version.

Reviewer comment (Writing):
"fine grain..." in the introduction should be "fine-grained..."

Author response:
We thank the reviewer for the careful reading of our paper, we will fix the typos and reference errors in the final version.

Reviewer comment (Experiments):
Although the whole pipeline is reasonable, it is complicated. Does it need to first use a face detector to detect faces? In the main architecture of O2S, there are CNNs for encoding video and audio, and then there are transformers for mixing video and audio and decoding them. Why not encode and mix video and audio in just a single transformer? In my view, all the three stages can be simplified in one single transformer in principle.

Author response:
We politely direct the reviewert to the general discussion regarding the mAP metric to clarify the use of face detections in the validation set. 

We also clarify that, while we welcome the reviewer's suggestion, we must also bring into consideration that we can not simply  plug a multimodal transformer into the proposed head, we explain in detail:

First, Since vASD predictions are made over individual face-crops. We still have to figure out a way to model features for each individual face-crop. Since the number of faces per frame can vary, we must resort either to simultaneous transformer forward passes (as is the case in O2S, but with CNN architectures), or a series of special tokens  delimiting the different faces and the individual face crops on each.

Second,  it must be empirically determined if these facial features should be mixed prior to the encoder-decoder head proposed in O2S or earlier in the visual stream. This constitutes an additional hyper-parameter that should be explored. If it is determined that no attention is required between the individual face crops, we regress to the same scenario of O2S where  the CNN is replaced  with a visual transformer.
 
Third, as we integrate the location data  of the face crops, an even more complex input stream for the visual transformer must be built. One might think about simply inputting the entire frame and encoding some ROIs corresponding to the faces. This will clearly incrates the memory allocation for the proposed network, and might be impractical as a portion of the visual objects on the frame have no direct relation with the task. For example the visual feature of a supermarket will not be truly useful in identifying the actual source of a speech event.

Fourth, some of the structure proposed in the encoder-decoder transformer must remain to generate predictions which are temporally consistent and directly correspond to the input faces, in other words we can not simply mix all the data without any strategy to generate individual predictions over the inputs. Part of the complexity introduced in O2S is due to the requirement of directly supervising each token in the output with its corresponding ground-truth, if these correspondence is lost, the network supervision becomes significantly more challenging 

Certainly we share the same opinion of the reviewer regarding the versatility of transformers architectures for ASD tasks, but we also consider that it also brings extra challenges in a problem where visual and audio streams must be carefully aligned. We think this suggestion aligns better with the future work.

Reviewer comment (Experiments):
"In other words, the active speaker is more likely to appear near the center of the frame" I agree in most cases the active speaker appear around the center. But in some cases, the camerawearer may not look at the speaker or only turn eyeballs to look at the speaker. It may be more accurate to incorporate eye gaze location in the Visual Token Representation. After all, the XR device should already detected eye gaze.

Author response:
We fully agree that the camera Field of View is not a direct estimation of the person's gaze, but rather a correlated variable. As outlined in the paper this empirical improvement of explicitly locating the face crops has already been explored  in datasets composed of  commercial movies.
We see this additional information as common prior in the ASD task, regardless of the setup, egocentric or commercial movies. In the egocentric domain, we are the first to show the usefulness of this spatial prior.