  In this paper, we address the problem of separating individual speech signals
from videos using audio-visual neural processing. Most conventional approaches
utilize frame-wise matching criteria to extract shared information between
co-occurring audio and video. Thus, their performance heavily depends on the
accuracy of audio-visual synchronization and the effectiveness of their
representations. To overcome the frame discontinuity problem between two
modalities due to transmission delay mismatch or jitter, we propose a
cross-modal affinity network (CaffNet) that learns global correspondence as
well as locally-varying affinities between audio and visual streams. Given that
the global term provides stability over a temporal sequence at the
utterance-level, this resolves the label permutation problem characterized by
inconsistent assignments. By extending the proposed cross-modal affinity on the
complex network, we further improve the separation performance in the complex
spectral domain. Experimental results verify that the proposed methods
outperform conventional ones on various datasets, demonstrating their
advantages in real-world scenarios.
