  Gram-based and patch-based approaches are two important research lines of
image style transfer. Recent diversified Gram-based methods have been able to
produce multiple and diverse reasonable solutions for the same content and
style inputs. However, as another popular research interest, the diversity of
patch-based methods remains challenging due to the stereotyped style swapping
process based on nearest patch matching. To resolve this dilemma, in this
paper, we dive into the core style swapping process of patch-based style
transfer and explore possible ways to diversify it. What stands out is an
operation called shifted style normalization (SSN), the most effective and
efficient way to empower existing patch-based methods to generate diverse
results for arbitrary styles. The key insight is to use an important intuition
that neural patches with higher activation values could contribute more to
diversity. Theoretical analyses and extensive experiments are conducted to
demonstrate the effectiveness of our method, and compared with other possible
options and state-of-the-art algorithms, it shows remarkable superiority in
both diversity and efficiency.
