  Speaker-conditioned target speaker extraction systems rely on auxiliary
information about the target speaker to extract the target speaker signal from
a mixture of multiple speakers. Typically, a deep neural network is applied to
isolate the relevant target speaker characteristics. In this paper, we focus on
a single-channel target speaker extraction system based on a CNN-LSTM separator
network and a speaker embedder network requiring reference speech of the target
speaker. In the LSTM layer of the separator network, we propose to customize
the LSTM cells in order to only remember the specific voice patterns
corresponding to the target speaker by modifying the information processing in
the forget gate. Experimental results for two-speaker mixtures using the
Librispeech dataset show that this customization significantly improves the
target speaker extraction performance compared to using standard LSTM cells.
