  Deep learning has shown a great potential for speech separation, especially
for speech and non-speech separation. However, it encounters permutation
problem for multi-speaker separation where both target and interference are
speech. Permutation Invariant training (PIT) was proposed to solve this problem
by permuting the order of the multiple speakers. Another way is to use an
anchor speech, a short speech of the target speaker, to model the speaker
identity. In this paper, we propose a simple strategy to train a long
short-term memory (LSTM) model to solve the permutation problem in speaker
separation. Specifically, we insert a short speech of target speaker at the
beginning of a mixture as guide information. So, the first appearing speaker is
defined as the target. Due to the powerful capability on sequence modeling,
LSTM can use its memory cells to track and separate target speech from
interfering speech. Experimental results show that the proposed training
strategy is effective for speaker separation.
