Paper:
The expressive power of end-to-end automatic speech recognition (ASR) systems enables direct estimation of the character or word label sequence from a sequence of acoustic features. Direct optimization of the whole system is advantageous because it not only eliminates the internal linkage necessary for hybrid systems, but also extends the scope of potential application use cases by training the model for multiple objectives. Several multi-lingual ASR systems were recently proposed based on a monolithic neural network architecture without language-dependent modules, showing that modeling of multiple languages is well within the capabilities of an end-to-end framework. There has also been growing interest in multispeaker speech recognition, which enables generation of multiple label sequences from single-channel mixed speech. In particular, a multi-speaker end-to-end ASR system that can directly model one-to-many mappings without additional auxiliary clues was recently proposed. In this paper, we propose an all-in-one end-to-end multi-lingual multi-speaker ASR system that integrates the capabilities of these two systems. The proposed model is evaluated using mixtures of two speakers generated by using 10 languages, including mixed-language utterances.

Review:
This paper presents an end-to-end system that can recognize single-channel multiple-speaker speech with multiple languages.
Pros:
- The paper is well written.
- It shows the existing end-to-end multi-lingual ASR (Seki et al., 2018b) and end-to-end multi-speaker ASR (Seki et al., 2018a) techniques can be combined without any change to achieve reasonable performance.
- It demonstrates the challenge of single-channel multi-lingual multiple-speaker speech recognition, and compares the performance of the multiple-speaker system on the mixed speech and the single-speaker system on the isolated speech.
Cons:
- It lacks novelty: the proposed framework just simply combines the two existing techniques as mentioned above.
- The training and evaluation data are both artificially created by randomly concatenating utterances with different languages from different speakers with different context. I am not sure of how useful the evaluation is, since this situation is not realistic. Also, currently it cannot test the real code-switching since the utterances are not related and not from the same speaker.
- There are not enough analyses. E.g. it would be good to analyze what contributes to the gap between the single-speaker ASR system performance on the isolated speech and the multi-lingual multi-speaker ASR system on the mixed speech. How well does the proposed end-to-end framework perform compared to a two-step framework with speaker separation followed by multi-lingual single-speaker ASR?