Paper:
The input method is an essential service on every mobile and desktop devices that provides text suggestions. It converts sequential keyboard inputs to the characters in its target language, which is indispensable for Japanese and Chinese users. Due to critical resource constraints and limited network bandwidth of the target devices, applying neural models to input method is not well explored. In this work, we apply a LSTM-based language model to input method and evaluate its performance for both prediction and conversion tasks with Japanese BCCWJ corpus. We articulate the bottleneck to be the slow softmax computation during conversion. To solve the issue, we propose incremental softmax approximation approach, which computes softmax with a selected subset vocabulary and fix the stale probabilities when the vocabulary is updated in future steps. We refer to this method as incremental selective softmax. The results show a two order speedup for the softmax computation when converting Japanese input sequences with a large vocabulary, reaching real-time speed on commodity CPU. We also exploit the model compressing potential to achieve a 92% model size reduction without losing accuracy.

Review:
The paper demonstrates the main challenge of using LSTM-based language models for input method in real time is the huge amount of computation in the softmax. The authors present a system to speed up the inference by avoiding computing the full softmax in the Japanese conversion task, where the number of output words can be limited from the mapping of the input sequence through a lexicon. The experiment result is encouraging in that the proposed incremental selective softmax approach significantly reduces latency over the standard inference with the full softmax computation while not hurting accuracy much. The paper also evaluates the effect of quantization for LSTM LM model compression in terms of size and accuracy.
However, there are a few major problems of the paper as follows:
1. The main weakness in the experiment setup is that it misses a few competitive baselines in terms of inference speed, notably hierarchical softmax[1] and self normalization[2]. In the Japanese conversion task in the paper, it only needs to evaluate the scores of limited output words that are given from the mapping of the input sequence through the lexicon. This is exactly like the rescoring setup in speech recognition and machine translation, where self-normalization is typically used for efficient inference to avoid computing the expensive softmax normalization term [2,3]. Assuming the number of selected output words is K and the entire vocabulary size is V, then the time complexity is O(K logV) for the hierarchical softmax, O(K) for self normalization, but O(V) for all the baselines in the paper. Self normalization is simple to implement and works well in practice, while the proposed incremental selective softmax approach in the paper needs an additional step to sample most frequent words to adjust the normalization term. Without showing the self normalization result, I am not convinced that the proposed approach is better and needed.
[1] F. Morin and Y. Bengio. "Hierarchical Probabilistic Neural Network Language Model," in Proc. of AISTATS, 2005,
[2] J. Devlin et al., "Fast and Robust Neural Network Joint Models for Statistical Machine Translation," in Proc. of ACL, 2014.
[3] Y. Shi, W. Zhang, M. Cai and J. Liu, "VARIANCE REGULARIZATION OF RNNLM FOR SPEECH RECOGNITION," in Proc. of ICASSP, 2014.
2. The proposed approach would only be useful in speeding up the conversion task, but not applicable to the prediction task where it needs to evaluate all words and choose the top hypotheses. Also how is the latency of the prediction task compared to conversion task? Please also add it to the experiment result.
3. The idea of using quantization for neural network model compression is not novel (even for language model), although it is listed as one of the main contributions in Section 1.
So in general, I think the paper is insufficient in novelty and missing competitive baselines.
Some specific comments:
4. Figure 2(b) is not clear what it means, and not referenced anywhere in the paper.
5. The last 3 lines in Section 3: "as each path has different missing vocabularies": Why is that? The candidates of the output words should only depend on the input sequence and the lexicon, based on Eq(1)(2).
6. It is not clear how to adjust the probability in the second pass of incremental selective softmax. The description "we compute a union of all missing vocabularies, and then recompute the logits of them in batch." is unclear what it means.
7. Section 4.2: "is measure with numpy" -> "is measured with numpy".
8. Section 4.4: It is not clean how "76x speedup" is computed from Table 2 since all the time numbers are rounded. Consider also showing one digit after the decimal point.