Paper:
Middle-Out Decoding
Despite being virtually ubiquitous, sequence-to-sequence models are challenged by their lack of diversity and inability to be externally controlled. In this paper, we speculate that a fundamental shortcoming of sequence generation models is that the decoding is done strictly from left-to-right, meaning that outputs values generated earlier have a profound effect on those generated later. To address this issue, we propose a novel middle-out decoder architecture that begins from an initial middle-word and simultaneously expands the sequence in both directions. To facilitate information flow and maintain consistent decoding, we introduce a dual self-attention mechanism that allows us to model complex dependencies between the outputs. We illustrate the performance of our model on the task of video captioning, as well as a synthetic sequence de-noising task. Our middleout decoder achieves significant improvements on de-noising and competitive performance in the task of video captioning, while quantifiably improving the caption diversity. Furthermore, we perform a qualitative analysis that demonstrates our ability to effectively control the generation process of our decoder.

Review:
This paper describes a middle-out decoding scheme for the encoder-decoder frameworks, and its application to a synthetic symmetric task and the video captioning task. The main idea is that the decoder starts from an externally provided middle word (either an oracle, or a word predicted using a classifier over the vocabulary), and then alternates between expanding to the left and to the right. The two directions are connected through a self attention mechanism which uses two heads, one over the previous decoder hidden states, and one over the decoder inputs. The use of two attention heads for self attention is claimed as a novel contribution, but seems to be more of a design decision. The use of two heads instead of one is never tested directly. Strong results are shown on a synthetic task, and they are able to match the baseline on image captioning while increasing diversity. They can also easily control the output by changing the provided middle word.   This is a clear paper with an interesting main idea and sound experimental design. I’m not sure the idea is sufficiently developed to warrant publication at this stage, but it is very close. I’m also not sure it’s a very good idea, which is affecting my grade somewhat, though I am trying my best to judge the work primarily based on its evaluation of this idea.  At its core, middle-out decoding re-introduces some amount of pipelining into the encoder-decoder architecture. The system designer must decide how the middle word will be selected from the reference sentence (it isn’t the exact middle word by token count, but some high-information content word like a main verb or the subject or or object noun phrase), and they must decide how that middle word will be predicted before decoding begins. Those decisions will have a large impact on the ceiling for the complete system that will be built around them. I consider this pipeline to be a pretty steep cost. What we are buying with this cost is controllability and diversity. I do not see controllability as a large gain. Controllable output has been studied in NMT, and I’ll provide some references below. And the diversity effect isn’t compared against other methods of introducing diversity (again, I’ll provide references). So, I’m not sure if this cost is worth what we gain. Hence my middling score.  There are a number of things the authors could do to make this a stronger paper:  Implement the proposed future work and use a strong, proven action classification architecture to predict the middle word, and beat the baseline of Seq2Seq + Attention + Self-Attention. I suspect that if this had happened, I would not be focusing so much on the side-benefits. Another way to potentially improve your score would be to try incorporating the middle word choice into the beam, so with a beam of 5, the system starts with the 5-best choices for the middle word from the current model. That is, if you are not already doing so (it isn’t clear from the paper).  Compare to known methods for controlling the words in the output sequence by altering the beam search: https://arxiv.org/abs/1804.06609 https://arxiv.org/abs/1704.07138 I suspect you will win here, as you are altering both the search and the model, while the above techniques alter only the search, but without experiments it’s hard to know. Providing both the embedding of the constraint-word to the decoder while also constraining the search to output the word would be an interesting and novel way to extend the work cited above to change both the model and search.  Compare to a known methods for introducing diversity, for example (though you can probably do better than these, there may well be techniques specific to the image captioning domain that I am unaware of): https://arxiv.org/abs/1611.08562 https://arxiv.org/abs/1605.03835  Other, smaller concerns:  Equation (3) has a circular dependency on h^d_t, you probably meant h^d_{t-1}.  Do you share encoders for the middle word classifier and the middle-out decoder? If so, please specify how you balance the two loss functions. If not, it might be worth considering to strength the middle-word classifier.  Your results on the synthetic data are impressive, but it would be worthwhile to specify how many more parameters are used with this new model. Likewise for the captioning experiments.  ===  Thanks very much for addressing my comments in your response. Given these additional experiments and comments, I will gladly raise my scores. Using a sampled oracle to simulate a higher accuracy classifier was particularly clever. Good luck with your future research.  ===