Paper:
Recurrently Controlled Recurrent Networks
Recurrent neural networks (RNNs) such as long short-term memory and gated recurrent units are pivotal building blocks across a broad spectrum of sequence modeling problems. This paper proposes a recurrently controlled recurrent network (RCRN) for expressive and powerful sequence encoding. More concretely, the key idea behind our approach is to learn the recurrent gating functions using recurrent networks. Our architecture is split into two components -a controller cell and a listener cell whereby the recurrent controller actively influences the compositionality of the listener cell. We conduct extensive experiments on a myriad of tasks in the NLP domain such as sentiment analysis (SST, IMDb, Amazon reviews, etc.), question classification (TREC), entailment classification (SNLI, SciTail), answer selection (WikiQA, TrecQA) and reading comprehension (NarrativeQA). Across all 26 datasets, our results demonstrate that RCRN not only consistently outperforms BiLSTMs but also stacked BiLSTMs, suggesting that our controller architecture might be a suitable replacement for the widely adopted stacked architecture.

Review:
Update:  Thanks for your response. I'm glad to hear that you are going to open source the optimizations; I look forward to playing with this. Best of luck with the follow-up work, and I look forward to seeing how the RCRN performs on SNLI in the cross-attention task setting (really hoping to see this in the camera-ready!).  Original Review:  The core idea behind this paper is to use RNNs (LSTMs or GRUs in this work) to compute the gates (input, forget, etc.) for a higher level RNN. They use this idea to show how to improve performance on a large number of tasks (mostly classification) with relatively simple models that have little to no loss in efficiency compared to models that perform similarly. Some of the performances achieved new state-of-the-art results. When implemented well, they can even be faster despite the apparent addition of complexity.   The setup as controller and listener cells seems unnecessarily confusing. It seems clearer to describe this as a collection of three RNNs, two that compute gates for the state produced by the third. If it was presented that way, then it would be clearer that Eq. 15 and 16 are really the meat of the RCRN. The first 14 equations are really just normal (or bidirectional adaptations of normal) RNN equations (7-9 are nice to see for clarity). I understand that it can be helpful to write out all of those equations some times, but I would recommend defining an RNN cell once, and then focus on describing how you use the three different states, rather than duplicating the same equations 2 times in Eq. 1-6 and then again in 10-14. On my first read, I found myself looking for potential difference in all those equations because I assumed they were written out so explicitly for that reason.  In equations 1-3, not sure if it is a typo, but either way: it would be clearer if the W_i, W_f, and W_o used to get i_t^2, f_t^2, and o_t^2 respectively had a numerical superscript like the other matrices. It sounds like it would be a 2, but without the superscript it kind of feels like a typo since all the other matrices have superscripts. U_i^1 is also used for i_t^2, but it seems like that should be U_i^2 instead. Would like a little clarification or a note if these aren’t typos clarifying that some of the weights are indeed shared.  Why not run on SQuAD? That is a pretty standard QA dataset now, so much so that its absence makes me wonder whether that was a failure case for RCRNs. It would be great if you could assuage that concern for future readers. I could almost say the same for MultiNLI over SNLI now as well, but I understand that both of these have external testing systems that can make using them difficult some times.  Why do you only compare with models that do not use semi-supervised learning on external corpora? If you are using CoVe, which is using supervised learning on external corpora and GloVe, then it might seem a bit arbitrary to rule out semi-supervised augmentations in those tables. I think the claims are still effective for RCRN because the BiLSTM and 3l-BiLSTM also use CoVe, POS embeddings, etc.  What exactly does CUDA optimized mean? How hard would these be to make on your own? Or will you be open sourcing these essential optimizations or even better building into something like PyTorch or Tensorflow? If you want maximum impact, definitely help people get that super fast CUDA optimized version. If that existed open source, I would definitely start trying it out.  I’m not convinced about the claim that this is different than ‘going deeper’. This just seems like a different (apparently smarter/better) way of going deeper than by just stacking RNNs as layers.  I wonder whether you could just use a single RNN cell as the Controller and do a simpler transformation of that state to get the different gate values. Perhaps this could be even faster?  Can we have RCRNs controlling RCRNs?  Character level embeddings are using a BiGRU model; is this better than using pertained character embeddings?  Lots of the true SOTA models for entailment do use sentence cross attention, and you chose not to. Why? How do the RCRN’s purported benefits stand up when attention is in the mix for entailment? Or perhaps the Highway networks are particularly important and you don’t get any benefit from the RCRN without them?   It seems insane at first glance that your 3L-BiLSTM is as good as all those other more complex models even without the RCRN. Is that coming from the GRU character embeddings or the POS embeddings or the CoVe embeddings? Embeddings are just really that important it seems, and as we’ve created better representations, simple BILSTMs and stacked versions are getting better.   Overall is a 7, with a confidence of 4.  I'm reserving very confident for reproducibility of systems that have been declared as being open source in the future. This submission includes a lot of detail, but it seems unlikely that everyone who wants to use this work will implement those custom CUDA ops themselves. The RCRN seems simple enough to reproduce, but at the speeds presented, it seems potentially difficult.