Paper:
MergedFile
We study the problem of learning representations of sequence data. Recent work has built on variational autoencoders to develop variational recurrent models for generation. Our main goal is not generation but rather representation learning for downstream prediction tasks. Existing variational recurrent models typically use stochastic recurrent connections to model the dependence among neighboring latent variables, while generation assumes independence of generated data per time step given the latent sequence. In contrast, our models assume independence among all latent variables given non-stochastic hidden states, which speeds up inference, while assuming dependence of observations at each time step on all latent variables, which improves representation quality. In addition, we propose and study extensions for improving downstream performance, including hierarchical auxiliary latent variables and prior updating during training. Experiments show improved performance on several speech and language tasks with different levels of supervision, as well as in a multi-view learning setting.

Review:
This is largely an experimental paper, proposing and evaluating various modifications of variational recurrent models towards obtaining sequence data representations that are effective in downstream tasks. The highlighted contribution is a "stochastic generation" training procedure in which the training objective evaluates the reconstruction of output sequence elements from individual latent variables independently. The main claim is that the resulting model, augmented with prior updating and/or hierarchical latent variables, improves results w.r.t. the baselines.
My main concern is that the various choices are not motivated well, e.g. with examples or detailed descriptions of the issues addressed and that the resulting implications are not discussed in detail (see detailed comments below). This could perhaps be alleviated during the rebuttal discussion.
Empirically, when used in conjunction with prior updating and/or hierarchical latent variables, the proposed "stochastic generation" approach improves upon the baselines, but not when used in isolation. This is OK, but it weakens the contribution since it's more unclear what the exact advantage "stochastic generation" is, how it takes advantage of prior updating, and so on. Could you maybe discuss this in the rebuttal? The fact that not all model variants considered are evaluated on all settings also contributes to this problem (again, see below).
General questions:
- "dependence of observations at each time step on all latent variables": Unfortunately, this means that the complexity of evaluating the model during training is O(n^2), where n is the sequence size, rather than linear in the standard case. Is that correct? I think this is what is alluded to on the top on page 4. Could you discuss this trade-off?
- regarding section 2.1.: Multi-modal marginal probabilities are also used due their increased modeling power, and this again seems like a potential limitation of the proposed approach w.r.t. the baseline, and is not discussed.
- "the mean of z_t may have very small probability and thus may not be a good choice": I think this statement requires more context. The mean of z_t can have low probability in both cases (e.g. if the posterior has a high variance). Are you suggesting that the low probability issue is exacerbated by to the sampling of previous z_{t-1}? Or are you comparing to the case where the mean z_{t-1} is used instead of sampling as well?
Stochastic generation:
- While I understand where it's coming from, the term "stochastic generation" is somewhat misleading, since stochasticity is already present in the generation process for VAEs;
- Stochastic generation is introduced as a way to approximate the generation process. However, when it's introduced, it's not clear what the generation process that needs to be approximated is. Introducing the model in eq. (6-7), motivating its use and then showing how it is obtained through stochastic generation second would improve the clarity of the paper.
- Related to the point above, the implications of using the model in eq. (6-7) are not discussed. The graphical model in Figure 1 suggests that x_k depends jointly on all the (z_t)_{t=1 ... sequence_size}. Instead, in eq. (6-7), each x_k is generated independently from each z_t (for t = 1 ... T, and k sampled from a distribution which depends on t). In particular, if I understand this correctly, the distribution p(x_k | z) = p(x_k | z_1 \dots z_T) factorizes as p(x_k | z_1) p(x_k | z_2) ... p(x_k | z_T). Could you motivate this choice and its expected effect? It seems to me that this encourages each z_t to capture all the information needed to reconstruct each x_k in the corresponding window.
Experimental results:
- Table 2: I think this table since it includes most models, but it still misses RecRep (without delta = 0) and StocCon. Could you confirm whether StocCon vs. RecRep have the same setting except the use of recurrent stochastic connections in StocCon vs. using eq. (4) in RecRep with window size 1?
- In Table 4, the difference between line 5 and line 6 is interesting and I wish it was discussed more, maybe used in the visualization experiment to show how/why "stochastic generation" with a larger window improves performance.
- Figure 3, could it be that the use of hierarchical latent variables (H) accounts for the visual difference? Is a difference still observed when comparing lines 3 and 7 in Table 4, whose settings seem more comparable?
- 
Minor issues:
- the lack of parenthesis around citations makes the text hard to follow at times (maybe use \citep whenever the citation mixes with the text?);
- typo: "for use in a downstream tasks"
- typo: "with graphical model as described" => "with the/a graphical model as described"