Paper:
A General Method for Amortizing Variational Filtering
We introduce the variational filtering EM algorithm, a simple, general-purpose method for performing variational inference in dynamical latent variable models using information from only past and present variables, i.e. filtering. The algorithm is derived from the variational objective in the filtering setting and consists of an optimization procedure at each time step. By performing each inference optimization procedure with an iterative amortized inference model, we obtain a computationally efficient implementation of the algorithm, which we call amortized variational filtering. We present experiments demonstrating that this general-purpose method improves performance across several deep dynamical latent variable models.

Review:
Thank you for your thorough response.  I appreciate the additional analysis and I think it will definitely make for a stronger paper.  I also appreciate the run time discussion --- I was confused about the computational costs of the amortization, and the author response cleared that up for me.  I am still a bit confused about why AVF would be less susceptible to local minima (and more likely to use the latent codes) than the baseline comparisons, and it would be great to include some discussion about this improvement.  I have increased my score.   ------------------ The authors present a method for fitting dynamic latent variable models with a variational inference objective.  They use a variational family factorized sequentially to approximate the posterior over the latent variables. They describe the inference procedure using this posterior sequence approximation, which involves solving T optimization problems at each step of the sequence, approximating filtering.  Learning the global parameters requires performing this as an inner loop.  To reduce computation, they use iterative inference models as an amortized inference scheme.  They present experimental comparisons to baseline methods in a variety of domains involving sequence modeling.    *Clarity*:  The presentation is relatively clear --- the introduction clearly states the problem to be solved and approach they take to solve it. Some aspects were confusing, particularly equations 5 and 13 (and further details of iterative inference mode).  *Quality*: The paper appears to be technically correct, and the proposed method is tested in a variety of domains.  The idea of using a sequentially structured inference network is interesting.  I think the results section would be more compelling if alternate inference network schemes were compared (or if a baseline contains an amortization strategy, how does the structure compare?).  *Originality+Impact*:  The approach showed big empirical gains in some real data examples.  I think the potential impact of this method is contingent on some practical aspects, such as run time.    *Questions and Comments*:  - I am a little confused about the term empirical priors, as used in line 82.  Inference models of the form in Eq (4) have difficulty with which particular distribution?  Is the idea that dynamical priors (or even hierarchical priors) induce some structure that is ignored by a generic inference network?  Is the type of structure that the amortized filtering inference approach exploits?  - Iterative inference models are a big part of the proposed method, however they are not well described in this manuscript.  Could some part of section 2.3 explain Eq (5) in greater detail.    - I am a bit confused about the units used in the experiment log-likelihood reporting.  Do the experiments report the probability of test sequences (having marginalized out the latent variables)?    - How does optimization differ between the using the existing filtering methods vs AVF?  Is it a matter of optimization (e.g. would the existing baselines given more time or tuning rise to the performance of AVF?  Is AVF uniformly better?    - For the video modeling --- what is driving this big improvement?  Is the posterior approximation more accurate?  In general, does the use of the AVF framework enable more expressive posterior approximations than the baselines?  Or does it afford more efficient inference?  Or both?    - How much does the amortization afford you?  If the same variational approximation were used within a stochastic variational inference framework (say for a small dataset), we would expect it to outperform an amortized framework in terms of ELBO objective value.  How does including the amortization structure affect the final model fit?    - For learning global parameters, would a filter + smoothing approach be sensible?  Can this approach be used to update global parameters using information from the entire sequence to do learning?