Paper:
Equilibrium Propagation (EP) is a learning algorithm that bridges Machine Learning and Neuroscience, by computing gradients closely matching those of Backpropagation Through Time (BPTT), but with a learning rule local in space. Given an input x and associated target y, EP proceeds in two phases: in the first phase neurons evolve freely towards a first steady state; in the second phase output neurons are nudged towards y until they reach a second steady state. However, in existing implementations of EP, the learning rule is not local in time: the weight update is performed after the dynamics of the second phase have converged and requires information of the first phase that is no longer available physically. In this work, we propose a version of EP named Continual Equilibrium Propagation (C-EP) where neuron and synapse dynamics occur simultaneously throughout the second phase, so that the weight update becomes local in time. Such a learning rule local both in space and time opens the possibility of an extremely energy efficient hardware implementation of EP. We prove theoretically that, provided the learning rates are sufficiently small, at each time step of the second phase the dynamics of neurons and synapses follow the gradients of the loss given by BPTT (Theorem 1). We demonstrate training with C-EP on MNIST and generalize C-EP to neural networks where neurons are connected by asymmetric connections. We show through experiments that the more the network updates follows the gradients of BPTT, the best it performs in terms of training. These results bring EP a step closer to biology by better complying with hardware constraints while maintaining its intimate link with backpropagation.

Review:
I think it is an intriguing paper, but unfortunately left me a bit confused. I have to admit is not a topic I'm really versed in, so it might be that this affected my evaluation of the work. But also, as a paper submitted to ICLR I would expect the paper to be self-contained and be able to provide all the details needed. 
I do appreciate the authors providing in the appendix the proofs of the other theorems even if they come form other works. 
The paper introduces C-EP, an extension of  a previously introduced algorithm EP, such that it becomes biologically plausible. In particular EP is local in space but not in time (you need the steady state of the recurrent state after the first stage at the end of the second stage to get your gradients). I think this is fair, and the need for biological plausibility is well motivated in the beginning of the work.  
My first issue is with the proof for the equivalence between EP and C-EP. This is done by taking the limit of eta goes to 0. I think I must be missing something. But the proof relies on eta being small enough such that \theta_i = \theta (i.e. theta does not change). Given this state evolves the same way as for EP, because we basically not changing theta. 
Yet the crux of my issue is exactly here. The proof relies on the fact that we don't change theta. So then when you converged on the second phase, isn't theta the same as theta_0? So you haven't actually learned anything!? Basically looking at the delta isn't this just misleading? 
Ok lets assume that on the last step you allow yourself to change eta to be non-zero. (I.e. we are just after the delta in theta, and what to show we can get the same delta in theta as EP which is how the proof is phrased). Then in that difference aren't you looking at s_{t+1} and s_t rather than s_{t+1} and s_0, which is what EP would do? In EP you have s^\beta_* - s_*. This is not what you get if you don't update theta and apply C-EP? 
I think there might be something I'm missing about the mathematical argument here. 
At a higher-level question, we talk about the transition function F as being a gradient vector field, i.e. there exist a phi such that F is d phi/d theta.  Why is this assumption biologically plausable ? Parametrizing gradient vector fields in general is far from trivial, and require very specific structure of the neural implementation of F to be true. Several works have looked at parametrizing gradient vector fields (https://arxiv.org/abs/1906.01563, https://arxiv.org/pdf/1608.05343.pdf) and the answer is that without parametrizing it by actually taking the gradient of a function there is not much of a choice. 
Incidentally, here we exploit that  F = sigma (Wx), with W symmetric. This is a paramtrization of a gradient vector field, i.e. of xU, where UU^T =W I think. But if you want to make F deep than it becomes non-trivial to restrict it to gradient vector field. Is the assumption that we never want to move away from vanilla RNNs? And W symmetric is also not biologically plausible. In C-EP you say is not needed to be symmetric, but that implicitly means there exist no phi and everything that follows breaks, no? 
I'm also confused by how one has access to d phi / ds and d phi / d theta. Again I feel like I'm missing information and the formalism is not introduced in a way that it is easy to parse. My understand is that you have an RNN that updates the state s. And the transfer function of this RNN is meant to be d phi / ds, which is trues if the recurrent weight is symmetric. Fine. But then why do we have access to d phi/ dtheta? Who is this function? Is the assumption that d s / dtheta is something we can compute in a biologically plausible way? Is this something that is obvious?