Recurrent neural networks (RNNs) have been extraordinarily successful for prediction with sequential data. To tackle highly variable and noisy real-world data, we introduce Particle Filter Recurrent Neural Networks (PF-RNNs), a new RNN family that explicitly models uncertainty in its internal structure: while an RNN relies on a long, deterministic latent state vector, a PF-RNN maintains a latent state distribution, approximated as a set of particles. For effective learning, we provide a fully differentiable particle filter algorithm that updates the PF-RNN latent state distribution according to the Bayes rule. Experiments demonstrate that the proposed PF-RNNs outperform the corresponding standard gated RNNs on a synthetic robot localization dataset and 10 real-world sequence prediction datasets for text classification, stock price prediction, etc. * equal contribution Preprint. Under review. F L x 4 r m L b Q h r L Z b t q l m 0 3 Y f R F K 6 G / w 4 k E R r / 4 g b / 4 b t 2 0 O 2 j q w M M y 8 Y d + b M J X C o O t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S N k m m G f d Z I h P d D a n h U i j u o 0 D J u 6 n m N A 4 l 7 4 S T u 7 n f e e L a i E Q 9 4 j T l Q U x H S k S C U b S S 3 x 8 m a A b V m l t 3 F y D r x C t I D Q q 0 B t U v m 2 N Z z B U y S Y 3 p e W 6 K Q U 4 1 C i b 5 r N L P D E 8 p m 9 A R 7 1 m q a M x N k C + W n Z E L q w x J l G j 7 F J K F + j u R 0 9 i Y a R z a y Z j i 2 K x 6 c / E / r 5 d h d B P k Q q U Z c s W W H 0 W Z J J i Q + e V k K D R n K K e W U K a F 3 Z W w M d W U o e 2 n Y k v w V k 9 e J + 2 r u m f 5 w 3 W t e V v U U Y Y z O I d L 8 K A B T b i H F v j A Q M A z v M K b o 5 w X 5 9 3 5 W I 6 W n C J z C n / g f P 4 A 8 W q O w g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 1 X f z o B L f z 7 F P V / w Q N r Y I e i F Z x Q = " > A A A B 7 H i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 K I U I 9 F L x 4 r m L b Q h r L Z b t q l m 0 3 Y f R F K 6 G / w 4 k E R r / 4 g b / 4 b t 2 0 O 2 j q w M M y 8 Y d + b M J X C o O t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S N k m m G f d Z I h P d D a n h U i j u o 0 D J u 6 n m N A 4 l 7 4 S T u 7 n f e e L a i E Q 9 4 j T l Q U x H S k S C U b S S 3 x 8 m a A b V m l t 3 F y D r x C t I D Q q 0 B t U v m 2 N Z z B U y S Y 3 p e W 6 K Q U 4 1 C i b 5 r N L P D E 8 p m 9 A R 7 1 m q a M x N k C + W n Z E L q w x J l G j 7 F J K F + j u R 0 9 i Y a R z a y Z j i 2 K x 6 c / E / r 5 d h d B P k Q q U Z c s W W H 0 W Z J J i Q + e V k K D R n K K e W U K a F 3 Z W w M d W U o e 2 n Y k v w V k 9 e J + 2 r u m f 5 w 3 W t e V v U U Y Y z O I d L 8 K A B T b i H F v j A Q M A z v M K b o 5 w X 5 9 3 5 W I 6 W n C J z C n / g f P 4 A 8 W q O w g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 1 X f z o B L f z 7 F P V / w Q N r Y I e i F Z x Q = " > A A A B 7 H i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 K I U I 9 F L x 4 r m L b Q h r L Z b t q l m 0 3 Y f R F K 6 G / w 4 k E R r / 4 g b / 4 b t 2 0 O 2 j q w M M y 8 Y d + b M J X C o O t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S N k m m G f d Z I h P d D a n h U i j u o 0 D J u 6 n m N A 4 l 7 4 S T u 7 n f e e L a i E Q 9 4 j T l Q U x H S k S C U b S S 3 x 8 m a A b V m l t 3 F y D r x C t I D Q q 0 B t U v m 2 N Z z B U y S Y 3 p e W 6 K Q U 4 1 C i b 5 r N L P D E 8 p m 9 A R 7 1 m q a M x N k C + W n Z E L q w x J l G j 7 F J K F + j u R 0 9 i Y a R z a y Z j i 2 K x 6 c / E / r 5 d h d B P k Q q U Z c s W W H 0 W Z J J i Q + e V k K D R n K K e W U K a F 3 Z W w M d W U o e 2 n Y k v w V k 9 e J + 2 r u m f 5 w 3 W t e V v U U Y Y z O I d L 8 K A B T b i H F v j A Q M A z v M K b o 5 w X 5 9 3 5 W I 6 W n C J z C n / g f P 4 A 8 W q O w g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 6 1 X f z o B L f z 7 F P V / w Q N r Y I e i F Z x Q = " > A A A B 7 H i c b V B N S 8 N A F H y p X 7 V + V T 1 6 W S y C p 5 K I U I 9 F L x 4 r m L b Q h r L Z b t q l m 0 3 Y f R F K 6 G / w 4 k E R r / 4 g b / 4 b t 2 0 O 2 j q w M M y 8 Y d + b M J X C o O t + O 6 W N z a 3 t n f J u Z W / / 4 P C o e n z S N k m m G f d Z I h P d D a n h U i j u o 0 D J u 6 n m N A 4 l 7 4 S T u 7 n f e e L a i E Q 9 4 j T l Q U x H S k S C U b S S 3 x 8 m a A b V m l t 3 F y D r x C t I D Q q 0 B t U v m 2 N Z z B U y S Y 3 p e W 6 K Q U 4 1 C i b 5 r N L P D E 8 p m 9 A R 7 1 m q a M x N k C + W n Z E L q w x J l G j 7 F J K F + j u R 0 9 i Y a R z a y Z j i 2 K x 6 c / E / r 5 d h d B P k Q q U Z c s W W H 0 W Z J J i Q + e V k K D R n K K e W U K a F 3 Z W w M d W U o e 2 n Y k v w V k 9 e J + 2 r u m f 5 w 3 W t e V v U U Y Y z O I d L 8 K A B T b i H F v j A Q M A z v M K b o 5 w X 5 9 3 5 W I 6 W n C J z C n / g f P 4 A 8 W q O w g = = < / l a t e x i t > K particles stochastic Bayesian update PF-RNNs combine the strengths of RNNs and particle filtering by learning a latent particle belief representation and updating the belief with a particle filter. We apply the idea specifically to LSTM and GRU and construct PF-LSTM and PF-GRU, respectively. Our results show that PF-RNNs outperform the corresponding RNNs on various sequence prediction tasks, including text classification, activity recognition, stock price prediction, robot localization, etc. Various aspects of the PF-RNN approach provide opportunities for further investigation. Currently, PF-RNNs make a prediction with the mean of the particle belief. An alternative is to aggregate particles, e.g., by estimating higher-order moments, estimating the entropy, attaching an additional RNN [15], or applying an attention mechanism [34]. PF-RNNs can serve as drop-in replacements for RNNs. While this work focuses on sequence prediction and classification, the idea can be applied equally well to sequence generation and sequence-to-sequence prediction, with application to, e.g., image captioning and machine translation.
REF integrated the particle filter with standard RNNs, e.g., the LSTM, and introduced PF-RNNs for sequence prediction.
Particle Filter Recurrent Neural Networks
ArXiv
ArXiv
Mathematics Computer Science