Extrapolative Continuous-time Bayesian Neural Network for Predictive Streaming Domain Adaptation
Abstract
This work is inspired by internal predictive modeling in neuroscience.
Such amechanism allows humans to process non-stationary streaming sensory data withremarkably low latency and high precision by continuously and instantly generatinginternal predictive models.
This leads us to rethink why existing approaches inunsupervised domain adaptation (UDA) fail to achieve what humans are capableof when handling the streaming data.
In this paper, we explore the possibilityof introducing such a mechanism in UDA.
We propose to formulate predictivemodeling as a continuous-time Bayesian filtering problem.
We aim to infer latentstates of a dynamical system that describes the dynamics of model parameters of a
UDA model progressively receiving non-stationary streaming data.
To this end, wethen propose extrapolative continuous-time Bayesian neural networks (ECBNN),which generalize existing Bayesian neural networks to represent temporal dynamicsand allow us to extrapolate the distribution of model parameters over the time axiswhen only historical streaming data is available.
Remarkably, our empirical resultsshow that ECBNN is capable of continuously generating better distributions ofmodel parameters along the time axis given historical data only, thereby achieving(1) gradually improved alignment between the source and target features, and (2)gradually improved model performance over time during the real-time testing stage.
Introduction
Human consciousness requires building an internal predictive model that describes the external world[1, 2].
Numerous neuroscience studies have suggested that humans continually update their internalpredictive model to represent the statistics of the fluctuating environments accurately [3, 4, 5, 6].
Internal predictive modeling is crucial for humans perceiving continuous stream of sensory inputs(e g. sound and sight) in real time given constraints imposed by neural transmission delays [7].
For example, to catch a fast-moving object, the internal predictive model is required to forecast itstrajectory ahead to compensate for the response latency of neurons; to recognize words from lipmovements at various view angels in real time, again, the internal predictive model is required toensure that decision outcomes are not delayed [8].
Such a mechanism allows humans to process non-stationary streaming sensory data with remarkablylow latency and high precision by continuously and instantly generating internal predictive models.
Incontrast, most existing UDA algorithms either require collecting additional data from a large amountof increasingly more domains [9] or using streaming data batches for fine-tuning [10, 11], which isimpractical for real-time streaming situations.
This leads us to rethink why existing approaches in unsupervised domain adaptation (UDA) fail to achieve what humans are capable of when handlingthe streaming data.
This work aims to introduce “internal predictive modeling” to augment traditional UDA algorithmsin handling non-stationary streaming data.
UDA augmented with such a mechanism should be ableto instantly extrapolate the model parameters on the fly throughout the real-time testing stage, ratherthan merely performing offline adaptation or online adaptation with unacceptable latency due to datacollection and training or fine-tuning.
To achieve this, we propose to formulate predictive modeling as a continuous-time Bayesian filteringproblem.
More specifically, we model the neural network (NN) parameters of UDA evolving withnon-stationary streaming data as a continuous-time stochastic dynamical system.
We aim to infermodel distributions over time given the historical streaming data.
To our knowledge, such Bayesian treatment of time-evolving NN parameters is mostly unexplored.
Existing continuous-time Bayesian filtering approaches, e.g. CTBN [12], assume discrete-state forthe dynamical system; and thus cannot scale to NNs.
Furthermore, most of Bayesian neural networks,including SDE-BNN [13] are designed to reason about static distributions of NN parameters.
Inthis paper, we propose extrapolative continuous-time Bayesian neural network (ECBNN), whichgeneralizes existing static BNNs to represent temporal dynamics and allows us to extrapolate thedistribution of model parameters over the time axis when only historical streaming data is available.
Specifically, by adopting ECBNN as encoder of UDA for streaming data, the new model is capableof efficiently inferring multi-step ahead model distributions before observing the incoming data,therefore effectively reducing the latency.
The multi-step ahead Bayesian inference is achieved bydeveloping a particle-filter-like probabilistic differential equation, dubbed Particle Filter Differential Equation (PFDE), which allows reasoning about time-evolving model distributions suitable forhandling each incoming data point.
Such distributions are produced by efficiently solving PFDE.
Furthermore, PFDE is based on a continuous-time Markovian assumption for the dependency ofmodel distributions, which can be evaluated in real-time, allowing us to align source and targettemporal domains in the entire timeline by optimizing an upper bound for temporal-domain-invariantrepresentation learning.
Related Work
Unsupervised Domain Adaptation with Streaming Data.
There is a vast body of prior work[14, 15, 16, 17, 18, 19] exploring domain-invariant representation learning for the purposes ofunsupervised domain adaptation (UDA).
These works’ key idea is harnessing expressive powerof neural networks to transform the input data to latent representations that are further alignedacross static domains.
However, in real-world streaming applications, streaming data usually followsnon-stationary distributions.
Consequently, these work and a few other offline adaptation methods[20, 21, 22] mostly fail to handle such data due to either the assumption of static domains or therequirements of low latency.
The UDA for streaming data with non-stationary distribution have been studied before but in differentproblem settings with different focuses.
For example, continuous manifold adaption (CMA)
[9]addresses this problem by adapting the model with the additional unlabeled data from a fixed numberof pre-defined intermediate domains; predictive domain adaptations [23, 10] attempt to solve aspecific scenario where target data is unavailable but the meta data that characterizes the targetdomain is provided.
However, these methods cannot be directly applied to our problem settings dueto the unavailability of the meta-data, e.g. absolute time stamps of each data sample.
The closestwork to ours is [11], in which a meta adaptation framework is introduced to extract the temporaldomain-invariant representation from the streaming data.
However, [11] requires collecting partialstreaming data for fine-tuning, greatly limiting the applicability to real-time streaming situations.
In contrast, our work derive a new PF-based differential equation, allowing to continuously inferdistributions of NN parameters from historical streaming data and achieve much lower latency andtemporal-domain-invariant feature learning during real-time testing.
Bayesian Neural Networks with Neural Differential Equations.
Bayesian neural networks(BNNs)
[24] aim to infer the posterior distribution of the NN parameters given observations and aprior imposed on these parameters.
With a different aim, early work in Neural differential equations,e.g. neural ordinary differential equation (ODE) [25], focuses on adopting non-Bayesian NNs toparameterize the dynamics involved in differential equations over depth or layers.
Bayesian Neural
ODE [26, 27] improve upon Neural ODE by using BNNs to achieve a similar goal.
Unlike the abovestudies, this work aims to develop a BNN that captures the evolution of NN parameters over time.
The closest work to ours is [13], which focuses on building a continuous-depth BNN using stochasticdifferential equations.
However, their model assumes a static distribution for NN parameters; andthus cannot scale to capture the time-varying distribution of NN parameters.
Addressing this issue isone of our model’s focuses.
(Please refer to the Supplement for related work on particle filtering withneural networks)
Method
In this section, we first review the background of particle filtering, formalize the problem of predictivestreaming adaptation for non-stationary streaming data, and then describe our methods for addressingthe problem.
3.1 Background: Particle Filtering
Particle filtering (PF) provides an effective solution for state estimation problem of dynamical systems,in which sequential importance sampling (SIS) is used to approximate the evolution of the posteriordistribution of the state given observations, p ( θ t | x 1: t ) , where θ t represents the latent state at time tand x 1: t denotes observations from time 1 to t .
PF adopts a weighted set of particles { ( w ( j ) t , θ ( j ) t ) } kj =
1to construct an approximate posterior:
[Equation]
where δ denotes the Dirac-delta function; θ ( j ) t is the j -th particle (sample) drawn from an approximatedistribution called importance distribution q ( θ 1: t ) , which can be assumed to have a simple parametricform, e.g. a Gaussian; w ( j ) t is the corresponding importance weight.
By assuming a Markov chain for both the importance distribution and the state space as well as usingtransition distribution p ( θ t | θ t − 1 ) as the importance distribution, one can construct a bootstrap filter[28] with simplified importance weights as: w ( j ) t ∝ w ( j ) t − 1 p ( x t | θ t ) , where p ( x t | θ t ) is the likelihoodand (cid:80) j w ( j ) t = 1 .
3.2 Problem Setting: Predictive Streaming Domain Adaptation
Problem.
We consider the unsupervised domain adaptation setting for non-stationary streamingdata.
More specifically, given a source stream { B si } Ii =1 , a target stream { B ai } Ii =1 and a testingstream { B ei } Ii =1 , where I denotes the number of data batches; the source data batches B si ={ x st , y st } i · N − 1 t =( i − 1) · N are labeled, while the target and testing data batches B ai = { x at } i · N − 1 t =(
i − 1) · N ,
B ei = { x et } i · N − 1 t =( i − 1) · N are unlabeled, where N , the number of frames in each batch, is usually smallfor machine learning applications that require streaming inputs, e.g. streaming video processingsystems.
Note that the target and test stream share the same categorical domain, while source streamdiffers from them.
We first train a UDA based on the source and target streams { B si } Ii =1 and { B ai } Ii =1 .
Based on such a model, we aim to perform adaptation as each testing stream data batch B ei arrives.
Low Latency and Timeline.
Notably, during the real-time testing stage, our problem setting has strictrequirements for the latency caused by adaptation.
Therefore, existing offline adaptation methodsor online adaptation methods that require training or fine-tuning, are not applicable to our problemsetting.
Furthermore, the source stream, target stream and testing stream may not necessarily sharethe same timeline and the absolute time stamps of the data are unknown; and thus DA methods[18, 29] that require absolute time stamps fail to handle our problem setting as well.
3.3 Overview
Below we provide an overview of our method (Fig. 1).
[Figure]
(cid:54)(cid:82)(cid:88)(cid:85)(cid:70)(cid:72)(cid:172)(cid:3)(cid:82)(cid:85)(cid:3)(cid:55)(cid:68)(cid:85)(cid:74)(cid:72)(cid:87)
[Figure]
Training.
Our approach adopts both source stream and target stream data for training: (i) it starts by adopting PFDE,taking as input the historical streaming batches B 1: i − 1 ,to infer the posterior distributions of NN parameters ofthe encoder f e ; based on such distributions, particles aresampled to update the encoder f e (updated before processing B i ); (ii) it further take the updated encoder togenerate z , the encoder output for B i , and (cid:101) z , the encoderoutput for the approximated data required for calculatingthe proposed analytical upper bound (see Sec.3.7 for moredetails of the approximated data ); (iii) finally, we followthe typical adversarial UDA training procedure to evaluate the training loss given z and (cid:101) z and updatethe whole framework (see Sec.
3.6 for more detailed description).
Real-time Testing During the real-time testing stage, the testing stream cannot be revisited andshould be processed as each data batch arrives.
The testing procedure is mostly similar to the training,except that discriminator are removed.
3.4 Particle Filter Differential Equation (PFDE)
Traditional PF methods, e.g. bootstrap filters, require calculating likelihood for observations at eachtime point, which is infeasible in our problem setting since only historical data is available whenperforming multi-step ahead Bayesian inference.
We therefore develop, by the following theorem,a new type of differential equation (DE), dubbed Particle Filter Differential Equation (PFDE), thatdescribes the evolution of the state in a continuous-time stochastic dynamical system.
Theorem 1.
(Informal)
Suppose we are given a time series { x t } nτt
= τ , where τ is small constant.
Assume { x t } nτt = τ are conditionally independent given a sequence of latent states { θ ( t ) }
nτt = τ , which isitself assumed to be Markovian.
Assume marginal distribution of the state p ( θ ( t )) is stationary.
Letp ( θ ( t ) | θ ( t − τ )) be the transition distribution.
Let w ( t ) ( j ) be the importance weight.
Then we have:
[Equation]
where ( · ) ′ and ( · ) ′′ denote the first-order and second-order temporal derivatives respectively.
Thederivation of PFDE in Theorem 1 (See the Supplement for the proof) is based on the bootstrap filter.
By assuming the marginal distribution of the state p ( θ ( t )) as a stationary distribution (which is amild assumption as conditional distribution can still be non-stationary), we successfully build the linkbetween the transition distribution and the importance weight , whose calculation no longer requirescalculating the intractable likelihood term.
3.5 Extrapolative Continuous-time Bayesian Neural Networks (ECBNN)
At ECBNN’s core is a PFDE, which provides a Bayesian treatment of time-evolving NN parametersand inherits several advantages from both particle filters and differential equations.
For instance,
PFDE enables inferring posteriors of NN parameters along the time axis by solving itself at any giventime points.
Specifically, due to the intractable parametric form of the transition distribution p ( θ t | θ t − 1 ) in oursetting, we use an approximate distribution (cid:101) q ( θ ( t )) as the importance distribution.
Let w ( t ) ( j ) bethe importance weight; let θ ( t ) ( j ) be the particle and let (cid:101) q
( θ ( t ))
=
N ( m ( t ) , s ( t )) be the importancedistribution, where m ( t ) and s ( t ) are the mean and variance of the Gaussian.
By continuous-time
Markovian assumption and Theorem 1, the PFDE can be written as:
[Equation]
where f ( · ) is a nonlinear function, whose detailed form is described in the Supplement.
Let thesolution of PFDE be G ( t ) = { m ( t ) , s ( t ) , { w ( t ) ( j ) } kj =1 } .
Then ECBNN takes the following form toapproximate the posterior of the NN parameters q ( θ ( t )) :
[Equation]
We further demonstrate the inference and learning of ECBNN by applying it to address the problemof predictive streaming DA.
3.6 Application of ECBNN for Predictive Streaming Domain Adaptation
In this section, we demonstrate how to embed ECBNN into a typical UDA framework to performpredictive streaming DA.
Learning.
An UDA usually contains an encoder f e to project the input to a domain-invariant spacez with the help of an discriminator f d , such that the target data can be accurately predicted by thepredictor f p .
Let θ be the parameters of the encoder f e , an UDA performs a minimax optimizationwith the value function V ( f e , f d , f p ) as:
[Equation]
(4) where V p is the expectation of the prediction loss (e.g., crossentropy loss for classification tasks)over
the source data batches; λ d is a hyperparameter balancing both losses ; V d is the expectation ofthe discriminator loss over source and target data batches.
For simplicity, we only emphasize theparameters of the encoder f e and omit other parameters.
We extend an UDA with ECBNN to handle predictive streaming DA by adopting ECBNN as encoderof UDA.
The architecture of our model is shown in Fig.
Recall that ECBNN adopts an approximatedistribution (cid:101) q ( θ ( t )) as the importance distribution due to the intractability of the transition distribution.
This leads us to adopt variational inference to optimize our model, in which (cid:101) q
( θ ( t )) is treated asthe variational distribution of the transition distribution.
Let p ( θ ( t )) be the prior.
Let l u ( · ; θ, f p , f d )be the function inside the minimax optimization of Eq.(4).
Since both terms in l u can be treatedas negative log-likelihood, we directly combine their negation’s expectation with KL terms as theevidence lower bound (ELBO), which can be written as:
[Equation]
We further include in the ELBO the following additional loss term l de , which aims to learn the NNparameters involved in PFDE by attempting to satisfy the DE defined in Eq. [Equation]
[Equation]
In the following subsections, we will show how to parameterize the variables and its temporalderivatives involved in DE using neural networks.
Inference.
The key to reducing latency is to force the inference of variables involved in PFDE todepend on historical batch data only.
Specifically, suppose we are given up to i − 1 -th batch B 1: i − 1 ,we aim to infer the encoder distribution for each frame of the incoming batch through solving the
PFDE.
However, existing ODE numerical solvers fail to give the approximate solution as the initialor boundary values of the variables are computationally intractable.
Inspired by [25, 30], we adoptneural networks to approximate the solution of these variables, such that:
[Equation]
where f θ ∗ are neural networks ( implementations detail in the Supplement).
To process B 1: i − 1 , f θ ∗contains a recurrent neural network (RNN), which maintains hidden states to represent the historicalbatches, without need to repeatedly process the sequence from the start.
Notably, our solution doesn’trequire the absolute time stamps, e.g. to solve for the encoder distribution for the incoming batch at the first frame, we can set t = 1 .
Furthermore, we specifically design these neural networks as anon-linear function of time t , allowing all temporal derivatives required by Eq.
(6) to be computed bythe chain rule and evaluated by means of automatic differentiation [31].
Prior.
We use NNs to parameterize the mean m 0 ( t ) and variance s 0 ( t ) of the prior p ( θ ( t )) :
[Equation]
where f θ ∗ are neural networks (implementations detail in the Supplement).
We specifically use thecurrent batch B i to estimate the parameters of the prior during training.
Consequently, KL term in the
ELBO tends to encourage the posterior to extrapolate the future based on what has observed before.
It’s worth noting that this will not affect the latency as prior will be removed during testing.
3. Toward Temporal-domain-invariance for the Entire Time Line
Similar to particle filters, PFDE is also capable of capturing the latent time-varying distribution of the non-stationary streaming data.
Therefore, the solution of PFDE G ( t ) ={ m ( t ) , s ( t ) , { w ( t ) ( j ) } kj =1 } + ∞can be treated as a stochastic process that describes the generationmechanism of the non-stationary data.
To impose the discriminator to consider such inductive biasand encourage our model to generalize to unseen streaming data, we give a temporal-domain-invariantloss that is the expectation of the discriminator loss V d over the stochastic process G ( t ) :
[Equation]
However, the upper limit of integration in above loss function approaching infinity results in animproper integral which may not converge.
To overcome above challenges, we propose the followingtheorem which provides an analytical upper bound for the temporal-domain-invariant loss.
Theorem 2.
Suppose we are given an arbitrary function, π ( t ) with t ∈ [0 , + ∞ ) .
Let β = − e − t 2 /α 2 .
Let ϵ be a positive real constant and let h : [ − 1 , 0) → R be a continuous function.
There exists an
H : [ − 1 , 0) → [0 , + ∞ ) that satisfies an initial value problem:
[Equation]
, where h (
[Equation]
Such that as ϵ → 0 , we have:
[Equation]
Let π ( t ) be the integrand of the right-hand-side of the Eq.
(9), Theorem 2 (proof given in the
Supplement) states that we can construct a H : [ − 1 , 0) →
[0 , + ∞ ) that satisfies an initial valueproblem (IVP):
[Equation]
where h ( β ) can be written as:
[Equation]
where ϕ
= (cid:112) − log( − β ) ; (cid:101) x
( ϕ ) is the approximated data.
Note that we use (cid:101) z to denote the encoderoutput for the approximated data.
Then by Theorem 2, the upper bound of the temporal-domaininvariant loss, (cid:101) l i , can be written as:
[Equation]
11) where H ( − ϵ ) and H ( − 2 ϵ ) can be obtained by solving the IVP.
For instance, these two terms canbe approximated using the ODE numerical solvers such as the Euler method and the Runge-Kuttamethods [25].
In this work, we adopt Euler method to calculate these two terms, in which ϵ is set asthe step size of the Euler method.
Notably, by setting a suitable step size, (cid:101) x
( ϕ ) , where ϕ ∈ (0 , N )are required to calculate the bound; and thus (cid:101) x
( ϕ ) can be approximated by simply using linear
[Table]
[Figure]
[Table] [Table]
(cid:11)(cid:4)(cid:9) (cid:6)(cid:5)(cid:4)(cid:5) (cid:13)(cid:7) (cid:5) (cid:7) (cid:8) (cid:10) (cid:12) (cid:6)(cid:5)
[Figure]
ADDA
CIDA
DSAN
ECBNN
[Figure]
interpolation over the hidden representation of the current batch.
In summary, the final loss of ourmodel can be written as a minimax optimization problem:
[Equation]
(12) where λ de and λ i are importance weights; f θ = { f θ m , f θ s , f θ w } , f θ
= { f θ m 0 , f θ s 0 } .
Experiments
We evaluate ECBNN on both synthetic and real-world streaming data.
Specifically, we conductpreliminary experiments on low dimensional synthetic toy data (Growing Circle), and then study thegeneralization of ECBNN in handling high dimensional synthetic video dataset based on handwrittendigits (Streaming Rotating MNIST → USPS).
We finally evaluate our model on OuluVS2, a realisticmulti-view lip reading dataset.
(See the Supplement for the detailed data configuration of datasets)
4.1 Baselines
We compare ECBNN with state-of-the-art UDA methods including DANN [15], ADDA [16], CIDA[18], and DSAN [17].
Since these UDAs assume static domains, we extend them to handle streamingdata in our experiments through adopting recurrent neural network (RNN) in the encoder.
Forexample, we extend CIDA by using both the frame index and categorical domain label as the inputto the encoder and as target of the discriminator.
Notice that all above methods can only performoffline adaptation, while ECBNN is able to perform online adaptation by continuously producingencoder distributions.
We also compare our model with a typical online UDA method, ( EDA )
[11],which is able to perform online adaptation but requires collecting data batches for fine-tuning.
Forfair comparisons, all models have similar numbers of weight parameters, with the similar neuralnetwork architecture of encoder f c , predictor f p , and discriminator f d .
Following [18], λ d is chosenfrom { 0 . 2 , 0 . 5 , 1 . 0 , 2 . 0 , 5 . 0 } (see implementations details in the Supplement).
4.2 Toy Dataset
Growing Circle.
We conduct a preliminary study on a toy dataset, Growing Circle, for a binaryclassification task.
Fig.
2 shows the initial data samples at the 0 -th frame, in which a green sectorseparates the data on a circle into source and target domain.
Fig.
2 (b) shows the motion dynamicsof decision boundary and data, in which time ranges of source and target domain are indicated bydifferent colors.
We include 0 -th to 39 -th frame to source domain and 50 -th to 89 -th to target domain.
Results and Analysis.
We visualize the predicted classification boundaries generated by differentmethods at frames t = 50 and t = 60 , shown in Fig.
3 (more results in the the Supplement).
We cansee that ECBNN almost captures the ground truth (GT) decision boundaries at different time stamps,
[Table]
while other methods perform poorly when handlingdata with time-varying distributions.
To evaluate
ECBNN’s capability of continuously generating better distribution of model parameters from historicalstream data in real-time, we test our model by varyingthe length of the historical stream data ranging from 0to 50 frames on the testing set.
Fig. 2(c) shows the results for ECBNN and baselines.
ECBNN consistentlyoutperforms baselines.
Interestingly, ECBNN’s performance grows increasingly better with the accumulation of observed history, demonstrating ECBNN’spotential in continuously learning from streamingdata during real-time testing.
4.3 Streaming Rotating MNIST → USPS
Streaming Rotating MNIST → USPS.
We further evaluate ECBNN on high-dimensional streaming data.
To do this, we construct a synthetic video dataset based on MNIST and USPS:
Streaming Rotating MNIST → USPS.
The video sequences are generated from the rotating
[Figure]
MNIST and USPS handwritten digits with sinusoidal angular velocity.
We take the first 40 frames of the Streaming
Rotationg MNIST video sequence as source data, whoserotating angles range from 0 ◦ to 80 ◦ ; we take the 50-thto 90-th frames of the Streaming Rotationg USPS videosequence as target data, whose rotating angles range from110 ◦ to 190 ◦ ; we also take the 90-th to 110-th frames of the
Streaming Rotationg USPS video sequence to constructan additional Out-of-Domain (OOD) testing set.
Results and Analysis.
We report the frame-level classification accuracy on the source-testing set, the target-testingset and the OOD-testing set, from the source domain, thetarget domain and out-of-domain respectively, as shown in Table 1.
We can see that Source-Onlyachieves accuracy of 27.7%, which indicates this task is very challenging.
ECBNN achieves thebest accuracy of 60.9%.
It dramatically outperforms other methods, e.g. it outperforms the state-ofthe-art domain adaptation method (DSAN) by 14 . 1% absolute.
For domain generalization, ECBNNoutperforms the best-performing baselines by 5 . 5% absolute, demonstrating ECBNN’s potential oftemporal-domain-invariant representation learning in the entire timeline.
We further evaluate ECBNN’s ability of continuously generating better distribution of model parameters from high-dimensional stream data in the target domain.
We evaluate ECBNN and baselines on
Rotating USPS testing set and report the accuracy at each frame in Fig.
Remarkably, while handlingthe real-time stream, ECBNN achieves gradually better performance over time.
This demonstratesthat ECBNN is capable of continuously generating better encoder distribution over time based onhistorical data, whereby it becomes better and better at handling the data from target domain.
[Table]
OuluVS2 dataset was originally proposed by [32], which is, as far as we know, the only publicly available dataset that contains multi-view lip motion data.
It includes lipreading data recorded by 52 speakers from5 different views.
The train, valid and test sets include5250, 750 and 1800 sequences, respectively.
We adoptexperiment settings similar to [33], e.g. the sequencelevel class label is voted by class labels of all frames.
Toformulate the streaming domain adaptation scenes, weconsider the sequences from 0 ◦ view as the source domainand sequences from [30 ◦ , 45 ◦ , 60 ◦ , 90 ◦ ] as target domains.
Besides, each data batch contains 10 frames, and eachframe is a greyscale image of size 44 × 50 .
Quantitative evaluation of ECBNN on OuluVS2.
Table 2 shows the sequence-level classificationaccuracy of the baseline models and our new ECBNN on Test of OuluVS2.
We can see that ECBNNperforms the best among all models in terms of accuracy in both source and target domains.
ECBNNoutperforms EDA, the online adaptation baseline, by 5 . 0% absolute, while having remarkably lowerlatency in terms of generating model parameters.
We also report the detailed accuracy of various DAmethods on different views in the Supplement.
Qualitative evaluation of ECBNN on temporal-domain-invariant feature learning.
The production of lip motion data is governed by highly nonlinear dynamics over time, e
[Table]
g. the movementsof various articulators.
Therefore, aligning such data in terms of highly nonlinear temporal domain
[Table]
(cid:19)(cid:17)(cid:21) (cid:19)(cid:17)(cid:23) (cid:19)(cid:17)(cid:25) (cid:19)(cid:17)(cid:27) (cid:20)(cid:17)(cid:19) t= t= t=
Here,we aim to qualitatively evaluate whether
ECBNN can produce encoder distributionsthat achieves temporal-domain-invariantfeature learning for real data.
We randomly select data from 4 classes for boththe source domain ( 0 ◦ ) and the target domain ( 30 ◦ ), and use t-SNE to visualize thecorresponding encoder output z at timet = 9 , 15 , and 21 .
Fig.
5 shows the visualization.
Surprisingly, the alignment qualityincreases over time and features at t = 21 align much better than features at t = 9 .
This suggeststhat ECBNN can gradually improve the alignment given sequentially arriving data streams, therebygradually improving accuracy on the target streaming data over time.
Conclusion
Inspired by internal predictive modeling in neuroscience, this paper builds on the connection between
Bayesian neural networks and continuous-time stochastic dynamical system.
We derive a novelparticle-filter-based differential equation, thereby proposing extrapolative continuous-time Bayesianneural network (ECBNN), which generalizes existing Bayesian neural networks to represent temporaldynamics.
We then augment the unsupervised domain adaptation framework by adopting ECBNNas its encoder for handling non-stationary streaming data.
We further provide an upper bound withtheoretical guarantee for ECBNN to achieve temporal-domain-invariant representation learning.
Weshow that ECBNN gradually improves model performance over time with low latency during thereal-time testing stage.
Our current approach assumes continuous dynamics of NN parameters; weleave it for future work to be able to learn more complex discrete dynamics introduced by discreteevents.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] The Supplement
contains detailed instructions to reproduce the main experimental results and the code
will be released when the paper is accepted.
(b) Did you specify all the training details (e
g., data splits, hyperparameters,
how they
were chosen)?
[Yes] Please see more details in the Supplement.
(c) Did you report error bars (e
g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] Please see more details in the Supplement.
(d) Did you include the total amount of compute and the type of resources used (e
g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] Please see more details in the
Supplement.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, ifapplicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]