V ARIATIONAL C OMPONENT D ECODER FOR S OURCE E XTRACTION FROM N ONLINEAR M IXTURE
Anonymous authors
Paper under double-blind review
A BSTRACT
In many practical scenarios of signal extraction from a nonlinear mixture, only one (signal) source is intended to be extracted.
However, modern methods involving Blind Source Separation are inefﬁcient for this task since they are designed to recover all sources in the mixture.
In this paper, we propose supervised Variational
Component Decoder (sVCD) as a method dedicated to extracting a single source from nonlinear mixture.
sVCD leverages the sequence-to-sequence (Seq2Seq) translation ability of a specially designed neural network to approximate a nonlinear inverse of the mixture process, assisted by priors of the interested source.
In order to maintain the robustness in the face of real-life samples, sVCD combines Seq2Seq with variational inference to form a deep generative model, and it is trained by optimizing a variant of variational bound on the data likelihood concerning only the interested source.
We demonstrate that sVCD has superior performance on nonlinear source extraction over a state-of-the-art method on diverse datasets, including artiﬁcially generated sequences, radio frequency (RF) sensing data, and electroencephalogram (EEG) results.
I NTRODUCTION
Signal extraction from nonlinear mixture is a recurring yet hard research problem in signal processing and representation learning.
To tackle this problem, conventional methods can be divided into two categories.
One is Blind Source Separation (BSS) that recovers all sources in the mixture (Comon & Jutten, 2010), the other is to leverage available prior knowledge to extract only the desired components from the mixture (Leong et al., 2008).
Following the hardness results of nonlinear BSS based on nonlinear Independent Component Analysis (ICA) (Hyv ¨ arinen & Pajunen, 1999), many attempts have been made to realize nonlinear ICA and to further connect it with representation learning (Hyv ¨ arinen & Morioka, 2016; 2017; Khemakhem et al., 2020).
However, in many practical scenarios, only one or few sources are supposed to be retrieved for the concerned application; for example, heartbeat monitoring, especially that achieved by contact-free approaches (Ha et al., 2020), demands the heartbeat signal (waveform) to be extracted out of the nonlinear mixture with other signals such as body movements and respiration.
Compared with nonlinear BSS, nonlinear source extraction can be far more efﬁcient and effective under these scenarios: on one hand, extracting only the source of interest could signiﬁcantly reduce the computation cost for the same category of algorithms (e.g., deep-learning based signal extraction would be computationally cheaper than deep-learning based signal separation).
On the other hand, if prior knowledge of the source is available, the training may substantially leverage such knowledge so as to sufﬁciently improve the (later) inference accuracy.
Recently, there is a rising interest in investigating disentangled representation learning exploiting deep learning techniques.
Different from conventional source separation methods that demand hardcoded rules, a deep learning model aims to learn the underlying generating factors automatically and adaptively.
To this end, the frameworks of variational auto-encoder (VAE) (Kingma & Welling, 2013) and generative adversarial network (GAN) (Goodfellow et al., 2014) are widely used, and variants modifying designs and training techniques to enhance disentanglement have also been proposed, such as β -VAE (Higgins et al., 2016).
However, as the existing proposals mostly focus on disentangling on the representation in image or video data (Kingma & Welling, 2013; Higgins et al., 2016; Fabius & Van Amersfoort, 2014; Yingzhen & Mandt, 2018) where signal mixture never take place, we cannot directly apply the idea of disentangled representation learning to our problem of [Figure]
SourcesSignals
ObservedSignals 𝐱
Encoder 𝑞 # = 𝑝(𝐳|𝐱)
Decoder 𝑝 * = 𝑝(𝐲,|𝐳)
TargetSignal 𝐲
ExtractedSignal 𝐲, ~ … … t t n 𝐳~𝑁(𝝁,𝚺 2 ) source extraction (a special form of disentanglement) from nonlinear time-series mixture.
Though a few results have been obtained for disentangling the sources in nonlinear mixing using deep learning, they fall into the category of BSS (Hyv¨arinen & Morioka, 2016; Khemakhem et al., 2020).
In this paper, we propose supervised Variational Component Decoder (sVCD) as a robust deeplearning framework for source extraction from nonlinear mixture.
As illustrated in Figure 1, sVCD aims to recover the interested source y from observed signals x formed by source signals mixed in a nonlinear manner.
In particular, sVCD involves an encoder to analyze the temporal features of the observed signals and hence to encode them to the hidden state (posterior distributions in a latent space), and it then decodes the hidden state to recover the interested source.
A modiﬁed variational inference is applied to help approximate the intractable posterior and to achieve a disentangled representation in the latent space dedicated to the interested source, guided by the prior knowledge applied to train sVCD.
Essentially, while y is available for supervised training, sVCD takes x as the sole input during the testing phase to obtain ˆ y .
To summarize, our main contributions include:
• We prove that nonlinear source extraction can be formulated as modiﬁed variational inference and achieved by maximizing a tighter variational lower bound.
• We design a novel sequence-to-sequence (Seq2Seq) translation architecture; it leverages the modiﬁed variational inference to extract the desired source from nonlinear mixture.
• We provide empirical evidence that sVCD outperforms a state-of-the-art method based on extensive evaluations under different nonlinear mixing scenarios.
The rest of the paper is organized as follows.
Section 2 presents related works.
Section 3 introduces the mathematical framework for source extraction from nonlinear mixture.
Section 4 presents the details sVCD model.
Section 5 introduces the datasets and reports the evaluation results.
Finally,
Section 6 concludes this paper and points out potential future applications.
R ELATED W ORKS
Numerous algorithms have been developed to solve the BSS problem (Comon & Jutten, 2010): they all aim to decompose signal mixtures into individual components without knowing information about the source signals or the mixing process.
Among them, Non-negative Matrix Factorization (NMF) decomposes the signal by using low-rank approximations under the non-negative constraint (Wang & Zhang, 2012).
Independent Component Analysis (ICA) assumes that the underlying signals are statistically independent to separate them (Hyv¨arinen & Oja, 2000).
BSS achieved by masking and re-weighting the frequency spectrograms have also been implemented by Hidden
Markov Models (HMMs) (Roweis, 2000) and segmentation methods (Bach & Jordan, 2005).
One common assumption of regular BSS is that the mixing process is linear, under which the statistical independence of the underlying sources is a sufﬁcient condition to constrain the linear unmixing function.
However, as we extend BSS to nonlinear mixture, the statistical independence of the sources can no longer be treated as a sufﬁcient constraint for the unmixing (Hyv¨arinen & Pajunen, 1999).
Harmeling et al.
and Hyv¨arinen & Morioka (2016) have proposed to utilize temporal structure in the mixture to overcome this challenge.
Furthermore, several properties on the mixed sources, e.g., autocorrelation (Sprekeler et al., 2014), general non-Gaussian temporal dependency (Hyv¨arinen & Morioka, 2017), or non-stationarity (Hyv¨arinen & Morioka, 2016), can also provide sufﬁcient constraints for solving the problem.
Aother broadly related topic is disentanglement representation learning (Bengio et al., 2013).
The word “disentanglement” means decoupling of generating factors.
Unlike BSS performing separation in signal space , disentanglement representation learning uncovers the underlying factors of variation in latent space .
With the development of deep learning, disentanglement representation learning based on VAEs has gained momentum (Kingma & Welling, 2013).
By augmenting the lower bound formulation with the coefﬁcient that regulates the independence prior, Higgins et al.
(2016) proposes β -VAE, a framework to discover interpretable latent representations automatically.
Subsequent works (Burgess et al., 2018; Rolinek et al., 2019; Chen et al., 2018) have led to signiﬁcant accomplishments in understanding the capabilities of nonlinear disentanglement in VAEs.
Researchers have also been seeking the formal deﬁnition of disentanglement in several works (Ridgeway, 2016;
Higgins et al., 2018; Eastwood & Williams, 2018).
The aforementioned algorithms assume that the sources are completely unknown, however, in many practical cases, we have some prior knowledge available and do not actually have to perform source separation in a completely blind manner.
As such, these algorithms can be extended to incorporate the knowledge from the priors.
In Calhoun et al.
(2005), paradigm information was employed to aid
ICA analysis, and the results are shown to be more robust to noise and outperform basic ICA.
In another work (Jeong et al., 2009), the original NMF algorithm for BSS is signiﬁcantly improved by enforcing a constraint that requires disjointness under the semi-blind denoising framework.
Last but not least, Kameoka et al.
(2018) achieves source separation by leveraging conditional VAE (Sohn et al., 2015) with source class as labels in priors during training.
All the aforementioned approaches do not seem to suit our need for extracting speciﬁc source(s).
V ARIATIONAL I NFERENCE FOR S IGNAL E XTRACTION
Unlike VAE (Kingma & Welling, 2013) that aims to recover input by ﬁnding an efﬁcient encoding in latent space, our sVCD focuses on extracting a speciﬁc component from nonlinear mixture.
Therefore, if we directly borrow the idea of variational inference from VAE, the variational lower bound will not be as tight.
As such, we propose and prove a new lower bound for our problem setup, so as to endow a new mathematical framework for sVCD.
Let x be a nonlinear mixture of several independent sources.
In order to ﬁnd a representation of x ,
Bayesian modeling (Lee, 1989) can be used to encode the beliefs about the processes that generate x from all components into a model M with latent vectors z and parameters θ M .
The model is then “learned” by inferring z and adjusting the parameters θ M .
In other words, given the observed mixture x , Bayesian modeling aims to infer the posterior p ( z | x ) .
However, since the posterior distribution is intractable, an optimization approach, which is called variational inference (Blei et al., 2017), can be leveraged to approach the problem.
To be speciﬁc, a surrogate distribution q ( z ) is employed to approximate p ( z | x ) by minimizing their KL divergence (Van Erven & Harremos, 2014).
However, the KL divergence itself still involves the intractable posterior p ( z | x ) .
To cope with this issue, the KL divergence is decomposed into:
[Equation]
Since the marginal log-likelihood log p ( x ) is independent of the variational distribution q ( z ) , the
KL divergence can be minimized by maximizing the variational lower bound E q ( z ) (cid:104) log p ( x , z ) q ( z ) (cid:105) .
As a particular type of variational inference method, VAE uses deep neural networks to approximate the generative models.
It should be noted that in VAE, the distribution q is conditioned on the observation x , and approximated by an encoder network q ( z | x ) .
As such, the variational lower bound of VAE can be expressed as:
[Equation]
By slight modiﬁcations (Higgins et al., 2016; Kim & Mnih, 2018), the latent representation z of VAE shows disentanglement properties, suggesting that a regular VAE can be used for separating mixed sources.
However, unlike the scenario faced by VAE which is unsupervised and source-agnostic, our task is not completely “blind”, i.e., prior knowledge about a speciﬁcally interested component y is available.
In this case, the variational lower bound should be E ˜ q
( z | y ) (cid:104) log p ( y , z )
˜ q ( z | y ) (cid:105) , where ˜ q ( z | y ) is an approximate posterior distribution in the form ˜ q ( z | y ) =
(cid:82) p ( z | x ) q ( x | y )
d x .
It can be observed that the posterior ˜ q ( z | y ) takes into account the effect of the mixing process q ( x | y ) , thus covering a broader class of distributions and becoming more expressive.
We also notice that there is still the desired component y in the denominator.
Considering the goal of extraction signal component from a nonlinear mixture, we would like to maximize E
[Equation]
q ( z | y ) (cid:104)
[Equation]
( y , z ) q ( z | x ) (cid:105) instead, in which case x and y should be used as the input and output of the network, respectively.
Therefore, we need to study this new lower bound speciﬁcally.
Lemma 3.1.
We get a tighter lower bound for the source extraction problem:
[Equation]
Proof.
For the left inequality, by using Jensen’s inequality, we have:
[Equation]
For the right inequality, by using Jensen’s inequality, we have:
[Equation]
As a result, we can optimize the surrogate lower bound E ˜ q ( z | y ) (cid:104) log p ( y , z ) q ( z | x ) (cid:105) of sVCD, which is tighter than that of VAE, for the task of source extraction.
This variational inference can be readily implemented by the sVCD network in Section 4.
S UPERVISED V ARIATIONAL C OMPONENT D ECODER A RCHITECTURE
To implement the variational inference framework for extracting signals from nonlinear mixture as presented in Section 3, we speciﬁcally design sVCD network to take observed nonlinear signal mixtures (containing multiple components) as input and to output the temporal sequence of the desired component.
It inherits from encoder-decoder architecture with an encoder and decoder, both based on Recurrent Neural Network (RNN) (Cho et al., 2014), sandwiching a latent space.
The architecture of sVCD is shown in Figure 2, and we elaborate the details in the following.
4. E NCODER
We employ an RNN-based encoder to implement the q ( z | x ) of the variational inference as introduced in Section 3.
The encoder is parameterized by φ .
As demonstrated in Figure 3a, the RNNbased encoder q φ ( z 1: T | x 1: T ) of sVCD takes in the temporal signal x 1: T =
[ x 1 , · · · , x t , · · · , x T ] of length T , where x t is abbreviated for x t ∈ R N , N represents number of observed signal mixtures.
The encoder takes the responsibility of decomposing and selecting the desired component, which is in turn encoded to a compressed latent distribution z .
Samples drawn from z are then used to drive the decoder for recovering the desired component.
We use RNN to form the encoder as it is naturally suitable for sequence modeling.
At each timestamp t , the RNN reads the symbol x t and previous hidden state denoted as e t − 1 that summarizes the past information, and updates the new hidden state e t .
In general, RNNs model sequences by parameterizing a factorization of the joint sequence probability distribution as a product of conditional probabilities such that:
[Equation]
where g τ is a function that maps the RNN hidden state e t − 1 to a probability distribution over possible outputs, and τ is the parameter set of g .
RNN can be implemented with gated activation functions for longer dependecies, such as Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) or Gated Recurrent Unit (GRU) (Cho et al., 2014).
In this paper, we apply bidrectional GRU to for sequential learning.
Given an input sequence x 1: T to the bidirectional GRU, we obtain the encoded features e 1: T =
[ −→ e 1 , ←− e T ; · · · ; −→ e T , ←− e 1 ] .
Input signal 𝒙 1:T
Linear
Linear
Linear 𝒒 1:𝑇 𝒌 1:𝑇 𝒗 1:𝑇
Attention 𝒄 1:𝑇
Encoder GRU 𝒂 1:𝑇
Linear 𝒆 1:𝑇
Linear
Sample 𝝁 1:𝑇 log 𝜮 21:𝑇
Linear
Decoder GRU 𝒛 1:𝑇 𝒉 1:𝑇 𝒅 1:𝑇
Linear
Output signal 𝒚 1:𝑇
We let the prior distribution z t ∼ N ( µ t , Σ 2 t ) be a Gaussian with a diagonal covariance matrix, whose mean and logvariance are parameterized by neural networks (usually implemented by fully connected layer), and the values depend on deterministic encoded features e t :
[Equation] [Equation]
Then z t is sampled from mean and variance by reparameterization tricks (Kingma & Welling, 2013).
We denote the encoder parameters for computing the input features as φ x and the parameters for computing the mean and variance as φ e , we can model encoder as:
[Equation]
As such, the encoder has transformed the input temporal sequence to distributions on the latent space, which can be then readily sampled to drive the decoder for recovering the desired component y .
4. A TTENTION M ECHANISM
Although RNN introduced in Section 4.1 is a good way to handle sequential data, they may still lack the ability to discover very long dependencies across thousands of steps.
To solve this problem, we employ the scaled dot product attention mechanism (Vaswani et al., 2017) to add more contexts to the temporal sequence; the modiﬁed encoder architecture is shown in Fig.
3b.
The core idea of attention is to focus on the most relevant parts of the input sequence for each output.
An attention function can be described as mapping a query and a set of key-value pairs to an output.
Speciﬁcally, the query q 1: T , key k 1: T , and value v 1: T can be deemed as modiﬁed versions of the input x 1: T by three different linear transformations as follows:
[Equation]
[Equation]
12) where W and b (with respective superscripts) are parameters that are trained to transform the input to its corresponding query q 1: T , key k 1: T , and value v 1: T , whose dimensions are denoted by d q , d k , d v .
The output context c 1: T is obtained by calculating a weighted sum of the values, where the weight of each value is a normalized product of the query and its corresponding key.
Then the context is concatenated to the query to form the ﬁnal output of our attention layer.
In summary, the attention mechanism can be expressed as follows:
[Equation] [Equation]
where ⊕ represents concatenation.
Compared with its input x 1: T , the output of the attention layer c 1: T contains more contexts of the whole signal by taking into account the interactions among the elements, thus forming a better input for the ensuing RNN.
𝑥 𝑡− 𝑥 𝑡 𝑥 𝑡+ 𝑒 𝑡− 𝑒 𝑡 𝑒 𝑡+ 𝑧 𝑡− 𝑧 𝑡 𝑧 𝑡+ (a) Encoder.
𝑎 𝑡− 𝑎 𝑡 𝑎 𝑡+ 𝑒 𝑡− 𝑒 𝑡 𝑒 𝑡+ 𝑥 𝑡− 𝑥 𝑡 𝑥 𝑡+ 𝑧 𝑡− 𝑧 𝑡 𝑧 𝑡+ (b) Encoder with self-attention.
𝑦 𝑡− 𝑦 𝑡 𝑦 𝑡+ 𝑑 𝑡− 𝑑 𝑡 𝑑 𝑡+ ℎ 𝑡− ℎ 𝑡 ℎ
𝑡+ 𝑧 𝑡− 𝑧 𝑡 𝑧 𝑡+ (c) Decoder.
4. D ECODER
To realize p θ ( y | z ) in the variational inference problem discussed in Section 3, we implement a decoder using RNN-based network, as shown in Figure 3c.
The decoder p θ ( y 1: T | z 1: T ) is parameterized by θ , and serves to reconstruct the source of interest y 1: T from a latent vector z 1: T .
The decoding consists of three steps.
Firstly the latent vector z 1: T is expanded to a hidden feature dimension by a fully connected layer:
[Equation]
(15) with W h and b h being the weight matrix and bias represented by the linear layer.
Then h is sent to the sequence decoding layer implemented by a forward directional GRU that produces output d .
Finally, we have the decoded output sequence d 1: T =
[ −→ d 1 , · · · , −→ d t , · · · , −→ d T ] , where feature at each step is connected to a two linear layers with rectiﬁed linear units (ReLU) to the dimension of y 1: T .
[Equation]
16) where W y 1 , W y 2 , b y 1 , and b y 2 are the weights and biases of the linear layer.
Overall we have the decoder represented as: p θ ( y 1: T | z 1: T ) =
T (cid:89) t = p θ h ( h t | z t )
T (cid:89) t = p θ d ( d t | h <t )
T (cid:89) t =
p θ y ( y t | d t ) .
4. L OSS F UNCTIONS
As stated in Section 3, we maximize the variational lower bound E ˜ q
( z | y ) (cid:104) log p ( y , z ) q ( z | x ) (cid:105) in Equation (3) to solve the variational inference problem.
Since we have parameterized the probabilities with deep neural networks, we rewrite the variational lower bound as E ˜ q
( z | y ) (cid:104)
ln p θ ( y , z ) q φ ( z | x ) (cid:105) , and it can be decomposed as:
[Equation]
This lower bound can be separated into two parts, the ﬁrst term is a reconstruction error between the sVCD output and the ground truth, and the second term as a regularizer which enforces the approximate posterior to be close to the speciﬁed prior.
To practically implement the reconstruction loss E q φ ( z | x ) [ln p θ ( y | z )] , we use the L 2 distance between the separated component ˆy and y :
[Equation] [Equation]
Since the prior distribution is deﬁned to be unit Gaussian, the second term can be practically implemented by:
[Equation]
And the loss function of sVCD can be deﬁned as:
[Equation]
where λ is the weight controlling the strength of the regularization.
E VALUATION
In this section, we ﬁrst report experimental results on three datasets for sVCD compared with stateof-the-art baselines.
We then perform an ablation study on sVCD by removing its certain parts, in order to gaining more understanding of the network.
5. D ATASETS FOR N ONLINEAR M IXTURES
Traditional datasets for BSS, e.g., those created for audio source separation (Garofolo et al., 1993;
Vincent et al., 2007) are mostly linear.
To demonstrate the nonlinear mixture separation efﬁcacy of sVCD, we employ three datasets of nonlinear mixtures, with details elaborated in the following.
Artiﬁcially Generated Dataset One artiﬁcially generated dataset is employed in our evaluation due to its special statistical properties.
For this dataset, both source signals and observed signals are generated according to the settings in (Hyv ¨ arinen & Morioka, 2017).
All source signals are randomly produced from an AutoRegressive (AR) model with a non-Gaussian process.
Then, we obtain nonlinear mixtures from the source signals leveraging a nonlinear mixing function comprised of a multi-layer perceptron (MLP) with nonlinear activation functions, i.e., leaky ReLU units.
For each of the 4,000 trials, we leverage the model to generate 12 temporally dependent sources (independent of each other) that are in turn mixed to produce 12 observed signals; we select one as source of interest to be extracted by sVCD.
RF
-PPG Dataset Contact-free vital sign sensing is the problem of extracting vital signs of human
subjects via contact-free sensors, a representative and practical scenario for signal extraction from the nonlinear mixture.
To be speciﬁc, the micromovements of the skin caused by heartbeats are mixed non-linearly with respiration-induced movements and other body movements, and the mixture is carried on RF signals reﬂected back from human bodies.
In this dataset, a high-resolution radar sensor (Novelda AS, 2017) is used to collect the RF signals.
The ground truth heartbeat photoplethysmography (PPG) waveforms are measured by NeuLog (2017).
The data collection is conducted with 12 healthy subjects (6 females and 6 males) under the IRB approval of our institute.
The subjects are asked to sit at quasi-static and natural states.
The RF-PPG dataset includes 48-hour data, approximately 180,000 heart cycles in total, and is divided into 20-seconds segments.
EEG-EOG Dataset Electroencephalography (EEG) is an electrophysiological monitoring technique for recording electrical activity on the scalp, which has been proven to reﬂect the activity of the brain’s surface layer underneath (Henry, 2006).
Due to the sophisticated structure of the brain, the signals at different locations of the underneath brain (whose ground truths cannot be obtained directly), when manifested as EEG signals, are often superimposed nonlinearly and exhibit complex behavior (Steyrl et al., 2013), and the nonlinearity is escalated by the nonlinear properties of the brain (Nunez et al., 2006) and the skull (Zhang et al., 2014).
Moreover, the electrical activity of the eye, which is the main source of artifacts of brain signals, complicates the EEG recordings.
As such, it is often required to isolate and remove electrical signals of the eyes, i.e., electrooculography (EOG) signals, from EEG recordings.
In the collected dataset (Torkamani-Azar et al., 2020), EEG and EOG signals are recorded simultaneously to form 2.5-minute segments.
The EEG signals have 64 channels corresponding to electrodes mounted on the scalp, and the EOG signals have 3 channels corresponding to electrodes connected to the outer corners of both left and right eyes,
5. E XPERIMENT S ETUP
We employ the aforementioned datasets to evaluate the performance of sVCD.
The collected datasets are divided into 80% training and 20% test sets.
For the model parameters, we set the size for d q , d k and d v as 64; e , z , h , d are set respectively as: 64, 8, 128, 128; and dropout rate at 0.2.
The weight λ is set to 0.1 by line search.
For the training process, we employ the Adam optimizer (Kingma &
Ba, 2014) with a starting learning rate of 0.001 to update the parameters according to equation 21.
The batch size is set to 20.
To enable proper evaluation, all signals are scaled to [ − 1 , 1] beforehand.
To evaluate the performance of the method we calculate the cosine similarity between the recovered component ˆ y and the ground truth component y .
The cosine similarity is deﬁned as S (ˆ y , y ) =
[Equation]
t , with value in the range of [ − 1 , 1] .
A high similarity indicates that better performance of the model, and vice versa.
To compare sVCD with previous works, we choose ICA (Hyv¨arinen & Oja, 2000) and ConvTasNet (Luo & Mesgarani, 2019) as baseline methods; they are both designed for separating linear signal mixture.
ICA is chosen because it is a widely used blind source separation technique, and the ﬁtting of ICA and the training of sVCD are similar since both consider the context of the signal including both the past and the future.
Conv-TasNet is chosen as the baseline because it is a stateof-the-art deep-learning method for automatic signal separation.
We follow the parameter settings in the original papers, and for ICA, we separate 12, 5, and 64 independent components from the artiﬁcially generated, RF-PPG, and EEG-EOG datasets, respectively.
5. R ESULTS
In Figure
[Figure]
we show the waveforms of the ground truth
[Figure]
and the signals extracted by ICA, ConvTasNet (abbreviated as C-TasNet) and sVCD signal on the three datasets.
For the artiﬁcially generated data, sVCD retrieves a component approximately the same as the ground truth, while ICA and Conv-TasNet obtain very noisy results.
For the RF-PPG data, sVCD successfully recovers the heartbeat waveform bearing correct heart rate, while ICA barely separate heartbeat signal from respiration as the breath waveform is still dominating, and Conv-TasNet fails to recover event details in the waveform.
For the EEG-EOG dataset, sVCD extracts a source signal bearing high similarity to the ground truth, while ICA and Conv-TasNet seem to have deduced totally wrong outcomes.
[Figure] [Figure]
o s i n e
S i m il a r it y
[Figure]
In Figure 5, we show the cosine similarities between the recovered waveforms and the ground truth of ICA, Conv-TasNet, and sVCD.
It can be observed that the performance of sVCD substantially surpasses those of ICA and Conv-TasNet on all three datasets, and the median cosine similarities of sVCD are all above 0.8 on all three datasets, indicating a strong resemblance between the sVCD extracted source signals and the ground truth.
5. A BLATION S TUDIES
We provide an ablation study of the main design decisions for sVCD in this Section.
In the ﬁrst study labeled as “Seq2Seq”, we skip the variational inference part, and make the output of the encoder
GRU directly driving the decoder GRU.
Meanwhile, in the second study “sVCD with only mean regularization”, instead of regularizing the distributions to the standard Gaussian, we regularize only the mean value towards 0.
Lastly, the study labeled as “sVCD without attention” holds out the attention mechanism.
Our goal is to see the respective accuracy drops to measure their importance in the proposed sVCD network.
Given the cost of training a single model, we did not compute conﬁdence intervals for each variation.
The result of the ablation study is shown in Table 1.
We can observe that our proposed sVCD achieves cosine similarities of 0.9358, 0.8634, and 0.
on the three datasets, respectively.
Performance drops can be observed for the Seq2Seq model on Artiﬁcially generated, RF-PPG and EEG-EOG datasets, indicating that a discrete latent space without regularization encodes less useful information that can be used by the decoder to recover the source.
For the sVCD with only mean regularization, further performance decrease can be observed, proving that the distribution mean and variance should better off be regularized together.
As for the sVCD without attention, some clear drop in cosine similarity can be observed on all three datasets, demonstrating that the attention mechanism helps boosting the performance of sVCD.
[Table]
C ONCLUSION
Taking an important step toward deep learning enabled signal processing, we proposed a new approach for source extraction from nonlinear mixture.
We started by formulating nonlinear signal extraction as a modiﬁed variational inference problem.
By detailed derivation, we have proved that the mathematical framework of signal extraction can be implemented by an sVCD network, in which the modiﬁed variational lower bound is maximized using backpropagation.
Experiments on artiﬁcially generated data, RF-PPG data, and EEG-EOG data have proved the signal extraction efﬁcacy of sVCD.
The sVCD in this paper is used as a vehicle to validate the overall mathematical framework, but better architectures are likely to be found.
In the future, we plan to apply the sVCD network to other promising applications such as source extraction from video data, econometric data, and biomedical data (such as EMG and ECG), in which source signals have nonlinear mixtures.
E THICS S TATEMENT
All experiments involve human subjects have been approved by IRB of our institute, and the experiments have been conducted strictly followed the standard procedures.
The RF-PPG data will not be made available to the public to protect the privacy of the subjects.
R EPRODUCIBILITY S TATEMENT
In this section we brieﬂy reference the details needed to help reproducibility.
For the dataset: we have uploaded the artiﬁcially generated dataset in the link in our supplementary materials; due to privacy concerns we could not make the RF-PPG dataset to public; the EEG-EOG dataset is a public dataset acquired from Torkamani-Azar et al.
The model implementation, settings are described as in Section 5.2.
We have also uploaded the source code to allow for reproducing our results in supplementary materials.
A E XPERIMENTS ON L ENGTH OF S EQUENCE
As sVCD is built on RNN structure, there is no strict requirement on the length of input sequence.
We provide an experiment on the adaptability of sVCD by training with sequence length of L , then testing with different lengths from 2 L to 0 .
01 L .
For the three datasets mentioned in Section 5.1,
L is 800 time steps which equivalent to: 16 seconds for artiﬁcially generated data, 20 seconds for RF-PPG data, 4 seconds for EEG-EOG data.
The cosine similarities are presented in Figure 6.
It can be observed that generally tested with sequence lengths from 0 . 5 L to 2 L , resulting statistics do not have much variation for all three datasets.
For artiﬁcially generated dataset, test data length with only 0 . 01 L still has median of similarity above 0.9, but much larger error can be observed.
For RF-PPG dataset, test data that have length lower than 0 .
4 L have similarities much lower than those longer. While for EEG-EOG dataset, test data length lower than 0 . 5 L will make the performance drops. Different minimum sequence length of test data may be due to different source signals have distinct time-correlation features.
Therefore, it is better to take a sufﬁciently long sequence as it is hard to judge the minimum yet sufﬁcient length a priori.
Overall, the results show a promising future of sVCD that after training with an adequate length, the model could adapt to not only short time observation but also long time monitoring without extra training efforts.
[Figure]
B E XPERIMENTS ON L INEAR AND N ONLINEAR R EGRESSIONS
We have performed linear and nonlinear regression for the signal extraction purpose.
For nonlinear regression, we employ least square regression and support vector regression.
For each category of nonlinear regression, we employ polynomial, radial basis function and sigmoid kernel.
For nonlinear regression, we only report the results of support vector regression with polynomial kernel because this combination delivers the best performance.
The recovered waveforms of linear regression (denoted as “LinReg”), nonlinear regression (denoted as “NLinReg”), and sVCD are shown in Figure
[Figure]
and their respective cosine similarities are shown in Figure 8.
It can be observed that sVCD signiﬁcantly outperforms these two regression methods, while both regressions fail to recover meaningful results.
The reason for their inferior performance can be readily explained by their omission of temporal correlations
[Figure] but they do offer instantaneous point-to-point conversion instead of the seq-to-seq manner operated by sVCD [Figure] [Figure]