D OMAIN A DAPTATION VIA D ISTRIBUTION AND R EPRESENTATION M ATCHING : A C ASE S TUDY ON T RAINING D ATA S ELECTION VIA R EINFORCEMENT L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Supervised models suffer from domain shifting where distribution mismatch across domains greatly affect model performance.
Particularly, noise scattered in each domain has played a crucial role in representing such distribution, especially in various natural language processing (NLP) tasks.
In addressing this issue, training data selection (TDS) has been proven to be a prospective way to train supervised models with higher performance and efﬁciency.
Following the TDS methodology, in this paper, we propose a general data selection framework with representation learning and distribution matching simultaneously for domain adaptation on neural models.
In doing so, we formulate TDS as a novel selection process based on a learned selection distribution from the input data, which is produced by a trainable selection distribution generator (SDG) that is optimized by reinforcement learning (RL).
Then, the model trained by the selected data not only predicts the target domain data in a speciﬁc task, but also provides input for the value function of the
RL, which exactly helps update the parameters of SDG in a policy gradient way.
Experiments are conducted on three typical NLP tasks, namely, part-of-speech tagging, dependency parsing, and sentiment analysis.
Results demonstrate the validity and effectiveness of our approach.
I NTRODUCTION
Learning with massive data suffers from “Pyrrhic victory” where huge amounts of resource, e.g., computation, annotation, storage, etc., are consumed with many issues.
One well-known issue for data-driven approaches is that data quality considerably affects the performance of learned models.
Especially in NLP, such phenomenon is incredibly signiﬁcant where noise and inaccurate annotations are demolishing models’ robustness when applying them to various scenarios (Akopyan & Khashba, 2017; Zhang et al., 2016), one of which is cross-domain NLP tasks.
In this scenario, many data and their labels are considered noisy and inaccurate when they are used across domains, resulting in the mismatch of distribution between training and test data.
To reduce the impact of the noise in data and annotations, reasonable selection of training samples is proved to be an useful strategy that can help supervised models achieving equivalent performance with less computational efforts (Fan et al., 2017; Feng et al., 2018).
Particularly, it is also conﬁrmed that appropriate data selection strategy can beneﬁt domain adaptation (Ruder & Plank, 2017) by preventing negative transfer (Rosenstein et al., 2005) from irrelevant data and noisy labels, meanwhile providing a more efﬁcient learning process compared with other domain adaptation methods such as sample reweighing (Borgwardt et al., 2006), feature distribution matching (Tzeng et al., 2014) and representation learning (Csurka, 2017).
Although various data selection-based domain adaptation approaches were proposed for NLP tasks (III, 2007; Blitzer et al., 2007; Søgaard, 2011), most of them only consider scoring or ranking learning samples under a certain metric over the entire dataset, then select the top n (or a proportion, which is usually a predeﬁned hyper-parameter) items to learn.
However, such pre-designed metrics is not always able to cover effective characteristics for transferring domain knowledge and meanwhile suffers from their inﬂexibility.
Even though some methods learn measures across domains and models (Ruder & Plank, 2017), their hyper-parameter setting, such as the n , still demands further explorations.
Fortunately, recent studies conﬁrmed that RL is an effective solution for data selection in training neural networks with improving their efﬁciency (Fan et al., 2017) and
accuracy (Feng et al., 2018).
A natural extension of using RL for data selection can be considered for the cross-domain scenario.
Yet, to apply RL to data selection for domain adaptation is not a straightforward mission as those
RL application for in-domain tasks.
To tackle this mission, in this paper, we propose a general data selection approach for domain adaptation via RL that jointly learns data representations and matches them from the source and target domain.
There are two major components in our approach: a selection distribution generator (SDG) and a task-speciﬁc predictor with a feature extractor and a classiﬁer 1 .
The SDG and the predictor are pipelined by taking each others’ output as input and optimized accordingly via RL.
As a result, in our approach, two major problems in domain adaptation are addressed: some redundant or noisy data across domains gives little or negative contribution to adaptation; useful information from the source domain is usually not properly organized and exploited for the target domain.
Experimental results from three NLP tasks, namely, part-of-speech (POS) tagging, dependency parsing and sentiment analysis, illustrate that our approach achieve state-of-the-art performance, which conﬁrm the validity and effectiveness of our approach.
T HE A PPROACH
Our approach is in the line of the unsupervised domain adaptation (Roark & Bacchiani, 2003), where one has no access to any labels of target domain data.
Particularly, for a task T , we take labeled data instances for the task from a source domain D S as the input to our approach, and some unlabeled data from a target domain D T as the guidance.
The approach is then expected to generate an optimal selected data set and a model trained on the set for T .
An overview illustration of the entire architecture for our approach is demonstrated in Figure 1.
In detail, for each step, the key component for selection, SDG, produces a distribution vector based on the representation of the selected source data from last step, then data instances are selected according to the vector and new reward is generated for next round data selection.
To update the SDG, different measurements can be used to assess the discrepancy between the representations of the selected source data and the guidance set and then approximates the value function for updating.
For the predictor, the loss from the classiﬁer on the domain-speciﬁc task is used to guide the parameter updating for the feature extractor and the classiﬁer.
The details is unfolded in the following subsections, in which we ﬁrstly give the details of the two major components and then the joint training algorithm.
2. T HE P REDICTOR
The predictor is the main component to perform a particular task T .
In our approach we decompose the predictor into two parts, the feature extractor and a classiﬁer, and use them separately.
The feature extractor serves as the representation learning module that transform input data to dense representations, while the classiﬁer takes the representations for T in the target domain.
In doing so, the predictor is a neural model in this study so that the aforementioned separation can be done.
Therefore, the feature extractor is normally the ﬁrst n-1 layers of the predictor given that it has n layers in total.
As a result, the classiﬁer is the last layer dedicated for model output.
The Feature Extractor Data in its original form, especially as natural language, is usually difﬁcult to be exploited in neural computation.
The feature extractor is thus serving as a critical component in our approach to transform the data into distributed representations for their efﬁcient use and computation.
There are two-way inputs for the feature extractor.
One is the guidance set X
Tg = { x T 1 , x T 2 , ..., x Tm } , a collection of unlabeled data drawn from the target domain, serving as the reference for the selection process.
The other input is the selected data from the source domain in a “data bag”, which is a batch of a certain amount of samples to facilitate the selection in this study.
Let X S
= { x S 1 , x S 2 , ..., x Sn } , ∀ x Si ∈ X S denote the data from the source domain, we uniformly partitions the entire data set into N disjoint data bags marked as { B 1 , B 2 , ..., B N } , with
B j = { x S ( j − 1) n/N +1 , x S ( j − 1) n/N +2 , ..., x Sjn/N } , j ∈ { 1 , 2 , ..., N } .
Through the feature extractor, the outputs of the guidance set and the selected data are two collections of distribution vectors.
The Classiﬁer Once data selection is done, the classiﬁer is combined with the feature extractor and trained on the selected data for a particular task.
Note that the training is conducted in the source domain and the output of the classiﬁer is used to generate gradients for back-propagation.
Then the parameters of the classiﬁer and the feature extractor are updated accordingly (with a learning rate β ).
2. T HE S ELECTION D ISTRIBUTION G ENERATOR
A multi-layer perceptron (MLP) model is used as the SDG, which learns the selection policy optimized by the reward from the representations of the guidance set and the selected data by RL.
In doing so, at each time step, the SDG is fed by a collection of data representations for a data bag from the feature extractor.
We denote the collection as Φ B j = { r j 1 , r j 2 , ..., r j | B j | } , where r jl
( l = 1 , 2 , ..., | B j | ) is the vector of the l -th sample 3 in B j .
Then Φ B j is mapped by the SDG to a vector D B j = ( p j 1 , p j 2 , ..., p j | B j | ) , p jl ( l = 1 , 2 , ..., | B j | ) , which represents the probability for each sample that whether it should be selected individually.
To learn the SDG, each Φ B j is measured with Φ t to give the reward in an RL framework, which is described as follows.
RL Components in Learning the SDG
• State ( s 1 , s 2 , ...
s j , ...
s N ) includes a collection of states for all j with respect to N data bags, where each s j indicates a status including data bag ˆ B j and parameters of the feature extractor for the selected instances ˆ B j from B j .
For simplicity we use Φ ˆ B j and Φ t to represent state s j . • Action For each state, the action space A is a 0-1 judgment to decide if selecting an instance (1) or not (0).
An action a = { a k } | B j | k =1 ∈ { 0 , 1 } | B j | , which is obtained from D ˆ B j .
After each action, data selection gives new Φ ˆ B j , then transforms state s into s (cid:48) .
Thus we deﬁne the policy as P W ( a | s ) .
• Reward Since the data selection process is to ensure that the selected data ﬁtting the distribution of the target domain
we set a reward r ( s, a, s (cid:48) ) to assess the distance between Φ ˆ B j and Φ t in the current state ( s (cid:48) ) and its previous state ( s ):
[Equation]
where d ( · , · ) is a distribution discrepancy measurement, which can be implemented by different information-bearing functions.
γ ∈ (0 , 1) is a discounting constant that decreases the impact from future distribution difference.
Note that Eq.
(1) is conducted in a sequence based on two adjacent data bags B j − 1 and B j , of which Φ s (cid:48)
ˆ B j are impacted by Φ s ˆ B j − 1 via parameters Ψ of the feature extractor updated by ˆ B j − 1 .
Consequently, the state transition probability P ( s (cid:48) | s, a ) is determined by stochastic optimization and other randomness in training, e.g., dropout (Srivastava et al., 2014).
When better instances are selected, the reward is then expected to produce a higher value because the measurement for the previous state d ( Φ s ˆ B j − 1 , Φ st ) supposes to give a larger distance between Φ ˆ B j − 1 and Φ t than that for the current state.
Distribution Discrepancy Measurement To measure each ˆ B j and the X Tg , let P be the elementwise average of Φ ˆ B j and Q the average of Φ t , we use the following measurements for d ( · , · ) in this paper:
• JS :
The Jensen-Shannon divergence (Lin, 1991).
d ( P, Q ) = 12 [ D KL ( P || M ) + D KL
( Q || M )] where D KL ( P || Q ) =
(cid:80) ni
=1 p i log p
i q i , with M = 12 ( P + Q ) .
• MMD : The maximum mean discrepancy (Borgwardt et al., 2006).
d ( P, Q ) =
(cid:107) P − Q (cid:107) .
• R ´ ENYI :
The symmetric R´enyi divergence (R´enyi, 1961).
d ( P, Q ) = 12 [ Ry ( P, M )
+ Ry ( Q, M )] ,
Ry ( P, Q ) =
α − 1 log
( (cid:80) ni
=1 p αi q α − 1 i ) .
We set α =0.99 following Van Asch & Daelemans (2010).
[Table]
[Table] [Table]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
• L OSS :
The guidance loss.
d = − 1 m (cid:80) mi =1 (cid:80)
y t ∈Y t y t log p Φ ( y t | x Ti ) where y t
is the
label
of instance t from the guidance set, and
p φ
the learned conditional probability of the predictor.
Note that
, different from aforementioned measurements, L OSS requires labels from the target domain, thus is only set as a comparison to our approach with using annotated data through the predictor.
Optimization The following object is optimized to obtain the optimal distribution generation policy.
[Equation]
Then the parameters of the SDG, i.e., W , is updated via policy gradient (Sutton et al., 1999) by
[Equation]
where τ is the discounting learning rate 5 , the gradient ∇ W J ( W ) is approximated by ∇ W (cid:101) J ( W ) =
n J
[Equation]
s
[Equation] [Equation]
) , where j refers to the j -th step (corresponding to the j -th data bag) in RL; i denotes the i -th selection process to estimate ∇ W J ( W ) .
Note that ∇ W J ( W ) is estimated after n J times of selection over all N data bags, where n J is a predeﬁned hyper-parameter.
2. J OINT T RAINING
We jointly train the SDG and the predictor with policy gradient method (Sutton et al., 1999), which favors actions with high rewards from better selected samples.
The entire joint training process is described in Algorithm 1.
Following Feng et al.
(2018), we choose the whole B j for the predictor to compute r ( s j − 1 , a j , s j ) when ˆ B j = ∅ at time step j to help exclude noisy bags.
E XPERIMENT
To evaluate our approach, we conduct experiments on three typical NLP tasks: POS tagging, dependency parsing, and sentiment analysis.
Details about our experiments are described as follows.
3. D ATASETS
Two popular datasets are used in our experiments.
For POS tagging and dependency parsing, we use the dataset from the SANCL 2012 shared task (Petrov & McDonald, 2012), with six different domains.
For sentiment analysis, we use the product review dataset from Blitzer et al.
(2006), with
T ASK
POS T AGGING /D
EPENDENCY P ARSING
S ENTIMENT
A NALYSIS
D OMAIN
A
E M
N
R
[Table]
WSJ
[Table]
D
K
E
[Table]
four domains.
Note that for all datasets, there exists both labeled and unlabeled samples.
The statistics and the domains for the aforementioned datasets are reported in Table 1.
3. C OMMON S ETTINGS
A major difference between our approach and other data selection methods is that the number of instances to be selected, n , is not ﬁxed in our approach.
Instead, it chooses the most effective one automatically.
For fair comparison, we record the resulted n from our approach in different tasks and use it in other methods to guide their selection.
In all experiments, the source domain includes all labeled data from the dataset except that for the target domain, i.e., we take turns selecting a domain as the target domain, and use the union of the rest as the source domain.
N is set according to a uniform bag size of around 1K. For the guidance set, we follow Ruder & Plank (2017) and randomly select half of the instances from all the test data in the target domain discarding their labels.
[Table]
Considering that the starting reward needs to be calculated from a reliable feature extractor, before the regular training, we pre-train the predictor on all source data forepochs, then initialize parameters of SDG with Gaussian variables.
Later the predictor and SDG follows ordinary updating paradigm in each training epoch.
In all experiments, we use Adam (Kingma &
Ba, 2014) as the optimizer
[Table]
and set γ to 0.99 following Fan et al.
(2017) and n J to 3.All the deep learning based methods are conﬁgured with the same neural networks as our corresponding experiments.
3. POS TAGGING
The Predictor We use the Bi-LSTM tagger proposed in Plank et al.
(2016) as the predictor.
Baselines Following Ruder & Plank (2017), we compare our approach to ﬁve baselines: 1) JS-E : top instances selected using Jensen-Shannon divergence.
2) JS-D : top instances selected from the most similar source domain, where the similarity between domains are determined by Jensen-Shannon divergence.
Bayesian optimization (Brochu et al., 2010) with the following settings: T-S , term distribution similarity; T O -S , topic distribution similarity; T+T O -S , joint term and topic distribution similarity; T-S+D , term distribution similarity and diversity; T O -S+D , topic distribution similarity and diversity.
4) MDAN : a recent multiple source domain adaptation with adversarial learning (Zhao et al., 2018).
5) R ANDOM : a random selection model that selects the same number of instances according to our proposed approach.
6) A LL :
The predictor is trained on all available source data.
Results POS tagging results are reported in Table 2.
Overall, our approach with different distribution discrepancy metrics outperforms all baselines based on the same feature extractor.
This observation demonstrates the excellent adaptability of our approach on this tasks with complicated structural variance in sentences.
Among the four metrics, R´enyi divergence achieve the best overall performance, which is slightly surpassed by MMD in the A NSWER and W EBLOGS domain.
It is observed that the convergence of our approach results in around 50% of the source data selected.
3. D EPENDENCY P ARSING
[Table]
The Predictor
The Bi-LSTM parser proposed by Kiperwasser &
Goldberg (2016) is the predictor.
Baselines For dependency parsing, we use the same baselines introduced in the POS tagging task.
Results The performance (labeled attachment scores, LAS) of dependency parsing is reported in Table 3.
Similar to POS tagging, MDAN stand out among the baselines and the term distribution-based method (T-S) as well as its combination with diversity features (T-S+D) surpass other Bayesian optimization baselines, while our models outperform the measurementbased as well as neural network based ones signiﬁcantly in most domains.
However, different from POS tagging, in this task [Table] the predictor trained on the entire source data still performs the best on some domains, which can be explained by the complexity of the task.
Yet, our models, e [Table] g., the SDG (JS) and SDG (R´ ENYI ), outperform the A LL model in the last two domains, with only half of the source data used.
3. S ENTIMENT A NALYSIS
[Table]
The Predictor We use the CNN classiﬁer proposed by Kim (2014) as the predictor in this task.
Baselines In addition to the baselines used in
POS tagging and dependency parsing, we use a series of extra baselines from previous studies: 1) SCL , the structural correspondence learning proposed by Blitzer et al.
(2006); 2) SST , the sentiment sensitive thesaurus method (Bollegala et al., 2011); 3) DAM , a general-purpose multiple source domain adaptation method proposed by Mansour et al.
(2008); 4) SDAMS-LS and SDAMS-SVM , the specially designed sentiment domain adaptation approach (Wu & Huang, 2016) for multiple sources with square loss and hinge loss, respectively.
Results Table 4 presents the results of sentiment analysis.
Similar to previous tasks, it is observed that our approach still performs competitively in this task, even though compared with the algorithms dedicatedly designed for sentiment analysis (e.g., SDAMS) and a general deep learning based multiple source domain adaptation method (MDAN).
A potential reason for our weaker results on Electronics target domain is that dedicatedly designed methods include relation graphs among key words as prior knowledge, while our model aims for a wider application without task-speciﬁc consideration.
Slightly different from previous tasks, in this task, around 40% source data are selected upon the convergence of our models.
3. D ISCUSSION
In all three tasks, our approach achieve the best overall performance when there are half or less than half source data selected to train the predictor.
The comparisons between our approach and the basic distribution measure-based methods, the general-purpose multiple source approach as well as the previous models (in sentiment analysis) across all tasks illustrate the effectiveness of our approach in selecting the most useful instances for the target domain while eliminating the noise.
However, domain variance still plays an important role affecting model performance, where it is also taskspeciﬁc.
Compared to POS tagging and dependency parsing, in sentiment analysis there exists more signiﬁcant bias across domains, e.g., “small” could be positive in one domain but negative in another.
[Figure]
Consequently, topic related domains express similar sentiment expressions.
The investigation of the selected data indicates that our approach chooses more instances from the similar domains in sentiment analysis (e.g.,
B OOK ⇒ D VD ), while the selected instances in POS tagging and dependency parsing are more balanced across domains.
This observation suggests the effectiveness of our approach in adapting different tasks with the most appropriate strategy.
In addition, there still exist side effects while leveraging noise ﬁltering and relevant instance selection, which can be observed from the slightly weaker results on Electronics domain in sentiment analysis and the fact that our approach is outperformed by training on all of the sources in parsing task.
Such phenomenon implies that ﬁltering some irrelevant instances may lose intrinsic beneﬁcial information for target domain classiﬁcation, and policy gradient method with partial data sometimes converges to a local optima given structured data with many indirect relations.
A BLATION S TUDIES 4. P ERFORMANCE AND E FFICIENCY OF D IFFERENT M ODELS
To better understand the behavior of our model with different measurements, we investigate their performance using a case study on the DVD domain in sentiment analysis.
We record the accuracy curves of different models with respect to their training epochs, as shown in Figure 2(a).
In general, our models show similar performance and are signiﬁcantly better than the R ANDOM one.
Interestingly, their curves are also similar to the A LL model but shows a much stable performance with the epoch increasing.
This observation demonstrates that there exist severe noise when using all source data directly, while our models are able to overcome such limitation.
Another investigation is to study how much data are selected by different measurements in our approach.
We display the number of instances selected by the four measurements in Figure 2(b), using the same domain and task setting as that in Figure 2(a).
Overall our models with different measurements share similar behavior in selecting source data in terms of selection numbers.
They tend to select more data at the beginning stage, i.e., before 10 epochs, then reduces selected instances to a smaller set and keeps the performance of the predictor (comparing with the curve in Figure 2(a)).
Among all measurements, R ´ enyi divergence tend to select less data instances while achieving a better performance when we match the numbers with the results reported in Table 4.
4. D ISTRIBUTION V ISUALIZATION
To better demonstrate the effect of our approach, we use the sentiment analysis for the D VD domain with SDG (R´ ENYI ) for visualized comparison among the distributions of instances in different scenarios.
Following Tzeng et al.
(2014), we plot in Figure 3 the t-SNE visualizations of the feature vectors learned from the feature extractor in four settings: vectors before training (learning with initialized weights) (Figure 3(a)), trained with SDG (Figure 3(b)), trained with JS-E (Figure 3(c)), and directly trained on all source data (Figure 3(d)).
It is observed that, in original vectors, D VD and B OOKS are similar, while E LECTRONICS and K ITCHEN are different from them as well as to each other.
When trained with all source data, feature vectors could show some change in their distributions where instances from different domains are mixed and closer to the target domain.
In the case where JS divergence is minimized for each domain, we can see a further mixture with closer representation matching.
However, their domain adaptation capability is limited since the vectors are not optimized for the target domain.
On the contrary, when trained with our approach, the selected instances result in a highly similar distribution as that in the target domain, as shown in Figure 3(b) the matched shape between the points in red and other colors, which is more beneﬁcial than the structure in non-RL based embeddings (3(c)).
Consider that it is a dynamic selection process in our
[Figure]
approach, the observation from Figure 3 conﬁrms that such process ensures that the selected data could shape a target domain alike distribution without redundant instances.
R ELATED W ORK
Many studies have been conducted recently for domain adaptation with neural networks (Long et al., 2015; 2016; Shu et al., 2018; Shankar et al., 2018).
Their methodologies include representation learning (Glorot et al., 2011;
Chen et al., 2012; Baktashmotlagh et al., 2013; Zhao et al., 2018), reweighing samples from the source domain (Borgwardt et al., 2006; III, 2007;
Gong et al., 2013), and seeking a way for explicit feature space transformation (Gopalan et al., 2011; Pan et al., 2011;
Long et al., 2013), etc.
Normally, the transferable knowledge across domains are derived from some particular data, while others contribute less and are costly to be learned from (Axelrod et al., 2011; Ruder & Plank, 2017).
Thus, previous studies conduct domain adaptation through selecting relative and informative source data according to the nature of the target domain, such as entropy-based source data selection (Song et al., 2012), Bayesian optimization based selection (Ruder & Plank, 2017), etc.
Especially in various
NLP tasks, Training data selection has been widely used, such as multilingual NER (Murthy et al., 2018), machine translation (Chen et al., 2016; van der Wees et al., 2017), and language model (Moore & Lewis, 2010).
Consider that data selection in general is a combinatorics’ optimization problem with exponential complexity, it thus needs to be reduced to a series of sequential decisions, while
RL provides a good solution in this case.
For example, in Feng et al.
(2018), they used sequential one-step actions for each single instance where every action is decided based on the previous one.
In doing so, the selection is a consuming process where the complexity is related to the amount of the source data.
Fan et al.
(2017) sought to allocate appropriate training data at different training stages, which helps achieving comparative accuracy with less computational efforts compared with the model trained on the entire data.
Another line of research related to this work is the representation adaptation across domains (Mansour et al., 2008), which is the main strategy of domain adaptation for neural network models.
Compared to the aforementioned studies, the proposed approach in this paper combines dynamic data selection and transferable representation learning in a uniﬁed RL framework, and is conducted in an effective way on data batches.
Especially different from previous work that requires a predeﬁned selection number, our approach not only ensures the selection of related instances in a dynamic manner, but also effectively controls negative transfer (Rosenstein et al., 2005) with limited noise.
C ONCLUSION
In this paper, we proposed a general data selection framework for domain adaptation via reinforcement learning.
In detail, the framework matches the representations of the selected data from the source domain and the guidance set from the target domain and pass their similarity in different time steps as rewards to guide the selection distribution generator.
Through the generator, different instances from the source domain are selected to train the task-speciﬁc predictor.
Experiments were conducted on
POS tagging, dependency parsing, and sentiment analysis.
Results from all tasks demonstrate that our approach outperforms different baselines, especially the one trained on all source data.
Ablation study was also provided in analyzing the factors that affects the performance of our approach.
Investigations on model convergence, selection numbers, as well as distribution visualizations conﬁrm the validity and effectiveness of our approach.
A I NSIGHT OF REWARD FUNCTION DESIGN
In this section, we will brieﬂy illustrate the motivation to formulate the one-step reward function as
Eq.
The main idea is to reduce the variance for exploring a path in the state space.
By summing up the discounted reward in a exploring path, we can obtain the value function:
[Equation]
where s 0 denotes the the ﬁnal state in last epoch served as an initial state for the current iteration.
It shows that the value function for our reinforcement learning process only rely on the initial state and the ﬁnal state, with a stronger dependency on the initial state as γ < 1 .
Such a result complies with the intuition to minimize the current distribution discrepancy between source and target representations, i.e., d ( Φ s 0 , Φ s 0 t ) , as the objective, which implies that different exploring paths with similar initial and ﬁnal states gives alike total value function for policy updates.
Hence we achieve the crucial idea to minimize the trivial discrepancy loss in a reinforced exploring way that allows ﬁnding optima closer to the global one and a lower path sampling variance.