Evidential Reasoning with Expert-Guided Machine Learning
Abstract
Evidential reasoning aims to infer hidden causes from observed effects.
In the context of fault detection, it is possible to trace the cause of anomalies by combining evidential reasoning with physical knowledge.
However, experts may not have full knowledge of the physical systems, and data may not be large enough to learn from scratch.
We develop an online evidential reasoning algorithm that blends machine learning with expert-provided physical knowledge about causal structure and functional form.
The expert ﬁrst represents possible causes and effects in the structure of a Bayesian Network, then provides physics-informed priors about the model relating failure modes to effects, allowing inference in the absence of strong supervision.
As data are sampled from the physical system, predictions are generated using a quasi-Bayesian mixture of the expert’s judgment and a datadriven estimate.
With simulated datasets, we evaluate the conditions under which the system converges to correct causal inferences under weak supervision, and small amounts of strong supervision from the expert.
We ﬁnd that the approach is able to make accurate inferences with little or no data unless the expert’s physical model is very incorrect or the signal to noise ratio across error modes is small.
Introduction
Evidential reasoning aims to make inferences from observed effects to hidden causes [14], and is important in many applications, from medical diagnosis, to analyzing the root cause of a manufacturing machine’s failure (a.k.a root cause analysis or RCA).
In the manufacturing case, patterns in measurable quantities (e.g., sound, vibration, force) captured during operation will vary over time due to changes of operation, component degradation, corrosion, overheating and other effects. A
Bayesian Network (Bayes Net) can connect the observed sensor measurements (effects) to their hidden causes.
A maintenance technician or engineer forms expectations about the patterns of observed variables given different failure modes, such as degradation after a speciﬁed number of cycles.
The expert’s task is to study the patterns (sensor readings) to determine whether the system is healthy, and if not, remedy the cause of the problem before failure occurs.
Predictive maintenance approaches that combine expert knowledge and machine learning to solve the evidential reasoning problem have the potential to reduce costs, improve situational awareness, and optimize machine operation [15, 2, 18]. Schedule based maintenance, on the other hand, is suboptimized as it risks both performing maintenance tasks when not needed, resulting in excess cost, and missing failure in progress, resulting in lost of productivity due to unplanned downtime.
Because machine failure is costly, algorithms would ideally be able to detect faults before they occur, with little supervision.
Physical knowledge about expected patterns may be provided by a knowledgeable expert or collection of experts.
In some cases, this knowledge may be represented as a system of ordinary differential equations (ODEs) or partial differential equations (PDEs).
Recent research has used neural networks to leverage this kind of prior physical knowledge [23, 4, 16].
An effective algorithms should also be able to adapt to streaming data, reduce the burden of system operators, and be robust to operator mis-speciﬁcation of the relationship between signal patterns and hidden causes. Leveraging expert knowledge about the potential causes and physical functional relationships underlying machine operation, we propose an approach that: 1) incorporates expertprovided physical knowledge into the modeling of machine failure modes, 2) uses an expert-informed Bayes Net to simplify the evidential reasoning inference problem, and 3) combines an expert’s judgment with a data-driven model to solve the root cause analysis problem.
Related Work
Machine learning approaches to root-cause analysis may use supervised learning combined with physics-based feature engineering, such as using a signal’s frequency spectrum in a vibration model.
Many features in both the time and frequency domains have proved useful, including the energy ratio, side-band index, sideband level factor, root mean squared error, energy operator, skewness, kurtosis, crest, energy ratios of wavelet coefﬁcients, ensemble empirical mode decomposition, wavelet kernel local ﬁsher discriminant analysis, among others [9, 24].
Feng et al .
[9] use a local connection network with normalized sparse auto-encoders, and Hong et al .
[8] use Gaussian Process regression to model bearing degradation, where the rMSE of the GP ﬁt on normal data and tested on new data is used to classify failure one day in advance.
Another important tool is the use of Bayes Nets. In more general settings, Kaufman et al . [10] propose a method for estimating the Markov blanket for some variables in large networks, which improves subsequent Bayesian inference using Gibbs sampling. Louizos et al.
[17] use variational auto-encoders to represent confounders as latent variables to estimate causal effects.
Inclusion of experts in machine learning has gained interest in recent years.
Abdollahi et al .
[1] use expert guidance to optimize the selection of parameter spaces and hyper-parameters in the challenging task of 3D printing soft materials.
Gennatas et al .
[7] use expert decision rules as priors in a machine learning model.
Xu et al .
[26] use mis-speciﬁed expert functions in the form of pairwise comparisons for optimization.
Using expert-provided Bayes Nets for root cause analysis
We next describe an approach to use expert-provided Bayes Nets in the evidential reasoning task.
Assume a physical system has S sensors s 1 , s 2 , . . .
s S .
For any time interval [0 , t ] , we are interested in the quantity p ( m | s 1 , 0 , s 2 , 0 , ...s 1 ,t , s 2 ,t , . . .
s S,t ) where m is a set of causes (normal or failure modes) with categorical input space M , and s i,t denotes the measurements of sensor i at time t .
If both the sensors and failure modes are observed, one can train physics-based neural network models { ψ i m } (see next section for ψ ) to ﬁt the sensor i ’s readings to a pattern, according to speciﬁc failure modes.
Bayes Nets capture the fact that it is unlikely that all failure modes affect all sensors.
Causal relationships between the hidden states m and the observed variables s are initially given by experts as a Bayes Net G .
Using the Bayes Net we can factorize the joint distribution p ( m , s 1 , 0 , s 2 , 0 , ...s 1 ,t , s 2 ,t , . . .
s S,t ) into a product of conditional distributions based on the parents ( Pa ) of each sensor: p
[Equation]
m , s 1 , 0 , s 2 , 0 , ... s 1 ,t , s 2 ,t , . . . [Equation]
t ) [Equation] i [Equation]
[Equation]
m )) · p ( m ) .
With this factorization it is possible to do likelihood-based inference if robust likelihoods are available for the conditional distributions of p ( s i, 0 , s i, 1 , ...
s i,t , | Pa ( s i ) ⊆
m )) .
In our model we impose the assumption that the distribution of sensors with respect to the fault vector m satisﬁes a Gaussian distribution, such that p ( s i, 0 , s i, 1 , ...s it | m ) ∼ (cid:81) tj =0 N ( ψ i m ( j ) , σ 2 i ) .
A sequence is predicted based on the neural network ψ i m with input [0 , t ] and compared with each s i,j , j ∈ [0 , t ] to see how similar two functions behave in terms of mean squared error.
Our goal is to: 1) Estimate ˜ m by the conditional probabilities, and 2) update ψ i m based on streaming data.
The approach works as follows:
Construct Bayes Net G from the expert’s engineering analysis of the system 2.
For S sensors s 1 , 0 , s 2 , 0 , ...
s 1 ,t , s 2 ,t , . . .
s S,t and hidden state m , factorize their joint distribution according to G
[Equation]
where Pa ( s i ) ⊆
m are the parents of sensor i in G 3.
Initialize ψ
i m with an expert’s assessment of the time series model for each sensor i and each combination of its parents 4.
For each of the sensors and failure mode combinations, label the sensor data s t observation with:
[Equation]
Label the data vector s t with the label ˜ m , which indicates the failure modes that cause the machine to produce this data.
(Optional strong expert supervision: Ask the expert to verify the accuracy of the label by inspecting the machine.)
With n ˜ m as the number of labels for class ˜ m , use learning rate 0 ≤ α ( n ˜ m ) ≤ α ( n ˜ m +1) ≤to update ψ i ˜ m using backpropagation:
[Equation]
Integrating expert-provided physics into learning time-series functions
Machinery is governed by the classical mechanics and electrodynamics and consequently, the machine’s sensor readings and signals will be governed by a systems of ODEs or PDEs that represent the underlying physics.
The aim of predictive analytics is thusto solve for the parameters of the equations that govern the patterns of the data, under various anomaly scenarios.
Our approach aids the experts by training the neural network ψ on a series of time-dependent readings for each sensor and failure mode.
We leverage recent advances in physics-based modeling to improve machinery models using neural networks that require little or no data for accurate estimation.
Speciﬁcally, we use physics-constrained loss functions to approximate the solution to an ODE/PDE using a neural network [21, 22, 20].
The approach learns a hidden function u ( t, x ) such that [20]:
[Equation]
[Equation] where N [ u ; λ ] is an arbitrary non-linear differential operator parameterized by λ and x ∈ R D .
For example, the 1D Berger’s equation is N [ u ; λ ] =
λ 1 u ∂u∂x − λ 2 ∂ 2 u ∂ 2 x .
To enforce the physics-based constraint, the approach deﬁnes f ( t, x ) = ∂u∂t + N [ u ; λ ] and adds (cid:80) Tt
=1 f ( t, x ) 2 as a regularizer, where ∂u∂t and N [ u ; λ ] are calculated using auto-differentiation on the ﬁtted neural network.
In our approach, we treat this differential equation information as imperfect prior knowledge provided by the expert.
Model predictions are then based partly on a data-driven model, and partly on the expert’s guess about the system’s physical dynamics ψ expert .
To capture the data-driven component, we use a neural network with two hidden layers and sine activation functions [25] to predict physical quantities of interest at each timestep.
Prior research [22] ﬁnds that such an architecture can learn to reproduce not only the positions in a ﬂuid ﬂow problem, but also the governing PDE equations.
The use of sinusoidal activation functions in physics-informed neural networks was rigorously studied in the SIREN architecture to deal with the inﬁnite local minima problem [25], with the ability to solve boundary value problems including Poisson, Helmholtz, and wave equations.
Summary of the role of the expert
The expert plays a key role in several stages of the proposed approach.
Here we provide an overview of the expert’s role in each part of the algorithm.
Bayes Net .
The expert provides a Bayesian network G for the algorithm.
For failure mode analysis, this often takes the form of a bipartite graph, where the ﬁrst part of the graph consists of the failure modes F , and the second part includes the observable sensors S .
The directed edges E link failure modes to some subset of the sensors.
Experts should be able to construct such a graph based on their engineering knowledge of the system, particularly if they are experienced with the machine.
Assessment of the causal model .
The expert provides an assessment of how the sensor observations behave given different conﬁgurations of failure modes.
For example, in the case of a turning or cutting machine, there is usually a fundamental frequency at which the system vibrates, and chattering or other problems with the system add new frequencies.
This additivity of signals can be used to simplify the combinatorial problem of specifying the conditional distribution of each sensor for all combinations of its parents.
For the algorithm to be useful, the expert needs to be able to give precise enough estimates such that the difference between the signals across normal and failure modes is larger than the noise variance in the system.
Estimation of failure modes distribution .
Although not used in our current approach, the expert can also provide base rate (unconditional) guesses about the probability of different failure modes.
As the system is continuously ﬁtting data-driven models and weakly classifying hidden states as failure modes at each timestep, it is possible for these estimates to also be data driven.
Strong supervision .
We assume the expert can inspect the machine to determine whether the algorithm correctly classiﬁes the failure modes.
This might be visual inspection, diagnostic testing, or other relevant engineering analyses.
Strong supervision is particularly important when the expert is inaccurate in the initial estimates (or is unable to provide estimates).
If a previously unobserved malfunctioning cause is detected, the expert can add additional nodes ( F, S ) and edges ( E ) to reﬁne the Bayesian network G .
The ideal scenario is that the expert can give strong supervision with the system; that is, the expert can detect all misclassiﬁed labels and provide correct feedback.
Synthetic experiments
We examine the ability of the proposed approach to recover time series functions and accurately classify error modes with little or no data, focusing on a) the amount of data required, b) noise in the data generating process, c) bias in the expert’s judgments, and d) strong supervision.
In a real machine deployed in a manufacturing environment, the Bayesian Network and functional relationships can be quite complex.
For the purpose of initial demonstration, we use a simple Bayesian network with two sensors S =
{ s 1 , s 2 } and three failure modes F = { f 1 , f 2 , f 3 } .
The directed edges depict the failure modes that are parents of each sensor, as shown by Figure 1.
The probability of each failure mode occurring is characterized by independent Bernoulli random variables with probability p of being active during any time window T =
[ T 1 , T 2 ] .
Each sensor outputs either the normal operation signal (no failure modes are active) or a sum of the signals produced by the active failure modes.
This means that each element m ∈ m is also m ∈ F .
Let s iT denote the signal observed by sensor i during the time interval T , then we have
[Equation]
where ψ mi is the function that produces signals when fault m occurs, ψ {+
+} i produces the normal functioning signals.
Before data are collected from the machine, experts provide an assessment of the functional causal model for each sensor for all combinations of its parent failure modes, with a maximum of 2 | F | assessments.
Ideally, given an arbitrary time window t , the estimate for sensor i follows ( Pr ( | ˆ s iT − s iT | ) > (cid:15) ) < δ ).
As we have more data collected and with correct expert’s intervention, we want our estimated ˆ s i to get closer to the true s i .
6.1 Experiment 1: Time windows
First we consider the case where the expert provides nearly accurate estimates of the signal for each failure mode in the presence of small amounts of noise.
We examine the amount of data required for the algorithm to accurately learn the sum of periodic functions.
Our key variable manipulated in the experiment is the time window size, proportional to the number of observations collected after the failure occurs.
Ideally this number will be small so that faults can be identiﬁed early and machine down-time will be small.
Data Generation .
Our assumption is that the expert already knows how s 1 and s 2 behave during normal operation ( sin( t ) and cos( t ) , respectively).
When a failure occurs, the two sensors s 1 and s 2 are generated with the same periodic functions, according to Table 1.
This simpliﬁcation does not affect the generality of the approach, because different neural networks are still required to learn the periodic functions for each combination of failure modes.
We assume all combinations of errors occur with equal probability.
Sensor measurements are collected for 1 observation per 0 .
2 seconds, with T samples are collected for a time window of T × 0 .
2 seconds.
The experiment varies the length of time T for collection of sensor data.
Our primary outcomes are the accuracy of failure mode classiﬁcation and resulting average MSE’s between the modeled functions and data.
[Table]
Training .
We train independent SIREN models for each combination of sensors and failure modes.
For each of the SIREN models, we initialize 2 internal layers of 256 nodes with sine activation andoutput sine layer.
We also use the expert’s strong supervision when the model misclassiﬁes the labels.
We repeat this experiment for 25 iterations(assuming the machine is breaking down 25 times) for each of the failure modes.
Results .
Table 2 reports the accuracy of root cause classiﬁcations and Figure 4 shows the MSE for s 1 .
We ﬁnd that a small time window causes the classiﬁer to slowly converge, as shown in Figure 4.
The accuracy improves as the time window expands, but the small time window does not affect the classiﬁcation accuracy too much, reﬂecting the expert’s initial accuracy.
Function recovery is possible with around 10 to 15 iterations, regardless of the time window sizes.
[Table]
6.2 Experiment 2: Noise variance
Here we examine how noise in the sensor measurements affects function recovery and failure mode classiﬁcation.
The approach is the same as Experiment 1, but with a ﬁxed window size of T = 25 , and the actual equation has a range of different noise variances σ 2 i ∈ { 0 . 1 , 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 , 1 } .
Results .
The failure mode classiﬁcation accuracy is shown in Table 3, with MSEs for ﬁtting s 1 in Figure 6.
The classiﬁcation accuracy drops as the sensor observations become more noisy.
For larger variances, the MSEs vacillate instead of converging to their lower bound, meaning that the model fails to recover the underlying mean function.
Note that this happens with a σ 2 i of approximately 0.4, where accuracy drops signiﬁcantly across SIREN models. [Table]
6.3 Experiment 3: Expert accuracy and supervision
There are two critical cases where the model’s performance depends on expert judgment: 1) when the experts do not have full knowledge of the underlying physical equations for the failure modes
[Table]
and 2) when the experts are unavailable for providing strong supervision to correct misclassiﬁcations.
We examine the model’s performance in each scenario.
To illustrate model performance, we focus only on the classiﬁcation of failure mode F 2 .
Expert’s prior knowledge .
We vary two components of the expert’s assessment: 1) the amplitude of the sine or cosine functions and the frequency of the sine or cosine functions.
We use a training strategy similar to the previous experiments, see Table 4.
Scale parameters θare generated from {− 10 , − 5 , − 2 , 0 , 2 , 5 , 10 } and frequency parameters θ 2 are generated from { 0 . 1 , 1 , 1 . 5 , 2 , 2 . 5 , 3 , 3 . 5 } .
When we alter θ 1 , θ 2 is ﬁxed to be the same as the frequency parameter in actual equation ( 2 . 2 ).
When we alter θ 2 , θ 1 is taken to be the same as the scale parameter in the actual equation ( − 0 .
We use 30 iterations instead of 25 , to fully observe F 2 .
We assume the experts give correct classiﬁcation labels when the model misclassiﬁes.
Results .
We show the performance of the model under different prior knowledge, with its accuracy in detecting F 2 and the MSE convergence for F 2 .
For simple periodic functions, if enough data are given to the SIRENs for each of the iterations and an expert corrects misclassiﬁed labels, the model is able to give relatively accurate predictions.
In both cases, the MSE between the true function and predicted function converges in Figure 5.
We further compare the accuracy results without expert supervision (labels), provided in Table 5. [Table] [Table]
Supervision .
We conduct the same experiments as shown in Table 4, varying the scale θand frequency θ 2 with two common ranges of parameters, θ 1 ∈ {− 5 , − 2 , 0 , 2 , 5 } , and θ 2 ∈ { 1 , 1 . 5 , 2 , 2 . 5 , 3 } .
However, here we assume that the expert cannot provide any labels.
Results .
The MSE for convergence is shown in Figure 7 and the accuracy in comparison with strong supervision is shown in Table 5.
Without the expert’s correction of the wrongly classiﬁed labels, the model does not properly modify the internal physical neural network associated with the failure mode, leading to non-convergence of the MSE.
Our experiment shows that the expert’s strong supervision is important, especially in the event of limited data, to initialize the physical models.
Turning data and chatter diagnosis
In this section we apply our approach to a real machining dataset by Khasawneh et.
al.
[11] who examine tool chattering during a turning task under different settings for workpiece and cutting edge.
The sensor signals are simultaneously collected at a sampling frequency of 160 kHz.
The processed data are low-pass ﬁltered using a Butterworth ﬁlter of order 100, and then subsampled to 10 kHz. We use the processed and tagged x-axis signals from the triaxial accelerometer. In addition to desired normal operation (stable, s), there are three non-stable modes: intermediate chatter (precursor to failure) (i), chatter (failure mode) (c), and unknown (failures other than chattering) (u), see Figure 2. In prior work, classifying the chattering data was accomplished by extracted features from signals based on Topological Data Analysis (TDA) and nonlinear time series analysis [12] (accuracy score of 0 . 97 ), or the usage of wavelet packet transform (WPT) and ensemble empirical mode decomposition (EEMD) [27] (accuracy score of 0 . 94 ).
While the methods in [12] and [27] train and test the model on the entire dataset, we simulate a zero-shot environment with few or no data in the beginning and train our classiﬁcation model sequentially.
We have also made use of different levels of EEMD[5] to denoise the input chatter data.
The chattering data consists of variable length time-series data-points which are split into non-overlapping chunks with time window of 500 observations each for consistency.
The model observes one chunk at each instance which is classiﬁed and subsequently used to update the model.
For SIREN, we use 3 hidden layers of 256 nodes, and an output linear layer.
For LSTM, we applyhidden layers of 256 nodes, and an output linear layer.
The process proceeds for 1000 iterations, with expert’s supervision once every 10 iterations, as well as each of the ﬁrst 10 iterations.
For expert’s prior guesses, we provide approximate equations based on the judgement of a single chunk(500 data) for each scenario, see Table 6.
[Table]
7.1 Result
The average classiﬁcation accuracy for all scenarios in the chattering dataset is shown by Table 7 for
SIREN and LSTM models respectively.
The rows indicate the original data, and the denoised data decomposed by 1 , 2 , 3 , 4 intrinsic mode functions(IMFs) during each EEMD decomposition.
Figure 3 shows the accuracy over the number of iterations for the SIREN and LSTM models on the original data.
Data with different IMFs
[Table]
Both SIREN and LSTM models achieve best accuracy for denoised data at 2 IMFs ( 0 . 86 and 0 .respectively).
As expected, the models perform better with increasing data updates.
Both models underperform due to increased data complexity (data generated from complex mechanical systems) and noise present in the chattering dataset.
Our model performs less accurately than state-of-the-art models introduced by Khasawneh et. al. [12] and Yesilli et. al.
[27], which rely on speciﬁc physical knowledge or feature extractions that are tailored by experts for the chattering processes. Moreover, their classiﬁcation results are reported for entire variable length time sequences. Our model does not rely heavily on prior physical knowledge (underlying PDEs, which are often unknown in practice) and feature extraction. We believe this setup allows for better generalization to more than just chattering signals, for example, acoustic signals and imagery data.
Conclusion
We provide an initial test of an online evidential reasoning machine learning model under various window sizes at each time of query, different levels of noise, inaccurate prior knowledge, and expert supervision.
Overall, our results suggest that 1) function recovery and accuracy are high when the expert proposes a reasonably accurate model of the physical process and is able to supervise the model when it makes errors, even when the number of observations are small; 2) function recovery and accuracy may degrade when the expert is either very wrong or there is too much noise in the system; 3) a small amount of strong supervision from the expert can realign the algorithm in the presence of inaccurate initial estimates or noise.
Broader impact and future work
Modern machine learning models used for research have increased in complexity and size, leading for example to signiﬁcant energy consumption and resource use.
The implications of such have not been fully explored and are still in nascent stages.
García-Martín et al .
[6] describe methods to estimate energy consumption of machine learning models.
At the same time many emerging applications suffer from constrained or sparse data sets that prevent supervised learning models from being fully effective.
Our future work aims to continue study of the effects of expert-guidance in learning systems to make use of all possible knowledge of the system being modeled, hopefully leading to both reduced model complexity and improved model inferencing.
Another area of interest is joint learning of experts and machines.
Interesting questions include "Can machines guide experts in learning?" and "Can machines teach experts new physics?".
Feedback loops through which machines can communicate to experts with information and suggestions can improve learning and teaching.
In the current framework, this would be advising the experts on the effectiveness of the priors provided or the causal relations.
Finally, inclusion of neural differential equations (ODEs, PDEs) is an area where experts may be able to provide stronger functional form assumptions to the network.
Chen et al .
[3] represent the transformations of inputs through a neural network as a continuous ordinary differential equation rather than discrete layers.
Maulik et al .
[19] apply the approach to the viscous Berger’s equation.
Recently, Kidger et al .
[13] use controlled differential equations to allow neural differential equation time series models with missing data and irregularly spaced temporal observations.
It is an interesting direction to explore the utility of experts in suggesting such differential equation based regularizers for neural networks.