Robustness of Neural Networks used in Electrical Motor Time-Series
Abstract
Electrical motors are widely used in industrial and emerging applications such as electrical automotive.
Industrial 4.0 has led to the usage of neural networks for electrical motor tasks like fault detection, monitoring, and control of electrical motors.
The growing increase of neural networks in safety-critical systems requires an in-depth analysis of their robustness and stability.
This paper studies the robustness of neural networks used in time-series tasks like system modeling, signal denoising, speed-torque estimation, temperature estimation, and fault detection.
The dataset collected for these problems has all types of noise from the operating environment, sensors, and the system itself.
This affects the performance of different network architectures during training and inference.
We train and analyze under perturbations several different architectures that range from simple linear, convolutional and sequential networks to complex networks like 1D ResNet and Transformers.
Code is available at https://github.com/sagarverma/robust-motor.
Introduction
Electrical motors are one of the most used heavy devices in industrial and non-industrial places.
In heavy industry, they can be found in cranes, tunnel boring machines and trains, while at home, they can be found inside all electrical appliances like washing machines, fans and rotational drives.
Each of these applications requires different operating reliability from the motor used.
These motors also have different types of operating environments and scenarios.
A brushless DC motor inside a rotational drive has a consistent operating paradigm.
In contrast, an induction motor inside a tunnel boring machine has to face extreme heat, humidity and dust while having a very inconsistent operation cycle.
Given the nature of the operation, it has been vital for the drive control community to use fault detection and observation methods (1; 2).
Recently neural network based methods have been proposed for such tasks (3; 4; 5).
The scale with which neural network based methods have been proposed for electrical motor tasks does not match the very much required robustness and stability analysis of such methods.
Figure 1 shows how naturally occurring perturbation in inputs can cause a neural network to give the wrong output.
When such neural networks are cascaded to drive a control system, it can lead to catastrophic failures.
In this case, meta-denoiser (6) and speed-torque estimator (7) together take denoised currents and voltages and estimate speed and torque which can then be used to drive the control system.
Currents and voltages are measured using sensors that can be affected by extreme heat, water, dust, or material faults.
Sensors can also perform poorly due to aging.
Motors can age and get affected due to their environment and will behave differently.
Given that meta-denoiser and
DiagBiRNN have not been trained on a dataset that considers such varied inputs and motor states, the networks are bound to give wrong predictions, which can affect the downstream task of control.
C on v
Encoder
Decoder
C on v
C on v
D c on v1 D c on v2 D c on v
S e g m e n t a t i on
H ea d
D e no i se r H ea d
Noisy Input
Segmented Signal
Denoised Signal
Predicted Speed-Torque
Meta-Denoiser
DiagBiRNN
Speed-Torque Estimator
[Figure]
PERTURBED
[Figure]
In (8), convolutional, feed-forward, and sequential neural networks have been used to model electrical motor dynamics by modeling the input-output relationship of its quantities in a data-driven manner.
An encoder-decoder based network called DiagBiRNN has also been proposed which is then used in (7) to estimate speed-torque from currents and voltages.
To apply this network in real-world setting where currents and voltages are noisy, a meta-denoiser network has been proposed in (6) to denoise noisy currents and voltages.
To model electrical motor dynamics, a multi-scale pyramid and lightweight residual networks have been proposed in (9; 10).
Estimating the temperature of different parts of an electrical motor is important to understand its thermal evolution.
Given the complexity of rotating parts, (11) proposed an induction motor temperature dataset.
They use convolution and recurrent network-based benchmarks to predict permanent magnet temperature using several electrical motor quantities and temperatures of different motor parts, which are easy to record.
Several electrical motor fault datasets cover different categories of faults occurring in different types of motors. (12) proposed fault detection and classification dataset to identify the number of broken bars from electrical motor quantities and vibrations recorded using accelerometers.
Robustness Issue in Neural Networks
Test time instability of neural networks is a well-known and active area of research leading to a more explainable and reliable A.I.
In (13), the concept of adversarial attacks was first proposed to fool neural networks.
Adding a well-crafted subtle perturbation to the input of the neural network produces a misclassification.
This scenario is possible even when the model has good clean accuracy.
These attacks pose a huge threat to the performance of neural networks.
There has been a multitude of works introducing stronger adversarial attacks and their defenses.
Goodfellow et al.
(14) proposed the
Fast Gradient Sign Method (FGSM) to generate ℓ ∞ bounded adversarial attacks.
This is a white box attack, i.e., it has access to network structure, parameter weights, and all the related training details.
The generated inputs are misclassified by adding perturbations and linearizing the cost function in the gradient direction.
FGSM is a single-step attack, Madry et al.
(15) proposed a multi-step variant of FGSM called Projected Gradient Descent (PGD) attack.
Kurakin et al.
(16) proposed an optimized FGSM, Iterative Gradient Sign Method (IGSM), which adds perturbations in multiple smaller steps and clips the results after each iteration ensuring that the perturbations are restricted to the neighborhood of the example.
Dong et al.
(17) added momentum to IGSM attacks.
Moosavi et al. (18) proposed Deepfool as a non-targeted attack that tries to find the decision boundary closest to the sample in the input space and then uses the classification boundary to fool the classifier.
Moosavi et al.
(19) proposed a universal image-agnostic perturbation attack method that fools the classifier by adding a single perturbation to all images in the dataset.
Carlini et al.
(20) proposed a powerful attack based on L-BFGS.
In (21), the authors propose a general framework for the generation of adversarial examples in both classification and regression tasks for applications in the image domain.
Most of these methods belong to the class of white box attackers, i.e., the attacker has access to the information related to the trained neural network model, including the model architecture and its parameters.
A black box attacker is introduced in (22).
Such attackers do not know the model but can interact with it.
Ballet et al.
proposed a white box attacker specifically for handling tabular (23) for classification tasks.
A similar approach using the Jacobian property of the neural network for tabular data is proposed in Gupta et al.(24) for regression tasks.
These attacks and defenses constitute the adversarial robustness of neural networks.
Robustness of time-series networks due to thermal evolution can be considered as a generalization problem.
Such problem have been very well studied in (25; 26).
Experimental Setup
We use 9 different architectures proposed in (8) for motor dynamics , denoise , and speed-torque tasks.
We use networks named FNN, RNN, LSTM, and CNN for baselines.
We also use encoderdecoder variants from the paper named Deep, Skip, RNN-Skip, BiRNN-Skip, and DiagBiRNN-Skip to understand robustness behavior with respect to network complexity.
Details of these networks can be found in (8).
For the temperature task we use DiagBiRNN-Skip and FedFormer (27).
For the broken bars task, we use 1D variant of three classification networks namely CRNN (28), ResNet- (29), and RegNet-20 (30).
For generating adversarial attacks, we use FGSM and DeepFool.
The Value of ϵ is taken at 0.01 and 0.1.
In the case of DeepFool number of iterations used is 100.
For the broken bars task, we report clean accuracy and FGSM and DeepFool attack accuracy.
For all the regression tasks, we report clean and FGSM attack values of Mean Absolute Error (MAE), Symmetric Mean Absolute Percentage
Error (SMAPE), coefficient of determination ( R 2 ) (31), and Root Mean Squared Error (RMSE).
[Table]
Tables 1 shows results obtained by training encoder-decoder networks proposed in (8) for motor dynamics task .
It shows MAE, SMAPE, R 2 , and RMSE for clean predictions and FGSM predictions at ϵ = 0 .
01 and ϵ = 0 .
Skip encoder-decoder variant achieves the best clean predictions SMAPE (4.62%) and R 2 (0.77).
When attacked, RNN-Skip is most robust at ϵ = 0 .
01 with SMAPE (20.36%), followed by DiagBiRNN-Skip with SMAPE (21.31%).
Skip achieves the second worst performance when attacked.
In case of denoise task, RNN-Skip obtains the best SMAPE (0.14%) among all the variants.
Ris 0.99 for all the networks.
However, when attacked with FGSM at ϵ = 0 . 01 , DiagBiRNN-Skip outperforms every other network with the lowest SMAPE (1.53%).
With a more aggressive attack of ϵ = 0 . 1 , DiagBiRNN-Skip still outperforms every other network with SMAPE (13.6%) and R(0.72).
Table 2 shows results of the encoder-decoder variants trained for speed-torque estimation task.
Encoder-decoder variant Skip shows the best SMAPE (0.18%) among all the variants.
R 2 is 0.99 for all the networks.
However, when attacked with FGSM at ϵ = 0 . 01 , DiagBiRNN-Skip outperforms every other network with the lowest SMAPE (2.70%).
With a more aggressive attack of ϵ = 0 . 1 , RNNSkip outperforms every other network with SMAPE (17.96%) and R 2 (-0.56).
DiagBiRNN-Skip achieves the best R 2 of -0.49.
[Table] [Table]
and adversarial predictions for the two networks trained for permanent magnet temperature prediction task.
Table 3 shows results for FedFormer and DiagBiRNN-SKip trained to do permanent magnet temperature prediction task.
In this case, it can be seen that FedFormer has performed very poorly when compared to DiagBiRNN in terms of clean predictions.
However, when attacked with FGSM at ϵ = 0 . 01 , FedFormer manages to be robust and give the same result as DiagBiRNN.
This shows that FedFormer is very robust to adversarial examples.
It should be noted that the clean metrics of
FedFormer can be improved with a better training strategy as we have used default hyperparameters to train the network.
[Table]
Table 4 shows results obtained by CRNN, ResNet, and RegNet on broken bars task.
In this case,
RegNet gives the best clean accuracy.
When attacked with FGSM and DeepFool at ϵ = 0 .
01 and ϵ = 0 .
1 , the accuracy of all three networks decreases, but the order remains the same.
Although at ϵ = 0 .
1 , RegNet is still more robust compared to the other two networks.
Conclusion and Future Work
We present a robustness analysis of neural networks used in five different electrical motor tasks.
We train a wide array of networks and generate adversarial attacks on them to show their instability.
Although our experiments show that networks are somewhat unstable to input perturbations, we still need more sophisticated perturbations to understand the robustness of neural network in electrical motors.
In future, it will would be interesting to consider domain knowledge and sequential dynamics of data to generate better attacks and improve the robustness of neural networks for electrical motors.
It is also essential to understand the sensitivity of the individual inputs with respect to neural network stability.
A Appendix A Datasets
We use dataset proposed in (6) for motor dynamics , denoise , and speed-torque tasks.
The dataset consists of following quantities currents ( i d , i q ), voltages ( u d , u q ), noisy currents ( ˆ i d , ˆ i q ), noisy voltages ( ˆ u d , ˆ u q ), rotor speed ( ω r ) and mechanical torque ( τ em ).
The dataset consists of simulations using the control law proposed in (32).
The dataset contains 100 hours of simulation data which cover a wide range of operating trajectories.
All three tasks are regression problems where some motor quantities are estimated from other quantities.
In case of motor dynamics inputs are voltages ( u d , u q ) and rotor speed ( ω r ).
The quantities that have to be predicted are currents ( i d , i q ) and mechanical torque ( τ em ).
The denoise problem deals with denoising noisy currents ( ˆ i d , ˆ i q ), noisy voltages ( ˆ u d , ˆ u q ).
For the speed-torque estimation we predict rotor speed ( ω r ) and mechanical torque ( τ em ) from currents ( i d , i q ) and voltages ( u d , u q ).
Temperature dataset (11) has been used for data-driven thermal modeling to remove or reduce the cost of placing thermal sensors deep inside moving parts of motors.
It consists of different experiments where the temperature of the stator and rotor were measured in real operating conditions.
Currents ( i d , i q ), voltages ( u d , u q ), speed ( ω r ), and torque ( τ em ) are the electrical motor quantities.
Permanent magnet ( ϑ PM ), stator yoke ( ϑ SY ), stator tooth ( ϑ ST ), stator winding ( ϑ SW ), ambient temperature outside of stator ( ϑ a ), and coolant temperature ( ϑ c ) are the recorded temperatures.
The objective is to predict permanent magnet ( ϑ PM ) temperature from all other quantities, making this a regression task.
Broken bars dataset (12) is a multiclass classification dataset.
The data set contains currents, voltages, and torque as electrical signals.
Accelerometers placed in 5 different motor parts are used to collect vibrations/mechanical signals.
In total, 400 experiments, each lasting 20 seconds, are performed.
The objective is to predict how many broken bars are in the motor from its electrical and mechanical signals.
[Equation]
[-R 2 ( y, ˆ y ) = 1 − (cid:80) Tt =
-]
[-( ˆ y t − ¯ y )(cid:80) Tt =
-]
4) where y t is the ground truth, ˆ y t is the predicted output of the model at time t
[Figure]
and T is the total experiment duration.
¯ y denotes the mean of ground truth y
[Figure]
C Results
[Figure]
prediction.
Figure 2 shows an example trajectory from the validation set of the motor dynamics dataset.
In the top sub-figure normalized values of u d , u q , and ω r are showed as normal lines.
FGSM is used with ϵ = 0 .
01 to attack the DiagBiRNN-Skip network to generate an adversarial example shown using the dotted lines in the same sub-figure.
It can be seen that the adversarial example has an offset when compared to the clean input, but this offset can be positive and negative.
There is also intermittent noise like big offsets, for example, one in the 90s.
The middle sub-figure shows the ground truth of output signals i d , i q , and τ em using normal lines and DiagBiRNN-Skip predictions using dotted lines.
It can be seen that the network has poor predictions during the first ramp between 5s and 20s.
The bottom figure shows ground truth as normal lines and dotted lines are the predictions of DiagBiRNN when the generated adversarial example is given as the input.
It can be seen that the adversarial [Table] [Table] predictions are very noisy compared to clean predictions when the perturbations in the input are minimal.
This shows that the attacks on the networks are strong and establishes that robustness study of such networks is important.
Tables 5 shows results obtained by simple networks proposed in (8) for motor dynamics task .
It shows MAE, SMAPE, R 2 , and RMSE for clean predictions and FGSM predictions at ϵ = 0 .
01 and ϵ = 0 .
CNN variant achieves the best clean predictions MAE (0.02), SMAPE (4.86%) and R(0.76).
When attacked, RNN is most robust at both epsilon values.
Table 6 shows results of the encoder-decoder variants trained for denoise task.
RNN-Skip obtains the best SMAPE (0.14%) among all the variants.
R 2 is 0.99 for all the networks.
However, when attacked with FGSM at ϵ = 0 . 01 , DiagBiRNN-Skip outperforms every other network with the lowest
SMAPE (1.53%).
With a more aggressive attack of ϵ = 0 . 1 , DiagBiRNN-Skip still outperforms every other network with SMAPE (13.6%) and R 2 (0.72).