Original: Activation Function: Absolute Function,One Function Behaves more Individualized
Revised: Activation Function: Absolute Function,One Function Behaves more Individualized

Original: Abstract: Inspire by nature world mode, a activation function is proposed.
Revised: Abstract: Inspire by nature world mode, a activation function is proposed.

Original: It is absolute function.
Revised: It is absolute function.

Original: Through test on mnist dataset and fully-connected neural network and convolutional neural network, some conclusions are put forward.
Revised: Through test on mnist dataset and fully-connected neural network and convolutional neural network, some conclusions are put forward.

Original: The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu.
Revised: The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu.

Original: The absolute function can keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function.
Revised: The absolute function can keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function.

Revised: In order to generalization, the individualization is the reason of shake, the accuracy may be good in some set and may be worse in some set.

Original: The absolute function is less likely to be over-fitting.
Revised: The absolute function is less likely to be over-fitting.

Revised: The batch size is small, the individualization is clear, vice versa.

Original: Through test on mnist and autoencoder, It is that the leaky relu function can do classification task well, while the absolute function can do generation task well.
Revised: Through test on mnist and autoencoder, It is that the leaky relu function can do classification task well, while the absolute function can do generation task well.

Original: Because the classification task need more universality and generation task need more individualization.
Revised: Because the classification task need more universality and generation task need more individualization.

Original: The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, so the negative parts should keep as a part.
Revised: The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, so the negative part should keep as a part.

Original: Stimulation which happens frequently is low value, it is showed around zero in figure 1 .
Revised: Stimulation which happens frequently is low value, it is showed around zero in figure 1 .

Original: Stimulation which happens accidentally is high value, it is showed far away from zero in figure 1.
Revised: Stimulation which happens accidentally is high value, it is showed far away from zero in figure 1.

Original: So the high value is the big stimulation, which is individualization.
Revised: So the high value is the big stimulation, which is individualization.

Original: 1.1. Preface
Revised: 1.1. Preface

Original: There are many activation function, such as sigmoid, tanh, relu, leaky relu, elu.
Revised: There are many activation function, such as sigmoid, tanh, relu, leaky relu, elu.

Original: The function frequently used is relu[1].
Revised: The function frequently used is relu[1].

Original: The relu function has dead relu problem that is some neuron never be activated, the related parameter never be updated.
Revised: The relu function has dead relu problem that is some neuron never be activated, the related parameter never be updated.

Original: The absolute function doesn' t have dead relu problem and gradient disappearance and gradient explosion, however it has some interesting characteristics.
Revised: The absolute function doesn' t have dead relu problem, and gradient disappearance and gradient explosion, however it has some interesting characteristics.

Original: The image of absolute function is showed below.
Revised: The image of absolute function is showed below.

Original: The formula is y=|x|.
Revised: The formula is y=|x|.

Original: 2.1. The test of absolute function on fully-connected neural network
Revised: 2.1. The test of absolute function on fully-connected neural network

Original: First, I test the activation function which include relu , leaky relu and absolute on mnist dataset and fully-connected neural network.
Revised: First, I test the activation function which include relu , leaky relu and absolute on mnist dataset and fully-connected neural network.

Original: The input should be processed by Z-Score.

Original: The formula is (X-mean)/std.

Original: The network is 5 layers.
Revised: The network is 5 layers.

Original: The firstsamples are validation set, the left are training set[7].
Revised: The first 5000 samples are validation set, the left are training set[7].

Original: Optimizer is adam[6], loss is SparseCategoricalCrossentropy, metric is accuracy[2].
Revised: Optimizer is adam[6], loss is SparseCategoricalCrossentropy, metric is accuracy[2].

Original: The following  is the test result.
Revised: The following picture is the test result.

Original: From the figure 2, the accuracy of absolute function in validation set is lower than the accuracy of relu and leaky relu in validation set when epochs are small.
Revised: From the figure 2, the accuracy of absolute function in validation set is lower than the accuracy of relu and leaky relu in validation set when epochs are small.

Original: But when the epochs grow, the accuracy of absolute function in validation set is almost equal with the accuracy of relu and leaky relu in validation set.
Revised: But when the epochs grow, the accuracy of absolute function in validation set is almost equal with the accuracy of relu and leaky relu in validation set.

Original: The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu.
Revised: The line of accuracy of absolute function is a little shaken that is different from the line of accuracy of relu and leaky relu.

Original: The loss of absolution function is almost equal with the loss of relu and leaky relu.
Revised: The loss of absolution function is almost equal with the loss of relu and leaky relu.

Original: Because the absolute function keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function.
Revised: Because the absolute function keep the negative parts as equal as the positive parts, so the individualization is more active than relu and leaky relu function.

Original: Why?
Revised: Why?

Original: Let’s change the batch size, the results are in figure 3.
Revised: Let’s change the batch size, the results are in figure 3.

Original: In figure 3, batch size is 64, the line is no longer shaken like before, it behaves like the line of relu and leaky relu.
Revised: In figure 3, batch size is 64, the line is no longer shaken like before, it behaves like the line of relu and leaky relu.

Original: The line of relu and leaky relu function are more universality and stable than the line of absolute function.
Revised: The line of relu and leaky relu function are more universality and stable than the line of absolute function.

Original: It filters the negative parts, so it loss the individualization partly.
Revised: It filters the negative parts, so it loses the individualization partly.

Original: In order to generalization, the individualization is the reason of shake, the accuracy may be good in some set and may be worse in some set.
Revised: In order to generalization, the individualization is the reason of shake, the accuracy may be good in some set and may be worse in some set.

Original: The absolute function is less likely to be over-fitting.
Revised: The absolute function is less likely to be over-fitting.

Original: 2.2. The test of absolute function on convolutional neural network
Revised: 2.2. The test of absolute function on convolutional neural network

Original: Let’s change the neural network, the result on convolutional neural network and mnist dataset is showed in figure 4.
Revised: Let’s change the neural network, the result on convolutional neural network and mnist dataset is showed in figure 4.

Revised: The convolutional neural network has 3 layers convolutional neural network and 2 layers fully connected network.

Original: The shake phenomenon is more clear than on fully-connected neural network when batch size is 32.
Revised: The shake phenomenon is more clear than on fully-connected neural network when batch size is 32.

Original: When batch_size increase, the shake phenomenon is more weaker.
Revised: When batch_size increase, the shake phenomenon is more weaker.

Original: It can be anti-over-fitting when batch_size is small.
Revised: It can be anti-over-fitting when batch_size is small.

Original: The loss which decrease firstly but increase then is odd when batch size is 32, however the loss is stable like relu and leaky relu when batch size is 128.
Revised: The loss which decrease firstly but increase then is odd when batch size is 32, however the loss is stable like relu and leaky relu when batch size is 128.

Original: Correspondingly, the accuracy is increase step by step.
Revised: Correspondingly, the accuracy is increase step by step.

Original: In order to increase the accuracy, the model has to learn the individualization which increases the loss.
Revised: In order to increase the accuracy, the model has to learn the individualization which increases the loss.

Original: The relu and leaky relu function can not learn the individualization like the absolute function, so its loss is stable.
Revised: The relu and leaky relu function can not learn the individualization like the absolute function, so its loss is stable.

Original: The batch size is small, the individualization is clear, vice versa.
Revised: The batch size is small, the individualization is clear, vice versa.

Original: If you want to change the individualization of absolute function, just change the batch size.
Revised: If you want to change the individualization of absolute function, just change the batch size.

Original: In order to visualize the intermediate activation, one more test is conducted.
Revised: In order to visualize the intermediate activation, one more test is conducted.

Revised: The convolutional neural network has 3 layers convolutional neural network and 2 layers fully connected network.

Original: The test result is showed in Fig4.7 and Fig 4.8.
Revised: The test result is showed in Fig4.7 and Fig 4.8.

Original: In Fig 4.7,the intermediate activation is sparse and blur with relu activation; Oppositely in Fig 4.8, the intermediate activation is dense and clear with absolute function.
Revised: In Fig 4.7,the intermediate activation is sparse and blur with relu activation; Oppositely in Fig 4.8, the intermediate activation is dense and clear with absolute function.

Original: Because the relu activation function filter the negative part and the absolute activation function keep the negative part.
Revised: Because the relu activation function filter the negative part and the absolute activation function keep the negative part.

Original: Because the feature the absolute function shows is clear , so the generation work can be done by absolute function.
Revised: Because the feature the absolute function shows is clear , so the generation work can be done by absolute function.

Original: Fig4.7.1 Visualize Intermediate Activation with Relu Activation(First Convolution
Revised: Fig4.7.1 Visualize Intermediate Activation with Relu Activation(First Convolution

Original: Layer)
Revised: Layer)

Original: Fig4.
Revised: Fig4.

Original: 3. The test of absolute function on autoencoder generation
Revised: 3. The test of absolute function on autoencoder generation

Original: Because the absolute function has more individualization, so I choose it to do autoencoder’s generation[3] work.
Revised: Because the absolute function has more individualization, so I choose it to do autoencoder’s generation[3] work.

Original: Abstract network(namely prediction network, always used as classification and regression) is common now.
Revised: Abstract network(namely prediction network, always used as classification and regression) is common now.

Original: But the concrete network(namely generation network) which generates concrete information from concept or label is rare.
Revised: But the concrete network(namely generation network) which generates concrete information from concept or label is rare.

Original: Its principle is showed in figure 5.
Revised: Its principle is showed in figure 5.

Original: The test is on mnist dataset and convolutional neural network[5].
Revised: The test is on mnist dataset and convolutional neural network[5].

Original: The convolutional neural network(abstract network) is like LeNet, it has 6 layers.
Revised: The convolutional neural network(abstract network) is like LeNet, it has 4 layers convolutional neural network and 2 layers fully connected network.

Original: The concrete network is the inverse function of abstract network, it has 6 layers.
Revised: The concrete network is the inverse function of abstract network, it has 6 layers.

Original: Optimizer is adam, loss is SparseCategoricalCrossentropy and mse, metric is accuracy and mae and cosinesimilarity.
Revised: Optimizer is adam, loss is mse.

Original: You can read my paper[3] for the detail.
Revised: You can read my paper[3] for the detail.

Original: The test result is showed in figure 6.
Revised: The test result is showed in figure 6.

Original: In figure 6,the left image is input, the label is  the predict of input which conducted by abstract network, the right image is the output of concrete network.
Revised: In figure 6, the left image is input, the label is argmax function of the predict of output of abstract network, the right image is the output of concrete network.

Original: In figure 6.1, when epoch=1, the labels of input are all right.
Revised: In figure 6.1, when epoch=1, the labels of input are all right.

Original: But the image of output is very blurred.
Revised: But the image of output is very blurred.

Original: Meanwhile, in figure 6.2, the labels of input are not all right.
Revised: Meanwhile, in figure 6.2, the labels of input are not all right.

Original: But the image of output is clear.
Revised: But the image of output is clear.

Original: One conclusion can easily draw.
Revised: One conclusion can easily draw.

Original: It is that the leaky relu function can do classification task well, while the absolute function can do generation task well.
Revised: It is that the leaky relu function can do classification task well, while the absolute function can do generation task well.

Original: Because the classification task need more universality and generation task need more individualization.
Revised: Because the classification task need more universality and generation task need more individualization.

Original: In figure 6.2 and 6.3, epoch=30, the result is almost the same between the two activation function.
Revised: In figure 6.2 and 6.3, epoch=30, the result is almost the same between the two activation function.

Original: One different is the background color of absolute function is whiter than leaky relu function.
Revised: One different is the background color of absolute function is whiter than leaky relu function.

Original: So the absolute function can do generation task well.
Revised: So the absolute function can do generation task well.

Original: 4.Some Thinking
Revised: 4.Some Thinking

Original: Should we keep the negative parts of the inputs?
Revised: Should we keep the negative parts of the inputs?

Original: I think we can learn from other fields.
Revised: I think we can learn from other fields.

Original: The color has white and black, the temperature has hot and cool, the smell has fragrant and smelly, the taste has bitter and sweet.
Revised: The color has white and black, the temperature has hot and cool, the smell has fragrant and smelly, the taste has bitter and sweet.

Original: All information our body received are opposite mode.
Revised: All information our body received are opposite mode.

Original: The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, so the negative parts should keep as a part.
Revised: The pleasure irritation and painful irritation is not only the magnitude differences, but also the sign differences, so the negative parts should keep as a part.

Original: According to probability principle, nature word is normal distribution[4].
Revised: According to probability principle, nature word is normal distribution[4].

Original: Stimulation which happens frequently is low value, it is showed around zero in figure 1 .
Revised: Stimulation which happens frequently is low value, it is showed around zero in figure 1 .

Original: Stimulation which happens accidentally is high value, it is showed far away from zero in figure 1.
Revised: Stimulation which happens accidentally is high value, it is showed far away from zero in figure 1.

Original: So the high value is the big stimulation, which is individualization.
Revised: So the high value is the big stimulation, which is individualization.