Logical Activation Functions: Logit-space equivalents of Probabilistic Boolean Operators
Abstract
The choice of activation functions and their motivation is a long-standing issue within the neural network community.
Neuronal representations within artiﬁcial neural networks are commonly understood as logits, representing the log-odds score of presence of features within the stimulus.
We derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities.
Such theories are important to formalize more complex dendritic operations in real neurons, and these operations can be used as activation functions within a neural network, introducing probabilistic Boolean-logic as the core operation of the neural network.
Since these functions involve taking multiple exponents and logarithms, they are computationally expensive and not well suited to be directly used within neural networks.
Consequently, we construct efﬁcient approximations named AND AIL (the AND operator Approximate for Independent
Logits), OR AIL , and XNOR AIL , which utilize only comparison and addition operations, have well-behaved gradients, and can be deployed as activation functions in neural networks.
Like MaxOut, AND AIL and OR AIL are generalizations of
ReLU to two-dimensions.
While our primary aim is to formalize dendritic computations within a logit-space probabilistic-Boolean framework, we deploy these new activation functions, both in isolation and in conjunction to demonstrate their effectiveness on a variety of tasks including image classiﬁcation, transfer learning, abstract reasoning, and compositional zero-shot learning.
Introduction
Non-linear activation functions are essential in artiﬁcial neural networks (ANNs) to form higher-order representations, since otherwise the network would be degeneratively equivalent to a single linear layer.
Most activation functions represent a simple non-linearity despite evidence of much more complex non-linear integration and computations in dendrites (Hentschel et al., 2004; London &
Häusser, 2005; Payeur et al., 2019).
For example, Gidon et al.
(2020) recently demonstrated that a single biological neuron can compute the XOR of its inputs, a property long known to be lacking from artiﬁcial neurons (Minsky & Papert, 1969).
While simple activation functions work in ANNs in the sense that more complex operations can be formed from the combination of several layers of neurons, understanding the function and impact of advanced dendritic operations in networks is important.
In this work, we add some of this behaviour to neural activations, corresponding to shifting some of the network’s complexity from its global structure to the neural level.
There is far more complexity in biological neurons than in the abstractions that we consider here, but we make a step in the direction of using more complex neurons in ANNs.
We also develop a theoretical underpinning for higher-order activation functions (e.g. like MaxOut, Goodfellow et al., 2013) in a probabilistic framework.
We hypothesize that such architectures will be more parameter-efﬁcient in situations where their assumptions hold.
Neuronal representations within ANNs are commonly understood as logits, representing the log-odds score of presence (versus absence) of features within the stimulus.
From a Bayesian perspective, a ReLU-like operation corresponds to the removal of all evidence for the lack of a feature.
Under the logit interpretation of ANN potentiations, this seems unreasonable.
This can be seen in some of the ways we interact with neural networks: we must apply batch-norm before activations and not after them; when doing transfer learning from an embedding space we must use pre-activation potentiations instead of activations.
Can ANNs do better if we design an architecture which treats potentiations as logits?
We thus derive logit-space operators equivalent to probabilistic Boolean logic-gates AND, OR, and XNOR for independent probabilities.
Networks constructed in this way can be interpreted as performing logical operations using point-estimates of probabilities , in a similar manner to a
Bayesian network.
This brings operations from the symbolism framework of AI (which is more similar to deliberative thinking, or System 2 of Kahneman, 2011) into the connectionist framework of ANNs (which is more like instinctive, System 1, thinking).
We also construct computationally feasible approximations to these functions with well-behaved gradients.
These new activation functions, which are generalizations of ReLU to two-dimensions, are then applied on benchmark datasets to demonstrate their effectiveness on a diverse range of tasks including image classiﬁcation, transfer learning, abstract reasoning, and compositional zero-shot learning.
The new principled approach we present introduces new ways to redistribute computation from the network into the neuronal mechanisms, and build more parameter-efﬁcient models.
We demonstrate the effectiveness of activation functions based on these ideas, and expect future work to build on this.
Background
Early artiﬁcial neural networks featured either logistic-sigmoid or tanh as their activation function, motivated by the idea that each layer of the network is building another layer of abstraction of the stimulus space from the last layer.
Each neuron in a layer identiﬁes whether certain properties or features are present within the stimulus, and the pre-activation (potentiation) value of the neuron indicates a score or logit for the presence of that feature.
The sigmoid function, σ ( x ) = 1 / (1+ e − x ) , was hence a natural choice of activation function, since as with logistic regression, this will convert the logits of features into probabilities.
There is evidence that this interpretation has merit.
Previous work has been done to identify which features neurons are tuned to.
Examples include LSTM neurons tracking quotation marks, line length, and brackets (Karpathy et al., 2015) and sentiment (Radford et al., 2017); projecting features back to the input space to view them (Olah et al., 2017); and interpretable combinations of neural activities (Olah et al., 2020).
Sigmoidal activation functions are no longer commonly used within machine learning between layers of representations, though sigmoid is still widely used for gating operations which scale the magnitude of other features in an attention-like manner.
The primary disadvantage of the sigmoid activation function is its vanishing gradient — as the potentiation rises, activity converges to a plateau, and hence the gradient goes to zero.
This prevents feedback information propagating back through the network from the loss function to the early layers of the network, which consequently prevents it from learning to complete the task.
The Rectiﬁed Linear Unit activation function (Fukushima, 1980; Jarrett et al., 2009; Nair & Hinton, 2010), ReLU( x ) = max(0 , x ) , does not have this problem, since in its non-zero regime it has a gradient of 1.
Another advantage of ReLU is it has very low computational demands.
Since it is both effective and efﬁcient, it has proven to be a highly popular choice of activation function.
The chief drawback to ReLU is that it has no sensitivity to changes across half of its input domain, which prevents updates on stimuli which trigger its “off” state and can even lead to neuronal death 1 .
Variants of ReLU have emerged, aiming to smooth out its transition between domains and provide a gradient in its inactive regime.
These include ELU (Clevert et al., 2016), CELU (Barron, 2017),
SELU (Klambauer et al., 2017), GELU (Hendrycks & Gimpel, 2020), SiLU (Elfwing et al., 2018;
Ramachandran et al., 2017), and Mish (Misra, 2019).
However, all these activation functions still bear the general shape of ReLU and truncate negative logits.
Fuzzy logic operators are generalizations of Boolean logic operations to continuous variables, using rules similar to applying logical operators in probability space.
Prior work has explored networks of fuzzy logic operations, including some which use an activation function constituting a learnable interpolation between fuzzy logic operators (Godfrey & Gashler, 2017).
The activation functions we introduce here are motivated similarly to fuzzy logic operators, but designed to operate in logit space instead of in probability space, which better reﬂects the behaviour and space of pre-activation units.
Bayesian neural networks (BNNs) are probabilistic models which can represent the uncertainty in their model parameters, and hence uncertainty in their outputs.
This differs from standard ANNs which use only a single point-estimate of each model parameter, and hence also in their neural activations.
By making their priors explicit and modelling their uncertainty, Bayesian networks can be better calibrated and less vulnerable to overﬁtting than ANNs.
However, BNNs are more challenging to train, and cannot reasonably be scaled up to the deep architectures which are possible with standard
ANNs and necessary in order to learn a sufﬁciently complex model to solve highly complex tasks.
In this work we contribute to the theoretical underpinning of neural activation functions by developing activation functions based on the principle that neurons encode logits — scores that represent the presence of features in the log-odds space.
In §3 we derive and deﬁne these functions in detail for different logical operators, and then consider their performance on numerous task types including parity (§4.1), image classiﬁcation (§4.3 and §4.4), transfer learning (§4.5), abstract reasoning (§A.18), soft-rule guided classiﬁcation as exempliﬁed by the Bach chorale dataset (§4.2), and compositional zero-shot learning (§A.19).
These tasks were selected to (1) survey the performance of the new activations on existing benchmark tasks, and (2) evaluate their performance on tasks which we suspect in particular may require logical reasoning and hence beneﬁt from activation functions which apply these logical operations to logits.
Derivation
Manipulation of probabilities in logit-space is known to be more efﬁcient for many calculations.
For instance, the log-odds form of Bayes’ Rule (Eq. 9) states that the posterior logit equals the prior logit plus the log of the likelihood ratio for the new evidence (the log of the Bayes factor).
Thus, working in logit-space allows us to perform Bayesian updates on many sources of evidence simultaneously, merely by summing together the log-likelihood ratios for the evidence (see §A.3).
A weighted sum may be used if the amount of credence given to the sources differs — and this is precisely the operation performed by a linear layer in a neural network.
When considering sets of probabilities, a natural operation is to measure the joint probability of two events both occurring— the AND operation.
Suppose our input space is x ∈ [0 , 1] 2 , and the goal is to output y >
if x i = 1 ∀ i , and y < 0 otherwise, using model with a weight vector w and bias term b , such that y = w T x + b .
This can be trivially solved with the weight matrix w =
[1 , 1] and bias term b = − 1 . 5 .
However, since this is only a linear separator, the solution can not generalize to the case y >
iff x i > 0 ∀ i .
Similarly, let us consider how the OR function is solved with a linear layer.
Our goal is to output y > 0 if ∃ x i = 1 , and y < 0 otherwise.
The binary case can be trivially solved with the weight matrix w =
[1 , 1] and bias term b = − 0 .
The difference between the solution for
OR and the solution for AND is only an offset to our bias term.
In each case, if the input space is expanded beyond binary to R 2 , the output can be violated by changing only one of the arguments.
3.1 AND
Suppose we are given x and y as the logits for the presence (vs absence) of two events, X and Y .
These logits have equivalent probability values, which can be obtained using the sigmoid function, σ ( u ) =
(1 + e − u ) − 1 .
Let us assume that the events X and Y are independent of each other.
In this case, the probability of both events occurring (the joint probability) is P( X, Y ) =
P( X ∧ Y ) =
P( X ) P( Y ) =
σ ( x ) σ ( y ) .
However, we wish to remain in logit-space, and must determine the logit of the joint probability, logit(P( X, Y ) .
This is given by
AND IL :
= logit(P( X ∧ Y ) x ⊥⊥ y ) = log (cid:18) p 1 − p (cid:19) , where p = σ ( x ) σ ( y ) , = log (cid:18) σ ( x ) σ ( y ) 1 − σ ( x ) σ ( y ) (cid:19) , (1) which we coin as AND IL , the AND operator for independent logits (IL).
This 2d function is illustrated as a contour plot (Fig. 1, left panel).
Across the plane, the order of magnitude of the output is the same as at least one of the two inputs, scaling approximately linearly.
The approximately linear behaviour of the function is suitable for use as an activation function (no vanishing gradient), however taking exponents and logs scales poorly from a computational perspective.
Hence, we developed a computationally efﬁcient approximation as follows.
Observe that we can loosely approximate AND IL with the minimum function (Fig. 1, right panel).
This is equivalent to assuming the probability of both X and Y being true equals the probability of the least likely of X and Y being true — a naïve approximation which holds well in three quadrants of the plane, but overestimates the probability when both X and Y are unlikely.
In this quadrant, when both
X and Y are both unlikely, a better approximation for AND IL is the sum of their logits.
We thus propose AND AIL , a linear-approximate AND function for independent logits (AIL, i.e. approximate IL).
AND
[Equation]
As shown in Fig.
1 (left, middle), we observe that their output values and shape are very similar.
[Figure]
3.2 OR
Similarly, we can construct the logit-space OR function, for independent logits.
For a pair of logits x and y , the probability that either of the corresponding events is true is given by p = 1 − σ ( − x ) σ ( − y ) .
This can be converted into a logit as
OR
[Equation] [Equation]
p
[Equation] [Equation]
3) which can be roughly approximated by the max function.
This is equivalent to setting the probability of either of event X or Y occurring to be equal to the probability of the most likely event.
This underestimates the upper-right quadrant (below), which we can approximate better as the sum of the input logits, yielding
[Equation]
[Figure]
− −
OR IL −4 −
OR AIL −4 −    max −10 − −6 − −2 + +2 + +6 + +
3.3 XNOR
We also consider the construction of a logit-space XNOR operator.
This is the probability that X and
Y occur either together, or not at all, given by
[Equation]
where p = σ ( x ) σ ( y ) + σ ( − x ) σ ( − y ) .
We can approximate this with
[Equation]
which focuses on the logit of the feature most likely to ﬂip the expected parity (see Fig. 3).
We could use other approximations, such as the sign-preserving geometric mean,
[Equation]
7) but this matches XNOR IL less closely
[Figure]
and has a divergent gradient along both x = 0 and y =
[Figure]
) ; our constructed approximation, XNOR AIL ( x, y ) ; and SignedGeomean( x, y ) .
3.4 Discussion
By working via probabilities, and assuming inputs encode independent events, we have derived logit-space equivalents of the Boolean logic functions, AND, OR, and XNOR.
Since these are computationally demanding to compute repeatedly within a neural network, we have constructed approximations of them: AND AIL , OR AIL , and XNOR AIL .
Like ReLU, these involve only comparison, addition, and multiplication operations which are cheap to perform.
In fact, AND AIL and
OR AIL are a generalization of ReLU to an extra dimension, since OR AIL ( x, y = 0)
=
max ( x, 0) .
The majority of activation functions are one-dimensional, f : R → R .
In contrast to this, our proposed activation functions are all two-dimensional, f : R 2 → R .
They must be applied to pairs of features from the embedding space, and will reduce the dimensionality of the input space by a factor of 2.
This behaviour is the same as seen in MaxOut networks (Goodfellow et al., 2013) which use max as their activation function; MaxOut( x, y ; k ) :
= max( x, y ) .
Similar to MaxOut , our activation functions could be generalized to higher dimensional inputs, f : R k → R , by considering the behaviour of the logit-space AND, OR, XNOR operations with regard to more inputs.
For simplicity, we restrict this work to consider only k =2 , but note these activation functions also generalize to higher dimensions.
3.5 Ensembling
By using multiple logit-Boolean activation functions simultaneously alongside each other, we permit the network multiple options of how to relate features together.
When combining the activation functions, we considered two strategies (illustrated in §A.6).
In the partition (p) strategy, we split the n c dimensional pre-activation embedding equally into m partitions, apply different activation functions on each partition, and concatenate the results together.
Using AIL activation functions under this strategy, the output dimension will always be half that of the input, as it is for each AIL activation function individually.
In the duplication (d) strategy, we apply m different activation functions in parallel to the same n c elements.
The output is consequently larger, with dimension m n c .
If desired, we can counteract the 2 → 1 reduction of AIL activation functions by using two of them together under this strategy.
A negative weight in a network is equivalent to the logit-NOT operator.
Hence with sufﬁcient width and depth, a multi-layer network using only the OR IL activation function should be able to replicate any probabilistic logic circuit.
Utilizing AND AIL , OR AIL and XNOR AIL simultaneously allows our networks to access logit-space equivalents of 12 of the 16 Boolean logical operations with only a single sign inversion (in either one of the inputs or the output).
Including the bias term and skip connections, the network has easy access to logit-space equivalents of all 16 Boolean logical operations.
We hypothesised that training a network with all three of our activation functions in an ensemble in this manner could yield better results since the network would not need to expend layers having to combine OR IL operations together to yield other Boolean operations.
3.6 Normalization
magnitudes (orange: +ve, blue: -ve).
MLP with ReLU: 60% accuracy;
XNOR AIL : 100% accuracy.
Our AIL activation functions are close approximations to exact
AND, OR, and XNOR operations in logit-space.
However, when deploying non-linearities within a neural network, it is important that the activation functions have a gain of 1 in order to improve stability during training (Klambauer et al., 2017), a property the AIL activations do not possess.
We constructed normalized variants of the exact and approximate logit operators, dubbed NIL and NAIL respectively, by subtracting the expected mean and dividing by the expected standard deviation, assuming the operands are sampled independently from the standard normal distribution, N (0 , 1) .
For more details, see §A.7.
Experiments
We evaluated the performance of our AIL activations, both individually and together in an ensemble, on a range of benchmark tasks.
Since AND AIL and OR AIL are equivalent when the sign of operands and outputs can be freely chosen, we only show results for OR AIL .
We compared against three primary baselines: (1) ReLU, (2) max( x, y ) =
MaxOut([ x, y ]; k =2) , and (3) the concatenation of max( x, y ) and min( x, y ) , denoted { Max , Min (d)}.
The { Max , Min (d)} ensemble is equivalent to GroupSort with a group size of 2 (Chernodub & Nowicki, 2017; Anil et al., 2019), sometimes referred to as the MaxMin operator; it is comparable to the concatenation of OR AIL and AND AIL under our duplication strategy.
Source code for our experiments can be found at https: //github.com/DalhousieAI/logical_activation_experiments .
4.1 Parity
In a simple initial experiment, we constructed a synthetic dataset whose labels could be derived directly by stacking the logical operation XNOR.
Each sample had four input logits, and target value equal to the parity of the number of positive inputs.
A very small model using XNOR with two hidden layers (of 4, then 2 neurons) should be capable of perfect classiﬁcation accuracy on this dataset with a sparse weight matrix by learning to nest pairwise binary relationships.
We trained such an MLP with either ReLU or XNOR AIL activations.
The MLP with XNOR AIL activation learned a sparse weight matrix able to perfectly classify any input combination, shown in Fig.
In comparison, with ReLU the network was only able to produce 60% classiﬁcation accuracy.
The accuracy with ReLU was improved by increasing the MLP width/depth, but this did not result in a sparse weight matrix.
This experiment demonstrates that
XNOR AIL can be utilized by a network to ﬁnd the simplest relationship between inputs that satisﬁes the objective.
For additional results, see §A.11.
4.2 MLP on Bach Chorales and Logit Independence
The Bach Chorale dataset (Boulanger-Lewandowski et al., 2012) consists of 382 chorales composed by JS Bach, each ∼ 12 measures long, totalling ∼ 83 000 notes.
Represented as discrete sequences of tokens, it has served as a benchmark for music processing for decades, from heuristic methods to
HMMs, RNNs, and CNNs (Mozer, 1990; Hild et al., 1991; Allan & Williams, 2005; Liang, 2016;
Hadjeres et al., 2017; Huang et al., 2019).
The chorales are comprised of 4 voices (melodic lines) whose behaviour is guided by soft musical rules depending on the prior movement of that and other voices, e.g. “two voices a ﬁfth apart ought not to move in parallel with one another”.
We tasked 2-layer MLPs with determining whether a short four-part musical excerpt is taken from a Bach chorale.
Negative examples were created by stochastically corrupting chorale excerpts (see §A.13).
We found { OR , AND , XNOR AIL (d)} had highest accuracy, but the results were not statistically signiﬁcant ( p< 0 .
1 between best and worst, two-tailed Student’s t -test, 10 random inits; see §A.10).
Additionally, we investigated the independence between logits in the trained pre-activation embeddings.
We expect that an MLP which is optimally engaging its neurons would maintain independence between features in order to maximize information.
We measured correlations between adjacent pre-activations (paired operands for the logical activation functions), and also between non-adjacent pairs of pre-activations.
Our results indicate the network learns features which are independent when they are not passed to the same 2D activation, and anti-correlated features when they are.
For more details, see §A.14.
4.3 CNN and MLP on MNIST
We trained 2-layer MLP and 6-layer CNN models on MNIST with ADAM (Kingma & Ba, 2015), 1-cycle schedule (Smith & Topin, 2017; Smith, 2018), and using hyperparameters tuned through a random search against a validation set comprised of the last 10k images of the training partition.
The MLP used two hidden layers, the widths of which were varied together to evaluate the performance for a range of model sizes.
The CNN used six layers of 3x3 convolution layers, with 2x2 max pooling (stride 2) after every other conv layer, followed by ﬂattening and three MLP layers.
The layer widths were scaled up to explore a range of model sizes (see §A.15 for more details).
For the MLP, XNOR AIL performed best along with SignedGeomean ( p< 0 .
1 , two-tailed Student’s t -test), ahead of all other activations ( p< 0 .
01 for each; Fig.
5 left panel) when considering the best performance across all widths (see §A.10 for methodology).
However when the width is reduced below 2 × 10 5 there is a transition and XNOR-shaped activations perform worst.
We hypothesize this may be because smaller widths embeddings are over-saturated and have individual units corresponding to multiple features, whilst XNOR activations may require single-feature units to perform best.
With the CNN, ﬁve activation conﬁgurations ({ OR , AND , XNOR AIL (p)}, { OR , XNOR AIL (d/p)},
Max, and SiLU) performed best ( p< 0 .
05 for other activations, two-tailed Student’s t -test; Fig. 5,
Number of parameters 98.
T e s t a cc u r a c y ( % )
MNIST, MLP (2 hidden layers)
ReLU SiLU
Max Max, Min (d) SignedGeomean
XNOR AIL OR AIL
OR AIL , AND AIL (d) OR AIL , XNOR AIL (p) OR AIL , XNOR AIL (d) OR AIL , AND AIL , XNOR AIL (p)
OR AIL , AND AIL , XNOR AIL (d)
Number of parameters 99.
T e s t a cc u r a c y ( % )
MNIST, CNN (6 layer)
4.4 ResNet50 on CIFAR-10/
We explored the impact of our activation functions on deep networks by deploying them in a preactivation ResNet50 model (He et al., 2016a,b).
We exchanged all ReLU activations in the network to a candidate activation while maintaining the size of the pass-through embedding.
We experimented with changing the width of the network, scaling up the embedding space and all hidden layers by a common factor, w .
The network was trained on CIFAR-10/-100 for 100 epochs using ADAM (Kingma & Ba, 2015), 1-cycle (Smith, 2018; Smith & Topin, 2017).
See §A.16 for further details.
Number of parameters 91.
T e s t a cc u r a c y ( % )
CIFAR10, ResNet
ReLU PReLU
SiLU Max
Max, Min (d) OR AIL
OR AIL , AND AIL (d) OR AIL , XNOR AIL (p) OR AIL , XNOR AIL (d) OR AIL , AND AIL , XNOR AIL (p)
OR AIL , AND AIL , XNOR AIL (d)
Number of parameters
T e s t a cc u r a c y ( % )
CIFAR100, ResNet
ReLU PReLU
SiLU Max
Max, Min (d) OR AIL
OR AIL , AND AIL (d) OR AIL , XNOR AIL (p) OR AIL , XNOR AIL (d) OR AIL , AND AIL , XNOR AIL (p)
OR AIL , AND AIL , XNOR AIL (d)
For both CIFAR-10 and 100, SiLU, OR AIL , and Max outperform ReLU across a wide range of width values (see Fig. 6).
These three activation functions hold up their performance best as the number of parameters is reduced.
Since SiLU was discovered by a search of 1 → 1 activation functions for this type of architecture and task (Ramachandran et al., 2017), we expected it to perform well in this setting.
Yet, we ﬁnd OR AIL and Max perform better than ReLU and comparable with SiLU generally.
Meanwhile, other AIL activations perform similarly to ReLU when the width is thin, and slightly worse than ReLU when the width is wide.
When used on its own and not part of an ensemble, the XNOR AIL activation function performed poorly (off the bottom of the chart), indicating it is not suited for this task.
4.5 Transfer learning
We considered transfer learning on several image classiﬁcation datasets.
We used a ResNet model (He et al., 2016a) pretrained on ImageNet-1k.
The weights were frozen (not ﬁne-tuned) and used to generate embeddings of samples from other image datasets.
We trained a two-layer MLP to classify images from these embeddings using various activation functions.
For a comprehensive set of baselines, we compared against every activation function built into PyTorch 1.10 (see §A.17).
To make the number of parameters similar, we used a width of 512 for activation functions with 1 → 1 mapping (e.g. ReLU), a width of 650 for activation functions with a 2 → 1 mapping (e.g. Max,
OR AIL ), and a width of 438 for { OR , AND , XNOR AIL (d)}.
See §A.17 for more details.
[Table]
OR
[Table]
OR
[Table]
OR , AND
[Table]
(d)
[Table]
OR , XNOR
[Table]
d)
[Table]
OR , AND , XNOR NIL
[Table]
d)
[Table]
Our results are shown in Table 1.
We found that all our (N)AIL activation functions outperformed
ReLU on every transfer task.
Normalization had little impact on the performance of OR ∗ IL
[Table]
but a large impact on XNOR ∗ IL .
The best overall performance was attained by the exact XNOR NIL , closely followed by our approximate XNOR NAIL and ensembles containing either of these.
Surprisingly, our approximation OR ∗ AIL outperformed the exact form OR ∗ IL .
The proposed activation functions were beaten only by PReLU, on Stanford Cars.
On Caltech101, all MLPs were beaten by a linear layer, suggesting our MLPs were overﬁtting for that dataset.
For further discussion, see §A.17.
4.6 Additional results
For results on Covertype, abstract reasoning and compositional zero-shot learning, please see §A.12, §A.18 and §A.19, respectively.
Conclusion
In this work, we motivated and introduced novel activation functions analogous to Boolean operators in logit-space.
We designed the AIL functions, fast approximates to the true logit-space probabilistic
Boolean operations, and demonstrated their effectiveness on a wide range of tasks.
Although our activation functions assume independence (which is generally approximately true for pre-activation features learnt with 1D activations), we found the network learnt to induce anticorrelations between features which were paired together by our activations (§A.14).
This suggests exact independence of the features is not essential to the performance of our proposed activations.
We found XNOR ∗ AIL was highly effective for shallow MLPs.
Meanwhile, OR AIL was highly effective for representation learning in the setting of a deep ResNet architecture trained on images.
In scenarios which involve manipulating high-level features extracted by an embedding network, we ﬁnd that using XNOR ∗ IL or an ensemble of AIL activation functions together works best, and that the duplication ensembling strategy outperforms partitioning.
In this work we restricted ourselves to only considering using a single activation function (or ensemble) throughout the network, however our results together indicate stronger results may be found by using OR ∗ AIL for feature extraction and either XNOR ∗ IL or ensemble { OR , XNOR ∗ IL (d)} for later higher-order reasoning layers.
Our work shows there is more to learn about the importance of more complex activation functions, both for ANN applications and for non-linear dendritic integration in biological neuronal networks.
Acknowledgments and Disclosure of Funding
We are grateful to Chandramouli Shama Sastry, Eleni Triantaﬁllou, and Finlay Maguire for insightful discussions.
Resources used in preparing this research were provided, in part, by the Province of Ontario, the
Government of Canada through CIFAR, and companies sponsoring the Vector Institute https: //vectorinstitute.ai/partners/ , and in part by ACENET https://ace-net.ca/ and Compute Canada https://www.computecanada.ca/ .
Additionally, we gratefully acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPU used for this research.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[No] The methodology we discuss is general-purpose and any societal impacts we could discuss would be highly general and completely speculative at this stage.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See §4.1–4.5 and §A.11–A.19.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All results ﬁgures and tables have error over several seeds, and the number of seeds used, clearly indicated.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See §A.9.
The time to run each experiment is also indicated in the README ﬁles which accompany the code.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See §A.8, Table 3.
(b) Did you mention the license of the assets?
[No] It is surprisingly difﬁcult to discover the license under which popular ML datasets, such as CIFAR-10, are released!
Most of the popular datasets within the community do not appear to be released under open licenses.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] Code to replicate our results is included in the supplemental material zip.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[No] We describe the nature of the datasets we are using.
There is no reason to expect that the data we are using contains personally identiﬁable information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A.1 Approximating logistic AND and OR using ReLU
In Fig.
7 (upper panels), we show a representation of a 2D linear layer followed by the ReLU activation function.
Changing projection used in the linear layer allows us to rotate and stretch the output only; each unit is, by construction, empty in half the plane.
If we apply a second linear layer on top of the outputs of the ﬁrst two units, we can try to approximate the logit AND or OR function with z = ReLU( x ) + ReLU( y ) .
However, as shown in Fig. 7, lower panel, the solution using ReLU leaves a quadrant of the output space set to zero due to its behaviour at truncating away information.
This illustrates why a ReLU-based neural network can not accurately approximate the logistic AND and OR functions between a pair of logits with only two hidden units.
Comparing this with Figures 1 and 2 demonstrates the advantage of AND AIL and OR AIL in situations where the network needs to use probabilistic Boolean logic as its basis.
A.2 Solving XOR
A long-standing criticism of artiﬁcial neural networks is their inability to solve XOR with a single layer (Minsky & Papert, 1969).
Of course, adding a single hidden layer allows a network using ReLU to solve XOR.
However, the way that it solves the problem is to join two of the disconnected regions together in a stripe (see Fig.
Meanwhile, our XNOR AIL is trivial able to solve the XOR problem without any hidden layers.
For comparison here, we include one hidden layer with 2 units for each network.
Including a layer before the activation function makes the task harder for XNOR AIL , which must learn how to project the input space in order to compute the desired separation.
Also, including the linear layer allows the network to generalize to rotations and offset versions of the task.
[Figure]
A.3 Linear layers and Bayes’ Rule in log-odds form
Bayes’ Theorem or Bayes’ Rule is given by
[Equation]
In this case, we update the probability of our hypothesis, H , based on the event of the observation of a new piece of evidence, X .
Our prior belief for the hypothesis is P( H ) , and posterior is P( H | X ) .
To update our belief from the prior and yield the posterior, we multiply by the Bayes factor for the evidence which is given by P( X | H ) / P( X ) .
Converting the probabilities into log-odds ratios (logits) yields the following representation of Bayes’
Rule.
[Equation]
Here, H C is the complement to H (the event that the hypothesis is false), and P( H C )
= 1 −
P( H ) .
Our prior log-odds ratio is log (cid:0) ( P( H ) / P( H C ) (cid:1) , and our posterior after updating based on the observation of new evidence X is log (cid:0) P( H | X ) / P( H C | X ) (cid:1) .
To update our belief from the prior and yield the posterior, we add the log-odds Bayes factor for the evidence which is given by log (cid:0) P( X | H ) /
P( X | H C ) (cid:1) .
In log-odds space, a series of updates with multiple pieces of independent evidence can be performed at once with a summation operation.
[Equation]
This operation can be represented by the linear layer in an artiﬁcial neural network, z k = b k + w Tk x .
Here, the bias term b k = log (cid:0) ( P( H ) / P( H C ) (cid:1) is the prior for hypothesis (the presence of the feature represented by the k-th neuron), and the series of weighted inputs from the previous layer, w ki x i provide evidential updates.
This is also equivalent to the operation of a multinomial naïve Bayes classiﬁer, expressed in log-space, if we choose w ki = log
p ki
(Rennie et al., 2003).
A.4 Difference between AIL and IL functions
Here, we measure and show the difference between the true logit-space operations and our AIL approximations, shown in Fig.
9, Fig. 10, and Fig.
In each case, we observe that the magnitude of the difference is never more than 1, which occurs along the boundary lines in AIL.
Since the magnitude of the three functions increase as we move away from the origin, the relative difference decreases in magnitude as the size of x and y increase.
A.5 Gradient of AIL and IL functions
We show the gradient of each of the logit-space Boolean operators and their AIL approximates in
Fig.
12, Fig. 13, and Fig.
By the symmetry of each of the functions, the derivative with respect to y is a reﬂected copy of the gradient with respect to x .
We ﬁnd that the gradient of each AIL function closely matches that of the exact form.
Whilst there are “dead” regions where the gradient is zero, this only occurs for one of the derivatives at a time (there is always a gradient with respect to at least one of x and y ).
A.6 Activation pathing diagrams
Here we illustrate the implications of using a 2 → 1 activation function on the network achitecture, and the layout of the partition and duplication ensembling methods described in §3.5.
A standard 1D activation function with a 1 → 1 mapping such as ReLU processes each pre-activation logit independently, and the number of post-activation channels equals the pre-activation channels (Fig. 15a).
The total number of parameters per layer is C 2 + C ∼ C 2 .
[Figure]
a function such as OR function requires two
If we use a combination of 2 → 1 activation functions with the partition ensembling strategy, each pre-activation neuron is used once, and only seen by one activation function.
The number of output channels is again half that of the input, as seen in Fig. 15c and e.
Using the duplication ensembling strategy, each 2 → 1 activation function is applied in parallel to the same set of operands (Fig. 15d and f).
The number of output channels scales up with the number of activation functions used in the ensemble.
With two activation functions, the ensemble performs a 2 → 2 mapping; the number of channels is unchanged by the layer and the number of parameters per layer is equal to that of a 1 → 1 activation function, C 2 + C ∼ C 2 .
A.7 Normalization methodology
To normalize the IL and AIL functions, we assumed the pair of arguments to the activation function were drawn independently from the standard normal distribution, N (0 , 1) , with probability density function
[Equation]
We found the expected value of the operator, and the standard deviation, and then subtracted by the mean and divided by the standard deviation,
[Equation]
For the normalized version of the exact operators (OR NIL , etc.), the mean and standard deviation were estimated empirically using 600 million samples.
For our AIL functions, we determined the mean and standard deviation analytically by integration.
The values are shown in Table 2.
[Table]
AND
[Table]
XNOR
[Table]
OR
[Table]
AND AIL
[Table]
XNOR AIL
[Table]
A.7.1 Derivations for OR AIL
Here, we analytically derive the mean and variance of OR AIL .
Recall
OR AIL
[Equation] [Equation]
x > 0 , y >max( x, y ) otherwise which can be rewritten as
OR
[Equation]
Proposition A.1.
The expected value of OR
AIL
( x, y ) for independently sampled x, y ∼ N (0 , 1) is
[Equation]
OR
[Equation]
Proof.
Using Eq. 13,
[Equation]
Let us consider the ﬁrst component, where x > 0 y > 0 .
[Equation]
Note that the integral of the probability density function φ ( x ) is its cumulative distribution function,
[Equation]
[Equation]
which has the limits lim u →−∞ Φ( u )
=
and lim u →∞ Φ( u ) = 1 , and central value Φ(0)
= 0 .
The probability density function φ ( x ) has limits lim u →−∞ φ ( u ) =
lim u →−∞ φ ( u ) = 0 , and central value Φ(0)
= 1 / √ 2 π .
From the integrals of Gaussian functions given in Owen (1980), we have
[Equation]
Thus,
[Equation]
=
[Equation]
Next let us consider the second component of the integral.
[Equation]
By change of variables,
[Equation]
Thus we conclude
[Equation]
Proposition A.2.
The variance of OR AIL ( x, y ) for independently sampled x, y ∼ N (0 , 1) is
Var
[Equation]
OR AIL
[Equation]
Proof.
Using Eq. 13,
E [OR AIL
[Equation] [Equation]
Let us consider the ﬁrst component, where x > 0 , y >
[Equation]
From Owen (1980) we note the identity
[Equation]
Additionally, we note that
[Equation]
Using this in addition to Eq.
[Equation]
Next we consider the second component,
[Equation]
From Equation n1 of Owen (1980),
[Equation]
Additionally from Owen (1980),
[Equation] [Equation]
Thus
[Equation]
By change of variables,
[Equation]
Thus we conclude
E [OR AIL
[Equation]
[Equation] [Equation]
−
[Equation]
π =4 + 2 π
Recall that
Var( X ) =
E [( X − E
[ X ]) 2 ] =
E [ X 2 ] − E
[ X ] 2 .
Hence
Var(OR
AIL ( x, y ))
= 5 4 + 1 2 π − (cid:18) 1 √ 2 π + 2 √ π (cid:19)=4 + 2 π − (cid:18)2 π +1 √ 2 π 2 √ π + 4 π (cid:19)
=4 + 2 π − (cid:18)4 π + 1 √ 2 π (cid:19) =
4 − 1 √ 2 π − 4 π
A.7.2 Derivations for AND AIL
Follows from AND AIL ( x, y )
=
− OR AIL ( − x, − y ) .
A.7.3 Derivations for XNOR AIL
Here, we analytically derive the mean and variance of XNOR AIL .
Recall
XNOR
[Equation] [Equation]
Proposition A.3.
The expected value of XNOR AIL ( x, y ) for independently sampled x, y ∼ N (0 , 1) is 0 .
Proof.
The expected value is given by
[Equation]
Note that φ ( x ) is an even function, i.e. φ ( x ) =
φ ( − x ) .
Moreover, XNOR AIL ( x, y ) is an odd function with XNOR AIL ( − x, y )
= − XNOR AIL ( x, y ) .
Thus XNOR AIL ( x, y ) φ ( x ) φ ( y ) is an odd function.
Therefore,
[Equation]
Proposition A.4.
The variance of XNOR AIL ( x, y ) for independently sampled x, y ∼ N (0 , 1) is
[Equation]
Proof.
Recall that
[Equation]
Using Proposition A.
[Equation]
since x ∈ R , y ∈ R .
[Equation]
The integrand has three symmetries: f ( − x ) =
f
( x ) , f ( − y ) =
f ( y ) , and f ( x, y ) =
f ( y, x ) .
Using this, we can break the integral up into 8 slices, each of which have an equal area, A .
We next integrate over one of these slices to ﬁnd A .
[Equation]
Equation (1;012-1) from Owen (1980) provides us with the identity (cid:90) ∞
[Equation]
Thus,
[Equation]
=
[Equation]
 ∞
y
[Equation]
(cid:35)
[Equation]
Corollary A.4.1.
The standard deviation of XNOR AIL ( x, y ) for independently sampled x, y ∼
N (0 , 1) is
[Equation]
A.8 Dataset summary
The datasets used in this work are summarized in Table
[Table] [Table]
We used the same splits for Caltech101 as used in Kabir et al.
The Covertype dataset was chosen as a dataset that contains only simple features (and not pixels of an image) on which we could study a simple MLP architecture, and was selected based on its popularity on the UCI ML repository.
The Bach Chorales dataset was chosen because — in addition to being in continued use by ML researchers for decades — it presents an interesting opportunity to consider a task where logical activation functions are intuitively applicable, as it is a relatively small dataset that has also been approached with rule-based frameworks, e.g. the expert system by Ebcioglu (1988).
MNIST, CIFAR-10, CIFAR-100 are standard image datasets, commonly used.
We used small MLP and CNN architectures for the experiments on MNIST so we could investigate the performance of the network for many conﬁgurations (varying the size of the network).
We used ResNet-50, a very common deep convolutional architecture within the computer vision ﬁeld, on CIFAR-10/100 to evaluate the performance in the context of a deep network.
The datasets used for the transfer learning task are all common and popular natural image datasets, with some containing coarse-grained classiﬁcation (CIFAR-10), others ﬁne-grained (Stanford Cars), and with a varying dataset size ( 5000 — 75 000 training samples).
We chose to do an experiment involving transfer learning because it is a common practical situation where one must train only a small network that handles high-level features, and is the sort of situation which involves manipulating high-level features, relying on the pretrained network to do the feature extraction.
We considered other domains where logical reasoning is involved as a component of the task, and isolated abstract reasoning and compositional zero-shot learning as suitable tasks.
For abstract reasoning, we wanted to use an IQ style test, and determined that I-RAVEN was a state-of-the-art dataset within this domain (having ﬁxed some problems with the pre-existing RAVEN dataset).
We determined that the SRAN architecture from the paper which introduced I-RAVEN was still state-of-the-art on this task, and so used this.
Another problem domain in which we thought it would be interesting to study our activation functions was compositional zero-shot learning (CZSL).
This is because the task inherently involves combining an attribute with an object (i.e. the AND operation).
For CZSL, we looked at SOTA methods on https://paperswithcode.com .
The best performance was from SymNet, but this was only implemented in TensorFlow and our code was set up in PyTorch so we built our experiments on the back of the second-best instead, which is the TMN architecture.
In the TMN paper, they used two datasets: MIT-States and UT-Zappos-1.
In our preliminary experiments, we found that the network started overﬁtting on MIT-States after around 6 epochs, but on UT-Zappos-1 it was overﬁtting after the ﬁrst or second epoch (one can not tell beyond the fact the val performance is best for the ﬁrst epoch).
In the context of zero-shot learning, an epoch uses every image once, but there are also only a ﬁnite number of tasks in the dataset.
Because there are multiple samples for each noun/adjective pair, and each noun only appears with a handful of adjectives and vice versa, there is in a way fewer tasks in one epoch than there are images.
Hence it is possible for a zero-shot learning model to overﬁt to the training tasks in less than one epoch (recall that the network includes a pretrained ResNet model for extracting features from the images).
For simplicity, we dropped UT-Zappos-1 and focused on
MIT-States.
A.9 Experiment conﬁguration and hardware
Experiments were run using PyTorch (typically v1.10) on an internal cluster of NVIDIA RTXand
Tesla T4 GPUs.
Our experiments each used either 1 or 4 GPUs.
A.10 Statistical Methodology
Here we provide information on how we carried out some of the detailed comparisons between activation functions.
In our experiments on Bach chorales (§4.2), MNIST (§4.3), CIFAR-10/100 (§4.4), and Covertype (§A.12), we explored the performance of networks using various activation functions whilst changing the number of parameters in the network.
This was achieved by scaling up the width of the networks, whilst keeping the depth (number of layers) ﬁxed.
Since the different structure of 1-to-1 and 2-to-1 activations (e.g. ReLU vs Max and the activations introduced in this paper) imply a different organization of the network, to explore the same range of values for the parameter count, the width values spanned different ranges depending on the activation function.
Note that in other cases, it made more sense to preserve a characteristic such as the size of the embedding space, in which case we discuss this explicitly (e.g. §A.19).
In the cases where we explore a range of values for the parameter count, we adopt the following methodology for our statistical tests for the “best” activation function.
First we collapsed across the number of params dimension by selecting the best width value for each activation function.
Typically this is the widest value, but sometimes some activation functions had their performance peak with fewer parameters (especially on Covertype, where we had disabled weight decay).
Then we identiﬁed the best performing activation function as the one with the highest accuracy.
For example, in the case of MLP on MNIST (§4.3), the best activation function (over all num.
params considered) was XNOR AIL .
We compared its performance to each of the other activation functions using a (non-paired) two-tailed Student’s t -test, implemented with scipy.stats.ttest_ind , one test for each of the other activations.
The total number of tests performed is ( num_actfuns − 1) .
We did not use a Bonferroni correction for multiple comparisons.
In the case of the MLP on MNIST, the p-value vs SignedGeomean was between 0.05 and 0.1 and hence did not meet our threshold for statistical signiﬁcance.
For all the other activations, the p-value was less than 0.01, more than exceeding our threshold for signiﬁcance.
Hence, we found that XNOR AIL was signiﬁcantly better than all other activations besides SignedGeomean .
Note that this methodology (collapsing across the number of parameters by selecting the best for each activation function) compares each of the activation functions at their best.
This is in keeping with how results are frequently reported in the literature (i.e. in a table showing one value per comparator, after performing a hyperparam search), and makes it possible to do a simple statistical test, such as a t -test.
However, such a simpliﬁcation does not tell the whole story about the results.
For example,
Fig.
5 also shows that for the MLP, XNOR-shaped activation functions also lose performance sooner than other activation functions when the number of parameters is reduced.
Meanwhile for the CNN, some activation functions which perform highly with a large number of parameters maintain high performance even as the number of parameters is reduced, while others do not.
This seemed to be associated with those activation functions that included Max/OR operations as opposed to those that did not.
The methodology we used to evaluate the performance was the same as described above for the MLP.
Selecting the best width for each activation function, we found there arefunctions/ensembles which have indistinguishably best performance: (OR, AND, XNOR AIL
(p),
OR, XNOR AIL (d/p), Max, and SiLU) performed best, whilst other activations had signiﬁcantly lower performance than top-performing activation function, p < 0 .
A.11 Parity Experiments
Our parity experiment shown in §4.1 was trained for 100 epochs using ADAM, one-cycle learning rate schedule, max LR 0 . 01 , weight decay 1 × 10 − 4 .
Following on from the parity experiment described in the main text, we also introduced a second synthetic dataset with a labelling function that, while slightly more complex than the ﬁrst, was still solvable by applying the logical XNOR operation to the network inputs.
In this dataset increased our number of inputs to 8, and derived our labels by applying a set of nested XNOR IL operations:
[Equation]
For this more difﬁcult task we also reformulated our initial experiment into a regression problem, as the continuous targets produced by this labelling function are more informative than the rounded binary targets used in the ﬁrst experiment.
We also adjusted our network setup to have an equal number of neurons at each hidden layer as we found that this signiﬁcantly improved model performance 2 .
We again trained using the same model hyperparameters for 100 epochs.
While this time the model was not able to learn a sparse weight matrix that exactly reﬂected our labelling function (see Fig. 16), the model was again able to leverage the XNOR AIL activation function to signiﬁcantly outperform an identical model utilizing the ReLU activation function.
We found that a simple model with three hidden layers, each with eight neurons, utilizing XNOR AIL was able to go from a validation RMSE of 0.287 at the beginning of training to a validation RMSE of 0.016 after 100 epochs.
Comparatively, an identical model utilizing the ReLU activation function was only able to achieve a validation RMSE of 0.271 after 100 epochs.
In order for our ReLU network to match the validation RMSE of our 8-neuron-per-layer XNOR AIL model, we had to increase the model size by 32 times to 256 neurons at each hidden layer.
[Figure]
A.12 MLP on Covertype
We trained small one, two, and three-layer MLP models on the Covertype dataset (UCI Machine
Learning Repository, 1998; Blackard & Dean, 1999) from the UCI Machine Learning Repository.
The Covertype dataset is a classiﬁcation task consisting of 581 012 samples of forest coverage across 7 classes.
Each sample has 54 attributes.
Networks were trained using one-cycle (Smith & Topin, 2017; Smith, 2018) for 50 epochs, with a batch size of 1024 , without weight decay.
We used a ﬁxed random 80:20 train:test split throughout our experiments.
The learning rate was selected using an automated learning rate ﬁnder approach.
No data augmentation was performed.
For each activation function, we varied the number of hidden units per layer to investigate how the activation functions affected the performance of the networks as its capacity changed.
We did not use weight decay or data augmentation for this experiment, and so the network exhibits clear overﬁtting with larger architectures when the network is over-parameterized.
As shown in Fig.
17, we found that XNOR NAIL performed well on this task (outperforming ReLU), whilst OR NAIL and Max were less successful, performing worse than ReLU.
SignedGeomean performed well when using 1 hidden layer, but its relative performance drastically decreased as the
MLP depth was increased.
Tanh performed well throughout, and was best when using 2 hidden layers, but the margin vs XNOR NAIL was not statistically signiﬁcant (two-sided Student’s t -test, p> 0 . 05 ).
SiLU performed poorly at this task, and the ELU family of activation functions (of which SELU is shown in the ﬁgures) performed even worse (reaching only 94 .
15 ± 0 .
09% with 3 hidden layers).
A.13 Bach Chorale training details
For each of the 4 voices, we restricted the available pitches to 3 octaves, resulting in 37 one-hot tokens (including silence).
Since we only planned on feeding small time-windows of the chorale into our model, for pre-processing, we converted the data into a shape of ( L, 4 , 37) , where we set the sequence length to be L = 4 .
To generate training examples for the discriminator, we transposed by {− 5 , − 4 , . . .
, 5 , 6 } semitones, chosen uniformly at random, and there was a 0 .
5 probability that the sample was corrupted by the following method:
[Table]
OR
IL
[Table]
OR ,
[Table]
OR ,
[Table] [Table]
OR ,
[Table] [Table]
[Table]
OR , AND ,
[Table]
[Table] [Table]
OR , AND ,
[Table]
[Table]
OR
[Table]
OR , AND
[Table] [Table]
[Table]
OR , XNOR
[Table]
(p)
[Table]
OR , XNOR NIL (d)
[Table]
OR , AND , XNOR NIL (p)
[Table]
OR , AND , XNOR NIL (d)
[Table]
XNOR AIL
[Table]
OR
AIL
[Table]
OR , AND AIL (d)
[Table]
OR , XNOR AIL (p)
[Table]
OR , XNOR [Table] (d)
[Table]
OR , AND , XNOR
[Table]
(p)
[Table]
OR , AND ,
XNOR AIL (d)
[Table]
OR , AND
[Table] [Table]
OR ,
[Table]
[Table]
[Table]
OR , XNOR
[Table]
(d)
[Table]
OR , AND , XNOR NAIL (p)
[Table]
OR , AND , XNOR NAIL (d)
[Table]
Number of parameters
T e s t a cc u r a c y ( % )
Covertype, MLP (1 hidden layer)
ReLU PReLU
SELU SiLU
Tanh Max
Max, Min (d) SignedGeomean
XNOR NAIL OR NAIL
OR NAIL , AND NAIL (d) OR NAIL , XNOR NAIL (p) OR NAIL , XNOR NAIL (d) OR NAIL , AND NAIL , XNOR NAIL (p)
OR NAIL , AND NAIL , XNOR NAIL (d)
Number of parameters
T e s t a cc u r a c y ( % )
Covertype, MLP (2 hidden layers)
Number of parameters 95.
T e s t a cc u r a c y ( % )
Covertype, MLP (3 hidden layers)
• Choose 2–3 notes in the 4 L window to be corrupted • For each note, corrupt by the following mixture distribution: – ( p = 0 .
Sample a pitch from a Gaussian centered on the existing note, with σ =semitones, forcing the new pitch to be distinct – ( p = 0 .
2) Copy a pitch from the current voice, forcing the new pitch to be distinct – ( p = 0 . 2) Extend the previous note in time – ( p = 0 .
1) Sample uniformly from all 37 possible tokens
A.14 Correlations between pre-activations
To capture the existence of correlations, we took the cosine similarity between rows of the weight matrix.
Since the inputs to all features in a given layer are the same, this is equivalent to measuring the similarity between corresponding pair of pre-activation features.
Results on correlations between weights in the JSB Chorale models are shown in Fig.
We found that when taking all pre-activations into account, every activation function generally showed independence between features.
Interestingly, the cosine similarities between inputs that were paired together for the bivariate activation functions showed anticorrelation in almost all cases where Max or OR AIL were used, and other cases generally showed more correlation than ReLU .
í í         5H/ í í         35H/ í í         6L/ í í         0D[ í í         0D[0LQG
í í         ;125 AIL í í         25 AIL í í         25$1' AIL G í í         25;125 AIL S í í         25;125 AIL G í í         25$1';125 AIL S í í         25$1';125 AIL G
&RVLQHVLPLODULWLHVEHWZHHQDOOSUHDFWLYDWLRQVVWOD\HU í í         5H/ í í         35H/ í í         6L/ í í         0D[ í í         0D[0LQG
í í         ;125 AIL í í         25 AIL í í         25$1' AIL G í í         25;125 AIL S í í         25;125 AIL G í í         25$1';125 AIL S í í         25$1';125 AIL G &RVLQHVLPLODULWLHVEHWZHHQSDLUHGSUHDFWLYDWLRQV
VWOD\HU
We found that randomly selected pairs of preactivation features within the same layer have correlations that are given by a Gaussian-like distribution centered around zero.
This was the case for all of the activation functions we tested.
The behaviour of randomly selected pairs of features is thus reasonably consistent with the assumption of independence which we have made.
We also investigated the correlation between the pairs of preactivation features which were passing into our two-dimensional activation functions.
Here, we found the correlation structure is different, and the correlation depends on the activation function being used.
With Max and OR AIL activations, the network learns to make the columns of the weight matrix (and hence the preactivation scores for the pair of features) be inversely correlated.
With XNOR AIL , the network learns features which are either positively or negatively correlated (a wider distribution of correlations than seen with random pairs of features).
We observe that (in all cases) the network learns to make the features passed to the AIL activation functions be correlated instead of independent, despite our assumption of independence.
So it appears clear that the assumption of independence is violated, but also that it doesn’t really matter because the network is choosing to break the assumption and induce these correlations between the features to get better performance.
A.15 CNN and MLP on MNIST
In this experiment we trained MLP and CNN models for 10 epochs on the MNIST dataset using
ADAM optimizer, one-cycle learning rate schedule, and cross entropy loss, with batch size of 256.
We augmented our training samples using a random afﬁne transformation from: rotation of ±degrees, scale of factor 0 .
8 to 1 . 2 , translation with max absolute fraction for horizontal and vertical directions of 0 .
08 , and shear parallel to the x-axes of ± 0 .
The hyperparameters for the optimizer and scheduler were selected through a random search of the hyperparameter space.
We chose to do random search instead of grid search because it typically yields better results for the same number of test cases.
For the hyperparameter search, we trained on the ﬁrst 50 000 samples of the training partition and used the ﬁnal 10 000 samples as a validation set.
We ran the search for four iterations, with each iteration sampling 120 different hyperparameter settings.
The initial bounds for our hyperparameter samples were set as described in Table 5.
These bounds were chosen to be suitably wide such that the optimal conﬁguration should be contained within them for all activation functions considered.
[Table]
The bounds on our uniform random variable x were tightened at each iteration by selecting the topperforming hyperparameter settings, taking the mean ¯ x and weighted standard deviation σ across those settings, and re-setting the bounds for the next iteration to be equal to ¯ x ± 1 .
5 σ .
After the fourth iteration, we ran a ﬁnal iteration where we selected the top-10 performing hyperparameter settings across the four previous iterations, and then re-ran these for another 120 seeds (i.e. random weight initializations).
We then selected the hyperparameter settings which had the highest performance across all 120 seeds.
We found that the XNOR AIL activation function required quite different hyperparameters than the other activation functions, but AND AIL and Max activation functions used similar hyperparameters to ReLU.
However, we have no reason to believe that the proposed AIL activation functions are more susceptible than others to the choice of hyperparameters or the way in which hyperparameters are selected.
Our MLP model consisted of two hidden layers of equal size with batch norm applied to each.
The number of neurons in each layer was set, taking into consideration the current activation function being tested, to ensure the number of trainable parameters in the network remained static across experiments
Our CNN model consisted of six layers, where each layer was comprised of a set of 2D convolution ﬁlters (kernel size 3, stride 1, padding 1), batch normalization, and a non-linear activation.
The network also applied a pooling layer (kernel size 2, stride 2) at the end of every second layer.
After the 6 CNN layers the network ﬂattens the output and applies three linear layers.
The number of output channels (i.e. pre-activations) for the six 2D convolutions were [ c, 2 c, 4 c, 4 c, 8 c, 8 c ] , and the number of pre-activation neurons produced by the subsequent linear layers were [32 c, 16 c ] .
Similar to the
MLP model, c was chosen to ensure the number of trainable parameters in the network remained ﬁxed across experiments.
When varying the number of network parameters in our experiments, if the number of parameters dropped below 1 × 10 6 in the CNN model, we changed the structure of the convolution layers and linear layers to [ c, c, c, c, 1 . 5 c, 1 . 5 c ] and [2 c, 1 .
5 c ] , respectively.
This ensured that each layer had at least two channels for our activation functions to aggregate.
Once our optimal hyperparameters were selected for both our MLP and CNN models, we then trained each model several times with varying number of trainable parameters.
The reason we vary the number of parameters in the network instead of the network size/structure is because a network using our logical activation functions can have a signiﬁcantly different number of parameters than an identical network using ReLU activation, because our logical activations aggregate across neurons at each layer.
Our results are plotted in Fig.
5, and tabulated in Table 6 and Table 7.
[Table]
[Table]
OR
AIL
[Table]
OR , AND AIL (d)
[Table]
OR , XNOR AIL (
[Table]
[Table]
OR , XNOR AIL (d)
[Table]
OR , AND , XNOR AIL (
[Table]
[Table]
OR , AND
, XNOR AIL (d
[Table]
[Table]
[Table]
OR
[Table]
OR , AND AIL (d)
[Table] OR , XNOR AIL (p)
[Table]
OR , XNOR
[Table]
(d)
[Table]
[Table]
OR , AND ,
XNOR AIL (p)
[Table]
OR , AND ,
XNOR AIL (d)
[Table] A.16 ResNet50 on CIFAR-10/
In this experiment, we explored the impact of the choice of activation functions on the performance of a pre-activation ResNet50 model (He et al., 2016a,b) applied to the CIFAR-10 and -100 datasets.
Our base network was a pre-activation ResNet50 with 4 layers comprised of [3 , 4 , 6 , 3] bottleneck residual blocks
(He et al., 2016b) with an expansion factor of 4.
As described in the main text, we exchanged all ReLU activation functions in the network to a candidate activation function while maintaining the size of the pass-through embedding, with the exception of the activation function between the ﬁrst convolutional layer (stem) and the ﬁrst residual block which was ReLU throughout. We did not use any activation on the identity/pass-through branches of the residual blocks.
The base widths for the residual blocks were [64 , 128 , 256 , 512] , respectively.
We experimented with changing the width of the network, scaling up the embedding space and all hidden layers by a common factor, w .
We used width factors of 0.25, 0.5, 1, 2, and 4 for 1 → 1 activation functions (ReLU, etc), widths of 0.375, 0.75, 1.5, 3, and 6 for 2 → 1 activation functions (Max, etc), and widths of 0.2, 0.4, 0.75, 1.5, and 3 for { OR AIL , AND AIL , XNOR AIL (d)}.
The stem width was held constant at 64 channels throughout.
For this experiment we trained a ResNet50 model for 100 epochs on CIFAR-10 and CIFAR-100 using
ADAM optimizer, one-cycle learning rate schedule, cross entropy loss, and augmentations derived for
CIFAR-10 by AutoAugment (Cubuk et al., 2018), with batch size of 128.
The hyperparameters for the optimizer and scheduler were determined through a random search of the hyper parameter space on the CIFAR-100 dataset for a ﬁxed width factor w = 2 only.
We used the same set of hyperparameters from this search for both the CIFAR-10 and CIFAR-100 experiments.
The hyperparameter search was tuned against a partition of 10% of the CIFAR-100 training samples. Our hyperparameter search follows a similar approach to the one used for our MLP and CNN models on MNIST (§A.15), but due to compute constraints with our larger models here we only trained 100 seeds each iteration, and only re-examined the top 10 seeds in the ﬁnal round.
For PReLU, we did not perform a new hyperparameter search and simply re-used the same hyperparameters as discovered in the search with ReLU.
[Table]
OR
AIL
[Table]
OR , AND AIL (d)
[Table]
OR , XNOR AIL (p)
[Table]
OR , XNOR
[Table]
[Table]
OR , AND ,
XNOR AIL (p)
[Table]
OR , AND ,
XNOR AIL (d)
[Table]
The results are plotted in Fig. 6, and tablated in Table 8 and Table 9.
[Table]
OR , XNOR AIL (d)
[Table]
OR , AND ,
XNOR AIL (p)
[Table]
OR , AND ,
XNOR AIL (d)
[Table]
A.17 Transfer learning
We used a pretrained ResNet50 model taken from the PyTorch hub.
The 2-layer MLP head was trained for 25 epochs on each dataset.
Since we are interested in transfer learning in a data-limited regime for these experiments, we used only a small amount of data augmentation.
We applied horizontal ﬂip ( p =0 . 5 ), scaling (0.7 to 1), aspect ratio stretching ( 3 / 4 to 4 / 3 ), and colour jitter (intensity 0.4) only.
For the SVHN dataset, horizontal ﬂip was not performed. Pixel intensity normalization was done against the ImageNet mean and standard deviation.
The MLP head was optimized using SGD, momentum 0.9.
We used a batch size of 128, maximum learning rate 0.01, and weight decay 1 × 10 − 4 .
Before training began, we passed one epoch worth of inputs through the network without updating the weights in order to refresh the batch normalization statistics to the new dataset.
We also performed one epoch of training with a warmup learning rate of 1 × 10 − 5 before commencing training at the maximum learning rate of 0.01.
The learning rate was decayed with a cosine annealing schedule over 24 epochs.
We report the performance of the ﬁnal model at the end of the 25 epochs.
We used a pre-activation width of 512 neurons for ReLU and other and activation functions which map 1 → 1 .
To approximately match up the number of parameters, we used a width of 650 for activation functions which map 2 → 1 , and 438 for { OR , AND , XNOR AIL (d)} which maps features 3 → 2 .
These values control the total number of parameters in the head to be the same for datasets with 100 classes (the median number of classes in the datasets we considered).
Our results with widths 438/512/650 and ∼ 600k parameters are shown in the main results, Table 1.
To investigate a comprehensive set of baselines, we compared against every activation function implemented in
PyTorch v1.10.
This was too many comparisons to include in the main results, so for brevity we narrowed the activation functions down by omitting some activation functions where there were several similar functions and kept the better performing one (SiLU vs Mish, LeakyReLU vs PReLU, etc).
The full set of results is shown in Table 10.
Additionally, the precise widths and number of trainable parameters used for these experiments are shown Table 11.
Since the transfer learning experiments were performed without retraining the base network, we found (Table 1) the performance is limited by the features which the base network produces, whose relevance depends on the overlap between the domain of the pretraining task (ImageNet) and the new task.
This information bottleneck, and variation in its utility, means the variation between performance on different datasets is much larger than the variation for different activation functions within the same dataset.
For transfer tasks which involve coarse-grained discrimination on images which are similar to
ImageNet, the embedding generated by the pretrained model is already sufﬁcient to separate the
[Table]
OR ,
[Table]
, XNOR
[Table]
OR ,
[Table]
, XNOR
[Table]
OR
[Table]
OR , AND NIL (
[Table]
[Table]
OR , XNOR
[Table]
[Table]
[Table]
OR ,
[Table]
[Table]
OR , AND , XNOR NIL (p)
[Table]
OR , AND , XNOR NIL (d)
[Table]
OR AIL
[Table]
OR , AND AIL (d)
[Table]
OR , XNOR AIL (p)
[Table]
OR ,
[Table]
OR , AND ,
XNOR AIL (p)
[Table] OR , AND , XNOR
[Table]
[Table]
OR [Table]
OR , AND
[Table] classes in the new dataset.
Examples of this are Caltech101, where linear layer beats out using additional layers with non-linearities; and STL-10, where our best model is on-par with the linear model.
The results on these two transfer learning tasks are too simple to be of interest to study.
The performance is limited by the information retained by the pretrained embedding, but it appears that what task-relevant information is there is readily available with a linear layer without needing additional logic to interpret it.
Other transfer learning tasks we attempted are less trivial and show a larger difference in performance across the activation functions
[Table]
and a gap from the linear layer readout model.
The Stanford
Cars dataset involves ﬁne-grained discrimination between different car models, for which features generated by a model pretrained on ImageNet are not effective.
The SVHN dataset, which contains
[Table]
OR
[Table]
OR ,
[Table]
OR ,
[Table]
OR , [Table]
[Table]
p
[Table]
OR , AND , XNOR IL (d)
[Table]
OR
[Table]
549k 549k OR , AND NIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , XNOR NIL (p) 2 → 576k 546k 576k 576k 607k 546k 546k OR , XNOR NIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , AND , XNOR NIL (p) 2 → 576k 546k 576k 576k 607k 546k 546k OR , AND , XNOR NIL (d) 2 → 579k 519k 579k 580k 642k 519k 519k
XNOR AIL 2 → 578k 549k 578k 579k 609k 549k 549k OR AIL 2 → 578k 549k 578k 579k 609k 549k 549k OR , AND AIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , XNOR AIL (p) 2 → 576k 546k 576k 576k 607k 546k 546k OR , XNOR AIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , AND , XNOR AIL (p) 2 → 576k 546k 576k 576k 607k 546k 546k OR , AND , XNOR AIL (d) 2 → 579k 519k 579k 580k 642k
519k
[Table]
images of house numbers, is coarse-grained but uses images which are outside the domain of
ImageNet, which makes the transfer-learning task more difﬁcult.
As shown in Table
[Table]
we found the duplication ensemble strategy consistently outperformed the partition strategy on the transfer learning experiments.
Compared to using OR NAIL , there was no beneﬁt to using an ensemble with the partition strategy.
However, it was beneﬁcial to ensemble
OR NAIL with AND NAIL , and better still with XNOR NAIL , under the duplication strategy.
The performance of duplication-ensembles using XNOR NAIL ,
i
[Table]
e. { OR NAIL , AND NAIL , XNOR NAIL (d)} and { OR NAIL , AND NAIL , XNOR NAIL (d)}, was comparable to using XNOR NAIL alone, so in this case an ensemble was neither beneﬁcial nor detrimental. Additionally, we ran the experiment again with a pre-activation width of w = 512 for all activation functions.
The results with constant pre-activation width w = 512 are shown in Table 12.
By using a constant pre-activation width, the number of parameters used is not consistent across activation functions.
The number of trainable parameters in each experiment is shown in Table 13.
Note: some experiments on SVHN are missing due to an issue with the job scheduler which was not noticed in time to run the missing experiments.
[Table]
OR , AND IL
[Table]
d)
[Table]
OR , XNOR
[Table]
OR , XNOR
[Table]
OR , AND , XNOR IL (
[Table]
[Table]
OR , AND , XNOR IL (d)
[Table]
OR NIL
[Table]
OR , AND NIL (d)
[Table]
OR , XNOR NIL (p)
[Table]
OR ,
[Table]
OR , AND , XNOR NIL (p)
[Table]
OR , AND , XNOR NIL (d)
[Table]
XNOR AIL
[Table]
OR
[Table]
OR , AND AIL (d)
[Table]
OR , XNOR AIL (
[Table]
[Table]
OR , XNOR AIL (
[Table]
[Table]
OR , AND ,
XNOR AIL (p)
[Table]
OR , AND , XNOR AIL
[Table]
[Table]
OR , AND NAIL (d)
[Table]
OR ,
XNOR NAIL (p)
[Table]
OR , XNOR NAIL (d)
[Table]
OR , AND , XNOR NAIL (p)
[Table]
OR , AND , XNOR NAIL (d)
[Table]
We found that reducing the width from 650 to 512 (and hence reducing total number of trainable parameters) reduced the performance of 2 → 1 activation functions XNOR AIL , OR AIL , { OR , XNOR AIL (p)}, and { OR , AND , XNOR AIL (p)}.
Increasing the width from 438 to 512 increased the performance of { OR , AND , XNOR AIL (d)}. [Table]
530k 530k
GLU 2 → 420k 397k 420k 420k 445k 397k 397k Max 2 → 420k 397k 420k 420k 445k 397k 397k Max , Min (d) 2 → 577k 530k 577k 578k 626k 530k 530k
SignedGeomean 2 → 420k 397k 420k 420k 445k 397k 397k
XNOR IL 2 → 420k 397k 420k 420k 445k 397k 397k OR IL 2 → 420k 397k 420k 420k 445k 397k 397k OR , AND IL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , XNOR IL (p) 2 → 420k 397k 420k 420k 445k 397k 397k OR , XNOR IL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , AND , XNOR IL (p) 2 → 418k 395k 418k 418k 442k 395k 395k OR , AND , XNOR IL (d) 2 → 734k 664k 733k 735k 807k 664k 664k
XNOR NIL 2 → 420k 397k 420k 420k 445k 397k 397k OR NIL 2 → 420k 397k 420k 420k 445k 397k 397k OR , AND NIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , XNOR NIL (p) 2 → 420k 397k 420k 420k 445k 397k 397k OR , XNOR NIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , AND , XNOR NIL (p) 2 → 418k 395k 418k 418k 442k 395k 395k OR , AND , XNOR NIL (d) 2 → 734k 664k 733k 735k 807k 664k 664k
XNOR AIL 2 → 420k 397k 420k 420k 445k 397k 397k OR AIL 2 → 420k 397k 420k 420k 445k 397k 397k OR , AND AIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , XNOR AIL (p) 2 → 420k 397k 420k 420k 445k 397k 397k OR , XNOR AIL (d) 2 → 577k 530k 577k 578k 626k 530k 530k OR , AND , XNOR AIL (p) 2 → 418k 395k 418k 418k 442k 395k 395k OR , AND , XNOR AIL (d) 2 → 734k 664k 733k 735k 807k 664k 664k
XNOR NAIL 2 → 420k 397k 420k 420k 445k 397k 397k OR NAIL 2 → 420k 397k 420k
[Table]
[Table]
[Table]
A.18 Abstract reasoning
Abstract reasoning is challenging for neural networks to learn because their structure and objective function are more effective for tasks which come instinctively to humans (“System 1” of Kahneman, 2011), such as object recognition, as opposed to demanding (“System 2”) logic tasks.
In recent years, several challenges have been proposed to evaluate the ability of neural networks to perform abstract reasoning.
The Raven’s Progressive Matrices (Raven & Court, 1938) are a long-standing IQ test, which have been emulated by Barrett et al.
(2018) with Procedurally Generated
Matrices (PGM), and then the RAVEN task by (Zhang et al., 2019).
Recently Hu et al.
(2021) and
Benny et al.
(2021) have improved on that task with I-RAVEN and RAVEN-FAIR, respectively, both aiming to make the task more balanced.
Other abstract reasoning tasks have also been proposed (Fleuret et al., 2011; Johnson et al., 2017; Barrett et al., 2018).
We considered the application of AIL activation functions in the context of the I-RAVEN task, by adapting the Stratiﬁed Rule-Aware Network (SRAN) of Hu et al.
(2021) to include our AIL activation functions.
We ﬁrst added LayerNorm to the network, and then swapped out the seven
ReLU activations in the gating module.
The architecture for the three ResNet-18 (He et al., 2016a) base models were unchanged.
Where necessary, the number of units per layer was modiﬁed to facilitate the change in dimensionality caused by our activation function.
The networks were trained using the same procedure as described by Hu et al.
[Table]
Max
[Table]
[Table] XNOR AIL
[Table]
OR AIL
[Table]
OR , AND
[Table]
[Table]
[Table]
OR , XNOR AIL (p)
[Table]
OR , XNOR AIL (
[Table]
[Table]
OR , AND , XNOR AIL (
[Table]
[Table]
We found the network using OR AIL activation function performed best overall, and across most of the subtasks.
The second best performing activation functions were { OR AIL , XNOR AIL (d)} and
ReLU.
A.19 Compositional Zero-Shot Learning
Zero-shot learning encompasses all problems which involve completing novel tasks which the subject has never seen before.
The subject must infer both the task and its solution based on their previous experiences (a meta-learning task). Compositional zero-shot learning is a subset of zero-shot learning which involves combining knowledge about multiple stimulus properties together in novel pairings.
For instance, if the network has been trained on “sliced bread,” “sliced pear,” and “caramelized pear,” is it able to classify images of “caramelized bread” despite having never seen an example of this before?
We based our experiments on the Task-driven Modular Networks (TMN) proposed by Purushwalkam et al.
We used the code shared by the authors, but were unable to replicate the results they reported in the paper when using a different random seed (see Table 15).
We adapted this network by changing out all the ReLU activation functions in the gate and module networks with a different activation function.
Because the modules each terminate with an activation function, we needed to double the size of the hidden layer for some of our networks in order to maintain the dimensionality of the output.
Consequently, some experiments had around 50% more parameters in total than others. Experiments were performed on the MIT-States dataset (Isola et al., 2015).
We trained and tested on the corresponding partitions of the dataset as introduced by Purushwalkam et al.
We used the same paradigm as Purushwalkam et al.
(2019): ADAM (Kingma & Ba, 2015) with a learning rate of 0.001 for the module network and 0.01 for the gating network, momentum 0.9, batch size 256, weight decay 5 × 10 − 5 .
We evaluated the network using the AUC between seen and unseen samples (Purushwalkam et al., 2019).
The network was trained until the validation AUC had plateaued, determined by not increasing for 5 epochs. We selected the model from the epoch with highest validation AUC to apply to the test set.
The best performance was typically attained after aroundepochs.
As shown in Table [Table]
we ﬁnd that OR AIL performs best, but uses more parameters, and is very closely followed by { OR , XNOR AIL (d)},.
[Table]
Since we could not ﬂexibly scale the total number of parameters in the network with the original architecture
[Table]
we modiﬁed the TMN architecture by adding an additional linear layer to the end of each module which projects from the activation function to the embedding space.
The modules would otherwise terminate with an activation function, which makes it difﬁcult to handle activation functions which map 2 → 1 .
We dub the modiﬁed version of the network “TMNx.”
Comparing the TMN results in Table 15 to TMNx in Table 16, we can see that adding the extra linear layer improved performance of the network in of itself.
Intuitively, this makes sense since the output of the TMN modules are weighted with the output of the gating network and then summed, and this weighting and summing of evidence is best performed with on logits instead of the truncated output of ReLU units.
But also, performance may have improved just because the model became larger.
We performed a hyperparameter search across the training parameters for the new network against the validation set using the ReLU activation function only.
We adopted the hyperparameters discovered for ReLU for all other activation functions.
The batch size was reduced to 128 due to the increase in size of the model.
The discovered hyperparameters were a learning rate of 3 × 10 − 3 for both the module and gating network, and a weight decay of 1 × 10 − 5 .
Other training hyperparameters, such as the ratio of negative samples to present, were left unchanged.
In order to match the number of parameters in the network, we used the original TMN hidden width of 64 for the module and gater networks with activation functions which map 1 → 1 , and increased this to 70 for activation function which map 2 → 1 , to maintain the total number of trainable parameters.
To investigate a comprehensive set of baselines, we compared against every activation function implemented in PyTorch v1.10.
The results are shown in Table 16.
We found that the ELU family of activations (ELU, CELU, SELU) performed best across all three top-k values.
Our best activation functions were OR (N)IL and { OR , AND (N)IL (d)}, which came next alongside Softplus and Tanh, outperforming ReLU.
Activation functions using XNOR and our
AIL approximations performed less well on this task, and were did not surpass ReLU.
These results suggests this is a domain where logical activation functions are less well suited.
[Table]
OR , AND IL (
[Table]
[Table]
OR , XNOR IL (p)
[Table]
OR , XNOR IL (d)
[Table]
OR , AND , XNOR IL (p)
→ 1.19M 2 .
50 ± 0 .6 .
34 ± 0 .10 .
05 ± 0 .
24 OR , AND , XNOR IL (d) 2 → 1.15M 2 .
39 ± 0 .6 .
31 ± 0 .10 .
17 ± 0 .
XNOR NIL
→ 1.16M 2 .
08 ± 0 .5 .
55 ± 0 .9 .
11 ± 0 .
10 OR
NIL
→ 1.16M 2 .
92 ± 0 .7.42 ± 0 .11 . 74 ± 0 .
14 OR , AND NIL (d) 2 → 1.15M 2 .
82 ± 0 .7.43 ± 0 .11 .
69 ± 0 .
19 OR , XNOR NIL (p)
→ 1.19M 2 .
50 ± 0 .6 .
51 ± 0 .10 .
40 ± 0 .
17 OR , XNOR NIL (d) 2 → 1.15M 2 .
48 ± 0 .6 .
56 ± 0 .10 .
63 ± 0 .
12 OR , AND , XNOR NIL (p) 2 → 1.19M 2 .
48 ± 0 .6 .
54 ± 0 .10 .
38 ± 0 .
08 OR , AND , XNOR NIL (d) 2 → 1.15M 2 .
54 ± 0 .6 .
74 ± 0 .10 .
72 ± 0 .
XNOR AIL 2 → 1.16M 1 . 88 ± 0 .4 .
94 ± 0 .7 .
99 ± 0 .
19 OR
AIL 2 → 1.16M 2 .
61 ± 0 .6 .
83 ± 0 .10 .
97 ± 0 .
20 OR , AND AIL (d) 2 → 1.15M 2 .
81 ± 0 .7 .
12 ± 0 .11 .
27 ± 0 .
14 OR , XNOR AIL (p)
→ 1.19M 2 .
50 ± 0 .6 .
34 ± 0 .10 .
07 ± 0 .
21 OR , XNOR AIL (d) 2 → 1.15M 2 .
46 ± 0 .6 .
51 ± 0 .10 .
45 ± 0 .
17 OR , AND , XNOR AIL (p) 2 → 1.19M 2 .
45 ± 0 .6 .
33 ± 0 .10 .
09 ± 0 .
23 OR , AND , XNOR AIL (d) 2 → 1.15M 2 .
57 ± 0 .6 .
83 ± 0 .10 .
97 ± 0 .
XNOR NAIL 2 → 1.16M 1 .
78 ± 0 .4 .
84 ± 0 .7 .
95 ± 0 .
19 OR
NAIL 2 → 1.16M 2 .
51 ± 0 .6 .
58 ± 0 .10 .
56 ± 0 .
24 OR , AND NAIL (d) 2 → 1.15M 2 .
67 ± 0 .6 .
98 ± 0 .11 .
02 ± 0 .
16 OR , XNOR NAIL (p)
→ 1.19M 2 .
29 ± 0 .6 .
13 ± 0 .9 .
95 ± 0 .
17 OR , XNOR NAIL (d) 2 → 1.15M 2 .
43 ± 0 .6 .
32 ± 0 .10 .
28 ± 0 .
18 OR , AND , XNOR NAIL (p) 2 → 1.19M 2 .
45 ± 0 .6 .
34 ± 0 .10 .
11 ± 0 .
12 OR , AND , XNOR NAIL (d) 2 → 1.15M 2 .
58 ± 0 .6 .
85 ± 0 .10 .
94 ± 0 .