A Deep Architecture for Log-Linear Models
Abstract
We present a novel perspective on deep learning architectures using a partial order structure, which is naturally incorporated into the information geometric formulation of the log-linear model.
Our formulation provides a different perspective of deep learning by realizing the bias and weights as different layers on our partial order structure.
This formulation of the neural network does not require any gradients and can efﬁciently estimate the parameters using the EM algorithm
Introduction
Log-linear models and energy based models (EBMs) are a family of models which are widely used in statistics and machine learning.
Log-linear models provide the relationship between probabilistic models and structured frameworks, such as a graphical model.
In this paper, we investigate the use of log-linear models as a means of framing a deep-learning architecture in a probabilistic manner.
Our approach is based on the em -projection algorithm (which stands for exponential-mixture) to estimate parameters of the log-linear model (Amari, 1995).
The m -projection estimates the expectation of the latent variables given the current parameters of the model, while the e -projection estimates the parameters in the model via maximum likelihood estimation (MLE).
(Amari, 1995), show that the em -projection algorithm in information geometry coincides with the statistical EM algorithm (which stands for Expectation-Maximization)
(Dempster et al., 1977), in a majority of cases.
However, the em -projection algorithm usually requires an iterative approach to estimate the parameters in the e -projection (Amari, 1995), making the algorithm less efﬁcient than the iterative approach.
In this paper, we formulate the deep-learning architecture as a poset (partially ordered set)
(Gierz et al., 2003).
The partial order structure decomposes the model-parameters into two hierarchical layers that represent the bias and the edge weights to allow for a closed form analytical expression for the for the maximum likelihood estimation which is typically computed by the e -projection in the projection algorithm or M-step in the statistical EM algorithm.
Our proposed approach allows us to efﬁciently update parameters without any gradients and therefore does not have any issues such as vanishing gradients are not encountered in our formulation.
Preliminary on Neural Networks
A perceptron (Rosenblatt, 1958) is a mathematical neuron deﬁned as an element which receives
M ( t − 1) ∈
Z + inputs u
[Equation]
t
[Equation]
=
[Equation]
[Equation] [Equation] ) and emits an output o ( t ) j ∈ R , where t denotes the layer of the current neuron and t − 1 denotes the layer with all the incoming neurons, and M ( t − 1) is the number of neurons at layer t − 1 .
Parameters w ( t − 1) i ∈ R of the neuron represents the edge weights of each input u ( t − 1)
i ∈ R and b ( t ) j ∈ R to be the threshold for activation.
Then the integrated input of neuron j at layer t , u ( t ) j is calculated by the weighted sum of inputs at layer t − 1 and is given by
[Equation]
where the output of the neuron, given some activation function σ , is computed by o ( t )
j = σ (cid:16) u ( t ) j (cid:17) , where σ is typically chosen to be a sigmoid function or a Rectiﬁed linear Unit (ReLU) for the hidden layers and a softmax function for the output layer.
Perceptrons can be hierarchically stacked with each other to construct a multi-layered perception where each layer is denoted as t with a total of T layers in the entire network.
This deﬁnition of the multi-layered perception is often referred to as a vanilla neural network.
For this study, we deﬁne the input at layer t = 0 to be x = [ x 1 , . . .
, x M (0) ] ∈ R M (0) , where x k
= o (0) k
= σ ( u (0) k ) with the activation function that linearly scales the feature between 0 and 1 .
We denote the output of the hidden layers to be h ( t ) =
[ h ( t ) 1 , . . .
, h ( t ) M ( t ) ]
∈ R M
( t ) for t ∈ { 1 , . . .
, T − 1 } , where h ( t ) k
= o ( t ) k
= σ ( u ( t ) k ) with the sigmoid or the ReLU activation function for k = 1 , . . .
, M ( t ) , for t = 1 , . . .
T − 1 .
The output for layer T is deﬁned to be y =
[ y 1 , . . .
, y M ( T ) ] , where y k
= o ( T ) k
= σ ( u ( T ) k ) with a softmax activation function for classiﬁcation with the domain (0, 1) and ReLU for regressions with the domain R .
Connection Between Binary Log-Linear Model and Partial Orders
In this section we introduce the relationship between the log-linear model, (Agresti, 2012), and a poset (Ω , (cid:22) )
(Sugiyama et al., 2016, 2017).
We ﬁrst let Ω be the sample space of distributions and (cid:22) denotes the partial order between elements in Ω .
If interactions between variables are represented as a partial order between elements in Ω ; that is, if each element in Ω represents a variable conﬁguration and the resulting set Ω is a poset, the interactions can naturally be treated as a log-linear model.
In the poset we denote the least element by ⊥ and Ω + = Ω \ {⊥} .
For a given subset, S ⊆ Ω + , the log-linear model with parameters ( θ ( s ))
s ∈S is given as,
[Equation]
for each ω (cid:48) ∈ Ω .
The parameters θ in the model coincide with the deﬁnition of the natural parameter in the exponential family distribution.
The partition function ψ ( θ ) ∈ R can be uniquely obtained by ψ ( θ )
= log (cid:80) ω
[Equation]
+ exp[ (cid:80) [Equation] [Equation] (cid:22) ω (cid:48) ]
θ ( ω )]
= − θ ( ⊥ ) .
The information geometric structure of the set of distributions, G = { p | 0 < p ( ω ) < 1 for all ω ∈ Ω and (cid:80) ω ∈ Ω + p ( ω ) = 1 } , arises when we introduce the expectation parameter, given as
[Equation]
Then the pair ( θ, η ) becomes a dual coordinate system of the statistical manifold G , and they are orthogonal, because they are connected via a Legendre transformation.
The orthogonality will be used in optimization of the model, where θ and η are jointly used to achieve minimization of the KL divergence via the projection algorithm.
Implementing Deep Architecture as Partial Orders
We introduce our key technical contribution by formulating a neural network on a partial order structure.
First we deﬁne the sample space Ω of the log-linear model to be Ω + = X ∪ Y ∪ H ∪ W , where
X is the set of input nodes, Y the set of output nodes, H the set of hidden nodes and W the set of edge weights.
Each entry in the set is denoted as, x ∈ X , y ∈ Y , h ∈ H and w ∈ W representing an input node, an output node, a hidden node, and an edge weight, respectively.
We use an underline to distinguish an element of Ω and the corresponding node in a neural network.
We also denote by xxxh (1) h (1) h (2) h (2) yyyw (1)11 w (1) w (1)21 w (1) w (1)31 w (1) w (2)11 w (2) w (2)21 w (2) w (3)11 w (3)12 w (3) w (3)21 w (3)22 w (3) ⊥
FirstOrderInteractionLayer(Bias)
SecondOrderInteractionLayer(EdgeWeights)
PosetNormalizationTerm n ∈ N = X ∪ Y ∪ H the set of visible and hidden nodes in our model. We deﬁne the parameters in our model that we estimate to be θ associated with s [Equation]
We deﬁne a partial order on Ω inspired by a multi-layered perceptron studied in deep learning to be,
[Equation]
where the index for each layer is t ∈ T and T is the total number of layers in the network.
The input layer is deﬁned to be t = 0 and the output layer is deﬁned to be t
= T .
This partial order structure of Ω is illustrated in Figure 1 for the neural network architecture shown in Figure 2.
We focus on the conventional neural network which has the parameters of the bias and edge weights to represent the ﬁrst- and second-order interaction, respectively.
We rearrange Equation (1) to deﬁne the integrated input u ( t ) k of each node k for k = 1 , . .
, M ( t ) to be
[Equation]
where (cid:15) ( ω ; θ ) :
= log p ( ω ; θ ) + ψ ( θ )
= (cid:80) s ∈S
[ s (cid:22) ω ] θ ( s ) .
The value of the edge weight, w ( t ) ij is
[Equation]
The relationship between the classical neural network and our partially ordered structure is apparent by deﬁning the expectation of each node in X and Y to be,
[Equation]
In the following section we deﬁne the optimal values of θ and η , and show how to estimate them.
Optimization on Posets
We optimize the log-linear model by minimizing the KL divergence between the distribution of a given input and output pair ( x , y ) =
{ ( x i , y i ) } Ni =1 , and the joint distribution ˆ p ( X , Y , H , W ; θ ) .
The objective function is given as
[Equation]
see Appendix for the derivation.
In Equation 8, ˆ p is the estimated probability of the poset structure and can be determined from the data points using Equation (7).
We note that p ( x , y ) =
η ( x , y ) because there are no hidden elements in our input dataset.
Our optimization only requires the ﬁrst term because the second term does not depend on θ .
So our objective function can be written as a function of θ =
( θ ( s )) s ∈S and η = ( η ( s ))
s ∈S .
[Equation]
which is equivalent to Maximum Likelihood Estimation (MLE).
Our optimization is performed using the statistical EM-algorithm (which stands for expectation-maximization), which alternately iterates
M-step and the E-step.
The E-step minimizes the objective function with respect to the expectation parameter η ( s ) and the
M-step minimizes the objective function with respect to the natural parameter θ ( s ) ,
[Equation]
We now present the equations for the E-step and the M-step.
5.1 Expectation Step
The E-step estimates the expectation of each of the variables, given the parameters estimated at the previous step.
The probability estimated by the E-step is, ˆ p ( ω ; ˆ
[Equation]
) =
[Equation]
The expectation of the edge weights can be computed by ˆ η ( w ij )
=
ˆ p ( w ij
[Equation]
ˆ [Equation] ) = (1 / exp[ ψ (ˆ θ )]) exp[ (cid:15) ( w ij ; ˆ θ )] .
We now compute the expectation parameter ˆ η for the input nodes, hidden nodes and the output node
5.1.1 Updating Probabilities in the Input and Output Nodes.
The probabilities x k ∈ X and y k ∈ Y are required to be updated after each step using the logistic function (see Appendix for derivation).
[Equation]
5.1.2 Forward Propagation on a Poset
Forward propagation can be computed directly on the poset structure.
From Equation (1), we substitute the parameter values for the edge weights and the neuron to obtain the forward equation to compute the energy of the hidden nodes.
This is given by,
[Equation]
See Appendix for details.
Then the expectation of the hidden node can be computed as
[Equation]
The forward propagration is used to estimate the parameters in the model during the training phase.
It is an optimizer for Equation (10) because it updates η given the model parameters θ in the previous eprojection.
After the model has been trained, the forward propagation is also used to make predictions by propagating the values of the input through the network until it reaches the output layer.
5.2 Maximization Step
The M-step computes the MLE of the parameters given the expectation of the variables in the model.
For the second-order model, the M-step has closed form analytical solution.
The parameter θ for the edge weight can be updated with the following relationship,
[Equation]
Similarly, the parameters for the hidden nodes can be updated with the following relationship,
[Equation]
We repeat the E- and M-step until the model has converged.
A Example of Classical Neural Network Architecture
xxxh (1) h (1) h (2) h (2) yyyw (1) w (1) w (1) w (1) w (1) w (1) w (2) w (2) w (2) w (2) w (3) w (3) w (3) w (3) w (3) w (3)
[Figure]
[Figure]
[Figure]
B Derivations B.1 Optimization: Minimizing KL Divergence
[Equation]
B.2 Update for Input and Ouput Nodes
[Equation] [Equation]
Similarly for y k ,
[Equation] [Equation]
B.3 Forward Propagation
From Equation (1) [Equation] we substitute the equivalent values for our partial order formulation [Equation]
( t )
[Equation]
t )
[Equation]
n
[Equation]
[Equation]
(cid:88)
i
[Equation]
[Equation]
i
[Equation]
i
[Equation]
t
[Equation]
[Equation]
The equation can be vectorized for more efﬁcient implementation.
The vectorized form is given as,
[Equation]
where the boldfaced symbol denotes vectorization across the ﬁrst dimension of the edge weight index.
C Discussion on a Universal Approximator
We have drawn parallels between our approach proposed approach and the neural network.
We now provide the connection that our partial order structure is a universal approximator by using the
Kolmogorov-Arnold representation theorem which states as follows:
Theorem 1 (Kolmogorov–Arnold Representation Theorem (Braun and Griebel, 2009; Kolmogorov, 1957)) .
Any multivariate continuous function can be represented as a superposition of one–dimensional functions,
[Equation]
Braun (2009) showed that the Generalized Additive Model (GAM) is an approximation to the general form presented in Kolmogorov-Arnold representation theorem by replacing the range l ∈ { 1 , . .
. , 2 M ( t − 1) + 1 } with our partial order structure and the inner function g l,m by the identity if m = l and zero otherwise, yielding u ( t ) k
[Equation]
) , where f m ( · ) is a smooth monotonic function.
For our model this is the function for forward propagation in Equation (12) which is a smooth monotonic function with respect to the input u ( t − 1) m .
The model is able to approximate the input ˆ o to be | ˆ o − σ ( u ( t ) k ) | < δ , where δ is the error in the approximation.
By applying the forward propagation function through different layers, we can generalize the expression to be u
[Equation]
t )
[Equation]
( u ( τ ) m ) , where τ represents a layer in the neural network that is τ < t .
The function F ( t,τ ) m remains a smooth monotonic function because when we stack multiple smooth monotonic functions together, the overall transformation by the function is still smooth and monotonic.
This means for any giving input x we are always able to learn a representation for the given output y .
D Summary
We have presented a novel perspective on deep learning architectures using a partial order structure which can be naturally represented as a log-linear model studied in information geometry.
We ﬁrst have shown that a partial order structure can be used to represent deep architectures.
We then formulated our optimization is formulated by minimizing the KL-divergence between the set of inputs and our partial order structure.
We use the EM algorithm for optimization and it has a closed form analytical solution for both the E- and the M-step and does not require any gradients for optimization.
Our approach has clear advantages as it does not have the same drawbacks in the classical deep learning models such as vanishing gradients.