Interpreting Word Embeddings with Generalized Low Rank Models
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Glove and Skip-gram word embedding methods learn word vectors by decomposinga matrix of word co-occurrences into a product of low-rank matrices.
In thiswork, we propose an iterative algorithm for computing word embeddings basedon constructing word occurrence matrices via Generalized Low Rank Models.
Our algorithm generalizes both Skip-gram and GloVe as well as giving rise toother embedding methods based on the speciﬁed occurrence matrix distributionand the number of iterations in the iterative algorithm.
In particular, using a
Tweedie distribution with one-step in the iterative framework results in the GloVeembedding whereas a multinomial distribution with full-convergence mode resultsin the skip-gram model.
Experimental results demonstrate improved results of ourmodel over the GloVe method on a word analogy similarity task
Introduction
Word embeddings are low dimensional vector representations of words or phrases.
They are applied toword analogy tasks and used as feature vectors in numerous tasks within natural language processing,computational linguistics, and machine learning.
They are constructed by various methods whichrely on the distributional hypothesis popularized by Firth: “words are characterized by the companythey keep” [Firth, 1962].
Two seminal methodological approaches to ﬁnding word embeddings are
Skip-gram [Mikolov et al., 2013a] and GloVe [Pennington et al., 2014].
Both methods input a corpus
D , process it into a word co-occurence matrix Y , then output word vectors with some dimension d .
Skip-gram processes a corpus with w words and c contexts into a co-occurence matrix Y ∈ R w × c ,where y ij is the number of times word w i is used in context c j in the corpus.
Next, Skip-gram[Pennington et al., 2014, Section 3.1] estimates
[Equation]
then deﬁnes the word vectors to be the rows of ˆ U and the context vectors are the rows of ˆ V .
GloVe processes a corpus with w words into a co-occurence matrix Y ∈ R w × w where y ij is theharmonic sum of the number of tokens between words w i and w j for each co-occurrences.
That is,
[Equation]
where D ( p 1 ) is the p th1 word in the corpus and l c is the length of the context window .
Next, GloVeestimates
[Equation]
where uuu i is the i th row of the matrix U , vvv j is the j th row of the matrix V , ψ i and ω
j are bias terms,and y max is some prespeciﬁed cutoff.
GloVe then deﬁnes the estimated word vectors to be the rowsof 12 ˆ U + 12 ˆ V .
In both Skip-gram and GloVe, a matrix of co-occurences Y is introduced by processing the corpus,and an objective function is introduced to ﬁnd a low rank factorization related to the co-occurences
Y .
In this paper, we derive the corpus processing techniques and loss functions from principledmodel-based assumptions.
We introduce a model-based iterative algorithm, and show that (1) resultsfrom running the iterative algorithm until completion for a Multinomial model and (2) is a one stepof the iterative algorithm for a Tweedie model.
This algorithms allows us to introduce methodsintimately related to Skip-gram and GloVe and to introduce altogether new methods.
Related Work
In the introduction, we saw that Skip-gram and GloVe compute a co-occurence matrix Y whichresults from processing the corpus D and an objective function L to relate the matrix Y to a productof low rank matrices.
Many existing approaches for explaining word embedding methods do so byidentifying or deriving the co-occurence matrix Y or the objective function L .
In this section, wereview relevant work in this area, which helps frame our approach discussed in Section 4.
Much of the related work involves using the co-occurence matrix from Skip-gram, where the rowsrepresent words, the columns represent context, and the entry of the matrix represents the frequencyof the word-context pair: Deﬁne Y to be this matrix.
Early approaches to ﬁnding low-dimensional embeddings of words relied on the singular value decomposition [Landauer et al., 1998, Turney and Pantel, 2010].
These methods would truncate the singular value decomposition by zeroing out the small singular values.
Eckart and Young [1936] show that this is equivalent to using a loss function L which is invariant to orthogonal transformation.
For simplicity, these early approaches ﬁnd arg
[Equation]
where the Frobenius norm is the choice of loss function.
The co-occurence matrix and the loss function for Skip-gram can be read off from equation (1).
Cotterell et al.
ﬁnd a probabilistic interpretation of this loss function related to a Multinomialdistribution.
Mikolov et al.
[2013b] introduce Skip-gram with negative sampling (SGNS), a variant of Skip-gram.
If we view Skip-gram as maximizing the true positive rate of predicting a word will be in a givencontext, we can view SNGS as maximizing the true positive rate plus k times an approximation ofthe true negative rate.
When k = 0 , Skip-gram and SGNS coincide.
Levy and Goldberg [2014] use a heuristic argument to interpret SGNS as using a co-occurence matrix which is a shifted PMI.
Deﬁne the total number of times word w
i appears t i
= (cid:80) c
˜ j =
y i ˜ j , the total number of times context c
j appears t j = (cid:80) w ˜ i =1 y ˜ ij , and the total number of words t
= (cid:80) w,c ˜ i, ˜ j
=1 y ˜ i ˜
j .
The shifted PMI matrix has has entries log y ij t t i t j − log k .
However, they were unable to determine the loss function.
Later, Li et al.
[2015] and Landgraf and Bellay [2017] were able to explicitly identify both the loss function and the co-occurence matrix.
They ﬁnd a different co-occurence matrix than
Levy and Goldberg [2014], one that does not depend on k , while their loss function does depend on k .
Surprisingly, they argue that SGNS is using Y , the same matrix that Skip-gram uses.
The loss function is
[Equation]
Landgraf and Bellay [2017] explain that this loss function has a probabilistic interpretation, and theyuse that interpretation to recover the shifted PMI matrix as a plug in estimator within their model.
The approach in this paper will be to view the co-occurence matrix as having each entry be a randomvariable and connect the loss function to the likelihood of the random variable.
This approach wasshown to be effective by Cotterell et al.
[2017] and Landgraf and Bellay [2017].
Before this, we needto review some relevant modeling background.
Generalized Low Rank Models
Generalized linear models model a response vector y as estimated given an input matrix X and anassumed link-linear relationship with the mean of y given X .
A brief introduction to generalized linearmodels is given in the Appendix in Section 7.2.
We now consider two important generalizations ofgeneralized linear models: (1) extending the response to a matrix Y ∈ R n × m , and (2) an unsupervisedlow-rank relationship for the systematic component η .
Principal components analysis (PCA) Jolliffe [2011], is one well-known method for computing alow-rank approximation to a matrix Y which satisﬁes both (1) and (2).
In PCA, we write y ij ∼
Normal( uuu T
i vvv j , σ 2 ) where each y ij is independent, and uuu i ,
vvv j ∈ R d for some dimension d (cid:28) n, m .
The maximum likelihood estimator for uuu i is taken to be a low-dimensional embedding of the i th rowof the response Y .
A similar extension from linear models to generalized linear models allows usto generalize from PCA to generalized low rank models [Udell et al., 2014] allowing us to estimatemodel-based low-dimensional embeddings of non-normal data.
Deﬁnition 1 For some exponential dispersion family ED( µ, ϕ ) with mean parameter µ and dispersion parameter ϕ , the model for Y ∈ R n × m is a generalized low rank model with link function g when [Equation] ED
[Equation]
where uuu i ,vvv j ∈ R
d are the rows of matrices U ∈ R n × d and V ∈ R m × d , respectively, and ψψψ,ωωω ∈ R dare offset or bias terms.
The difference between the generalized low rank model, and the generalized linear model from
Deﬁnition (3) is in the systematic component (third line of Deﬁnition (4)).
In generalized lowrank models, the link-transformed mean is modeled as a low rank matrix which is the product oftwo matrices with small number of columns d .
In other words, generalized low rank models ﬁndlow-dimensional embeddings of the mean of the response on a scale determined by the link function.
In Section 4 we present an iterative algorithm inspired by IRLS which to estimate the maximumlikelihood estimator of a generalized low rank model.
In Section 5, we show that this method allowsus to compute word embeddings.
Methodology
In Section 7.2 of the Appendix following our introduction of generalized linear models, we introducedthe IRLS method for computing the MLE ˆ βββ , and in Section 3 we introduced generalized low rankmodels.
We now present Algorithm 1 for performing an algorithm analogous to IRLS on a cooccurrence matrix Y to produce embedding matrices ˆ U and ˆ V .
This algorithm gives us an effectiveprocedure for constructing word embeddings ˆ U and ˆ V from a corpus D .
4.1 Our Proposed Method
Our proposed method has three steps:
Choose a co-occurence matrix
Y ∈ R w × c to summarize the document.
(Note, in some casesthat c = w .)
Choose a plausible exponential dispersion family to model the entries of the co-occurencematrix3.
Run IRLS using the chosen exponential dispersion family to output word vectors
We will in practice always make a default choice of the link function that only depends on thedistribution.
This makes it so that there is no need to input the link function into the algorithm.
Data: Co-occurrence Matrix Y ∈ R w × c
Require: Distribution ED ( x ; µ, ϕ ) , number of steps T , Dimension d
Result: ˆ U ∈ R w × d , ˆ V ∈ R c × d , ˆ ψψψ ∈ R w , ˆ ωωω ∈ R c
Initialize µ (0) =
Y ;
Compute W (0) and Z (0) according to Equations (8) and (9); for t = 0 : T do
Evaluate the least squares problem arg
[Equation]
Update µ ( t +1) according to µ ij = uuu
Ti vvv j + ω i + ψ j ;
Update W ( t +1) according to Equation (8);
Update Z ( t +1) according to Equation (9); end return ˆ U ( T ) , ˆ V ( T ) , ˆ ψψψ ( T ) , ˆ ωωω ( T )
We regularize the word vectors by including the penaltyλ
[Equation]
(cid:0) (cid:107)
[Equation]
(cid:107) 2 F + (cid:107)
[Equation] (cid:107) [Equation] F (cid:1) (3) with λ = .
002 for two reasons.
One is to reduce some noise in the estimation of the word vectors.
Udell et al.
[2014, Lemma 7.3] show that penalizing by (3) is equivalent to penalizing by λ (cid:107) UV T (cid:107) ∗ ,the nuclear norm of UV T .
Since penalizing the nuclear norm UV T shrinks the dimension of theembedding and larger dimensional embeddings tend to be better [Melamud et al., 2016], we choose asmall tuning parameter to reduce noise while still preserving the dimension.
Another reason is to symmetrically distribute the singular values of ˆ U ˆ V T to both matrices ˆ U and ˆ V .
Write the singular value decomposition ˆ U ˆ V T = U Σ V T , for U and V orthogonal and Σ diagonal.
Mu et al.
[2018, Theorem 1] shows that using penalty (3) results in having ˆ U
= U Σ 1 / 2 Q andˆ V = V Σ 1 / 2 Q for some orthogonal matrix Q .
This is desirable since it was argued by Levy et al.[2015] that it has empirically been shown that a symmetric distribution of singular values worksoptimally on semantic tasks.
4.2 Example 1: GloVe
The ﬁrst step of our algorithm is to pick a co-occurence matrix Y ∈ R w × w that summarizes the corpus.
We will pick the one used by GloVe, so that y ij =
[Equation]
Now we must determine a plausible distribution for the co-occurence matrix that lies within theexponential dispersion family.
Recall that the Tweedie distribution has the property mentioned inequation (5) that it is a sum of Poisson many independent Gamma distributions.
An informal way to write this is that
Tweedie =
Poisson (cid:88) i =
Gamma i =
Poisson (cid:88)
[Equation]
where the Inverse gamma ( InvGamma ) distribution is deﬁned to be the reciprocal of the Gammadistribution.
The InvGamma distribution is supposed on the positive real line.
We argue that the Tweedie distribution is a reasonable distribution for the entries of the co-occurencematrix.
We do this by connecting the Poisson and Gamma distributions in displayed above toattributes of the corpus.
Intuitively it is reasonable that the number of times word w i and wordw j co-occur within the corpus can be modeled as having a Poisson distribution.
Another intuitivedistribution is that of an Inverse Gamma distribution for the number of tokens between word w iand word w j at some co-occurence for some choice of rate and scale parameters, although it is anapproximation as the number of tokens must be integer but the Inverse Gamma has support over thepositive reals.
Next, we ﬁnd the weight W and the pseudo-response Z that the Tweedie distribution provides.
This amounts to plugging in the cumulant generating function b from 7.1.
This gives
[Equation]
When the algorithm is intialized with ˆ µ (0)
=
Y , the pseudo response simpliﬁes to z ij = log y ij .
Further taking p = 1 . 25 , the weight simpliﬁes to y 3 / 4 ij .
In summary, we’ve shown that:
Result {+
+}
Inputting the harmonic co-occurence matrix used by GloVe, Tweedie distribution withpower p = 1.25, number of iterations T = 1 into our algorithm results in GloVe (without any weighttruncation.)
Given this connection, we can extend GloVe by either early stopping the algorithm some time afterthe ﬁrst iteration or running the algorithm until completion.
4.3 Example 2: SVD, Skip-gram, and More
We consider another co-occurence matrix to summarize the corpus.
In this case, we will pick theco-occurence matrix
Y ∈ R w × c such that Y ij is the number of times word w i appears in context c j .
4.3.1 The SVD
For step 2 of our method, one proposed distribution for the entries of Y is a Gaussian distribution.
For step 3, the IRLS algorithm will always converge in one iteration, so our method recovers themethod of computing a truncated SVD by Eckart and Young [1936].
4.3.2 Skip-gram
For step 2 of our method, another proposed distribution for the entries of Y is a multinomial distribution.
Speciﬁcally, we could propose that the the rows of Y corresponding to the i th word have the distribution
[Equation]
where πππ ∈ R c is vector of probabilities of appearing within each context and (cid:80) wi =1 y ij is the totalnumber of times word w i appears in the corpus.
This is the approach taken by Cotterell et al.
We suspect that some weight and pseudo-reponse with this distribution will yeild a one-step estimatorwhich is the estimator of Arora et al.
[2016], since they ﬁnd their estimator via the second order
Taylor expansion of the likelihood, completely analogous to Newton’s method, which is the basisfor IRLS.
One iteration of algorithm would result in an estimator which is equivalent to doing asecond-order Taylor expansion around the co-occurence matrix Y .
4.3.3 Poisson Estimator
In Sections 4.3.2 and 4.3.1, we note some model mis-speciﬁcation.
The co-occurence matrix Yhad entries appearing on both the left-hand and right-hand side, since it is unknown what the totalnumber of times word w i will appear in the corpus.
This means that the Poisson distribution is amore appropriate model for the co-occurence matrix Agresti [2014, Section 7.2.1].
For step 2, wecould instead propose a Poisson distribution to model the entries of the co-occurence matrix, whichshould yield improved results over the multinomial model.
Experiments
In Section 4 we introduced an IRLS algorithm to compute word embeddings such as those producedby GloVe.
We now conduct quantitative evaluation experiments on an English word analogy task,a variety of word similarity tasks [Mikolov et al., 2013a] to demonstrate the performance of thealgorithm.
First, in Section 5.1 we introduce the analogy similarity task for evaluating word vectors.
In Section 5.2 our parameter conﬁgurations and training procedure.
In Sections 5.3-5.6 we presentresults of our algorithm in numerous scenarios showcasing improvement through multiple steps androbustness to other model parameters.
5.1 Word Analogies
We introduce the word analogy task following the presentation of [Pennington et al., 2014].
Theword analogy task is a dataset of 19 , 544 statements of the basic form “a is to b as c is to __”, whichare divided into a semantic and syntactic subsets.
The semantic statements are typically analogiesrelating to people, places, or nouns such as “Athens is to Greece as Berlin is to __”, while the syntacticquestions relate to verb or adjective forms such as “dance is to dancing as ﬂy is to __”.
The basicanalogy statement is answered by ﬁnding the closest vector uuu d to uuu b − uuu a + uuu c 1 in the embeddingspace via cosine similarity 2 .
The task has been shown under speciﬁc assumptions to be provablysolvable by methods such as GloVe and Skip-gram [Ethayarajh et al., 2018, Gittens et al., 2017] andas such is closely related to solving the objectives introduced in Sections 1 and 4.
5.2 Training Details
We train our models on the English text8 corpus 3 with approximately 17 million tokens.
We ﬁlter outword types that occur fewer than 50 times to obtain a vocabulary size of approximately 11 , 000 ; aratio consistent with other embedding literature experiments 4 .
The adjustable model conﬁgurations in our algorithm are the choice of power parameter p , penaltytuning parameter λ , and co-occurrence processing step.
We experiment with different choices ofp ∈ { 1 . 1 , 1 . 25 , 1 . 5 , 1 . 75 , 1 . 9 } , different choices of processing including no processing, clampingthe weights (as in GloVe) and truncating the outliers in the co-occurrence matrix (elaborated on in
Section 5.5, and set the penalty tuning parameter λ = 0 . 002 .
The estimated word vectors are therows of 12 ˆ U + 12 ˆ V .
For all of our experiments, we set the dimension of the word vectors to d = 150 , and the objectivefunction at each iteration is optimized using Adagrad [Duchi et al., 2011] with a ﬁxed learning rate
[Table]
of 0 .
Models are trained for up to 50 epochs ( 50 passes through the co-occurrence matrix) withbatches of size 512 .
5.3 Experiment 1: Multiple Iteration Effects
We present results of multiple iterations of our IRLS algorithm compared to the GloVe method.
Inparticular, to match the GloVe objective function and processing, we set the weight function in ourmodel from f ( y ) =
y 2 − p to f ( y ) =
(cid:16) y y max (cid:17) 2 − p with y max = 10 and p = 1 . 25 , and note wheninitializing µ = y , the solution to 2 is exactly the solution of the IRLS algorithm at the ﬁrst step.
Results of the IRLS algorithm with these conﬁgurations is shown in Table 1.
We remark on a few observations based on the results in Table 1.
First, as the number of stepsincreases, the accuracy on the analogy task increases as well with the models at steps 3 - 5 doingrelatively similarly and over 1% higher than the GloVe model.
Second, relatively few steps are neededwith the accuracy being relatively similar for steps 3 - 5 in comparison to steps 1 and 2 .
Finally, wenote that in this case running the algorithm for only 2 iteration is sufﬁcient to see an improvementover the one-step estimates.
5.4 Experiment 2: Effects of Varying p
Here we examine the effect of the choice of the power p in the tuning parameter when you run a
Tweedie generalized low rank model.p
Iterations [Table]
The Results in Table 2 show that values of p which are high perform poorly, while values of p around1 . 25 and 1 . 5 perform similarly.
We ﬁnd that p = 1 . 5 performs the best, and view this value of p as agood choice as it accounts for zero-inﬂation present in Y .
5.5 Experiment 3: Effects of Co-Occurrence Matrix Transformations
With power p = 1 . 25 , we explore the effect of different strategies to handle large entries in theco-occurrence matrix Y .
One strategy is to simply input Y into step 3 of our algorithm.
A secondstrategy is to truncate the weight W that results from step 3 of our algorithm, in a strategy similar to GloVe.
A third strategy is to input min { Y, Y max } into step 3 of our algorithm, where for us
Y max = 10010 = 10 , where 100 is the threshold chosen by GloVe Pennington et al.
[2014] and ourcorpus is 110th of the size of their corpus.
[Table]
We see that the clear best strategy is to truncate the weights at each step in the iterative algorithm, ina manner similar to GloVe.
5.6 Experiment 4: Regularization Effects
[Table]
Finally, we experiment with whether the penalty introduced in Equation (3) improves results andaccurately reduces noise in the estimate.
We also consider not including the diagonal elements of
Y as a form of regularization and experiment here as well, as these terms are often large (can beconsidered as outliers) and do not contain a great deal of information.
Table 4 demonstrates theincluded regularization improves results.
Conclusion
We presented a general methodology for computing word vectors from a corpus, and demonstrate thatmany commonly used methods are special cases this methodology.
Our methodology allows us toanalyze the distributional properties of the co-occurrence matrices and yields better objective functionsfor estimating word functions more particular to the characteristics of the corpora.
Experimentalresults on a small corpus demonstrates our method improves results on a word analogy similaritytask.
It is our hope that this methodology can lead to the development of better, statistically soundword embeddings, and improve results on many other downstream tasks.
Acknowledgements
We thank YA, BA, DM, and WZ for their helpful contributions.
The work in this paper is supportedby an AWS grant.
Appendix
In this section, we develop some background in exponential dispersion families [Jorgensen, 1997],generalized linear models [Agresti, 2014], and generalized low rank models [Udell et al., 2014].
7.1 Exponential dispersion families and the Tweedie distribution
We begin by discussing exponential dispersion families, the distribution of the response in generalizedlinear models.
Deﬁnition 2 Let y ∈ R be a random variable.
If the density function f
( y ; θ, φ ) of y satisﬁes
[Equation]
over its support, then the distribution of y is in the exponential dispersion family .
The parameter θ isthe natural parameter , ϕ is the dispersion parameter , and the function b is the cumulant generatingfunction .
In many cases, the function a ( ϕ ) is very simple, meaning that, for instance, a ( ϕ ) = 1 or a ( ϕ ) = ϕ .
The function c ( y ; ϕ ) can be viewed as the normalizing constant ensuring that the density integrates toone.
When y follows a distribution in the exponential dispersion family with natural parameter θ , itsmean µ = b (cid:48) ( θ ) , so we can equivalently specify the mean µ or the natural parameter θ .
Many classical distributions such as the Poisson, Normal, Binomial, and Gamma distribution areexponential dispersion families.
For example, when y ∼ Normal( µ, σ 2 ) is a normal distribution withmean µ and variance σ 2 , its log density satisﬁes
[Equation]
showing that here the natural parameter θ = µ , the dispersion parameter ϕ
= σ 2 , the functionsb ( θ ) = 12 θ 2 , a
The Tweedie distribution [Jorgensen, 1997], of particular importantance to us, also lies within theexponential dispersion family.
Instead of deﬁning the Tweedie distribution through the form of itsdensity, we will deﬁne it through the relationship between its mean and variance.
This relies on aresult from [Jorgensen, 1987, Theorem 1] that distributions within the exponential dispersion familyare deﬁned by the relationship between their mean and variance.
Deﬁnition
A random variable y has a Tweedie distribution with power parameter p ∈ { 0 }∪ [1 , ∞ ) when var
[Equation]
and the distribution of y is an exponential dispersion family.
In this case, we write y ∼
Tweedie p ( µ, ϕ ) , where µ = E ( y ) is the mean.
The Normal distribution discussed above has a variance that does not depend on the mean.
In ournew notation, this means that the Normal distribution is a Tweedie distribution with power parameterp = 0 .
The Poisson distribution has variance equal to the mean and is in the exponential dispersionfamily, so is a Tweedie distribution with power parameter p
= 1 and dispersion parameter ϕ
= 1 .
A Gamma distribution with shape parameter α and rate parameter β is a Tweedie distribution withpower p = 2 , mean µ = αβ , and dispersion parameter ϕ
= α − 1 .
We will only consider Tweedie distributions with power parameter p ∈ (1 , 2) .
These distributions arealso known as compound Poisson-Gamma distributions due to the representation
[Equation]
(5) where n ∼ Poisson( λ ) and g i i .
i .
d . ∼ Gamma( α, β ) , and λ = µ 2 − p (2 − p ) ϕ , α = 2 − p p − 1 , and
β = µ 1 − p ( p − 1) ϕ[Smyth, 1996].
It is important to note that the Tweedie distribution has positive mass at zero, an important characteristic for capturing the zero-inﬂation prominent in some co-occurence matrices.
Speciﬁcally,
[Equation]
Using other arguments related to representations of the mean and variance in terms of the cumulant generating function b , Jorgensen [1997] show that the Tweedie distribution has b ( θ ) =
12 − p ((1 − p ) θ ) 2 − p 1 − p .
7.2 Generalized linear models, the MLE, and iteratively re-weighted least squares
We start by introducing the linear model.
Given a response yyy ∈ R n comprising n observations,the model for yyy is a linear model with covariates xxx i ∈
R p when y i ind .
∼ Normal( xxx Ti βββ, σ 2 ) for alli ∈ { 1 , . . . , n } .
In vector notation, this reads that yyy ∼ Normal
( Xβββ, σ 2 I ) , where X ∈ R n × p is amatrix with i th row
xxx i .
Generalized linear models remove the the assumptions of normality and that the mean is a linearfunction of the coefﬁcients βββ .
Deﬁnition
For some exponential dispersion family ED( µ, ϕ ) with mean parameter µ and dispersion parameter ϕ , the model for yyy ∈ R n is a generalized linear model with link function g when
[Equation]
In the ﬁrst line of the displayed relationships, the distribution of the response yyy is described.
In thethird line, the systematic component η i expresses the effect of the covariates xxx i .
The second lineconnects the distribution to the covariates through the link function.
That is, the covariates effect alink-transformed mean.
Generalized linear models are used as the default modeling framework in many ﬁelds of appliedscience for non-normal distributions [McCullagh and Nelder, 1989].
When g ( µ ) = µ is the identitymap and ED is the Normal distribution, the generalized linear model is simply the linear model.
When g ( µ ) = logit( µ ) = log 1 − µµ and ED is the Binomial distribution, the generalized linear modelis logistic regression.
Further, a generalized linear model can be viewed as a no-hidden-layer neuralnetwork with activation function g .
The coefﬁcient βββ is unknown and a target of estimation in the generalized linear model.
The standardapproach to compute βββ in a generalized linear model is maximum likelihood estimation [Fisher,1922, Section 7] to produce the maximum likelihood estimator , or MLE ˆ βββ , with ˆ βββ deﬁned to bethe maximizer of the likelihood of yyy given X .
Deﬁne (cid:96) ( βββ ) to be the negative log likelihood.
Anatural way to estimate the MLE is through Fisher scoring, a version of Newton’s method on thelog likelihood which uses the expectation of the Hessian in place of the Hessian [Agresti, 2014,
Section 4.5].
In notation, Fisher scoring produces a sequence of estimates { ˆ βββ ( t ) } ∞ t =1 starting withsome initialization ˆ βββ (0) so that
[Equation]
where ∇ (cid:96) is the gradient and D 2 (cid:96) is the Hessian of the likelihood (cid:96) .
Upon plugging in the gradientand expected hessian of (cid:96) for an exponential family, a surprising identity emerges: each iteration of
Fisher scoring is equivalent to minimizing a weighted least squares objective:
[Equation]
where the weight W ( t ) is diagonal with
[Equation]
and the pseudo-response for observation i is
[Equation]
where ηηη ( t ) =
X ˆ βββ ( t ) and µ ( t )
i = g − 1 ( η ( t ) i ) .