G AUSSIAN
C ONDITIONAL
R ANDOM
F IELDS
FOR
C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
In this paper, a Gaussian conditional random ﬁeld model for structured binary classiﬁcation (GCRFBC) is proposed.
The model is applicable to classiﬁcation problems with undirected graphs, intractable for standard classiﬁcation CRFs.
The model representation of GCRFBC is extended by latent variables which yield some appealing properties.
Thanks to the GCRF latent structure, the model becomes tractable, efﬁcient, and open to improvements previously applied to
GCRF regression.
Two different forms of the algorithm are presented: GCRFBCb (GCRGBC - Bayesian) and GCRFBCnb (GCRFBC - non-Bayesian).
The extended method of local variational approximation of sigmoid function is used for solving empirical Bayes in GCRFBCb variant, whereas MAP value of latent variables is the basis for learning and inference in the GCRFBCnb variant.
The inference in GCRFBCb is solved by Newton-Cotes formulas for one-dimensional integration.
Both models are evaluated on synthetic data and real-world data.
It was shown that both models achieve better prediction performance than relevant baselines.
Advantages and disadvantages of the proposed models are discussed.
I NTRODUCTION
Increased quantity and variety of sources of data with correlated outputs, so called structured data, created an opportunity for exploiting additional information between dependent outputs to achieve better prediction performance.
One of the most successful probabilistic models for structured output classiﬁcation problems are conditional random ﬁelds (CRF) (Sutton & McCallum, 2006).
The main advantages of CRFs lie in their discriminatory nature, resulting in the relaxation of independence assumptions and the label bias problem that are present in many graphical models.
Aside of many advantages, CRFs also have many drawbacks mostly resulting in high computational cost or intractability of inference and learning.
A wide range of different approaches of tackling these problems has been proposed, and they motivate our work, too.
One of the popular methods for structured regression based on CRFs – Gausian conditional random ﬁelds (GCRF) – has the form of multivariate Gaussian distribution (Radosavljevic et al., 2010).
The main assumption of the model is that the relations between outputs are presented in quadratic form.
It has convex loss function and, consequently, efﬁcient inference and learning, and expensive sampling methods are not used.
In this paper, a new model of Gaussian conditional random ﬁelds for binary classiﬁcation is proposed (GCRFBC).
GCRFBC builds upon regression GCRF model which is used to deﬁne latent variables over which output dependencies are deﬁned.
The model assumes that discrete outputs y i are conditionally independent conditioned on continuous latent variables
z i which follow a distribution modeled by a GCRF.
That way, relations between discrete outputs are not expressed directly.
Two different inference and learning approaches are proposed in this paper.
The ﬁrst one is based on evaluating empirical Bayes by marginalizing latent variables (GCRFBCb), whereas MAP value of latent variables is the basis for learning and inference in the second model (GCRFBCnb).
In order to derive GCRFBCb model and its learning procedure the variational approximation of Bayesian logistic regression (Jaakkola & Jordan, 2000) is generalized.
Compared to CRFs and structured SVM classiﬁers, the GCRFBC models have some appealing properties: • The model is applicable to classiﬁcation problems with undirected graphs, intractable for standard classiﬁcation CRFs.
Thanks to the GCRF latent structure, the model becomes tractable, efﬁcient and open to improvements previously applied to GCRF regression models.
• Deﬁning correlations directly between discrete outputs may introduce unnecessary noise to the model (Tan et al., 2010). This problem can be solved by deﬁning structured relations on a latent continuous variable space.
• In case that unstructured predictors are unreliable, which is signaled by their large variance (diagonal elements in the covariance matrix), it is simple to marginalize over latent variable space and obtain better results.
GCRFBC model is relying on the assumption that the underlying distribution of latent variables is multivariate normal distribution, due to that in the case when this distribution cannot be ﬁtted well to the data (e.g. when the distribution of latent variables is multimodal) the model will not perform as well as it is expected.
The proposed models are experimentally tested on both synthetic and real-world datasets in terms of predictive performance and computation time.
In experiments with synthetic datasets, the results clearly indicate that the the empirical Bayes approach (GCRFBCb) better exploits output dependence structure, more so as the variance of the latent variables increases.
We also tested both approaches on real-world datasets of predicting ski lift congestion, gene function classiﬁcation, classiﬁcation of music according to emotion and highway congestion.
Both GCRFBC models outperformed ridge logistic regression, lasso logistic regression, neural network, random forest, and structured SVM classiﬁers, demonstrating that the proposed models can exploit output dependencies in a real-world setting.
R ELATED W ORK
An extensive review of binary and multi-label classiﬁcation with structured output is provided in Su (2015).
A number of different studies related to graph based methods for regression can be found in the literature (Fox, 2015).
CRFs were successfully applied on a variety of different structured tasks (Cotterell & Duh, 2017; Zhang et al., 2015; Masada & Bunescu, 2017; Zia et al., 2018) and different model adaptations can be found in literature Kim (2017); Maaten et al.
Recently, successful uniﬁcations of deep learning and CRFs have been proposed Chen et al.
Kosov et al.
Moreover, implementation of deep neural networks as potential functions is presented in form of structure prediction energy networks (SPEN)
Belanger & McCallum (2016); Belanger et al.
Adaptation of normalazing ﬂows in SPEN structure is presented in Lu & Huang (2019).
An extensive review on topic of binary and multi-label classiﬁcation with structured output is provided in Su (2015).
Large number of different studies related to graph based methods for regression can be found in the literature (Fox, 2015). CRFs were successfully applied on a variety of different structured tasks, such as: low-resource named entity recognition (Cotterell & Duh, 2017), image segmentation (Zhang et al., 2015), chord recognition (Masada & Bunescu, 2017) and word segmentation (Zia et al., 2018). The mixture of CRFs capable to model data that come from multiple different sources or domains is presented in Kim (2017). The method is related to the well known hidden-unit CRF (HUCRF) (Maaten et al., 2011).
The conditional likelihood and expectation minimization (EM) procedure for learning have been derived there.
The mixtures of CRF models were implemented on several real-world applications resulting in prediction improvement.
Recently, a model based on uniﬁcation of deep learning and CRF was developed by Chen et al.
The deep CRF model showed better performance compared to either shallow CRFs or deep learning methods on their own.
Similarly, the combination of CRFs and deep convolutional neural networks was evaluated on an example of environmental microorganisms labeling (Kosov et al., 2018).
The spatial relations among outputs were taken in consideration and experimental results have shown satisfactory results.
The GCRF model was ﬁrst implemented for the task of low-level computer vision (Tappen et al., 2007).
Since then, various different adaptations and approximations of GCRF were proposed (Radosavljevic et al., 2014).
The parameter space for the GCRF model is extended to facilitate joint modelling of positive and negative inﬂuences (Glass et al., 2016).
In addition, the model is extended by bias term into link weight and solved as a part of convex optimization.
Semi-supervised marginalized Gaussian conditional random ﬁelds (MGCRF) model for dealing with missing variables was proposed by Stojanovic et al.
The beneﬁts of the model were proved on partially observed data and showed better prediction performance than alternative semi-supervised structured models.
A comprehensive review of continuous conditional random ﬁelds (CCRF) was provided in
Radosavljevic et al.
The sparse conditional random ﬁelds obtained by l 1 regularization are ﬁrst proposed and evaluated by Wytock & Kolter (2013).
Additionaly, Frot et al.
(2018) presented
GCRF with the latent variable decomposition and derived convergence bounds for the estimator that is well behaved in high dimensional regime.
An adaptation of GCRF on discrete output was brieﬂy discussed in Radosavljevic (2011), as a part of future work.
This discussion motivates our work, but our approach is different in technical aspects.
M ETHODOLOGY
In this section we ﬁrst present already known GCRF model for regression and then we propose
GCRFBC model for binary classiﬁcation and two approaches to inference and learning.
3. B ACKGROUND M ATERIAL
GCRF is a discriminative graph-based regression model (Radosavljevic et al., 2010).
Nodes of the graph are variables y
=
( y 1 , y 2 , . . .
, y N ) , which need to be predicted given a set of features x .
The attributes x = ( x 1 , x 2 , . . .
, x N ) interact with each node
y i independently of one another, while the relations between outputs are expressed by pairwise interaction function.
In order to learn parameters of the model, a training set of vectors of attributes x and real-valued response variables y are provided.
The generalized form of the conditional distribution P (cid:0) y | x , α , β (cid:1) is:
P
[Equation]
y | x ,
[Equation]
, β (cid:1) =
[Equation]
First sum models relations between outputs y i and corresponding input vector x
i and the second one models pairwise relations between nodes.
R k ( x i ) represents an unstructured predictor of y i for each node in the graph and S lij is value that expresses similarity between nodes i and j in graph l .
Unstructured predictor can be any regression model that gives prediction of output y i for given attributes x i .
K is the total number of unstructured predictors.
L is the total number of graphs (similarity functions).
Graphs can express any kind of binary relations between nodes e.g., spatial and temporal correlations between outputs.
Z is a partition function and vectors α and β are learnable parameters.
One of the main advantages of GCRF is the ability to express different relations between outputs by variety of graphs and ability to learn which graphs are signiﬁcant for prediction.
The quadratic form of interaction and association potential enables conditional distribution
P ( y | x , α , β ) to be expressed as multivariate Gaussian distribution (Radosavljevic et al., 2010):
[Equation]
Precision matrix Σ − 1 = 2 Q and distribution mean µ = Σ b are deﬁned as, respectively:
[Equation]
Due to concavity of multivariate Gaussian distribution, the inference task argmax y
P ( y | x , α , β ) is straightforward.
The maximum posterior estimate of y is the distribution expectation µ .
The objective of the learning task is to optimize parameters α and β by maximizing conditional log likelihood argmax α , β (cid:80)
y log P ( y | x , α , β ) .
One way to ensure positive deﬁniteness of the covariance matrix of GCRF is to require diagonal dominance (Strang et al., 1993).
This can be ensured by imposing constraints that all elements of α and β be greater than 0 (Radosavljevic et al., 2010).
3. GCRFBC MODEL REPRESENTATION
One way of adapting GCRF to classiﬁcation problem is by approximating discrete outputs by suitably deﬁning continuous outputs.
Namely, GCRF can provide dependence structure over continuous variables which can be passed through sigmoid function.
That way the relationship between regression GCRF and classiﬁcation GCRF is similar to the relationship between linear and logistic regression, but with dependent variables.
Aside from allowing us to deﬁne a classiﬁcation variant of GCRF, this may result in additional appealing properties: (i)
The model is applicable to classiﬁcation problems with undirected graphs, intractable for standard classiﬁcation CRFs.
Thanks to the
GCRF latent structure, the model becomes tractable, efﬁcient and open to improvements previously applied to GCRF regression models.
(ii) Deﬁning correlations directly between discrete outputs may introduce unnecessary noise to the model (Tan et al., 2010).
We avoid this problem by deﬁning structured relations on a latent continuous variable space.
(iii)
In case that unstructured predictors are unreliable, which is signaled by their large variance (diagonal elements in the covariance matrix), it is simple to marginalize over latent variable space and obtain better results.
It is assumed that y i are discrete binary outputs and z i are continuous latent variables assigned to each y i .
Each output y i is conditionally independent of the others, given z i .
The conditional probability distribution P ( y i |
z i ) is deﬁned as Bernoulli distribution:
[Equation]
[Equation] where σ ( · ) is sigmoid function.
Due to conditional independence assumption, the joint distribution of outputs y i can be expressed as:
[Equation]
Furthermore, the conditional distribution P ( z | x ) is the same as in the classical GCRF model and has canonical form deﬁned by multivariate Gaussian distribution.
Hence, joint distribution of continuous latent variables z and outputs y given x and θ = ( α 1 , . .
. , α K , β 1 , . . .
, β L ) is is the general form of the GCRFBC model deﬁned as:
[Equation]
We consider two ways of inference and learning in GCRFBC model: (i) GCRFBCb - with conditional probability distribution P ( y | x , θ ) , in which variables z are marginalized over, and (ii) GCRFBCnb - with conditional probability distribution P (cid:0)
y | x , θ , µ z (cid:1) , in which variables z are substituted by their expectations.
3. I NFERENCE IN GCRFBC B M ODEL
Prediction of discrete outputs y for given features x and parameters θ is analytically intractable due to integration of the joint distribution P ( y, z | x , θ ) with respect to latent variables.
However, due to conditional independence between nodes, it is possible to obtain P ( y i =
| x , θ ) .
[Equation]
where σ ( z i ) models P ( y i | z ) .
As a result of independence properties of the distribution, it holds
P ( y i = 1 | z ) =
P ( y i =
| z i ) , and it is possible to marginalize P ( z | x , θ ) with respect to latent variables z (cid:48)
[Equation]
where (cid:82) z (cid:48)
P ( z (cid:48) , z i | x , θ ) d z (cid:48) is normal distribution with mean µ = µ i and variance σ 2 i = Σ ii .
Therefore, it holds:
[Equation]
The evaluation of P ( y i =
| x , θ ) is straightforward: P ( y i =
| x , θ ) =
1 − P ( y i =
| x , θ ) .
The one-dimensional integral is still analytically intractable, but can be effectively evaluated by onedimensional numerical integration.
The proposed inference approach can be effectively used in case of huge number of nodes, due to low computational cost of one-dimensional numerical integration.
3. I NFERENCE IN GCRFBC NB M ODEL
The inference procedure in GCRFBCnb is much simpler, because marginalization with respect to latent variables is not performed.
To predict y , it is necessary to evaluate posterior maximum of latent variable z max = argmax z
P ( z | x , θ ) , which is straightforward due to normal form of GCRF.
Therefore, it holds z max = µ z,i .
The conditional distribution P ( y i = 1 | x , µ z,i , θ ) , where µ z,i is expectation of latent variable z i , can be expressed as:
[Equation]
3. L EARNING IN GCRFBC B M ODEL
In comparison with inference, learning procedure is more complicated.
Evaluation of the conditional log likelihood is intractable, since latent variables cannot be analytically marginalized.
The conditional log likelihood is expressed as:
[Equation]
where Y ∈ R M × N is complete dataset of outputs, X ∈ R M × N × A is complete dataset of features,
M is the total number of instances and A is the total number of features.
Please note that each instance is structured, so while different instances are independent of each other, variables within one instance are dependent.
One way to approximate integral in conditional log likelihood is by local variational approximation.
Jaakkola & Jordan (2000) derived lower bound for sigmoid function, which can be expressed as:
[Equation]
(14) where λ ( ξ ) = − 12 ξ · (cid:2) σ ( ξ ) − 12 (cid:3) and ξ is a variational parameter.
The Eq.
14 is called ξ transformation of sigmoid function and it yields maximum value when ξ = x .
This approximation can be applied to the model deﬁned by Eq. 13, but the variational approximation has to be further extended because of the product of sigmoid functions, such that:
[Equation]
The Eq.
16 can be arranged in the form suitable for integration.
Detailed derivation of lower bound of conditional log likelihood is presented in Appendix A.
The lower bound of conditional log likelihood
L ( y j | x j , θ , ξ j ) is deﬁned as:
[Equation]
where:
[Equation]
GCRFBCb uses the derivative of conditional log likelihood in order to ﬁnd the optimal values for parameters α , β and matrix of variational parameters ξ ∈ R M × N .
In order to ensure positive deﬁniteness of normal distribution involved, it is sufﬁcient to constrain parameteres α > 0 and β > 0 .
The partial derivatives of lower bound of conditional log likelihood are presented in Appendix B.
For constrained optimization, the truncated Newton algorithm was used Nocedal & Wright (2006);
Facchinei et al.
The target function is not convex, so ﬁnding a global optimum cannot be guaranteed.
3. L EARNING IN GCRFBC NB M ODEL
In GCRFBCnb the mode of posterior distribution of continuous latent variable z is evaluated directly, so there is no need for approximation.
The conditional log likelihood can be expressed as:
[Equation]
| X , θ , µ ) =
M (cid:88) j
=
N (cid:88) i
= log P ( y ji | [Equation] j , θ , µ ji ) =
M (cid:88) j
=
N (cid:88) i =
L ji ( y ji | x j , θ , µ ji ) (20)
[Equation]
The partial derivatives of conditional log likelihood are presented in Appendix C.
E XPERIMENTAL E VALUATION
Both proposed models were tested and compared on synthetic data and real-world tasks.
1 All compared classiﬁers were compared in terms of the area under ROC curve (AUC) and accuracy 2 (ACC).
Moreover, the lower bound (in case of GCRFBCb) of conditional log likelihood L (cid:0) Y | X , θ , µ (cid:1) and actual value (in case of GCRFBCnb) of conditional log likelihood L (cid:0) Y | X , θ (cid:1) of obtained values on synthetic test dataset were also reported.
4. S YNTHETIC D ATASET
The main goal of experiments on synthetic datasets was to examine models under various controlled conditions, and show advantages and disadvantages of each. In all experiments on synthetic datasets two different graphs were used (hence β ∈ R 2 ) and two unstructured predictors (hence α ∈ R 2 ).
The results of experiments on synthetic datasets are presented in Appendix D.
It can be noticed, that in cases where norm of the variances of latent variables is small, both models have equal performance considering AUC and conditional log likelihood L (cid:0) Y | X , θ (cid:1) .
This is the case when values of parameters α used in data generating process are greater or equal to the values of parameters β .
This means that the information provided by unstructured predictors is more important for classiﬁcations task than the information provided by output structure.
Therefore, conditional distribution P ( y, z | x , θ ) is concentrated around mean value and MAP estimate is a satisfactory approximation.
However, when data is generated from distribution with signiﬁcantly higher values of β than α , the GCRFBCb performs signiﬁcantly better than GCRFBCnb.
For the larger values of variance norm, this difference is also large.
This means that the structure between outputs has signiﬁcant contribution to solving the classiﬁcation task.
It can be concluded that GCRFBCb has at least equal prediction performance as GCRFBCnb.
Also, it can be argued that the models were generally able to utilize most of the information (from both features and the structure between outputs), which can be seen through AUC values. In addition, distribution of local variational parameters were analyzed during learning.
It is noticed that in each epoch, the variance of this distribution is small and that the parameters can be clustered and their number signiﬁcantly reduced. Therefore, it is possible to signiﬁcantly lower down computational and memory costs of GCRFBCb learning procedure, but that’s out of the scope of this paper. 4. P ERFORMANCE ON R EAL -W ORLD D ATASETS 4.2. S KI LIFTS CONGESTION
Data used in this research includes information on ski lift gate entrances in Kopaonik ski resorts, for the period March 15 to March 30 for the seasons from 2006 to 2011.
The goal is to predict occurrence of crowding on ski lifts for 40 minutes in advance.
Total number of instances in dataset was 4,850 for each ski lift, which is 33,950 in total.
Relatively simple method for crowding detection was devised for labelling data.
We assume that, if the crowding at some gate occurs, distributions of skiing times from other gates to that gate within some time window get shifted towards larger values.
We model probability distribution of skiing time between two gates by the well-known parametric method of kernel density estimation (KDE) (Silverman, 2018).
The distribution shift is measured with respect to the mode of the distribution.
The dataset is generated by observing shifts in time windows of 5 minutes.
When the mode of the distribution of skiing times within that window is greater than the mode for the whole time-span, the instance is labeled by 1 (crowding) and otherwise, it is labeled by 0 (no crowding).
In order to obtain more information from the data distribution, additional 18 features were extracted.
Four different unstructured predictors that were trained on each class separately were used: ridge logistic regression, LASSO logistic regression, neural network and random forest, whereas additional two unstructured predictors: decision tree and neural network were trained on all nodes together.
Additionally, three structural support vector machine and two CRFs classiﬁers were used (M ¨ uller & Behnke, 2014).
Fully connected graph of SSVM and CRF models are deﬁned as SSVM-full and
CRF-full, whereas Chow-Liu tree method for specifying edge connections are deﬁned as SSVMtree and CRF-tree, respectively.
In the SSVM-independent model the nodes of the graph are not connected.
Six different weighted graphs were used to capture dependence structure between ski lifts (nodes): χstatistics on labels of training set, mutual information between labels, correlation matrix between outputs of over-ﬁtted neural networks, norm of difference between vectors of labels and two graphs were deﬁned based on difference of vectors of historical labels and on differences of historical averages of skier times.
The AUC score and ACC of structured and unstructured predictors, along with the total computational time are shown in Table 1.
It can be observed that GCRFBCb and GCRBCnb outperformed unstructured and other structured predictors in all cases.
Based on evaluated parameters it could be concluded that dependence structure has signiﬁcant impact on overall prediction performance, even though, due to low values of norm of variance, GCRFBCb and GCRFBCnb have equal AUC scores. It can be summarized that advantages of structured models compared to unstructured are obvious, but in this particular task due to equal prediction performance and its lower computational and memory complexity, GCRFBCnb is the best choice for this speciﬁc application.
[Table]
[Table]
4.2. M ULTI - LABEL CLASSIFICATION OF MUSIC ACCORDING TO EMOTION
The dataset used for this work consists of 100 songs from 7 different genres.
The collection was created from 233 musical albums choosing three songs from each album.
8 rhythmic and 64 timbre features are extracted.
The music is labeled in 6 categories of emotions: amazed-surprised, happypleased, relaxing-calm, quiet-still, sad-lonely and angry-fearful (Trohidis et al., 2008).
Total number of instances in dataset was 593.
Four different weighted graphs were used: statistics on labels of training set, mutual information between labels, correlation matrix between outputs of over-ﬁtted neural networks and norm of difference between vectors of labels.
Same unstructured predictors as in ski lift congestion problem were used, along with three structural support vector machine classiﬁers.
The performances of models are evaluated by 10 fold cross validation.
The AUC score and ACC of structured and unstructured predictors, along with the total computational time are shown in
Table 2.
It can be seen that GCRFBCb has achieved the best prediction performances.
The ACC of GCRFBC models are signiﬁcantly better than the SSVM performances.
The AUC score and
ACC of GCRGBCb are higher than the best result (AUC = 0 . 8237 ) presented in original paper (Trohidis et al., 2008).
As in previous cases, computational time of GCRFBCb is signiﬁcantly longer compared to GCRFBCnb and SSVM models.
4.2. G ENE FUNCTION CLASSIFICATION
This dataset is formed by micro-array expression data and phylogenetic proﬁles with 2417 genes (instances).
The number of features is 103, whereas each gene is associated with the set of 14 groups (Elisseeff & Weston, 2002).
The same unstructured, structured predictors and weighted graphs, as [Table] in music according to emotion classiﬁcation, were used.
The 10-fold cross validation results of the classiﬁcation are shown in Table [Table]
It can be observed that both GCRFBCb and GCRFBCnb achieved signiﬁcantly better results in comparison with unstructured predictors.
However, neural network trained on all data together achieved the same ACC scores as GCRFBCb.
The AUC of GCRFBCb has outperformed Random forest classiﬁer by 19%
[Table]
whereas SSVM - tree has better ACC compared to GCRFBCnb. It also outperformed
GCRFCnb
[Table]
but as expected, its computation time was longer. In addition, the computation time of CRFs models are longer compared to GCRFBCb
4.2. H IGHWAY CONGESTION
The E70-E75 motorway is a major transit motorway in Serbia. With 504 kilometers, it is the one the major transit motorway in Serbia. It crosses the country from north-west to south, starting at Batrovci border crossing with the Republic of Croatia and ending with Pre ˇ sevo border crossing with the Republic of North Macedonia.
One of the biggest problems in E70-E75 motorway is high congestion that frequently occurs. One of the reasons lies in lack of open toll stations. In order to mitigate congestion problem, it is necessary to predict its occurrence and open enough toll stations. Data used in this research includes information of car entrance and exit for the year 2017.
Two different sections were analyzed: Belgrade - Adaˇsevci and Niˇs - Belgrade.
The section Belgrade - Adaˇsevci was analyzed for the period of
January 2017, whereas section Niˇs - Belgrade was analyzed for the period of April - July 2017.
The congestion was labeled using the similar technique based on KDE as presented in the ski lifts congestion problem.
Based on raw datasets for sections Niˇs - Belgrade and Belgrade - Adaˇsevci with 5,132,918 and 487,767 instances, respectively, a new dataset for section Niˇs - Belgrade is generated by observing shifts in time windows of 10 minutes due to large number of vehicles, whereas in the case of section Belgrade - Adaˇsevci the shifts are observed in time windows of 20 minutes.
Total numbers of instances for sections Belgrade - Adaˇsevci and Niˇs - Belgrade are 50,964 and 235,872, whereas numbers of highway exits (outputs) are 6 and 18, respectively.
The extracted features are similar to the ones presented in ski congestion problem.
The χ 2 statistics, mutual information, correlation matrix and difference of vectors of historical labels were used to capture dependence structure, whereas the same unstructured predictors as in ski lifts congestion problem were evaluated.
The classiﬁcation results, validated by 10 fold cross validation, are presented in Table 4.
The GCRFBCnb achieved the highest AUC and ACC scores in the section Belgrade - Adaˇsevci, whereas GCRFBCb has better prediction performance in section Niˇs - Belgrade.
Moreover, in case of section Niˇs - Belgrade, GCRFBCb has worse ACC score than fully connected CRF, whereas
CRF-tree outperformed GCRFBCnb in section Belgrade - Adaˇsevci
[Table]
In this paper [Table] a new model, called Gaussian Conditional Random Fields for Binary Classiﬁcation (GCRFBC) is presented.
The model is based on latent GCRF structure, which means that intractable structured classiﬁcation problem can become tractable and efﬁciently solved.
Moreover, the improvements previously applied to regression GCRF can be easily extended to GCRFBC.
Two different variants of GCRFBC were derived: GCRFBCb and GCRFBCnb.
Empirical Bayes (marginalization of latent variables) by local variational methods is used in optimization procedure of GCFRBCb, whereas MAP estimate of latent variables is applied in GCRFBCnb.
Based on presented methodology and obtained experimental results on synthetic and real-world datasets it can be concluded that both GCRFBCb and GCRFBCnb models have better prediction performance compared to the analysed structured unstructured predictors.
Additionaly, GCRFBCb has better performance considering AUC score, ACC and lower bound of conditional log likelihood L (cid:0) Y | X , θ (cid:1) compared to GCRFBCnb, in cases where norm of the variances of latent variables is high.
However, in cases where norm of the variances is close to zero, both models have equal prediction performance.
Due to high memory and computational complexity of GCRFBCb compared to GCRFBCnb, in cases where norm of the variances is close to zero, it is reasonable to use GCRFBCnb.
Additionally, the trade off between complexity and accuracy can be made in situation where norm of the variances is high.
Further studies should address extending GCRFBC to structured multi-label classiﬁcation problems, and lower computational complexity of GCRFBCb by considering efﬁcient approximations.
A D ERIVATION OF LOWER BOUND OF CONDITIONAL LIKELIHOOD
In this section we derive lower bound of conditional likelihood.
In order to obtain suitable form of joint distribution that can be easily integrated, the lower bound for sigmoid function was used (Jaakkola & Jordan, 2000).
The lower bound of joint distribution P ( y j , z
j | x j , θ ) can be expressed as:
[Equation]
The simpliﬁed form of Eq.
23 can be represented by rearranging terms in the following form:
[Equation]
The lower bound of likelihood P ( y j | x j , θ , ξ j ) can be obtained by marginalization of z j as:
[Equation]
The lower bound of likelihood P ( y j | x j , θ , ξ j ) can be transformed in the following form:
[Equation]
where S − 1 j = Σ
[Equation]
2Λ j
[Equation] m j = [Equation] This integration is easily performed by noting that it is the integral over an unnormalized Gaussian distribution, which yields:
[Equation]
The ﬁnal form of the lower bound of conditional log likelihood L j ( y j | x j , θ , ξ j ) is:
[Equation]
B P ARTIAL DERIVATIVE OF LOWER BOUND OF CONDITIONAL LOG LIKELIHOOD
The partial derivative of lower bound of conditional log likelihood (GCRFBCb) ∂ L j ( y j | x j , θ , ξ j ) ∂α k is computed as:
[Equation]
where:
[Equation] [Equation]
Similarly partial derivatives with respect to β can be deﬁned as:
[Equation]
where:
[Equation]
In the same manner partial derivatives of conditional log likelihood with respect to ξ ji are:
[Equation]
where:
[Equation]
C P ARTIAL DERIVATIVE OF CONDITIONAL LOG LIKELIHOOD
The derivatives of the conditional log likelihood (GCRFBCnb) with respect to α and β are deﬁned as, respectively:
[Equation]
[Equation] (43) where ∂µ ji ∂α k and ∂µ ji ∂β l are elements of the vectors ∂ µ j ∂α k and ∂ µ j ∂β l and can be obtained by Eqs.
33 and 37, respectively.
D S YNTHETIC DATASET RESULTS
In order to generate and label graph nodes, edge weights S and unstructured predictor values R were randomly generated from uniform distribution.
Besides, it was necessary to choose values of parameters α and β .
Greater values of α indicate that the model is more conﬁdent about performance of unstructured predictors, whereas for the larger value of β the model is putting more emphasis on the dependence structure of output variables.
Six different values of parameters α and β were used.
In the ﬁrst group α and β have similar values, so unstructured predictors and dependence structure between outputs have similar importance.
In the second group, α has higher values compared to β , which means that unstructured predictors are more important than the dependence structure.
In the third group β has higher values than α , meaning that dependence structure is more important than unstructured predictors.
Along with the AUC and conditional log likelihood, norm of the variances of latent variables (diagonal elements in the covariance matrix) is evaluated and presented in Table 5. In addition, the results of experiments are presented in Fig.
1, where for different values of α and β we show differences between GCRFBCb and GCRFBCnb (a) AUC scores, (b) log likelihoods, and (c) norm of the variances of latent variables.
[Table]
α = [5 , 4] β
=
[Table]
α = [1 , 18] β = [1 , 18] 0.
[Table]
[Table]
[Table]
α = [22 , 21] β = [5 , 22] 0.
α = [22 , 21] β = [0 . 1 , 0 .
α = [0 . 8 , 0 . 5] β =
α = [0 . 2 , 0 . 4] β = [1 , 18] 0.