N EURAL
M ESSAGE
P ASSING
FOR
M ULTI -L ABEL
C LASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Multi-label classiﬁcation (MLC) is the task of assigning a set of target labels for a given sample.
Modeling the combinatorial label interactions in MLC has been a long-haul challenge.
Recurrent neural network (RNN) based encoder-decoder models have shown state-of-the-art performance for solving MLC.
However, the sequential nature of modeling label dependencies through an RNN limits its ability in parallel computation, predicting dense labels, and providing interpretable results.
In this paper, we propose Message Passing Encoder-Decoder (MPED) Networks, aiming to provide fast, accurate, and interpretable MLC.
MPED networks model the joint prediction of labels by replacing all RNNs in the encoder-decoder architecture with message passing mechanisms and dispense with autoregressive inference entirely.
The proposed models are simple, fast, accurate, interpretable, and structure-agnostic (can be used on known or unknown structured data).
Experiments on seven real-world MLC datasets show the proposed models outperform autoregressive RNN models across ﬁve different metrics with a signiﬁcant speedup during training and testing time.
I NTRODUCTION
Multi-label classiﬁcation (MLC) is receiving increasing attention in tasks such as text categorization and image classiﬁcation.
Accurate and scalable MLC methods are in urgent need for applications like assigning topics to web articles, classifying objects in an image, or identifying binding proteins on DNA.
The most common and straightforward MLC method is the binary relevance (BR) approach that considers multiple target labels independently (Tsoumakas and Katakis, 2006).
However, in many MLC tasks there is a clear dependency structure among labels, which BR methods ignore.
Accordingly, probabilistic classiﬁer chain (PCC) models were proposed to model label dependencies and formulate MLC in an autoregressive sequential prediction manner (Read et al., 2009).
One notable work in the PCC category was from Nam et al.
(2017) which implemented a classiﬁer chain using a recurrent neural network (RNN) based sequence to sequence (Seq2Seq) architecture, Seq2Seq
MLC.
This model uses an encoder RNN encoding elements of an input sequence, a decoder RNN predicting output labels one after another, and beam search that computes the probability of the next
T predictions of labels and then chooses the proposal with the max combined probability.
However, the main drawback of classiﬁer chain models is that their inherently sequential nature precludes parallelization during training and inference.
This can be detrimental when there are a large number of positive labels as the classiﬁer chain has to sequentially predict each label, and often requires beam search to obtain the optimal set.
Aside from time-cost disadvantages, PCC methods have several other drawbacks.
First, PCC methods require a deﬁned ordering of labels for the sequential prediction, but MLC output labels are an unordered set, and the chosen order can lead to prediction instability (Nam et al., 2017).
Secondly, even if the optimal ordering is known, PCC methods struggle to accurately capture long-range dependencies among labels in cases where the number of positive labels is large (i.e., dense labels).
For example, the Delicious dataset has a median of 19 positive labels per sample, so it can be difﬁcult to correctly predict the labels at the end of the prediction chain.
Lastly, many real-world applications prefer interpretable predictors.
For instance, in the task of predicting which proteins (labels) will bind to a DNA sequence based binding site, users care about how a prediction is made and how the interactions among labels inﬂuence the predictions 1 .
Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017) introduce a class of methods that model joint dependencies of variables using neural message passing rather than an explicit representation such as a probabilistic classiﬁer chain.
Message passing allows for efﬁcient inference by modelling conditional independence where the same local update procedure is applied iteratively to propagate information across variables.
MPNNs provide a ﬂexible method for modeling multiple variables jointly which have no explicit ordering (and can be modiﬁed to incorporate an order, as explained in section 3).
To handle the drawbacks of BR and PCC methods, we propose a modiﬁed version of MPNNs for MLC by modeling interactions between labels using neural message passing.
We introduce Message Passing Encoder-Decoder (MPED)
Networks aiming to provide fast, accurate, and interpretable multi-label predictions.
The key idea is to replace RNNs and to rely on neural message passing entirely to draw global dependencies between input components, between labels and input components, and between labels.
The proposed MPED networks allow for signiﬁcantly more parallelization in training and testing.
The main contributions of this paper are:
• Novel approach for MLC.
To the authors’ best knowledge, MPED is the ﬁrst work using neural message passing for MLC.
• Accurate MLC .
Our model achieves similar, or better performance compared to the previous state of the art across ﬁve different MLC metrics.
We validate our model on seven MLC datasets which cover a wide spectrum of input data structure: sequences (English text, DNA), tabular (bag-of-words), and graph (drug molecules), as well as output label structure: unknown and graph.
• Fast .
Empirically our model achieves an average 1.7x speedup over the autoregressive seq2seq MLC at training time and an average 5x speedup over its testing time.
• Interpretable .
Although deep-learning based systems have widely been viewed as “black boxes” due to their complexity, our attention based MPED models provide a straightforward way to explain label to label, input to label, and feature to feature dependencies.
M ETHOD : MPED N ETWORKS 2. B ACKGROUND : M ESSAGE P ASSING N EURAL N ETWORKS
Message Passing Neural Networks (MPNNs) (Gilmer et al., 2017) are a generalization of graph neural networks (GNNs)
(Gori et al., 2005; Scarselli et al., 2009), where variables are represented as nodes on a graph G and joint dependencies are modelled using message passing rather than explicit representations, which allows for efﬁcient inference.
MPNNs model the joint dependencies using message function M t and node update function U t for T time steps, where t is the current time step.
The hidden state v ti ∈ R d of node i ∈ G is updated based on messages m ti from its neighboring nodes { v tj ∈N ( i ) } deﬁned by neighborhood N ( i ) :
[Equation]
After T updates, a readout function R is used on the updated nodes for a prediction (e.g., node classiﬁcation or graph classiﬁcation) on the graph G .
Many possibilities exist for functions M t and U t .
For example, one can pass messages using neural attention in which nodes are able to attend over their neighborhoods differentially (Veliˇckovi´c et al., 2017).
This allows for the network to learn different weights for different nodes in a neighborhood, without depending on knowing the graph structure a priori.
In this formulation, messages for node v ti are obtained by a weighted sum of all neighboring nodes { v tj ∈N ( i ) } where the weights are obtained by attention (Bahdanau et al., 2014).
In our implementation, we implement neural message passing with attention.
In the rest of the paper, we use “graph attention” and “neural message passing” interchangeably.
Neural message passing with attention works as follows.
Attention weights α tij for pair of nodes ( v ti , v tj ) are computed using some attention function a ( · ) :
[Equation]
where e tij represents the importance of node j for node i .
For our attention function, we used a scaled dot product with node-wise linear transformations W q ∈ R d × d on node v t
i and W u ∈ R
d
×
d on yyyy 4 cc 2 cyxx2 xhhhhhhhhhhcccee 13 eee 12 e
MPNN XX eee
R
MPNN XY
MPNN YY t t t t t t t t t t t t t t t t node v tj .
Scaling by √ d is used to mitigate training issues (Vaswani et al., 2017):
[Equation]
Attention coefﬁcients e tij are then normalized across all neighboring nodes of node i using a softmax function:
[Equation] [Equation]
In our method we use a so called attention message function M t atn to produce the message from node j to node i using the learned attention weights α tij and another transformation matrix W v ∈ R d × d .
Then we compute the full message m ti by linearly combining messages from all nodes j ∈ N ( i ) with a residual connection on the current v ti :
[Equation]
Lastly, node v ti is updated to state v t
+
i using message m ti by a multi-layer perceptron (MLP) update function U mlp with matrices W r ∈ R 2 d × d , W b ∈ R 2 d × d , and a residual connection on m ti :
[Equation]
It is important to note that matrices W are shared (i.e., separately applied) across all nodes.
This can be viewed as 1-dimensional convolution with kernel and stride sizes of 1.
Weight sharing across nodes is a key aspect of MPNNs, where node dependencies are learned in an order-invariant manner.
2. MPED: M ESSAGE P ASSING E NCODER -D ECODER N ETWORK FOR MLC
Notations: We deﬁne the following notations, used throughout the paper.
Let D = { ( x n , y n ) }
Nn = be the set of data samples with inputs x ∈ X and outputs y ∈ Y .
Inputs x are a (possibly ordered) set of S components { x 1 , x 2 , ..., x S } , and outputs y are a set of L labels { y 1 , y 2 , ..., y L } .
MLC involves predicting the set of binary labels { y 1 , y 2 , ..., y L } , y i ∈ { 0 , 1 } given input x .
Input features are represented as embedded vectors { c t =01 , c t =02 , ..., c t =0 S } , c ti ∈ R d , using embedding matrix W x ∈ R δ × d , where d is the embedding size, δ is the vocabulary size, and t represents the ‘state’ of the embedding after t updates.
Similarly, labels are represented as an embedded vectors { h t =01 , h t =02 , ..., h t =0 L } , h ti ∈ R d , using embedding
matrix
W y ∈ R L × d , where L is the number of labels, and t represents the ‘state’ of the embedding after t updates.
Setup:
In MLC, each output label is determined by a joint probability of other labels and the input features.
Our goal is to achieve the performance of explicit joint probability methods such as PCCs (Eqs.
22 and 23)
, at the test speed of BR methods (Eq. 21).
We introduce Message Passing Encoder-Decoder (MPED) networks, where we formulate MLC using an encoder-decoder architecture.
In MPED Networks, input components are represented as nodes in encoder graph G ENC using embedding vectors { c t 1: S } , and labels are represented as nodes in decoder graph G DEC using embedding vectors { h t 1: L } .
MPED networks use three MPNN modules with attention to pass messages within G ENC , from G ENC to G DEC , and within G DEC to model the joint prediction of labels.
The ﬁrst module, MPNN xx , is used to update input component nodes { c t 1: S } by passing messages within the encoder (between input nodes).
The second module, MPNN xy , is used to update output label nodes { h t 1: L } by passing messages from the encoder to decoder (from input nodes { c t 1: S } to output nodes { h t 1: L } ).
The third module, MPNN yy , is used to update output label nodes { h t 1: L } by passing messages within the decoder (between label nodes).
Once messages have been passed to update input and label nodes, a readout function R is then used on the label nodes to make a binary classiﬁcation prediction for each label, { ˆ y 1 , ˆ y 2 , ..., ˆ y L } .
An overview of our model is shown in Fig.
2. E NCODER : M ODELING F EATURE I NTERACTIONS V IA I NPUT C OMPONENT TO C OMPONENT M ESSAGE P ASSING
For a particular input x , we ﬁrst assume that the input features { x 1: S } are nodes on a graph, we call G ENC .
G ENC =
( V, E ) , V = { x 1: S } , and E includes all undirected pairwise edges connecting node c i and node c j .
MPNN xx , parameterized by W xx , is used to pass messages between the input embeddings in order to update their states.
x i can be any component of a particular input (e.g. words in a sentence, patches of an image, nodes of a known graph, or tabular features).
Nodes on G ENC are represented as embedding vectors { c t 1 , c t 2 , ..., c tS } , where the initial states { c 01: S } are obtained using embedding matrix W x on the input components { x 1 , x 2 , ..., x S } .
The embeddings are then updated by MPNN xx using message passing for T time steps to produce { c T 1 , c T 2 , ..., c TS } .
To update input embedding c ti , MPNN xx uses attention message function M t atn (Eq. 6) on all neighboring input embeddings { c tj ∈N ( i ) } to produce messages m ti , and MLP update function U mlp (Eq.
8) to produce updated embedding c t
+
i
[Equation]
If there exists a known G ENC graph, message m ti for node i is computed using its neighboring nodes j ∈ N ( i ) , where the neighbors N ( i ) are deﬁned by the graph.
If there is no known graph, we assume a fully connected G ENC graph, which means N ( i ) =
{ j (cid:54) = i } .
Inputs with a sequential ordering can be modelled as a fully connected graph using positional embeddings (Battaglia et al., 2018).
2. D ECODER
Similar to the input components in the encoder, we assume that the labels { y 1: L } are nodes on a decoder graph called G DEC .
Nodes on G DEC are represented as embedding vectors { h t 1 , h t 2 , ..., h tL } , where the initial states { h 01: L } are obtained using label embedding matrix W y .
The decoder MPNNs update the label embeddings { h t 1: L } by passing messages from the encoder to the decoder, and then pass messages within the decoder.
MPNN xy , is used to pass messages from input embeddings { c T 1: S } to label embeddings, and then MPNN yy is used to pass messages between label embeddings.
2.4. U PDATING L ABEL E MBEDDINGS THROUGH I NPUT TO L ABEL M ESSAGE P ASSING
In order to update the label nodes given a particular input x , the decoder uses MPNN xy , parameterized by W xy , to pass messages from input x to labels y .
At the equation level, this module is identical to MPNN xx except that it updates the i th label node’s embedding h i using the embeddings of all the components of an input.
That is, we update each h ti by using a weighted sum of all input embeddings { c T 1: S } , in which the weights represent how important an input component is to the i th label node and the weights are learned via attention.
Messages are only passed from the encoder nodes to the decoder nodes, and not vice versa (i.e. encoder to decoder message passing is directed).
More speciﬁcally, to update label embedding h ti , MPNN xy uses attention message function M t atn on all embeddings of the input { c T 1: S } to produce messages m ti , and another MLP update function U mlp to produce the updated intermediate embedding state h t (cid:48) i :
[Equation]
The key advantage of input-to-label message passing with attention is that each label node can attend to different input nodes (e.g. different words in the sentence).
2.4. U PDATING L ABEL E MBEDDINGS T HROUGH L ABEL TO L ABEL M ESSAGE P ASSING
At this point, the decoder can make an independent prediction for each label conditioned on x .
However, in order to make more accurate predictions, we model interactions between the label nodes { h t (cid:48) 1: L } using message passing and update them accordingly.
To do this we use a a third message passing module, MPNN yy .
At the equation level, this layer is identical to MPNN xx except that it replaces the input embeddings with label embeddings.
In other words, label embedding h t (cid:48)
i is updated by a weighted combination through attention of all its neighbor label nodes { h t (cid:48)
j ∈N ( i ) } .
To update each label embedding h t (cid:48)
i , MPNN yy uses attention message function M t (cid:48) atn on all neighbor label embeddings { h t (cid:48)
j ∈N ( i ) } to produce message m ti , and MLP update function U t (cid:48)
mlp to compute updated embedding h t
+
i
[Equation]
If there exists a known G DEC graph, message m ti for node i is computed using its neighboring nodes j ∈ N ( i ) , where the neighbors N ( i ) are deﬁned by the graph.
If there is no known G DEC graph, we assume a fully connected graph, which means N ( i ) =
{ j (cid:54) = i } .
In our implementation, the label embeddings are updated by MPNN xy and MPNN xx for T time steps to produce { h T 1 , h T 2 , ..., h TL } .
2. R EADOUT L AYER (MLC P REDICTIONS FROM THE D ECODER )
The last module of the decoder predicts each label { ˆ y 1 , ...
ˆ y L }.
A readout function R projects each of the L label embeddings h
Ti using projection matrix W o ∈ R L × d , where row W oi ∈ R d is the learned output vector for label i .
The calculated vector of size L × 1 is then fed through an element-wise sigmoid function to produce probabilities of each label being positive:
[Equation]
In MPED networks we use binary the mean cross entropy on the individual label predictions to train the model.
p ( y i |{ y j (cid:54) = i } , c T 1: S ; W ) is approximated in MPED networks by jointly representing { y 1: L } using message passing from { c T 1: S } and from the embeddings of all neighboring labels { y j ∈N ( i ) } . 2. M
[Table]
Multi-head Attention
In order to allow a particular node to attend to multiple other nodes (or multiple groups of nodes) at once, MPED uses multiple attention heads.
Inspired by Vaswani et al.
(2017), we use K independent attention heads for each W · matrix during the message computation, where each matrix column W · ,kj is of dimension d/K .
The generated representations are concatenated (denoted by (cid:107) ) and linearly transformed by matrix W z ∈ R d × d .
Multi-head attention changes
[Table]
[Table]
mlp
[Table]
same.
[Equation]
Graph Time Steps To learn more complex relations among nodes, we compute T time steps of graph updates.
This is essentially a stack of T MPNN layers.
Matrices W q · , W u · , W v · , W r · , W b · , W z · , are not shared across time steps (but are shared across nodes).
Label Embedding Weight Sharing To enforce each label’s input embedding to correspond to that particular label, the label embedding matrix weights W y are shared with the readout projection matrix W o .
In other words, W y is used to produce the initial node vectors for G DEC , and then is used again to calculate the pre-sigmoid output values for each label, so W o ≡ W y .
This was shown beneﬁcial in Seq2Seq models for machine translation (Press and Wolf, 2016).
2. A DVANTAGES OF MPED M ODELS
Speed.
In MPED models, the joint probability of labels isn’t explicitly estimated using the chain rule.
This enables making predictions in parallel and decreases test time drastically, especially when the number of labels is large.
We model the joint probability implicitly using the MPED decoder, at the beneﬁt of a substantial speedup.
Time complexities of different types of models are compared in
Table 1.
The biggest advantage of MPED networks is constant training and testing times.
[Table]
Handling dense label predictions.
Motivated by the drawbacks of autoregressive models for MLC (Section 5.6), the proposed MPED model removes the dependencies on a chosen label ordering and beam search.
This is particularly beneﬁcial when the number of positive output labels is large (i.e. dense).
MPED networks predict the output set of labels all at once, which is made possible by the fact that inference doesn’t use a probabilistic chain, but there is still a representation of label dependencies via label to label attention.
As an additional beneﬁt, as noted by Belanger and McCallum (2016), it may be useful to maintain ‘soft’ predictions for each label in MLC.
This is a major drawback of the
PCC models which make ‘hard’ predictions of the positive labels, defaulting all other labels to 0.
Interpretability.
One advantage of MPED models is that interpretability is “built in” via graph attention.
Speciﬁcally, we can visualize 3 aspects: input-to-input attention (word dependencies), input-to-label attention (label/word dependencies), and label-to-label attention (label dependencies), as shown in the Appendix.
2. C ONNECTING TO R ELATED T OPICS
Structured Output Predictions
The use of graph attention in MPED models is closely connected to the literature of structured output prediction for MLC.
Ghamrawi and McCallum (2005) used conditional random ﬁelds (Lafferty et al., 2001) to model dependencies among labels and features for MLC by learning a distribution over pairs of labels to input features.
MPED also models similar dependencies through attention mechanisms on embeddings.
In another research direction, recently proposed SPENs (structured prediction energy network (Belanger and McCallum, 2016; Tu and Gimpel, 2018)) and Deep Value Networks (Gygli et al., 2017) tackled MLC by optimizing different variants of structured loss formulations.
As a future direction, we plan to improve MPED models by integrating a similar structured loss in our formulation.
Graph Neural Networks (GNNs)
Passing embedding messages from node to neighbor nodes connects to a large body of literature on graph neural networks (Battaglia et al., 2018) and embedding models for structures (Dai et al., 2016).
The key idea is that instead of conducting probabilistic operations (e.g., product or re-normalization), the proposed models perform nonlinear function mappings in each step to learn feature representations of structured components.
Neural message passing networks (Gilmer et al., 2017), graph attention networks (Veliˇckovi´c et al., 2017) and neural relation models (Battaglia et al., 2016) follow similar ideas to pass the embedding from node to neighbor nodes or neighbor edges.
There have been many recent works extending the basic GNN framework to update nodes using various message passing, update, and readout functions (Kipf and
Welling, 2016; Hamilton et al., 2017; Duvenaud et al., 2015; Li et al., 2015; Gilmer et al., 2017;
Battaglia et al., 2016; Kearnes et al., 2016; Zheng et al., 2018).
We refer the readers to (Battaglia et al., 2018) for a survey.
However, none of these have used GNNs for MLC.
E XPERIMENTS
In the appendix, we have added the details of training/hyperparameters (5.2), datasets (5.1), evaluation metrics (5.3), and explained how we selected baseline models from previous work (5.8).
We explain our own MPED variations in 5.7 and previous work baselines in section 5.8.
In short, we compare MPED to (1)
MPED
Prior G DEC (by using a known label graph), (2) MPED
Edgeless G DEC (by removing label-to-label message passing), (3) MPED Autoregressive (by predicting labels sequentially), (4) MP BR (binary relevance MLP output used on the mean of MPNN xx embeddings), and the baselines reported in related works.
We compare our models on seven real world datasets, which vary in the number of samples, number of labels, input type (sequential, tabular, graph), and output type (unknown, known label graph), as summarized in Table 3.
Across all datasets, MPED outperforms or achieves similar results as the baseline models.
Most importantly, we show that autoregressive models are not crucial in MLC for most metrics, and non-autoregressive models result in a signiﬁcant speedup at test time.
Accuracy Table 2 shows the performance of different models across the seven datasets.
Metric signiﬁcance across models is shown in Appendix Table 4.
For subset accuracy, autoregressive models perform the best, but at a small margin of increase.
More importantly, autoregressive models that predict only positive labels are targeted at maximizing subset accuracy, but they perform poorly on other metrics.
For all other metrics, autoregressive models are not essential.
One important observation is that for most datasets, MPED outperforms the autoregressive models in both miF1 (frequent labels) and more importantly, maF1 (rare labels).
Since maF1 favors models which can predict the rare labels, this shows that autoregressive models with beam search often make the wrong predictions on the rare labels (which are ordered last in the sequence during training).
MPED is a solid choice across all metrics as it comes the closest to subset accuracy as the autoregressive models, but also performs well in other metrics.
While MPED does not explicitly model label dependencies as autoregressive or structured prediction models do, it seems as though the attention weights do learn some dependencies among labels (Visualizations, Tables 3 and 4).
This is indicated by the fact that MPED, which uses label-to-label attention, mostly outperforms the ones which don’t, indicating that it is learning label dependencies.
Table 2 shows 3 time step models results, but a comparison of time steps is shown in Figure 2.
Speed Table 2 shows the per epoch train and test times for each model.
All models are trained and tested on the same GPU using a batch size of 32.
At test time, since the autoregressive model cannot be parallelized, MPED and other non-autoregressive models are signiﬁcantly faster.
During training, the autoregressive model can be parallelized because the true labels are fed as the previous label.
Since the autoregressive model only predicts the ρ positive labels, it can be faster at training time, whereas the MPED model is predicting the probability for all labels.
MPED results in a mean of 1.7x and 5.0x training and testing speedups, respectively, over Seq2Seq autoregressive models.
Visualizations We present visualizations of the input-to-label and label-to-label attention weights (averaged across the 4 attention heads) in the Appendix.
In the visualizations, we show the positive labels only, and the darker lines show higher attention weights to the corresponding label or word.
The attention weights clearly learn certain relationships between input-label pairs as well as the
[Table]
label-label pairs, which is all done in an unsupervised manner.
In future work, we plan to add a structured prediction loss function which will likely improve the attention mechanisms and the model’s ability to estimate the joint probability.
C ONCLUSION
In this work we present Message Passing Encoder-Decoder (MPED) Networks which achieve a signiﬁcant speedup at close to the same performance as autoregressive models for MLC.
We open a new avenue of using neural message passing to model label dependencies in MLC tasks.
In addition, we show that our method is able to handle various input data types (sequence, tabular, graph), as well various output label structures (known vs unknown).
One of our future extensions is to adapt the current model to predict more dynamic outputs.
A PPENDIX :
[Table] [Table]
(Read et al., 2009; Demšar, 2006)
5. D ATASETS
We test our method against baseline methods on seven different multi-label sequence classiﬁcation datasets.
The datasets are summarized in Table 3.
We use Reuters-21578, Bibtex (Tsoumakas et al., 2009), Delicious (Tsoumakas et al.),
Bookmarks (Katakis et al.), RCV1-V2 (Lewis et al., 2004), our own DNA protein binding dataset (TFBS) from Consortium et al.
(2012), and SIDER (Kuhn et al., 2015), which is side effects of drug molecules.
As shown in the table, each dataset has a varying number of samples, number of labels, positive labels per sample, and samples per label.
For BibTex and Delicious, we use 10% of the provided training set for validation.
For the TFBS dataset, we use 1 layer of convolution at the ﬁrst layer to extract “words” from the DNA characters (A,C,G,T), as commonly done in deep learning models for DNA.
For datasets which have sequential ordering of the input components (Reuters, RCV1), we add a positional encoding to the word embedding as used in Vaswani et al.
(2017) (sine and cosine functions of different frequencies) to encode the location of each word in the sentence.
For datasets with no ordering or graph stucture (Bibtex, Delicious, Bookmarks, which use bag-of-word input representations) we do not use positional encodings.
For inputs with an explicit graph representation (SIDER), we use the known graph structer.
5. M ODEL H YPERPARAMETERS
We validate our model on seven MLC datasets.
These datasets cover a wide spectrum of input data types, including: raw English text (sequential form), bag-of-words (tabular form), and drug molecules (graph form).
For all 6 datasets except SIDER, we use the same MPED model with T =3 time steps, d = 512 , and K =4 attention heads.
Since SIDER is signiﬁcantly smaller, we use T =1 time step, d = 64 , and K =4 attention heads.
We trained our models on an NVIDIA TITAN
X Pascal with a batch size of 32.
We used Adam (Kingma and Ba, 2014) with betas = (0 . 9 , 0 . 999) , eps = 1e-08, and a learning rate of 0.0002 for each dataset.
We used dropout of p = 0 .
2 for all models.
The MPED models also use layer normalization (Ba et al., 2016) around each of the attention and feedforward layers.
The non-autoregressive models are trained with binary cross-entropy on each label and the autoregressive models are trained with cross entropy across all possible labels at each position.
5. E VALUATION M ETRICS
Multi-label classiﬁcation methods can be evaluated with many different metrics which each evaluate different strengths or weaknesses.
We use the same 5 evaluation metrics from Nam et al.
All of our autoregressive models predict only the positive labels before outputting a stop signal.
This is a special case of PCC models (explained as PCC + in section 5.4), which have been shown to outperform the binary prediction of each label in terms of performance and speed.
These models use beam search at inference time with a beam size of 5.
For the non-autoregressive models, to convert the labels to { 0 , 1 } we chose the best threshold on the validation set from the same set of thresholds used in Tu and Gimpel (2018).
Example-based measures are deﬁned by comparing the target vector y to the prediction vector ˆ y .
Subset
Accuracy (ACC) requires an exact match of the predicted labels and the true labels: ACC ( y , ˆ y )
=
I [ y = ˆ y ] .
Hamming Accuracy (HA) evaluates how many labels are correctly predicted in ˆ y : HA ( y , ˆ y )
=
1 L (cid:80)
Lj =
I [ y j = ˆ y j ] .
Example-based F1 (ebF1) measures the ratio of correctly predicted labels to the sum of the total true and predicted labels:
[Table]
Label-based measures treat each label y j as a separate two-class prediction problem, and compute the number of true positives ( tp j ), false positives ( fp j ), and false negatives ( fn j ) for a label.
Macro-averaged F1 (maF1) measures the
[Table] [Table]
averaged over all labels:
[Table]
measures the label-
[Table]
F1 averaged over each sample:
[Table]
usually indicate
[Table]
performance
[Table] [Table]
usually
[Table]
5. B ACKGROUND OF M ULTI -L ABEL C LASSIFICATION :
MLC has a rich history in text (McCallum; Ueda and Saito, 2003), images (Tsoumakas and Katakis, 2006; Elisseeff and Weston, 2002), bioinformatics (Tsoumakas and Katakis, 2006; Elisseeff and Weston, 2002), and many other domains.
MLC methods can roughly be broken into several groups, which are explained as follows.
Label powerset models (LP) (Tsoumakas and Vlahavas, 2007; Read et
al., 2011), classify each input into one label combination from the set of all possible combinations Y = {{ 1 } , { 2 } , ..., { 1 , 2 , ..., L }} .
LP explicitly models the joint distribution by predicting the one subset of all positive labels.
Since the label set Y grows exponentially in the number of total labels ( 2 L ), classifying each possible label set is intractable for a modest L .
In addition, even in small L tasks, LP suffers from the “subset scarcity problem” where only a small amount of the label subsets are seen during training, leading to bad generalization.
Binary relevance (BR) methods predict each label separately as a logistic regression classﬁer for each label (Zhang and Zhou, 2005; Godbole and Sarawagi, 2004).
The naïve approach to BR prediction is to predict all labels independently of one another, assuming no dependencies among labels.
That is, BR uses the following conditional probability parameterized by learned weights W :
[Equation]
Probabilistic classiﬁer chain (PCC) methods (Dembczynski et al., 2010; Read et al., 2009) are autoregressive models that estimate the true joint probability of output labels given the input by using the chain rule, predicting one label at a time:
[Equation]
Two issues with PCC models are that inference is very slow if L is large, and the errors propagate as L increases (Montañes et al., 2014).
To mitigate the problems with both LP and PCC methods, one solution is to only predict the true labels in the LP subset.
In other words, only predicting the positive labels (total of ρ for a particular sample) and ignoring all other labels, which we call PCC + .
Similar to PCC, the joint probability of PCC + can be computed as product of conditional probabilities, but unlike PCC, only ρ < L terms are predicted as positive:
[Equation]
This can be beneﬁcial when the number of possible labels L is large, reducing the total number of prediction steps.
However, in both PCC and PCC + , inference is done using beam search, which is a costly dynamic programming step to ﬁnd the optimal prediction.
MPED methods approximate the following factored formulation, where N ( Y i ) denotes the neighboring nodes of Y i .
[Equation]
5. S EQ 2S EQ M ODELS
In machine translation (MT), sequence-to-sequence (Seq2Seq) models have proven to be the superior method, where an encoder RNN reads the source language sentence into an encoder hidden state, and a decoder RNN translates the hidden state into a target sentence, predicting each word autoregressively (Sutskever et al., 2014).
Bahdanau et al.
(2014) improved this model by introducing “neural attention” which allows the decoder RNN to “attend” to every encoder word at each step of the autoregressive translation.
Recently, Nam et al.
(2017) showed that, across several metrics, state-of-the-art MLC results could be achieved by using a recurrent neural network (RNN) based encoder-to-decoder framework for Equation 23 (PCC + ).
They use a Seq2Seq RNN model (Seq2Seq Autoregressive) which uses one RNN to encode x , and a second RNN to predict each positive label sequentially, until it predicts a ‘stop’ signal.
This type of model seeks to maximize the ‘subset accuracy’, or correctly predict every label as its exact 0/1 value.
Vaswani et al.
(2017) eliminated the need for the recurrent network in MT by introducing the Transformer.
Instead of using an RNN to model dependencies, the Transformer explicitly models pairwise dependencies among all of the features by using attention (Bahdanau et al., 2014; Xu et al.) between signals.
This speeds up training time because RNNs can’t be fully parallelized but, the transformer still uses an autoregressive decoder.
5. D RAWBACKS OF A UTOREGRESSIVE M ODELS
Autoregressive models have been proven effective for machine translation and MLC (Sutskever et al., 2014;
Bahdanau et al., 2014;
Nam et al., 2017).
However, predictions must be made sequentially, eliminating parallelization.
Also, beam search is typically used at test time to ﬁnd optimal predictions.
But beam search is limited by the time cost of large beams sizes, making it difﬁcult to optimally predict many output labels (Koehn and Knowles, 2017).
In addition to speed constraints, beam search for autoregressive inference introduces a second drawback: initial wrong predictions will propagate when using a modest beam size (e.g. most models use a beam size of 5).
This can lead to signiﬁcant decreases in performance when the number of positive labels is large.
For example, the Delicious dataset has a median of 19 positive labels per sample, and it can be very difﬁcult to correctly predict the labels at the end of the prediction chain.
Autoregressive models are well suited for machine translation because these models mimic the sequential decoding process of real translation.
However, for MLC, the output labels have no intrinsic ordering.
While the joint probability of the output labels is independent of the label ordering via autoregressive based inference, the chosen ordering can make a difference in practice (Vinyals et al., 2015; Nam et al., 2017).
Some ordering of labels must be used during training, and this chosen ordering can lead to unstable predictions at test time.
Our non-autoregressive version connects to Gu et al.
(2017) who removed the autoregressive decoder in MT with the Non-Autoregressive Transformer.
In this model, the encoder makes a proxy prediction, called “fertilities”, which are used by the decoder to predict all translated words at once.
The difference between their model and ours is that we have a constant label at each position, so we don’t need to marginalize over all possible labels at each position.
5. M ODEL V ARIATIONS
MPED :
In the full MPED model, we use 3 encoder time steps and 3 decoder time steps with node to node attention in both the encoder and decoder graphs, and K =4 attention heads.
MPED Prior G DEC : The same as MPED except that we use a prior representation of G DEC rather than fully connected.
For Reuters, Bibtex, Delicious, and Bookmarks we have extracted label graphs using label similarity (using ﬁxed WUP scores) from WordNet.
For RCV1, we use the topological graph from the RCV1 dataset.
For TFBS, we use String-DB (Szklarczyk et al., 2016) to obtain the label graph representing TF-TF protein interactions.
MPED Edgeless G DEC : In this baseline, we remove MPNN yy , and only use MPNN xy to pass messages from the inputs to the labels, and not between labels.
This model assumes no dependencies among output labels.
MPED Autoregressive: Instead of predicting all labels simultaneously and modelling label dependencies using graph attention, the MPED Autoregressive variation models p ( y i +1 | y 1: i , x 1: S ; W ) by predicting each positive label
y i sequentially and using attention on { c T 1: S } and the previously predicted labels (ˆ y 1 , ..., ˆ y i ) until a stop signal is predicted (see PCC + in Appendix).
While it is possible to predict the probability for each label (i.e. regular PCC), this is extremely time consuming when the number of possible labels is large (and in general, doesn’t increase performance (Nam et al., 2017)).
BR :
The mean node output of a MPNN xx is used as the input to a 2 layer multilayer perceptron (MLP) readout module.
This model assumes no dependencies among output labels.
RNN
Seq2Seq
A
layer RNN Seq2Seq
model with attention.
This is similar to
RNN
Seq2Seq from Nam et al.
(2017) (RNN m in their paper), except with 3 layers.
5. B ASELINE C OMPARISONS
For Reuters and RCV1-V2, we compare against the MLP and autoregressive model used in (Nam et al., 2017), which use these two datasets.
MLP standard MLP without the self attention input representation
[Table]
5. V ISUALIZATIONS
[Figure]
Label
Label corn grain oilseed soybean wheat − / .
according added adds agriculture and &apos;s began begins cancelled cancels china commitments corn covering delivery department ended export february for has in its june num of outstanding purchases report reuter said sales season september soybeans the to tonnes total transactions u.s. usda week wheat
[Figure]