ME-GCN: M ULTI - DIMENSIONAL E DGE -E MBEDDED G RAPH C ONVOLUTIONAL N ETWORKS FOR S EMISUPERVISED T EXT C LASSIFICATION
A BSTRACT
Compared to sequential learning models, graph-based neural networks exhibit excellent ability in capturing global information and have been used for semisupervised learning tasks.
Most Graph Convolutional Networks are designed with the single-dimensional edge feature and failed to utilise the rich edge information about graphs.
This paper introduces the ME-GCN (Multi-dimensional Edgeenhanced Graph Convolutional Networks) for semi-supervised text classification
A text graph for an entire corpus is firstly constructed to describe the undirected and multi-dimensional relationship of word-to-word, document-document and word-to-document.
The graph is initialised with corpus-trained multi-dimensional word and document node representation and the relations are represented according to the distance of those words/documents nodes.
Then, the generated graph is trained with ME-GCN which considers the edge features as multi-stream signals and each stream performs a separate graph convolutional operation.
Our
ME-GCN can integrate a rich source of graph edge information of the entire text corpus.
The results have demonstrated that our proposed model has significantly outperformed the state-of-the-art methods across eight benchmark datasets.
The code is available on: https://github.com/usydnlp/ME GCN
I NTRODUCTION
Deep Learning models have performed well and have been widely used for text classification; however, the performance is not always satisfactory when utilising small labelled datasets.
In many practical scenarios, the labelled dataset is very scarce as human labelling is time-consuming and may require domain knowledge.
There is a pressing need for studying semi-supervised text classification with a relatively small number of labelled training data in deep learning paradigm.
For the successful semi-supervised text classification, it is crucial to maximize effective utilization of structural and feature information of unlabelled data.
Graph Neural Networks have recently received lots of attention as it can analyse rich relational structure, prioritize global features exploitation, and preserve global structure of a graph in embeddings.
Due to these benefit, there have been successful attempts to revisit semi-supervised learning with Graph Convolutional Networks (GCN) (Kipf & Welling, 2017).
TextGCN (Yao et al., 2019) initialises the whole text corpus as a document-word graph and applies GCN.
It shows potential of
GCN-based semi-supervised text classification.
Hu et al.
(2019) worked on semi-supervised short text classification using GCN with topic-entity, and Liu et al.
(2020) proposed tensorGCN with semantic, syntactic, and sequential information.
One major problem in those existing GCN-based text classification models is that edge features are restricted to be one-dimensional, which are the indication about whether there is edge or not (e.g. binary connectedness) or often one-dimensional real-value representing similarities (e.g. pmi, tf-idf).
Instead of being a binary indicator variable or a single-dimensional value, edge features can possess rich information and fully incorporated by using multi-dimensional vectors.
Addressing this problem is likely to benefit several graph-based classification problems but is particularly important for the text classification task.
This is because the relationship between words and documents can be better represented in a multi-dimensional vector space rather than a single value.
For example, word-based vector space models embed the words in a vector space where similarly defined words are mapped near to each other.
Rather than using the lexical-based syntactic parsers or additional resources, words that share semantic or syntactic relationships will be represented by vectors of similar magnitude and be mapped in close proximity to each other in the word embedding.
Using this multi-dimensional word embedding as node and edge features, it would be more effective to analyse rich relational information and explore global structure of a graph.
Then, what would be the best way to exploit edge features in a text graph convolutional network?
According to the recently reported articles (Gong & Cheng, 2019; Khan & Blumenstock, 2019; Huang et al., 2020; Liu et al., 2020; Schlichtkrull et al., 2018), more rich information should be considered in the relations in the graph neural networks.
In this paper, we propose a new multi-dimensional edge enhanced text graph convolutional networks (ME-GCN), which is suitable for the semi-supervised text classification.
Note that the focus of our semi-supervised text classification task is on small proportion of labelled text documents with no other resource, i.e. no pre-trained word embedding or language model, syntactic tagger or parser.
We construct a single large textual graph from an entire corpus, which contains words and documents as nodes.
The graph describes the undirected and multi-dimensional relationship of word-to-word, document-document, and word-to-document.
Each word and document are initialised with corpustrained multi-dimensional word and document embedding, and the relations are represented based on the semantic distance of those representations.
Then, the generated graph is trained with MEGCN, which considers edge features as multi-stream signals, and each stream performs a separate graph convolutional operation.
We conduct experiments on several semi-supervised text classification datasets.
Our model can achieve strong text classification performance with a small proportion of labelled documents with no additional resources.
The main contributions: (1) To the best of our knowledge, this is the first attempt to apply multi-dimensional edge features on GNN for text classification.
(2) ME-GCN is proposed to use corpus-trained multi-dimensional word and document-based edge features for the semi-supervised text classification.
(3) Experiments are conducted on several semi-supervised text classification datasets to illustrate the effectiveness of ME-GCN.
R ELATED W ORKS
Semi-supervised text classification: Due to the high cost of human labelling and the scarcity of fully-labelled data, semi-supervised models have received attention in text classification.
Latent variable models (Chen et al., 2015) apply topic models by user-oriented seed information and infer the documents’ labels based on category-topic assignment.
The embedding-based model (Tang et al., 2015; Meng et al., 2018) utilise seed information to derive text (word or document) embeddings for documents and labels for text classification.
Yang et al.
(2017) leveraged sequence-to-sequence
Variational AutoEncoders (VAEs), and Miyato et al.
(2017) utilized adversarial training to the text domain by applying perturbations to the word embeddings.
Graph convolutional networks (GCN) have been popular in semi-supervised learning as it shows superior global structure understanding ability (Kipf & Welling, 2017).
GNN for Text Classification: Graph Neural Networks have successfully used in various NLP tasks (Bastings et al., 2017; Tu et al., 2019;
Cao et al., 2019; Xu et al.).
Yao et al.
(2019) proposed the Text
Graph Convolutional Networks by applying a basic GCN (Kipf & Welling, 2017) to the text classification task.
In their work, a text graph for the whole corpus is constructed; word and document nodes are initialised with one-hot representation and edge features are represented as one-dimensional real values, such as PMI, TF-IDF.
Several studies have attempted multiple different graph alignments using knowledge graph or semantic/syntactic graph.
Vashishth et al.
(2019) applied GCN to incorporate syntactic/semantic information for word embedding training.
Cao et al.
(2019) proposed an alignment-oriented knowledge graph embedding for entity alignment.
TensorGCN (Liu et al., 2020) proposed semantic, syntactic, and sequential contextual information.
In their framework, multiple aspect graphs are constructed from external resources, and those graph are jointly trained.
There are several Multi-aspect, Multi-dimension edge research have been published but none of them are working on the Natural Language Processing field (Schlichtkrull et al., 2018; Khan & Blumenstock, 2019; Ma et al., 2020;
He et al., 2020).
Recently, graph attention mechanism has been applied in text classification tasks (Mei et al., 2021; Liu et al., 2021;
Yang et al., 2021a).
Others focus on using both local and global information (Jin et al., 2021), multi-modality with text and image information (Yang et al., 2021b), enhancing TextGCN with other models (Ragesh et al., 2021) and combining with external knowledge (Dai et al., 2022).
ME-GCN
We propose the Multi-dimensional Edge-enhanced Graph Convolutional Networks (ME-GCN) for semi-supervised text classification.
Note that all graph components are only based on the given text corpus without using any external resources.
We utilize the GCN as a base component, due to its simplicity and effectiveness.
Figure 1 shows the overall architecture of ME-GCN.
Graph A GCN (Kipf & Welling, 2017) is a generalised version of the convolutional neural networks for semi-supervised learning that operates directly on the graph-structured data and induces embedding vectors of nodes based on properties of their neighbourhoods.
Consider a graph
G =
( V, E, A ) , where V ( | V | = N ) is the set of graph nodes, E is the set of graph edges, and
A ∈ R N × N is the graph adjacency matrix.
3. T EXTUAL G RAPH C ONSTRUCTION
We first describe how to construct a textual graph that contains word/document node representation and multi-dimensional edge features for a whole text corpus.
We apply a straightforward textual construction approach that treats words and documents as nodes in the graph.
Unlike (Yao et al., 2019), we have three types of edges, namely word-document edge, word-word edge, and documentdocument edge with the aim to investigate all possible relations between nodes.
Formally, we define a ME-GCN graph G ME =
( V, E ( t ) , ME ( t ) ) , where t denotes the t th dimensional edge, V ( | V | =
N ) is the set of graph nodes of word/document, E ( t ) are the set of graph edges, which can be one of the three types, and ME ( t ) is the set of adjacency matrix at the t th dimension.
The details of node and edge features construction are presented as follows.
3.1. T EXTUAL N ODE C ONSTRUCTION
From an entire textual corpus, we construct word and document nodes in a graph so that the global word and document distance can be explicitly modeled and graph convolution can be easily adapted.
ME-GCN considers the word and document nodes as components for preserving rich information and representing the global structure of a whole corpus, which can fully support for the successful semi-supervised text classification.
With this in mind, ME-GCN trains word/node feature by using a Word2Vec (Mikolov et al., 2013) for word nodes, and a Doc2Vec (Le & Mikolov, 2014) for document nodes.
For instance, Word2Vec takes as its input a whole corpus of words, and the trained word vectors are positioned in a vector space such that words that share common contexts in the corpus are located in close proximity to one another in the space.
This is well-aligned with the role of graph neural networks, representing the global structure of the corpus, and preserving rich semantic information of the corpus.
Most importantly, those word/document embeddings are distributed representations of text in an T -dimensional space so the distance between words and documents can be represented as a multi-dimensional vector.
Formally, the word/document node features in ME-GCN are initialised as follows.
Note that the negative sampling is applied to reduce the training time.
Word Node Construction We train the Word2Vec CBOW (Mikolov et al., 2013) using context words to predict the centre word.
Assume we have a given text corpus consisting of K documents and U unique words.
The input is a set of context words X ik in document k ∈ K encoded as one-hot vector of size U .
Then the hidden layer H and output layer Output are formulated in equation (1) and (2), in which W U × T and W ′ T
× U are two projection matrix.
After training, we extract the U vectors of dimension T from the updated matrix W U × T representing the corresponding U unique words in the whole corpus.
[Equation]
Document Node Construction Doc2Vec CBOW (Le & Mikolov, 2014) is essentially the same as
Word2Vec.
In Doc2Vec, we feed the context words X ik together with the current document k to the model, which is also encoded as one-hot vector based on the document id, and the vector size becomes ˆ U
=
U + K .
We have the projection matrix W ˆ U × T containing U + K vectors.
After training, those K vectors in the updated W T × ˆ U are used for representing the corresponding K document.
[Equation]
3.1. M ULTI - DIMENSIONAL E DGE C ONSTRUCTION
In this section, we describe how to construct a multi-dimensional edge feature in a graph.
A traditional textual graph edge (Yao et al., 2019) was based on word occurrence in documents (documentword edges), and word co-occurrence in the whole corpus (word-word edges), however, the occurrence information is not enough to extract how close two pieces of text are in both surface proximity and meaning.
According to Mikolov et al.
; Kusner et al.
, the distance between word/document embeddings learn semantically meaningful representations for words from local cooccurrences in sentences and each dimension of word2vec and doc2vec represents the same aspect of word/document representations.
Inspired by this, we utilise the distance between each dimension of word/document embeddings to preserve the rich semantic information captured by edges, which are also presented as multi-dimensional vectors.
To represent all possible edge types, we propose three types of edges: word-word edges, document-document edges, and word-document edges.
Our goal is to incorporate the semantic similarity between individual node pairs (each unique word and document) into multi-dimensional edge features.
One such measure for word/document node similarity is provided by their Euclidean distance in the Word2Vec or Doc2Vec embedding space.
We separately use each dimension space in the node feature (Word2Vec/Dec2Vec) for representing each of the dimension in the multi-dimensional node edge.
Thus, we have T dimensional edges between nodes of T dimensional features and each t ∈ { 1 , 2 , ..., T } is represented by one dimensional Euclidean distance calculation in the t th dimensional space.
This edge calculation method is applied to word-word and doc-doc edges.
Word-Word Edge Feature We draw on the learned semantics in each feature dimension of the word embedding of size T to calculate the edge weight for each dimension.
Concretely, the Tdimensional word-word edge E ( t ) w i ,w j , t ∈ { 1 , 2 , ..., T } between word i and word j is formulated as in equation (5), in which W ( t ) i and W ( t ) j represents the feature value at the dimension t of the word embedding W i
[Equation]
Doc-Doc Edge Feature The document-document edge is constructed in a way similar to the wordword edge.
As is shown in equation (6), the T -dimensional document-document edge E ( t ) d i ,d j is calculated based on the normalized Euclidean distance between the values D ( t ) i and D ( t ) j at each dimension t of the features for document i and j .
To relieve over-smoothing issue, we only consider edges between two documents having over u overlapping words.
[Equation]
Word-Doc Edge Feature We use the same calculation method for a single-dimension worddocument edge as in TextGCN while repeating it for each dimension t .
Thus, the T -dimensional word-document edge E ( t ) w i ,d j is simply represented as the TF-IDF value of word i and document j .
This is repeated for each dimension t , as is formulated in equation (7).
We also found using TF-IDF weight is better than using term frequency only.
[Equation]
3. ME-GCN L EARNING
After constructing the multi-dimensional edge enhanced text graph, we focus on applying effective learning framework to perform GCN on the textual graph with multi-dimensional edge features.
The traditional GCN learning takes into the initial input matrix H (0)
∈ R N × d 0 containing N node features of size d 0 .
Then the propagation through layers is made based on the rule in equation (9), which takes into consideration both node features and the graph structure in terms of connected edges. [Equation]
[Equation]
The l and ( l +1) represents the two subsequent layers, ˆ A
= ˜ D − 12 ˜
A ˜ D − 12 is the normalized symmetric adjacency matrix ˜
A =
A
+ I
( I is an identity matrix for including self-connection), ˜ D is the diagonal node degree matrix with ˜ D
( i, i ) =
Σ j ˜ A ( i, j ) , and W ( l ) ∈ R
d l × d l +1 is a layer-specific trainable weight matrix for l th layer.
d l and d l +1 indicates the node feature dimension for l th layer and ( l + 1) th respectively.
σ denotes a non-linear activation function for each layer such as Leaky
ReLu/ReLU except for the output layer where softmax is normally used for the classification.
Our goal is to represent the node representation by aggregating neighbour information with each edge features in a multi-stream manner.
Hence, we generalize the traditional GCN learning approach to perform multi-stream(MS) learning for the multi-dimensional edge enhanced graph.
The overall
MS learning procedure is in equation (10), for each node feature in H ( l ) ∈
R N × d l , we apply the multi-stream GCN learning f MS that formulates t streams of traditional GCN learning in equation (9) through the t dimensions of the connected edge, resulting in the multi-stream hidden feature
H ( l +1) t ∈ R N × d ( l +1) ms at ( l + 1) th layer.
Here t ∈ { 1 , 2 , ..., T } and d ( l +1) ms is the multi-stream feature size for each edge dimension at this layer.
Then a multi-stream aggregation function ϕ MS is applied over the t streams, producing the feature matrix H ( l +1)
∈ R N × d ( l +1) that contains the aggregated feature for each node in N .
Here we use concatenation function as ϕ MS for the hidden layer in the multi-stream aggregation, leading us to have d l +
=
t ∗ d ( l +1) ms .
Specifically, for the output layer, pooling method is used instead and the details are provided in later paragraph.
Accordingly, the updated propagation rule is provided in equation (11).
Unlike the original GCN propagation in equation (9), we have T streams of GCN learning in each layer, sharing the same input H ( l ) and propagating based on the T adjacency matrices ME ( t ) , which involves a set of layer and stream specific trainable weight matrices denoted as W ( l )( t ) .
We also tried the shared-stream learning that shares the trainable weight matrices across each stream but found that separate stream-specific trainable weight matrices have better performance.
The comparison of the two learning mechanisms is provided in Appendix Section B.
[Equation]
[Table] [Equation]
Unlike the hidden layers where we use concatenation to aggregate the node features over each stream to continue propagation to next layer, we instead apply the pooling at the output layer to further synthesize the multi-stream features of each node to do the final classification. Equation (12) formulizes max pooling , in which H ( l O ) t ∈ R N × d lOms , t ∈ { 1 , 2 , ..., T } denotes the T streams of node features for N nodes at the output layer l O , and here d l O ms is the node feature dimension that equals to the classification label number C .
Through max pooling , we select the best valued features over the T streams for each node in N before the final classification. We also tried other pooling and provide the comparison in Appendix Section B.
[Equation]
E VALUATION S ETUP
We evaluate our ME-GCN on text classification in semi-supervised settings, and examine the effectiveness of corpus-based multi-dimensional edge features.
Baselines: We compare ME-GCN with state-of-the-art semi-supervised text classification models, which do not use any external resources.
Additionally, we also include four baseline models, which use pretrained embedding or language model: CNN-Pretrained, LSTM-Pretrained, BERT, and TMix.
1)TF-IDF+LR, 2)TF-IDF+SVM : Term frequency inverse document frequency for feature engineering with Logistic Regression or SVM with rbf kernel.
3)CNN-Rand, 4)-Pretrained : Text-CNN (Kim, 2014) is used as the classifier.
Both CNN-Rand using random initialized word embedding and CNN-Pretrained using pretrained word embedding are evaluated.
We used English
Glove-pretrained (Pennington et al., 2014) and Chinese Word Vectors (Li et al., 2018) for Chinese dataset-zh.
5)LSTM-Rand, 6)-Pretrained :
We apply the same set-up as the CNN, but with Long
Short-Term Memory (LSTM).
7)TextGCN : We follow the same hyperparameters of the TextGCN (Yao et al., 2019).
8)BERT :
We use huggingface(Wolf et al., 2020) BERT BASE (Devlin et al., 2019) in our experiments (‘bert-base-chinese’ model is used for Chinese).
9)TMix :
TMix(Chen et al., 2020) generates new training text data by interpolating over labelled text encoded using BERT hidden representation and train on the generated text data for text classification.
We use the default setting provided.
Dataset: We evaluated our experiments on five widely used text classification benchmark datasets (Yao et al., 2019), 20NG, R8, R52, MR and Ohsumed, and three additional semi-supervised text classification datasets (Hu et al., 2019), Agnews, Twitter nltk and Waimai.
All the data is split based on the extreme low resource text classification enviornment- 1% training and 99% test set.
The summary statistics of the datasets can be found in Appendix Table 3.
For the data sample selection, we randomly select them but the class distribution is followed by the original datasets.
1)20NG is a 20-class news classification dataset and we select 3,000 samples from the original dataset. 2)R8 , 3)R52 are from Reuters which is a topic classification dataset with 8 classes and 52 classes.
3, samples from each dataset are selected.
4)MR (Pang & Lee, 2005) is a binary classification dataset about movie comments and we use all samples from the dataset.
5)Ohsumed is a medical dataset with 23 classes, and we select 3,000 samples from the original dataset.
6)Agnews (Zhang et al.,
[Figure]
2015) is a 4-class news classification dataset and 6,000 samples are selected.
7)Twitter nltk is a binary classification sentiment analysis from Twitter, we sampled 1,500 positive and 1,500 test samples from the original dataset.
8)Waimai is a binary sentiment analysis dataset about food delivery service comments from a Chinese online food ordering platform.
The dataset is pre-tokenized.
R ESULTS A NALYSIS 5. P ERFORMANCE E VALUATION
Table 1 presents a comprehensive performance experiment, conducted on the benchmark datasets.
The most bottom row shows the accuracy from our best models using either max or average pooling.
Overall, our proposed model significantly outperforms the baseline models on all eight datasets, demonstrating the effectiveness of our ME-GCN on semi-supervised text classification for various length of text.
With in-depth analysis, CNN/LSTM-Rand is quite low in performance on several datasets but increases significantly when using pretrained embeddings.
While TextGCN achieves better accuracy than above baselines on most datasets, the performance is all lower than ME-GCN.
This shows the efficiency of preserving rich information using multi-dimensional edge features.
The merit of pre-training stands out with BERT and TMix, producing better accuracy than the baseline
TextGCN on most datasets.
Especially, BERT achieves the best and second best performance on
MR and Waimai, which are short-text sentiment analysis datasets.
This would be because of the two aspects of sentiment classification: (1) compared to topic-specific text classification, sentiment analysis task may benefit from the pretrained general semantics learned from a large external text; (2) word order matters for sentiment analysis, which could be missing in GNNs.
Nevertheless, our
ME-GCN, with no external resources, still outperforms those pertrained models in seven datasets, illustrating the potential superiority of self-exploration on the corpus via multi-dimensional edge graph in comparison of pretraining on large external resource.
5. I MPACT OF E DGE F EATURE D IMENSION
To evaluate the effect of the dimension size of the edge features, we tested ME-GCN with different dimensions.
Figure 2 shows the test accuracy of our ME-GCN model on the four dataset, including
R8, R52, MR, Waimai(zh).
The bottom right corner for each subgraph includes the average number of the words per document.
We noted that the test accuracy is related to the average number of words per document in the corpus.
For instance, for ‘MR’ (avg #: 18.4), test accuracy first increases with the increase of the size of edge feature dimensions, reaching the highest value at 10; it falls when its dimension is higher than 15.
However, for R8 and R52 (avg #: 84.2 and 104.5), got the highest value at 20 or 25.
This is consistent with the intuition that the average number of words per document in the corpus should align with the dimension size of the edge features in ME-GCN.
The trend is different in waimai dataset as it is Chinese, this is because different languages would have different nature of choosing the efficient edge feature dimension.
Moreover, in order to analyse the impact of the edge feature dimension, we present an illustrative visualisation of the document embeddings learned by ME-GCN.
We use the t-SNE tool (Van der
Maaten & Hinton, 2008) in order to visualise the learned document embeddings.
Figure 3a and
Figure 3b shows the visualisation of test set document embeddings in AgNews learned by ME-GCN
[Figure]
[Table]
[-Instead of dim=
-]
5. I MPACT OF R ATIO OF L ABELLED D OCS
We choose 3 representative methods with the best performance from Table 1: CNN-Pretrained,
TextGCN and our ME-GCN, in order to study the impact of the number of labelled documents.
Particularly, we vary the ratio of labelled documents and compare their performance on the two datasets, Twitter nltk and R52, that have the smallest number and largest number of classes.
Figure 3c and Figure 3d reports test accuracies with 1%, 10%, and 33% of the R52 and Twitter nltk training set.
We note that our ME-GCN outperforms all other methods consistently.
For instance, MEGCN achieves a test accuracy of 0.8232 on Twitter nltk with only 1% training documents and a test accuracy of 0.8552 on R52 with only 10% training documents which are higher than other models with even the 33% training documents.
It demonstrates that our method can more effectively take advantage of the limited labelled data for text classification.
5. C OMPARISON OF E MBEDDING V ARIANTS
ME-GCN apply a Word2Vec CBOW in order to train the word node embedding and the related multi-dimensional edge feature.
We compare our model with three different word embedding techniques, Word2Vec, fastText, and Glove in Table 2.
We noted that using Word2Vec and Glove, word-based models, is comparatively higher than applying the fastText, a character n-gram-based model.
This would be affected because the node and edge of ME-GCN are based on words, not characters.
C ONCLUSION
We introduced ME-GCN (Multi-dimensional Edge-enhanced Graph Convolutional Networks) for semi-supervised text classification, which takes full advantage of both limited labelled and large unlabelled data by rich node and edge information propagation.
We propose corpus-trained multidimensional edge features to efficiently handle the distance/closeness between words and documents as multi-dimensional edge features, and all graph components are based on the given text corpus only.
ME-GCN demonstrates promising results by outperforming numerous state-of-the-arts on eight semi-supervised text classification datasets consistently.
In the future, it would be interesting to make this multi-aspect graph under inductive learning.
A S ETTINGS A. H YPERPARAMETER S ETTING
All documents are tokenized using NLTK tokenizer(Bird et al., 2009) and words occurring no more than 5 times have been excluded.
Both word2vec and Dec2vec are trained on the corpus we get using gensim package with window size = 5 and iter = 200 .
The initial feature dimension for node and document is set to d 0 = 25 , which is same to the multi-dimension number for edge features and multi-stream number T in ME-GCN learning.
Different multi-stream numbers are tested and discussed in Section 5.2.
The threshold u = 5 is used for document-document edge construction.
We use two-layers of multi-stream GCN learning with d l 1 ms = 25 (thus d l 1 = 625 ) for the first multistream GCN layer and d l O ms = C (no. of label in the datasets) for the output layer.
In the training process, following Liu et al.
(2020), we use dropout rate as 0.5 and learning rate as 0.002 with Adam optimizer.
The number of epochs is 2000 and 10% of the training set is used as the validation set for early stopping when there is no decreasing in validation set’s loss for 100 consecutive epochs. [Table]
A. H YPERPAREMETER S EARCH
For each dataset we use grid search to find the best set of hyperparameters and select the base model based on the average accuracy by running each model for 5 times.
The number of stream: 5,10,20,25,30,40,50.
The document edge threshold: 3,5,10,15.
The pooling method: max pooling, min pooling, average pooling.
The number of hyperparameter search trials is 72( = 6 ∗ 4 ∗ 3 ) for each dataset.
The best hyperparameters for each dataset and their average accuracy on test set shows in Table 4.
And the trend of validation performance is very similar to the testing performance trend.
[Table]
A. R UNNING D ETAILS
All the models are trained by using 16 Intel(R) Core(TM) i9-9900X CPU @
3.50GHz and NVIDIA
Titan RTX 24GB using Pytorch (Paszke et al., 2019).
The number of parameters for each part of the model is: Word Node (Word2vec): 2 UT , Document
Node (Doc2vec): 2 T ( U + K ) , ME-GCN Learning: T 2 d l 1 ms (1 + C ) .
The default value of d l 1 ms is 25.
The number of parameters of TextGCN is ( U + K ) ∗ D
+ D ∗ C and the default value of D is 200.
Comparison of the number of parameters between TextGCN and our ME-GCN shows in Figure 4. [Table] [Table]
B L EARNING AND P OOLING V ARIANT T ESTING
We compare ME-GCN with three different pooling approaches (max, average, and min pooling) and the result is shown in Table 5.
Most datasets produce better results when using max pooling [Table] and the result with max and average pooling outperforms that with min pooling.
This is very obvious because the min pooling captures the minimum value of each graph component.
We also compare two multi-stream graph learning methods, including separated and shared stream learning to examine the effectiveness of ME-GCN learning with multi-dimensional edge features.
Table 6 presents that the separated stream learners significantly outperforms the shared learners.
This shows it is much efficient to learn each dimensional stream with an individual learning unit and initially understand the local structure, instead of learning all global structures at once.
C L INKS R ELATED TO D ATASETS AND B ASELINE M ODELS
The links for Datasets:
• 20NG : http://qwone.com/ ∼ jason/20Newsgroups/ • R8 , R52 : https://www.cs.umb.edu/ ∼ smimarog/textmining/datasets/ • MR : http://www.cs.cornell.edu/people/pabo/movie-review-data/ • Ohsumed :
http://disi.unitn.it/moschitti/corpora.htm • Agnews : http://www.di.unipi.it/ ∼ gulli/AG corpus of news articles • Twitter nltk : http://nltk.org/howto/twitter.html • Waimai : https://github.com/SophonPlus/ChineseNlpCorpus/
The links for Baseline Models:
• TextCNN : https://github.com/DongjunLee/text-cnn-tensorflow • TextGCN : https://github.com/yao8839836/text gcn • BERT BASE : https://huggingface.co/bert-base-uncased
• Tmix : https://github.com/GT-SALT/MixText • Chinese BERT : https://huggingface.co/bert-base-chinese • GloVe-pretrained : https://nlp.stanford.edu/projects/glove/ • Chinese Word Vectors : https://github.com/Embedding/Chinese-Word-Vectors
The tokenizer used:
• English Tokenizer - NLTK : https://www.nltk.org/api/nltk.tokenize.html • Chinese Tokenizer - Jieba : https://github.com/fxsjy/jieba