Public Wisdom Matters ! Discourse-Aware Hyperbolic Fourier Co-Attention for Social-Text Classification
Abstract
Social media has become the fulcrum of all forms of communication.
Classifying social texts such as fake news, rumour, sarcasm, etc. has gained significant attention.
The surface-level signals expressed by a social-text itself may not be adequate for such tasks; therefore, recent methods attempted to incorporate other intrinsic signals such as user behavior and the underlying graph structure.
Oftentimes, the ‘public wisdom’ expressed through the comments/replies to a social-text acts as a surrogate of crowd-sourced view and may provide us with complementary signals.
State-of-the-art methods on social-text classification tend to ignore such a rich hierarchical signal.
Here, we propose Hyphen , a discourse-aware hyperbolic spectral co-attention network.
Hyphen is a fusion of hyperbolic graph representation learning with a novel Fourier co-attention mechanism in an attempt to generalise the social-text classification tasks by incorporating public discourse .
We parse public discourse as an Abstract Meaning Representation (AMR) graph and use the powerful hyperbolic geometric representation to model graphs with hierarchical structure.
Finally, we equip it with a novel Fourier co-attention mechanism to capture the correlation between the source post and public discourse.
Extensive experiments on four different social-text classification tasks, namely detecting fake news, hate speech, rumour, and sarcasm, show that Hyphen generalises well, and achieves state-of-the-art results on ten benchmark datasets.
We also employ a sentence-level fact-checked and annotated dataset to evaluate how Hyphen is capable of producing explanations as analogous evidence to the final prediction.
Code is available at: https://github.com/LCS2-
IIITD/Hyphen .
Introduction
Social media has become a significant source of communication and information sharing.
Mining texts shared on social media ( aka social-texts) are indispensable for multiple tasks – online offence detection, sarcasm identification, sentiment analysis, fake news detection, etc.
Despite the proliferation of research in social computing, there is a gap in capturing the heterogeneous signals beyond the standalone source text processing.
Predictive models incorporating signals such as user profiles [1, 2, 3, 4], underlying user interaction networks [5, 6, 7, 8, 9] and metadata information [10, 11, 12, 13], are far and few in between.
These heterogeneous signals are challenging to obtain and may not always be available on different platforms (e.g., Reddit does not provide explicit user interaction network; YouTube does not release user activities publicly).
On the other hand, comment threads following a source post are an equally rich source of heterogeneous signals, which are easier to obtain and uniformly available across social media platforms and forums.
We hypothesise that such public discourse carries complementary and rich latent signals (public wisdom, worldly knowledge, fact busting, opinions, emotions, etc.), which would otherwise be difficult to obtain from just standalone source-post analysis.
Therefore, public discourse can be used in unison with the source posts to enhance social-text classification tasks.
Figure 1 hints towards the motivation behind using public discourse as an implicit proxy for social-text classification.
In this work, we propose Hyphen , a discourse-aware hyperbolic spectral co-attention network that amalgamates the source post and its corresponding public discourse through a novel framework to perform generalised social-text classification.
We parse individual comments on a source post as separate Abstract Meaning Representation (AMR) graphs [14], and merge them into one macro-AMR, representing mass perception and public wisdom about the source post.
The AMR representation inherently abstracts away from syntactic representations, i.e., sentences which are similar in meaning are assigned similar AMRs, even if they are not identically worded [15].
The resultant macro-AMR graph represents the semantic information in a rich hierarchical structure [16, 17].
Hyphen aims to effectively utilise the hierarchical properties of the macro-AMR graph by using the hyperbolic manifold [18] for representation learning.
In order to fuse the source post with the public discourse, we propose a novel Fourier co-attention mechanism on the hyperbolic space.
It computes pair-wise attention between user comments and the source post, thereby capturing the correlation between them.
On a typical social media post, several users express their opinions, and there are several messages being conveyed by the source post itself, some of which are more relevant and/or common than the others.
We use a novel discrete
Fourier transform based [19] sublayer to filter the most-common user opinions expressed in the macro-AMR and most prominent messages being conveyed by the source post.
Fourier transform is essentially a measurement of energy (i.e., strength of prevalence) of a particular frequency within a signal.
We can extend this notion to quantify how dominant a particular frequency is within a signal.
Building on this, we hypothesise that the time-domain signal isomorphically represents various user comments on the source post, and the Fourier transform over the comment representations yields the most-commonly occurring user frequencies (stance, opinions, interpretation, wisdom, etc.).
Similarly, the Fourier transform over the sentence-level representations of the source post renders the most intense messages and facts being conveyed by it.
We perform extensive experiments with Hyphen on four social-text classification tasks – detecting fake news, hate speech, rumour, and sarcasm, on ten benchmark datasets.
Hyphen achieves state-ofthe-art results across all datasets when compared with a suite of generic and data-specific baselines.
Further, to evaluate the efficacy of hyperbolic manifold and Fourier co-attention in Hyphen , we perform extensive ablation studies, which provide empirical justification behind the superiority of
Hyphen .
Finally, we show how Hyphen excels in producing explainability.
Related Work
Generic social-text classification.
There have been some attempts to arrive at a general architecture for social-text classification.
Bi-RNODE [20] proposes to use recurrent neural ordinary differential equations by considering the time of posting.
CBS-L [21] considers transformation of document representation from the traditional n -gram feature space to a center-based similarity (CBS) space to solve the issue of co-variate shift.
Pre-trained Transformer-based models like RoBERTa-base [22],
BERTweet [23], ClinicalBioBERT [24], etc. also deliver benchmark results on generic social-text classification [25].
FNet [26] proves to be competent at modeling semantic relationships by replacing the self-attention layer in a Transformer encoder with a standard, non-parametric Fourier transform.
Use of public discourse in social-text classification.
Multiple approaches have been proposed to use public discourse as an attribute for classifying the social media posts.
TCNN-URG [27] utilises a CNN-based network to encode the content, and a variational autoencoder for modeling user comments in fake news detection.
CSI [28] is a hybrid deep learning model that utilizes subtle clues from text, user responses, and the source post, while modeling the source post representation using an LSTM-based network.
Zubiaga et al.
[29] use public discourse for rumour stance detection using sequential classifiers.
Lee et al.
[30] propose sentence-level distributed representation for the source post guided by the conversational structure.
CASCADE [31] and CUE-CNN [32] use stylometric and personality traits of users in unison with the discussion threads to learn contextual representations for sarcasm detection.
dEFEND [33] and GCAN [9] propose to use co-attention over user comments and other social media attributes for detecting fake news and other social texts.
The performance of most of these models deteriorate when extended to multiple tasks and fail to filter out the least relevant parts of their respective input modalities.
Moreover, they operate on the Euclidean manifold, and therefore, overlook the representation strength of hyperbolic geometry in modeling hierarchical structures.
Hyphen overcomes these limitations of the existing methods.
Hyperbolic representation learning.
Hyperbolic representation learning has gained significant attention in tasks in which the data inherently exhibits a hierarchical structure.
HGCN [34] and HAT [35] achieve state-of-the-art results in graph classification owing to their powerful representation ability to model graphs with hierarchical structure.
Unlike these two, H2H-GCN [36] directly works on the hyperbolic manifold to keep global hyperbolic structure, instead of relying on the tangent space.
Furthermore, the recent GIL model [37] captures more informative internal structural features with low dimensions while maintaining conformal invariance of both Euclidean and hyperbolic spaces.
However, for social-text classification, none of the above approaches simultaneously consider the source- and discourse-guided representations.
We build on this limitation and use public comments in unison with the source post to further contextualise and improve a social-text classifier.
Background
Hyperbolic geometry.
A Riemannian manifold ( M , g ) of dimension n is a real and smooth manifold equipped with an inner product on tangent space
g x : T x M × T x M
→ R at each point x ∈ M , where the tangent space T
x M is an n -dimensional vector space and can be seen as a first-order local approximation of M around point x .
In particular, hyperbolic space ( H nc , g c ) , a constant negative curvature Riemannian manifold, is defined by the manifold H nc = { x ∈ R n : c ∥ x ∥ < 1 } equipped with the following Riemannian metric: g c x = λ 2 x g E , where λ x = 2 1 − c ∥ x ∥ 2 , and g E =
I n is the
Euclidean metric tensor.
The connections between hyperbolic space and tangent space are established by the exponential map exp c x : T x H nc
→ H nc , and the logarithmic map log c x : H nc → T x H nc , as follows,
[Equation]
where x , y ∈ H nc , v ∈ T x H nc , and ⊕ c represents M ¨ obius addition as follows,
[Equation]
Further, the generalization for multiplication in hyperbolic space can be defined by the M ¨ o bius matrix-vector multiplication between vector x ∈ H nc \ { 0 } and matrix M ∈ R m × n as shown below,
[Equation]
Hyperbolic space has been studied in differential geometry under five isometric models [18].
This work mostly confines to the Poincaré ball model.
It is a compact representation of the hyperbolic space and has the principled generalizations of basic operations (e.g., addition, multiplication).
We use P , E in the superscript, to denote the Poincaré and Euclidean manifolds, respectively.
We provide more insights into these models in Appendix A.1.
Discrete Fourier Transform.
The Fourier transform decomposes a function into its constituent frequencies.
Given a sequence { x n } with n ∈ [0 , N − 1] , the Discrete Fourier Transform (DFT) is defined as, X k = (cid:80) N − 1 n =0 x n e − 2 πiN nk , where 0 ≤ k ≤ N − 1 .
For each k , the DFT generates a new representation X k as a sum of the original input tokens x n , with the twiddle factors [38, 19, 39].
Architecture of Hyphen
In this section, we lay out the structural details of Hyphen (see Figure 2 for the schematic diagram).
We propose individual pipelines for learning representations of the source post and the user comments on the hyperbolic space.
We then combine both the representations using a novel hyperbolic Fourier co-attention mechanism that helps in simultaneously attending to both the representations.
Lastly, we pass it to a feed-forward network for the final classification.
Without loss of generality, we denote the
Poincaré ball model ( P ) as H nc (hyperbolic space) throughout the paper.
4.1 Encoding public discourse
In this section, we discuss the pipeline for encoding the public discourse.
We parse the user comments into an AMR (Abstract Meaning Representation)
[14] graph.
The individual comment-level AMR graphs are merged to form a macro-AMR (discussed below), representing the global public wisdom and latent frequencies in the discourse.
Next, we learn representations of the macro-AMR using a HGCN (Hyperbolic Graph Convolutional Network) [40].
This yields a representation for public discourse containing rich latent signals.
Macro-AMR graph creation .
Considering a social media post containing several user comments
C =
[ c 1 , c 2 , ..., c m ] , we obtain an AMR (Abstract Meaning Representation) [14] graph for each user comment.
We merge all the comment-level AMR graphs into one macro-AMR (post-level) while preserving the structural context of the subgraphs (comment-level).
Figure 2(a) contains the schematic for an example macro-AMR graph.
In particular, we adopt three strategies – (a)
Add a global dummy-node : We add a dummy node and connect it to all the root nodes of the comment-level AMRs, and add a comment tag :COMMENT to the edges.
The dummy node ensures that all the AMRs are connected, so that information can be exchanged during graph encoding.
(b) Concept merging : Since we consider comments made on a particular post, these comments will essentially discuss the same topic.
Therefore, multiple user comments can have identical mentions, resulting in repeated concept nodes in the comment-level AMRs.
We identify such repeated concepts, and add an edge with label :
SAME starting from earlier nodes to later nodes (here later and earlier refer to the temporal order of the ongoing conversation on a social media post).
(c) Intercomment co-reference resolution : A major challenge for conversational understanding is posed by pronouns, which occur quite frequently in such social media comments.
We conduct co-reference resolution on the comment-level AMRs to identify co-reference clusters containing concept nodes that refer to the same entity.
We add edges labeled with the label :COREF between them, starting from earlier nodes to later nodes in a co-reference cluster to indicate their relation.
Such types of connections can further enhance cross-comments information exchange.
This step results in a post-level AMR graph G amr =
[ g 1 s , g 2 s , ..., g ms ] , representing relations between various subgraphs { g is =
( v is , e is ) | 1 ≤ i ≤ m } (different user comments will correspond to different subgraphs).
The merged-AMR presents a global view of the public wisdom and interpretations.
Hyperbolic graph encoder .
We adopt the Poincaré ball model of HGCN [40] to encode the post-level
AMR graph and form user comment representations.
Since different comments correspond to different
T r a n s po s e
Fourier Transform
E i n s t e i n
M i dpo i n t
A gg .
M ob i u s G
R U
H yp e r bo
li c
A c ti v a ti on
E u c li d ea n G R U
H yp e r bo li c G C N
M e r g e d
A
M R
Hyperbolic User Interpretation Encoder
Hyperbolic Fourier Co-Attention
Hyperbolic Candidate Post Encoder
AMR Parsing and Merging
Feed Forward :
SAME :SAME :COREF :COREF :COMMENT :COMMENT :COMMENT:COREF
Transpose subgraphs of the post-level AMR, we ultimately aggregate the node representations to form subgraph embeddings.
Each subgraph embedding represents how individual users interpret the source post (their opinion).
In this section, we summarize the graph encoder architecture.
Given a post-level AMR graph G amr
= ( V , E ) and the Euclidean input node features, denoted by ( x 0 , E ) ∈ R d g , where d g is the input embedding dimension for entities in the AMR graph, we first map the input from Euclidean to Hyperbolic space.
Therefore, we interpret x E as a point in the tangent space T
o H d g c and map it to
H d g c with x 0 , P = exp c 0 ( x 0 , E ) .
Our graph encoder then stacks multiple hyperbolic graph convolution layers to perform message passing (see Appendix A.2 for the background on HGCN).
Finally, we aggregate the hyperbolic node embeddings ( x L, P i ) i ∈V at the last layer to form subgraph (comments) embeddings as shown in Figure 2(d).
We take the mean of the node embeddings for nodes present in a subgraph to yield the aggregated subgraph embedding: g P j = exp c 0 ( (cid:80) ni ∈ v js log c 0 ( x L, P i ) / | v js | ) .
Here, g P j ∈ H 2 d g c , the operator | .
| represents the number of nodes present in subgraph v js , and L is the number of layers of HGCN.
Therefore, the output for this encoder is G P =
[ g 1 P , g 2 P , ..., g m P ] , where G P ∈ H 2 d g × m is the matrix containing the learned representations for user interpretations (comments).
4.2 Hyperbolic Candidate Post Encoder
Inspired by [41], we propose to learn the source post content representations through a hierarchical attention network in the hyperbolic space.
We know that not all sentences in a source post might contain relevant information.
We thus employ a hierarchical attention-based network to capture the relative importance of various sentences.
Consider the input embedding of the t th word appearing in the i th sentence as x it , in the candidate post.
We utilise a hyperbolic word-level encoder (see
Appendix A.3 for the background of Hyperbolic Hierarchical Attention Network (HHAN)) to learn s K w i , the representation of the i th sentence.
Now, similar to the word-level encoder, we utilize M ¨ o biusGRU units to encode each sentence in the source post.
We capture the sentence-level context to learn the sentence representation s P i from the sentence vector s P w i obtained from the word-level encoder.
Specifically, we use Poincaré ball model based M ¨ o bius -GRU to encode different sentences.
We obtain outputs from the M ¨ o bius -GRU as s P i =
[ −−−→ GRU mob ( s P w i ) , ←−−− GRU mob ( s P w i )] as shown in Figure 2(e).
Here, s P i is the final context-aware representation for the i th sentence in the source post in the hyperbolic space (Poincaré ball model), i.e., s P i ∈
H 2 d s c , where d s is the input embedding dimension for the words x it in the source post’s sentences.
This finally gives us S P =
[ s 1 P , s 2 P , ..., s n P ] , where S P ∈ H
2 d s × n c is the matrix containing the learned candidate post representations.
4.3 Hyperbolic Fourier Co-Attention
We hypothesise that the evidence for various social-text classification tasks can be unveiled by investigating how different parts of the post are interpreted by different users, and how they correlate to different user opinions.
Therefore, we develop a hyperbolic Fourier co-attention mechanism to model the mutual influence between the source social media post (i.e., S P =
[ s 1 P , s 2 P , ..., s n P ] ) and user comments (interpretation) embeddings (i.e., G P =
[ g 1 P , g 2 P , ..., g m P ] , where S P ∈ H 2 d s × n c and G P ∈
H 2 d g × m c ).
Co-Attention [42] enables the learning of pairwise attentions, i.e., learning to attend based on computing word-level affinity scores between two representations.
Once we have the public discourse (Section 4.1) and the social media text (Section 4.2) embeddings in the hyperbolic space, the next step is a Fourier sublayer, which applies a 2D DFT to its (sequence length, hidden dimension) embedding input – one 1D DFT along the sequence dimension, F seq , and one 1D DFT along the hidden dimension, F h :
[Equation]
The intuition behind taking the Fourier transform over the user interpretation embeddings can be thought of as an attempt to capture the most commonly occurring frequencies (public wisdom, worldly knowledge, fact busting, opinions, emotions, etc.) in the public discourse.
These frequencies signify how the source post is being received by most of the people.
Further, the Fourier transform over the source post embeddings hints towards the most prominent messages conveyed by the source post.
This is depicted in Figure 2(b).
Next, we compute a proximity matrix F E ∈ R m × n .
The affinity (proximity) matrix F E can be thought to transform the user-interpretation attention space to the candidate post attention space, and vice versa for its transpose F E⊤ .
It is computed as:
[Equation]
where W sg ∈ R 2 d s × 2 d g is a matrix of learnable parameters.
The operator ⊗ c is the M ¨ o bius
Multiplication operator (Equation 4), and ⊗ E is the simple euclidean matrix multiplication.
By treating the affinity matrix as a feature, we can learn to predict candidate post and user interpretation attention maps H s P ∈ H k × n c and H g P ∈
H k × m c , given by
[Equation]
where W s ∈ R k × 2 d s , W g ∈ R k × 2 d g are learnable parameters, k is the latent-dimension used in computing co-attention and ⊕ c is the M ¨ o bius Addition operator (Equation 3).
We can then generate the attention weights of source words and interaction users through the Softmax function:
[Equation]
where a s E ∈ R 1 × m and a g E ∈ R 1 × n are the vectors of attention probabilities for each sentence in the source story and each user comment, respectively.
w hs , w hg ∈ R 1 × k are learnable weights.
Eventually, we can generate the attention vectors of source sentences and user interpretation through weighted sum using the derived attention weights, given by
[Equation]
where ˆ s E ∈ R 1 × 2 d s and ˆ g E ∈ R 1 × 2 d g are the learned co-attention feature vectors that depict how sentences in the source post are correlated to the user interpretations.
Finally, we have a feed forward network which yields the final classification output as ˆ y =
FFN [ˆ s E , ˆ g E ] , where [ . ] is the concatenation operator.
Equipped with co-attention learning, our model is further capable of generating suitable explanations (Section 6) by looking into the co-attention weights between different frequencies of users and in the source post.
Experiments
Datasets.
We evaluate the performance of Hyphen on four different social-text classification tasks across ten datasets (c.f. Table 1) – (i) fake news detection (Politifact [43], Gossipcop [43], AntiVax [44]), (ii) hate speech detection (HASOC [45]), (iii) rumour detection (Pheme [46], Twitter15 [47],
Twitter16 [47], RumourEval [48]), and (iv) sarcasm detection (FigLang-Twitter[49], FigLang-Reddit [49]).
We augmented the datasets with public comments/replies to suite our experimental setting (see the Appendix B for details on dataset preparation).
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
FiglangTwitter
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
FiglangReddit
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Experimentation details
For both the hyperbolic encoders in Figures 2(c)- (d), we adopt the Poincaré model of the respective frameworks.
Due to limited machine precision, it is possible that the exp c 0 ( . ) and log c 0 ( . )
maps might sometimes return points that are not exactly located on the manifold.
To avoid this and to ensure that points remain on the manifold and tangent vectors remain on the right tangent space, we clamp the maximum norm to 1 − e − 14 .
For optimization on the hyperbolic space, we use Riemannian Adam from Geoopt [61].
To find the optimal k (latent dimension, see Equation 5) for hyperbolic co-attention, we run grid search over k = 50 , 80 , 128 , 256 , and finally use k = 128 .
For HGCN, we use two layers with curvatures K 1 = K 2 = -1.
We detail all other hyper-parameters in the Appendix B.
We run all experiments for 100 epochs with early stopping patience of 10 epochs, on a NVIDIA RTX A6000 GPU.
Curvature for our implementation .
Hyphen learns the hyperbolic representations for public discourse and source-post text simulataneously and applies a novel Fourier co-attention mechanism over the obtained embeddings.
However, to be able to do so, we need to ensure that the curvatures of the hyperbolic manifolds (in our case Poincaré ball model) are same (or a product space of both manifolds).
To ensure consistency across both the pipelines (public discourse encoder (Section 4.1) and source-post encoder (Section 4.2)), in Hyphen we take the constant negative curvature c = − 1 .
As addressed in the limitations (See Section 7), another promising approach for Hyphen could be to consider the product space of both the manifolds before applying the co-attention mechanism.
Baseline methods.
We compare Hyphen with two sets of baselines (c.f. Table 1) – (i) Generic neural baselines: We employ those models that are often used for social-text classification tasks and have
[Table] [Table]
Rec.
[Table] [Table]
[Table]
Hate Speech Detection
HASOC
F
[Table] [Table]
Pre.
[Table]
Rec.
[Table]
F
[Table] [Table]
[Table]
Twitter15 F
[Table]
Pre.
[Table]
Rec.
[Table]
Twitter16 F
[Table] [Table]
[Table] [Table]
RumourDetection
[Table]
SarcasmDetection
FigLangReddit
[Table]
Table been shown to perform comparatively.
We consider different variations of the Transformer model and those who use social context as an auxiliary signal for social-text classification (dEFEND [33]).
(ii)
Data-specific baselines :
We experimented with many data-specific and task-specific baselines and chose top three for every dataset based on the performance.
Since top three models are data-specific, we call them with generic names – (a) SOTA-1, (b) SOTA-2, and (c) SOTA-3, respectively.
Performance comparison.
Table 5 shows the performance comparison.
The content-based pretrained models, BERT and RoBERTa, outperform dEFEND which uses both the source content and user comments.
We observe that dEFEND performs better than all the data-specific baselines because of the sophisticated use of co-attention.
By incorporating public comments along with the social post, Hyphen shows significant 2 performance improvement over all the baselines.
We observe that while the performance improvement over baselines is significant ( ∼ 4% ; p < 0 .
005 ) on datasets like
Politifact, Gossipcop, and Twitter15, the performance improvement is not that significant ( p < 0 . 05 ) on AntiVax and FigLang (Reddit).
This is due to the fact that in the latter datasets, there are less number of comments available per the source posts (see Table 1).
On Politifact and Gossipcop,
Hyphen - hyperbolic has a performance gain of 3 . 9% and 3 .
8% , over the best baselines models,
RoBERTa and dEFEND, respectively.
Note that even when compared to the pre-trained Transformer architectures, Hpyhen shows decent improvement, while for the non-Transformer based baselines like
HAN , there is a performance gain of 11 .
2% even on the AntiVax dataset.
We explain the data-specific baselines, their modalities, and detailed analyses of their performance in the Appendix B.
Ablation study.
We perform ablations with two variants of our model, namely Hyphen-hyperbolic and Hyphen-euclidean , in which Hyperbolic and Euclidean represent the underlying manifold.
■
Effect of public wisdom.
When we remove user comments ( Hyphen w/o comments : we consider only source post, get rid of the co-attention block for this analysis as we have just one modality, and keep the Fourier transform layer to capture the latent messages in the candidate post), the performance degrades.
Table 3 shows that for Gossipcop and Politifact datasets, Hyphen - hyperbolic has a performance degradation of 7 . 23% and 7 . 4% , respectively.
Due to the presence of less number of comments per post in the AntiVax dataset, the performance degradation is not that significant (i.e., 1 .
05% , p < 0 . 1 ).
On some datasets like FigLang (Twitter), Pheme and Twitter15, Hyphenhyperbolic w/o comments records a significant performance degradation ( p < 0 .
001 ) of 8 .
7 . 4% and 6 .
24% , respectively.
Even Hyphen - euclidean w/o comments sees a fall in F1 score of 6 .
38% , 6 . 4% and 5 .
45% for Twitter15, Twitter16 and FigLang (Twitter), respectively.
Since this is a content-only pipeline, in many cases, the model is outperformed by pre-trained Transformer models.
■
Effect of hyperbolic space.
We evaluate Hyphen ’s performance by replacing the hyperbolic manifold with Euclidean.
We observe that in support of our initial hypothesis, Hyphen - hyperbolic outperforms Hyphen - euclidean (see Table 3).
The former records a considerable gain of 3 .
55% and 2 .
85% F1 score on Gossipcop and Politifact datasets, respectively, over the latter.
For the
AntiVax dataset, a smaller increment of 0 .
83% can be attributed to the less number of user comments available in the dataset.
Note that for the variant, Hyphen - hyperbolic w/o comments , there is a performance degradation as compared to Hyphen - euclidean on Gossipcop and Politifact.
This is intuitive as the sole advantage of hyperbolic space lies in capturing the inherent hierarchy of the macro-AMR graphs.
Therefore, in case of a content-only model, Hyphen - euclidean performs better.
On Pheme and Twitter15, the former achieves a significant F1 score gain ( p < 0 . 005 ) of 3 .
12% and 3 .
18% respectively.
Due to less number of user comments in RumourEval, FigLang (Twitter) and
Figlang (Reddit), the performance gain is less significant ( p < 0 . 05 ), i.e., 1 . 44% , 1 . 01% and 0 .
47% respectively.
It should be noted that this behaviour demonstrates the effectiveness of Hyphen in early detection .
Even with less number of user comments available, Hyphen achieves performance boost over the baselines, and thus can be extremely effective in tasks like detecting fake news, where early detection is of great significance.
■ Effect of Fourier transform layer.
Table 3 shows that including the Fourier transform layer to capture the most prominent user opinions about the source post and the most common (latent) messages conveyed by the source post, boosts the overall performance of Hyphen .
There is an improvement of 5 . 6% F1 score on Gossipcop and 1 .
74% on Politifact due to the Fourier layer in
Hyphen - hyperbolic .
Because of the less number of comments per post in AntiVax, there is a smaller increment of 0 .
87% F1 score.
Even for Hyphen - euclidean , there is an increase of 2 .
28% on Gossipcop and ∼ 1 % on Politifact.
Hyphen - hyperbolic shows a significant improvement ( p < 0 .
001 ) of 4 .
34% , and 4 .
13% F1 score on FigLang (Twitter), Pheme and Twitter15, respectively.
For Hyphen - euclidean , there is an increase of 5 .
10% on FigLang (Twitter) and 3 .
53% on FigLang (Reddit).
Hyphen - euclidean and Hyphen - hyperbolic record an average increase of 4 .
49% and 4 .
34% in F1 score, respectively, over all datasets.
On applying co-attention over the outputs of Fourier transform layer, we are able to attend better to both the representations simultaneously, and thus the model’s ability to capture the correlation between the two increases.
Explainability
Here, we demonstrate how Hyphen excels at providing explanations for social-text classification tasks.
Using the hyperbolic co-attention weights a s E (Equation 8), we can provide an implicit rank list of sentences present in the source post in the order of their relevance to the final prediction.
For instance, consider the scenario of fake news detection.
A fake news is often created by manipulating selected parts of a true information.
The generated rank list of sentences in this case would correspond to the sentences in the news article, which are possible misinformation.
Furthermore, manual verification of all sentences in a news article is tedious, and therefore, a rank list based on the level of check-worthiness of sentences is convenient.
[Table]
To evaluate the performance of Hyphen in generating explanations, we consider Politifact, and for each source post, we manually annotate sentences present in the source post based on their relevance to the final level (fake/real) (see Appendix C for dataset annotation details).
The annotators also rank sentences of a source post in the order of their check-worthiness.
We expect our model to produce a similar list of sentences for the source post.
We use dEFEND [33] as a baseline for comparing the rank correlations, and evaluate the rank list produced by Hyphen against this ground-truth annotated rank list using Kendall’s τ and
Spearman’s ρ rank correlation coefficients.
dEFEND is the only model among the chosen baselines, which produces a similar rank link using attention weights in an attempt to provide explanations (See Appendix C for sample rank lists generated by dEFEND and Hyphen ).
Table 4 shows that the explanations produced by dEFEND have almost no correlation ( τ = 0 .
0231 , ρ = 0 .
to the annotated rank list.
On the contrary, Hyphen - hyperbolic shows a high positive correlation
[Table]
[Table] [Table]
[Table] [Table]
( τ = 0 . 4983 , ρ = 0 . 5532) .
Hyphen - euclidean also shows comparable performance.
The present the efficacy of Hyphen in providing decent explanations for social-text classification.
Model augmentation for explainability.
To provide explanations, we rule out the Fourier sub-layer from Hyphen .
This is done because on taking the Fourier transform of source-post and comments’ representations (Equation 5), we cannot assert an ordered mapping from the spectral domain to the sentence representations.
Such an order is necessary for us to have a mapping between the co-attention weights and the source-post sentences they were derived from.
Without such a mapping,
Hyphen would not be able to generate a rank-list based on the sentences in the source-post.
Conclusion
Public wisdom on social media carries diverse latent signals which can be used in unison with the source post to enhance the social-text classification tasks.
Our proposed Hyphen model uses a novel hyperbolic Fourier co-attention network to amalgamate both these information.
Apart from the state-of-the-art performance in social-text classification, Hyphen shows the potential of generating suitable explanations to support the final prediction and works well in a generalised discourse-aware setting.
In the future, mixed-curvature learning in product spaces [62] and hyperbolic-to-hyperbolic [36] representations could be employed to boost the learning capabilities of Hyphen .
Limitations .
Hyphen resorts to using tangent spaces for computing Fourier co-attention which is inferior because tangent spaces are only a local approximation of the manifold.
One may further incorporate other signals such as user interaction network and user credibility into the model.
Acknowledgment
T. Chakraborty would like to acknowledge the support of the LinkedIn faculty research grant.
We would like to acknowledge the support of the data annotators - Arnav Goel, Samridh Girdhar, Abhijay
Singh, and Siddharth Rajput, for their help in annotating the Politifact dataset.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] See Section(c)
Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We release the Code and Data used as a part of the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See experimentation details in Section(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] See Section(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See experimentation details in Section4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]