HYPHEN : Hyperbolic H awkes Attention For Text Streams (2022)

Abstract: Analyzing the temporal sequence of texts from sources such as social media, news, and parliamentary debates is a challenging problem as it exhibits time-varying scale-free properties and fine-grained timing irregularities. We propose a Hyperbolic Hawkes Attention Network (HYPHEN), which learns a data-driven hyperbolic space and models irregular powerlaw excitations using a hyperbolic Hawkes process. Through quantitative and exploratory experiments over financial NLP, suicide ideation detection, and political debate analysis we demonstrate HYPHEN’s practical applicability for modeling online text sequences in a geometry agnostic manner.

## introduction text stream modeling is a critical problem that helps analyze trends over a variety of applications spanning finance @xcite , healthcare @xcite , and political discourses @xcite . however, analyzing such text sequences poses several challenges. first, modeling individual text items may not be informative enough since text sequences display a sequential context dependency, where analyzing them together in succession provides better contextual representation @xcite . second, timing plays an essential role in online stream modeling as users quickly react to new information @xcite . for instance, in stock markets, reacting a second slower than other investors can lead to massive losses @xcite . a fundamental limitation in existing rnn methods is that it ignores the natural fine-grained timing irregularities in streams @xcite . social theories show that from a vast volume of texts in a stream, only a few are powerful enough to heavily influence the overall trend @xcite . such texts are rare and the * equal contribution. excitation induced by them follows a powerlaw distribution which gives rise to scale-free properties @xcite . for example, in political debates, there are a few rare highly-influential debates that heavily impact the overall voting decisions of citizens @xcite . further, the impact of such powerlaw excitations varies for each event. the presence of varying powerlaw dynamics from highly influential texts correlates with natural hierarchies and scale-free dynamics in text streams, making them difficult to model @xcite . the good news is that hyperbolic learning has shown to better model such powerlaw dynamics compared to euclidean learning over domains, including vision @xcite and nlp @xcite . however, existing works face two major limitations, 1) they ignore the timing irregularities in scale-free sequences and 2) they use a single hyperbolic space to encode varying levels of hyperbolic dynamics. building on social theories, our contributions can be summarized as: problem formulation: for a sequence of texts [p 1 . . . , p n ] released at times [t 1 , . . . , t n ] sequentially, with [t 1 < • • • < t n ], our target is to model this sequence in a time-sensitive fashion for a variety of downstream applications ( §3). ## applications and tasks political stance prediction parliamentary debates consist of responses from politicians over a motion. following @xcite , we aim to classify the stance of a speaker as 'aye'/'no' on a motion based on their historic speeches. we evaluate on the parlvote dataset (abercrombie and batista-navarro, 2020) comprising of 33,461 uk debate transcripts of 1,346 politicians. financial nlp we aim to predict future stock trends based on the historic texts about a stock. following @xcite we regress the future volatility of a stock defined as λ = ln(| p i -p i-1 p i-1 |), where p i is the closing price. we evaluate on the s&p @xcite containing 88 stocks with 109,915 tweets and the china stock exchange (cse) @xcite containing 90,361 chinese news articles for 85 stocks. suicide ideation following @xcite , we aim to detect suicidal intent in a tweet given historic tweets from a user. we use the data from @xcite containing 32,558 user timelines and 2.3m texts. ## conclusion we explore the scale-free dynamics and timing irregularities of text streams. we propose hyphen which uses hyperbolic hawkes attention and learns data-driven geometries to represent varying hyperbolic properties of streams. through experiments on political, financial nlp, and healthcare tasks, we show the applicability of hyphen on 4 datasets.