Detecting Trending Terms in Cybersecurity Forum Discussions (2020)

Abstract: We present a lightweight method for identifying currently trending terms in relation to a known prior of terms, using a weighted log-odds ratio with an informative prior. We apply this method to a dataset of posts from an English-language underground hacking forum, spanning over ten years of activity, with posts containing misspellings, orthographic variation, acronyms, and slang. Our statistical approach supports analysis of linguistic change and discussion topics over time, without a requirement to train a topic model for each time interval for analysis. We evaluate the approach by comparing the results to TF-IDF using the discounted cumulative gain metric with human annotations, finding our method outperforms TF-IDF on information retrieval.

## introduction underground hacking forums contain a large collection of noisy text data around various topics, with misspellings, changing lexicons, and slang phrases. the evolving domain-specific lexicon includes homonyms, where "rat" may be identified as an animal by off-the-shelf tools, but is typically defined as a "remote access trojan" in this context, a type of malware used to gain access to a victim's computer. we work with texts from the hackforums site foot_0 , the largest english-language hacking forum, with multiple bulletin boards arranged around various topics, and many active users submitting thousands of new posts every day. the dataset contains over a decade of text data, but detecting trends is nontrivial, due to the informal language used by members, not only technical terms such as "rat", but also misspellings, slang, orthographic variation, and acronyms. for instance, the following texts demonstrate how posts are structured into threads on given top-ics, and how users both deliberately and accidentally use noisy language foot_1 : user1 ransomware infects hospitals all over uk: link user2 anyone think they made some money from this? user1 they might of done but idk they'll get caught eventually, it's stupid to commit crimes like this user3 who tf targets hospitals for ransomeware user1 i dont believe they actually went for the nhs.. the ransom would be more $$$ lol user4 i looked up a few btc addresses and can confirm they made money researchers interested in analysing hot topics on the forum will find it hard to gain a clear perspective on this due to the volume of data going through the forum every day. therefore, an overview of trending topics with natural language processing and statistical techniques is useful for identifying what may be of interest to security researchers. we propose a tool to identify tokens from trending topics, by pre-tokenising post data, followed by adapting a statistical technique for measuring changes, which can be used to scan across the dataset. the tool builds upon a weighted log-odds ratio @xcite with an informative bayesian prior @xcite , used to compare differences in two corpora. in our case the corpora represents two distinct time periods of interest within the same subforum 3 . for known events, one period can be a set of texts preceding the event (the prior) and the other period can be texts following the event (the target). also, the tool can be used for live listings of trending terms in the present day, by comparing new posts against some fixed prior. our method identifies the relative importance of tokens to each time period. the log-odds ratio indicates whether terms are more likely to appear in a given corpus over others. a log-odds score is higher for terms that are both unique and more frequent to a given period. other nlp methods require the removal of pre-defined stopwords. however, for our approach, as stopwords have a similar distribution across both time periods, they will have a low log-odds score and rank. the tool looks at "bursty" events: for a token to be trending, frequency of the token should be significantly different between the prior and target periods, and be more frequent than other terms in the target period. for identifying topics, our method uses a feature-pivot approach (a topic is a cluster of keywords) over a document-pivot approach (a topic is a cluster of documents). the latter may struggle with documents about multiple topics, whereas the former may incorrectly identify correlations between words as topics. a major challenge in developing the tool is that it is to be used on a large dataset of noisy data, for exploring the evolution of underground hacking forums. we take mitigating steps such as storing the pre-tokenised and part-of-speech tagged text, to decrease computation time for longitudinal analysis. while we focus on a cybercrime context, we note this type of data has similarities to twitter data: short posts, and informal language. however, while twitter data has some minimal inter-tweet connections through hashtags, quoting comments and replies, forums have a rigid discussion-based structure set by the forum administrators. our contributions are: ## evaluation we evaluate the results of the tool by carrying out an information retrieval task with human annotators. we compare the log-odds approach with tf-idf using discounted cumulative gain and the human annotations as a ground-truth ranking of identified terms. we use both a known cybersecurity event to define our target window, as well as a randomly-selected target window. ## discussion detecting trending topics on noisy social media data is not a new problem for information retrieval and nlp. however, we believe our application of an existing statistical method onto a longitudinal dataset provides a novel lightweight approach to detecting trending terms, which returns terms of more relevance than tf-idf, and remains computationally less expensive than topic modelling such as lda. this work provided an initial step towards detecting temporal linguistic changes over time, by preprocessing text data, followed by using a bayesian approach with a moving prior and target window depending on whether a user is observing short or long term trends. while our method does not identify the relevant windows itself, the tool can be combined with trending topic detection techniques to identify lexically distinct events, where some terms may re-trend. having shown that the statistical model is strong, and using a bayesian approach can support new and evolving slang in the dataset without fine tuning a language model, we recognise that there are ways to further improve the nlp of cybersecurity forum texts. for instance, we can improve preprocessing in order to better deal with noisy texts: this includes the detection of misspellings, orthographic variation, acronyms and abbreviation, and deliberate obfuscation such as leetspeak. in addition, we can start to incorporate the detection of multiword expressions and named entity recognition techniques for noisy language, since both are likely to be of interest to researchers analysing language use in cybersecurity forums. in future work we aim to increase understanding of the evolution of forums, changing language over time, and the changing topics of discussion by forum members. we also aim to automatically detect and extract events in the crimebb dataset. although we have focused on analysing forum data, the tool can be used to explore trends in other cor-pora. in future work, we plan to use this approach to analyse how spam emails have changed following the covid-19 pandemic. ## conclusion in this work, we presented a new use-case for the log-odds tool introduced by @xcite and implemented in the tidylo r library by @xcite , for detecting trending terms in longitudinal historical noisy text data of an underground hacking forum. the tool can be used for both detecting short term and long term trends depending on the time windowing and separation of windows selected. using annotations of salient terms during both discussion of wannacry, and a randomly chosen duration, we found our approach to produce more relevant salient terms over tf-idf.