Abstract. Despite several benefits to modern communities and businesses, Twitter has attracted many spammers overwhelming legitimate users with unwanted and disruptive advertising and fake information. Detecting spammers is always challenging because there is a huge volume of data that needs to be analyzed while at the mean time spammers continue learning and changing their ways to avoid being detected by anti-spammer systems. Several spam classification systems are proposed using various features extracted from the content and user's information from their Tweets. Nevertheless, no comprehensive study has been done to compare and evaluate the effectiveness and efficiency of these systems. It is not known what the best anti-spammer system is and why. This paper proposes an evaluation framework that allows researchers, developers, and practitioners to access existing user-based and content-based features, implement their own features, and evaluate the performance of their systems against other systems. Our framework helps identify the most effective and efficient spammer detection features, evaluate the impact of using different numbers of recent tweets, and therefore obtaining a faster and more accurate classifier model. Keywords: Spam detection; Evaluation workbench; Feature selection; Machine learning Spams are unwanted activities such as when marketers send members unwanted advertisements, post fake reviews, or steal user information by directing users to malicious external pages [11] . As Social Network Services (SNS) becoming an important mode of communication, it attracts spammers who overwhelms users with unwanted content. Among these sites, Twitter, which was started in 2006, has grown to be one of the most popular SNS [22] . There are 500 million number of messages (called tweets) produced by 328 million active Twitter users (called twitterers) every day. Unlike other popular SNS, tweets can be read by anyone and people can follow a user without their consent. To attract users to their target websites, spammers post a large 2 number of coordinated messages containing specific URLs and sometimes describing them with unrelated words [26] . Because SNS helps build intrinsic trust between their users, 45% of them will click on links posted by their online friends even though they do not know those people in real life [24] . Twitterers also tend to post shortened URLs and write in abbreviated forms that rarely appear in conventional text documents or e-mails as a tweet can only contain up to 140 characters. Consequently, it is difficult for users to know the source URL and identify the content of the URL without clicking the link and loading the page. The noisy, unstructured, and informal expressions, such as "2mo is a new daaaaay!" or "TIL DC Comics stands for Detective Comics", used in the text also made it difficult for automatic spam detection system to accurately identify the semantic meaning of the tweets. Hence, social spamming is more harmful and complex than SMS, email or Web spams. It is becoming an important problem for users and service providers. Around 83% of users of social networks have received at least one unwanted friend request or message and over 3% of tweets are spam [7] . To make Twitter a spam-free platform, Twitter enable twitterers to report spam URLs, tweets and accounts which after being verified will be included in the Twitter's blacklist. All URLs, tweets or accounts in the blacklist will be automatically filtered, suspended, or deleted by Twitter. However, due to time lag, 90% of users may visit a new spam link before it is included in the blacklist [26] . Furthermore, twitterers identify spammers manually based on experience that could lead to false positives. Therefore, it is important to have a tool that can automatically identify spammers. The approach must be scalable too, i.e. it can handle a large amount of data in a short amount of time with limited computation resources. We can divide anti-social spammers systems into two types: tweet-level detection and account-level detection [24] . The tweet-level detection system check each tweet for spam text content or URLs. If an account has posted a certain number of spam tweets, it is flagged as spammers. As around 350,000 tweets are generated per minute [22] , tweet-level detection consumes too much computing resources and is harder to be run in real-time. Account-level detection checks individual accounts profile and activity patterns for evidence of them sending spam tweets or is a fake account. Because there is very limited amount of imbalanced-labeled data, account-level detection system tends to have high precision and accuracy but low recall. When it predicts that an account is a spammer account, it has a high probability of it being true. However, there are many more spammer accounts out there that were not considered as spammer candidates, i.e. they are classified as legitimate. This of course is not useful to us as we are interested in detecting all spammers. Spam detection is a never-ending game of cat and mouse. Although security companies, as well as Twitter, are working on creating systems to detect spam and spammers, spammers are always trying to avoid being detected. They deploy different techniques to post unwanted messages to users on SNS for advertisement, frauds or spreading of malware through the malicious URLs [12] . For instance, spammers create many fake accounts to post spam tweets for a specific purpose (or known as spam campaign), send message with different text to convey the same meanings or pay some users to follow their accounts [15] . Thus, the statistical attributes of spammers 3 and spam tweets vary over time. System that relies on old samples may struggle to detect the new spammers or spam tweets. As there are many spam detection systems proposed, it is hard for users and providers to decide which the best one is. We have also found that different work uses different evaluation metrics and datasets, so it is hard to achieve a standard evaluation. This brings in a research challenge about comparing and evaluating the performance of various spam detection systems and identifying the best technique w.r.t effectiveness (i.e., accuracy, true positive rate, and precision) and efficiency (run-time execution for training and classification). This is particularly beneficial to the research community as any newly proposed techniques can be evaluated against the existing ones allowing knowing if the technique is an improvement. In our study, we have reviewed 172 content-based and user-based features from the majority of existing literature. Based on these features, we aim to develop a workbench, namely WEST (Workbench Evaluation Spammer detection system in Twitter) to evaluate their proposed features against the ones in defined in WEST as well as to set the best number of recent tweets and find the best possible subset from all the features available. We have designed an evaluation method with a set of experiments to help select the optimal subset of features. We organize the rest of this paper as follows. Section 2 introduce background and existing work related to spam detection for Twitter. Section 3 discusses our proposed evaluation workbench and Section 4 discusses experimental results from our study. Lastly, a conclusion and future work are given in Section 5. This section provides an overview of related work with approaches and methods for SNS spammer detection. [25] shows that machine learning methods demonstrated by [12] can be utilized with significant success in spammer detection on Twitter. Such methods are able to extract user or context-based features from user-behavioral patterns or linguistic features in a tweet [1, 6] . [2] shows that supervised machine learning techniques such as Support Vector Machine [10, 24] are able to train features extracted from user profiles in order to find profiles linked to spam activity. Performance is evaluated based on precision (the percentage of correct positive prediction), recall (the percentage of positive instances that were predicted as positive), and accuracy (overall percentage of correct prediction). [7] demonstrates a method of extracting the user and contextbased features from the dataset before running this through Meda et al.'s Random Forest classifier [25] . The output was evaluated based on precision and f-measure, the harmonic mean of precision and recall. [17] uses information gain and relief methods to determine the five best features from the dataset. [17] uses Information Gain, and Relief methods to find the best five from features. These approaches all use different features, datasets and classifiers, and as such, we are unable to evaluate and compare their performance to ours. Context-based features are linguistic features extracted from tweet context [6] . Twitter performs no checks on the legitimacy of shortened URLs, so spammers often exploit this by using a shortened URL service in an attempt to lure in legitimate Twitter users. [19] points out that spammers often use the same URLs in multiple tweets in order to increase the chance of it being clicked on by legitimate users. A number of researchers have utilized features related to URLs. Examples of these are the number of URLs [9] , the number of URLs per word [6] , and the number of unique URLs [10] . In Twitter, the #hashtag is used to describe a term, event, or emotion. If multiple tweets occur with the same #hashtag, it will become a trending topic [7] . Spammers often include a trending #hashtag with their tweets (though with unrelated content) in order to lure in legitimate users [17] . #Hashtags can be manipulated in the same way as URL features, and expanded to other forms such as the number of #hashtags per work on a tweet [6] . Twitter users are able to include @username in their tweets (called a "mention"). This enables the tweet to be sent to the user in the @username, regardless of whether or not they are followers of or followed by the user who tweeted. This is a feature that spammers also exploit, enabling them to push tweets to users [17] . This feature has been explored by several authors [7, 19] . It has been noted that tweets from spammers often include a larger number of spam-related words (up to 39%) while legitimate users around 4% [1] . Because of this, some papers use a spam word feature based on spam words from sites such as Wordpress.org [1, 7] . Other methods and approaches include features such as percentage of words not contained in a dictionary in their system [8] . User-based features are derived from properties related to user behaviour [1] . Generally, spammers follow as many users as possible to gain their attention, and increase the likelihood of success with spam attacks [7] . Common user-based features include number of followers (users following the user in question), number of followings (the users the user in question is following), and reputation (determining the user's influence on Twitter). These features are used in different combinations with varying success. [7] uses the number following and number followed features, and achieve 95.7% precision, whereas [14] uses number following, number followed, and reputation, but only achieves 91% precision. Both [12] and [2] use the followers-tofollowings ratio. The reason for this feature is that while spammers attempt to follow as many accounts as possible, it is difficult to achieve "follow-backs", and the features ensures a healthy ratio of followers and followings is maintained. The approach used in [12] was able to achieve 93.6% precision. However, [15] points out followers can be purchased from certain websites, effectively reducing the reliability of the followers-to-followings ratio feature. They introduce a new feature called bidirectional links ratio. This is defined as "mutual followings" -i.e., two accounts following each other. This feature is more difficult for spammers to evade, since it results in them having to purchase more followers. The only way this is evaded is through reflexive reciprocity -when a user follows someone back out of courtesy [27] . 95% of spam tweets contain shortened malicious URLs. [16] proposes URL rate and Interaction Rate -two features to address URL-based spam attacks. Interaction rate notes the lack of normal interaction behavior in spammers, while URL rate com-5 pares the ratio of URL-based tweets to normal tweets. This is a particularly effective feature since is it almost impossible to evade. As fundamentally formulated in the existing literature, given a set of users in the dataset , a spammer detection system is intended to build a classifier model, , to predict whether a user, is a spammer based on a set of features extracted from the user's social activities , relations , profiles and/or textual contents gathered from the user's N recent tweets. Spammers True Positive Rate, Spammers False Positive Rate, Precision, Accuracy, F 1 -measure, and Time are then used to measure the efficiency and effectiveness of the model. it is unclear which system is better. Spammers are continually changing their strategies to fool the anti-spam systems [3] . Current effective features now might not be effective in the future. Thus, it is imperative to be able to quickly find out the ineffective features and test the performance of new extracted features based on a new dataset. The main motivation for our Workbench for Evaluating Spammers in Twitter (WEST) is to provide a comprehensive collection of social activities, relations, profiles and textual content features that researchers can quickly select and evaluate on new data sets. People can then easily determine which features or which overall system is the most effective and efficient. WEST is written in JAVA and it has extensible architecture that enables new features to be easily implemented and integrated to the workbench. It relies on WEKA [20] to perform the feature selection and classification on the dataset. Therefore, users can choose any attribute selection technique and supervised learning algorithm available in WEKA or include their own implementation to WEKA when building their anti-spammer model. With WEST, we are able to answer the following important questions. 1. What are the most effective sets of features for identifying spammers? 2. What is the most efficient model for detecting spammers? 3. Can the number of recent tweets used significantly affect the system performance? Figure 1 illustrates the overall framework of WEST and this section outlines the set of features included in the workbench and the way researchers or practitioners can utilize and extend WEST. To avoid confusion, from now on, we will use the term user to signify the users of WEST and twitterer to represent the users of Twitter. The input dataset must be comprised of two folders, spam and ham, containing XML files for spammers and ham twitterers respectively. Data about each twitterer is stored in a separate XML file as shown in Figure 2 . WEST then extracts features that were selected by the users from the file. Based on the collection of features that WEST currently has, Table 1 displays the XML tags read by WEST and the list of information that can be extracted from each tag. Given a twitterer ID, we utilize Twitter4J [21] to get the age of a Twitterer's account and the list of the twitterer's followees. For each followee, we find out their name and if they are followed back by the twitterer. The result is stored in a CSV file illustrated in Table 2 .  XML Tags Information Extracted It is computationally expensive to analyze every tweet that a twitterer send to decide if the account is a spammer account. Furthermore, one solution for tackling the drift problem of twitter spams is by re-training a spammer classifier model every day based on the new spam tweets. Consequently, anti-spammer detection system must be able to identify spammer from the least number of recent tweets possible. Different researchers extract features from different number of recent tweets. For instance, [19] used 200 tweets and achieved 81% precision while [7] extracts feature from 100 most 8 recent tweets and obtained 95.7% precision. However, it is still unclear whether the difference in performance is due to the differing number of recent tweets or due to the set of features and dataset that they used. As shown in Section 4, with WEST, researchers can easily define the number of recent tweets to be included from the dataset and evaluate whether changing that number increases the model's performance. In WEST, users can either select which features they want to extract from the dataset, or select the name of an existing spammer detection system. When the user selects the name of an existing anti spammer system, WEST automatically selects the set of features used by that system. WEST also allows for new features to be added by the user and presently there are 17 systems [1, 2, 4, [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] and 173 features. Each of those features can be grouped into four types: profile, activities, relations, and tweet contents. Profile features are information obtained from the twitterer's profile page such as screen name, description, age of the account, profile's URL, and reputation. Relations features represent the twitterer's friendship status and activities like followers, followee, friends, bi-directional links (followers that is followed by the twitterer) and interaction (twitterer reply or mention of a follower or a non-follower's name or tweet). Content features capture all the linguistic properties of the text in a tweet such as URLs, URLs to a social media domain,
framework proposed by Kenny Ho et al. helps to identify the most effective and efficient spammer detection features, evaluate the impact of using different numbers of recent tweets, and therefore, obtaining a faster and more accurate classifier model REF .
A Framework for Evaluating Anti Spammer Systems for Twitter
OTM Conferences
Computer Science