How vulnerable are you? A Novel Computational Psycholinguistic Analysis for Phishing Influence Detection (2021)

Abstract: This document contains our work and progress regarding phishing detection by searching for proper influential sentences. Currently, the world is becoming smart, as a result most of the transactions and posting offers happen online. So, human beings have become the most vulnerable to security breach or hacking through phishing attacks, or being persuaded through influential texts in social media sites. We have analyzed influential and non-influential sentences and populated our dataset with those. We have proposed a computational model for implementing Cialdini and we got state of the art accuracy with our model. Our approach is language independent and domain independent and it is applicable to any problem where persuation detection is important. Our dataset and proposed computational psycholinguistic approach will motivate researchers to work more in the area of persuasion detection.

## introduction in some ways we humans are incredibly durable, and in others we are incredibly fragile. most of the humans can be easily swayed by beautiful rewards. we are the most vulnerable to phishing attacks and such rackets. some people take these opportunities to persuade people to click some bad links through some deceitful messages. susceptibility (pierre o. jacquet, 2018), persuasion @xcite @xcite and gullibility @xcite , these three words are very much related to human behaviour. humans are susceptible to treachery, they can be persuaded easily, and they are gullible too to believe anything in what others say. depending on a person's behaviour we can influence him/her using different types of methods. robert cialdini did research on all those methods and ways and published his famous principles. we created a classifier to detect types of influence by fine-tuning a pre-trained bert model. in this paper our major contributions are: i. our research on phishing detection was using computational psycholinguistic approach. here we have modeled the cialdini principles on influence analysis. the rest of the paper is organised as follows. section 2 discusses about the related works and how those are different. section 3 discusses about cialdini principles and how we applied them in our work. section 4 describes how we made our own dataset and related literature survey. section 5 provides an insight into the pre-trained model and why we used it. section 6 discusses about our experiment with different methods and tools to reach higher accuracy. section 7 provides a detail about our result and section 8 concludes the paper. ## related work the closest research to our work is on sentiment analysis and persuasion detection based on cialdini principles. kiemute oyibo and julitavassileva (2018) focused on how culture and gender influence the effectiveness of cialdini's principles of persuasion. they investigated on how the culture, gender and age differences affect individual's susceptibility to cialdini's persuasive strategies. but their work centred on the people of nigeria. sridhar ramaswamy (2018) used unstructured data available for survey voice recording of customer interactions and chat transcripts and explored different technologies of deep learning @xcite and natural language processing (nlp) that would help better analyze the contextual information to capture customer feedback. christopher hidey (2018) worked with detection of persuasion in online discussion or in some argument. merton lansley (2019) developed a method that detects social engineering attacks that are based on nlp and artificial neural networks. ifeoma @xcite proposed the use of shoppers' online shopping motivation in tailoring six commonly used influence strategies: scarcity, authority, consensus, liking, reciprocity, and commitment and they identified how these influence strategies can be tailored or personalized to e-commerce shoppers based on the online customers' motivation when shopping. shilpa p c (2021) used deep learning techniques to classify the sentiments of an expression into positive or negative emotions which were further classified into more atomic emotions. vansh gupta (2021) worked with persuasion detection but their work was based on biased or misleading information or propaganda classification. some researchers have also worked on sentiment analysis and persuasion detection based on twitter data. olhakaminska and hoste (2021) developed an approach for the semeval-2018 emotion detection task, based on the fuzzy rough nearest neighbor (frnn) classifier enhanced with ordered weighted average (owa) operators. neha jadav and khamparia (2018) focused more on improvement of accuracy of sentiment system 1 to 5 star, 1 being the most negative. so many works on persuasion detection have been done over the years. everyone has focused on some particular domain, either customer feedbacks, online shopping sites, or some particular culture or area. our goal was to identify phishing attacks through influential texts. therefore we used persuasion detection and cialdini principles and we tried to be more general in determining effects of these influences. so, we collected example sentences from as many diverse places as possible. ## our technique while searching for the ways by which people can be persuaded or phishing attacks can be detected we came across cialdini's principles. foot_0 so we used mainly those principles in our classifier. before going further let's first discuss about cialdini principles. ## making dataset we studied cialdini's theory to understand how a person can be influenced in order to detect phishing attacks. we did not find any dataset regarding our research, so we had to make our own dataset and categorize the texts by ourselves. we used some pre-defined definitions in order to sort those sentences out. for identifying the six influence classes, we followed the following conventions -• give away or giving something without any charge on some occasion, such messages show reciprocity type of influence. we collected as many influential texts as possible from advertisements from different companies and grouped them in those classes. we gathered about 100 texts from each class and made a dataset with 735 texts. we created a model using this dataset. 3 still the variety in our dataset was very small. so we collected more examples using twitter scraping of companies like -amazon, tesla, microsoft, flipkart, tinder etc. from amazon, flipkart, we got more examples on scarcity and reciprocity. from tesla, we got more examples on commitment types. tinder gave us examples on liking. big companies like these are also perfect for finding examples on authority and consensus. these sentences helped us understand how social sites can persuade people through influential advertisements and messages. after collecting these sentences on various domains we used our temporary model to make predictions. then we rechecked, verified and corrected the predictions that were made above 95% and discarded others. finally we appended those texts with our original dataset (please see transformers use self-attention mechanism that allows the decoder to look back at the entire sentence and selectively extract the information it needs during decoding. this mechanism helps to know the context better. with rnn, one has to go word by word to access the cell of the last word. this becomes a major problem for gpus, sentence influence class number influence class name @lifeisahandful you're most welcome! we're happy we can help brighten your day! 1 reciprocity @rachelliveslife thanks for the support! we are hiring, have a look. 1 reciprocity @jaggipagal apologies for the unpleasant experience with your order. could you confirm if we've missed the estimated delivery date? 2 scarcity @funsizedel time to bust out those happy moves, and dance all night long! enjoy your order! #deliveringsmiles 2 scarcity @nhuebecker alexa play the fame by lady gaga 3 authority you guessed it -it's their next look! with the latest trends and top brands at flipkart fashion -india ka fashion capital, there's no other way than to wear the next! 3 authority you can now contribute to on-ground covid relief services with #check-outgiving . donate rs. 10 to @giveindia when you check out with just a click of a button. #flipkartcares #flipkartforindia. 4 commitment if you're not 100% in love after the first 30 nights, we'll pick it up, do all the packing, and give you a full refund. we do our best to donate returned mattresses and give them a new home. 4 commitment @stephthegroupie you've got to be squidding! we're shrimply lobsessed with these slides. a pair of these would make anyone jelly(fish)! 5 liking @cutenfeisty-there's nothing quite like a good book journey! tell us, what's your favorite genre of books to read on your amazon kindle device? 5 liking for the last 10 years we are the number 1 in this industry by our customer review 6 consensus 57% of consumers will buy this or use a this service because it has at least a 4-star rating. 6 consensus having zelda's approval on the box just brightened up our caturday! what does she enjoy doing when she's not lounging? 7 normal @pww3777 someone sure looks comfurtable! what's this little cutie's name? 7 normal elmo follows a more traditional design and uses lstm to compute vocabulary when it comes to art. bert uses transformers in two approaches for language improvement. elmo uses two layers each to include forward and backward passages to calculate the middle word vectors. this helps elmo to achieve high efficiency compared to other traditional language models.(ezen-can, 2020) though transformer uses a decoder and an encoder, bert only uses an encoder and it does bidirectional context search, whereas gpt does it in one direction. bert can also be used for multimasked sentences, and next sentence prediction, and bi-directional search really makes it more reliable. moreover, bert model reads at most 512 words in one iteration. gpt-3 has almost 175b parameters and t5 has 11b parameters, whereas bert has only 110m parameters in its base model and 340m parameters in its large model. based on our subject we did not require so many parameters as gpt and t5 provide, that is why we preferred bert over other models. we picked bert-base-uncased over cased as influence detection should not be sensitive to cases. ## experiments with different methods we trained a wide range of different models for the task. as discussed in sections 3 and 4 we made our own dataset and used them to train the models. as we discussed in section 5, we used bert model from transformers rather than rnn. after making the train and validation datasets we applied different pre-trained models, fine-tuned them and used the best one to make our final model. we fine tuned bert-base-uncased with our neural network which is taking input from the fine tuned bert layer and processing it in this extra 9 layers out of which first 7 layers are simple dense layers with the unit numbers of 1024, 512, 256, 128, 128, 64, 32, a dropout layer with 0.3 dropout rate and lastly an output dense layer with unit number of 7. we fine tuned another bert-base-uncased model with previous configurations but this time without the dropout layer. we wanted to check how it behaves after removing the dropout layer. then we also fine tuned a bert-base-cased model with a total of 9 layers among which 7 layers are simple dense layers with unit numbers 1024, 512, 256, 128, 128, 64, 32, an output dense layer with unit number of 7 and one dropout layer with rate 0.3. we checked by removing the dropout layer also for this model. then we also fine tuned a distilled version of bert model or distilbert. our deep neural network model used a vocabulary size of 10,000, a batch size of 32 and was trained over 10 epochs. the system consisted of two input layers, one main bert layer, six dense layers with varying sizes decreasing from 1024 to 32, one dropout layer with regularization of 0.3 and one output layer. we used rectified linear unit (relu) 4 activation function for dense layers and softmax function for the output layer. the main advantage of using 4 https://bit.ly/32yjplx relu over other activation functions is that it does not activate all the neurons at the same time. this means that the neurons will only be deactivated if the output of the linear transformation is less than 0. we experimented with all these models using different learning rates, because not only the method, but how fast the model is learning, is also very crucial in gaining more accuracy. we started with 0.01 and decreased the rate each time until we got the maximum accuracy. then we collected the learning rate and accuracy for each model. among all these models, we got the best result in bert-baseuncased without dropout layer model with learning rate of 0.0001 or 1e-4. foot_3 ## result and discussion there are many datasets on whether a sentence is ham or spam 67 and datasets on detecting phishing sites foot_6 (justinas rastenis, 2021; patrick lawson, 2020; kiemute oyibo and julitavassileva, 2018). but there is no dataset on influence analysis using cialdini principles, so we had to make our dataset. then we experimented with different deep neural network models and found bert-base-uncased to overwhelm others and be suited for our work. figure 2 shows epoch count versus loss for different models. as we got the best accuracy in accuracy graph for bert-base-uncased without dropout, similarly we got the least loss in case of this model. we used bert-base-uncased model of transformer to make our model with a learning rate of 0.0001 as it reached the highest accuracy. we achieved 97.46% accuracy with our small dataset. after appending it with more example texts, we got 93.76% from our model using the new dataset. 9 we have also kept a sub-part 10 of the dataset, as our test data, and used that to evaluate our model. it turned out to be 87.5%. our model does not show which type of influence is being used, but rather it shows each type of influence along with their probabilities of being present in the given text. figure 3 shows the prediction of our proposed model by displaying the probabilities of each influence category in a given sentence. to check this model is predicting well in the dataset, we used confusion matrix (see figure 4 ). true positive, false negative, false positive, true negative for each class are shown in to make the understanding better we use f1-score which depends on both recall and precision. in ## conclusion influence detection is a multi-level classification problem. many works have been done on this topic and on persuasion detection, but they have worked on a particular domain, area, or caste. we focused more on finding the example texts not only from we focused on detecting phishing and other social media attacks, because these are something to which people are the most vulnerable nowadays. we made our own dataset and we have not predicted only the type of influence; we have given all the probabilities for each type of influence. even for a human, we cannot predict exactly what type of influence is working in a sentence. sometimes, a sentence can be formed using more than one influence categories. advertisements include multiple influences in one sentence, so that becomes very difficult for anyone to determine the most effective type of the sentence. so, we have shown the probabilities as a pie chart (please see figure 3 ) and also mentioned the type of influence that is most probably working. where from here i. at first we did not find any suitable dataset for our work, so we collected data from different advertisements and sentences from twitter and we made our model. the final dataset that we have made is still not sufficient, so our model may be overfitted. we plan to increase the data more in the future using bootstrap mechanism. then by human intervention we will verify and rectify the predictions made by our model and train it with new data.