CL a C @ DEFT 2018: Sentiment analysis of tweets on transport from Î le-de- F rance (2018)

Abstract: CLaC @ DEFT 2018: Analysis of tweets on transport on the Île-de-France This paper describes the system deployed by the CLaC lab at Concordia University in Montreal for the DEFT 2018 shared task. The competition consisted in four different tasks; however, due to lack of time, we only participated in the first two. We participated with a system based on conventional supervised learning methods: a support vector machine classifier and an artificial neural network. For task 1, our best approach achieved an F-measure of 87.61%; while at task 2, we achieve 51.03%, situating our system below the average of the other participants.

## introduction this paper describes the system deployed by the clac lab at concordia university for the deft 2018 shared task. for this 14 th edition of the défi fouille de textes (deft), the main goal was to analyze the sentiments in french tweets regarding transport on the île-de-france. as described in @xcite , four tasks were proposed: task 3 (t3) -sentiment marker and target: given a tweet about transport that expresses at least one sentiment, for each sentiment expressed, determine (1) the target of the sentiment, and (2) the sentiment marker (i.e. the linguistic expression that expresses the sentiment). task 4 (t4) -full annotation: given a tweet about transport that expresses at least one sentiment, for each sentiment expressed, determine (1) the target of the sentiment, (2) the sentiment marker, in addition to ( 3 ) the source of the sentiment. as each task builds on top of the previous one, due to lack of time, we only participated in the first two tasks (t1 and t2). ## datasets the deft 2018 organizers @xcite put at our disposal a training corpus of 68,916 tweets already annotated with their topic label (transport/non-transport) and their polarity label (positive, negative, neutral, mixed). to train our classifiers, we first split the dataset randomly to create two distinct sub-sets: (i) 80% was used as training set, (ii) and 20% was used as validation set. since the polarity was identified only on transport tweets, the second task consisted of 28,374 tweets for training and 7,094 for validation. tables 1 and 2 show the distribution of the datasets for tasks t1 and t2, respectively. ## pre-processing given the original datasets (see section 2), we first performed various pre-processing steps to clean and normalize the tweets. these steps were inspired by the work of @xcite @xcite . figure 1 illustrates the pre-processing of a sample tweet. as shown in as part of the pre-processing, we also experimented with marking negation, in order to increase our performance on task t2. as shown by several previous work(e.g. ( reitan et al., 2015 kouloumpis @xcite , negation is an important feature for sentiment analysis on tweets. we therefore tried to mark the scope of negation by marking each expression indicating a negation (e.g. pas, ne, n') until the next punctuation. unfortunately, our elementary method to mark negation seemed to lower the performance with our baseline model, a naive bayes classifier (see section 4.2) on the the validation set. with the negation marking, the f-measure scored approximately 15% lower than without it. hence, we dropped our negation marking in the pre-processsing. ## results and analysis for the shared task, we submitted 2 runs for task 1 and 2 runs for task 2. ## conclusion this paper described our first participation to the deft shared task. due to lack of time, we only participated to the first two tasks: transport / non-transport classification and global polarity. we deployed models based on standard hand-crafted features and used off-the-shelf toolkits to pre-process the tweets and experiment with a variety of supervised learning models. although our results with the validation set seemed somewhat acceptable, most of our runs underperformed with the actual test set. our results at the shared tasks clearly indicate that training with the support vector machine classifiers seemed to over-fit the training set with the large feature set that we used; whereas the artificial neural network seemed more robust as is reached 85.54% at task 1 with very little fine-tuning of hyper-parameters.