UIC - NLP at S em E val-2020 Task 10: Exploring an Alternate Perspective on Evaluation (2020)

Abstract: In this work we describe and analyze a supervised learning system for word emphasis selection in phrases drawn from visual media as a part of the Semeval 2020 Shared Task 10. More specifically, we begin by briefly introducing the shared task problem and provide an analysis of interesting and relevant features present in the training dataset. We then introduce our LSTM-based model and describe its structure, input features, and limitations. Our model ultimately failed to beat the benchmark score, achieving an average match() score of 0.704 on the validation data (0.659 on the test data) but predicted 84.8% of words correctly considering a 0.5 threshold. We conclude with a thorough analysis and discussion of erroneous predictions with many examples and visualizations.

## data analysis this task made use of a dataset obtained from adobe spark ( https://spark.adobe.com ). the provided training set contained 2742 unique phrases, comprising a total of 32400 words. these phrases were annotated manually by nine individuals who decided, for each word, whether the word should be emphasized. the individual labels were aggregated by the task organizers, with the final gold standard being provided as ratios of votes for emphasized over votes for not emphasized, ranging from {0.0, 0.11, ..., 1.0}. a similarly-labeled validation set containing 392 phrases (a total of 4385 words) was also provided. before creating models, we analyzed the data to better understand the underlying trends and how they might impact model training and performance. below we detail three interesting observations that directly impact model learnability. second, we noticed that the training set contains 85 duplicate phrases. this presented the opportunity for some interesting additional analyses; we focused specifically on those repeated exactly twice, which was the vast majority of the duplicates. we sought to find how often the annotators agreed with themselves between pairs of repeated phrases to provide further evidence regarding the subjectivity of the task. absolute word-wise agreement -cases when all annotators voted each word the same way in a duplicate phrase -was very low, around 27%. we illustrate this in ## model design and training procedures we developed and trained several models using keras in python 3. generally, our goal was to create a stacked bidirectional lstm structure and feed it a descriptive word embedding and additional information for every input word to aid its learning capability. our input features are described in detail below. while we experimented with many different word embeddings, including word2vec @xcite and elmo @xcite , we observed the most consistently positive results using pre-trained 100-length glove @xcite embeddings. we introduced this information into the network using a keras embedding layer, where every missing word was given the average embedding of any word present in the three datasets (training, validation, and test). in addition to the word embedding, a one-hot encoded part of speech (pos) tag was generated for each word using the stanfordpostagger @xcite version from 2018-10-16. while the training and validation datasets came with provided pos tags, these tags were not provided for the test dataset. therefore, we chose to overwrite all pos tags so there would be no issues with consistency. the second supplemental feature was a vector of three scores corresponding to valence, arousal, and dominance (vad) scores for each input word, to attempt to capture emotion-related attributes. we extracted this information from the sizeable nrc vad lexicon @xcite ; we refer readers to the original paper for a detailed explanation of how these scores are determined. in short, valence corresponds to the positivity or negativity of a word, arousal corresponds to the word's intensity, and dominance refers to the amount of affective control the word exerts. any words which were missing vad scores were given an array of zeroes. these three pieces of information (word embedding, pos vector, and vad vector) were fed into a neural network utilizing the following structure: concatenation @xmath0. the output layer contains 38 nodes corresponding with the maximum document length across the three datasets. we optimized the network using logcosh (log cosine similarity) loss, linear activations, and a batch size of 1. this structure was not difficult to implement, but the nature of keras concatenation layers forced us to pad input sequences to a constant length (max phrase length) and predict 38 outputs for every phrase. this is likely one of the contributing factors limiting model performance. we also experimented with many other architectural variations, including introducing dropout layers, dimension and number of stacked lstm layers, more dense layers at the end of the network, and as previously noted, modifying the word embedding type. ultimately, the described setup yielded the most consistent, albeit low-ranking in the context of the shared task, results. we found that higher-dimensional word embeddings increased network training time and caused significant issues with overfitting. ## results and error analysis the shared task evaluation metric was the match(n) statistic, defined by @xcite . this metric considers the relative ranking of prediction labels, rather than the values themselves. here n refers to the number of top predicted values to be compared -in other words, for @xmath1, the metric looks at the two words having the highest gold standard scores for an instance, and computes their overlap with the two words having the highest predicted scores. our model achieved an average match score of 0.704 on the validation set, which shows that some patterns have been learned, while others remain unknown. for our error analysis, we chose to consider an accuracy measurement that takes into account how these models would be practically deployed in a downstream application. we used a 0.5 threshold to signify a positive (> 0.5) or negative (< 0.5) emphasis label. considering predictions for each of the 4385 individual words in the validation set, 3718 are a 0.5 match; in other words, the validation label and prediction agree at a 0.5 threshold that the word should/shouldn't be emphasized. this 84.8% measurement is highly promising, foot_3 and leaves us with 667 errors to analyze. as we are treating the analysis from the scope of binary classification, we also computed other standard classification metrics, finding that in this context precision is 0.527, recall is 0.742, and f1 is 0.616. after analyzing approximately one-third of the errors manually, repetitive categories became quickly apparent. these categories are explained, with frequency percentages and examples, below. close misses on low-confidence labels made up a significant proportion of the errors present. we define this category to consist of incorrect predictions ranging between 0.35 and 0.65 where the true label was of low confidence (0.44, 0.55, or 0.66). as we expected from the analysis of the training data, in a vast majority of these misses, the model predicted too low rather than too high. this category accounts for approximately 33% of all misses. we chose to filter these errors from the remainder of our analysis so that subsequent error categories only include more descriptive misses (scaling the future categories up to a fraction of the 667 total). late-phrase misses were very common and are likely fueled by the wide distribution of phrase lengths and relatively low average number of emphasized words per phrase. these errors are defined to be a 0.5 match threshold disagreement between the predicted label and validation label, where the word with the error is at @xmath2. this category accounted for around 17% of the total number of errors. abnormal sentence structure seemed to cause some difficulty for our model. while this category is somewhat hard to define, around 8% of our model's errors can be attributed to phrases which are poetic or sound like advertisements -not necessarily following the typical grammar rules or flow. an example of such a sentence is shown in two final categories with a significant number of errors are early-phrase (word in position 0 through 9) and late-phrase (word in @xmath3. these errors occurred when the model failed to predict multiple words correctly in a row. an interesting distinction is that the model struggled far more with these sequences late-phrase than early-phrase. an example is shown in we end this section by reiterating some optimism regarding the model's performance. at this 0.5 cutoff threshold, the model predicted 84.8% of the words correctly. increasing the threshold to 0.6 brings the accuracy up to 87.8%, and increasing it to 0.7 results in 91.3% correct predictions. although the model fell short of baseline performance when measured using the match(n) statistic, when considering the downstream task of actually selecting words to be emphasized, the model has clearly learned many accurate patterns. the task also remains subjective, leaving many "incorrect" predictions to not necessarily seem nonsensical (e.g., "jump into retirement" (model) vs. "jump into retirement" (true label)). ## conclusion for this shared task, we sought to develop a system which accurately selects words to be emphasized in short phrases of varying length and subject matter. we used keras to develop a stacked bidirectional lstm network and trained it using pre-trained glove embeddings and additional, supplemental information including pos tags and vad scores for every input word. unfortunately, despite many iterations on design and inputs, our model fell short when compared to the baseline using the match(n) metrics, ranking 22nd. through our error analysis, we have identified a number of common patterns with which the model struggled, including late-phrase words, sequences of words, and abnormal/poetic sentence structure. we plan to introduce additional strategies for addressing those limitations in follow-up work, and hope that our analysis provides useful guidance for others pursuing this task as well.