A n I nvestigation into the C ontribution of L ocally A ggregated D escriptors to F igurative L anguage I dentification (2021)

Abstract: In natural language understanding, topics that touch upon figurative language and pragmatics are notably difficult. We probe a novel use of locally aggregated descriptors – specifically, an architecture called NeXtVLAD – motivated by its accomplishments in computer vision, achieve tremendous success in the FigLang2020 sarcasm detection task. The reported F1 score of 93.1% is 14% higher than the next best result. We specifically investigate the extent to which the novel architecture is responsible for this boost, and find that it does not provide statistically significant benefits. Deep learning approaches are expensive, and we hope our insights highlighting the lack of benefits from introducing a resource-intensive component will aid future research to distill the effective elements from long and complex pipelines, thereby providing a boost to the wider research community.

## introduction natural language understanding often goes beyond the syntactic and semantic layers, and perhaps nowhere is this more palpable than in the use of figurative language. a better understanding of figurative language use, such as metaphors, irony, or sarcasm, can not only lead to advances in computational creativity @xcite , but also in understanding social media content, where users often employ such pragmatic tools as irony or sarcasm @xcite . this type of figurative language is difficult to identify, however, at least partly due to what the influential literary poet and critic william empson called "ambiguities" @xcite in the language. in particular, figurative language use with sarcasm or irony completely decouples -and even contrasts -the communicator's intent from the communicated content @xcite , rendering shallow syntactic or semantic features unsuitable. the poor fit of such features is further exacerbated in social media posts due to the ubiquity of grammatical errors, hashtags, emojis, etc. the deeper, context-dependent inferential nature of figurative language, together with the poor fit of shallow syntactic and semantic features, makes deep neural networks a natural candidate for downstream nlp tasks like sarcasm detection @xcite . unfortunately, with increasing popularity of deep learning, the reliability of findings in publications that extensively employ deep learning can be expected, in general, to decrease @xcite . in light of this seminal empirical observation and the general difficulty of accurately identifying figurative language, it is reasonable to not expect outright success on a benchmark corpus simply based on the use of a deep network. the concerns about reliability, and thus, about reproducibility, are particularly acute in deep learning. for instance, @xcite demonstrated that the hyperparameter settings have a significant impact on the final results obtained by a model. crane (2018) further showed that other confounding factors such as variation of gpus, the exact version of a framework, the randomness of a seed value provided to a learning algorithm, and the interaction between multiple such factors, can all impact the obtained results. beyond reproducibility, however, lies another pertinent factor: the use of increasingly complex pipelines where multiple sophisticated components are glued together for an important downstream nlp task. in such scenarios, it is not always clear which components within the complex system may be responsible for improved outcomes. a simple change in data preprocessing may lead to a significant difference in the final result, for example @xcite . in publications that introduce complex nlp pipelines, however, such details have sometimes been omitted. turn tweet label context-1 the [govt] just confiscated a $180 million boat shipment of cocaine from drug traffickers. sarcastic context-2 people think 5 tonnes is not a load of cocaine. response man! i've seen more than that on a friday night. ## exemplar task and data within the limited scope of this paper, our goal is to specifically investigate the state-of-the-art sarcasm detection system presented by @xcite which reported an f 1 score of 93.1%, 14% higher than the next best result reported to the figlang 2020 workshop @xcite for the twitter track -and to distill a novel deep learning component used in their pipeline in order to investigate its contribution to the final result. through a comprehensive series of experiments, we find that this novel architecture (discussed in sections 3 and 4) does not lead to any significant improvement. the improvement may thus be attributed to components other than deep learning, such as augmenting the corpus by using additional data. investigating the other components, however, is not in the scope of the work being presented here. the task is to determine if the final response in a thread (i.e., a sequence of tweets where each post is in response to its previous post) is sarcastic. one such thread is shown in ## background the architecture we investigate has recently been used in downstream nlp tasks, motivated by its success in computer vision. its origins, however, can be traced back to nlp research, when @xcite borrowed from the bag-ofwords approach used in text retrieval. since then, a significant body of work in computer vision has developed this approach further. the core idea being the treatment of an image as a document, and lowdimensional features foot_0 extracted from them forming the visual vocabulary, thus enabling a vector representation of each image, subsequently used in classification or ranking tasks. a key advancement came in the form of vector of locally aggregated descriptors (vlad), introduced by @xcite . in this work, too, low-dimensional features were extracted from images, but k clusters of the features were created, and only the difference of each feature from the cluster center was recorded. instead of a single n -dimensional feature vector, each image would thus be represented by a k × n matrix. the non-differentiable hard cluster assignment, however, renders it unsuitable for training a neural network. netvlad @xcite resolves this by using the softmax function, whose parameters can be learned during training. since the cluster assignments of a feature are not known prior to training, their approach requires k ndimensional difference vectors to encode each feature. this increase in the number of parameters impedes model optimization, and may lead to overfitting -drawbacks discussed and subsequently addressed by nextvlad @xcite by introducing a step prior to the soft cluster assignments. in this step, the input is expanded to λn size by a fully-connected layer, and then decomposed into g groups of lower-dimensional vectors. further, a sigmoid function with range [0, 1] is used to assign attention scores to the groups for each vector. the process effectively provides a g λ reduction in the number of parameters, by aggregating lowerdimensional vectors. from a linear algebra perspective, this can be interpreted as representing the data using subspace projections of the original vector. context 1 e m e 3 e 2 e 1 . . . t m t 3 t 2 t 1 . . . tr tr tr tr tr tr tr tr lstm lstm lstm lstm . . . lstm lstm lstm lstm . . . h m h3 h2 h1 . . . cluster centroids fully connected for assignment .. . vlad vectors . . . . . . ## architecture for sarcasm detection for an analogous use of nextvlad in nlp, the token representation vectors take the place of the feature vectors used in computer vision literature. in particular, for sarcasm detection using the figlang corpus, one entire thread needs to be represented by a k ×n matrix. to achieve this, the context and response tweets (as shown in the locally aggregated feature vectors (i.e., the vlad vectors) are generated by computing the product of the attention, assignment, and the difference from the cluster center in the above equations, t, g, k, and i iterate over tokens, groups, clusters, and vector elements respectively, while w and b denote the weight and bias parameters of the linear transformations in the fully-connected layers. ## experiments and results we delve into several modifications of the model, as well as various hyperparameter settings, in order to investigate how much effect the nextvlad component has on the sarcasm detection task. our experiments initially use the same training configuration as @xcite , before exploring further. since @xcite employ additional unpublished data, an exact reproduction of the experiments is not possible. moreover, the partition of the corpus into training and validation set is left unspecified. thus, their results reported on the validation set are not truly comparable. some hyperparameter settings, like the number of epochs for training, are also omitted from their report. however, the primary aim of this work is not to focus on reproduction of the results, but to determine what role the nextvlad component played in the excellent final f 1 score of 93.1%. the performance of different configurations are shown in ## conclusion we investigate the extent to which nextvlad contributes to improved results in a recent sarcasm detection task, and find that it offers little in terms of additional benefits. our conjecture at this point is, thus, that the 14% improvement achieved by @xcite must entirely be due to the natural language augmentation techniques used. our work also indicates that local aggregators like nextvlad are unlikely to offer significant benefits to tasks related to figurative language identification, but more empirical work is needed to confirm this hypothesis. we hope that our insights can help future research in this direction by making it easier to channel their efforts into aspects of a pipeline that have tangible and attributable benefits to the final downstream nlp task.