The Devil is in the Details: Evaluating Limitations of Transformer-based Methods for Granular Tasks (2020)

Abstract: Contextual embeddings derived from transformer-based neural language models have shown state-of-the-art performance for various tasks such as question answering, sentiment analysis, and textual similarity in recent years. Extensive work shows how accurately such models can represent abstract, semantic information present in text. In this expository work, we explore a tangent direction and analyze such models’ performance on tasks that require a more granular level of representation. We focus on the problem of textual similarity from two perspectives: matching documents on a granular level (requiring embeddings to capture fine-grained attributes in the text), and an abstract level (requiring embeddings to capture overall textual semantics). We empirically demonstrate, across two datasets from different domains, that despite high performance in abstract document matching as expected, contextual embeddings are consistently (and at times, vastly) outperformed by simple baselines like TF-IDF for more granular tasks. We then propose a simple but effective method to incorporate TF-IDF into models that use contextual embeddings, achieving relative improvements of up to 36% on granular tasks.

## introduction in recent years, contextual embeddings @xcite have made immense progress in semantic understanding-based tasks. after being trained using large amounts of data, for example via a self-supervised task like masked language-modeling, such models learn crucial elements of language, such as syntax and semantics @xcite @xcite from just raw text. the best performing contextual embeddings are trained with transformerbased methods (tbm) @xcite . these embeddings have been shown to frequently achieve state-of-the-art results in downstream tasks like question answering and sentiment analysis @xcite . contextual embeddings are also often used to capture the similarity between pairs of documents; for example, on the semantic textual similarity (sts) task @xcite included in the glue benchmark @xcite , tbms have shown competitive performance, substantially outperforming embedding baselines like word2vec @xcite and glove @xcite . however, their performance on similarity tasks beyond abstract, semantic ones @xcite -for example, on granular news article matching -is less understood. in this work, we study the performance of tbms in textual similarity tasks with the following research question: are transformer-based methods as performant for granular tasks as they are for abstract ones? here, granular and abstract reflect varying amounts of coarseness in the concept of similarity. for example, consider the news domain: a granular notion of similarity might be whether a pair of articles both report the exact same news event. conversely, an abstract notion might be when the articles share the same topical category, like sports or finance. figure 1 illustrates this with an example for clarity. firstly, we define separate tasks to explore these two notions of similarity on two datasets from different domains -news articles, and bug reports. our analysis on both datasets reveals that contextual embeddings do not perform well on granular tasks, and are outperformed by simple baselines like tf-idf. secondly, we demonstrate that tbm contextual embeddings do in fact contain important semantic information, and a simple interpolation strategy between the two methods can help boost the relative individual performance of tbms (tf-idf) by up to 36% (6%) on the granular task. ## related work we discuss related work in two areas: textual similarity, and tbms. textual similarity has been studied from various perspectives -comparing documents of different lengths in order to capture varying levels of detail @xcite , evaluating semantic similarity between reference and generated corpus @xcite , and semantic similarity for long documents in a hierarchical fashion @xcite . it is also shown that sentence meta-embeddings (obtained from combining ensembles of sentence embeddings) perform better @xcite for semantic similarity tasks compared to the individual baselines. for duplicate detection, which is a more granular task compared to semantic similarity, @xcite show that detection of near-duplicates in news articles can be identified by evaluating n-gram level overlap in documents. in the news domain, @xcite shows that article similarity can be improved by extracting common 'concepts' from the two articles using graph-based approaches. tbms @xcite have been shown to consistently perform better in the semantic similarity tasks. peinelt et al. ( 2020 ) also shows that bert-based architectures appended with topic-related details from topic models lead to an increase in semantic similarity performance. however, few works have highlighted tbms' ability to capture granular information. @xcite shows that bert can be used for document retrieval by matching embeddings of each word in the query and document, capturing granular similarity. unlike previous approaches that focus on either a granular or abstract similarity task, we compare the performance of tbms with other baseline methods across the two tasks, and in addition, provide a simple method to improve the performance of tbms on granular similarity tasks. ## method in this section, we describe the methodology used to compare two documents from a granular and abstract perspective. further, we also define the granular and abstract text similarity tasks in detail. ## results despite the better results, the complete absence of semantic understanding is a disadvantage of tf-idf. to mitigate this issue, we propose a simple approach to merge the best performing indirect methods. let g t and g r be the similarity scores obtained from the tf-idf and the st-rt approaches respectively: we obtain a new, interpolated score g @xmath1, that is then used as an input to the classifier. as ## conclusion in this work, we study the use of contextual embeddings derived from transformer-based models (tbms) for semantic similarity tasks of varying granularity level. through empirical analysis, we show that while tbms achieve higher performance in the abstract similarity tasks, simple methods like tf-idf outperform these models for granular similarity tasks (like event matching). we then propose a simple but effective method to merge these two approaches, achieving relative improvements of 36% (6%) when compared to using only tbms (tf-idf). in future work, we plan to investigate the scope for integrating granular information into tbm contextual embeddings to toggle the granularity that such embeddings inherently encode.