ABSA -Bench: Towards the Unified Evaluation of Aspect-based Sentiment Analysis Research (2020)

Abstract: Aspect-Based Sentiment Analysis (ABSA)has gained much attention in recent years. It is the task of identifying fine-grained opinionpolarity towards a specific aspect associated with a given target. However, there is a lack of benchmarking platform to provide a unified environment under consistent evaluation criteria for ABSA, resulting in the difficulties for fair comparisons. In this work, we address this issue and define a benchmark, ABSA-Bench, by unifying the evaluation protocols and the pre-processed publicly available datasets in a Web-based platform. ABSA-Bench provides two means of evaluations for participants to submit their predictions or models for online evaluation. Performances are ranked in the leader board and a discussion forum is supported to serve as a collaborative platform for academics and researchers to discuss queries.

## introduction aspect-based sentiment analysis (absa) has gained a lot of attention in recent years from both industries and academic communities as it provides a more practical solution to real life problems. the goal of absa is to identify the aspects and infer the sentiment expressed for each aspect. for example, given a sentence i hated their service, but their food was great, the sentiment polarities for the aspect service and food are negative and positive respectively. conventional techniques for absa are mostly traditional machine learning models based on lexicons and syntactic features @xcite @xcite . therefore, the performance of such models depend on hand-crafted features. recent progresses have been made with the advancement of deep neural networks (dnn) with some of the models being considered as state-of-the-art (xu 1 https://absa-bench.com/ figure 1 : the general process of absa et al.). among them, attention mechanism has played an important role outperforming previous approaches by paying more attention to the context words that are semantically-closer with the aspect terms @xcite @xcite @xcite . the most recent approaches adopted pre-trained bidirectional encoder representations from transformers (bert) architecture @xcite generating significant performance gaps to other approaches due to bert's capability of capturing bi-directional contextual information and providing rich token-wise representation. introducing bert architecture into absa task naturally distinguishing the approaches to non-bert based models and bert-based models. figure 1 depicts the general processes of both of the two groups of supervised absa methods. although this research area has gained much attention in recent years, it lacks of unbiased comparisons overall. as deep learning based models perform differently on various hardware on different deep learning tools, existing works typically chose to either re-run or re-implement the selected comparative models under their own experimental environment. we also observe few works directly referring the results presented in the corresponding papers for comparison. this makes it difficult to have a general overview of the performances of the state-of-the-art models and has motivated us to build a benchmarking platform for absa research. existing benchmarking research works are mostly conducted on evaluating single tasks and none of them support aspect-based sentiment analysis @xcite @xcite @xcite . in this project, we fill this gap by proposing a unified evaluation process and building a united platform for comparing different absa models. we name our work as absa-bench. absa-bench particularly focuses on supervised approaches and is suitable for both dnn-based and conventional models. it provides two means of evaluations namely, results evaluation and model evaluation. results evaluation is done by comparing the ground-truth with the model-generated predictions submitted by the researchers. model evaluation supports the model submission and online evaluation which keeps the integrity of the predictions in a better way. to aid the model evaluation, a web based tool is developed to provide an objective evaluation environment. the background computation power of absa-bench is supported by the google cloud platform (gcp) 2 . after evaluation, the performance results are then ranked in the absa-bench leader board. absa-bench further supports a discussion forum for queries, comments and discussions regarding the model implementations, performances, ranking and new ideas. to the best of our knowledge, this is the first platform created with diverse functionalities to support the understanding of the state-of-the-art absa works. the contributions of the work includes: i) providing a unified absa evaluation platform which enables researchers to evaluate their models on the same benchmark dataset with a consistent metric under the same computation environment; ii) supporting a leader board for easy comparison, and a discussion forum for sharing ideas; iii) presenting the comparisons of several recent research works based on their performances on the absa-bench platform through a re-run or re-implementation. ## related works the related benchmarking platforms for natural language processing models can be categorized into two groups: single task benchmarks and mul-2 https://cloud.google.com/ tiple tasks benchmarks. squad @xcite foot_0 is a representative benchmark for a single task. it provides a platform for evaluating question answering models on the squad dataset. researchers could either submit the prediction results or their models which will be run on codalab worksheets foot_1 . a leader board ranks the performances of all the evaluated models. quac @xcite foot_2 imitates squad, but for contextaware question answering models for which the questions and answers are provided in the dialogue form. glue @xcite foot_3 provides a collection of tools for evaluating the natural language understanding models across a diverse set of existing tasks. it allows researchers to submit their prediction files for comparison. error analysis is also enabled. lince @xcite foot_4 is a centralized benchmark for linguistic code-switching evaluation that combines ten corpora covering four different code-switched language pairs and four sub-tasks. similar to glue, lince enables result submission, but does not support online model execution. textgen @xcite is a benchmarking platform to support research on open-domain text generation models. it implements a majority of text generation models and aims to standardize the research in this field. however, textgen does not allow online submission and evaluation. absa-bench is the most akin to squad but unlike squad, it focuses on absa task. absa-bench provides two means of evaluations that is similar to squad and quac. the online evaluation in absa-bench is supported by jupyterhub which has key features like customization, flexibility and scalability. this distinguishes it from other similar platforms. juptyerhub also serves a variety of environments. it can be easily containerised with any container, therefore can be scaled up for a greater number of users. a number of authentication protocols such as oauth and github are also supported, making it flexible for users. absa-bench also supports an online discussion forum for researchers to exchange their ideas. there are relatively less research efforts on providing a comprehensive benchmarking platform for multiple nlp tasks. @xcite 8 is the only one found in this category. it spans ten nlp tasks and recasts these tasks as question answering over a context using automatic transformations. therefore, decanlp evaluates the models under the rubrics of assessing question answering models. decanlp considers the general sentiment analysis, but does not include absa. ## taxonomy and the models aspect based sentiment analysis is a fundamental task in sentiment analysis research field @xcite which comprises of three sub-tasks: aspect extraction, sentiment extraction and aspect based sentiment classification. in recent years, deep neural network has gained a lot of attention in solving the problem of absa. more recently, bert @xcite , has shown its effectiveness to alleviate the effort of feature engineering and has shown state-of-the art results in the given task. however these performance improvements have been achieved at a high computational cost. as a result these models are costly to train and evaluate. to have a better understanding of the large number of dnn based absa models, a categorization is utmost essential. therefore, a taxonomy has been designed in this study which categorises different deep learning supervised technique, diving all approaches into broadly two categories: bert based and non-bert based models. note that we focus on supervised approaches in this work. ## the absa-bench this section introduces the absa-bench platform, including the two ways of absa benchmarking evaluations provided and our insights into the design and implementation of absa-bench. ## performance comparison discussion on the dataset including the motivation for choice, the implementation settings for the experiments and an objective comparison of the results have been presented in this section. ## conclusion and future work in this work, we design and implement an absa benchmarking evaluation process by providing two means of online evaluations and a web-based platform. leader board and discussion forums are enabled to rank the state-of-the-art absa research and share research ideas respectively. we examined some recent models and compared their actual differences under the unified platform absabench. this platform will help to understand the implementation of different deep learning models performing the task of absa. this understanding can then be utilised to improve the existing models. we intend to update our benchmarking platform with new tasks and datasets which will encourage quantitatively-informed research and learning.