What is S em E val evaluating? A Systematic Analysis of Evaluation Campaigns in NLP (2021)

Abstract: SemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.

## introduction a large portion of the empirical methods in natural language processing (nlp) are defined over canonical text interpretation tasks such as named entity recognition (ner), semantic role labeling (srl), sentiment analysis (sa), among others. the systematic creation of benchmarks and the comparative performance analysis of resources, representations and algorithms is instrumental for moving the boundaries of natural language interpretation. semeval @xcite @xcite @xcite @xcite is the primary venue in the nlp community for the organisation of shared nlp tasks and challenges. semeval is organised as an annual workshop co-located with the main nlp conferences and has attracted a large and growing audience of task organisers and participants. despite its recognition as a major driver in the creation of gold-standards and evaluation campaigns, there is no existing meta-analysis which interprets the overall contribution of semeval as a collective effort. this paper aims to address this gap by performing a systematic descriptive quantitative analysis of 96 tasks encompassing the se-meval campaigns between 2012-2019. this study targets understanding the evolution of semeval over this period, describing the core patterns with regard to task popularity, impact, task format (inputs, outputs), techniques, target languages and evaluation metrics. this paper is organised as follows: section 2 describes related work; 3 describes the methodology; 4 defines the underlying task macro-categories; 5 and 6 presents the number of tasks and popularity in 2012-2019; 7 discusses semeval impact in terms of citations; 8 shows targeted languages; then, sections 9, 10, 11 analyse input, output and evaluation metrics; 11 focuses on sentiment analysis architectures and representations; this is followed by a discussion section; we close the paper with recommendations and conclusions. ## related work each semeval task is described by an anthology, which contains: a summary of previous editions or similar tasks, references to previous works, detailed task description, evaluation methods, available resources, overview of submitted systems and final results of the competition. it is worth noting, there is a variation, or even inconsistency, in the structure and the level of detail in the description. participants are also encouraged to submit papers with systems architecture explanations. however, there is a lack of overall analysis across different tasks and years in semeval. there are existing studies on the analysis of specific semeval tasks. @xcite focuses on developing sentiment analysis tasks in @xcite @xcite is an example of a replication study of the top performing systems, in this case systems used in semeval twitter sentiment analysis @xcite @xcite , and focuses on architectures and performance. evolution and challenges in semantics similarity were described in @xcite . this is an example of a study on the performance of a given type of architecture across tasks of the same type. there also exist studies on shared tasks in given domain, specially in clinical application of nlp @xcite , @xcite . however, they refer to tasks outside the semeval and are more result oriented rather than task organization. some studies discuss ethical issues in the organisation and participation of shared tasks. an overview focusing on task competitive nature and fairness can be found in parra @xcite . in @xcite authors also relate to these issues, yet giving the priority to advancing the field over fair competition. comparatively, this paper covers a wider range of nlp topics, and compares sentiment analysis and semantic similarity as well as other task types/groups in a systematic manner. to the best to our knowledge this is the first systematic analysis on semeval. ## analysis methodology we build a corpus based on the acl anthology archive from the semeval workshops between the years 2012-2019. reference material included acl anthology papers covering the task description, tasks' websites and papers describing the participating systems. all the reference papers included in this analysis are reported in the appendix b. the pre-processing analysis consisted in manually extracting the target categories for the analysis which includes: task types, input and output types, as well as evaluation metrics, number of teams, languages and system architectures. tasks were grouped based on the similarity between task types. if the same team took part in several tasks the same year, we considered each participation as distinct. there are four missing tasks in the plotted indexes, due to cancellation (2015-task16, 2019-task11), task-sharing (2013-task6) or lack of supporting task description (2013-task14). numbers of citations are the numbers returned by google scholar, using publish and perish supporting api @xcite . the list of citations were manually validated and noisy entries were filtered out. a final table with all the values extracted from the corpus is included in the appendix b. ## task types and groups based on task description we group each task within a macro-category. then, due to a large number of task types, tasks were clustered within 6 groups: sentiment analysis (sa); semantic analysis (sem): semantic analysis, semantic difference, semantic inference, semantic role labeling, semantic parsing, semantic similarity, relational similarity; information extraction (ie): information extraction, temporal information extraction, argument mining, fact checking; machine translation (mt); question answering (qa); other (ot): hypernym discovery, entity linking, lexical simplification, word sense disambiguation, taxonomy extraction, taxonomy enrichment. there are also macro-categories defined by the semeval organizers, starting from 2015, but we found them not consistent enough for the purpose of this analysis. ## semeval tasks in years within 8 editions of semeval, a total of 96 tasks were successfully announced. the number of tasks within one group is roughly similar every year (except for mt), as well as distribution of tasks in each edition. according to fig. 1a , we observe decreasing number of sem tasks: 5 on average in 2012-2017, and only 2 in 2018-2019. moreover, there were no machine translation tasks in the last 2 years, and a low number of mt tasks in general (only 4 tasks in 8 years). although sa has a relatively limited task complexity when compared to sem or ie, which reflects a higher variety of task types and an abundance of more specific interpretation challenges, the number of sa tasks each year is high (4, 3, 3 and 4 in years 2016-2019). it is worth mentioning, that there are other 6 sa tasks in the forthcoming semeval 2020. the absence of some task types may be caused by the emergence of specialized workshops or conferences, e.g. low number of mt tasks in semeval is caused by the presence a separate venue for mt: the conference on machine translation @xcite , which attracts more participants than semeval in this field. 6 task popularity as a measure of task popularity, we analysed how many teams participated in a given task. as the number of teams signed up to the task is usually much higher than the number submitting a system result, we consider only the latter. the number of teams increased significantly from 62 in 2012 to 646 in 2019, which shows not only a popularity boost for semeval, but an increase in the general interest for nlp. so far, a total of 1883 teams participated in this period. in fig. 1b , we observe a gradual increase in se-meval popularity, 30% on average each year to 2018, with a +129% jump in 2019. this is associated mainly with a dramatic increase of interest for sa: 542 teams (84% of total) in 2019. however, at the same time, number of teams in non-sa tasks decreased from 132 in 2017, to 115 in 2018 and 104 in 2019. the most popular tasks groups along the years are sa and sem, which gather more than 75% of teams on average each year. the third most popular is ie, in which total of 235 teams participated in semeval from 2012 (12% of total). as a contrast, we observe a relatively low interest in qa and ot tasks. only 41 teams participated in the last 3 years (3% of a total of 1148 in 2017-2019). especially in ot tasks, which concentrates novel tasks, in many cases including novel formats. in the last 2 years, sa shows a major increase in popularity (76% of all teams, compared to 40% in 2013-2017). at the same time, in tasks such as 2019-10, 2018-4 and 2018-6, which are mathematical question answering, entity linking on multiparty dialogues and parsing time normalization, respectively, only 3, 4 and 1 teams submitted results. this divergence may be associated with an emergence of easily applicable ml systems and libraries, which better fit to standard classification tasks more prevalent in sa (in contrast to ot, qa nor ie). ## the impact of semeval papers as a measure of the impact of semeval outcomes in the nlp community, we analysed the numbers of citations per task description in google scholar. the task description paper was used as a proxy to analyse the task impact within the nlp community. papers submitted by participating teams describing systems and methods were not included on this analysis. we considered the cumulative citations from 2012 to 2019 (fig. 2a ), with additional distinction on citations of task description papers published in a given year (fig. 3a ). citations within semeval proceedings were treated separately, as we focused on the impact both outside (fig. 2a ) and inside (fig. 2b ) the semeval community. in other words, citations found in google scholar are split into numbers of papers out and in the semeval proceedings. sa and sem have the highest impact, being the most cited tasks along the years both inside and outside semeval community, what can be attributed to their high popularity. considering the external impact, in 2019 sa and sem anthologies contributed with 2847 (41%) and 2426 (35%) citations respectively. ie has 985 citations (14%) and qa contributed with 148 citations (2%). the ot group, which consists of less canonical tasks, accumulated 468 citations (7%). the impact of mt papers is noticeably lower -84 (1%). in terms of citations within the semeval community (in all semeval 2012-2019 proceedings), we observe a similar pattern: 41% and 37% citations in 2019 come from sa and sem (357 and 322), and for remaining task groups proportions are almost identical as in citations outside community (chi.sq. ## languages in tasks we analysed semeval in terms of languages used in the tasks (fig. 4 ). we can distinguish 3 clusters: english-only (except for 3 tasks entirely in chinese); multi-lingual, which define identical subtasks for several languages; cross-lingual (targeting the use of semantic relation across languages). in total of 96 tasks, 30 investigated more than one language (multi-lingual and cross-lingual tasks) and 63 tasks were using only english. the five most popular languages, excluding english are: spanish (16), french (10), italian (10), arabic (8), german (8). although chinese is the 1st language in number of speakers, only 4 tasks were organised for chinese. most of multi-lingual or cross-lingual tasks are related to sa ( 5 in 2016-2018) or sem (15 in 2012-2019 ## input and output analysis in order to better understand the evolution of the semantic complexity of the tasks, we analysed them in terms of the types used to represent input and output data in all subtasks. based on their descriptions, we devised a list of 25 different abstract types used, then assigning each subtask the most appropriate input and output types. ## evaluation metrics we counted a total of 29 different evaluation metrics used in semeval (fig. 6 ). at a subtask level, the most frequent metric is f1-score, with 105 usages, followed by recall and precision, with 51 and 49 usages respectively, and accuracy, with 26 usages. f1, recall and precision are frequently jointly used, the last two playing the role of supporting break-down metrics for f1 in 95% of cases. this combination is very popular, especially for ie tasks, almost half of the use coming from this task group. the top 5 evaluation metrics make up 84% of the total number of metrics used in all years, last 12 (almost half) being only used once. in 89% of cases when rare evaluation metrics (from kendall's t to the right) are used, they occur in sa and sem tasks e.g. jaccard index in affect in tweets (2018) or smatch in meaning representation parsing (2016). furthermore, 67% of the least used evaluation metrics (only used 3 times or less) appear in 2015-2017, the same period when we could see tasks experimenting the most with input and output types. 11 zooming in into sentiment analysis ## recommendations we believe that this paper can serve as a guideline for the selection and organisation of future semeval tasks. based on the analyses performed on this paper, these are the main recommendations: ## conclusions this paper reported a systematic quantitative analysis of semeval, which is an important venue for the empirical evaluation of nlp systems. the analysis, which provides a detailed breakdown of 96 tasks in the period between 2012-2019, provided quantitative evidence that: