MAGPIE : A Large Corpus of Potentially Idiomatic Expressions (2020)

Abstract: Given the limited size of existing idiom corpora, we aim to enable progress in automatic idiom processing and linguistic analysis by creating the largest-to-date corpus of idioms for English. Using a fixed idiom list, automatic pre-extraction, and a strictly controlled crowdsourced annotation procedure, we show that it is feasible to build a high-quality corpus comprising more than 50K instances, an order of a magnitude larger than previous resources. Crucial ingredients of crowdsourcing were the selection of crowdworkers, clear and comprehensive instructions, and an interface that breaks down the task in small, manageable steps. Analysis of the resulting corpus revealed strong effects of genre on idiom distribution, providing new evidence for existing theories on what influences idiom usage. The corpus also contains rich metadata, and is made publicly available.

## introduction idiomatic expressions are an established research topic within nlp, but progress has been hampered by a lack of large annotated corpora. existing corpora cover less than 5,000 instances of less than 100 different idiom types, which means they do not provide a good training and testing ground for methods aimed at capturing the phenomenon of idiom as a whole. idioms are a very large, constantly changing, fuzzy-boundaried category of expressions, which cannot be completely captured in a corpus by its very nature. however, the bigger and more varied the corpus, both in number of different idiom types and the number of idiom instances, the more likely it is that conclusions drawn from this corpus are valid for 'idioms' as a whole. larger corpora have indeed benefits from both the descriptive and computational linguistic perspectives. for the first, they allow for better evaluation of assumptions about idiomatic expressions regarding their distribution, frequency, variation, and more. for natural language processing purposes, they allow for more reliable, more fine-grained, and more representative evaluation of idiom disambiguation tools, and for better training of such tools, given the possibility of using more data-greedy machine learning tools. due to the desired size of the corpus, we could not rely on the tried and tested method of using expert annotators to label the potentially idiomatic expressions. instead, we used crowdsourcing to get the annotations from nonexperts. while this is an established and reliable method for large-scale annotation, crowdsourcing has not been applied to idioms specifically, likely because of the significant difficulty of the task and the resulting difficulty of instructing crowdworkers to perform the task reliably. we were however encouraged by related work by @xcite , who use crowdsourcing for a similar annotation task (verbal multiword expressions, including some idioms), showing that annotators agree in approximately 67% of cases. in this paper, we describe the whole corpus creation, namely the selection of idiom types and data to annotate, the crowdsourcing setup, procedure and difficulties, as well as a detailed analysis of the resulting corpus. contributions this paper describes the construction of the, to our knowledge, largest available corpus of idioms. 1 in addition to the practical advantage deriving from creating this resource, we we also able to answer the following research questions: terminological clarification the ambiguity of phrases like wake up and smell the coffee poses a terminological problem. usually, these phrases are called idiomatic expressions, which is suitable when they are used in an idiomatic sense, but not so much when they are used in a literal sense. therefore, we propose a new term: potentially idiomatic expressions, or pies for short. the term potentially idiomatic expression refers to those expressions which can have an idiomatic meaning, regardless of whether they actually have that meaning in a given context. so, see the light is a pie in both 'after another explanation, i finally saw the light' and 'i saw the light of the sun through the trees', while it is an idiomatic expression in the first context, and a literal phrase in the latter context. ## related work in this section, we discuss existing idiom corpora and the variety of approaches used in their creation. the idiom corpora contain both literal and idiomatic occurrences of idiomatic expressions, and are labelled with their meaning. the four biggest of these are discussed here in detail. in general, there is large variation in corpus creation methods, regarding pie definition, extraction method, annotation schemes, base corpus, and pie type inventory. depending on the goal of the corpus, the amount of deviation that is allowed from the pie's dictionary form to the instances can be very little @xcite , to quite a lot @xcite . the number of pie types covered by each corpus is limited, ranging from 17 to 65 types, often limited to one or more syntactic patterns. the extraction of pie instances is usually done in a semiautomatic manner, by manually defining patterns in a text or parse tree, and doing some manual filtering afterwards. this works well, but an extension to a large number of pie types (e.g. several hundreds) would also require a large increase in the amount of manual effort involved. considering the sense annotations done on the pie corpora, there is significant variation, with @xcite using only three tags (idiomatic, literal, unclear), whereas @xcite use six (idiomatic, literal, both, unclear, embedded, meta-linguistic). as for our approach, we allow a large amount of deviation from the pie's dictionary form, use a large amount of almost 2,000 pie types with no restriction on syntactic pattern, automatic pre-extraction, and five sense labels. as far as we know, crowdsourcing has not been utilised for creating an idiom corpus before. however, there is closely related work by @xcite , who create a corpus of verbal multiword expressions, a group which includes idioms as well, but is a lot broader, incorporating particle verbs, collocations, and other types of set phrases. kato et al. extract all instances of a set of mwe types taken from wiktionary from part of the ontonotes corpus @xcite . since simple extraction based on words can yield a lot of noise, i.e. non-instances, they refine those extractions based on the gold-standard part-of-speech tags and parse trees that are present in the ontonotes corpus. most novel, however, is their use of crowdsourcing for distinguishing between literal equivalents of mwe phrases like get up in 'he gets up early' and actual mwe instances like in 'he gets up a hill'. they frame the task as a sense annotation task, asking crowdworkers to label instances as either literal, non-literal, unclear, or 'none of the above'. using this procedure, they create a corpus of 7,833 verbal mwe instances, of 1,608 different types. ## method overall, the procedure for building the corpus is as follows: we select a set of idioms from a dictionary (section 3.1), and use a system to extract all forms of this idiom type from a base corpus (section 3.2). the resulting candidates are annotated using crowdsourcing, and the obtained annotations are aggregated and converted into a corpus of senseannotated pies (section 4). ## annotation procedure we make use of the figureeight crowdsourcing platform for annotation. 4 we see crowdsourcing as the only feasible method for creating a corpus of this size, in contrast to existing corpora, which were all created by experts. crowdsourcing does pose some challenges, especially for a relatively hard task like this one. given the set of candidate instances from the pre-extraction system, the challenge is to get high-quality sense annotations at manageable costs of time and money using the figureeight platform. therefore, we strive to make the task as easy as possible for crowdworkers. the basic setup of the task is three-tiered (see figure 1 ). given a highlighted pie instance in context, the annotators are asked to make a three-way decision: whether the instance is used idiomatically, literally, or in a way that does not fit the binary distinction. by only asking about nonbinary senses in a subquestion, the initial decision is kept as simple as possible. the dictionary definition of the idiom, extracted from the odei, is available on mouse-over of the highlighted text below the question. this is only displayed on-demand, in order to prevent information overload. 5 this constitutes the first tier of the annotation. the second tier of annotation is only triggered when the 'other'-sense is selected in the first tier (see figure 2 ). in this case, annotators are asked whether the instance is a false extraction ('not an instance of ...'), whether the given context is insufficient to interpret the instance ('unclear') or whether it can be interpreted but simply does not fit the binary sense distinction ('non-standard usage'). since this latter category is a 'miscellaneous'-category that can contain anything unforeseen, it triggers the third tier. the third tier is a plain text input asking annotators to describe the instance's usage or meaning in the context. ## the magpie corpus the annotated data is converted into a practically usable corpus by excluding those instances on which the majority of annotators agree that it is a false extraction from the data. aggregation of annotations, i.e. selecting the majority label and assigning a confidence score, is done on the annotation platform, so those labels and scores are used. an example data point is presented in listing 1. given that this is an annotated corpus of pies of unprecedented size, in addition to being an excellent resource for training disambiguation models, we we can gather substantial insights from it on idioms' characteristics and behaviour. in this analysis, we look into various aspects: annotation (dis)agreement, distribution of idiom types, sense distributions across types, composition of the 'other'-category, and influence of genre. please note that we carry out our analysis on the whole corpus. however, if one were interested only in the most reliable portions of the corpus, i.e. the cases all annotators agreed upon, different confidence thresholds can be set, as shown in ## conclusions we have built the largest corpus of sense-annotated pies to date using a crowdsourced annotation approach, and we make it available to the community to enable progress in idiom disambiguation and evaluation. based on a highaccuracy list of idioms created by combining idiom dictio-naries, and by using a strictly controlled crowdsourced annotation procedure, we created a high-quality corpus containing a total of 56,622 pie instances. based on the lessons learned during the creation of the corpus and the findings resulting from analysing its contents, we are now in a good position to answer the research questions posed in section 1. first, we consider whether crowdsourcing is a suitable method for large-scale, high-quality annotation of a large variety of potentially idiomatic expressions. the answer to this is a qualified yes: crowdsourcing is suitable, but the procedure has to be set up carefully to yield reliable results. we found that the most important characteristics were the procedure for selecting crowdworkers, dealing with untrustworthy annotators (cf. buchholz and latorre ( 2011 )), the writing of instructions which are both clear and comprehensive, and the development of an interface that breaks down the task in small, manageable steps. the fact that, taking care of these requirements, yielded a corpus which is an order of magnitude bigger than previous ones, both in number of types and instances, with a high-level of inter-annotator agreement, confirms the potential of crowdsourcing for complex linguistic tasks. as for the second question, which covers the relation between a corpus of this size and its potential for insight in idioms' behaviour and distribution, the answer is varied. on the one hand, the fact that the bnc is the base corpus means it has rich genre information, which provides insight into the matters of idiom usage in spoken vs. written language, and differences between academic disciplines. on the other hand, what we have done here in terms of analysis is limited to a relatively high-level view of idiom distributions and frequencies. as such, the true potential for linguistic insight of the corpus remains for further investigation, which likely includes more in-depth inspection of specific questions and manual encoding of specific idiom characteristics, such as a more fine-grained classification of variation types than we have used here. finally, we look at previously under-researched matters regarding idiom: the influence of genre on the proportion of literal vs. idiomatic pies. overall, we have found that almost all types are strongly skewed towards either idiomatic or literal usage, and that truly balanced types are rare. this implies that a corpus properly representative of idiom as a linguistic phenomenon will necessarily include many skewed types, which contrasts strongly with previous corpus-building approaches, which all explicitly aimed to include mostly balanced idioms. the strongest effect, however, is that of genre on the proportion of literal and idiomatic pies. the percentage of idiomatic pies ranges from over 85% in some genres to less than 40% in others, showing a clear pattern: technical, instructional language discussing concrete, physical topics have more literal pies, while argumentative, rhetorically rich language touching on more abstract concepts contains more idiomatic pies.