Using NLP to quantify the environmental cost and diversity benefits of in-person NLP conferences (2022)

Abstract: The environmental costs of research are progressively important to the NLP community and their associated challenges are increasingly debated. In this work, we analyse the carbon cost (measured as CO2-equivalent) associated with journeys made by researchers attending in-person NLP conferences. We obtain the necessary data by text-mining all publications from the ACL anthology available at the time of the study (n=60,572) and extracting information about an author’s affiliation, including their address. This allows us to estimate the corresponding carbon cost and compare it to previously known values for training large models. Further, we look at the benefits of in-person conferences by demonstrating that they can increase participation diversity by encouraging attendance from the region surrounding the host country. We show how the trade-off between carbon cost and diversity of an event depends on its location and type. Our aim is to foster further discussion on the best way to address the joint issue of emissions and diversity in the future.

## introduction figure 1 shows the increase in travel to the acl annual meeting over the past 40 years. whereas conferences used to be the privilege of a few academics, they are now attended by participants from companies, research institutes and universities across the world. this comes with an increase in the total volume of work published, and with it an increase in the carbon emissions attributed to travelling to in-person events. in this study we seek to quantify the impact of conferences that are increasingly diverse in terms of participation and location (undoubtedly beneficial) on the increased carbon emissions (undoubtedly detrimental). we base our analysis on publications spanning 55 years , taken from the acl anthology foot_0 . we use nlp tools to parse each document and identify the locations of the conference venues and lead researcher's institution. we answer the following questions: to the best of our knowledge, our work is the first to quantitatively explore the relationship between the location of conferences in a research field and diversity of participation. we make our dataset and code available foot_1 to enable further discussion on the costs and benefits of in-person meetings. the scientific discourse has included measuring and quantifying the emissions costs of conferences and the travel associated with them, from specific events (astudillo and azarijafari, 2018), to conference series @xcite , or indeed looking at the total emissions of an entire discipline @xcite . travel is not the only cost associated with academic conferences, or research in general, with one phd accounting for 21.5 tonnes of co2-equivalent emissions @xcite , of which 35% was attributed to conferences. recent work shows that in france, a typical research lab might dispense 64% of its carbon outputs on conference travel, with the remaining 36% made up mostly of commuting and energy usage @xcite in response to the pandemic, many conferences have moved temporarily online. a meta-analysis of these online conferences showed that a major result of online delivery was a reduction in the registration fee, promoting access @xcite . further, online delivery may allay fears of high travel costs (raby and madden, 2021) -as is often the case with top-tier conferences. the main barrier to online participation is a perception of reduced social (rather than academic) opportunities (raby and madden, 2021), although this may be overcome through facilitating interpersonal meetings, and social discussion @xcite . it should be noted that whilst travel is unnecessary in virtual conferences, there is still a quantifiable carbon cost due to the infrastructure required @xcite @xcite . academic conferences are not without their benefits and a clear advantage of in-person conferences rather than online is the perceived value in social interaction @xcite . this argu-ment is strengthened by the observation that citation rates are higher for work presented across longer distances @xcite ). an important benefit of conferences is providing an opportunity for researchers to interact with peers from diverse cultural, linguistic, demographic and academic backgrounds. this goal is also recognised within the nlp field. 3 the high climate cost of academic conferences has led to policy considerations @xcite , including the adoption of carbon offsetting programmes for participants @xcite , wise choices of locations to reduce the average journey distance @xcite and mandated reporting of climate costs for conferences @xcite . moving towards the adoption of any of these policies would help to begin the mitigation of the environmental impact of academic travel. similar discussion has already started in computer science conference communities, e.g. acm @xcite . environmental cost of ml and nlp research: in the field of ml and nlp, there has been an increasing trend towards openness in reporting of the emissions associated with ai research @xcite , especially that using deep learning @xcite . work has also been undertaken to estimate the overall cost of training machine learning (ml) models -taking into account not only the training time, but also the age of the hardware and server location @xcite . there have been a few efforts within our own field of nlp to better understand the impact that modern techniques are having on the environment and specifically to quantify the emissions costs of training ever larger neural networks @xcite . benchmarking of nlp systems in terms of their energy consumption is a viable way to better understand the carbon cost of training such a model @xcite . taking into account factors such as resource utilisation can give a more accurate picture of the energy consumption of nlp models @xcite . a recent trend in nlp is to create low-resource models that provide sufficient performance. for example, light transformer models are quicker to train and consequently have a lower carbon footprint @xcite . transfer learning presents an opportunity for massive carbon savings. if a model can be trained that requires only minimal retraining for various other subtasks, then this prevents further carbon expenditure down the line. maximising model reusability is a good strategy for reducing carbon emissions @xcite . ## methods to be able to answer the questions that motivate this work, we need certain data about the research process, in particular regarding the location of researchers' affiliations and conference venues. since no such single source of information existed, we decided to combine publicly available resources to create a new dataset containing the information we required. the process we used to create this resource is detailed below: data structure: a publication is an independent piece of research presented to the community as a journal article or a presentation at a conference. for the purposes of this work, each publication is described by: (1) an identifier; (2) the first author's affiliation (identified by the domain name in their e-mail address); (3) the location of the first author's affiliation and (4) an event, to which the publication is assigned. an event could be a track at a conference, a colocated meeting (e.g. a workshop) or a volume of a journal. it is described by: (1) an identifier; (2) a name and (3) a location -physical place name in case of in-person events or a special tag (@) in case of journals and virtual conferences. note that in this model, we always take into account the first author, while in fact one person may attend a conference to present several publications (resulting in less travels) or more than one author may attend to present a single publication (resulting in more travels). resolving this issue would require conference registration data, which are not publicly available. further, the address of the primary affiliation does not necessarily match the researcher's starting location when travelling to a conference. text mining: in the process of gathering the data we rely on the xml version of the acl anthology available on github foot_3 (we used the version from 17.02.2021). from there we obtain the publications (<@xmath0, associated events (<@xmath1. the crucial information missing from the xml structure is the author's affiliation and their location. this information is mined from the publication text: we download the publication pdf and use pymupdf foot_4 to convert it to plain text. next, we extract the first e-mail domain occurring in text through regular expressions (allowing for the curly brackets notation for account usernames) and treat it as affiliation identifier. then, we use spacy @xcite to process the text with the en_core_web_trf pipeline, based on roberta @xcite . among the text spans recognised by the named entity recogniser as belonging to the category gpe (geopolitical entity), the one occurring first after the first author's last name is considered their location. entities occurring close to each other are grouped, so that multipart names, such as cambridge, massachusetts (usa), are located correctly. finally, to interpret the location names for affiliations and events, we use the geocoding api of the google maps api. this allows us to obtain geographical coordinates (longitude and latitude) and country name for each location. we obtain continent information using the pycountry-convert python package. missing data: the process described above may leave some of the data fields empty. this may be caused by information being omitted in the xml (year or location for events) or pdf files (affiliation address not provided) or imperfect named entity recognition. in the case of events, we fill the missing data based on co-located events and manual investigation. we also check which of the conferences in 2020 took place as in-person events in the locations advertised. in the case of affiliations, we look at all other publications with the same affiliation and identify the most common location. we assume this location may also be used for the publication in question. note that some of the pdf files of the oldest publications are based on scanned typescripts. extracting information from these would require ocr techniques, but this was not attempted within the described work, resulting in a lower coverage of the earliest publications. ## results the process described above results in a dataset of 60,572 publications associated with 1,991 events. in the following subsections we analyse them to answer some of the important questions about the costs and benefits of the nlp conference system. where is nlp research done? regarding affiliations (e-mail domains), we see 5,501 different values in our dataset. unsurprisingly for literature dating back to 1965, no domain could be found in a significant portion (22%) of the publications foot_5 . for the known affiliations, the research output is unequally distributed between them, with the top 207 domains (3.76%) responsible for 50% of the publications. our diversity index d takes the value 0.2303. regarding addresses, they are associated with 135 countries. following the refining procedure described in the previous section, only 0.8% unknown values remain. the concentration here is even larger than in the case of affiliations: half of the output is generated by just 3 countries (us, china and germany) and the d coefficient equals 0.1087, indicating an even lower diversity amongst international publication in nlp venues. the contribution varying across years is shown in figure 2 . coloured bars show the fraction of publications from a given year associated with each country, sorted by their global contribution (@xmath2, @xmath3, @xmath4, @xmath5, @xmath6, @xmath7. additionally, we show the diversity coefficient for the years (white line, right axis). we can see the diversity was rising through most of the considered period, but since 2013, the trend is reversed. where is nlp research presented? in total, the 1,991 events were held in 48 different countries. the distribution of publications presented at each country is more uniform than previously covered, with diversity index of 0.3838. figure 3 shows how this distribution changed across the years. the bars correspond to the number of papers presented in each country in a given year, with the same colour coding as in figure 2 . we can see that the distribution changes drastically every year due to major conferences moving around the world. as previously, we see the increasing diversity through the increasing d coefficient. moreover, while the number of articles presented in the most common country (us) was consistently high throughout the studied period, its relative contribution to the overall publication volume was falling for many years. similarly to the previous plot, a new trend of falling diversity is visible from 2015. finally, we can observe the changing role of non-physical venues (light grey bars): the share generated by online journals falling over the years and the sudden change in 2020, when 96% of work was presented online. figure 5: the average emission per publication (over 5year periods) and total emission (yearly) between 1970 and 2019. what are the environmental costs? our dataset includes 51,116 publications, for which both the location of research centre and conference venue are known. the average journey distance was 4,988 km and the longest distance travelled was 19,888 km from new zealand to spain. 7 to convert from the number of kilometres travelled (to the conference and back) to the carbon emissions costs, we turned to data from the uk government for enabling companies to report their emissions 8 . this resource provided us with 5 years of historic emissions data (2016-2020) for short-haul and long-haul flights giving the co2 per passenger per kilometer for each given year. we trained a linear regression model to estimate the carbon cost of air travel beyond this time span. gains in flight efficiency have led to the reduction of carbon emission, resulting in higher costs for his7 note that this (and other journeys) may have involved a connecting flight, increasing its length. each event could be simply represented through its total emissions, but there are several issues with this approach. firstly, the size of a conference (number of attendees) dictates its overall emissions cost. therefore, we use the mean carbon cost of a publication at each event instead. secondly, we compared events according to their geographic reach. international conferences are those that can be hosted anywhere in the world. regional conferences are those that are restricted to a specific region (we included lrec, which typically happens around the mediterranean) and local conferences are those that happen in a single country (or a very narrow geographical region). the conferences included in each band are shown in appendix b. figure 4 shows that international and regional conferences are the main emitters of greenhouse gasses in the nlp field. local conferences emit around a quarter of the co2-equivalent (per publication) compared to international or regional conferences. whilst regional conferences have traditionally tracked below the average emissions of international conferences, the gap between them is narrowing, as these conferences are increasingly treated as international events. figure 5 shows the discrepancy between the total co2 emissions (in red, right axis) and the average co2 emissions (in blue, left axis) over the same period across our entire dataset. we can see that whilst the average emissions fluctuate, they are generally stable around 0.8-1.2 tonnes of co2 emitted per publication. this stability is possibly due to the fact that the increasing distances travelled are offset by increasing flight efficiency. in contrast, the total amount of co2-equivalent emitted by conferences has risen exponentially hitting 1 million kg in 1998, 2 million kg in 2006, 3 million kg in 2016 and then jumping to over 6 million kg in 2018. 6,000 tonnes of co2-equivalent equates to... 1,304 cars driven for a year 722 homes powered for a year 13,892 barrels of oil (energy production) 99,212 new trees planted (co2 capture) 339,172 nlp pipelines trained 168 nlp pipelines optimised 68,894 generic transformers trained 22 generic transformers optimised 71 instances of gpt-3 trained to put the value of around 6,000 tonnes of co2equivalent (total emissions of nlp conferences in 2018) into context, we can compare to emissions for other activities. these are shown in what are the diversity benefits? we hypothesise that series of events occurring in different locations have the benefit of encouraging local researchers to attend, increasing the diversity of participation. in this section we seek to quantify this effect. firstly, we verify this hypothesis by comparing the distances researchers travelled for conferences (blue bars) to the distances they would need to travel if they were choosing venues randomly (orange bars) in figure 6 . the results clearly confirm 9 https://www.epa.gov/energy/ greenhouse-gas-equivalencies-calculator our assumptions: the number of observed short trips, especially a few hundred kilometers, is much higher than expected in a random choice scenario. the number of long trips, especially around 10,000 km, is greatly reduced. using the data from the previous section, we can also estimate that thanks to these choices, the carbon cost of all travels was 27.21% lower (a total saving of 19,104 tons of co2 according to emission rates of 2020). next, we can ask whether the priority given to local conferences depends on what country a researcher comes from. to that end, we compute the relative travel length by dividing the observed mean travel distance by the travel distance in a 'random choice' scenario. figure 7 shows all countries with at least 15 publications according to their relative travel length and gdp per capita in 2018 @xcite . we can see that the longest travels are made by countries in the middle-east, most of them considerably wealthy. most countries that prefer nearby conferences have relatively low income, e.g. serbia, philippines or bulgaria. knowing that each event generates diversity by encouraging researchers from the nearby countries to participate, we can now measure how well this effect works for different conferences. it might be expected that achieving high diversity comes at a cost of longer journeys. we verify this by plotting the diversity of in-person events against travel distance (average per publication) in figure 8 . most events are indeed arranged along an upward direction, but some do not belong to that trend. for example, we can see that eacl conferences deliver more diversity than others for the same travel distance. some acl meetings foot_6 , on the contrary, are associated with very long travel and not so much diversity. lrec events are clear outliers here, since they have by far the highest diversity for low distances. the dashed line corresponding to the diversity index of journals indicates that the diversity observed in many in-person events is much higher. note that the online conferences are not included in this analysis, since their format was often unclear to authors in the moment of submission. in figure 9 we compare the mean participation diversity of events organised in a given continent across the years. consistently with figure 2 , we see an increasing diversity throughout most of the considered period for most continents. europe is the location of very diverse events, but the asian ones appear to be catching up. the journals have seen relatively slow growth and remain much less diverse than in-person events, except for south america or australia and oceania, where too few conferences took place for our analysis. ## discussion our work covers the carbon cost and diversity gain associated with conferences in the acl anthology. we consider that it is timely to perform this analysis, given the shutdown in physical meetings brought on by the global covid-19 pandemic and have focussed our analysis on conferences from before the pandemic began. figure 9 : diversity of events held on each continent between 1965 and 2019. '@' refers to journals. africa is not represented due to the lack of events there in the acl anthology. figure 2 shows that whilst the diversity index has grown consistently from 1970 to 2014, it has dropped since then, with 2020 having the lowest diversity index since 2008. we cannot give an explanation for the drop over this period without speculating, however tracking this index will allow us to measure the change in diversity over the coming years. whereas previous work has claimed that nonphysical venues promote diversity (raby and madden, 2021), our research broadens the picture, with figure 8 demonstrating that whilst some events are below the mean diversity index of online journals, many are above; in particular lrec and ranlp attract an audience from many countries. we chose not to make a direct comparison between in-person events and the pandemic-era online conferences of 2020 and 2021, since some events of the latter type were (at the point of submission) advertised as physical meetings, while others were in the hybrid format. however, extending our analysis to pure online and hybrid events is a clear direction for future work. we were also able to quantify the carbon cost of travelling to physical events in terms of co2equivalent. whilst this has unsurprisingly grown with the growth of the nlp field, the average carbon cost per paper has remained stable, indicating that gains in efficiency from better modes of transport are offset by an increased travel distance. the total emissions in recent years has been as high as 6,000 tonnes of co2-equivalent. it must also be noted that other activities of nlp research contribute to the total carbon cost generated by the nlp field. for example, the carbon cost of all travel in a single year of nlp research equates to about 22 fully optimised transformer models trained from scratch (see measuring the diversity impact contributed by a conference happening in a certain place is not possible directly, since we cannot know, who would participate, if the event took place elsewhere. however, our data indicate a preference for local events, which is the highest in low-income countries. holding conferences across the globe allows researchers from diverse locations to attend an event without flying as far as in a scenario where all conferences were located in one region (as was the case in the early days of the acl conferences). however, there is a cautionary tale to tell in our data relating to the year 2018. in figure 5 , a large spike on the right hand side corresponds to 2018, when a total of over 6 thousand tonnes of co2-equivalent was attributed to conference travel. in this year the acl annual meeting was held in melbourne, australia and lrec was held in miyazaki, japan. the effect of this is clear as researchers from europe and north america -who usually attend these conferences -needed to travel further, increasing the emissions. holding conferences in different locations will only lead to increased diversity if these events are advertised to and attended by a majority of people from the region they are held in. our definition of diversity index only takes into account the countries from which authors have attended, and does not measure other important factors of diversity (gender, race, economic status, native language, etc.). whilst some of this information may be discernable from our data, most of it would only be possible to discover by author disclosure, which was not possible in our context. reporting on the country-based diversity allows us to better understand the diversity of nlp research across the last 50 years. our work is designed as a focussed study on the acl anthology, and a similar analysis of a broader scope (e.g., all computer science, all science publications) would yield results allowing comparisons between disciplines. we were able to perform this analysis due to the provision of the acl anthology, which only covers papers in our field. whilst other resources indexing ai and wider computer science, or even generic scientific literature, do exist (e.g., dblp, google scholar, repositories such as openaire, event websites etc.), these each have their own limitations, such as not including pdf links (only dois which point to journal websites), lack of a public api or covering only a subset of the literature. event websites are a fruitful source for data mining, but each event has its own bespoke format and extracting data this way is slow. we have attempted to give a view of the data that allows policy makers to make informed decisions on where the next nlp conference should be. we have also made our data available to facilitate future research. policy makers may wish to consider the high emissions impact of locating a conference in an area far away from the typical attendance base, and also weigh this against the potential diversity gain of locating a conference in a lower-wealth area. we expect that conference organisers will make different decisions based on the relative importance of the above factors to their communities.