Principled Analysis of Energy Discourse across Domains with Thesaurus-based Automatic Topic Labeling (2021)

Abstract: With the increasing impact of Natural Language Processing tools like topic models in social science research, the experimental rigor and comparability of models and datasets has come under scrutiny. Especially when contributing to research on topics with worldwide impacts like energy policy, objective analyses and reliable datasets are necessary. We contribute toward this goal in two ways: first, we release two diachronic corpora covering 23 years of energy discussions in the U.S. Energy Information Administration. Secondly, we propose a simple and theoretically sound method for automatic topic labelling drawing on political thesauri. We empirically evaluate the quality of our labels, and apply our labelling to topics induced by diachronic topic models on our energy corpora, and present a detailed analysis.

## introduction policy-making in highly technical areas such as energy is deemed to require neutral input from specialised government agencies, e.g. the us's energy information administration (eia) or the international energy agency (iea). their publications are known as"grey literature", given that they are expertly produced but not peer-reviewed. in contrast to academic literature, grey literature is often freely available online and aims to be more accessible to lay readers. energy grey literature has been shown to display biases towards incumbent fossil fuel technologies @xcite , but no thorough exploration exists of the disconnect between grey and academic literature in their coverage of energy issues. in this paper, we provide a reproducible and automated assessment of topics across both literatures. we analyse and make available two diachronic corpora derived covering 23 years of energy discussions from two eia publications: the international energy outlook (ieo) and the more nationally focused annual energy outlook (aeo). parsing reports of government agencies is non-trivial due to their diverse layouts (changing over time), and text frequently disrupted with tables and graphs. we release a clean text corpus as a basis for future research on energy communication. we also make available a software tool-set for the creation and analysis of these corpora, easily generalisable to diachronic datasets in general. we analyse the corpora using dynamic topic models @xcite . we compare their discussion of various facets of energy politics against the discourse in the scientific community over the same period, drawing on a corpus of abstracts of articles published in established energy journals. by releasing our grey literature corpus, we address the reproducibility challenge of large-scale text analyses with unsupervised models like topic models in the social sciences @xcite . studies are often difficult to reproduce, compare against or build upon due to a lack of public resources, as well as ad-hoc subjective choices of the researchers. we address the latter problem by proposing a conceptually simple and theoretically sound method for automatic topic labelling, drawing from thesauri over the political domain. in summary, our contributions are: topic models and their dynamic counterparts have been extended to leverage the power of deep learning @xcite @xcite leading to a better data fit at the cost of substantial increase in compute cost and technical expertise. in this work we will leverage the dtm as introduced above to explore the discussion of energy technologies in scientific and government publications over the past 30 years. topic labelling topics as induced by the dtm are probability distributions over the vocabulary. while they are often visualized through the top n words with highest probability, a principled interpretation of their content remains a challenge. various methods have been proposed for labelling a topic, ranging from single best word selection @xcite over involved methods leveraging domain-general external resources like wikipedia @xcite by retrieving relevant phrases, which requires substantial ir and nlp overhead to process the potential label inventory. other work has employed graphbased methods over the structured dbpedia @xcite , generated candidate labels using wordnet @xcite or created descriptive labels as text summaries by extracting and aggregating candidate sentences from documents highly relevant to a topic @xcite . we propose a simpler solution by leveraging structured, and broadly domain-relevant thesauri as our label inventory. specifically, we use the eurovoc thesaurus, compiled by the european union which covers all areas of european parliament discussion (including energy policy), noting that our methods extend to any thesaurus. we propose methods for filtering the resource to a focused set of labels ( § 4.1); and mapping induced topics to one or more thesaurus labels ( § 4.3). ## topic labeling with thesauri in this section we propose a new way of assigning automatic labels to dtm topics. even though a variety of methods for automatic topic labeling exist @xcite @xcite , case studies in the social sciences have largely resorted to qualitative analysis and manual labeling @xcite , resulting in a bottleneck for analysis as well as the potential for introduction of human bias. we introduce a general and conceptually simple method, drawing on established domain-specific thesauri as a label inventory, and propose two methods for mapping topics to a small set of labels that reflect its content. we use the eurovoc thesaurus in our study ( § 2.1), however, our method generalizes to any domain-specific thesaurus which organizes related keyphrases into succinct labels. formally, we describe the set of eurovoc labels as l. each label @xmath0. our method consists of two steps: (1) thesaurus filtering, in order to retain only domain-relevant labels; and (2) an algorithm to map a topic (represented as a weighted list of words) to one or more ## experiment settings our experiments consist of two parts. the first empirically evaluates the effectiveness of our proposed topic labelling method ( § 6) and the second leverages these labels to support a large-scale diachronic investigation of the discussion of energy technologies in the grey-and scientific literature ( § 7). we use the official dtm implementation. 9 automatic model selection for topic models is an open research problem, where automatic methods such as normalized pointwise mutual information (npmi) do not always correlate well with human judgment. as a consequence, especially in nontechnical fields, researchers resort to their domain knowledge when selecting a parameterization. we selected a dtm parameterization based on a combination of npmi scores and modelers' expectation in terms of topic variation over time. as a result, we set the number of topics @xmath1, and the topic variance parameter σ=0.05, and use default values for all other parameters. we applied the dtm with the above parameters independently to the three corpora described in § 3. all experiments below are based on the induced topics. ## conclusions we presented a novel method for topic labeling leveraging domain-relevant structured resources. we empirically showed the quality of our approach through human evaluation, and through its application in a detailed analysis of discussions on en-ergy policy over the past 23 years. we highlighted differences in the discussions around electricity generation, sustainability and economy between nationally and internationally focused reports from the eia and scientific publications over the same period. we release our grey literature corpora and software tool-set to support future research. there are several areas of future work. in terms of down-stream analyses, our labelling framework can support additional comparisons for example across countries or other agencies such as nongovernmental organizations; and can be extended to different thesauri, with different focus or level of detail. for example, eurovoc captures all of renewable energy under a single label. future work could also involve automatic splitting of assigned labels for example based on further hierarchical clustering of keyphrases associated with a label.