Assessing Social License to Operate from the Public Discourse on Social Media (2020)

Abstract: Organisations are monitoring their Social License to Operate (SLO) with increasing regularity. SLO, the level of support organisations gain from the public, is typically assessed through surveys or focus groups, which require expensive manual efforts and yield quickly-outdated results. In this paper, we present SIRTA (Social Insight via Real-Time Text Analytics), a novel real-time text analytics system for assessing and monitoring organisations’ SLO levels by analysing the public discourse from social posts. To assess SLO levels, our insight is to extract and transform peoples’ stances towards an organisation into SLO levels. SIRTA achieves this by performing a chain of three text classification tasks, where it identifies task-relevant social posts, discovers key SLO risks discussed in the posts, and infers stances specific to the SLO risks. We leverage recent language understanding techniques (e.g., BERT) for building our classifiers. To monitor SLO levels over time, SIRTA employs quality control mechanisms to reliably identify SLO trends and variations of multiple organisations in a market. These are derived from the smoothed time series of their SLO levels based on exponentially-weighted moving average (EWMA) calculation. Our experimental results show that SIRTA is highly effective in distilling stances from social posts for SLO level assessment, and that the continuous monitoring of SLO levels afforded by SIRTA enables the early detection of critical SLO changes.

## introduction social license to operate (slo) represents the ongoing acceptance (or lack thereof) of an organisation's standard business practices or operating procedures by the general public (or the society at large) @xcite @xcite . it captures the opinion of the public towards a business. low slo levels can increase business risks significantly, and, in the worst case scenarios, prevent the operation of an organisation. to obtain a high slo level, organisations typically need to build trust with the community and then work to maintain that trust. traditionally, the slo of an organisation is evaluated using surveys and focus groups @xcite , during which a diversity of opinions is collected and the results then quantified. these effective techniques provide indepth analysis. they are, however, manual practices and thus expensive to do on a frequent basis @xcite . in addition, the samples of a survey are often limited, and, as the time intervals between consecutive surveys are usually long, an organisation might not detect critical changes in its slo levels in a timely fashion, leading to exposure to potential risks. the public discussions continuously taking place on social media, where people are not shy about expressing their opinions about a number of topics, including companies and specific projects, provide an opportunity to monitor slo in real-time, on a continuous basis and at scale. this is what we aim to do in this work. we first determined the possible facets of slo for our domain, specifically economic (e.g., the public is in favor of a project because it will create jobs), environmental (e.g., the public believes the company has a good/bad environmental record) and social (e.g., the public believes the company addresses -or not -its social responsibilities). we then built sirta (social insight via real-time text analytics), a novel automated system that combines advanced text analytics with real-time monitoring techniques to assess and monitor the slo levels of a collection of organisations (in the same industry) over time. by taking the "pulse" of the public towards an organisation in real time, through the lens of social media, this tool complements the in-depth analysis done through surveys and focus groups, providing an early indication of trends, and potentially informing the design of in-depth surveys. figure 1 : the dashboard of sirta for slo assessment and monitoring plus the sirta architecture. figure 1 shows the dashboard of sirta for monitoring several major mining companies in the country. its main functionality is demonstrated in three panels: 1) the slo weekly overview, a list of real-time (weekly) numerical scores representing the slo levels for the organisations under consideration, 2) the slo trend, which plots the long-term trend of the slo level of a selected organisation (here, rio tinto), compared with the general trend of the market (all the organisations together), and 3) the social feed, with the most recent social media posts (e.g., tweets) about the selected organisation, with both the stances and slo risk categories (i.e., environment, social, and economic) identified. to carry out the slo assessment, sirta extracts opinion information from posts published on social media (twitter), along the different slo facets, and then transforms that information into slo scores. in contrast to many opinion mining systems that rely primarily on sentiment analysis, e.g., @xcite , we focus on stance detection @xcite , which is more suitable for our task because it indicates whether someone is for, neutral or against a specific company, not just whether the surface sentiment of their posts is positive or negative. the novel aspect of sirta's slo assessment engine includes a specialised text classification pipeline (see the text analytic pipeline on the bottom of figure 1 ), where three chained text classification tasks are performed for opinion extraction: 1) relevance classification, for finding posts contributing to the slo assessment, 2) risk classification, for identifying the different facet(s), or slo risk(s), being discussed in the posts, and 3) risk-aware stance classification, for detecting stances in the posts that are specific to each slo risk. the outcome from text analytic pipeline is fed into the slo score computation component for converting the stances into numerical slo scores. to train and evaluate the classifiers for each task above, we created both a silver standard and a gold standard dataset and employed state-of-the-art language understanding models such as bert @xcite . to monitor the derived slo scores, a monitoring engine keeps track of the time series of the slo scores of multiple organisations operating in a market. specifically, it leverages control charts @xcite , a powerful tool for statistical process control. the monitoring engine discovers if an organisation is experiencing significant changes in its slo score by contrasting its time series with a benchmark of the market. we conduct several quantitative experiments to evaluate the performance of our classifiers, and thus the effectiveness of our text analysis pipeline. we then present a case study, which suggests that sirta can identify periods of unusual changes early. this confirms our original hypothesis that we could harness social media to monitor slo in real-time and at scale, in a relatively inexpensive manner, reserving the more expensive, traditional methods for circumstances where a more detailed assessment is required. ## training and evaluating the slo assessment engine we now present how we developed, trained and evaluated the classifiers for the slo assessment engine. we first created training and test data sets for each task in the text classification pipeline. we used these data sets to train a number of modules, experimenting with several state-of-the-art techniques. finally, we evaluated the classifiers, in order to choose the best ones to incorporate into sirta.