D igi C all: A Benchmark for Measuring the Maturity of Digital Strategy through Company Earning Calls (2022)

Abstract: Digital transformation reinvents companies, their vision and strategy, organizational structure, processes, capabilities, and culture, and enables the development of new or enhanced products and services delivered to customers more efficiently. Organizations, by formalizing their digital strategy attempt to plan for their digital transformations and accelerate their company growth. Understanding how successful a company is in its digital transformation starts with accurate measurement of its digital maturity levels. However, existing approaches to measuring organizations’ digital strategy have low accuracy levels and this leads to inconsistent results, and also does not provide resources (data) for future research to improve. In order to measure the digital strategy maturity of companies, we leverage the state-of-the-art NLP models on unstructured data (earning call transcripts), and reach the state-of-the-art levels (94%) for this task. We release 3.691 earning call transcripts and also annotated data set, labeled particularly for the digital strategy maturity by linguists. Our work provides an empirical baseline for research in industry and management science.

## introduction digital transformation (dt) has emerged as an important phenomenon and is expected to preserve its prominence for companies. international data corporation @xcite forecast that global spending on digital transformation will reach @xmath010 trillion over a five-year period. dt redefines how companies operate and enhances connectivity, and inclusion worldwide. according to united nations(nations, 2019), ai-enabled frontier technologies are helping to save lives, diagnose diseases and increase life expectancy, while ai-enabled education by enabling virtual learning environments, opens up programs to students who would otherwise be excluded. dt, at a high level, encompasses the profound changes taking place in society and industries due to the adaptation of digital technologies, while at the orga-nizational level, organizations by practicing strategies that do not only embrace the implications of digital transformation but also reach better operational performance @xcite . vial @xcite explores 282 works on digital transformation in the information systems literature and develops a conceptual definition of dt as 'a process that aims to improve an entity by triggering significant changes to its properties through combinations of information, computation, communication, and connectivity technologies (p.118). organizations, by formalizing their digital strategy and leveraging their digital resources, plan for their dt @xcite @xcite . in order to measure and evaluate the digital strategy of firms, the status quo of a company's digital transformation, and the digital maturity level are measured @xcite . in this research, our conceptual interest is centered on measuring the digital strategy maturity of firms. by considering this as a text classification task of earnings call transcripts, we leverage several transformer-based architectures for text classification, in addition to rule-based approaches. in the end we present our measure and release two data sets for future research foot_0 . ## related work digital transformation research is one of the most growing areas and has been studied by management scientists @xcite @xcite @xcite , economists @xcite , engineers @xcite , and social scientists @xcite in the literature. despite there are many studies investigating the implications or drivers of dt in different fields, our focus is limited to studies in management science, computer science, and their intersection. in this research, we predict the maturity of the digital strategy of s&p 500 companies by leveraging transformer-based models with domain knowledge. the most similar work to ours (al-ali et al., 2020)'s that uses earning call transcripts but does not release any data. therefore, we do not have chance to test our approach on their data. however, available earning calls data sets @xcite have limitations that make it infeasible to measure the digital strategy maturity of companies. moreover, there is no task-specific annotated data set available in the area particularly to measure the digital strategy maturity of companies. every quarter the executive leadership of a public company holds an earnings call meeting with investors and analysts to inform them about the status of the company, including their major digital initiatives. an earning calls transcript has mainly three parts, the first part consists of the names of company call participants, the second part consists of the presentation session of company executives, and the third part consists of a question-and-answer session. existing earning call transcripts data sets do not provide both sessions, but only provide the answers of the most spoken company executive in the q&a session @xcite . the executives might share more information during the first session than during the q&a session. one of the other major issues with the existing data-sets is that they only share one company executive's sentences by discarding other company representatives' comments @xcite . therefore, this would also yield inaccurate results leading to biased analysis when we attempt to learn about how companies are performing in their digital transformation. in this research, we release all sections of the earning call transcripts without any section removal. any public company has 4 (q1, q2, q3, q4) earning conference call meetings annually in the usa. however, existing data sets in the literature have missing transcripts for some of these meetings. measuring digital strategy maturity with missing earning conference call transcripts would yield inaccurate results. such that if we obtain apple's digital maturity in q1 and q2 of 2018, any digital strategy maturity analysis of apple would be biased given that apple might have disclosed several accomplishments in q3 and q4 of 2018. moreover, dataset digicall maec keith qin duration 2018/19 2015/18 2010/17 2017 companies 469 1,213 642 280 instances 3,691 3.443 12,285 576 data av. yes yes no yes ## problem definition and our hypotheses there are several research studies attempting to measure digital maturity in management science literature @xcite @xcite . however, among these studies, the only study that leverages transformer-based models is (al-ali et al., 2020)'s work, and therefore it is the most similar to ours. in their work (al-ali et al., 2020), that they identify two tasks; first, the prediction of the aspect of the digital strategy, and second, the prediction of the maturity of the digital strategy. for the first task, they disclose a dictionary of 350 terms in 17 topics to be used with the prediction task 2 . any sentence s1 contains a term for the aspect of digital strategy, they combine it with the preceding s0 and subsequent s2 sentence from the transcripts and feed the appended sentences s0+s1+s2 into the model. we hypothesize that appending s0 and s2 with s1 might increase the noise in the data. therefore, we only process sentences containing aspect maturity terms s1, and drop s0 and s2. in the second stage, they predict the maturity stage. if the digital initiative is being planned, it is labeled as plan, if the digital initiative is being developed or piloted, it is labeled as pilot, if the digital initiative is launched and making an active contribution to the business, it is labeled as release, if the digital initiative is being pioneered and making a significant business impact, it is labeled as pioneer. we hypothesize that a digital initiative released or pioneered completed in the past. hence, these two labels are combined under the past label by considering the temporal orientation @xcite ) (grammatical tenses); plan as future, pilot as present, and release and pioneer as past. ## approaches in this section, we provide details of different approaches applied for our task. for each following approach, the input is a textual string, i.e. a sentence from an earning call document, and the output is a label indicating the status of a company/a project. we consider three labels: past, present and future. ## experimental evaluation and findings by building on previous work @xcite and several transformer-based models, we considered predicting the aspect maturity of the organizations' digital strategy as a text classification task with past, present, and future as labels. consistent with the baseline study in the area (al-ali et al., 2020), we trained several domain-agnostic transformer-based models by applying fine-tuning with domain-specific data annotated by 4 different linguists. despite roberta provides the highest f-1 weighted at (al-ali et al., 2020)'s work, our 5 in all transformer-based models, we randomly split the data as train, validation, test into 75%*75%, 75%*25%, and 25% respectively. all experiments were conducted at google colab pro and randomly assigned to nvidia tesla p100-pcie (16gb) for gpu, and had the following settings: the learning rate 1e-5 (adamw), epochs 6, batch size 3. we used the huggingface library https://huggingface.co/ for all models. ## conclusions and future work predicting the maturity of the digital strategy of firms accurately might help researchers to explore organizations' digital transformation while providing insights on how to plan for digital transformation for organizations. in our supplementary analysis, we find that industrial trends with respect to the maturity of the digital strategy also change with time, such that organizations disclose more past and future digital strategies in q4 than in any other quarter. future research might consider the time-variant factors that influence companies' digital strategy maturity. such as, exogenous shocks, competitors' product releases might impact organizations' digital transformation. with the new proposed approach, it might be possible to predict which future digital strategy plans of organizations were suspended or accelerated by also considering the competitive behavior of organizations.