A Taxonomical NLP Blueprint to Support Financial Decision Making through Information-Centred Interactions (2022)

Abstract: Investment management professionals (IMPs) often make decisions after manual analysis of text transcripts of central banks’ conferences or companies’ earning calls. Their current software tools, while interactive, largely leave users unassisted in using these transcripts. A key component to designing speech and NLP techniques for this community is to qualitatively characterize their perceptions of AI as well as their legitimate needs so as to (1) better apply existing NLP methods, (2) direct future research and (3) correct IMPs’ perceptions of what AI is capable of. This paper presents such a study, through a contextual inquiry with eleven IMPs, uncovering their information practices when using such transcripts. We then propose a taxonomy of user requirements and usability criteria to support IMP decision making, and validate the taxonomy through participatory design workshops with four IMPs. Our investigation suggests that: (1) IMPs view visualization methods and natural language processing algorithms primarily as time-saving tools that are incapable of enhancing either discovery or interpretation and (2) their existing software falls well short of the state of the art in both visualization and NLP.

## introduction there are many stakeholders and agents that interact within the space of financial markets. investment management professionals (imps) play the most prominent role here. on a macro scale, imps are responsible for the long-term strategies of institutions such as mutual funds, pension funds, sovereign wealth funds, etc. at the core of their activities lies information seeking -staying well informed by understanding market trends through reading external reports and developing their own predictive models based on thorough statistical analysis of large and varied sources of data. within the technological space of tools that support such information-seeking activities, natural language processing (nlp) research is already tackling tasks that imps perform, e.g., trading securities based on sources such as newswire, company quarterly reports, financial blog posts, and social media text @xcite @xcite . our study has revealed that textual and spoken documents are highly valued by experienced analysts, because they yield nuanced insights not available in aggregated, numerical data. 1 our critical survey of major financial-analysis software (e.g., bloomberg terminal, factset) reveals, however, that while this software is ubiquitous, its use of tools that could amplify understanding or enable discovery within natural-language sources is extremely conservative. this is particularly noticeable against the backdrop of a general trend in the financial sector towards automation of information processes, and an abstract awareness that ever-expanding nl datasets can facilitate more nuanced decision making @xcite . nevertheless, as we elaborate upon below, we have found that the "boots on the ground," the imps themselves, seem to assess the value of visualization and nlp techniques, as applied to their own use of unstructured natural language artifacts, exclusively in terms of faster analysis, with no prospect of better analysis -a world of "little data," mostly disconnected from the "big data" that they have read about in the popular press, in which computers can be relied upon to fetch and render natural language content but are largely superfluous to the analysis and interpretation of that content as imps require. this view persists because of a commodified view of nlp in which the literal under-standing of speech and language is viewed as either trivial or at least a mostly solved problem, through the lens of commercial successes such as siri, ibm watson, and google now @xcite . in other words, zealous misrepresentations of what nlp research has already accomplished have tragically impaired imps' awareness of the goals and capabilities of contemporary nlp, and have been perhaps the major obstacle to a more pragmatic utilization of nlp within this community. as will shortly become apparent, this is not an nlp research paper, nor have we attempted to reform the perceptions of imps. but because a central goal of the financial nlp community is to design intelligent interfaces and software that will better support the information practices of these imps, the ethnographic hci research presented here is important to the financial nlp community, as it identifies critical aspects of the information practices of imps that are not being supported. the good news is that the problems being addressed by past offerings of this workshop series are well positioned to address many of these aspects. below, we first describe our investigation of the information practices and processes of imps from an information-seeking process (isp) perspective @xcite . we conducted a contextual inquiry (ci), from which we infer a taxonomy of information seeking tasks related to analyzing natural language documents (study 1). we then conducted a series of participatory design (pd) workshops to validate the taxonomy and explore how revisions to the current software interfaces can better support the isps captured in our proposed taxonomy (study 2). after presenting the insights from study 2, anchored in the proposed taxonomy, we suggest design approaches for using visualization and nlp tools to support the isps of imps. ## background central banks such as the us federal reserve (fed) or the european central bank (ecb) play a prominent role in deciding the monetary policy of a jurisdiction @xcite . the leaders of central banks hold several press conferences a year to inform the public about their activities, and to give guidance on how they might act going forward. similarly, publicly traded companies play a significant role in the capital markets by providing investment and risk mitigation opportunities to financial organizations. public companies are required to hold regular earning calls to update the public on their activities. to imps, such events are critical to their risk mitigation efforts; the transcripts of these calls or conferences are thus valuable. imps (often referred to as financial analysts) make investment decisions on behalf of their employers (buy-side analysts) or provide advice to large investment banks (sell-side analysts). the scale and complexity of their decision making sets them apart from retail analysts who advise individuals or small businesses on their investments. in our first study, we examine how imps make use of spoken records of central bank news conferences and earning calls in their professional activities. in the second study, we will use their input from this first study to investigate better design approaches for software that supports the use of such records in their information seeking practices. ## related work observational studies have investigated the workflow and information practices of imps, producing taxonomies of the information transfer process from sell-side to buy-side analysts @xcite or details of accounting practices @xcite . however, these do not capture the imps' information-seeking needs themselves. they also do not describe how imps interact with information systems to satisfy their information needs. under the banner of interaction capture and retrieval or icr @xcite , however, there have been observational studies of somewhat related information practices that use spoken records of events. @xcite , for example, investigated how recorded voicemail was used in a corporate setting, and incorporated their findings in the design of an improved voicemail indexing and retrieval system @xcite . @xcite studied why and how users review meeting records in order to guide their development of a cue-based meeting retrieval system. @xcite conducted another field study in which they observed how people were using records of meetings, and showed that although technology such as speech excision @xcite ) is effective, it was not incorporated into state-of-the-art meeting browsers. these prior studies, along with other research in the meeting domain @xcite @xcite , have confirmed the relatively limited utility of more traditional meeting artifacts such as minutes, personal notes, and raw audio-visual recordings, and point to software-enabled tools as more effective. these include speech recognition and speech excision for voicemail and meetings @xcite , but there are many other promising candidate technologies: speech alignment @xcite , disfluency detection @xcite , speaker segmentation @xcite , information extraction @xcite , answer selection, an important step in question answering (qa) systems (e.g., @xcite , machine comprehension @xcite , and sentiment analysis (e.g., @xcite . these technologies, furthermore, as well as the remarkable pace of their advancement, are known to software developers who support imps, despite the lack of a design investigation that explicitly connects these advancements to imps' needs and practices. the remarkable performance boost in answer selection between 2004 to 2016 on datasets containing financial news (from a mean reciprocal rank of 0.4939 @xcite to 0.877 @xcite ), for example, was well publicized among these vendors, as were the significant improvements to machine comprehension, which extracts exact answer phrases to questions from raw text, in the space of a single year -from 50.5% @xcite to 78.6% @xcite (3.6% shy of human performance). sentiment analysis of financial news was understood to have improved automatic trading from roughly 30% to 70.1% annualized returns @xcite , and the use of sentiment analysis in market analysis tools has been commonplace now for almost 10 years @xcite . with the exception of sentiment analysis, however, the absence of any serious, contemporary nlp functionality is notable. this paper takes a first step towards an explicit design investigation of the potential of this functionality by proposing (study 1) and validating (study 2) a design-minded taxonomy of information practices within the financial analysis domain. the information-seeking process has been characterized as a highly variable process shaped by information seeking factors such as the task and information domain @xcite . for differ-ent tasks and information seeking factors, different types of support are needed by information seekers @xcite . methods such as contextual inquiry (ci) are effective in uncovering such information seeking factors @xcite , while approaches such as participatory design (pd) @xcite are useful not only for engaging users in the design process but for refining the functional requirements of information support tools @xcite . ## study 1: observing spoken document use spoken documents contain unique and critical information for imps. they are rich with both factual and affective data, and yet this medium is not adequately supported by existing financial analysis software such as bloomberg or factset. moreover, these spoken documents contain both content authored by the institutions holding the events (e.g. federal reserve), as well as q&a from journalists and analysts that, as will be discussed, give transcript readers clues about their future publications, and thus about the markets' reaction to the events. hence, the focus of our taxonomy is on imps' use of spoken documents such as transcripts from the federal reserve. in particular, we focus on overall information and decision-making practices, instead of users' interaction, or the use of specific elements of the text, a topic extensively studied in linguistics. we conducted a contextual inquiry, observing how imps utilized spoken records. eleven analysts (4 female, 7 male) who actively use transcripts responded to our participation call, which had been distributed through our professional network and word of mouth. all participants had more than 5 years of experience in their field, and were currently working at wall street (new york, usa) hedge funds, asset management firms, central banks, and large multinational investment banks. the study was conducted at participant offices. participants were instructed to choose spoken records they would be interested in reading as part of their professional activities. the documents that they chose were transcripts of earning calls of publicly traded companies or of news conferences given by leaders of central banks. a researcher observed them during reading; after they read, he later conducted a semi-structured interview with the participants to gain insights into the 6 information-seeking factors defined by marchionini (1995) that characterize their isp, the lens through which we view their interactions with information systems. these are: setting, information seeker, domain, task, search systems, and outcomes. the interviews were recorded and transcribed. the study's data consist of these transcripts as well as the observation notes. an inductive thematic analysis was used to extract the major themes in the dataset @xcite , conducted under an essentialist epistemological approach, in which language is seen as a reflection of intended meaning and individual experience (versus a constructional perspective, in which meaning and experience are viewed as socially constructed). in this paradigm, one can theorise about individual motivations. no theoretical framework was used, however, as our goal was not to measure fit with a particular theory. the participants/readers all work for organizations that are market participants, entities that buy and/or sell assets in the investment markets. when describing their professional duties, the participants noted that they exclusively made decisions in a group, highlighting the collaborative nature of their isps. 8 of the 11 participants noted that they usually updated their team about what they learned after reading transcripts. their task can therefore be formulated as extracting from the content of these spoken records key takeaway points that could be referenced later or shared with colleagues. they furthermore noted that in this industry, time is of the essence. even minor delays in investment decisions could be very costly. this is the major reason that imps tolerate working with error-laden transcripts, so long as they are available sooner. it also explains their expert proficiency in skimming and skipping over information they already know or find irrelevant. the meta-goal of participants is to increase institutional returns. for this, participants need to develop insights about future actions (e.g., whether the fed would raise rates) and outcomes (e.g., whether a company's total revenue would appreciate over the next year) of the organizations they study (e.g. a company or a central bank). just as important, the users also need to develop a good understanding of the markets' expectations of those actions and outcomes. the success of an imp hinges not just upon a more accurate grasp of the organizations' futures actions and outcomes, but of a differentially better understanding than the general market consensus. ## study 2: participatory design the competitive nature of the financial markets has made this user group largely unavailable to participation in user studies. we were fortunate to be able to recruit four professionals to participate in participatory design (pd) workshops (3 males, 1 female, identified as d1 to d4). d1 and d2 had also participated in study 1. all had more than six years of experience. d3 and d4 were sellside analysts for investment banks and equity firms, while d1 and d2 were buy-side analysts for large global asset management firms and hedge funds. ## conclusion: hci-nlp co-design our studies have revealed many information practices of imps. several are not well supported by existing software marketed to imps, partly due to the complexity of the processes that imps typically carry out (figure 3 ). our studies suggest that imps need more and more detailed visualizations than what currently exists in their software. they also suggest that nlp technology will be most enthusiastically received when it is bundled with visualization techniques as an extraction mechanism that populates visualizations in such a way that preserves the imps' sense of agency over decision making proper. an extensive taxonomy (see appendix, figure 4 ) synthesizes our findings, capturing the requisite high-level information practices, the software functionality that would serve the typical cases envisioned by analysts, the available nlp tools to support this, and the common ui elements in which these tools can be encapsulated (as drawn or described by the pd workshop participants). note that the "desired" functionality here consists mainly of very close variants of problems that have already received considerable attention from the nlp community, such as aspect-based sentiment analysis, but recast as more vertical tasks that imps will assign value to. without that domain-specific context, the more abstract tasks that nlp researchers generally ascribe to their own work are more likely to be construed by imps as a combination of trite and insufficiently nuanced, because their own vocational expertise is more highly prized by them than the general cognitive mechanisms that the ai community focus on in popular representations of their accomplishments. finally, this investigation has shown the importance of conducting user studies to assess the usefulness of technology (in this case, for supporting isps) alongside the development of the technology. blindly pursuing a "deep-learning" crusade for general intelligence is unlikely to result in widespread adoption of black boxes, even at the level of speech recognition and sentiment analysis, by imps. to some extent, this is a catch-22. their current software does not incorporate advanced nlp, and so imps are unaware of its potential specific to their needs, and thus they are resigned to reserving agency over even the minutest of their decisionmaking tasks, which software vendors capitulate to in the design of their products. for their real potential to be embraced by imps, nlp tools need to be embedded in designs and visualizations in a manner that emphasizes superior extractive accuracy and generative quality over the time value of using the tools, while maintaining a sense of isp agency.