The Cycle of Trust and Responsibility in Outsourced AI (2022)

Abstract: Artificial Intelligence (AI) and Machine Learning (ML) are rapidly becoming must-have capabilities. According to a 2019 Forbes Insights Report, “seventy-nine percent [of executives] agree that AI is already having a transformational impact on workflows and tools for knowledge workers, but only 5% of executives consider their companies to be industry-leading in terms of taking advantage of AI-powered processes.” (Forbes 2019) A major reason for this may be a shortage of on-staff expertise in AI/ML. This paper explores the intertwined issues of trust, adoption, training, and ethics of outsourcing AI development to a third party. We describe our experiences as a provider of outsourced natural language processing (NLP). We discuss how trust and accountability co-evolve as solutions mature from proof-of-concept to production-ready.

## introduction our business unit specializes in providing ai/ml solutions to customers seeking to use nlp and other ai capabilities to augment human analysts. our typical use case involves customers with a small number of highly specialized subject matter experts (smes) who need to assess a large number of documents in a short amount of time, often in the context of high-stakes missions. our third-party nlp solution space is comprised of secure, clouddeployed processing pipelines that transform unstructured text collections into actionable insights using combinations of customized entity extraction and text classification. the pipelines produce a sortable and filterable data stream suitable for prioritized review by analytic endusers. with increasing frequency, we are approached by customers who have heard of our early successes in ai-based analyst augmentation and would like to achieve similar results in their own operations. regardless of perceived similarities among new opportunities and previous successful applications, we believe that ethically, the responsibility lies with us to assure that each potential use case for ai/ml adoption is appropriate, feasible, and sustainable. feasibility and sustainability play into the ethics of ai/ml solutions and also in our ethical dealings with customers. this includes ensuring that our customers have appropriately managed expectations for what ai can and should do in their context. it means working to understand the specifics of customers' requirements, including: the nature of their data the questions they need to ask of the data the availability of legacy data usable for training and evaluation the availability of smes to validate models the potential development of feedback loops for continuous model improvement. ## an eye opening first engagement one of our earliest customer engagements corresponded closely to the typical use case described in the introduction, in which we augment human workflows with automated nlp processing using a framework like that depicted in figure 2 . this particular engagement was small in terms of data size, typically under 10k documents per batch, but was nonetheless extremely impactful. the work was initiated by decision makers who believed that an ai-based solution using cloud services would provide a much-needed productivity boost for their highly valuable, specialized, yet under-staffed analytic workforce. they engaged our team to help create a cloudbased data processing pipeline to automate some analytic tasks performed by their staff, hoping to free up the smes to focus on other less automatable duties. our first trust-building challenge arose from our customer's initial expectation that our analytic pipeline would be fully automated, removing the human analyst from the loop completely. it became rapidly apparent that full task automation would not be advisable any time in the foreseeable future. the existing body of labeled data was produced by a single analyst responsible for producing a binary classification indicating whether or not reports were relevant to the team's mission. on a monthly basis, the analyst's process was kicked off by manual execution of a standing boolean database query. the boolean query returned an unranked list of documents numbering in the thousands or tens of thousands. each document would then be manually reviewed and tracked in a spreadsheet. any report deemed of interest would be subsequently annotated to highlight entities and key phrases of interest. the analyst would then generate visualizations to communicate his findings. processing a typical tranche of data in this way would take that sme analyst a minimum of three full business days, but often much more for larger document sets. this background scenario meant that we were starting our ml development with an initial data set that had an extremely skewed distribution of relevant reports to non-relevant ones. the data also featured annotations that had been produced without the benefits of standard annotation guidelines and automation. given this, we needed to help our customers understand why it would be beneficial to opt for a user-in-the-loop, active learning scenario. fortunately, the decision makers and the analyst grasped the proposal immediately and were eager to help create a sustainable solution. ## conclusion our goal has been to highlight important questions of trust, ethics, and responsibility that have arisen via our experience as third-party providers of ai/ml-based nlp. we have discussed how user engagement and accountability co-evolve with trust as a capability matures from proof-of-concept to production-ready. we conclude by listing a few of the key questions to be posed at various phases of a responsible engagement. by attending to these types of questions from the outset of each engagement and throughout, we strive to maximize successful nlp deployment and to build long term trust in ai/ml and nlp.