Bridging Multi-disciplinary Collaboration Challenges in ML Development via Domain Knowledge Elicitation (2021)

Abstract: Building a machine learning model in a sophisticated domain is a time-consuming process, partially due to the steep learning curve of domain knowledge for data scientists. We introduce Ziva, an interface for supporting domain knowledge from domain experts to data scientists in two ways: (1) a concept creation interface where domain experts extract important concept of the domain and (2) five kinds of justification elicitation interfaces that solicit elicitation how the domain concept are expressed in data instances.

## introduction in recent decades, machine learning (ml) technologies have been sought out by an increasing number of professionals to automate their work tasks or augment their decision-making @xcite . broad areas of applications are benefiting from integration of ml, such as healthcare @xcite , finance @xcite , employment @xcite , and so on. however, building an ml model in a specialized domain is still expensive and time-consuming for at least two reasons. first, a common bottleneck in developing modern ml technologies is the requirement of a large quantity of labeled data. second, many steps in an ml development pipeline, from problem definition to feature engineering to model debugging, necessitate an understanding of domain-specific knowledge and requirements @xcite . data scientists therefore often require input from domain experts to obtain labeled data, to understand model requirements, to inspire feature engineering, and to get feedback on model behavior. in practice, such knowledge transfer between domain experts and data scientists is very much ad-hoc, with few standardized practices or proven effective approaches, and requires significant direct interaction between data scientists and domain experts. building a high-quality legal, medical, or financial model will inevitably require a data scientist to consult with professionals in such domains. in practice, these are often costly and frustrating iterative conversations and labeling exercises that can go on for weeks and months, which usually still do not yield output in a form readily consumable by a model development pipeline. in this work, we set out to develop methods and interfaces that facilitate knowledge sharing from domain experts to data scientists for model development. we developed a domain-knowledge acquisition interface ziva (with zero knowledge, how do i develop a machine learning model?). instead of a data-labeling tool, ziva intends to provide a diverse set of elicitation methods to gather knowledge from domain experts, then present the results as a repository to data scientists to serve their domain understanding needs and to build ml models for specialized domains. ziva scaffolds the knowledge sharing in desired formats and allows asynchronous exchange between domain experts and data scientists. it also allows flexible re-use of the knowledge repository for different modeling tasks in the domain. specifically, ziva focuses on eliciting key concepts in the text data of a domain (concept creation), and rationale justifying a label that a domain expert gives to a representative data instance (justification elicitation). in the current version of ziva, we provide five different justification elicitation methods -bag of words, simplification, perturbation, concept bag of words, and concept annotation.