Interactive Construction of User-Centric Dictionary for Text Analytics (2020)

Abstract: We propose a methodology to construct a term dictionary for text analytics through an interactive process between a human and a machine, which helps the creation of flexible dictionaries with precise granularity required in typical text analysis. This paper introduces the first formulation of interactive dictionary construction to address this issue. To optimize the interaction, we propose a new algorithm that effectively captures an analyst’s intention starting from only a small number of sample terms. Along with the algorithm, we also design an automatic evaluation framework that provides a systematic assessment of any interactive method for the dictionary creation task. Experiments using real scenario based corpora and dictionaries show that our algorithm outperforms baseline methods, and works even with a small number of interactions.

## introduction since the emergence of practical interests in text analytics that finds insights from massive documents @xcite , there are several requirements for enhancing valuable discoveries. the one critical issue we tackle in this paper is an effective construction of a term dictionary @xcite . the term dictionary, which is an arbitrary set of terms, is used in text analytics to represent interesting analysis perspectives @xcite ; for example, dictionaries of "product names" and "evaluative description" are required for mining customer reputations about products. the motivation of this paper is how to reduce the human workload for the dictionary construction as much as possible. to this end, we establish a methodology of interactive dictionary construction that incrementally captures an analyst's intention starting from a small number of sample terms and enables him/her to effortlessly expand terms in the intended dictionary through suggestions by a machine. the term dictionary for text analytics is expensive to be constructed because we need to focus more on terms with flexible granularity for in-depth analysis @xcite @xcite . for instance, if the analyst wants to examine product evaluation from both its function and appearance, he/she then needs to separately create those dictionaries whose boundaries are vague and overlapped (figure 1 ). in short, we need to group any terms the analyst wants together depending on documents and the objective of analysis, which forces an ad hoc construction of the term dictionary. this situation is rather severe in the real-world tasks because the vocabulary size for an exhaustive search of the texts is vast, and the analyst will go through re-peated trial and error of creating dictionaries until he/she reaches findings. at present, there is a demand for a machine that decreases the cost of the ad hoc dictionary construction. as the dictionary construction can be considered as a type of collecting terms, there is a related research field -set expansion that expands a small set of terms by means of bootstrapping @xcite . this approach automatically finds new terms for the given set from documents in accordance with a predefined exploration strategy @xcite . although such an automatic procedure is advantageous for reducing the human workload, the quality of the collected terms is suspicious for a term dictionary. for example, a good analysis requires more fine-grained dictionaries than the original targets in set expansion such as distinct ontological terms (e.g., country name, @xcite . several studies have incorporated a human in the term collection process @xcite . specifically, dictionaries are built in an interactive process where the human gives feedback to the machine and the machine suggests candidates based on the given feedback @xcite . such a humanin-the-loop approach has been an active topic in other fields as well, for instance, image classification @xcite , dialogue system @xcite , and audio annotation @xcite . we can generally expect that a reliable feedback provided by human makes a system more accurate. with respect to dictionary construction, however, experimental results in this vein are limited due to the empirical evaluation by just a few participants and the use of a coarse dictionary as the test items. in short, it is a still open question -what is a critical issue for interactive construction of fine-grained term dictionary for text analytics? moving in the same promising direction of leveraging both a human and a machine, we establish a well-defined and effective methodology for constructing the term dictionary. in summary, our contribution in this paper is fourfold: (i) we formulate the interactive process of a term collection, which brings clarity to the problem to be solved ( §2). (ii) we develop a method that captures an analyst's intention from a small number of samples with our formulation as the basis ( §3). (iii) we propose an automatic evaluation framework that provides a systematic assessment for interactive methods ( §4). (iv) our experimental results show that the proposed method surpasses baseline methods such as set expansion, word embedding and a linear classifier on the crowdsourced dataset. the dataset emulates the real-world scenario of flexible and fine-grained dictionary construction, and we distribute the dataset to the public ( §5). ## task definition in this section, we provide the definitions and notations used throughout this paper. first, a term is a string representation of a certain notion such as "apple" and "new york". a dictionary is a collection of terms. a user denotes the person who wants to construct a dictionary, and system denotes the machine that helps the user. let w be the whole set of terms in documents. our objective is to rapidly find as many terms of the user's interest u ⊂ w as possible. as seen in figure 2 , interactive dictionary construction is defined as an iterative process in which each iteration consists of the following steps: 1) user feedback in which the user selects terms for the dictionary from the current candidate terms, and 2) candidate selection in which the system finds candidate terms for the next user feedback. for the i-th iteration (@xmath0, 1, 2, . . .), let c i be the set of terms that the system finds in the candidate selection step and u i be the set of terms that the user selects from c i-1 in the user feedback step as positive examples. here, u 0 is a special feedback we call seed terms that are directly given by the user first. note that, because we wish to expand the dictionary, each term in c i should be new to the user in the (i + 1)-th iteration. in the i-th step of the user feedback (@xmath1, we assume that the user can annotate which terms in c i-1 are in u without being aware of the whole 𝑊 𝑊 𝑊 𝑈 𝑈 # 𝑈 𝑈 𝑈 # 𝐶 # 𝐶 # 𝑈 # 𝑈 % set seed term(s) (user feedback) candidate selection user feedback (𝑈 # ∪ 𝑈 % = ( 𝑈 % ) figure 3 : task definition u . so, u i ⊂ u for each i. let u i := ∪ i @xmath2. however, it is impractical to define our objective as an optimization problem for the asymptotic convergence of u i because the user feedback is done by a human, and i cannot be large. hence, we try to maximize |c i ∩ u |, the number of suggested terms that match the user's interest. also, since c i is manually selected by a human user, the proper size of c i is practically limited to 5 ∼ 10. figure 3 shows the steps from setting the seed terms to giving the first feedback to the first candidates. using the example in figure 2 , u 0 is {formal}, c 0 is {nice, traditional}, u 1 is {traditional}, and c 0 \u is {nice}. the system then next selects c 1 based on u 0 and u 1 (i.e., u 1 ) from w except for the shown terms c 0 ∪ u 1 . it is important that we design the system to be effective so that the overlapped area of c i and u becomes larger. there are two major challenges for this problem; one is number of seed terms, and the other is term overlaps of different dictionaries. in terms of the first issue, we have only a few seed terms for the target dictionary at the first iteration. if the system requires more seed terms, the advantage of the system drops because it contradicts our purpose to decrease the human workload in constructing the dictionary. therefore, we need a method that captures the user's intention from a smaller number of samples. in terms of the second issue, identifying terms of user's interest is difficult because boundaries between dictionaries are often overlapped in text analytics as seen in figure 1 . in other words, the system need to be more sensitive to subtle semantic differences only with a few feedbacks. ## method in this section, we first describe a previous candidate selection model, setexpan algorithm @xcite that inspired our method ( §3.1). subsequently, we introduce our method as the weighted version of setexpan with improvements in dealing with interactive settings ( §3.2∼). throughout this section, we discuss the i-th step of candidate selection for a certain i. for simplicity, c i and u i are denoted as c and u , respectively. ## evaluation framework in this section, we explain an automatic evaluation framework for interactive dictionary construction. by using a predefined dictionary as the oracle dictionary u * , we emulate the manual feedback process and apply a new evaluation metric to estimate the effectiveness of building a dictionary with consideration of the human interaction. ## experiments we conduct an experiment following the automatic evaluation framework by using public datasets and oracle dictionaries created through crowdsourcing. in the experiment, we compare several methods in addition to our proposed method. as emulation parameters, we set number of seed terms (|u 0 | ), the number of terms in one suggestion (|c|), and number of total iterations (i) to 3, 10, and 30, respectively. note that we tried different numbers of seeds (1 and 5), but the overall tendencies were the same. ## conclusion to the best of our knowledge, this paper proposes the first formulation of interactive dictionary construction for text analytics, which clarifies the critical issues to resolve. in response to those issues, we provide the method, the evaluation framework, and the experimental dataset. also, our experimental results show the promising performances of our method in concern with real situations of text analytics. our systematic study will pave the way to future research about the effective construction of dictionaries for text analytics.