Source: {"pile_set_name": "USPTO Backgrounds"}

Categorizers are often used in data mining applications, where data contained in a database (or multiple databases) is analyzed and used for various purposes (such as to determine customer purchasing habits and preferences or for other purposes). A categorizer looks at a data item (e.g., article, product, customer, stock, support call, and so forth), and decides, based on information associated with the item (e.g., text, cost, date, duration, buying history, trade volume, and so forth), whether the item should be associated with a particular category or multiple categories. The categorizer decides which of a set of potential categories most likely applies, describes, or should be associated with the data item in question. A “category” refers to a label, annotation, or some other form of identifying indicator.
A categorizer has to be trained to enable the categorizer to perform categorization tasks. Various conventional algorithms exist to train categorizers. The categorizer implemented according to a given algorithm is associated with a set of parameters that the categorizer uses to make its decision. Often, the parameters are in the form of weights attached to the presence or absence of various features (words, purchases, and so forth). The parameters can be set manually (e.g., by an expert creating categorization rules) or deduced automatically from data. Most conventional techniques for training categorizers rely on the notion of a labeled training set, which is a set of training cases each annotated with a category (or multiple categories). The annotated category, or categories, constitutes the “correct answer” for each training case. The correctness of the categorizer is judged based on the errors the categorizer makes with respect to the categorizations of cases in the training set. Often the correctness will be judged based on the errors the categorizer makes with respect to a subset of the cases in the training set, the subset (called a “validation set”) not being used otherwise in training the categorizer.
With many conventional approaches, training a categorizer involves two separate stages. In a first stage, a training set is acquired. Then, the training set is used to train the categorizer. To acquire a training set, several approaches are used. A first approach involves acquiring training sets where the “correct answer” can be ascertained at some point after a prediction is made. However, this training approach may not be useful for circumstances where the correct answer cannot be objectively ascertained at some later point in time.
A second approach to acquire a training set involves obtaining desired information about a subset of observed cases by asking people to divulge information about themselves, such as purchasing information or other information. In one example, people can be asked to complete surveys. Alternatively, the information can be purchased from a third party or some other data source, or by physically measuring an object or plural objects. This second approach is usually expensive in terms of time and effort. Additionally, if surveys or other techniques of acquiring information are not designed properly, the training set acquired may not provide the desired level of accuracy in training categorizers.
In an approach used in the customer service call center context for acquiring a training set, customer support representatives are asked to choose categorizations (also referred to as issue paths) from a set of possibilities at the time that a customer call occurs. However, many customer support representatives may not be properly trained to recognize all possible answers, so that the categorization performed by such customer support representatives may not be accurate. Also, it may be expensive to train people to properly recognize all possible categorizations. Additionally, customer support representatives may not be properly motivated to spend the time to provide accurate categorizations, since the performance of customer support representatives may be measured on how quickly they resolve calls, not on how accurately they categorize the calls. Also, the set of possibilities may not include a category that accurately describes the customer call.
Another approach for acquiring a training set involves asking an expert to provide a correct answer for a randomly drawn subset of cases. The expert usually has to be prepared to distinguish among all possible categories (dozens or even hundreds of possible categories) to find the correct answer for each particular case. Also, an expert may not be qualified to recognize cases for certain categories. In addition, an expert may have to look at a large number of cases to accurately find a sufficient number of examples for certain types of categories. This training approach can be relatively expensive and may not produce sufficiently accurate results for training categorizers.