Knowing-how & Knowing-that: A New Task for Machine Comprehension of User Manuals (2023)

Abstract: The machine reading comprehension (MRC) of user manuals has huge potential in customer service. However, current methods have trouble answering complex questions. Therefore, we introduce the knowing-how & knowing-that task that requires the model to answer factoid-style, procedure-style, and inconsistent questions about user manuals. We resolve this task by jointly representing the sTeps and fActs in a gRAh (TARA), which supports a unified inference of various questions. Towards a systematical benchmarking study, we design a heuristic method to automatically parse user manuals into TARAs and build an annotated dataset to test the model’s ability in answering real-world questions. Empirical results demonstrate that representing user manuals as TARAs is a desired solution for the MRC of user manuals. An in-depth investigation of TARA further sheds light on the issues and broader impacts of future representations of user manuals. We hope our work can move the MRC of user manuals to a more complex and realistic stage.

## introduction user manuals are supposed to be helpful in using products, getting involved in promotions, or other goals of interest @xcite @xcite . though well-designed and instructive, they are seldom read by users because "life is too short to read manuals" @xcite . towards high user satisfaction, professional customer service representatives (csrs) are hired to do the reading and answer user questions about the manuals. the machine reading comprehension (mrc) of user manuals thus has huge potential, as it would not only reduce high labor costs but enable the service ready for customers 24/7. let's see how a csr approaches various questions given the user manual 1 in figure 1 . specifi-figure 1 : a user manual and three qa cases about it cally, she can answer q1 by looking up the arguments of facts, i.e., the value ( 100% ) of the hit rate. for q2, she needs to reason about the steps in the procedure as the answer is not explicitly exhibited in the user manual -the user manual directs the user to "sign in" and "scan", and eventually to the "payment page". after that, the subsequent description does not direct the user anywhere else. a user can infer from the procedure that one can scratch the card after immediately getting it on the payment page. therefore, the location to scratch the card is on the payment page . the most difficult one is q3, which raises a case inconsistent with the descriptions in the user manual -going along with the user manual (s-4), the user didn't get the scratch card. we call this type of question an inconsistent question. a possible response to the inconsistent question may be an answer (r1) or a high-utility question leading to an answer (r2-r4). towards such responses, the csr needs to reason about not only the steps but the facts binding the steps -the inconsistency may be caused by incorrect operations in the previous step (e.g., the user didn't pay by scanning the qr code ) or unsatisfiable constraints in the facts (e.g., the user already had 10 scratch cards ). therefore, aligned with the csr, a practical mrc model must also be able to draw a unified inference across steps and facts to properly answer various user questions. with a splendid library of studies @xcite , current mrc models have shown strong power to answer factoid-style questions (e.g., q1). more recent studies @xcite @xcite have mitigated the weakness of factoid-style mrc on procedure-style questions. however, these studies mainly focus on questions about the states of entities that can be answered by single spans. they have trouble answering more complex procedurestyle questions with multiple-span answers or about the dynamics of actions (e.g., q2), let alone inconsistent questions like q3. we also notice that existing models are only designed for single types of questions. this doesn't meet the real-world scenarios where factoid-style, procedure-style, and inconsistent questions are all involved (cf., to address these limitations, we propose the knowing-how & knowing-that task foot_0 -given a user manual, the model is required to answer factoid-style, procedure-style or inconsistent questions about it. we resolve this task by jointly representing the steps and facts in a heterogeneous graph (tara), cf., figure 2 . this representation allows us to make the unified inference of various questions. we further benchmark the proposed task with a heuristic method (hum) to automatically represent user manuals as taras and a denselyannotated dataset (oho) to test the model's ability in answering real-world questions. specifically, hum is designed to be independent of labelled data with the ultimate goal of building industryscale applications and particular attention to low costs and good generalization capability. the annotated dataset (oho) is derived from online help documents and real-world user questions from an e-commerce company. we provide the gold answers to user questions and gold taras of user manuals. experiments demonstrate the superiority of tara, the efficiency of hum, and the significant challenges of oho. with an in-depth investigation of tara, we discuss the issues and broader impact. we expect our work can advance the mrc of user manuals and inspire future research on smart customer service. we highlight our work as follows: datasets for user manuals we have done a thorough survey (cf., for user manuals. the types of user manuals include recipes @xcite @xcite @xcite , technical manuals (e.g., device maintenance, surgical practices) @xcite @xcite , and scientific experiment @xcite @xcite . there is a lack of a dataset about ecommerce scenarios where the mrc of user manuals is urgently needed due to the high labor costs for customer service. besides, we have noticed that existing annotations are all about user manuals. none of them has offered real-world qa pairs to test the practical performance of mrc models. to address these limitations, we collect the oho dataset with online helping documents from an e-commerce company. we then contribute annotations for datadriven evaluations of the model's ability to answer real-world questions and represent user manuals. ## representation of user manuals definitions we wish to design a representation for user manuals that supports the unified inference of various questions. it can be reduced to two fun-damental tasks: how to extract actions (entities) and their arguments from the user manual, and what is the relation between an action and an entity (two actions or two entities). this motivates us to propose the tara representation, which jointly describes steps and facts in a heterogeneous graph with two sets of nodes, three sets of arguments, and five sets of relations, as defined in ## the hum method towards practical industry-scale applications, it requires extra attention to labor costs and generalization. thus, we choose to design an efficient method based on pre-trained semantic dependency parsing tools and domain-independent heuristics. first, we segment the user manual sentence by sentence and feed the sentences into a pre-trained semantic dependence parsing tool. if a sentence is imperative, we leverage "user" as the subject of the sentence. for instance, "sign in the app" is modified as "user sign in the app". then, we eliminate the vertices to compose more meaningful phrases -the offsprings of a predicate's child are eliminated and combined with the child. an example foot_3 is shown in figure 3 . in addition, to maintain the leading role of the user, if the user is the patient of a predicate, we change its role with the agent and mark the predicate as "reverse". second, we use the active verbs, whose agents are the user, as the action nodes. these nodes are linked with the "next" relations based on their order in the user manual and semantic dependencies. besides the user node created by default, other entity nodes are initialized as the unique patients with the user as agents foot_4 . the relation from the user node to any action node is "agt". we also add an edge with the "pat" relation from an action node to the entity node, whose value serves as the patient of the action node. particularly, if an entity has attributives, we create a new entity with the "sub" relation to it by combining the attributives. the "same user" node is created in this way. most of action-arg, entity-arg, and arg-arg are generated by the alignment with the tags from off-the-shelf tools like ltp @xcite , hanlp (he and choi, 2020), etc. the alignment rules are defined in automated inference for qa to infer the answers to user questions, we leverage the hum method to represent the user manual and user question, simultaneously. then, we extract the subgraph that is most similar to the representation of the user question via the sub-graph matching algorithm @xcite . the extraction only considers the nodes in the graph. the similarity score between two nodes (x and y) is computed by s(x, y) = 1 -d(x, y)/ max(|x|, |y|), (1) where d computes the levenshtein distance, and | * | is the length of * . after getting the sub-graph matched to the user question, we do the inference described in section 3 to get the answer. ## the oho dataset we gather the frequently asked questions (faqs) summarized by the customer service department of an e-commerce company. we remove faqs that aren't attached to any documents and that are attached to documents with only images. finally, answer annotation each annotator is shown an faq and its attached user manual, which is displayed sentence by sentence. a toy example for the annotation of q3 is shown in figure 4 . the annotator is required to select which sentences in the user manual can be possible responses to the question and then select the type of the question. each sentence is treated as a text span in the answer. the statistics of the annotations are described in it reveals that the procedure-style and inconsistent questions take up more than half part (52.45%) of the real-world user questions about user manuals. representation annotation this annotation is done on 200 user manuals of oho. as some user manuals describe more than one task, there are 346 graphs that need to be annotated. it is not practical for the annotator to draw a heterogeneous graph for each user manual. following @xcite , we reduce the task as the annotation of answers to nine basic questions derived from quality control we employ three csrs with at least one-year working experience from the same department that summarizes the faqs. we give a training session to the workers to help them fully understand our annotation requirements. the annotation tasks are released to the workers via an online annotation platform managed by the e-commerce company. on average, it takes about 3 minutes for an annotator to annotate the answer to an faq and about 2.8 hours to annotate all answers to basic questions of a user manual. an annotated result is accepted only if all three workers agree; otherwise, we invite two experts (one is the manager of the customer service department and the other is a postdoc working on computational linguistics) to make the final decision. the experts closely discuss with each other to ensure the consistency of results. ## experiments to systematically benchmark the knowing-how &knowing-that task evaluation, we perform two experiments based on the oho dataset. first, with the help of answer annotations, we compare the performances of hum with seven baselines in answering real-world user questions, i.e., the faq-answering task. second, with the help of representation annotations, we investigate the maximum potential of hum, i.e., the b-answering task. specifically, we use the ltp tool @xcite for word segmentation and semantic dependency parsing. ## conclusion and future work we propose the knowing-how & knowing-that task that requires the model to answer various questions about a user manual. we resolve it by representing the steps and facts of a user manual in a unified graph (tara). to benchmark it, we design an efficient method and annotate a testing set derived from real-world customer service scenarios. experiments reveal the superiority of tara, the efficiency of hum, and the significant challenges of oho and the new task. we take the primary step to study the mrc of user manuals in a more challenging setting, where various questions are involved. we hope our work can benefit further research on smart customer service. there are several directions for further work. first, to improve the performance of hum, we will research resolutions to accumulative errors, co-reference problems, complex discourse parsing beyond sentences, etc. second, to explore the potential of the proposed representation, we will introduce it to other tasks that need better interpretability, like task-oriented information seeking. third, we will investigate the potential of unified inference for more complex user manuals, e.g., user manuals with multiple agents. lastly, we plan to deploy hum to online customer service to gain more insights for further improvements.