Towards Hybrid Human-Machine Workflow for Natural Language Generation (2021)

Abstract: In recent years, crowdsourcing has gained much attention from researchers to generate data for the Natural Language Generation (NLG) tools or to evaluate them. However, the quality of crowdsourced data has been questioned repeatedly because of the complexity of NLG tasks and crowd workers’ unknown skills. Moreover, crowdsourcing can also be costly and often not feasible for large-scale data generation or evaluation. To overcome these challenges and leverage the complementary strengths of humans and machine tools, we propose a hybrid human-machine workflow designed explicitly for NLG tasks with real-time quality control mechanisms under budget constraints. This hybrid methodology is a powerful tool for achieving high-quality data while preserving efficiency. By combining human and machine intelligence, the proposed workflow decides dynamically on the next step based on the data from previous steps and given constraints. Our goal is to provide not only the theoretical foundations of the hybrid workflow but also to provide its implementation as open-source in future work.

## introduction with the rapid development of internet technologies, crowdsourcing has become one of the primary resources to solve tasks such as image tagging, transcribing the text, or digitizing print documents that computers cannot yet solve and need human intelligence @xcite @xcite . further, the cost and time advantages of crowdsourcing have raised the interest of many nlg researchers to generate corpus or to evaluate the quality of nlg outputs @xcite @xcite @xcite . despite the increasing popularity of crowdsourcing, the quality of crowdsourced data has been many times questioned because of crowd worker's potential inaccuracy and the complexity of nlg tasks. as a solution, a variety of workflow approaches have been proposed with the aim of quality assurance, quality control, or cost optimization @xcite @xcite @xcite @xcite @xcite . however, all of these approaches are neither designed explicitly for the given nlg task nor integrate the nlg tools themselves into the workflow dynamically. therefore, we propose an automatic hybrid human-machine workflow that decides on the next step (when to use humans and when to use an nlg tool) based on the given constraints and the previous workflow step, optimizing the cost/quality trade-off. with this hybrid dynamic methodology, we aim to collect high-quality data while preserving efficiency. since this is a work-in-progress, we describe the logic and the theoretical aspects of the workflow in this paper and will provide its complete modeling and practical implementation as open-source in future work. ## related work many crowdsourcing platforms provide support for repetitive independent micro-tasks, which can be completed in a short amount of time @xcite . however, the recent advances of internet technologies require human intelligence for more complex tasks. as a solution, crowdsourcing workflows have been introduced to a variety of problems such as taxonomy creation @xcite , entity resolution @xcite , and complex work @xcite . the main focus of these crowdsourcing workflows are cost/quality optimization, task allocation, modeling the incentive mechanism, or modeling the crowd workers' skills @xcite . with artificial intelligence (ai) systems being an important part of our lives, combining crowdsourcing workflows with ai tools, hybrid intelligence, promise great potential for improving human-only workflows. therefore, researchers have developed intelligent hybrid systems for real-time speech transcribing @xcite @xcite @xcite , clustering data points @xcite @xcite , forecasting political or economic events @xcite @xcite or scheduling conference meetings @xcite @xcite . these hybrid workflows have been proven to perform better than human-only and machine-only systems. however, to this date, there is no hybrid humanmachine workflow that combines the human and machine intelligence with quality control mechanisms for crowd workers and with a methodology for cost/quality optimization. usage of crowdsourcing to nlg has been limited to single crowdsourcing studies for quality evaluation or data labeling for semantic parsing @xcite , information retrieval @xcite , translation @xcite and summarization @xcite @xcite , but the hybrid intelligence approach has not been applied in these works. therefore, we propose to combine the strength of the human-only workflows and nlg tools in the form of a hybrid humanmachine workflow with quality control mechanisms. such an integrative hybrid approach offers great promise for the development of practical applications by achieving high-quality data while preserving efficiency. ## expected contributions and future work the current crowdsourcing platforms offer minimal guidance and support on how to interpret the collected data or how to assure quality. with this hybrid dynamic methodology, we aim to overcome this challenge by providing the logic, modeling, and implementation of a hybrid human-machine workflow for nlg with quality control and cost optimization methods. in this way, the data creation and evaluation can be accelerated for many languages leading to the enhancement of multilingual nlg tools. since many nlg tasks usually require data creation and evaluation steps, the workflow can be adjusted easily to other nlg tasks such as translation, question-answering, or data-to-text. as future work, after completing the modeling and implementation, we plan to run experiments with the proposed workflow to finalize the workflow decision algorithm and test it empirically. to foster the development of the proposed workflow, we welcome researchers in the nlg community to join our experiments, use the proposed workflow to collect data or evaluate their tool, improve the workflow based on empirical data and serve high-quality results for the researchers.