F orm LM : Recommending Creation Ideas for Online Forms by Modelling Semantic and Structural Information (2022)

Abstract: Online forms are widely used to collect data from human and have a multi-billion market. Many software products provide online services for creating semi-structured forms where questions and descriptions are organized by predefined structures. However, the design and creation process of forms is still tedious and requires expert knowledge. To assist form designers, in this work we present FormLM to model online forms (by enhancing pre-trained language model with form structural information) and recommend form creation ideas (including question / options recommendations and block type suggestion). For model training and evaluation, we collect the first public online form dataset with 62K online forms. Experiment results show that FormLM significantly outperforms general-purpose language models on all tasks, with an improvement by 4.71 on Question Recommendation and 10.6 on Block Type Suggestion in terms of ROUGE-1 and Macro-F1, respectively.

## introduction online forms are widely used to collect data in everyday scenarios such as feedback gathering @xcite , application system @xcite , research surveys @xcite , etc. with a multi-billion market @xcite , many software products -such as survey monkey @xcite , google @xcite and microsoft forms (rhodes, 2019)provide services to help users create online forms which consist of multiple blocks (e.g., figure 1 ). however, there are obstacles preventing the creation of well-designed online forms, which could hurt response rate and quality @xcite . for each form question, form designers need to write an informative title, specify its type, and provide other required components. this process is tedious and time-consuming even for experienced users. also, non-experts may be unsure about what question to add or which question type to choose. to improve the experience and efficiency of form composing, it is desirable that online form services could recommend creation ideas to form designers. to address the above demands, in §3 we identify three machine learning (ml) tasks of form creation ideas, including question recommendation, block type suggestion, and options recommendation. for example, in figure 1 , when one adds a text field block as the second block, the question recommendation suggests "employee id" for the question based on the existing content (form title, description, and the first question "full name"). when editing the third choice question block, the options recommendation suggests "yes" and "no" as candidate options. finally, if the user types "how happy are you with your current job?" for the fourth block but hasn't selected a block type yet, the block type suggestion predicts it as a rating type block. the above tasks require a specifically designed model to understand semi-structured forms, where natural language (nl) text is organized by predefined structures. a form is composed of a title, a description, and a series of blocks. for each block, its subcomponents also follow unique structures. for example, a choice block contains a list of options which serve as candidate answers to the question displayed in the block title. existing pre-trained language models (plms) focus on general-purpose free-form nl text @xcite . they may provide a good starting point to model the rich semantic information within nl contents of a form. however, they cannot directly handle the extra structural information of the form. is it possible to infuse a plm with structural information of online forms? in this paper, we propose formlm to model both the semantic and structural information of online forms. as we will discuss in §4, there are three key parts of formlm. first, the form serialization procedure, which represents a form as a tree and converts it into a token sequence without information loss. second, inheriting existing plm with a small number of additional parameters: formlm inherits the parameters of bart @xcite to leverage its language modelling capabilities. also, by adding extra biases to the attention layers, formlm explicitly handles the structural information. third, continual pre-training with collected online forms: for better downstream application: we propose two structure-aware objectives -span masked language model and block title permutation -to continually pre-train formlm on top of the inherited and additional parameters. we evaluate formlm on form creation ideas tasks using our oof (open online forms) dataset. this dataset (see §2.2) is created by crawling and parsing public forms on the web. comparing to plms such as bart, formlm improves the rouge-1 score from 32.82 to 37.53 on question recommendation, and the macro-f1 score from 73.3 to 83.9 on block type suggestion. in summary, our main contributions are: ## preliminaries in this section, we further elaborate the predefined structure in online forms, and introduce our collected dataset. ## form creation ideas as illustrated in figure 1 , when adding a new block, one needs to specify its type and title in the first step. then, other required components -such as a list of options for a choice block -are added according to the block type. in this paper, we focus on the following three tasks which provide form creation ideas to users in the first and later steps. question recommendation the question recommendation aims at providing users with a recommended question based on the selected block type and the previous context. formally, the model needs to predict title i based on t , desc, b 1 , ..., b i-1 and type i . for example, in figure 1 , it is desirable that the model could recommend "employee id" when the form designer creates a text field block after the first block. block type suggestion different from the scenario of question recommendation, sometimes form designers may first come up with a block title without clearly specifying its block type. the block type suggestion helps users select a suitable type in this situation. for example, for the last block of figure 1 , the model will predict it as a rating block and suggest adding candidate rating scores if the form designer has not appointed the block type himself / herself. formally, given title i and the available context (t, desc, b 1 , ..., b i-1 ), the model should predict type i in this task. options recommendation as figure 2 shows, choice blocks are frequently used in online forms. when creating a choice block, one should additionally provide a set of options, and the options recommendation helps in this case. given the previous context (t, desc, b 1 , ..., b i-1 ) and title i , the model predicts c (1) i , ..., c in this work, we expect the model to recommend a set of possible options at the same time, so the desired output of this task is c (1) i , ..., c (n i ) i concatenated with a vertical bar. for example, in figure 1 , the model may output "yes | no" to recommend options for the third block. ## methodology as discussed in §1, we propose formlm to model forms for creation ideas. we select bart as the backbone model of formlm because it is widely used in nl-related tasks and supports both generation and classification tasks. in the rest of this section, we will describe the design and training details of formlm as demonstrated in figure 3 . ## conclusion in this paper, we present formlm for online form modeling. formlm jointly consider the semantic and structural information by leveraging the plm and designing form serialization and structural attention. furthermore, we continually pre-train formlm on our collected data with structure-aware objectives for better domain adaptation. an extensive set of experiments show that formlm outperforms baselines on form creation ideas tasks which assist users in the form creation stage.