C hat GPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustness (2023)

Abstract: The emergence of generative large language models (LLMs) raises the question: what will be its impact on crowdsourcing? Traditionally, crowdsourcing has been used for acquiring solutions to a wide variety of human-intelligence tasks, including ones involving text generation, modification or evaluation. For some of these tasks, models like ChatGPT can potentially substitute human workers. In this study, we investigate whether this is the case for the task of paraphrase generation for intent classification. We apply data collection methodology of an existing crowdsourcing study (similar scale, prompts and seed data) using ChatGPT and Falcon-40B. We show that ChatGPT-created paraphrases are more diverse and lead to at least as robust models.

## introduction crowdsourcing has been an established practice for collecting training or validation examples for nlp tasks, including the intent classification (i.e. determining the purpose behind a phrase or sentence). when crowdsourcing intent examples, workers typically create new phrases for some scenario @xcite . however, a data augmentation approach can also be used: by workers paraphrasing existing sentences with known intents. the aim is to increase the data diversity and subsequent model robustness. in particular, diversity can increase considerably with the use of taboo words that force workers to be more creative @xcite . without data diversity, models can lose their ability to generalize well @xcite @xcite . unfortunately, crowdsourcing has several downsides: 1) the workforce is costly, 2) output quality is difficult to achieve (which translates to further costs), and 3) there are overheads related to the design and organization of the process. the advent of generative large language models (llms) and chatgpt in particular opens up the possibility of replacement of crowd workers by ai. llms have been already investigated for a variety of nlp tasks: translation @xcite , question answering, sentiment analysis, or summarization @xcite , displaying strong performance against fine-tuned models. indeed, some crowd workers have already started to exploit llms to replace themselves @xcite . with this paper, we investigate whether chat-gpt can replace crowd workers in paraphrase generation for intent classification dataset creation. we answer the following research questions: to answer these questions, we have conducted a quasi-replication of an existing study @xcite , where paraphrases were crowdsourced to augment intent classification data (using also taboo words technique to induce more example diversity). in our own study, we followed the same protocol (seeds, taboo words, scale), but replaced the crowd workers with chatgpt (for approximately 1:600 lesser price) as the most widely used llm and falcon-40b @xcite as one of the best performing open source llm at the time of writing this paper. this allowed us to directly compare properties of crowd-and llm-generated data, with following findings: 1) chatgpt is highly reliable in generating valid paraphrases, 2) falcon-40b struggled in generating valid and unique paraphrases, 3) chatgpt outputs lexically and syntactically more diverse data than human workers, and 4) models trained on chatgpt data have comparable robustness to those trained on crowd-generated data. crowdsourcing of paraphrases is used for creating new datasets for various nlp tasks @xcite . in paraphrase crowdsourcing, the process typically entails showing of an initial seed sentence to the worker, who is then asked to paraphrase the seed to new variations @xcite . as training sample diversity is correlated with robustness of downstream models @xcite @xcite , various diversity incentives are used to encourage crowd workers to create more diverse paraphrases. in some approaches, workers are hinted with word suggestions @xcite . syntactic structures can also be suggested @xcite ). another approach is chaining, where paraphrases from previous workers are used as seeds @xcite . yet another technique is the use of taboo words, where users are explicitly told to avoid certain phrases. previous research has shown, that taboo words lead to more diverse paraphrases and more robust models @xcite . yet, despite all the advances, crowdsourcing remains expensive for dataset building. seq2seq models llms have already found their use for paraphrase generation from existing corpora (gpt-2 @xcite , bart @xcite ). llms can also be used to paraphrase using style transfer @xcite (resulting paraphrases adhere to a certain language style). to increase paraphrase diversity, syntax control approaches can be used @xcite . promising results have been achieved in zero-shot settings to produce paraphrases in a multi-lingual domains @xcite , or finetuning using only prompt tuning and low rank adaptation @xcite . these previous works showed the capabilities of llms to produce diverse and valid paraphrases, albeit with various syntax or semantic restrictions and additional mechanisms needed to produce good results. in a number of recent studies, chatgpt foot_0 has been applied to a variety of tasks. it has been used as a mathematical solver where it achieves below average human performance @xcite , sentiment analysis task with varying performance @xcite and also as a general nlp task solver where it shows good performance on reasoning tasks @xcite . another study @xcite measured chatgpts capabilites to evaluate outputs of several natural language generation tasks, achieving performance of human evaluators. chatgpt performs well in translation of high resource languages @xcite . as for some other typical crowdsourcing tasks, chatgpt outperforms the crowd in classification of political affiliation of twitter users (törnberg, 2023) and in-text annotation tasks @xcite . when (zero shot) chatgpt is compared with a finetuned bert, chatgpt falls short in detection of paraphrases and text similarity tasks, but performs well on sentiment analysis, qa and inference @xcite . these results indicate good capabilities of chatgpt for nlp tasks, and its potential to replace crowd workers in at least some tasks @xcite . ## chatgpt paraphrase validity and diversity to test whether chatgpt can be an alternative to crowd workers in paraphrase generation, we replicated the data collection process of @xcite , who crowdsourced paraphrases to create datasets for the intent classification task. in their study, workers created paraphrases of seed sentences (with known intents). we replicate their process using chatgpt and show that chatgpt generated data are more diverse both in terms of lexical and syntactical diversity than those created by humans. ## model robustness to compare the robustness of models trained on chatgpt and human paraphrases (rq3), we computed accuracy of these models over out-ofdistribution (ood) data. the comparison indicates higher or comparable accuracy of chatgpt-datatrained models. ## cost comparison: chatgpt vs crowdsourcing? we compare the crowdsourcing costs of the original study with our chatgpt replication and we do this for both the diversity experiments and for the model robustness experiments on ood data. in all of the experiments the authors of the original study estimate the cost of one paraphrase to be at @xmath00.002 per 1k tokens. for diversity experiments, the number of collected paraphrases from the original study is 10,050 samples. this results in an approximate cost of the original study at approximately @xmath10.5. this is a 1:1000 ratio for chatgpt when both studies are compared. for the model robustness experiments, the total data size from the original study for all the 5 benchmark datasets combined is 13,680 samples, which corresponds to an approximate cost of the original study at @xmath22.5. this results in an approximate ratio of 1:525 in favor of chatgpt. together, both experiments make up the price ratio of 1:600 for chatgpt. we conclude that the price of using chatgpt for collecting paraphrases for intent classification is much lower than with crowdsourcing. ## discussion given the results of our experiments, we perceive chatgpt as a viable alternative to human paraphrasing. however, we also will not argue for a complete replacement yet (for that, many more investigations are needed). chatgpt, while generally powerful, still has some weaknesses. during the manual validation of chatgpt generated data we noticed that chatgpt does not change named entities for their acronyms (or vice versa) or for their other known names. for example, the word 'ny', even though it refers to 'new york' in the context of the sentence, will always remain the same in all the paraphrases generated by chatgpt. this would indicate that while chatgpt is capable of producing very rich sentences in terms of lexical and syntactical diversity (as seen in section 3.3), it does not produce alternative names for named entities, e.g. locations, songs, people, which is something crowdsourced data handles well. the tendency of chatgpt to produce "odd" paraphrases also has potential implications: the divergence from typical human behavior can harm many applications. on the bright side, oddity can indicate easier machine-generated text detection, which may serve (crowdsourcing) fraud detection. figure 2 : the evaluation process on out-of-distribution (ood) data for one dataset on bert-large, same process was repeated for svm. this process has been repeated for all the datasets for a comparison of robustness for models trained on chatgpt and human collected data. ## conclusion in this work, we compared the quality of crowdsourced and llm generated paraphrases in terms of their diversity and robustness of intent classification models trained over them. we reused the data collection protocol of the previous crowdsourcing study of @xcite to instruct chatgpt and falcon-40b instead of crowd workers. our results show that chatgpt collected data yield higher diversity and similar model robustness to the data collected from human workers for a fraction of the original price, while falcon-40b struggled in creating valid and unique paraphrases. the effects of human-collected and chatgpt-collected data on robustness vary, which might indicate the need for their combination for best model performance. the much cheaper chatgpt data collection provided us with a better dataset (in many aspects). given our observations, it does appear beneficial to us to use chatgpt as a supplement to crowdsourcing for paraphrase collection. we do not rule out the use of human computation for this task, but at the same time, we speculate that the usage of chatgpt as a data enhancement method might be beneficial for dataset building. our results lead to new questions about chat-gpt's capabilities for the creation of paraphrases. a possible point of diminishing returns for this paraphrasing task in chatgpt might be explored in order to determine when too many duplicates are created from chatgpt. the usage of different diversity incentives on chatgpt could further improve the performance of chatgpt. another area of interest might be the usage of chatgpt paraphrasing for low-resource languages, where further investigation to the quality of created data is needed.