The Shifted and The Overlooked: A Task-oriented Investigation of User- GPT Interactions (2023)

Abstract: Recent progress in Large Language Models (LLMs) has produced models that exhibit remarkable performance across a variety of NLP tasks. However, it remains unclear whether the existing focus of NLP research accurately captures the genuine requirements of human users. This paper provides a comprehensive analysis of the divergence between academic research in NLP and the needs of real-world NLP applications via a large-scale collection of user-GPT conversations. We analyze a large-scale collection of real user queries to GPT. We compare these queries against existing NLP benchmark tasks and identify a significant gap between the tasks that users frequently request from LLMs and the tasks that are commonly studied in academic research. For example, we find that tasks such as “design” and “planning” are prevalent in user interactions but largely neglected or different from traditional NLP benchmarks. We investigate these overlooked tasks, dissect the practical challenges, and provide insights toward a roadmap to make LLMs better aligned with user needs.

## introduction over the past years, the nlp community has witnessed several paradigm shifts in technology followed by renewed research focus on applications that test the limits of this technology @xcite . for example, distributed word representations @xcite enabled a better characterization of the semantic similarity between words, entailing nlp research gravitated towards tasks like sentiment analysis and dependency parsing @xcite . subsequent technologies like the transformer architecture @xcite and contextual word representations @xcite further expanded the space of possible applications and the edge of nlp research, such * work partially done during internship at microsoft. as machine translation @xcite and document summarization @xcite . most recently, large language models (llms) @xcite such as chatgpt, emerged as powerful tools capable of achieving unprecedented success across a broad spectrum of nlp tasks @xcite @xcite . these models have become accessible and popular among non-nlp experts, opening the door for many new user applications. the flood of new applications and the sharing of user interactions with llms @xcite provide a great opportunity to closely examine the distribution of real applications users need on a daily basis. after a detailed analysis, we identify a conspicuous gap between real-world user queries and established nlp benchmarks, suggesting another shift in nlp focus is needed. to systematically analyze the phenomenon and to bridge the gap, we conduct a battery of experiments aiming to examine the following aspects: we start by investigating sharegpt foot_0 , a largescale collection of user-gpt conversations in the real world, containing 94,145 split data samples. sharegpt has been used for training powerful llms @xcite and incorporated into new datasets @xcite , both showing substantial advantages. specifically, we design an annotation framework where we employ @xcite to generate the related information for every user query that appears in sharegpt. we subsequently delve into the obtained data and conduct comprehensive analyses to answer the aforementioned questions foot_1 . we summarize our key findings as follows: ## methodology in this section, we employ gpt to annotate the topic/domain and task type of every sample in the sharegpt collection. the whole annotation and post-processing pipeline is illustrated in figure 1 . we use human evaluation to verify the quality of our automatic annotation. ## overall investigation in this section, we first present the overall statistics and analysis of the sharegpt clustering results. then we compare the results with conventional nlp datasets. for conventional nlp datasets, we investigate 2,911 datasets from the huggingface datasets foot_6 collected by @xcite . these 2,911 datasets are filtered and selected from the original around 50k datasets in the huggingface platform with the following conditions: (i) datasets with no license description or the license does not allow usage. for ethical considerations, the datasets collected are strictly following license restrictions. the number of all datasets fulfilling license requirements is restricted to around 16k. (ii) datasets that are non-english. since our investigation focuses on the english language and sharegpt contains only english samples, datasets collected from huggingface also follow this paradigm. (iii) datasets that are related to multi-modal. there is a large portion of datasets in the huggingface data platform related to multi-modal research. since we target at nlp community and related topics, we exclude those datasets. (iv) datasets that are invalid. some datasets in the huggingface data platform are empty or cannot be downloaded, which is ignored by our work. ## emerging trends and challenges in this section, we summarize the common trends and challenges shared by these prevalent or overlooked tasks we identified from the sharegpt data. ## conclusion and future works in this paper, we identified a discrepancy between the existing state of nlp research and the need for real-world applications by investigating large collections of sharegpt and huggingface data samples with gpt-4. we make this annotation resource public, which could be directly leveraged for further investigation of sharegpt data, or to fine-tune advanced models such as llama as a much cheaper alternative tool for annotation. based on our observation, we also provide insights into the challenges posed by real user needs and a potential roadmap for future work.