Paper: LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (ICLR-2024)

Reviewer comment (Presentation):
Figure 1 and 2 could be redrawn by leaving vicuna/English out because the distribution is left-skewed and therefore the number for the rest of the models/languages are hard to interpret.

Author response:
We will consider redrawing Figures 1 and 2, either by using a log scale or by splitting them into two separate figures.

Reviewer comment (Experiments):
How many prompts are there after filtering for section 3.2?

Author response:
We began with 100K conversations, each averaging two turns, resulting in 200K prompts. After filtering, half of the prompts were removed, leaving us with 100K prompts.

Reviewer comment (Experiments):
The dataset does not seem quite as diverse as the abstract suggests. Although 25 LLMs are used, Vicuna-13B is by far the most frequently used one. Similarly, while there may be a few instances of many languages, the vast majority is still in English.

Author response:
It is true that Vicuna-13B and English comprise a large fraction of our dataset. This is because Vicuna-13B is the first model we hosted on the platform and the fact that most of our users speak English. Meanwhile, our dataset will continue to be updated. This is expected to mitigate the bias present in earlier models and keep pace with the latest data from new models. We are actively maintaining our data collection website and plan to release more data dumps in the future. In the month following our paper submission, we received approximately 25K user queries per day for the latest models, including GPT-4 Turbo, Mistra, and Zephyr. We also plan to release more data on human preferences, such as votes.

Reviewer comment (Experiments):
As the authors point out, there are no human preference values which is one weakness compared to related datasets (see table 1).

Author response:
It is true that Vicuna-13B and English comprise a large fraction of our dataset. This is because Vicuna-13B is the first model we hosted on the platform and the fact that most of our users speak English. Meanwhile, our dataset will continue to be updated. This is expected to mitigate the bias present in earlier models and keep pace with the latest data from new models. We are actively maintaining our data collection website and plan to release more data dumps in the future. In the month following our paper submission, we received approximately 25K user queries per day for the latest models, including GPT-4 Turbo, Mistra, and Zephyr. We also plan to release more data on human preferences, such as votes.

Reviewer comment (Experiments):
It is not clear whether the use format (single model vs side-by-side) and IP address have similar balance issues to model and language above. Although there are 210K unique IP addresses, is some large portion of the dataset covered by only a few of these? And do most users access single models rather than the perhaps more interesting side-by-side mode? Having the answers to these questions would help evaluate this paper further.

Author response:
The IP addresses are actually very sparse, so there is no small set of IP addresses that can cover a large portion of the dataset. It is true that most users access single models rather than side-by-side. But the distribution is shifting towards side-by-side more recently because we are promoting more on the side-by-side mode and making it the landing page.

Reviewer comment (Experiments):
While the user prompts in this dataset are genuine, the responses are synthetic and do not have quality ratings. Thus, additional processing is necessary before use.

Author response:
Although limited, our data does include some user ratings (e.g., upvote/downvote) on model responses, in which the upvoted responses were used to train the model in Section 4.3. We will soon release the user feedback dataset after finishing the in-depth study and data cleaning.

Reviewer comment (Experiments):
Are there plans to enhance the website's interface design to increase its accessibility for a broader audience? Such improvements could lead to a more diverse user base and a richer dataset distribution.

Author response:
Yes, we’re actively enhancing the website’s interface and committed to growing this platform to a broader user base. In the past few weeks, our traffic has increased significantly due to active community engagement and now we’ve collected over five million user chat queries. Those new data will be included in our next dataset release. Please kindly let us know if you have any suggestions on UI/UX, thanks!