The Fourth Evaluation on C hinese Spatial Cognition (2024)

Abstract: “The Fourth Chinese Spatial Cognition Evaluation Task (SpaCE 2024) presents the first comprehensive Chinese benchmark to assess spatial semantic understanding and reasoning capabilities of Large Language Models (LLMs). It comprises five subtasks in the form of multiple-choice questions: (1) identifying spatial semantic roles; (2) retrieving spatial referents; (3) detecting spatial semantic anomalies; (4) recognizing synonymous spatial expression with different forms; (5) conducting spatial position reasoning. In addition to proposing new tasks, SpaCE 2024 applied a rule-based method to generate high-quality synthetic data with difficulty levels for the reasoning task. 12 teams submitted their models and results, and the top-performing team attained an accuracy of 60.24%, suggesting that there is still significant room for current LLMs to improve, especially in tasks requiring high spatial cognitive processing.”

## introduction spatial expressions have long been a focus of cognitive linguists because they are not only a highfrequency phenomenon in human language but also embody the fundamental mechanisms of how humans perceive the world @xcite . in recent years, some nlp evaluation tasks have also sought to explore machines' cognitive semantic understanding capabilities through the lens of spatial expressions by a single task such as spatial role labeling @xcite or spatial reasoning @xcite . as for evaluation in chinese, the spatial cognition evaluation task (space) has been held for 3 years since 2021 with the aim of comprehensively evaluating machine's capabilities with regard to chinese spatial semantic understanding by multi-task learning @xcite @xcite . the latest results indicate that machines' spatial semantic understanding capabilities lags significantly behind the average level of humans, especially in tasks requiring high cognitive processing. spatial semantic understanding remains a challenging task for nlp systems, even for large language models (llms). in this work, we construct a benchmark that comprehensively assesses llms' performance on understanding spatial expressions and conducting spatial reasoning in the following five subtasks: (1) identifying spatial semantic roles (isr); (2) retrieving spatial referents (rsr); (3) detecting spatial semantic anomalies (dsa); (4) recognizing synonymous spatial expression with different forms (rse); (5) spatial position reasoning (spr). space 2024 has released 10,373 multiple-choice questions (mcqs) of the benchmark, which is available at https://github.com/2030nlp/space2024 . summarized statistics are shown in the five subtasks of space 2024 have different difficulty levels of cognitive processing, which can be roughly divided into three levels: (1) basic: isr. this level requires understanding the lexical meaning, grammatical structure, and semantic structure of a sentence, focusing on local information processing. ## evaluation and results we use accuracy as the ranking metric. the higher the accuracy, the better the llm's capability of comprehensive spatial semantic understanding. there are 150 disturbance data in test set that use to evaluate the consistency of llms. a group of disturbance data contains an original data, a repeated version, a version with different order of options, and a version with different expression of question. a llm is considered stable only if all the answers in a group are the same. the higher the consistency score, the less the llm is affected by the data format, making the evaluation results more reliable. totally 30 teams enrolled and 12 teams stick to the end. the top 6 teams won the prize. they all employed llms either fine-tuned or utilized prompt engineering. the accuracy results are shown in ## overview of approaches only the top 6 teams submitted their models. model ensemble with majority voting is the main strategy. the team from teleai company manually selected 5 examples for each subtask to construct in-context learning (icl) training data, then fine-tuned qwen1.5-7b-chat 3 times to do the majority voting. teleai set weights on various models and tasks, thereby performing a more nuanced and context-sensitive aggregation to enhance the accuracy and consistency score of the final predictions. the team from shanghai university (shu) recategorized the dataset into four types of questions based on the training size and the difficulty of the tasks. for the questions having large training data with low difficulty, shu fine-tuned qwen1.5-7b-chat, yi-6b-chat, and intern2-chat-7b, then voted for the majority as answer. for the structured data in spr, shu assumed that it was too difficult and not suitable for training with the easy question, so they fine-tuned the above models again and voted. the remaining questions did not have enough training data to fine-tune well, so shu ran gpt-4o with chain of thought (cot) and qwen1.5-110b-chat with 5-shot to solve them. the team from peking university (pku) ran gpt-4o three times in isr, rsr, dsa, spr and conducted a majority vote. for rse, it ran gpt-4o and deepseek-chat, meanwhile set another deepseek c c l 2 0 2 4 as the judge to decide which one was better when two answers were not the same. pku also set an error correction mechanism to rerun the question in some situations like only outputted one answer in a multi-answer question. the team from zhongyuan university of technology (zut) used data augmentation by generating pseudo-labeled data. they used training data to fine-tune qwen1.5-7b-chat in different hyperparameter and chose high-score models to predict the unlabeled test data. after majority voting, nearly 3k reliable pseudo-labeled data were added to the training set to fine-tune again until no further improvement in the accuracy of the test set. zut also ran glm-4 in the test set and made model ensemble. the team from guangdong university of technology (gdut) employed 5-fold cross-validation and majority voting to produce the final prediction by fine-tuning chatglm3-6b-base. another model for ensemble was ernie-4.0-8k-0329, whose overall score was higher than chatglm3 in isr, dsa and rse. for icl, gdut dynamically selected 5-shot examples by computing the similarity of options between the target test data and the training data in same subtask. the top 5 of similarity would be chosen as examples. the team from beijing normal university (bnu) constructed prompt with one-shot. to select the most representative example for each task, they used sentence-bert @xcite to encode all training data into embedding and averaged all embedding to get the centroid of the task. the data closest to the centroid in terms of embedding distance was the most representative. in general, the models of the top 6 teams were primarily based on icl and the majority voting strategy. fine-tuning can bring about a more stable model, and it can improve the performance of a 7b model close to that of gpt-4o, but this improvement is limited to simple tasks. for the two most complex tasks, finetuning might not bridge the gap because it's fundamentally related to the reasoning capabilities of the base llms. ## additional analysis following an analysis based on the results of the top 6 teams, we studied each subtask, the shared text across tasks, and conducted a comparison between space 2024 and space 2023. ## conclusion this paper introduces the fourth evaluation of the chinese spatial cognition task. the benchmark consists of five subtasks covering 5 dimensions of spatial semantic understanding to comprehensively assess the machine's language comprehension and reasoning capabilities on spatial semantics. in addition to capability assessment, space 2024 concerns the consistency of llms' performance and sets a disturbance set. as for quality control, a novel rule-based method was used in the spatial reasoning task, successfully generating high-quality synthesis data with transparent intermediate processes and difficulty levels. the accuracy scores across 5 subtasks reveal the following trend: @xmath0. based on a fine-grained analysis, we have gained the following insights into the spatial semantic understanding capabilities of llms: in future work, our aim is to generate a large amount of structured, high-quality, and knowledgecontrollable data to train llms to enhance their semantic understanding and reasoning capabilities. to this end, we will further investigate data synthetic methods to incorporate more spatial relations such as 里(inside) and 外(outside) into the knowledge base and to extend this method to four other subtasks, exploring a technical path for data synthesis driven by knowledge.