Do great minds think alike? Investigating Human- AI Complementarity in Question Answering with CAIMIRA (2024)

Abstract: Recent advancements of large language models (LLMs)have led to claims of AI surpassing humansin natural language processing NLP tasks such as textual understanding and reasoning.%This work investigates these assertions by introducingCAIMIRA, a novel framework rooted in item response theory IRTthat enables quantitative assessment and comparison of problem-solving abilities inquestion-answering QA agents.%Through analysis of over 300,000 responses from ~ 70 AI systemsand 155 humans across thousands of quiz questions, CAIMIRA uncovers distinctproficiency patterns in knowledge domains and reasoning skills. %Humans outperform AI systems in knowledge-grounded abductive and conceptual reasoning,while state-of-the-art LLMs like GPT-4 Turbo and Llama-3-70B demonstrate superior performance ontargeted information retrieval and fact-based reasoning, particularly when information gapsare well-defined and addressable through pattern matching or data retrieval.%These findings identify key areas for future QA tasks and model development,highlighting the critical need for questions that not only challengehigher-order reasoning and scientific thinking, but also demand nuanced linguisticand cross-contextual application.

## introduction the nlp community has focused on human behavior emulation, treating human performance as ceiling for models. however, the latest wave of llms has turned the discussion to supremacy: models are purportedly acing tests @xcite ) that many humans find challenging. 1 a no ## background and preliminaries this section describes the source of the quizbowl qa data ( § 2.1) and preliminaries of irt and mirt ( § 2.2), the foundation of caimira ( § 3). ## question and agent analysis this section interprets the latent aspects of caimira, emphasizing their role in differentiating agent skills. it also examines the patterns of question difficulty and agent performance. ## conclusions caimira enables discovery and interpretation of latent aspects in qa datasets that highlight the skills of various qa agents. on contrasting ai systems with humans, we find notable disparities: systems like gpt-4-turbo and gemini pro excel at direct, context-rich queries that require connecting events and figures, but struggle with indirectly phrased questions lacking explicit entity references-domains where human acumen shines. although gpt-4-turbo matches individual human performance on complex abductive reasoning tasks, we caution against interpreting this as indicative of superhuman abilities. given that the protobowl dataset is publicly available and the models' training data is unknown, accurately assessing their near-perfect performance is challenging. future research should aim to develop stronger and innovative evaluations that better gauge ai systems' ability to understand implicit contexts, and systematically contrast with that of humans. lastly, this work opens up new avenues for research on estimating agent skills that can be combined to assess multi-agent system and collaborations.