Research Community Perspectives on “Intelligence” and Large Language Models (2025)

Abstract: Despite the widespread use of ‘artificial intelligence’ (AI) framing in Natural Language Processing (NLP) research, it is not clear what researchers mean by ”intelligence”. To that end, we present the results of a survey on the notion of ”intelligence” among researchers and its role in the research agenda. The survey elicited complete responses from 303 researchers from a variety of fields including NLP, Machine Learning (ML), Cognitive Science, Linguistics, and Neuroscience.We identify 3 criteria of intelligence that the community agrees on the most: generalization, adaptability, & reasoning.Our results suggests that the perception of the current NLP systems as ”intelligent” is a minority position (29%).Furthermore, only 16.2% of the respondents see developing intelligent systems as a research goal, and these respondents are more likely to consider the current systems intelligent.

## introduction whether machines can be considered intelligent has long been a question of interest in both academia and industry. researchers have a rich tradition of devising intelligence tests for machines, starting from the turing test @xcite , and the industry increasingly markets "smart" and "intelligent" systems. a recent iteration hereof are llm-based systems, also referred to as "generative ai", 1 which have been suggested to show "sparks of artificial general intelligence" @xcite . irrespective of one's position on whether systems based on current ml technology can be considered intelligent, there is undeniably an increase 1 we are only concerned with systems based on large language models (llms), i.e., models trained on large text corpora that are used for transfer learning @xcite . we use the term "ai" in line with its common use in the field, but we note that it entails that the term "intelligence" is applicable, which, as our results suggest, is not accepted by the majority of the respondents. in the phenomenon we refer to as seeming intelligence: the user's perception that a system is intelligent, based on their interaction with that system. this perception arises from the eliza effect: humans have a strong tendency to attribute to their conversational partners, even simple chatbots, "all sorts of background knowledge, insights and reasoning ability" @xcite , and the perception of intelligence is a driver of anthropomorphization, irrespective of whether a system is represented via an avatar or not @xcite . the tendency to anthropomorphize ai systems is not limited to the lay public or marketing by tech companies: 2 mitchell (2021) points out that even researchers mislead themselves by using terms like "understanding", which have long been criticized as "wishful mnemonics" @xcite . in particular, we have a longstanding tradition of evaluating "understanding" (e.g. @xcite , "reasoning" (e.g. @xcite , "behaviors" (e.g. @xcite and "capabilities" (e.g. @xcite . given the above tendency towards anthropomorphization and the long history of "wishful mnemonics" in the academic literature, the goal of this study is to provide a snapshot of how researchers think about "intelligence" two years after the release of @xcite . we pose the following research questions: rq1 what are key criteria for a system to be considered "intelligent", and to what degree do researchers in different fields, occupations, and career stages agree? rq2 what is the perception of the "intelligence" of current llm-based systems such as chatgpt? rq3 what role, if any, does the research agenda of respondents play in the notion of "intelligence"? our results indicate a high level of coherence across groups in terms of the criteria that are relevant for intelligence. independent of one's field of research, occupation, and career stage, most researchers believe that generalization, adaptability, and reasoning are key aspects of intelligence (rq1, illustrated in figure 1 ). we additionally observe a general skepticism regarding the attribution of intelligence to llm-based systems, although it decreases for "future systems based on a similar technology" (rq2). for rq3, we find that only a small subset of respondents consider it their goal to build intelligent technology, with the majority rather aiming to add to the scientific knowledge. interestingly, the survey shows that those who consider 'creating intelligent systems' as their research goal are also more likely to attribute intelligence to the current systems (rq2, rq3). ## survey structure and distribution the survey was developed using the surveyxact platform. we initially provide the introductory information regarding anonymity and informed consent, followed by questions related to basic demographic information relevant for the analysis. this is followed by questions pertaining to the respondents' perspectives on the notion of "intelligence". responses were collected between november 2024 and january 2025. the survey form and response data (with coarse-grained geographic data for in-creased anonymity) is provided on the accompanying github repository, and a full list of questions can be found in appendix b. the survey was distributed through social media (bluesky and linkedin) as well as the mailing lists seen in ## question design this section provides a brief overview of the questions included in the survey and the rationale for including them. an overview of the questions is provided in appendix b. ## discussion this survey provides concrete numbers to a core source of confusion in the field: what we mean when we use the term "intelligence". while most papers do not define the term, and many researchers struggle to specify what they mean (subsection 5.1), we do observe a degree of coherence. the majority of respondents (> 80%), regardless of their career stage, primary research area and occupation, agree that "intelligence" relies on the criteria of generalization, adaptability and reasoning (rq1). the high agreement on these criteria offers the community an opportunity to make the discussion of "intelligence" more specific, and encourage evaluation that focuses on them. this is also consistent with the fact that generalization tests, such as the arc challenge @xcite , was the most selected among respondents (≈ 37%). given that there is some consistency to the core criteria of "intelligence", we would expect significant agreement on whether this term applies to the current llm-based systems (rq2). indeed, this is what we find: the majority of respondents are skeptical about both current (≈ 71%) and future systems based on a similar technology (≈ 60%) (subsection 5.4). 7 this skepticism stems in part from the limitations of current systems to generalize and reason ( finally, the results of this survey provide context for the recent debate on the relation between the fields of "nlp" and "ai" @xcite @xcite . the majority of survey respondents do not see their goal as creating intelligent systems, focusing instead on advancing scientific knowledge. however, we found an interesting correlation between research goals and beliefs regarding the intelligence of current systems (rq3). the respondents who do aim to create intelligent systems were more likely to attribute intelligence to both the current and future llm-based systems. ## conclusion the term "intelligence" is often used, but rarely defined in the current research on llms. despite the lack of clear definitions, we find a high degree of consensus across research fields that generalization, adaptability, and reasoning are key to the notion of "intelligence". the majority of the survey respondents are skeptical of applying this term to the current and future systems based on llms. we find that the senior researchers tend to be more skeptical, and respondents who view creation of intelligent systems as part of their research agenda are more likely to attribute intelligence to both current and future systems. to the best of our knowledge, this is the most comprehensive effort to date to clarify what the experts currently mean by the term 'intelligence', and whether it applies to the current technology. these results provide a useful point of reference for both public discourse and evaluation of marketing claims.