Does GPT -4 pass the T uring test? (2024)

Abstract: We evaluated GPT-4 in a public online Turing test. The best-performing GPT-4 prompt passed in 49.7% of games, outperforming ELIZA (22%) and GPT-3.5 (20%), but falling short of the baseline set by human participants (66%). Participants’ decisions were based mainly on linguistic style (35%) and socioemotional traits (27%), supporting the idea that intelligence, narrowly conceived, is not sufficient to pass the Turing test. Participant knowledge about LLMs and number of games played positively correlated with accuracy in detecting AI, suggesting learning and practice as possible strategies to mitigate deception. Despite known limitations as a test of intelligence, we argue that the Turing test continues to be relevant as an assessment of naturalistic communication and deception. AI models with the ability to masquerade as humans could have widespread societal consequences, and we analyse the effectiveness of different strategies and criteria for judging humanlikeness.

## conclusion we conducted a large-scale public online turing test with human participants and gpt-4. one gpt-4 witness, dragon, deceived users into believing that it was human fairly robustly across 855 games. as far as we are aware this is the first empirical demonstration of an agent achieving a 50% success rate at the turing test on such a large sample. this suggests that successful deception and impersonation of humans is already possible, especially in contexts where the possibility of deception is less salient. there was wide variation in success rate by prompt, suggesting that further refinement of models, prompts, or the general setup could lead to higher success rates. the majority of interrogator reasons for ai decisions pertained to linguistic style and socio-emotional factors, suggesting that these are currently greater barriers to ai impersonating humans than traditional notions of intelligence. although our sample here was relatively large, the goal of our contribution was largely exploratory: to provide a framework and test a variety of parameters to understand the influence they have on turing test results. future work should confirm these findings in a pre-registered, randomized, controlled study. in addition, we only test a small number of models and prompting techniques here. future work should explore using other models and giving models access to external tools like web browsing and chain-of-thought reasoning. 9 5191 as a public online experiment, this work contains several limitations which could limit the reliability of the results. first, participants were recruited via social media, which likely led to a biased sample that is not representative of the general population (see figure 14 ). secondly, participants were not incentivised in any way, meaning that interrogators and witnesses may not have been motivated to competently perform their roles. some human witnesses engaged in 'trolling' by pretending to be an ai. equally some interrogators cited this behavior in reasons for human verdicts (see figure 20 . as a consequence, our results may underestimate human performance and overestimate ai performance. third, some interrogators mentioned that they personally knew the witness (e.g. they were sitting in the same room). we excluded games where interrogators mentioned this in their reason, but to the extent that this occurred and interrogators did not mention it, we may have overestimated human performance. fourth, sometimes only one participant was online at a time, meaning that they would be repeatedly matched up with ai witnesses. this led participants to have an a priori belief that a given witness was likely to be ai, which may have led to lower sr for all witness types. we tried to mitigate this by excluding games where an interrogator had played against an @xmath0, however, this bias likely had an effect on the presented results. fourth, we used closed-source openai models for analysis. because limited information is available about the architecture and training of gpt-3.5 and gpt-4, our results may be more difficult to reproduce with open-source models. finally, we used a relatively small sample of prompts, which were designed before we had data on how human participants would engage with the game. it seems very likely that much more effective prompts exist, and therefore that our results underestimate gpt-4's potential performance at the turing test.