G emini P ro at S em E val-2024 Task 9: B rain T easer on Gemini (2024)

Abstract: It is known that human thought can be distinguished into lateral and vertical thinking. The development of language models has thus far been focused on evaluating and advancing vertical thinking, while lateral thinking has been somewhat neglected. To foster progress in this area, SemEval has created and distributed a brainteaser dataset based on lateral thinking consist of sentence puzzles and word puzzle QA. In this paper, we test and discuss the performance of the currently known best model, Gemini, on this dataset.

## introduction human thought is known to be distinguished into lateral and vertical thinking. @xcite cites @xcite in mentioning, based on modern neuroscience, that vertical thinking is associated with the left hemisphere of the brain, while lateral thinking is associated with the right hemisphere. moreover, this paper notes that during the development of language models, there has been a focus on problem-solving abilities in vertical thinking, neglecting the capabilities based on lateral thinking. this paper anticipates that lateral thinking puzzles may not be easily solved with just additional adaptations and extensions of the llm (large language model) approach, yet this paper is prepared to counter that expectation. it evaluates the performance of google's ambitious model, gemini @xcite , which is said to surpass gpt-4, by measuring performance solely through changes in demonstration, as it was not possible to fine-tune gemini. gemini is anticipated to show increased performance due to the scaling law mentioned in @xcite , as it utilizes significantly more parameters than gpt-4. being a multimodal model trained with additional learning resources such as visual and auditory inputs, it is speculated that these characteristics might give rise to unique emergent abilities. @xcite these two aspects are expected to contribute to performance improvements in lateral thinking. our approach is straightforward. first, we formalize gemini's responses by adding demonstrations, following the same few-shot provision method as used by semeval, and second, we provide only the relevant task few-shot examples for the two brainteaser tasks: sentence puzzles and word puzzles. through the second method, we investigate whether providing clear few-shot examples for tasks alone can aid in performance improvement. ## system overview in this study, we used gemini, an ambitious model released by google, known to surpass chatgpt. we utilized the gemini-pro api and followed the chatgpt evaluation method provided by semeval. ## experimental setup although semeval provided aproxymately 1,000 training examples, this study did not use the training data as it did not involve fine-tuning. instead, we directly used the brain teaser test data to measure the intrinsic capabilities of gemini-pro. gemini-pro occasionally does not output an answer. in such cases, we considered (d) as the answer. if it does not output a response in the structured form of (a), (b), (c), (d), we also treated it as (d). for all other cases, we followed the chatgpt methodology as outlined by semeval. ## conclusion as observed in figure 2 of @xcite , increasing the number of examples in sentence puzzles did not consistently improve performance, and while an overall upward trend in performance for word puzzles was noted, it did not improve regularly. similarly, in the experiments of this paper, performance fluctuations with the number of examples were erratic, but it is clear that performance is generally higher compared to chatgpt. the leaderboard for brainteasers often shows many cases scoring over 90, which is likely due to the use of fine-tuning methods on the brainteaser training set. without training specialized for brainteasers, the effect of using the method of demonstration appears to be minimal or almost nonexistent in a pure model state, and it has been found that larger models exhibit more pronounced performance improvements. particularly, gemini, despite being a multimodal model trained with both visual and auditory inputs, significantly underperforms compared to human capabilities. contrary to the original paper's expectation, it was observed that merely increasing the model size could spontaneously develop problem-solving abilities for lateral thinking tasks, suggesting that even the capability for lateral thinking falls within the range of emergent abilities. according to @xcite , as the training of transformer models progresses, layers specialized for tasks are formed, with it being speculated that lateral associations are made in highly differentiated semantic layers in layers closer to the end. perhaps the improvement in performance in llms, as mentioned in the context of brainteasers, might simply be due to memorizing content from the corpus. @xcite it remains to be seen whether probing layers specialized for semantic tasks in the future could unveil the mechanism behind lateral thinking.