F t G - C o T at S em E val-2024 Task 9: Solving Sentence Puzzles Using Fine-Tuned Language Models and Zero-Shot C o T Prompting (2024)

Abstract: Recent large language models (LLMs) can solve puzzles that require creativity and lateral thinking. To advance this front of research, we tackle SemEval-2024 Task 9: BRAINTEASER: A Novel Task Defying Common Sense. We approach this task by introducing a technique that we call Fine-tuned Generated Chain-of-Thought (FtG-CoT). It is a novel few-shot prompting method that combines a fine-tuned BERT classifier encoder with zero-shot chain-of-thought generation and a fine-tuned LLM. The fine-tuned BERT classifier provides a context-rich encoding of each example question and choice list. Zero-shot chain-of-thought generation leverages the benefits of chain-of-thought prompting without requiring manual creation of the reasoning chains. We fine-tune the LLM on the generated chains-of-thought and include a set of generated reasoning chains in the final few-shot LLM prompt to maximize the relevance and correctness of the final generated response. In this paper, we show that FtG-CoT outperforms the zero-shot prompting baseline presented in the task paper and is highly effective at solving challenging sentence puzzles achieving a perfect score on the practice set and a 0.9 score on the evaluation set.

## introduction the brainteaser semeval-2024 task @xcite explores the ability of large language models (llms) to perform lateral thinking or "thinking outside the box", a topic that is currently under-explored by the natural language processing (nlp) community. unlike vertical thinking tasks that rely only on "common sense", solving this task requires a creative thinking process. the goal is to force llms to challenge their preconceptions and consider new perspectives. the task organizers propose a way to assess the ability of llms to think outside the box by creating a multiple-choice question-answering task designed to defy default commons-sense associations. for this task, the task organizers created a sentence puzzle dataset that contains sentence-type brain teasers centered on sentence snippets, and a word puzzle dataset that contains word-type brain teasers centered on the letter composition of the target question. both datasets are written in english. our approach, fine-tuned generated chain-of-thought (ftg-cot), as depicted in figure 1 , is a novel few-shot prompting method that combines a fine-tuned bert classifier encoder with zero-shot chain-of-thought (cot) generation and a fine-tuned gpt-3.5 llm. the bert @xcite classifier is fine-tuned by treating the training set as a multi-class classification task. each training set question and choice list is treated as the classification example. the index corresponding to the correct answer choice is the class label. reasoning chains for each example in the training set are generated using a variation of the classic zero-shot chain-of-thought prompt. for each example, in addition to the question and choice list, the llm is provided with the correct answer and asked to generate an explanation of why that is the correct answer. the generated chains of thought are then used to fine-tune the llm. for each example in the fine-tuning dataset, the prompt contains the question and choice list, and the response contains the generated chain-ofthought. to get the final generated response, the fine-tuned llm is queried with a few-shot prompt that contains a set of generated reasoning chains as example demonstrations. the set of training demonstrations provided in the few-shot prompt is chosen based on their cosine similarity to the test question. only the top 20 most similar training demonstrations are provided, ranked in order of increasing similarity. we chose to use openai's gpt-3.5 turbo model as the pretrained llm due to the ability to easily query and to start, the test question and choice list are fed to a fine-tuned bert classifier. the bert classifier outputs an encoding that is used to identify the top 20 training questions that are most similar to the test question. the corresponding training demonstrations are combined in order of increasing similarity with the test question and choice list to create a few-shot prompt. the few-shot prompt is sent to a fine-tuned llm, which outputs a generated response. a text parser matches the generated response with the most similar answer choice from the choice list. we provide a robust system that combines finetuned bert and gpt-3.5 together with zero-shot cot generation and ranked few-shot prompting. we found that this approach significantly improves the ability of the llm to solve brain teaser sentence puzzles compared to the baseline zero-shot prompting approach, achieving a perfect score on the practice set and a 0.9 score on the evaluation set. our method ranked 18th in the competition. we provide our code here: https://github.com/micahzhang/semeval-2024 ## background the brainteaser task @xcite consists of two sub-tasks: sentence puzzles and word puzzles. both datasets are written in english and each question is structured in a multiple-choice question format. sentence puzzles tend to be in a longer narrative story format. the correct answer challenges the common-sense interpretation of the question and the common-sense answer to the question. for word puzzles, the correct answer challenges the default meaning of a particular word and focuses on the letter composition of the question. both types of questions provide 4 different answer choices, including 1 correct answer and 3 distractors. figure 2 provides an example of a sentence puzzle and an example of a word puzzle for comparison. we chose to participate in the sentence puzzle sub-task. for this sub-task, a training dataset consisting of 507 sentence puzzles was provided by the task organizers. the training set contained 169 original sentence puzzle examples obtained from public websites via web crawlers. each example consisted of a question, a list of answer choices, the correct answer, and the distractors in the choice list. for each original question, a corresponding semantic reconstruction question and a context reconstruction question were generated. semantic reconstruction rephrases the original question without changing the correct answer and provided answer choices. context reconstruction changes both the question and answer to fit a new situational context, while keeping the original premise intact. the two adversarial question counterparts were created by the organizers to prevent an llm from being able to win the competition using memorization only. in addition to the training set, the organizers provide a practice evaluation and an official evaluation dataset for the sentence puzzle sub-task. both datasets contain 120 questions. to evaluate a model, a text file containing the answer choice index for each selected answer choice is submitted to codalab. codalab then automatically calculates the corresponding accuracy scores and posts the results on the leaderboard. our approach is inspired by @xcite , who introduced chain-of-thought prompting and demonstrated that providing a llm with a series of intermediate reasoning steps improves its ability to perform complex reasoning. our approach was also inspired by @xcite , who introduced zero-shot chain-of-thought prompting and demonstrated that llms could be made to generate their own series of intermediate reasoning steps by adding the words "let's think step by step" to the end of the prompt. bert @xcite is a popular transformer architecture commonly used as an sentence encoder for nlp tasks. the pretrained bert model can be fine-tuned by adding a classifier head to the end of the model and training it on a classification dataset. llms based on the gpt-3.5 architecture were popularized by @xcite , who introduced few-shot prompting a man shaves everyday, yet keeps his beard long. he is a barber. he wants his girlfriend to buy him a razor. he wants to maintain his appearance. ## system overview fine-tuned generated chain-of-thought prompting (ftg-cot) is a few-shot prompting method we developed for this competition. it consists of five steps. the first step is to fine-tune a pre-trained bert-small model on the training set as a multiclass classification task to use as an encoder. the second step is to use zero-shot chain-of-thought prompting @xcite to generate explanations for each demonstration in the training set. the third step is to fine-tune a large language model (llm) on the generated chain-of-thought @xcite demonstrations. the fourth step is to rank each encoded training demonstration based on its cosine similarity with the current encoded test question. the fifth step is to construct a few-shot (brown et al., 2020) prompt by stacking the top n most similar demonstrations with the test question prompt in order of increasing similarity. this few-shot prompt is sent to the llm, an answer choice is extracted from the llm's response, and the process is repeated for each question in the test set. this system is illustrated in figure 1 . fine-tuning bert: we fine-tuned a pre-trained bert model by converting it to a multi-class classifier and training it on the provided brain-teaser training set. this allowed us to create an encoder purpose-suited for solving sentence puzzles. during training the bert classifier learns contextual and semantic information that is relevant to solving the sentence puzzle task. when used to encode a training or test example, this information is captured in the encoding and aids in selecting highly relevant and useful demonstrations to use for few-shot prompting. zero-shot prompting: zero-shot prompting is used to generate chain-of-thought reasoning explanations for each training example. this is accomplished by prompting the llm using a custom zero-shot prompt for each training example. the first three lines of the prompt contain the training question, answer choices, and correct answers. the fourth line is "let's think step by step. what is the best answer? explain in detail why this is the best answer in 5 or less sentences". a custom fuzzy logic answer extractor parses the generated response from the llm by isolating the sentence containing the selected answer and then using a sequence matcher to compare the generated answer against each of the answer choices in the choice list. the index of the answer choice that most closely matches the generated answer is set as the predicted label. fine-tuning gpt-3.5: fine-tuning a pre-trained gpt-3.5 model on the 507 training examples and their corresponding generated reasoning chains improves the accuracy and consistency of the answers generated by the llm. the fine-tuning dataset contains an example prompt and response for each training question. the first two lines of the prompt contain the question and answer choices. the third line is "let's think step by step. what is the best answer?". the ground truth response is the generated response from the previous zero-shot prompting step. ## experimental setup training ftg-cot requires fine-tuning bert and fine-tuning gpt-3.5. we use all 507 examples from the sentence puzzle training set provided for the brainteaser task for fine-tuning. to fine-tune bert, we start with the bert-small pre-trained model from google @xcite and add a multi-class classifier head consisting of two linear hidden layers separated by a relu activation function to create a bert classifier. the input to the classifier head is the [cls] token from bert: the first token from bert's final hidden layer. the first hidden layer in the classifier head projects the bert output from dimensions 512 to 256. the output of the second hidden layer is passed through a log softmax function. to train the bert classifier on the sentence puzzle training set, we first concatenate the question and choice list from each example and provide them as the input string to the classifier. the index of the correct answer from the choice list is the label for the classifier. during training the bert classifier encodes each input string via the pre-trained bert model and passes the encoding to the classifier head, which then assigns a class label to the input. the model calculates the negative log-likelihood loss between the predicted and true label and back-propagates the loss through the entire model, including the bert layers. the bert classifier was trained using an adamw optimizer for 40 epochs with a learning rate of 0.00001. the final layer of the trained bert classifier produces encodings that have been fine-tuned to solve sentence puzzles. to fine-tune gpt-3.5, we start with the gpt-3.5turbo-1106 pre-trained model from openai and fine-tune the model using the openai fine-tuning api. we create a fine-tuning dataset according to the format required by the openai api. for each of the 507 training examples the system is set to "teacher", and the user prompt is defined as a con-catenation of the question and choice list. the ground truth assistant response is defined as the generated chain of thought. the fine-tuned gpt-3.5 model was trained for 3 epochs with a batch size of 1 and a learning rate multiplier of 2. all other training hyperparameters cannot be set externally by the user and are instead defined internally by openai. the model is evaluated using the test set provided for the brainteaser task. the test set consist of 120 questions, each with their own choice list. for each of the 120 questions, we use ftg-cot to create a few-shot prompt used to query the fine-tuned gpt-3.5 model via the openai api. we set the temperature to 1.0. the corresponding received response is then parsed using the custom fuzzy logic answer extractor described earlier. the predicted labels for all 120 test questions are then submitted to the brainteaser codalab website where it is automatically graded against the ground truth labels. the resulting accuracy score is displayed on the competition leaderboard. ## results the task organizers created 6 different accuracy metrics for evaluation. the first 3 metrics are instance-based accuracy scores that measure the accuracy of the model in solving the original questions, the semantic reconstruction questions, and the context reconstruction questions separately from one another. the next 2 metrics are groupbased accuracy scores that consider each original puzzle and its variants as a single group. for each group, the model will score 1 accuracy point only if it successfully solves all three puzzles in the group. otherwise, it will score 0 points. the last metric is an overall accuracy that is calculated as the average of the 3 instance-based accuracy scores. ftg-cot performs well at the task according to the official metrics, achieving a perfect score on the practice set and a 0.9 score on the evaluation set, ranking 18th overall in the competition. in this context, "score" refers to the accuracy score for the original practice and evaluation questions, not including the semantic and context reconstruction questions. furthermore, reversing the order of the demonstrations to be in order of decreasing similarity such as that the least similar demonstration is located closest in proximity to the test question in the prompt resulted in a lower score. this suggests that listing few-shot demonstrations in order of increasing similarity is the better approach. the results also illustrate the effectiveness of fine-tuning the llm on the training set. whereas without fine-tuning the score peaked at around 0.925 regardless of the number of demonstrations provided, fine-tuning the llm immediately resulted in a jump in performance to 0.975. finetuning the llm combined with increasing the number of provided demonstrations to 20 resulted in the highest achieved score of 1.0. however, increasing the number of provided demonstrations past 20 did not result in additional improved performance. ## error analysis for error analysis, we performed a 80/20 randomized train/test split on the sentence puzzle training set to create our own error analysis test set. we then fine-tuned ftg-cot on the remaining training examples and evaluated the model against this test set. early analysis revealed that many of the questions that the model answered incorrectly were false negatives resulting from using exact string matching for answer extraction. this motivated the creation of the fuzzy logic answer extractor. we also found that the llm struggled with answer consistency and repeatability. fine-tuning the model helped to lessen this issue. however, even with the fuzzy extractor and fine-tuning, the llm was not able to achieve perfect accuracy on the error analysis test set. figure 3 provides both an example of a sentence puzzle that the model answered correctly as well as an example of a sentence puzzle that the model answered incorrectly. both questions belong to the same group of sentence puzzles. the question the model answered correctly was the original sentence puzzle. the question the model answered incorrectly was the semantically reconstructed version of the sentence puzzle. the semantically reconstructed question rephrases the original question, keeps the same correct answer, and shuffles the provided answer choices. in this case, both questions share the same premise: a group of animals are racing to grab if a monkey, a squirrel, and a bird are racing to the top of a coconut tree, who will get the banana first? the squirrel. the bird. the monkey. none of above. ## conclusion the ftg-cot few-shot prompting method we developed for this competition combines a fine-tuned bert classifier encoder with zero-shot chain-ofthought generation and a fine-tuned llm. our experiments have demonstrated that ftg-cot is highly effective at solving sentence puzzles, achieving a perfect score on the practice set and a 0.9 score on the evaluation set, ranking 18th overall in the competition. the key takeaways from our experiments are that ftg-cot performs better with a larger number of demonstrations up to a certain point, fewshot demonstrations should e listed in order of increasing similarity, fine-tuning results in markedly improved performance, and that ftg-cot significantly outperforms zero-shot prompting on the sentence puzzle evaluation set. regarding future work, it is possible that the performance of ftg-cot can be further improved by using data augmentation techniques to expand the training set. for example, prompting an llm to rephrase the existing training questions could increase the size of the training set and potentially decrease the fine-tuned model's sensitivity to word choice and order. it is also possible that the performance of ftg-cot can be further improved by incorporating human feedback via reinforcement learning. a human could provide feedback on the quality and accuracy of the generated reasoning chains as well as manually rewrite incorrectly generated reasoning chains, improving the quality of both the fine-tuning dataset and the demonstrations.