Mothman at S em E val-2024 Task 9: An Iterative System for Chain-of-Thought Prompt Optimization (2024)

Abstract: Extensive research exists on the performance of large language models on logic-based tasks, whereas relatively little has been done on their ability to generate creative solutions on lateral thinking tasks. The BrainTeaser shared task tests lateral thinking and uses adversarial datasets to prevent memorization, resulting in poor performance for out-of-the-box models. We propose a system for iterative, chain-of-thought prompt engineering which optimizes prompts using human evaluation. Using this shared task, we demonstrate our system’s ability to significantly improve model performance by optimizing prompts and evaluate the input dataset.

## introduction the ability for language models to reason or possess common sense knowledge has become a controversial topic with far-reaching implications @xcite . large language models (llms) show remarkable results on vertical thinking tasks that require sequential logical inference @xcite but there have been relatively few studies done on lateral thinking puzzles-tasks that require more creative, "outside the box" problemsolving processes. as larger llms with the ability to memorize large corpora @xcite are developed, lateral thinking tasks become an increasingly important benchmark for analyzing and evaluating their reasoning capacities. the brain-teaser shared task @xcite is designed to elicit and evaluate lateral thinking through two english-language subtasks, using sentence puzzles and word puzzles respectively. in this paper, we propose a novel method for optimizing chain-of-thought (cot) prompting @xcite on the gpt-4 model which we use to tackle the sentence puzzle subtask. our system iteratively optimizes cot prompting by systematically evaluating input data and model output using human performance as a benchmark. we identify question types that are difficult for humans, informing the next iteration of prompt engineering. not only does this process optimize cot prompting for a specific task, our system also provides insights for improving future data collection and synthesis. our main contribution is the novel approach for identifying reasoning challenges to optimize prompting. for the sentence-based task, we develop a prompt engineering method which requires the model to reason over all answer choices and provide explanations for both correct and incorrect choices. in doing so, the model is more likely to refute choices that are semantically related to the question but logically incorrect. our methodology significantly improves performance for adversarial datasets and achieves more consistent results, which suggests that the model relies less on memorization when using these cot prompts. as part of our evaluation of the data, we also identify several questions in the adversarial datasets that are difficult to solve due to having multiple logical options or are unanswerable with the provided premises. by combining model reasoning with human evaluation, we can quickly identify and evaluate problematic questions. this process can further explain model performance and provide guidance for future data collection/generation. ## background question answering (qa) is a well-established task in natural language processing with broad applications both in academia and in industry @xcite . recent work such as commonsenseqa (csqa) @xcite and strategyqa @xcite focus on reasoning questions that require logical inference the data for this subtask is drawn from online english-language riddles and brainteasers, with incorrect choices created by handpicking entailments generated by comet @xcite using incorrect premises. each question has three unique answers, as well as a shared fourth option, "none of above". to counter memorization from llms trained on web crawls, the task authors generated two synthetic datasets using semantic reconstruction (sr) and context reconstruction (cr). the sr dataset rephrases the original question without changing the answer or premises while the cr dataset changes the situational context without changing the misleading premise. the sr dataset was generated with an open-source rephrasing tool while the cr dataset was generated using gpt-4; both sets were manually refined by human annotators. in total, 208 question/answer pairs were sampled for the base set resulting in 624 questions after sr and cr augmentation. the training set was split with 81.25% of the data with the same base/sr/cr questions kept together in the split. although the task is designed to elicit lateral thinking, we consider an alternative understanding of the task by thinking of the questions as noisy. questions are loaded with irrelevant, contradictory, or misleading information to distract the respondent. since transformers generally learn meaning by scoring tokens across the sentence or sentence pair, they are biased against long-tail knowledge @xcite , which is knowledge that occurs infrequently in the training set. brainteasers, by their nature, rely on the unconventional interpretation of the question to stump the respondent. this same property can trick the model into selecting a semantically similar answer choice that is logically incorrect. chain-of-thought (cot) prompt engineering @xcite is a recent method that has been shown to not only improve outcomes on similar problems, but also to provide an interpretable window for human review. cot prompts provide example questions with related reasoning to the model, which induces the model to provide reasoning for a given answer in the output. we utilize both of these properties to introduce an iterative method that optimizes cot prompting for a given task. ## system overview we propose an iterative system for optimizing the cot prompt engineering process: each step of our process is iterative, although independent human evaluation should only be performed when novel problem categories are identified in the model reasoning. once the human benchmark has been incorporated into the prompt engineering process, future iterations mainly rely on evaluating model outputs for gaps in logic. all evaluation steps can provide powerful insights into the dataset itself to inform future dataset creation, and is particularly useful for real-world applications where data selection is an open-ended problem. by identifying gaps and problems in the data, more representative data can be collected to improve model performance on the given application. this can be thought of as a backpropagating human and model outputs back to the prompt engineering and data curation steps. ## experimental setup all experiments were performed using gpt-4 via the openai api 2 with the same system prefix. each question was called separately with the respective prefix. initial experiments limited token count to 1 in order to force the model to output integer labels but the restriction was removed due to poor performance. output text was logged and labels were extracted deterministically; labels that could not be extracted this way were reviewed and manually entered. results shown in this paper were from api calls between 2024/01/15-2024/02/17; since openai models are updated regularly replication results may differ. the input data was first preprocessed by removing extra spaces, lines, punctuation, and spelling and grammatical errors. due to the variety of sentence-based problems in the data, some answer 2 https://platform.openai.com/ choices were multiple sentences long while others were single words; this discrepancy could potentially affect model performance. since semicolons do not occur in the data at all, they were selected as separators between answer choices for gpt-4 prompting to mitigate this issue. system prompts can be found in appendix c. ## results in this section, we compare results across four categories: human performance, zero/multi-shot performance, naive cot prompt performance, and the iterated cot prompt performance. for completion, we also provide the top competition result (abdelhak). our official submission for the shared task leaderboard used the gpt-4 new cot-base prompts. with 31 participants, our results scored 2nd overall on the base data, 7th on sr, cr, and base&sr accuracy, 9th on adversarial accuracy, and 9th overall. ## conclusions in this paper, we demonstrate a novel system for optimizing chain-of-thought prompt engineering using human evaluation. while naive cot prompts performed similarly to multi-shot prompting without guidance on reasoning, later iterations were able to approach state-of-the-art performance. significant improvement on the sr, cr, and groupbased metrics were shown on the test data, supporting the adoption for this method of prompt engineering. this system further provides guidance on identifying key problem areas in the dataset, specifically with regards to the generation of context reconstruction questions. this form of evaluation serves to inform decisions for improving dataset quality. for future work, we plan to implement different techniques to create synthetic data and perform the same evaluation across an open-ended dataset. married to more than one person at a time, even if her husband is bad. if the woman did not get divorced or legally separated, she cannot be legally married to more than one person. ministers perform marriages, they do not get married. the answer is 2 3. question: brad began by entering the office tower's 22nd story through the filthy common window. he slid the window open and leapt through it after becoming depressed. outside the building, there was a cliff-like drop to the ground. amazingly, he came to rest entirely unharmed. how could he have survived the fall if there was nothing to slow or cushion his descent? choices: 0 = brad was so sick and tired of window washing, he opened the window and jumped inside.; 1 = the ground outside the building is wet.; 2 = consistent exercise has made him a very strong man.; 3 = none of above.; response: even very strong men would not survive a fall from a 22-story building. the ground outside the building being wet would not cushion his fall. brad was so sick and tired of window washing, he opened the window and jumped inside. the answer is 0 4. question: danny had just passed under an overpass in his semi when he abruptly came to a stop. danny accidentally drove under the overpass that was only just tall enough for his truck because he wasn't paying enough attention. he was unable to move forward or backward in the semi due to how tightly it was wedged. when another tracker passed by, he was told how simple it would be to remove the semi from underneath the bridge. what did he recommend? choices: 0 = he told danny to left the bridge.; 1 = he told danny to overturn the track.; 2 = he told danny to let some air out of his tires.; 3 = none of above.; response: the bridge would not be lifted. overturning the truck would not help. letting some air out of his tires would help. the answer is 2 5. question: i excavate little caves and keep my gold and silver there. i also create gold crowns and silver bridges. they are the tiniest things you can imagine. everyone will eventually require my assistance, but many people are reluctant to accept it. why? choices: 0 = i am a dentist.; 1 = i am a thief.; 2 = i am a miner.; 3 = none of above.; response: thieves and miners do not create gold crowns or bridges. tiny crowns and bridges are dental work. people don't like going to the dentist. the answer is 0 mules are born sterile, so they cannot give birth to offspring. the answer is 2 8. question: six women were hiking on a trail when a sudden snowfall began. five of the women hurried along, but the sixth did not. still, they all reached the cabin at the same time, and all but the sixth were cold. how is that possible? choices: 0 = the 6th woman beated them at the end.; 1 = the sixth woman was in a photograph that one of the others was carrying.; 2 = the 6th woman walked slowly to dodge the snow.; 3 = none of above.; response: if the sixth woman beat them or walked slowly to dodge the snow she would not have reached the cabin at the same time. if the sixth woman was in a photograph that one of the others was carrying, then she would not be cold and would not have to hurry. the answer is 1 b new cot prompts b.1 new base prompts 1. question: a couple is having a disagreement over the man's error. the man kept apologizing and pleading with the woman to allow him to see her directly. the woman was still upset. thus, she wouldn't agree. the couple, though, was positioned on the same mat. how is that even doable? choices: 0 = the mat was folded in multiple times and end up with a special shape.; 1 = the woman kept turning her back to the man.; 2 = the couple was standing on two sides of the girl's apartment door, which had a rug under it.; 3 = none of above.; response: the couple are standing on the same mat but cannot see each other, which suggests that there is a barrier between them. regardless of how the mat was folded, it would not prevent the couple from seeing each other. even if the woman turned her back to the man he would still see her. therefore, the couple was standing on two sides of a door, which had a rug under it. the answer is 2 2. question: eight people were sitting under a large tree. suddenly, a gust of wind blows, yet none of them got hit by any falling leaves. how is this possible? choices: 0 = it was winter and the tree doesn't have any leaves.; 1 = people were camping under the tree.; 2 = the wind blows heavily.; 3 = none of above.; response: if the people were camping under the tree, they would still be hit by the falling leaves. the wind blowing heavily would cause the leaves to fall. if it was winter and the tree doesn't have any leaves, then the people would not be hit by any falling leaves. the answer is 0 3. question: the ship was in the central pacific ocean. the ship suddenly began to sink without being crushed. however, each team was still preoccupied with its own tasks, so none was threatening. why? choices: 0 = an underwater earthquake caused a rapid drop in the water level and resulted in the loss of buoyancy for the ship.; 1 = it was a submarine.; 2 = there are too many fish around the ship.; 3 = none of above.; response: an earthquake would cause people to feel threatened, and fish around the ship would not matter. if the ship were a submarine, it would be designed to sink. the answer is 1 4. question: two mothers and two daughters were asking for new state ids, but the agent only gave out three forms and instructed them on how to fill them out. why? choices: 0 = they are one daughters, one mother and one grandmother.; 1 = two girls filled the from together.; 2 = one mother is too old to apply for new ids.; 3 = none of above.; response: the agent only gave out three forms and instructed them on how to fill them out, which suggests that there are only three people. if they are one daughter, one mother, and one grandmother, then there would be three people. the answer is 0 5. question: every night, a man would sleep with a light on, as bright as the sun, dazzling the neighbors. but why did his neighbors never complain? choices: 0 = all his neighbors are blind.; 1 = the man would wake up early.; 2 = he lived in a light house.; 3 = none of above.; response: if all his neighbors are blind, they would not be dazzled by the light. if the man lived in a light house then the light would be expected. the answer is 2 6. question: each of the 30 participants in the masquerade had to wear a unique hat to distinguish themselves from one another. the host, however, only tallied 29 when he counted the number of hats to determine attendance. all 30 persons had signed their names on the spreadsheet, which confused him. he repeated the count. there are still 29. how is that even doable? choices: 0 = one person had a pretty beautiful hat.; 1 = the host had a hat himself and he forget to count it.; 2 = one person had moved away from the group when the host was counting.; 3 = none of above.; response: if one person had a pretty beautiful hat, then the host would still count 30 hats. if one person had moved away from the group when the host was counting, then the host would still count 30 hats. the host had a hat himself and he forget to count it. the answer is 1 7. question: a certain kind of animal has parents but no children, lives happily but can not give birth to offspring. this kind of animal has existed on earth for a long history. how is that possible? choices: 0 = the animal is the dinosaur. millions of years ago, dinosaurs roamed the earth, but eventually, they became extinct due to various factors such as climate change or asteroid impact.; 1 = the animal is the butterfly. during the pupal stage of their life cycle, butterflies are inside a protective casing, and it may appear as if they have completely disappeared or been wiped out.; 2 = the animal is the mule. since all mules are born sterile, you can only get a mule by crossing a donkey with a horse.; 3 = none of above.; response: dinosaurs are no longer alive, so they cannot live happily. butterflies can give birth to offspring, so they are not the answer. mules are born sterile, so they cannot give birth to offspring. the answer is 2 8. question: six women were hiking on a trail when a sudden snowfall began. five of the women hurried along, but the sixth did not. still, they all reached the cabin at the same time, and all but the sixth were cold. how is that possible? choices: 0 = the 6th woman beated them at the end.; 1 = the sixth woman was in a photograph that one of the others was carrying.; 2 = the 6th woman walked slowly to dodge the snow.; 3 = none of above.; response: if the sixth woman beat them or walked slowly to dodge the snow she would not have reached the cabin at the same time. if the sixth woman was in a photograph that one of the others was carrying, then she would not be cold and would not have to hurry. the answer is 1 b.2 new mixed prompts 1. question: a horse is tied to a five-meter rope in front of an old saloon. ten meters behind the horse is a bale of hay. without breaking his rope, the horse is able to eat the hay whenever he chooses. how is this possible? choices: 0 = the rope stretches proportionally, provid-ents but no children, lives happily but can not give birth to offspring. this kind of animal has existed on earth for a long history. how is that possible? choices: 0 = the animal is the dinosaur. millions of years ago, dinosaurs roamed the earth, but eventually, they became extinct due to various factors such as climate change or asteroid impact.; 1 = the animal is the butterfly. during the pupal stage of their life cycle, butterflies are inside a protective casing, and it may appear as if they have completely disappeared or been wiped out.; 2 = the animal is the mule. since all mules are born sterile, you can only get a mule by crossing a donkey with a horse.; 3 = none of above.; response: the animal is still alive, has parents, but cannot have children. mules are born sterile, so they cannot give birth to offspring. the answer is 2 8. question: six women were hiking on a trail when a sudden snowfall began. five of the women hurried along, but the sixth did not. still, they all reached the cabin at the same time, and all but the sixth were cold. how is that possible? choices: 0 = the 6th woman beated them at the end.; 1 = the sixth woman was in a photograph that one of the others was carrying.; 2 = the 6th woman walked slowly to dodge the snow.; 3 = none of above.; response: all the women hurried except the sixth and were cold. the sixth woman is not physically present. she must be in a photograph one of the others was carrying. the answer is 1