Decomposed Prompting : A M ODULAR A PPROACH FOR S OLVING C OMPLEX T ASKS
Anonymous authors
Paper under double-blind review
A BSTRACT
Few-shot prompting is a surprisingly powerful way to use Large Language Models (LLMs) to solve various tasks.
However, this approach struggles as the task complexity increases or when the individual reasoning steps of the task themselves are hard to learn, especially when embedded in more complex tasks.
To address this, we propose Decomposed Prompting, a new approach to solve complex tasks by decomposing them (via prompting) into simpler sub-tasks that can be delegated to a shared library of prompting-based LLMs dedicated to these sub-tasks.
This modular structure allows each prompt to be optimized for its specific sub-task, further decomposed if necessary, and even easily replaced with more effective prompts, trained models, or symbolic functions if desired.
We show that the flexibility and modularity of Decomposed Prompting allows it to outperform prior work on few-shot prompting using GPT-3.
On symbolic reasoning tasks, we can further decompose sub-tasks that are hard for LLMs into even simpler solvable sub-tasks.
When the complexity comes from the input length, we can recursively decompose the task into the same task but with smaller inputs.
We also evaluate our approach on textual multi-step reasoning tasks: on long-context multi-hop QA, we can more effectively teach the sub-tasks via our separate sub-tasks prompts; and on open-domain multi-hop QA, we can easily incorporate a symbolic information retrieval module within our decomposition framework, leading to improved performance on both tasks.
I NTRODUCTION
Large Language Models (LLMs) such as GPT-3 (Brown et al., 2020) have been shown to solve various tasks given only a few examples as prompts, also referred to as in-context learning.
These models can even perform more complex reasoning tasks when shown the sequence of simple reasoning steps needed to perform the complex task as a prompt Wei et al.
; Nye et al.
In essence, the sequence of reasoning steps, such as in Chains-of-Thought (CoT) prompting (Wei et al., 2022), demonstrates how to decompose the complex task as well as how each reasoning step should be performed.
However, as tasks become more complex, few demonstrations of the complex task aren’t sufficient for current models to learn to perform all necessary reasoning steps.
E.g., fewshot demonstrations of concatenating the k th letter of words in a string is insufficient for GPT-3 to learn to extract the k th letter, or learn to answer hard single-hop questions when only provided a few demonstrations of multi-hop questions.
Additionally, it is unclear whether tasks such as document retrieval and integration, for knowledge-intensive tasks, can even be done by few-shot prompts.
To address these limitations, we propose Decomposed Prompting (D ECOM P), a new approach to solve complex tasks by instead decomposing them into simpler sub-tasks and delegating these to sub-task specific LLMs, with both the decomposer and the sub-task LLMs (henceforth, sub-task handlers ) having their own few-shot prompts.
Fig 1 illustrates our approach.
The decomposer prompt only describes a sequence of sub-tasks (A, B, and C) needed to solve the complex tasks, indicated with the dashed lines.
Each sub-task is then delegated to the corresponding sub-task handler shown on the right.
[Figure]
Using a software engineering analogy, the decomposer defines the top-level program for the complex task using interfaces to simpler, sub-task functions.
The sub-task handlers serve as modular, debuggable, and upgradable implementations of these simpler functions, akin to a software library.
If a particular sub-task handler, say the one for identifying the k th letter or retrieving a document, is not performing well enough, we can debug this handler in isolation, explore alternative prompts or implementations, and seamlessly plug the improved module back into the overall system, as a systematic way to try to improve performance on the complex end-task.
This approach has several advantages over prior work (as also shown in the figure).
The sub-task handlers can be shown a broader and richer set of examples (of the simpler task) than the specific ones needed for the complex task prompt (task A).
If a sub-task is too complex, it can be further decomposed into simpler sub-tasks (task B).
Similar to software libraries, these sub-task handlers can be shared across multiple tasks; e.g., here tasks A and C are reused in the model for task B.
As noted above, a sub-task handler can be easily swapped with an improved implementation without any change to the rest of the system.
Finally, few-shot prompt based LLMs can be even replaced with a symbolic system for tasks more suited for non-neural methods; e.g., task C uses a symbolic retrieval system such as Elasticsearch that can handle very large-scale corpora.
To illustrate these advantages of D ECOM P, we empirically evaluate it against prior work on four challenging datasets using GPT3 models: 1) On a task of concatenating the k th letter, we show that our approach of factoring out each sub-task allows us to more effectively teach the sub-problem of extracting the k th letter(specifically, by decomposing it into even easier sub-tasks).
On a task of reversing a list, we show that D ECOM P allows us to extend the capabilities of a weaker model and build a scale-invariant system by recursively decomposing the task into reversal of smaller and smaller lists.
On a task of long-context QA (Khot et al., 2022), our approach allows each subtask handler to accommodate more examples than feasible with CoT prompting leading to better
QA performance.
On multi-hop open-domain QA dataset Yang et al.
(2018), we can incorporate a symbolic retrieval (ElasticSearch) API as the handler for the retrieval sub-task leading to better results than CoT systems.
R ELATED W ORK
Few-shot Prompts for Multi-Step Reasoning Large-scale Language models (LLMs) have been shown to learn various NLP tasks given just few examples as prompts (Brown et al., 2020).
Recently, they have also been successfully applied to various multi-step reasoning tasks by providing the intermediate reasoning steps, i.e. Chain-of-Thought (Wei et al., 2022; Chowdhery et al., 2022), needed to arrive at the answer.
Building of the successes of Chain-of-thought (CoT) style prompting on LLMs, many subsequent studies have focused on improvements of this basic model.
This includes, for example, diverse decoding and ensembling to improve self-consistency (Wang et al., 2022c), more complex and contrasting rationales (Marasovi´c et al., 2021;
Wang et al., 2022b), generating sub-questions and answering them sequentially in least-to-most complexity order (Zhou et al., 2022), integration with symbolic modules (Jung et al., 2022), selection inference (Creswell et al., 2022), coupling in-context learning with sparse fine-tuning (Zelikman et al., 2022), and even presented many of them under a single probabilistic programming framework (Dohan et al., 2022).
We view these prior works as specialized systems with a pre-defined decomposition structure.
For example, the closest work to our approach is the idea of least-to-most prompting (Zhou et al., 2022) where one prompt is used to generate the sub-questions needed to answer a complex question and a second prompt sequentially answers these sub-questions.
In contrast, our approach allows for diverse decomposition structures and can use different prompts to answer each sub-question, even including symbolic systems.
Moreover, rather than generating the entire set of sub-questions (i.e., the decomposition) at once, we can iteratively generate new questions based on the answers to previous steps (e.g., our prompt for HotpotQA in Sec.
4.4 is one such use case.)
Modular Approaches for Multi-Step Reasoning Our work follows a long literature in NLP on neural modular modeling architectures (Andreas et al., 2016;
Talmor & Berant, 2018;
Min et al., 2019; Jiang & Bansal, 2019; Gupta et al., 2020; Perez et al., 2020; Khot et al., 2021; Levine et al., 2022) for question-answering and other tasks.
We take particular inspiration from the Text Modular
Networks approach of Khot et al.
(2021), whereby problem decomposition consists of a learned next question generator trained to generate questions in the language of a collection of textual and symbolic agents.
Best-first search strategy was used to explore the space of possible decompositions during inference.
In contrast to this work, which largely centered around supervised training of the next-question generator given existing agents , we leverage the power and recent successes of fewshot LLMs to build both the decomposer and the sub-task agents that best fit the ideal decomposition.
This has the advantage of obviating the need for specialized supervised training data that may not always be available for all sub-tasks – a key bottleneck of this prior work.
Neural Programming Our work also relates to work on neural programming architectures (Reed & De Freitas, 2016;
Neelakantan et al., 2016;
Cai et al., 2017), which aims at building neural models that can represent and execute compositional programs, often focusing on algorithmic tasks of the type we investigate here.
Specifically, our top-level decomposer prompts with their corresponding sub-task handlers (see Figure 1), which get constructed incrementally and interpreted by the underlying LLM, can be interpreted as structured programs where the the underlying program construction and interpretation process is performed using LLMs.
D ECOMPOSED P ROMPTING
As with conventional few-shot prompting, the goal is to teach an LLM to find an answer A to a query Q using a small set of in-context examples D
= { E 1 , ..., E | D | } .
The answer A is obtained from the underlying distribution p
( A | Q, D, θ ) (Dohan et al., 2022).
In the most basic few-shot setup, examples take the form E j =
( Q j , A j ) .
In the case of CoT-style prompting, the goal is to obtain answers by first generating a sequence or chain of intermediate reasoning steps or “thoughts”
T , and then deriving the final answer based on T .
To teach this ability, one uses more sophisticated in-context examples that take the form E j = ( Q j , ( T j, 1 , . .
, T j,k ) , A j ) .
In D ECOM P, the core is a decomposer LLM that tries to solve a complex task by generating a prompting program P for it.
Each step of P directs a simpler sub-query to a function in an auxiliary set of sub-task functions F available to the system.
Given a query Q whose answer is
A , the program P is a sequence of the form (cid:0) ( f 1 , Q 1 , A 1 ) , ..., ( f k , Q k , A k ) (cid:1) where A k is the final answer predicted by the program and Q i is a sub-query directed to the sub-task function f i ∈ F .
The prompting program P is executed by a high-level imperative controller , which passes the inputs and outputs between the decomposer and sub-task handler until a stopping condition is met in P and a final output is obtained.
To teach the decomposer LLM in a few-shot prompting manner, we use in-context examples that take the form E j = (cid:0) ( Q j , (cid:0) f j, 1 , Q j, 1 , A j, 1 ) , ..., ( f j,k j , Q j,k j , A j,k j ) (cid:1)(cid:1) where A j,k j
=
A j is the final answer for Q j and ( Q j, 1 , . . .
, Q j,k j ) is a decomposition of Q j .
Each sub-task function f , in turn, is operationalized via a sub-task handler as an in-context prompting LLM (e.g., a separate CoT-style prompt or a additional prompting program dedicated to that sub-task), or any other symbolic or learned function (e.g., a calculator or specialized supervised trained model).
[Figure] [Figure]
3. D ECOMPOSED P ROMPTS
To illustrate this with an example
[Figure]
consider a multi-step task such as “Concatenate the first letter of every word in str using a space”.
We can solve this task by decomposing it into a sequence of three simple sub-tasks: 1) Collect the list of words in the str ; 2) For each word, extract the third letter; 3) Concatenate the extracted letters using space as the separator.
Figure 2 shows an example decomposition prompt for this task.
Much like a conventional structured program, the top-level decomp prompt provides an example program E
j using three sub-task functions: f 1 : split that splits words in an input string , f 2 : str pos that finds character positions in strings and f 3 : merge that concatenates characters .
In this case, we operationalize each sub-task function as a separate in-context prompt (e.g., using a standard prompting approach for split and merge on the right side), each containing a set of in-context examples that are independent of the original complex task.
In addition to the three functions described above, additional control structure is included, such as the symbolic function foreach , which iterates over arrays and references to previous answers such as #1.
We note that such a helper function is not strictly necessary (e.g., we could directly generate “Q2’:
What is the first letter of Jack?”
and “Q3’:
What is the first letter of Ryan?”
instead of Q in the figure)
and is added to reduce the manual effort needed to specify the decomposition and also reduce potential errors during decomposition.
In our experiments we use two of the compositional operators defined by Khot et al.
(2022) (see appendix for details), although it is capable of using all their operators (which also capture the QDMR operators from Wolfson et al. [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
3. P ROMPT E XECUTION AND I NFERENCE
Given a new question and a set of background in-context examples D the inference (i e. the program construction and execution) process is illustrated in Fig.
The new complex question is fed to the decomposer prompt to get the first sub-question to be asked to the split prompt.
With the help of our symbolic controller, the answer generated from this prompt is then appended to the decomposer prompt to get the second sub-question, Q 2 .
Due to the foreach operator in the generated question, Q 2 results in two questions (one for each word in the answer #1 ) to be fed to the str pos prompt.
The answers are combined into an array to get the answer #2 .
The entire decomposition history is used to generate Q 3 and passed to the merge prompt to get the final answer.
Since the task has been solved, the decomposition prompt produces the special end-of-sequence marker([EOQ]) and the last answer is returned as the final answer.
Formally, performing inference involves finding the best answer A to a new query Q , which in the simplest form of prompting involves computing the MAP answer using the LLMs predictive distribution for A , i.e., ˆ A = arg max A p
( A | D, Q, θ ) (Dohan et al., 2022).
For practicality, such computations are approximated using greedy search, which we pursue in our experiments.
3. D ECOM P C APABILITIES
Next we describe three types of few-shot prompting solutions that our approach enables.
Hierarchical Decomposition Certain sub-
tasks, even when given many examples, are not solvable with few-shot prompting.
E.g., we found identifying the k th letter of a string to be challenging for the GPT3 text-davinci-002 model.
In such a scenario, we can decompose the sub-task prompt further, to first identify the letters and their position and then select the k th element of this array (see Fig. 4).
We can also re-use existing sub-task prompts in our framework.
E.g., the split prompt can be reused since it was developed for the general task of splitting strings.
[Table]
), (o, 3), (g, 4), (l, 5), (e, 6)]" QS: [arr_pos] What is at position 1 in #1?
A: "G" QS: [EOQ] .... str_pos
Q: What is at position 1 in "[("Colorless", 1), ("green", 2), ("ideas", 3), ("sleep", 4), ("furiously", 5)]"?
A: "Colorless"
Q: What is at position 2 in "[(G, 1), (o, 2), (o, 3), (g, 4), (l, 5), (e, 6)]"?
A: "o" .... arr_pos
Recursive Decomposition
Some problems can be naturally broken down into one or more smaller problems of the same form.
Recursive algorithms such as merge sort use this idea to solve large problems efficiently, using a succinctly described method.
We apply this same principle in D ECOM P by allowing the decomposer prompt to recursively call itself, as shown in Fig.
5 for the task of list reversal.
By using recursion, we are able to generalize any base prompting approach (CoT in this figure) to much longer lists by breaking the input into smaller and smaller lists till we reach a list length where the model is highly accurate.
Such recursive approaches can not be described by current methods such as CoT and standard prompting.
Least-to-most prompting (Zhou et al., 2022) also proposes a similar solution but differs in two key aspects (a) it has to identify all the subproblems in one-shot instead of our iterative top-down decomposition (b) it has to learn to identify the relevant answers from the previous solutions which we get for free from our decomposition.
External API Calls In certain cases, the sub-tasks may not be feasible to solve using only a LLM.
E.g., retrieving knowledge from a KB or large corpus.
Such sub-tasks, however, can be easily solved using existing systems such as retrieving documents using an Elasticsearch index or webpages using
Google search Lazaridou et al.
Fig.
6 shows how D ECOM P can easily use such a system to retrieve the relevant documents and answer a single-hop open-domain question.
[Figure] [Figure]
this prompt for open-domain multihop questions is given in Figure 11.
C ASE S TUDIES
We showcase D ECOM P’s strengths through four tasks; two symbolic manipulation tasks similar to those investigated in Wei et al.
and two existing textual multi-hop reasoning tasks.
Unless specified, we use text-davinci-002 InstructGPT3 model Ouyang et al.
as the LLM and report the Exact Match (EM) numbers, following prior work.
For order-independent list answers, we evaluate set equality as EM.
To show the value of decomposed prompting, we compare our approach to CoT rather than each specific decomposition structure used in prior work.
The complete prompts for all our tasks are provided in App.
G
4. k th LETTER CONCATENATION (H IERARCHICAL D ECOMPOSITION )
We compare D ECOM P to CoT prompting for concatenating letters at the k th position.
All prompts contain examples of concatenating letters in position 1, 4, and last position of strings with 3 words.
We create three different prompts for all our baselines and present the average to account for variance due to the choice of examples following Perez et al.
We use the decomp , split , str pos (further decomposed as shown in Fig. 4), and merge prompts for decomposition prompting.
We adapt the CoT for last letter concatenation from prior work (Wei et al., 2022) for this task as shown below.
In addition, we consider a rolled out version of our decomposition prompts in terms of a
CoT, i.e., we describe the entire decomposition process (identify words, split each word into letters, take k th letter and concatenate) as a single CoT. e.g, for the question “Take the letters at positionof the words in ”Herbert Alexander Simon” and concatenate them using a space.”
, we use the CoT:
Chain-Of-Thought
The letter at position 4 of ”Herbert” is ”b”.
The letter at position 4 of ”Alexander” is ”x”.
The letter at position 4 of ”Simon” is ”o”.
Concatenating ”b”, ”x”, ”o”
using a space leads to ”b x o”.
So, ”Herbert Alexander Simon” outputs ”b x o”. ...
Chain-Of-Thought (rolled out)
(r t
We compare these three prompting techniques on 4 datasets to evaluate generalization along 3 axes: (1) unobserved letter position k
= 3 3 (2) longer inputs, #words=4 and 5 (3) new delimeter ”semicolon”.
The words in the test examples come from a list of most popular first and last names.
4 All evaluation datasets have 100 examples.
See Fig.
7 for results.
E M
N=3 (IID)
N=4 (OOD)
N=5 (OOD)
CoT
CoT w/ rollout
Decomp (a) Space
E M
N=3 (IID)
N=4 (OOD)
N=5 (OOD)
CoT
CoT w/ rollout
Decomp (b) Semi-Colon
D ECOM P outperforms chain-of-thought reasoning , even when the chain-of-thought uses the same reasoning procedure as the rolled out decomposition.
This shows that the separate prompts are more effective at teaching hard sub-tasks than a single CoT prompt.
D ECOM P generalizes perfectly to longer sequences.
As the length of the input sequence increases, our approach continues to achieve close to 100% accuracy on this task.
The CoT-based approaches drop noticeably in their scores with longer input lengths, widening the performance gap.
4. L IST R EVERSAL (R ECURSIVE D ECOMPOSITION )
We use the task of reversing lists of words 5 to show how recursive D ECOM P enables length generalization.
We adapt the relevant CoT prompt from Wei et al.
and integrate it in a decomposed prompt.
As a control, we also compare to a CoT version w/ rollout of our decomposed prompt.
All prompts contain the same 3 examples of reversing word sequences with 3-5 items.
We evaluate all prompts for generalization to 4, 6, 8, and 10-item sequences.
Here we use davinci-001 to show that D ECOM P enables a weaker model approach davinci-002 ’s performance (which does solve this task).
We use the strategy from Fig.
5 and provide our prompts in App.
G. Figure 8 shows the results of the prompting strategies on different input lengths.
D ECOM P improves the length generalization of few-shot prompting.
While our base CoT prompt does not generalize at all to longer sequences, our approach can recursively decompose the problem and achieve better length generalization.
CoT w/ rollout fails.
The CoT version of our decomposition strategy fails because the unrolled prompt becomes too long and convoluted without the ability to abstract away sub-modules.
[Figure]
Incorporating CoT in D ECOM P greatly increases the ability of the model to generalize to new sequence lengths.
[Figure]
4. L ONG -C ONTEXT Q UESTION A NSWERING
We next evaluate on the CommaQA-E dataset (Khot et al., 2022) under the reading comprehension setting.
The dataset consists of synthetically generated entities (e.g. Erowid award), facts (“Wetherality was an actor in the movie Dewbar.”) and multi-hop questions (e.g., “What awards have the actors of the Erowid winning movies received?”).
Due to the presence of many distractors and, as a result, longer context, this dataset has been shown to be hard for standard LMs even when fine-tuned.
To fit these questions within GPT3’s context limit (2049 tokens), we generate a smaller version of the CommaQA-E dataset and of the compositional generalization split such that we can fit at least four examples in the context for CoT prompts.
For the question “What awards have movies produced by people born in 1910 won?”, the CoT is: The people born in 1910 were Muntaril and Teeplemole.
Teeplemole produced the movies Featsaw and Zalate.
Muntaril produced the movie Premercy.
Featsaw was awarded the Zorgion award.
Premercy was awarded the Chowwurst award.
Zalate was awarded the Hallowcock award.
So the answer is [”Zorgion”, ”Chowwurst”, ”Hallowcock”].
[Table]
_qa] Who were born in the year 1910?
A: ["Teeplemole", "Muntaril"] QS: (foreach_merge) [pos_qa] For which movies was #1 the producer?
A: ["Featsaw", "Zalate", "Premercy"] QS: (foreach_merge)
[aw_qa] Which awards were given to #2?
A: ["Zorgion", "Chowwurst", "Hallowcock"]
[Figure] [Figure]
type allowing the
For D ECOM P, we can separate the task of decomposition (independent of the context) from the subtasks of single-hop question answering.
As shown in Fig.
10, we provide examples of the contextindependent decomposition in the decomposer prompt and use the separate sub-task prompts to teach the QA skill over the given context.
Additionally, we can choose the granularity of decomposition to trade off human effort for increased accuracy.
For example, we could have single QA prompt to handle all the questions or create QA prompts for different classes of questions.
In our experiments, each sub-task prompt contains 8 QA examples (2 questions/para)
(see App.
G for the prompts).
We evaluate three different prompts and report the average results in Fig. 9.
D ECOM P is more accurate than CoT on CommaQA.
Irrespective of the granularity of decomposition or the evaluate split, our approach outperforms CoT on this task.
Finer grained decomposition can help improve performance .
Our fine-grained decomposition can provide more examples for each class of questions resulting in increased single-hop QA performance and as a result increased performance on CommaQA.
D ECOM P also generalizes to new compositions.
Compositional generalization split of CommaQA uses the same relations as the training set but with new compositions.
While CoT has a drop in score, both decomposition-based approaches even have slightly better scores (the subset of relations used in Comp.
Gen split are easier for our specialized QA models).
[Figure]
4. O PEN -D OMAIN Q UESTION A NSWERING
Next, we demonstrate the ability of our approach to integrate external API calls on the task of opendomain multihop question answering.
We evaluate our approach on the HotpotQA dataset under the fullwiki setting that requires relevant paragraphs to be retrieved from a Wikipedia corpus.
We use the code-davinci-002 model here since it can fit the much longer contexts needed here.
Fig.
11 shows the decomposition prompt we use.
The decomposer generates (singlehop) subquestions and delegates them to retrieve odqa (described in Fig. 6).
As we showed earlier, this module retrieves relevant documents then uses an RC model to answer.
retrieve odqa returns both the answer and the documents, allowing subsequent sub-questions to use the answers (e.g.
“Mack Rides”) and the multihop rcqa model to use the documents.
We implement the final multihop rcqa model in two ways: (i) prompting the model to generate the answer directly ( Decomp-Ctxt QA ) as shown in Fig.
11 and (ii) prompting the model to generate the Chain-of-Thought ( Decomp-Ctxt CoT QA ) (e.g. “The Lost Gravity was manufactured by Mack
Rides.
Mack Rides is a German company.
So the answer is: Germany.”).
We compare our approach against two baselines: A. No Context (No-Ctxt), A closed-book setting baseline where the model must rely only on its parametric knowledge.
B. NoDecomp Context (NoDecomp-Ctxt), A simple retrieval baseline where we retrieve K (set to 8) paragraphs using the multi-hop question as the input and use that as context.
In both these baselines, we consider the QA and CoT QA variants.
We manually annotate CoTs and decompositions for 25 training set questions, and sample 3 prompts of 15 questions each for all approaches.
The detailed prompts are given in the Appendix G. We evaluate on the subset of bridge questions from our held-out dev questions (248/300) i.e. questions where answer for the first sub-question is needed for the second sub-question (as in our example).
On the comparison questions in HotpotQA (e.g. “Who is older Jeremy Horn or Renato Sobral ?”), the relevant entities are clearly mentioned in the question and do not necessitate multi-step retrieval.
We also evaluate our approach on 2WikiMultihopQA Ho et al.
and analyse how the model performance changes with increasing model sizes.
The results are provided in the Appendix A.
Decomp
Results are presented in Fig.
The NoDecomp-Ctxt models perform significantly better than NoCtxt models in both QA (i.e. standard prompting) and CoT variants showing that external knowledge can be leveraged to improve standard and CoT based open-domain mulithop QA.
Furthermore, our
Decomp-Ctxt models are yet better at leveraging external knowledge than a strong retrieval baseline (NoDecomp-Ctxt), regardless of whether we leverage CoT jointly with our approach or not.
C ONCLUSION
We proposed a new approach, Decomposed Prompting, to solve complex tasks using few-shot prompts, by decomposing them into a prompting program built out of simpler sub-tasks.
Drawing inspiration from software libraries, our decomposer and shared sub-tasks are designed in a modular fashion: they use their own few-shot prompts, allowing one to independently optimize each prompt, decompose a sub-task further if necessary or even seamlessly replace it with a symbolic system.
We show that Decomposed Prompting outperforms prior work on four different tasks and generalization settings, establishing it as an effective few-shot paradigm for solving complex tasks
A
[Equation]
IKI M ULTI H OP QA
We also evaluate DecomP for open-domain QA on 2WikiMultihopQA Ho et al.
This dataset is reading-comprehension multihop QA dataset created by composing entities and relations on WikiData.
So we first convert it to an open-domain setting by taking a union of all the paragraphs present in the dataset and treating as corpus.
In all it has 430225 paragraphs.
We manually annotate 20 questions with chain-of-thought prompts an DecomP prompts and manually sample 15 of them which are used as demonstrations.
To further analyse the effect of model-scale on our method, we also experiment with different sized
Flan-T5 models Chung et al.
(2022) (large, xl, xxl), which have been recently shown to exhibit chain-of-thought reasoning.
For DecomP approach, we still use GPT3 (text-davinci-002) for decomposition as Flan variations could not generate syntactically correct decompositions.
The results are show in the Figure 13.
[Figure]
B M ATH QA
We apply Decomposed Prompting to two math QA datasets: GSM8K Cobbe et al.
and MultiArith Roy & Roth (2015).
For Chain-of-thought, we used the original prompts for math reasoning Wei et al.
For example:
Q: There are 15 trees in the grove.
Grove workers will plant trees in the grove today.
After they are done, there will be 21 trees.
How many trees did the grove workers plant today?
A: There aretrees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 - 15 = 6.
The answer is 6.
Most CoT systems Wei et al.
; Wang et al.
(2022a) rely on extracting the answer by finding the number following “answer is”.
However, this may not always be accurate.
For example, the following CoT would be unanswerable by relying on simple patterns
Parker chews 4 pieces of gum a day.
There are 15 pieces of gum in a pack.
So he will need 4 * 30 / 15 = 8 packs of gum to last him 30 days.
Rather than relying on patterns with limited generalization, we can use a language model to extract the answer more reliably.
Specifically, we use Decomposed Prompting to decompose the task into first identifying the chain-of-thought reasoning and then using a second GPT3-based sub-module to extract the answer from the CoT. We show examples of our prompts here (full prompt in App.
G):
[Figure] [Figure]
Decomposition Prompt
QC:
There are 15 trees in the grove.
Grove workers will plant trees in the grove today.
After they are done, there will be 21 trees.
How many trees did the grove workers plant today?
QS: [cot]
There
are
15 trees in the grove.
Grove workers will plant trees in the grove today.
After they are done, there will be 21 trees.
How many trees did the grove workers plant today?
A: There are 15 trees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 - 15 = 6 trees planted.
QS: [gpt ans]
There are 15 trees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 - 15 = trees planted.
A:QS: [EOQ] gpt ans Q: There are 15 trees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 - 15 = 6 trees planted.
A:
We present our results in Fig.
On the GSM8K data set 8 , we outperform CoT by 14 points.
On the
MultiArith dataset 9 , we achieve a 17 pt improvement compare to CoT. While this is a simple change, it illustrates the possibility of using D ECOM P for other complex answer types, e.g. non-extractive answer generation from chain-of-thoughts.
C E FFECT OF S CALE ON C OMM AQA
We evaluate text-curie-001, text-davinci-001 and text-davinci-002 on the CommAQA dataset.
Since the curie-001 and davinci-001 have a smaller context window size, we further reduced our prompts to fit within their context windows (2048 tokens).
As shown in Fig.
15, both CoT and D ECOM P are effected by the model size.
D R ESULTS ON ALL PROMPTS D. P ER -P ROMPT R ESULT ON L ETTER C ONCATENATION
We present the results of the letter concatenation task (with space delimiter) for different values of N in Fig.
Our results are stable across the different prompts (P1, P2 and P3) and always outperform CoT. We also compare our system to least-to-most prompting Zhou et al.
(2022) (prompts in App. G).
Since least-to-most prompting can not decompose the k th letter extraction sub-task further, its performance is similar to CoT and much worse than D ECOM P.
[Figure]
M
[Figure]
D. P ER -P ROMPT R ESULTS OM C OMMA QA
We present the results of all the prompts on the CommAQA dataset in Fig.
[Figure]
E E FFECT OF D ECOMPOSITION S CHEME
To evaluate the effect of the decomposition scheme, we experiment with two other simple decomposition structures for the letter concatenation and reversal tasks.
For letter concatenation, we consider an alternate scheme where we use GPT3 to generate each question rather than loop over the answers, i.e.,
Replace
A: [”Augusta”, Ada”, King”] QS: (foreach) [str position]
What is the last letter in
with
QS: [str position]
What is the last letter in ”Augusta”?
A: ”a” QS: [str position]
What is the last letter in ”Ada”?
A: ”a” QS: [str position]
What is the last letter in ”King”?
[Equation]
By using the decomposer prompt model to generate these sub-questions, we can be more robust to formatting issues in the output answers, but at the cost of potential precision or recall mistakes in generating these questions.
For list reversal, instead of splitting into halves, we take the tail of the list, reverse it and then concatenate it to the head.
i.e. reverse(list) =
reverse(list[1:])
+ list[0].
This requires more GPT calls (O(n)) compared to the original approach of splitting the list into halves.
In both these cases, we noticed that the performance did not drop as shown in Fig.
18 and Fig.
On the letter concatenation task, the results were exactly the same.
The new reversal decomposition schema was actually stronger on longer inputs at the cost of more calls to GPT3 (O(ln(n)) using binary splits vs O(n) one element at a time).
Both these decomposition schemes are still better than
CoT.
[Figure] [Figure]
at
F E RRORS F. L ETTER C ONCATENATION F.1. D ECOM P
Q: Take
the
letters at position 3 of
the
words in ”Nancy Samina Abbas Caudhari Bano” and concatenate
them
using a space.
A: n m b u n
Prediction: c m b u n
Error: Incorrect letter extraction (Sub-task)
What is
at
position
in
”[(N, 1), (a, 2), (n, 3), (c, 4), (y, 5)]”?
⇒ ”c”
Q: Take the letters at position 3 of the words in
”Orlando Stephen Cho Teixeira Pierre”
and
concatenate
them using a space.
A: l e o i e
Prediction: leoie
Error: Incorrect concatenation (Sub-task)
Concatenate [”l”, ”e”, ”o”, ”i”, ”e”]
using
a space.
⇒ ”leoie”
F.1.
C O T W / ROLLOUT
Q: Take the letters at position 3 of the words in ”Sheila
Nicolas
Verma Sha Sousa” and concatenate them using a space.
A: e c r a u
Pred: i c r a u
Error: Incorrect letter extraction ...
The letters and their positions in ”Sheila” are ”[(S, 1), (h,
), (e, 3), (i, 4), (l, 5), (a, 6)]”.
The letter at position 3 in this sequence is ”i”...
Q:Take the letters at position 3 of the words in ”Shobha Kailash Nakamura Peter Benitez” and concatenate them using a space.
A:
o i k t n Pred: o l k t i Error: Incorrect letter extraction ...”Benitez” are ”[(B, 1), (e, 2), (n, 3), (i, 4), (t, 5), (e, 6), (z, 7)
The letter at position 3 in this sequence is ”i”...
[Table]
Similarly in CommaQA, the errors are mostly due to sub-task errors, which in this dataset correspond to answering single-hop questions.
CoT also makes the same types of errors but they are more frequent since this QA sub-task can not be delegated to a specialized prompt in CoT. Since all errors are of this type, we show only one example here.
Q: What awards have movies written by people born in 1933 won?
A: [”Hydrallium”, ”Pompasole”]
Pred: [”Pompasole”]
Error: Incorrect sub-question answer Sub-Q:What movies has Haldron written?
Sub-A: [”Polytetrafluoromethane”, ”Skia”, ”Warpstone”]
Pred: [”Skia”, ”Warpstone”]
Similar sub-task errors are present in the list reversal and open-domain QA datasets.
G T ASK P ROMPTS
We have provided the task prompts for all the datasets for COT and our Decomposed Prompting approach.
CoT Since CoT methods also perform 2-step reasoning: first generate the chain-of-thought and second extract the answer from the CoT, we use the same decomposition-based framework for COT baselines too.
For example, consider the following example in our COT prompt:
QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS: [extract] The letter at position 1 of ”Alan” is ”A”.
The letter at position 1 of ”Mathison” is ”M”.
The letter at position 1 of ”Turing” is ”T”.
Concatenating ”A”, ”M”, ”T” using a space leads to ”A M T”.
So, ”Alan Mathison Turing” outputs ”A M T”.
A: ”A M T” QS: [EOQ]
GPT3 generates the chain-of-thought during the ”decomposition” step and a regex-based answer extractor extract ( ’.
* outputs "(.*)" \ .’ )
then takes this CoT and generates the answer.
In some cases, the module name is skipped in the prompt (the CoT is sent to the extractor by default).
Operators In this work, we use the same operators as defined by Khot et al..
Their select operator is just the basic operator that replaces references to an answer index with its answer.
When not specified, select is assumed to be the default operator.
In addition, we consider two operators in our experiments: project values and project values flat unique .
• project values : This operator takes a list answer # i =
X
and iterates over it to generate new questions by replacing mentions of #
i
i
e. Q =
[q.replace(#i, x) for x ∈ X].
The answer to each question is simply concatenated to get the final answer i
e. A = [model(q) for q ∈ Q].
We refer to this as foreach for simplicity in the main text.
• project values flat unique : This operator performs the same steps as project values but then additionally flattens the list and only returns the unique entities in the flattened list.
We refer to this as foreach merge in the main text for simplicity.
G. L ETTER C ONCATENTATION
We show one of the prompts used for experiments here.
The entire set of prompts is provided as supplemetary material.
[Table]
[Table] [Table]
D
[Table]
QC: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
QS: [split] What are the words in ”Augusta Ada King”?
A: [”Augusta”, ”Ada”, ”King”] QS: (project values) [str position]
What is the last letter in ”#1”?
A: [”a”, ”a”, ”g”]
QS: [merge] Concatenate #2 using a space.
A: ”a a g” QS: [EOQ] QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS: [split] What are the words in ”Alan Mathison Turing”?
A: [”Alan”, ”Mathison”, ”Turing”]
QS: (project values) [str position]
What is the letter at position 1 in ”#1”?
A: [”A”, ”M”, ”T”] QS: [merge] Concatenate #2 using a space.
A: ”A M T” QS: [EOQ] QC: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
QS: [split] What are the words in ”Herbert Alexander Simon”?
A: [”Herbert”, ”Alexander”, ”Simon”]
QS: (project values) [str position]
What is the letter at position 4 in ”#1”?
A: [”b”, ”x”, ”o”] QS: [merge] Concatenate #2 using a space.
A: ”b x o” QS
[Table]
[EOQ]
[Table]
Q: What are the words in ”Alan Mathison Turing”?
A: [”Alan”, ”Mathison”, ”Turing”]
Q: What are the letters in ”Alan”?
A: [”A”, ”l”, ”a”, ”n”]
Q: What are the letters and their positions in ”Mathison”?
A: ”[(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]” Q: What are the words and their positions in ”Herbert Alexander Simon”?
A: ”[(Herbert, 1), (Alexander, 2), (Simon, 3)]”
str position
QC:
What is the letter at position 1 of the word ”Augusta”?
QS: (select) [split] What are the letters and their positions in ”Augusta”?
A: ”[(A, 1), (u, 2), (g, 3), (u, 4), (s, 5), (t, 6), (a, 7)]” QS: (select) [arr position]
What is at position 1 in #1?
A: ”A” QS: [EOQ] QC:
What is the last letter of the word ”Mathison”?
QS: (select) [split] What are the letters and their positions in ”Mathison”?
A: ”[(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]” QS: (select) [arr position]
What is the last letter in #1?
A: ”n”
[Table]
[EOQ] QC:
What is the word at the position 4 in ”Colorless green ideas sleep furiously”?
QS: (select) [split] What are the words and their positions in ”Colorless green ideas sleep furiously ”?
A: ”[(Colorless, 1), (green, 2), (ideas, 3), (sleep, 4), (furiously, 5)]” QS: (select) [arr position]
What is at the position 4 in #1?
A: ”sleep” QS: [EOQ]
[Table]
Q: Concatenate [”A”, ”l”, ”a”, ”n”].
A: ”Alan” Q: Concatenate [”b”, ”x”, ”o”] using a space.
A: ”b x o” Q: Concatenate [”a”, ”a”, ”g”] using a comma.
A: ”a,a,g” Q: Concatenate [”Alan”, ”Mathison”, ”Turing”] using a space.
A: ”Alan Mathison Turing” Q: Concatenate [”Allen”, ”Institute”].
A: ”AllenInstitute”
arr position
Q: What is at position 4 in ”
[(”Colorless”, 1), (”green”, 2), (”ideas”, 3), (”sleep”, 4), (”furiously”, 5)]”?
A: ”sleep” Q: What is at position 1 in ”
[(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]”?
A: ”M” Q: What is at the last position in ”[(A, 1), (u, 2), (g, 3), (u, 4), (s, 5), (t, 6), (a, 7)]”?
A: ”a” Q: What is at position 1 in ”
[(Herbert, 1), (Alexander, 2), (Simon, 3)]”?
A: ”Herbert” Q: What is at last position in ”[(Allen, 1), (Institute, 2), (for, 3), (Artificial, 4), (Intelligence, 5)]”?
A: ”Intelligence” Q: What is at position 4 in ”
[(A, 1), (l, 2), (e, 3), (x, 4), (a, 5), (n, 6), (d, 7), (e, 8), (r, 9)]”?
A: ”x”
G.1. COT WITH ROLLOUT COT w/ rollout
QC: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
QS:
The words in ”Augusta Ada King” are ”Augusta”, ”Ada” and ”King”.
The letters and their positions in ”Augusta” are ”
[(A, 1), (u, 2), (g, 3), (u, 4), (s, 5), (t, 6), (a, 7)]”.
The last letter in this sequence is ”a”.
The letters and their positions in ”Ada” are ”
[(A, 1), (d, 2), (a, 3)]”.
The last letter in this sequence is ”a”.
The letters and their positions in ”King” are ”
[(K, 1), (i, 2), ( n, 3), (g, 4)]”.
The last letter in this sequence is ”g”.
Concatenating ”a”, ”a”, ”g” using a space leads to ”a a g”.
So, ”Augusta Ada King” outputs ”a a g”.
A: ”a a g” QS: [EOQ]
QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS:
The words in ”Alan Mathison Turing” are ”Alan”, ”Mathison”, and ”Turing”.
The letters and their positions in ”Alan” are ”
[(A, 1), (l, 2), (a, 3), (n, 4)]”.
The letter at position 1 in this sequence is ”A”.
The letters and their positions in ”Mathison” are [(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]”.
The letter at position 1 in this sequence is ”M”.
The letters and their positions in ”Turing” are ”[(T, 1), (u, 2), (r, 3), (i, 4), (n, 5), (g, 6)]”.
The letter at position 1 in this sequence is ”T”.
Concatenating ”A”, ”M”, ”T” using a space leads to ”A M T”.
So, ”Alan Mathison Turing” outputs ”A M T”.
A: ”A M T” QS: [EOQ] QC: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
QS:
The words in ”Herbert Alexander Simon” are ”Herbert”, ”Alexander”, and ”Simon”.
The letters and their positions in ”Herbert” are ”
[(H, 1), (e, 2), (r, 3), (b, 4), (e, 5), (r, 6), (t, 7)]”.
The letter at position 4 in this sequence is ”b”.
The letters and their positions in ”Alexander” are ”
[(A, 1), (l, 2), (e, 3), (x, 4), (a, 5), (n, 6), (d, 7), (e, 8), (r, 9)]”.
The letter at position 4 in this sequence is ”x”.
The letters and their positions in ”Simon” are ”[(S, 1), (i, 2), (m, 3), (o, 4), (n, 5)]”.
The letter at position 4 in this sequence is ”o”.
Concatenating ”b”, ”x”, ”o” using a space leads to ”b x o”.
So, ”Herbert Alexander Simon” outputs ”b x o”.
A: ”b x o”
[Table] [Table]
QC: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
QS:
The last letter of ”Augusta” is ”a”.
The last letter of ”Ada” is ”a”.
The last letter of ”King” is ” g”.
Concatenating ”a”, ”a”, ”g” using a space leads to ”a a g”.
So, ”Augusta Ada King” outputs ”a a g”.
A: ”a a g” QS: [EOQ] QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS:
The letter at position 1 of ”Alan” is ”A”.
The letter at position 1 of ”Mathison” is ”M”.
The letter at position 1 of ”Turing” is ”T”.
Concatenating ”A”, ”M”, ”T” using a space leads to ”A M T”.
So, ”Alan Mathison Turing” outputs ”A M T”.
A: ”A M T” QS: [EOQ] QC: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
QS:
The letter at position 4 of ”Herbert” is ”b”.
The letter at position 4 of ”Alexander” is ”x”.
The letter at position 4 of ”Simon” is ”o”.
Concatenating ”b”, ”x”, ”o” using a space leads to ”b x o”.
So, ”Herbert Alexander Simon” outputs ”b x o”.
A
[Table]
”b x o”
[Table] [Table]
G.1. L EAST - TO - MOST W / ROLLOUT Least-to-most Decomp
QC: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
QS: [l2m] Take the last letters of the words in ”Augusta Ada” and concatenate them using a space.
A: The words in ”Augusta Ada” are ”Augusta” and ”Ada”.
The letters and their positions in ” Augusta” are ”
[(A, 1), (u, 2), (g, 3), (u, 4), (s, 5), (t, 6), (a, 7)]”.
The last letter in this sequence
is ”a”.
The letters and their positions in ”Ada” are ”
[(A, 1), (d, 2), (a, 3)]”.
The last letter in this sequence is ”a”.
Concatenating ”a”, ”a” using a space leads to ”a a”.
So, ”Augusta Ada” outputs ”a a”.
QS: [l2m] Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
A: ”Augusta Ada” outputs ”a a”.
The letters and their positions in ”King” are ”
[(K, 1), (i, 2), (n, 3), (g, 4)]”.
The last letter in this sequence is ”g”.
Concatenating ”a a”, ”g” using a space leads to ”a a g”.
So, ”Augusta Ada King” outputs ”a a g”.
QS: [ans ext]
So, ”Augusta Ada King” outputs ”a a g”.
A: ”a a g” QS: [EOQ] QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS: [l2m] Take the letters at position 1 of the words in ”Alan Mathison” and concatenate them using a space.
A: The words in ”Alan Mathison” are ”Alan” and ”Mathison”.
The letters and their positions in ” Alan” are ”
[(A, 1), (l, 2), (a, 3), (n, 4)]”.
The letter at position 1 in this sequence is ”A”.
The letters and their positions in ”Mathison” are [(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]”.
The letter at position 1 in this sequence is ”M”.
Concatenating ”A”, ”M” using a space leads to ”A M”.
So, ”Alan Mathison” outputs ”A M”.
QS: [l2m] Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
A: ”Alan Mathison” outputs ”A M”.
The letters and their positions in ”Turing” are ”[(T, 1), (u, 2), (r, 3), (i, 4), (n, 5), (g, 6)]”.
The letter at position 1 in this sequence is ”T”.
Concatenating ”A M”, ”T” using a space leads to ”A M T”.
So, ”Alan Mathison Turing” outputs ”A M T”.
QS: [ans ext]
So, ”Alan Mathison Turing” outputs ”A M T”.
A: ”A M T” QS: [EOQ] QC: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
QS: [l2m] Take the letters at position 4 of the words in ”Herbert Alexander” and concatenate them using a space.
A: The words in ”Herbert Alexander” are ”Herbert” and ”Alexander”.
The letters and their positions in ”Herbert” are ”
[(H, 1), (e, 2), (r, 3), (b, 4), (e, 5), (r, 6), (t, 7)]”.
The letter at position 4 in this sequence is ”b”.
The letters and their positions in ”Alexander” are ”
[(A, 1), ( l, 2), (e, 3), (x, 4), (a, 5), (n, 6), (d, 7), (e, 8), (r, 9)]”.
The letter at position 4 in this sequence is ”x”.
Concatenating ”b”, ”x” using a space leads to ”b x”.
So, ”Herbert Alexander” outputs ” b x”.
QS: [l2m] Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
A: ”Herbert Alexander” outputs ”b x”.
The letters and their positions in ”Simon” are ”[(S, 1), (i, 2) , (m, 3), (o, 4), (n, 5)]”.
The letter at position 4 in this sequence is ”o”.
Concatenating ”b x”, ” o” using a space leads to ”b x o”.
So, ”Herbert Alexander Simon” outputs ”b x o”.
QS: [ans ext]
So, ”Herbert Alexander Simon” outputs ”b x o”.
A: ”b x o” QS
[Table] [Table]
Least
[Table] [Table]
Q: Take the last letters of the words in ”Augusta Ada” and concatenate them using a space.
A: The words in ”Augusta Ada” are ”Augusta” and ”Ada”.
The letters and their positions in ” Augusta” are ”
[(A, 1), (u, 2), (g, 3), (u, 4), (s, 5), (t, 6), (a, 7)]”.
The last letter in this sequence is ”a”.
The letters and their positions in ”Ada” are ”
[(A, 1), (d, 2), (a, 3)]”.
The last letter in this sequence is ”a”.
Concatenating ”a”, ”a” using a space leads to ”a a”.
So, ”Augusta Ada” outputs ”a a”.
Q: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
A: ”Augusta Ada” outputs ”a a”.
The letters and their positions in ”King” are ”
[(K, 1), (i, 2), (n, 3), (g, 4)]”.
The last letter in this sequence is ”g”.
Concatenating ”a a”, ”g” using a space leads to ”a a g”.
So, ”Augusta Ada King” outputs ”a a g”.
Q: Take the letters at position 1 of the words in ”Alan Mathison” and concatenate them using a space.
A: The words in ”Alan Mathison” are ”Alan” and ”Mathison”.
The letters and their positions in ” Alan” are ”
[(A, 1), (l, 2), (a, 3), (n, 4)]”.
The letter at position 1 in this sequence is ”A”.
The letters and their positions in ”Mathison” are [(M, 1), (a, 2), (t, 3), (h, 4), (i, 5), (s, 6), (o, 7), (n, 8)]”.
The letter at position 1 in this sequence is ”M”.
Concatenating ”A”, ”M” using a space leads to ”A M”.
So, ”Alan Mathison” outputs ”A M”.
Q: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
A: ”Alan Mathison” outputs ”A M”.
The letters and their positions in ”Turing” are ”[(T, 1), (u, 2), (r, 3), (i, 4), (n, 5), (g, 6)]”.
The letter at position 1 in this sequence is ”T”.
Concatenating ”A M”, ”T” using a space leads to ”A M T”.
So, ”Alan Mathison Turing” outputs ”A M T”.
Q: Take the letters at position 4 of the words in ”Herbert Alexander” and concatenate them using a space.
A: The words in ”Herbert Alexander” are ”Herbert” and ”Alexander”.
The letters and their positions in ”Herbert” are ”
[(H, 1), (e, 2), (r, 3), (b, 4), (e, 5), (r, 6), (t, 7)]”.
The letter at position 4 in this sequence is ”b”.
The letters and their positions in ”Alexander” are ”
[(A, 1), ( l, 2), (e, 3), (x, 4), (a, 5), (n, 6), (d, 7), (e, 8), (r, 9)]”.
The letter at position 4 in this sequence is ”x”.
Concatenating ”b”, ”x” using a space leads to ”b x”.
So, ”Herbert Alexander” outputs ” b x”.
Q: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
A: ”Herbert Alexander” outputs ”b x”.
The letters and their positions in ”Simon” are ”[(S, 1), (i, 2) , (m, 3), (o, 4), (n, 5)]”.
The letter at position 4 in this sequence is ”o”.
Concatenating ”b x”, ” o” using a space leads to ”b x o”.
So, ”Herbert Alexander Simon” outputs ”b x o”.
G
[Table] [Table]
LT D ECOM P SCHEMA
[Table]
decomp
QC: Take the last letters of the words in ”Augusta Ada King” and concatenate them using a space.
QS: [split] What are the words in ”Augusta Ada King”?
A: [”Augusta”, ”Ada”, ”King”] QS: [str position]
What is the last letter in ”Augusta”?
A: ”a” QS: [str position]
What is the last letter in ”Ada”?
A: ”a” QS: [str position]
What is the last letter in ”King”?
A: ”g” QS: [merge] Concatenate [”a”, ”a”, ”g”] using a space.
A: ”a a g” QS: [EOQ] QC: Take the letters at position 1 of the words in ”Alan Mathison Turing” and concatenate them using a space.
QS: [split] What are the words in ”Alan Mathison Turing”?
A: [”Alan”, ”Mathison”, ”Turing”]
QS: [str position]
What is the letter at position 1 in ”Alan”?
A: ”A” QS: [str position]
What is the letter at position 1 in ”Mathison”?
A: ”M” QS: [str position]
What is the letter at position 1 in ”Turing”?
A: ”T”
QC: Take the letters at position 4 of the words in ”Herbert Alexander Simon” and concatenate them using a space.
QS: [split] What are the words in ”Herbert Alexander Simon”?
A: [”Herbert”, ”Alexander”, ”Simon”
QS
[str position]
What is the letter at position 4 in ”Herbert”?
QS
[str position]
What is the letter at position 4 in ”Alexander”?
”x”
[str
] What is the letter at position 4 in ”Simon”?
a
G. S EQUENCE R EVERSAL G.2. S PLIT R EVERSEAL
The prompts in this section implement Algorithm 1.
procedure S PLIT R EVERSE ( x )
▷
The reversal of x if | x | < 4 then ▷ The base case return x | x | , . . .
, x▷
The reversed sequence else
▷
The inductive case n ← | x | /▷
Half the length of x ℓ ← x 1 , . . .
, x n ▷ The first half of x ℓ R ← S PLIT R EVERSE ( ℓ )
▷
The reversed first half r ← x n +1 , . . .
, x | x | ▷ The second half of x r R ← S PLIT R EVERSE ( r ) ▷ The reversed second half return r R 1 , . . .
, r Rn , ℓ Rn +1 , . . . , ℓ R | x | ▷ The concatenated reversed halves end if end procedure reverse
QC: Reverse the sequence ”driving license, button, packet, identity card, shoe”.
QS: [extract]
The sequence is ”1.
driving license, 2. button, 3. packet, 4.
identity card, 5. shoe”.
The sequence is 5 items long, which is more than the minimum length of 4, so we split it.
Half of 5 is 5 / 2 = 2.5.
Dropping the decimal, we get that the first half will be 2 items long, ending in ”2. button”.
The first half (2 items) is ”1.
driving license, 2. button”.
A: ”1.
driving license, 2. button” QS: [extract] The first half of the sequence ends with ”2. button”, so the second half starts after ”2. button” with ”3. packet”.
The full sequence is 5 items long, and the first half is 2 items long, so the second half will be 5 − 2 = 3 items long.
The second half of the sequence (3 items) is ”3.
packet, 4.
identity card, 5. shoe”.
A: ”3.
packet, 4.
identity card, 5. shoe” QS: [remove numbers]
Remove the numbers from #1.
A: ”driving license, button” QS: [remove numbers] Remove the numbers from #2.
A: ”packet, identity card, shoe” QS: [reverse] Reverse the sequence #3.
A: ”button, driving license”
QS: [reverse] Reverse the sequence #4.
A: ”shoe, identity card, packet” QS: [join] #6 # A: ”shoe, identity card, packet, button, driving license” QS: [EOQ] QC: Reverse the sequence ”laptop, photo, clip”.
QS: [extract]
The sequence is ”1. laptop, 2.
photo, 3. clip”.
The sequence is 3 items long, which is less than the minimum length of 4, so we don’t need to split it.
All we need to do is reverse ” laptop, photo, clip”.
A: ”laptop, photo, clip” QS: [cot] Reverse the sequence #1.
A: ”clip, photo, laptop” QS: [EOQ] QC: Reverse the sequence ”newspaper, glasses, laptop, bottle”.
QS: [extract]
The sequence is ”1. newspaper, 2.
glasses, 3. laptop, 4. bottle”.
The sequence isitems long, which is equal to the minimum length of 4, so we split it.
Half of 4 is 4 / 2 = 2.0.
Dropping the decimal, we get that the first half will be 2 items long.
The first half (2 items) of the sequence is ”1. newspaper, 2. glasses”.
A: ”1. newspaper, 2.
glasses” QS: [extract] The first half of the sequence ends with ”2. glasses”, so the second half starts after ”2. glasses” with ”3. laptop”.
The full sequence is 4 items long and the first half is 2 items long, so the second half will be 4 − 2 = 2 items long, ending in ”2. glasses”.
The second half of the sequence (2 items) is ”3. laptop, 4.
bottle”.
A: ”3. laptop, 4.
bottle” QS: [remove numbers]
Remove the numbers from #1.
A: ”newspaper, glasses” QS: [remove numbers] Remove the numbers from #2.
A: ”laptop, bottle” QS: [reverse] Reverse the sequence #3.
A: ”glasses, newspaper” QS: [reverse] Reverse the sequence #4.
A: ”bottle, laptop” QS: [join] #6 # A: ”bottle, laptop, glasses, newspaper” QS: [EOQ]
remove numbers
Q: Remove the numbers from ”4.
bottle, 3. laptop, 2.
glasses, 1. newspaper”.
A: ”bottle, laptop, glasses, newspaper” Q: Remove the numbers from ”1.
identity card, 2.
packet, 3. button”.
A: ”identity card, packet, button” Q: Remove the numbers from ”1.
player, 2.
passport, 3. umbrella, 4. radio”.
A: ”player, passport, umbrella, radio”
join
Q: ”bottle, laptop” ”glasses, newspaper”.
A: ”bottle, laptop, glasses, newspaper” Q: ”identity card, packet, button” ”magazine, notebook, glasses”.
A: ”identity card, packet, button, magazine, notebook, glasses” Q: ”passport, umbrella, radio, mobile phone, photo” ”player”.
A: ”passport, umbrella, radio, mobile phone, photo, player”
Q: ”mirror, case” ”toothbrush, alarm clock”.
A: ”mirror, case, toothbrush, alarm clock”
Q: ”light bulb, clip, umbrella” ”driving licence
, watch”
A: ”light bulb, clip,
umbrella, driving
licence
, watch”
[Table]
QC: Reverse the sequence ”newspaper, glasses, laptop, bottle”.
QS: First is newspaper.
Second is glasses.
Third is laptop.
Fourth is bottle.
Now to reverse, change the order to: Fourth is bottle.
Third is laptop.
Second is glasses.
First is newspaper.
So the answer is ”bottle, laptop, glasses, newspaper”.
A: ”bottle, laptop, glasses, newspaper” QC: Reverse the sequence ”laptop, photo, clip”.
QS: First is laptop.
Second is photo.
Third is clip.
Now to reverse, change the order to: Third is clip .
Second is photo.
First is laptop.
So the answer is ”clip, photo, laptop”.
A: ”clip, photo, laptop” QC: Reverse the sequence ”driving license, button, packet, identity card, pineapple”.
QS:
First is driving license.
Second is button.
Third is packet.
Fourth is identity card.
Fifth is pineapple.
Now to reverse, change the order to: Fifth is pineapple.
Fourth is identity card.
Third is packet.
Second is button.
First is driving license.
So the answer is ”pineapple, identity card, packet, button, driving license”.
A: ”pineapple, identity card, packet, button, driving license”
unrolled decomp
QC
: Reverse the sequence ”driving license, button, packet, identity card, shoe”.
QS:
The sequence is ”1.
driving license, 2. button, 3. packet, 4.
identity card, 5. shoe”.
The sequence is 5 items long, which is more than the minimum length of 4, so we split it.
Half of 5 is 5 / 2 = 2.5.
Dropping the decimal, we get that the first half will be 2 items long, ending in ”2. button”.
The first half (2 items) is ”1.
driving license, 2. button”.
The first half of the sequence ends with ”2. button”, so the second half starts after ”2. button” with ”3. packet”.
The full sequence is 5 items long, and the first half is 2 items long, so the second half will be 5 − 2 = 3 items long.
The second half of the sequence (3 items) is ”3.
packet, 4.
identity card, 5. shoe”.
Removing the numbers from ”1.
driving license, 2. button”, we get ”driving license, button”.
Removing the numbers from ”3.
packet, 4.
identity card, 5. shoe”, we get ”packet, identity card, shoe”.
Reversing the sequence ”driving license, button”, First is driving license.
Second is button.
Now to reverse, change the order to: Second is button.
First is driving license.
So the answer is ”button, driving license”.
Reversing the sequence ”packet, identity card, shoe”, First is packet.
Second is identity card.
Third is shoe.
Now to reverse, change the order to: Third is shoe.
Second is identity card.
First is packet.
So the answer is ”shoe, identity card, packet”.
Joining ”shoe, identity card, packet” and ”button, driving license”, the answer is ”shoe, identity card, packet, button, driving license”.
A: ”shoe, identity card, packet, button, driving license” QS: [EOQ] QC: Reverse the sequence ”laptop, photo, clip”.
QS:
The sequence is ”1. laptop, 2.
photo, 3. clip”.
The sequence is 3 items long, which is less than the minimum length of 4, so we don’t need to split it.
All we need to do is reverse ”laptop, photo, clip”.
First is laptop.
Second is photo.
Third is clip.
Now to reverse, change the order to : Third is clip.
Second is photo.
First is laptop.
So the answer is ”clip, photo, laptop”.
A: ”clip, photo, laptop” QS: [EOQ] QC: Reverse the sequence ”newspaper, glasses, laptop, bottle”.
QS:
The sequence is ”1. newspaper, 2.
glasses, 3. laptop, 4. bottle”.
The sequence is 4 items long, which is equal to the
minimum length of 4, so we split it.
Half of 4 is 4 / 2 = 2.0.
Dropping the decimal, we get that the first half will be 2 items long, ending in ”2.
glasses”.
The first half
(2 items) is ”1. newspaper, 2. glasses”.
The first half of the sequence ends with ”2. glasses”, so the second half starts after ”2. glasses” with ”3. laptop”.
The full sequence is 4 items long, and the first half is 2 items long, so the second half will be 4 − 2 = 2 items long.
The second half of the sequence (2 items) is ”3. laptop, 4. bottle”.
Removing the numbers from ”1. newspaper, 2. glasses”, we get ”newspaper, glasses”.
Removing the numbers from ”3. laptop, 4. bottle”, we get ”laptop, bottle”.
Reversing the sequence ”newspaper, glasses”, First is newspaper.
Second is glasses.
Now to reverse, change the order to: Second is glasses.
First is newspaper.
So the answer is ”glasses, newspaper”.
Reversing the sequence ”laptop, bottle”,
First is laptop.
Second is bottle.
Now to reverse, change the order to: Second is bottle.
First is laptop.
So the answer is ”bottle, laptop”.
Joining ”bottle, laptop” and ”glasses, newspaper”, the answer is ”bottle, laptop, glasses, newspaper”.
A: ”bottle, laptop,
glasses
, newspaper”
QS: [EOQ] reverse (tail)
QC: Reverse the sequence ”driving license, button, packet, identity card, shoe”.
Q1:
[extract] The sequence is ”1.
driving license, 2. button, 3. packet, 4.
identity card, 5. shoe”.
The sequence is 5 items long, which is more than the minimum length of 4, so we split it.
We take the first element in the sequence which is ”1.
driving license”.
driving license” Q2:
extract
] The full sequence is 5 items long, so the remaining sequence will be
−
=
4 items long.
The
tail of
the sequence
with
4 items
is
button, 3. packet, 4.
identity card, 5. shoe”
2: ”2. button, 3.
packet, 4.
identity card,
. shoe”
Q
remove
numbers]
Remove
the numbers from #1.
#3: ”driving license” Q4: [remove numbers] Remove the numbers from #2.
#4: ”button, packet, identity card, shoe” Q5: [reverse] Reverse the sequence #4.
#5: ”shoe, identity card, packet, button” Q6: [join] #5 # #6: ”shoe, identity card, packet, button, driving license” Q7:
[EOQ] QC: Reverse the sequence ”laptop, photo, clip”.
Q1:
[extract] The sequence is ”1. laptop, 2.
photo, 3. clip”.
The sequence is 3 items long, which is less than the minimum length of 4, so we don’t need to split it.
All we need to do is reverse ” laptop, photo, clip”.
#1: ”laptop, photo, clip” Q2:
[cot] Reverse the sequence #1.
#2: ”clip, photo, laptop” Q3:
[EOQ] QC: Reverse the sequence ”newspaper, glasses, laptop, bottle”.
Q1:
[extract] The sequence is ”1. newspaper, 2.
glasses, 3. laptop, 4. bottle”.
The sequence isitems long, which is more than the minimum length of 4, so we split it.
We take the first element in the sequence which is ”1. newspaper”.
#1: ”1. newspaper” Q2:
[extract] The full sequence is 4 items long, so the remaining sequence will be 4 − 1 = 3 items long.
The tail of the sequence with 3 items is ”2.
glasses, 3. laptop, 4. bottle”.
glasses, 3. laptop, 4.
bottle” Q3:
[remove numbers] Remove the numbers from #1.
#3: ”newspaper” Q4:
[remove numbers] Remove the numbers from #2.
#4: ”glasses, laptop, bottle” Q5: [reverse] Reverse the sequence #4.
#5: ”bottle, laptop, glasses” Q6: [join] #5 # #6: ”bottle, laptop, glasses, newspaper” Q7: [EOQ]
G. L ONG -D OCUMENT QA
We show one of the prompts used for CommaQA experiments here.
The entire set of prompts is provided as supplemetary material.
G.3. D ECOMPOSED P ROMPTING ( COARSE ) decomp
What awards
have
movies produced by people born in
won?
QS: (select) [qa] Who were born in the year 1910?
A: [”Teeplemole”, ”Muntaril”]
QS: (project values flat unique) [qa] For which movies was #1 the producer?
A: [”Featsaw”, ”Zalate”, ”Premercy”]
QS: (project values flat unique) [qa] Which awards were given to #2?
A: [”Zorgion”, ”Chowwurst”, ”Hallowcock”] QS: [EOQ] QC:
What movies have people from the country Stridery acted in?
QS: (select) [qa] Who is from the country Stridery?
A: [”Gastrat”] QS: (project values flat unique) [qa] Which movies has #1 been an actor in?
A: [”Partnershipmaker”, ”Nilitude”, ”Warpstone”] QS: [EOQ] QC:
What awards have the actors of the Erowid winning movies received?
QS: (select) [qa] Which movies were given the Erowid award?
A: [”Dewbar”, ”Caudacite”] QS: (project values flat unique) [qa] Who are the actors in the movie #1?
A: [”Wetherality”, ”Lougeri ` ere”, ”Gigabut”]
QS: (project values flat unique)
[qa] Which awards were given to #2?
A: [”Aniconder”, ”Trifogation”] QS: [EOQ] QC:
What awards did the movies directed by the Modiparity winners receive?
QS: (select) [qa] Who has been awarded the Modiparity award?
A: [”Bioperatology”] QS: (project values flat unique) [qa] Which movies has #1 directed?
A: [”Pestok”, ”Vitrilateral”]
QS: (project values flat unique)
[qa] Which awards were given to #2?
A: [”Gutney”, ”Antipositive”] QS: [EOQ] QC:
What awards have movies written by people born in 1935 won?
QS: (select) [qa] Who were born in the year 1935?
A: [”Sclerocybin”, ”Zayage”] QS: (project values flat unique) [qa] What movies has #1 written?
A: [”Noenometer”, ”Tayenne”, ”Pneumodendron”] QS: (project values flat unique) [qa] Which awards were given to #2?
A: [”Brownbeard”, ”Goosehead”, ”Handt”] QS: [EOQ] QC:
What movies have the directors from Legault directed?
QS: (select) [qa] Who is from the country Legault?
A: [”Metatoun”, ”Sapien”] QS: (project values flat unique) [qa] What movies has #1 been the director of?
A: [”Coacheship”, ”Misapportionment”] QS: [EOQ]
[Table]
movie: Premercy ; directed by: Muntaril.
movie: Skirtsicine ; director: Teeplemole.
movie:
Featsaw ; directed by: Monsterscar.
movie: Zalate ; director: Monsterscar.
movie: Zalate ; awarded:
Hallowcock.
movie:
Featsaw ; awarded: Zorgion.
movie: Premercy ; award: Chowwurst.
movie: Skirtsicine ; award: Hallowcock.
award: Goatfly ; winner: Teeplemole.
person:
Monsterscar ; award: Glodome.
person: Muntaril ; award: Goatfly.
movie:
Featsaw ; release year: 1973.
movie: Zalate ; release year: 1964.
movie: Skirtsicine ; release year: 1973.
movie:
Premercy ; year: 1961.
Teeplemole was an actor in the movie Skirtsicine.
Muntaril was an actor in the movie Skirtsicine.
Monsterscar was an actor in the movie Premercy.
Muntaril was an actor in the movie Featsaw.
Teeplemole was an actor in the movie Zalate.
Muntaril was born in the year 1910.
Teeplemole was born in 1910.
Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish.
Monsterscar is from the country of Piperfish.
Muntaril is from the country of Clony.
Muntaril produced the movie Skirtsicine with others.
Monsterscar was one of the producers of the movie Featsaw.
Monsterscar produced the movie
Premercy with others.
Monsterscar produced the movie Zalate with others.
Teeplemole was one of the producers of the movie Featsaw.
Teeplemole produced the movie Zalate with others.
Muntaril produced the movie Premercy with others.
Monsterscar wrote for the movie
Premercy.
Muntaril was one of the writers for the movie Zalate.
Muntaril wrote for the movie
Featsaw.
Teeplemole wrote for the movie Featsaw.
Monsterscar was one of the writers for the movie Zalate.
Teeplemole was one of the writers for the movie Skirtsicine.
Q: For which movies was Teeplemole the producer?
A: [”Featsaw”, ”Zalate”]
Q:
[Table]
awards
[Table]
to Featsaw?
A: [”Zorgion”] movie: Misgendery ; directed by:
Wetherality.
movie: Dewbar ; director: Gigabut.
movie:
Caudacite ; director: Lougeri ` ere.
movie: Tayenne ; directed by: Lougeri ` ere.
movie:
Misgendery ; awarded: Microsouenesis.
movie: Dewbar ; awarded: Erowid.
movie: Tayenne ; awarded: Cockspit.
movie: Caudacite ; award: Erowid.
award: Aniconder ; winner:
Wetherality. award: Aniconder ; winner: Lougeri`ere.
person: Gigabut ; award: Trifogation.
movie: Dewbar ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Caudacite ; release year: 2008.
movie:
Misgendery ; year: 1991.
Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne.
Lougeri ` ere was an actor in the movie Tayenne.
Lougeri ` ere acted in the movie Caudacite.
Lougeri ` ere acted in the movie Misgendery.
Gigabut was an actor in the movie Caudacite.
Wetherality was an actor in the movie Misgendery.
Wetherality was born in the year 1917.
Lougeri ` ere was born in 1926.
Gigabut was born in the year 1917.
Gigabut grew up in the nation of Triclops.
Lougeri ` ere is from the country of Tatkin .
Wetherality grew up in the nation of Tatkin.
Lougeri ` ere produced the movie Dewbar with others.
Gigabut produced the movie Tayenne with others.
Gigabut produced the movie
Dewbar with others.
Lougeri`ere was one of the producers of the movie Misgendery.
Wetherality was one of the producers of the movie Caudacite.
Gigabut was one of the producers of the movie Caudacite.
Wetherality produced the movie Misgendery with others.
Wetherality produced the movie Tayenne with others.
Wetherality wrote
for the movie Tayenne.
Gigabut wrote for the movie Misgendery.
Lougeri
`
ere was one of the writers for the movie Caudacite
Wetherality wrote
for
the movie Misgendery
Gigabut
wrote
for
the movie Tayenne
Gigabut
wrote
for
the movie Dewbar
Lougeri
`ere
wrote
for the
movie
Dewbar
Wetherality wrote for the movie
Caudacite
Q: Who are the actors in the movie Caudacite?
A: [”Lougeri`ere, ”Gigabut”]
Q: Which movies were given the Erowid award?
A: [”Dewbar”, ”Caudacite”] movie: Pastillobox ; directed by:
Firmline.
movie: Clenestration ; directed by: Carblock.
movie: Pestok ; directed by:
Bioperatology. movie: Vitrilateral ; director: Bioperatology.
movie: Vitrilateral ; award: Antipositive.
movie: Clenestration ; awarded: Handt.
movie: Pastillobox ; awarded: Handt.
movie: Pestok ; awarded: Gutney.
movie:
Pestok ; writer: Firmline.
movie:
Clenestration ; written by: Carblock.
movie: Pastillobox ; written by: Bioperatology.
movie:
Pestok ; writer:
Bioperatology. movie: Clenestration ; written by:
Firmline.
movie: Vitrilateral ; writer: Bioperatology.
movie: Pastillobox ; writer: Carblock.
movie: Vitrilateral ; written by : Carblock.
movie: Pestok ; release year: 1986.
movie: Clenestration ; year: 1986.
movie:
Vitrilateral ; year: 1999.
movie: Pastillobox ; release year: 1984.
Carblock was an actor in the movie Pastillobox.
Firmline was an actor in the movie Vitrilateral.
Bioperatology was an actor in the movie Clenestration.
Firmline acted in the movie Pastillobox.
Carblock was an actor in the movie Clenestration.
Bioperatology was an actor in the movie Pestok.
Firmline was born in the year 1904.
Bioperatology was born in the year 1935.
Carblock was born in 1935.
Carblock grew up in the nation of Knoppock.
Firmline grew up in the nation of Tatkin.
Bioperatology grew up in the nation of Tatkin.
Bioperatology won the Modiparity award.
Halfbill was awarded to Firmline.
Halfbill was awarded to Carblock.
Bioperatology was one of the producers of the movie Pestok.
Bioperatology produced the movie Vitrilateral with others.
Firmline produced the movie Pastillobox with others.
Firmline produced the movie
Clenestration with others.
Carblock was one of the producers of the movie Pastillobox.
Carblock produced the movie Vitrilateral with others.
Carblock produced the movie
Clenestration with others.
Firmline was one of the producers of the movie Pestok.
Q: Who has been awarded the Modiparity award?
A: [”Bioperatology”]
Q: Which movies has Bioperatology directed?
A: [”Pestok”, ”Vitrilateral”] movie: Nohit ; director: Mimicocycle.
movie: Noenometer ; director: Mimicocycle.
movie:
Tayenne ; directed by: Zayage.
movie: Pneumodendron ; director: Sclerocybin.
movie:
Tayenne ; awarded: Goosehead.
movie: Nohit ; awarded: Handt.
movie: Pneumodendron ; award: Handt.
movie: Noenometer ; awarded: Brownbeard.
movie: Nohit ; writer:
Mimicocycle.
movie: Noenometer ; written by: Sclerocybin.
movie: Tayenne ; writer:
Sclerocybin.
movie: Pneumodendron ; written by: Zayage.
movie: Tayenne ; writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle.
movie: Noenometer ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Nohit ; year: 2005.
movie: Pneumodendron ; year: 2005.
Mimicocycle was an actor in the movie Tayenne.
Zayage acted in the movie Pneumodendron.
Zayage was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne.
Mimicocycle was an actor in the movie
Noenometer.
Zayage was born in 1935.
Sclerocybin was born in 1935.
Mimicocycle was born in 1930.
Mimicocycle is from the country of Calderita.
Sclerocybin grew up in the nation of
Calderita.
Zayage is from the country of Obility.
Quinion was awarded to Zayage.
Fannyxist was awarded to Sclerocybin.
Fannyxist was awarded to Mimicocycle.
Mimicocycle produced the movie Nohit with others.
Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne.
Sclerocybin produced the movie
Pneumodendron with others.
Zayage produced the movie Pneumodendron with others.
Mimicocycle was one of the producers of the movie Tayenne.
Sclerocybin was one of the producers of the movie Noenometer.
Zayage produced the movie Noenometer with others
Q: What movies has Sclerocybin written?
A: [”Noenometer”, ”Tayenne”]
Q: Who were born in the year 1935?
A: [”Sclerocybin”, ”Zayage”] G.3.
D ECOMPOSED P ROMPTING ( FINE ) decomp
What awards have movies produced by people born in 1910 won?
QS: (select) [simp qa] Who were born in the year 1910?
A: [”Teeplemole”, ”Muntaril”]
QS: (project values flat unique) [pos qa] For which movies was #1 the producer?
A: [”Featsaw”, ”Zalate”, ”Premercy”]
QS: (project values flat unique)
[aw qa] Which awards were given to #2?
A: [”Zorgion”, ”Chowwurst”, ”Hallowcock”]
QS: [EOQ] QC:
What movies have people from the country Stridery acted in?
QS: (select) [simp qa]
Who is from the country Stridery?
A: [”Gastrat”] QS: (project values flat unique) [pos qa] Which movies has #1 been an actor in?
A: [”Partnershipmaker”, ”Nilitude”, ”Warpstone”] QS: [EOQ] QC:
What awards have the actors of the Erowid winning movies received?
QS: (select)
[aw qa] Which movies were given the Erowid award?
A: [”Dewbar”, ”Caudacite”] QS: (project values flat unique) [pos qa]
Who are the actors in the movie #1?
A: [”Wetherality”, ”Lougeri`ere”, ”Gigabut”]
QS: (project values flat unique)
[aw qa] Which awards were given to #2?
A: [”Aniconder”, ”Trifogation”] QS: [EOQ] QC:
What awards did the movies directed by the Modiparity winners receive?
QS: (select) [aw qa] Who has been awarded the Modiparity award?
A: [”Bioperatology”] QS: (project values flat unique) [pos qa] Which movies has #1 directed?
A: [”Pestok”, ”Vitrilateral”]
QS: (project values flat unique)
[aw qa] Which awards were given to #2?
A: [”Gutney”, ”Antipositive”] QS: [EOQ] QC:
What awards have movies written by people born in 1935 won?
QS: (select) [simp qa] Who were born in the year 1935?
A: [”Sclerocybin”, ”Zayage”] QS: (project values flat unique) [pos qa] What movies has #1 written?
A: [”Noenometer”, ”Tayenne”, ”Pneumodendron”] QS: (project values flat unique)
[aw qa] Which awards were given to #2?
A: [”Brownbeard”, ”Goosehead”, ”Handt”] QS: [EOQ] QC:
What movies have the directors from Legault directed?
QS: (select) [simp qa]
Who is from the country Legault?
A: [”Metatoun”, ”Sapien”] QS: (project values flat unique) [pos qa] What movies has #1 been the director of?
A: [”Coacheship”, ”Misapportionment”] QS: [EOQ] aw qa
movie: Premercy ; directed by: Muntaril movie: Skirtsicine ; director: Teeplemole.
movie:
Featsaw ; directed by: Monsterscar movie: Zalate ; director: Monsterscar.
movie: Zalate ; awarded:
Hallowcock movie: Featsaw ; awarded: Zorgion.
movie: Premercy ; award: Chowwurst.
movie: Skirtsicine ; award: Hallowcock.
award: Goatfly ; winner: Teeplemole.
person:
Monsterscar ; award: Glodome.
person: Muntaril ; award: Goatfly.
movie:
Featsaw ; release year: 1973.
movie: Zalate ; release year: 1964.
movie: Skirtsicine ; release year: 1973.
movie:
Premercy ; year: 1961.
Teeplemole was an actor in the movie Skirtsicine.
Muntaril was an actor in the movie Skirtsicine.
Monsterscar was an actor in the movie Premercy.
Muntaril was an actor in the movie Featsaw.
Teeplemole was an actor in the movie Zalate.
Muntaril was born in the year 1910.
Teeplemole was born in 1910.
Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish.
Monsterscar is from the country of Piperfish.
Muntaril is from the country of Clony.
Muntaril produced the movie Skirtsicine with others.
Monsterscar was one of the producers of the movie Featsaw.
Monsterscar produced the movie
Premercy with others.
Monsterscar produced the movie Zalate with others.
Teeplemole was one of the producers of the movie Featsaw.
Teeplemole produced the movie Zalate with others.
Muntaril produced the movie Premercy with others.
Monsterscar wrote for the movie
Premercy.
Muntaril was one of the writers for the movie Zalate.
Muntaril wrote for the movie
Featsaw.
Teeplemole wrote for the movie Featsaw.
Monsterscar was one of the writers for the movie Zalate.
Teeplemole was one of the writers for the movie Skirtsicine.
Q: Which awards were given to Zalate?
A: [”Hallowcock”]
Q: Which awards were given to Featsaw?
A: [”Zorgion”] movie: Misgendery ; directed by:
Wetherality.
movie: Dewbar ; director: Gigabut.
movie:
Caudacite ; director: Lougeri`ere.
movie: Tayenne ; directed by: Lougeri`ere.
movie:
Misgendery ; awarded: Microsouenesis.
movie: Dewbar ; awarded: Erowid.
movie: Tayenne ; awarded: Cockspit.
movie: Caudacite ; award: Erowid.
award: Aniconder ; winner:
Wetherality. award: Aniconder ; winner: Lougeri`ere.
person: Gigabut ; award: Trifogation.
movie: Dewbar ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Caudacite ; release year: 2008.
movie:
Misgendery ; year: 1991.
Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne.
Lougeri`ere was an actor in the movie Tayenne.
Lougeri ` ere acted in the movie Caudacite.
Lougeri ` ere acted in the movie Misgendery.
Gigabut was an actor in the movie Caudacite.
Wetherality was an actor in the movie Misgendery.
Wetherality was born in the year 1917.
Lougeri`ere was born in 1926.
Gigabut was born in the year 1917.
Gigabut grew up in the nation of Triclops.
Lougeri`ere is from the country of Tatkin .
Wetherality grew up in the nation of Tatkin.
Lougeri ` ere produced the movie Dewbar with others.
Gigabut produced the movie Tayenne with others.
Gigabut produced the movie
Dewbar with others.
Lougeri ` ere was one of the producers of the movie Misgendery.
Wetherality was one of the producers of the movie Caudacite.
Gigabut was one of the producers of the movie Caudacite.
Wetherality produced the movie Misgendery with others.
Wetherality produced the movie Tayenne with others.
Wetherality wrote for the movie
Tayenne.
Gigabut wrote for the movie Misgendery.
Lougeri ` ere was one of the writers for the movie Caudacite.
Wetherality wrote for the movie Misgendery.
Gigabut wrote for the movie
Tayenne.
Gigabut wrote for the movie Dewbar.
Lougeri ` ere wrote for the movie Dewbar.
Wetherality wrote for the movie Caudacite.
Q: Which movies were given the Erowid award?
A: [”Dewbar”, ”Caudacite”] Q: Which awards were given to Wetherality?
A: [”Aniconder”] movie: Pastillobox ; directed by:
Firmline.
movie: Clenestration ; directed by: Carblock.
movie: Pestok ; directed by:
Bioperatology. movie: Vitrilateral ; director: Bioperatology.
movie: Vitrilateral ; award: Antipositive.
movie: Clenestration ; awarded: Handt.
movie: Pastillobox ; awarded: Handt.
movie: Pestok ; awarded: Gutney.
movie:
Pestok ; writer: Firmline.
movie: Clenestration ; written by: Carblock.
movie: Pastillobox ; written by: Bioperatology.
movie:
Pestok ; writer:
Bioperatology. movie: Clenestration ; written by:
Firmline.
movie: Vitrilateral ; writer: Bioperatology.
movie: Pastillobox ; writer: Carblock.
movie: Vitrilateral ; written by : Carblock.
movie: Pestok ; release year: 1986.
movie: Clenestration ; year: 1986.
movie: Vitrilateral ; year: 1999.
movie: Pastillobox ; release year: 1984.
Carblock was an actor in the movie Pastillobox.
Firmline was an actor in the movie Vitrilateral.
Bioperatology was an actor in the movie Clenestration.
Firmline acted in the movie Pastillobox.
Carblock was an actor in the movie Clenestration.
Bioperatology was an actor in the movie Pestok.
Firmline was born in the year 1904.
Bioperatology was born in the year 1935.
Carblock was born in 1935.
Carblock grew up in the nation of Knoppock.
Firmline grew up in the nation of Tatkin.
Bioperatology grew up in the nation of Tatkin.
Bioperatology won the Modiparity award.
Halfbill was awarded to Firmline.
Halfbill was awarded to Carblock.
Bioperatology was one of the producers of the movie Pestok.
Bioperatology produced the movie Vitrilateral with others.
Firmline produced the movie Pastillobox with others.
Firmline produced the movie
Clenestration with others.
Carblock was one of the producers of the movie Pastillobox.
Carblock produced the movie Vitrilateral with others.
Carblock produced the movie
Clenestration with others.
Firmline was one of the producers of the movie Pestok.
Q: Who has been awarded the Modiparity award?
A: [”Bioperatology”]
Q: Which awards were given to Pestok?
A: [”Gutney”] movie: Nohit ; director: Mimicocycle.
movie: Noenometer ; director: Mimicocycle.
movie:
Tayenne ; directed by: Zayage.
movie: Pneumodendron ; director: Sclerocybin.
movie:
Tayenne ; awarded: Goosehead.
movie: Nohit ; awarded: Handt.
movie: Pneumodendron ; award: Handt.
movie: Noenometer ; awarded: Brownbeard.
movie: Nohit ; writer:
Mimicocycle.
movie: Noenometer ; written by: Sclerocybin.
movie: Tayenne ; writer:
Sclerocybin.
movie: Pneumodendron ; written by: Zayage.
movie: Tayenne ; writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle.
movie: Noenometer ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Nohit ; year: 2005.
movie: Pneumodendron ; year: 2005.
Mimicocycle was an actor in the movie Tayenne.
Zayage acted in the movie Pneumodendron.
Zayage was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne.
Mimicocycle was an actor in the movie
Noenometer.
Zayage was born in 1935.
Sclerocybin was born in 1935.
Mimicocycle was born in 1930.
Mimicocycle is from the country of Calderita.
Sclerocybin grew up in the nation of
Calderita.
Zayage is from the country of Obility.
Quinion was awarded to Zayage.
Fannyxist was awarded to Sclerocybin.
Fannyxist was awarded to Mimicocycle.
Mimicocycle produced the movie Nohit with others.
Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne.
Sclerocybin produced the movie
Pneumodendron with others.
Zayage produced the movie Pneumodendron with others.
Mimicocycle was one of the producers of the movie Tayenne.
Sclerocybin was one of the producers of the movie Noenometer.
Zayage produced the movie Noenometer with others
Q: Which awards were given to Noenometer?
A: [”Brownbeard”]
Q: Which awards were given to Pneumodendron?
A: [”Handt”]
[Table]
movie: Premercy ; directed by: Muntaril.
movie: Skirtsicine ; director: Teeplemole.
movie:
Featsaw ; directed by: Monsterscar.
movie: Zalate ; director: Monsterscar.
movie: Zalate ; awarded:
Hallowcock.
movie:
Featsaw ; awarded: Zorgion.
movie: Premercy ; award: Chowwurst.
movie: Skirtsicine ; award: Hallowcock.
award: Goatfly ; winner: Teeplemole.
person:
Monsterscar ; award: Glodome.
person: Muntaril ; award: Goatfly.
movie:
Featsaw ; release year: 1973.
movie: Zalate ; release year: 1964.
movie: Skirtsicine ; release year: 1973.
movie:
Premercy ; year: 1961.
Teeplemole was an actor in the movie Skirtsicine.
Muntaril was an actor in the movie Skirtsicine.
Monsterscar was an actor in the movie Premercy.
Muntaril was an actor in the movie Featsaw.
Teeplemole was an actor in the movie Zalate.
Muntaril was born in the year 1910.
Teeplemole was born in 1910.
Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish.
Monsterscar is from the country of Piperfish.
Muntaril is from the country of Clony.
Muntaril produced the movie Skirtsicine with others.
Monsterscar was one of the producers of the movie Featsaw.
Monsterscar produced the movie
Premercy with others.
Monsterscar produced the movie Zalate with others.
Teeplemole was one of the producers of the movie Featsaw
Q: For which movies was Teeplemole the producer?
A: [”Featsaw”, ”Zalate”]
Q: For which movies was Muntaril the producer?
A: [”Premercy]
movie: Misgendery ; directed by: Wetherality.
movie: Dewbar ; director: Gigabut.
movie:
Caudacite ; director: Lougeri`ere.
movie: Tayenne ; directed by: Lougeri`ere.
movie:
Misgendery ; awarded: Microsouenesis.
movie: Dewbar ; awarded: Erowid.
movie: Tayenne ; awarded: Cockspit.
movie: Caudacite ; award: Erowid.
award: Aniconder ; winner:
Wetherality. award: Aniconder ; winner: Lougeri`ere.
person: Gigabut ; award: Trifogation.
movie: Dewbar ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Caudacite ; release year: 2008.
movie:
Misgendery ; year: 1991.
Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne.
Lougeri`ere was an actor in the movie Tayenne.
Lougeri`ere acted in the movie Caudacite.
Lougeri`ere acted in the movie Misgendery.
Gigabut was an actor in the movie Caudacite.
Wetherality was an actor in the movie Misgendery.
Wetherality was born in the year 1917.
Lougeri`ere was born in 1926.
Gigabut was born in the year 1917.
Gigabut grew up in the nation of Triclops.
Lougeri`ere is from the country of Tatkin .
Wetherality grew up in the nation of Tatkin.
Lougeri`ere produced the movie Dewbar with others.
Gigabut produced the movie Tayenne with others.
Gigabut produced the movie
Dewbar with others.
Lougeri`ere was one of the producers of the movie Misgendery.
Wetherality was one of the producers of the movie Caudacite.
Gigabut was one of the producers of the movie Caudacite.
Wetherality produced the movie Misgendery with others.
Wetherality produced the movie Tayenne with others.
Wetherality wrote for the movie
Tayenne.
Gigabut wrote for the movie Misgendery.
Lougeri ` ere was one of the writers for the movie Caudacite.
Wetherality wrote for the movie Misgendery.
Gigabut wrote for the movie
Tayenne.
Gigabut wrote for the movie Dewbar.
Lougeri`ere wrote for the movie Dewbar.
Wetherality wrote for the movie Caudacite.
Q: Who are the actors in the movie Dewbar?
A: [”Wetherality”]
Q: Who are the actors in the movie Caudacite?
A: [”Lougeri ` ere, ”Gigabut”] movie: Pastillobox ; directed by:
Firmline.
movie: Clenestration ; directed by: Carblock.
movie:
Pestok ; directed by:
Bioperatology. movie: Vitrilateral ; director: Bioperatology.
movie:
Vitrilateral ; award: Antipositive.
movie: Clenestration ; awarded: Handt.
movie: Pastillobox ; awarded: Handt.
movie: Pestok ; awarded: Gutney.
movie:
Pestok ; writer: Firmline.
movie:
Clenestration ; written by: Carblock.
movie: Pastillobox ; written by: Bioperatology.
movie:
Pestok ; writer:
Bioperatology. movie: Clenestration ; written by:
Firmline.
movie: Vitrilateral ; writer: Bioperatology.
movie: Pastillobox ; writer: Carblock.
movie: Vitrilateral ; written by : Carblock.
movie: Pestok ; release year: 1986.
movie: Clenestration ; year: 1986.
movie:
Vitrilateral ; year: 1999.
movie: Pastillobox ; release year: 1984.
Carblock was an actor in the movie Pastillobox.
Firmline was an actor in the movie Vitrilateral.
Bioperatology was an actor in the movie Clenestration.
Firmline acted in the movie Pastillobox.
Carblock was an actor in the movie Clenestration.
Bioperatology was an actor in the movie Pestok.
Firmline was born in the year 1904.
Bioperatology was born in the year 1935.
Carblock was born in 1935.
Carblock grew up in the nation of Knoppock.
Firmline grew up in the nation of Tatkin.
Bioperatology grew up in the nation of Tatkin.
Bioperatology won the Modiparity award.
Halfbill was awarded to Firmline.
Halfbill was awarded to Carblock.
Bioperatology was one of the producers of the movie Pestok.
Bioperatology produced the movie Vitrilateral with others.
Firmline produced the movie Pastillobox with others.
Firmline produced the movie
Clenestration with others.
Carblock was one of the producers of the movie Pastillobox.
Carblock produced the movie Vitrilateral with others.
Carblock produced the movie
Clenestration with others.
Firmline was one of the producers of the movie Pestok.
Q: Which movies has Bioperatology directed?
A: [”Pestok”, ”Vitrilateral”]
Q: Which movies has Carblock directed?
A: [”Clenestration”] movie: Nohit ; director: Mimicocycle.
movie: Noenometer ; director: Mimicocycle.
movie: Tayenne ; directed by: Zayage.
movie: Pneumodendron ; director: Sclerocybin.
movie: Tayenne ; awarded: Goosehead.
movie: Nohit ; awarded: Handt.
movie: Pneumodendron ; award: Handt.
movie: Noenometer ; awarded: Brownbeard.
movie: Nohit ; writer:
Mimicocycle.
movie: Noenometer ; written by: Sclerocybin.
movie: Tayenne ; writer:
Sclerocybin.
movie: Pneumodendron ; written by: Zayage.
movie: Tayenne ; writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle.
movie: Noenometer ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Nohit ; year: 2005.
movie: Pneumodendron ; year: 2005.
Mimicocycle was an actor in the movie Tayenne.
Zayage acted in the movie Pneumodendron.
Zayage was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne.
Mimicocycle was an actor in the movie
Noenometer.
Zayage was born in 1935.
Sclerocybin was born in 1935.
Mimicocycle was born in 1930.
Mimicocycle is from the country of Calderita.
Sclerocybin grew up in the nation of
Calderita.
Zayage is from the country of Obility.
Quinion was awarded to Zayage.
Fannyxist was awarded to Sclerocybin.
Fannyxist was awarded to Mimicocycle.
Mimicocycle produced the movie Nohit with others.
Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne.
Sclerocybin produced the movie
Pneumodendron with others.
Zayage produced the movie Pneumodendron with others.
Mimicocycle was one of the producers of the movie Tayenne.
Sclerocybin was one of the producers of the movie Noenometer.
Zayage produced the movie Noenometer with others
Q: What movies has Sclerocybin written?
A: [”Noenometer”, ”Tayenne”]
Q: What movies has Zayage written?
A: [”Pneumodendron”, ”Tayenne”] simp qa movie: Premercy ; directed by: Muntaril.
movie: Skirtsicine ; director: Teeplemole.
movie:
Featsaw ; directed by: Monsterscar.
movie: Zalate ; director: Monsterscar.
movie: Zalate ; awarded:
Hallowcock.
movie:
Featsaw ; awarded: Zorgion.
movie: Premercy ; award: Chowwurst.
movie: Skirtsicine ; award: Hallowcock.
award: Goatfly ; winner: Teeplemole.
person:
Monsterscar ; award: Glodome.
person: Muntaril ; award: Goatfly.
movie:
Featsaw ; release year: 1973.
movie: Zalate ; release year: 1964.
movie: Skirtsicine ; release year: 1973.
movie:
Premercy ; year: 1961.
Teeplemole was an actor in the movie Skirtsicine.
Muntaril was an actor in the movie Skirtsicine.
Monsterscar was an actor in the movie Premercy.
Muntaril was an actor in the movie Featsaw.
Teeplemole was an actor in the movie Zalate.
Muntaril was born in the year 1910.
Teeplemole was born in 1910.
Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish.
Monsterscar is from the country of Piperfish.
Muntaril is from the country of Clony.
Muntaril produced the movie Skirtsicine with others.
Monsterscar was one of the producers of the movie Featsaw.
Monsterscar produced the movie
Premercy with others.
Monsterscar produced the movie Zalate with others.
Teeplemole was one of the producers of the movie Featsaw.
Teeplemole produced the movie Zalate with others.
Muntaril produced the movie Premercy with others.
Monsterscar wrote for the movie
Premercy.
Muntaril was one of the writers for the movie Zalate.
Muntaril wrote for the movie
Featsaw.
Teeplemole wrote for the movie Featsaw.
Monsterscar was one of the writers for the movie Zalate.
Teeplemole was one of the writers for the movie Skirtsicine.
Q: Who were
born in the year
A: [”Teeplemole”, ”Muntaril”]
Q: From which country is Monsterscar?
A
: [”Piperfish”] movie: Nilitude ; director: Monsterscar.
movie: Dewbar ; directed by: Metatoun.
movie: Warpstone ; directed by:
Gastrat.
movie: Partnershipmaker ; director: Metatoun.
movie: Dewbar ; award:
Tachychronograph.
movie: Partnershipmaker ; awarded: Tachychronograph.
movie: Nilitude ; award: Paleodactyl.
movie:
Warpstone ; award: Sabonade.
person: Gastrat ; award:
Trifogation. award: Polyquadrase ; winner: Monsterscar.
award: Trifogation ; winner:
Metatoun.
movie:
Warpstone ; release year: 1956.
movie: Dewbar ; release year: 1984.
movie:
Nilitude ; year: 1984.
movie: Partnershipmaker ; year: 1962.
Gastrat was an actor in the movie Partnershipmaker.
Metatoun was an actor in the movie Partnershipmaker.
Metatoun was an actor in the movie Nilitude.
Gastrat acted in the movie Nilitude.
Monsterscar was an actor in the movie Dewbar.
Gastrat acted in the movie Warpstone.
Metatoun acted in the movie Warpstone.
Metatoun was born in 1939.
Gastrat was born in the year 1933.
Monsterscar was born in 1933.
Metatoun grew up in the nation of Moulole.
Gastrat is from the country of Stridery.
Monsterscar grew up in the nation of Moulole.
Monsterscar produced the movie Nilitude with others.
Monsterscar was one of the producers of the movie Warpstone .
Metatoun was one of the producers of the movie Warpstone.
Gastrat was one of the producers of the movie Nilitude.
Metatoun produced the movie Partnershipmaker with others.
Metatoun produced the movie Dewbar with others.
Monsterscar was one of the producers of the movie Partnershipmaker.
Gastrat produced the movie Dewbar with others.
Metatoun wrote for the movie Partnershipmaker.
Gastrat wrote for the movie Warpstone.
Gastrat was one of the writers for the movie Dewbar.
Monsterscar was one of the writers for the movie
Nilitude.
Metatoun wrote for the movie Warpstone.
Q: Who is from the country Stridery?
A: [”Gastrat”
Q: Which movies were released in 1984?
A
: [”Dewbar”, ”Nilitude”
] movie: Nohit ; director: Mimicocycle.
movie: Noenometer ; director: Mimicocycle.
movie:
Tayenne ; directed by: Zayage.
movie: Pneumodendron ; director: Sclerocybin.
movie:
Tayenne ; awarded: Goosehead.
movie: Nohit ; awarded: Handt.
movie: Pneumodendron ; award: Handt.
movie: Noenometer ; awarded: Brownbeard.
movie: Nohit ; writer:
Mimicocycle.
movie: Noenometer ; written by: Sclerocybin.
movie: Tayenne ; writer:
Sclerocybin.
movie: Pneumodendron ; written by: Zayage.
movie: Tayenne ; writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle.
movie: Noenometer ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Nohit ; year: 2005.
movie: Pneumodendron ; year: 2005.
Mimicocycle was an actor in the movie Tayenne.
Zayage acted in the movie Pneumodendron.
Zayage was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne.
Mimicocycle was an actor in the movie
Noenometer.
Zayage was born in 1935.
Sclerocybin was born in 1935.
Mimicocycle was born in 1930.
Mimicocycle is from the country of Calderita.
Sclerocybin grew up in the nation of
Calderita.
Zayage is from the country of Obility.
Quinion was awarded to Zayage.
Fannyxist was awarded to Sclerocybin.
Fannyxist was awarded to Mimicocycle.
Mimicocycle produced the movie Nohit with others.
Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne.
Sclerocybin produced the movie
Pneumodendron with others.
Zayage produced the movie Pneumodendron with others.
Mimicocycle was one of the producers of the movie Tayenne.
Sclerocybin was one of the producers of the movie Noenometer.
Zayage produced the movie Noenometer with others
Q: Who were born in the year 1935?
A: [”Sclerocybin”, ”Zayage”]
Q: When was the movie Nohit released?
A: [”2005”] movie: Coacheship ; director: Metatoun.
movie: Assamplifier ; director: Kapod.
movie:
Misapportionment ; director: Sapien.
movie:
Quinsid ; director: Kapod.
movie:
Assamplifier ; award: Zorgion.
movie: Quinsid ; awarded: Airpipe.
movie: Coacheship ; award: Electrodesal.
movie: Misapportionment ; award: Airpipe.
movie: Coacheship ; written by: Metatoun.
movie: Misapportionment ; written by: Kapod.
movie: Coacheship ; written by: Kapod.
movie :
Quinsid ; writer: Sapien.
movie: Misapportionment ; written by: Metatoun.
movie:
Assamplifier ; written by: Kapod.
movie: Assamplifier ; written by: Sapien.
movie:
Assamplifier ; release year: 2000.
movie: Coacheship ; year: 2001.
movie: Quinsid ; year: 2005.
movie: Misapportionment ; year: 2005.
Sapien was an actor in the movie
Misapportionment.
Sapien acted in the movie Assamplifier.
Kapod acted in the movie Quinsid .
Sapien acted in the movie Coacheship.
Metatoun was an actor in the movie Quinsid.
Kapod acted in the movie Misapportionment.
Metatoun acted in the movie Coacheship.
Kapod acted in the movie Assamplifier.
Sapien was born in the year 1910.
Metatoun was born in 1928.
Kapod was born in the year 1910.
Metatoun is from the country of Legault.
Kapod grew up in the nation of Tatkin.
Sapien is from the country of Legault.
Malwarp was awarded to Sapien.
Metatoun won the Monkeynote award.
Kapod won the Monkeynote award.
Kapod was one of the producers of the movie Quinsid.
Metatoun was one of the producers of the movie
Misapportionment.
Metatoun was one of the producers of the movie Quinsid.
Sapien was one of the producers of the movie Assamplifier.
Sapien produced the movie Coacheship with others.
Metatoun was one of the producers of the movie Assamplifier.
Kapod was one of the producers of the movie Misapportionment.
Kapod was one of the producers of the movie
Coacheship.
[Table]
Legault
[Table]
[”Metatoun”, ”Sapien”
[Table]
movie: Premercy ; directed by: Muntaril.
movie: Skirtsicine ; director: Teeplemole.
movie:
Featsaw ; directed by: Monsterscar.
movie: Zalate ; director: Monsterscar.
movie: Zalate ; awarded:
Hallowcock.
movie:
Featsaw ; awarded: Zorgion.
movie: Premercy ; award: Chowwurst.
movie: Skirtsicine ; award: Hallowcock.
award: Goatfly ; winner: Teeplemole.
person:
Monsterscar ; award: Glodome.
person: Muntaril ; award: Goatfly.
movie:
Featsaw ; release year: 1973.
movie: Zalate ; release year: 1964.
movie: Skirtsicine ; release year: 1973.
movie:
Premercy ; year: 1961.
Teeplemole was an actor in the movie Skirtsicine.
Muntaril was an actor in the movie Skirtsicine.
Monsterscar was an actor in the movie Premercy.
Muntaril was an actor in the movie Featsaw.
Teeplemole was an actor in the movie Zalate.
Muntaril was born in the year 1910.
Teeplemole was born in 1910.
Monsterscar was born in 1942.
Teeplemole is from the country of Piperfish.
Monsterscar is from the country of Piperfish.
Muntaril is from the country of Clony.
Muntaril produced the movie Skirtsicine with others.
Monsterscar was one of the producers of the movie Featsaw.
Monsterscar produced the movie
Premercy with others.
Monsterscar produced the movie Zalate
with others.
Teeplemole was one of the producers of the movie Featsaw.
Teeplemole produced the movie Zalate with others.
Muntaril produced the movie Premercy with others.
Monsterscar wrote for the movie Premercy.
Muntaril was one of the writers for the movie Zalate.
Muntaril wrote for the movie Featsaw.
Teeplemole wrote for the movie Featsaw.
Monsterscar was one of the writers for the movie Zalate.
Teeplemole was one of the writers for the movie Skirtsicine.
QC:
What awards have movies produced by people born in 1910 won?
QS:
The people born in 1910 were Muntaril and Teeplemole.
Teeplemole produced the movies Featsaw and Zalate.
Muntaril produced the movie Premercy.
Featsaw was awarded the Zorgion award.
Premercy was awarded the Chowwurst award.
Zalate was awarded the Hallowcock award.
So the answer is [”Zorgion”, ”Chowwurst”, ”Hallowcock”].
A: [”Zorgion”, ”Chowwurst”, ”Hallowcock”] QS: [EOQ] movie: Misgendery ; directed by: Wetherality.
movie: Dewbar ; director: Gigabut.
movie: Caudacite ; director: Lougeri`ere.
movie: Tayenne ; directed by: Lougeri`ere.
movie: Misgendery ; awarded: Microsouenesis.
movie: Dewbar ; awarded: Erowid.
movie: Tayenne ; awarded: Cockspit.
movie: Caudacite ; award: Erowid.
award: Aniconder ; winner: Wetherality. award: Aniconder ; winner: Lougeri`ere.
person: Gigabut ; award: Trifogation.
movie: Dewbar ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Caudacite ; release year: 2008.
movie:
Misgendery ; year: 1991.
Wetherality was an actor in the movie Dewbar.
Gigabut was an actor in the movie Tayenne.
Lougeri`ere was an actor in the movie Tayenne.
Lougeri
`
ere
acted in the movie Caudacite.
Lougeri`ere acted in the movie Misgendery.
Gigabut was an actor in the movie Caudacite.
Wetherality was an actor in the movie Misgendery.
Wetherality was born in the year 1917.
Lougeri`ere was born in 1926.
Gigabut was born in the year 1917.
Gigabut grew up in the nation of Triclops.
Lougeri`ere is from the country of Tatkin .
Wetherality grew up in the nation of Tatkin.
Lougeri`ere produced the movie Dewbar with others.
Gigabut produced the movie Tayenne with others.
Gigabut produced the movie
Dewbar with others.
Lougeri`ere was one of the producers of the movie Misgendery.
Wetherality was one of the producers of the movie Caudacite.
Gigabut was one of the producers of the movie Caudacite.
Wetherality produced the movie Misgendery with others.
Wetherality produced the movie Tayenne with others.
Wetherality wrote for the movie
Tayenne.
Gigabut wrote for the movie Misgendery.
Lougeri`ere was one of the writers for the movie Caudacite.
Wetherality wrote for the movie Misgendery.
Gigabut wrote for the movie Tayenne.
Gigabut wrote for the movie Dewbar.
Lougeri`ere wrote for the movie Dewbar.
Wetherality wrote for the movie Caudacite.
QC:
What awards have the actors of the Erowid winning movies received?
QS:
The movies that won the Erowid award are Dewbar and Caudacite.
Wetherality was an actor in Dewbar.
Lougeri`ere and Gigabut acted in Caudacite.
Wetherality won the Aniconder award.
Lougeri`ere also won the Aniconder award.
Gigabut won the Trifogation award.
So the answer is [”Aniconder”, ”Trifogation”].
A: [”Aniconder”, ”Trifogation”] QS: [EOQ] movie: Pastillobox ; directed by: Firmline.
movie: Clenestration ; directed by: Carblock.
movie: Pestok ; directed by:
Bioperatology. movie: Vitrilateral ; director: Bioperatology.
movie: Vitrilateral ; award: Antipositive.
movie: Clenestration ; awarded: Handt.
movie: Pastillobox ; awarded: Handt.
movie: Pestok ; awarded: Gutney.
movie:
Pestok ; writer: Firmline.
movie: Clenestration ; written by: Carblock.
movie: Pastillobox ; written by: Bioperatology.
movie:
Pestok ; writer:
Bioperatology. movie: Clenestration ; written by:
Firmline.
movie: Vitrilateral ; writer: Bioperatology.
movie: Pastillobox ; writer: Carblock.
movie: Vitrilateral ; written by : Carblock.
movie: Pestok ; release year: 1986.
movie: Clenestration ; year: 1986.
movie: Vitrilateral ; year: 1999.
movie: Pastillobox ; release year: 1984.
Carblock was an actor in the movie Pastillobox.
Firmline was
an actor in the movie Vitrilateral.
Bioperatology was an actor in the movie Clenestration.
Firmline acted in the movie Pastillobox.
Carblock was an actor in the movie Clenestration.
Bioperatology was an actor in the movie Pestok.
Firmline was born in the year 1904.
Bioperatology was born in the year 1935.
Carblock was born in 1935.
Carblock grew up in the nation of Knoppock.
Firmline grew up in the nation of Tatkin.
Bioperatology grew up in the nation of Tatkin.
Bioperatology won the Modiparity award.
Halfbill was awarded to Firmline.
Halfbill was awarded to Carblock.
Bioperatology was one of the producers of the movie Pestok.
Bioperatology produced the movie Vitrilateral with others.
Firmline produced the movie Pastillobox with others.
Firmline produced the movie
Clenestration with others.
Carblock was one of the producers of the movie Pastillobox.
Carblock produced the movie Vitrilateral with others.
Carblock produced the movie
Clenestration with others.
Firmline was one of the producers of the movie Pestok.
QC
What awards did the movies directed by the Modiparity winners receive?
QS:
The Modiparity winners are
Bioperatology.
Bioperatology has directed the movies Pestok and
Vitrilateral.
Pestok has won the Gutney
award
Vitrilateral has won the Antipositive award.
So the
answer
is
[”Gutney”, ”Antipositive”
A: [”Gutney”, ”Antipositive”] QS: [EOQ
] movie: Nohit ; director: Mimicocycle.
movie: Noenometer ; director: Mimicocycle.
movie:
Tayenne ; directed by: Zayage.
movie: Pneumodendron ; director: Sclerocybin.
movie:
Tayenne ; awarded: Goosehead.
movie: Nohit ; awarded: Handt.
movie: Pneumodendron ; award: Handt.
movie: Noenometer ; awarded: Brownbeard.
movie: Nohit ; writer:
Mimicocycle.
movie: Noenometer ; written by: Sclerocybin.
movie: Tayenne ; writer:
Sclerocybin.
movie: Pneumodendron ; written by: Zayage.
movie: Tayenne ; writer: Zayage.
movie: Pneumodendron ; written by: Mimicocycle.
movie: Noenometer ; release year: 1991.
movie: Tayenne ; year: 2013.
movie: Nohit ; year: 2005.
movie: Pneumodendron ; year: 2005.
Mimicocycle was an actor in the movie Tayenne.
Zayage acted in the movie Pneumodendron.
Zayage was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Nohit.
Sclerocybin was an actor in the movie Tayenne.
Mimicocycle was an actor in the movie
Noenometer.
Zayage was born in 1935.
Sclerocybin was born in 1935.
Mimicocycle was born in 1930.
Mimicocycle is from the country of Calderita.
Sclerocybin grew up in the nation of
Calderita.
Zayage is from the country of Obility.
Quinion was awarded to Zayage.
Fannyxist was awarded to Sclerocybin.
Fannyxist was awarded to Mimicocycle.
Mimicocycle produced the movie Nohit with others.
Zayage was one of the producers of the movie Nohit.
Sclerocybin was one of the producers of the movie Tayenne.
Sclerocybin produced the movie
Pneumodendron with others.
Zayage produced the movie Pneumodendron with others.
Mimicocycle was one of the producers of the movie Tayenne.
Sclerocybin was one of the producers of the movie Noenometer.
Zayage produced the movie Noenometer with others
QC:
What awards have movies written by people born in 1935 won?
QS:
The people born in 1935 were Sclerocybin and Zayage.
Sclerocybin has written the movies Noenometer and Tayenne.
Zayage has written the movies Pneumodendron and Tayenne.
Noenometer has won the Brownbeard award.
Tayenne has won the Goosehead award.
Pneumodendron has won the Handt award.
So the answer is [”Brownbeard”, ”Goosehead”, ” Handt”
A
”Brownbeard”
, ”Goosehead”,
”Handt”]
[Equation] [Equation]
Q: There are 15 trees in the grove.
Grove workers will plant trees in the grove today.
After they are done, there will be 21 trees.
How many trees did the grove workers plant today?
A:
There are 15 trees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 − 15 = 6.
The answer is 6.
Q: If there are 3 cars in the parking lot and 2 more cars arrive, how many cars are in the parking lot ?
A:
There are originally 3 cars.
2 more cars arrive.
3 + 2 = 5.
The answer is 5.
Q: Leah had 32 chocolates and her sister had 42.
If they ate 35, how many pieces do they have left in total?
A: Originally, Leah had 32 chocolates.
Her sister had 42.
So in total they had 32 + 42 = 74.
After eating 35, they had 74 − 35 = 39.
The answer is 39.
Q: Jason had 20 lollipops.
He gave Denny some lollipops.
Now Jason has 12 lollipops.
How many lollipops did Jason give to Denny?
A:
Jason started with 20 lollipops.
Then he had 12 after giving some to Denny.
So he gave Denny 20 − 12 = 8.
The answer is 8.
Q: Shawn has five toys.
For Christmas, he got two toys each from his mom and dad.
How many toys does he have now?
A:
Shawn started with 5 toys.
If he got 2 toys each from his mom and dad, then that is 4 more toys.
5 + 4 = 9.
The answer is 9.
Q: There were nine computers in the server room.
Five more computers were installed each day, from monday to thursday.
How many computers are now in the server room?
A:
There were originally 9 computers.
For each of 4 days, 5 more computers were added.
So 5 *= 20 computers were added.
9 + 20 is 29.
The answer is 29.
Q: Michael had 58 golf balls.
On tuesday, he lost 23 golf balls.
On wednesday, he lost 2 more.
How many golf balls did he have at the end of wednesday?
A: Michael started with 58 golf balls.
After losing 23 on tuesday, he had 58 − 23 = 35.
After losing 2 more, he had 35 − 2 = 33 golf balls.
The answer is 33.
Q: Olivia has $23.
She bought five bagels for $3 each.
How much money does she have left?
A: Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 x 3 = 15 dollars.
So she has 23 −dollars left.
23 − 15 is 8.
The answer is 8.
gpt
ans
Q: There are 15 trees originally.
Then there were 21 trees after some more were planted.
So there must have been 21 − 15 = 6 trees planted.
A:Q: There are originally 3 cars.
2 more cars arrive.
3 + 2 = 5.
The answer is 5.
A:Q: Originally, Leah had 32 chocolates.
Her sister had 42.
So in total they had 32 + 42 = 74.
After eating 35, they had 74 − 35 = 39.
The answer is 39.
A:Q: Jason started with 20 lollipops.
Then he had 12 after giving some to Denny.
So he gave Denny 20 − 12 = 8 apples.
A:Q: Shawn started with 5 toys.
If he got 2 toys each from his mom and dad, then that is 4 more toys.
5 + 4 = 9.
The answer is 9.
A:Q: There were originally 9 computers.
For each of 4 days, 5 more computers were added.
So 5 *= 20 computers were added.
9 + 20 is 29.
The answer is 29.
A:Q: Michael started with 58 golf balls.
After losing 23 on tuesday, he had 58 − 23 = 35.
After losing 2 more, he had 35 − 2 = 33 golf balls.
A:Q: Olivia had 23 dollars.
5 bagels for 3 dollars each will be 5 x 3 = 15 dollars.
So she has 23 −= 8 dollars left.
A:
G.
H OTPOT QA
The prompts in this section implement Decomposed Prompting approach to open-domain multihop
QA.
For brevity we’ve included prompts for 5 of 25 randomly sampled questions.
The full prompts are attached with the submission and will also be released with the code.
Note that we selected a set of 100 questions from the development set to tune the hyperparameter (number of paragraphs to retrieve for all of the retrieval-based approaches).
decomp
QC: who is older Jeremy Horn or Renato Sobral ?
QS: (select) [retrieve odqa]
When was Jeremy Horn born?
A: { ”titles”: [”Jeremy Horn”, ”Train horn”], [”August 25, 1975”] } QS: (select) [retrieve odqa]
When was Renato Sobral born?
A: { ”titles”: [”Sobral Base”, Renato Sobral”, ”Luta Livre”], [”September 7, 1975”] } QS: (select) [multihop titleqa] Titles: [”Jeremy Horn”, ”Train horn”, ”Sobral Base”, Renato Sobral ”, ”Luta Livre”].
Question: Who is older Jeremy Horn or Renato Sobral?
A: [”Jeremy Horn”]
QS: [EOQ] QC:
In which country did this Australian who was detained in Guantanamo Bay detention camp and published ”Guantanamo: My Journey” receive para−military training?
QS: (select) [retrieve odqa]
Which Australian was detained in Guantanamo Bay detention camp and published ”Guantanamo: My Journey”?
A: { ”titles”: [”Shawali Khan”, ”Guantanamo: My Journey”, ”Yasim Muhammed Basardah”], ” answer”: [”David Hicks”] } QS: (select) [retrieve odqa] In which country did David Hicks receive para−military training?
A: { ”titles”: [”David Hicks”, ”Train (military)”], ”answer”: [”Afghanistan”] } QS: (select) [multihop titleqa] Titles: [”Shawali Khan”, ”Guantanamo: My Journey”, ”Yasim Muhammed Basardah”, ”David Hicks”, ”Train (military)”].
Question:
In which country did this Australian who was detained in Guantanamo Bay detention camp and published ” Guantanamo: My Journey” receive para−military training?
A: [”Afghanistan”]
QS: [EOQ]
QC: Which former Canadian Professional Ice Hockey Player endorses MLX Skates?
QS: (select) [retrieve odqa]
Who endorses MLX Skates?
A: { ”titles”: [”Marit Henie”, ”MLX Skates”, ”Puma Clyde”, ”MLX (gene)”], [”Mario Lemieux”] } QS: (select) [retrieve odqa]
Is Mario Lemieux a former Canadian Professional Ice Hockey Player?
A: { ”titles”: [”Mario Lemieux”, ”Claude Lemieux”, ”Mario Simioni”], [”yes”] } QS: (select) [multihop titleqa] Titles: [”Marit Henie”, ”MLX Skates”, ”Puma Clyde”, ”MLX (gene )”, ”Mario Lemieux”, ”Claude Lemieux”, ”Mario Simioni”].
Question: Which former Canadian Professional Ice Hockey Player endorses MLX Skates?
A: [”Mario Lemieux”] QC: Mister Magoo’s Christmas Carol was produced by the same studio that produced a film that featured the only animated−film role by who?
QS: (select) [retrieve odqa] Mister Magoo’s Christmas Carol was produced by what studio?
A: { ”titles”: [”UPA (animation studio)”, ”ImageMovers”, ”Jane Kean”, ”List of Christmas carols ”], [”United Productions of America”] } QS: (select) [retrieve odqa] United Productions of America studio produced a film that featured the only animated−film role by who?
A
[Table]
{ ”titles”: [”Gay Purr−ee”, ”Walt Disney Pictures”, ”Action Synthese”, ”A. Film A/S”], [”Judy Garland”] } QS: (select) [multihop titleqa] Titles: [”UPA (animation studio)”, ”ImageMovers”, ”Jane Kean”, ” List of Christmas carols”, ”Gay Purr−ee”, ”Walt Disney Pictures”, ”Action Synthese”, ”
A
Film A
/S”].
Question: Mister Magoo’s Christmas Carol was produced by the same studio that produced a film that featured the only animated−film role by who?
A: [”Judy Garland”] QS: [EOQ] QC:
What was the 2014 population of the city where Lake Wales Medical Center is located?
QS: (select) [retrieve odqa]
In which city is Lake Wales Medical Center is located?
A: { ”titles”: [”Lake Wales Medical Center”, ”Salt Lake Regional Medical Center”, ”Sky Lakes Medical Center”, ”Which Medical Device”, ”Lake Wales Medical Center”], ”answer”: [”Lake Wales”] } QS: (select) [retrieve odqa]
What was the population of Lake Wales in 2014?
A: { ”titles”: [”Lake Wales, Florida”, ”Wales Act 2014”, ”2014 in Wales”, ”Lake Albert (New South Wales)”, ”Saddlebag Lake Resort”, ”List of localities in Wales by population”], ” answer”: [”15,140”] } QS: (select) [multihop titleqa] Titles: [”Lake Wales Medical Center”, ”Salt Lake Regional Medical Center”, ”Sky Lakes Medical Center”, ”Which Medical Device”, ”Lake Wales Medical Center”, ”Lake Wales, Florida”, ”Wales Act 2014”, ”2014 in Wales”, ”Lake Albert (New South Wales)”, ”Saddlebag Lake Resort”, ”List of localities in Wales by population”].
Question:
What was the 2014 population of the city where Lake Wales Medical Center is located?
A: [”15,140”]
QS: [EOQ]
retrieve odqa
QC: When was Jeremy Horn born?
QS: (select) [retrieve] When was Jeremy Horn born?
A: [”Jeremy Horn”, ”Train horn”]
QS: (select) [singlehop titleqa] Titles: [”Jeremy Horn”, ”Train horn”].
Question: When was Jeremy Horn born?
A: { ”titles”: [”Jeremy Horn”, ”Train horn”], [”August 25, 1975”] } QS: [EOQ] QC: When was Renato Sobral born?
QS: (select) [retrieve] When was Renato Sobral born?
A: [”Sobral Base”, Renato Sobral”, ”Luta Livre”] QS: (select) [singlehop titleqa] Titles: [”Sobral Base”, Renato Sobral”, ”Luta Livre”].
Question: When was Renato Sobral born?
A: { ”titles”: [”Sobral Base”, Renato Sobral”, ”Luta Livre”], [”September 7, 1975”] } QS
[Table]
[EOQ] QC:
In which country did David Hicks receive para−military training?
QS: (select) [retrieve] In which country did David Hicks receive para−military training?
A: [”David Hicks”, ”Train (military)”]
QS: (select) [singlehop titleqa] Titles: [”David Hicks”, ”Train (military)”].
Question:
In which country did David Hicks receive para−military training?
A: { ”titles”: [”David Hicks”, ”Train (military)”], ”answer”: [”Afghanistan”] } QS: [EOQ] QC: Which Australian was detained in Guantanamo Bay detention camp and published ” Guantanamo: My Journey”?
QS: (select) [retrieve] Which Australian was detained in Guantanamo Bay detention camp and published ”Guantanamo: My Journey”?
A: [”Shawali Khan”, ”Guantanamo: My Journey”, ”Yasim Muhammed Basardah”] QS: (select) [singlehop titleqa] Titles: [”Shawali Khan”, ”Guantanamo: My Journey”, ”Yasim Muhammed Basardah”].
Question: Which Australian was detaindin Guantanamo Bay detention camp and published ”Guantanamo: My Journey”?
A: { ”titles”: [”Shawali Khan”, ”Guantanamo: My Journey”, ”Yasim Muhammed Basardah”], ” answer”: [”David Hicks”] } QS: [EOQ] QC:
Who endorses MLX Skates?
QS: (select) [retrieve] Who endorses MLX Skates?
A: [”Marit Henie”, ”MLX Skates”, ”Puma Clyde”, ”MLX (gene)”]
QS: (select) [singlehop titleqa] Titles: [”Marit Henie”, ”MLX Skates”, ”Puma Clyde”, ”MLX ( gene)”].
Question: Who endorses MLX Skates?
A: { ”titles”: [”Marit Henie”, ”MLX Skates”, ”Puma Clyde”, ”MLX (gene)”], [”Mario Lemieux”] } QS: [EOQ] QC: Is Mario Lemieux a former Canadian Professional Ice Hockey Player?
QS: (select) [retrieve] Is Mario Lemieux a former Canadian Professional Ice Hockey Player?
A: [”Mario Lemieux”, ”Claude Lemieux”, ”Mario Simioni”]
QS: (select) [singlehop titleqa] Titles: [”Mario Lemieux”, ”Claude Lemieux”, ”Mario Simioni”].
Question:
Is Mario Lemieux a former Canadian Professional Ice Hockey Player?
A: { ”titles”: [”Mario Lemieux”, ”Claude Lemieux”, ”Mario Simioni”], [”yes”] } QS: [EOQ] QC: Mister Magoo’s Christmas Carol was produced by what studio?
QS: (select) [retrieve] Mister Magoo’s Christmas Carol was produced by what studio?
A: [”UPA (animation studio)”, ”ImageMovers”, ”Jane Kean”, ”List of Christmas carols”]
QS: (select) [singlehop titleqa] Titles: [”UPA (animation studio)”, ”ImageMovers”, ”Jane Kean”, ” List of Christmas carols”].
Question: Mister Magoo’s Christmas Carol was produced by what studio?
A: { ”titles”: [”UPA (animation studio)”, ”ImageMovers”, ”Jane Kean”, ”List of Christmas carols ”], [”United Productions of America”] } QS: [EOQ] QC: United Productions of America studio produced a film that featured the only animated−film role by who?
QS: (select) [retrieve] United Productions of America studio produced a film that featured the only animated−film role by who?
A: [”Gay Purr−ee”, ”Walt Disney Pictures”, ”Action Synthese”, ”A. Film A/S”]
: United Productions of America studio produced a film that featured the only animated−film role by
Pictures”
[Table]
QC
: What was the population of Lake Wales in 2014?
QS: (select)
retrieve
What was the population of Lake Wales in
A: [”Lake
Wales
, Florida”, ”Wales Act 2014”, ”
in
Wales”, ”Lake Albert (
New South Wales
) ”, ”Saddlebag
Lake
Resort”, ”List
of localities
in
Wales
by population”]
QS: (select) [singlehop
titleqa
] Titles: [”Lake
Wales
, Florida”, ”Wales Act 2014”, ”
in Wales
”, ”Lake Albert (
New South Wales
)”, ”Saddlebag Lake Resort”, ”List of
localities
in
Wales
by population”].
Question:
What was the population of Lake
Wales
in 2014?
A: { ”titles”: [”Lake
Wales
, Florida”, ”Wales
[Table]
2014”, ”2014 in
Wales”
, ”Lake Albert (New
South
Wales
)”, ”Saddlebag Lake
[Table]
”List of localities in
[Table]
by
[Table] [Table]
[Table] [Table]
[Table]
QC
: In which city is Lake Wales Medical Center is located?
QS
[Table]
select
[retrieve
In
which city is Lake Wales Medical Center is located
A: [”Lake
Wales
Medical Center”, ”Salt Lake Regional Medical Center”, ”Sky Lakes Medical
Center”, ”Which Medical
Device”
, ”Lake
Wales
Medical Center”]
QS: (select) [singlehop titleqa] Titles: [”Lake
Wales
Medical Center”, ”Salt Lake Regional
Medical Center”, ”Sky Lakes Medical Center”, ”Which Medical
Device”
, ”Lake Wales
Medical Center”].
Question
: In
which city is Lake Wales
Medical Center is
located
A: { ”titles”: [”Lake Wales Medical Center”, ”Salt Lake Regional Medical Center”, ”Sky Lakes
Medical Center”, ”Which Medical Device”, ”Lake Wales Medical Center”], ”answer”:
[”Lake
Wales”
[Table]
singlehop titleqa
Wikipedia Title: Jeremy Horn < hidden for brevity >
Wikipedia Title: Renato Sobral < hidden for brevity >
Wikipedia Title: Renato Aragao < hidden for brevity >
Q: When was Jeremy Horn born?
A: [”August 25, 1975”]
Q: When was Renato Sobral born?
A: [”September 7, 1975”]
Wikipedia Title: Guantanamo: My Journey < hidden for brevity >
Wikipedia Title: David Hicks < hidden for brevity >
Wikipedia Title: John Adams Project < hidden for brevity >
Q: Which Australian was detained in Guantanamo Bay detention camp and published ” Guantanamo: My Journey”?
A: [”David Hicks”]
Q: In which country did David Hicks receive para−military training?
A: [”Afghanistan”]
Wikipedia Title: Brian Smith (ice hockey, born 1940)
< hidden for brevity >
Wikipedia Title: MLX Skates < hidden for brevity >
Wikipedia Title: Mario Lemieux < hidden for brevity >
Q: Is Mario Lemieux a former Canadian Professional Ice Hockey Player?
A: [”yes”]
Q: Who endorses MLX Skates?
A: [”Mario Lemieux”]
Wikipedia Title: Gay Purr−ee < hidden for brevity >
Wikipedia Title: A Christmas Carol < hidden for brevity >
Wikipedia Title: UPA (animation studio) < hidden for brevity >
Q: Mister Magoo’s Christmas Carol was produced by what studio?
A: [”United Productions of America”]
Q: United Productions of America studio produced a film that featured the only animated−film role by who?
A: [”Judy Garland”]
Wikipedia Title: Lake Wales Medical Center < hidden for brevity >
Wikipedia Title: Church of the Holy Spirit (Lake Wales, Florida) < hidden for brevity >
Wikipedia Title: Lake Wales, Florida < hidden for brevity >
Q: In which city is Lake Wales Medical Center is located?
A: [”Lake Wales”]
Q: What was the population of Lake Wales in 2014?
A: [”15,140”] multihop titleqa (direct)
Wikipedia Title: Jeremy Horn < hidden for brevity >
Wikipedia Title: Renato Sobral < hidden for brevity >
Wikipedia Title: Renato Aragao < hidden for brevity >
Q: who is older Jeremy Horn or Renato Sobral ?
A: [”Jeremy Horn”]
Wikipedia Title: Guantanamo: My Journey
< hidden for brevity >
Wikipedia Title: David Hicks < hidden for brevity >
Wikipedia Title: John Adams Project < hidden for brevity >
Q: In which country did this Australian who was detained in Guantanamo Bay detention camp and published ”Guantanamo: My Journey” receive para−military training?
A: [”Afghanistan”]
Wikipedia Title: Brian Smith (ice hockey, born 1940) < hidden for brevity >
Wikipedia Title: MLX Skates < hidden for brevity >
Wikipedia Title: Mario Lemieux < hidden for brevity >
Q: Which former Canadian Professional Ice Hockey Player endorses MLX Skates?
A: [”Mario Lemieux”]
Wikipedia Title: Gay Purr−ee < hidden for brevity >
Wikipedia Title: A Christmas Carol < hidden for brevity >
Wikipedia Title: UPA (animation studio) < hidden for brevity >
Q: Mister Magoo’s Christmas Carol was produced by the same studio that produced a film that featured the only animated−film role by who?
A: [”Judy Garland”]
Wikipedia Title: Lake Wales Medical Center < hidden for brevity >
Wikipedia Title: Church of the Holy Spirit (Lake Wales, Florida) < hidden for brevity >
Wikipedia Title: Lake Wales, Florida < hidden for brevity >
Q: What was the 2014 population of the city where Lake Wales Medical Center is located?
A: [”15,140”] multihop titleqa
cot)
Wikipedia Title: Jeremy Horn < hidden for brevity >
Wikipedia Title: Renato Sobral < hidden for brevity >
Wikipedia Title: Renato Aragao < hidden for brevity >
Q: who is older Jeremy Horn or Renato Sobral ?
A: Jeremy Horn was born on August 25, 1975.
Renato Sobral was born on September 7, 1975.
Thus, Jeremy Horn is older.
So the answer is: Jeremy Horn.
Wikipedia Title: Guantanamo: My Journey < hidden for brevity >
Wikipedia Title: David Hicks < hidden for brevity >
Wikipedia Title: John Adams Project < hidden for brevity >
Q
: In which country did this Australian who was detained in Guantanamo Bay detention camp and published ”Guantanamo: My Journey” receive para−military training
A
: The Australian who was detained in Guantanamo Bay detention camp and published
Guantanamo: My Journey” is David Hicks.
David Hicks received his para−military training in Afghanistan.
So the answer is: Afghanistan
Wikipedia Title: Brian Smith (ice hockey, born 1940) < hidden for brevity >
Wikipedia Title: MLX Skates < hidden for brevity >
Wikipedia Title: Mario Lemieux < hidden for brevity >
Q: Which former Canadian Professional Ice Hockey Player endorses MLX Skates?
A: MLX Skates is endorsed
by
Mario Lemieux
Mario Lemieux is
a former Canadian Professional Ice Hockey Player.
So the answer is: Mario Lemieux
Wikipedia Title: Gay Purr−ee < hidden for brevity >
Wikipedia Title: A Christmas Carol < hidden for brevity >
Wikipedia Title: UPA (animation studio) < hidden for brevity >
Q: Mister Magoo’s Christmas Carol was produced by the same studio that produced a film that featured the only animated−film role by who?
A: Mister Magoo’s Christmas Carol was produced by United Productions of America studio.
United Productions of America studio produced a film Gay Purr−ee, which features the voice of Judy Garland in her only animated−film role.
So the answer is: Judy Garland.
Wikipedia Title: Lake Wales Medical Center < hidden for brevity >
Wikipedia Title: Church of the Holy Spirit (Lake Wales, Florida) < hidden for brevity >
Wikipedia Title: Lake Wales, Florida < hidden for brevity >
Q: What was the 2014 population of the city where Lake
Wales Medical Center is located?
A: Lake Wales Medical Center is located in the city of Polk County, Florida.
The population of
Polk County in 2014 was 15,140.
So the answer is: 15,140.