Original: PARSE: split ( theme = boy ) DONE (A monster) (studied [study])
Revised: PARSE: split ( theme = boy ) DONE (A monster) (studied [study])

Original: PARSE: study ( agent = monster ) DONE (A girl) (stuttered [stutter])
Revised: PARSE: study ( agent = monster ) DONE (A girl) (stuttered [stutter])

Original: PARSE: stutter ( agent = girl ) DONE (A crocodile) (talked [talk]) PARSE: talk ( agent = crocodile ) DONE (A baby) (walked [walk])
Revised: PARSE: stutter ( agent = girl ) DONE (A crocodile) (talked [talk]) PARSE: talk ( agent = crocodile ) DONE (A baby) (walked [walk])

Original: PARSE: walk ( agent = baby ) DONE
Revised: PARSE: walk ( agent = baby ) DONE

Original: Once the sentences are fully decomposed, COGS only contains only 9 different types of verb phrases.
Revised: Once the sentences are fully decomposed, COGS only contains only 9 different types of verb phrases.

Original: We select for the pool 3 examples for each of them, which adds another 27 exemplars to the pool (see below) and hence results in a pool size of 89 exemplars in total.
Revised: We select for the pool 3 examples for each of them, which adds another 27 exemplars to the pool (see below) and hence results in a pool size of 89 exemplars in total.

Original: (The
Revised: (The

Original: * girl) (fed [feed])
Revised: * girl) (fed [feed])

Original: (the * dog) (the * mandarin)
Revised: (the * dog) (the * mandarin)

Original: PARSE:
Revised: PARSE:

Original: feed ( agent =
Revised: feed ( agent =

Original: * girl , recipient =
Revised: * girl , recipient =

Original: * dog ,
Revised: * dog ,

Original: theme
Revised: theme

Original: = * mandarin )
Revised: = * mandarin )

Original: DONE (The
Revised: DONE (The

Original: * cat) (gave [give]) (the * host) (the * yogurt)
Revised: * cat) (gave [give]) (the * host) (the * yogurt)

Original: PARSE:
Revised: PARSE:

Original: give ( agent = * cat , recipient =
Revised: give ( agent = * cat , recipient =

Original: * host , theme = * yogurt )
Revised: * host , theme = * yogurt )

Original: DONE (
Revised: DONE (

Original: The * mouse) (loaned [loan]) (the * turkey)
Revised: The * mouse) (loaned [loan]) (the * turkey)

Original: (the * chalk)
Revised: (the * chalk)

Original: PARSE: loan
Revised: PARSE: loan

Original: ( agent = * mouse , recipient =
Revised: ( agent = * mouse , recipient =

Original: * turkey , theme = * chalk )
Revised: * turkey , theme = * chalk )

Original: DONE (
Revised: DONE (

Original: The *
Revised: The *

Original: [Table]

Original: [Table]

Original: the

Original: [Table]
Revised: [Table]

Original: ) PARSE: like

Original: [Table]

Original: [Table]

Original: DONE

Original: [Table]
Revised: [Table]

Original: The
Revised: the

Original: [Table]
Revised: [Table]

Original: missed

Original: [Table]
Revised: [Table]

Original: miss
Revised: like

Original: [Table]
Revised: [Table]

Original: the

Original: [Table]
Revised: [Table]

Original: : miss [Table] baby ,
Revised: DONE

Original: [Table]
Revised: [Table]

Original: DONE(The
Revised: DONE

Original: [Table]
Revised: [Table]

Original: creature
Revised: creature

Original: [Table]
Revised: [Table]

Original: [Table]

Original: appreciate])
Revised: appreciate

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table]
Revised: [Table]

Original: PARSE: appreciate( agent
Revised: : appreciate

Original: [Table]
Revised: [Table]

Original: creature , theme=
Revised: creature , theme

Original: [Table]
Revised: [Table]

Original: present

Original: [Table]

Original: DONE (The
Revised: DONE (The

Original: [Table]

Original: ) (was found

Original: [Table]

Original: by (the

Original: [Table]

Original: ) PARSE:

Original: [Table]

Original: ( theme =

Original: [Table]

Original: cake ,

Original: [Table]

Original: =

Original: [Table]
Revised: [Table]

Original: DONE [Table] The [Table]
Revised: (was

Original: seed

Original: [Table]
Revised: [Table]

Original: by
Revised: by

Original: [Table]
Revised: [Table]

Original: the
Revised: the

Original: [Table]

Original: : eat

Original: [Table]
Revised: [Table]

Original: seed ,

Original: [Table]
Revised: [Table]

Original: DONE
Revised: DONE

Original: [Table]
Revised: [Table]

Original: by
Revised: eat

Original: [Table]
Revised: [Table]

Original: DONE
Revised: DONE

Original: [Table]
Revised: [Table]

Original: : like
Revised: DONE

Original: [Table]
Revised: [Table]

Original: DONE
Revised: DONE

Original: [Table]
Revised: [Table]

Original: discover
Revised: discover

Original: [Table]
Revised: [Table]

Original: DONE
Revised: [Table] [Table] [Table] [Table]

Original: DONE
Revised: DONE

Original: B.2. E XEMPLAR MATCHING
Revised: B.2. E XEMPLAR MATCHING

Original: For matching, we use essentially the same heuristics as we used for CFQ with slightly adjusted parameters.
Revised: For matching, we use essentially the same heuristics as we used for CFQ with slightly adjusted parameters.

Original: In particular, we only look at the inner-most subclause when selecting exemplars.
Revised: In particular, we only look at the inner-most subclause when selecting exemplars.

Original: This is sufﬁcient because the handling of the nested subclauses and nested prepositional clauses is demonstrated with a static prompt that we provide in addition to the dynamically selected exemplars (see
Revised: This is sufﬁcient because the handling of the nested subclauses and nested prepositional clauses is demonstrated with a static prompt that we provide in addition to the dynamically selected exemplars (see

Original: Section B.3 below).
Revised: Section B.3 below).

Original: If the inner-most subclause has a subject but no object (e.g., “a donut snapped”), we select the exemplar for the verb of this subclause, which is either unaccuastive or unergarive.
Revised: If the inner-most subclause has a subject but no object (e.g., “a donut snapped”), we select the exemplar for the verb of this subclause, which is either unaccusative or unergartive.

Original: This tells the model whether the subject should be annotated as the agent or the theme (e.g., since “snap” is unaccusative “a donate” is annotated as the theme).
Revised: This tells the model whether the subject should be annotated as the agent or the theme (e.g., since “snap” is unaccusative “a donate” is annotated as the theme).

Original: If the inner-most subclause has both a subject and an object, we select 3 exemplars corresponding to the subclause structure.
Revised: If the inner-most subclause has both a subject and an object, we select 3 exemplars corresponding to the subclause structure.

Original: Example 1.
Revised: Example 1.

Original: In our ﬁrst example “James said that a manager liked that Aiden appreciated that
Revised: In our ﬁrst example “James said that a manager liked that Aiden appreciated that

Original: Emily believed that the girl was posted a cake beside a table by Olivia .”
Revised: Emily believed that the girl was posted a cake beside a table by Olivia .”

Original: , the inner-most subclause is “the girl was posted a cake by Olivia”, which means that we select the following three exemplars:
Revised: , the inner-most subclause is “the girl was posted a cake by Olivia”, which means that we select the following three exemplars:

Original: Q: (The * frog) (was mailed [mail]) (the * ball) by (
Revised: Q: (The * frog) (was mailed [mail]) (the * ball) by (

Original: the * child) A: PARSE: mail ( recipient = * frog , theme =
Revised: the * child) A: PARSE: mail ( recipient = * frog , theme =

Original: * ball , agent =
Revised: * ball , agent =

Original: * child ) DONE Q: (The * dog) (was given [give]) (the * cookie) by (the * duck)
Revised: * child ) DONE Q: (The * dog) (was given [give]) (the * cookie) by (the * duck)

Original: A: PARSE: give ( recipient =
Revised: A: PARSE: give ( recipient =

Original: * dog , theme = * cookie , agent =
Revised: * dog , theme = * cookie , agent =

Original: * duck ) DONE Q: (The * prince) (was passed [pass])
Revised: * duck ) DONE Q: (The * prince) (was passed [pass])

Original: (the * box) by (the * president)
Revised: (the * box) by (the * president)

Original: A: PARSE: pass ( recipient =
Revised: A: PARSE: pass ( recipient =

Original: * prince , theme = * box , agent =
Revised: * prince , theme = * box , agent =

Original: * president )
Revised: * president )

Original: DONE
Revised: DONE

Original: Example 2.
Revised: Example 2.

Original: In our second example “The boy shortened the donut beside the bed in the car in the garden in the can on the tree .”, the inner-most subclause is “the boy shortened the donit”, which means that we select the following three exemplars:
Revised: In our second example “The boy shortened the donut beside the bed in the car in the garden in the can on the tree .”, the inner-most subclause is “the boy shortened the donut”, which means that we select the following three exemplars:

Original: Q: (The * pig) (liked [like]) (the * zebra) A: PARSE: like ( agent = * pig , theme =
Revised: Q: (The * pig) (liked [like]) (the * zebra) A: PARSE: like ( agent = * pig , theme =

Original: * zebra ) DONE Q: (The * baby) (missed [miss])
Revised: * zebra ) DONE Q: (The * baby) (missed [miss])

Original: (the * cake) A: PARSE: miss ( agent =
Revised: (the * cake) A: PARSE: miss ( agent =

Original: * baby , theme = * cake ) DONE Q: (The * creature) (appreciated [appreciate]) (the * present)
Revised: * baby , theme = * cake ) DONE Q: (The * creature) (appreciated [appreciate]) (the * present)

Original: A: PARSE: appreciate ( agent =
Revised: A: PARSE: appreciate ( agent =

Original: * creature , theme = * present ) DONE
Revised: * creature , theme = * present ) DONE

Original: B. COGS S OLUTION : D ETAILS AND P ROMPTS
Revised: B. COGS S OLUTION : D ETAILS AND P ROMPTS

Original: As we discussed in Section 3.3, we linearize the composition tree into a sequence of sentences that correspond to increasingly complex subproblems.
Revised: As we discussed in Section 3.3, we linearize the composition tree into a sequence of sentences that correspond to increasingly complex subproblems.

Original: Like with CFQ (see Section A.3, this linearization is performed by a walk over the parse tree to obtain a linear sequence of well-formed sentences.
Revised: Like with CFQ (see Section A.3, this linearization is performed by a walk over the parse tree to obtain a linear sequence of well-formed sentences.

Original: For each sentence, we then query the language model using a prompt that consists of three parts.
Revised: For each sentence, we then query the language model using a prompt that consists of three parts.

Original: Static prompt context illustrating the composition of subclauses and prepositional phrases 2.
Revised: Static prompt context illustrating the composition of subclauses and prepositional phrases 2.

Original: Dynamically selected exemplars as additional context 3.
Revised: Dynamically selected exemplars as additional context 3.

Original: Sequential subproblems.
Revised: Sequential subproblems.

Original: These parts are detailed throughout the rest of this section.
Revised: These parts are detailed throughout the rest of this section.

Original: B.3. P ART 1: S TATIC PROMPT CONTEXT
Revised: B.3. P ART 1: S TATIC PROMPT CONTEXT

Original: The compositions of subclauses and prepositional phrases are translated very systematically.
Revised: The compositions of subclauses and prepositional phrases are translated very systematically.

Original: Furthermore, because of the sequential prompting, the model only needs to perform one step of compposition at a time.
Revised: Furthermore, because of the sequential prompting, the model only needs to perform one step of compposition at a time.

Original: As a consequence, it is sufﬁcient to demonstrate this behavior with the following static prompt context that is used for all inputs.
Revised: As a consequence, it is sufﬁcient to demonstrate this behavior with the following static prompt context that is used for all inputs.

Original: Noe that this prompt does not contain any nesting beyond level 2.
Revised: Noe that this prompt does not contain any nesting beyond level 2.

Original: Note that we again use a parsed version of the natural language sentences, which contain additional parentheses and other annotations that make it easier for the model to perform the translation.
Revised: Note that we again use a parsed version of the natural language sentences, which contain additional parentheses and other annotations that make it easier for the model to perform the translation.

Original: Q: (the * song) (needed [need]) (to help)
Revised: Q: (the * song) (needed [need]) (to help)

Original: A: PARSE: need ( agent =
Revised: A: PARSE: need ( agent =

Original: * song , xcomp = help ( agent = * song ) ) DONE Q: (The * camera) (hoped [hope]) (to run)
Revised: * song , xcomp = help ( agent = * song ) ) DONE Q: (The * camera) (hoped [hope]) (to run)

Original: A: PARSE: hope ( agent =
Revised: A: PARSE: hope ( agent =

Original: * camera , xcomp = run ( agent = * camera ) ) DONE Q: (the * atom) (wanted [want]) (to hunt)
Revised: * camera , xcomp = run ( agent = * camera ) ) DONE Q: (the * atom) (wanted [want]) (to hunt)

Original: A: PARSE: wanted ( agent = * atom , xcomp = hunt ( agent = * atom ) ) DONE Q: (a helmet) (preferred [prefer]) (to cook)
Revised: A: PARSE: wanted ( agent = * atom , xcomp = hunt ( agent = * atom ) ) DONE Q: (a helmet) (preferred [prefer]) (to cook)

Original: A: PARSE: prefer ( agent = helmet , xcomp = cook ( agent = helmet ) ) DONE Q: (Fred) (expected [expect]) (to paint)
Revised: A: PARSE: prefer ( agent = helmet , xcomp = cook ( agent = helmet ) ) DONE Q: (Fred) (expected [expect]) (to paint)

Original: A: PARSE: expect ( agent =
Revised: A: PARSE: expect ( agent =

Original: Fred , xcomp = paint ( agent =
Revised: Fred , xcomp = paint ( agent =

Original: Fred ) )
Revised: Fred ) )

Original: DONE Q: (A camera) (intended [intend]) (to clean) A: PARSE: intend ( agent = camera ,
Revised: DONE Q: (A camera) (intended [intend]) (to clean) A: PARSE: intend ( agent = camera ,

Original: xcomp = clean ( agent = camera ) ) DONE Q: ((A camera) (beside) (the * song))
Revised: xcomp = clean ( agent = camera ) ) DONE Q: ((A camera) (beside) (the * song))

Original: (intended [intend]) (to clean) A: PARSE: intend ( agent = camera ( nmod .
Revised: (intended [intend]) (to clean) A: PARSE: intend ( agent = camera ( nmod .

Original: beside = * song ) , xcomp = clean ( agent = camera
Revised: beside = * song ) , xcomp = clean ( agent = camera

Original: ( nmod . beside = * song ) ) ) DONE Q: (Lucas) (liked [like]) (to eat)
Revised: ( nmod . beside = * song ) ) ) DONE Q: (Lucas) (liked [like]) (to eat)

Original: A: PARSE: like ( agent =
Revised: A: PARSE: like ( agent =

Original: Lucas ,
Revised: Lucas ,

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: PARSE
Revised: PARSE

Original: DONE
Revised: DONE

Original: A
Revised: A

Original: DONE
Revised: DONE

Original: Q
Revised: Q

Original: PARSE
Revised: PARSE

Original: DONE
Revised: DONE

Original: PARSE
Revised: PARSE

Original: DONE DONE
Revised: DONE DONE

Original: DONE DONE
Revised: DONE DONE

Original: A
Revised: A

Original: DONE
Revised: DONE

Original: Q
Revised: Q

Original: A: PARSE
Revised: A: PARSE

Original: DONE
Revised: DONE

Original: Q
Revised: Q

Original: DONE DONE
Revised: DONE DONE

Original: DONE
Revised: DONE

Original: B.3. P ART 2: D YNAMICALLY SELECTED EXEMPLARS
Revised: B.3. P ART 2: D YNAMICALLY SELECTED EXEMPLARS

Original: After the constant prompt context, we add as additional context the exemplars that we dynamically selected for the given input using the process described in Section B.2.
Revised: After the constant prompt context, we add as additional context the exemplars that we dynamically selected for the given input using the process described in Section B.2.

Original: We provide these exemplars for both of our runnning examples in Section B.2.2 and do not repeat them here.
Revised: We provide these exemplars for both of our runnning examples in Section B.2.2 and do not repeat them here.

Original: B.3. P ART 3: S EQUENTIAL SUBPROBLEMS
Revised: B.3. P ART 3: S EQUENTIAL SUBPROBLEMS

Original: After provding the static and dynamic prompt context, we perform the sequential prompting.
Revised: After providing the static and dynamic prompt context, we perform the sequential prompting.

Original: We start by appending to the prompt the simplest subproblem and send it as a request to the language model.
Revised: We start by appending to the prompt the simplest subproblem and send it as a request to the language model.

Original: We then append the model’s reply before we append the next subproblem and make another request to the language model.
Revised: We then append the model’s reply before we append the next subproblem and make another request to the language model.

Original: This is done until we obtain the result for the ﬁnal subproblem, which corresponds to the solution of the original problem.
Revised: This is done until we obtain the result for the ﬁnal subproblem, which corresponds to the solution of the original problem.

Original: Example 1.
Revised: Example 1.

Original: For the ﬁrst example (“James said that a manager liked that Aiden appreciated that
Revised: For the ﬁrst example (“James said that a manager liked that Aiden appreciated that

Original: Emily believed that the girl was posted a cake beside a table by Olivia .”), this part of the prompt looks as follows.
Revised: Emily believed that the girl was posted a cake beside a table by Olivia .”), this part of the prompt looks as follows.

Original: Note that thetext that is not bolded  corresponds tothe answers provided by the model.
Revised: Note that the not bolded text corresponds to answers provided by the model.

Original: Q: (the * girl) (was posted [post]) (a cake) by (Olivia)
Revised: Q: the * was posted post A: PARSE: post ( recipient Olivia

Original: A: PARSE: post ( recipient = * girl , theme = cake , agent =
Revised: DONE

Original: Olivia ) DONE Q: (the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia)
Revised: Q: (the * girl) (was posted [post]) ((a cake) (beside) (a )) by (Olivia)

Original: A: PARSE: post ( recipient = * girl , theme = cake ( nmod .
Revised: A: PARSE: post ( recipient = * girl , theme = cake ( nmod .

Original: beside = table ) , agent =
Revised: beside = table ) , agent =

Original: Olivia ) DONE Q: (Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia))
Revised: Olivia ) DONE Q: (Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia))

Original: A: PARSE: believe ( agent =
Revised: A: PARSE: believe ( agent =

Original: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .
Revised: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .

Original: beside = table ) , agent = Olivia ) ) DONE Q: (Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia)))
Revised: beside = table ) , agent = Olivia ) ) DONE Q: (Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia)))

Original: A: PARSE: appreciate ( agent =
Revised: A: PARSE: appreciate ( agent =

Original: Aiden , ccomp = believe ( agent =
Revised: Aiden , ccomp = believe ( agent =

Original: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .
Revised: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .

Original: beside = table ), agent = Olivia ) ) ) DONE Q: (a manager) (liked [like]) that ((Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia))))
Revised: beside = table ) agent = Olivia ) ) ) DONE Q: (a manager) (liked [like]) that ((Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia))))

Original: A: PARSE: like ( agent = manager , ccomp = appreciate ( agent =
Revised: A: PARSE: like ( agent = manager , ccomp = appreciate ( agent =

Original: Aiden , ccomp = believe ( agent =
Revised: Aiden , ccomp = believe ( agent =

Original: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .
Revised: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .

Original: beside = table ) , agent = Olivia ) ) ) ) DONE
Revised: beside = table ) , agent = Olivia ) ) ) ) DONE

Original: Q: (James) (said [say]) that ((a manager) (liked [like]) that ((Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia)))))
Revised: Q: (James) (said [say]) that ((a manager) (liked [like]) that ((Aiden) (appreciated [appreciate]) that ((Emily) (believed [believe]) that ((the * girl) (was posted [post]) ((a cake) (beside) (a table)) by (Olivia)))))

Original: A: PARSE: say ( agent =
Revised: A: PARSE: say ( agent =

Original: James , ccomp = like ( agent = manager , ccomp = appreciate ( agent =
Revised: James , ccomp = like ( agent = manager , ccomp = appreciate ( agent =

Original: Aiden , ccomp = believe ( agent =
Revised: Aiden , ccomp = believe ( agent =

Original: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .
Revised: Emily , ccomp = post ( recipient = * girl , theme = cake ( nmod .

Original: beside = table ) , agent = Olivia ) ) ) ) ) DONE
Revised: beside = table ) , agent = Olivia ) ) ) ) ) DONE

Original: Example 2.
Revised: Example 2.

Original: For the second example (“The boy shortened the donut beside the bed in the car in the garden in the can on the tree .”), the sequential part of the prompt looks as follows.
Revised: For the second example (“The boy shortened the donut beside the bed in the car in the garden in the can on the tree .”), the sequential part of the prompt looks as follows.

Original: Q: (The * boy) (shortened [shorten])
Revised: Q: (The * boy) (shortened [shorten])

Original: (the * donut) A: PARSE: shorten ( agent =
Revised: (the * donut) A: PARSE: shorten ( agent =

Original: * boy , theme = * donut ) DONE Q: (The * boy) (shortened [shorten])
Revised: * boy , theme = * donut ) DONE Q: (The * boy) (shortened [shorten])

Original: ((the * donut) (beside) (the * bed))
Revised: ((the * donut) (beside) (the * bed))

Original: A: PARSE: shorten ( agent =
Revised: A: PARSE: shorten ( agent =

Original: * boy , theme =
Revised: * boy , theme =

Original: * donut ( nmod .
Revised: * donut ( nmod .

Original: beside = * bed ) ) DONE Q: (The * boy) (shortened [shorten])
Revised: beside = * bed ) ) DONE Q: (The * boy) (shortened [shorten])

Original: ((the * donut) (beside) ((the * bed) (in) (the * car)))
Revised: ((the * donut) (beside) ((the * bed) (in) (the * car)))

Original: A: PARSE: shorten ( agent =
Revised: A: PARSE: shorten ( agent =

Original: * boy , theme =
Revised: * boy , theme =

Original: * donut ( nmod .
Revised: * donut ( nmod .

Original: beside =
Revised: beside =

Original: * bed ( nmod .
Revised: * bed ( nmod .

Original: in = * car ) ) )
Revised: in = * car ) ) )

Original: DONE
Revised: DONE

Original: Q: (The * boy) (shortened [shorten]) ((the * donut) (beside) ((the * bed) (in) ((the * car) (in) (the * garden))))
Revised: Q: (The * boy) (shortened [shorten]) ((the * donut) (beside) ((the * bed) (in) ((the * car) (in) (the * garden))))

Original: A: PARSE: shorten ( agent =
Revised: A: PARSE: shorten ( agent =

Original: * boy , theme =
Revised: * boy , theme =

Original: * donut ( nmod .
Revised: * donut ( nmod .

Original: beside =
Revised: beside =

Original: * bed ( nmod .
Revised: * bed ( nmod .

Original: in = * car ( nmod .
Revised: in = * car ( nmod .

Original: in = * garden ) ) ) ) DONE Q: (The * boy) (shortened [shorten]
Revised: in = * garden ) ) ) ) DONE Q: (The * boy) (shortened [shorten]) ((the

Original: [Table]
Revised: [Table]

Original: A: PARSE: shorten ( agent = * boy , theme =
Revised: C C HAIN - OF - THOUGHT P ROMPTING FOR CFQ

Original: * donut ( nmod .
Revised: This section includes the chain-of-thought prompts introduced in Section 4.

Original: beside =
Revised: A prompt in chainof-thought format includes intermediate steps that are processed before predicting the ﬁnal answer.

Original: * bed ( nmod . in = * car ( nmod .
Revised: Although the exemplars include these intermediate steps already, the model must predict the steps for new input.

Original: in = * garden ( nmod .
Revised: Also, it is assumed that the exemplar pool has not previously been annotated with chainof-thought, so a procedure to bootstrap these chain-of-thought (also called rationale) is required.

Original: in = * can ( nmod . on = * tree ) ) ) ) ) ) DONE
Revised: In practice, we can do this by annotating a small set of exemplars (in our case 5), then using these to teach the model to predict new chain-of-thought through prompting.

Original: This is a hybrid prompt containing 15 vanilla input/output exemplars selected by bag-of-words similarity with the input, and a static set of 5 exemplars manually annotated inthe chain-of-thought format. This prompt is used to generate new chain-of-thought for the rest of the exemplar pool.
Revised: The prompt we use for bootstrapping is in Appendix C.1. For evaluation, we use prompts including a mix of exemplars, some in chain-of-thought format, and the rest in vanilla format, this allows us to keep the prompt relatively short, since chain-of-thought can be verbose, often the same length or longer than the original exemplar.

Original: An example  is shownbelow in reduced font size, since the chain-of-thought can be verbose. [Table] [Table]
Revised: An example chain-ofthought prompt uused in evaluation is shown in Appendix C.2. Note, the vanilla few-shot prompts we use are similar but only includes exemplars in the vanilla format.

Original: So [Table]
Revised: We only evaluate chain-of-thought and vanilla few-shot prompting against CFQ, not COGS.

Original: C. P REDICTION P ROMPT
Revised: C. B OOTSTRAP P ROMPT

Original: This is an example of the chain-of-thought prompt used to predict semantic parse output on evaluation data. It includes 10 exemplars in the vanilla input/output format, then 5 exemplars  in the chainof-thought format. 
Revised: This is a hybrid prompt containing 15 vanilla input/output exemplars selected by bag-of-words similarity with the input, and a static set of 5 exemplars manually annotated in the chain-of-thought format. This prompt is used to generate new chain-of-thought for the rest of the exemplar pool.

Original: The example is shown below in reduced font size, since the chain-of-thought can be verbose.
Revised: An example is shown below in reduced font size, since the chain-of-thought can be verbose.

Original: [Table]
Revised: [Table]

Original: Were

Original: [Table]

Original: [Table]

Original: [Table]

Original: input

Original: [Table]
Revised: [Table]

Original: Was

Original: [Table] screenwriter [Table] French
Revised: C. P REDICTION P ROMPT

Original: [Table]

Original: [Table]

Original: There is

Original: [Table]

Original: is

Original: [Table]

Original: So the  parse of this query is: Parse: [Table] count(*) WHERE { ?x [Table]
Revised: This is an example of the chain-of-thought prompt used to predict semantic parse output on evaluation data.

Original: _of ?

Original: x1 . ? x0 has_nationality British [Table] x1 a writer .
Revised: It includes 10 exemplars in the vanilla input/output format, then 5 exemplars in the chainof-thought format.

Original: M2 parent_of ? x0 } [Table] : Was M
Revised: The example is shown below in reduced font size, since the chain-of-thought can be verbose

Original: [Table]

Original: by a Swedish

Original: [Table]

Original: ’s

Original: [Table]

Original: and

Original: [Table]

Original: by M

Original: [Table]

Original: Type

Original: [Table]

Original: was/were =

Original: [Table]

Original: There is a Swedish

Original: [Table]
Revised: [Table]

Original: has

Original: [Table]

Original: nationality

Original: [Table]
Revised: [Table]

Original: So the parse of this query is: Parse:

Original: [Table]

Original: (*) WHERE { ?

Original: [Table]

Original: _to ?

Original: [Table]

Original: has

Original: [Table]

Original: nationality

Original: [Table]

Original: M

Original: [Table]

Original: [Table] : Was [Table] Query Type: was/were =
Revised: ﬁctional universe.ﬁctional character.married to/ns:ﬁctiona married to ˆ ns:people.person.nationality nationality of ns:people.person.children | ns:ﬁctional universe.ﬁctional character.children | ns:organization.organization.child/ns:org parent of ns:ﬁlm.producer.ﬁlm | ns:ﬁlm.production company.ﬁlms produced ns:ﬁlm.ﬁlm.produced by | ns:ﬁlm.ﬁlm.production companies produced by ns:people.person.sibling s/ns:people.sibling relationship.sibling | ns:ﬁctional universe.ﬁctional character.siblings sibling of ns:ﬁlm.ﬁlm.starring/ns:ﬁlm.performance.actor starred ns:ﬁlm.ﬁlm.written by written by ns:ﬁlm.writer.ﬁlm wrote ns:ﬁlm.actor actor ns:ﬁlm.ﬁlm art director art director

Original: [Table] count( [Table] There is a male executive
Revised: ns:ﬁlm.cinematographer cinematographer ns:ﬁlm.ﬁlm costumer designer costume designer ns:ﬁlm.director ﬁlm director ns:ﬁlm.editor ﬁlm editor ns:business.employer employer ns:ﬁctional universe.ﬁctional character ﬁctional character ns:ﬁlm.ﬁlm ﬁlm ns:ﬁlm.ﬁlm distributor ﬁlm distributor ns:people.person person ns:

Original: [Table]
Revised: [Table]

Original: directed by

Original: [Table]

Original: >

Original: [Table]

Original: _by

Original: [Table]

Original: x

Original: M2 is written by M

Original: [Table]

Original: >

Original: [Table]

Original: _by M4 M2 is a film

Original: [Table]

Original: >

Original: [Table] film So the parse of this query is: Parse [Table]
Revised: .producer ﬁlm producer ns:ﬁlm.production company production company ns:ﬁlm.writer writer

Original: WHERE [Table] [Table] D D ATA P REPARATION , E VALUATION , AND H YPERPARAMETERS D. CFQ P ROCESSING AND E VALUATION
Revised: D D ATA P REPARATION , E VALUATION , AND H YPERPARAMETERS D. CFQ P ROCESSING AND E VALUATION

Original: To make the CFQ benchmark more appropriate for processing with large language models we use the processing steps and evaluation method detailed below.
Revised: To make the CFQ benchmark more appropriate for processing with large language models, we use the processing steps and evaluation method detailed below.

Original: We veriﬁed that these steps do not alter the performance for fully supervised methods by reproducing experiments with T5-base.
Revised: We veriﬁed that these steps do not alter the performance for fully supervised methods by reproducing experiments with T5-base.

Original: The results match previously published results within 1-2 points on MCD1, MCD2, and MCD3.
Revised: The results match previously published results within 1-2 points on MCD1, MCD2, and MCD3.

Original: D.1. CFQ P REPROCESSING
Revised: D.1. CFQ P REPROCESSING

Original: To prepare the CFQ data we applied two preprocessing steps:
Revised: To prepare the CFQ data we apply two preprocessing steps:

Original: We replaced the excessively long freebase identiﬁers with human-readable strings (see Table 3).
Revised: We replace excessively long freebase identiﬁers with human-readable strings (see Table 4).

Original: We stripped FILTER statements because they always appear with the “sibling of” and “married to” properties, and would be trivial to add to the output after prediction
Revised: We strip FILTER statements because they always appear with the “sibling of” and “married to” properties, and would be trivial to add to the output after prediction

Original: (i
Revised: (i

Original: e., they can be considered to be part of the abbreviated property).
Revised: e., they can be considered to be part of the abbreviated property).

Original: For example, if “?x0 married to M0” appeared then so would “FILTER(?x0 != M0)”.
Revised: For example, if “?x0 married to M0” appeared, then so would “FILTER(?x0 ! = M0)”.

Original: Essentially, one can not be married to themselves, nor a sibling of themselves.
Revised: Essentially, one cannot be married to themselves, nor a sibling of themselves.

Original: D.1. CFQ P OSTPROCESSING AND E VALUATION
Revised: D.1. CFQ P OSTPROCESSING AND E VALUATION

Original: For CFQ, we apply the following post-processing to both gold and predicted semantic parses.
Revised: For CFQ, we apply the following post-processing to both gold and predicted semantic parses.

Original: Clause sanitization : We discard any malformed clause.
Revised: Clause sanitization : We discard any malformed clause.

Original: To measure well-formedness we check all of the following.
Revised: To measure well-formedness we check all of the following.

Original: (a) A clause should have 3 white-space separated tokens; (b) A clause should either have “a” as its second token or a string, so symbols such as “=” can be safely discard.
Revised: (a) A clause should have 3 white-space separated tokens; (b) A clause should either have “a” as its second token or a string, so symbols such as “=” can be safely discarded.

Original: Inverse properties : For properties that have an ofﬁcial inverse property in Freebase (e.g. “directed” vs. “directed by”) we deterministically pick one of its variants and ﬂip the arguments if needed.
Revised: Inverse properties : For properties that have an ofﬁcial inverse property in Freebase (e.g. “directed” vs. “directed by”), we deterministically pick one of its variants and ﬂip the arguments if needed.

Original: However, note that in some error cases the model will predict “sequel of” which does not get corrected since “sequel of” is not in the original property vocabulary, only “has sequel” and “has prequel”.
Revised: However, note that in some error cases the model will predict “sequel of”, which does not get corrected, since “sequel of” is not in the original property vocabulary, only “has sequel” and “has prequel”.

Original: Argument ordering : For clauses that are bidirectional (“sibling of”, “married to”), we sort the arguments alphabetically.
Revised: Argument ordering : For clauses that are bidirectional (“sibling of”, “married to”), we sort the arguments alphabetically.

Original: Statement ordering : Since order does not matter in SPARQL statements, we sort statements alphabetically.
Revised: Statement ordering : Since order does not matter in SPARQL statements, we sort statements alphabetically.

Original: Variable normalization : Since variable labels are arbitrary (?x0, ?
Revised: Variable normalization : Since variable labels are arbitrary (?x0, ?

Original: x1, etc.)
Revised: x1, etc.)

Original: in SPARQL, we re-label the variables so that they appear in increasing order, “SELECT ?
Revised: in SPARQL, we re-label the variables so that they appear in increasing order.

Original: x1 { ?x2 directed ?
Revised: E.g., “SELECT ?x1 { ?x2 directed ?

Original: x1 . ?
Revised: x1 . ?

Original: x1 inﬂuenced ?
Revised: x1 inﬂuenced ?

Original: x0 } ” gets converted to “SELECT ?
Revised: x0 } ” gets converted to “SELECT ?

Original: x0 { ?x1 directed ?
Revised: x0 { ?x1 directed ?

Original: x0 . ?
Revised: x0 . ?

Original: x0 inﬂuenced ?x2 } ”.
Revised: x0 inﬂuenced ?x2 } ”.

Original: We alternate running variable normalization and statement ordering until no change is detected, since re-labeling variables might impact the sort order of clauses.
Revised: We alternate running variable normalization and statement ordering until no change is detected, since re-labeling variables might impact the sort order of clauses.

Original: Stripping of implied types : CFQ is constructed such that types that are directly implied by a accompanied relation are dropped from the SPARQL even if these types are explicitly mentioned in the natural language question.
Revised: Stripping of implied types : CFQ is constructed such that types that are directly implied by a accompanied relation are dropped from the SPARQL even if these types are explicitly mentioned in the natural language question.

Original: For example, the clause “?
Revised: For example, the clause “?

Original: x0 a actor” is dropped from the translation of the question “Did a male actor play M0 and play M1” because the relation “?x0 portrayed M0” implies that ?
Revised: x0 a actor” is dropped from the translation of the question “Did a male actor play M0 and play M1” because the relation “?x0 portrayed M0” implies that ?

Original: x0 has type actor.
Revised: x0 has type actor.

Original: For a pretrained language[Table] inﬂuenced by ns:people.person.spouse s/ns:people.marriage.spouse | ns:ﬁctional universe.ﬁctional character.married to/ns:ﬁctiona married to ˆns:people.person.nationality nationality of ns:people.person.children | ns:ﬁctional universe.ﬁctional character.children | ns:organization.organization.child/ns:org parent of ns:ﬁlm.producer.ﬁlm | ns:ﬁlm.production company.ﬁlms produced ns:ﬁlm.ﬁlm.produced by | ns:ﬁlm.ﬁlm.production companies produced by ns:people.person.sibling s/ns:people.sibling relationship.sibling | ns:ﬁctional universe.ﬁctional character.siblings sibling of ns:ﬁlm.ﬁlm.starring/ns:ﬁlm.performance.actor starred ns:ﬁlm.ﬁlm.written by written by ns:ﬁlm.writer.ﬁlm wrote ns:ﬁlm.actor actor ns:ﬁlm.ﬁlm art director art director ns:ﬁlm.cinematographer cinematographer ns:ﬁlm.ﬁlm costumer designer costume designer ns:ﬁlm.director ﬁlm director ns:ﬁlm.editor ﬁlm editor ns:business.employer employer ns:ﬁctional universe.ﬁctional character ﬁctional character ns:ﬁlm.ﬁlm ﬁlm ns:ﬁlm.ﬁlm distributor ﬁlm distributor ns:people.person person ns:ﬁlm.producer ﬁlm producer ns:ﬁlm.production company production company ns:ﬁlm.writer writer model, this is quite unnatural and hurts accuracy even though keeping the type leads to a 
Revised: For a pretrained language model, this is quite unnatural and hurts accuracy, even though keeping the type leads to a SPARQL query that is semantically equivalent.

Original: SPARQL query that has the is semantically equivalent.

Original: We therefore strip implied types when they’re predicted by the language model.
Revised: We therefore strip implied types when they are predicted by the language model.

Original: After completing these steps, we do the standard approach and measure accuracy using exact string match.
Revised: After completing these steps we do the standard approach of measuring accuracy using exact string match.

Original: D. COGS P OSTPROCESSING AND E VALUATION
Revised: D. COGS P OSTPROCESSING AND E VALUATION

Original: For longer outputs, the language model sometimes fails to match closing parentheses.
Revised: For longer outputs, the language model sometimes fails to match closing parentheses.

Original: When this happens, we add closing parentheses at the end of output until all opening parenthesis are matched.
Revised: When this happens, we add closing parentheses at the end of output until all opening parenthesis are matched.

Original: This trivial ﬁx improved exact match accuracy on COGS generalization test set from 97.8% to 99.2%.
Revised: This trivial ﬁx improved exact match accuracy on COGS generalization test set from 97.8% to 99.2%.

Original: It’s worth noting  50 examples in the original COGS generalization test set are mislabeled (the authors have since released a new version).
Revised: It is worth noting that 50 examples in the original COGS generalization test set are mislabeled (the authors have since released a new version).

Original: We evaluate using the original data in order to compare with previous work.
Revised: We evaluate using the original data in order to compare with previous work.

Original: One would not expect a model to accurately predict the idiosyncrasies associated with the mislabeled data, so upper bound on performance for the generalization test should be about 99.7% accuracy.
Revised: As one would not expect a model to accurately predict the idiosyncrasies associated with the mislabeled data, an upper bound on performance for the generalization test should be about 99.7% accuracy.

Original: D. H YPERPARAMETERS
Revised: D. H YPERPARAMETERS

Original: There are two sets of hyperparameters. Those associated with prompts and those associated with generation.
Revised: There are two sets of hyperparameters: those for prompts and those for generation.

Original: When designing the initial prompts, we performed  minimal hyperparameter tuning using a 100-sentence subset of the validation data.
Revised: We performed initial minimal hyperparameter tuning using a 100-sentence subset of the validation data

Original:  D.3. P ROMPT H YPERPARAMETERS
Revised: . D.3. P ROMPT H YPERPARAMETERS

Original: Dynamic Least-to-Most Described in Section 3.
Revised: Dynamic Least-to-Most Described in Section 3.

Original: • Number of Static Exemplars = 12 for CFQ, 28 for COGS • Number of Dynamic Exemplars = 4-35 for CFQ, between 1-3 for COGS (these are determined automatically based on the decomposition tree) • Number of Exemplar Lists =
Revised: • Number of Static Exemplars = 12 for CFQ, 28 for COGS • Number of Dynamic Exemplars = 4-35 for CFQ, between 1-3 for COGS (these are determined automatically based on the decomposition tree) • Number of Exemplar Lists =

Original: • Number of Generations per List =•
Revised: • Number of Generations per List =•

Original: Generation Mode =
Revised: Generation Mode =

Original: Greedy
Revised: Greedy

Original: Chain-of-Thought Selects exemplars according to bag-of-words similarity (Section 4.2), but to be effective, at least some of the exemplars must use rationale.
Revised: Chain-of-Thought Selects exemplars according to bag-of-words similarity (Section 4.2), but to be effective, at least some of the exemplars must use rationale.

Original: To keep the prompt compact, we use a hybrid approach where 5 exemplars are in chain-of-thought format and the rest are vanilla.
Revised: To keep the prompt compact, we use a hybrid approach where 5 exemplars are in chain-of-thought format and the rest are vanilla.

Original: The original data does not have alignments,so we need to manually create some chain-of-thought exemplars which we can then use to generate chain-of-thought for the full exemplar pool (Zelikman et al., 2022).
Revised: As the original data does not have alignments, we need to manually create some chain-of-thought exemplars, which we can then use to generate chain-of-thought for the full exemplar pool (Zelikman et al., 2022).

Original: In our case, we manually labeled 5 sentences with chain-of-thought, then appended these to the 15 exemplars we already retrieve for each sentence.
Revised: In our case, we manually labeled 5 sentences with chain-of-thought, then appended these to the 15 exemplars we already retrieve for each sentence.

Original: We did this to predict chain-ofthought for the exemplar pool, and discarded any chain-of-thought that did not produce the correct semantic parse.
Revised: We did this to predict chain-ofthought for the exemplar pool, and discarded any chain-of-thought that did not produce the correct semantic parse.

Original: 884 out of 1000 successfully predicted the correct semantic parse.
Revised: 884 out of 1000 successfully predicted the correct semantic parse.

Original: When constructing chain-of-thought prompts, we only represent an exemplar with chain-of-thought if it succeeded in this previous step, otherwise we use its vanilla format.
Revised: When constructing chain-of-thought prompts, we only represent an exemplar with chain-of-thought if it succeeded in this previous step, otherwise we use its vanilla format.

Original: • Bootstrap Pool Size =• Number of Exemplars per List ( k ) =
Revised: • Bootstrap Pool Size =• Number of Exemplars per List ( k ) =

Original: • Number of Exemplar Lists ( n ) =
Revised: • Number of Exemplar Lists ( n ) =

Original: • Number of Generations per List ( s ) =
Revised: • Number of Generations per List ( s ) =

Original: • Generation Mode =
Revised: • Generation Mode =

Original: Sample • Number of Chain -of-Thought Exemplars = 5 (the other 10 are in vanilla format)
Revised: Sample • Number of Chain-of-Thought Exemplars = 5 (the other 10 are in vanilla format)

Original: Vanilla Few-Shot Vanilla few-shot is a simple input/output exemplar-based prompts.
Revised: Vanilla Few-Shot Vanilla few-shot is a simple input/output exemplar-based prompt.

Original: The exemplars are chosen identically as chain-of-thought.
Revised: The exemplars are chosen identically as chain-of-thought.

Original: • Number of Exemplars per List ( k ) =
Revised: • Number of Exemplars per List ( k ) =

Original: • Number of Exemplar Lists ( n ) =
Revised: • Number of Exemplar Lists ( n ) =

Original: • Number of Generations per List ( s ) =
Revised: • Number of Generations per List ( s ) =

Original: • Generation Mode =
Revised: • Generation Mode =

Original: Sample
Revised: Sample

Original: D.3. G ENERATION H YPERPARAMETERS
Revised: D.3. G ENERATION H YPERPARAMETERS

Original: We did not require any extensive search over these hyperparamters.
Revised: We did not require any extensive search over these hyperparamters.

Original: We tried reasonable settings based on previously published works, and the only real toggle here is temperature, which wasn’t used at all by dynamic least-to-most.
Revised: We tried reasonable settings based on previously published works, and the only real toggle here is temperature, which wasn’t used at all by dynamic least-to-most.

Original: • Model = code-davinci- • Temperature = 0.7 when sampling, or 0.0 when greedy.
Revised: • Model = code-davinci- • Temperature = 0.7 when sampling, or 0.0 when greedy.

Original: • Top-P = 1. • Presence Penalty =• Frequency Penalty =
Revised: • Top-P = 1.0, Presence Penalty = 0, Frequency Penalty =

Original: E A DDITIONAL A NALYSIS E. I NITIAL P ROMPTING A TTEMPTS FOR CFQ
Revised: E A DDITIONAL A NALYSIS E. I NITIAL P ROMPTING A TTEMPTS FOR CFQ

Original: We initially tried various prompting attempts for CFQ where we did not provide any examples as part of the context.
Revised: We initially tried various prompting attempts for CFQ where we did not provide any examples as part of the context.

Original: Instead, we provided an instruction along the lines of, “Translate English to
Revised: Instead, we provided an instruction along the lines of, “Translate English to

Original: SPARQL”.
Revised: SPARQL”.

Original: While the language model was still able to produce SPARQL-like output, it scored 0% accuracy on CFQ.
Revised: While the language model was still able to produce SPARQL-like output, it scored 0% accuracy on CFQ.

Original: This indicates that the language model was exposed to SPARQL in pretraining but has not seen or memorized speciﬁc CFQ examples.
Revised: This indicates that the language model was exposed to SPARQL in pretraining but has not seen or memorized speciﬁc CFQ examples.

Original: # Translate English to SPARQL.
Revised: to SPARQL.

Original: English: What did a production company ’s Japanese founder direct , edit , and 
Revised: English: What did a production company ’s Japanese founder direct , edit , and executive produce

Original: [Table] produce SPARQL: 
Revised: SPARQL: ‘‘‘SELECT?director ?editor ?

Original: [Table]
Revised: executive_producer

Original: WHERE { ?director a dbo:FilmDirector .
Revised: WHERE { ?director a dbo:FilmDirector .

Original: [Table]
Revised: director dbo:nationality dbr:Japan . ? director dbo:birthPlace dbr:Japan . director dbo:birthDate ?birthDate .

Original: FILTER 
Revised: FILTER (?birthDate < "1940-01-01T00:00:00Z"ˆˆxsd:dateTime) ? director dbo:film ?film . ? film dbo:editor ?editor . film dbo:executiveProducer ?

Original: [Table] [Table]
Revised: executive_producer . } ‘‘‘

Original: E
Revised: Acc. Type Cartesian product Property direction

Original: [Table]
Revised: Wrong property /type Entity reference

Original: D YNAMIC L EAST - TO
Revised: Full dynamic L2M (0) 94. − Delta highlights (a) 93. − Decomposition hints (b) 92. − Question

Original: [Table]
Revised: [Table]

Original:  MOST P ROMPTING ON 
Revised: E. D YNAMIC L EAST - TO - MOST P ROMPTING ON CFQ: A BLATIONS AND E RROR A NALYSIS

Original: AND During the development of dynamic least-to-most prompting, we performed a qualitative error analysis for various ablations on a 100-example subset of the CFQ validation set (MCD1 split).

Original: This provides us with interesting insights about the impact of different components that make up the dynamic least-to-most prompting technique.
Revised: Here we include 1-step ablations of the most important components of the dynamic least-to-most method (explained in detail in Appendix A.3).

Original: We compare the following 4 setups: Decomposition-based exemplar retrieval. We use the decomposition to retrieve the exemplars, then execute a vanilla few-shot prompt. We do not perform any form of sequential least-to-most prompting.
Revised: These components are:

Original: + Decomposition hints. Same as above, but adds brackets to sentences indicating compositional structure and hinting at relations between exemplars (see Appendix A for examples). + Least-to-most prompting.
Revised: • Delta highlights are textual hints (such as “# Extend the answer above. Add: ...” and “# Copy the answer from above.”)

Original: Adds sequential least-to-most prompting, including subproblems in the prompt that were used toselect the exemplars. + Chain-of-thought (CoT) grounding.
Revised: that highlight the change between subsequent subquestions. • Decomposition hints are parenthesis that we add to the natural language questions to indicate the syntactic parse.

Original: A constant prompt preﬁx with basic subproblems solved in a chain-of-thought-like way. This corresponds to the full implementation of dynamic least-to-most prompting, also described in Appendix A.3 with concrete examples. Table 4 shows the accuracy and error types associated with these setups.
Revised: • Question grounding is a constant prompt preﬁx that illustrates to the model how to translate the most basic subquestions. • Exemplar selection .

Revised: Bag of words (BoW) exemplar selection is an alternative method to select relevant exemplars.

Revised: Instead of using the parse tree, it aims to cover with exemplars as many words as possible of the target question.

Revised: Constant exemplars uses a static set of exemplars for all questions (i

Revised: e., no dynamically selected exemplars).

Revised: • Decomposition granularity .

Revised: No decomposition does not decompose the question at all, which means that we then use standard prompting rather than least-to-most prompting.

Revised: 2step decomposition decomposes the question into at most 2 steps, while 50% decomposition keeps half the decomposition steps of the most granular decomposition.

Revised: The second column of Table 5 shows the accuracies of these ablations on a 500-example subset of

Revised: MCD1.

Original: We observe that the most basic approach using decomposition-based exemplar retrieval only achieves 55% accuracy.
Revised: It turns out that ablating delta highlights (93.8% accuracy) and decomposition hints (92.4% accuracy) only lead to a small regression. This matches our intuition because these components cause a relatively small change to the overall prompt. Ablating question grounding on the other hand leads to a much larger regression (79.2% accuracy). The reason for this is that the partial exemplars turn out to be important for the model to correctly translate the ﬁrst subquestion.

Original: The majority of the errors (24) are  caused by unnecessary or missing types and by incorrect expansion of cartesian products (11).
Revised: The performance of the other ablations are discussed in Section 6. In addition to the accuracies, Table 5 also contains a qualitative error analysis. For the baseline (ﬁrst row), we analyzed 20 random errors. For each of the ablations, we analyzed 20 errors that were caused by the ablation (i.e., the examples were correct in the baseline).

Original: Adding decomposition hints in form of parentheses to the question eliminates some of the errors across the board and achieves 67% accuracy.
Revised: We make the following observations: • Delta highlights (a) and BoW exemplar selection (d) have a similar error proﬁle as the baseline. Property direction is a bit more prominent for BoW exemplar selection, which indicates that we do not always retrieve exemplars that illustrate the property in as much detail as in the baseline. • Decomposition hints (b)

Original: Adding least-to-most prompting further increases accuracy to 83%.
Revised: mostly help with cartesian products, which makes sense because they make it easier for the model to understand the grouping and connections between the various words of a question. • Question grounding (c) helpw a lot with types, which is the primary reason why we added them. • Using constant exemplars (e) leads to a lot of issues with wrong properties/types and property direction. This makes sense since the constant exemplars cannot illustrate all properties and types that are used in the dataset.

Original: The error analysis shows that this mostly because it eliminates all the errors related to  cartesian products. Finally, adding CoT grounding pushes the accuracy to 99%, which is mostly due to eliminating type errors.
Revised: • Reducing the decomposition granularity (f1, f2, f3) mostly leads to problems with types and cartesian products. Especially the issues with cartesian products are to be expected because decomposition allows the model to unfold those incrementally.

Original: Below are examples to illustrate the different error types.
Revised: Below, we provide a few examples to illustrate the different error buckets.

Original: Unnecessary or missing types
Revised: Unnecessary or missing types

Original: Question: What Dutch spouse of M2 was M0 ’s employee and founder
Revised: Question: What Dutch spouse of M2 was M0 ’s employee and founder

Original: Answer: SELECT DISTINCT ?
Revised: Answer: SELECT DISTINCT ?

Original: x0 WHERE { ?
Revised: x0 WHERE { ?

Original: x0 a person . ?
Revised: x0 a person . ?

Original: x0 married to M2 . ?
Revised: x0 married to M2 . ?

Original: x0 founded M0 .
Revised: x0 founded M0 .

Original: x0 employed by M0 .
Revised: x0 employed by M0 .

Original: x0 has nationality Dutch }
Revised: x0 has nationality Dutch }

Original: Golden: SELECT DISTINCT ?
Revised: Golden: SELECT DISTINCT ?

Original: x0 WHERE { ?x0 founded M0 . ?
Revised: x0 WHERE { ?x0 founded M0 . ?

Original: x0 employed by M0 . ?
Revised: x0 employed by M0 . ?

Original: x  has nationality Dutch . ?
Revised: x  has nationality Dutch . ?

Original: x0 married to M2 }
Revised: x0 married to M2 }

Original: Question: Who was a writer , costume designer , editor , and star of M1 and M
Revised: Question: Who was a writer , costume designer , editor , and star of M1 and M

Original: Answer: SELECT DISTINCT ?
Revised: Answer: SELECT DISTINCT ?

Original: x0 WHERE { ?x0 starred in M1 . ?
Revised: x0 WHERE { ?x0 starred in M1 . ?

Original: x0 starred in M2 . ?
Revised: x0 starred in M2 . ?

Original: x0 wrote M1 .
Revised: x0 wrote M1 .

Original: ?x0 wrote M2 . ?
Revised: ?x0 wrote M2 . ?

Original: x0 edited M1 . ?
Revised: x0 edited M1 . ?

Original: x0 edited M2 . ?
Revised: x0 edited M2 . ?

Original: x0 costume designed M1 .
Revised: x0 costume designed M1 .

Original: x0 costume designed
Revised: x0 costume designed

Original: M2 }
Revised: M2 }

Original: Golden: SELECT DISTINCT ?
Revised: Golden: SELECT DISTINCT ?

Original: x0 WHERE { ?
Revised: x0 WHERE { ?

Original: x0 a person . ?
Revised: x0 a person . ?

Original: x0 starred in M1 . ?
Revised: x0 starred in M1 . ?

Original: x0 starred in M2 .
Revised: x0 starred in M2 .

Original: x0 edited M1 . ?
Revised: x0 edited M1 . ?

Original: x0 edited M2 . ?
Revised: x0 edited M2 . ?

Original: x0 costume designed M1 .
Revised: x0 costume designed M1 .

Original: x0 costume designed M2 . ?
Revised: x0 costume designed M2 . ?

Original: x0 wrote
Revised: x0 wrote

Original: M1 .
Revised: M1 .

Original: ?x0 wrote M2 }
Revised: ?x0 wrote M2 }

Original: Incorrect cartesian product
Revised: Incorrect cartesian product

Original: Question:
Revised: Question:

Original: Did M1 ’s costume designer , cinematographer , director , art director , producer , and editor produce , write , and edit M
Revised: Did M1 ’s costume designer , cinematographer , director , art director , producer , and editor produce , write , and edit M

Original: Answer: SELECT count(*) WHERE { ?x0 directed M1 . ?
Revised: Answer: SELECT count(*) WHERE { ?x0 directed M1 . ?

Original: x0 executive produced M0 . ?
Revised: x0 executive produced M0 . ?

Original: x0 edited
Revised: x0 edited

Original: M0 . ?
Revised: M0 . ?

Original: x0 wrote M0 . ?
Revised: x0 wrote M0 . ?

Original: x0 cinematographer of M1 .
Revised: x0 cinematographer of M1 .

Original: x0 costume designed M1 . ?
Revised: x0 costume designed M1 . ?

Original: x0 produced M1 }
Revised: x0 produced M1 }

Original: Golden: SELECT count(*) WHERE { ?x0 cinematographer of M1 . ?
Revised: Golden: SELECT count(*) WHERE { ?x0 cinematographer of M1 . ?

Original: x0 directed M0 . ?
Revised: x0 directed M0 . ?

Original: x0 directed
Revised: x0 directed

Original: M1 .
Revised: M1 .

Original: x0 edited M0 . ?
Revised: x0 edited M0 . ?

Original: x0 costume designed M1 .
Revised: x0 costume designed M1 .

Original: x0 executive produced M0 . ?
Revised: x0 executive produced M0 . ?

Original: x0 produced M1 .
Revised: x0 produced M1 .

Original: ?x0 wrote M0 }
Revised: ?x0 wrote M0 }

Original: Question:
Revised: Question:

Original: Did M2 ’s editor , director , star , art director , and cinematographer produce , direct , and edit M0 and M
Revised: Did M2 ’s editor , director , star , art director , and cinematographer produce , direct , and edit M0 and M

Original: Answer: SELECT count(*) WHERE { ?x0 edited M0 . ?
Revised: Answer: SELECT count(*) WHERE { ?x0 edited M0 . ?

Original: x0 edited M1 . ?
Revised: x0 edited M1 . ?

Original: x0 directed M0 . ?
Revised: x0 directed M0 . ?

Original: x directed M1 .
Revised: x directed M1 .

Original: x0 produced M0 . ?
Revised: x0 produced M0 . ?

Original: x0 produced M1 . ?
Revised: x0 produced M1 . ?

Original: x0 starred in M2 . ?
Revised: x0 starred in M2 . ?

Original: x0 art directed M2 . ?
Revised: x0 art directed M2 . ?

Original: x cinematographer of M2 }
Revised: x cinematographer of M2 }

Original: Golden: SELECT count(*) WHERE { ?x0 starred in M2 . ?
Revised: Golden: SELECT count(*) WHERE { ?x0 starred in M2 . ?

Original: x0 cinematographer of M2 . ?
Revised: x0 cinematographer of M2 . ?

Original: x0 directed
Revised: x0 directed

Original: M0 . ?
Revised: M0 . ?

Original: x0 directed M1 .
Revised: x0 directed M1 .

Original: x0 directed M2 . ?
Revised: x0 directed M2 . ?

Original: x0 edited M0 . ?
Revised: x0 edited M0 . ?

Original: x0 edited M1 . ?
Revised: x0 edited M1 . ?

Original: x0 edited M2 . ?
Revised: x0 edited M2 . ?

Original: x art directed M2 . ?
Revised: x art directed M2 . ?

Original: x0 produced M0 . ?
Revised: x0 produced M0 . ?

Original: x0 produced M1 }
Revised: x0 produced M1 }

Original: Wrong property direction
Revised: Wrong property direction

Original: Question: Which actor that a cinematographer was inﬂuenced by was M1 ’s spouse
Revised: Question: Which actor that a cinematographer was inﬂuenced by was M1 ’s spouse

Original: Answer: SELECT DISTINCT ?
Revised: Answer: SELECT DISTINCT ?

Original: x0 WHERE { ?x0 a actor . ?
Revised: x0 WHERE { ?x0 a actor . ?

Original: x0 married to M1 . ?
Revised: x0 married to M1 . ?

Original: x0 inﬂuenced by ?x1 . ?
Revised: x0 inﬂuenced by ?x1 . ?

Original: x1 a cinematographer }
Revised: x1 a cinematographer }

Original: Golden: SELECT DISTINCT ?
Revised: Golden: SELECT DISTINCT ?

Original: x0 WHERE { ?x0 a actor . ?
Revised: x0 WHERE { ?x0 a actor . ?

Original: x0 inﬂuenced ?
Revised: x0 inﬂuenced ?

Original: x1 . ?
Revised: x1 . ?

Original: x0 married to M1 .
Revised: x0 married to M1 .

Original: x1 a cinematographer }
Revised: x1 a cinematographer }

Revised: Question: What prequel of M0 was executive produced , written , directed , edited , and produced by M

Original: Mixing up
Revised: Answer: SELECT DISTINCT ? x0 WHERE { ?x0 has prequel M0 . x0 executive produced by M . ? x0 produced by M1 .

Revised: x0 edited by M1 . ?

Revised: x0 directed by M1 . ?

Revised: x0 written by M1 }

Revised: Golden: SELECT DISTINCT ?

Revised: x0 WHERE { ?x0 directed by M1 . ?

Revised: x0 edited by M1 . ?

Revised: x executive produced by M1 .

Revised: x0 produced by M1 .

Original: entities
Revised: x0 has sequel M0 . ? x0 written by M1 } Wrong property / type

Original: {+Question: Was a ﬁlm director ’s child M
+}
Revised: {+Question: Was a ﬁlm director ’s child M
+}

Revised: ’s cinematographer

Revised: Answer: SELECT count(*) WHERE { ?

Revised: x0 child of ?x1 . ?

Revised: x1 a ﬁlm director . ?

Revised: x0 cinematographed

Revised: M0 }

Revised: Golden: SELECT count(*) WHERE { ?x0 cinematographer of M0 .

Revised: x0 child of ?x1 . ?

Revised: x1 a ﬁlm director }

Revised: Question: Was a company whose employee wrote M2 M1 ’s employer

Revised: Answer: SELECT count(*) WHERE { ?

Revised: x0 a company . ?

Revised: x0 employed M1 .

Revised: x0 employed ?

Revised: x1 .

Revised: x1 wrote M2 }

Revised: Golden: SELECT count(*) WHERE { ?

Revised: x0 a employer . ?

Revised: x0 employed ?

Revised: x1 . ?

Revised: x0 employed M1 . ?

Revised: x wrote M2 }

Revised: Mixing up entity references

Original: Question: What costume designer was a ﬁlm director ’s Canadian female parent
Revised: Question: What costume designer was a ﬁlm director ’s Canadian female parent

Original: Answer: SELECT DISTINCT ?
Revised: Answer: SELECT DISTINCT ?

Original: x0 WHERE { ?
Revised: x0 WHERE { ?

Original: x0 a costume designer . ?
Revised: x0 a costume designer . ?

Original: x0 a ﬁlm director . ?
Revised: x0 a ﬁlm director . ?

Original: x has gender female . ?
Revised: x has gender female . ?

Original: x0 has nationality Canadian . ?
Revised: x0 has nationality Canadian . ?

Original: x0 parent of ? x1 } }
Revised: x0 parent of ?x1 } }

Original: Golden : SELECT DISTINCT ?
Revised: Golden: SELECT DISTINCT ?

Original: x0 WHERE { 
Revised: x0 WHERE { ?

Original: x0 a costume designer . ?
Revised: x0 a costume designer . ?

Original: x0 parent of ?x1 . ?
Revised: x0 parent of ?x1 . ?

Original: x has gender female . ?
Revised: x has gender female . ?

Original: x0 has nationality Canadian . ?
Revised: x0 has nationality Canadian . ?

Original: x1 a ﬁlm director }
Revised: x1 a ﬁlm director }

Original: E. F AIR C OMPARISON AGAINST O THER P ROMPTING T ECHNIQUES
Revised: Question: Did a male actor whose spouse married M2 play M Answer: SELECT count(*) WHERE { ?x0 portrayed M0 . ? x0 married to ?x1 . ? x1 married to M2 .

Revised: x1 a actor . ?

Revised: x1 has gender male }

Revised: Golden: SELECT count(*) WHERE { ?x0 portrayed M0 . ?

Revised: x0 has gender male . ?

Revised: x0 married to ?x1 . ?

Original:  In the comparison in Figure 5, vanilla few-shot and chain-of-thought prompting have an advantage over least-to-most prompting because we sample multiple exemplar lists ( n = 4 ) and multiple outputs per list ( s = 4 ) using temperature-based decoding.
Revised: x1 a actor . ? x1 married to M2 } E. F AIR C OMPARISON AGAINST O THER P ROMPTING T ECHNIQUES In the comparison in Figure 5, vanilla few-shot and chain-of-thought prompting have an advantage over least-to-most prompting because we sample multiple exemplar lists ( n = 4 ) and multiple outputs per list ( s = 4 ) using temperature-based decoding.

Original: This yields n · s = 16 outputs per input, which are aggregated using self-consistency.
Revised: This yields n · s = 16 outputs per input, which are aggregated using self-consistency.

Original: When using n = 1 and s = 1 with greedy decoding, the comparison between these prompting techniques and dynamic least-to-most is more fair, and the beneﬁts of dynamic least-to-most are more prominent.
Revised: When using n = 1 and s = 1 with greedy decoding, the comparison between these prompting techniques and dynamic least-to-most is more fair, and the beneﬁts of dynamic least-to-most are more prominent.

Original: Chain-of-thought achieves 75.4% accuracy (down from 87.2%), and vanilla few-shot achieves 69.8% accuracy (down from 80.8%).
Revised: Chain-of-thought achieves 75.4% accuracy (down from 87.2%), and vanilla few-shot achieves 69.8% accuracy (down from 80.8%).

Original: Dynamic least-to-most substantially outperforms both of these, and achieves 94.4% on the same 500-sentence subset of MCD1 validation data.
Revised: Dynamic least-to-most substantially outperforms both of these without using self-consistency, and achieves 94.6% on the same 500-sentence subset of MCD1 validation data.

Original: F R EPRODUCIBILITY O UTLINE
Revised: F R EPRODUCIBILITY S UMMARY

Original: At all points in this work, we aim to make our methods and experiments easily reproducible, and we hope our ﬁndings will have impact in part through others using the same or similar methods for new tasks.
Revised: At all points in this work, we aim to make our methods and experiments easily reproducible, and we hope our ﬁndings will have impact in part through others using the same or similar methods for new tasks.

Original: Here we summarize critical components of our work and where their relevant details are described:
Revised: Here we summarize critical components of our work and where their relevant details are described:

Original: • Main Prompts: In lieu of code, we provide the exact prompts that we executed to obtain model predictions.
Revised: • Main Prompts: In lieu of code, we provide the exact prompts that we executed to obtain model predictions.

Original: Prompts for CFQ: syntactic parsing (Appendix A.1), dynamic leastto-most (Appendix A.3).
Revised: Prompts for CFQ: syntactic parsing (Appendix A.1), dynamic leastto-most (Appendix A.3).

Original: Prompts for COGS: syntactic parsing (Appendix B.1), dynamic least-to-most (Appendix B.3).
Revised: Prompts for COGS: syntactic parsing (Appendix B.1), dynamic least-to-most (Appendix B.3).

Original: Exemplars are chosen using the methods described in Section 3.2, and Appendix A.2 (for CFQ) and Appendix B.2 (for COGS).
Revised: Exemplars are chosen using the methods described in Section 3.2, and Appendix A.2 (for CFQ) and Appendix B.2 (for COGS).

Original: The prompts include the exemplars found from a concrete input.
Revised: The prompts include the exemplars found from a concrete input.

Original: • Additional Prompts: We only use vanilla few-shot and chain-of-thought to compare against dynamic least-to-most on CFQ.
Revised: • Additional Prompts: We only use vanilla few-shot and chain-of-thought to compare against dynamic least-to-most on CFQ.

Original: The chain-of-thought prompt used to bootstrap rationale is in Appendix C.1, and the one used for evaluation data is in Appendix C.2.
Revised: The chain-of-thought prompt used to bootstrap rationale is in Appendix C.1, and the one used for evaluation data is in Appendix C.2.

Original: The prompt includes the exemplars found from a concrete input.
Revised: The prompt includes the exemplars found from a concrete input.

Original: The vanilla few-shot prompt is similar, but all examples use the vanilla input/output format.
Revised: The vanilla few-shot prompt is similar, but all examples use the vanilla input/output format.

Original: Exemplars are chosen using the methods described in Section 4.2.
Revised: Exemplars are chosen using the methods described in Section 4.2.

Original: • Exemplar Pool: We will provide a link to the exact sentences we included in the exemplar pools in the deanonymized version of our work.

Original: • Dataset Preparation and Evaluation: We performed both pre-processing and a postprocessing output normalization step in order to make semantic parsing with freebase identiﬁers more feasible for prompting.
Revised: • Dataset Preparation and Evaluation: We performed both pre-processing and a postprocessing output normalization step in order to make semantic parsing with freebase identiﬁers more feasible for prompting.

Original: Described in Section 5.1, Appendix D.1.1, D.1.2, D.2.
Revised: Described in Section 5.1, Appendix D.1.1, D.1.2, D.2.

Revised: • Exemplar Pool: Methodology for constructing exemplar pools is described in Section 3.2.

Original: • Hyperparameters: We didn’t require any extensive hyperparameter search.
Revised: • Hyperparameters: We didn’t require any extensive hyperparameter search.

Original: The hyperparameters we used are described in Section 5.2 and Appendix D.3.
Revised: The hyperparameters we used are described in Section 5.2 and Appendix D.3.