Paper: Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation (ICLR-2024)

Reviewer comment (Experiments):
One key weakness of the proposed decoding method is that the generations happen for different bullet points of a reply are independent from each other, limiting its use to answers that require chain-of-thoughts. Hence, it is applicable to specific type of replies and is sensitive to the instruction following abilities of the model that is being applied to, which limits its generality.

Author response:
**W1, about general applicability** One key weakness of the proposed decoding method is that the generations happen for different bullet points of a reply are independent from each other, limiting its use to answers that require chain-of-thoughts.

**A**: Yes, the current SoT method is suitable for only a range of questions. Therefore, we designed a simple extension SoT-R to trigger SoT only for suitable questions by a router module. If we'd like to produce the answer for a broader range of questions in some "structured way" for acceleration and quality, we need to extend the method, for example, by combining the prompting-for-efficiency idea and other structuring prompting techniques to get a better pipeline. Besides prompting, the LLMs themselves might need some adaptation to enable this too. See more discussions of these directions in the global response and Section 6.

Reviewer comment (Evaluation):
The proposed method requires prompting the model with a large number of additional tokens to generate continuations for each different bullet point per reply; the number of tokens increases linearly with the number of bullet points. This has both memory/inference cost and latency implications that are not clearly exposed when discussing the main results.

Author response:
**W2, about the overhead**: The proposed method requires prompting the model with a large number of additional tokens to generate continuations for each different bullet point per reply; the number of tokens increases linearly with the number of bullet points. This has both memory/inference cost and latency implications that are not clearly exposed when discussing the main results.

**Q5**: From 3.1.1, it is not entirely clear what is the overhead in terms of memory and or latency introduced by the additional prompting required especially for replies that require expansion of many bullet points with and without preflls. Can the authors quantify this? It would show a more clear picture of the tradeoffs involved.

**A to W2&Q5**: Yes, due to the long prompt, the current SoT introduces additional computation in the prefilling stage.

**In terms of inference latency**, our reported results have already taken the additional prefilling overhead into account. Figure 12 and Figure 13 have shown the latency breakdown of the two phases (prefill and decode). One can see that the prefilling latency is very small compared with the decoding latency.

**In terms of the additional memory footprint brought by batched inference**, Figure 10(c) showed a profiling result. It shows that the peak memory overhead due to the increasing size of the KV cache and activation grows at a slow pace as batch size B increases. Besides the profiling results, following your suggestion, we report the actual peak memory overhead in Appendix K.2 in the revision. The results show that in all models and categories, the peak memory overhead of both the prefilling and decoding phases is quite small (<1.1x).

In Sec. 6 and Appendix H, we discussed the influences caused by the additional computation in different scenarios. 
* When the original concurrent query number is low (i.e., the workload is memory-access-bounded), using SoT can help improve latency without sacrificing throughput, even though it introduces additional computation, as it improves computation utilization.
* When the original query concurrency is already high (i.e., the workload is already computation-bounded), using SoT trades off the throughput for latency.

Such overhead can be mitigated by the following approaches. (1) Only prefilling the common prefix of the multiple point expansion prompts once, and using the KV cache for batch decoding. (2) Triggering SoT based on the current system workloads. (3) Using shorter prompts to activate the SoT mode. Actually, in our experiments, we found that one can correctly trigger SoT with a much shorter skeleton prompt (without the demonstrations) for strong models such as GPT-4, GPT-3.5, and LLaMA2. (In our experiment, we used one unified prompt for all open-source models without applying this trick.) In addition, applying prompt compression techniques to stronger models could also largely reduce the SoT prompt length, thus lowering the computation overhead.

Reviewer comment (Evaluation):
When the method is applied without the backoff strategy its quality benefit is not very clear as the improvement is observed in about half the models and categories according to FastChat and LLMZoo evaluations combined.

Author response:
**W3**: When the method is applied without the backoff strategy its quality benefit is not very clear as the improvement is observed in about half the models and categories according to FastChat and LLMZoo evaluations combined.

**A**: We are not sure if we understood this question correctly. Did you mean that "the quality improvement is *only* observed in about half the models and categories, and *not* the other half"? If this is the question, SoT was originally designed for efficiency improvements. So, as long as the performance does not degrade much, we are satisfied. We regard quality improvements on half the models and categories as a bonus. If we misunderstood this question, please do not hesitate to correct us.

> Regarding W3 my point is that on half of them there is improvement but on the other half it leads to degradation; it would be nice if improvement doesn't come at the expense of other tasks. The level of degradation is substantial in my view.

Oh, we got your point. Yes, the current SoT is not suitable for some question types. Currently, we went with a "router" solution, SoT-R, that simply doesn't use SoT for these questions. Making answer planning work for broader question types indeed needs future improvements.

Reviewer comment (Experiments):
There is no direct comparison to alternative methods that have been proposed for speeding up decoding in autoregressive models such as quantization, batch inference, and speculative decoding. For instance, a simple baseline would be to perform quantization and then apply multi-stage prompting to improve quality for certain categories.

Author response:
**W4**: There is no direct comparison to alternative methods that have been proposed for speeding up decoding in autoregressive models such as quantization, batch inference, and speculative decoding. For instance, a simple baseline would be to perform quantization and then apply multi-stage prompting to improve quality for certain categories.

**A**: Thanks for this valuable suggestion. We follow the suggestion to experiment with model quantization. Model quantization is a model-level optimization approach to accelerate the LLM inference process, which is orthogonal to our SoT and thus can be combined with SoT to achieve a higher speed-up. The following table shows the speed-ups of combining SoT-R and quantization w.r.t. normal generation without quantization. The numbers between brackets are the SoT speed-ups without quantization. More detailed results (e.g., speed-up on different categories, speed-up w.r.t. the quantized model) can be found in Appendix J in our revision.
| Router | OpenChat-13B | Vicuna-7B V1.3 | UltraLM-13B | LLaMA2-Chat-13B | LLaMA2-Chat-7B | Vicuna-7B V1.1 | Vicuna-13B V1.3 | Vicuna-33B V1.3 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompting router | 1.90x (1.66x) | 1.95x (1.39x) | 1.82x (1.51x) | 1.93x (1.70x) | 1.84x (1.67x) | 1.71x (1.55x) | 1.63x (1.32x) | 1.59x (1.62x) |
| Trained router | 2.07x (1.82x) | 1.98x (1.42x) | 1.92x (1.59x) | 1.91x (1.70x) | 1.92x (1.69x) | 1.76x | 1.64x (1.59x) | 1.54x (1.57x) |


> The additional results with quantization show that the proposed method is competitive but performs worse than quantization. I am still not convinced in which case one would prefer SoT in place of quantization or other efficient methods, especially if some of the evaluation categories are going to get a hit in performance.

Maybe a point we didn't make clear enough is that we didn't compare SoT with quantization. As quantization and SoT are orthogonal methods, we apply SoT generation to quantized models and report (1) the overall speed-up w.r.t. *the FP model with normal generation* (newly added Appendix J.2 and the table in the response); and (2) the speed-up w.r.t. *the quantized model with normal generation* (newly added Appendix J.1). 

Another thing is that we already use SoT-R in all these experiments, i.e., not triggering SoT for unsuitable questions, to avoid hurting the answer quality.

That is to say, the table in the response reports the overall speed-ups of SoT + quantization, where the SoT-alone speed-ups are reported in parentheses. For example, the "1.84x (1.67x)" for LLaMA2-Chat-7B in the table means that SoT-R achieves a 1.67x speed-up , and quantization brings an additional 1.1x speed-up, which multiplies up to 1.84x.

Reviewer comment (Evaluation):
From 3.1.1, it is not entirely clear what is the overhead in terms of memory and or latency introduced by the additional prompting required especially for replies that require expansion of many bullet points with and without preflls. Can the authors quantify this? It would show a more clear picture of the tradeoffs involved.

Author response:
**Q5**: From 3.1.1, it is not entirely clear what is the overhead in terms of memory and or latency introduced by the additional prompting required especially for replies that require expansion of many bullet points with and without preflls. Can the authors quantify this? It would show a more clear picture of the tradeoffs involved.

**A to W2&Q5**: Yes, due to the long prompt, the current SoT introduces additional computation in the prefilling stage.

**In terms of inference latency**, our reported results have already taken the additional prefilling overhead into account. Figure 12 and Figure 13 have shown the latency breakdown of the two phases (prefill and decode). One can see that the prefilling latency is very small compared with the decoding latency.

**In terms of the additional memory footprint brought by batched inference**, Figure 10(c) showed a profiling result. It shows that the peak memory overhead due to the increasing size of the KV cache and activation grows at a slow pace as batch size B increases. Besides the profiling results, following your suggestion, we report the actual peak memory overhead in Appendix K.2 in the revision. The results show that in all models and categories, the peak memory overhead of both the prefilling and decoding phases is quite small (<1.1x).

In Sec. 6 and Appendix H, we discussed the influences caused by the additional computation in different scenarios. 
* When the original concurrent query number is low (i.e., the workload is memory-access-bounded), using SoT can help improve latency without sacrificing throughput, even though it introduces additional computation, as it improves computation utilization.
* When the original query concurrency is already high (i.e., the workload is already computation-bounded), using SoT trades off the throughput for latency.

Such overhead can be mitigated by the following approaches. (1) Only prefilling the common prefix of the multiple point expansion prompts once, and using the KV cache for batch decoding. (2) Triggering SoT based on the current system workloads. (3) Using shorter prompts to activate the SoT mode. Actually, in our experiments, we found that one can correctly trigger SoT with a much shorter skeleton prompt (without the demonstrations) for strong models such as GPT-4, GPT-3.5, and LLaMA2. (In our experiment, we used one unified prompt for all open-source models without applying this trick.) In addition, applying prompt compression techniques to stronger models could also largely reduce the SoT prompt length, thus lowering the computation overhead.

Reviewer comment (Experiments):
What is the speed up of the proposed method for different prompt sizes, especially the large ones? It would be useful to show the improvement for replies that require an increasing number of bullet points to answer.

Author response:
**Q6**: What is the speed up of the proposed method for different prompt sizes, especially the large ones? It would be useful to show the improvement for replies that require an increasing number of bullet points to answer.

**A**: We show the speed-ups under different numbers of bullet points in Appendix K.3 in our revision. The speed-up does not increase with the number of bullet points. The reason is that there exist other variables that would affect the speed-up. For example, for questions that yield more bullet points, SoT might give out a much longer response compared with the normal generation process. This will lower the speed-up.

Reviewer comment (Experiments):
Have you tried prompting the model to generate arbitrary partial answer segments other than bullet points to perform continuations on them in parallel? That could potentially make the method applicable beyond answers that require specific bullet point structure.

Author response:
**Q8**: Have you tried prompting the model to generate arbitrary partial answer segments other than bullet points to perform continuations on them in parallel? That could potentially make the method applicable beyond answers that require a specific bullet point structure.

**A**: Thanks for proposing this interesting idea. We did not try this. We think it is a valid pathway to improve general applicability. This could also help with the coherence, if the LLM can understand the requirement that the output of one segment will be followed by the given partial segment in the skeleton.

Reviewer comment (Evaluation):
For the quality results in section 3.2.1, what is the average absolute scores achieved per category and across all categories overall when showing quality per model?

Author response:
**Q9**: For the quality results in section 3.2.1, what is the average absolute scores achieved per category and across all categories overall when showing quality per model? 

**A**: Our original submission gave a per-category-model heatmap on answer quality in Figure 27, which is evaluated by the GPT-3.5 judge. To further answer your question, we added an answer quality heatmap evaluated by the GPT-4 judge to the revision at Figure 22.

Reviewer comment (Novelty):
The main contribution of this paper is on the experiments part rather than the idea. In other words, the idea of this article is not insightful because it is very straightforward. Although I believe that skeleton can be generated in different ways, such as through a tree, rather than directly prompting the LLM.

Author response:
Thanks for your recognition of our evaluation! We'd like to share our opinion on the idea: Although the current method design is straightforward, the idea behind SoT -- **prompting an explicit answer planning for efficiency** --  is novel and worth future study, which is enabled by the fact that LLM has increasingly stronger capabilities including instruction following, planning, etc. The idea is to do explicit (by language) answer planning to help make better use of hardware resources (i.e., improving computation utilization) and thus improve efficiency. We think explicitly making the generation workload parallelizable might become a more relevant efficiency-oriented methodology as the LLM gets more widely adopted and stronger.

Reviewer comment (Experiments):
If the purpose of this paper is acceleration, then SoT is not universally applicable. The experiments in the article demonstrate it can lead to a decline in the quality of tasks such as code generation. More importantly, because the length (autoregressive steps) of each point the LLM needs to generate varies, the actual acceleration ratio in online-serving may differ from the test scenarios presented in the article, e.g., batch size=1 or batch size > 1.

Author response:
Yes, the current SoT method is not suitable for all questions. Therefore, we propose a simple extension SoT-R to only trigger SoT on suitable questions. If we'd like to produce the answer for a broader range of questions in some "structured way" for acceleration and quality, we need to extend the method, for example, by combining the prompting-for-efficiency idea and other prompting techniques (e.g., CoT) to get better quality. Besides prompting, the LLMs themselves might need some adaptation to enable this too. See more discussions of these directions in the global response and Section 6. 

Thanks for pointing out this valuable concern. Speed-up controllability of this data-level technique is a valid concern that we did not cover in our original paper. Unlike most model-level (quantization, static sparse, etc.) and system-level acceleration (fusing, offloading, etc.) techniques, SoT's acceleration ratio depends more on the SoT prompt, the model, and the question. We add a limitation discussion to the "Data-centric efficiency optimization" paragraph in Sec. 6.

As for the case with different batch sizes, Section 6 has discussed the overhead of SoT in two scenarios: memory-access bounded (with a moderate batch size) and computation-bounded (with a large batch size>>1).

Reviewer comment (Presentation):
The content relies too much on supplementary materials. The manuscript is preferred to be self-contained.

Author response:
Many thanks for the suggestion. Given the extensive results on hardware profiling, efficiency analysis, answer examples, failure case analysis, and more, it is difficult to include everything in 9 pages. We tried hard to make sure that the discussion and experiments in the main text were sufficient for readers to understand the key points. Although the main text indeed refers to the appendix in many places (where the appendix provides more supporting experiments and detailed discussions), these references mainly serve as a convenient pointer to the corresponding appendix section. 

That being said, if the reviewer thinks any of the contents in the appendix should be elevated to the main text, please let us know and we will make the changes.

Reviewer comment (Evaluation):
As mentioned above, as far as I understood this algorithm may use more resource than normal sequential generation. Do you have numbers about overall resource time used for a single generation? i.e. for API-based experiments it would be (skeleton generation time + sum of all parallel execution times) and for batch-based experiments it would be (skeleton generation time + (batch generation time * sequences per minibatch * number of minibatches)).

Author response:
You are right that the SoT prompts introduce computation overhead. Will this overhead actually cause throughput degradation? We gave the computation overhead results in Appendix H, and gave a short analysis in Section 6 by looking at two types of scenarios (with or without a large number of concurrent queries).

The discussion said that when the original concurrent query number is low (i.e., the workload is memory-access-bounded), using SoT can help improve latency without sacrificing throughput, even though it introduces additional computation, as it improves the computation utilization. When the original query concurrency is already high (i.e., the workload is already computation-bounded), using SoT might hurt the throughput, and does not help so much with the latency either.

Such overhead can be mitigated by the following approaches.
1. Only prefilling the common prefix of the multiple point expansion prompts once, and using the KV cache for batch decoding.
2. Triggering SoT based on the current system workloads.
3. Using shorter prompts to activate the SoT mode. Actually, in our experiments, we found that one can correctly trigger SoT with a much shorter skeleton prompt (without the demonstrations) for strong models such as GPT-4, GPT-3.5, and LLaMA2. (In our experiment, we used one unified prompt for all open-source models without applying this trick.) In addition, applying prompt compression techniques to stronger models could also largely reduce the SoT prompt length, thus lowering the computation overhead.

Reviewer comment (Evaluation):
For router experiments, do you have numbers about the ratio of each instance being classified as "capable of being generated via SoT"? For example in Figure 7, StableVicuna-13B with SoT-R experiments achieved ~1.0x speedup, which means that most of them would be generated via normal sequential generation.

Author response:
Thanks for this question. On Vicuna-80, on the counterfactual, common-sense, knowledge, and generic categories, the number of problems that are suitable for SoT mode is very high (>7/10).  While on the other categories, the number of suitable examples for SoT is quite small, especially on the math and coding categories. More detailed statistics can be found in Appendix K.1 in the revision.

The reason that StableVicuna cannot achieve good speed-ups is not relevant to the number of questions suitable for SoT, but lies in the shortage of StableVicuna in understanding the point expansion task (it generates long answers in the point-expanding stage regardless of the short answer requirement in the prompt). We can see from Figure 2(a) that plain SoT can only achieve an on-average 1.1x speed up on StableVicuna.

Reviewer comment (Evaluation):
Is naturalness of generated text included in metrics of automatic quality evaluation? The prompt used feels not very natural for me; since as far as I understood each sentence should always start with the "skeleton point," according to Prompt 2 ("... do not continue with other points! [Assistant:] {point index}. {point skeleton}").

Author response:
Thanks for this great question. You are right that each point starts with a partial answer. We used the partial answer to avoid useless outputs from LLMs such as "Sure, I'll help you to expand the 2nd point", so that we can directly concatenate the point responses as the final response. For some types of questions, this might result in unnatural responses. We have included a discussion on this pattern in Appendix I.1.2 in the original manuscript.

> Writing problems usually require a coherent passage without embedded skeleton points, whereas our current SoT pipeline (§ 2) concatenates skeleton points as part of the answer In such cases, simply removing the skeleton points would greatly improve the answer quality. To make SoT more general, one future direction is to let the LLM itself decide whether the point index and point skeleton should be included to make the final answer more natural and fluent.

Regarding the metric definitions: Although the two sets of metrics (FastChat, LLMZoo) don't have exactly the word "naturalness" in their evaluation prompts, the definition of the "Coherence" and "Immersion" metrics are actually related. And indeed, our evaluation in Figure 6 found that the current SoT solution hurt immersion and coherence.

In our early experiment, we made some attempts to mitigate this issue: we asked the model itself to judge if the point skeleton should exist in the final response, and exclude the partial answer part in the final response if the judgment is no. We found that strong models such as GPT-3.5 can give judgments that match manual justifications, but unfortunately, the open-source models cannot give good judgments on this aggregation scheme. As the attempt is not thorough and quantitive, we did not include it in the manuscript and deferred it to future work.

Reviewer comment (Theory):
The comparison with human thinking is merely intuitive and not founded by any references, hence speculation. I do not think this makes the method any more attractive, and would prefer to see a discussion with scientific evidence, if this should be a benefit of this method. It is clear that the inspiration comes from how humans draft a text or the like - but this does not result in humans and LLMs functioning in the same way. The phrasing that LLMs are made “thinking in the same way” just adds fuel to the pseudoscientific discussions around whether LLMs are “thinking” or not (especially the phrase “pushing LLMs to think more like a human”). In addition, there is no connection between the argument of efficiency and human thinking - is it for efficiency that humans derive a skeleton first?

Author response:
**Q1**: The comparison with human thinking is merely intuitive. It is clear that the inspiration comes from how humans draft a text or the like - but this does not result in humans and LLMs functioning in the same way. The phrasing that LLMs are made “thinking in the same way” just adds fuel to the pseudoscientific discussions around whether LLMs are “thinking” or not (especially the phrase “pushing LLMs to think more like a human”). In addition, there is no connection between the argument of efficiency and human thinking - is it for efficiency that humans derive a skeleton first?

**A**: Thanks for this valuable suggestion! We used these descriptions to give readers an intuitive understanding of the idea, but we agree with the reviewer that they are not well-defined and are not suitable for papers. Upon your suggestion, we also realize that these sentences could mislead the readers to think if we are claiming there is a connection between the argument of efficiency and human thinking (which is not). We revised the paper according to your suggestion. 

Except for these two sentences, we think the statements and logic in our introduction and discussion section are solid. The main logic we want to express in the introduction is: (1) The fully sequential decoding is one of the key reasons that cause slow generation. Is this fully sequential way necessary? (2) Humans usually write the answer in a structured, non-sequential way. For example, we usually have a skeleton of the idea in our minds or on paper first. This intuition has our back to question the necessity of fully sequential decoding. (3) Inspired by this, we propose SoT as an attempt towards "data-level organization for efficiency".

Another place we relate SoT to human thinking is in the discussion in Sec. 6: We first relate SoT’s potential of enhancing answer quality to other recent research like CoT, ToT, and ReAct: These work shows that "explicitly articulating the thought process (e.g., CoT) or structure (e.g., SoT) in language can elicit high-quality answers". Then, we claim that "these findings resemble human thinking: rather than relying solely on first intuition or purely sequential thinking, we often document step-by-step reasoning or thought organization to attain high-quality answers".

Reviewer comment (Experiments):
Missing comparison to previous orthogonal methods proposed for speedups: Where on Figure 1b would they be located? This is necessary to understand the impact of this work in context of the previously proposed and adopted solutions.

Author response:
**Q4**: Missing comparison to previous orthogonal methods proposed for speedups: This is necessary to understand the impact of this work in context of the previously proposed and adopted solutions.

**A**: Thanks for this valuable suggestion. Here, we add an experiment using the widely used quantization optimization. Model quantization is a model-level optimization approach, which is orthogonal to SoT and thus can be combined with SoT to achieve a higher speed-up. The following table shows the speed-ups of combining SoT-R and quantization w.r.t. normal generation without quantization for different models. The numbers between brackets are the SoT-R speed-ups without quantization. More detailed results (e.g., speed-up on different categories, speed-up w.r.t. the quantized model) can be found in Appendix J in our revision.
| Router | OpenChat-13B | Vicuna-7B V1.3 | UltraLM-13B | LLaMA2-Chat-13B | LLaMA2-Chat-7B | Vicuna-7B V1.1 | Vicuna-13B V1.3 | Vicuna-33B V1.3 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompting router | 1.90x (1.66x) | 1.95x (1.39x) | 1.82x (1.51x) | 1.93x (1.70x) | 1.84x (1.67x) | 1.71x (1.55x) | 1.63x (1.32x) | 1.59x (1.62x) |
| Trained router | 2.07x (1.82x) | 1.98x (1.42x) | 1.92x (1.59x) | 1.91x (1.70x) | 1.92x (1.69x) | 1.76x | 1.64x (1.59x) | 1.54x (1.57x) |

Reviewer comment (Writing):
The title is misleading and alludes to a new ability of LLMs (“LLMs can do”): The actual LLM decoding process is technically still sequential and there is no modification to the LLMs decoding loop (not to be confused with non-autoregressive methods), it’s just that the model is prompted to provide a decomposable answer to be elaborated in parallel. A more concise (but probably not great for selling the paper) reformulation of the title would be “LLMs can be prompted to provide decomposable answers”.

Author response:
**Q2**: The title is misleading and alludes to a new ability of LLMs (“LLMs can do”): The actual LLM decoding process is technically still sequential and there is no modification to the LLMs decoding loop (not to be confused with non-autoregressive methods).

**A**: We understand and carefully discussed your concern. As we want to convey the key messages including "prompting", "answer planning", "parallel generation", and "efficiency", we are considering a new title "Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation", which contains the messages and avoids the confusion you brought up. At the current review stage, we are not allowed to modify the title. We will make the change in the final revision.  If the reviewer has more comments on this new title, please let us know!

Reviewer comment (Presentation):
The limitations of the method are not clearly articulated in the introduction: until Section 3 it is not clear that this method is limited to specific types of questions and that there is prompt-specific sensitivity. It might be obvious to the authors, but for positioning the paper in the right scope, this should be mentioned in the introduction.

Author response:
**Q3**: The limitations of the method are not clearly articulated in the introduction: until Section 3 it is not clear that this method is limited to specific types of questions and that there is prompt-specific sensitivity. It might be obvious to the authors, but for positioning the paper in the right scope, this should be mentioned in the introduction.

**A**: Thanks for this suggestion. We add an emphasis to the introduction that SoT is not suitable for some types of questions, which motivates us to develop the SoT-R extension. About prompt-specific sensitivity, we mention it in the "Data-centric efficiency optimization" paragraph in Sec. 6 in the revision.

Reviewer comment (Experiments):
An analysis of the effect of added length is missing: Since skeleton-of-thought prompting produces longer answers than the usual prompting (3.1.1), there might be a bias of certain evaluations to prefer more wordy answers even if they have the same contents. Similarly, it remains an open question how the length part (“3-5 words”/”3-10 points”/… see prompt examples) of the prompt should be optimally configured.

Author response:
**Q5**: An analysis of the effect of added length is missing: Since skeleton-of-thought prompting produces longer answers than the usual prompting (3.1.1), there might be a bias of certain evaluations to prefer more wordy answers even if they have the same contents. Similarly, it remains an open question how the length part of the prompt should be optimally configured.

**A**: 
**Answer lengths might bias the evaluation.** Thanks for raising this valuable concern. Following your suggestion, we added an experiment: we let the LLM sequentially output a longer answer by using the prompt template "Please give a long answer for the following question: {request}" so that the answer length is comparable to the one from SoT, and then compare the lengthy sequential answer and the SoT answer. The figures are shown in Figure 26-27 in the revision (Appendix I.3). We can see that SoT does not cause severe quality degradation with roughly the same answer length.

**How the length part of the prompt should be optimally configured.** These words and concrete numbers were crafted on Vicuna-7B V1.1 manually and directly used for other open-source models. That being said, we did observe some differences in how models obey the constraints described by these words (e.g., see the total lengths of answers in Figure 11). How different models can obey these number-of-words or number-of-points commands is indeed an interesting direction to study on its own. We note that a recent paper [1] has studied this topic, and it will be interesting to incorporate their insights to improve SoT. 

[1] Sun, Jiao, et al. "Evaluating Large Language Models on Controlled Generation Tasks." arXiv preprint arXiv:2310.14542 (2023).

Reviewer comment (Evaluation):
How would Figure 1b look like for the LLMZoo evaluation? This kind of plot is answering the question for the speed-quality trade-off at one glance, so it would be great to have as well for LLMZoo. It looks like the model with the largest wins (StableVicuna-13B) also has the lowest speed-up for LLMZoo (3.2.2), so it would be nice to see if this is a trend more than for FastChat.

Author response:
**Q6**: How would Figure 1b look like for the LLMZoo evaluation? This kind of plot is answering the question for the speed-quality trade-off at one glance, so it would be great to have as well for LLMZoo.

**A**: We have added the speed-quality trade-off plot to Figure 25 in the revision. LLMZoo and FastChat give different net win rate results. Nevertheless, the general message (which categories and models are more suitable for SoT) is consistent.

Reviewer comment (Presentation):
I do not think that Figure 3 is very expressive, as it is an accumulated result across all models, which in turn (3.2.2) vary largely. Perhaps this could be replaced with a deeper analysis of the loss cases? This would allow the reader to get a better idea of the potential risks.

Author response:
**Q7, writing suggestion**: I do not think that Figure 3 is very expressive, as it is an accumulated result across all models, which in turn (3.2.2) vary largely. Perhaps this could be replaced with a deeper analysis of the lose cases? This would allow the reader to get a better idea of the potential risks.

**A**: We agree with the reviewer: the fact that different models have large quality variances makes Figure 3 less informative, and qualitative knowledge should be discussed on a per-model basis. Nevertheless, when we wrote the draft, we finally decided to put Figure 3 so that it is easy for readers to grasp the overall answer quality of SoT without the need to read more complicated discussions and figures later on. A full analysis of the failure case is discussed in the main paper with examples attached in the appendix for readers to look up.

Reviewer comment (Experiments):
The method reminds me of the map-reduce paradigm, but the reduce step is a simple concatenation rather than a joint processing of the whole list of answers. How much would a final “summarize” decoding step add to the quality and remove from the efficiency? I could imagine this increasing the coherence of the answers.

Author response:
**Q8, about method extension**: How much would a final “summarize” decoding step add to the quality and remove from the efficiency? I could imagine this increasing the coherence of the answers.

**A**: Thanks for this valuable idea. We had some considerations and attempts in this direction -- using another aggregation method other than simple concatenation. (1) Post-processing relying on a language model (i.e., a summarization decoding step): Using the autoregressive decoding method in this summarization step will incur significant latency. As this work's main objective is to maximize hardware utilization and accelerate generation, we did not experiment with this method. An optimization to this idea could be to use a smaller model for summarization, which is feasible since the main structure and facts are already there. (2) Prompting the LLM to choose from a few manually designed aggregation choices, for example, whether or not to keep the "1. 2. 3." indexes in the final response. We find that GPT-3.5 can give reasonable judgment according to our human justification, but open-source models cannot give reasonable choices yet. We did not conduct large-scale experiments at this point and defer this to future work.

Reviewer comment (Evaluation):
For 4.3.1 the most relevant question seems to be which ratio of questions the router decides to process with skeleton-of-thought. This directly affects efficiency, since speedups can only possibly be obtained if the skeleton-of-thought route is chosen. Quality and efficiency both seem to be a predictable combination of normal decoding and skeleton-of-thought decoding quality and efficiency once the ratio is known. It would be interesting to report this ratio by category in order to evaluate if it agrees with the categories that were identified to work well for skeleton-of-thought in 3.1.2.

Author response:
**Q9, SoT-R statistics**: For 4.3.1 the most relevant question seems to be which ratio of questions the router decides to process with skeleton-of-thought. It would be interesting to report this ratio by category in order to evaluate if it agrees with the categories that were identified to work well for skeleton-of-thought in 3.1.2.

**A**: Thanks for this suggestion. On the Vicuna-80 dataset, on the counterfactual, common-sense, knowledge, and generic categories, the number of questions that are suitable for SoT mode is very high (>7/10) based on our routers. While on the other categories (especially math and coding), the number of suitable questions for SoT is quite small. The results show agreement with the results in 3.1.2. We've added the statistics to Appendix K.1 in the revision.

Reviewer comment (Experiments):
There are requirements for tasks, and multiple points of tasks must be able to be generated at ordinary times without affecting each other.

Author response:
Thanks for this question! You are right that the current SoT method is mainly suitable for questions that require long answers whose structure can be planned ahead and contain multiple points that are relatively independent, while not suitable for questions that require step-by-step reasoning.

We want to note that the the three datasets used in this paper (Vicuna, WizardLM, LIMA) *already contain both types of questions that are suitable and not suitable for SoT*. In fact, a large fraction of the questions in these datasets are not suitable for SoT (see Appendix K.1). That is why we see in Figure 8 that SoT compromises the answer quality in some question categories. Therefore, we designed a simple extension SoT-R to trigger SoT only for suitable questions by a router module, and we see that SoT-R does not compromise answer quality in all question categories in these datasets.

Beyond SoT-R, there are many other potential pathways to expand SoT's applicability. For example, we discussed the outlook of extending "skeleton-of-thoughts" to "graph-of-thoughts", and "ahead-of-time planning" to "just-in-time planning (a.k.a. dynamic graph-of-thoughts)". We can not fully explore all these potentials in this single paper, but all of them are worth exploring in the future.

Reviewer comment (Experiments):
This method of generation may compromise the quality of the results generated.

Author response:
Thanks for this question! You are right that the current SoT method is mainly suitable for questions that require long answers whose structure can be planned ahead and contain multiple points that are relatively independent, while not suitable for questions that require step-by-step reasoning.

We want to note that the the three datasets used in this paper (Vicuna, WizardLM, LIMA) *already contain both types of questions that are suitable and not suitable for SoT*. In fact, a large fraction of the questions in these datasets are not suitable for SoT (see Appendix K.1). That is why we see in Figure 8 that SoT compromises the answer quality in some question categories. Therefore, we designed a simple extension SoT-R to trigger SoT only for suitable questions by a router module, and we see that SoT-R does not compromise answer quality in all question categories in these datasets.

Beyond SoT-R, there are many other potential pathways to expand SoT's applicability. For example, we discussed the outlook of extending "skeleton-of-thoughts" to "graph-of-thoughts", and "ahead-of-time planning" to "just-in-time planning (a.k.a. dynamic graph-of-thoughts)". We can not fully explore all these potentials in this single paper, but all of them are worth exploring in the future.

Reviewer comment (Evaluation):
Is there anything wrong with the content generated this way? Such as repeating generation or generating blank.

Author response:
We did not observe issues like repeating generation or generating blank. Nevertheless, applying SoT on unsuitable questions indeed results in unreasonable answer skeleton and wrong reasoning (for math and fermi), and empty talk about the strategy without giving the actual code (for code). These cases are discussed in Appendix I.1.

Reviewer comment (Novelty):
The core idea is quite straightforward: first list the key points, then elaborate each point. I am afraid that similar ideas have been extensively studied in the literature of NLP (if the authors search hierarchical text generation or planning for text generation or long text generation). Yet this paper gives no discussion. Below I list several references [1][2] (but clearly not sufficient).

Author response:
**Q1**: The core idea is quite straightforward. I am afraid that similar ideas have been extensively studied in the literature of NLP. Yet this paper gives no discussion.

**A**: Thanks for referring to these works on hierarchical text generation. We have added the discussion to Appendix D.3 in the revision. As we position our method as a data-level efficient LLM method, we previously discussed SoT in the efficient LLM literature. However, we agree that previous hierarchical text generation studies indeed share some ideas with SoT (e.g., structured generation), and we should discuss them.

The high-level differences between SoT from this literature are:
- Objective: Prior studies in hierarchical text generation all focus on enhancing the answer quality, including improving the long-range coherence, relevance to the topic, or reducing redundancy. They still employ sequential word-by-word generation without parallelization between sentences.
- Methodology: Instead of designing new hierarchical architectures or planning modules, SoT exploits the emerging planning and instruction-following abilities of LLMs to do explicit (which means the plan is described by interpretable language) and free-form planning. This allows SoT to be applied to off-the-shelf LLMs for producing structured answers.

As we summarized in the global response, we think this **"prompting an answer plan for efficiency"** idea has its own merits and is worth future study. We appreciate this suggestion and think the hierarchical text generation literature might be a mine of insights for further extending SoT's idea to enable high-quality structured generation in the LLM era.

Reviewer comment (Evaluation):
As a result, the proposed method inevitably hurts the overall coherence of the answer. The experiments in this paper also confirm that the proposed method hurts the immersion and coherence judged by GPT4 (Section 3.2.4).

Author response:
**Q3**: The authors just ignored my concerns on the experiment results regarding the coherence of generated answers.

**A**: We would like to humbly point out that we **didn't** ignore this question. Just this question is saying the same thing as the second question, so we merge them into one question. As one can see, we have already merged the "coherence" question into Q2:

> **Q2, about the general applicability**: The proposed method only makes sense when the answer is long, decomposable into several key points, and the interdependence of these key points is low (because the writing of each point only sees the skeleton). **As a result, the proposed method inevitably hurts the overall coherence of the answer.**

We've already answered this question in the response to Q2. Current SoT is mainly suitable for questions that require answers whose structure contains points that are relatively independent. If it is not the case, SoT hurt the immersion and coherence, as we said in Section 3.2.4. Also, qualitative failure pattern analyses have been shown in the appendix of our original manuscript.