Original: Conﬁdent Adaptive Language Modeling
Revised: Conﬁdent Adaptive Language Modeling

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Recent advances in Transformer-based large language models (LLMs) have led tosigniﬁcant performance improvements across many tasks.
Revised: Recent advances in Transformer-based large language models (LLMs) have led tosigniﬁcant performance improvements across many tasks.

Original: These gains come with adrastic increase in the models’ size, potentially leading to slow and costly use at inference time.
Revised: These gains come with adrastic increase in the models’ size, potentially leading to slow and costly use at inference time.

Original: In practice, however, the series of generations made by LLMs is composed of varying levels of difﬁculty.
Revised: In practice, however, the series of generations made by LLMs is composed of varying levels of difﬁculty.

Original: While certain predictions truly beneﬁt from themodels’ full capacity, other continuations are more trivial and can be solved withreduced compute.
Revised: While certain predictions truly beneﬁt from themodels’ full capacity, other continuations are more trivial and can be solved withreduced compute.

Original: In this work, we introduce Conﬁdent Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of computeper input and generation timestep.
Revised: In this work, we introduce Conﬁdent Adaptive Language Modeling (CALM), a framework for dynamically allocating different amounts of computeper input and generation timestep.

Original: Early exit decoding involves several challengesthat we address here, such as: (1) what conﬁdence measure to use; (2) connectingsequence-level constraints to local per-token exit decisions; and (3) attending backto missing hidden representations due to early exits in previous tokens.
Revised: Early exit decoding involves several challengesthat we address here, such as: (1) what conﬁdence measure to use; (2) connectingsequence-level constraints to local per-token exit decisions; and (3) attending backto missing hidden representations due to early exits in previous tokens.

Original: Through theoretical analysis and empirical experiments on three diverse text generation tasks,we demonstrate the efﬁcacy of our framework in reducing compute—potentialspeedup of up to ⇥ 3 —while provably maintaining high performance.
Revised: Through theoretical analysis and empirical experiments on three diverse text generation tasks,we demonstrate the efﬁcacy of our framework in reducing compute—potentialspeedup of up to ⇥ 3 —while provably maintaining high performance.

Original:  Introduction
Revised:  Introduction

Original: Recent advances in Large Language Models (LLMs) have led to breakthroughs in language understanding and language generation across almost every widely-used Natural Language Processing(NLP) task considered in the ﬁeld today [5; 15; 17; 20; 51; 52; 53; 75; 89; 73].
Revised: Recent advances in Large Language Models (LLMs) have led to breakthroughs in language understanding and language generation across almost every widely-used Natural Language Processing(NLP) task considered in the ﬁeld today [5; 15; 17; 20; 46; 47; 48; 68; 82; 66].

Original: Autoregressivelanguage modeling provides a ﬂexible framework for solving complex tasks with a uniﬁed naturallanguage input and output format, while also relaxing the need for large-scale task-speciﬁc datacollection and training [67; 15; 17; 58; 80].
Revised: Autoregressivelanguage modeling provides a ﬂexible framework for solving complex tasks with a uniﬁed naturallanguage input and output format, while also relaxing the need for large-scale task-speciﬁc datacollection and training [13; 15; 17; 53; 73].

Original: The large size of LLMs, however, results in massivecomputational load that might be limiting for certain real-world applications (e g., machine translation) [9; 30; 42; 49; 59; 63; 71].
Revised: The large size of LLMs, however, results in massivecomputational load that might be limiting for certain real-world applications (e g., machinetranslation) [9; 29; 38; 44; 54; 58; 64].

Original: This is especially pronounced in the autoregressive decoding processwhere the full stack of Transformer layers is repeatedly computed for each output token [37; 40; 86].
Revised: This is especially pronounced in the autoregressive decodingprocess where the full stack of Transformer layers is repeatedly computed for each output token [79].

Original: While large models do better in general, the same amount of computation may not be required forevery input to achieve similar performance (e.g., depending on if the input is easy or hard) [66].
Revised: While large models do better in general, the same amount of computation may not be required forevery input to achieve similar performance (e.g., depending on if the input is easy or hard) [60].

Original: Earlyexiting is a promising approach to decreasing the computational cost of multilayered architecturessuch as those used in Transformer-based LLMs, where the number of layers used by the model isdynamically decided on an input-by-input basis [18; 23; 57; 60; 70].
Revised: Earlyexiting is a promising approach to decreasing the computational cost of multilayered architecturessuch as those used in Transformer-based LLMs, where the number of layers used by the model isdynamically decided on an input-by-input basis [18; 23; 52; 55; 63].

Original: In this setting, an LLM canchoose to generate a new token based off the representation at an intermediate layer instead of usingthe full model, and save computation as a result.
Revised: In this setting, an LLM canchoose to generate a new token based off the representation at an intermediate layer instead of usingthe full model, and save computation as a result.

Original: A natural question that arises, however, is when is ita good decision to exit early, as opposed to wait?
Revised: A natural question that arises, however, is when is ita good decision to exit early, as opposed to wait?

Original: Naively choosing when to exit can be suboptimal interms of saving computation time, and also result in unpredictable degradations to model performance,especially when predictions depend on each other, as in autoregressive language generation.
Revised: Naively choosing when to exit can be suboptimal interms of saving computation time, and also result in unpredictable degradations to model performance,especially when predictions depend on each other, as in autoregressive language generation.

Original: In this work, we analyze the early exiting paradigm for LLMs, and present a principled method forincreasing model efﬁciency while remaining conﬁdent in the quality of the resulting predictions.
Revised: In this work, we analyze the early exiting paradigm for LLMs, and present a principled method forincreasing model efﬁciency while remaining conﬁdent in the quality of the resulting predictions.

Original: Speciﬁcally, we develop a method for calibrating local, per-token, exit decisions such that global,
Revised: Speciﬁcally, we develop a method for calibrating local, per-token, exit decisions such that global,

Original: Layer Layer
Revised: Layer Layer

Original: Grant
Revised: Grant

Original: Elliott
Revised: Elliott

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layerhit a
Revised: Layerhit a

Original: Layersix to put <S> …
Revised: Layersix to put <S> …

Original: Example of “ state propagation ” (Dashed arrows show the hidden- states of previous steps used to compute layer 5 at position 6).
Revised: Example of “ state propagation ” (Dashed arrows show the hiddenstates of previous steps used to compute layer 5 at position 6).

Original: Early Exit?
Revised: Early Exit?

Original: Layer
Revised: Layer

Original: User-defined !,
Revised: User-defined !,

Original: # global tolerance constraints guide the local exiting decisions
Revised: # global tolerance constraints guide the local exiting decisions

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer Layer
Revised: Layer Layer

Original: Layer!
Revised: Layer!

Original: $%&'( ! )
Revised: $%&'( ! )

Original: *'' or $ +$,+
Revised: *'' or $ +$,+

Original: Guaranteed !,# textual or risk consistency of the full sequence ≈ sequence-level constraints—as determined by lexical or semantic sequence-level metrics like ROUGEor BLEURT score—are provably maintained with arbitrarily high probability (e.g., 95%).
Revised: Guaranteed !,# textual or risk consistency of the full sequence ≈ sequence-level constraints—as determined by lexical or semantic sequence-level metrics like ROUGEor BLEURT score—are provably maintained with arbitrarily high probability (e.g., 95%).

Original: Thisprocess, which we call C onﬁdent
Revised: Thisprocess, which we call C onﬁdent

Original: A daptive L anguage M odeling (CALM), is illustrated in Figure 1.
Revised: A daptive L anguage M odeling (CALM), is illustrated in Figure 1.

Original: Our approach leverages recent techniques in distribution-free risk control in order to create conﬁdentgenerations with strong statistical guarantees [2; 3; 10].
Revised: Our approach leverages recent techniques in distribution-free risk control in order to create conﬁdentgenerations with strong statistical guarantees [2; 3; 10].

Original: Concretely, suppose we have been givena calibration set S cal :
Revised: Concretely, suppose we have been givena calibration set S cal :

Original: = { P i } n i =1 2 P n of independent and identically distributed (i.i.d.) prompts 44 to our LLM (e.g., paragraphs to be summarized, sentences to be translated, or questions to beanswered via language modeling).
Revised: = { P i } ni =1 2 P n of independent and identically distributed (i.i.d.) prompts 44 to our LLM (e.g., paragraphs to be summarized, sentences to be translated, or questions to beanswered via language modeling).

Original: Let P test be a new i.i.d.
Revised: Let P test be a new i.i.d.

Original: test prompt to our LLM, where Y early :=
Revised: test prompt to our LLM, where Y early :=

Original: LLM early ( P test ) and Y full :=
Revised: LLM early ( P test ) and Y full :=

Original: LLM full ( P test ) are the adaptive and standard outputs of our LLM,respectively.
Revised: LLM full ( P test ) are the adaptive and standard outputs of our LLM,respectively.

Original: In order to be satisﬁed with Y early , we might require it to be textually consistent with
Revised: In order to be satisﬁed with Y early , we might require it to be textually consistent with

Original: Y full .
Revised: Y full .

Original: Given any bounded text dissimilarity function D , we aim to calibrate the early-exiting LLMsuch that its predictions agree to a tolerance  with the full model in expectation with high probability,
Revised: Given any bounded text dissimilarity function D , we aim to calibrate the early-exiting LLMsuch that its predictions agree to a tolerance  with the full model in expectation with high probability,

Original: [Equation]
Revised: [Equation]

Original: where the randomness is over draws of S cal , and ✏ 2 (0 , 1) .
Revised: where the randomness is over draws of S cal , and ✏ 2 (0 , 1) .

Original: Eq.
Revised: Eq.

Original: (1) has the signiﬁcant advantageof being achievable using only unlabeled calibration data S cal (a quality that is critical for fewshot tasks, for example).
Revised: (1) has the signiﬁcant advantageof being achievable using only unlabeled calibration data S cal (a quality that is critical for fewshot tasks, for example).

Original: Enforcing textual consistency with the original Y full , however, may beunnecessarily strict for certain tasks, especially where multiple generations may be acceptable.
Revised: Enforcing textual consistency with the original Y full , however, may beunnecessarily strict for certain tasks, especially where multiple generations may be acceptable.

Original: Asan alternative, given a calibration set of prompts paired with a set of (potentially multiple) targetreferences, S cal :
Revised: Asan alternative, given a calibration set of prompts paired with a set of (potentially multiple) targetreferences, S cal :

Original: =
Revised: =

Original: { ( P i , Z i ) } ni =1 2 ( P ⇥ 2 Y ) n , and any bounded risk function R , we also consider 57 an objective that enforces risk consistency by limiting the relative increase in risk of the predictions
Revised: { ( P i , Z i ) } ni =1 2 ( P ⇥ 2 Y ) n , and any bounded risk function R , we also consider 57 an objective that enforces risk consistency by limiting the relative increase in risk of the predictions

Original: Y early compared to Y full , with respect to the set of test-time references Z test , i.e.,
Revised: Y early compared to Y full , with respect to the set of test-time references Z test , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Within the constraints of either Eq.
Revised: Within the constraints of either Eq.

Original: (1) or Eq.
Revised: (1) or Eq.

Original: (2), the goal of our work is to ﬁnd the most computationally efﬁcient Y early , i.e., generations that exit as early as possible while still maintaining our desiredperformance guarantees.
Revised: (2), the goal of our work is to ﬁnd the most computationally efﬁcient Y early , i.e., generations that exit as early as possible while still maintaining our desiredperformance guarantees.

Original: In order to achieve this, it is necessary to develop a reliable signal for howlikely local, per-token early-exit decisions are to disrupt the global properties of the complete sequence.
Revised: In order to achieve this, it is necessary to develop a reliable signal for howlikely local, per-token early-exit decisions are to disrupt the global properties of the complete sequence.

Original: Here, we ﬁrst analyze how errors are propagated in Transformer-based LLMs, and then present aneffective and efﬁcient scoring mechanism for assigning “consistent early-exit” conﬁdence scores aftereach layer used during the generation of a new token.
Revised: Here, we ﬁrst analyze how errors are propagated in Transformer-based LLMs, and then present aneffective and efﬁcient scoring mechanism for assigning “consistent early-exit” conﬁdence scores aftereach layer used during the generation of a new token.

Original: The decision to exit or not is based on thesescores, and is carefully calibrated using S cal such that our performance bounds are provably satisﬁed.
Revised: The decision to exit or not is based on thesescores, and is carefully calibrated using S cal such that our performance bounds are provably satisﬁed.

Original: Finally, we empirically validate our method on multiple, diverse NLP generation tasks, includingtext summarization, machine translation, and question answering.
Revised: Finally, we empirically validate our method on multiple, diverse NLP generation tasks, includingtext summarization, machine translation, and question answering.

Original: Our experiments demonstrate thepotential of CALM in reducing the average complexity of the model and accelerating inference byabout ⇥ 3 while reliably controlling for high performance.
Revised: Our experiments demonstrate thepotential of CALM in reducing the average complexity of the model and accelerating inference byabout ⇥ 3 while reliably controlling for high performance.

Original: Contributions.
Revised: Contributions.

Original: In summary, our main contributions are as follows:
Revised: In summary, our main contributions are as follows:

Original: • A framework (CALM) for reliably accelerating Transformer-based LLM generations.
Revised: • A framework (CALM) for reliably accelerating Transformer-based LLM generations.•

Original: • A systematic analysis of the token-wise early exit mechanism that motivates a simple-but-effectiveclass of conﬁdence measures and threshold functions that are used as part of the CALM framework.•
Revised: A systematic analysis of the token-wise early exit mechanism that motivates a simple-but-effectiveclass of conﬁdence measures and threshold functions that are used as part of the CALM framework.

Original:  An empirical demonstration of CALM’s efﬁciency gains on three diverse generation datasets.
Revised: • An empirical demonstration of CALM’s efﬁciency gains on three diverse generation datasets.

Original:  Related Work
Revised:  Related Work

Original: Improving inference-time efﬁciency of LLMs has been an ongoing effort of the research communityover the past several years [49; 72; 85], leveraging techniques such as knowledge distillation [6; 32;36; 69; 69; 78; 56], ﬂoating point quantization [71; 65], layer pruning [24], vector dropping [38], andothers [41].
Revised: Improving inference-time efﬁciency of LLMs has been an ongoing effort of the research communityover the past several years [44; 65; 78], leveraging techniques such as knowledge distillation [6; 31;34; 62; 62; 71; 51], ﬂoating point quantization [64; 59], layer pruning [24], vector dropping [35], andothers [37].

Original: Another line of work involves conditional computation to train larger models that onlyuse a sparser subset of the full network during inference, for example by routing over mixture-ofexperts [9; 22; 39; 91], recurring modules [18; 29; 35], or accessing external memory [82].
Revised: Another line of work involves conditional computation to train larger models that onlyuse a sparser subset of the full network during inference, for example by routing over mixture-ofexperts [9; 22; 36; 84], recurring modules [18], or accessing external memory [75].

Original: Thesemodels, however, still use the same amount of compute for all input examples.
Revised: These models,however, still use the same amount of compute for all input examples.

Original: Here, we focus on adaptive compute , a speciﬁc kind of conditional compute that aims to dynamicallyallocate different computational power per example, with the goal of reducing the overall complexitywhile maintaining high performance.
Revised: Here, we focus on adaptive compute , a speciﬁc kind of conditional compute that aims to dynamicallyallocate different computational power per example, with the goal of reducing the overall complexitywhile maintaining high performance.

Original: This approach, often referred to as early-exiting [16; 25; 47;74; 79; 87], is complementary to many of the solutions above and can potentially be combined withthem.
Revised: This approach, often referred to as early-exiting [16; 25; 67;72; 80], is complementary to many of the solutions above and can potentially be combined withthem.

Original: Multiple early-exit techniques for encoder-only Transformers (e.g., BERT [20]) have beenrecently proposed [8; 34; 43; 44; 45; 60; 68; 83; 90; 92].
Revised: Multiple early-exit techniques for encoder-only Transformers (e.g., BERT [20]) have beenrecently proposed [8; 33; 39; 40; 41; 55; 61; 76; 83; 85].

Original: Most of these methods rely on intrinsicconﬁdence measures (e.g., based on the softmax distribution), while others try to predict the routingin advance [46; 70], or train a small early-exit classiﬁer [57; 84], as we also examine here.
Revised: Most of these methods rely on intrinsicconﬁdence measures (e.g., based on the softmax distribution), while others try to predict the routingin advance [42; 63], or train a small early-exit classiﬁer [52; 77], as we also examine here.

Original: Thesemeasures can be calibrated to reliably guarantee consistency of the early prediction with the fullmodel [57].
Revised: Thesemeasures can be calibrated to reliably guarantee consistency of the early prediction with the fullmodel [52].

Original: However, the techniques used for encoder-only classiﬁers are unsuitable for globalconsistency constraints with a sequence of dependent predictions, which are inherent in the decodingprocess of autoregressive language models, which we address here.
Revised: However, the techniques used for encoder-only classiﬁers are unsuitable for globalconsistency constraints with a sequence of dependent predictions, which are inherent in the decodingprocess of autoregressive language models, which we address here.

Original: Our work is also motivated by recent ﬁndings on the existence of saturation events in LMs, where thetop-ranked prediction is unchanged after some layer and is propagated upward.
Revised: Our work is also motivated by recent ﬁndings on the existence of saturation events in LMs, where thetop-ranked prediction is unchanged after some layer and is propagated upward.

Original: Geva et al.
Revised: Geva et al.

Original: [28] examined interactions of the hidden-state with feed-forward layers to predict these events.
Revised: [28] examined interactions of the hidden-state with feed-forward layers to predict these events.

Original: However, theyonly consider local single predictions and do not address the challenges involved with sequence generation.
Revised: However, theyonly consider local single predictions and do not address the challenges involved with sequence generation.

Original: Our early-exit LM architecture most closely relates to Elbayad et al.
Revised: Our early-exit LM architecture most closely relates to Elbayad et al.

Original: [23], who found a tokenlevel early-exit classiﬁer to provide the best efﬁciency-performance tradeoffs on machine translation.
Revised: [23], who found a tokenlevel early-exit classiﬁer to provide the best efﬁciency-performance tradeoffs on machine translation.

Original: Here, we introduce a theoretically-grounded calibration method for provably controlling the qualityof the full sequence.
Revised: Here, we introduce a theoretically-grounded calibration method for provably controlling the qualityof the full sequence.

Original: By doing so, we provide reliable efﬁciency gains—deriving local early exitingdecisions from the global desirable constraints.
Revised: By doing so, we provide reliable efﬁciency gains—deriving local early exitingdecisions from the global desirable constraints.

Original: Moreover, we introduce several model improvementsand empirical analyses, including (1) analyzing the primary sources of performance degradation, leading us to propose a decaying threshold function for better tradeoff control without inﬂating the searchspace; (2) improving the early-exit classiﬁer training; and (3) experimenting with two new tasks.
Revised: Moreover, we introduce several model improvementsand empirical analyses, including (1) analyzing the primary sources of performance degradation, leading us to propose a decaying threshold function for better tradeoff control without inﬂating the searchspace; (2) improving the early-exit classiﬁer training; and (3) experimenting with two new tasks.

Original: Our calibration procedure for connecting global constraints to local decisions, relates to recentresearch around distribution-free uncertainty quantiﬁcation [1; 62; 77].
Revised: Our calibration procedure for connecting global constraints to local decisions, relates to recentresearch around distribution-free uncertainty quantiﬁcation [1; 57; 70].

Original: Several methods weredeveloped in recent studies to expand and adjust the theoretical framework for obtaining practicalefﬁciency gains on target applications [4; 7; 21; 26; 27; 48; 88].
Revised: Several methods weredeveloped in recent studies to expand and adjust the theoretical framework for obtaining practicalefﬁciency gains on target applications [4; 7; 21; 26; 27; 43; 81].

Original: Here, we frame our consistencyrequirements around the Learn then Test (LTT) framework [3], and leverage the approximatelymonotonic behavior of our conﬁdence measures and the nested structure of our problem, that bydeﬁnition guarantees consistency with large enough threshold, to form tight and effective bounds.
Revised: Here, we frame our consistencyrequirements around the Learn then Test (LTT) framework [3], and leverage the approximatelymonotonic behavior of our conﬁdence measures and the nested structure of our problem, that bydeﬁnition guarantees consistency with large enough threshold, to form tight and effective bounds.

Original:  Early Exiting for Adaptive Language Modeling
Revised:  Early Exiting for Adaptive Language Modeling

Original: In the following, we describe and analyze the early-exiting Transformer LM.
Revised: In the following, we describe and analyze the early-exiting Transformer LM.

Original: We begin with a briefrecap of the Transformer architecture (§3.1) and early exiting (§3.2) for convenience, followingprevious work [23; 70; 76].
Revised: We begin with a briefrecap of the Transformer architecture (§3.1) and early exiting (§3.2) for convenience, followingprevious work [23; 63; 69].

Original: We then investigate the effects of early exiting on model performance,and identify primary sources of performance degradation and how to alleviate them (§3.3)—whichguide our architecture and training design (§3.4) and proposed per-token conﬁdence measures (§3.5).
Revised: We then investigate the effects of early exiting on model performance,and identify primary sources of performance degradation and how to alleviate them (§3.3)—whichguide our architecture and training design (§3.4) and proposed per-token conﬁdence measures (§3.5).

Original: 3.1 The Transformer architecture
Revised: 3.1 The Transformer architecture

Original: We use the Transformer sequence-to-sequence model, based on the T5x implementation [55].
Revised: We use the Transformer sequence-to-sequence model, based on the T5x implementation [50].

Original: Here,we only review simpliﬁed details of the Transformer architecture relevant to early-exiting, and referthe reader to Vaswani et al.
Revised: Here,we only review simpliﬁed details of the Transformer architecture relevant to early-exiting, and referthe reader to Vaswani et al.

Original: [76] for full details.
Revised: [69] for full details.

Original: At a high level, both encoder and decoder networkscontain L stacked layers, where each layer is composed of a multi-head self-attention sub-layer,followed by a feedforward sub-layer, each with residual connections and layer normalization.
Revised: At a high level, both encoder and decoder networkscontain L stacked layers, where each layer is composed of a multi-head self-attention sub-layer,followed by a feedforward sub-layer, each with residual connections and layer normalization.

Original: Thedecoder network has an additional multi-head attention sub-layer that attends to the encoder states.
Revised: Thedecoder network has an additional multi-head attention sub-layer that attends to the encoder states.

Original: Consider a prompt x = ( x 1 , . . .
Revised: Consider a prompt x = ( x 1 , . . .

Original: , x p ) , processed by the encoder to yield encoder states ( e 1 , . . .
Revised: , x p ) , processed by the encoder to yield encoder states ( e 1 , . . .

Original: , e p ) ,and the current, partially generated response ( y 1 , . . .
Revised: , e p ) ,and the current, partially generated response ( y 1 , . . .

Original: , y t ) .
Revised: , y t ) .

Original: When generating the next token y t +1 , thedecoder computes a decoder state d it for layer i out of L as:
Revised: When generating the next token y t +1 , thedecoder computes a decoder state d it for layer i out of L as:

Original: [Equation]
Revised: [Equation]

Original: FeedForward
Revised: FeedForward

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Internal to each of the attention mechanisms, written as Attention( x, z 1: m ) for some input x andsequence of m states z 1: m , x is ﬁrst projected to a query vector q :=
Revised: Internal to each of the attention mechanisms, written as Attention( x, z 1: m ) for some input x andsequence of m states z 1: m , x is ﬁrst projected to a query vector q :=

Original: W Q x 2 R dim k , while z isprojected to a matrix of key-value vectors, K :=
Revised: W Q x 2 R dim k , while z isprojected to a matrix of key-value vectors, K :=

Original: W K z 1: m 2 R m ⇥ dim k and V :=
Revised: W K z 1: m 2 R m ⇥ dim k and V :=

Original: W V z 1: m 

Revised: W V z 1: m

Original: R m ⇥ dim v .
Revised: R m ⇥ dim v .

Original: The output o is then computed as o :=
Revised: The output o is then computed as o :=

Original: softmax  q K > / p dim k 
Revised: softmax  q K > / p dim k

Original: V .
Revised: V .

Original: Multi-head and normalization components are omitted for brevity.
Revised: Multi-head and normalization components are omitted for brevity.

Original: Each layer uses different projections W iQ , W iK , and W iV (which are also unique for computing h it versus a it ).
Revised: Each layer uses different projections W iQ , W iK , and W iV (which are also unique for computing h it versus a it ).

Original: Finally, after layer L , a distribution over vocabulary tokens y t +1 2 Y is computed via a softmaxnormalized linear classiﬁer W L , where p ( y t +1 | d Lt ) =
Revised: Finally, after layer L , a distribution over vocabulary tokens y t +1 2 Y is computed via a softmaxnormalized linear classiﬁer W L , where p ( y t +1 | d Lt ) =

Original: softmax( W L d Lt ) .
Revised: softmax( W L d Lt ) .

Original: 3.2 Decoding with early exiting
Revised: 3.2 Decoding with early exiting

Original: Instead of always making a prediction based on the representation at the ﬁnal layer, d Lt , the keyidea in early-exiting is to choose y t +1 more quickly, if conﬁdent, by computing p ( y t +1 | d i t ) =
Revised: Instead of always making a prediction based on the representation at the ﬁnal layer, d Lt , the keyidea in early-exiting is to choose y t +1 more quickly, if conﬁdent, by computing p ( y t +1 | d i t ) =

Original: softmax( W i d it ) for some intermediate layer i < L .
Revised: softmax( W i d it ) for some intermediate layer i < L .

Original: Concretely, let c it 2 [0 , 1] denote some localconﬁdence score for layer i while processing token t , where higher values indicate a higher propensityto exit early (we will propose effective instantiations of c it in §3.5).
Revised: Concretely, let c it 2 [0 , 1] denote some localconﬁdence score for layer i while processing token t , where higher values indicate a higher propensityto exit early (we will propose effective instantiations of c it in §3.5).

Original: Let  it 2 [0 , 1] denote some localearly-exiting threshold, where the model exits early if c it   it , or otherwise proceeds to compute thenext representation, d i +1 t .
Revised: Let  it 2 [0 , 1] denote some localearly-exiting threshold, where the model exits early if c it   it , or otherwise proceeds to compute thenext representation, d i +1 t .

Original: The (greedily chosen) prediction y t +1 can then be written as:
Revised: The (greedily chosen) prediction y t +1 can then be written as:

Original: [Equation]
Revised: [Equation]

Original: Note that due to the self-attention mechanism of the Transformer, computing the input hidden stateh it for layer i depends on d i  1 1: t  1 , i.e., the output hidden states of the previous layer for all the tokensthat have been generated so far.
Revised: Note that due to the self-attention mechanism of the Transformer, computing the input hidden stateh it for layer i depends on d i  1 1: t  1 , i.e., the output hidden states of the previous layer for all the tokensthat have been generated so far.

Original: Therefore, if the model has early exited at some layer j < i  1 fora token s < t , then d i  1 s is not available.
Revised: Therefore, if the model has early exited at some layer j < i  1 fora token s < t , then d i  1 s is not available.

Original: As an approximation, we set d ks = d
Revised: As an approximation, we set d ks = d

Original: js for all layers k > jfollowing Elbayad et al.
Revised: js for all layers k > jfollowing Elbayad et al.

Original: [23], with the understanding that this will introduce some error.
Revised: [23], with the understanding that this will introduce some error.

Original: In the nextsection, in addition to other factors, we will analyze the impact of this copied state on performance.
Revised: In the nextsection, in addition to other factors, we will analyze the impact of this copied state on performance.

Original: 3.3 The effects of early exiting on error propagation
Revised: 3.3 The effects of early exiting on error propagation

Original: We perform several controlled experiments to investigate the behavior and the potential of earlyexiting during decoding.
Revised: We perform several controlled experiments to investigate the behavior and the potential of earlyexiting during decoding.

Original: We use an 8-layer T5 encoder-decoder and the CNN/DM dataset for theseexperiments.
Revised: We use an 8-layer T5 encoder-decoder and the CNN/DM dataset for theseexperiments.

Original: See §5 for more details on this model and data.
Revised: See §5 for more details on this model and data.

Original: 3.3.1 State propagation
Revised: 3.3.1 State propagation

Original: First, we control for the correctness of the predicted tokens to examine the effect of state copying(§3.2), and also measure an approximate upper bound for compute reduction.
Revised: First, we control for the correctness of the predicted tokens to examine the effect of state copying(§3.2), and also measure an approximate upper bound for compute reduction.

Original: We use an oracleconﬁdence measure that exits at the earliest layer that agrees with the top prediction (i.e., replacingthe conditions in Eq.
Revised: We use an oracleconﬁdence measure that exits at the earliest layer that agrees with the top prediction (i.e., replacingthe conditions in Eq.

Original: 4 with arg max p ( y t +1 | d it ) =
Revised: 4 with arg max p ( y t +1 | d it ) =

Original: arg max p ( y t +1 | d Lt ) ).
Revised: arg max p ( y t +1 | d Lt ) ).

Original: Hence, the only factorthat can cause divergence in the generation is the state copying mechanism for skipped layers.
Revised: Hence, the only factorthat can cause divergence in the generation is the state copying mechanism for skipped layers.

Original: Theresults of this experiment are highly encouraging.
Revised: Theresults of this experiment are highly encouraging.

Original: This oracle achieves an ROUGE-L score of 38.24,compared to 38.32 with the full model, while only using an average of 1.53 layers per token.
Revised: This oracle achieves an ROUGE-L score of 38.24,compared to 38.32 with the full model, while only using an average of 1.53 layers per token.

Original: Wealso try an oracle that always uses d 11: t  1 and it reaches 38.

Revised: Wealso try an oracle that always uses d 11: t  1 and it reaches 38.

Original: ROUGE-L.
Revised: ROUGE-L.

Original: These results indicate that(1) the model is robust to state copying from lower layers, and (2) there is remarkable potential forsaving compute—by up to ⇥ 5 . 2 —while preserving performance, given a good conﬁdence measure.
Revised: These results indicate that(1) the model is robust to state copying from lower layers, and (2) there is remarkable potential forsaving compute—by up to ⇥ 5 . 2 —while preserving performance, given a good conﬁdence measure.

Original: We also experiment with copying the projected states K j , V j to skipped layers k > j .
Revised: We also experiment with copying the projected states K j , V j to skipped layers k > j .

Original: This versionof the oracle results in a signiﬁcant drop in performance to 23.02 ROUGE-L. Overall, we conjecture
Revised: This versionof the oracle results in a signiﬁcant drop in performance to 23.02 ROUGE-L. Overall, we conjecture

Original: [Figure]
Revised: [Figure]

Original: that the self-attention at layer i for token t can safely use hidden-states d js for j < i  1 as key-valuesof tokens s < t , as long as the projections W iK/V of layer i are used.
Revised: that the self-attention at layer i for token t can safely use hidden-states d js for j < i  1 as key-valuesof tokens s < t , as long as the projections W iK/V of layer i are used.

Original: Notably, this projection cannow be computed concurrently for all skipped layers as they all use the same d from the exited layer.
Revised: Notably, this projection cannow be computed concurrently for all skipped layers as they all use the same d from the exited layer.

Original: 3.3.2 Sensitivity to local errors
Revised: 3.3.2 Sensitivity to local errors

Original: Next, we examine the impact of local token modiﬁcations—which might occur due to early exits—onthe whole generated sequence.
Revised: Next, we examine the impact of local token modiﬁcations—which might occur due to early exits—onthe whole generated sequence.

Original: We experiment with two kinds of perturbations: sampling-based ,where we select the 10th-ranked token according to layer L ; and layer-based , where we select the theﬁrst layer’s prediction at timestep t .
Revised: We experiment with two kinds of perturbations: sampling-based ,where we select the 10th-ranked token according to layer L ; and layer-based , where we select the theﬁrst layer’s prediction at timestep t .

Original: All other tokens are predicted greedily by layer L .
Revised: All other tokens are predicted greedily by layer L .

Original: As shown in
Revised: As shown in

Original: Figure 2a, earlier perturbations result in lower sequence-level scores as there are more tokens thatmight suffer from the divergence.
Revised: Figure 2a, earlier perturbations result in lower sequence-level scores as there are more tokens thatmight suffer from the divergence.

Original: The degradation, though, is much smaller with layer- compared tosampling-based perturbations since, in practice, the early exit predictions are mostly accurate.
Revised: The degradation, though, is much smaller with layer- compared tosampling-based perturbations since, in practice, the early exit predictions are mostly accurate.

Original: Decaying threshold.
Revised: Decaying threshold.

Original: Following the above observation, we introduce a decaying early-exitingthreshold that is more permissive towards exiting as the decoding process continues.
Revised: Following the above observation, we introduce a decaying early-exitingthreshold that is more permissive towards exiting as the decoding process continues.

Original: Motivated by thelogarithmic behavior in Figure 2a, we use an exponential function with a user-deﬁned temperature ⌧ :
Revised: Motivated by thelogarithmic behavior in Figure 2a, we use an exponential function with a user-deﬁned temperature ⌧ :

Original: [Equation]
Revised: [Equation]

Original: where N is the maximum output length.
Revised: where N is the maximum output length.

Original: Figure 2b illustrates this function.
Revised: Figure 2b illustrates this function.

Original: Essentially, this functionpresents an effective compromise between simply using the same threshold for all tokens, andsearching over a huge space of per-position different thresholds.
Revised: Essentially, this functionpresents an effective compromise between simply using the same threshold for all tokens, andsearching over a huge space of per-position different thresholds.

Original: Practically, it supports ﬁner andbetter control over the performance-efﬁciency tradeoff compared to a single threshold.
Revised: Practically, it supports ﬁner andbetter control over the performance-efﬁciency tradeoff compared to a single threshold.

Original: Figure 2cpresents the outcomes of a search over  with steps of 0 .
Revised: Figure 2cpresents the outcomes of a search over  with steps of 0 .

Original: 01 and softmax-based conﬁdence (§3.5).
Revised: 01 and softmax-based conﬁdence (§3.5).

Original: With the single threshold variant ( ⌧ = 0 ), attempting to improve the efﬁciency will lead to a drasticdrop of more than 10 points in the textual similarity against the full model’s prediction.
Revised: With the single threshold variant ( ⌧ = 0 ), attempting to improve the efﬁciency will lead to a drasticdrop of more than 10 points in the textual similarity against the full model’s prediction.

Original: In contrast,the decaying thresholds reveal several intermediate points with desirable tradeoffs to consider.
Revised: In contrast,the decaying thresholds reveal several intermediate points with desirable tradeoffs to consider.

Original: 3.4 Training early exit classiﬁers for local consistency
Revised: 3.4 Training early exit classiﬁers for local consistency

Original: While our goal is to preserve the quality of the complete output sequence, we note that this doesn’tnecessarily demand local token-level consistency.
Revised: While are goal is to preserve the quality of the complete output sequence, we note that this doesn’tnecessarily demand local token-level consistency.

Original: Consider the target sequence “the concert waswonderful and long.”
Revised: Consider the target sequence “the concert waswonderful and long.”

Original: An output that switches the order of adjectives to “the concert was long andwonderful” would be called consistent by most semantic measures (and obtain 100 token- F 1 score).
Revised: An output that switches the order of adjectives to “the concert was long andwonderful” would be called consistent by most semantic measures (and obtain 100 token- F 1 score).

Original: Yet, the sentences diverge at the ﬁrst adjective long which is semantically different from wonderful .
Revised: Yet, the sentences diverge at the ﬁrst adjective long which is semantically different from wonderful .

Original: Training for global consistency, however, could be challenging [81] as it depends on possibly noisysignals that might affect the learning, and also breaks the efﬁcient teacher-forcing training strategyof LMs that relies on local-decisions.
Revised: Training for global consistency, however, could be challenging [74] as it depends on possibly noisysignals that might affect the learning, and also breaks the efﬁcient teacher-forcing training strategyof LMs that relies on local-decisions.

Original: On the other hand, perfect local consistency implies globalconsistency.
Revised: On the other hand, perfect local consistency implies globalconsistency.

Original: Therefore, we opt to train for local consistency, which requires minimal changes to thetraining procedure, and relax the local requirement to a global one during inference.
Revised: Therefore, we opt to train for local consistency, which requires minimal changes to thetraining procedure, and relax the local requirement to a global one during inference.

Original: Speciﬁcally, similar to Elbayad et al.
Revised: Speciﬁcally, similar to Elbayad et al.

Original: [23], we average losses for each layer to obtain the objective
Revised: [23], we average losses for each layer to obtain the objective

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: L is the negative log-likelihood loss.
Revised: L is the negative log-likelihood loss.

Original: We set !
Revised: We set !

Original: i = i/
Revised: i = i/

Original: P
Revised: P

Original: Lj =1 j to favor higher layers, and ﬁnd this objective to mostly preserve the full model’s performance compared to regular training.
Revised: Lj =1 j to favor higher layers, and ﬁnd this objective to mostly preserve the full model’s performance compared to regular training.

Original: We note that thereis some misalignment between this training and inference behavior due to the hidden states of skippedlayers.
Revised: We note that thereis some misalignment between this training and inference behavior due to the hidden states of skippedlayers.

Original: However, as discussed in §3.3.1, the performance is not affected if the hidden-state is copied.
Revised: However, as discussed in §3.3.1, the performance is not affected if the hidden-state is copied.

Original: 3.5 Local conﬁdence measures
Revised: 3.5 Local conﬁdence measures

Original: We experiment with three conﬁdence measures for Eq.
Revised: We experiment with three conﬁdence measures for Eq.

Original: (4) that differ in their parameter and computeoperation efﬁciencies.
Revised: (4) that differ in their parameter and computeoperation efﬁciencies.

Original: Our experiments (§6) will also show that they differ in their predictive power.
Revised: Our experiments (§6) will also show that they differ in their predictive power.

Original: Softmax response.
Revised: Softmax response.

Original: We take the difference between the top two values of Softmax( W i d it ) .
Revised: We take the difference between the top two values of Softmax( W i d it ) .

Original: With alarge output vocabulary, this results in many ﬂoating point operations (FLOPs)—though, the nextlayer i + 1 can start its computation in parallel, avoiding additional runtime.
Revised: With alarge output vocabulary, this results in many ﬂoating point operations (FLOPs)—though, the nextlayer i + 1 can start its computation in parallel, avoiding additional runtime.

Original: Hidden-state saturation.
Revised: Hidden-state saturation.

Original: As a simple parameter-free and fast to compute alternative, we take thecosine similarity sim( d it , d i  1 t ) for i > 1 .
Revised: As a simple parameter-free and fast to compute alternative, we take thecosine similarity sim( d it , d i  1 t ) for i > 1 .

Original: By deﬁnition, the ﬁrst possible exit is at the second layer(unless  = 0 ).
Revised: By deﬁnition, the ﬁrst possible exit is at the second layer(unless  = 0 ).

Original: This measure tries to identify early saturation events of the hidden-state [28].
Revised: This measure tries to identify early saturation events of the hidden-state [28].

Original: Early exit classiﬁer.
Revised: Early exit classiﬁer.

Original: We train a dedicated linear classiﬁer M to predict the likelihood of exitingwith local consistency given the current hidden-state: c i t =
Revised: We train a dedicated linear classiﬁer M to predict the likelihood of exitingwith local consistency given the current hidden-state: c i t =

Original: M ( d i t ) .
Revised: M ( d i t ) .

Original: This measure is very fast tocompute at inference, and adds only | d | + 1 new parameters.
Revised: This measure is very fast tocompute at inference, and adds only | d | + 1 new parameters.

Original: To avoid any impact on the core model’sperformance, we train it as a second step where we freeze all parameters other than M .
Revised: To avoid any impact on the core model’sperformance, we train it as a second step where we freeze all parameters other than M .

Original: We simplyuse a per-layer independent cross-entropy loss against a consistency oracle [arg max( p ( y t +1 | d it ) =
Revised: We simplyuse a per-layer independent cross-entropy loss against a consistency oracle [arg max( p ( y t +1 | d it ) =

Original: arg max( p ( y t +1 | d Lt )] , and average across the L  1 layers.
Revised: arg max( p ( y t +1 | d Lt )] , and average across the L  1 layers.

Original: We also experimented with the geometriclike training of Elbayad et al.
Revised: We also experimented with the geometriclike training of Elbayad et al.

Original: [23], but ﬁnd it to be less effective here (see App. D).
Revised: [23], but ﬁnd it to be less effective here (see App. D).

Original: The two objectivesare closely related, but the geometric one ignores any signal from the states post the ﬁrst oracle exit.
Revised: The two objectivesare closely related, but the geometric one ignores any signal from the states post the ﬁrst oracle exit.

Original:  Calibrating Local Early Exits from Global Constraints
Revised:  Calibrating Local Early Exits from Global Constraints

Original: We now describe our calibration procedure for ﬁnding a shared exit threshold  2 [0 , 1] that can beused directly in Eq.
Revised: We now describe our calibration procedure for ﬁnding a shared exit threshold  2 [0 , 1] that can beused directly in Eq.

Original: (4), or via Eq.
Revised: (4), or via Eq.

Original: (5), such that we provably satisfy our desired global constraintsover the fully generated sequences.
Revised: (5), such that we provably satisfy our desired global constraintsover the fully generated sequences.

Original: At a high level, our approach uses the following basic recipe:
Revised: At a high level, our approach uses the following basic recipe:

Original: We specify a grid of possible values of ⇤ =
Revised: We specify a grid of possible values of ⇤ =

Original: (  1 , . . .
Revised: (  1 , . . .

Original: ,  k ) that
Revised: ,  k ) that

Original: may
Revised: may

Original: result in acceptable generations;
Revised: result in acceptable generations;

Original: We choose the lowest valid  2 ⇤ that we can identify with rigorous statistical testing tools.
Revised: We choose the lowest valid  2 ⇤ that we can identify with rigorous statistical testing tools.

Original: Let P test be an i.i.d.
Revised: Let P test be an i.i.d.

Original: prompt given to the LLM at test time, and let Y full :
Revised: prompt given to the LLM at test time, and let Y full :

Original: = LLM full ( P test )2 Y and
Revised: = LLM full ( P test ) and

Original: Y early :=
Revised: Y early :=

Original: LLM early ( P test ,  )2 Y denote the full and adaptive responses, respectively.
Revised: LLM early ( P test ,  ) denote the full and adaptive responses, respectively.

Original: Optionally,let Z test be a set of gold references for our task, if assumed.
Revised: Optionally, let Z test be a set of gold references for our task, if assumed.

Original: Our goal, as introduced in §1, is to ﬁnd avalid  using S cal such that we satisfy either of two types of global “consistency” constraints:
Revised: Our goal, as introduced in §1, is to ﬁnd avalid  using S cal such that we satisfy either of two types of global “consistency” constraints:

Original: Deﬁnition 1 (Textual consistency) .
Revised: Deﬁnition 1 (Textual consistency) .

Original: An adaptive LLM is textually consistent if given any bounded
Revised: An adaptive LLM is textually consistent if given any bounded

Original: text dissimilarity function, D : Y ⇥ Y !
Revised: text dissimilarity function, D : Y ⇥ Y !

Original: R , and tolerance  2 R , E ⇥ D ( Y early , Y full )
Revised: [Equation] and tolerance [Equation]

Original: ⇤   .

Original: Deﬁnition 2 (Risk consistency) .
Revised: Deﬁnition 2 (Risk consistency) .

Original: An adaptive LLM is risk consistent if given any bounded riskfunction, R :
Revised: An adaptive LLM is risk consistent if given any bounded riskfunction, R :

Original: Y ⇥ 2 Y !
Revised: Y ⇥ 2 Y !

Original: R , and tolerance  2 R , E [ R ( Y early , Z test )]  E [ R ( Y full , Z test )] +  .
Revised: R , and tolerance  2 R , E [ R ( Y early , Z test )]  E [ R ( Y full , Z test )] +  .

Original: Without loss of generality, we will assume that D and R are always normalized to the unit interval 240 [0 , 1] , and therefore will only be considering tolerances  2 (0 , 1) .
Revised: Without loss of generality, we will assume that D and R are always normalized to the unit interval 240 [0 , 1] , and therefore will only be considering tolerances  2 (0 , 1) .

Original: At a glance, to ﬁnd a  thatproduces a consistent LLM early , we cast our problem as a multiple hypothesis testing problemover a large array of k candidate classiﬁer exit thresholds, ⇤ =
Revised: At a glance, to ﬁnd a  thatproduces a consistent LLM early , we cast our problem as a multiple hypothesis testing problemover a large array of k candidate classiﬁer exit thresholds, ⇤ =

Original: (  1 , . . .
Revised: (  1 , . . .

Original: ,  k ) , and apply the Learnthen Test (LTT) framework of Angelopoulos et al.
Revised: ,  k ) , and apply the Learnthen Test (LTT) framework of Angelopoulos et al.

Original: [3] to identify a subset of statistically valid,constraint-satisfying thresholds ⇤ valid ✓ ⇤ .
Revised: [3] to identify a subset of statistically valid,constraint-satisfying thresholds ⇤ valid ✓ ⇤ .

Original: Our ﬁnal  is then chosen as  := min( ⇤ valid [ { 1 } ) .
Revised: Our ﬁnal  is then chosen as  := min( ⇤ valid [ { 1 } ) .

Original: 4.1 The Learn then Test calibration framework
Revised: 4.1 The Learn then Test calibration framework

Original: Choosing a value of  that rigorously satisﬁes our consistency objectives is challenging, as theperformance impact of increasing or decreasing  is not necessarily monotonic.
Revised: Choosing a value of  that rigorously satisﬁes our consistency objectives is challenging, as theperformance impact of increasing or decreasing  is not necessarily monotonic.

Original: Naively setting  ,for example, based simply on average calibration set performance, can lead to statistically invalidresults in our ﬁnite-sample, distribution-free setting.
Revised: Naively setting  ,for example, based simply on average calibration set performance, can lead to statistically invalidresults in our ﬁnite-sample, distribution-free setting.

Original: The LTT framework proposed by Angelopouloset al.
Revised: The LTT framework proposed by Angelopouloset al.

Original: [3] solves this problem by reframing hyper-parameter selection as a multiple testing problem.
Revised: [3] solves this problem by reframing hyper-parameter selection as a multiple testing problem.

Original: Let ⇤ =
Revised: Let ⇤ =

Original: (  1 , . . .
Revised: (  1 , . . .

Original: ,  k ) be a ﬁnite grid of hyper-parameter values that may, or may not, obtain validconsistency.
Revised: ,  k ) be a ﬁnite grid of hyper-parameter values that may, or may not, obtain validconsistency.

Original: For example, when searching for a value of  2 [0 , 1] , we might consider the evenlyspaced set ⇤ =
Revised: For example, when searching for a value of  2 [0 , 1] , we might consider the evenlyspaced set ⇤ =

Original: { i k +1 : i = 1 , . . .
Revised: { i k +1 : i = 1 , . . .

Original: , k } .
Revised: , k } .

Original: LTT then identiﬁes a subset of values, ⇤ valid ✓ ⇤ , where
Revised: LTT then identiﬁes a subset of values, ⇤ valid ✓ ⇤ , where

Original: [Equation]
Revised: [Equation]

Original: LLM early
Revised: LLM early

Original: [Equation]
Revised: [Equation]

Original: P test
Revised: P test

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: and LLM
Revised: and LLM

Original: [Equation]
Revised: [Equation]

Original: ( P test
Revised: ( P test

Original: [Equation]
Revised: [Equation]

Original: are not consistent
Revised: are not consistent

Original: [Equation]
Revised: [Equation]

Original: Here, we are using consistency to refer to either textual consistency or risk consistency.
Revised: Here, we are using consistency to refer to either textual consistency or risk consistency.

Original: Eq.
Revised: Eq.

Original: (7) canbe satisﬁed by applying standard multiple hypothesis testing techniques as long as super-uniformp-values, p j , are supplied for each value  j 2 ⇤ that support the null hypothesis
Revised: (7) canbe satisﬁed by applying standard multiple hypothesis testing techniques as long as super-uniformp-values, p j , are supplied for each value  j 2 ⇤ that support the null hypothesis

Original: [Equation]
Revised: [Equation]

Original:  j is placed in ⇤ valid if H j is rejected, and discarded otherwise.
Revised:  j is placed in ⇤ valid if H j is rejected, and discarded otherwise.

Original: This yields a consistent LLM early .
Revised: This yields a consistent LLM early .

Original: Proposition 1 (LTT for CALM) .
Revised: Proposition 1 (LTT for CALM) .

Original: Suppose p j is super-uniform for all j under H j for some speciﬁedtolerance  2 (0 , 1) .
Revised: Suppose p j is super-uniform for all j under H j for some speciﬁedtolerance  2 (0 , 1) .

Original: Let A be any family-wise error rate (FWER) controlling procedure at a level✏ 2 (0 , 1) , where A ( p 1 , . .
Revised: Let A be any family-wise error rate (FWER) controlling procedure at a level✏ 2 (0 , 1) , where A ( p 1 , . .

Original: , p k ) selects H j to reject.
Revised: , p k ) selects H j to reject.

Original: Choosing  :=
Revised: Choosing  :=

Original: min( ⇤ valid [ { 1 } ) then yieldsa consistent LLM early with probability at least 1  ✏ .
Revised: min( ⇤ valid [ { 1 } ) then yieldsa consistent LLM early with probability at least 1  ✏ .

Original: Note that a FWER-controlling procedure at a level ✏ is an algorithm that decides to accept or rejecthypotheses { H i } ki =1 , while ensuring that the probability of falsely rejecting any H j is less than ✏ .
Revised: Note that a FWER-controlling procedure at a level ✏ is an algorithm that decides to accept or rejecthypotheses { H i } ki =1 , while ensuring that the probability of falsely rejecting any H j is less than ✏ .

Original: Theproof of Proposition 1, given in Appendix A.1, follows directly from Theorem 1 of Angelopoulos et al.[3], and the fact that LLM early ( P test , 1) =
Revised: Theproof of Proposition 1, given in Appendix A.1, follows directly from Theorem 1 of Angelopoulos et al.[3], and the fact that LLM early ( P test , 1) =

Original: LLM full ( P test ) by construction per Eq.
Revised: LLM full ( P test ) by construction per Eq.

Original: (4), so that we canalways use  = 1 as a valid fallback if we fail to identify non-empty ⇤ valid .
Revised: (4), so that we canalways use  = 1 as a valid fallback if we fail to identify non-empty ⇤ valid .

Original: In the next sections, wedescribe how we calculate valid p-values using S cal , and our choice of FWER-controlling procedure.
Revised: In the next sections, wedescribe how we calculate valid p-values using S cal , and our choice of FWER-controlling procedure.

Original: 4.2 Deﬁning p-values for
Revised: 4.2 Deﬁning p-values for

Original: consistent
Revised: consistent

Original: early-exiting
Revised: early-exiting

Original: LTT relies on valid p-values p j , where p j is a random variable satisfying P ( p j  u )  u under H jfor all u 2 [0 , 1] .
Revised: LTT relies on valid p-values p j , where p j is a random variable satisfying P ( p j  u )  u under H jfor all u 2 [0 , 1] .

Original: For our purposes, we can obtain valid p-values from the empirical consistency of
Revised: For our purposes, we can obtain valid p-values from the empirical consistency of

Original: LLM early ( P i ,  ) measured over the random calibration sample, S cal .
Revised: LLM early ( P i ,  ) measured over the random calibration sample, S cal .

Original: Since we have assumed w.l.o.g.that either of our bounded consistency functions D and R from Defs.
Revised: Since we have assumed w.l.o.g.that either of our bounded consistency functions D and R from Defs.

Original: 1 and 2 have been normalized tolie in [0 , 1] , we can, for example, obtain a valid p-value by simply inverting Hoeffding’s inequality: 

Revised: 1 and 2 have been normalized tolie in [0 , 1] , we can, for example, obtain a valid p-value by simply inverting Hoeffding’s inequality:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: ni =
Revised: ni =

Original: 1 L i (  j ) is the empirical average of random variable L i (  j ) 2 [0 , 1] , with
Revised: 1 L i (  j ) is the empirical average of random variable L i (  j ) 2 [0 , 1] , with

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: for textual consistency versus risk consistency, respectively.
Revised: for textual consistency versus risk consistency, respectively.

Original: Note that, as a technicality of enforcingthe r.v.
Revised: Note that, as a technicality of enforcingthe r.v.

Original: L i (  j ) to be within [0 , 1] , Eq.
Revised: L i (  j ) to be within [0 , 1] , Eq.

Original: (11) computes a conservative estimate of the difference in theempirical risk that doesn’t reward instances in which the risk of the early-exit model is lower.
Revised: (11) computes a conservative estimate of the difference in theempirical risk that doesn’t reward instances in which the risk of the early-exit model is lower.

Original: 4.3 Efﬁcient ﬁxed sequence testing
Revised: 4.3 Efﬁcient ﬁxed sequence testing

Original: The more values of  we test, the higher the chance that we might accidentally choose a  thatdoes not in fact result in consistent generations, despite whatever misleading performance we mighthave measured by chance on S cal .
Revised: The more values of  we test, the higher the chance that we might accidentally choose a  thatdoes not in fact result in consistent generations, despite whatever misleading performance we mighthave measured by chance on S cal .

Original: As part of LTT, we must select a multiple testing procedure thatcorrects for this (i.e., that controls the FWER at level ✏ ).
Revised: As part of LTT, we must select a multiple testing procedure thatcorrects for this (i.e., that controls the FWER at level ✏ ).

Original: Though the precise dependence betweenthe early-exit LLM’s performance and  is unknown, in practice we ﬁnd that it tends to be fairlysmooth and roughly monotonic.
Revised: Though the precise dependence betweenthe early-exit LLM’s performance and  is unknown, in practice we ﬁnd that it tends to be fairlysmooth and roughly monotonic.

Original: That is, nearby thresholds  ⇡  0 tend to perform similarly, whereas >  0 tends to result in relatively more consistent performance.
Revised: That is, nearby thresholds  ⇡  0 tend to perform similarly, whereas >  0 tends to result in relatively more consistent performance.

Original: Taking advantage of this structure,we choose to employ ﬁxed sequence testing (FST) as our FWER-controlling procedure [3; 11].
Revised: Taking advantage of this structure,we choose to employ ﬁxed sequence testing (FST) as our FWER-controlling procedure [3; 11].

Original: Here we deﬁne a sequence of descending thresholds  1 >  2 > . . .
Revised: Here we deﬁne a sequence of descending thresholds  1 >  2 > . . .

Original:  k with a relatively coarse stepsize (e.g., increments of 0 . 05 ).
Revised:  k with a relatively coarse stepsize (e.g., increments of 0 . 05 ).

Original: For each  j in order , we compute p j , and reject H j if p j  ✏ .
Revised: For each  j in order , we compute p j , and reject H j if p j  ✏ .

Original: The ﬁrsttime we fail to reject H j , we immediately terminate our search, and return  j  1 to use as our calibratedthreshold (or 1 , if we fail to reject H 1 ).
Revised: The ﬁrsttime we fail to reject H j , we immediately terminate our search, and return  j  1 to use as our calibratedthreshold (or 1 , if we fail to reject H 1 ).

Original: An Algorithm of the full procedure is provided in Appendix E.
Revised: An Algorithm of the full procedure is provided in Appendix E.

Original: [Figure]
Revised: [Figure]

Original:  Experimental Setting
Revised:  Experimental Setting

Original: [Table]
Revised: [Table]

Original: We empirically evaluate our methods on three popular text generation tasks that vary in their target generation length and extractive degrees against the input.
Revised: We empirically evaluate our methods on three popular text generation tasks that vary in their target generation length and extractive degrees against the input.

Original: CNN/DM [31] is a collectionof news articles to be summarized in few sentences.
Revised: CNN/DM [30] is a collectionof news articles to be summarized in few sentences.

Original: WMT

Revised: WMT

Original: EN-FR [13] contains English sentences (one per example) to bemachine translated to French.
Revised: EN-FR [14] contains English sentences (one per example) to bemachine translated to French.

Original: Open-book SQ U AD 1.1 [54] isa QA dataset with Wikipedia paragraphs paired with questions,where the target answer is a text span from the input.
Revised: Open-book SQ U AD 1.1 [49] isa QA dataset with Wikipedia paragraphs paired with questions,where the target answer is a text span from the input.

Original: Lengthstatistics of the validation sets are summarized in Table 1.
Revised: Lengthstatistics of the validation sets are summarized in Table 1.

Original: Model.
Revised: Model.

Original: We implement CALM on top of the T5 encoder-decoder model that showed good performanceon the tasks above [53], using the T5X framework [55].
Revised: We implement CALM on top of the T5 encoder-decoder model that showed good performanceon the tasks above [48], using the T5X framework [50].

Original: We use the 8 layers T5 1.1 model that doesn’tshare input and output embeddings.
Revised: We use the 8 layers T5 1.1 model that doesn’tshare input and output embeddings.

Original: We share all output embeddings for the softmax predictions, andthe early-exit classiﬁer across all decoder layers.
Revised: We share all output embeddings for the softmax predictions, andthe early-exit classiﬁer across all decoder layers.

Original: Based on validation results, we set the temperatureof our decaying threshold to ⌧ = 4 for the softmax and classiﬁer measures of CNN/DM and WMT.
Revised: Based on validation results, we set the temperatureof our decaying threshold to ⌧ = 4 for the softmax and classiﬁer measures of CNN/DM and WMT.

Original: In other settings, we use ⌧ = 0 .
Revised: In other settings, we use ⌧ = 0 .

Original: See App. C for more details, and App.
Revised: See Appendix C for more details.

Original: B.3 for a 12 layers T5 model.

Original: Evaluation metrics.
Revised: Evaluation metrics.

Original: We use the standard metrics for each task: ROUGE-L for CNN/DM, BLEU [50]for WMT, and Token-F1 [54] for SQ U AD.
Revised: We use the standard metrics for each task: ROUGE-L for CNN/DM, BLEU [45]for WMT, and Token-F1 [49] for SQ U AD.

Original: We rely on the same metrics for computing the riskand textual distance, other than BLEU which is a corpus-level metric that doesn’t directly enable expectation control.
Revised: We rely on the same metrics for computing the riskand textual distance, other than BLEU which is a corpus-level metric that doesn’t directly enable expectation control.

Original: Instead, we use the BLEURT learned metric [61].
Revised: Instead, we use the BLEURT learned metric [56].

Original: For a given metricm ( y early , y full or z test ) 2 [0 , 1] , we use 1  m for distance or risk computation, respectively.
Revised: For a given metricm ( y early , y full or z test ) 2 [0 , 1] , we use 1  m for distance or risk computation, respectively.

Original: Our main efﬁciency metric is the average number of decoder layers used per output token, as itdirectly measures complexity reduction without conﬂating with implementation or infrastructure speciﬁc details [19].
Revised: Our main efﬁciency metric is the average number of decoder layers used per output token, as itdirectly measures complexity reduction without conﬂating with implementation or infrastructure speciﬁc details [19].

Original: For reference, we also report the average decoder FLOPs reduction per token [23].
Revised: For reference, we also report the average decoder FLOPs reduction per token [23].

Original: Also, we compute an estimated speedup potential of the whole encoder-decoder model for the full sequence, assuming parallel implementation, based on TPUv3 timings in Colab (see App.
Revised: Also, we compute an estimated speedup potential of the whole encoder-decoder model for the full sequence, assuming parallel implementation, based on TPUv3 timings in Colab (see App.

Original: C for details).
Revised: C for details).

Original: Calibration experiments.
Revised: Calibration experiments.

Original: For each task, we use the validation and test sets to evaluate our calibrationmethod (§4) (for SQ U AD we only use the validation set as the test answers are hidden).
Revised: For each task, we use the validation and test sets to evaluate our calibrationmethod (§4) (for SQ U AD we only use the validation set as the test answers are hidden).

Original: We run 50random trials per target tolerance  and consistency objective (textual or risk), where we partition thedata to 80% calibration ( S cal ) and 20% test ( P test ).
Revised: We run 50random trials per target tolerance  and consistency objective (textual or risk), where we partition thedata to 80% calibration ( S cal ) and 20% test ( P test ).

Original: We set ✏ = 0 . 05 for all experiments.
Revised: We set ✏ = 0 . 05 for all experiments.

Original: Baselines.
Revised: Baselines.

Original: We emphasize that the CALM framework is general for any autoregressive multi-layered
Revised: We emphasize that the CALM framework is general for any autoregressive multi-layered

Original: LM with any conﬁdence measure, allowing controlled consistency by Eq.
Revised: LM with any conﬁdence measure, allowing controlled consistency by Eq.

Original: (1) or Eq.
Revised: (1) or Eq.

Original: Toempirically evaluate the efﬁciency gains enabled by our proposed conﬁdence measures, we comparewith static baselines that use the same number of layers for all tokens.
Revised: Toempirically evaluate the efﬁciency gains enabled by our proposed conﬁdence measures, we comparewith static baselines that use the same number of layers for all tokens.

Original: We also compare ourearly-exit classiﬁer training with the geometric method of [23] in Appendix D.
Revised: We also compare ourearly-exit classiﬁer training with the geometric method of [23] in Appendix D.

Original: Also, we computean oracle local measure (§3.3.1) as an upper-bound estimate of the performance-efﬁciency tradeoff.
Revised: Also, we computean oracle local measure (§3.3.1) as an upper-bound estimate of the performance-efﬁciency tradeoff.

Original:  Experimental Results
Revised:  Experimental Results

Original: We ﬁrst report the empirical performance-efﬁciency tradeoff achieved with each conﬁdence 
Revised: We ﬁrst report the empirical performance-efﬁciency tradeoff achieved with each conﬁdence measure.

Revised: For each task and measure, we evaluate the full

Original: [Table]
Revised: [Table]

Original: The results, presented in Figure 3, show the power of the softmax response measure, allowing onlyminor performance loss while reducing more than half of the layers in all three tasks.
Revised: The results, presented in Figure 3, show the power of the softmax response measure, allowing onlyminor performance loss while reducing more than half of the layers in all three tasks.

Original: The early-exitclassiﬁer, that is more FLOP-efﬁcient, is also effective, mostly when targeting high performance(right hand side of plots).
Revised: The early-exitclassiﬁer, that is more FLOP-efﬁcient, is also effective, mostly when targeting high performance(right hand side of plots).

Original: The simple and parameter-free state saturation measure is competitive, butoften falls bellow the static baseline, despite enabling per-token exit decisions.
Revised: The simple and parameter-free state saturation measure is competitive, butoften falls bellow the static baseline, despite enabling per-token exit decisions.

Original: The dynamic oracle obtains compelling efﬁciency gains, using only 1.5, 1.3, and 1.2 layers on averagefor summarization, WMT, and QA, respectively, without losing any performance.
Revised: The dynamic oracle obtains compelling efﬁciency gains, using only 1.5, 1.3, and 1.2 layers on averagefor summarization, WMT, and QA, respectively, without losing any performance.

Original: This illustrates thefull potential of CALM and leaves further room for improvements with better conﬁdence measures.
Revised: This illustrates thefull potential of CALM and leaves further room for improvements with better conﬁdence measures.

Original: It also shows the effectiveness of inference-time state propagation for skipped layers (§3.3.1).
Revised: It also shows the effectiveness of inference-time state propagation for skipped layers (§3.3.1).

Original: Next, we examine the outcomes of the calibration process.
Revised: Next, we examine the outcomes of the calibration process.

Original: Since the obtained risk is guaranteed to bevalid (i.e.,   at least 95% of the time), we focus here on efﬁciency gains per chosen  .
Revised: Since the obtained risk is guaranteed to bevalid (i.e.,   at least 95% of the time), we focus here on efﬁciency gains per chosen  .

Original: We refer thereader to Appendix B for empirical validation and for additional results and qualitative examples.
Revised: We refer thereader to Appendix B for empirical validation and for additional results and qualitative examples.

Original: Table 2 presents the potential efﬁciency gains per choice of  for each consistency objective andconﬁdence measure.
Revised: Table 2 presents the potential efﬁciency gains per choice of  for each consistency objective andconﬁdence measure.

Original: We examine larger  values for textual consistency as this is generally a stricterrequirement since the full model’s error is not considered.
Revised: We examine larger  values for textual consistency as this is generally a stricterrequirement since the full model’s error is not considered.

Original: Across all, the softmax conﬁdence measureleads to the greatest decrease in number of decoder layers required for all conﬁgurations.
Revised: Across all, the softmax conﬁdence measureleads to the greatest decrease in number of decoder layers required for all conﬁgurations.

Original: Accordingly,softmax enables the highest speedup potential of up to about three times faster than the full model.
Revised: Accordingly,softmax enables the highest speedup potential of up to about three times faster than the full model.

Original: These encouraging efﬁciency gains are enabled even with the rigorous performance guarantees thatare sometimes conservative (e.g., Eq.
Revised: These encouraging efﬁciency gains are enabled even with the rigorous performance guarantees thatare sometimes conservative (e.g., Eq.

Original: We note that relaxing these constraints, or tightening theconﬁdence intervals (e.g., with larger calibration sets), can further improve the empricial gains.
Revised: We note that relaxing these constraints, or tightening theconﬁdence intervals (e.g., with larger calibration sets), can further improve the empricial gains.

Original: The softmax operation over the full output vocabulary is FLOPs heavy (though, this compute canpotentially be paralleled), sometime leading to increased total FLOPs, even with fewer used layers.
Revised: The softmax operation over the full output vocabulary is FLOPs heavy (though, this compute canpotentially be paralleled), sometime leading to increased total FLOPs, even with fewer used layers.

Original: The state-based and early-exit classiﬁer measures require minimal FLOPs and provide a goodalternative with compelling efﬁciency gains, if total (parallizeable, or not) FLOPs is of concern.
Revised: The state-based and early-exit classiﬁer measures require minimal FLOPs and provide a goodalternative with compelling efﬁciency gains, if total (parallizeable, or not) FLOPs is of concern.

Original:  Conclusion
Revised:  Conclusion

Original: We present conﬁdent adaptive language modeling (CALM) for dynamically allocating differentamounts of compute per generated token, following explicitly deﬁned tolerance levels on the full generation output.
Revised: We present conﬁdent adaptive language modeling (CALM) for dynamically allocating differentamounts of compute per generated token, following explicitly deﬁned tolerance levels on the full generation output.

Original: This paper covers both modeling solutions and analyses towards this goal, as well asa theoretically-grounded framework for provably controlling the quality of the full output to meet theuser-speciﬁed tolerance levels.
Revised: This paper covers both modeling solutions and analyses towards this goal, as well asa theoretically-grounded framework for provably controlling the quality of the full output to meet theuser-speciﬁed tolerance levels.

Original: We investigate the effects of local early exiting during decoding on theﬁnal output, leading us to propose a decaying function over the initial threshold that enables ﬁner control over the performance-efﬁciency tradeoffs without inﬂating the search space.
Revised: We investigate the effects of local early exiting during decoding on theﬁnal output, leading us to propose a decaying function over the initial threshold that enables ﬁner control over the performance-efﬁciency tradeoffs without inﬂating the search space.

Original: We also study different solutions for addressing missing computations of early-exited tokens that are dependent upon forfuture tokens.
Revised: We also study different solutions for addressing missing computations of early-exited tokens that are dependent upon forfuture tokens.

Original: Overall, our complete adaptive compute framework for LMs requires minimal modiﬁcations to the underlying model and enables efﬁciency gains while satisfying rigorous quality guaranteesfor the output.
Revised: Overall, our complete adaptive compute framework for LMs requires minimal modiﬁcations to the underlying model and enables efﬁciency gains while satisfying rigorous quality guaranteesfor the output.

Original: Also, our oracle experiments and runtime analysis demonstrates the full potential of thisframework and leave room for future research to further improve the efﬁciency in a controllable way.
Revised: Also, our oracle experiments and runtime analysis demonstrates the full potential of thisframework and leave room for future research to further improve the efﬁciency in a controllable way.