Original: L IQUID S TRUCTURAL S TATE -S PACE M ODELS
Revised: L IQUID S TRUCTURAL S TATE -S PACE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks.
Revised: A proper parametrization of state transition matrices of linear state-space models (SSMs) followed by standard nonlinearities enables them to efficiently learn representations from sequential data, establishing the state-of-the-art on a large series of long-range sequence modeling benchmarks.

Original: In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model.
Revised: In this paper, we show that we can improve further when the structured SSM, such as S4, is given by a linear liquid time-constant (LTC) state-space model.

Original: LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference.
Revised: LTC neural networks are causal continuous-time neural networks with an input-dependent state transition module, which makes them learn to adapt to incoming inputs at inference.

Original: We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structured state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the LongRange Arena benchmark.
Revised: We show that by using a diagonal plus low-rank decomposition of the state transition matrix introduced in S4, and a few simplifications, the LTC-based structured state-space model, dubbed Liquid-S4, achieves the new state-of-the-art generalization across sequence modeling tasks with long-term dependencies such as image, text, audio, and medical time-series, with an average performance of 87.32% on the LongRange Arena benchmark.

Original: On the full raw Speech Command recognition dataset,
Revised: On the full raw Speech Command recognition dataset,

Original: Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4.
Revised: Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameter counts compared to S4.

Original: The additional gain in performance is the direct result of the
Revised: The additional gain in performance is the direct result of the

Original: Liquid-S4’s kernel structure that takes into account the similarities of the input sequence samples during training and inference.
Revised: Liquid-S4’s kernel structure that takes into account the similarities of the input sequence samples during training and inference.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Learning representations from sequences of data requires expressive temporal and structural credit assignment.
Revised: Learning representations from sequences of data requires expressive temporal and structural credit assignment.

Original: In this space, the continuous-time neural network class of liquid time-constant networks (LTC) (Hasani et al., 2021b) has shown theoretical and empirical evidence for their expressivity and their ability to capture the cause and effect of a given task from high-dimensional sequential demonstrations (Lechner et al., 2020a;
Revised: In this space, the continuous-time neural network class of liquid time-constant networks (LTC) (Hasani et al., 2021b) has shown theoretical and empirical evidence for their expressivity and their ability to capture the cause and effect of a given task from high-dimensional sequential demonstrations (Lechner et al., 2020a;

Original: Vorbach et al., 2021).
Revised: Vorbach et al., 2021).

Original: Liquid networks are nonlinear statespace models (SSMs) with an input-dependent state transition module that enables them to learn to adapt the dynamics of the model to incoming inputs, at inference, as they are dynamic causal models (Friston et al., 2003).
Revised: Liquid networks are nonlinear statespace models (SSMs) with an input-dependent state transition module that enables them to learn to adapt the dynamics of the model to incoming inputs, at inference, as they are dynamic causal models (Friston et al., 2003).

Original: Their complexity, however, is bottlenecked by their differential equation numerical solver that limits their scalability to longer-term sequences.
Revised: Their complexity, however, is bottlenecked by their differential equation numerical solver that limits their scalability to longer-term sequences.

Original: How can we take advantage of LTC’s generalization and causality capabilities and scale them to competitively learn long-range sequences without gradient issues, compared to advanced recurrent neural networks (RNNs) (Rusch & Mishra, 2021; Erichson et al., 2021;
Revised: How can we take advantage of LTC’s generalization and causality capabilities and scale them to competitively learn long-range sequences without gradient issues, compared to advanced recurrent neural networks (RNNs) (Rusch & Mishra, 2021a; Erichson et al., 2021;

Original: Gu et al., 2020a), convolutional networks (CNNs) (Lea et al., 2016; Romero et al., 2021b; Cheng et al., 2022), and attention-based models (Vaswani et al., 2017)?
Revised: Gu et al., 2020a), convolutional networks (CNNs) (Lea et al., 2016; Romero et al., 2021b; Cheng et al., 2022), and attention-based models (Vaswani et al., 2017)?

Original: In this work, we set out to leverage the elegant formulation of structured state-space models (S4)
Revised: In this work, we set out to leverage the elegant formulation of structured state-space models (S4)

Original: (Gu et al., 2022a) to obtain linear liquid network instances that possess the approximation capabilities of both S4 and LTCs.
Revised: (Gu et al., 2022a) to obtain linear liquid network instances that possess the approximation capabilities of both S4 and LTCs.

Original: This is because structured SSMs are shown to largely dominate advanced RNNs,
Revised: This is because structured SSMs are shown to largely dominate advanced RNNs,

Original: CNNs, and Transformers across many data modalities such as text, sequence of pixels, audio, and time series (Gu et al., 2021; 2022a;b; Gupta, 2022).
Revised: CNNs, and Transformers across many data modalities such as text, sequence of pixels, audio, and time series (Gu et al., 2021; 2022a;b; Gupta, 2022).

Original: structured SSMs achieve such impressive performance by using three main mechanisms: 1) High-order polynomial projection operators (HiPPO) (Gu et al., 2020a) that are applied to state and input transition matrices to memorize signals’ history, 2) diagonal plus low-rank parametrization of the obtained HiPPO (Gu et al., 2022a), and 3) an efficient (convolution) kernel computation of an SSM’s transition matrices in the frequency domain, transformed back in time via an inverse Fourier transformation (Gu et al., 2022a).
Revised: structured SSMs achieve such impressive performance by using three main mechanisms: 1) High-order polynomial projection operators (HiPPO) (Gu et al., 2020a) that are applied to state and input transition matrices to memorize signals’ history, 2) diagonal plus low-rank parametrization of the obtained HiPPO (Gu et al., 2022a), and 3) an efficient (convolution) kernel computation of an SSM’s transition matrices in the frequency domain, transformed back in time via an inverse Fourier transformation (Gu et al., 2022a).

Original: To combine S4 and LTCs, instead of modeling sequences by linear state-space models of the form ˙ x = A x + B u , y = C x , (as done in structured and diagonal SSMs (Gu et al., 2022a;b)), we propose to use a linearized LTC state-space model (Hasani et al., 2021b), given by the following dynamics: ˙ x
Revised: To combine S4 and LTCs, instead of modeling sequences by linear state-space models of the form ˙ x = A x + B u , y = C x , (as done in structured and diagonal SSMs (Gu et al., 2022a;b)), we propose to use a linearized LTC state-space model (Hasani et al., 2021b), given by the following dynamics: ˙ x

Original: = ( A + B u ) x + B u , y =
Revised: = ( A + B u ) x + B u , y =

Original: C x .
Revised: C x .

Original: We show that this dynamical system can also be efficiently solved via the same parametrization of S4, giving rise to an additional convolutional
Revised: We show that this dynamical system can also be efficiently solved via the same parametrization of S4, giving rise to an additional convolutional

Original: Kernel that accounts for the similarities of lagged signals.
Revised: Kernel that accounts for the similarities of lagged signals.

Original: We call the obtained model LiquidS4.
Revised: We call the obtained model LiquidS4.

Original: Through extensive empirical evaluation, we show that Liquid-S4 consistently leads to better generalization performance compared to all variants of S4, CNNs, RNNs, and Transformers across many time-series modeling tasks.
Revised: Through extensive empirical evaluation, we show that Liquid-S4 consistently leads to better generalization performance compared to all variants of S4, CNNs, RNNs, and Transformers across many time-series modeling tasks.

Original: In particular, we achieve SOTA performance on the Long Range
Revised: In particular, we achieve SOTA performance on the Long Range

Original: Arena benchmark (Tay et al., 2020b).
Revised: Arena benchmark (Tay et al., 2020b).

Original: To sum up, we make the following contributions:
Revised: To sum up, we make the following contributions:

Original: We introduce Liquid-S4, a new state-space model that encapsulates the generalization and causality capabilities of liquid networks as well as the memorization and scalability of S4.
Revised: We introduce Liquid-S4, a new state-space model that encapsulates the generalization and causality capabilities of liquid networks as well as the memorization and scalability of S4.

Original: We achieve state-of-the-art performance on pixel-level sequence classification, text, speech recognition, and all six tasks of the long-range arena benchmark with an average accuracy of 87.32%.
Revised: We achieve state-of-the-art performance on pixel-level sequence classification, text, speech recognition, and all six tasks of the long-range arena benchmark with an average accuracy of 87.32%.

Original: On the full raw Speech Command recognition dataset, Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameters.
Revised: On the full raw Speech Command recognition dataset, Liquid-S4 achieves 96.78% accuracy with a 30% reduction in parameters.

Original: Finally, on the BIDMC vital signs dataset, Liquid-S4 achieves SOTA in all modes.
Revised: Finally, on the BIDMC vital signs dataset, Liquid-S4 achieves SOTA in all modes.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Learning Long-Range Dependencies with RNNs.
Revised: Learning Long-Range Dependencies with RNNs.

Original: Sequence modeling can be performed autoregressively with RNNs which possess persistent states (Little, 1974) originated from Ising (Brush, 1967) and Hopfield networks (Hopfield, 1982; Ramsauer et al., 2020).
Revised: Sequence modeling can be performed autoregressively with RNNs which possess persistent states (Little, 1974) originated from Ising (Brush, 1967) and Hopfield networks (Hopfield, 1982; Ramsauer et al., 2020).

Original: Discrete RNNs approximate continuous dynamics step-by-steps via dependencies on the history of their hidden states, and continuous-time (CT) RNNs use ordinary differential equation (ODE) solvers to unroll their dynamics with more elaborate temporal steps (Funahashi & Nakamura, 1993).
Revised: Discrete RNNs approximate continuous dynamics step-by-steps via dependencies on the history of their hidden states, and continuous-time (CT) RNNs use ordinary differential equation (ODE) solvers to unroll their dynamics with more elaborate temporal steps (Funahashi & Nakamura, 1993).

Original: CT-RNNs can perform remarkable credit assignment in sequence modeling problems both on regularly sampled, irregularly-sampled data (Pearson et al., 2003;
Revised: CT-RNNs can perform remarkable credit assignment in sequence modeling problems both on regularly sampled, irregularly-sampled data (Pearson et al., 2003;

Original: Li & Marlin, 2016; Belletti et al., 2016; Roy & Yan, 2020; Foster, 1996; Amig ´ o et al., 2012; Kowal et al., 2019), by turning the spatiotemproal dependencies into vector fields (Chen et al., 2018), enabling better generalization and expressivity (Massaroli et al., 2020; Hasani et al., 2021b).
Revised: Li & Marlin, 2016; Belletti et al., 2016; Roy & Yan, 2020; Foster, 1996; Amig ´ o et al., 2012; Kowal et al., 2019), by turning the spatiotemproal dependencies into vector fields (Chen et al., 2018), enabling better generalization and expressivity (Massaroli et al., 2020; Hasani et al., 2021b).

Original: Numerous works have studied their characteristics to understand their applicability and limitations in learning sequential data and flows (Lechner et al., 2019;
Revised: Numerous works have studied their characteristics to understand their applicability and limitations in learning sequential data and flows (Lechner et al., 2019;

Original: Dupont et al., 2019; Durkan et al., 2019; Jia & Benson, 2019; Grunbacher et al., 2021; Hanshu et al., 2020;
Revised: Dupont et al., 2019; Durkan et al., 2019; Jia & Benson, 2019; Grunbacher et al., 2021; Hanshu et al., 2020;

Original: Holl et al., 2020; Quaglino et al., 2020; Kidger et al., 2020; Hasani et al., 2020; Liebenwein et al., 2021; Gruenbacher et al., 2022).
Revised: Holl et al., 2020; Quaglino et al., 2020; Kidger et al., 2020; Hasani et al., 2020; Liebenwein et al., 2021; Gruenbacher et al., 2022).

Original: However, when these RNNs are trained by gradient descent (Rumelhart et al., 1986; Allen-Zhu & Li, 2019; Sherstinsky, 2020), they suffer from the vanishing/exploding gradients problem, which makes difficult the learning of long-term dependencies in sequences (Hochreiter, 1991; Bengio et al., 1994).
Revised: However, when these RNNs are trained by gradient descent (Rumelhart et al., 1986; Allen-Zhu & Li, 2019; Sherstinsky, 2020), they suffer from the vanishing/exploding gradients problem, which makes difficult the learning of long-term dependencies in sequences (Hochreiter, 1991; Bengio et al., 1994).

Original: This issue happens in both discrete RNNs such as GRU-D with its continuous delay mechanism (Che et al., 2018) and Phased-LSTMs (Neil et al., 2016), and continuous RNNs such as ODE-RNNs (Rubanova et al., 2019), GRU-ODE (De Brouwer et al., 2019), Log-ODE methods (Morrill et al., 2020) which compresses the input time-series by time-continuous path signatures (Friz & Victoir, 2010), and neural controlled differential equations (Kidger et al., 2020), and liquid time-constant networks (LTCs)
Revised: This issue happens in both discrete RNNs such as GRU-D with its continuous delay mechanism (Che et al., 2018) and Phased-LSTMs (Neil et al., 2016), and continuous RNNs such as ODE-RNNs (Rubanova et al., 2019), GRU-ODE (De Brouwer et al., 2019), Log-ODE methods (Morrill et al., 2020) which compresses the input time-series by time-continuous path signatures (Friz & Victoir, 2010), and neural controlled differential equations (Kidger et al., 2020), and liquid time-constant networks (LTCs)

Original: (Hasani et al., 2021b).
Revised: (Hasani et al., 2021b).

Original: Numerous solutions have been proposed to resolve these gradient issues to enable long-range dependency learning.
Revised: Numerous solutions have been proposed to resolve these gradient issues to enable long-range dependency learning.

Original: Examples include discrete gating mechanisms in LSTMs (Hochreiter & Schmidhuber, 1997; Greff et al., 2016; Hasani et al., 2019), GRUs (Chung et al., 2014), continuous gating mechanisms such as CfCs (Hasani et al., 2021a), hawks LSTMs (Mei & Eisner, 2017), IndRNNs (Li et al., 2018), state regularization (Wang & Niepert, 2019), unitary RNNs (Jing et al., 2019), dilated RNNs (Chang et al., 2017), long memory stochastic processes (Greaves-Tunnell & Harchaoui, 2019), recurrent kernel networks (Chen et al., 2019), Lipschitz RNNs (Erichson et al., 2021), symmetric skew decomposition (Wisdom et al., 2016), infinitely many updates in iRNNs (Kag et al., 2019), coupled oscillatory RNNs (coRNNs) (Rusch & Mishra, 2021), mixed-memory RNNs (Lechner & Hasani, 2021), and Legendre Memory Units (Voelker et al., 2019).
Revised: Examples include discrete gating mechanisms in LSTMs (Hochreiter & Schmidhuber, 1997; Greff et al., 2016; Hasani et al., 2019), GRUs (Chung et al., 2014), continuous gating mechanisms such as CfCs (Hasani et al., 2021a), hawks LSTMs (Mei & Eisner, 2017), IndRNNs (Li et al., 2018), state regularization (Wang & Niepert, 2019), unitary RNNs (Jing et al., 2019), dilated RNNs (Chang et al., 2017), long memory stochastic processes (Greaves-Tunnell & Harchaoui, 2019), recurrent kernel networks (Chen et al., 2019), Lipschitz RNNs (Erichson et al., 2021), symmetric skew decomposition (Wisdom et al., 2016), infinitely many updates in iRNNs (Kag et al., 2019), coupled oscillatory RNNs (coRNNs) (Rusch & Mishra, 2021a), mixed-memory RNNs (Lechner & Hasani, 2021), and Legendre Memory Units (Voelker et al., 2019).

Original: Learning Long-range Dependencies with CNNs and Transformers.
Revised: Learning Long-range Dependencies with CNNs and Transformers.

Original: RNNs are not the only solution to learning long-range dependencies.
Revised: RNNs are not the only solution to learning long-range dependencies.

Original: Continuous convolutional kernels such as CKConv (Romero et al., 2021b) and (Romero et al., 2021a), and circular dilated CNNs (Cheng et al., 2022) have shown to be efficient in modeling long sequences faster than RNNs.
Revised: Continuous convolutional kernels such as CKConv (Romero et al., 2021b) and (Romero et al., 2021a), and circular dilated CNNs (Cheng et al., 2022) have shown to be efficient in modeling long sequences faster than RNNs.

Original: There has also been a large series of works showing the effectiveness of attention-based methods for modeling spatiotemporal data.
Revised: There has also been a large series of works showing the effectiveness of attention-based methods for modeling spatiotemporal data.

Original: A large list of these models is listed in Table 1.
Revised: A large list of these models is listed in Table 6.

Original: These baselines have recently been largely outperformed by the structured state-space models (Gu et al., 2022a).
Revised: These baselines have recently been largely outperformed by the structured state-space models (Gu et al., 2022a).

Original: State-Space Models.
Revised: State-Space Models.

Original: SSMs are well-established frameworks to study deterministic and stochastic dynamical systems (Kalman, 1960).
Revised: SSMs are well-established frameworks to study deterministic and stochastic dynamical systems (Kalman, 1960).

Original: Their state and input transition matrices can be directly learned by gradient descent to model sequences of observations (Lechner et al., 2020b; Hasani et al., 2021b;
Revised: Their state and input transition matrices can be directly learned by gradient descent to model sequences of observations (Lechner et al., 2020b; Hasani et al., 2021b;

Original: Gu et al., 2021).
Revised: Gu et al., 2021).

Original: In a seminal work, Gu et al.
Revised: In a seminal work, Gu et al.

Original: (2022a) showed that with a couple of fundamental algorithmic methods on memorization and computation of input sequences, SSMs can turn into the most powerful sequence modeling framework to-date, outperforming advanced RNNs, temporal and continuous CNNs (Cheng et al., 2022; Romero et al., 2021b;a) and a wide variety of Transformers (Vaswani et al., 2017), available in Table 1 by a significant margin.
Revised: (2022a) showed that with a couple of fundamental algorithmic methods on memorization and computation of input sequences, SSMs can turn into the most powerful sequence modeling framework to-date, outperforming advanced RNNs, temporal and continuous CNNs (Cheng et al., 2022; Romero et al., 2021b;a) and a wide variety of Transformers (Vaswani et al., 2017), available in Table 6 by a significant margin.

Original: The key to their numerical performance is their derivation of higher-order polynomial projection (HiPPO) matrix (Gu et al., 2020a) obtained by a scaled Legendre measure (LegS) inspired by the
Revised: The key to their numerical performance is their derivation of higher-order polynomial projection (HiPPO) matrix (Gu et al., 2020a) obtained by a scaled Legendre measure (LegS) inspired by the

Original: Legendre Memory Units (Voelker et al., 2019) to memorize input sequences.
Revised: Legendre Memory Units (Voelker et al., 2019) to memorize input sequences.

Original: Their efficient runtime and memory is derived from their normal plus-low rank representation.
Revised: Their efficient runtime and memory is derived from their normal plus-low rank representation.

Original: It was also shown recently that diagonal SSMs (S4D) (Gupta, 2022) could be as performant as S4 in learning long sequences when parametrized and initialized properly (Gu et al., 2022b;c).
Revised: It was also shown recently that diagonal SSMs (S4D) (Gupta, 2022) could be as performant as S4 in learning long sequences when parametrized and initialized properly (Gu et al., 2022b;c).

Original: There was also a new variant of
Revised: There was also a new variant of

Original: S4 introduced as simplified-S4 (S5) Smith et al.
Revised: S4 introduced as simplified-S4 (S5) Smith et al.

Original: (2022) that tensorizes the 1-D operations of S4 to gain a more straightforward realization of SSMs.
Revised: (2022) that tensorizes the 1-D operations of S4 to gain a more straightforward realization of SSMs.

Original: Here, we introduce Liquid-S4, which is obtained by a more expressive SSM, namely liquid time-constant (LTC) representation (Hasani et al., 2021b) which achieves SOTA performance across many benchmarks.
Revised: Here, we introduce Liquid-S4, which is obtained by a more expressive SSM, namely liquid time-constant (LTC) representation (Hasani et al., 2021b) which achieves SOTA performance across many benchmarks.

Original:  S ETUP AND M ETHODOLOGY
Revised:  S ETUP AND M ETHODOLOGY

Original: In this section, we first revisit the necessary background to formulate our liquid structured statespace models.
Revised: In this section, we first revisit the necessary background to formulate our liquid structured statespace models.

Original: We then set up and sketch our technical contributions.
Revised: We then set up and sketch our technical contributions.

Original: 3. B ACKGROUND : S TRUCTURED S TATE -S PACE M ODELS (S4)
Revised: 3. B ACKGROUND : S TRUCTURED S TATE -S PACE M ODELS (S4)

Original: We aim to design an end-to-end sequence modeling framework built by SSMs.
Revised: We aim to design an end-to-end sequence modeling framework built by SSMs.

Original: A continuous-time
Revised: A continuous-time

Original: SSM representation of a linear dynamical system is given by the following set of equations:
Revised: SSM representation of a linear dynamical system is given by the following set of equations:

Original: [Equation]
Revised: [Equation]

Original: Here, x ( t ) is an N -dimensional latent state, receiving a 1-dimensional input signal u ( t ) , and computing a 1-dimensional output signal y ( t ) .
Revised: Here, x ( t ) is an N -dimensional latent state, receiving a 1-dimensional input signal u ( t ) , and computing a 1-dimensional output signal y ( t ) .

Original: A ( N × N ) , B ( N × 1) , C (1 × N ) and D (1 × 1) are system’s parameters.
Revised: A ( N × N ) , B ( N × 1) , C (1 × N ) and D (1 × 1) are system’s parameters.

Original: For the sake of brevity, throughout our analysis, we set D = 0 as it can be added eventually after construction of our main results in the form of a skip connection (Gu et al., 2022a).
Revised: For the sake of brevity, throughout our analysis, we set D = 0 as it can be added eventually after construction of our main results in the form of a skip connection (Gu et al., 2022a).

Original: Discretization of SSMs.
Revised: Discretization of SSMs.

Original: In order to create a sequence-to-sequence model similar to a recurrent neural network (RNN), we discretize the continuous-time representation of SSMs by the trapezoidal rule (bilinear transform) as follows (sampling step = δt )
Revised: In order to create a sequence-to-sequence model similar to a recurrent neural network (RNN), we discretize the continuous-time representation of SSMs by the trapezoidal rule (bilinear transform) as follows (sampling step = δt )

Original: (Gu et al., 2022a):
Revised: (Gu et al., 2022a):

Original: [Equation]
Revised: [Equation]

Original: This is obtained via the following modifications to the transition matrices:
Revised: This is obtained via the following modifications to the transition matrices:

Original: [Equation]
Revised: [Equation]

Original: With this transformation, we constructed a discretized seq-2-seq model that can map the input u k to output y k , via the hidden state x k ∈ R N .
Revised: With this transformation, we constructed a discretized seq-2-seq model that can map the input u k to output y k , via the hidden state x k ∈ R N .

Original: A is the hidden transition matrix, B and C are input and output transition matrices, respectively.
Revised: A is the hidden transition matrix, B and C are input and output transition matrices, respectively.

Original: Creating a Convolutional Representation of SSMs.
Revised: Creating a Convolutional Representation of SSMs.

Original: The system described by Eq. 2 and Eq. 3, can be trained via gradient descent to learn to model sequences, in a sequential manner which is not scalable.
Revised: The system described by Eq. 2 and Eq. 3, can be trained via gradient descent to learn to model sequences, in a sequential manner which is not scalable.

Original: To improve this, we can write the discretized SSM in Eq.
Revised: To improve this, we can write the discretized SSM in Eq.

Original: 2 as a discrete convolutional kernel.
Revised: 2 as a discrete convolutional kernel.

Original: To construct the convolutional kernel, let us unroll the system (2) in time as follows, assuming a zero initial hidden states x − 1 = 0 :
Revised: To construct the convolutional kernel, let us unroll the system of Eq. 2 in time as follows, assuming a zero initial hidden states x − 1 = 0 :

Original: [Equation]
Revised: [Equation]

Original: The mapping u k → y k can now be formulated into a convolutional kernel explicitly:
Revised: The mapping u 0 ,k → y k can now be formulated into a convolutional kernel explicitly:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 5 is a non-circular convolutional kernel.
Revised: 5 is a non-circular convolutional kernel.

Original: Gu et al.
Revised: Gu et al.

Original: (2022a) showed that under the condition that
Revised: (2022a) showed that under the condition that

Original: K is known, it can be solved very efficiently by a black-box Cauchy kernel computation pipeline.
Revised: K is known, it can be solved very efficiently by a black-box Cauchy kernel computation pipeline.

Original: Computing S4 kernel Efficiently:
Revised: Computing S4 Kernel

Original:  Gu et al.
Revised: Efficiently: Gu et al.

Original: (2022a) showed that the S4 convolution kernel could be computed efficiently using the following elegant parameterization tricks:
Revised: (2022a) showed that the S4 convolution kernel could be computed efficiently using the following elegant parameterization tricks:

Original: • To obtain better representations in sequence modeling schemes by SSMs, instead of ran- domly initializing the transition matrix A , we can use the Normal Plus Low-Rank (NPLR) matrix below, called the Hippo Matrix (Gu et al., 2020a) which is obtained by the Scaled Legendre Measure (LegS) (Gu et al., 2021; 2022a): 
Revised: • To obtain better representations in sequence modeling schemes by SSMs, instead of ran- domly initializing the transition matrix A , we can use the normal plus low-Rank (NPLR) matrix below, called the Hippo Matrix (Gu et al., 2020a) which is obtained by the scaled Legendre measure (LegS) (Gu et al., 2021; 2022a): [Equation] HiPPO Matrix

Original: [Equation]
Revised: [Equation]

Original: • The NPLR representation of this matrix is the following (Gu et al., 2022a):
Revised: • The NPLR representation of this matrix is the following (Gu et al., 2022a

Original: [Equation]
Revised: [Equation]

Original: Here, V ∈ C N × N is a unitary matrix, Λ is diagonal, and P , Q ∈ R N × r are the lowrank factorization.
Revised: Here, V ∈ C N × N is a unitary matrix, Λ is diagonal, and P , Q ∈ R N × r are the lowrank factorization.

Original: Eq.
Revised: Eq.

Original: 7 is Normal plus low rank with r = 1 (Gu et al., 2022a).
Revised: 7 is normal plus low rank with r = 1 (Gu et al., 2022a).

Original: With the decomposition (8), we can obtain A over complex numbers in the form of Diagonal plus low-rank (DPLR) (Gu et al., 2022a).
Revised: With the decomposition in Eq. 8, we can obtain A over complex numbers in the form of diagonal plus low-rank (DPLR) (Gu et al., 2022a).

Original: • Vectors B n and P n are initialized by B n = (2 n + 1) 12 and P n = ( n + 1 / 2) 

Revised: • Vectors B n and P n are initialized by B n = (2 n + 1) 12 and P n = ( n + 1 / 2)

Original: (Gu et al., 2022b).
Revised: (Gu et al., 2022b).

Original: Both vectors are trainable.
Revised: Both vectors are trainable.

Original: Furthermore, it was shown in Gu et al.
Revised: Furthermore, it was shown in Gu et al.

Original: (2022b)
Revised: (2022b)

Original: that with decomposition (8), the eigenvalues of A might be on the right half of the complex plane, resulting in numerical instability.
Revised: that with Eq. 8, the eigenvalues of A might be on the right half of the complex plane, resulting in numerical instability.

Original: To resolve this, Gu et al. (2022b) recently proposed to use the parametrization Λ − P P ∗ instead of Λ − P Q ∗ .
Revised: To resolve this, Gu et al. (2022b) recently proposed to use the parametrization Λ − P P ∗ instead of Λ − P Q ∗ .

Original: • Computing the powers of A in direct calculation of the S4 kernel K is computationally expensive.
Revised: • Computing the powers of A in direct calculation of the S4 kernel K is computationally expensive.

Original: S4 computes the spectrum of K instead of direct computations, which reduces the problem of matrix powers to matrix inverse computation Gu et al. (2022a).
Revised: S4 computes the spectrum of K instead of direct computations, which reduces the problem of matrix powers to matrix inverse computation (Gu et al., 2022a).

Original: S4 then computes this convolution kernel via a black-box Cauchy Kernel efficiently, and recovers K by an inverse Fourier Transform (iFFT) (Gu et al., 2022a).
Revised: S4 then computes this convolution kernel via a black-box Cauchy Kernel efficiently, and recovers K by an inverse Fourier Transform (iFFT) (Gu et al., 2022a).

Original: 3. L IQUID S TRUCTURAL S TATE -S PACE M ODELS
Revised: 3. L IQUID S TRUCTURAL S TATE -S PACE M ODELS

Original: In this work, we construct a convolutional kernel corresponding to a linearized version of LTCs (Hasani et al., 2021b); an expressive class of continuous-time neural networks that demonstrate attractive generalizability out-of-distribution and are dynamic causal models (Vorbach et al., 2021;
Revised: In this work, we construct a convolutional kernel corresponding to a linearized version of LTCs (Hasani et al., 2021b); an expressive class of continuous-time neural networks that demonstrate attractive generalizability out-of-distribution and are dynamic causal models (Vorbach et al., 2021;

Original: Friston et al., 2003; Hasani et al., 2020).
Revised: Friston et al., 2003; Hasani et al., 2020).

Original: In their general form, the state of a liquid time-constant network at each time-step is given by the set of ODEs described below (Hasani et al., 2021b):
Revised: In their general form, the state of a liquid time-constant network at each time-step is given by the set of ODEs described below (Hasani et al., 2021b):

Original: [Equation]
Revised: [Equation]

Original: In this expression, x ( N × 1) ( t ) is the vector of hidden state of size N , u
Revised: In this expression, x ( N × 1) ( t ) is the vector of hidden state of size N , u

Original: ( m × 1) ( t ) is an input signal with m features, A ( N × 1) is a time-constant state-transition mechanism, B ( N × 1) is a bias vector, and ⊙ represents the Hadamard product.
Revised: ( m × 1) ( t ) is an input signal with m features, A ( N × 1) is a time-constant state-transition mechanism, B ( N × 1) is a bias vector, and ⊙ represents the Hadamard product.

Original: f ( . ) is a bounded nonlinearity parametrized by θ .
Revised: f ( . ) is a bounded nonlinearity parametrized by θ .

Original: Our objective is to show how the liquid time-constant (i.e., an input-dependent state transition mechanism in state-space models can enhance its generalization capabilities by accounting for the covariance of the input samples.
Revised: Our objective is to show how the liquid time-constant (i.e., an input-dependent state transition mechanism in state-space models can enhance its generalization capabilities by accounting for the covariance of the input samples.

Original: To do this, we linearize the LTC formulation of Eq. 9 in the following to better connect the model to SSMs.
Revised: To do this, we linearize the LTC formulation of Eq. 9 in the following to better connect the model to SSMs.

Original: Let’s dive in:
Revised: Let’s dive in:

Original: Linear Liquid Time-Constant State-Space Model.
Revised: Linear Liquid Time-Constant State-Space Model.

Original: A Linear LTC SSM can be presented by the following coupled bilinear (first order bilinear Taylor approximation (Penny et al., 2005)) equation:
Revised: A Linear LTC SSM can be presented by the following coupled bilinear (first order bilinear Taylor approximation (Penny et al., 2005)) equation:

Original: [Equation]
Revised: [Equation]

Original: Similar to Eq.
Revised: Similar to Eq.

Original: 1, x ( t ) is an N -dimensional latent state, receiving a 1-dimensional input signal u ( t ) , and computing a 1-dimensional output signal y ( t ) .
Revised: 1, x ( t ) is an N -dimensional latent state, receiving a 1-dimensional input signal u ( t ) , and computing a 1-dimensional output signal y ( t ) .

Original: A ( N × N ) , B ( N × 1) , and C (1 × N ) .
Revised: A ( N × N ) , B ( N × 1) , and C (1 × N ) .

Original: Note that D is set to zero for simplicity.
Revised: Note that D is set to zero for simplicity.

Original: In Eq.
Revised: In Eq.

Original: 10, J N is an N × N unit matrix that adds B u ( t ) element-wise to
Revised: 10, J N is an N × N unit matrix that adds B u ( t ) element-wise to

Original: A .
Revised: A .

Original: This dynamical system allows the coefficient (state transition compartment) of state vector x ( t ) to be input dependent which, as a result, allows us to realize more complex dynamics.
Revised: This dynamical system allows the coefficient (state transition compartment) of state vector x ( t ) to be input dependent which, as a result, allows us to realize more complex dynamics.

Original: Discretization of Liquid-SSMs.
Revised: Discretization of Liquid-SSMs.

Original: Similar to SSMs, Liquid-SSMs can also be discretized by a bilinear transform (trapezoidal rule) to construct a sequence-to-sequence model as follows:
Revised: We can use a forward Euler transformation to discretize Eq.into the following discretization:

Original: [Equation]
Revised: [Equation]

Original: The discretized parameters  A , B , and C are identical to that of Eq.
Revised: The discretized parameters would then correspond to: A =

Original: 3, which are function of the continuous-time coefficients A , B , and C , and the discretization step δt . 
Revised: I + δt 2 A , B = δt B , and C = C , which are function of the continuous-time coefficients A , B , and C , and the discretization step δt . Given the properties of the transition matrices A and B , and ranges of δt , we could use the more stable bilinear discretization of matrices A and B , of Eq. 3 as well, as the Forward Euler discretization and the bilinear transformation of A and B presented in Eq. 3 stay close to each other.

Original: Creating a Convolutional Representation of Liquid-SSMs.
Revised: Creating a Convolutional Representation of Liquid-SSMs.

Original: Similar to Eq.
Revised: Similar to Eq.

Original: 4, we first unroll the Liquid-SSM in time to construct a convolutional kernel of it.
Revised: 4, we first unroll the Liquid-SSM in time to construct a convolutional kernel of it.

Original: By assuming x − 1 = 0 , we have:
Revised: By assuming x − 1 = 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: The resulting expressions of the Liquid-SSM at each time step consist of two types of weight configurations: 1.
Revised: The resulting expressions of the Liquid-SSM at each time step consist of two types of weight con-

Original: Weights corresponding to the mapping of individual time instances of inputs independently, shown in black in Eq.

Original: 12, and 2. Weights associated with all orders of auto-correlation of the input signal, shown in violet in Eq. The first set of weights corresponds to the convolutional kernel of the simple SSM, shown by Eq. 5 and Eq. 6, whereas the secondset leads to the design of an additional input correlation kernel, which wecall the liquid kernel.
Revised: whereas the second which we

Original: Thesekernels generate the following input-output mapping:
Revised: These generate the following [Equation]

Original: [Equation]
Revised: [Equation]

Revised: for i and

Revised: 0 , → y = K ∗ u + K liquid ∗ u correlations .

Original: Here, Π( k + 1 , p ) represents (cid:0)
Revised: Here, Π( k + 1 , p ) represents (cid:0)

Original: k +1 p (cid:1) permuted indices.
Revised: k +1 p (cid:1) permuted indices.

Original: For instance, let us assume we have a 1dimensional input signal u ( t ) of length L = 100 on which we run the liquid-SSM kernel.
Revised: For instance, let us assume we have a 1dimensional input signal u ( t ) of length L = 100 on which we run the liquid-SSM kernel.

Original: We set the hyperparameters P = 4 .
Revised: We set the hyperparameters P = 4 .

Original: This value represents the maximum order of the correlation terms wewould want to take into account to output a decision.
Revised: This value represents the maximum order of the correlation terms we

Original: This means that the signal u correlations in Eq.will contain all combinations of 2 order correlation signals (cid:0) L +12 (cid:1) , u i u j , 3 order (cid:0) L +13 (cid:1) , u i u j u k and 4 order signals (cid:0) L +14 (cid:1) , u i u j u k u l . The kernel weights corresponding to this auto-correlation signal are given in Appendix A. This additional kernel takes the temporal similarities of incoming input samples into consideration. This way, Liquid-SSM gives rise to a more general sequence modeling framework.
Revised: [Table]

Original: The liquid convolutional kernel, K liquid is as follows:

Original: [Equation]
Revised: [Table]

Original: How to compute Liquid-S4 kernel efficiently? K liquid possess similar structure to the S4 kernel. In particular, we have:
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table][Table] [Table] [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: SSM generating function (
Revised: SSM generating function (

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: is a backward identity matrix
Revised: is a backward identity matrix

Original: [Table]
Revised: [Table]

Original: ( K liquid = p ) 10: end for 11: else if Mode ==
Revised: ( K liquid = p ) 10: end for 11: else if Mode ==

Original: PB then ▷ Liquid-S4 Kernel of Eq.
Revised: PB then ▷ Liquid-S4 Kernel of Eq.

Original: 14 with A reduced to Identity.
Revised: 14 with A reduced to Identity.

Original: 12: for p in { 2 , . .
Revised: 12: for p in { 2 , . .

Original: . , P} do 13:
Revised: . , P} do 13:

Original:  Kliquid = p =
Revised: [Equation] K

Original: C ⊙ B p − 1 ( L − ˜ L,L ) 14:
Revised: [Equation] .append ( K liquid = p ) 15: end for 16: end if would want to take into account to output a decision. This means that the signal u correlations in Eq.will contain all combinations of 2 order correlation signals (cid:0) L +12 (cid:1) , u i u j , 3 order (cid:0) L +13 (cid:1) , u i u j u k and 4 order signals (cid:0) L +14 (cid:1) , u i u j u k u l .

Original: K liquid .append
Revised: The kernel weights corresponding to this auto-correlation signal are given in Appendix A. This additional kernel takes the temporal similarities of incoming input samples into consideration. This way, Liquid-SSM gives rise to a more general sequence modeling framework.

Original: ( K liquid = p ) 15: end for 16: end if
Revised: The liquid convolutional kernel, K liquid is as follows:

Original: Proposition 1.
Revised: [Equation] How to compute Liquid-S4 kernel efficiently? K liquid possess similar structure to the S4 kernel. In particular, we have:

Revised: Proposition 1.

Original: The liquid-S4 kernel for each order p ∈ P , K liquid , can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K = (cid:0) CB , CAB , . . .
Revised: The Liquid-S4 kernel for each order p ∈ P , K liquid , can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K = (cid:0) CB , CAB , . . .

Original: , CA L − 1 B (cid:1) , and a vector B p − 1 ∈ R N .
Revised: , CA L − 1 B (cid:1) , and a vector B p − 1 ∈ R N .

Original: The proof is given in Appendix.
Revised: The proof is given in Appendix.

Original: Proposition 1 indicates that the liquid-s4 kernel can be obtained from the precomputed S4 kernel and a Hadamard product of that kernel with the transition vector B powered by the chosen liquid order.
Revised: Proposition 1 indicates that the Liquid-s4 kernel can be obtained from the precomputed S4 kernel and a Hadamard product of that kernel with the transition vector B powered by the chosen liquid order.

Original: This is illustrated in Algorithm 1, lines 6 to 10, corresponding to a mode we call KB, which stands for Kernel ×
Revised: This is illustrated in Algorithm 1, lines 6 to 10, corresponding to a mode we call KB, which stands for Kernel ×

Original: B.
Revised: B.

Original: Additionally, we introduce a simplified Liquid-S4 kernel that is easier to compute while is as expressive as or even better performing than the KB kernel.
Revised: Additionally, we introduce a simplified Liquid-S4 kernel that is easier to compute while is as expressive as or even better performing than the KB kernel.

Original: To obtain this, we set the transition matrix
Revised: To obtain this, we set the transition matrix

Original: A in Liquid-S4 of Eq.
Revised: A in Liquid-S4 of Eq.

Original: 14, with an identity matrix, only for the input correlation terms.
Revised: 14, with an identity matrix, only for the input correlation terms.

Original: This way, the liquid-s4 Kernel for a given liquid order p ∈ P reduces to the following expression: 
Revised: This way, the Liquid-s4 Kernel for a given liquid order p ∈ P reduces to the following expression: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We call this kernel Liquid-S4 - PB, as it is obtained by powers of the vector B .
Revised: We call this kernel Liquid-S4 - PB, as it is obtained by powers of the vector B .

Original: The computational steps to get this kernel is outlined in Algorithm 1 lines 11 to 15.
Revised: The computational steps to get this kernel is outlined in Algorithm 1 lines 11 to 15.

Revised: [Table]

Original: Computational Complexity of the Liquid-S4 Kernel.
Revised: Computational Complexity of the Liquid-S4 Kernel.

Original: The computational complexity of the S4Legs Convolutional kernel solved via the Cauchy Kernel is ˜ O
Revised: The computational complexity of the S4Legs Convolutional kernel solved via the Cauchy Kernel is ˜ O

Original: ( N + L ) , where N is the state-size, and L is the sequence length
Revised: ( N + L ) , where N is the state-size, and L is the sequence length

Original: [Gu et al.
Revised: [Gu et al.

Original: (2022a), Theorem 3].
Revised: (2022a), Theorem 3].

Original: Liquid-S4 both in KB and PB modes can be computed in ˜ O
Revised: Liquid-S4 both in KB and PB modes can be computed in ˜ O

Original: ( N + L + p max ˜ L ) .
Revised: ( N + L + p max ˜ L ) .

Original: The added time complexity in practice is tractable.
Revised: The added time complexity in practice is tractable.

Original: This is because we usually select the liquid orders, p , to be less than 10 (typically p max = 3 , and ˜ L which is the number of terms we use to compute the input correlation vector, u correlation , is typically two orders of magnitude smaller than the seq length.[Table]
Revised: This is because we usually select the liquid orders, p , to be less than 10 (typically p max = 3 , and ˜ L which is the number of terms we use to compute the input correlation vector, u correlation , is typically two orders of magnitude smaller than the sequence length.

Original:  E XPERIMENTS WITH L IQUID -S 
Revised:  E XPERIMENTS WITH L IQUID -S

Original: In this section [Table] we present an extensive evaluation of Liquid-S4 on sequence modeling tasks with very long-term dependencies and compare its performance to a large series of baselines ranging from advanced Transformers and Convolutional networks to many variants of state-space models [Table]
Revised: In this section, we present an extensive evaluation of Liquid-S4 on sequence modeling tasks with very long-term dependencies and compare its performance to a large series of baselines ranging from advanced Transformers and Convolutional networks to many variants of state-space models.

Original: In the following [Table] we first outline the baseline models we compare against.
Revised: In the following, we first outline the baseline models we compare against.

Original: We then list the datasets we evaluated these models on and finally present results and discussions.
Revised: We then list the datasets we evaluated these models on and finally present results and discussions.

Original: Baselines.
Revised: Baselines.

Original: We consider a broad range of advanced models to compare liquid-S4 with.
Revised: We consider a broad range of advanced models to compare Liquid-S4 with.

Original: These baselines include transformer variants such as vanilla Transformer, Sparse Transformers, a Transformer model with local attention, Longformer, Linformer, Reformer, Sinkhorn Transformer, BigBird, Linear Transformer, and Performer.
Revised: These baselines include transformer variants such as vanilla and Sparse Transformers, a Transformer model with local attention, Longformer, Linformer, Reformer, Sinkhorn, BigBird, Linear Transformer, and Performer.

Original: We also include architectures such as FNets, Nystro ¨ mformer, Luna-256, H-Transformer-1D, and Circular Diluted Convolutional neural networks (CDIL).
Revised: We also include architectures such as FNets, Nystro ¨mformer, Luna-256, H-Transformer1D, and Circular Diluted Convolutional neural networks (CDIL).

Original: We then include a full series of state-space models and their variants such as diagonal SSMs (DSS), S4, S4-legS, S4-FouT, S4-LegS/FouT, S4D-LegS, S4D-Inv, S4D-Lin and the Simplified structured state-space models (S5).
Revised: We then include a full series of state-space models and their variants such as diagonal SSMs (DSS), S4, S4-legS, S4-FouT, S4LegS/FouT, S4D-LegS, S4D-Inv, S4D-Lin and the Simplified structured state-space models (S5).

Original: Datasets.
Revised: Datasets.

Original: We first evaluate Liquid-S4’s performance on the well-studied Long Range Arena (LRA) benchmark (Tay et al., 2020b), where Liquid-S4 outperforms other S4 and S4D variants in every task pushing the state-of-the-art further with an average accuracy of 87.32% .
Revised: We first evaluate Liquid-S4’s performance on the well-studied Long Range Arena (LRA) benchmark (Tay et al., 2020b), where Liquid-S4 outperforms other S4 and S4D variants in every task pushing the state-of-the-art further with an average accuracy of 87.32% .

Original: LRA dataset includes six tasks with sequence lengths ranging from 1k to 16k.
Revised: LRA dataset includes six tasks with sequence lengths ranging from 1k to 16k.

Original: We then report Liquid-S4’s performance compared to other S4, and S4D variants, as well as other models, on the BIDMC Vital Signals dataset (Pimentel et al., 2016; Goldberger et al., 2000).
Revised: We then report Liquid-S4’s performance compared to other S4, and S4D variants, as well as other models, on the BIDMC Vital Signals dataset (Pimentel et al., 2016; Goldberger et al., 2000).

Original: BIDMC uses bio-marker signals of length 4000 to predict Heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2).
Revised: BIDMC uses bio-marker signals of length 4000 to predict Heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2).

Original: We also experiment with the sCIFAR dataset that consists of the classification of flattened images in the form of 1024-long sequences into ten classes. 
Revised: We also experiment with the sCIFAR dataset that consists of the classification of flattened images in the form of 1024-long sequences into ten classes. [Table]

Original: Finally, we perform Raw Speech Command (SC) recognition with full 35 labels as conducted very recently in the updated S4 article (Gu et al., 2022a).
Revised: Finally, we perform Raw Speech Command (SC) recognition with full 35 labels as conducted very recently in the updated S4 article (Gu et al., 2022a).

Original: It is essential to denote that there is a modified speech command dataset that restricted the dataset to only ten output classes and is used in a couple of works (see for example Kidger et al.
Revised: It is essential to denote that there is a modified speech command dataset that restricted the dataset to only ten output classes and is used in a couple of works (see for example Kidger et al.

Original: Gu et al.
Revised: Gu et al.

Original: ; Romero et al.
Revised: ; Romero et al.

Original: (2021b;a)).
Revised: (2021b;a)).

Original: [Table]

Original: Aligned with the updated results reported in Gu et al.
Revised: Aligned with the updated results reported in Gu et al.

Original: (2022a)
Revised: (2022a)

Original: and Gu et al.
Revised: and Gu et al.

Original: (2022b)
Revised: (2022b)

Original: , we choose not to break down this dataset and use the full-sized benchmark.
Revised: , we choose not to break down this dataset and use the full-sized benchmark.

Original: SC dataset contains sequences of length 16k to be classified into 35 commands.
Revised: SC dataset contains sequences of length 16k to be classified into 35 commands.

Original: Gu et al.
Revised: Gu et al.

Original: (2022a) introduced a new test case setting to assess the performance of models (trained on 16kHz sequences) on sequences of length 8kHz.
Revised: (2022a) introduced a new test case setting to assess the performance of models (trained on 16kHz sequences) on sequences of length 8kHz.

Original: S and S4D perform exceptionally well in this zero-shot test scenario.
Revised: S4 and S4D perform exceptionally well in this zero-shot test scenario.

Original: [Figure]
Revised: [Figure]

Revised: Order

Revised: Liquid

Revised: ListOps, and B) IMDB datasets.

Revised: More in Appendix.

Revised: (n=3)

Original: 4.1 Results on Long Range Arena
Revised: 4.1 Results on Long Range Arena

Original: Table 1 depicts a comprehensive list of baselines benchmarked against each other on six long-range sequence modeling tasks in LRA.
Revised: Table 6 depicts a comprehensive list of baselines benchmarked against each other on six long-range sequence modeling tasks in LRA.

Original: We observe that Liquid-S4 instances (all use the PB kernel with a scaled Legendre (LegS) configuration) with a small liquid order, p , ranging fromto 6, consistently outperform all baselines in all six tasks, establishing the new SOTA on LRA with an average performance of 87.32% .
Revised: We observe that Liquid-S4 instances (all use the PB kernel with a scaled Legendre (LegS) configuration) with a small liquid order, p , ranging fromto 6, consistently outperform all baselines in all six tasks, establishing the new SOTA on LRA with an average performance of 87.32% .

Original: In particular, on ListOps, Liquid-S4 improves S4-LegS performance by more than 3%, on character-level IMDB by 2.2%, and on 1-D pixel-level classification (CIFAR) by 0.65%, while establishing the-state-of the-art on the hardest LRA task by gaining 96.66% accuracy.
Revised: In particular, on ListOps, Liquid-S4 improves S4-LegS performance by more than 3%, on character-level IMDB by 2.2%, and on 1-D pixel-level classification (CIFAR) by 0.65%, while establishing the-state-of the-art on the hardest LRA task by gaining 96.66% accuracy.

Original: Liquid-S4 performs on par with improved S4 and S4D instances on both AAN and Pathfinder tasks.
Revised: Liquid-S4 performs on par with improved S4 and S4D instances on both AAN and Pathfinder tasks.

Original: The performance of SSM models is generally well-beyond what advanced Transformers, RNNs, and
Revised: The performance of SSM models is generally well-beyond what advanced Transformers, RNNs, and

Original: Convolutional networks achieve on LRA tasks, with the Liquid-S4 variants standing on top.
Revised: Convolutional networks achieve on LRA tasks, with the Liquid-S4 variants standing on top.

Original: The impact of increasing Liquid Order p .
Revised: The impact of increasing Liquid Order p .

Original: Figure 1 illustrates how increasing the liquid order, p , can improve performance on ListOps and IMDB tasks from LRA.
Revised: Figure 1 illustrates how increasing the liquid order, p , can improve performance on ListOps and IMDB tasks from LRA (More results in Appendix).

Original: 4.2 Results on BIDMC Vital Signs
Revised: 4.2 Results on BIDMC Vital Signs

Original: Table 2 demonstrates the performance of a variety of classical and advanced baseline models on the BIDMC dataset for all three heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2) level prediction tasks.
Revised: Table 2 demonstrates the performance of a variety of classical and advanced baseline models on the BIDMC dataset for all three heart rate (HR), respiratory rate (RR), and blood oxygen saturation (SpO2) level prediction tasks.

Original: We observe that Liquid-s4 with a PB kernel of order p = 3 , p = 2 , and p = 4 , perform better than all S4 and S4D variants.
Revised: We observe that Liquid-s4 with a PB kernel of order p = 3 , p = 2 , and p = 4 , perform better than all S4 and S4D variants.

Original: It is worth denoting that Liquid-S4 is built by the same parametrization as S4-LegS (which is the official S4 model reported in the updated S report (Gu et al., 2022a)).
Revised: It is worth denoting that Liquid-S4 is built by the same parametrization as S4-LegS (which is the official S4 model reported in the updated S report (Gu et al., 2022a)).

Original: In RR, Liquid-S4 outperforms S4-LegS by a significant margin of 36%.
Revised: In RR, Liquid-S4 outperforms S4-LegS by a significant margin of 36%.

Original: On SpO2, Liquid-S4 performs 26.67% better than S4-Legs.
Revised: On SpO2, Liquid-S4 performs 26.67% better than S4-Legs.

Original: On HR, Liquid-S4 outperforms S4-Legs by 8.7% improvement in performance.
Revised: On HR, Liquid-S4 outperforms S4-Legs by 8.7% improvement in performance.

Original: [Table]
Revised: [Table]

Original: 4.3 Results on Image Classification
Revised: 4.3 Results on Image Classification

Original: Similar to the previous tasks [Table] a Liquid-S4 network with PB kernel of order p = 3 outperforms all variants of S4 and S4D while being significantly better than Transformer and RNN baselines as summarized in Table 3.
Revised: Similar to the previous tasks, a Liquid-S4 network with PB kernel of order p = 3 outperforms all variants of S4 and S4D while being significantly better than Transformer and RNN baselines as summarized in Table 3.

Original: 4.4 Results on Speech Commands
Revised: 4.4 Results on Speech Commands

Original: Table 4 demonstrates that Liquid-S4 withliquid order p = 2 achieves the best performance amongst all benchmarks on the 16KHz testbed with a full dataset.
Revised: Table 4 demonstrates that Liquid-S4 with p = 2 achieves the best performance amongst all benchmarks on the 16KHz testbed.

Original: Liquid-S4 also performs competitively on the half-frequency zero-shot
Revised: LiquidS4 also performs competitively on the halffrequency zero-

Original: experiment, while it does not realize the best performance.
Revised: shot experiment , while it does not realize the best performance.

Original: Although the task is solved to a great degree, the reason could be that liquid kernel accounts for covariance terms.
Revised: Although the task is solved to a great degree, the reason could be that liquid kernel accounts for covariance terms.

Original: This might influence the learned representations in a way that hurts performance by a smallmargin in this zero-shot experiment.
Revised: This might influence the learned repre - sentations in a way that hurts performance by a small

Revised: margin

Original:  The hyperparameters are given in Appendix.
Revised: in this zero -shot experiment. The hyperparameters are given in Appendix

Original: It is essential to denote that there is a modified speech command dataset that restricts the dataset to only ten output classes, namely SC10, and is used in a couple of works (see for example (Kidger et al., 2020; Gu et al., 2021; Romero et al., 2021b;a)).
Revised: It is essential to denote that there is a modified speech command dataset that restricts the dataset to only ten output classes, namely SC10, and is used in a couple of works (see for example (Kidger et al., 2020; Gu et al., 2021; Romero et al., 2021b;a)).

Original: Aligned with the updated results reported in (Gu et al., 2022a) and (Gu et al., 2022b), we choose not to break down this dataset and report the full-sized benchmark in the main paper.
Revised: Aligned with the updated results reported in (Gu et al., 2022a) and (Gu et al., 2022b), we choose not to break down this dataset and report the full-sized benchmark in the main paper.

Original: Nevertheless, we conducted an experiment with SC10 and showed that even on the reduced dataset, with the same hyperparameters, we solved the task with a
Revised: Nevertheless, we conducted an experiment with SC10 and showed that even on the reduced dataset, with the same hyperparameters, we solved the task with a

Original: SOTA accuracy of 98.51%.
Revised: SOTA accuracy of 98.51%.

Original: The results are presented in Table 6.
Revised: The results are presented in Table 7.

Original:  CONCLUSIONS
Revised:  CONCLUSIONS

Original: We showed that the performance of structured state-space models could be considerably improved if they are formulated by a linear liquid time-constant kernel, namely Liquid-S4.
Revised: We showed that the performance of structured state-space models could be considerably improved if they are formulated by a linear liquid time-constant kernel, namely Liquid-S4.

Original: Liquid-S4 kernels are obtainable with minimal effort, with their kernel computing the similarities between time-lags of the input signals in addition to the main S4 diagonal plus low-rank parametrization.
Revised: Liquid-S4 kernels are obtainable with minimal effort, with their kernel computing the similarities between time-lags of the input signals in addition to the main S4 diagonal plus low-rank parametrization.

Original: Liquid-S4 kernels with smaller parameter counts achieve SOTA performance on all six tasks of the Long-range arena dataset, on BIDMC heart rate, respiratory rate, and blood oxygen saturation, on sequential 1-D pixellevel image classification, and on Speech command recognition.
Revised: Liquid-S4 kernels with smaller parameter counts achieve SOTA performance on all six tasks of the Long-range arena dataset, on BIDMC heart rate, respiratory rate, and blood oxygen saturation, on sequential 1-D pixellevel image classification, and on Speech command recognition.

Original: As a final note, our experimental evaluations suggest that for challenging multivariate time series and modeling complex signals with long-range dependencies, SSM variants such as Liquid-S4 dominate other baselines, while for image and text data, a combination of SSMs and attention might enhance model quality. 
Revised: As a final note, our experimental evaluations suggest that for challenging multivariate time series and modeling complex signals with long-range dependencies, SSM variants such as Liquid-S4 dominate other baselines, while for image and text data, a combination of SSMs and attention might enhance model quality. 2022b)

Original: [Table]
Revised: [Table]

Original: A E XAMPLE L IQUID -S4 K ERNEL
Revised: A E XAMPLE L IQUID -S4 K ERNEL

Original: [Equation]
Revised: [Equation]

Original: Here, u correlations is a vector of length (cid:0) k +12 (cid:1) + (cid:0) k +13 (cid:1) + (cid:0)
Revised: Here, u correlations is a vector of length (cid:0) k +12 (cid:1) + (cid:0) k +13 (cid:1) + (cid:0)

Original: k
Revised: k

Original: +14 (cid:1) , and the kernel K liquid ∈
Revised: +14 (cid:1) , and the kernel K liquid ∈

Original: R ( k +12 )
Revised: R ( k +12 )

Original: + ( k +13 )
Revised: + ( k +13 )

Original: + ( k +14 ) .
Revised: + ( k +14 ) .

Original: B P ROOF OF P ROPOSITION
Revised: B P ROOF OF P ROPOSITION

Original: Proposition.
Revised: Proposition.

Original: The liquid-S4 kernel for each order p ∈ P , K liquid , can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K = (cid:0) CB , CAB , . . .
Revised: The Liquid-S4 kernel for each order p ∈ P , K liquid , can be computed by the anti-diagonal transformation (flip operation) of the product of the S4 convolution kernel, K = (cid:0) CB , CAB , . . .

Original: , CA L − 1 B (cid:1) , and a vector B p − 1 ∈ R N .
Revised: , CA L − 1 B (cid:1) , and a vector B p − 1 ∈ R N .

Original: Proof.
Revised: Proof.

Original: This can be shown by unrolling the S4 convolution kernel and multiplying its components with B p − 1 , performing an anti-diagonal transformation to obtain the corresponding liquid S4 kernel:
Revised: This can be shown by unrolling the S4 convolution kernel and multiplying its components with B p − 1 , performing an anti-diagonal transformation to obtain the corresponding liquid S4 kernel:

Original: [Equation]
Revised: [Equation]

Original: For p = 2 (correlations of order 2), S4 kernel should be multiplied by B .
Revised: For p = 2 (correlations of order 2), S4 kernel should be multiplied by B .

Original: The resulting kernel would be:
Revised: The resulting kernel would be:

Original: [Equation]
Revised: [Equation]

Original: We obtain the liquid kernel by flipping the above kernel to be convolved with the 2-term correlation terms (p=2):
Revised: We obtain the liquid kernel by flipping the above kernel to be convolved with the 2-term correlation terms (p=2):

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can obtain liquid kernels for higher liquid orders and obtain the statement of the proposition.
Revised: Similarly, we can obtain liquid kernels for higher liquid orders and obtain the statement of the proposition.

Original: C H YPERPARAMETERS
Revised: C H YPERPARAMETERS

Original: Learning Rate.
Revised: Learning Rate.

Original: Liquid-S4 generally requires a smaller learning rate compared to S4 and S4D blocks.
Revised: Liquid-S4 generally requires a smaller learning rate compared to S4 and S4D blocks.

Original: Setting ∆ t max and ∆ t min We set ∆ t max for all experiments to 0.2, while the ∆ t min was set based on the recommendations provided in (Gu et al., 2022c) to be proportional to ∝ 1 seq length .
Revised: Setting ∆ t max and ∆ t min We set ∆ t max for all experiments to 0.2, while the ∆ t min was set based on the recommendations provided in (Gu et al., 2022c) to be proportional to ∝ 1 seq length .

Original: Causal Modeling vs. Bidirectional Modeling Liquid-S4 works better when it is used as a causal model, i.e., with no bidirectional configuration.
Revised: Causal Modeling vs. Bidirectional Modeling Liquid-S4 works better when it is used as a causal model, i.e., with no bidirectional configuration.

Original: d s tate We observed that liquid-S4 PB kernel performs best with smaller individual state sizes d s tate .
Revised: d s tate We observed that Liquid-S4 PB kernel performs best with smaller individual state sizes d s tate .

Original: For instance, we achieve SOTA results in ListOps, IMDB, and Speech Commands by a state size set to 7, significantly reducing the number of required parameters to solve these tasks.
Revised: For instance, we achieve SOTA results in ListOps, IMDB, and Speech Commands by a state size set to 7, significantly reducing the number of required parameters to solve these tasks.

Original: Choice of Liquid-S4 Kernel
Revised: Choice of Liquid-S4 Kernel

Original: In all experiments, we choose our simplified PB kernel over the KB kernel due to the computational costs and performance.
Revised: In all experiments, we choose our simplified PB kernel over the KB kernel due to the computational costs and performance.

Original: We recommend the use of PB kernel.
Revised: We recommend the use of PB kernel.

Original: Choice of parameter p in liquid kernel.
Revised: Choice of parameter p in liquid kernel.

Original: In all experiments, start off by setting p or the liquidity order to 2.
Revised: In all experiments, start off by setting p or the liquidity order to 2.

Original: This means that the liquid kernel is going to be computed only for correlation terms of order 2.
Revised: This means that the liquid kernel is going to be computed only for correlation terms of order 2.

Original: In principle, we observe that higher p values consistently enhance the representation learning capacity of liquid-S4 modules, as we showed in all experiments.
Revised: In principle, we observe that higher p values consistently enhance the representation learning capacity of Liquid-S4 modules, as we showed in all experiments.

Original: We recommend p = 3 as a norm to perform experiments with Liquid-S4. 
Revised: We recommend p = 3 as a norm to perform experiments with Liquid-S4. [Table]

Original: The kernel computation pipeline uses the PyKeops package (Charlier et al., 2021) for large tensor computations without memory overflow.
Revised: The kernel computation pipeline uses the PyKeops package (Charlier et al., 2021) for large tensor computations without memory overflow.

Original: All reported results are validation accuracy (similar to Gu et al.
Revised: All reported results are validation accuracy (similar to Gu et al.

Original: (2022a)) performed with 2 todifferent random seeds, except for the BIDMC dataset, which reports accuracy on the test set.
Revised: (2022a)) performed with 2 todifferent random seeds, except for the BIDMC dataset, which reports accuracy on the test set.

Original: [Table]
Revised: [Table]

Original: dataset.

Original: Numbers indicate validation accuracy.

Original: The accuracy of baseline models is reported from

Original: Table 5 of (Gu et al., 2022a). x standsfor infeasible computation on a single GPU or not applicable as stated in Table 10 of (Gu et al., 2022a).
Revised: Table 5 of ( et al., 2022a) stands on a single GPU or not applicable as stated in Table 10 of (Gu et al., 2022a).

Original: The hyperparameters for Liquid-S4 are the same as the ones reported for Speech Commands Full Dataset reported in Table5.
Revised: The hyperparameters for Liquid-S4 are the same as the ones reported for Speech Commands Full Dataset reported in Table

Original: [Table] [Figure]
Revised: [Table] [Figure]