Original: ATLAS: Universal Function Approximator for Memory Retention
Revised: ATLAS: Universal Function Approximator for Memory Retention

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Artificial neural networks (ANNs), despite their universal function approximationcapability and practical success, are subject to catastrophic forgetting.
Revised: Artificial neural networks (ANNs), despite their universal function approximationcapability and practical success, are subject to catastrophic forgetting.

Original: Catastrophicforgetting refers to the abrupt unlearning of a previous task when a new task islearned.
Revised: Catastrophicforgetting refers to the abrupt unlearning of a previous task when a new task islearned.

Original: It is an emergent phenomenon that plagues ANNs and hinders continuallearning.
Revised: It is an emergent phenomenon that plagues ANNs and hinders continuallearning.

Original: Existing universal function approximation theorems for ANNs guaranteefunction approximation ability, but seldom touch on the model details and do notpredict catastrophic forgetting.
Revised: Existing universal function approximation theorems for ANNs guaranteefunction approximation ability, but seldom touch on the model details and do notpredict catastrophic forgetting.

Original: This paper presents a novel universal approximationtheorem for multi-variable functions using only single-variable functions andexponential functions.
Revised: This paper presents a novel universal approximationtheorem for multi-variable functions using only single-variable functions andexponential functions.

Original: Furthermore, we present ATLAS —a novel ANN architecturebased on the exponential approximation theorem and B-splines.
Revised: Furthermore, we present ATLAS —a novel ANN architecturebased on the exponential approximation theorem and B-splines.

Original: It is shown that
Revised: It is shown that

Original: ATLAS is a universal function approximator capable of memory retention and,therefore, continual learning.
Revised: ATLAS is a universal function approximator capable of memory retention and,therefore, continual learning.

Original: The memory retention of ATLAS is imperfect,with some off-target effects during continual learning, but it is well-behaved andpredictable.
Revised: The memory retention of ATLAS is imperfect,with some off-target effects during continual learning, but it is well-behaved andpredictable.

Original: An efficient implementation of ATLAS is provided.
Revised: An efficient implementation of ATLAS is provided.

Original: Experimentsare conducted to evaluate both the function approximation and memory retentioncapabilities of ATLAS.
Revised: Experimentsare conducted to evaluate both the function approximation and memory retentioncapabilities of ATLAS.

Original:  Introduction
Revised:  Introduction

Original: Catastrophic forgetting [7, 13, 22] is an emergent phenomenon where an ANN learns a new task andthe subsequent parameter updates interfere with the model’s performance on previously learned tasks.
Revised: Catastrophic forgetting [7, 13, 22] is an emergent phenomenon where an ANN learns a new task andthe subsequent parameter updates interfere with the model’s performance on previously learned tasks.

Original: Catastrophic forgetting is also called catastrophic interference [19].
Revised: Catastrophic forgetting is also called catastrophic interference [19].

Original: If an ANN cannot effectivelylearn many tasks, it has limited utility in the context of continual learning [9, 12].
Revised: If an ANN cannot effectivelylearn many tasks, it has limited utility in the context of continual learning [9, 12].

Original: Catastrophicforgetting is like learning to pick up a cup, but simultaneously forgetting how to breathe.
Revised: Catastrophicforgetting is like learning to pick up a cup, but simultaneously forgetting how to breathe.

Original: Even linearfunctions are susceptible to catastrophic forgetting, as illustrated in Figure 

Revised: Even linearfunctions are susceptible to catastrophic forgetting, as illustrated in Figure

Original: The simple example of a linear regression model being susceptible to catastrophic forgetting mightbe due to the non-linearity of the target function, noise in the data or parameter sharing across the input.
Revised: The simple example of a linear regression model being susceptible to catastrophic forgetting mightbe due to the non-linearity of the target function, noise in the data or parameter sharing across the input.

Original: Avoiding parameter sharing is easily accomplsihed with piece-wise defined functions such assplines [26].
Revised: Avoiding parameter sharing is easily accomplsihed with piece-wise defined functions such assplines [26].

Original: ANNs can be explained in many different ways; a useful analogy would be to compare
Revised: ANNs can be explained in many different ways; a useful analogy would be to compare

Original: ANNs to very large lookup tables that store information.
Revised: ANNs to very large lookup tables that store information.

Original: Removing and updating values has off-targeteffects throughout the table or ANN.
Revised: Removing and updating values has off-targeteffects throughout the table or ANN.

Original: Universal function approximation theorems are a cornerstone of machine learning, and prove thatartificial neural networks (ANNs) can approximate any given continuous target function [10, 11, 15].
Revised: Universal function approximation theorems are a cornerstone of machine learning, and prove thatartificial neural networks (ANNs) can approximate any given continuous target function [10, 11, 15].

Original: The theorems do not specify how to find an ANN with sufficient performance for problems in practice.
Revised: The theorems do not specify how to find an ANN with sufficient performance for problems in practice.

Original: Gradient descent optimisation is the standard practice for finding/training neural networks, but otheroptimisation and learning procedures exist.
Revised: Gradient descent optimisation is the standard practice for finding/training neural networks, but otheroptimisation and learning procedures exist.

Original: ATLAS models trained with gradient descent methodsexhibit desirable properties.
Revised: ATLAS models trained with gradient descent methodsexhibit desirable properties.

Original: However, other optimisation techniques based on evolutionary algorithmsmight not exhibit the same properties in general for ATLAS models.
Revised: However, other optimisation techniques based on evolutionary algorithmsmight not exhibit the same properties in general for ATLAS models.

Original: This paper introduces ATLAS—a novel universal function approximator based on B-splines thathas intrinsic memory retention, even in the absence of other training and regularisation techniques.
Revised: This paper introduces ATLAS—a novel universal function approximator based on B-splines thathas intrinsic memory retention, even in the absence of other training and regularisation techniques.

Original: ATLAS has well-behaved parameter gradients that are sparse, bounded and orthogonal betweeninput points that are far enough from each other.
Revised: ATLAS has well-behaved parameter gradients that are sparse, bounded and orthogonal betweeninput points that are far enough from each other.

Original: The accompanying representation and universalapproximation theorems are also provided.
Revised: The accompanying representation and universalapproximation theorems are also provided.

Original:  Relevant Studies
Revised:  Relevant Studies

Original: It is conjectured that overlapping representations in ANNs lead to catastrophic forgetting [12].
Revised: It is conjectured that overlapping representations in ANNs lead to catastrophic forgetting [12].

Original: Catastrophic forgetting occurs when parameters necessary for one task change while training to meetthe objectives of another task [14, 20].
Revised: Catastrophic forgetting occurs when parameters necessary for one task change while training to meetthe objectives of another task [14, 20].

Original: The least desirable strategy to mitigate catastrophic forgettingis retraining a model over all tasks.
Revised: The least desirable strategy to mitigate catastrophic forgettingis retraining a model over all tasks.

Original: Data augmentation approaches such as rehearsal and pseudorehearsal have also been employed [22].
Revised: Data augmentation approaches such as rehearsal and pseudorehearsal have also been employed [22].

Original: Other ideas from optimal control theory in combinationwith dynamic programming have also been applied to counteract catastrophic forgetting, with a costfunctional similar in form to the action integral from physics and Lagrangian mechanics [16].
Revised: Other ideas from optimal control theory in combinationwith dynamic programming have also been applied to counteract catastrophic forgetting, with a costfunctional similar in form to the action integral from physics and Lagrangian mechanics [16].

Original: Orthogonal Gradient Descent (OGD) is a training augmentation or optimisation technique thatmodifies the gradient updates of subsequent tasks to be orthogonal to previous tasks [6, 1].
Revised: Orthogonal Gradient Descent (OGD) is a training augmentation or optimisation technique thatmodifies the gradient updates of subsequent tasks to be orthogonal to previous tasks [6, 1].

Original: Onecan describe data in terms of a distribution defined over the input space, target values, and time (theorder of data or tasks that are presented during training).
Revised: Onecan describe data in terms of a distribution defined over the input space, target values, and time (theorder of data or tasks that are presented during training).

Original: OGD attempts to make gradient updatesorthogonal to each other over time.
Revised: OGD attempts to make gradient updatesorthogonal to each other over time.

Original: ATLAS, in contrast, possesses distal orthogonality, meaning thatif two inputs are far enough from each other in the input space, then corresponding gradient updateswill be orthogonal.
Revised: ATLAS, in contrast, possesses distal orthogonality, meaning thatif two inputs are far enough from each other in the input space, then corresponding gradient updateswill be orthogonal.

Original: A corollary of this is that if the data distribution between tasks shifts in the inputspace, then the subsequent gradient updates will tend to be orthogonal.
Revised: A corollary of this is that if the data distribution between tasks shifts in the inputspace, then the subsequent gradient updates will tend to be orthogonal.

Original: ATLAS does not use externalmemory like OGD.
Revised: ATLAS does not use externalmemory like OGD.

Original: Extensions of OGD include PCA-OGD, which compresses gradient updates intoprincipal components to reduce memory requirements [4].
Revised: Extensions of OGD include PCA-OGD, which compresses gradient updates intoprincipal components to reduce memory requirements [4].

Original: The Neural Tangent Kernel (NTK) overlapmatrices, as discussed by Doan et al.
Revised: The Neural Tangent Kernel (NTK) overlapmatrices, as discussed by Doan et al.

Original: [4], could be a useful tool for analysing ATLAS architectures.
Revised: [4], could be a useful tool for analysing ATLAS models.

Original: The survey by Delange et al.
Revised: The survey by Delange et al.

Original: [3] gives an extensive overview of continual learning to addresscatastrophic forgetting.
Revised: [3] gives an extensive overview of continual learning to addresscatastrophic forgetting.

Original: ATLAS is a model that implements parameter isolation, because of its use ofpiece-wise defined splines.
Revised: ATLAS is a model that implements parameter isolation, because of its use ofpiece-wise defined splines.

Original: Particularly relevant to ATLAS is the work on scale of initialisation andextreme memorisation [21].
Revised: Particularly relevant to ATLAS is the work on scale of initialisation andextreme memorisation [21].

Original: Increasing the density of basis functions in ATLAS can lead to bettermemorisation, and increases the scale of some parameters in ATLAS.
Revised: Increasing the density of basis functions in ATLAS can lead to bettermemorisation, and increases the scale of some parameters in ATLAS which may affect generalisation.

Original: Pi-sigma neural networks use nodes that compute products instead of sums [25].
Revised: Pi-sigma neural networks use nodes that compute products instead of sums [25].

Original: Pi-sigma neuralnetworks have some similarities with the global structure of ATLAS.
Revised: Pi-sigma neuralnetworks have some similarities with the global structure of ATLAS.

Original: B-splines, which form the basisof ATLAS, have been applied for machine learning [5].
Revised: B-splines, which form the basisof ATLAS, have been applied for machine learning [5].

Original: Scardapane et al.
Revised: Scardapane et al.

Original: [24] investigated trainableactivation functions parameterised by splines.
Revised: [24] investigated trainableactivation functions parameterised by splines.

Original: Uniform cubic B-splines have basis functions that aretranslates of one another [2].
Revised: Uniform cubic B-splines have basis functions that aretranslates of one another [2].

Original: Uniform cubic B-splines have been tested for memory retention, and
Revised: Uniform cubic B-splines have been tested for memory retention, and

Original: ATLAS can be considered as an improvement on existing spline models [26].
Revised: ATLAS can be considered as an improvement on existing spline models [26].

Original: B-splines, and by extension ATLAS, can be trained to fit lower frequency components, expanded andtrained again until a network is found with sufficient accuracy and generalisation, similar to othertechniques [17, 18].
Revised: B-splines, and by extension ATLAS, can be trained to fit lower frequency components, expanded andtrained again until a network is found with sufficient accuracy and generalisation, similar to othertechniques [17, 18].

Original: It is not necessary to expand the capacity of an ATLAS model to learn newtasks, as with some other approaches [23].
Revised: It is not necessary to expand the capacity of an ATLAS model to learn newtasks, as with some other approaches [23].

Original: ATLAS does in practice demonstrate something akin to"graceful forgetting" asdiscussed in Golkar
Revised: ATLAS does in practice demonstrate something akin to"graceful forgetting" as

Original:  Notation
Revised:  Notation

Original: Vector quantities like ⃗ x are clearly indicated with a bar or arrow for legibility.
Revised: Vector quantities like ⃗ x are clearly indicated with a bar or arrow for legibility.

Original: Parameters, inputs,functions etc.
Revised: Parameters, inputs,functions etc.

Original: without a bar or arrow are scalar quantities like S ( x ) .
Revised: without a bar or arrow are scalar quantities like S ( x ) .

Original: Some scalar quantities withindices are the scalar components of a vector like x j or scalar parameters in the model like θ i .
Revised: Some scalar quantities withindices are the scalar components of a vector like x j or scalar parameters in the model like θ i .

Original: Thegradient operator that acts on a scalar function like ⃗ ∇ ⃗θ A ( ⃗ x ) yields a vector-valued function ⃗ ∇ ⃗θ A ( ⃗ x )as is typical of multi-variable calculus.
Revised: Thegradient operator that acts on a scalar function like ⃗ ∇ ⃗θ A ( ⃗ x ) yields a vector-valued function ⃗ ∇ ⃗θ A ( ⃗ x )as is typical of multi-variable calculus.

Original:  Exponential Representation Theorem
Revised:  Exponential Representation Theorem

Original: Any continuous multi-variable function on a compact space can be uniformly approximated withmulti-variable polynomials by the Stone-Weierstrass Theorem.
Revised: Any continuous multi-variable function on a compact space can be uniformly approximated withmulti-variable polynomials by the Stone-Weierstrass Theorem.

Original: Let I denote an index set of tuples ofnatural numbers including zero such that i j ∈ N 0 for all j ∈ N with i =
Revised: Let I denote an index set of tuples ofnatural numbers including zero such that i j ∈ N 0 for all j ∈ N with i =

Original: ( i 1 , .., i n ) ∈ I and a i ∈ R .
Revised: ( i 1 , .., i n ) ∈ I and a i ∈ R .

Original: Multi-variable polynomials can be represented as:
Revised: Multi-variable polynomials can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Each monomial term a i Π nj
Revised: Each monomial term a i Π nj

Original: =1 x i j j is a product of single-variable functions in each variable.
Revised: =1 x i j j is a product of single-variable functions in each variable.

Original: It isdesirable to rewrite products as sums. The product of single-variable functions can be rewritten as asum using exponentials and logarithms.
Revised: It isdesirable to rewrite products as sums using exponentials and logarithms.

Original: Lemma 1.
Revised: Lemma 1.

Original: For any a i ∈ R , there exists γ i > 0 and β i > 0 , such that: a i = γ i − β
Revised: For any a i ∈ R , there exists γ i > 0 and β i > 0 , such that: a i = γ i − β

Original: i
Revised: i

Original: Theorem 1 (Exponential representation theorem) .
Revised: Theorem 1 (Exponential representation theorem) .

Original: Any multi-variable polynomial function y ( ⃗ x )of n variables over the positive orthant, can be exactly represented by continuous single-variablefunctions g i,j ( x j ) and h i,j ( x j ) in the form:
Revised: Any multi-variable polynomial function y ( ⃗ x )of n variables over the positive orthant, can be exactly represented by continuous single-variablefunctions g i,j ( x j ) and h i,j ( x j ) in the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any monomial term a i Π nj =1 x i j j with a i ∈ R , then by Lemma 1 there exist strictlypositive numbers γ i > 0 and β i > 0 , such that:
Revised: Consider any monomial term a i Π nj =1 x i j j with a i ∈ R , then by Lemma 1 there exist strictlypositive numbers γ i > 0 and β i > 0 , such that:

Original: [Equation]
Revised: [Equation]

Original: The argument of each exponential function is a sum of single-variable functions and constants.
Revised: The argument of each exponential function is a sum of single-variable functions and constants.

Original: Without loss of generality, a set of single-variable functions can be defined such that:
Revised: Without loss of generality, a set of single-variable functions can be defined such that:

Original: [Equation]
Revised: [Equation]

Original: Since this holds for any a i Π nj =1 x i j j and all i ∈
Revised: Since this holds for any a i Π nj =1 x i j j and all i ∈

Original: I , it follows that:
Revised: I , it follows that:

Original: [Equation]
Revised: [Equation]

Original: This result is fundamental to the paper.
Revised: This result is fundamental to the paper.

Original: Since every continuous function can be approximated withmulti-variable polynomials, it follows that every continuous function can be approximated withpositive and negative exponential functions.
Revised: Since every continuous function can be approximated withmulti-variable polynomials, it follows that every continuous function can be approximated withpositive and negative exponential functions.

Original: Some consideration must be given to single-variablefunction approximators, since it is the main building block for
Revised: Single-variable function approximators are pivotal andmust be reconsidered.

Revised: Universal function approximation can also be proven with the sub-algebraformulation of the Stone-Weierstrass theorem, but it’s not as delightful and simple as the firstconstructive proof given above.

Original:  Single-Variable Function Approximation
Revised:  Single-Variable Function Approximation

Original: Splines are piece-wise defined single-variable functions over some interval.
Revised: Splines are piece-wise defined single-variable functions over some interval.

Original: Each sub-interval of aspline is most often locally given by a low degree polynomial, even though the global structure is nota low degree polynomial.
Revised: Each sub-interval of aspline is most often locally given by a low degree polynomial, even though the global structure is nota low degree polynomial.

Original: B-splines are polynomial splines that are defined in a way that resemblesother basis function formulations [2].
Revised: B-splines are polynomial splines that are defined in a way that resemblesother basis function formulations [2].

Original: Each single-variable function in ATLAS is approximatedwith uniform cubic B-spline basis functions, shown in Figure 2.
Revised: Each single-variable function in ATLAS is approximatedwith uniform cubic B-spline basis functions, shown in Figure 2.

Original: B-splines can approximate anysingle-variable function, similar to using the Fourier basis.
Revised: B-splines can approximate anysingle-variable function, similar to using the Fourier basis.

Original: With uniform B-splines, each basisfunction is scaled so that the unit interval is uniformly partitioned, as in Figure 2.
Revised: With uniform B-splines, each basisfunction is scaled so that the unit interval is uniformly partitioned, as in Figure 2.

Original: [Figure]
Revised: [Figure]

Original: possible 
Revised: possible to

Original: The activation function to implement B-splines is given by:
Revised: The activation function to implement B-splines is given by:

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: The choice was made to use uniform cubic B-splines due to their excellent performance and robustnessto catastrophic forgetting, illustrated in Figure 3.
Revised: The choice was made to use uniform cubic B-splines due to their excellent performance and robustnessto catastrophic forgetting, illustrated in Figure 3.

Original: Using uniform B-splines instead of arbitrary subinterval partitions (also called knots in literature) makes optimisation easier.
Revised: Using uniform B-splines instead of arbitrary subinterval partitions (also called knots in literature) makes optimisation easier.

Original: Optimising partitions isnon-linear, but optimising only coefficient (also called control points) is linear and thus convex.
Revised: Optimising partitions isnon-linear, but optimising only coefficient (also called control points) is linear and thus convex.

Original: Bythe contra-positive: If the basis functions do not have the same shape, then non-uniform B-splines areused.

Original: (a) The activation is sparse and bounded.

Original: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Each basis function is multiplied by a parameter and summed together.
Revised: Each basis function is multiplied by a parameter and summed together.

Original: The total number of basisfunctions is typically fixed.
Revised: The total number of basisfunctions is typically fixed.

Original: Cubic B-splines are 3 rd order polynomials, and thus require a minimumof 3 + 1 = 4 control points or basis functions.
Revised: Cubic B-splines are 3 rd order polynomials, and thus require a minimumof 3 + 1 = 4 control points or basis functions.

Original: Instead of considering arbitrary densities of uniform cubic B-splines, we look at powers of two times the minimum number of basis functions, calledρ -density B-spline functions.
Revised: Instead of considering arbitrary densities of uniform cubic B-splines, we look at powers of two timesthe minimum number of basis functions, called ρ -density B-spline functions.

Original: Definition 1 ( ρ -density B-spline function) .
Revised: Definition 1 ( ρ -density B-spline function) .

Original: A ρ -density B-spline function is a uniform cubic B-splinefunction with 2 ρ +2 basis functions:
Revised: A ρ -density B-spline function is a uniform cubic B-splinefunction with 2 ρ +2 basis functions:

Original: [Equation]
Revised: [Equation]

Original: Consider the problem of expanding a single-variable function approximator with more basis functionsto increase its expressiveness.
Revised: Consider the problem of expanding a single-variable function approximator with more basis functionsto increase its expressive power.

Original: Using the Fourier basis makes it trivially easy by adding higherfrequency sines and cosines with coefficients initialised to zero.
Revised: Using the Fourier basis makes it trivially easy by adding higherfrequency sines and cosines with coefficients initialised to zero.

Original: It is trickier to achieve somethingsimilar with  cubic B-splines.
Revised: It is trickier to achieve somethingsimilar with uniform cubic B-splines.

Original: There are algorithms for creating new splines from existing splineswith knot insertion, but the intermediate steps result in non-uniform knots and splines.
Revised: There are algorithms for creating new splines from existingsplines with knot insertion, but the intermediate steps result in non-uniform knots and splines.

Original: A simple andpractical compromise that we propose is to use mixtures of different ρ -density B-spline functions, asillustrated in Figure 2.
Revised: Asimple and practical compromise that we propose is to use mixtures of different ρ -density B-splinefunctions, as illustrated in Figure 2.

Original: Definition 2 (mixed-density B-spline function) .
Revised: Definition 2 (mixed-density B-spline function) .

Original: A mixed-density B-spline function is a singlevariable function approximator that is obtained by summing together different ρ -density B-splinefunctions.
Revised: A mixed-density B-spline function is a singlevariable function approximator that is obtained by summing together different ρ -density B-splinefunctions.

Original: Only the maximum ρ -density B-spline function has trainable parameters, the others areconstant.
Revised: Only the maximum ρ -density B-spline function has trainable parameters, the others areconstant.

Original: Mixed-density B-spline functions are of the form:
Revised: Mixed-density B-spline functions are of the form:

Original: [Equation]
Revised: [Equation]

Original: Only the maximum r = ρ -density B-spline has trainable coefficients.
Revised: Only the maximum r = ρ -density B-spline has trainable coefficients.

Original: All lower density r > ρdensity B-spline have frozen and constant coefficients.
Revised: All lower density r > ρdensity B-spline have frozen and constant coefficients.

Original: The maximum r = ρ -density B-spline hastrainable coefficients with gradient updates that are orthogonal if the distance between two inputs islarge enough.
Revised: The maximum r = ρ -density B-spline hastrainable coefficients with gradient updates that are orthogonal if the distance between two inputs islarge enough.

Original: Similar to increasing the expressiveness of a Fourier basis function approximator by adding higherfrequency terms, one can add larger density cubic B-spline functions.
Revised: Similar to increasing the expressiveness of a Fourier basis function approximator by adding higherfrequency terms, one can add larger density cubic B-spline functions.

Original: Analytically, we can initialiseall the new  parameters θ r +1 ,i = 0 , ∀ i ∈ N such that:
Revised: Analytically, we can initialiseall the new scalar parameters θ r +1 ,i = 0 , ∀ i ∈ N such that:

Original: [Equation]
Revised: [Equation]

Original: It is therefore possible to create a minimal model with r = 0 initialised at zero, and train the modeluntil convergence.
Revised: It is therefore possible to create a minimal model with r = 0 initialised at zero, and train the modeluntil convergence.

Original: Then one can create a new model with r = 1 , by subsuming the previous model’sparameters, and train this more expressive model until convergence.
Revised: Then one can create a new model with r = 1 , by subsuming the previous model’sparameters, and train this more expressive model until convergence.

Original: This process of training andexpansion can be continued indefinitely, and is shown in Figure 4.
Revised: This process of training andexpansion can be continued indefinitely, and is shown in Figure 4.

Original:  ATLAS
Revised:  ATLAS

Original: ATLAS is named for carrying the burden of all it must remember, after the Titan god Atlas in Greekmythology who was tasked with holding the weight of the world.
Revised: ATLAS is named for carrying the burden of all it must remember, after the Titan god Atlas in Greekmythology who was tasked with holding the weight of the world.

Original: ATLAS is also an acronym for
Revised: ATLAS is also an acronym for

Original: AddiTive exponentiaL Additive Splines.
Revised: AddiTive exponentiaL Additive Splines.

Original: Definition 3 (ATLAS) .
Revised: Definition 3 (ATLAS) .

Original: ATLAS is a function approximator of n variables, with mixed-density
Revised: ATLAS is a function approximator of n variables, with mixed-density

Original: B-spline functions f j ( x j ) , g i,j ( x j ) , and h i,j ( x j ) in the form: 
Revised: B-spline functions f j ( x j ) , g i,j ( x j ) , and h i,j ( x j ) in the form: A ( ⃗ x ) :

Original: [Equation]
Revised: [Equation]

Original: ( ⃗ x ) :
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: ATLAS is equivalently given by the compact notation:
Revised: ATLAS is equivalently given by the compact notation:

Original: [Equation]
Revised: [Equation]

Original: The absolutely convergent series of scale factors k − 2 was chosen for numerical stability and to ensurethe model is absolutely convergent.
Revised: The absolutely convergent series of scale factors k − 2 was chosen for numerical stability and to ensurethe model is absolutely convergent.

Original: Another feature is that the series of scale factors also breaks thesymmetry that would otherwise exist if all mixed-density B-spline functions were initialised to zero.
Revised: Another feature is that the series of scale factors also breaks thesymmetry that would otherwise exist if all mixed-density B-spline functions were initialised to zero.

Original: Initialising all the parameters to be zero is a departure from the conventional approach of randominitialisation.
Revised: Initialising all the parameters to be zero is a departure from the conventional approach of randominitialisation.

Original: The number of exponential terms can be increased without changing the output of themodel.
Revised: The number of exponential terms can be increased without changing the output of themodel.

Original: We can choose to initialise G M +1 ( ⃗ x ) = 0 and H M +1 ( ⃗ x )
Revised: We can choose to initialise G M +1 ( ⃗ x ) = 0 and H M +1 ( ⃗ x )

Original: = 0 , such that the model capacitycan be increased at will.
Revised: = 0 , such that the model capacitycan be increased at will.

Original: ATLAS is a universal function approximator with robust memory retention.
Revised: ATLAS is a universal function approximator with some inherent memory retention.

Original: It possesses threeproperties atypical of most universal function approximators:
Revised: It possesses threeproperties atypical of most universal function approximators:

Original: The activity within ATLAS is sparse – most neural units are zero and inactive.
Revised: The activity within ATLAS is sparse – most neural units are zero and inactive.

Original: The gradient vector with respect to trainable parameters is bounded regardless of the size
Revised: The gradient vector with respect to trainable parameters is bounded regardless of the size

Original: and capacity of the model, so training is numerically stable for many possible training
Revised: and capacity of the model, so training is numerically stable for many possible training

Original: hyper-parameters.
Revised: hyper-parameters.3.

Original: Inputs that are sufficiently far from each other have orthogonal representations.
Revised: Inputs that are sufficiently far from each other have orthogonal representations.

Original: The proofs of the three properties follows from the single-variable case, the assumption of boundedsingle-variable functions and parameters, and the absolutely convergent k − 2 scale factors.
Revised: The proofs of the three properties follows from the single-variable case, the assumption of boundedsingle-variable functions and parameters, and the absolutely convergent k − 2 scale factors.

Original: Property 1 (Sparsity) .
Revised: Property 1 (Sparsity) .

Original: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parameters θ i with indexset Θ , the gradient vector of trainable parameters (for ATLAS) is sparse:
Revised: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parameters θ i with indexset Θ , the gradient vector of trainable parameters (for ATLAS) is sparse:

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.
Revised: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.

Original: The gradient vector has a maximum of 4 n (2 M +1) non-zero entries, which is independentof r .
Revised: The gradient vector has a maximum of 4 n (2 M +1) non-zero entries, which is independentof r .

Original: Recall that only the maximum density ( ρ = r ) cubic B-spline function has trainable parameters.
Revised: Recall that only the maximum density ( ρ = r ) cubic B-spline function has trainable parameters.

Original: The fraction of trainable basis functions that are active is at most 2 − r .
Revised: The fraction of trainable basis functions that are active is at most 2 − r .

Original: Sparsity entails efficientimplementation, and suggests possible memory retention and robustness to catastrophic forgetting.
Revised: Sparsity entails efficientimplementation, and suggests possible memory retention and robustness to catastrophic forgetting.

Original: Property 2 (Gradient flow attenuation) .
Revised: Property 2 (Gradient flow attenuation) .

Original: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i with index set Θ : if all the mixed-density B-spline functions are bounded, then the gradient vectorof trainable parameters for ATLAS is bounded:
Revised: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i with index set Θ : if all the mixed-density B-spline functions are bounded, then the gradient vectorof trainable parameters for ATLAS is bounded:

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.
Revised: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.

Original: The factor of k − 2 inside the expression for ATLAS is necessary to ensure the sum isconvergent in the limit of infinitely many exponential terms M → ∞ .
Revised: The factor of k − 2 inside the expression for ATLAS is necessary to ensure the sum isconvergent in the limit of infinitely many exponential terms M → ∞ .

Original: Only the maximum density( ρ = r ) cubic B-spline function has trainable parameters, so that the gradient vector is bounded in thelimit of arbitrarily large densities r → ∞ .
Revised: Only the maximum density( ρ = r ) cubic B-spline function has trainable parameters, so that the gradient vector is bounded inthe limit of arbitrarily large densities r → ∞ .

Original: It is worth recalling that at most four basis functions areactive for uniform cubic B-spline functions, regardless of the density, but the smaller densities cannotbe trainable, otherwise this property does not hold.The gradient vector has bounded L1 norm for anynumber of basis functions and exponential terms.
Revised: Smaller densities cannot be trainable, otherwise thisproperty does not hold.

Original: The bounded gradient vector implies that ATLASis numerically stable during training, regardless of its size or parameter count.
Revised: The bounded gradient vector implies that ATLAS is numerically stable duringtraining, regardless of its size or parameter count.

Original: Property 3 (Distal orthogonality) .
Revised: Property 3 (Distal orthogonality) .

Original: For any ⃗ x , ⃗ y ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i for an ATLAS model A ( ⃗ x ) :
Revised: For any ⃗ x , ⃗ y ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i for an ATLAS model A ( ⃗ x ) :

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: Two points that sufficiently differ in each input variable have orthogonal parameter gradients.
Revised: Two points that sufficiently differ in each input variable have orthogonal parameter gradients.

Original: A more general result holds for other choices of piece-wise defined single-variable functions.

Original: Distal orthogonality means ATLAS is reasonably robust to catastrophic forgetting, without otherregularisation and training techniques.
Revised: Distal orthogonality means ATLAS is reasonably robust to catastrophic forgetting, without otherregularisation and training techniques.

Original: However, memory retention can still potentially be improvedwhen used in conjunction with other techniques.
Revised: However, memory retention can still potentially be improvedwhen used in conjunction with other techniques.

Original: ATLAS can be implemented with 1D convolution, reshaping, embedding, multiplication and denselayers.
Revised: ATLAS can be implemented with 1D convolution, reshaping, embedding, multiplication and denselayers.

Original: The same basis functions have to be computed for each input variable, hence 1D convolutions.
Revised: The same basis functions have to be computed for each input variable, hence 1D convolutions.

Original: By correctly scaling, shifting, and rounding inputs one can compute only the non-zero basis functionswith embedding layers.
Revised: By correctly scaling, shifting, and rounding inputs one can compute only the non-zero basisfunctions with embedding layers.

Revised: The number of basis functions are chosen from powers of two forconvenience, with the maximum density B-spline function having exactly λ = 4 × 2 r basis functions.

Revised: Summing over all densities the total number of all basis functions in each input variable is at most2 λ , because a geometric series was used.

Original: ATLAS models with input dimension n , output dimension p , O (2 r ) cubic B-spline basis functions, and M positive and negative exponential functions have time complexity 
Revised: ATLAS models with input dimension n , output dimensionp , approximately 2 λ cubic B-spline basis functions over each input, and M positive and negativeexponential functions have time complexity O

Original: O ( rMnp ) , and O (2 r Mnp ) space complexity.
Revised: ( Mnp log λ ) , and O ( Mnpλ ) space complexity.

Original:  Methodology
Revised:  Methodology

Original: The 2- and 8-dimensional models were considered for evaluation, in combination with a fixed widthfor the update region in Task 2 from 0 .
Revised: The 2- and 8-dimensional models were considered for evaluation, in combination with a chosenwidth for the update region in Task 2 from 0 .

Original: 1 to 0 .
Revised: 1 to 0 .

Original: 9 inincrements of 0 . 1 .
Revised: 9 in 0 . 1 increments.

Original: 30 trials were performed for eachcombination update region width and model dimension.
Revised: 30 trials were performed foreach combination of model dimension and update region width.

Original: Mean Absolute Error (MAE) loss function,the Adam optimiser, and mini batch sizes of 100 are used throughout all experiemnts.
Revised: Mean Absolute Error (MAE) lossfunction, the Adam optimiser, and mini batch sizes of 100 are used throughout all experiments.

Original: At the beginning of each trial (for a given dimension and update region width) a random learning ratewas sampled uniformly between 10 − 6 and 0 .
Revised: At the beginning of each trial (for a given dimension and update region width) a random learning ratewas sampled uniformly between 10 − 6 and 0 .

Original: + 10 − 6 .
Revised: + 10 − 6 .

Original: A random noise level was sampled from anexponential distribution with scale parameter equal to one.
Revised: A random noise level was sampled from anexponential distribution with scale parameter equal to one.

Original: The Task 1 target function is constructedfrom 1000 Euclidean radial basis functions (RBFs) with locations chosen uniformly over the entireinput domain, with RBF scale parameters sampled independently from an exponential distribution(scale parameter equal to 10).
Revised: The Task 1 target function is constructedfrom 1000 Euclidean radial basis functions (RBFs) with locations chosen uniformly over the entireinput domain, with RBF scale parameters sampled independently from an exponential distribution(scale parameter equal to 10).

Original: The weights of each radial basis function are sampled from a normaldistribution with mean zero and variance equal to one.
Revised: The weights of each radial basis function are sampled from a normaldistribution with mean zero and standard deviation equal to one.

Original: The Task 2 target function is exactly the sameas the Task 1 target function - except for a square-like region with width equal to update region width.
Revised: The Task 2 target function is exactlythe same as the Task 1 target function – except for a square-like region with width equal to updateregion width.

Original: The location of the update region is chosen uniformly at random, and such that it is completely insidethe domain of the model.
Revised: The location of the update region is chosen uniformly at random, and such that it iscompletely inside the domain of the model.

Original: The updated region masks the Task 1 target function and instead replacesthe values inside it with another function that is sampled from the same distribution as the Task 1target function, but independently from the Task 1 target function.
Revised: The updated region masks the Task 1 target function andinstead replaces the values inside it with another function that is sampled from the same distributionas the Task 1 target function, but independently from the Task 1 target function.

Original: After the generation of the target functions 10000 data points are sampled for training, validation, andtest sets for Task 1 and Task 2.
Revised: After the generation of the target functions 10000 data points are sampled for training, validation, andtest sets for Task 1 and Task 2.

Original: To simulate the effect of learning unrelated tasks, the training data for
Revised: To simulate the effect of learning unrelated tasks, the training data for

Original: Task 2 is only sampled from update region - with no training data outside of it being presented again,by contrast the validation and test sets for Task 2 were sampled over the entire input domain.
Revised: Task 2 is only sampled from update region - with no training data outside of it being presented again,by contrast the validation and test sets for Task 2 were sampled over the entire input domain.

Original: Gaussiannoise with standard deviation equal to the randomly chosen noise level is added to all training data.
Revised: Gaussiannoise with standard deviation equal to the randomly chosen noise level is added to all training data.

Original: An ATLAS model ( M = 10 positive and M = 10 negative exponential functions, maximum basisfunction density r = 4 ) with guaranteed distal orthogonality is trained and evaluated on Task 1and
Revised: An ATLAS model ( M = 10 positive and M = 10 negative exponential functions, maximum basisfunction density r = 4 ) with guaranteed distal orthogonality is trained and evaluated on Task 1and

Original: Task 2.
Revised: Task 2.

Original: Then a modified ATLAS model ( 
Revised: Then a modified ATLAS model ( M =

Original:  Results
Revised:  Results

Original: As shown in Figure 5 the effect of distal orthogonality is clear and crisp boundaries that limit theeffect of Task 2 on the memory of Task 1.
Revised: As shown in Figure 5 the effect of distal orthogonality is clear and crisp boundaries that limit theeffect of Task 2 on the memory of Task 1.

Original: Without distal orthogonality there are more off-targeteffects that can be visualised.
Revised: Without distal orthogonality there are more off-targeteffects that can be visualised.

Original: [Figure]
Revised: [Figure]

Original: The effect of distal orthogonality on the averaged MAE for various trials for 2- and 8-dimensionalproblems are presented as scatter plots of the averaged MAE over 30 trials for different update regionwidths as shown in Figure 6.
Revised: The effect of distal orthogonality on the averaged MAE for various trials for 2- and 8-dimensionalproblems are presented as scatter plots of the averaged MAE over 30 trials for different update regionwidths as shown in Figure 6.

Original: The expected off-target error depends on the dimension of the problemand the width of the updated regions.
Revised: The expected off-target error depends on the dimension of the problemand the width of the updated regions.

Original: [Figure]
Revised: [Figure]

Original: Initial estimates of analytical results to the expected off-target error require simplification, but areasonable assumption in light of other evidence is that each input dimension has equal contributionon the unit hyper-cube.
Revised: Analytical results to the expected off-target error require simplification, but a reasonable assumptionin the absence of other evidence is that each input dimension has equal contribution on the unithyper-cube.

Original: Assume for a fixed input dimension n and some region of width 0 < δ < 1where the target function Y is changed such that | ∆ Y | = 1 is one larger than it was originally.
Revised: Assume for a fixed input dimension n and some region of width 0 < δ < 1 where thetarget function Y is changed such that | ∆ Y | = 1 is one larger than it was originally.

Original: Theexpected off-target error depends on k the number of input variables inside the update region:
Revised: The expectedoff-target error depends on k the number of input variables inside the updated region:

Original: [Equation]
Revised: [Equation]

Original: A binomial coefficient is needed to correctly account for all permutations with the same magintude ofchange:
Revised: A binomial coefficient is needed to correctly account for all permutations with the same magnitude ofchange:

Original: [Equation]
Revised: [Equation]

Original: One can calculate expected change values:
Revised: One can calculate expected change values:

Original: [Equation]
Revised: [Equation]

Original: However if one assumes that the target function inside the updated region of width δ is correct, withprobability δ n of sampling from the entire input-domain, then the expected off-target error should be:
Revised: However if one assumes that the target function inside the updated region of width δ is correct, withprobability δ n of sampling from the entire input-domain, then the expected off-target error should be:

Original: [Equation]
Revised: [Equation]

Original: This seems consistent with some of the experimental results, but further investigation is needed.
Revised: This seems consistent with some of the experimental results, but further investigation is needed.

Original:  Conclusion
Revised:  Conclusion

Original: The main contribution of the paper is theoretical and technical.
Revised: The main contribution of the paper is theoretical and technical.

Original: A representation theorem is presentedthat outlines how to approximate multi-variable functions with single-variable functions (splines andexponential functions).
Revised: A representation theorem is presentedthat outlines how to approximate multi-variable functions with single-variable functions (splines andexponential functions).

Original: ATLAS approximates all arbitrary single-variable functions with mixturesof B-spline functions.
Revised: ATLAS approximates all arbitrary single-variable functions with mixturesof B-spline functions.

Original: ATLAS is constructed in such a way that the gradient vector with respectto trainable parameters is bounded, regardless of how large an ATLAS model is.
Revised: ATLAS is constructed in such a way that the gradient vector with respectto trainable parameters is bounded, regardless of how large an ATLAS model is.

Original: The activation ofunits in ATLAS is sparse, and allowed for an efficient implementation that only computes non-zeroactivation values with the aid of embedding layers.
Revised: The activation ofunits in ATLAS is sparse, and allowed for an efficient implementation that only computes non-zeroactivation values with the aid of embedding layers.

Original: The gradient update vector with respect totrainable parameters is orthogonal for different inputs as long as the inputs are sufficiently differentfrom each other.
Revised: The gradient update vector with respect totrainable parameters is orthogonal for different inputs as long as the inputs are sufficiently differentfrom each other.

Original: ATLAS models with input dimension n , output dimension p , O (2 r ) cubic B-spline basis functions 
Revised: ATLAS models with input dimension n , output dimension p , 2 λ cubic B-spline basis functions overeach input, and M positive and negative exponential functions have time complexity O

Original: ,and M positive and

Original: negative exponential

Original: functions have

Original: time complexity O ( rMnp ) , and O (2 r Mnp ) space complexity.
Revised: ( Mnp log λ ) ,and O ( Mnpλ ) space complexity.

Original: ATLAS was shown to exhibit some memory retention, without the assistance of other techniques.
Revised: ATLAS was shown to exhibit some memory retention, without the assistance of other techniques.

Original: This is a good indication of the potential for combining it with other techniques and models forcontinual learning.
Revised: This is a good indication of the potential for combining it with other techniques and models forcontinual learning.

Original: The chosen experiments demonstrated the theoretically derived predictions andcontrasted two models, incuding a variant of ATLAS without distal orthogonality guarantees.
Revised: The chosen experiments demonstrated the theoretically derived predictions andcontrasted two models, incuding a variant of ATLAS without distal orthogonality guarantees.

Original: As far as societal impacts are concerned: It is possible that ATLAS could allow for the creation ofmore powerful machine learning algorithms, that require less resources to train and deploy.
Revised: As far as societal impacts are concerned: It is possible that ATLAS could allow for the creation ofmore powerful machine learning algorithms, that require less resources to train and deploy.

Original: Furthertesting is needed to make any concrete claim.Checklist
Revised: Furthertesting is needed to make any concrete claim.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes]

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes]

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes]

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes]

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [TODO]

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [TODO]

Original: (b) Did you mention the license of the assets?

Original: [TODO]

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [TODO]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?

Original: [N/A]

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Original: [N/A]