Original: ATLAS: Universal Function Approximator for Memory Retention
Revised: ATLAS: Universal Function Approximator for Memory Retention

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AffiliationAddress email
Revised: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Artificial neural networks (ANNs), despite their universal function approximationcapability and practical success, are subject to catastrophic forgetting.
Revised: Artificial neural networks (ANNs), despite their universal function approximationcapability and practical success, are subject to catastrophic forgetting.

Original: Catastrophicforgetting refers to the abrupt unlearning of a previous task when a new task islearned.
Revised: Catastrophicforgetting refers to the abrupt unlearning of a previous task when a new task islearned.

Original: It is an emergent phenomenon that plagues ANNs and hinders continuallearning.
Revised: It is an emergent phenomenon that plagues ANNs and hinders continuallearning.

Original: Existing universal function approximation theorems for ANNs guaranteefunction approximation ability, but seldom touch on the model details and do notpredict catastrophic forgetting.
Revised: Existing universal function approximation theorems for ANNs guaranteefunction approximation ability, but seldom touch on the model details and do notpredict catastrophic forgetting.

Original: This paper presents a novel universal approximationtheorem for multi-variable functions using only single-variable functions andexponential functions.
Revised: This paper presents a novel universal approximationtheorem for multi-variable functions using only single-variable functions andexponential functions.

Original: Furthermore, we present ATLAS —a novel ANN architecturebased on the exponential approximation theorem and B-splines.
Revised: Furthermore, we present ATLAS —a novel ANN architecturebased on the exponential approximation theorem and B-splines.

Original: It is shown that
Revised: It is shown that

Original: ATLAS is a universal function approximator capable of memory retention and,therefore, continual learning.
Revised: ATLAS is a universal function approximator capable of memory retention and,therefore, continual learning.

Original: The memory retention of ATLAS is imperfect,with some off-target effects during continual learning, but it is well-behaved andpredictable.
Revised: The memory retention of ATLAS is imperfect,with some off-target effects during continual learning, but it is well-behaved andpredictable.

Original: An efficient implementation of ATLAS is provided.
Revised: An efficient implementation of ATLAS is provided.

Original: Experimentsare conducted to evaluate both the function approximation and memory retentioncapabilities of ATLAS.
Revised: Experimentsare conducted to evaluate both the function approximation and memory retentioncapabilities of ATLAS.

Original:  Introduction
Revised:  Introduction

Original: Catastrophic forgetting [7, 13, 22] is an emergent phenomenon where an ANN learns a new task andthe subsequent parameter updates interfere with the model’s performance on previously learned tasks.
Revised: Catastrophic forgetting [7, 13, 22] is an emergent phenomenon where an ANN learns a new task andthe subsequent parameter updates interfere with the model’s performance on previously learned tasks.

Original: Catastrophic forgetting is also called catastrophic interference [19].
Revised: Catastrophic forgetting is also called catastrophic interference [19].

Original: If an ANN cannot effectivelylearn many tasks, it has limited utility in the context of continual learning [9, 12].
Revised: If an ANN cannot effectivelylearn many tasks, it has limited utility in the context of continual learning [9, 12].

Original: Catastrophicforgetting is like learning to pick up a cup, but simultaneously forgetting how to breathe.
Revised: Catastrophicforgetting is like learning to pick up a cup, but simultaneously forgetting how to breathe.

Original: Even linearfunctions are susceptible to catastrophic forgetting, as illustrated in Figure 

Revised: Even linearfunctions are susceptible to catastrophic forgetting, as illustrated in Figure

Original: The simple example of a linear regression model being susceptible to catastrophic forgetting mightbe due to the non-linearity of the target function, noise in the data or parameter sharing across the input.
Revised: The simple example of a linear regression model being susceptible to catastrophic forgetting mightbe due to the non-linearity of the target function, noise in the data or parameter sharing across the input.

Original: Avoiding parameter sharing is easily accomplsihed with piece-wise defined functions such assplines [26].
Revised: Avoiding parameter sharing is easily accomplsihed with piece-wise defined functions such assplines [26].

Revised: ANNs can be explained in many different ways; a useful analogy would be to compare

Revised: ANNs to very large lookup tables that store information.

Original:  Universal function approximation theorems are a cornerstone of machine learning, and prove thatartificial neural networks (ANNs) can approximate any given continuous target function [10, 11, 15].
Revised: Removing and updating values has off-targeteffects throughout the table or ANN. Universal function approximation theorems are a cornerstone of machine learning, and prove thatartificial neural networks (ANNs) can approximate any given continuous target function [10, 11, 15].

Original: The theorems do not specify how to find an ANN with sufficient performance for problems in practice.
Revised: The theorems do not specify how to find an ANN with sufficient performance for problems in practice.

Original: Gradient descent optimisation is the standard  for training neural networks, but other optimisationand learning procedures exist.
Revised: Gradient descent optimisation is the standard practice for finding/training neural networks, but otheroptimisation and learning procedures exist.

Original: The numerical stability of gradient descent methods is of practicalimportance.
Revised: ATLAS models trained with gradient descent methodsexhibit desirable properties.

Original: Gradient update vectors that are bounded and orthogonal to each other improve numericalstability and can mitigate catastrophic forgetting.
Revised: However, other optimisation techniques based on evolutionary algorithmsmight not exhibit the same properties in general for ATLAS models.

Original: This paper introduces ATLAS—a novel universal function approximator based on B-splines thathas intrinsic memory retention.
Revised: This paper introduces ATLAS—a novel universal function approximator based on B-splines thathas intrinsic memory retention, even in the absence of other training and regularisation techniques.

Original: ATLAS has well-behaved parameter gradients that do not vanishor explode. ATLAS has methods to expand model capacity in a systematic way without loosingthe previously learned information.
Revised: ATLAS has well-behaved parameter gradients that are sparse, bounded and orthogonal betweeninput points that are far enough from each other.

Original: The accompanying representation and universal approximationtheorems are also provided.
Revised: The accompanying representation and universalapproximation theorems are also provided.

Original:  Relevant Studies
Revised:  Relevant Studies

Original: It is conjectured that overlapping representations in ANNs lead to catastrophic forgetting [12].
Revised: It is conjectured that overlapping representations in ANNs lead to catastrophic forgetting [12].

Original: Catastrophic forgetting occurs when parameters necessary for one task change while training to meetthe objectives of another task [14, 20].
Revised: Catastrophic forgetting occurs when parameters necessary for one task change while training to meetthe objectives of another task [14, 20].

Original: Strategies to mitigate catastrophic forgetting can includeshielding important weights from changes, or regularisation to keep weights close to their initialvalues.

Original: The least desirable option is retraining a model over all tasks.
Revised: The least desirable strategy to mitigate catastrophic forgettingis retraining a model over all tasks.

Original: Data augmentation approachessuch as rehearsal and pseudo-rehearsal have also been employed [22].
Revised: Data augmentation approaches such as rehearsal and pseudorehearsal have also been employed [22].

Original: Other ideas from optimal control theory in combination with dynamic programming have also beenapplied to counteract catastrophic forgetting, with a cost functional similar in form to the actionintegral from physics and Lagrangian mechanics [16].
Revised: Other ideas from optimal control theory in combinationwith dynamic programming have also been applied to counteract catastrophic forgetting, with a costfunctional similar in form to the action integral from physics and Lagrangian mechanics [16].

Original: Orthogonal Gradient Descent (OGD) is a training augmentation or optimisation technique thatmodifies the gradient updates of subsequent tasks to be orthogonal to previous tasks [6, 1].
Revised: Orthogonal Gradient Descent (OGD) is a training augmentation or optimisation technique thatmodifies the gradient updates of subsequent tasks to be orthogonal to previous tasks [6, 1].

Original: Onecan describe data in terms of a distribution defined over the input space, target values, and time (theorder of data or tasks that are presented during training).
Revised: Onecan describe data in terms of a distribution defined over the input space, target values, and time (theorder of data or tasks that are presented during training).

Original: OGD attempts to make gradient updatesorthogonal to each other over time.
Revised: OGD attempts to make gradient updatesorthogonal to each other over time.

Original: ATLAS, in contrast, possesses distal orthogonality, meaning thatif two inputs are far enough from each other in the input space, then corresponding gradient updateswill be orthogonal.
Revised: ATLAS, in contrast, possesses distal orthogonality, meaning thatif two inputs are far enough from each other in the input space, then corresponding gradient updateswill be orthogonal.

Original: A corollary of this is that if the data distribution between tasks shifts in the inputspace, then the subsequent gradient updates will tend to be orthogonal.
Revised: A corollary of this is that if the data distribution between tasks shifts in the inputspace, then the subsequent gradient updates will tend to be orthogonal.

Original: ATLAS does not use externalmemory like OGD.
Revised: ATLAS does not use externalmemory like OGD.

Original: Extensions of OGD include PCA-OGD, which compresses gradient updates intoprincipal components to reduce memory requirements [4].
Revised: Extensions of OGD include PCA-OGD, which compresses gradient updates intoprincipal components to reduce memory requirements [4].

Original: The Neural Tangent Kernel (NTK) overlapmatrices, as discussed by Doan et al.
Revised: The Neural Tangent Kernel (NTK) overlapmatrices, as discussed by Doan et al.

Original: [4], could be a useful tool for analysing ATLAS architectures.
Revised: [4], could be a useful tool for analysing ATLAS architectures.

Original: The survey by Delange et al.
Revised: The survey by Delange et al.

Original: [3] gives an extensive overview of continual learning to addresscatastrophic forgetting.
Revised: [3] gives an extensive overview of continual learning to addresscatastrophic forgetting.

Original: ATLAS is a model that implements parameter isolation, because of its use ofpiece-wise defined splines.
Revised: ATLAS is a model that implements parameter isolation, because of its use ofpiece-wise defined splines.

Original: Particularly relevant to ATLAS is the work on scale of initialisation andextreme memorisation [21].
Revised: Particularly relevant to ATLAS is the work on scale of initialisation andextreme memorisation [21].

Original: Increasing the density of basis functions in ATLAS can lead to bettermemorisation, and increases the scale of some parameters in ATLAS.
Revised: Increasing the density of basis functions in ATLAS can lead to bettermemorisation, and increases the scale of some parameters in ATLAS.

Original: It is not necessary to expand the capacity of an ATLAS model to learn new tasks, as with some otherapproaches [23].

Original: ATLAS does in practice demonstrate something akin to "graceful forgetting" asdiscussed in Golkar et al.

Original: Pi-sigma neural networks use nodes that compute products instead of sums [25].
Revised: Pi-sigma neural networks use nodes that compute products instead of sums [25].

Original: Pi-sigma neuralnetworksreportedly have potential utility in building more efficient and expressive models than thestandard ANNs. ATLAS can similarly compute products, and is more easily trained than the Pi-sigmaneural networks.
Revised: Pi-sigma neuralnetworks have some similarities with the global structure of ATLAS.

Original: B-splines, which form the basis of ATLAS, have been applied for machine learning [5].
Revised: B-splines, which form the basisof ATLAS, have been applied for machine learning [5].

Original: Scardapaneet al.
Revised: Scardapane et al.

Original: [24] investigated trainable activation functions parameterised by splines.
Revised: [24] investigated trainableactivation functions parameterised by splines.

Original: Uniform cubic Bsplines have basis functions that are translates of one another [2].
Revised: Uniform cubic B-splines have basis functions that aretranslates of one another [2].

Original: Uniform cubic B-splines have beentested for memory retention, and ATLAS can be considered as an improvement on existing splinemodels [26].
Revised: Uniform cubic B-splines have been tested for memory retention, and ATLAS can be considered as an improvement on existing spline models [26].

Original: B-splines, and by extension ATLAS, can be trained to fit lower frequency components, expanded and trained again until a network is found with sufficient accuracy and generalisation,similar to other techniques [17, 18]
Revised: B-splines, and by extension ATLAS, can be trained to fit lower frequency components, expanded andtrained again until a network is found with sufficient accuracy and generalisation, similar to othertechniques [17, 18]. It is not necessary to expand the capacity of an ATLAS model to learn newtasks, as with some other approaches [23]. ATLAS does in practice demonstrate something akin to"graceful forgetting" as discussed in Golkar

Original:  Notation
Revised:  Notation

Original: Vector quantities like ⃗ x are clearly indicated with a bar or arrow for legibility.
Revised: Vector quantities like ⃗ x are clearly indicated with a bar or arrow for legibility.

Original: Parameters, inputs,functions etc.
Revised: Parameters, inputs,functions etc.

Original: without a bar or arrow are scalar quantities like S ( x ) .
Revised: without a bar or arrow are scalar quantities like S ( x ) .

Original: Some scalar quantities withindices are the scalar components of a vector like x j or scalar parameters in the model like θ i 
Revised: Some scalar quantities withindices are the scalar components of a vector like x j or scalar parameters in the model like θ i .

Revised: Thegradient operator that acts on a scalar function like ⃗ ∇ ⃗θ A ( ⃗ x ) yields a vector-valued function ⃗ ∇ ⃗θ A ( ⃗ x )as is typical of multi-variable calculus.

Original:  Exponential Representation Theorem
Revised:  Exponential Representation Theorem

Original: Any continuous multi-variable function on a compact space can be uniformly approximated withmulti-variable polynomials by the Stone-Weierstrass Theorem.
Revised: Any continuous multi-variable function on a compact space can be uniformly approximated withmulti-variable polynomials by the Stone-Weierstrass Theorem.

Original: Let I denote an index set of tuples ofnatural numbers including zero such that i j ∈ N 0 for all j ∈ N with i =
Revised: Let I denote an index set of tuples ofnatural numbers including zero such that i j ∈ N 0 for all j ∈ N with i =

Original: ( i 1 , .., i n ) ∈ I and a i ∈ R .
Revised: ( i 1 , .., i n ) ∈ I and a i ∈ R .

Original: Multi-variable polynomials can be represented as:
Revised: Multi-variable polynomials can be represented as:

Original: [Equation]
Revised: [Equation]

Original: Each monomial term a i Π nj
Revised: Each monomial term a i Π nj

Original: =1 x i j j is a product of single-variable functions in each variable.
Revised: =1 x i j j is a product of single-variable functions in each variable.

Original: It isdesirable to rewrite products as sums.
Revised: It isdesirable to rewrite products as sums.

Original: The product of single-variable functions can be rewritten as asum using exponentials and logarithms.
Revised: The product of single-variable functions can be rewritten as asum using exponentials and logarithms.

Original: Lemma 1.
Revised: Lemma 1.

Original: For any a i ∈ R , there exists γ i > 0 and β i > 0 , such that: a i = γ i − β
Revised: For any a i ∈ R , there exists γ i > 0 and β i > 0 , such that: a i = γ i − β

Original: i
Revised: i

Original: Theorem 1 (Exponential representation theorem) .
Revised: Theorem 1 (Exponential representation theorem) .

Original: Any multi-variable polynomial function y ( ⃗ x )of n variables over the positive orthant, can be exactly represented by continuous single-variablefunctions g i,j ( x j ) and h i,j ( x j ) in the form:
Revised: Any multi-variable polynomial function y ( ⃗ x )of n variables over the positive orthant, can be exactly represented by continuous single-variablefunctions g i,j ( x j ) and h i,j ( x j ) in the form:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Consider any monomial term a i Π nj =1 x i j j with a i ∈ R , then by Lemma 1 there exist strictlypositive numbers γ i > 0 and β i > 0 , such that:
Revised: Consider any monomial term a i Π nj =1 x i j j with a i ∈ R , then by Lemma 1 there exist strictlypositive numbers γ i > 0 and β i > 0 , such that:

Original: [Equation]
Revised: [Equation]

Original: The argument of each exponential function is a sum of single-variable functions and constants.
Revised: The argument of each exponential function is a sum of single-variable functions and constants.

Original: Without loss of generality, a set of single-variable functions can be defined such that:
Revised: Without loss of generality, a set of single-variable functions can be defined such that:

Original: [Equation]
Revised: [Equation]

Original: Since this holds for any a i Π nj =1 x i j j and all i ∈
Revised: Since this holds for any a i Π nj =1 x i j j and all i ∈

Original: I , it follows that:
Revised: I , it follows that:

Original: [Equation]
Revised: [Equation]

Original: This result is fundamental to the paper.
Revised: This result is fundamental to the paper.

Original: Since every continuous function can be approximated withmulti-variable polynomials, it follows that every continuous function can be approximated withpositive and negative exponential functions.
Revised: Since every continuous function can be approximated withmulti-variable polynomials, it follows that every continuous function can be approximated withpositive and negative exponential functions.

Original: Some consideration must be given to single-variablefunction approximators, since it is the main building block forthe representation theorem.
Revised: Some consideration must be given to single-variablefunction approximators, since it is the main building block for

Original:  Single-Variable Function Approximation
Revised:  Single-Variable Function Approximation

Original: Splines are piece-wise defined single-variable functions over some interval.
Revised: Splines are piece-wise defined single-variable functions over some interval.

Original: Each sub-interval of aspline is most often locally given by a low degree polynomial, even though the global structure is nota low degree polynomial.
Revised: Each sub-interval of aspline is most often locally given by a low degree polynomial, even though the global structure is nota low degree polynomial.

Original: B-splines are polynomial splines that are defined in a way that resemblesother basis function formulations [2].
Revised: B-splines are polynomial splines that are defined in a way that resemblesother basis function formulations [2].

Original: Each single-variable function in ATLAS is approximatedwith uniform cubic B-spline basis functions, shown in Figure 2.
Revised: Each single-variable function in ATLAS is approximatedwith uniform cubic B-spline basis functions, shown in Figure 2.

Original: B-splines can approximate anysingle-variable function, similar to using the Fourier basis.
Revised: B-splines can approximate anysingle-variable function, similar to using the Fourier basis.

Original: With uniform B-splines, each basisfunction is scaled so that the unit interval is uniformly partitioned, as in Figure 2. 
Revised: With uniform B-splines, each basisfunction is scaled so that the unit interval is uniformly partitioned, as in Figure 2. [Figure]

Original: possibleto
Revised: possible

Original: The activation function to implement B-splines is given by:
Revised: The activation function to implement B-splines is given by:

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: The choice was made to use uniform cubic B-splines due to their excellent performance and robustnessto catastrophic forgetting, illustrated in Figure 3.
Revised: The choice was made to use uniform cubic B-splines due to their excellent performance and robustnessto catastrophic forgetting, illustrated in Figure 3.

Original: Using uniform B-splines instead of arbitrary subinterval partitions (also called knots in literature) makes optimisation easier.
Revised: Using uniform B-splines instead of arbitrary subinterval partitions (also called knots in literature) makes optimisation easier.

Original: Optimising partitions isnon-linear, but optimising only coefficient (also called control points) is linear and thus convex.
Revised: Optimising partitions isnon-linear, but optimising only coefficient (also called control points) is linear and thus convex.

Original: Bythe contra-positive: If the basis functions do not have the same shape, then non-uniform B-splines areused. 
Revised: Bythe contra-positive: If the basis functions do not have the same shape, then non-uniform B-splines areused. (a) The activation is sparse and bounded. [Equation]

Original: [Figure]
Revised: [Figure]

Original: Each basis function is multiplied by a parameter and summed together.
Revised: Each basis function is multiplied by a parameter and summed together.

Original: The total number of basisfunctions is typically fixed.
Revised: The total number of basisfunctions is typically fixed.

Original: Cubic B-splines are 3 rd order polynomials, and thus require a minimumof 3 + 1 = 4 control points or basis functions.
Revised: Cubic B-splines are 3 rd order polynomials, and thus require a minimumof 3 + 1 = 4 control points or basis functions.

Original: Instead of considering arbitrary densities of uniformcubic B-splines, we look at powers of two times the minimum number of basis functions, calledρ -density B-spline functions.
Revised: Instead of considering arbitrary densities of uniform cubic B-splines, we look at powers of two times the minimum number of basis functions, calledρ -density B-spline functions.

Original: Definition 1 ( ρ -density B-spline function) .
Revised: Definition 1 ( ρ -density B-spline function) .

Original: A ρ -density B-spline function is a uniform cubic B-splinefunction with 2 ρ +2 basis [-functions:f ( x ) = 2 ρ +2 (cid:88) i = θ i S i ( x ) = 
 ρ-]
Revised: A ρ -density B-spline function is a uniform cubic B-splinefunction with 2 ρ +2 basis [-functions:f ( x ) = 2 ρ +2 (cid:88) i = θ i S i ( x ) = 
 ρ-]functions:

Original: +2 (cid:88) i

Original: = θ i S ( w i x + b i ) = 2 ρ +2 (cid:88) i = θ i S ((2 ρ +2 − 3) x + 4 − i )
Revised: [Equation]

Original: Consider the problem of expanding a single-variable function approximator with more basis functionsto increase its expressiveness.
Revised: Consider the problem of expanding a single-variable function approximator with more basis functionsto increase its expressiveness.

Original: Using the Fourier basis makes it trivially easy by adding higherfrequency sines and cosines with coefficients initialised to zero.
Revised: Using the Fourier basis makes it trivially easy by adding higherfrequency sines and cosines with coefficients initialised to zero.

Original: It is trickier to achieve somethingsimilar with cubic B-splines.
Revised: It is trickier to achieve somethingsimilar with cubic B-splines.

Original: There are algorithms for creating new splines from existing splineswith knot insertion, but the intermediate steps result in non-uniform knots and splines.
Revised: There are algorithms for creating new splines from existing splineswith knot insertion, but the intermediate steps result in non-uniform knots and splines.

Original: A simple andpractical compromise that we propose is to use mixtures of different ρ -density B-spline functions, asillustrated in Figure 4. [Figure]
Revised: A simple andpractical compromise that we propose is to use mixtures of different ρ -density B-spline functions, asillustrated in Figure 2.

Original: Definition 2 (mixed-density B-spline function) .
Revised: Definition 2 (mixed-density B-spline function) .

Original: A mixed-density B-spline function is a singlevariable function approximator that is obtained by summing together different ρ -density B-splinefunctions.
Revised: A mixed-density B-spline function is a singlevariable function approximator that is obtained by summing together different ρ -density B-splinefunctions.

Original: Only the maximum ρ -density B-spline function has trainable parameters, the others areconstant.
Revised: Only the maximum ρ -density B-spline function has trainable parameters, the others areconstant.

Original: Mixed-density B-spline functions are of the form:
Revised: Mixed-density B-spline functions are of the form:

Original: [Equation]
Revised: [Equation]

Original: Only the maximum r = ρ -density B-spline has trainable coefficients.
Revised: Only the maximum r = ρ -density B-spline has trainable coefficients.

Original: All lower density r > ρdensity B-spline have frozen and constant coefficients.
Revised: All lower density r > ρdensity B-spline have frozen and constant coefficients.

Original: The maximum r = ρ -density B-spline hastrainable coefficients with gradient updates that are orthogonal if the distance between two inputs islarge enough.
Revised: The maximum r = ρ -density B-spline hastrainable coefficients with gradient updates that are orthogonal if the distance between two inputs islarge enough.

Original: Similar to increasing the expressiveness of a Fourier basis function approximator by adding higherfrequency terms, one can add larger density cubic B-spline functions.
Revised: Similar to increasing the expressiveness of a Fourier basis function approximator by adding higherfrequency terms, one can add larger density cubic B-spline functions.

Original: Analytically, we can initialiseall the new parameters θ r +1 ,i = 0 , ∀ i ∈ N such that:
Revised: Analytically, we can initialiseall the new parameters θ r +1 ,i = 0 , ∀ i ∈ N such that:

Original: [Equation]
Revised: [Equation]

Original: It is therefore possible to create a minimal model with r = 0 initialised at zero, and train the modeluntil convergence.
Revised: It is therefore possible to create a minimal model with r = 0 initialised at zero, and train the modeluntil convergence.

Original: Then one can create a new model with r = 1 , by subsuming the previous model’sparameters, and train this more expressive model until convergence.
Revised: Then one can create a new model with r = 1 , by subsuming the previous model’sparameters, and train this more expressive model until convergence.

Original: This process of training andexpansion can be continued indefinitely, and is shown in Figure 5.
Revised: This process of training andexpansion can be continued indefinitely, and is shown in Figure 4.

Original:  ATLAS
Revised:  ATLAS

Original: ATLAS is named for carrying the burden of all it must remember, after the Titan god Atlas in Greekmythology who was tasked with holding the weight of the world.
Revised: ATLAS is named for carrying the burden of all it must remember, after the Titan god Atlas in Greekmythology who was tasked with holding the weight of the world.

Original: ATLAS is also an acronym for
Revised: ATLAS is also an acronym for

Original: AddiTive exponentiaL Additive Splines.
Revised: AddiTive exponentiaL Additive Splines.

Original: Definition 3 (ATLAS) .
Revised: Definition 3 (ATLAS) .

Original: ATLAS is a function approximator of n variables, with mixed-density
Revised: ATLAS is a function approximator of n variables, with mixed-density

Original: B-spline functions f j ( x j ) , g i,j ( x j ) , and h i,j ( x j ) in the form: 
Revised: B-spline functions f j ( x j ) , g i,j ( x j ) , and h i,j ( x j ) in the form: [Equation] ( ⃗ x ) :

Original: [Equation]
Revised: [Equation]

Original: ATLAS is equivalently given by the compact notation:
Revised: ATLAS is equivalently given by the compact notation:

Original: [Equation]
Revised: [Equation]

Original: The absolutely convergent series of scale factors k − 2 was chosen for numerical stability and to ensurethe model is absolutely convergent.
Revised: The absolutely convergent series of scale factors k − 2 was chosen for numerical stability and to ensurethe model is absolutely convergent.

Original: Another feature is that the series of scale factors also breaks thesymmetry that would otherwise exist if all mixed-density B-spline functions were initialised to zero.
Revised: Another feature is that the series of scale factors also breaks thesymmetry that would otherwise exist if all mixed-density B-spline functions were initialised to zero.

Original: Initialising all the parameters to be zero is a departure from the conventional approach of randominitialisation.
Revised: Initialising all the parameters to be zero is a departure from the conventional approach of randominitialisation.

Original: The number of exponential terms can be increased without changing the output of themodel.
Revised: The number of exponential terms can be increased without changing the output of themodel.

Original: We can choose to initialise G M +1 ( ⃗ x ) = 0 and H M +1 ( ⃗ x )
Revised: We can choose to initialise G M +1 ( ⃗ x ) = 0 and H M +1 ( ⃗ x )

Original: = 0 , such that the model capacitycan be increased without changing the output of the model.
Revised: = 0 , such that the model capacitycan be increased at will.

Original: ATLAS is a universal function approximator with robust memory retention.
Revised: ATLAS is a universal function approximator with robust memory retention.

Original: It possesses threeproperties atypical of most universal function approximators:
Revised: It possesses threeproperties atypical of most universal function approximators:

Original: The activity within ATLAS is sparse – most neural units are zero and inactive.
Revised: The activity within ATLAS is sparse – most neural units are zero and inactive.

Original: The gradient vector with respect to trainable parameters is bounded regardless of the size
Revised: The gradient vector with respect to trainable parameters is bounded regardless of the size

Original: and capacity of the model, so training is numerically stable for many possible training
Revised: and capacity of the model, so training is numerically stable for many possible training

Original: hyper-parameters.
Revised: hyper-parameters.

Original: Inputs that are sufficiently far from each other have orthogonal representations.
Revised: Inputs that are sufficiently far from each other have orthogonal representations.

Revised: The proofs of the three properties follows from the single-variable case, the assumption of boundedsingle-variable functions and parameters, and the absolutely convergent k − 2 scale factors.

Original: Property 1 (Sparsity) .
Revised: Property 1 (Sparsity) .

Original: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parameters θ i with indexset Θ , the gradient vector of trainable parameters (for ATLAS) is sparse:
Revised: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parameters θ i with indexset Θ , the gradient vector of trainable parameters (for ATLAS) is sparse:

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.
Revised: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.

Original: The gradient vector has a maximum of 4 n (2 M +1) non-zero entries, which is independentof r .
Revised: The gradient vector has a maximum of 4 n (2 M +1) non-zero entries, which is independentof r .

Original: Recall that only the maximum density ( ρ = r ) cubic B-spline function has trainable parameters.
Revised: Recall that only the maximum density ( ρ = r ) cubic B-spline function has trainable parameters.

Original: The fraction of trainable basis functions that are active is at most 2 − r .
Revised: The fraction of trainable basis functions that are active is at most 2 − r .

Original: Sparsity entails efficientimplementation, and suggests possible memory retention and robustness to catastrophic forgetting.
Revised: Sparsity entails efficientimplementation, and suggests possible memory retention and robustness to catastrophic forgetting.

Original: Property 2 (Gradient flow attenuation) .
Revised: Property 2 (Gradient flow attenuation) .

Original: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i with index set Θ : if all the mixed-density B-spline functions are bounded, then the gradient vectorof trainable parameters for ATLAS is bounded:
Revised: For any ⃗ x ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i with index set Θ : if all the mixed-density B-spline functions are bounded, then the gradient vectorof trainable parameters for ATLAS is bounded:

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.
Revised: For a fixed number of variables n , the model has a total of n 2 r +2 (2 M + 1) trainableparameters.

Original: The factor of k − 2 inside the expression for ATLAS is necessary to ensure the sum isconvergent in the limit of infinitely many exponential terms M → ∞ .
Revised: The factor of k − 2 inside the expression for ATLAS is necessary to ensure the sum isconvergent in the limit of infinitely many exponential terms M → ∞ .

Original: Only the maximum density( ρ = r ) cubic B-spline function has trainable parameters, so that the gradient vector is bounded in thelimit of arbitrarily large densities r → ∞ .
Revised: Only the maximum density( ρ = r ) cubic B-spline function has trainable parameters, so that the gradient vector is bounded in thelimit of arbitrarily large densities r → ∞ .

Original: It is worth recalling that at most four basis functions areactive for uniform cubic B-spline functions, regardless of the density, but the smaller densities cannotbe trainable, otherwise this property does not hold.
Revised: It is worth recalling that at most four basis functions areactive for uniform cubic B-spline functions, regardless of the density, but the smaller densities cannotbe trainable, otherwise this property does not hold.

Original: The gradient vector has bounded L1 norm for anynumber of basis functions and exponential terms.
Revised: The gradient vector has bounded L1 norm for anynumber of basis functions and exponential terms.

Original: The bounded gradient vector implies that ATLASis numerically stable during training, regardless of its size or parameter count.
Revised: The bounded gradient vector implies that ATLASis numerically stable during training, regardless of its size or parameter count.

Original: Property 3 (Distal orthogonality) .
Revised: Property 3 (Distal orthogonality) .

Original: For any ⃗ x , ⃗ y ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i , there exists a δ > 0 such that:
Revised: For any ⃗ x , ⃗ y ∈ D ( A ) ⊂ R n and bounded trainable parametersθ i for an ATLAS model A ( ⃗ x ) :

Original: [Equation]
Revised: [Equation]

Original: Remark.
Revised: Remark.

Original: Two points that sufficiently differ in each input variable have orthogonal parameter gradients.
Revised: Two points that sufficiently differ in each input variable have orthogonal parameter gradients.

Original: It is worth mentioning that the condition resembles a cross-like region in two variables, and planesthat intersect in higher dimensions.
Revised: A more general result holds for other choices of piece-wise defined single-variable functions.

Original: Distal orthogonality means ATLAS is reasonably robust tocatastrophic forgetting.
Revised: Distal orthogonality means ATLAS is reasonably robust to catastrophic forgetting, without otherregularisation and training techniques.

Revised: However, memory retention can still potentially be improvedwhen used in conjunction with other techniques.

Original: ATLAS can be implemented with 1D convolution, reshaping, embedding, multiplication and denselayers.
Revised: ATLAS can be implemented with 1D convolution, reshaping, embedding, multiplication and denselayers.

Original: The samekind of basis functions have to be computed for each input variable, hencethe choiceof 1D convolutions.
Revised: The same basis functions have to be computed for each input variable, hence 1D convolutions.

Original: By correctly scaling, shifting, and rounding inputs one can compute onlythenon-zero basis functions with the aid of convolution and embedding layers to lookup the parameters.
Revised: By correctly scaling, shifting, and rounding inputs one can compute only the non-zero basis functionswith embedding layers.

Original: ATLAS has time complexity O ( ρMn ) ,and space complexity O 
Revised: ATLAS models with input dimension n , output dimension p , O (2 r ) cubic

Original: (2 ρ Mn ) ,thanks to efficient usage ofembedding layers. It is worth mentioning that the engineering process settled on using 1D convolutionand embedding layers as an efficient method for approximating high-dimensional functions, whichpotentially suggests a deeper connection to other high-dimensional convolutional models (with smallfilters) and embedding layers.
Revised: B-spline basis functions, and M positive and negative exponential functions have time complexity O ( rMnp ) , and O (2 r Mnp ) space complexity.

Original:  Methodology
Revised:  Methodology

Original: The main point of the experimentation was to demonstrate that ATLAS exhibits distal orthogonality,and slightly improved memory retention without any regularisation or augmentation techniques.
Revised: The 2- and 8-dimensional models were considered for evaluation, in combination with a fixed widthfor the update region in Task 2 from 0 . 1 to 0 . 9 in increments of 0 . 1 .

Original: Demonstrating this for every possible target function is not feasible, thus a random set of problemswere considered instead. Two variants of ATLAS are considered for comparison: one with guaranteed distal orthogonality,and another without distal orthogonality that lets lower density (or lower frequency) basis functioncoefficients change.
Revised: 30 trials were performed for eachcombination update region width and model dimension. Mean Absolute Error (MAE) loss function,the Adam optimiser, and mini batch sizes of 100 are used throughout all experiemnts. At the beginning of each trial (for a given dimension and update region width) a random learning ratewas sampled uniformly between 10 − 6 and 0 .

Original: This modification leads to a variant that does not possess any of the threeproperties of ATLAS as outlined in the theory sections.
Revised: + 10 − 6 .

Original: The ATLAS and variant models have
Revised: A random noise level was sampled from anexponential distribution with scale parameter equal to one.

Original: M = 10 exponential terms that are strictly positive, and M = 10 exponential terms that are strictlypositive. The maximum density r = 4 = ⇒ 4 × 2 r = 64 is the number of basis functions for eachinput variable, with  the greatest expressive power.
Revised: The Task 1 target function is constructedfrom 1000 Euclidean radial basis functions (RBFs) with locations chosen uniformly over the entireinput domain, with RBF scale parameters sampled independently from an exponential distribution(scale parameter equal to 10).

Original: The lower density basis functions do not addexpressive power, and can alter training dynamics, potentially including greater off-target effects. To test memory retention, two tasks, presented to ATLAS one after the other, were constructed.
Revised: The weights of each radial basis function are sampled from a normaldistribution with mean zero and variance equal to one. The Task 2 target function is exactly the sameas the Task 1 target function - except for a square-like region with width equal to update region width.

Original: Thedetails of each task are given below. Task 1The target function is a randomly generated function with 1000 radial basis functions withpositions chosen uniformly over the input domain, using the Euclidean metric. The scale parametersand weights are also randomly sampled, and all radial basis functions are added together to obtain a target function.
Revised: The location of the update region is chosen uniformly at random, and such that it is completely insidethe domain of the model. The updated region masks the Task 1 target function and instead replacesthe values inside it with another function that is sampled from the same distribution as the Task 1target function, but independently from the Task 1 target function.

Original:  10000 data points are sampled for training, validation and testing.
Revised: After the generation of the target functions 10000 data points are sampled for training, validation, andtest sets for Task 1 and Task 2.

Revised: To simulate the effect of learning unrelated tasks, the training data for

Revised: Task 2 is only sampled from update region - with no training data outside of it being presented again,by contrast the validation and test sets for Task 2 were sampled over the entire input domain.

Original: All sampled datapoints have added Gaussian noise with a variance chosen randomly for each trial.
Revised: Gaussiannoise with standard deviation equal to the randomly chosen noise level is added to all training data.

Revised: An ATLAS model ( M = 10 positive and M = 10 negative exponential functions, maximum basisfunction density r = 4 ) with guaranteed distal orthogonality is trained and evaluated on Task 1and

Original: Task [-
-]
Revised: Task [-
-]2.

Original: The target function is the same as Task 1, except for a randomly positioned region of a fixedwidth, where the function values for Task 1 are replaced with another randomly sampled function of1000 radial basis functions that is sampled independently of the target function for Task 1.
Revised: Then a modified ATLAS model (

Original:  Results
Revised:  Results

Original: A randomly chosen trial is presented for visual inspection. As shown in Figure 6 and Figure 7 theeffect of distal orthogonality is clear and crisp boundaries, with the absolute change in the modeloutput before and after training on  Task 2.
Revised: As shown in Figure 5 the effect of distal orthogonality is clear and crisp boundaries that limit theeffect of Task 2 on the memory of Task 1. Without distal orthogonality there are more off-targeteffects that can be visualised. [Figure]

Original: Theabsolute difference between the target functions andthe 2nd output of each model is shown in Figure 8 and Figure 9. The effect of distal orthogonality on the averaged MAE for various trials is presented  in Figure 10and Figure 11.
Revised: The effect of distal orthogonality on the averaged MAE for various trials for 2- and 8-dimensionalproblems are presented as scatter plots of the averaged MAE over 30 trials for different update regionwidths as shown in Figure 6. The expected off-target error depends on the dimension of the problemand the width of the updated regions. [Figure]

Original: The removal of distal orthogonality lead to increased errors and  off-target effects onaverage.
Revised: Initial estimates of analytical results to the expected off-target error require simplification, but areasonable assumption in light of other evidence is that each input dimension has equal contributionon the unit hyper-cube. Assume for a fixed input dimension n and some region of width 0 < δ < 1where the target function Y is changed such that | ∆ Y | = 1 is one larger than it was originally. Theexpected off-target error depends on k the number of input variables inside the update region:

Original: Conclusion
Revised: [Equation] A binomial coefficient is needed to correctly account for all permutations with the same magintude ofchange: [Equation] One can calculate expected change values:

Revised: [Equation]

Revised: However if one assumes that the target function inside the updated region of width δ is correct, withprobability δ n of sampling from the entire input-domain, then the expected off-target error should be:

Revised: [Equation]

Revised: This seems consistent with some of the experimental results, but further investigation is needed.

Original: The main contribution of the paper is theoretical and technical.
Revised: The main contribution of the paper is theoretical and technical.

Original: A representation theorem is presentedthat outlines how to approximate multi-variable functions with single-variable functions, and exponential functions.
Revised: A representation theorem is presentedthat outlines how to approximate multi-variable functions with single-variable functions (splines andexponential functions).

Original: ATLAS approximates all arbitrary single-variable functions with mixed-density
Revised: ATLAS approximates all arbitrary single-variable functions with mixturesof B-spline functions.

Original: B-spline functions.
Revised: ATLAS is constructed in such a way that the gradient vector with respectto trainable parameters is bounded, regardless of how large an ATLAS model is. The activation ofunits in ATLAS is sparse, and allowed for an efficient implementation that only computes non-zeroactivation values with the aid of embedding layers.

Original: ATLAS is constructed in such a way that the gradient  vector with respect totrainable parameters is bounded, regardless of how large an ATLAS model is.
Revised: The gradient update vector with respect totrainable parameters is orthogonal for different inputs as long as the inputs are sufficiently differentfrom each other. ATLAS models with input dimension n , output dimension p , O (2 r ) cubic B-spline basis

Original: ATLAS was shownto exhibit intrinsic memory retention, except for a small cross-shaped region of interference andoff-target effects.
Revised: functions ,and M positive and negative exponential functions have

Revised: time complexity O ( rMnp ) , and O (2 r Mnp )

Revised: space complexity.

Revised: ATLAS was shown to exhibit some memory retention, without the assistance of other techniques.

Revised: This is a good indication of the potential for combining it with other techniques and models forcontinual learning.

Original: The chosen experiments demonstrated the theoretically derived predictions and contrasted two models,incuding a variant of ATLAS without distal orthogonality guarantees.
Revised: The chosen experiments demonstrated the theoretically derived predictions andcontrasted two models, incuding a variant of ATLAS without distal orthogonality guarantees.

Original: As far as societal impacts are concerned: It is possible that ATLAS could allow for the creation ofmore powerful machine learning algorithms, that require less resources to train and deploy.
Revised: As far as societal impacts are concerned: It is possible that ATLAS could allow for the creation ofmore powerful machine learning algorithms, that require less resources to train and deploy.

Original: Furthertesting is needed to make any concrete claim. 
Revised: Furthertesting is needed to make any concrete claim. Checklist

Revised: For all authors...

Revised: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Revised: contributions and scope?

Revised: [Yes]

Revised: (b) Did you describe the limitations of your work?

Revised: [Yes]

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes]

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Revised: [Yes]

Revised: If you are including theoretical results...

Revised: (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes]

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes]

Revised: If you ran experiments...

Revised: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Revised: mental results (either in the supplemental material or as a URL)?

Revised: [Yes]

Revised: (b) Did you specify all the training details (e

Revised: g., data splits, hyperparameters,

Revised: how they

Revised: were chosen)?

Revised: [Yes]

Revised: (c) Did you report error bars (e

Revised: g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes]

Revised: (d) Did you include the total amount of compute and the type of resources used (e

Revised: g., typeof GPUs, internal cluster, or cloud provider)?

Revised: [TODO]

Revised: If you are using existing assets (e

Revised: g., code, data, models) or curating/releasing new assets...

Revised: (a) If your work uses existing assets, did you cite the creators?

Revised: [TODO]

Revised: (b) Did you mention the license of the assets?

Revised: [TODO]

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [TODO]

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Revised: using/curating?

Revised: [N/A]

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Revised: [Table]

Revised: information or offensive content?

Revised: [N/A]

Revised: [Table]

Revised: If you used crowdsourcing or conducted research with human subjects...

Revised: [Table]

Revised: (a) Did you include the full text of instructions given to participants and screenshots, if

Revised: [Table]

Revised: applicable?

Revised: [N/A]

Revised: [Table]

Revised: (b) Did you describe any potential participant risks, with links to Institutional Review

Revised: [Table]

Revised: Board (IRB) approvals, if applicable?

Revised: [N/A]

Revised: [Table]

Revised: (c) Did you include the estimated hourly wage paid to participants and the total amountspent on participant compensation?

Revised: [N/A]