Original: C ONTINUOUS -D ISCRETE C ONVOLUTION FOR G EOMETRY -S EQUENCE M ODELING IN P ROTEINS
Revised: C ONTINUOUS -D ISCRETE C ONVOLUTION FOR G EOMETRY -S EQUENCE M ODELING IN P ROTEINS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The structure of proteins involves 3D geometry of amino acid coordinates and 1D sequence of peptide chains.
Revised: The structure of proteins involves 3D geometry of amino acid coordinates and 1D sequence of peptide chains.

Original: The 3D structure exhibits irregularity because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables.
Revised: The 3D structure exhibits irregularity because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables.

Original: In contrast, the 1D structure is regular because amino acids are arranged uniformly in the chains and their sequential positions (orders) are discrete variables.
Revised: In contrast, the 1D structure is regular because amino acids are arranged uniformly in the chains and their sequential positions (orders) are discrete variables.

Original: Moreover, geometric coordinates and sequential orders are in two types of spaces and their units of length are incompatible.
Revised: Moreover, geometric coordinates and sequential orders are in two types of spaces and their units of length are incompatible.

Original: These inconsistencies make it challenging to capture the 3D and 1D structures while avoiding the impact of sequence and geometry modeling on each other.
Revised: These inconsistencies make it challenging to capture the 3D and 1D structures while avoiding the impact of sequence and geometry modeling on each other.

Original: This paper proposes a Continuous-Discrete Convolution (CDConv) that uses irregular and regular approaches to model the geometry and sequence structures, respectively.
Revised: This paper proposes a Continuous-Discrete Convolution (CDConv) that uses irregular and regular approaches to model the geometry and sequence structures, respectively.

Original: Specifically, CDConv employs independent learnable weights for different regular sequential displacements but directly encodes geometric displacements due to their irregularity.
Revised: Specifically, CDConv employs independent learnable weights for different regular sequential displacements but directly encodes geometric displacements due to their irregularity.

Original: In this way, CDConv significantly improves protein modeling by reducing the impact of geometric irregularity on sequence modeling.
Revised: In this way, CDConv significantly improves protein modeling by reducing the impact of geometric irregularity on sequence modeling.

Original: Extensive experiments on a range of tasks, including protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction, demonstrate the effectiveness of the proposed CDConv.
Revised: Extensive experiments on a range of tasks, including protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction, demonstrate the effectiveness of the proposed CDConv.

Original: Our code will be publicly available.
Revised: Our code will be publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Proteins are large biomolecules and are essential for life.
Revised: Proteins are large biomolecules and are essential for life.

Original: Understanding their function is significant for life sciences.
Revised: Understanding their function is significant for life sciences.

Original: However, it usually requires enormous experimental efforts (Wüthrich, 2001;
Revised: However, it usually requires enormous experimental efforts (Wüthrich, 2001;

Original: Jaskolski et al., 2014; Bai et al., 2015; Thompson et al., 2020) to find out their function.
Revised: Jaskolski et al., 2014; Bai et al., 2015; Thompson et al., 2020) to find out their function.

Original: Recently, with the development of deep learning, emerging computational and data-driven approaches are particularly useful for efficient protein understanding (Derevyanko et al., 2018; Ingraham et al., 2019; Strokach et al., 2020; Cao et al., 2021;
Revised: Recently, with the development of deep learning, emerging computational and data-driven approaches are particularly useful for efficient protein understanding (Derevyanko et al., 2018; Ingraham et al., 2019; Strokach et al., 2020; Cao et al., 2021;

Original: Jing et al., 2021; Jumper et al., 2021; Shanehsazzadeh et al., 2020), including protein design, structure classification, model quality assessment, function prediction, etc .
Revised: Jing et al., 2021; Jumper et al., 2021; Shanehsazzadeh et al., 2020), including protein design, structure classification, model quality assessment, function prediction, etc .

Original: Because the function of proteins is based on their structure, accurately modeling protein structures can facilitate a mechanistic understanding of their function to life.
Revised: Because the function of proteins is based on their structure, accurately modeling protein structures can facilitate a mechanistic understanding of their function to life.

Original: Proteins are made up of different amino acids.
Revised: Proteins are made up of different amino acids.

Original: There are 20 types of amino acids (residues) commonly found in plants and animals and a typical protein is made up of 300 or more amino acids.
Revised: There are 20 types of amino acids (residues) commonly found in plants and animals and a typical protein is made up of 300 or more amino acids.

Original: Because these amino acids are linked by peptide bonds and form a chain (shown in Fig. 1), proteins exhibit a 1D sequence structure.
Revised: Because these amino acids are linked by peptide bonds and form a chain (shown in Fig. 1), proteins exhibit a 1D sequence structure.

Original: Moreover, because amino acids are arranged uniformly in the chains and their orders are discrete, the sequence structure is regular.
Revised: Moreover, because amino acids are arranged uniformly in the chains and their orders are discrete, the sequence structure is regular.

Original: In this case, 1D Convolutional Neural
Revised: In this case, 1D Convolutional Neural

Original: Network (CNN) (Kulmanov et al., 2018;
Revised: Network (CNN) (Kulmanov et al., 2018;

Original: Hou et al., 2018; Kulmanov & Hoehndorf, 2021) and Long
Revised: Hou et al., 2018; Kulmanov & Hoehndorf, 2021) and Long

Original: Short-Term Memory (LSTM) (Bepler & Berger, 2019;
Revised: Short-Term Memory (LSTM) (Bepler & Berger, 2019;

Original: Rao et al., 2019; Alley et al., 2019; Strodthoff et al., 2020) are widely used to model the regular 1D sequence structure of proteins.
Revised: Rao et al., 2019; Alley et al., 2019; Strodthoff et al., 2020) are widely used to model the regular 1D sequence structure of proteins.

Original: In addition to the 1D sequential order in peptide chains, each amino acid is with a 3D coordinate that specifies its spatial position in a protein.
Revised: In addition to the 1D sequential order in peptide chains, each amino acid is with a 3D coordinate that specifies its spatial position in a protein.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1, those 3D coordinates describe a geometry structure, which is also crucial for protein recognition.
Revised: 1, those 3D coordinates describe a geometry structure, which is also crucial for protein recognition.

Original: As mentioned by Alexander et al.
Revised: As mentioned by Alexander et al.

Original: (2009), proteins with similar peptide chains may fold into very different 3D geometry structures.
Revised: (2009), proteins with similar peptide chains may fold into very different 3D geometry structures.

Original: Conversely, proteins with similar 3D geometry structures may have entirely different amino acid chains (Agrawal & Kishan, 2001).
Revised: Conversely, proteins with similar 3D geometry structures may have entirely different amino acid chains (Agrawal & Kishan, 2001).

Original: Therefore, it is necessary to consider both the 1D and 3D structures in protein amino acid chain (1D sequence) amino acid coordinates (3D point set/cloud) protein (3+1)D structure (3D point list) + ⋯ protein data structure regular 1D sequentialorder irregular 3D coordinate aminoacidtype (3+1)D position feature 1 2 ⋮ 𝑁 18.09,5.83,18.78 16.25,9.24,18.87 −18.37,8.50,−6.
Revised: Therefore, it is necessary to consider both the 1D and 3D structures in protein amino acid chain (1D sequence) amino acid coordinates (3D point set/cloud) protein (3+1)D structure (3D point list) + ⋯ protein data structure regular 1D sequentialorder irregular 3D coordinate aminoacidtype (3+1)D position feature 1 2 ⋮ 𝑁 18.09,5.83,18.78 16.25,9.24,18.87 −18.37,8.50,−6.

Original: P F E ⋮ ⋮ regular 1D sequence
Revised: P F E ⋮ ⋮ regular 1D sequence

Original: + ⋯ regular 1D position irregular 3D coordinate
Revised: + ⋯ regular 1D position irregular 3D coordinate

Original: aminotype (3+1)D structure feature 1 2 ⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)
Revised: aminotype (3+1)D structure feature 1 2 ⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)

Original: P F E ⋮ ⋮
Revised: P F E ⋮ ⋮

Original: [Figure]
Revised: [Figure]

Original: protein (3+1)D structure protein data structure discrete1D order continuous 3D coordinate residue type (3+1)D position feature  ⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)
Revised: protein (3+1)D structure protein data structure discrete1D order continuous 3D coordinate residue type (3+1)D position feature  ⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)

Original: 𝑃 𝐹 𝐸 ⋮ ⋮
Revised: 𝑃 𝐹 𝐸 ⋮ ⋮

Original: [Figure]
Revised: [Figure]

Original: ⋯ modeling.
Revised: ⋯ modeling.

Original: However, different from the sequence structure, the geometry structure is irregular because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables.
Revised: However, different from the sequence structure, the geometry structure is irregular because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables.

Original: This increases the challenge for neural networks to understand proteins.
Revised: This increases the challenge for neural networks to understand proteins.

Original: To model the 1D and 3D structures in proteins, Gligorijevi ´ c et al. (2021) employed an LSTM and a
Revised: To model the 1D and 3D structures in proteins, Gligorijevi ´ c et al. (2021) employed an LSTM and a

Original: Graph Convolutional Network (GCN) for sequence and geometry modeling, respectively.
Revised: Graph Convolutional Network (GCN) for sequence and geometry modeling, respectively.

Original: Because the two structures are processed separately, it may not properly understand proteins’ local geometrysequence structure.
Revised: Because the two structures are processed separately, it may not properly understand proteins’ local geometrysequence structure.

Original: In contrast, a few unified networks try to model the two types of structures together (Baldassarre et al., 2021;
Revised: In contrast, a few unified networks try to model the two types of structures together (Baldassarre et al., 2021;

Original: Jing et al., 2021; Zhang et al., 2022; Hermosilla & Ropinski, 2022).
Revised: Jing et al., 2021; Zhang et al., 2022; Hermosilla & Ropinski, 2022).

Original: However, those methods process geometric and sequential displacements together or model the sequence structure in a similar way to geometry modeling, thus neglecting the regularity of the 1D structure.
Revised: However, those methods process geometric and sequential displacements together or model the sequence structure in a similar way to geometry modeling, thus neglecting the regularity of the 1D structure.

Original: Moreover, because the length units of the sequential and geometric spaces in proteins are not compatible, treating their distances similarly may mislead deep neural networks.
Revised: Moreover, because the length units of the sequential and geometric spaces in proteins are not compatible, treating their distances similarly may mislead deep neural networks.

Original: In this paper, we first propose and formulate a new class of convolution operation, named ContinuousDiscrete Convolution (CDConv), to make the most of the dual discrete and continuous nature of the data to avoid the impact of regular and irregular modeling on each other.
Revised: In this paper, we first propose and formulate a new class of convolution operation, named ContinuousDiscrete Convolution (CDConv), to make the most of the dual discrete and continuous nature of the data to avoid the impact of regular and irregular modeling on each other.

Original: Specifically, CDConv employs independent learnable weights to reflect different regular and discrete displacements but directly encodes continuous displacements due to their irregularity.
Revised: Specifically, CDConv employs independent learnable weights to reflect different regular and discrete displacements but directly encodes continuous displacements due to their irregularity.

Original: Then, we implement a (3+1)D
Revised: Then, we implement a (3+1)D

Original: CDConv and use this operation to construct a hierarchical (3+1)D CNN for geometry-sequence modeling in proteins.
Revised: CDConv and use this operation to construct a hierarchical (3+1)D CNN for geometry-sequence modeling in proteins.

Original: We apply our network to protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction.
Revised: We apply our network to protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction.

Original: Experimental results demonstrate the effectiveness of the proposed method.
Revised: Experimental results demonstrate the effectiveness of the proposed method.

Original: The contributions of this paper are fourfold:
Revised: The contributions of this paper are fourfold:

Original: • We propose a new class of convolution,
Revised: • We propose a new class of convolution,

Original: i
Revised: i

Original: e ., CDConv, which unifies continuous and discrete convolutions and makes the most of the regularity and irregularity in data, respectively.
Revised: e ., CDConv, which unifies continuous and discrete convolutions and makes the most of the regularity and irregularity in data, respectively.

Original: • We implement a (3+1)D CDConv for geometry-sequence modeling in proteins.
Revised: • We implement a (3+1)D CDConv for geometry-sequence modeling in proteins.

Original: Based on the convolution, we construct deep neural networks for protein representation learning.
Revised: Based on the convolution, we construct deep neural networks for protein representation learning.

Original: • We conduct extensive experiments on four tasks and the proposed method surpasses the previous methods by large margins, resulting in the new state-of-the-art accuracy.
Revised: • We conduct extensive experiments on four tasks and the proposed method surpasses the previous methods by large margins, resulting in the new state-of-the-art accuracy.

Original: • We find that amino acids in central regions may be
Revised: • We find that amino acids in central regions may be

Original: more important than those in surface areas.
Revised: more important than those in surface areas.

Original: This can be verified via experimental efforts in the future.
Revised: This can be verified via experimental efforts in the future.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Protein Representation Learning.
Revised: Protein Representation Learning.

Original: Protein representation learning attracts increasing attention in the fields of protein modeling and structural bioinformatics and is critical in biology.
Revised: Protein representation learning attracts increasing attention in the fields of protein modeling and structural bioinformatics and is critical in biology.

Original: Because proteins are sequences of amino acids, 1D CNN, LSTM and Transformer are employed for sequence-based protein representation learning (Amidi et al., 2018;
Revised: Because proteins are sequences of amino acids, 1D CNN, LSTM and Transformer are employed for sequence-based protein representation learning (Amidi et al., 2018;

Original: Kulmanov et al., 2018;
Revised: Kulmanov et al., 2018;

Original: Hou et al., 2018;
Revised: Hou et al., 2018;

Original: Rao et al., 2019; Bepler & Berger, 2019; Alley et al., 2019; Strodthoff et al., 2020; Shanehsazzadeh et al., 2020;
Revised: Rao et al., 2019; Bepler & Berger, 2019; Alley et al., 2019; Strodthoff et al., 2020; Shanehsazzadeh et al., 2020;

Original: Kulmanov & Hoehndorf, 2021).
Revised: Kulmanov & Hoehndorf, 2021).

Original: Besides the sequence structure, the 3D geometry structure is also used to enhance protein representations, based on amino acid or atom 3D coordinates (Derevyanko et al., 2018;
Revised: Besides the sequence structure, the 3D geometry structure is also used to enhance protein representations, based on amino acid or atom 3D coordinates (Derevyanko et al., 2018;

Original: Ingraham et al., 2019; Gligorijevi´c et al., 2021; Baldassarre et al., 2021; Jing et al., 2021; Wang et al., 2021; Hermosilla et al., 2021; Zhang et al., 2022; Chen et al., 2022; Hermosilla & Ropinski, 2022) or protein surfaces (Gainza et al., 2020; Sverrisson et al., 2021; Dallago et al., 2021).
Revised: Ingraham et al., 2019; Gligorijevi´c et al., 2021; Baldassarre et al., 2021; Jing et al., 2021; Wang et al., 2021; Hermosilla et al., 2021; Zhang et al., 2022; Chen et al., 2022; Hermosilla & Ropinski, 2022) or protein surfaces (Gainza et al., 2020; Sverrisson et al., 2021; Dallago et al., 2021).

Original: Our method is based on 1D sequential orders and 3D geometric coordinates of amino acids.
Revised: Our method is based on 1D sequential orders and 3D geometric coordinates of amino acids.

Original: Different from those existing methods, we propose to employ discrete and continuous manners to model regular orders and irregular coordinates, respectively.
Revised: Different from those existing methods, we propose to employ discrete and continuous manners to model regular orders and irregular coordinates, respectively.

Original: Discrete and Continuous Convolution.
Revised: Discrete and Continuous Convolution.

Original: Convolution is one of the most significant operations in the deep learning field and has made impressive achievements in many areas, including but not limited to computer vision and natural language processing.
Revised: Convolution is one of the most significant operations in the deep learning field and has made impressive achievements in many areas, including but not limited to computer vision and natural language processing.

Original: Convolution can be defined as functions on a discrete or continuous space.
Revised: Convolution can be defined as functions on a discrete or continuous space.

Original: For example, images can be seen in a 2D discrete space of pixels and thus a number of 2D discrete CNNs are used for image processing (Krizhevsky et al., 2012;
Revised: For example, images can be seen in a 2D discrete space of pixels and thus a number of 2D discrete CNNs are used for image processing (Krizhevsky et al., 2012;

Original: Simonyan & Zisserman, 2015; Szegedy et al., 2016; He et al., 2016).
Revised: Simonyan & Zisserman, 2015; Szegedy et al., 2016; He et al., 2016).

Original: Similarly, languages can be seen in a 1D discrete space of words or characters and therefore can be modeled by 1D discrete CNNs (Kim, 2014;
Revised: Similarly, languages can be seen in a 1D discrete space of words or characters and therefore can be modeled by 1D discrete CNNs (Kim, 2014;

Original: Yin et al., 2017; Bai et al., 2018).
Revised: Yin et al., 2017; Bai et al., 2018).

Original: Continuous convolutions are usually used to process irregular data (Schütt et al., 2017; Simonovsky & Komodakis, 2017;
Revised: Continuous convolutions are usually used to process irregular data (Schütt et al., 2017; Simonovsky & Komodakis, 2017;

Original: Wang et al., 2018; Romero et al., 2022), especially to 3D point clouds (Thomas et al., 2018; Wu et al., 2019;
Revised: Wang et al., 2018; Romero et al., 2022), especially to 3D point clouds (Thomas et al., 2018; Wu et al., 2019;

Original: Hu et al., 2020; Fuchs et al., 2020; Shi et al., 2021), or sometimes used to improve flexibility for discrete convolutions (Shocher et al., 2020).
Revised: Hu et al., 2020; Fuchs et al., 2020; Shi et al., 2021), or sometimes used to improve flexibility for discrete convolutions (Shocher et al., 2020).

Original: When implementing an instance of the proposed (3+1)D CDConv for protein modeling, the 3D processing part is inspired by those point cloud methods.
Revised: When implementing an instance of the proposed (3+1)D CDConv for protein modeling, the 3D processing part is inspired by those point cloud methods.

Original:  P ROPOSED C ONTINUOUS -D ISCRETE C ONVOLUTION
Revised:  P ROPOSED C ONTINUOUS -D ISCRETE C ONVOLUTION

Original: In this section, we first briefly review the discrete and continuous convolutions for regular and irregular structure modeling, respectively.
Revised: In this section, we first briefly review the discrete and continuous convolutions for regular and irregular structure modeling, respectively.

Original: Then, we present the proposed Continuous-Discrete Convolution (CDConv) in detail.
Revised: Then, we present the proposed Continuous-Discrete Convolution (CDConv) in detail.

Original: Third, we introduce the (3+1)D CDConv for geometry-sequence modeling in proteins.
Revised: Third, we introduce the (3+1)D CDConv for geometry-sequence modeling in proteins.

Original: Fourth, we present an implementation for the proposed (3+1)D CDConv.
Revised: Fourth, we present an implementation for the proposed (3+1)D CDConv.

Original: Last, we incorporate our (3+1)D CDConv into a hierarchical architecture for protein representation learning.
Revised: Last, we incorporate our (3+1)D CDConv into a hierarchical architecture for protein representation learning.

Original: 3. P RELIMINARY : C ONVOLUTION
Revised: 3. P RELIMINARY : C ONVOLUTION

Original: The power of convolutional neural networks (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical or pyramid architectures.
Revised: The power of convolutional neural networks (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical or pyramid architectures.

Original: Convolution is defined as the integral of the product of two functions after one is reversed and shifted.
Revised: Convolution is defined as the integral of the product of two functions after one is reversed and shifted.

Original: The convolution of f and g is written as follows,
Revised: The convolution of f and g is written as follows,

Original: [Equation]
Revised: [Equation]

Original: where Ω is a neighborhood in a space.
Revised: where Ω is a neighborhood in a space.

Original: In deep learning, f ( t ) usually is the feature at position t , i.e ., f ( t ) =
Revised: In deep learning, f ( t ) usually is the feature at position t , i.e ., f ( t ) =

Original: f t ∈ R C × 1 , where C is the number of input feature channels, and g ( τ ) ∈ R C
Revised: f t ∈ R C × 1 , where C is the number of input feature channels, and g ( τ ) ∈ R C

Original: ×
Revised: ×

Original: C is usually implemented as a parametric kernel function, where C ′ is the number of output feature channels.
Revised: C is usually implemented as a parametric kernel function, where C ′ is the number of output feature channels.

Original: Traditionally, convolutions are mainly used for image processing (Krizhevsky et al., 2012) and natural language processing (Kim, 2014).
Revised: Traditionally, convolutions are mainly used for image processing (Krizhevsky et al., 2012) and natural language processing (Kim, 2014).

Original: Because positions of pixels and words or characters are regular, discrete convolutions are used to model their structures as follows,
Revised: Because positions of pixels and words or characters are regular, discrete convolutions are used to model their structures as follows,

Original: [Equation]
Revised: [Equation]

Original: where · is matrix multiplication, and D is a regular and discrete neighborhood and can be seen as the kernel size of convolution.
Revised: where · is matrix multiplication, and D is a regular and discrete neighborhood and can be seen as the kernel size of convolution.

Original: The key to discrete convolution is to employ an independent group of learnable weights for every displacement ∆ .
Revised: The key to discrete convolution is to employ an independent group of learnable weights for every displacement ∆ .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2(a), for a 2D discrete convolution with kernel size 3 × 3 , there are nine independent groups of weights for the convolution: { θ ( − 1 , − 1) , θ ( − 1 , 0) , θ ( − 1 , 1) , θ (0 , − 1) , θ (0 , 0) , θ (0 , 1) , θ (1 , − 1) , θ (1 , 0) , θ (1 , 1) } .
Revised: 2(a), for a 2D discrete convolution with kernel size 3 × 3 , there are nine independent groups of weights for the convolution: { θ ( − 1 , − 1) , θ ( − 1 , 0) , θ ( − 1 , 1) , θ (0 , − 1) , θ (0 , 0) , θ (0 , 1) , θ (1 , − 1) , θ (1 , 0) , θ (1 , 1) } .

Original: For discrete convolution, g is usually an identity function.
Revised: For discrete convolution, g is usually an identity function.

Original: Continuous convolutions are mainly used to handle irregular data, especially to point clouds.
Revised: Continuous convolutions are mainly used to handle irregular data, especially to point clouds.

Original: For an irregular and continuous space, even in an extremely small neighborhood C , there can be countless
Revised: For an irregular and continuous space, even in an extremely small neighborhood C , there can be countless

Original: [Table]
Revised: [Table]

Original: 𝑟
Revised: 𝑟

Original: [Table]
Revised: [Table]

Original: 𝑟
Revised: 𝑟

Original: [Table]
Revised: [Table]

Original: −
Revised: −

Original: [Table]
Revised: [Table]

Original: ) 𝜽 (d) displacements.
Revised: ) 𝜽 (d) displacements.

Original: Therefore, it is impossible to assign an independent group of learnable weights for every displacement δ .
Revised: Therefore, it is impossible to assign an independent group of learnable weights for every displacement δ .

Original: In this case, the kernel function g is parameterized by the same weights θ for displacements but takes the displacement as an input,
Revised: In this case, the kernel function g is parameterized by the same weights θ for displacements but takes the displacement as an input,

Original: [Equation]
Revised: [Equation]

Original: We illustrate a continuous convolution in Fig. 2(b).
Revised: We illustrate a continuous convolution in Fig. 2(b).

Original: 3. C ONTINUOUS -D ISCRETE C ONVOLUTION
Revised: 3. C ONTINUOUS -D ISCRETE C ONVOLUTION

Original: Besides exclusively regular or irregular, some data may simultaneously exhibit both regularity and irregularity.
Revised: Besides exclusively regular or irregular, some data may simultaneously exhibit both regularity and irregularity.

Original: In this case, the neighborhood can be hybrid, i.e ., Ω = C
Revised: In this case, the neighborhood can be hybrid, i.e ., Ω = C

Original: × D , and each displacement in such a space consists of a discrete part ∆ and a continuous part δ .
Revised: × D , and each displacement in such a space consists of a discrete part ∆ and a continuous part δ .

Original: To model regular and irregular structures, we propose a continuous-discrete convolution (CDConv),
Revised: To model regular and irregular structures, we propose a continuous-discrete convolution (CDConv),

Original: [Equation]
Revised: [Equation]

Original: The crucial design of CDConv is to employ independent weights { θ ∆ } ∆ ∈D for different regular displacements ∆ but directly encode continuous displacements δ due to their irregularity.
Revised: The crucial design of CDConv is to employ independent weights { θ ∆ } ∆ ∈D for different regular displacements ∆ but directly encode continuous displacements δ due to their irregularity.

Original: In this way, CDConv facilitates regular-irregular modeling and reduces the impact of the irregularity of continuous displacements on the regular and discrete structure modeling.
Revised: In this way, CDConv facilitates regular-irregular modeling and reduces the impact of the irregularity of continuous displacements on the regular and discrete structure modeling.

Original: The proposed CDConv is a new class of convolution and we can derive different variants based on different combinations of continuous and discrete spaces.
Revised: The proposed CDConv is a new class of convolution and we can derive different variants based on different combinations of continuous and discrete spaces.

Original: In this paper, we define ( i + j )D as a hybrid space with an i D continuous structure and a j D discrete structure.
Revised: In this paper, we define ( i + j )D as a hybrid space with an i D continuous structure and a j D discrete structure.

Original: The (1+2)D and (2+1)D continuous-discrete spaces are illustrated in Fig. 2(c) and Fig. 2(d), respectively.
Revised: The (1+2)D and (2+1)D continuous-discrete spaces are illustrated in Fig. 2(c) and Fig. 2(d), respectively.

Original: 3. (3+1)D CDC ONV FOR P ROTEIN M ODELING
Revised: 3. (3+1)D CDC ONV FOR P ROTEIN M ODELING

Original: In a protein, the amino acids are linked by peptide bonds and form a chain.
Revised: In a protein, the amino acids are linked by peptide bonds and form a chain.

Original: Let ( f 1 , · · · , f N ) denote the features of the amino acids in a protein, where f t ∈ R C × 1 denotes the feature of the t -th amino 𝑡 +𝑡 𝑡 −𝑡 +𝑡 + 4 𝑡 + 𝑡 +(a) −    2 −  𝑟 0 − 𝑙 2 = − 𝑙 2 =𝑙 =(b) 𝑔(𝒑 𝑡−1 − 𝒑 𝑡 ;𝜽 −1 ) ∙𝒇 𝑡− 𝑔(𝒑 𝑡+4 −𝒑 𝑡 ;𝜽 4 ) ∙𝒇 𝑡+ 𝑔(𝟎;𝜽 0 )
Revised: Let ( f 1 , · · · , f N ) denote the features of the amino acids in a protein, where f t ∈ R C × 1 denotes the feature of the t -th amino 𝑡 +𝑡 𝑡 −𝑡 +𝑡 + 4 𝑡 + 𝑡 +(a) −    2 −  𝑟 0 − 𝑙 2 = − 𝑙 2 =𝑙 =(b) 𝑔(𝒑 𝑡−1 − 𝒑 𝑡 ;𝜽 −1 ) ∙𝒇 𝑡− 𝑔(𝒑 𝑡+4 −𝒑 𝑡 ;𝜽 4 ) ∙𝒇 𝑡+ 𝑔(𝟎;𝜽 0 )

Original: ∙𝒇 𝑡 ⨁ (c) multiplication acid
Revised: ∙𝒇 𝑡 ⨁ (c) multiplication acid

Original: and N is the number of amino acids.
Revised: and N is the number of amino acids.

Original: In addition to the 1D order in the sequence, each amino acid has a 3D coordinate that specifies its spatial position, i.e ., p t ∈ R 3 × 1 .
Revised: In addition to the 1D order in the sequence, each amino acid has a 3D coordinate that specifies its spatial position, i.e ., p t ∈ R 3 × 1 .

Original: The values of 1D sequential orders are regular and discrete while those of 3D coordinates are irregular and continuous non-integers.
Revised: The values of 1D sequential orders are regular and discrete while those of 3D coordinates are irregular and continuous non-integers.

Original: In this case, amino acids can be seen in a (3+1)D continuous-discrete space and we can employ the proposed convolution to capture the structure of a geometry-sequence neighborhood, resulting in the (3+1)D CDConv, (3+1)D CDConv: f ′ t = (cid:88)
Revised: In this case, amino acids can be seen in a (3+1)D continuous-discrete space and we can employ the proposed convolution to capture the structure of a geometry-sequence neighborhood, resulting in the (3+1)D CDConv, (3+1)D CDConv: f ′ t = (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where the neighborhood is defined with a 1D sequential kernel size l and a 3D geometric radius r .
Revised: where the neighborhood is defined with a 1D sequential kernel size l and a 3D geometric radius r .

Original: We illustrate how (3+1)D CDConv captures the geometry-sequence structure of a local area in Fig.
Revised: We illustrate how (3+1)D CDConv captures the geometry-sequence structure of a local area in Fig.

Original: Discussion
Revised: Discussion

Original: Comparison with existing methods.
Revised: Comparison with existing methods.

Original: The proposed (3+1)D CDConv is substantially different from existing methods that usually process geometric and sequential displacement together or in similar irregular manners.
Revised: The proposed (3+1)D CDConv is substantially different from existing methods that usually process geometric and sequential displacement together or in similar irregular manners.

Original: For example, GraphQA (Baldassarre et al., 2021) concatenates the 3D distance with a radial basis function and the 1D distance with the one-hot representation.
Revised: For example, GraphQA (Baldassarre et al., 2021) concatenates the 3D distance with a radial basis function and the 1D distance with the one-hot representation.

Original: The edge in
Revised: The edge in

Original: GVP (Jing et al., 2021) embeds the unit vector in the 3D direction, the 3D distance with a radial basis function and the sinusoidal encoding of the 1D displacement.
Revised: GVP (Jing et al., 2021) embeds the unit vector in the 3D direction, the 3D distance with a radial basis function and the sinusoidal encoding of the 1D displacement.

Original: IEConv (Hermosilla & Ropinski, 2022) encodes normalized 1D displacements.
Revised: IEConv (Hermosilla & Ropinski, 2022) encodes normalized 1D displacements.

Original: GearNet
Revised: GearNet

Original: (Zhang et al., 2022) directly concatenates 3D and 1D distances.
Revised: (Zhang et al., 2022) directly concatenates 3D and 1D distances.

Original: Those methods can be summarised as γ ([ δ, ∆]) or [ γ ( δ ) , η (∆)] , where [ · , · ] demotes concatenation and γ and η are encoding functions.
Revised: Those methods can be summarised as γ ([ δ, ∆]) or [ γ ( δ ) , η (∆)] , where [ · , · ] demotes concatenation and γ and η are encoding functions.

Original: In contrast, (3+1)D CDConv does not encode or embed 1D displacements but employs independent weights for different discrete displacements.
Revised: In contrast, (3+1)D CDConv does not encode or embed 1D displacements but employs independent weights for different discrete displacements.

Original: In this way, CDConv makes the most of the regularity of sequential orders.
Revised: In this way, CDConv makes the most of the regularity of sequential orders.

Original: Moreover, because the units of length of the sequential and geometric spaces are not compatible, capturing the two structures in different ways improves the proposed CDConv for protein understanding.
Revised: Moreover, because the units of length of the sequential and geometric spaces are not compatible, capturing the two structures in different ways improves the proposed CDConv for protein understanding.

Original: Comparison with two-stream 3D-CNN+1D-CNN architectures.
Revised: Comparison with two-stream 3D-CNN+1D-CNN architectures.

Original: To make the most of the regularity of sequential orders and the irregularity of geometric coordinates, we can first employ an independent 3D continuous CNN branch and an independent 1D discrete CNN branch to extract the global geometric and sequential features, respectively, and then fuse the two features as the final protein representation (illustrated in Appendix D).
Revised: To make the most of the regularity of sequential orders and the irregularity of geometric coordinates, we can first employ an independent 3D continuous CNN branch and an independent 1D discrete CNN branch to extract the global geometric and sequential features, respectively, and then fuse the two features as the final protein representation (illustrated in Appendix D).

Original: However, because the two types of structures in proteins are modeled separately, the local (3+1)D structure is not properly captured.
Revised: However, because the two types of structures in proteins are modeled separately, the local (3+1)D structure is not properly captured.

Original: In contrast, CNNs based on our (3+1)D CDConv can simultaneously model the local geometry-sequence structure and thus extract more discriminative local features than 3D-CNN+1D-CNNs.
Revised: In contrast, CNNs based on our (3+1)D CDConv can simultaneously model the local geometry-sequence structure and thus extract more discriminative local features than 3D-CNN+1D-CNNs.

Original: As shown in Sec.
Revised: As shown in Sec.

Original: 4.5.2, our (3+1)D
Revised: 4.5.2, our (3+1)D

Original: CDConv-based CNN significantly outperforms the corresponding 3D-CNN+1D-CNN approach.
Revised: CDConv-based CNN significantly outperforms the corresponding 3D-CNN+1D-CNN approach.

Original: 3. A N I MPLEMENTATION FOR (3+1)D CDC ONV
Revised: 3. A N I MPLEMENTATION FOR (3+1)D CDC ONV

Original: There can be many ways to implement the kernel function g for the proposed (3+1)D CDConv.
Revised: There can be many ways to implement the kernel function g for the proposed (3+1)D CDConv.

Original: In this paper, inspired by PointConv (Wu et al., 2019), we implement g as follows,
Revised: In this paper, inspired by PointConv (Wu et al., 2019), we implement g as follows,

Original: [Equation]
Revised: [Equation]

Original: However, the above implementation is not rotationally invariant.
Revised: However, the above implementation is not rotationally invariant.

Original: Inspired by Hermosilla & Ropinski (2022), we replace the 3D displacement encoding with relative spatial encoding (Ingraham et al., 2019) as follows,
Revised: Inspired by Hermosilla & Ropinski (2022), we replace the 3D displacement encoding with relative spatial encoding (Ingraham et al., 2019) as follows,

Original: [Equation]
Revised: [Equation]

Original: where O t = ( b t , n t , b t × n t ) and u t = p t − p t − 1 ∥
Revised: where O t = ( b t , n t , b t × n t ) and u t = p t − p t − 1 ∥

Original: p t − p t − 1 ∥ , b t = u t − u t +1 ∥
Revised: p t − p t − 1 ∥ , b t = u t − u t +1 ∥

Original: u t − u t +1 ∥ , n t =
Revised: u t − u t +1 ∥ , n t =

Original: u t × u t +1 ∥
Revised: u t × u t +1 ∥

Original: u t × u t +1 ∥ and θ ∆ ∈ R C ′ × 7 .
Revised: u t × u t +1 ∥ and θ ∆ ∈ R C ′ × 7 .

Original: Finally, our (3+1)D CDConv is implemented as follows,
Revised: Finally, our (3+1)D CDConv is implemented as follows,

Original: [Equation]
Revised: [Equation]

Original: The matrix multiplication result is then reshaped from R C ′
Revised: The matrix multiplication result is then reshaped from R C ′

Original: × C to R C ′ C
Revised: × C to R C ′ C

Original: × 1 .
Revised: × 1 .

Original: 3. C ONTINOUS -D ISCRET C ONVOLUTIONAL N EURAL N ETWORK
Revised: 3. C ONTINOUS -D ISCRET C ONVOLUTIONAL N EURAL N ETWORK

Original: Based on the proposed CDConv, hierarchical deep neural networks can be built for global protein representation learning.
Revised: Based on the proposed CDConv, hierarchical deep neural networks can be built for global protein representation learning.

Original: To this end, we stack multiple CDConv layers and downsample amino acids as the number of layers increases.
Revised: To this end, we stack multiple CDConv layers and downsample amino acids as the number of layers increases.

Original: To construct hierarchical networks, late layers should have larger receptive fields than early layers.
Revised: To construct hierarchical networks, late layers should have larger receptive fields than early layers.

Original: For the 1D sequential kernel size l , we fix it because convolutions at late layers will capture relatively longer sequential regions due to amino acid downsampling.
Revised: For the 1D sequential kernel size l , we fix it because convolutions at late layers will capture relatively longer sequential regions due to amino acid downsampling.

Original: However, for the geometry modeling, we need to progressively enlarge the radius of each layer so that late layers have larger receptive fields.
Revised: However, for the geometry modeling, we need to progressively enlarge the radius of each layer so that late layers have larger receptive fields.

Original: Following Hermosilla et al.
Revised: Following Hermosilla et al.

Original: (2021) and Hermosilla & Ropinski (2022), we build a ResNet (He et al., 2016) with multiple ( h ) CDConv layers.
Revised: (2021) and Hermosilla & Ropinski (2022), we build a ResNet (He et al., 2016) with multiple ( h ) CDConv layers.

Original: After every two CDConv layers, a sequence sum pooling with downsampling rate 2 is performed to halve the protein resolution and build a pyramid architecture.
Revised: After every two CDConv layers, a sequence sum pooling with downsampling rate 2 is performed to halve the protein resolution and build a pyramid architecture.

Original: For the i -th CDConv layer, the radius is set to ⌈ i 2 + 1 ⌉ r o and the output dimension is set to ⌈ i 

Revised: For the i -th CDConv layer, the radius is set to ⌈ i 2 + 1 ⌉ r o and the output dimension is set to ⌈ i

Original: ⌉ c o , where r o and c
Revised: ⌉ c o , where r o and c

Original: o are the initial radius and the initial number of feature channels, respectively.
Revised: o are the initial radius and the initial number of feature channels, respectively.

Original: In this way, the geometric receptive field and feature dimension progressively increase.
Revised: In this way, the geometric receptive field and feature dimension progressively increase.

Original:  E XPERIMENTS 4. E VALUATION T ASKS AND D ATASETS
Revised:  E XPERIMENTS 4. E VALUATION T ASKS AND D ATASETS

Original: Following Hermosilla et al.
Revised: Following Hermosilla et al.

Original: (2021); Hermosilla & Ropinski (2022); Zhang et al.
Revised: (2021); Hermosilla & Ropinski (2022); Zhang et al.

Original: (2022), we evaluate the proposed method on four tasks: protein fold classification, enzyme reaction classification, gene ontology (GO) term prediction and enzyme commission (EC) number prediction.
Revised: (2022), we evaluate the proposed method on four tasks: protein fold classification, enzyme reaction classification, gene ontology (GO) term prediction and enzyme commission (EC) number prediction.

Original: Protein fold classification includes three evaluation scenarios: fold, superfamily and family.
Revised: Protein fold classification includes three evaluation scenarios: fold, superfamily and family.

Original: GO term prediction includes three sub-tasks: biological process (BP), molecular function (MF) and cellular component (CC) ontology term prediction.
Revised: GO term prediction includes three sub-tasks: biological process (BP), molecular function (MF) and cellular component (CC) ontology term prediction.

Original: More details about these tasks and datasets are shown in Appendix B.
Revised: More details about these tasks and datasets are shown in Appendix B.

Original: Protein fold and enzyme reaction classification are single-label classification tasks.
Revised: Protein fold and enzyme reaction classification are single-label classification tasks.

Original: Mean accuracy is used as the evaluation metric.
Revised: Mean accuracy is used as the evaluation metric.

Original: GO term and EC number prediction are multi-label classification tasks.
Revised: GO term and EC number prediction are multi-label classification tasks.

Original: The F max accuracy is used as the evaluation metric (details are described in Appendix C).
Revised: The F max accuracy is used as the evaluation metric (details are described in Appendix C).

Original: 4. T RAINING S ETUP
Revised: 4. T RAINING S ETUP

Original: We implement our method with the PyTorch-Geometric (PyG) 1 library.
Revised: We implement our method with the PyTorch-Geometric (PyG) 1 library.

Original: The number of CDConv layers h is set to 8.
Revised: The number of CDConv layers h is set to 8.

Original: The sequential kernel size l is set to 11 for fold classification, 25 for reaction
Revised: The sequential kernel size l is set to 11 for fold classification, 25 for reaction

Original: [Table]
Revised: [Table]

Original: classification and 15 for GO term and EC number prediction.
Revised: classification and 15 for GO term and EC number prediction.

Original: The initial radius r o is set to 4.
Revised: The initial radius r o is set to 4.

Original: More details of implementation and training setup are provided in Appendix E.
Revised: More details of implementation and training setup are provided in Appendix E.

Original: 4. C OMPARISON WITH S TATE - OF - THE -A RT
Revised: 4. C OMPARISON WITH S TATE - OF - THE -A RT

Original: We compare our method with existing 1D-only, 3D-only and (3+1)D methods.
Revised: We compare our method with existing 1D-only, 3D-only and (3+1)D methods.

Original: Results are shown in
Revised: Results are shown in

Original: Table 1 and Table 2.
Revised: Table 1 and Table 2.

Original: The IEConv method includes an atom-level variant (Hermosilla et al., 2021) and an amino-acid-level variant (Hermosilla & Ropinski, 2022).
Revised: The IEConv method includes an atom-level variant (Hermosilla et al., 2021) and an amino-acid-level variant (Hermosilla & Ropinski, 2022).

Original: Because the atom-level IEConv is based on the 3D coordinates of atoms and the topology structure of bonds between atoms, we denote the input of atom-level IEConv as “3D+Topo”.
Revised: Because the atom-level IEConv is based on the 3D coordinates of atoms and the topology structure of bonds between atoms, we denote the input of atom-level IEConv as “3D+Topo”.

Original: Note that none of the existing methods achieved state-of-the-art accuracy on all of the four tasks.
Revised: Note that none of the existing methods achieved state-of-the-art accuracy on all of the four tasks.

Original: In contrast, our method significantly outperforms all the existing methods.
Revised: In contrast, our method significantly outperforms all the existing methods.

Original: For example, on the superfamily fold classification, our CDConv outperforms the previous state-of-the-art method, i.e .,
Revised: For example, on the superfamily fold classification, our CDConv outperforms the previous state-of-the-art method, i.e .,

Original: GearNet-Edge-IEConv, by 6.9%.
Revised: GearNet-Edge-IEConv, by 6.9%.

Original: On the BP ontology term prediction, our CDConv outperforms the previous best method, i.e ., the residue-level IEConv, by 3.2%.
Revised: On the BP ontology term prediction, our CDConv outperforms the previous best method, i.e ., the residue-level IEConv, by 3.2%.

Original: This demonstrates the effectiveness of the proposed CDConv that employs discrete and continuous approaches for regular sequence and irregular geometry modeling, respectively.
Revised: This demonstrates the effectiveness of the proposed CDConv that employs discrete and continuous approaches for regular sequence and irregular geometry modeling, respectively.

Original: 4. H OW DOES C ONTINUOUS -D ISCRETE CNN C APTURE P ROTEIN S TRUCTURE ?
Revised: 4. H OW DOES C ONTINUOUS -D ISCRETE CNN C APTURE P ROTEIN S TRUCTURE ?

Original: To investigate the behavior of our CNN, we visualize the outputs of CDConvs.
Revised: To investigate the behavior of our CNN, we visualize the outputs of CDConvs.

Original: We treat the norm ∥ f ∥ of the output features as the activation strength of amino acids and visualize the activation strength in Fig. 4.
Revised: We treat the norm ∥ f ∥ of the output features as the activation strength of amino acids and visualize the activation strength in Fig. 4.

Original: Visualizations of proteins of different sizes for all CDConv layers are illustrated in
Revised: Visualizations of proteins of different sizes for all CDConv layers are illustrated in

Original: Appendix R. Recall that, to build a pyramid architecture, the CNN downsamples amino acids after every two layers.
Revised: Appendix S. Recall that, to build a pyramid architecture, the CNN downsamples amino acids after every two layers.

Original: The early layers ( e.g ., CDConv 2 and CDConv 4) tend to focus on the central region of the protein.
Revised: The early layers ( e.g ., CDConv 2 and CDConv 4) tend to focus on the central region of the protein.

Original: Then, the activation spreads to the entire protein so that the late layers ( e.g ., CDConvand CDConv 8) can capture the whole structure.
Revised: Then, the activation spreads to the entire protein so that the late layers ( e.g ., CDConvand CDConv 8) can capture the whole structure.

Original: This indicates that the central structure may provide the most informative clues in early recognition and may be more important than other areas.
Revised: This indicates that the central structure may provide the most informative clues in early recognition and may be more important than other areas.

Original: To verify the explanation for the above phenomenon, we mask the same number (20%) of amino acids in the central and surface areas, respectively.
Revised: To verify the explanation for the above phenomenon, we mask the same number (20%) of amino acids in the central and surface areas, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5(a), masking the central area leads to more accuracy drops, indicating that amino acids in central regions may be more important than those in surface areas.
Revised: 5(a), masking the central area leads to more accuracy drops, indicating that amino acids in central regions may be more important than those in surface areas.

Original: More details and results are shown in Appendix J.
Revised: More details and results are shown in Appendix J.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: The early layers tend to focus on the central region and the late layers try to capture the whole structure.
Revised: The early layers tend to focus on the central region and the late layers try to capture the whole structure.

Original: This may indicate that the central region is more important than other areas to proteins.
Revised: This may indicate that the central region is more important than other areas to proteins.

Original: 4. A BLATION S TUDY 4.5. I MPACT OF C ONTINUOUS G EOMETRY AND D ISCRETE S EQUENCE M ODELING
Revised: 4. A BLATION S TUDY 4.5. I MPACT OF C ONTINUOUS G EOMETRY AND D ISCRETE S EQUENCE M ODELING

Original: In this paper, we propose to use continuous and discrete approaches to model the geometry and sequence structures, respectively.
Revised: In this paper, we propose to use continuous and discrete approaches to model the geometry and sequence structures, respectively.

Original: To verify the effectiveness, we conduct cross-experiments.
Revised: To verify the effectiveness, we conduct cross-experiments.

Original: To model the regular sequence structure in a continuous manner, we directly encode the sequential displacement ∆ .
Revised: To model the regular sequence structure in a continuous manner, we directly encode the sequential displacement ∆ .

Original: To capture the irregular geometry structure in a discrete manner, 3D coordinates are voxelized with size 0.5.
Revised: To capture the irregular geometry structure in a discrete manner, 3D coordinates are voxelized with size 0.5.

Original: Results are shown in Table 3.
Revised: Results are shown in Table 3.

Original: First, when replacing discrete modeling with directly encoding on the sequence structure capture, the accuracy drops.
Revised: First, when replacing discrete modeling with directly encoding on the sequence structure capture, the accuracy drops.

Original: This verifies our motivation that regular structure should be modeled in a regular manner.
Revised: This verifies our motivation that regular structure should be modeled in a regular manner.

Original: Second, when using the discrete method to modeling the geometry structure, the accuracy drops significantly.
Revised: Second, when using the discrete method to modeling the geometry structure, the accuracy drops significantly.

Original: This is because quantization errors are inevitable during voxelization and voxelized coordinates cannot precisely describe the geometry structure.
Revised: This is because quantization errors are inevitable during voxelization and voxelized coordinates cannot precisely describe the geometry structure.

Original: 4.5. C OMPARISON WITH T WO -S TREAM 3D-CNN+1D-CNN
Revised: 4.5. C OMPARISON WITH T WO -S TREAM 3D-CNN+1D-CNN

Original: To avoid the impact of irregular and regular modeling on each other, one can employ an independent continuous 3D-CNN and an independent discrete 1D-CNN to build a two-stream network, resulting in 3D-CNN+1D-CNN.
Revised: To avoid the impact of irregular and regular modeling on each other, one can employ an independent continuous 3D-CNN and an independent discrete 1D-CNN to build a two-stream network, resulting in 3D-CNN+1D-CNN.

Original: We compare our CDConv-based CNN with the corresponding 3D-CNN+1DCNN.
Revised: We compare our CDConv-based CNN with the corresponding 3D-CNN+1DCNN.

Original: The 3D-CNN is implemented by PointConv with relative spatial encoding.
Revised: The 3D-CNN is implemented by PointConv with relative spatial encoding.

Original: Each of the 3D-CNN stream and the 1D-CNN stream consist of the same number of layers as our CDConv-based
Revised: Each of the 3D-CNN stream and the 1D-CNN stream consist of the same number of layers as our CDConv-based

Original: CNN’s.
Revised: CNN’s.

Original: As shown in Table 3, our CDConv-based CNN surpasses the 3D-CNN+1D-CNN by a large margin.
Revised: As shown in Table 3, our CDConv-based CNN surpasses the 3D-CNN+1D-CNN by a large margin.

Original: Moreover, compared to the 3D geometry-only continuous CNN, the two-stream method
Revised: Moreover, compared to the 3D geometry-only continuous CNN, the two-stream method

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: does not significantly improve the accuracy or even leads to worse accuracy.
Revised: does not significantly improve the accuracy or even leads to worse accuracy.

Original: These indicate that the 3D-CNN+1D-CNN strategy is not effective to capture the local (3+1)D structure of proteins and the global representations of the two structures may not benefit each other.
Revised: These indicate that the 3D-CNN+1D-CNN strategy is not effective to capture the local (3+1)D structure of proteins and the global representations of the two structures may not benefit each other.

Original: 4.5. I MPACT OF G EOMETRIC R ADIUS AND S EQUENTIAL K ERNEL S IZE
Revised: 4.5. I MPACT OF G EOMETRIC R ADIUS AND S EQUENTIAL K ERNEL S IZE

Original: The geometric radius and sequential kernel size control the range of the spatial and chain structure to be modeled, respectively.
Revised: The geometric radius and sequential kernel size control the range of the spatial and chain structure to be modeled, respectively.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5(b) and Fig. 5(c), using too small radius and kernel size cannot capture sufficient structure information and using too large radius and kernel size will decrease the discriminativeness of spatial local structure for modeling, thus leading to inferior accuracy.
Revised: 5(b) and Fig. 5(c), using too small radius and kernel size cannot capture sufficient structure information and using too large radius and kernel size will decrease the discriminativeness of spatial local structure for modeling, thus leading to inferior accuracy.

Original: The impact of the number of channels and CDConv layers are shown in Appendix F and in Appendix G.
Revised: The impact of the number of channels and CDConv layers are shown in Appendix F and in Appendix G.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We first propose a new class of convolutions, i.e ., CDConv, which uses continuous and discrete approaches to model the irregular and regular structures in data, respectively.
Revised: We first propose a new class of convolutions, i.e ., CDConv, which uses continuous and discrete approaches to model the irregular and regular structures in data, respectively.

Original: Then, we implement a (3+1)D CDConv to model the sequence and geometry structures in proteins.
Revised: Then, we implement a (3+1)D CDConv to model the sequence and geometry structures in proteins.

Original: We apply CDConv to a range of protein modeling tasks and achieve the new state-of-the-art accuracy.
Revised: We apply CDConv to a range of protein modeling tasks and achieve the new state-of-the-art accuracy.

Original: Moreover, by investigating the behavior of CDConv, we find that amino acids in central regions may be more important than those in surface areas to proteins, which can be further verified via experimental efforts.
Revised: Moreover, by investigating the behavior of CDConv, we find that amino acids in central regions may be more important than those in surface areas to proteins, which can be further verified via experimental efforts.

Original: As a new class of convolutions, the applications of CDConv are not limited to protein modeling and can be other scenarios where data exhibits both regularity and irregularity.
Revised: As a new class of convolutions, the applications of CDConv are not limited to protein modeling and can be other scenarios where data exhibits both regularity and irregularity.

Original: The proposed (3+1)D
Revised: The proposed (3+1)D

Original: CDConv can also be used to recognize non-coding RNAs and may be potentially improved with different implementations.
Revised: CDConv can also be used to recognize non-coding RNAs and may be potentially improved with different implementations.

Original: However, as a convolution, our CDConv requires a predefined and fixed geometric radius and sequential kernel size to capture local structure, which is somewhat rigid.
Revised: However, as a convolution, our CDConv requires a predefined and fixed geometric radius and sequential kernel size to capture local structure, which is somewhat rigid.

Original: This can be alleviated in future studies by integrating self-attention to adaptively capture the local structure.
Revised: This can be alleviated in future studies by integrating self-attention to adaptively capture the local structure.

Original: A P ROBLEM F ORMULATION
Revised: A P ROBLEM F ORMULATION

Original: To evaluate the proposed CDConv, we apply it to the protein classification problem.
Revised: To evaluate the proposed CDConv, we apply it to the protein classification problem.

Original: Classification is a fundamental task for pattern recognition and representation learning.
Revised: Classification is a fundamental task for pattern recognition and representation learning.

Original: Specifically, given a protein, which consists of a list of amino acid coordinates P ∈ R 3 × N and a list of amino acid types
Revised: Specifically, given a protein, which consists of a list of amino acid coordinates P ∈ R 3 × N and a list of amino acid types

Original: F ∈ R 1 × N , where N is the number of amino acids, the goal of protein classification is to map P and F to a prediction vector
Revised: F ∈ R 1 × N , where N is the number of amino acids, the goal of protein classification is to map P and F to a prediction vector

Original: y ∈
Revised: y ∈

Original: R
Revised: R

Original: k , where k is the number of protein classes or categories, y =
Revised: k , where k is the number of protein classes or categories, y =

Original: Φ( P , F ; Θ) .
Revised: Φ( P , F ; Θ) .

Original: The Φ is a deep neural network and Θ is its parameters.
Revised: The Φ is a deep neural network and Θ is its parameters.

Original: Then, for single-label classification, the softmax function is usually used to convert y to prediction probabilities, i.e ., softmax( y ) .
Revised: Then, for single-label classification, the softmax function is usually used to convert y to prediction probabilities, i.e ., softmax( y ) .

Original: The class with the highest probability will be considered as the final prediction.
Revised: The class with the highest probability will be considered as the final prediction.

Original: For multi-label classification, the sigmoid function is usually used to convert y to binary prediction probabilities, i.e ., σ ( y ) .
Revised: For multi-label classification, the sigmoid function is usually used to convert y to binary prediction probabilities, i.e ., σ ( y ) .

Original: Those classes whose probabilities are higher than a predefined threshold, e.g ., 0.5, will be considered as the final predictions.
Revised: Those classes whose probabilities are higher than a predefined threshold, e.g ., 0.5, will be considered as the final predictions.

Original: B D ETAILS OF E VALUATION T ASKS AND D ATASETS
Revised: B D ETAILS OF E VALUATION T ASKS AND D ATASETS

Original: Protein Fold Classification.
Revised: Protein Fold Classification.

Original: Protein fold classification is important in the study of the relationship between protein structure and protein evolution.
Revised: Protein fold classification is important in the study of the relationship between protein structure and protein evolution.

Original: The fold classes indicate protein secondary structure compositions, orientations and connection orders.
Revised: The fold classes indicate protein secondary structure compositions, orientations and connection orders.

Original: We follow Hermosilla et al.
Revised: We follow Hermosilla et al.

Original: (2021) to conduct protein fold classification on the training/validation/test splits of the SCOPe 1.75 data set of Hou et al.
Revised: (2021) to conduct protein fold classification on the training/validation/test splits of the SCOPe 1.75 data set of Hou et al.

Original: (2018), which in total contains 16,712 proteins with 1,195 fold classes.
Revised: (2018), which in total contains 16,712 proteins with 1,195 fold classes.

Original: The 3D coordinates of the proteins were collected from the SCOPe 1.75 database (Murzin et al., 1995).
Revised: The 3D coordinates of the proteins were collected from the SCOPe 1.75 database (Murzin et al., 1995).

Original: The data set provides three different evaluation scenarios.
Revised: The data set provides three different evaluation scenarios.

Original: 1) Fold, in which proteins from the same superfamily are not used during training.
Revised: 1) Fold, in which proteins from the same superfamily are not used during training.

Original: Superfamily, in which proteins from the same family are not provided during training.
Revised: Superfamily, in which proteins from the same family are not provided during training.

Original: 3) Family, in which proteins of the same family are available during training.
Revised: 3) Family, in which proteins of the same family are available during training.

Original: Mean accuracy is used as the evaluation metric.
Revised: Mean accuracy is used as the evaluation metric.

Original: Enzyme Reaction Classification.
Revised: Enzyme Reaction Classification.

Original: Enzyme reaction classification can be seen as a protein function classification task, which is based on the enzyme-catalyzed reaction according to all four levels of the Enzyme Commission (EC) number (Webb, 1992).
Revised: Enzyme reaction classification can be seen as a protein function classification task, which is based on the enzyme-catalyzed reaction according to all four levels of the Enzyme Commission (EC) number (Webb, 1992).

Original: We use the dataset collected by Hermosilla et al.
Revised: We use the dataset collected by Hermosilla et al.

Original: (2021), which includes 384 four-level EC classes and 29,215/2,562/5,651 proteins for training/validation/test, respectively.
Revised: (2021), which includes 384 four-level EC classes and 29,215/2,562/5,651 proteins for training/validation/test, respectively.

Original: Mean accuracy is used as the evaluation metric.
Revised: Mean accuracy is used as the evaluation metric.

Original: Gene Ontology Term Prediction.
Revised: Gene Ontology Term Prediction.

Original: This task aims to predict the functions of a protein via multiple
Revised: This task aims to predict the functions of a protein via multiple

Original: Gene Ontology (GO) terms, which can be seen as a multi-label classification task.
Revised: Gene Ontology (GO) terms, which can be seen as a multi-label classification task.

Original: Following Gligorijevi ´ c et al.
Revised: Following Gligorijevi ´ c et al.

Original: (2021), we classify proteins into hierarchically related functional classes organized into three ontologies: biological process (BP) with 1,943 classes, molecular function (MF) withclasses and cellular component (CC) with 320 classes.
Revised: (2021), we classify proteins into hierarchically related functional classes organized into three ontologies: biological process (BP) with 1,943 classes, molecular function (MF) withclasses and cellular component (CC) with 320 classes.

Original: The dataset contains 29,898/3,322/3, proteins for training/validation/test, respectively.
Revised: The dataset contains 29,898/3,322/3, proteins for training/validation/test, respectively.

Original: The F max accuracy is used as the evaluation metric.
Revised: The F max accuracy is used as the evaluation metric.

Original: Enzyme Commission Number Prediction.
Revised: Enzyme Commission Number Prediction.

Original: Different from enzyme reaction classification, this task aims to predict the three-level and four-level 538 EC numbers.
Revised: Different from enzyme reaction classification, this task aims to predict the three-level and four-level 538 EC numbers.

Original: We use the training/validation/test splits of Gligorijevi´c et al.
Revised: We use the training/validation/test splits of Gligorijevi´c et al.

Original: (2021), which contains 15,550/1,729/1,919 proteins in total.
Revised: (2021), which contains 15,550/1,729/1,919 proteins in total.

Original: EC number prediction is also a multi-label classification task.
Revised: EC number prediction is also a multi-label classification task.

Original: We use F max as the evaluation metric.
Revised: We use F max as the evaluation metric.

Original: For GO term and EC number prediction, we follow the multi-cutoff splits in (Gligorijevi´c et al., 2021) to ensure that the test set only contains PDB chains with sequence identity no more than 95% to the training set, which is also used in (Wang et al., 2021; Zhang et al., 2022).
Revised: For GO term and EC number prediction, we follow the multi-cutoff splits in (Gligorijevi´c et al., 2021) to ensure that the test set only contains PDB chains with sequence identity no more than 95% to the training set, which is also used in (Wang et al., 2021; Zhang et al., 2022).

Original: Accuracy with all cutoffs is shown in Appendix K.
Revised: Accuracy with all cutoffs is shown in Appendix K.

Original: C E VALUATION M ETRIC F max
Revised: C E VALUATION M ETRIC F max

Original: Multi-label classification can be seen as consisting of a series of binary classification tasks.
Revised: Multi-label classification can be seen as consisting of a series of binary classification tasks.

Original: The protein-centric maximum F-Score, i.e ., F max , as defined in Gligorijevi´c et al. (2021), can be used to measure the accuracy of multi-label classification.
Revised: The protein-centric maximum F-Score, i.e ., F max , as defined in Gligorijevi´c et al. (2021), can be used to measure the accuracy of multi-label classification.

Original: Suppose λ ∈ [0 , 1] is a decision threshold,
Revised: Suppose λ ∈ [0 , 1] is a decision threshold,

Original: p j
Revised: p j

Original: i is the prediction probability for the j -th class of the i -th protein, b ji ∈ { 0 , 1 } is the corresponding binary class label and J is the number of classes.
Revised: i is the prediction probability for the j -th class of the i -th protein, b ji ∈ { 0 , 1 } is the corresponding binary class label and J is the number of classes.

Original: F-Score is based on the precision and recall of the predictions for each protein,
Revised: F-Score is based on the precision and recall of the predictions for each protein,

Original: [Equation]
Revised: [Equation]

Original: and the average precision and recall over all proteins are defined as follows,
Revised: and the average precision and recall over all proteins are defined as follows,

Original: [Equation]
Revised: [Equation]

Original: where N is the number of proteins.
Revised: where N is the number of proteins.

Original: Then, F max is the maximum F-Score among the different thresholds
Revised: Then, F max is the maximum F-Score among the different thresholds

Original: λ tested in the range [0 , 1] ,
Revised: λ tested in the range [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: D F RAMEWORK I LLUSTRATION
Revised: D F RAMEWORK I LLUSTRATION

Original: The power of CNNs comes from local structure modeling by convolutions and global representation learning via hierarchical architectures.
Revised: The power of CNNs comes from local structure modeling by convolutions and global representation learning via hierarchical architectures.

Original: We illustrate our CDConv-based CNN in Fig. 6(a).
Revised: We illustrate our CDConv-based CNN in Fig. 6(a).

Original: Our
Revised: Our

Original: CDConv captures the (3+1)D structure in each local area.
Revised: CDConv captures the (3+1)D structure in each local area.

Original: To build the pyramid framework, amino acids are downsampled along CDConv layers.
Revised: To build the pyramid framework, amino acids are downsampled along CDConv layers.

Original: The final (3+1)D feature is used for classification.
Revised: The final (3+1)D feature is used for classification.

Original: To avoid the impact of the regularity of sequential orders and the irregularity of geometric coordinates on each other, we can employ a two-stream 3D-CNN+1D-CNN framework, as illustrated in Fig. 6(b).
Revised: To avoid the impact of the regularity of sequential orders and the irregularity of geometric coordinates on each other, we can employ a two-stream 3D-CNN+1D-CNN framework, as illustrated in Fig. 6(b).

Original: First, an independent 3D continuous CNN branch and an independent 1D discrete CNN branch are used to extract the global geometric and sequential features, respectively.
Revised: First, an independent 3D continuous CNN branch and an independent 1D discrete CNN branch are used to extract the global geometric and sequential features, respectively.

Original: Then, the two features are fused as the final protein representation.
Revised: Then, the two features are fused as the final protein representation.

Original: However, because the two types of structures in proteins are modeled separately, the local (3+1)D structure is not properly captured.
Revised: However, because the two types of structures in proteins are modeled separately, the local (3+1)D structure is not properly captured.

Original: In contrast, our (3+1)D
Revised: In contrast, our (3+1)D

Original: CDConv-based CNNs can simultaneously model the local geometry-sequence structure and thus extract more discriminative local features than 3D-CNN+1D-CNN architectures.
Revised: CDConv-based CNNs can simultaneously model the local geometry-sequence structure and thus extract more discriminative local features than 3D-CNN+1D-CNN architectures.

Original: E M ORE D ETAILS OF I MPLEMENTATION AND T RAINING S ETUP
Revised: E M ORE D ETAILS OF I MPLEMENTATION AND T RAINING S ETUP

Original: We provide more details of implementation and training setup in Table 4.
Revised: We provide more details of implementation and training setup in Table 4.

Original: Our model is implemented based on PyTorch-Geometric 2.0.4 and PyTorch 1.10.0 with CUDA 11.3.1 and cuDNN 8.2.0.
Revised: Our model is implemented based on PyTorch-Geometric 2.0.4 and PyTorch 1.10.0 with CUDA 11.3.1 and cuDNN 8.2.0.

Original: Experiments are conducted on a single Nvidia Quadro RTX A5000 GPU.
Revised: Experiments are conducted on a single Nvidia Quadro RTX A5000 GPU.

Original: [Table]
Revised: [Table]

Original: In data pre-processing, amino acid coordinates are normalized as follows,
Revised: In data pre-processing, amino acid coordinates are normalized as follows,

Original: [Equation]
Revised: [Equation]

Original: where p i is the coordinate of i -th amino acid and N is the number of amino acids.
Revised: where p i is the coordinate of i -th amino acid and N is the number of amino acids.

Original: During training, we add the simple isotropic additive gaussian noise (Jitter) to amino acid coordinates for data augmentation.
Revised: During training, we add the simple isotropic additive gaussian noise (Jitter) to amino acid coordinates for data augmentation.

Original: In practice, we find that directly neglecting the neighboring amino acids whose sequential distances are greater than ⌊ l/ 2 ⌋ but geometric distances are less than r is not conducive to optimization.
Revised: In practice, we find that directly neglecting the neighboring amino acids whose sequential distances are greater than ⌊ l/ 2 ⌋ but geometric distances are less than r is not conducive to optimization.

Original: In the
Revised: In the

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: feature 1⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)
Revised: feature 1⋮ 𝑁 (18.09, 5.83, 18.78) (16.25, 9.24, 18.87) (−18.37, 8.50, −6.16)

Original: 𝑃 𝐹 𝐸
Revised: 𝑃 𝐹 𝐸

Original: ⋮ ⋮ ⋯ classification
Revised: ⋮ ⋮ ⋯ classification

Original: CDConv
Revised: CDConv

Original: CDConv
Revised: CDConv

Original: CDConv ⋯ (3+1)D feature
Revised: CDConv ⋯ (3+1)D feature

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ) (−18.
Revised: ) (−18.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ⋯ ⋯ classification ⋯ ⋯
Revised: ⋯ ⋯ classification ⋯ ⋯

Original: [Figure]
Revised: [Figure]

Original: implementation, we include those geometric but non-sequential neighbors as follows,
Revised: implementation, we include those geometric but non-sequential neighbors as follows,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where θ − and θ + are two additional groups of learnable weights for those “special” neighbors.
Revised: where θ − and θ + are two additional groups of learnable weights for those “special” neighbors.

Original: In this way, networks are not significantly sensitive to the sequential kernel size hyper-parameter l , as shown in Fig. 5(c).
Revised: In this way, networks are not significantly sensitive to the sequential kernel size hyper-parameter l , as shown in Fig. 5(c).

Original: F I MPACT OF THE N UMBER OF F EATURE C HANNELS
Revised: F I MPACT OF THE N UMBER OF F EATURE C HANNELS

Original: Similar to most CNNs, we use multiple CDConv layers to build hierarchical or pyramid frameworks for protein representation learning.
Revised: Similar to most CNNs, we use multiple CDConv layers to build hierarchical or pyramid frameworks for protein representation learning.

Original: For the i -th CDConv layer, the number of channels is set to ⌈ i 2 ⌉ c o , where c o is the initial number of channels.
Revised: For the i -th CDConv layer, the number of channels is set to ⌈ i 2 ⌉ c o , where c o is the initial number of channels.

Original: In this way, the feature dimension progressively increases when CNNs become deep.
Revised: In this way, the feature dimension progressively increases when CNNs become deep.

Original: Fold
Revised: Fold

Original: Super
Revised: Super

Original: Family
Revised: Family

Original: Reaction
Revised: Reaction

Original: GO-MF
Revised: GO-MF

Original: GO-BP
Revised: GO-BP

Original: GO-CC
Revised: GO-CC

Original: EC (a) number of channels c o     0.
Revised: EC (a) number of channels c o     0.

Original: Fold
Revised: Fold

Original: Super
Revised: Super

Original: Family
Revised: Family

Original: Reaction
Revised: Reaction

Original: GO-MF
Revised: GO-MF

Original: GO-BP
Revised: GO-BP

Original: GO-CC
Revised: GO-CC

Original: EC (
Revised: EC (

Original: [Figure]
Revised: [Figure]

Original: h
Revised: h

Original: Usually
Revised: Usually

Original: As shown in Fig. 7(a), using too few channels cannot carry enough information and therefore lead to inferior accuracy.
Revised: As shown in Fig. 7(a), using too few channels cannot carry enough information and therefore lead to inferior accuracy.

Original: However, when employing too many channels, the accuracy does not significantly increase.
Revised: However, when employing too many channels, the accuracy does not significantly increase.

Original: This means that the number of weights or parameters is enough.
Revised: This means that the number of weights or parameters is enough.

Original: To further improve the accuracy, we may need to develop more effective CDConv implementations or network architectures.
Revised: To further improve the accuracy, we may need to develop more effective CDConv implementations or network architectures.

Original: G I MPACT OF THE N UMBER OF L AYERS (N ETWORK D EPTH )
Revised: G I MPACT OF THE N UMBER OF L AYERS (N ETWORK D EPTH )

Original: By default, we use eight CDConv layers ( h = 8 ) to build our CNN for protein representation learning.
Revised: By default, we use eight CDConv layers ( h = 8 ) to build our CNN for protein representation learning.

Original: In this section, we investigate the influence of the number of layers.
Revised: In this section, we investigate the influence of the number of layers.

Original: As shown in Fig. 7(b), when h is too small, networks fail to effectively capture the protein structure.
Revised: As shown in Fig. 7(b), when h is too small, networks fail to effectively capture the protein structure.

Original: However, when networks are too deep, the accuracy even slightly decreases.
Revised: However, when networks are too deep, the accuracy even slightly decreases.

Original: This is because, with depth increasing, the receptive fields of late layers are too large, which decreases the discriminativeness of local structures.
Revised: This is because, with depth increasing, the receptive fields of late layers are too large, which decreases the discriminativeness of local structures.

Original: The other reason is that too many layers may lead to overfitting.
Revised: The other reason is that too many layers may lead to overfitting.

Original: Note that for protein fold classification and GO-CC term prediction, using six CDConv layers ( h = 6 ) already achieves satisfactory accuracy.
Revised: Note that for protein fold classification and GO-CC term prediction, using six CDConv layers ( h = 6 ) already achieves satisfactory accuracy.

Original: However, for the consistency in visualization, we still use eight CDConv layers.
Revised: However, for the consistency in visualization, we still use eight CDConv layers.

Original: H I MPACT OF G EOMETRY AND S EQUENCE M ODELING ON P ROTEIN U NDERSTANDING
Revised: H I MPACT OF G EOMETRY AND S EQUENCE M ODELING ON P ROTEIN U NDERSTANDING

Original: The proposed CDConv employs the 3D geometry structure and the 1D sequence structure in proteins for recognition.
Revised: The proposed CDConv employs the 3D geometry structure and the 1D sequence structure in proteins for recognition.

Original: In this section, we analyze the influence of each structure.
Revised: In this section, we analyze the influence of each structure.

Original: As shown in Table 3, both the two structures contribute to protein understanding.
Revised: As shown in Table 3, both the two structures contribute to protein understanding.

Original: For example, without the sequence structure modeling, the fold classification accuracy decreases from 56.7% to 41.9%.
Revised: For example, without the sequence structure modeling, the fold classification accuracy decreases from 56.7% to 41.9%.

Original: Without the geometry structure modeling, the GO-BP term prediction F max decreases from 0.453 to 0.371.
Revised: Without the geometry structure modeling, the GO-BP term prediction F max decreases from 0.453 to 0.371.

Original: For the two types of modeling, geometry structure contributes more than sequence structure.
Revised: For the two types of modeling, geometry structure contributes more than sequence structure.

Original: For example, the 3D-only CDConv achieves 65.5% on superfamily fold classification while the 1D-only
Revised: For example, the 3D-only CDConv achieves 65.5% on superfamily fold classification while the 1D-only

Original: CDConv achieves 20.8%.
Revised: CDConv achieves 20.8%.

Original: This indicates that geometry structure is more discriminative than sequence structure for protein recognition.
Revised: This indicates that geometry structure is more discriminative than sequence structure for protein recognition.

Original: I I MPACT OF R ELATIVE S PATIAL E NCODING
Revised: I I MPACT OF R ELATIVE S PATIAL E NCODING

Original: [Table]
Revised: [Table]

Original: To make our method rotationally invariant
Revised: To make our method rotationally invariant

Original: [Table]
Revised: [Table]

Original: we replace the 3D displacement encoding in PointConv (Wu et al., 2019) with relative spatial encoding (Ingraham et al., 2019).
Revised: we replace the 3D displacement encoding in PointConv (Wu et al., 2019) with relative spatial encoding (Ingraham et al., 2019).

Original: The details are as follows,
Revised: The details are as follows,

Original: [Equation]
Revised: [Equation]

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we conduct experiments to evaluate the effectiveness of relative spatial encoding.
Revised: we conduct experiments to evaluate the effectiveness of relative spatial encoding.

Original: As shown in Table
Revised: As shown in Table

Original: [Table]
Revised: [Table]

Original: relative spatial encoding significantly improves the accuracy, demonstrating the effectiveness of rotational invariance for protein modeling
Revised: relative spatial encoding significantly improves the accuracy, demonstrating the effectiveness of rotational invariance for protein modeling

Original: [Table]
Revised: [Table]

Original: J M ORE D ETAILS AND R ESULTS OF THE S TUDY ON THE I MPORTANCE OF S URFACE AND C ENTRAL A MINO A CIDS
Revised: J M ORE D ETAILS AND R ESULTS OF THE S TUDY ON THE I MPORTANCE OF S URFACE AND C ENTRAL A MINO A CIDS

Original: [Table]
Revised: [Table]

Original: To investigate which of the surface and central regions is more important, we mask the same number (20%) of amino acids in the surface and central areas, respectively.
Revised: To investigate which of the surface and central regions is more important, we mask the same number (20%) of amino acids in the surface and central areas, respectively.

Original: Specifically, we first calculate the center of a protein as follows,
Revised: Specifically, we first calculate the center of a protein as follows,

Original: [Equation]
Revised: [Equation]

Original: where N is the number of amino acids and p i denotes the 3D coordinate of the i -th amino acid.
Revised: where N is the number of amino acids and p i denotes the 3D coordinate of the i -th amino acid.

Original: For surface region masking, we remove the 20% farthest amino acids from the center based on the geometry distance.
Revised: For surface region masking, we remove the 20% farthest amino acids from the center based on the geometry distance.

Original: For central region masking, we remove the 20% closest amino acids to the center.
Revised: For central region masking, we remove the 20% closest amino acids to the center.

Original: The results of family fold classification, GO term prediction and EC number prediction are shown in
Revised: The results of family fold classification, GO term prediction and EC number prediction are shown in

Original: Fig. 8.
Revised: Fig. 8.

Original: Note that amino acids in central areas are usually denser than those in surface regions.
Revised: Note that amino acids in central areas are usually denser than those in surface regions.

Original: This means that, when masking the same number of amino acids, we actually mask more surface regions.
Revised: This means that, when masking the same number of amino acids, we actually mask more surface regions.

Original: Even so, masking central area leads to more drops of accuracy.
Revised: Even so, masking central area leads to more drops of accuracy.

Original: This indicates that amino acids in central regions may be more important than those in surface areas.
Revised: This indicates that amino acids in central regions may be more important than those in surface areas.

Original: However, future experimental efforts are required to further verify this phenomenon
Revised: However, future experimental efforts are required to further verify this phenomenon

Original: [Table]
Revised: [Table]

Original: K GO T ERM AND EC N UMBER P REDICTION UNDER D IFFERENT S EQUENCE C UTOFFS
Revised: K GO T ERM AND EC N UMBER P REDICTION UNDER D IFFERENT S EQUENCE C UTOFFS

Original: [Table]
Revised: [Table]

Original: For GO term and EC number prediction, we can employ different cutoff splits (Gligorijevi´c et al., 2021): the proteins in the test set are split to have < 30%, < 40%, < 50%, < 70 % and < 95 % similarity to the training set, respectively.
Revised: For GO term and EC number prediction, we can employ different cutoff splits (Gligorijevi´c et al., 2021): the proteins in the test set are split to have < 30%, < 40%, < 50%, < 70 % and < 95 % similarity to the training set, respectively.

Original: The results are shown in Table 6.
Revised: The results are shown in Table 6.

Original: Our CDConv significantly outperforms the existing methods in most experimental scenarios.
Revised: Our CDConv significantly outperforms the existing methods in most experimental scenarios.

Original: For example, for GO-MF term prediction under cutoff 40%, our CDConv outperforms GearNet-Edge by 9.2%.
Revised: For example, for GO-MF term prediction under cutoff 40%, our CDConv outperforms GearNet-Edge by 9.2%.

Original: For GO-CC term prediction under cutoff 70%, our CDConv outperforms GearNet-Edge by 4.3%.
Revised: For GO-CC term prediction under cutoff 70%, our CDConv outperforms GearNet-Edge by 4.3%.

Original: This demonstrates the effectiveness of the proposed method.
Revised: This demonstrates the effectiveness of the proposed method.

Original: L C OMPARISON WITH P RETRAINING M ETHODS
Revised: L C OMPARISON WITH P RETRAINING M ETHODS

Original: Recently, pretraining or self-supervised learning on 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics.
Revised: Recently, pretraining or self-supervised learning on 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics.

Original: The goal of those methods is to exploit unlabeled protein data to boost accuracy by designing self-supervised or contrastive objective functions.
Revised: The goal of those methods is to exploit unlabeled protein data to boost accuracy by designing self-supervised or contrastive objective functions.

Original: For example, Hermosilla & Ropinski (2022) proposed to train networks by maximizing the similarity between representations from sub-structures of the same protein, and minimizing the similarity between sub-structures from different proteins.
Revised: For example, Hermosilla & Ropinski (2022) proposed to train networks by maximizing the similarity between representations from sub-structures of the same protein, and minimizing the similarity between sub-structures from different proteins.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) proposed to maximize the similarity between representations of different augmented views of the same protein while minimizing the agreement between views of different proteins, and to predict amino acid types, the
Revised: (2022) proposed to maximize the similarity between representations of different augmented views of the same protein while minimizing the agreement between views of different proteins, and to predict amino acid types, the

Original: Euclidean distance between nodes connected in the protein graph, the angle between nodes and the dihedral angles between edges.
Revised: Euclidean distance between nodes connected in the protein graph, the angle between nodes and the dihedral angles between edges.

Original: However, as mentioned by Fan et al.
Revised: However, as mentioned by Fan et al.

Original: (2022), it is unclear whether one or a few self-supervision tasks are sufficient for learning effective representations and which
Revised: (2022), it is unclear whether one or a few self-supervision tasks are sufficient for learning effective representations and which

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: task would be considered more.
Revised: task would be considered more.

Original: Doersch & Zisserman (2017) even attempted to combine several self-supervised tasks to obtain better visual representations
Revised: Doersch & Zisserman (2017) even attempted to combine several self-supervised tasks to obtain better visual representations

Original: [Table]
Revised: [Table]

Original: Different from those pretraining or self-supervised learning works, our method focuses on designing an effective fundamental operation to capture the protein structure, instead of task-related objective or loss functions.
Revised: Different from those pretraining or self-supervised learning works, our method focuses on designing an effective fundamental operation to capture the protein structure, instead of task-related objective or loss functions.

Original: An effective fundamental operation is general and independent of specific training strategies and may improve the baseline for the pretraining or self-supervised learning works.
Revised: An effective fundamental operation is general and independent of specific training strategies and may improve the baseline for the pretraining or self-supervised learning works.

Original: In this section, to show the effectiveness, we compare our CDConv to pretraining or self-supervised learning methods: DeepFRI (Gligorijevi´c et al., 2021), DeepFRI (Gligorijevi´c et al., 2021), ESM1b (Rives et al., 2021), ProtBERT-BFD (Elnaggar et al., 2021), LM-GVP (Wang et al., 2021), amino-acid-level IEConv (Hermosilla & Ropinski, 2022) and GearNet-based methods (Zhang et al., 2022).
Revised: In this section, to show the effectiveness, we compare our CDConv to pretraining or self-supervised learning methods: DeepFRI (Gligorijevi´c et al., 2021), DeepFRI (Gligorijevi´c et al., 2021), ESM1b (Rives et al., 2021), ProtBERT-BFD (Elnaggar et al., 2021), LM-GVP (Wang et al., 2021), amino-acid-level IEConv (Hermosilla & Ropinski, 2022) and GearNet-based methods (Zhang et al., 2022).

Original: As shown in Table 7 and Table 8, without any pretraining or self-supervised learning, our CDConv achieves comparable accuracy with those methods and even outperforms them on fold and enzyme reaction classification.
Revised: As shown in Table 7 and Table 8, without any pretraining or self-supervised learning, our CDConv achieves comparable accuracy with those methods and even outperforms them on fold and enzyme reaction classification.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: M R OTATION E QUIVARIANCE FOR O THER S CENARIOS
Revised: M R OTATION E QUIVARIANCE FOR O THER S CENARIOS

Original: In this paper
Revised: In this paper

Original: [Table]
Revised: [Table]

Original: we propose a new class of convolution, i
Revised: we propose a new class of convolution, i

Original: [Table]
Revised: [Table]

Original: e., CDConv, to make the most of the dual discrete and continuous nature of the data to avoid the impact of regular and irregular modeling on each other.
Revised: e., CDConv, to make the most of the dual discrete and continuous nature of the data to avoid the impact of regular and irregular modeling on each other.

Original: The key design of CDConv is to employ independent learnable weights for different regular displacements but directly encode irregular displacements.
Revised: The key design of CDConv is to employ independent learnable weights for different regular displacements but directly encode irregular displacements.

Original: As a prototype, CDConv can be instantiated to various instances.
Revised: As a prototype, CDConv can be instantiated to various instances.

Original: In this paper, we implement a (3+1)D CDConv for protein representation learning.
Revised: In this paper, we implement a (3+1)D CDConv for protein representation learning.

Original: When implementing the 3D geometry modeling part, we use the relative spatial encoding technique.
Revised: When implementing the 3D geometry modeling part, we use the relative spatial encoding technique.

Original: The relative spatial encoding is independent of our CDConv and can be replaced with other techniques.
Revised: The relative spatial encoding is independent of our CDConv and can be replaced with other techniques.

Original: When for other scenarios, we can integrate the corresponding rotation equivariance technique into
Revised: When for other scenarios, we can integrate the corresponding rotation equivariance technique into

Original: CDConv.
Revised: CDConv.

Original: Moreover, there can be a low-bound solution to completely avoid the rotational variance problem: only use the distance information and do not model direction,
Revised: Moreover, there can be a low-bound solution to completely avoid the rotational variance problem: only use the distance information and do not model direction,

Original: [Equation]
Revised: [Equation]

Original: where the distance is normalized with the neighbor searching radius r .
Revised: where the distance is normalized with the neighbor searching radius r .

Original: Finally, the new (3+1)D
Revised: Finally, the new (3+1)D

Original: CDConv is implemented as follows,
Revised: CDConv is implemented as follows,

Original: [Equation]
Revised: [Equation]

Original: Then, we evaluate this distance-only-based “completely-rotationally-invariant” (3+1)D CDConv on protein fold classification.
Revised: Then, we evaluate this distance-only-based “completely-rotationally-invariant” (3+1)D CDConv on protein fold classification.

Original: As shown in Table 9, the new (3+1)D CDConv still achieves comparable accuracy with the previous state-of-the-art methods.
Revised: As shown in Table 9, the new (3+1)D CDConv still achieves comparable accuracy with the previous state-of-the-art methods.

Original: N C OMPUTATIONAL E FFICIENCY AND S CALABILITY
Revised: N C OMPUTATIONAL E FFICIENCY AND S CALABILITY

Original: To investigate the performance of our method on relatively large proteins, we collect the proteins that contain more than 500 amino acids from the enzyme reaction classification dataset, leading to a subset with large proteins, as shown in Table 10.
Revised: To investigate the performance of our method on relatively large proteins, we collect the proteins that contain more than 500 amino acids from the enzyme reaction classification dataset, leading to a subset with large proteins, as shown in Table 10.

Original: However, because the training data significantly reduces (from 29,215 to 2,498 proteins), the subset of large proteins may lead to inferior accuracy.
Revised: However, because the training data significantly reduces (from 29,215 to 2,498 proteins), the subset of large proteins may lead to inferior accuracy.

Original: Therefore, for a fair comparison, we randomly sample a normal subset that contains the same number of proteins as that of large proteins.
Revised: Therefore, for a fair comparison, we randomly sample a normal subset that contains the same number of proteins as that of large proteins.

Original: Then, we conduct enzyme reaction classification on the two subsets with a single Nvidia Quadro
Revised: Then, we conduct enzyme reaction classification on the two subsets with a single Nvidia Quadro

Original: RTX A5000 GPU and Intel(R)
Revised: RTX A5000 GPU and Intel(R)

Original: Xeon(R)
Revised: Xeon(R)

Original: Silver 4214R CPU @ 2.40GHz.
Revised: Silver 4214R CPU @ 2.40GHz.

Original: We also report the mean running time per prediction.
Revised: We also report the mean running time per prediction.

Original: As shown in Table 11, our CDConv can still effectively and efficiently recognize enzyme reactions.
Revised: As shown in Table 11, our CDConv can still effectively and efficiently recognize enzyme reactions.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: O D ISCUSSION ON A TTENTION - BASED M ETHODS
Revised: O D ISCUSSION ON A TTENTION - BASED M ETHODS

Original: Recently, attention-based methods (Jumper et al., 2021; Baek et al., 2021; Humphreys et al., 2021;
Revised: Recently, attention-based methods (Jumper et al., 2021; Baek et al., 2021; Humphreys et al., 2021;

Original: Fuchs et al., 2020) are also widely used for protein-related problems, such as protein structure prediction or generation.
Revised: Fuchs et al., 2020) are also widely used for protein-related problems, such as protein structure prediction or generation.

Original: As we discussed in Sec. 5, attention-based techniques may potentially improve our method.
Revised: As we discussed in Sec. 5, attention-based techniques may potentially improve our method.

Original: Using deep learning to model a kind of data can be divided into two parts: 1) searching related elements ( e.g ., amino acids) and 2) encoding the local structure of the related elements.
Revised: Using deep learning to model a kind of data can be divided into two parts: 1) searching related elements ( e.g ., amino acids) and 2) encoding the local structure of the related elements.

Original: Attentionbased methods can adaptively reflect the correlation degree of two elements and therefore are good at searching related elements.
Revised: Attentionbased methods can adaptively reflect the correlation degree of two elements and therefore are good at searching related elements.

Original: Our CDConv focuses on encoding the region of the related elements by making the most of the dual discrete and continuous nature of the data.
Revised: Our CDConv focuses on encoding the region of the related elements by making the most of the dual discrete and continuous nature of the data.

Original: The two kinds of methods may facilitate each other, which can be studied in the future.
Revised: The two kinds of methods may facilitate each other, which can be studied in the future.

Original: P I NFLUENCE OF M ISSING C OORDINATES
Revised: P I NFLUENCE OF M ISSING C OORDINATES

Original: When some residues’ coordinates are not available, we can set their coordinates to a pre-defined value, e.g ., (0 , 0 , 0) .
Revised: When some residues’ coordinates are not available, we can set their coordinates to a pre-defined value, e.g ., (0 , 0 , 0) .

Original: In this way, our CDConv can still be used for protein modeling.
Revised: In this way, our CDConv can still be used for protein modeling.

Original: To evaluate the effectiveness of this strategy, we randomly select a certain percentage of amino acids and set their coordinates to (0 , 0 , 0) in training and test and conduct protein fold classification.
Revised: To evaluate the effectiveness of this strategy, we randomly select a certain percentage of amino acids and set their coordinates to (0 , 0 , 0) in training and test and conduct protein fold classification.

Original: As shown in Table 12, even with 20% missing coordinates, our CDConv still achieves promising results.
Revised: As shown in Table 12, even with 20% missing coordinates, our CDConv still achieves promising results.

Original: Q I NFLUENCE OF V OXEL S IZE FOR D ISCRETIZING 3D C OORDINATES
Revised: Q I NFLUENCE OF V OXEL S IZE FOR D ISCRETIZING 3D C OORDINATES

Original: In our experiments, when using discrete convolutions for 3D geometry structure modeling, we follow the practice in most discrete CNNs and employ kernel size 3, which means there are 3 × 3 × 3 =group of learnable weights for each convolution.
Revised: In our experiments, when using discrete convolutions for 3D geometry structure modeling, we follow the practice in most discrete CNNs and employ kernel size 3, which means there are 3 × 3 × 3 =group of learnable weights for each convolution.

Original: Because the computation resource is limited, there is a trade-off, which is also a limitation, to use discrete convolutions for 3D geometry modeling:
Revised: Because the computation resource is limited, there is a trade-off, which is also a limitation, to use discrete convolutions for 3D geometry modeling:

Original: • If based on high-resolution voxelized proteins, small kernel sizes are inevitable for modeling.
Revised: • If based on high-resolution voxelized proteins, small kernel sizes are inevitable for modeling.

Original: • If increasing the receptive field, i.e ., kernel size, networks have to reduce the resolution.
Revised: • If increasing the receptive field, i.e ., kernel size, networks have to reduce the resolution.

Original: In summary, due to the limited computation resource, we have to fix the kernel size and set it to a small range.
Revised: In summary, due to the limited computation resource, we have to fix the kernel size and set it to a small range.

Original: In this way, if we want to increase the receptive field, we can accordingly reduce the resolution.
Revised: In this way, if we want to increase the receptive field, we can accordingly reduce the resolution.

Original: If we want a higher resolution, the receptive field has to be relatively “shrunk”.
Revised: If we want a higher resolution, the receptive field has to be relatively “shrunk”.

Original: In experiments, we found setting the voxel size for discretizing 3D coordinates to 0.5 achieves the best accuracy.
Revised: In experiments, we found setting the voxel size for discretizing 3D coordinates to 0.5 achieves the best accuracy.

Original: The results for other voxel sizes are shown in Table 13.
Revised: The results for other voxel sizes are shown in Table 13.

Original: When increasing the voxel size,
Revised: When increasing the voxel size,

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: although the 3 × 3 × 3 can capture a relatively large receptive field
Revised: although the 3 × 3 × 3 can capture a relatively large receptive field

Original: [Table]
Revised: [Table]

Original: the accuracy does not increase.
Revised: the accuracy does not increase.

Original: This is because increasing voxel size results in low resolution, which cannot properly describe protein structures.
Revised: This is because increasing voxel size results in low resolution, which cannot properly describe protein structures.

Original: R V ISUALIZATION OF CDC ONV O UTPUTS
Revised: R E MPLOYING P ROTEIN S TRUCTURE P REDICTION M ODELS FOR P ROTEIN C LASSIFICATION

Revised: In recent years [Table] deep-learning-based methods have made tremendous achievements in protein structure prediction, such as AlphaFold (Jumper et al., 2021) and RoseTTAFold (Baek et al., 2021).

Revised: In this section

Revised: [Table]

Revised: we investigate using protein structure prediction methods for protein classification.

Revised: There is a difference in output between protein structure prediction and classification.

Revised: • Protein structure prediction .

Revised: The goal of structure prediction is to make residue-wise predictions.

Revised: The output is R N × C or R N × C × C , where N is the number of amino acids and C is the number of prediction channels.

Revised: For structure prediction, C = 3 .

Revised: • Protein classification .

Revised: Classification aims to make global predictions, in which a global protein representation vector is learned, i.e ., R C .

Revised: In our method, C = 1024 or C = 2048 .

Revised: Because CNNs are usually hierarchical and pyramid, which captures structures from local to global,

Revised: it is natural for CNNs to learn global representations.

Revised: Another difference is that protein structure prediction usually exploits external knowledge, e.g .,

Revised: Multiple Sequence Alignments (MSAs) and protein templates.

Revised: For classification, existing methods are usually based on the given limited training data.

Revised: In this section, we convert RoseTTAFold (Baek et al., 2021) for protein classification.

Revised: Specifically, we use the outputs of the feature extractor in RoseTTAFold: MSA feature ∈ R N × 384 , pair representation ∈ R N × N × 288 and xyz ∈ R N × 3 × 3 .

Revised: To generate a global representation vector based on these outputs, we first perform sum on the second dimension of the pair representation and get R N ×and flatten xyz as R N × 9 .

Revised: Then, we concatenate MSA feature, processed pair representation and xyz:

Revised: R N × (384+288+9)

Revised: =

Revised: R N × 681 .

Revised: Third, we append a fully-connected layer with ReLU to increase the dimension of the concatenation from 681 to 1024, i.e ., R N × 1024 .

Revised: Last, we sum those features and generate a global representation R 1024 .

Revised: In this way, classification can be performed.

Revised: We use the modified RoseTTAFold for protein fold classification.

Revised: The results are shown in Table 14.

Revised: When MSAs are not exploited, we follow Baek & Baker (2022) to only use the peptide chain itself.

Revised: When protein templates are not exploited, we use the input protein geometry structure as the only template.

Revised: Probably because the architecture of RoseTTAFold is not suitable for global representation

Revised: [Table]

Revised: learning, it achieves limited accuracy.

Revised: Converting RoseTTAFold to a pyramid architecture may boost the accuracy for protein classification, which can be studied in the future

Revised: [Table]

Revised: We notice that MSAs and protein templates help classification.

Revised: They may play a data augmentation role in protein classification.

Revised: Moreover, protein templates help more than MSAs.

Revised: This may indicate that the 3D geometry structure is more important than the 1D sequence structure for protein recognition.

Revised: S V ISUALIZATION OF CDC ONV O UTPUTS

Original: We visualize the outputs of all the CDConv layers in our CNNs in Fig.
Revised: We visualize the outputs of all the CDConv layers in our CNNs in Fig.

Original: Recall that the CNN downsamples amino acids after every two layers.
Revised: Recall that the CNN downsamples amino acids after every two layers.

Original: No matte for large or small proteins, the early layers mainly focus on the central regions of the proteins.
Revised: No matte for large or small proteins, the early layers mainly focus on the central regions of the proteins.

Original: Then, the activation spreads to the entire proteins to capture the whole structure.
Revised: Then, the activation spreads to the entire proteins to capture the whole structure.

Original: This indicates that the central region of proteins may be more important than other areas.
Revised: This indicates that the central region of proteins may be more important than other areas.

Original: However, experimental efforts are required to further verify this phenomenon.
Revised: However, experimental efforts are required to further verify this phenomenon.

Original: i npu
Revised: i npu

Original: t
Revised: t

Original: C D C o n v C D C o n v
Revised: C D C o n v C D C o n v

Original: C D C o n v C D C o n v
Revised: C D C o n v C D C o n v

Original: C D C o n v C D C o n v
Revised: C D C o n v C D C o n v

Original: C D C o n v C D C o n v
Revised: C D C o n v C D C o n v

Original: F i gu r e 9 :
Revised: F i gu r e 9 :

Original: V i s u a l i za ti on o f ho w t
Revised: V i s u a l i za ti on o f ho w t

Original: h e p r opo s e d n e t w o r k ca p t u r e s p r o t
Revised: h e p r opo s e d n e t w o r k ca p t u r e s p r o t

Original: e
Revised: e

Original: i n s t r u c t u r e .
Revised: i n s t r u c t u r e .

Original: T h
Revised: T h

Original: e c
Revised: e c

Original: o l o r
Revised: o l o r

Original: o
Revised: o

Original: f
Revised: f

Original: i npu t do t s i nd
Revised: i npu t do t s i nd

Original: i ca t e s a m
Revised: i ca t e s a m

Original: i
Revised: i

Original: no ac i d t yp e a nd t h e c
Revised: no ac i d t yp e a nd t h e c

Original: o l o r o f
Revised: o l o r o f

Original: C D
Revised: C D

Original: C onv ou t pu t s d e no t e s t h e s t
Revised: C onv ou t pu t s d e no t e s t h e s t

Original: r e ng t h
Revised: r e ng t h

Original: o f t h e ac ti v a ti on .
Revised: o f t h e ac ti v a ti on .

Original: T h
Revised: T h

Original: e ea r l y l a y e r s t e nd t o f o c u
Revised: e ea r l y l a y e r s t e nd t o f o c u

Original: s on t
Revised: s on t

Original: h e ce n t r a l r e g
Revised: h e ce n t r a l r e g

Original: i on a nd t h e l a t e l a y e r s t e nd t
Revised: i on a nd t h e l a t e l a y e r s t e nd t

Original: o ca p t u r e t
Revised: o ca p t u r e t

Original: h e w ho l e s t r u c t u r e .
Revised: h e w ho l e s t r u c t u r e .

Original: T h
Revised: T h

Original: i s m a y i nd
Revised: i s m a y i nd

Original: i ca t e t h a t t
Revised: i ca t e t h a t t

Original: h e ce n t r a l r e g
Revised: h e ce n t r a l r e g

Original: i on i s m o r e i m po r t a n t t h a n o t
Revised: i on i s m o r e i m po r t a n t t h a n o t

Original: h e r a r ea s f o r p r o t
Revised: h e r a r ea s f o r p r o t

Original: e
Revised: e

Original: i n r ec ogn iti on .
Revised: i n r ec ogn iti on .