Original: Graph Neural Networks for Selection of Preconditioners and Krylov Solvers Anonymous Author(s)
Revised: Graph Neural Networks for Selection of Preconditioners and Krylov Solvers ∗

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Solving large sparse linear systems is ubiquitous in science and engineering, generally requiring iterative solvers and preconditioners, as many problems cannotbe solved efficiently by using direct solvers.
Revised: Solving large sparse linear systems is ubiquitous in science and engineering, generally requiring iterative solvers and preconditioners since many problems cannot be solved efficiently by using direct solvers.

Original: However, the practical performanceof solvers and preconditioners is sometimes beyond theoretical analysis and anoptimal choice calls for intuitions from domain experts, knowledge of the hardware, as well as trial and error.
Revised: The practical performance of solvers and preconditioners is sometimes beyond theoretical analysis, however, and an optimal choice calls for intuition from domain experts and knowledge of the hardware, as well as trial and error.

Original: In this work, we propose a new method for optimalsolver-preconditioner selection using Graph Neural Networks (GNNs), as a complementary solution to laborious expert efforts.
Revised: In this work we propose a new method for optimal solverpreconditioner selection using graph neural networks as a complementary solution to laborious expert efforts.

Original: The method is based upon the graphrepresentation of the problem and the idea of integrating node features with graphfeatures via graph convolutions.
Revised: The method is based on the graph representation of the problem and the idea of integrating node features with graph features via graph convolutions.

Original: We show that our models outperform traditionalmachine learning models investigated bythe prior literature by a margin of 25%under the NDCG evaluation metrics.
Revised: We show that our models outperform traditional machine learning models investigated by a margin of 25% under the NDCG evaluation metric.

Original:  Introduction
Revised:  Introduction

Original: Solving linear systems in the form:
Revised: Solving linear systems of the form

Original: [Equation]
Revised: [Equation]

Original: where A is a large and sparse matrix, is a core computing task in numerous scientific and engineeringproblems, such as piezoelectric crystal vibration simulation [1], superconductor modeling [2], andairborne electromagnetic problems [3].
Revised: where A is a large and sparse matrix, is a core computing task in numerous scientific and engineering problems, such as piezoelectric crystal vibration simulation [1], superconductor modeling [2] and airborne electromagnetic problems [3].

Original: These problems typically scale poorly for direct solversdue to the constraints of memory and computational resources.
Revised: These problems typically scale poorly for direct solvers because of constraints of memory and computational resources.

Original: Therefore, iterative methods,aswell as preconditioners, are continuously being developed to improve effectiveness and acceleratethe computations [4].
Revised: Therefore, iterative methods, as well as preconditioners are continuously being developed to improve effectiveness and accelerate the computations [4].

Original: However, because of the wide variety of algorithms, data structures, andhardware architectures,  selecting the optimal combination of a solver and a preconditioner to a givensystem is difficult for application developers and researchers and may require extensive backgroundand expertise in numerical analysis and high-performance computing.
Revised: Because of the wide variety of algorithms, data structures and hardware architectures, however, selecting the optimal combination of a solver and a preconditioner for a given linear system is difficult for application developers and researchers and may require extensive background and expertise in numerical analysis and high-performance computing.

Original: For example, the scientificcomputing library PETSc [5, 6, 7] already includes 17 iterative methods, 17 preconditioners, and 6matrix formats (not counting support for third-party packages).
Revised: For example, the scientific computing library PETSc [5, 6, 7] already includes 17 iterative methods, 17 preconditioners, and 6 matrix formats (not counting support for third-party packages).

Original: The growing pool of new techniquesbecomes a barrier to productivity.
Revised: The growing pool of new techniques becomes a barrier to productivity.

Original: Therefore,it has been a long-standing aim to automate the processand ease the reliance on numerical analysis and hardware knowledge.
Revised: Therefore, a long-standing goal has been to automate the process and ease the reliance on numerical analysis and hardware knowledge.

Original: It becomes more crucial whenthe problem size is enormous and its solution requires parallel computation resources.
Revised: It becomes more crucial when the problem size is enormous and its solution requires parallel computation resources.

Original: However, this  is a challenging task because the decision cannot be made based on the algorithmsanalysis alone and the problem characteristics  must be taken into account.
Revised: Achieving this goal is a challenging task, however, because the decision cannot be made based on the algorithm analysis alone: the problem characteristics also must be taken into account.

Original: While most existingmachine learning methods rely on finding features that are relevant for the performance predictionand easy to compute, we propose a machine learning method that uses graph neural networks (GNNs)for fast selection of preconditioners and sparse iterative solvers adaptive to matrix properties.
Revised: While most existing machine learning methods rely on finding features that are relevant for the performance prediction but often expensive to compute, we propose a machine learning method that uses graph neural networks (GNNs) for fast selection of preconditioners and sparse iterative solvers based upon features that are easy to compute on graphs.

Original: Matricesof sparse linear systems entail a graph interpretation. Hence, GNNs are a promising and feasible framework for learning and encapsulating the domain knowledge required for efficient solution ofsparse linear systems.
Revised: Our work is motivated by the observations that matrices of sparse linear systems entail a graph interpretation and GNNs has shown promises for learning and encapsulating the domain knowledge required for efficient solution of sparse linear systems.

Original: For a proof of concept, only selected built-in Krylov solvers and preconditioners in PETSc areconsidered here for the performance benchmark.
Revised: For a proof of concept, only selected built-in Krylov solvers and preconditioners in PETSc are considered here for the performance benchmark.

Original: Users accessible to other software libraries canfollow a similar procedure and create a suitable training dataset to obtain specific predictions forsolver and preconditioner selection in their computing environment.
Revised: Users with access to other software libraries can follow a similar procedure and create a suitable training dataset to obtain specific predictions for solver and preconditioner selection in their computing environment.

Original: Our main contributions are listed below.
Revised: Our main contributions are listed below.

Original: • We introduce a general graph representation of large sparse matrices so that the local structure of the matrix is exploited by the graph convolution process.
Revised: • We introduce a general graph representation of large sparse matrices so that the local structure of the matrix is exploited by the graph convolution process.

Original: • We formulate a scored-based metric that account for the budget of running time and accuracyrequirement of the target problem, thus allowing for a trade-off between these two factors.
Revised: • We formulate a score-based metric that accounts for the budget of running time and accuracy requirement of the target problem, thus allowing for a trade-off between these two factors.

Original: • We analyze four known graph convolution models through the proposed GNN architecture that combines node features with graph 
Revised: • We analyze four known graph convolution models through the proposed GNN architecture that combines node features with graph features.

Original: features.• We introduce a two-level pooling strategy to simulate the mechanism by which human experts select iterative solvers and preconditioners.
Revised: • We introduce a two-level pooling strategy to simulate the mechanism by which human experts select iterative solvers and preconditioners.

Original: The remaining contents of this paper are organized as follows.

Original: Section 2 reviews the literature.

Original: Section 3 introduces the main procedure of the proposed model, from building dataset to generatingclassification.

Original: Section 4 shows the numerical results and comparisons.

Original: Section 5 describes conclusionsand future work.

Original:  Related work
Revised:  Related work

Original: Over the years, the literature has studied automatic methodologies for selecting preconditioned linearsolvers for sparse linear systems.
Revised: Over the years, the literature has reported automatic methodologies for selecting preconditioned linear solvers for sparse linear systems.

Original: In 1996, Barrett et al.
Revised: In 1996, Barrett et al.

Original: [8] proposed a poly-iterative approach thatexecuted several Krylov methods simultaneously to the same problem to predict the best solver.
Revised: [8] proposed a poly-iterative approach that executed several Krylov methods simultaneously for the same problem to predict the best solver.

Original: Bhowmick et al.
Revised: Bhowmick et al.

Original: [9, 10] present ideas of applying alternating decision tree to choose linear solversadaptively.
Revised: [9, 10] presented ideas for applying alternating decision trees to choose linear solvers adaptively.

Revised: Holloway and Chen [11] extended the methodology to neural networks, while Kuefler and

Revised: Chen [12] evaluated the effectiveness of reinforcement learning.

Original: In the follow-up work of Bhowmick et al.
Revised: Bhowmick et al.

Original: [11], other machine learning (ML) methods,including k -nearest neighbor, naive Bayes, and support vector machines, are investigated.
Revised: [13] subsequently investigated other machine learning (ML) methods, including k -nearest neighbor, naive Bayes, and support vector machines.

Original: Hollowayand Chen [12] extend the methodology to neural networks, while Kuefler and Chen [13] evaluatethe effectiveness of reinforcement learning.

Original: Other similar research includes [14], in which Eller etal.
Revised: Similar research includes that of Eller et al.

Original: introduce a dynamic model, especially for adaptive hydraulics(ADH) problems during transientsimulations.
Revised: [14], who introduced a dynamic model especially for adaptive hydraulics problems during transient simulations.

Original: Kotthoff et al.
Revised: Kotthoff et al.

Original: [15] compare ML techniques to address the more general problem ofalgorithm selection.
Revised: [15] compared ML techniques to address the more general problem of algorithm selection.

Original: Software and system for tuning and recommending numerical solvers have also been developed.
Revised: Software and systems for tuning and recommending numerical solvers have also been developed.

Original: Inparticular, PYTHIA [16] is a knowledge-based recommender system for elliptic partial differentialequations (PDEs).
Revised: In particular, PYTHIA [16] is a knowledge-based recommender system for elliptic partial differential equations.

Original: In 2010, Eijkhout and Fuentes [17] develop a self-adapting large-scale solverarchitecture (SALSA) for multi-stage solver selection, with the help of their matrix feature extractionsoftware named AnaMod [18].
Revised: In 2010, Eijkhout and Fuentes [17] developed a self-adapting large-scale solver architecture (SALSA) for multistage solver selection, with the help of their matrix feature extraction software named AnaMod [18].

Original: Later, a comprehensive taxonomy and framework are given in  project
Revised: Later, a comprehensive taxonomy and framework were given in the project

Original: Lighthouse [19, 20, 21], which targetson preconditioned solvers in PETSc and Trilinos libraries [22].
Revised: Lighthouse [19, 20, 21], which targets preconditioned solvers in PETSc and Trilinos libraries [22].

Original: Method
Revised: These prior works rely heavily on a number of carefully chosen features to predict the performance of preconditioners and solvers. However, useful features such as condition numbers are expensive to compute, and inexpensive features such as average node degree are not very effective in distinguishing matrices with different sparse structures. It is difficult to balance the effectiveness and cost of feature extraction with traditional methods. In this work, we consider GNN mainly for two reasons.

Revised: First, it can efficiently utilize features related to spatial structure, which are usually easy to compute.

Revised: Second,

Revised: GNNs allow us to explore popular techniques in graph learning, such as virtual nodes [23].

Revised: These techniques offer opportunities to improve graph classification performance, thus potentially beneficial to the preconditioners and solvers selection problem.

Revised:  Method

Original: Figure 1 shows an overview of the proposed GNN architecture.
Revised: Figure 1 shows an overview of the proposed GNN architecture.

Original: An input matrix of the COO formatis processed through two paths.
Revised: An input matrix of the coordinate (COO) format is processed through two paths.

Original: In the first path, a graph with well-defined node features is firstgenerated.
Revised: In the first path, a graph with well-defined node features is generated.

Original: Then, by applying graph convolutional layers and pooling layers, we obtain a graphembedding.
Revised: Then, by applying graph convolutional layers and pooling layers, we obtain a graph embedding.

Original: In the second path, matrix features are extracted and treated as graph features.
Revised: In the second path, matrix features are extracted and treated as graph features.

Original: A featurevector is processed by a neural network, i.e., a multi-layer perceptron (MLP).
Revised: A feature vector is processed by a neural network such as a multilayer perceptron (MLP).

Original: Thus, the second pathalso generates a graph embedding.
Revised: Thus, the second path also generates a graph embedding.

Original: Two separate embeddings are concatenated and integrated into anew graph embedding through another MLP.
Revised: The two separate embeddings are concatenated and integrated into a new graph embedding through another MLP.

Original: We may regard this output embedding as an input ofaclassifier or a prediction, after applying a scaling function, e.g. Sigmoid.
Revised: One may regard this output embedding as an input of a classifier, after applying a scaling function such as a sigmoid function.

Original: Note that, these two pathsare independent until the mixing step, and we can remove either path or add new paths of similarstructure. The remaining content of this section describes the implementation details of this work.
Revised: Note that these two paths are independent until the mixing step is taken, and in this framework one can remove either path or add new paths of a similar structure.

Original: 3.1 Problem formulation
Revised: 3.1 Problem formulation

Original: The objective of the selection problem is defined asa multi-label classification , where each classrepresents a combination of a solver and a preconditioner.
Revised: The objective of the selection problem is defined as multi-label classification , where each class represents a combination of a solver and a preconditioner.

Original: Table 1 shows the Krylov iterative solversand preconditioners used in our settings.
Revised: Table 1 shows the Krylov iterative solvers and preconditioners used in our settings.

Original: In total, 33 classes are considered, excluding combinationswhere solvers and preconditioners are incompatible.
Revised: In total, 33 classes are considered, excluding combinations where solvers and preconditioners are incompatible.

Original: Therefore, for any input matrix, the output ofour GNN model is a binary vectorˆ y ∈ { 0 , 1 } 33 , with a value of 1 representing recommendation, and0 otherwise.
Revised: Therefore, for any input matrix, the target label of our GNN model is a binary vector y ∈ { 0 , 1 } 33 , with a value of 1 representing recommendation, and 0 otherwise.

Original: [Table]
Revised: [Table]

Revised: CGGMRES(30)

Revised: BiCGStabLSQRFlexibleGMRES (inner GMRES)

Original: The performance of solvers and preconditioners on a given matrix A arises from solving a linearsystem A x = b  where all elements in b are 1’s.
Revised: The performance of solvers and preconditioners on a given matrix A arises from solving a linear system A x = b , where all elements in b are 1’s.

Original: By applying various combinations, we record therunning time t and the residual r = ∥
Revised: By applying various combinations, we record the running time t and the residual r = ∥

Original: b − A ˆ x ∥ 2 for each choice, where ˆ x is the computed solution.
Revised: b − A ˆ x ∥ 2 for each choice, where ˆ x is the computed solution.

Original: Then [Table] we use the following metrics to compute a score:[Equation]
Revised: Then, we use the following metrics to compute a score:

Original: [Equation]
Revised: [Equation]

Original: where w 1 , w 2 are user-defined weights (default 1) based on the budgets of running time and toleranceof residual error.
Revised: where w 1 , w 2 are user-defined weights (default 1) based on the budgets of running time and tolerance of residual error.

Original: For the cases that run out of time or fail to solve the system, the score will be0.
Revised: For the cases that run out of time or fail to solve the system, the score will be 0.

Original: Eq. 2 is designed to distinguish between the classes with exponential differences in residual orruntime cost.
Revised: Equation 2 is designed to distinguish between the classes with exponential differences in residual or runtime cost.

Original: We define an empirical threshold equal to 90% of the highest score.
Revised: We define an empirical threshold equal to 90% of the highest score.

Original: Any class that has a score higherthan the threshold will be labeled as 1, and others will be labeled as 0.
Revised: Any class that has a score higher than the threshold will be labeled as 1, and others will be labeled as 0.

Original: 3.2 Graph representation and preprocessing
Revised: 3.2 Graph representation and preprocessing

Original: For any input matrix A , a graph is first generated based on the adjacency matrix of A .
Revised: For any input matrix A , a graph is first generated based on the adjacency matrix of A .

Original: Then a self-loopis added to each node.
Revised: Then a selfloop is added to each node.

Original: The off-diagonal nonzero elements of A are attached to corresponding edgesas edge features and diagonal elements are attached to self-loops respectively.
Revised: The off-diagonal nonzero elements of A are attached to corresponding edges as edge features, and diagonal elements are attached to self-loops respectively.

Original: As a rule of thumb,diagonal elements are deemed more important compared with off-diagonal elements.
Revised: As a rule of thumb, diagonal elements are deemed more important than off-diagonal elements.

Original: Our graph representation emphasizes diagonal elements, as self-loops are visited more frequently throughoutthe message passing process.
Revised: Our graph representation emphasizes diagonal elements, since self-loops are visited more frequently throughout the message-passing process.

Original: Naturally, diagonal elements with larger magnitudes corresponds tolarger edge values on self-loops and thus have a larger impact on the computation graph.
Revised: Diagonal elements with larger magnitudes correspond to larger edge values on self-loops and thus have a larger impact on the computation graph.

Original: Here we demonstrate this idea with an example of a 3 × 3 matrix.
Revised: Here we demonstrate this idea with an example of a 3 × 3 matrix.

Original: Recalling that the essence of our GNN model is to predict the performance of solvers and preconditioners in solving linear systems, any modification  is reasonable as long as it does not affect theselection.
Revised: Recalling that the essence of our GNN model is to predict the performance of solvers and preconditioners in solving linear systems, any modification on a linear system is reasonable as long as it does not affect the selection.

Original: Therefore, all columns and rows that contain only diagonal nonzero elements can beremoved.
Revised: Therefore, all columns and rows that contain only diagonal nonzero elements can be removed.

Original: In the context of graph interpretation, this refers to the operation of deleting isolated nodes.
Revised: In the context of graph interpretation, this refers to the operation of deleting isolated nodes.

Original: Diagonal matrices are ignored from the dataset for the same reason.
Revised: Diagonal matrices are removed from the dataset for the same reason.

Original: Furthermore, we add a smallconstant value to every self-loop in the graph because diagonal shifting is commonly applied tosingular matrices in iterative linear solvers.
Revised: Furthermore, we add a small constant value to every self-loop in the graph because diagonal shifting is commonly applied to singular matrices in iterative linear solvers.

Original: 3.3 Node and graph features
Revised: 3.3 Node and graph features

Original: As mentioned in the beginning of Section 3, the graph embedding is generated by processing twotypes of features.
Revised: As mentioned at the beginning of Section 3, the graph embedding is generated by processing two types of features.

Original: Node features attached to graph nodes are processed through graph convolutionallayers while graph features are processed via other neural network layers, e.g., MLP.
Revised: Node features attached to graph nodes are processed through graph convolutional layers, while graph features are processed via other neural network layers, for example, MLP.

Original: Then a mixinglayer, e.g. another MLP, is performed to render a final graph embedding.
Revised: Then a mixing layer, for example another MLP, is performed to render a final graph embedding.

Original: We introduce a node feature to describe diagonal dominance of row i .
Revised: We introduce a node feature to describe diagonal dominance of row i .

Original: Assume that α i denotes theratio between magnitudes of diagonal and off-diagonal elements for each row,
Revised: Assume that α i denotes the ratio between magnitudes of diagonal and off-diagonal elements for each row,

Original: [Equation]
Revised: [Equation]

Original: Then, the node feature x i for node i is defined as
Revised: Then, the node feature x i for node i is defined as

Original: [Equation]
Revised: [Equation]

Original: such that x i ∈ [0 , 1] and x i = 0 .
Revised: such that x i ∈ [0 , 1] and x i = 0 .

Original: 5 is the critical point of whether row i is diagonally dominant.
Revised: 5 is the critical point of whether row i is diagonally dominant.

Original: Similarly, the diagonal decay of row i can be defined by replacing (cid:80)
Revised: Similarly, the diagonal decay of row i can be defined by replacing (cid:80)

Original: j ̸
Revised: j ̸

Original: = i |
Revised: = i |

Original: [Equation]

Original:  ij | in Eq. 3 with max j ̸ = i | A ij | .
Revised: A ij | in Eq. 3 with max j ̸ = i | A ij | .

Original: Both diagonal dominance and decay can also be computed for each column.
Revised: Both diagonal dominance and decay can also be computed for each column.

Original: Moreover, the localdegree profile [23] is also considered.
Revised: Moreover, the local degree profile [24] is also considered.

Original: For graph features, readers can refer to Table 2 for a complete list of  features used in this paper.
Revised: Table 2 gives a complete list of graph features used in this paper.

Original: Toreduce the cost of extracting features that have no significant impact on the classification accuracy, weuse the reduced feature set provided by [20, 21].
Revised: To reduce the cost of extracting features that have no significant impact on the classification accuracy, we use the reduced feature set provided by [20, 21].

Original: Additionally, it is essential to perform normalizationor standardization  when concatenating various features.
Revised: Additionally, normalization or standardization must be performed when concatenating various features.

Revised: [Table]

Original: 3.4 Graph convolutional layers
Revised: 3.4 Graph convolutional layers

Original: We analyze four conventional graph convolution models:
Revised: We analyze four conventional graph convolution models:

Original: • Graph Attention Networks (GAT).
Revised: • Graph attention networks (GATs).

Original: GAT and its successor [24, 25] support taking edgefeatures as input, where edge features are multiplied with transformation matrices and then concatenated with node features.
Revised: GATs [25, 26] support taking edge features as input, where edge features are multiplied with transformation matrices and then concatenated with node features.

Original: • The edge version of Graph Isomorphism Networks (GINE) [26, 27].
Revised: • The edge version of graph isomorphism networks (GINEs) [27, 28].

Original: GINE aggregates edge features along with node features.
Revised: GINEs aggregate edge features along with node features.

Original: • Graph Convolutional Networks (GCN) [28].
Revised: • Graph convolutional networks (GCNs) [29].

Original: GCN is rather different from  above models.
Revised: GCNs are different from the above models.

Original: GCN canonly take  non-negative edge weights, so a Sigmoid or softmax functionon edgefeatures is required.
Revised: GCNs can take only non-negative edge weights, so a sigmoid or softmax function is required for edge features.

Original: [Table] • A modified version of GraphSAGE [29].
Revised: • A modified version of GraphSAGE [30].

Original: The aggregator is represented by
Revised: The aggregator is represented by

Original: [Equation]
Revised: [Equation]

Original: where x i is the node embedding of node i , N ( i ) is the neighbor of node i , W 1 , W 2 aretraining weights, and e ji is the edge feature from node j to node i .
Revised: where x i is the node embedding of node i ; N ( i ) is the neighbor of node i ; W 1 , W 2 are training weights; and e ji is the edge feature from node j to node i .

Original: We make slight modifications to GCN and graphSAGE in order to utilize both node and edge features.
Revised: We make slight modifications to GCN and GraphSAGE in order to utilize both node and edge features.

Original: Edge features, in other words, nonzero elements of the matrix, can be arbitrary real values.
Revised: Edge features, in other words, nonzero elements of the matrix, can be arbitrary real values.

Original: Thisrequires that the desired convolutional networkshould be able to distinguish between A and itsabsolute counterpart | A | if there are both positive and negative elements in A .
Revised: This requires that the desired convolutional network be able to distinguish between A and its absolute counterpart | A | if there are both positive and negative elements in A .

Original: It also should producethe same results for A and kA for any scalar k ̸ = 0 .
Revised: It also should produce the same results for A and kA for any scalar k ̸ = 0 .

Original: 3.5 Two-level pooling
Revised: 3.5 Two-level pooling

Original: By adapting the idea of Multiset Pooling technique [30], we designed a two-level pooling basedon the fact that computations between distinct connected components are independent.
Revised: By adapting the idea of Multiset Pooling [31], we designed a two-level pooling based on the fact that computations between distinct connected components are independent.

Original: In ourapproach, the graph embedding is generated in the following steps: (1) the node embeddings are firstaggregated within the connected components to which they belong; (2) then the connected componentembeddings are summarized to yield the graph embedding.
Revised: In our approach, the graph embedding is generated in the following steps. First, node embeddings are aggregated within the connected components to which they belong. Then, the connected component embeddings are summarized to yield the graph embedding.

Original: The message passing mechanism can be viewed as a voting procedure that the central node collectsmessages from its neighbor nodes and votes for the classification of the graph from its perspective,i.e., via node embedding.
Revised: The message-passing mechanism can be viewed as a voting procedure in which the central node collects messages from its neighbor nodes and votes for the classification of the graph from its perspective via node embedding.

Original: However, since each node can only reach the connected component towhich it belongs,  we aggregate the node embeddings within each connected component.
Revised: Since each node can only reach the connected component to which it belongs, however, we aggregate the node embeddings within each connected component.

Original: Commonembedding techniques include mean-pooling, histogram of voting, and virtual node embedding [31].
Revised: Common embedding techniques include mean pooling, histogram of voting, and virtual node embedding.

Original: Without treating each connected component independently, the selection of solvers and preconditioners for the whole matrix heavily depends on the most difficult connected components to solve, e.g.,the most ill-conditioned one.
Revised: Without treating each connected component independently, the selection of solvers and preconditioners for the whole matrix heavily depends on the most difficult connected components to solve, for example, the most ill-conditioned one.

Original: Therefore, we apply a min-pooling technique to connected componentembeddings to generate the graph embedding, which indicates the solver and preconditioner that aresuitable and efficient for all connected components.
Revised: Therefore, we apply a min-pooling technique to connected component embeddings to generate the graph embedding, which indicates the solver and preconditioner that are suitable and efficient for all connected components.

Original: To illustrate the second level of pooling, we use a simple example where we choose from twocandidate preconditioned solvers for a block diagonal matrix that consists of two square matrices,i.e.,
Revised: To illustrate the second level of pooling, we use a simple example where we choose from two candidate preconditioned solvers for a block diagonal matrix that consists of two square matrices,

Original: [Equation]
Revised: [Equation]

Original: Here A and B are two distinct connected components of C .
Revised: Here A and B are two distinct connected components of C .

Original: If both solvers can be applied to Aand only the first solver is applicable to B , it gives rise to multi-labels y A , y B ∈ { 0 , 1 } 2 wherey A =
Revised: If both solvers can be applied to A and only the first solver is applicable to B , it gives rise to multi-labels y A , y B ∈ { 0 , 1 } 2 , where y A =

Original: [1 , 1] and y B =
Revised: [1 , 1] and y B =

Original: Then, the multi-label for C is y C
Revised: Then, the multi-label for C is y C

Original: =
Revised: =

Original: [1 , 0] =
Revised: [1 , 0] =

Original: min-pooling ( y A , y B ) .
Revised: min-pooling ( y A , y B ) .

Original:  Numerical results
Revised:  Numerical results

Original: The experiments for the GNN models were performed on a workstation with an NVIDIA RTX[-
-]
Revised: The experiments for the GNN models were performed on a workstation with an NVIDIA RTX[-
-]

Original: GPU and an Intel i7-11700KF CPU.
Revised: GPU and an Intel i7-11700KF CPU.

Original: Scikit-learn [32] is used to implement traditional ML methods.
Revised: Scikit-learn [32] was used to implement traditional ML methods.

Original: GNN frameworks are mainly based  on PyTorch [33] and PyTorch-Geometric [34], and evaluationmetrics are provided by TorchMetrics [35].
Revised: GNN frameworks were based mainly on PyTorch [33] and PyTorch-Geometric [34], and evaluation metrics were provided by TorchMetrics [35].

Original: 4.1 Benchmark dataset
Revised: 4.1 Benchmark dataset

Original: To assesses the performance of the GNN models, we built a benchmark using matrices from the
Revised: To assess the performance of the GNN models, we built a benchmark using matrices from the

Original: SuiteSparse Matrix Collection [36].
Revised: SuiteSparse Matrix Collection [36].

Original: Only square matrices A ∈ R n × n with a number of rows1000 ≤ m ≤ 10000 and a number of nonzero elements nnz ( A ) ≤ 200000 are selected.
Revised: Only square matrices A ∈ R n × n with a number of rows 1000 ≤ m ≤ 10000 and a number of nonzero elements nnz ( A ) ≤ 200000 were selected.

Original: The datasetcontains 614 matrices and is split into training and testing datasets using 5-fold cross-validation.
Revised: The dataset contained 614 matrices and was split into training and testing datasets using 5-fold cross-validation.

Original: Eachmatrix results in 33 entries because of the 33 combinations of solvers and preconditioners.
Revised: Each matrix resulted in 33 entries because of the 33 combinations of solvers and preconditioners.

Original: Each entryin the dataset consists of the matrix ID, the solver choice, the preconditioner choice, the solution time(with negative values indicating failure reasons such as using an incompatible preconditioner, timeout and divergence) and the ℓ 2 residual norm.
Revised: Each entry in the dataset consisted of the matrix ID, the solver choice, the preconditioner choice, the solution time (with negative values indicating failure reasons such as using an incompatible preconditioner, time out, or divergence) and the ℓ 2 residual norm.

Original: Each linear solve was run in parallel using 64 MPIprocesses on the KNL partition on Theta [37],on Theta, a Cray XC40 supercomputer atArgonne
Revised: Each linear solve was run in parallel using 64 MPI processes on the KNL partition on Theta [37], a Cray XC40 supercomputer at

Original:  National Laboratory.
Revised: Argonne National Laboratory.

Original: 4.2 Evaluation metrics
Revised: 4.2 Evaluation metrics

Original: The evaluation metrics for multi-label classification in this paper includes Label Ranking Average Precision (LRAP)
Revised: The evaluation metrics for multilabel classification in this paper includes label ranking average precision (LRAP)

Original: [38] and Normalized Discounted Cumulative Gain (NDCG) [39, 40, 41].
Revised: [38] and normalized discounted cumulative gain (NDCG) [39, 40, 41].

Original: Bothmethods enable taking prediction probability as input and return scores in [0 , 1] , where higher scoresresult in better classification accuracy.
Revised: Both methods enable taking prediction probability as input and return scores in [0 , 1] , where higher scores indicate better classification accuracy.

Original: Here, we do not use true positive rate (TPR)  because it ismeaningless for multi-label classification tasks.
Revised: Note that the metric true positive rate (TPR) was occasionally used in the related literature, but we do not use it because it is meaningless for multi-label classification tasks.

Original: TPR can be arbitrarily high if one sets the decisionthreshold sufficiently low.
Revised: TPR can be arbitrarily high if one sets the decision threshold sufficiently low.

Original: We denote the true labels by y and the prediction by ˆ y .
Revised: We denote the true labels by y and the prediction by ˆ y .

Original: The definition of LRAP is as follows:
Revised: The definition of LRAP is as follows:

Original: [Equation]
Revised: [Equation]

Original: where n is the number of samples, L ij = { k :
Revised: where n is the number of samples, L ij = { k :

Original: y ik
Revised: y ik

Original: = 1 , ˆ y ik > ˆ y ij } , rank ij = |{ k : ˆ y ik > ˆ y ij }| , and∥ · ∥ 0 computes the l 0 -norm which equals to the number of nonzeros.
Revised: = 1 , ˆ y ik > ˆ y ij } , rank ij = |{ k : ˆ y ik > ˆ y ij }| , and ∥ · ∥ 0 computes the l 0 -norm, which is equal to the number of nonzeros.

Original: The DCG score is defined as:
Revised: The DCG score is defined as follows:

Original: [Equation]
Revised: [Equation]

Original: where m is the number of classes, f is a ranking function induced from y and ˆ y .
Revised: where m is the number of classes and f is a ranking function induced from y and ˆ y .

Original: The NDCG score isthe DCG score divided by the DCG score of y , which is considered as the ideal DCG score.
Revised: The NDCG score is the DCG score divided by the DCG score of y , which is considered as the ideal DCG score.

Original: 4.3 Comaparison with traditional ML methods
Revised: 4.3 Comparison with traditional ML methods

Original: The implementations of the proposed GNN modelshave been compared with traditional ML methodssupporting multi-label classification, such as Random Forest (RF) [42], k -nearest neighbor (kNN) [43], Multi-layer Perceptron (MLP) [44, 45], and Ridge Classifier.
Revised: We compared the proposed GNN models with traditional ML methods supporting multi-label classification, such as random forest (RF) [42], k -nearest neighbor (kNN) [43], multilayer perceptron (MLP) [44, 45], and Ridge Classifier.

Original: GNN models in our experiments follow the same architecture and differ solely from the convolutionallayers.
Revised: GNN models in our experiments follow the same architecture and differ solely from the convolutional layers.

Original: Each graph convolution model consists of 5 layers.
Revised: Each graph convolution model consists of five layers.

Original: The MLP for processing graph featureshas 2 layers, and the MLP for combining node and graph embeddings has 2 layers.
Revised: The MLP for processing graph features has two layers, and the MLP for combining node and graph embeddings has two layers.

Original: More GNN layerscan cause over-smoothing, and effective training strategies [46] need to be used for such situations.
Revised: More GNN layers can cause oversmoothing, and effective training strategies [46] need to be used for such situations.

Original: Table 3 gives 5-fold average LRAP and NDCG scores and standard errors of  considered methodson the test dataset.
Revised: Table 3 gives 5-fold average LRAP and NDCG scores and standard errors of the considered methods on the test dataset.

Original: It shows that GNN models are comparable to traditional ML-based modelson the LRAP metric, with an exception that RF noticeably beats the rest. However, GNN modelssignificantly outperform traditional models on the NDCG metric.
Revised: It shows that GNN models are comparable to traditional ML-based models under the LRAP metric and significantly outperform traditional models under the NDCG metric.

Original: Specifically, the best-performing GNN model, GAT, outperfms the best-performing traditional method, RF, bya margin of more than25%.
Revised: Specifically, the best-performing GNN model, GAT, outperforms the best-performing traditional method, RF, by more than 25%.

Revised: 4.4 Overhead of the GNN inference

Original: [Table]
Revised: The overhead of the GNN inference consists of two parts: building the input graph from an input matrix and processing the data through our GNN model. Table 4 shows the detailed time costs for several representative matrices. As we can see, the processing time t p is negligible compared with the solving time. Although the cost of building the graphs is significantly higher than the processing time, it is still lower than the worst solving time of the linear system.

Original: [Figure]
Revised: Note that the same linear system is often solved many times in real applications that use PETSc. These observations imply that the cost of using the GNN models is justifiable and worthwhile relative to the solving time of the linear systems. [Table] [Table]

Original: 4.4 Case studies
Revised: 4.5 Case studies

Original: To interpret our experimental results and explain the large discrepancy between LRAP and NDCGscores, we perform a case study with the matrix rbd3200l  selected from the test dataset.
Revised: To interpret our experimental results and explain the large discrepancy between LRAP and NDCG scores, we performed case studies with the matrices rbd3200l and Trefethen_2000 selected from the test dataset.

Original: Its evaluationscores are  LRAP = 0 .
Revised: We notice that both the LRAP and NDCG metrics focus on the ranked positions of “good” classes, which are the classes associated with ground-truth label 1. The difference is that, when a “good” class is classified as “bad” and ranked low, the |L ij | in Eq. (7) of LRAP can still be large once other “good” classes are well-predicted, while in NDCG metric, the numerator is always 1. Therefore,

Original: 1809 and NDCG = 0 .
Revised: LRAP is more tolerant of the ground-truth “yes” labels that are ranked low than NDCG.

Original: Figure 2 shows the prediction results made bythe GAT model.
Revised: Figureshows the prediction results made by the GAT model.

Original: We notice that the LRAP metric focuses on the "good" classes, which are the classesassociated with labels equal to 1.
Revised: Its evaluation scores are LRAP =

Original: The metric penalizes heavily the cases when "good" classes arepredicted to be "bad" classes while false prediction for "bad" classes are ignored. Therefore, thereare four "good" classes (10, 11, 22, and 33) in rbd3200l as shown in Figure 2, and none of them iscorrectly classified, leading to a very low LRAP score. In contrast, the NDCG metric computes ascore according to the ranking induced by the prediction. For all six misclassified classes, only themiddle two classes (18 and 19) have large prediction values (or higher ranks), thus leading to a low
Revised: 0 .and

Original: NDCG score.
Revised: NDCG = 0 .

Original: Nevertheless, the other four  classes (10, 11, 22, and 33) do not have significant impacton NDCG scores for the same reason. In practice, "good" classes (the classes with label 1) attract more attention.
Revised: There are four “good” classes (10, 11, 22, and 33) in rbd3200l , as shown in

Original: The properties of LRAPand NDCG suggest that our GNN model tends to provide safe recommendations, that is, avoid false
Revised: Figure 2, and none of them are correctly classified, leading to low LRAP and NDCG scores.

Original: [Figure]
Revised: [Figure]

Original: positive predictions. In other words, our model may predict fewer "good" classes, but it tends notto mislead users to "bad" classes. Figure 3 also confirms our explanations.
Revised: A case study with the matrix Trefethen_2000 further confirms this finding. Figure 3 shows the classwise prediction results for this matrix.

Original: For classifying matrix Trefethen_2000 , GAT accomplishes less error in either "good" classes (class 18-20), therefore, ahigher LRAP score.
Revised: GAT provides a better ranking for the “good" classes (18, 19 and 20), therefore significantly scoring higher on LRAP compared with the prediction of Random

Original: We can also consider top 5 classes with the highest probabilities for  GAT(class19, 22, 18, 17, 21) and RF (class 19, 21, 33, 1, 20).
Revised: Forest (RF). As for the NDCG score, the advantage of ranking well is mitigated and the NDCG scores of GAT and RF are closer. In other words, doing well on NDCG conveys stronger effectiveness than doing well on LRAP. In practice, predicting “good" classes (with label 1) correctly is more important than predicting “bad" classes (with label 0).

Original: In this context, GAT also has less error comparingwith RF, which leads to higher NDCG scores.
Revised: High NDCG scores in Table 3 suggest that our GNN models tend to provide safe recommendations and make fewer false positive predictions. Therefore, our models are less likely to misclassify “good" classes as “bad" classes. [Figure]

Original:  Conclusions
Revised:  Conclusions

Original: Graph neural networks show great potential in classification tasks where inputs are sparse matrices ofvarying sizes.
Revised: We have shown that graph neural networks have great potential in classification tasks where inputs are sparse matrices of varying sizes.

Original: With the help of the message passing mechanism and node features that are properlydefined, the structural characteristics of the matrix, e.g., theinfluence of diagonal elements, andpatterns, are exploited inherently and expressed by the output embedding.
Revised: By using the message-passing mechanism, defining node features properly, and using the output embedding, we can exploit the structural characteristics of the matrix such as the diagonal elements and sparsity patterns.

Original: Our model also exploitsgraph features together with a two-level pooling, serving as a meaningful extension of standard GNNs.
Revised: Our methods also exploit graph features together with a two-level pooling, serving as a meaningful extension of standard GNNs.

Original: For future work, as seen in Table 3 that the performances of 4 GNN models are indistinguishable, arefinement is to explore models of a different nature (e.g., transformers) that have the potential tooutperform existing GNNs by a noticeable margin.

Original: In addition, graph coarsening techniques [47]will be deployed for arbitrary-sized (especially exceedingly large) matrices to improve the robustnessand efficiency of model training.

Original: Checklist

Original: The checklist follows the references.

Original: Please read the checklist guidelines carefully for information onhow to answer these questions.

Original: For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .

Original: You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.

Original: For example:

Original: • Did you include the license to the code and datasets?

Original: [Yes] See Section ?? .•

Original: Did you include the license to the code and datasets?

Original: [No] The code and the data are

Original: proprietary.• Did you include the license to the code and datasets?

Original: [N/A]

Original: Please do not modify the questions and only use the provided macros for your answers.

Original: Note that the

Original: Checklist section does not count towards the page limit.

Original: In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes] See Section

Original: (b) Did you describe the limitations of your work?

Original: [Yes] See Section

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [N/A]

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [N/A]

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [No] The code and

Original: data are available upon request.

Original: (b) Did you specify all the training details (e

Original: g., data splits, hyperparameters,

Original: how they were chosen)? [Yes] See Section 3.2, 3.3 and 4. (c) Did you report error bars (e
Revised: Table 3 shows that the performances of 4 GNN models are indistinguishable.

Original: g., with respect to the random seed after running experiments multiple times)?

Original: [Yes] See Table

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., typeof GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Section 4 and 4.14.

Original: If you are using existing assets (e g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] See Section 3.3, 3.4,
Revised: Future refinement includes exploring models of a different nature (e.g., transformers) that have the potential to outperform existing GNNs by a noticeable margin.

Original: and

Original: (b) Did you mention the license of the assets?

Original: [Yes] See Section 3.3, 3.4, and

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [N/A]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [N/A]

Original: [Table]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: [Table]

Original: information or offensive content?

Original: [N/A]

Original: If you used crowdsourcing or conducted research with human subjects... [Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
Revised: In addition, graph coarsening techniques [47] will be deployed for arbitrary-sized (especially exceedingly large) matrices to improve the robustness and efficiency of the GNN models.

Original: applicable?

Original: [N/A]

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?

Original: [N/A]

Original: [Table]