Original: W HAT ’ S

Original: W RONG

Original: WITH

Original: D EEP

Original:  L EARNING IN T REE S EARCH FOR C OMBINATORIAL O PTIMIZATION
Revised: W HAT ’ S W RONG WITH D EEP L EARNING IN T REE S EARCH FOR C OMBINATORIAL O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Combinatorial optimization lies at the core of many real-world problems.
Revised: Combinatorial optimization lies at the core of many real-world problems.

Original: Especially since the rise of graph neural networks (GNNs), the deep learning community has been developing solvers that derive solutions to NP-hard problems by learning the problem-speciﬁc solution structure.
Revised: Especially since the rise of graph neural networks (GNNs), the deep learning community has been developing solvers that derive solutions to NP-hard problems by learning the problem-specific solution structure.

Original: However, reproducing the results of these publications proves to be difﬁcult.
Revised: However, reproducing the results of these publications proves to be difficult.

Original: We make three contributions.
Revised: We make three contributions.

Original: First, we present an open-source benchmark suite for the NP-hard M AXIMUM I NDEPENDENT S ET problem, in both its weighted and unweighted variants.
Revised: First, we present an open-source benchmark suite for the NP-hard M AXIMUM I NDEPENDENT S ET problem, in both its weighted and unweighted variants.

Original: The suite offers a uniﬁed interface to various state-of-the-art traditional and machine learning-based solvers.
Revised: The suite offers a unified interface to various state-of-the-art traditional and machine learning-based solvers.

Original: Second, using our benchmark suite, we conduct an in-depth analysis of the popular guided tree search algorithm by Li et al.
Revised: Second, using our benchmark suite, we conduct an in-depth analysis of the popular guided tree search algorithm by Li et al.

Original: [NeurIPS 2018], testing various conﬁgurations on small and large synthetic and real-world graphs.
Revised: [NeurIPS 2018], testing various configurations on small and large synthetic and real-world graphs.

Original: By re-implementing their algorithm with a focus on code quality and extensibility, we show that the graph convolution network used in the tree search does not learn a meaningful representation of the solution structure, and can in fact be replaced by random values.
Revised: By re-implementing their algorithm with a focus on code quality and extensibility, we show that the graph convolution network used in the tree search does not learn a meaningful representation of the solution structure, and can in fact be replaced by random values.

Original: Instead, the tree search relies on algorithmic techniques like graph kernelization to ﬁnd good solutions.
Revised: Instead, the tree search relies on algorithmic techniques like graph kernelization to find good solutions.

Original: Thus, the results from the original publication are not reproducible.
Revised: Thus, the results from the original publication are not reproducible.

Original: Third, we extend the analysis to compare the tree search implementations to other solvers, showing that the classical algorithmic solvers often are faster, while providing solutions of similar quality.
Revised: Third, we extend the analysis to compare the tree search implementations to other solvers, showing that the classical algorithmic solvers often are faster, while providing solutions of similar quality.

Original: Additionally, we analyze a recent solver based on reinforcement learning and observe that for this solver, the GNN is responsible for the competitive solution quality.
Revised: Additionally, we analyze a recent solver based on reinforcement learning and observe that for this solver, the GNN is responsible for the competitive solution quality.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Various communities have been dealing with the question of how to efﬁciently solve combinatorial problems, which frequently are NP-hard.
Revised: Various communities have been dealing with the question of how to efficiently solve combinatorial problems, which frequently are NP-hard.

Original: These problems often have real-world applications in industry, for instance in staff assignment (Peters et al., 2019),
Revised: These problems often have real-world applications in industry, for instance in staff assignment (Peters et al., 2019),

Original: supply chain optimization (Eskandarpour et al., 2015), and trafﬁc optimization (B ¨ other et al., 2021).
Revised: supply chain optimization (Eskandarpour et al., 2015), and traffic optimization (B ¨ other et al., 2021).

Original: In recent years, also the machine learning community has been engaged in solving combinatorial problems.
Revised: In recent years, also the machine learning community has been engaged in solving combinatorial problems.

Original: The reason why machine learning might be able to solve such problems faster compared to classical solvers like Gurobi (Gurobi Optimization LLC, 2021)  is thatthe solutions to a speciﬁc problem often follow a certain structure, which we might be able to learn given enough data.
Revised: One reason that learning-based approaches are interesting, even though commercial general purpose solvers like Gurobi (Gurobi Optimization LLC, 2021) exist, is that we might be able to learn the solution structure of a specific family of instances , e.g., similar vehicle routing problems instances are solved everyday (Khalil et al., 2017; Dong et al., 2021).

Original: Commonly used machine learning approaches for combinatorial optimization include Deep Reinforcement
Revised: Such a trained model can be used together with an algorithmic component to find feasible solutions quickly; we outline in Section 2.

Original: Learning (Khalil et al., 2017; Ahn et al., 2020) and Graph Neural Networks (GNNs) (Cappart et al., 2021; Li et al., 2018).
Revised: the designs that past works have used.

Original: However, these statistics-based techniques have the well-known problem of reproducibility (Ioannidis, 2005; Baker, 2016), which is a particular problem in machine learning research with untested or even unpublished code and data sets (Kapoor & Narayanan, 2020;
Revised: However, these statistics-based techniques have the well-known problem of reproducibility (Ioannidis, 2005; Baker, 2016), which is a particular problem in machine learning research with untested or even unpublished code and data sets (Kapoor & Narayanan, 2020;

Original: Ding et al., 2020), and is the reason for the interest in initiatives such as Papers With Code 1 and the ReScience Journal 2 .
Revised: Ding et al., 2020), and is the reason for the interest in initiatives such as Papers With Code 1 and the ReScience Journal 2 .

Original: Keeping the importance of reproducibility in mind, we evaluate various machine learning approaches for combinatorial optimization, and compare them to traditional solvers.
Revised: Keeping the importance of reproducibility in mind, we evaluate various machine learning approaches for combinatorial optimization, and compare them to traditional solvers.

Original: Following previous research (Li et al., 2018; Ahn et al., 2020), we focus on the M AXIMUM (W EIGHTED ) I NDEPENDENT S ET problem (Miller & Muller, 1960; Karp, 1972).
Revised: We focus on the M AXIMUM (W EIGHTED ) I NDEPENDENT S ET problem (Miller & Muller, 1960; Karp, 1972), as the previous works we analyze have centered around variants of this problem.

Original: Overall, we contribute the following.
Revised: Overall, we contribute the following.

Original: • We provide an open-source, extensible benchmark suite for M AXIMUM (W EIGHTED )
Revised: • We provide an open-source, extensible benchmark suite for M AXIMUM (W EIGHTED )

Original: I NDEPENDENT S ET solvers.
Revised: I NDEPENDENT S ET solvers.

Original: The software currently supports ﬁve state-of-the-art solvers, I NTEL -T
Revised: The software currently supports five state-of-the-art solvers, I NTEL -T

Original: REE S EARCH (Li, Chen, and Koltun, 2018), G UROBI (Gurobi Optimization LLC, 2021), K A MIS (Lamm et al., 2017; Hespe et al., 2019; Lamm et al., 2019), L EARNING W HAT T O D EFER (Ahn, Seo, and Shin, 2020), and DGL-T REE S EARCH .
Revised: REE S EARCH (Li, Chen, and Koltun, 2018), G UROBI (Gurobi Optimization LLC, 2021), K A MIS (Lamm et al., 2017; Hespe et al., 2019; Lamm et al., 2019), L EARNING W HAT T O D EFER (Ahn, Seo, and Shin, 2020), and DGL-T REE S EARCH .

Original: Our DGLT REE S EARCH is a modern re-implementation of the I NTEL -T
Revised: Our DGLT REE S EARCH is a modern re-implementation of the I NTEL -T

Original: REE S EARCH , implemented in PyTorch (Paszke et al., 2019) and the Deep Graph Library (Wang et al., 2019), with a focus on clean, readable code, as well as performance, and it ﬁxes various issues of the original code.
Revised: REE S EARCH , implemented in PyTorch (Paszke et al., 2019) and the Deep Graph Library (Wang et al., 2019), with a focus on clean, readable code, as well as performance, and it fixes various issues of the original code.

Original: Our evaluation suite lays the ground for further research on hard combinatorial (graph) problems and aims at providing a fair and comparable environment for further evaluations.
Revised: Our evaluation suite lays the ground for further research on hard combinatorial (graph) problems and aims at providing a fair and comparable environment for further evaluations.

Original: • Using our re-implementation of the tree search, we propose and analyze additional techniques aiming at improving the guided search.
Revised: • Using our re-implementation of the tree search, we propose and analyze additional techniques aiming at improving the guided search.

Original: Employing the benchmark suite, we conduct an exhaustive analysis of various conﬁgurations of the tree search algorithms, showing that the results of the highly-cited I NTEL -T
Revised: Employing the benchmark suite, we conduct an exhaustive analysis of various configurations of the tree search algorithms, showing that the results of the highly-cited I NTEL -T

Original: REE S EARCH approach are not reproducible, neither with the original code nor with our re-implementation.
Revised: REE S EARCH approach are not reproducible, neither with the original code nor with our re-implementation.

Original: When exploring the design space further, we show that the various techniques used by the tree search algorithm to improve the results, like graph kernelization, are the reason for good performance, especially on hard data sets.
Revised: When exploring the design space further, we show that the various techniques used by the tree search algorithm to improve the results, like graph kernelization, are the reason for good performance, especially on hard data sets.

Original: In fact, replacing the GNN output with random values performs similar to using the trained network.
Revised: In fact, replacing the GNN output with random values performs similar to using the trained network.

Original: • Having analyzed the conﬁguration space, we compare the tree search approaches to the classical solvers like G UROBI and K A MIS, showing that problem-tailored solvers are often the superior approach.
Revised: • Having analyzed the configuration space, we compare the tree search approaches to the classical solvers like G UROBI and K A MIS, showing that problem-tailored solvers are often the superior approach.

Original: Without using techniques like graph reduction in the tree search, classical solvers are superior.
Revised: Without using techniques like graph reduction in the tree search, classical solvers are superior.

Original: The classical solvers show to be more efﬁcient even when accessing these routines – that have been implemented for the algorithmic solvers in the ﬁrst place – in the tree search.
Revised: The classical solvers show to be more efficient even when accessing these routines – that have been implemented for the algorithmic solvers in the first place – in the tree search.

Original: Last, we show that L EARNING W HAT T O D EFER seems to be able to ﬁnd good results very quickly, indicating that unsupervised reinforcement learning for combinatorial problems is a promising direction for future research.
Revised: Last, we show that L EARNING W HAT T O D EFER seems to be able to find good results very quickly, indicating that unsupervised reinforcement learning for combinatorial problems is a promising direction for future research.

Original:  The remainder of this paper is structured as follows. In Section 2, we introduce the various solvers and how they are integrated into our open-source benchmark suite, before evaluating them in Section 3. We discuss some related work in Section 4, and conclude in Section 5.
Revised:  I NDEPENDENT S ET S OLVERS AND M ACHINE L EARNING FOR C OMBINATORIAL O PTIMIZATION

Original: In this section, we formally introduce the M AXIMUM I NDEPENDENT S ET (MIS) problem as well as the solvers included in our analysis.
Revised: In this section, we formally introduce the M AXIMUM I NDEPENDENT S ET (MIS) problem as well as the solvers included in our analysis, and discuss related work in the broader space of deep learning for combinatorial optimization.

Original: Given an undirected graph G =
Revised: Given an undirected graph G =

Original: ( V, E ) , an independent set is a set of vertices S ⊆ V for which for all vertices u, v ∈ S , ( u, v ) (cid:54)∈ E .
Revised: ( V, E ) , an independent set is a set of vertices S ⊆ V for which for all vertices u, v ∈ S , ( u, v ) ̸∈ E .

Original: For u ∈ V , let w u be its weight, and let IS ( G ) be all independent sets of G , then the M AXIMUM W EIGHTED I NDEPENDENT S ET (MWIS) problem aims at determining 
Revised: For u ∈ V , let w u be its weight, and let IS ( G ) be all independent sets of G , then the M AXIMUM W EIGHTED I NDEPENDENT S ET (MWIS) problem aims at determining arg max S ∈ IS ( G ) (cid:80)

Original: [Equation]
Revised: u ∈ S w u .

Original: The unweighted MIS problem is equivalent to the MWIS problem, where f.a. u ∈ V , w u = 1 .
Revised: The unweighted MIS problem is equivalent to the MWIS problem, where f.a. u ∈ V , w u = 1 .

Original: Both problems are strongly NP-complete (Garey & Johnson, 1978).
Revised: Both problems are strongly NP-complete (Garey & Johnson, 1978).

Original: Next, we brieﬂy explain the solvers that we use to ﬁnd such maximum independent sets.
Revised: Next, we briefly explain the solvers that we use to find such maximum independent sets.

Original: Gurobi.
Revised: Gurobi.

Original: G UROBI is a commercial mathematical optimization solver.
Revised: G UROBI is a commercial mathematical optimization solver.

Original: We access G UROBI in Python using the abstraction layer PuLP 3 , and formulate the MWIS problem as a linear program.
Revised: There are various ways of formulating the M(W)IS problem mathematically (Butenko, 2003). In the main paper, we formulate the MWIS problem as the linear program above, and discuss other variants in Appendix D.

Original: KaMIS.
Revised: KaMIS.

Original: K A MIS is an open-source solver tailored towards the MIS and MWIS problems.
Revised: K A MIS is an open-source solver tailored towards the MIS and MWIS problems.

Original: It offers support both for the unweighted case (Lamm et al., 2017; Hespe et al., 2019) as well as the weighted case (Lamm et al., 2019).
Revised: It offers support both for the unweighted case (Lamm et al., 2017; Hespe et al., 2019) as well as the weighted case (Lamm et al., 2019).

Original: It employs graph kernelization and an optimized branch-and-bound algorithm to efﬁciently ﬁnd independent sets.
Revised: It employs graph kernelization and an optimized branch-and-bound algorithm to efficiently find independent sets.

Original: Note that the algorithms and techniques differ between the weighted and unweighted cases.
Revised: Note that the algorithms and techniques differ between the weighted and unweighted cases.

Original: We use the code unmodiﬁed from the ofﬁcial repository 4 .
Revised: We use the code unmodified from the official repository 3 .

Original: Intel-TreeSearch.
Revised: Intel-TreeSearch.

Original: In their inﬂuential paper, Li et al.
Revised: In their influential paper, Li et al.

Original: (2018) propose a guided tree search algorithm to ﬁnd maximum independent sets of a graph.
Revised: (2018) propose a guided tree search algorithm to find maximum independent sets of a graph.

Original: The idea is to train a graph convolutional network (GCN) (Kipf & Welling, 2017), which assigns each vertex a probability of belonging to the independent set, and then greedily and iteratively assign vertices to the set.
Revised: The idea is to train a graph convolutional network (GCN) (Kipf & Welling, 2017), which assigns each vertex a probability of belonging to the independent set, and then greedily and iteratively assign vertices to the set.

Original: They furthermore employ the reduction and local search algorithms by K A MIS to speed up the computation.
Revised: They furthermore employ the reduction and local search algorithms by K A MIS to speed up the computation.

Original: We use their published code 5 , which unfortunately is not runnable in its default state.
Revised: We use their published code 4 , which unfortunately is not runnable in its default state.

Original: We apply a git patch 6 to make the code runnable, enable further evaluation by collecting statistics, and add command-line ﬂags for more ﬁne-grained control of the solver conﬁguration.
Revised: We apply a git patch 5 to make the code runnable, enable further evaluation by collecting statistics, and add command-line flags for more fine-grained control of the solver configuration.

Original: For a detailed explanation of the tree search algorithm and details of the model, we refer to Appendix B as well as the original paper. In Appendix C, we give some details on possible complications with the original code. 
Revised: In Appendix C, we give some details on possible complications with the original code. Because a good knowledge of the algorithm is important to follow the remainder of this paper, we briefly describe the algorithm. A pseudocode description of the algorithm is found in Algorithm 1.

Revised: The core element of the tree search is a queue 6 P of partial solutions S ∈ { 0 , 1 , ? } | V | to the MIS problem, i.e., labelings of the graph at hand marking each vertex as either included in the MIS (1), excluded (0), or unlabeled (a decision is still to be made, ?).

Revised: To be a valid (partial) solution to the MIS problem, each element in the queue fulfills the constraint that no two adjacent vertices can both be included.

Revised: Furthermore, all vertices adjacent to an included one must be excluded .

Original: DGL-TreeSearch.
Revised: Given a graph G = ( V, E ) , we start with an empty solution, i.e., f.a. v ∈ V , S v = ? . In each step of the tree search, a partial labeling is popped from the queue, and the residual graph G residual consisting only of the unlabeled vertices is constructed. Next, we obtain a predefined number m of probability maps , each of which contains a value for all vertices of the residual graph, posing the “probability of being in the MIS”, by calling the trained GCN, which outputs its assignments from vertices to probabilities, i.e., GCN ( G ) ∈ [0 , 1] | V |× m .

Revised: From each of these probability maps, a new partial solution is derived as follows:

Revised: The probabilities of the maps are sorted in descending order.

Revised: The vertex with the highest probability gets labeled as included , and all adjacent vertices as excluded .

Revised: This step is repeated until we would have to label an already excluded vertex as included , in which case we break, and add the partial solution to the queue.

Revised: In case all vertices are labeled we obtained a full solution.

Revised: This procedure is repeated for all probability maps.

Revised: For further modifications of this algorithm (e.g., reduction), we refer to Appendix B.

Revised: DGL-TreeSearch.

Original: Because the code provided by Li et al.
Revised: Because the code provided by Li et al.

Original: (2018) might be difﬁcult to read and maintain, and hence is prone to errors in the evaluation, we re-implement the tree search using
Revised: (2018) might be difficult to read and maintain, and hence is prone to errors in the evaluation, we re-implement the tree search using

Original: PyTorch (Paszke et al., 2019) and the established Deep Graph Library (Wang et al., 2019).
Revised: PyTorch (Paszke et al., 2019) and the established Deep Graph Library (Wang et al., 2019).

Original: Our implementation aims at offering a more readable and modern implementation, which beneﬁts from improvements in the two deep learning libraries during recent years.
Revised: Our implementation aims at offering a more readable and modern implementation, which benefits from improvements in the two deep learning libraries during recent years.

Original: Furthermore, it ﬁxes various issues of the original implementation that sometimes deviates from the paper.
Revised: Furthermore, it fixes various issues of the original implementation that sometimes deviates from the paper.

Original: Additionally, we implement further techniques to improve the search, like queue pruning, and weighted selection of the next element, as well as multi-GPU functionality.
Revised: Additionally, we implement further techniques to improve the search, like queue pruning, and weighted selection of the next element, as well as multi-GPU functionality.

Original: The code is delivered in conjunction with the benchmarking suite.

Original: Learning What To Defer.
Revised: Learning What To Defer.

Original: We test L EARNING W HAT T O D EFER (L W D), an unsupervised deep reinforcement learning-based solution introduced by Ahn et al.
Revised: We test L EARNING W HAT T O D EFER (L W D), an unsupervised deep reinforcement learning-based solution introduced by Ahn et al.

Original: Their idea is similar to the tree search, as the algorithm iteratively assigns vertices to the independent set.
Revised: Their idea is similar to the tree search, as the algorithm iteratively assigns vertices to the independent set.

Original: However, this is not done using a supervised GCN, but instead by an unsupervised agent built upon the GraphSAGE architecture (Hamilton et al., 2017) and trained by Proximal Policy Optimization (Schulman et al., 2017).
Revised: However, this is not done using a supervised GCN, but instead by an unsupervised agent built upon the GraphSAGE architecture (Hamilton et al., 2017) and trained by Proximal Policy Optimization (Schulman et al., 2017).

Original: There is no queue of partial solutions.
Revised: There is no queue of partial solutions.

Original: We refer to the original paper for details on the algorithm.
Revised: We refer to the original paper for details on the algorithm.

Original: As their code 7 does not work with generic input, we patch their code.
Revised: As their code 7 does not work with generic input, we patch their code.

Original: Our open-source benchmarking suite 8 integrates all these solvers in one easily accessible commandline interface using Anaconda (Anaconda Inc., 2020), with a uniﬁed input and output format.
Revised: Our open-source benchmarking suite 8 integrates all these solvers in one easily accessible commandline interface using Anaconda (Anaconda Inc., 2020), with a unified input and output format.

Original: We provide our code for DGL-T REE S EARCH and our G UROBI interface directly, and download, compile, and patch the other solvers on-demand.
Revised: We provide our code for DGL-T REE S EARCH and our G UROBI interface directly, and download, compile, and patch the other solvers on-demand.

Original: It handles the correct invocation of the solvers and allows to quickly run experiments on various solvers in different conﬁgurations.
Revised: It handles the correct invocation of the solvers and allows to quickly run experiments on various solvers in different configurations. what defer 2. R ELATED W ORK

Revised: AI for Combinatorial Optimization.

Original: E VALUATION
Revised: Research at the intersection of artificial intelligence and combinatorial optimization is not limited to MIS. These methods differ in the problem they tackle, and how algorithmic components and learned components interact. The evolutionary computation community, for example, has researched various NP-hard routing problems, like the V EHICLE R OUTING P ROBLEM (Berger & Barkaoui, 2003; Potvin, 2009) and the M ULTIPLE R OUTES problem (B¨other et al., 2021).

Revised: In a nutshell, evolutionary optimization is an iterative improvement that is not guided by a learned component, but instead tries to mutate and combine existing solutions randomly, in order to find better solutions.

Revised: ML-based solvers often are variants of branch-and-bound solvers.

Revised: In general, such solvers partition the search space by some rule set; for ML-based solvers, the partition rules are not pre-defined, but given by a model that learned how to branch (Balcan et al., 2018).

Revised: The concrete algorithms differ in the model used, how they utilize the model to branch within the search space, and how the model is trained.

Revised: For example, the I NTEL -T REE S EARCH trains a GCN on pre-labeled data, and expects from its guidance model a probability for each vertex that describes how likely it is that the vertex belongs to the MIS, and then greedily operates on these probabilities.

Revised: In comparison to that, L W D instead models the branching process as an unsupervised agent that can choose vertices in a Markov Decision Process, that similarly to the tree search iteratively picks vertices, but for example allows to roll-back invalid solutions, instead of greedily pushing forward.

Revised: This methodology applies to other combinatorial problems as well.

Revised: For example, Kool et al.

Revised: (2019) propose an attention-based encoder-decoder-architecture as a model for solving T

Revised: RAVELLING S ALESMAN P ROBLEM

Revised: TSP

Revised: ) instances, where the decoder iteratively outputs probabilties for each vertex to be visited next.

Revised: They analyze different algorithmic components that utilize that model, for example, greedily following the most probable vertex, or sampling multiple tours and picking the best.

Revised: Other research has shown that the question of scaling such architectures to real-world instances poses a challenge by itself, and that currently, algorithmic solvers often outperform ML-based solvers for larger instances (Joshi et al., 2021).

Revised: A theoretical understanding of how such models extrapolate to different instance families is also topic of current research (Xu et al., 2021).

Revised: With this discussion, we want to show that the design space for ML-based solvers is broad, as on the one hand, one has to design a model that learns how to solve a problem, and on the other hand build an efficient algorithmic component that utilizes that model to actually find a solution.

Revised: These components can become very complex, as seen in recent work by Nair et al.

Revised: (2021) which aims at solving Mixed Integer Programs and uses two neural network components instead of one; the neural diving component finds variable assignments, while the neural branching component guides the branch-and-bound algorithm in its next step.

Revised: Other Solvers for Maximum Independent Set.

Revised: Next to the solvers included in this paper, there are some other solvers available.

Revised: Khalil et al.

Revised: (2017) propose S2V-DQN, in which they use Q-learning to solve the minimum vertex cover problem 9 .

Revised: Compared to LwD, which marks multiple vertices as part of the MIS in a single step, S2V-DQN only labels a single vertex in each step.

Revised: We mention the recent publication by Hespe et al.

Revised: (2021) introducing some new reduction rules for MIS, which have not yet been included in K A MIS.

Revised: Another heuristic for MIS that we do not consider in this paper in favor of

Revised: K A MIS is GRASP (Feo et al., 1994).

Revised: Note that all of these solvers are heuristics and hence only solve MIS approximately; exact solvers have been proposed by Jain & Seshadhri (2020);

Revised: Xiao &

Revised: Nagamochi (2017); Tomita et al.

Revised: (2010), for example, and theory has been working on understanding why and in what models MIS poses to be difficult (Censor-Hillel et al., 2017).

Revised:  E VALUATION

Original: In this section, we ﬁrst introduce our experimental setup and then focus on the analysis of the supervised tree search approach for combinatorial optimization in Section 3.1.
Revised: In this section, we first introduce our experimental setup and then focus on the analysis of the supervised tree search approach for combinatorial optimization in Section 3.1.

Original: After having derived a good conﬁguration for the tree search algorithm, we continue to compare the tree search algorithms to the other classical and reinforcement learning solvers in Section 3.2.
Revised: After having derived a good configuration for the tree search algorithm, we continue to compare the tree search algorithms to the other classical and reinforcement learning solvers in Section 3.2.

Original: We investigate the scalability of the approaches in Section 3.3, and analyze the behavior on the weighted MIS problem in Section 3.4.
Revised: We investigate the scalability of the approaches in Section 3.3, and analyze the behavior on the weighted MIS problem in Section 3.4.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: We run all our experiments on an NVIDIA DGX-1 with two 20-Core Intel
Revised: We run all our experiments on an NVIDIA DGX-1 with two 20-Core Intel

Original: Xeon E5-2698 CPUs at 2.2 GHz, leading to overall 40 physical and 80 logical cores, 512 GB of  memory, and a total of eight NVIDIA Tesla V100 GPUs.
Revised: Xeon E5-2698 CPUs at 2.2 GHz, leading to overall 40 physical and 80 logical cores, 512 GB of [Table] memory, and a total of eight NVIDIA Tesla V100 GPUs.

Original: For each experiment, we explicitly state the number of threads and GPUs used.
Revised: For each experiment, we explicitly state the number of threads and GPUs used.

Original: We run Ubuntu 20.04.1 LTS, using Linux kernel 4.15.0-124 .
Revised: We run Ubuntu 20.04.1 LTS, using Linux kernel 4.15.0-124 .

Original: Datasets.
Revised: Datasets.

Original: We evaluate the various solvers and conﬁgurations using both real-world datasets as well as generated random graphs in various sizes.
Revised: We evaluate the various solvers and configurations using both real-world datasets as well as generated random graphs in various sizes.

Original: We make use of the random graph models by Erd˝os &
Revised: We make use of the random graph models by Erd ˝ os &

Original: R´enyi (1960) (ER), Albert & Barab´asi (2002) (BA), Holme & Kim (2002) (HK), Watts & Strogatz (1998) (WS) and the Hyperbolic Random Graph model (Krioukov et al., 2010) (HRG).
Revised: R ´ enyi (1960) (ER), Albert & Barab ´ asi (2002) (BA), Holme & Kim (2002) (HK), Watts & Strogatz (1998) (WS) and the Hyperbolic Random Graph model (Krioukov et al., 2010) (HRG).

Original: The graph the time limit is 5 minutes.

Original: We refer to Appendix B for detailed explanations of the techniques used.

Original: We mark the conﬁguration/solver that has the best average MIS in bold, and grey out conﬁgurations for which solutions were found for less than 20 % of all graphs.

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table]

Original: [Figure]

Original: [Table] [Figure] [Table] generation functionality, employing NetworkX (Hagberg et al., 2008) and girgs (Bl¨asius et al., 2019) as backends, is integrated into our benchmark suite.
Revised: The graph generation functionality, employing NetworkX (Hagberg et al., 2008) and girgs (Bl ¨ asius et al., 2019) as backends, is integrated into our benchmark suite.

Original: For real-world datasets, we focus on the  SATLIB dataset (Hoos & St¨utzle, 2000), which consists of synthetic 3-SAT instances,and the vertex cover benchmark (VC-BM) (Xu et al., 2007) consisting of 40 graphs on which ﬁnding the maximum independent set is synthetically made hard; we also test various other graphs, like citation networks.
Revised: For real-world datasets, we focus on the hard SATLIB dataset (Hoos & St¨utzle, 2000), which consists of synthetic 3-SAT instances, the vertex cover benchmark (VC-BM) (Xu et al., 2007) consisting of 40 graphs on which finding the maximum independent set is synthetically made hard, and the DIMACS challenge graphs; we also test various other graphs, like citation networks.

Original: Details on these datasets, as well as detailed descriptions and splits of all mentioned datasets, and hyperparameters of graph generation, can be found in Appendix D.
Revised: Details on these datasets, as well as detailed descriptions and splits of all mentioned datasets, and hyperparameters of graph generation, can be found in Appendix E.

Original: 3. A NALYSIS OF THE T REE S EARCH
Revised: 3. A NALYSIS OF THE T REE S EARCH

Original: In this subsection, we analyze the I NTEL -T
Revised: In this subsection, we analyze the I NTEL -T

Original: REE S EARCH and DGL-T REE S EARCH .
Revised: REE S EARCH and DGL-T REE S EARCH .

Original: These supervised approaches require training a GCN (Kipf & Welling, 2017).
Revised: These supervised approaches require training a GCN (Kipf & Welling, 2017).

Original: Following Li et al.
Revised: Following Li et al.

Original: (2018), we train both on the SATLIB dataset for 20 epochs using the Adam optimizer (Kingma & Ba, 2015).
Revised: (2018), we train both on the SATLIB dataset for 20 epochs using the Adam optimizer (Kingma & Ba, 2015).

Original: As the
Revised: As the

Original: GCN outputs multiple probability maps (c.f. Appendix B), we employ the hindsight loss, which, for multiple choices, outputs the loss of the best choice (Guzm´an-Rivera et al., 2012; Chen & Koltun, 2017).
Revised: GCN outputs multiple probability maps (c.f. Appendix B), we employ the hindsight loss, which, for multiple choices, outputs the loss of the best choice (Guzm´an-Rivera et al., 2012; Chen & Koltun, 2017).

Original: We ﬁx the number of probability maps to 32, to enable comparison with Li et al.
Revised: We fix the number of probability maps to 32, to enable comparison with Li et al.

Original: We test the solvers in different conﬁgurations on various graphs; these conﬁgurations are assorted with increasing levels of complexity and intuitively should improve the solution quality.
Revised: We test the solvers in different configurations on various graphs; these configurations are assorted with increasing levels of complexity and intuitively should improve the solution quality.

Original: Details on what effects the individual conﬁguration options have are given in Appendix B.
Revised: Details on what effects the individual configuration options have are outlined in Appendix B.

Original: The results can be found in Table 1.
Revised: An excerpt of our results can be found in Table 1; the full table can be found in the Appendix A in Table 2.

Original:  First, we discuss the results of the random graphs.
Revised: Random Graph Results. First, we discuss the results of the random graphs.

Original: For all small graphs, both tree searches in all conﬁgurations ﬁnd solutions most of the time. 
Revised: For all small graphs, both tree searches in all configurations find solutions most of the time. These are close to optimal as soon [Table] [Table]

Revised: I n t e l (

Revised: [Table]

Original: These are close to optimal as soon as reduction ( +r ) and local search ( +ls ) are involved.
Revised: ) I n t e l r + l s ( c ) [Table] L [Table]

Revised: [Table]

Revised: L

Revised: [Table]

Revised: l s

Revised: [Table]

Revised: q

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: l

Revised: [Table]

Revised: l

Revised: [Table]

Revised: [Table]

Revised: s ( e ) D

Revised: G L q p + w p

Revised: f ) D

Revised: G L r + l s + q p + w p + r a nd 253545 as reduction ( +r ) and local search ( +ls ) are involved.

Original: Notably, on larger  graphs, in the default variants, where neither reduction nor local search is enabled, both I NTEL -T
Revised: Notably, on larger non-ER random graphs, in the default variants, where neither reduction nor local search is enabled, both I NTEL -T

Original: REE S EARCH and DGL-T REE S EARCH do not often ﬁnd a solution.
Revised: REE S EARCH and DGL-T REE S EARCH do not often find a solution.

Original: This shows that for medium-sized graphs, vanilla tree searches cannot discover solutions within a feasible time limit.
Revised: This shows that for medium-sized graphs, vanilla tree searches cannot discover solutions within a feasible time limit.

Original: The reductions often single-handedly solve the problem instance, as seen for example on the simple BA graphs, where a solution is found instantaneously.
Revised: The reductions often single-handedly solve the problem instance, as seen for example on the simple BA graphs, where a solution is found instantaneously.

Original:  To approach the issue of requiring hand-tailored reduction techniques to obtain any solution, with DGL-T REE S EARCH , we analyze queue pruning and the weighted queue pop.
Revised: Queue Pruning and Weighted Queue Pop. To approach the issue of requiring hand-tailored reduction techniques to obtain any solution, with DGL-T REE S EARCH , we analyze queue pruning and the weighted queue pop.

Original: While queue pruning itself performs similar to the default conﬁguration for all graphs, the weighted pop vastly increases the number of solutions found.
Revised: While queue pruning itself performs similar to the default configuration for all graphs, the weighted pop vastly increases the number of solutions found.

Original: For example, for large hyperbolic random graphs, in the default conﬁguration, the DGL-T REE S EARCH was only able to ﬁnd solutions for 5 graphs (and the I NTELT REE S EARCH was not able to ﬁnd any solution), whereas using queue pruning together with the weighted queue pop, we ﬁnd solutions for 98 % of all large HRGs.
Revised: For example, for large hyperbolic random graphs, in the default configuration, the DGL-T REE S EARCH was only able to find solutions for 5 graphs (and the I NTEL -T REE S EARCH was not able to find any solution), whereas using queue pruning together with the weighted queue pop, we find solutions for 98 % of all large HRGs.

Original: A similar observation can be made for WS graphs.
Revised: A similar observation can be made for WS graphs.

Original: Interestingly, it seems like the MIS problem is harder to address with the default approach on graphs that try to model real-world networks, such as the just mentioned HRGs and WS graphs (Bl¨asius et al., 2018).
Revised: Interestingly, it seems like the MIS problem is harder to address with the default approach on graphs that try to model real-world networks, such as the just mentioned HRGs and WS graphs (Bl¨asius et al., 2018).

Original: Overall, queue pruning might be desirable to reduce memory consumption, but its impact on solving quality and time is limited; on the other hand, weighted queue popping is a general idea that brings some more depth-search-like behavior into the breadth-search-like approach at hand, and thus enables the solver to ﬁnd a lot more solutions. 
Revised: Overall, queue pruning might be desirable to reduce memory consumption, but its impact on solving quality and time is limited; on the other hand, weighted queue popping is a general idea that brings some more depth-search-like behavior into the breadth-search-like approach at hand, and thus enables the solver to find a lot more solutions. Importance of Reduction.

Original: For small and large BA, HK, WS graphs as well as HRGs, the reduction itself already leads to an achieved average approximation of 1.
Revised: For small and large BA, HK, WS graphs as well as HRGs, the reduction itself already leads to an achieved average approximation of 1.

Original: Only for the large ER graphs, for which G UROBI was not able to ﬁnd provably optimal assignments, the local search further improves the average independent set size.
Revised: Only for the large ER graphs, for which G UROBI was not able to find provably optimal assignments, the local search further improves the average independent set size.

Original: Multithreading cannotfurther improve the results of the random graphs. 
Revised: Multithreading cannot improve the results of the random graphs. Hard Real-World Graph Results.

Original: Now, we discuss the results for real-world graphs.
Revised: Now, we discuss the results for the hard benchmark datasets.

Original: Unlike previously, where we just aimed at ﬁnding an MIS on random graphs, solving the MIS problem on SATLIB is equivalent to ﬁnding a satisﬁable assignment to synthesized hard 3-SAT instances.
Revised: Unlike previously, where we just aimed at finding an MIS on random graphs, solving the MIS problem on SATLIB is equivalent to finding a satisfiable assignment to synthesized hard 3-SAT instances.

Original: Thus, one can expect this data set to be particularly hard.
Revised: Thus, one can expect this data set to be particularly hard.

Original: Our experiments conﬁrm this, as both implementations rarely ﬁnd solutions within the time limit without reduction enabled.
Revised: Our experiments confirm this, as both implementations rarely find solutions within the time limit without reduction enabled.

Original: We can see that, similar to the random graphs, the weighted queue pop enables the search to at least ﬁnd some solution, as this conﬁguration is at least able to ﬁndinstead of no results at all.
Revised: We can see that, similar to the random graphs, the weighted queue pop enables the search to at least find some solution, as this configuration is at least able to find 211 instead of no results at all.

Original: However, the average independent set size is 357, which is not very good, considering each MIS in this dataset contains at least 403 vertices.
Revised: However, the average independent set size is 357, which is not very good, considering each MIS in this dataset contains at least 403 vertices.

Original: Consideringthe other hard dataset, VC-BM, due to the higher time limit, the default conﬁgurations are already able to ﬁnd some solution.
Revised: Considering VC-BM, due to the higher time limit, the default configurations are already able to find some solution.

Original: Interestingly, for VC-BM, the reductions do not improve the performance of the solvers.
Revised: Interestingly, for VC-BM, the reductions do not improve the performance of the solvers.

Original: Hence, the weighted queue pop is very important for ﬁnding results quickly, shrinking the time needed from the default 140 seconds to under 70 seconds.
Revised: Hence, the weighted queue pop is very important for finding results quickly, shrinking the time needed from the default 140 seconds to under 70 seconds.

Original: We also see that the weighted  pop enables discovering a solution for the Wikipedia datasets and the bitcoin-alpha graph.
Revised: For DIMACS, we see similar behavior to SATLIB, i.e., the reduction and local search techniques improve the average MIS, and the weighted queue pop enables faster discovery of solutions. Impact of the Findings.

Original: These results are interesting for various reasons.
Revised: These results are interesting for various reasons.

Original: First, the models are trained onthe SATLIB data, so they do not have to generalize to a different solution structure for this dataset. Still, the default conﬁguration does not ﬁnd any independent set for  this dataset.
Revised: First, the models are trained on SATLIB instances, but the default configuration does not find any independent set for other instances of the this family, which was one motivation for using machine learning for optimization (c.f. Section 1).

Original: This contradicts the original paper by Li et al.
Revised: The results contradict the original paper by Li et al.

Original: (2018), which claims an average MIS of 426 and related work by Ahn et al.
Revised: (2018), which claims an average MIS of 426 and related work by Ahn et al.

Original: (2020), claiming an average MIS of 418.
Revised: (2020), claiming an average MIS of 418.

Original: Additionally to our own trained weights, we test the model weights provided in the ofﬁcial I NTEL -T
Revised: Additionally to our own trained weights, we test the model weights provided in the official I NTEL -T

Original: REE S EARCH repository, but do not ﬁnd any differences in the results.
Revised: REE S EARCH repository, but do not find any differences in the results.

Original: Ahn et al.
Revised: Ahn et al.

Original: (2020) mention that they modiﬁed the ofﬁcial I NTEL -T
Revised: (2020) mention that they modified the official I NTEL -T

Original: REE S EARCH code.
Revised: REE S EARCH code.

Original: Unfortunately, neither the paper documents how exactly their queue pruning has been implemented, nor were the authors themselves able to provide us their modiﬁcations to the original
Revised: Unfortunately, neither the paper documents how exactly their queue pruning has been implemented, nor were the authors themselves able to provide us their modifications to the original

Original: I NTEL -T
Revised: I NTEL -T

Original: REE S EARCH repository when we contacted them.
Revised: REE S EARCH repository when we contacted them.

Original: Hence, it remains unclear whether the difference in the SATLIB results between Ahn et al.
Revised: Hence, it remains unclear whether the difference in the SATLIB results between Ahn et al.

Original: and our experiments stems from how they implemented queue pruning, or from some unwanted side effects in their experiments (e.g., the reduction may have been still enabled).
Revised: and our experiments stems from how they implemented queue pruning, or from some unwanted side effects in their experiments (e.g., the reduction may have been still enabled).

Original: Furthermore, we unsuccessfully contacted Li et al.
Revised: Furthermore, we unsuccessfully contacted Li et al.

Original: (2018) about our ﬁndings.
Revised: (2018) about our findings.

Original: As both the I NTEL -T
Revised: As both the I NTEL -T

Original: REE S EARCH and our re-implementation DGL-T REE S EARCH , which was written from scratch, exhibit this behavior, we suspect that there must be an undocumented modiﬁcation or problem in the experiments that leads to Ahn et al.
Revised: REE S EARCH and our re-implementation DGL-T REE S EARCH , which was written from scratch, exhibit this behavior, we suspect that there must be an undocumented modification or problem in the experiments that leads to Ahn et al.

Original: (2020) obtaining rather good results with the default conﬁguration.
Revised: (2020) obtaining rather good results with the default configuration.

Original: Overall, neither the original code with the original weights, nor the original code with newly trained weights, nor our reimplementation are performing as originally claimed.
Revised: Overall, neither the original code with the original weights, nor the original code with newly trained weights, nor our reimplementation are performing as originally claimed.

Original: In order to be as transparent as possible about our ﬁndings, we provide all of our code changes applied to the Intel repository as a patch and provide the entire source code of the re-implemented DGL-T REE S EARCH as well.
Revised: In order to be as transparent as possible about our findings, we provide all of our code changes applied to the Intel repository as a patch and provide the entire source code of the re-implemented DGL-T REE S EARCH as well.

Original:  Next, we analyze the replacement of the outputs of the GCN with random values.
Revised: Replacing the GCN with Random Values. Next, we analyze the replacement of the outputs of the GCN with random values.

Original: We start with the randomized default conﬁguration, i.e., no techniques like reduction are enabled.
Revised: We start with the randomized default configuration, i.e., no techniques like reduction are enabled.

Original: In this case, we see that for all small random graphs, the tree search is still able to ﬁnd solutions; however, the quality of these MIS seems to be slightly worse than the default results.
Revised: In this case, we find that for all small random graphs, the tree search is still able to find solutions; however, the quality of these MIS seems to be slightly worse than the default results.

Original: For 80 % of the larger random graphs – except for ER graphs – the randomized tree search is not able to ﬁnd solutions.
Revised: For 80 % of the larger random graphs – except for ER graphs – the randomized tree search is not able to find solutions.

Original: As the default conﬁguration is not able to ﬁnd any solution for SATLIB, it is to be expected that the random conﬁguration does not ﬁnd any solution either.
Revised: As the default configuration is not able to find any solution for SATLIB, it is to be expected that the random configuration does not find any solution either.

Original:  For the other real-world graphs, the reddit datasets are the only datasets where the default conﬁguration performed reasonably well.
Revised: Randomness for Other Real-World Graphs. For the other real-world graphs, the reddit datasets are the only datasets where the default configuration performed reasonably well.

Original: We ﬁnd that the randomness here in fact increases the performance of the algorithm. 
Revised: We find that the randomness here in fact increases the performance of the algorithm. Robustness.

Revised: Most ouf our datasets consist of multiple instances.

Revised: In order to shed light onto the distribution of both the solution size as well as the time until the solution was found, in Figure 1, we show violin plots for some data sets and tree search configurations.

Revised: Note in most real world scenarios, we do not know when we can terminate, hence the run time will always be the maximum time limit configured.

Original: Hence, the guidance by the GCN does not help our tree search algorithm; for some datasets, it is not able to generalize well (randomness beats the GCN), for others, there is no performance difference between random values and the GCN.
Revised: In these plots, we can for example again confirm that the combination of queue pruning and weighted popping enables us to find solutions quicker. We also see that as soon as reduction and local search are enabled, the solution distributions of the tree searches are very similar, no matter whether whether we query the GCN or use random values; for SATLIB, we see that using random values the distribution of the time to solution is even narrower towards 0, because no GPU computation is required. Final Considerations. We conclude that the guidance by the GCN does not help our tree search algorithm; for some datasets, it is not able to generalize well (randomness beats the GCN), for others, there is no performance difference between random values and the GCN.

Original: If we focus on the conﬁguration of the tree search that could be considered the production version 9 , i.e., with at least reduction and local search enabled, we ﬁnd that random outputs lead to identical performance, with very minor differences in the time until the ﬁnal solution discovery.
Revised: If we focus on the configuration of the tree search that could be considered the production version 10 , i.e., with at least reduction and local search enabled, we find that random outputs lead to identical performance, with very minor differences in the time until the final solution discovery.

Original: Overall, major contributing factors to solution quality are the reduction and local search by K A MIS, and in fact, the tree search algorithm is a “smarter brute-force” approach that does not gain any performance by being guided by a machine learning model.
Revised: Overall, major contributing factors to solution quality are the reduction and local search by K A MIS, and in fact, the tree search algorithm is a “smarter brute-force” approach that does not gain any performance by being guided by a machine learning model.

Original: Instead, the tree-search is almost like a classic branch-and-bound solver, and techniques such as data reduction and weighted queue popping, that narrow the search space, lead to the discovery of good solutions.
Revised: Instead, the search space is only narrowed by techniques such as data reduction and weighted queue popping, instead of good guidance by the ML model.

Original: Lastly, we brieﬂy note that the experimental results are inﬂuenced by randomness, as the queue popping happens randomly; this can for example be seen in the two cases, where a solution was discovered in the default conﬁguration of the DGL-T REE S EARCH , but no solutions for the conﬁguration with queue pruning enabled.

Original: 3. C OMPARING T REE S EARCH S OLVERS TO O THER S OLVERS
Revised: 3. C OMPARING T REE S EARCH S OLVERS TO O THER S OLVERS

Original: Having understood the implications of the various possible conﬁgurations of tree searches, next, we compare I NTEL -T REE S EARCH and DGL-T REE S EARCH to K A MIS, a heuristic solver optimized for the M AXIMUM I NDEPENDENT S ET problem, the mathematical optimization tool G UROBI , and the reinforcement learning tool L EARNING W HAT T O D EFER .
Revised: Having understood the implications of the various possible configurations of tree searches, next, we compare I NTEL -T REE S EARCH and DGL-T REE S EARCH to K A MIS, a heuristic solver optimized for the M AXIMUM I NDEPENDENT S ET problem, the mathematical optimization tool G UROBI , and the reinforcement learning tool L EARNING W HAT T O D EFER .

Original: For L W D, we train for 20 000 iterations of proximal policy optimization on the SATLIB dataset, using the hyperparameters given for SATLIB in Table 5 (Appendix A.1) of Ahn et al.
Revised: For L W D, we train for 20 000 iterations of proximal policy optimization on the SATLIB dataset, using the hyperparameters given for SATLIB in Table 5 (Appendix A.1) of Ahn et al.

Original: The results are given in  Table 2.
Revised: Due to space constraints, the detailed results are given in Appendix A in Table 3.

Original:  As we can see, the sophisticated state-of-the-art solver K A MIS can solve almost all instances perfectly; for example, for the large ER graphs, the multithreaded I NTELT REE S EARCH has a slightly higher average MIS, and a faster time until a solution is found.
Revised: K A MIS and G UROBI . As we can see, the sophisticated state-of-the-art solver K A MIS can solve almost all instances perfectly; for example, for the large ER graphs, the multithreaded I NTELT REE S EARCH has a slightly higher average MIS, and a faster time until a solution is found.

Original: For other graphs, especially the SATLIB dataset, K A MIS is very fast, while obtaining very high-quality results. 
Revised: For other graphs, especially the SATLIB dataset, K A MIS is very fast, while obtaining very high-quality results. These results are in line with Ahn et al.

Original:  Regarding the general-purpose solver G UROBI , we ﬁnd that it performs similar to K A MIS on simple instances, however, on harder datasets such as SATLIB or VC-BM, we see that Gurobi [Table]
Revised: (2020), who also observed K A MIS outperforming the tree search. Regarding the general-purpose solver G UROBI , we find that it performs similar to K A MIS on simple instances, however, on harder datasets such as SATLIB or VC-BM, we see that Gurobi takes significantly more time to find good solutions, and the average MIS is a little smaller (e.g., for large ER graphs, K A MIS achieves 44.57, compared to 37.79 for G UROBI ). Note that G UROBI and

Original: bitcoin [Table] takes signiﬁcantly more time to ﬁnd good solutions, and the average MIS is a little smaller (e.g., for large ER graphs, K A MIS achieves 44.57, compared to 37.79 for G UROBI ).
Revised: K A MIS have to rely on the CPU, while the tree search employs one one or more GPUs, unless it uses random values. Evening when assigning eight V100 GPUs to the multithreaded tree search, the computational advantage of the tree search does not help to outperform the other solvers, showing that the workload is not GPU-bound.

Original:  We analyze L W D ﬁrst in its default conﬁguration and then, similar to the tree search, replace the output of the graph neural network with a random tensor.
Revised: L W D. We analyze L W D first in its default configuration and then, similar to the tree search, replace the output of the graph neural network with a random tensor.

Original: First, L W D is very fast, even though it requires neural network inference.
Revised: First, L W D is very fast, even though it requires neural network inference.

Revised: For example, on the DIMACS data set, L W D finds solutions on average in just 4 seconds, while K A MIS takes 121 seconds.

Original: Quality-wise, except for the VC-BM dataset, it always ﬁnds near-optimal solutions.
Revised: Quality-wise, except for the VC-BM dataset, it always finds near-optimal solutions.

Original: Note that we did not use the local search or reduction techniques for L W D, which could further improve solution quality.
Revised: Note that we did not use the local search or reduction techniques for L W D, which could further improve solution quality.

Original: When using random output instead of the neural network, unlike the tree search algorithms, the solution quality degrades noticeably.
Revised: When using random output instead of the neural network, unlike the tree search algorithms, the solution quality degrades noticeably.

Original: These promising results show that L W D did  learn the solution structure of the MIS problem and is able to generalize over different datasets. 
Revised: These promising results show that L W D did not only learn the solution structure of the SATLIB instance family, but additionally is able to generalize over different datasets. Robustness.

Revised: Due to space constraints, we show violin plots of the results in Appendix A in Figure 2.

Revised: We clearly see the impact of randomness on L W D, and can visually compare the time difference between G UROBI and K A MIS, for example on VC-BM.

Original:  For tree search approaches, we see that state-of-the-art algorithmic techniques are required to ﬁnd good solutions.
Revised: Final Considerations. For tree search approaches, we see that state-of-the-art algorithmic techniques are required to find good solutions.

Original: As the fully-conﬁgured versions of the tree searches employ these K A MIS-internal routines, one can argue that the purely algorithmic solvers are the better choice, because the quality difference is negligible, while K A MIS is faster than the tree searches.
Revised: As the fully-configured versions of the tree searches employ these K A MIS-internal routines, one can argue that the purely algorithmic solvers are the better choice, because the quality difference is negligible, while K A MIS is faster than the tree searches.

Original: Purely algorithmic solvers do not come with the overhead of a machine learning environment (training as well as execution are more complicated), and the important algorithmic techniques need to be developed in any case.
Revised: Purely algorithmic solvers do not come with the overhead of a machine learning environment (training as well as execution are more complicated), and the important algorithmic techniques need to be developed in any case.

Original: However, the good results for L W D indicate that future work could focus on deepreinforcement learning solutions as well.
Revised: Our results are in line with observations by Joshi et al. (2021) who have compared algorithmic and classical solvers to deep learning solutions for TSP, and observe that DL-based solutions are often outperformed, especially on larger instances.

Original: 3. L ARGE -S CALE G RAPHS
Revised: 3. L ARGE -S CALE G RAPHS

Original: However, the question is whether these observations also hold true for large-scale graphs.
Revised: However, the question is whether these observations also hold true for large-scale graphs.

Original: To this end, we evaluate the solvers on random graph instances from 500 000 to 5 000 000 vertices, and on huge real-world graphs.
Revised: To this end, we evaluate the solvers on random graph instances from 500 000 to 5 000 000 vertices, and on huge real-world graphs.

Original: Detailed results can be found in Appendix A in Table 3.
Revised: Detailed results can be found in Appendix A in Table 4.

Original: Overall, we observe similar performance characteristics on large-scale graphs.
Revised: Overall, we observe similar performance characteristics on large-scale graphs.

Original: Only K A MIS and
Revised: Only K A MIS and

Original: G UROBI are able to ﬁnd solutions for all scenarios.
Revised: G UROBI are able to find solutions for all scenarios.

Original: K A MIS performs best, both with respect to solution quality and time required to ﬁnd these solutions; in many cases, it is more than one order of magnitude faster than other solvers.
Revised: K A MIS performs best, both with respect to solution quality and time required to find these solutions; in many cases, it is more than one order of magnitude faster than other solvers.

Original: Compared to the I NTEL -T
Revised: Compared to the I NTEL -T

Original: REE S EARCH , the DGLT REE S EARCH time outs more often due to VRAM limits; the I NTEL -T REE S EARCH beneﬁts from lower-level implementation using numpy arrays and sparse adjacency matrices, while the DGLT REE S EARCH employs the higher level Deep Graph Library graph abstraction, consuming more memory.
Revised: REE S EARCH , the DGLT REE S EARCH time outs more often due to VRAM limits; the I NTEL -T REE S EARCH benefits from lower-level implementation using numpy arrays and sparse adjacency matrices, while the DGLT REE S EARCH employs the higher level Deep Graph Library graph abstraction, consuming more memory.

Original: For L W D, we see good results for graphs up to 500 000 vertices, but cannot verify the scalability up to 2 000 000 vertices of the original paper 10 .
Revised: For L W D, we see good results for graphs up to 500 000 vertices, but cannot verify the scalability up to 2 000 000 vertices of the original paper 11 .

Original: 3. W EIGHTED G RAPHS
Revised: 3. W EIGHTED G RAPHS

Original: Having analyzed the solvers on unweighted graphs, we brieﬂy want to see how the solvers perform on weighted graphs.
Revised: Having analyzed the solvers on unweighted graphs, we briefly want to see how the solvers perform on weighted graphs.

Original: Intuitively, the weighted problem is harder, because each vertex can have a different value; hence specialized reduction techniques have to be developed.
Revised: Intuitively, the weighted problem is harder, because each vertex can have a different value; hence specialized reduction techniques have to be developed.

Original: Lamm et al.
Revised: Lamm et al.

Original: (2019) were the ﬁrst to present such rules also for the weighted case that are implemented in K A MIS, however, only integer weights are currently supported. 
Revised: (2019) were the first to present such rules also for the weighted case that are implemented in K A MIS, however, only integer weights are currently supported. We test weighted random graphs and three Amazon MWIS data sets.

Original: The results and details on the weighted graph generation are given in Appendix A in Table 4; note that
Revised: The results and details on the weighted graph generation are given in Appendix A in Table 5; note that

Original: I NTEL -T
Revised: I NTEL -T

Original: REE S EARCH does not support the weighted case, and while Ahn et al.
Revised: REE S EARCH does not support the weighted case, and while Ahn et al.

Original: (2020) evaluate L W D for MWIS, they do not provide the code to do so.
Revised: (2020) evaluate L W D for MWIS, they do not provide the code to do so.

Original: We observe that on HK, WS, and HRG graphs, where K A MIS is able to utilize its reduction techniques, it is able to instantaneously solve the weighted problem as well.
Revised: We observe that on HK, WS, and HRG graphs, where K A MIS is able to utilize its reduction techniques, it is able to instantaneously solve the weighted problem as well.

Original: However, on ER graphs, the reduction times out, showing that the suitability of the algorithmic techniques is graph-dependent.
Revised: However, on ER graphs, the reduction times out, showing that the suitability of the algorithmic techniques is graph-dependent.

Original: In the weighted case on ER graphs, the DGL-T REE S EARCH with queue pruning enabled ﬁnds the best result.
Revised: In the weighted case on ER graphs, the DGL-T REE S EARCH with queue pruning enabled finds the best result.

Original: G UROBI ﬁnds solutions of similar quality to K A MIS very quickly.
Revised: G UROBI finds solutions of similar quality to K A MIS very quickly.

Original: Overall, we see that the vanilla tree search has the advantage of being problem-agnostic, i.e., an extension to the weighted case was easily possible, while for the algorithmic solver, new reductions are needed, that currently cannot deal with some graphs.

Original: However, on graphs where the reductions are successful, K A MIS ﬁnds solutions of the highest quality, showing the trade-off between general and specialized solvers.  R ELATED W ORK In this section, we brieﬂy discuss some related work. Solvers for Maximum Independent Set.
Revised: For the Amazon data sets, all solvers except G UROBI reach their limits.

Original: Li et al. (2018) propose the I NTEL -T REE S EARCH , a supervised framework that determines an MIS over several steps while being guided in its choice of vertices by a GCN. As an alternative unsupervised approach, Khalil et al. (2017) propose S2V-DQN, in which they use Q-learning to solve the minimum vertex cover problem 11 .
Revised: K A MIS does not support the large node weights that are used in the Amazon instances and hence is not able to solve them.

Original: Building on their work,

Original: Ahn et al.

Original: (2020) propose L EARNING WHAT TO D EFER (LwD), a reinforcement learning approach with a focus on scalability on large graphs. Similar to the tree search, LwD can mark multiple vertices as part of the MIS in a single step, which makes it faster compared to S2V-DQN, that only labels a single vertex in each step. A solver that does not utilize machine learning is K A MIS (Lamm et al., 2017; 2019; Hespe et al., 2019); we mention the recent publication by Hespe et al.
Revised: The DGL-T REE S EARCH goes out of memory and crashes while solving these instances.

Original: (2021) introducing some new reduction rules. Note that all of these solvers are heuristics and hence only solve MIS approximately; exact solvers have been proposed by Jain & Seshadhri (2020); Xiao & Nagamochi (2017); Tomita et al. (2010), for example, and theory has been working on understanding why and in what models MIS poses to be difﬁcult (Censor-Hillel et al., 2017). Solvers for Other NP-hard Problems.
Revised: Overall, we see that both G UROBI as well as the vanilla tree search have the advantage of being problem-agnostic, i.e., an extension to the weighted case was easily possible, while for the algorithmic solver, new reductions are needed, that currently cannot deal with some graphs.

Original: Of course, other NP-hard problems that, unlike minimum vertex cover, are not closely related to maximum independent set, have also been subject to intensive research. The evolutionary computation community, for example, has researched various NPhard routing problems, like the V EHICLE R OUTING P ROBLEM (Berger & Barkaoui, 2003; Potvin, 2009) and the M ULTIPLE R OUTES problem (B ¨ other et al., 2021). Another routing problem, the T RAVELLING S ALESMAN P ROBLEM (TSP), has already been tackled using neural networks in the 1980s (Hopﬁeld & Tank, 1985).
Revised: On smaller random graphs where the reductions are successful, K A MIS finds solutions of the highest quality, showing the trade-off between general and specialized solvers.

Original: There have been two notable approaches by Bello et al.

Original: (2017) and Kool et al.

Original: (2019) that propose neural network-based solutions that  are tailored towards the TSP. This means, that unlike the tree search approach, their solutions focus more on the problem at hand and cannot easily be generalized to other NP-hard problems. Last, we note the recent work by Nair et al. (2021) which aims at solving Mixed Integer Programs using Deep Learning.
Revised: G UROBI is the only solver that is able to solve large, weighted, real-world MWIS instances, and shows that industry-standard optimizers are very robust towards different inputs of different sizes, while smaller algorithmic solvers need more time to reach that level of robustness.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: We present our comprehensive, open-source benchmark environment for the M AXIMUM (W EIGHTED )
Revised: We present our comprehensive, open-source benchmark environment for the M AXIMUM (W EIGHTED )

Original: I NDEPENDENT S ET problem.
Revised: I NDEPENDENT S ET problem.

Original: Using this environment, we run several experiments on both real-world and synthetic graphs of different sizes.
Revised: Using this environment, we run several experiments on both real-world and synthetic graphs of different sizes.

Original: Our analysis shows that guided tree searches, such as the I NTEL -T REE S EARCH by Li et al.
Revised: Our analysis shows that guided tree searches, such as the I NTEL -T REE S EARCH by Li et al.

Original: (2018), owe their good results not to the trained neural network, but instead to the various techniques used to make a “better” brute-force algorithm.
Revised: (2018), owe their good results not to the trained neural network, but instead to the various techniques used to make a “better” brute-force algorithm.

Original: To verify this, we show that the GCN that guides the search can be replaced by random values without a noticeable performance impact.
Revised: To verify this, we show that the GCN that guides the search can be replaced by random values without a noticeable performance impact.

Original: Furthermore, we are not able to reproduce the results of previous work in the default conﬁguration of the algorithm and claim that without algorithmic techniques, the tree search algorithms are not able to solve hard MIS instances.
Revised: Furthermore, we are not able to reproduce the results of previous work in the default configuration of the algorithm and claim that without algorithmic techniques, the tree search algorithms are not able to solve hard MIS instances.

Original: We believe this to be an important insight for the community researching at the intersection of combinatorial optimization and machine learning.
Revised: We believe our results to be an important insight for the community researching at the intersection of combinatorial optimization and machine learning.

Original: The benchmark suite lays the ground fur future reproducible evaluations for new MIS solvers, and the promising results for L EARNING W HAT T O D EFER indicate that reinforcement learning is superior to supervised approaches.
Revised: The benchmark suite lays the ground fur future reproducible evaluations for new MIS solvers, and the promising results for L EARNING W HAT T O D EFER indicate that reinforcement learning is superior to supervised approaches.

Original: This might be kept in mind when developing solving techniques using machine learning in the future.
Revised: This might be kept in mind when developing solving techniques using machine learning in the future.

Revised: For future work, further variants of the M AXIMUM

Revised: I NDEPENDENT S ET problem, such as the G ENERALIZED I NDEPENDENT S ET problem (Colombi et al., 2017; Hosseinian & Butenko, 2019), and other problems, such as TSP, should be considered, to further understand for what kind of problem what solver architecture should be used.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: We adhere to the ICLR Code of Ethics and declare that we have no conﬂicts of interest.
Revised: We adhere to the ICLR Code of Ethics and declare that we have no conflicts of interest.

Original: We tried to contact the authors of the related work, as stated in the respective sections.
Revised: We tried to contact the authors of the related work, as stated in the respective sections.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: As our results contradict previous work, we try to be as transparent as possible about our process.
Revised: As our results contradict previous work we try to be as transparent as possible about our process.

Original: The solvers we use are documented in Section 2.
Revised: The solvers we use are documented in Section 2.

Original: The changes we apply to L W D and I NTEL - T REE S EARCH are supplied as a git patch with our benchmark suite that can be accessed via the anonymous repository 12 or in the supplementary material in OpenReview.
Revised: The changes we apply to L W D and I NTELT REE S EARCH are supplied as a git patch with our benchmark suite that can be accessed via the anonymous repository 12 or in the supplementary material in OpenReview.

Original: The datasets we use are listed in Appendix D; we supply the code required to preprocess the datasets.
Revised: The datasets we use are listed in Appendix E; we supply the code required to preprocess the datasets

Original:  T ABLES 
Revised: A A DDITIONAL T ABLES AND P LOTS

Original: In this section, you can ﬁnd the tables  omitted in the main paper due to space constraints.
Revised: In this section you can find the tables and plots omitted in the main paper due to space constraints.

Revised: for detailed explanations of the techniques used.

Revised: We mark the configuration/solver that has the best average MIS in bold, and grey out configurations for which solutions were found for less than 20 % of all graphs.

Original: [Table]
Revised: [Figure]

Revised: +r +

Revised: [Figure]

Revised: +

Revised: [Figure]

Revised: +

Revised: [Figure]

Revised: +

Revised: [Figure]

Revised: +

Revised: [Figure]

Revised: +r +

Revised: [Figure]

Revised: r

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Original: [Table]
Revised: [Table]

Revised: bitcoin-alpha

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: berlin

Revised: [Figure]

Revised: All multithreaded experiments in this table use 8 threads, and all threads share a single GPU.

Revised: Time limits are set equally to Table 2.

Revised: We note that K A MIS and L W D always run single-threadedly, while

Revised: G UROBI employs up to 8 threads, as needed.

Revised: For L W D, we test the default configuration ( d ) and replace the output of the GNN with a random tensor ( rand ); the random graphs are executed with per episode and all other graphs with

Revised: We mark the configuration/solver that has the best average MIS in bold, and grey out configurations for which solutions were found for less than 20 % of graphs

Revised: [Table]

Revised: roadnet

Revised: [Table]

Revised: berlin

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: ( a ) I n t e l ( b ) I n t e l r + l s ( c ) D

Revised: G L ( d ) D

Revised: G L r + l s + q p + w p + r a nd ( e ) L w

Revised: D ( f ) L w

Revised: D r a nd ( g ) K a

Revised: M I S ( h ) G u r o b i

Revised: Intel

Revised: DGL

Revised: LwD

Revised: KaMIS

Revised: Gurobi

Revised: [Table]

Revised: The weights on the random garphs have been sampled per vertex from a normal distribution N with µ = 100 , σ = 30 , capped to be larger than 0, and rounded to integer values.

Revised: For the explanation of the various configuration flags, we refer to the caption of Table 2.

Revised: All multithreaded experiments in this table use 8 threads and all threads share a single GPU.

Revised: The time limit for all graphs was set to has the best were found

Revised: [Table]

Revised: [Table]

Original: B T REE S EARCH
Revised: OOM OOM OOM OOM

Revised: OOM

Revised: [Table]

Revised: OOM

Revised: [Table]

Revised: [Table]

Revised: OOM

Revised: OOM

Revised: OOM

Revised: OOM

Revised: OOM

Revised: [Table]

Original: In the following, we describe the guided tree search approach introduced by Li et al.
Revised: OOM OOM OOM OOM

Original: (2018) in more detail, and explain our additional modiﬁcations (e.g., the queue pruning). A pseudocode description of the algorithm is found in Algorithm 1; for a visualization, we refer to Figure 1 of Li et al. The core element of the tree search is a queue 13 P of partial solutions S ∈ { 0 , 1 , ? } | V | to the MIS problem, i.e., labelings of the graph at hand marking each vertex as either included in the MIS (1), excluded (0), or unlabeled (a decision about that vertex is still to be made, ?).
Revised: OOM OOM

Original: To be a valid (partial) solution to the MIS problem, each element in the queue fulﬁlls the constraint that no two adjacent vertices can both be included.
Revised: OOM

Original: Furthermore, all vertices adjacent to an included one must be excluded .
Revised: OOM notsupported 706635840.75(-)

Original: Given a graph G = ( V, E ) , we start with an empty solution, i.e., f.a. v ∈ V , S v = ? .
Revised: B T REE S EARCH

Original: Ineach step of the tree search, a partial labeling is popped from the queue ( random pop ), and the residual graph
Revised: In the following, we describe some additional details on the guided tree search approach introduced by Li et al.

Original: G residual consisting only of the unlabeled vertices is constructed (keeping only those edges between unlabeled vertices). Next, we obtain a predeﬁned number m of probability maps , each of which contains a value for all vertices of the residual graph, posing the “probability of being in the MIS”.
Revised: A textual description can be found in Section 2, and a pseudocode description in Algorithm 1; for a visualization, we refer to Figure 1 of Li et al.

Original: This is done using a function gen probmaps : G → [0 , 1] | V |× m , that can take arbitrary unweighted graphs G ∈ G as input.
Revised: The pseudocode makes use of a function gen probmaps : G → [0 , 1] | V |× m , that can take arbitrary unweighted graphs G ∈ G as input.

Original: In the default case, this function calls the trained GCN, which outputs its assignments from vertices to probabilities, i.e., GCN ( G ) ∈ [0 , 1] | V |× m .From each of these probability maps, a new partial solution is derived as follows: The probabilities of the maps are sorted in descending order. The vertex with the highest probability gets labeled as included , and all adjacent vertices as excluded .
Revised: In the default case, this function calls the trained GCN, which outputs its assignments from vertices to probabilities, i.e., GCN ( G ) ∈ [0 , 1] | V |× m .

Original: This step is repeated until we would have to label an already excluded vertex as included , in which case we break, and add the partial solution to the queue.

Original: In case all vertices are labeled we obtained a full solution, which we yield and do not add back into the queue.

Original: This procedure is repeated for all probability maps.

Original: A non-pseudocode implementation – like DGL-T REE S EARCH –could, for example, choose the largest independent set that is yielded to obtain a potential MIS.
Revised: We obtain the final solution in a non-pseudocode implementation – like DGL-T REE S EARCH – for example by choosing the largest independent set that is yielded.

Original: Furthermore, there are several modiﬁcations (ﬂags) discussed in Section 3.1.
Revised: Furthermore, there are several modifications (flags) discussed in Section 3.1.

Original: In the following, we brieﬂy discuss how these ﬂags impact the tree search algorithm.
Revised: In the following, we briefly discuss how these flags impact the tree search algorithm.

Original: • Reduction.
Revised: • Reduction.

Original: Reduction or graph kernelization is an addition that uses algorithmic techniques to ﬁnd vertices which (provably) have to be in the independent set.
Revised: Reduction or graph kernelization is an addition that uses algorithmic techniques to find vertices which (provably) have to be in the independent set.

Original: After determining the residual graph, before generating the probability maps, we optionally can determine whether some vertices can be labeled due to the reduction rules, and then continue to determine the probability maps with an even smaller residual graph.
Revised: After determining the residual graph, before generating the probability maps, we optionally can determine whether some vertices can be labeled due to the reduction rules, and then continue to determine the probability maps with an even smaller residual graph.

Original: Lamm et al.
Revised: Lamm et al.

Original: (2017) and Hespe et al.
Revised: (2017) and Hespe et al.

Original: (2019) provide reduction rules for the unweighted case, which both I NTELT REE S EARCH and DGL-T REE S EARCH access using a Python-C++ interface.
Revised: (2019) provide reduction rules for the unweighted case, which both I NTELT REE S EARCH and DGL-T REE S EARCH access using a Python-C++ interface.

Original: Furthermore, for the M AXIMUM W EIGHTED I NDEPENDENT S ET problem, DGL-T REE S EARCH uses the recently proposed reduction rules by Lamm et al.
Revised: Furthermore, for the M AXIMUM W EIGHTED I NDEPENDENT S ET problem, DGL-T REE S EARCH uses the recently proposed reduction rules by Lamm et al.

Original: • Local Search.
Revised: • Local Search.

Original: A local search is an addition based on small mutations which try to further improve a full solution before it is yielded.
Revised: A local search is an addition based on small mutations which try to further improve a full solution before it is yielded.

Original: It can be understood as ﬁne-tuning a solution.
Revised: It can be understood as fine-tuning a solution.

Original: The I NTEL -T REE S EARCH and DGL-T REE S EARCH use the local search implementation by Lamm et al.
Revised: The I NTEL -T REE S EARCH and DGL-T REE S EARCH use the local search implementation by Lamm et al.

Original: (2017) and Hespe et al.
Revised: (2017) and Hespe et al.

Original: (2019), and
Revised: (2019), and

Original: for the weighted case, DGL-T REE S EARCH uses the local search by Lamm et al.
Revised: for the weighted case, DGL-T REE S EARCH uses the local search by Lamm et al.

Original: • Multithreading.
Revised: • Multithreading.

Original: To ﬁnd the MIS faster, one can use multiple threads that search for independent sets.
Revised: To find the MIS faster, one can use multiple threads that search for independent sets.

Original: The parallelization idea is very simple, each thread just follows the same procedure as stated in Algorithm 1.
Revised: The parallelization idea is very simple, each thread just follows the same procedure as stated in Algorithm 1.

Original: In the I NTEL -T
Revised: In the I NTEL -T

Original: REE S EARCH , the threads share the queue P , making push and pop operations a critical section.
Revised: REE S EARCH , the threads share the queue P , making push and pop operations a critical section.

Original: Because we want to avoid this critical section, and due to issues with shared state in Python multiprocessing, in the DGL-T REE S EARCH, we give each thread its own queue, and ﬁrst initialize enough partial solutions, such that each thread has one partial solution to start on.
Revised: Because we want to avoid this critical section, and due to issues with shared state in Python multiprocessing, in the DGL-T REE S EARCH we give each thread its own queue, and first initialize enough partial solutions, such that each thread has one partial solution to start on.

Original: • Queue Pruning.
Revised: • Queue Pruning.

Original: As we see in Section 3.1, on difﬁcult datasets like SATLIB, the tree search almost cannot ﬁnd any solution.
Revised: As we see in Section 3.1, on difficult datasets like SATLIB, the tree search almost cannot find any solution.

Original: Furthermore, in their experiments, Ahn et al.
Revised: Furthermore, in their experiments, Ahn et al.

Original: (2020) state that the I NTEL -T
Revised: (2020) state that the I NTEL -T

Original: REE S EARCH runs out of memory, something that we did not observe for medium-sized graphs.
Revised: REE S EARCH runs out of memory, something that we did not observe for medium-sized graphs.

Original: Queue pruning is a technique used to tackle these two issues.
Revised: Queue pruning is a technique used to tackle these two issues.

Original: Unfortunately, we could not obtain the information on how the queue pruning approach by Ahn et al.
Revised: Unfortunately, we could not obtain the information on how the queue pruning approach by Ahn et al.

Original: (2020) is implemented via private communication, hence we propose our own solution.
Revised: (2020) is implemented via private communication, hence we propose our own solution.

Original: We set a maximum number of elements on our queue P , and after appending a new partial solution to the queue, we remove the oldest elements (at position 0) from the queue, until the queue is small enough again.
Revised: We set a maximum number of elements on our queue P , and after appending a new partial solution to the queue, we remove the oldest elements (at position 0) from the queue, until the queue is small enough again.

Original: The intuition is that we ﬁnd more solutions faster, because there is a higher chance to pop an almost labeled (more recent) solution from the queue, and we also limit the memory used.
Revised: The intuition is that we find more solutions faster, because there is a higher chance to pop an almost labeled (more recent) solution from the queue, and we also limit the memory used.

Original: Queue pruning was not proposed in the original paper by Li et al.
Revised: Queue pruning was not proposed in the original paper by Li et al.

Original: • Weighted Queue Pop.
Revised: • Weighted Queue Pop.

Original: By default, the random pop used in Algorithm 1 chooses an element from the queue uniformly at random.
Revised: By default, the random pop used in Algorithm 1 chooses an element from the queue uniformly at random.

Original: Weighted Queue Pop, on the other hand, shifts the probability of each element to be chosen according to how many unlabeled vertices are left in it, favoring fewer unlabeled vertices.
Revised: Weighted Queue Pop, on the other hand, shifts the probability of each element to be chosen according to how many unlabeled vertices are left in it, favoring fewer unlabeled vertices.

Original: Similar to queue pruning, the intuition is to ﬁnd fully labeled solutions faster while keeping some randomness in the popping behavior.
Revised: Similar to queue pruning, the intuition is to find fully labeled solutions faster while keeping some randomness in the popping behavior.

Original: It can be freely combined with queue pruning or used on its own.
Revised: It can be freely combined with queue pruning or used on its own.

Original: To the best of our knowledge, no other paper has proposed this addition to the tree search algorithm yet.
Revised: To the best of our knowledge, no other paper has proposed this addition to the tree search algorithm yet.

Original: • Random Values as Probability Maps.
Revised: • Random Values as Probability Maps.

Original: This ﬂag changes gen probmaps to a random generator, effectively replacing the GCN output with probability values chosen uniformly at random.
Revised: This flag changes gen probmaps to a random generator, effectively replacing the GCN output with probability values chosen uniformly at random.

Original: We note that the original implementation by Li et al.
Revised: We note that the original implementation by Li et al.

Original: (2018) does not exactly follow our description and Algorithm 1.
Revised: (2018) does not exactly follow our description and Algorithm 1.

Original: We note differences between their paper and the published code in the next section.
Revised: We note differences between their paper and the published code in the next section.

Original: C R EMARKS ON THE I NTEL -T REE S EARCH I MPLEMENTATION
Revised: C R EMARKS ON THE I NTEL -T REE S EARCH I MPLEMENTATION

Original: In this section, we note some important facts to keep in mind when dealing with the original tree search implementation, next to the rather difﬁcult understandability and maintainability of the code.
Revised: In this section, we note some important facts to keep in mind when dealing with the original tree search implementation, next to the rather difficult understandability and maintainability of the code.

Original: First, for difﬁcult instances, with reduction and local search both disabled, the search queue appears to grow indeﬁnitely without ever ﬁnding a single solution.
Revised: First, for difficult instances, with reduction and local search both disabled, the search queue appears to grow indefinitely without ever finding a single solution.

Original: To circumvent this effect and to be able to ﬁnd solutions, it might be necessary to limit the search queue to a ﬁxed maximum size, as observed by Ahn et al.
Revised: To circumvent this effect and to be able to find solutions, it might be necessary to limit the search queue to a fixed maximum size, as observed by Ahn et al.

Original: They furthermore stress the importance of pruning to limit memory usage.
Revised: They furthermore stress the importance of pruning to limit memory usage.

Original: We discuss our approach to queue pruning in Appendix B.
Revised: We discuss our approach to queue pruning in Appendix B.

Original: Second, in the paper, it is stated that the GCN is trained for 200 epochs.
Revised: Second, in the paper, it is stated that the GCN is trained for 200 epochs.

Original: However, in the implementation, one epoch iterates over just 2000 randomly chosen samples of the 38000 samples in the training set 14 .
Revised: However, in the implementation, one epoch iterates over just 2000 randomly chosen samples of the 38000 samples in the training set 13 .

Original: This is not what you would expect given the term “epoch”.
Revised: This is not what you would expect given the term “epoch”.

Original: Furthermore, the original code has some hardcoded values (e.g., sometimes it expects the number of input graphs to be ﬂexible, and sometimes it is hardcoded within the same ﬁle).
Revised: Furthermore, the original code has some hardcoded values (e.g., sometimes it expects the number of input graphs to be flexible, and sometimes it is hardcoded within the same file).

Original: Our patch provided in the benchmark suite ﬁxes these issues.
Revised: Our patch provided in the benchmark suite fixes these issues.

Original: Third, the multi-threaded variant of the algorithm ( demo parallel.py ) uses just one randomly chosen probability map of the GCN outputs instead of all.
Revised: Third, the multi-threaded variant of the algorithm ( demo parallel.py ) uses just one randomly chosen probability map of the GCN outputs instead of all.

Original: This makes this variant’s performance in terms of search steps per second, appear to scale superlinearly, and furthermore is not documented in the paper 15 , and needs to be kept in mind for evaluation. 
Revised: This makes this variant’s performance in terms of search steps per second, appear to scale superlinearly, and furthermore is not documented in the paper 14 , and needs to be kept in mind for evaluation. D D ETAILS ON USING G UROBI AS A M(W)IS S OLVER

Revised: Gurobi is a general purpose, off-the-shelf, mathematical optimization tool.

Revised: Hence, many parameters can be tuned, and the performance can very on how the problem to be solved is inputted into the solver.

Revised: In the following two subsections, we investigate the impact of these two dimensions.

Original: D D ATASETS
Revised: D. G UROBI T UNING

Revised: Similar to training machine learning models, we can try to optimize Gurobi’s performance for the problem at hand.

Revised: This is called hyperparameter tuning or algorithm configuration , and often done using heuristic searches, e.g., by evolutionary algorithms.

Revised: One well known optimizer for this is

Revised: SMAC (Lindauer & Hutter, 2018); however, in this subsection, we use grbtune 15 , as it is the native tool supported by Gurobi for parameter tuning.

Revised: Our methodology is as follows.

Revised: We generate a Gurobi configuration based on 10 instances of the

Revised: VC-BM dataset.

Revised: We choose this data set because for most other datasets like SATLIB, grbtune states that it is not able to improve over the default parameters.

Revised: We note that grbtune decides when it is done, and we do not impose a time limit on the tuning process.

Revised: The resulting found parameters are Heuristics = 0.001, PumpPasses = 0, and ZeroObjNodes =

Revised: We then run Gurobi both with the default configuration and the tuned configuration on various data sets, and split the VC-BM data set into the training and test sets.

Revised: The results can be seen in Table 6.

Revised: We find that tuning does not improve the results, and even on the training set leads to worse performance, although the optimization was performed on exactly those instances.

Revised: Only for the DIMACS data set, the tuned Gurobi finds the solution faster than the default

Revised: Gurobi configuration.

Revised: As Gurobi states 16 , “The bottom line is that automated performance tuning is meant to give suggestions for parameters that could produce consistent, reliable improvements on your models.

Revised: It is not meant to be a replacement for efficient modeling or careful performance testing.”

Revised: One possible explanation might be that grbtune might be more effective on more complex linear programs.

Revised: Additionally, on most data sets, it even states it cannot improve the default parameters.

Revised: Our html

Revised: (c.f. Appendix D.1)

Revised: For the explanation of the various configuration flags, we refer to the caption of Table 2.

Revised: Time limits are set equally to Table

Revised: G UROBI employs up to 8 threads, as needed

Revised: We

Revised: [Table]

Revised: experiments show that at least for the linear MIS program, the default configuration of a commercial solver like Gurobi are sufficient.

Revised: This may differ for other problems and hyperparameter tuning should always be considered

Revised: [Table]

Revised: but we do not find any benefit on our data sets

Revised: [Table]

Revised: D. A LTERNATIVE MIS F ORMULATION

Revised: The configuration of the solver is one way we can impact performance.

Revised: Another way is reformulating the mathematical optimization task into something that the solver can solver more efficiently.

Revised: Until now, we have used a linear formulation of the MWIS problem, i.e., arg max S ∈ IS ( G ) (cid:80)

Revised: u ∈ S w u .

Revised: Literature has discussed different formulations of the MIS problem (Butenko, 2003).

Revised: We want to see whether using a different program makes a difference on our benchmark datasets.

Revised: To this end, we use a quadratic formulation of the MIS problem (Pardalos & Rodgers, 1992).

Revised: Following the notation of Butenko (2003), let G be a graph with n vertices, A G be its adjacency matrix, and J the n × n identity matrix.

Revised: Let A =

Revised: J − A G .

Revised: Then, we can calculate the MIS as

Revised: [Equation]

Revised: which is a quadratic program instead of a linear one, that however might be easier to solve in practice.

Revised: To understand the impact of this formulation, we run Gurobi with this quadratic program instead of the linear one again on our data sets.

Revised: The results can be seen in Table 7.

Revised: We find that, with the exception of the DIMACS data set, the linear formulation always performs equally or better, with both respect to solution size and run time.

Revised: On DIMACS, the quadratic formulation performs maginally better (74.35 average MIS vs 74.62 average MIS), but also requires more time (202 seconds vsseconds).

Revised: On all other graphs, especially the hard benchmark instances, the linear formulation is faster, for example for VC-BM, where the linear program requires 269 seconds, and the quadratic program 300 seconds.

Revised: For the explanation of the various configuration flags we refer to the caption of Table 2.

Revised: All multithreaded experiments in this table use 8 threads and all threads share a single GPU.

Revised: Time limits are set equally to Table 2.

Revised: We note that K A MIS and L W D always run single-threadedly, while

Revised: G UROBI employs up to 8 threads, as needed.

Revised: For L W D, the random graphs are executed with per and all other graphs with

Revised: We mark the configuration solver has the best

Revised: MIS and out for which solutions were found

Revised: [Table]

Revised: wiki

Revised: [Table]

Revised: Vote

Revised: [Table]

Revised: [Table]

Revised: Cora

Revised: [Table]

Revised: [Table]

Revised: Citeseer -(-) 1808.

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: bitcoin-alpha -(-)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: bitcoin-otc -

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ego-facebook -(-)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: roadnet-berlin -(-)

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original:  In this section, we give some details on the datasets, groups of datasets, and random graphs models used in this paper.
Revised: [Table] [Table] E D ATASETS In this section, we give some details on the datasets, groups of datasets, and random graphs models used in this paper.

Original: For all datasets, we treat all graphs as undirected.
Revised: For all datasets, we treat all graphs as undirected.

Original: The data generation module of our benchmarking suite integrates the download/generation and conversion of these graphs.
Revised: The data generation module of our benchmarking suite integrates the download/generation and conversion of these graphs.

Original: We delete all self-loops, if there are any, because the solvers deal with self-loops differently (for example, with silent failures, errors, or just ignoring the vertex).
Revised: We delete all self-loops, if there are any, because the solvers deal with self-loops differently (for example, with silent failures, errors, or just ignoring the vertex).

Original: Erd˝os -R´enyi.
Revised: Erd˝os-R´enyi.

Original: This well-known random graph model by Erd˝os & R´enyi (1960) connects each pair of vertices with probability p .
Revised: This well-known random graph model by Erd˝os & R´enyi (1960) connects each pair of vertices with probability p .

Original: We use p = 0 .
Revised: We use p = 0 .

Original: 15 and for testing, generate 500 graphs with 50 tovertices and 100 graphs with 700 to 800 vertices.
Revised: 15 and for testing, generate 500 graphs with 50 tovertices and 100 graphs with 700 to 800 vertices.

Original: Barab´asi -Albert.
Revised: Barab´asi-Albert.

Original: This random graph model by Albert & Barab ´ asi (2002) iteratively adds nodes, connecting them to m already existing nodes.
Revised: This random graph model by Albert & Barab´asi (2002) iteratively adds nodes, connecting them to m already existing nodes.

Original: We use m = 2 and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs with 700 to 800 vertices.
Revised: We use m = 2 and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs with 700 to 800 vertices.

Original: Holme-Kim.
Revised: Holme-Kim.

Original: A random graph model by Holme & Kim (2002) similar to the BA-model, with an extra step for each randomly created edge that creates a triangle with probability p .
Revised: A random graph model by Holme & Kim (2002) similar to the BA-model, with an extra step for each randomly created edge that creates a triangle with probability p .

Original: We use m =and p = 0 .
Revised: We use m =and p = 0 .

Original: and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs withto 800 vertices.
Revised: and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs withto 800 vertices.

Original: For the large-scale experiments, we test on one graph per ﬁxed amount of vertices, c.f.
Revised: For the large-scale experiments, we test on one graph per fixed amount of vertices, c.f.

Original: Table 3.
Revised: Table 4.

Original: Watts-Strogatz.
Revised: Watts-Strogatz.

Original: A random graph model by Watts & Strogatz (1998) that starts with a wellstructured ring-lattice with mean degree k and in the following step replaces each edge with probability p with another edge sampled uniformly at random.
Revised: A random graph model by Watts & Strogatz (1998) that starts with a wellstructured ring-lattice with mean degree k and in the following step replaces each edge with probability p with another edge sampled uniformly at random.

Original: This way it tries to keep “small-world properties” while maintaining a random structure similar to Erd ˝ os-R ´ enyi graphs.
Revised: This way it tries to keep “small-world properties” while maintaining a random structure similar to Erd ˝ os-R ´ enyi graphs.

Original: We use k = 2 and p = 0 . 15 .
Revised: We use k = 2 and p = 0 . 15 .

Original: For the large-scale experiments, we test on one graph per ﬁxed amount of vertices, c.f.
Revised: For the large-scale experiments, we test on one graph per fixed amount of vertices, c.f.

Original: Table 3.
Revised: Table 4.

Original: Hyperbolic Random Graph.
Revised: Hyperbolic Random Graph.

Original: A random graph model by Krioukov et al.
Revised: A random graph model by Krioukov et al.

Original: (2010), which generates graphs by randomly putting nodes in a disk in the hyperbolic plane, and connecting them if their (hyperbolic) distance is below a certain threshold.
Revised: (2010), which generates graphs by randomly putting nodes in a disk in the hyperbolic plane, and connecting them if their (hyperbolic) distance is below a certain threshold.

Original: Recent works have it described to be the best model for modeling real-world networks, due to their heterogeneity and interdependency (Friedrich, 2019).
Revised: Recent works have it described to be the best model for modeling real-world networks, due to their heterogeneity and interdependency (Friedrich, 2019).

Original: As generation parameters, we use α = 0 .
Revised: As generation parameters, we use α = 0 .

Original: 75 , T = 0 .
Revised: 75 , T = 0 .

Original: 1 , deg = 10 and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs with 700 to 800 vertices.
Revised: 1 , deg = 10 and for testing, generate 500 graphs with 50 to 100 vertices and 100 graphs with 700 to 800 vertices.

Original: For the large-scale experiments, we test on one graph per ﬁxed amount of vertices, c.f.
Revised: For the large-scale experiments, we test on one graph per fixed amount of vertices, c.f.

Original: Table 3.
Revised: Table 4.

Original: SATLIB.
Revised: SATLIB.

Original: This is a well-known set of SAT instances in CNF commonly used as a benchmark for
Revised: This is a well-known set of SAT instances in CNF commonly used as a benchmark for

Original: SAT solvers (Hoos & St¨utzle, 2000).
Revised: SAT solvers (Hoos & St¨utzle, 2000).

Original: A SAT instance can easily be reduced to a graph, which has a MIS as large as the number of clauses, should the SAT instance be satisﬁable.
Revised: A SAT instance can easily be reduced to a graph, which has a MIS as large as the number of clauses, should the SAT instance be satisfiable.

Original: In short, for each clause a clique is created, each node of which stands for a variable or its negation, e.g., x or ¬ x .
Revised: In short, for each clause a clique is created, each node of which stands for a variable or its negation, e.g., x or ¬ x .

Original: Afterwards, for each variable x , all its nodes are connected to nodes referring to its negation ¬ x .
Revised: Afterwards, for each variable x , all its nodes are connected to nodes referring to its negation ¬ x .

Original: We use the “Random-3-SAT Instances with Controlled Backbone Size” dataset 16 , which consists of 40 000 instances, of which we train on 39 500 and test on 500.
Revised: We use the “Random-3-SAT Instances with Controlled Backbone Size” dataset 17 , which consists of 40 000 instances, of which we train on 39 500 and test on 500.

Original: Each instance has between 403 toclauses.
Revised: Each instance has between 403 toclauses.

Original: PPI.
Revised: PPI.

Original: The PPI dataset, introduced by Hamilton et al.
Revised: The PPI dataset, introduced by Hamilton et al.

Original: (2017), contains of 24 graphs whose nodes describe proteins and edges describe interactions between them.
Revised: (2017), contains of 24 graphs whose nodes describe proteins and edges describe interactions between them.

Original: They contain 591 to 3 480 nodes.
Revised: They contain 591 to 3 480 nodes.

Original: We use the data set provided by DGL 17 , as it provides a more easily understandable separation between the graphs than the original source by GraphSAGE 18 .
Revised: We use the data set provided by DGL 18 , as it provides a more easily understandable separation between the graphs than the original source by GraphSAGE 19 .

Original: REDDIT.
Revised: REDDIT.

Original: The REDDIT datasets, introduced by Yanardag & Vishwanathan (2015), contain graphs constructed from online discussion threads on Reddit 19 .
Revised: The REDDIT datasets, introduced by Yanardag & Vishwanathan (2015), contain graphs constructed from online discussion threads on Reddit 20 .

Original: Vertices represent users and edges represent at least one response by one of the users to the other.
Revised: Vertices represent users and edges represent at least one response by one of the users to the other.

Original: The difference between REDDIT-BINARY,
Revised: The difference between REDDIT-BINARY,

Original: REDDIT-MULTI-5K, and REDDIT-MULTI-12K is how many subreddits have been crawled, and whether the classiﬁcation task that is usually associated with these datasets is binary or multi-class classiﬁcation.
Revised: REDDIT-MULTI-5K, and REDDIT-MULTI-12K is how many subreddits have been crawled, and whether the classification task that is usually associated with these datasets is binary or multi-class classification.

Original: We obtain the data from TU Dortmund’s data set collection 20 .
Revised: We obtain the data from TU Dortmund’s data set collection 21 .

Original: For each dataset, we sample 500 graphs for testing, and they contain 41-3 648 nodes.
Revised: For each dataset, we sample 500 graphs for testing, and they contain 41-3 648 nodes.

Original: as-Caida.
Revised: as-Caida.

Original: The as-Caida dataset of autonomous systems, introduced by Leskovec et al.
Revised: The as-Caida dataset of autonomous systems, introduced by Leskovec et al.

Original: (2005), consists of 122 graphs derived from a set of RouteViews BGP table snapshots.
Revised: (2005), consists of 122 graphs derived from a set of RouteViews BGP table snapshots.

Original: They contain 8 020 to 26 475 nodes.
Revised: They contain 8 020 to 26 475 nodes.

Original: We obtain the data from Stanford’s SNAP repository 21 .
Revised: We obtain the data from Stanford’s SNAP repository 22 .

Original: Citation.
Revised: Citation.

Original: The Citation dataset consists of the three citation network graphs Cora (McCallum et al., 2000), Citeseer (Giles et al., 1998) and PubMed (Sen et al., 2008).
Revised: The Citation dataset consists of the three citation network graphs Cora (McCallum et al., 2000), Citeseer (Giles et al., 1998) and PubMed (Sen et al., 2008).

Original: We obtain all data from
Revised: We obtain all data from

Original: NetworkRepository (Rossi & Ahmed, 2015).
Revised: NetworkRepository (Rossi & Ahmed, 2015).

Original: DBLP- Coauthorship
Revised: DBLP-Coauthorship.

Original: This dataset contains the [-com -DBLP 
-] (Yang & Leskovec, 2013) graph.
Revised: This dataset contains the [-com -DBLP 
-]com-DBLP 23 (Yang & Leskovec, 2013) graph.

Original: Vertices represent authors and an edge exists if two authors have published at least one paper together.
Revised: Vertices represent authors and an edge exists if two authors have published at least one paper together.

Original: Road Networks.
Revised: Road Networks.

Original: The Road Networks dataset contains two road networks of different sizes.
Revised: The Road Networks dataset contains two road networks of different sizes.

Original: The network roadNet-PA obtained from SNAP 23 represents the state of Pennsylvania (Leskovec et al., 2009).
Revised: The network roadNet-PA obtained from SNAP 24 represents the state of Pennsylvania (Leskovec et al., 2009).

Original: The other network maps the city of Berlin, Germany, and is extracted from OpenStreetMap (OpenStreetMap Contributors, 2017) using OSMnx (Boeing, 2017).
Revised: The other network maps the city of Berlin, Germany, and is extracted from OpenStreetMap (OpenStreetMap Contributors, 2017) using OSMnx (Boeing, 2017).

Original: Social Networks.
Revised: Social Networks.

Original: The Social Networks dataset contains of 6 social network graphs we gathered.
Revised: The Social Networks dataset contains of 6 social network graphs we gathered.

Original: Vertices represent users and edges represent some kind of relationship between them.
Revised: Vertices represent users and edges represent some kind of relationship between them.

Original: The exact kind of relationship depends on the graph.
Revised: The exact kind of relationship depends on the graph.

Original: The ego-Facebook 24 and ego-Gplus 25 graphs represent “friendships” between users (Leskovec & Mcauley, 2012).
Revised: The ego-Facebook 25 and ego-Gplus 26 graphs represent “friendships” between users (Leskovec & Mcauley, 2012).

Original: The bitcoin-otc 26 and bitcoin-alphagraphs represent Bitcoin’s web-of-trust network (Kumar et al., 2016; 2018).
Revised: The bitcoin-otc 27 and bitcoin-alphagraphs represent Bitcoin’s web-of-trust network (Kumar et al., 2016; 2018).

Original: The wiki-Vote 28 and wiki-RfA 29 graphs are derived from interactions in the governing process of Wikipedia (Leskovec et al., 2010; West et al., 2014).
Revised: The wiki-Vote 29 and wiki-RfA 30 graphs are derived from interactions in the governing process of Wikipedia (Leskovec et al., 2010; West et al., 2014).

Original: VC-Benchmark (VC-BM).
Revised: VC-Benchmark (VC-BM).

Original: A dataset consisting of 40 graphs of different sizes from 450 to 1nodes, on which ﬁnding the maximum independent set is synthetically made hard (Xu et al., 2007).
Revised: A dataset consisting of 40 graphs of different sizes from 450 to nodes

Original: Li et al.
Revised: , on which

Revised: finding the maximum independent set is synthetically made hard (

Original:  (2018) call this dataset BUAA-MC.
Revised: Xu et al., 2007). Li et al. (2018) call this dataset BUAA-MC, and sometimes it us also called BHOSLIB 31 .

Original: As the original download 30 is not available anymore, we use a GitHub mirror 31 . 
Revised: As the original download 32 is not available anymore, we use a GitHub mirror 33 . DIMACS.

Revised: Graphs used for the evaluation of the second DIMACS implementation challenge 34 in 1992, on which it is particular hard to solve the MIS problem.

Revised: The dataset consists of 37 graphs from 125 to 4 000 nodes.

Revised: We use the complementary benchmark 35 , as the original graphs were build for the C LIQUE problem.