Given a user-specified minimum degree threshold \gamma , a%DIFDELCMD < {%%% \gamma -quasi-clique is a subgraph  where each vertex  connects to at least \gamma fraction of the other vertices . Mining maximal\lceil \rceil  quasi-cliques is notoriously expensive with the state-of-the-art algorithm scaling only to small graphs with thousands of vertices.This has hampered its popularity in real applications involving big graphs. We developed a task-based system called G-thinker for massively parallel graph mining, which is the first graph mining system that scales with the number of CPU cores. G-thinker provides a unique opportunity to scale the compute-intensive quasi-clique mining . This paperdesigns parallel algorithms for mining maximal quasi-cliques on G-thinker that scale to big graphs. Our algorithms follow the idea of divide and conquer which partitions the problem of mining a big graph into tasks that mine smaller subgraphs . However, we find that a direct application of G-thinker is insufficient due to the drastically different running time of different tasks that violates the original design assumption of G-thinker , requiring a system URLe. We also observe that the running time of a task is highly unpredictable solely from the features extracted from its subgraph, leading to difficulty in pinpoint expensive tasks to decompose for concurrent processing, and size-threshold based partitioning under-partitions some tasks but over-partitions others, leading to bad load balancing and enormous task partitioning overheads. We address this issue by proposing a novel time-delayed divide-and-conquer strategy that strikes a balance between the workloads spent on actual mining and the cost of balancing the workloads. Extensive experiments verify that our G-thinker algorithm scales perfectly with the number of CPU cores, achieving over 300x speedup when running on a graph with over 1M vertices in a small cluster.
Given a user-specified minimum degree threshold %DIFDELCMD < {%%% \gamma, a \gamma -quasi-clique is a subgraph g=(V_g,E_g) where each vertex v\in V_g connects to at least \gamma fraction of the other vertices (i.e.,\lceil \gamma\cdot(|V_g|-1)\rceil vertices) in g. Quasi-clique is one of the most natural definitions for dense structures useful in finding communities in social networks and discovering significant biomolecule structures and pathways. However, mining maximal quasi-cliques is notoriously expensive. In this paper, we design parallel algorithms for mining maximal quasi-cliques on G-thinker , a recent distributed framework targeting divide-and-conquer graph mining algorithms that decomposes the mining into compute-intensive tasks to fully utilize CPU cores . However, we found that directly using G-thinker results in the straggler problem due to (i) the drastic load imbalance among different tasks and (ii) the difficulty of predicting the task running time and the time growth with task-subgraph size. We address these challenges by redesigning G-thinker 's execution engine to prioritize long-running tasks for mining, and by utilizing a novel timeout strategy to effectively decompose the mining workloads of long-running tasks to improve load balancing. While this system redesign applies to many other expensive dense subgraph mining problems, this paper verifies the idea by adapting the state-of-the-art quasi-clique algorithm, Quick, to our redesigned G-thinker. We improve Quick by integrating new pruning rules, and fixing some missed boundary cases that could lead to missed results. Extensive experiments verify that our new solution scales well with the number of CPU cores, achieving 201\times runtime speedup when mining a graph with 3.77M vertices and 16.5M edges in a 16-node cluster.
D
\gamma
D
, a
R
\gamma
\gamma, a \gamma
A
g=(V_g,E_g)
A
v\in V_g
R
\gamma
\gamma
R
. Mining maximal
(i.e.,
A
\gamma\cdot(|V_g|-1)
A
vertices) in g. Quasi-clique is one of the most natural definitions for dense structures useful in finding communities in social networks and discovering significant biomolecule structures and pathways. However, mining maximal
R
is notoriously expensive with the state-of-the-art algorithm scaling only to small graphs with thousands of vertices.This has hampered its popularity in real applications involving big graphs. We developed a task-based system called G-thinker for massively parallel graph mining, which is the first graph mining system that scales with the number of CPU cores. G-thinker provides a unique opportunity to scale the compute-intensive quasi-clique mining . This paperdesigns
is notoriously expensive. In this paper, we design
R
that scale to big graphs. Our algorithms follow the idea of divide and conquer which partitions the problem of mining a big graph into tasks that mine smaller subgraphs
, a recent distributed framework targeting divide-and-conquer graph mining algorithms that decomposes the mining into compute-intensive tasks to fully utilize CPU cores
R
find that a direct application of
found that directly using
R
is insufficient due to the drastically different running time of different tasks that violates the original design assumption of
results in the straggler problem due to (i) the drastic load imbalance among different tasks and (ii) the difficulty of predicting the task running time and the time growth with task-subgraph size. We address these challenges by redesigning
R
, requiring a system URLe. We also observe that the running time of a task is highly unpredictable solely from the features extracted from its subgraph, leading to difficulty in pinpoint expensive tasks to decompose for concurrent processing, and size-threshold based partitioning under-partitions some tasks but over-partitions others, leading to bad load balancing and enormous task partitioning overheads. We address this issue by proposing a novel time-delayed divide-and-conquer strategy that strikes a balance between the workloads spent on actual mining and the cost of balancing the workloads.
's execution engine to prioritize long-running tasks for mining, and by utilizing a novel timeout strategy to effectively decompose the mining workloads of long-running tasks to improve load balancing. While this system redesign applies to many other expensive dense subgraph mining problems, this paper verifies the idea by adapting the state-of-the-art quasi-clique algorithm, Quick, to our redesigned G-thinker. We improve Quick by integrating new pruning rules, and fixing some missed boundary cases that could lead to missed results.
R
G-thinker algorithm scales perfectly
new solution scales well
R
over 300x speedup when running on
201\times runtime speedup when mining
R
over 1M vertices in a small
3.77M vertices and 16.5M edges in a 16-node