omkarenator
commited on
Commit
•
aba6489
1
Parent(s):
2f4db88
add example citation, fix formatting
Browse files- common.py +4 -2
- curated.py +0 -0
common.py
CHANGED
@@ -299,7 +299,7 @@ global_div = Div(
|
|
299 |
style="margin-bottom: 5px",
|
300 |
),
|
301 |
Li("Normailzation Form C Discussion", style="margin-bottom: 5px"),
|
302 |
-
),
|
303 |
id="section1",
|
304 |
),
|
305 |
Section(
|
@@ -376,7 +376,9 @@ global_div = Div(
|
|
376 |
H3("Finding Connected Components using MapReduce"),
|
377 |
Img(src="images/findcc.svg", style="max-width: 100%;"),
|
378 |
P(
|
379 |
-
"The purpose of this step is to create a set of clusters of matching pairs. For example, a list of pairs (A, B), (B, C), (D, E) is merged into a list of components (A, B, C) and (D, E). Using a third-party library like NetworkX to find connected components would require all pairs to fit into the memory of a single machine, which is not feasible. Instead, we implemented a distributed connected component finder
|
|
|
|
|
380 |
),
|
381 |
P(
|
382 |
"We needed to partition the duplicate pairs generated in the third stage into three groups to reduce memory pressure on the final stage. We observed that the second stage itself generates partial components which have some overlap. These overlapping clusters cause some documents to appear in the delete set multiple times. However, our deletion code handled this overlap."
|
|
|
299 |
style="margin-bottom: 5px",
|
300 |
),
|
301 |
Li("Normailzation Form C Discussion", style="margin-bottom: 5px"),
|
302 |
+
),
|
303 |
id="section1",
|
304 |
),
|
305 |
Section(
|
|
|
376 |
H3("Finding Connected Components using MapReduce"),
|
377 |
Img(src="images/findcc.svg", style="max-width: 100%;"),
|
378 |
P(
|
379 |
+
"The purpose of this step is to create a set of clusters of matching pairs. For example, a list of pairs (A, B), (B, C), (D, E) is merged into a list of components (A, B, C) and (D, E). Using a third-party library like NetworkX to find connected components would require all pairs to fit into the memory of a single machine, which is not feasible. Instead, we implemented a distributed connected component finder",
|
380 |
+
D_cite(bibtex_key="6785473"),
|
381 |
+
"using the Dask framework, which can scale across multiple machines. The algorithm works by mapping edges by both the source and destination of pairs and reducing only edges where the source is greater than the destination. It performs successive iterations of this MapReduce computation until convergence, meaning the number of new edges produced becomes zero. In the end, every document in a cluster points to the smallest document within the cluster. Later, we compile a list of duplicate documents that need deletion and gather statistics about each component.",
|
382 |
),
|
383 |
P(
|
384 |
"We needed to partition the duplicate pairs generated in the third stage into three groups to reduce memory pressure on the final stage. We observed that the second stage itself generates partial components which have some overlap. These overlapping clusters cause some documents to appear in the delete set multiple times. However, our deletion code handled this overlap."
|
curated.py
CHANGED
The diff for this file is too large to render.
See raw diff
|
|