Spaces:

training-transformers-together
/

calc

Runtime error

App Files Files Community

robot-bengali-2 commited on Dec 5, 2021

Commit

92a6755

•

1 Parent(s): 2068f16

Write "Make Your Own" section

Browse files

Files changed (6) hide show

app.py +4 -4
charts.py +1 -1
st_helpers.py +1 -1
static/content_style.css +2 -2
static/header_style.css +2 -2
static/tabs.html +128 -34

app.py CHANGED Viewed

@@ -20,10 +20,10 @@ content_text(f"""
 There was a time when you could comfortably train state-of-the-art vision and language models at home on your workstation.
 The first convolutional neural net to beat ImageNet
 (<a target="_blank" href="https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf">AlexNet</a>)
-was trained for 5-6 days on two gamer-grade GPUs. Today's TOP-1 ImageNet model
 (<a target="_blank" href="https://arxiv.org/abs/2106.04803">CoAtNet</a>)
 takes 20,000 TPU-v3 days. And things are even worse in the NLP world: training
-<a target="_blank" href="https://arxiv.org/abs/2005.14165">GPT-3</a> on a top-tier server
 with 8x A100 would take decades.""")
 content_text(f"""
@@ -47,7 +47,7 @@ content_title("How do I join?")
 content_text("""
 That's easy. First, make sure you're logged in at Hugging Face. If you don't have an account, create one <b>TODO</b>.<br>
-<ul style="text-align: left; list-style-position: inside; margin-top: 12px; margin-left: -32px;">
     <li style="margin-top: 4px;">
         Join our organization on Hugging Face here: <b>TODO</b>. </li>
     <li style="margin-top: 4px;">
@@ -62,7 +62,7 @@ Please note that we currently limit the number of colab participants to <b>TODO<
 with other users. If there are too many active peers, take a look at alternative starter kits here <b>TODO</b>
 """)
-content_title("But how does it work?")
 content_text("<b> TODO </b> General Story That Weaves Together Three Tabs Below . Lorem ipsum dolor sit amet, "
              "consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim"
              " ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. "

 There was a time when you could comfortably train state-of-the-art vision and language models at home on your workstation.
 The first convolutional neural net to beat ImageNet
 (<a target="_blank" href="https://proceedings.neurips.cc/paper/2012/file/c399862d3b9d6b76c8436e924a68c45b-Paper.pdf">AlexNet</a>)
+was trained for 5-6 days on two gamer-grade GPUs. In contrast, today's TOP-1 ImageNet model
 (<a target="_blank" href="https://arxiv.org/abs/2106.04803">CoAtNet</a>)
 takes 20,000 TPU-v3 days. And things are even worse in the NLP world: training
+<a target="_blank" href="https://arxiv.org/abs/2005.14165">GPT&#8209;3</a> on a top-tier server
 with 8x A100 would take decades.""")
 content_text(f"""
 content_text("""
 That's easy. First, make sure you're logged in at Hugging Face. If you don't have an account, create one <b>TODO</b>.<br>
+<ul style="text-align: left; list-style-position: inside; margin-top: 12px; margin-left: -24px;">
     <li style="margin-top: 4px;">
         Join our organization on Hugging Face here: <b>TODO</b>. </li>
     <li style="margin-top: 4px;">
 with other users. If there are too many active peers, take a look at alternative starter kits here <b>TODO</b>
 """)
+content_title("How does it work?")
 content_text("<b> TODO </b> General Story That Weaves Together Three Tabs Below . Lorem ipsum dolor sit amet, "
              "consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim"
              " ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. "

charts.py CHANGED Viewed

@@ -12,7 +12,7 @@ def draw_current_progress():
         source, {
             "height": 200,
             "title": {
-                "text": "Training DALLE with volunteers. Updated every few minutes during NeurIPS.",
                 "dy": 6,
             },
             "$schema": "https://vega.github.io/schema/vega-lite/v5.json",

         source, {
             "height": 200,
             "title": {
+                "text": "Training DALL-E with volunteers (updated every few minutes during NeurIPS 2021)",
                 "dy": 6,
             },
             "$schema": "https://vega.github.io/schema/vega-lite/v5.json",

st_helpers.py CHANGED Viewed

@@ -30,7 +30,7 @@ def make_header():
 def make_tabs():
-    components.html(f"{tabs_html}", height=400, scrolling=True)
 def make_footer():

 def make_tabs():
+    components.html(f"{tabs_html}", height=1000, scrolling=True)
 def make_footer():

static/content_style.css CHANGED Viewed

@@ -60,11 +60,11 @@
     border-color:rgba(27,31,35,.35);
     background-image:linear-gradient(180deg, #f0f3f6, #e6ebf1 90%)
 }
-a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
-}

     border-color:rgba(27,31,35,.35);
     background-image:linear-gradient(180deg, #f0f3f6, #e6ebf1 90%)
 }
+/* a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
+} */

static/header_style.css CHANGED Viewed

@@ -100,14 +100,14 @@ canvas {
     border-color:rgba(27,31,35,.35);
     background-image:linear-gradient(180deg, #f0f3f6, #e6ebf1 90%)
 }
-a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
-}
 .tooltip {
   position: relative;
   display: inline-block;

     border-color:rgba(27,31,35,.35);
     background-image:linear-gradient(180deg, #f0f3f6, #e6ebf1 90%)
 }
+/* a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
+} */
 .tooltip {
   position: relative;
   display: inline-block;

static/tabs.html CHANGED Viewed

@@ -32,25 +32,37 @@
    font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,Arial,
                 sans-serif,Apple Color Emoji,Segoe UI Emoji;
 }
-a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
-}
   </style>
 </head>
 <body>
-<div id="tabs" style="width: 100%; margin:0 auto;">
     <div>
         <!-- Nav tabs -->
         <ul class="nav nav-tabs" role="tablist">
             <li role="presentation" class="active"><a href="#tab1" aria-controls="tab1" role="tab" data-toggle="tab">"Efficient Training"</a></li>
             <li role="presentation"><a href="#tab2" aria-controls="tab2" role="tab" data-toggle="tab">Security</a></li>
-            <li role="presentation"><a href="#tab3" aria-controls="tab3" role="tab" data-toggle="tab">Make Your Own (TBU)</a></li>
         </ul>
         <!-- Tab panes -->
@@ -86,44 +98,126 @@ a:visited {
                 The same can happen due to broken hardware or misconfiguration.
                 </p>
-                <p>
-                One possible defense is using <b>authentication</b> combined with <b>model checkpointing</b>.
-                In this case, participants should log in (e.g. with their Hugging Face account) to interact with the rest of the collaboration.
-                In turn, moderators can screen potential participants and add them to an allowlist.
-                If something goes wrong (e.g. if a participant sends invalid gradients and the model diverges),
-                the moderators remove them from the list and revert the model to the latest checkpoint unaffected by the attack.
-                </p>
-                <p><b>Spoiler: How to implement authentication in a decentralized system efficiently?</b></p>
                 <p>
-                Nice bonus: using this data, the moderators can acknowledge the personal contribution of each participant.
-                </p>
-                <p>
-                Another defense is replacing the naive averaging of the peers' gradients with an <b>aggregation technique robust to outliers</b>.
-                <a href="https://arxiv.org/abs/2012.10333">Karimireddy et al. (2020)</a>
-                suggested such a technique (named CenteredClip) and proved that it does not significantly affect the model's convergence.
-                </p>
-                <p><b>Spoiler: How does CenteredClip protect from outliers? (Interactive Demo)</b></p>
-                <p>
-                In our case, CenteredClip is useful but not enough to protect from malicious participants,
-                since it implies that the CenteredClip procedure itself is performed by a trusted server.
-                In contrast, in our decentralized system, all participants can aggregate a part of the gradients and we cannot assume all of them to be trusted.
-                </p>
-                <p>
-                Recently, <a href="https://arxiv.org/abs/2106.11257">Gorbunov et al. (2021)</a>
-                proposed a robust aggregation protocol for decentralized systems that does not require this assumption.
-                This protocol uses CenteredClip as a subroutine but is able to detect and ban participants who performed it incorrectly.
-                </p>
             </div>
             <div role="tabpanel" class="tab-pane" id="tab3">
-                <span class="padded faded text">
-                <b> TODO 123</b> Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
-                </span>
             </div>
         </div>

    font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Helvetica,Arial,
                 sans-serif,Apple Color Emoji,Segoe UI Emoji;
 }
+.tab-group {
+    font-size: 16px;
+}
+.tab-content {
+    margin-top: 16px;
+}
+ul > li {
+    margin: 3px 0;
+}
+ol > li {
+    margin: 5px 0;
+}
+/* a:link {
     color: #00194a;
     text-decoration: none;
 }
 a:visited {
     color: #3f004a;
     text-decoration: none;
+} */
   </style>
 </head>
 <body>
+<div class="tab-group" style="width: 100%; margin:0 auto;">
     <div>
         <!-- Nav tabs -->
         <ul class="nav nav-tabs" role="tablist">
             <li role="presentation" class="active"><a href="#tab1" aria-controls="tab1" role="tab" data-toggle="tab">"Efficient Training"</a></li>
             <li role="presentation"><a href="#tab2" aria-controls="tab2" role="tab" data-toggle="tab">Security</a></li>
+            <li role="presentation"><a href="#tab3" aria-controls="tab3" role="tab" data-toggle="tab">Make Your Own</a></li>
         </ul>
         <!-- Tab panes -->
                 The same can happen due to broken hardware or misconfiguration.
                 </p>
+                <ul>
+                <li>
+                    <p>
+                    One possible defense is using <b>authentication</b> combined with <b>model checkpointing</b>.
+                    In this case, participants should log in (e.g. with their Hugging Face account) to interact with the rest of the collaboration.
+                    In turn, moderators can screen potential participants and add them to an allowlist.
+                    If something goes wrong (e.g. if a participant sends invalid gradients and the model diverges),
+                    the moderators remove them from the list and revert the model to the latest checkpoint unaffected by the attack.
+                    </p>
+                    <p><b>Spoiler: How to implement authentication in a decentralized system efficiently?</b></p>
+                    <p>
+                    Nice bonus: using this data, the moderators can acknowledge the personal contribution of each participant.
+                    </p>
+                </li>
+                <li>
                 <p>
+                    Another defense is replacing the naive averaging of the peers' gradients with an <b>aggregation technique robust to outliers</b>.
+                    <a href="https://arxiv.org/abs/2012.10333">Karimireddy et al. (2020)</a>
+                    suggested such a technique (named CenteredClip) and proved that it does not significantly affect the model's convergence.
+                    </p>
+                    <p><b>Spoiler: How does CenteredClip protect from outliers? (Interactive Demo)</b></p>
+                    <p>
+                    In our case, CenteredClip is useful but not enough to protect from malicious participants,
+                    since it implies that the CenteredClip procedure itself is performed by a trusted server.
+                    In contrast, in our decentralized system, all participants can aggregate a part of the gradients and we cannot assume all of them to be trusted.
+                    </p>
+                    <p>
+                    Recently, <a href="https://arxiv.org/abs/2106.11257">Gorbunov et al. (2021)</a>
+                    proposed a robust aggregation protocol for decentralized systems that does not require this assumption.
+                    This protocol uses CenteredClip as a subroutine but is able to detect and ban participants who performed it incorrectly.
+                    </p>
+                </li>
+                </ul>
             </div>
             <div role="tabpanel" class="tab-pane" id="tab3">
+                <ol>
+                    <li>
+                        Set up dataset streaming:
+                        <ul>
+                            <li>
+                                <a href="https://huggingface.co/docs/datasets/share_dataset.html">Upload</a> your dataset to Hugging Face Hub
+                                in a streaming-friendly format (<a href="https://huggingface.co/datasets/laion/laion_100m_vqgan_f8">example</a>).
+                            </li>
+                            <li>Set up dataset streaming (see the "Efficient Training" section).</li>
+                        </ul>
+                    </li>
+                    <li>
+                        Write code of training peers (<a href="https://github.com/learning-at-home/dalle-hivemind/blob/main/run_trainer.py">example</a>):
+                        <ul>
+                            <li>Implement your model, set up dataset streaming, and write the training loop.</li>
+                            <li>
+                                Get familiar with the hivemind library
+                                (e.g., via the <a href="https://learning-at-home.readthedocs.io/en/latest/user/quickstart.html">quickstart</a>).
+                            </li>
+                            <li>
+                                In the training loop, wrap up your PyTorch optimizer with
+                                <a href="https://learning-at-home.readthedocs.io/en/latest/modules/optim.html#hivemind.optim.experimental.optimizer.Optimizer">hivemind.Optimizer</a>
+                                (<a href="https://github.com/learning-at-home/dalle-hivemind/blob/main/task.py#L121">example</a>).
+                            </li>
+                        </ul>
+                    </li>
+                    <li>
+                        <b>(optional)</b> Write code of auxiliary peers (<a href="https://github.com/learning-at-home/dalle-hivemind/blob/main/run_aux_peer.py">example</a>):
+                        <ul>
+                            <li>
+                                It is convenient to create a special kind of peers responsible for
+                                logging loss values and metrics (e.g. to <a href="https://wandb.ai/">Weights & Biases</a>)
+                                and uploading model checkpoints (e.g. to Hugging Face Hub).
+                            </li>
+                            <li>
+                                Such peers don't need to calculate gradients and may be run on cheap machines without GPUs.
+                            </li>
+                            <li>
+                                They can serve as a convenient entry point to
+                                <a href="https://learning-at-home.readthedocs.io/en/latest/modules/dht.html">hivemind.DHT</a>
+                                (i.e., their address can be specified as <code>initial_peers</code>).
+                            </li>
+                            <li>
+                                It is useful to fix their address by providing <code>host_maddrs</code> and <code>identity_path</code>
+                                arguments to <code>hivemind.DHT</code>
+                                (these are forwarded to the underlying <a href="https://libp2p.io/">libp2p</a> daemon).
+                            </li>
+                        </ul>
+                    </li>
+                    <li>
+                        <b>(optional)</b> Make it easier for other people to join:
+                        <ul>
+                            <li>
+                                Create notebooks for free GPU providers (Google Colab, Kaggle, AWS SageMaker, etc.).
+                                People may run them online and/or download and run them on their own hardware.
+                            </li>
+                            <li>
+                                <a href="https://huggingface.co/organizations/new">Create</a> a Hugging Face organization
+                                with all resources related to the training
+                                (dataset, model, inference demo, links to a dashboard with loss values and metrics, etc.).
+                                Look at <a href="https://huggingface.co/training-transformers-together">ours</a> as an example.
+                            </li>
+                            <li>
+                                Set up an authentication system (see the "Security" section).
+                                For example, you can ask people to join your organization with their Hugging Face accounts
+                                (Hugging Face allows to share a link for joining or manually approve new participants).
+                                This allows you to screen participants,
+                                acknowledge their contributions (e.g., make a leaderboard), and
+                                ban accounts who behave maliciously.
+                            </li>
+                            <li>
+                                Set up an inference demo for your model (e.g., using <a href="https://huggingface.co/spaces">Spaces</a>) or
+                                a script that periodically uploads the inference results to show the training progress.
+                            </li>
+                        </ul>
+                    </li>
+                </ol>
+                <p>
+                    <b>Got confused?</b> Feel free to ask any questions at our <a href="https://discord.gg/uGugx9zYvN">Discord</a>!
+                </p>
             </div>
         </div>