added force push capability

README.md

@@ -16,14 +16,19 @@ A basic example of dynamic adversarial data collection with a Gradio app.
 
 *Setting up the Space*
 1. Clone this repo and deploy it on your own Hugging Face space.
-2. Add one of your Hugging Face tokens to the secrets for your space, with the
-   name `HF_TOKEN`. Now, create an empty Hugging Face dataset on the hub. Put
-   the url of this dataset in the secrets for your space, with the name
-   `DATASET_REPO_URL`. It can be a private or public dataset. When you run this
-   space on mturk and when people visit your space on huggingface.co, the app
-   will use your token to automatically store new HITs in your dataset. NOTE:
-   if you push something to your dataset manually, you need to reboot your space
-   or it could get merge conflicts when trying to push HIT data.
+2. Add the following secrets to your space:
+   - `HF_TOKEN`: One of your Hugging Face tokens.
+   - `DATASET_REPO_URL`: The url to an empty dataset that you created the hub. It
+    can be a private or public dataset.
+   - `FORCE_PUSH`: "yes"
+   When you run this space on mturk and when people visit your space on
+   huggingface.co, the app will use your token to automatically store new HITs
+   in your dataset. Setting `FORCE_PUSH` to "yes" ensures that your repo will
+   force push changes to the dataset during data collection. Otherwise,
+   accidental manual changes to your dataset could result in your space gettin
+   merge conflicts as it automatically tries to push the dataset to the hub. For
+   local development, add these three keys to a `.env` file, and consider setting
+   `FORCE_PUSH` to "no".
 
 *Running Data Collection*
 1. On your local repo that you pulled, create a copy of `config.py.example`,
@@ -38,11 +43,9 @@ Now, you should be watching hits come into your Hugging Face dataset
 automatically!
 
 *Tips and Tricks*
-- If you are developing and running this space locally to test it out, try
-deleting the data directory that the app clones before running the app again.
-Otherwise, the app could get merge conflicts when storing new HITs on the hub.
-When you redeploy your app on Hugging Face spaces, the data directory is deleted
-automatically.
+- Use caution while doing local development of your space and
+simultaneously running it on mturk. Consider setting `FORCE_PUSH` to "no" in
+your local `.env` file.
 - huggingface spaces have limited computational resources and memory. If you
 run too many HITs and/or assignments at once, then you could encounter issues.
 You could also encounter issues if you are trying to create a dataset that is

app.py

@@ -12,11 +12,13 @@ from dotenv import load_dotenv
 from pathlib import Path
 import json
 from filelock import FileLock
+from utils import force_git_push
 
 # These variables are for storing the mturk HITs in a Hugging Face dataset.
 if Path(".env").is_file():
     load_dotenv(".env")
 DATASET_REPO_URL = os.getenv("DATASET_REPO_URL")
+FORCE_PUSH = os.getenv("FORCE_PUSH")
 HF_TOKEN = os.getenv("HF_TOKEN")
 DATA_FILENAME = "data.jsonl"
 DATA_FILE = os.path.join("data", DATA_FILENAME)
@@ -106,7 +108,16 @@ with demo:
                 json_data_with_assignment_id =\
                     [json.dumps(dict({"assignmentId": state["assignmentId"]}, **datum)) for datum in state["data"]]
                 jsonlfile.write("\n".join(json_data_with_assignment_id) + "\n")
-            repo.push_to_hub()
+
+            if repo.is_repo_clean():
+                logger.info("Repo currently clean.  Ignoring push_to_hub")
+                return None
+            repo.git_add(auto_lfs_track=True)
+            repo.git_commit("Auto commit by space")
+            if FORCE_PUSH == "yes":
+                force_git_push(repo)
+            else:
+                repo.git_push()
         finally:
             lock.release()
         return state

utils.py

@@ -0,0 +1,39 @@
+import subprocess
+from huggingface_hub.repository import _lfs_log_progress
+
+def force_git_push(
+        repo,
+    ):
+        """
+        force a simple git push
+        Blocking. Will return url to commit on remote
+        repo.
+        """
+        command = "git push --force"
+
+        try:
+            with _lfs_log_progress():
+                process = subprocess.Popen(
+                    command.split(),
+                    stderr=subprocess.PIPE,
+                    stdout=subprocess.PIPE,
+                    encoding="utf-8",
+                    cwd=repo.local_dir,
+                )
+
+                stdout, stderr = process.communicate()
+                return_code = process.poll()
+                process.kill()
+
+                if len(stderr):
+                    print(stderr)
+
+                if return_code:
+                    raise subprocess.CalledProcessError(
+                        return_code, process.args, output=stdout, stderr=stderr
+                    )
+
+        except subprocess.CalledProcessError as exc:
+            raise EnvironmentError(exc.stderr)
+
+        return repo.git_head_commit_url()