created outline of pipeline in log

project_log.ipynb

@@ -68,6 +68,27 @@
     "\n",
     "\n"
    ]
+  },
+  {
+   "attachments": {},
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 07/02/2023\n",
+    "\n",
+    "-Read medium article about using config files to set up highly modular data analysis pipelines.\n",
+    "-Interested in setting this up here\n",
+    "\n",
+    "#### Outline of pipeline architecture\n",
+    "\n",
+    "1. Load dataset \n",
+    "    - option to load from file or from querying arxiv directly\n",
+    "    - stores raw title and abstract, id #s, msc_tags as english, and categories (OHE) as a separate dataframe\n",
+    "2. Load embeddings\n",
+    "    - option to load from file or generate using sentence transformers directly.\n",
+    "    - any data cleaning procedures will occur in the pipeline here\n",
+    "3. Plug into topic model(s)"
+   ]
   }
  ],
  "metadata": {