added readme

README.md

@@ -1,19 +1,25 @@
+---
+license: gpl-3.0
+title: Kudasai
+sdk: gradio
+emoji: 🈷️
+python_version: 3.10.0
+app_file: webgui.py
+colorFrom: gray
+colorTo: gray
+short_description: Japanese-English preprocessor with automated translation.
+pinned: true
+---
+
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 **Table of Contents**
 
 - [**Notes**](#notes)
-- [**Dependencies**](#dependencies)
-- [**Quick Start**](#quick-start)
-- [**Command Line Interface (CLI)**](#command-line-interface-cli)
-  - [Usage](#usage)
-    - [Preprocess Mode](#preprocess-mode)
-    - [Translate Mode](#translate-mode)
-  - [Additional Notes](#additional-notes)
-- [**Preprocessing**](#preprocessing)
+- [**General Usage**](#general-usage)
+- [**Indexing and Preprocessing**](#indexing-and-preprocessing)
 - [**Translator**](#translator)
 - [**Translator Settings**](#translator-settings)
 - [**Web GUI**](#web-gui)
-- [**Hugging Face**](#hugging-face)
 - [**License**](#license)
 - [**Contact**](#contact)
 - [**Acknowledgements**](#acknowledgements)
@@ -21,11 +27,7 @@
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 ## **Notes**<a name="notes"></a>
 
-Windows 10 and Linux Mint are the only tested operating systems, feel free to test on other operating systems and report back to me. I will do my best to fix any issues that arise.
-
-To see the README for the Hugging Face hosted version of Kudasai, please see [here](https://github.com/Bikatr7/Kudasai/blob/main/lib/gui/HUGGING_FACE_README.md). Further WebGUI documentation can be found there as well.
-
-Python version: 3.10+
+This readme is for the Hugging Space instance of Kudasai's WebGUI and the WebGUI itself, to run Kudasai locally or see any info on the project, please see the [GitHub Page](https://github.com/Bikatr7/Kudasai).
 
 Streamlining Japanese-English Translation with Advanced Preprocessing and Integrated Translation Technologies.
 
@@ -33,170 +35,60 @@ Preprocessor and Translation logic is sourced from external packages, which I al
 
 Kudasai has a public trello board, you can find it [here](https://trello.com/b/Wsuwr24S/kudasai) to see what I'm working on and what's coming up.
 
+The WebGUI on huggingface does not save anything through runs, so you will need to download the output files or copy the text out of the webgui. API keys are not saved, and the output folder is overwritten every time you run it. Archives deleted every run as well.
+
 Kudasai is proud to have been a Backdrop Build v3 Finalist:
 https://backdropbuild.com/builds/v3/kudasai
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-## **Dependencies**<a name="dependencies"></a>
-
-backoff==2.2.1
-
-gradio==4.20.0
-
-kairyou==1.5.0
-
-easytl==0.3.3
-
-or see requirements.txt
-
-Also requires spacy's ja_core_news_lg model, which can be installed via the following command:
-
-```bash
-python -m spacy download ja_core_news_lg
-```
-
-or on Linux
-
-```bash
-python3 -m spacy download ja_core_news_lg
-```
-
----------------------------------------------------------------------------------------------------------------------------------------------------
-## **Quick Start**<a name="quick-start"></a>
-
-Windows is assumed for the rest of this README, but the process should be similar for Linux. This is for the console version, for something less linear, see the [Web GUI](#webgui) section.
-
-Due to PyPi limitations, you need to install SpaCy's JP Model, which can not be included automatically due to it being a direct dependency link which PyPi does not support. Make sure you do this after installing the requirements.txt file as it requires Kairyou/SpaCy to be installed first.
-
-```bash
-python -m spacy download ja_core_news_lg
-```
-
-Simply run Kudasai.py, enter a txt file path to the text you wish to preprocess/translate, and then insert a replacement json file path if you wish to use one. If you do not wish to use a replacement json file, you can simply input a blank space and Kudasai will skip preprocessing and go straight to translation.
-
-Kudasai will offer to index the text, which is useful for finding new names to add to the replacement json file. This is optional and can be skipped.
-
-After preprocessing is completed (if triggered), you will be prompted to choose a translation method.
-
-You can choose between OpenAI, Gemini, and DeepL. Each have their own pros and cons, but OpenAI is the recommended translation method. DeepL and Gemini currently offer free versions, but all three require an api key, you will be prompted to enter this key when you choose to run the translation module.
-
-Next, Kudasai will ask you to confirm it's settings. This can be overwhelming, but you can simply enter 1 to confirm and use the default settings. If you wish to change them, you can do so here.
-
-See the [**Translator Settings**](#translator-settings) section for more information on Kudasai's Translation settings, but default should run fine. Inside the demo folder is a copy of the settings I use to translate COTE should you wish to use them. There is also a demo txt file in the demo folder that you can use to test Kudasai.
-
-Kudasai will then ask if you want to change your api key, simply enter 2 for now. 
-
-Next Kudasai will display an estimated cost of translation, this is based on the number of tokens in the preprocessed text as determined by tiktoken for OpenAI, by Google for Gemini, and by DeepL for DeepL. Kudasai will then prompt for confirmation, if this is fine, enter 1 to run the translation module otherwise 2 to exit.
-
-Kudasai will then run the translation module and output the translated text and other logs to the output folder in the same directory as Kudasai.py.
-
-These files are:
-
-    "debug_log.txt" : A log of crucial information that occurred during Kudasai's run, useful for debugging or reporting issues as well as seeing what was done.
-
-    "error_log.txt" : A log of errors that occurred during Kudasai's run if any, useful for debugging or reporting issues.
 
-    "je_check_text.txt" : A log of the Japanese and English sentences that were paired together, useful for checking the accuracy of the translation and further editing of a machine translation.
+## **General Usage**<a name="general-usage"></a>
 
-    "preprocessed_text.txt" : The preprocessed text, the text output by Kairyou (preprocessor).
+Kudasai's WebGUI is pretty easy to understand for the general usage, most incorrect actions will be caught by the system and a message will be displayed to the user on how to correct it.
 
-    "preprocessing_results.txt" : A log of the results of the preprocessing, shows what was replaced and how many times.
+Normally, Kudasai would save files to the local system, but on Hugging Face's servers, this is not possible. Instead, you'll have to click the 'Save As' button to download the files to your local system.
 
-    "translated_text.txt" : The translated text, the text output by Kaiseki or Kijiku.
+Or you can click the copy button on the top right of textbox modals to copy the text to your clipboard.
 
-Old runs are stored in the archive folder in output as well.
-
-If you have any questions, comments, or concerns, please feel free to open an issue.
+For further details, see below chapters.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
-## **Command Line Interface (CLI)**<a name="cli"></a>
-
-Kudasai provides a Command Line Interface (CLI) for preprocessing and translating text files. This section details how to use the CLI, including the required and optional arguments for each mode.
-
-### Usage
-
-The CLI supports two modes: `preprocess` and `translate`. Each mode requires specific arguments to function properly.
-
-#### Preprocess Mode
-
-The `preprocess` mode preprocesses the text file using the provided replacement JSON file. 
-
-**Command Structure:**
-
-```bash
-python path_to_kudasai.py preprocess <input_file> <replacement_json> [<knowledge_base>]
-```
 
-**Required Arguments:**
-- `<input_file>`: Path to the text file to preprocess.
-- `<replacement_json>`: Path to the replacement JSON file.
+## **Indexing and Preprocessing**<a name="kairyou"></a>
 
-**Optional Arguments:**
-- `<knowledge_base>`: Path to the knowledge base file (directory, file, or text).
+This section can be skipped if you're only interested in translation or do not know what indexing or preprocessing is.
 
-**Example:**
+Indexing is not for everyone, only use it if you have a large amount of previous text and want to flag new names. It can be a very slow and long process, especially on Hugging Face's servers. It's recommended to use a local version of Kudasai for this process.
 
-```bash
-python C:\\path\\to\\kudasai.py preprocess "C:\\path\\to\\input_file.txt" "C:\\path\\to\\replacement_json.json" "C:\\path\\to\\knowledge_base"
-```
+You'll need a txt file or some text to index. You'll also need a knowledge base, this can either be a single txt file or a directory of them, as well as a replacements json. Either Kudasai or Fukuin Type works. See [this](https://github.com/Bikatr7/Kairyou?tab=readme-ov-file#kairyou) for further details on replacement jsons.
 
-#### Translate Mode
+Please do indexing before preprocessing, output is neater that way.
 
-The `translate` mode translates the text file using the specified translation method.
+For Preprocessing, you'll need a txt file or some text to preprocess. You'll also need a replacements json. Either Kudasai or Fukuin Type works like with indexing.
 
-**Command Structure:**
+For both, text is put in the textbox modals, with the output text being in the first field, and results being in the second field. 
 
-```bash
-python path_to_kudasai.py translate <input_file> <translation_method> [<translation_settings_json>] [<api_key>]
-```
-
-**Required Arguments:**
-- `<input_file>`: Path to the text file to translate.
-
-**Optional Arguments:**
-- `<translation_method>`: Translation method to use (`'deepl'`, `'openai'`, or `'gemini'`). Defaults to `'deepl'`.
-- `<translation_settings_json>`: Path to the translation settings JSON file (overrides current settings).
-- `<api_key>`: API key for the translation service. If not provided, it will use the in the settings directory or prompt for it if that's not found.
-
-**Example:**
-
-```bash
-python C:\\path\\to\\kudasai.py translate "C:\\path\\to\\input_file.txt" gemini "C:\\path\\to\\translation_settings.json" "YOUR_API_KEY"
-```
-
-### Additional Notes
-- All arguments should be enclosed in double quotes if they contain spaces. Double quotes are optional and will be stripped. Single quotes are not allowed.
+They both have a debug field, but neither module really uses it.
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 
-## **Preprocessing**<a name="preprocessing"></a>
-
-Preprocessing is the act of preparing text for translation by replacing certain words or phrases with their translated counterparts. 
-
-Kudasai uses Kairyou for preprocessing, which is a powerful preprocessor that can replace text in a text file based on a json file. This is useful for replacing names, places, and other things that may not translate well or to simply speed up the translation process.
-
-You can run the preprocessor by using the CLI or simply running kudasai.py as instructed in the [Quick Start](#quick-start) section.
-
-Many replacement json files are included in the jsons folder, you can also make your own if you wish provided it follows the same format. See an example below
-Kudasai/Kairyou works with both Kudasai and Fukuin Json's, the below is a Kudasai type json.
-
-![Example JSON](https://i.imgur.com/u3FnUia.jpg)
+## **Translator**<a name="translator"></a>
 
----------------------------------------------------------------------------------------------------------------------------------------------------
+Kudasai supports 3 different translation methods at the moment, OpenAI's GPT, Google's Gemini, and DeepL. 
 
-## **Translator**<a name="translator"></a>
+For OpenAI, you'll need an API key, you can get one [here](https://platform.openai.com/docs/api-reference/authentication). This is a paid service with no free tier.
 
-Kudasai uses EasyTL for translation, which is a versatile translation library that uses several translation APIs to translate text.
+For Gemini, you'll also need an API key, you can get one [here](https://ai.google.dev/tutorials/setup). Gemini is free to use under a certain limit, 2 RPM for 1.5 and 15 RPM for 1.0.
 
-Kudasai currently supports OpenAI, Gemini, and DeepL for translation. OpenAI is the recommended translation method, but DeepL and Gemini are also good alternatives.
+For DeepL, you'll need an API key too, you can get one [here](https://www.deepl.com/pro#developer). DeepL is also a paid service but is free under 500k characters a month.
 
-You can run the translator by running kudasai.py as instructed in the [Quick Start](#quick-start) section.
+I'd recommend using GPT for most things, as it's generally better at translation.
 
-Note that you need an API key for OpenAI, Gemini, and DeepL. You will be prompted to enter this key when you choose to run the translation module.
+Mostly straightforward, choose your translation method, fill in your API key, and select your text. You'll also need to add your settings file if on HuggingFace if you want to tune the output, but the default is generally fine.
 
-The translator has a lot of settings, simply using the default settings is fine or the one provided in the demo folder. You can also change these manually when confirming your settings, as well as loading a custom json as your settings by pressing c at this window, with the settings in the script directory.
+You can calculate costs here or just translate. Output will show in the appropriate fields.
 
-The settings are fairly complex, see the below section [Translator Settings](#translator-settings) for more information.
+For further details on the settings file, see [here](#translation-with-llms-settings).
 
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 
@@ -285,14 +177,8 @@ The settings are fairly complex, see the below section [Translator Settings](#tr
 
 ## **Web GUI**<a name="webgui"></a>
 
-Kudasai also offers a Web GUI. It has all the main functionality of the program but in an easier and non-linear way.
-
-To run the Web GUI, simply run webgui.py which is in the same directory as kudasai.py
-
 Below are some images of the Web GUI.
 
-Detailed Documentation for this can be found on the Hugging Face hosted version of Kudasai [here](https://huggingface.co/spaces/Bikatr7/Kudasai/blob/main/README.md).
-
 Name Indexing | Kairyou:
 ![Name Indexing Screen | Kairyou](https://i.imgur.com/QCPqjrw.jpeg)
 
@@ -311,16 +197,6 @@ Translation Settings Page 2:
 Logging Page:
 ![Logging Page](https://i.imgur.com/vDPCUQC.jpeg)
 
----------------------------------------------------------------------------------------------------------------------------------------------------
-
-## **Hugging Face**<a name="huggingface"></a>
-
-For those who are interested, or simply cannot run Kudasai locally, a instance of Kudasai's WebGUI is hosted on Hugging Face's servers. You can find it [here](https://huggingface.co/spaces/Bikatr7/Kudasai).
-
-It's a bit slower than running it locally, but it's a good alternative for those who cannot run it locally. The webgui on huggingface does not save anything through runs, so you will need to download the output files or copy the text out of the webgui. API keys are not saved, and the output folder is overwritten every time it loads. Archives deleted every run as well.
-
-To see the README for the Hugging Face hosted version of Kudasai, please see [here](https://huggingface.co/spaces/Bikatr7/Kudasai/blob/main/README.md).
-
 ---------------------------------------------------------------------------------------------------------------------------------------------------
 ## **License**<a name="license"></a>